Update workshop/README.md, workshop/HandsOn.md, README.md,...

Update workshop/README.md, workshop/HandsOn.md, README.md, workshop/doc/Singularity-Pull-Example.gif, workshop/MPI/mpitest.c, definition-files/MPI/Singularity.ubuntu-18.04-OMPI-gcc
Deleted workshop/Introduction.md

README.md

@@ -11,11 +11,25 @@ Based on [naked-singularity](https://github.com/mkandes/naked-singularity). Than
 
 Install Singularity on your Linux desktop, laptop, or virtual machine. 
 
+There is a general install script that installs version 3.5.3 (can be easily changed to 3.8 or 3.9) from source, but there are also binary packages build for modern OS (Server OS).
+e.g. Red Hat, Arch, ...
+
+[Installing Singularity Documentation](https://sylabs.io/guides/3.6/admin-guide/installation.html#)
+
 ```bash
+# Centos / Red Hat
+sudo yum update -y && \
+    sudo yum install -y epel-release && \
+    sudo yum update -y && \
+    sudo yum install -y singularity
+# Arch / Manjaro
+pamac install singularity-container
+
+# Manual using this script (builds from source)
 sudo ./install-singularity.sh install
 ```
 
-## Build a Singularity container from a definition file
+## Build your first Singularity container from a definition file
 
 Build an Ubuntu Singularity container from one of the definition files
 available in this repository.
@@ -23,3 +37,25 @@ available in this repository.
 ```bash
 sudo singularity build ubuntu.sif definition-files/ubuntu/Singularity.ubuntu-18.04
 ```
+
+## Build your first Singularity container from a DockerHub
+
+Build an Ubuntu Singularity container from the [DockerHub repository](https://hub.docker.com/_/ubuntu)
+
+```bash
+# Using Ubuntu 18.04
+sudo singularity build ubuntu.sif docker://ubuntu:18.04
+# Using Ubuntu 20.04
+sudo singularity build ubuntu.sif docker://ubuntu:20.04
+```
+
+## Pull your first Singularity container from IMGW Sylab Cloud
+
+We try to build custom Singularity containers with signatures to help keeping track of changes and interoperability.
+
+```bash
+singularity pull ubuntu.sif library://mblaschek/imgw/ubuntu:20.04 
+```
+
+**Note:** With the *pull* command you do **not** need root access. Everybody can pull a container and execute it as ones own process. 
+

definition-files/MPI/Singularity.ubuntu-18.04-OMPI-gcc

+Bootstrap: library
+From: mblaschek/imgw/ubuntu:18.04
+
+%files
+    mpitest.c /opt
+
+%environment
+    export OMPI_DIR=/opt/ompi
+    export SINGULARITY_OMPI_DIR=$OMPI_DIR
+    export SINGULARITYENV_APPEND_PATH=$OMPI_DIR/bin
+    export SINGULAIRTYENV_APPEND_LD_LIBRARY_PATH=$OMPI_DIR/lib
+
+%post
+    echo "Installing required packages..."
+    apt-get update && apt-get install -y wget git bash gcc gfortran g++ make file
+
+    echo "Installing Open MPI"
+    export OMPI_DIR=/opt/ompi
+    export OMPI_VERSION=4.0.5
+    export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-$OMPI_VERSION.tar.bz2"
+    mkdir -p /tmp/ompi
+    mkdir -p /opt
+    # Download
+    cd /tmp/ompi && wget -O openmpi-$OMPI_VERSION.tar.bz2 $OMPI_URL && tar -xjf openmpi-$OMPI_VERSION.tar.bz2
+    # Compile and install
+    cd /tmp/ompi/openmpi-$OMPI_VERSION && ./configure --prefix=$OMPI_DIR && make install
+    # Set env variables so we can compile our application
+    export PATH=$OMPI_DIR/bin:$PATH
+    export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
+    export MANPATH=$OMPI_DIR/share/man:$MANPATH
+
+    echo "Compiling the MPI application..."
+    cd /opt && mpicc -o mpitest mpitest.c

workshop/HandsOn.md

+# Hands-On
+
+In order to build a singularity container you require a **root** environment, such as on a
+- Personal computer (Linux) -  (development: Windows, Mac) - [Instructions](https://sylabs.io/guides/3.0/user-guide/installation.html#overview)
+- Virtual Machine (all OS) - VMWare, KVM, VirtualBox - install a recent Linux and install singularity
+- Build Service - [Sylab Cloud](https://cloud.sylabs.io/home) - advanced
+
+## IMGW Resources
+Singularity is installed on all IMGW servers and on VSC4 via module (`module load singularity`). Most other HPC environments have a version installed as well.
+
+Based on a repository from M. Kandas, there is an installation routine 
+```bash
+git clone https://gitlab.phaidra.org/imgw/singularity.git 
+sudo ./install-singularity.sh install
+```
+and there are example **definition-files** that help to build containers by themselves. These files are called *Receipes*. It is of course by design that these receipes can be shared and adopted to ones needs. If you have a good one. Send it to us. Thanks. :)
+
+
+## Using a VM to build containers
+
+We are using a VM to build our containers with root privileges.
+Steps:
+- ssh to **jet01**
+- ssh using **rocky##@192.168.122.230**
+
+Please note that you will recieve the username and the password in the course.
+
+## Singularity Tutorial
+
+There are a few things that you need to learn about singularity.
+
+```bash
+singularity [options] <subcommand> [subcommand options] ...
+```
+
+has three essential subcommands:
+- **build**: Build your own container from scratch using a Singularity definition file; download and assemble any existing Singularity container; or convert your containers from one format to another (e.g., from Docker to Singularity).
+- **shell**: Spawn an interactive shell session in your container.
+- **exec**: Execute an arbitrary command within your container.
+
+### Definition File
+
+Typically a Singularity Recipe or Definition files is made of:
+- Bootstrap - start source
+- Files - Files to copy into the container
+- Post - install instructions
+- Labels - Information for the User
+- Environment - Runtime environment variables
+- Run - Runtime instructions, script, executable
+- Apps - Special executables defined as apps
+- Setup - 
+- Test - Tests to check if the container runs optimal
+
+
+e.g. [miniconda3](../definition-files/miniconda/Singularity.miniconda3-py39-4.9.2-ubuntu-18.04)
+
+```singularity
+Bootstrap: library
+From: mblaschek/imgw/ubuntu:18.04
+
+
+%labels
+
+    APPLICATION_NAME miniconda3
+    APPLICATION_VERSION py39-4.9.2-Linux-x86_64
+    APPLICATION_URL https://docs.conda.io
+
+    AUTHOR_NAME Michael Blaschek
+    AUTHOR_EMAIL michael.blaschek@univie.ac.at
+
+    LAST_UPDATED 20211118
+
+%setup
+
+%environment
+
+    # Set the conda distribution type, its version number, the python
+    # version it utilizes, the root and installation directories where
+    # the distribution will be installed within the container, and the
+    # root URL to the installer
+    export CONDA_DISTRIBUTION='miniconda'
+    export CONDA_VERSION='3'
+    export CONDA_PYTHON_VERSION='py39'
+    export CONDA_INSTALLER_VERSION='4.9.2'
+    export CONDA_ARCH='Linux-x86_64'
+    export CONDA_INSTALL_DIR="/opt/${CONDA_DISTRIBUTION}${CONDA_VERSION}"
+
+    # Set PATH to conda distribution
+    export PATH="${CONDA_INSTALL_DIR}/bin:${PATH}"
+
+%post -c /bin/bash
+
+    # Set operating system mirror URL
+    export MIRRORURL='http://at.archive.ubuntu.com/ubuntu'
+
+    # Set operating system version
+    export OSVERSION='bionic'
+
+    # Set system locale
+    export LC_ALL='C'
+
+    # Set debian frontend interface
+    export DEBIAN_FRONTEND='noninteractive'
+
+    # Upgrade all software packages to their latest versions
+    apt-get -y update && apt-get -y upgrade
+
+    cd /tmp
+
+    # Set the conda distribution type, its version number, the python
+    # version it utilizes, the root and installation directories where
+    # the distribution will be installed within the container, and the
+    # root URL to the installer
+    export CONDA_DISTRIBUTION='miniconda'
+    export CONDA_VERSION='3'
+    export CONDA_PYTHON_VERSION='py39'
+    export CONDA_INSTALLER_VERSION='4.9.2'
+    export CONDA_ARCH='Linux-x86_64'
+    export CONDA_INSTALLER="${CONDA_DISTRIBUTION^}${CONDA_VERSION}-${CONDA_PYTHON_VERSION}_${CONDA_INSTALLER_VERSION}-${CONDA_ARCH}.sh"
+    export CONDA_INSTALL_DIR="/opt/${CONDA_DISTRIBUTION}${CONDA_VERSION}"
+    export CONDA_ROOT_URL='https://repo.anaconda.com'
+
+    # Download and install conda distribution
+    wget "${CONDA_ROOT_URL}/${CONDA_DISTRIBUTION}/${CONDA_INSTALLER}"
+    chmod +x "${CONDA_INSTALLER}"
+    "./${CONDA_INSTALLER}" -b -p "${CONDA_INSTALL_DIR}"
+
+    # Remove conda installer
+    rm "${CONDA_INSTALLER}"
+
+    # Cleanup
+    apt-get -y autoremove --purge
+    apt-get -y clean
+
+    # Update database for mlocate
+    updatedb
+
+%files
+
+%runscript
+
+%test
+```
+
+### Singularity Variables
+
+
+`SINGULARITY_CACHEDIR` `SINGULARITY_TMPDIR`
+
+Limited space in home directories.
+Set to `$TMPDIR` to avoid quota limits.
+
+```bash
+export SINGULARITY_CACHEDIR=$TMPDIR
+export SINGULARITY_TMPDIR=$TMPDIR
+```
+

workshop/Introduction.md

-class: center, middle
-
-# My Awesome Presentation
-
-???
-
-Notes for the _first_ slide!
-
----
-
-# Agenda
-
-1. Introduction
-2. Deep-dive
-3. ...
-
-[NOTE]: Note that you need remark.js alongside this html file, but no internet connection.
----
-
-# Introduction
-
-Welcome to the workd of tomorrow
-
-
----
-
-# End

workshop/MPI/mpitest.c

+#include <mpi.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+int main (int argc, char **argv) {
+        int rc;
+        int size;
+        int myrank;
+
+        rc = MPI_Init (&argc, &argv);
+        if (rc != MPI_SUCCESS) {
+                fprintf (stderr, "MPI_Init() failed");
+                return EXIT_FAILURE;
+        }
+
+        rc = MPI_Comm_size (MPI_COMM_WORLD, &size);
+        if (rc != MPI_SUCCESS) {
+                fprintf (stderr, "MPI_Comm_size() failed");
+                goto exit_with_error;
+        }
+
+        rc = MPI_Comm_rank (MPI_COMM_WORLD, &myrank);
+        if (rc != MPI_SUCCESS) {
+                fprintf (stderr, "MPI_Comm_rank() failed");
+                goto exit_with_error;
+        }
+
+        fprintf (stdout, "Hello, I am rank %d/%d", myrank, size);
+
+        MPI_Finalize();
+
+        return EXIT_SUCCESS;
+
+ exit_with_error:
+        MPI_Finalize();
+        return EXIT_FAILURE;
+}

workshop/README.md

 # Singularity Workshop
 @IMGW
 
+**Date:** 24.11.2021
 
-- [Introduction](https://srvx1.img.univie.ac.at/webdata/imgw/Singularity-Workshop-Introduction.html)
-- [Examples]()
-- [Advanced]()
+**Lecturer:** @blaschm7
+
+## Summary
+Introduction to Singularity and its applications on HPC. The workshop aims at bringing the tools to the users and trying first hand what can be done. Of course this is just a tiny introduction to a much larger topic.
+
+Literature:
+- [Singularity: Scientific containers for mobility of compute](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459)
+- Cloud Computing for Science and Engineering - Book - [UB](https://web-p-ebscohost-com.uaccess.univie.ac.at/ehost/detail/detail?vid=0&sid=6dbe42f5-29a9-4256-bfc6-bece2b66646e%40redis&bdata=JnNpdGU9ZWhvc3QtbGl2ZQ%3d%3d#AN=2517979&db=nlebk) 
+- [A container for HPC](https://www.admin-magazine.com/HPC/Articles/Singularity-A-Container-for-HPC)
+
+## Schedule
+
+| Time | Task |
+| --- | --- |
+| 10:30 - 11:00 | [Introduction to Singularity](Introduction.pdf) |
+| 11:00 - 11:30 | [Hands-on](HandsOn.md), build a container |
+| 11:30 - 12:00 | [Problems](Introduction-Advanced.pdf), advanced examples |