update singularity cloud, build spack

README.md

-# Singularity
+# Singularity / Apptainer
 
-A repository of definition files for building 
-[Singularity](https://sylabs.io/guides/latest/user-guide) containers
-around the software applications, frameworks, and libraries you need to
-run on high-performance computing systems.
+A repository of definition files for building [Singularity](https://sylabs.io/guides/latest/user-guide) containers around the software applications, frameworks, and libraries you need to run on  high-performance computing systems.
 
 Based on [naked-singularity](https://github.com/mkandes/naked-singularity). Thanks M. Kandas
 
-## Install Singularity
+**Please note that since 2022 there is a Linux Foundation Version of singularity called apptainer. Currently developments are similar.**
 
-Install Singularity on your Linux desktop, laptop, or virtual machine. 
+## Install Singularity / Apptainer
+
+Install Singularity on your Linux desktop, laptop, or virtual machine. You need *root* permissions to use and install the package.
 
 There is a general install script that installs version 3.5.3 (can be easily changed to 3.8 or 3.9) from source, but there are also binary packages build for modern OS (Server OS).
 e.g. Red Hat, Arch, ...
 
-[Installing Singularity Documentation](https://sylabs.io/guides/3.6/admin-guide/installation.html#)
+more informaiton on building from source can be found here: [Installing Singularity Documentation](https://sylabs.io/guides/3.6/admin-guide/installation.html#)
 
 ```bash
 # Centos / Red Hat
@@ -22,13 +21,33 @@ sudo yum update -y && \
     sudo yum install -y epel-release && \
     sudo yum update -y && \
     sudo yum install -y singularity
+
 # Arch / Manjaro
-pamac install singularity-container
+sudo pamac install singularity-container
 
 # Manual using this script (builds from source)
 sudo ./install-singularity.sh install
 ```
 
+e.g. Ubuntu does not have the package in the repos, but a deb file exists in the releases. See below.
+
+Please note that these versions can vary substantially. Please check that at least version 3.5 is available or for newer distributions use **apptainer**.
+
+### prebuild binary releases 
+
+If you can not find singularity or apptainer in your repos, there are prebuild packages available as well. Go to the github release [singularity](https://github.com/sylabs/singularity/releases) or [apptainer](https://github.com/apptainer/apptainer/releases) page.
+
+```bash
+# e.g. for Ubuntu
+wget https://github.com/apptainer/apptainer/releases/download/v1.1.0/apptainer_1.1.0_amd64.deb
+sudo dpkg -i apptainer_1.1.0_amd64.deb
+# gives some errors of missing dependencies. solve running:
+sudo apt-get -f install
+# Retry and see that is completes successfully
+sudo dpkg -i apptainer_1.1.0_amd64.deb
+# done
+```
+
 ## Build your first Singularity container from a definition file
 
 Build an Ubuntu Singularity container from one of the definition files
@@ -54,6 +73,23 @@ sudo singularity build ubuntu.sif docker://ubuntu:20.04
 We try to build custom Singularity containers with signatures to help keeping track of changes and interoperability.
 
 ```bash
+# You need to do this only once.
+# add the library (no login required)
+singularity remote add --no-login sycloud cloud.sylabs.io
+# make it the default one
+singularity remote use sycloud
+# check if you get access
+singularity remote status
+    INFO:    Checking status of default remote.
+    SERVICE    STATUS  VERSION                 URI
+    Builder    OK      v1.6.7-0-gebd61128      https://build.sylabs.io
+    Consent    OK      v1.6.6-0-gd8f171e       https://auth.sylabs.io/consent
+    Keyserver  OK      v1.18.9-0-g76cbd56      https://keys.sylabs.io
+    Library    OK      v0.3.7-rc.2-0-gd7bd65e  https://library.sylabs.io
+    Token      OK      v1.6.6-0-gd8f171e       https://auth.sylabs.io/token
+
+    No authentication token set (logged out).
+# Now pull the container
 singularity pull ubuntu.sif library://mblaschek/imgw/ubuntu:20.04 
 ```
 

definition-files/almalinux/Singularity-alma8-spack

+Bootstrap: docker
+From: almalinux:8
+
+%labels
+    MAINTAINER Spack Maintainers <maintainers@spack.io>
+
+%post
+    # Every line will be a layer in the container
+    # See https://fedoraproject.org/wiki/EPEL#Quickstart for powertools
+    yum update -y \
+    && yum install -y dnf-plugins-core \
+    && dnf config-manager --set-enabled powertools \
+    && yum install -y epel-release \
+    && yum update -y \
+    && yum --enablerepo epel groupinstall -y "Development Tools" \
+    && yum --enablerepo epel install -y \
+            curl \
+            findutils \
+            gcc-c++ \
+            gcc \
+            gcc-gfortran \
+            git \
+            gnupg2 \
+            hostname \
+            iproute \
+            make \
+            patch \
+            python38 \
+            python38-pip \
+            python38-setuptools \
+            unzip \
+    && pip3 install boto3 \
+    && rm -rf /var/cache/yum \
+    && yum clean all
+    # Define Spack root
+    export SPACK_ROOT=/opt/spack
+    # Get the version of spack that you want
+    mkdir -p /opt && cd /opt && \
+        git clone --depth 1 --single-branch --branch 'v0.17.3' https://github.com/spack/spack.git
+    # Enable spack shell extensions
+    ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
+        /usr/local/bin/docker-shell && \
+        ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
+        /usr/local/bin/interactive-shell && \
+        ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
+        /usr/local/bin/spack-env
+    # only for docker as root
+    #mkdir -p /root/.spack \
+    #&& cp $SPACK_ROOT/share/spack/docker/modules.yaml \
+    #/root/.spack/modules.yaml \
+    rm -rf /root/*.* /run/nologin $SPACK_ROOT/.git
+
+    # [WORKAROUND]
+    # https://superuser.com/questions/1241548/
+    #     xubuntu-16-04-ttyname-failed-inappropriate-ioctl-for-device#1253889
+    [ -f ~/.profile ] && \
+    sed -i 's/mesg n/( tty -s \\&\\& mesg n || true )/g' ~/.profile || true
+
+    #cd /root
+    # TODO: add a command to Spack that (re)creates the package cache
+    . /opt/spack/share/spack/setup-env.sh
+    spack bootstrap untrust spack-install && \
+    spack spec hdf5+mpi && \
+    spack gc -y 
+
+%environment
+    export SPACK_ROOT=/opt/spack
+    export DEBIAN_FRONTEND=noninteractive
+
+%runscript
+    exec /bin/bash /opt/spack/share/spack/docker/entrypoint.bash interactive-shell "$@"
+
+%startscript
+    exec /bin/bash /opt/spack/share/spack/docker/entrypoint.bash interactive-shell "$@"

definition-files/centos/Singularity-centos8-IMPI

@@ -20,4 +20,5 @@ gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.
 EOF
     mv /tmp/oneAPI.repo /etc/yum.repos.d
     yum install -y intel-oneapi-mpi*
-    yum install -y sudo wget vi which numactl bzip2 tar gcc hostname util-linux redhat-lsb
+    # yum install -y sudo wget vi which numactl bzip2 tar gcc hostname util-linux redhat-lsb
+    yum install -y wget numactl bzip2 tar gcc hostname util-linux redhat-lsb
\ No newline at end of file

development/custom/build_base.sh

+#!/bin/bash
+# By MB
+# Build the required SPACK container images locally
+
+CPATH=$(dirname $0)
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+report() {
+    if [ $? -eq 0 ]; then
+        printf "%-68s[$GREEN%10s$NC]\n" "$@" "OK"
+        return 0
+    else
+        printf "%-68s[$RED%10s$NC]\n" "$@" "FAILED"
+        return 1
+    fi
+}
+
+centos7def() {
+    cat <<EOF >Singularity.centos7.$gccversion
+Bootstrap: docker
+From: spack/centos7:v${spackversion}
+Stage: build
+
+%post
+  # takes about 1 hour to build
+  # Create the manifest file for the installation in /opt/spack-environment
+  mkdir /opt/spack-environment && cd /opt/spack-environment
+  cat << EOM > spack.yaml
+spack:
+  specs:
+  - gcc@${gccversion}
+  concretization: together
+  config:
+    install_tree: /opt/software
+  view: /opt/view
+EOM
+
+  # Install all the required software
+  . /opt/spack/share/spack/setup-env.sh
+  spack env activate .
+  spack install --fail-fast
+  spack compiler add $(spack location -i gcc@${gccversion})
+  spack gc -y
+  spack env deactivate
+  spack env activate --sh -d . >> /opt/spack-environment/environment_modifications.sh
+
+  # Strip the binaries to reduce the size of the image
+  find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
+    xargs file -i | \
+    grep 'charset=binary' | \
+    grep 'x-executable\|x-archive\|x-sharedlib' | \
+    awk -F: '{print \$1}' | xargs strip -s
+
+
+Bootstrap: docker
+From: centos:7
+Stage: final
+
+%files from build
+  /opt/spack-environment /opt
+  /opt/software /opt
+  /opt/spack /opt
+  /opt/view /opt
+  /opt/spack-environment/environment_modifications.sh /opt/spack-environment/environment_modifications.sh
+
+%post
+  # Modify the environment without relying on sourcing shell specific files at startup
+  cat /opt/spack-environment/environment_modifications.sh >> \$SINGULARITY_ENVIRONMENT
+EOF
+}
+
+alma8def() {
+    cat <<EOF >Singularity.alma8.$gccversion
+# Bootstrap: library
+# From: mblaschek/spack/alma8:${spackversion}
+Bootstrap: localimage
+From: alma8-spack${spackversion}
+Stage: build
+
+%post
+  # takes about 1 hour to build
+  # Create the manifest file for the installation in /opt/spack-environment
+  mkdir /opt/spack-environment && cd /opt/spack-environment
+  cat << EOM > spack.yaml
+spack:
+  specs:
+  - gcc@${gccversion}
+  concretization: together
+  config:
+    install_tree: /opt/software
+  view: /opt/view
+EOM
+
+  # Install all the required software
+  . /opt/spack/share/spack/setup-env.sh
+  spack env activate .
+  spack install --fail-fast
+  spack compiler add $(spack location -i gcc@${gccversion})
+  spack gc -y
+  spack env deactivate
+  spack env activate --sh -d . >> /opt/spack-environment/environment_modifications.sh
+
+  # Strip the binaries to reduce the size of the image
+  find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
+    xargs file -i | \
+    grep 'charset=binary' | \
+    grep 'x-executable\|x-archive\|x-sharedlib' | \
+    awk -F: '{print \$1}' | xargs strip -s
+
+# Bootstrap: library
+# From: mblaschek/spack/alma8:${spackversion}
+Bootstrap: localimage
+From: alma8-spack${spackversion}
+Stage: final
+
+%files from build
+  /opt/spack-environment /opt
+  /opt/software /opt
+  /opt/spack /opt
+  /opt/view /opt
+  /opt/spack-environment/environment_modifications.sh /opt/spack-environment/environment_modifications.sh
+
+%post
+  # Modify the environment without relying on sourcing shell specific files at startup
+  cat /opt/spack-environment/environment_modifications.sh >> \$SINGULARITY_ENVIRONMENT
+EOF
+}
+
+question() {
+	read -p "$@ (y/n)" yn
+	case $yn in
+	[Yy]*) return 0 ;;
+	*)
+		warning "skipping"
+		return 1
+		;;
+	esac
+}
+
+# Requires sudo/root privileges
+test $EUID -eq 0
+report "Requires sudo/root privileges" || exit 1
+status=$(type singularity)
+report "Singularity: $status" || exit 1
+echo "Singularity Version: $(singularity version)"
+# Find Recipes
+DEFPATH=$(realpath $CPATH/../..)/definition-files
+test -d $DEFPATH
+report "Definitions directory" || exit 1
+echo "Skipping CentOS 7 builds ... supplied by spack"
+# Check declared Functions
+echo "Checking functions ..."
+declare -F | grep -v "\-fx" | cut -d" " -f3 | awk '{print "- ", $0}'
+echo ""
+# for two stage builds use:
+# Bootstrap: docker   -> localimage
+# From: spack/centos7:v0.17.3  -> centos7-spack
+# Stage: build
+
+# for final
+# Bootstrap: docker   -> localimage
+# From: centos:7    -> centos7-dev
+# Stage: final
+
+spackversion=
+read -p "Spack Version (default: 0.17.3): " spackversion
+if [ -z "$spackversion" ]; then
+    spackversion=0.17.3
+fi
+gccversion=
+read -p "GCC Compiler Version (default: 8.4.0): " gccversion
+if [ -z "$gccversion" ]; then
+    gccversion=8.4.0
+fi
+#
+# Build AlmaLinux spack container
+#
+test -f alma8-spack${spackversion}
+report "AlmaLinux 8 spack image available"
+if [ $? -ne 0 ]; then
+    test -f $DEFPATH/almalinux/Singularity-alma8-spack
+    report "AlmaLinux 8 spack definition found" || exit 1
+    sed "s/v0.17.3/${spackversion}/" definition-files/almalinux/Singularity-alma8-spack >Singularity-alma8-spack
+    singularity build alma8-spack${spackversion} Singularity-alma8-spack
+    report "AlmaLinux 8 spack container ready" || exit 1
+fi
+#
+# Build Centos 7 GCC build container
+#
+echo "Building CentOS 7 with spack $spackversion and gcc $gccversion"
+test -f centos7-spack${spackversion}-gcc${gccversion}
+report "Singularity Image: centos7-spack${spackversion}-gcc${gccversion}"
+question "Build Centos 7 spack with gcc $gccversion ?"
+if [ $? -eq 0 ]; then
+    echo "Preparing definitions ... "
+    centos7def
+    test -f Singularity.centos7.$gccversion
+    report "Definitions ready." || exit 1
+    echo "Building ..."
+    singularity build centos7-spack${spackversion}-gcc${gccversion} Singularity.centos7.$gccversion
+    report "Singularity CentOS 7 image $spackversion / $gccversion"
+fi
+#
+# Build AlmaLinux 8 GCC build container
+#
+test -f alma8-spack${spackversion}-gcc${gccversion}
+report "Singularity Image: alma8-spack${spackversion}-gcc${gccversion}"
+if [ $? -ne 0 ]; then
+    echo "Preparing definitions ... "
+    alma8def
+    test -f Singularity.alma8.$gccversion
+    report "Definitions ready." || exit 1
+    echo "Building ..."
+    singularity build alma8-spack${spackversion}-gcc${gccversion} Singularity.alma8.$gccversion
+    report "Singularity AlmaLinux 8 image $spackversion / $gccversion"
+fi
+# singularity build centos8-spack-gcc${gccversion} Singularity.centos8.$gccversion
+#
+# Verify containers work
+# not as root
+su mblaschek -c 'singularity key list | grep IMGW'
+report "IMGW GPG key" || exit 1
+echo "Please supply GPG passphrase for your key"
+su mblaschek -c "singularity sign alma8-spack${spackversion}"
+su mblaschek -c "singularity sign alma8-spack${spackversion}-gcc${gccversion}"
+su mblaschek -c "singularity sign centos7-spack${spackversion}-gcc${gccversion}"
+#
+# Upload to cloud / repository for open access
+#
+# Spack primary development container
+singularity push alma8-spack${spackversion} library://mblaschek/spack/alma8:${spackversion}
+# Spack development container with gcc versions
+singularity push alma8-spack${spackversion}-gcc${gccversion} library://mblaschek/spack/${spackversion}/alma8/gcc:${gccversion}
+singularity push centos7-spack${spackversion}-gcc${gccversion} library://mblaschek/spack/${spackversion}/centos7/gcc:${gccversion}
+# singularity push centos8-spack-icc${iccversion} library://mblaschek/spack/alma8/icc:${iccversion}