conda activate function revised!

32e39ea7 · Michael Blaschek · d5baffd4 · 32e39ea7 · 32e39ea7
Commit 32e39ea7 authored Dec 1, 2021 by Michael Blaschek
--- a/Python/QA-003-Conda-Environment.ipynb
+++ b/Python/QA-003-Conda-Environment.ipynb
@@ -259,14 +259,38 @@
    "will edit your `.bashrc` and that is not nice, especially when you might move to another module. \n",
    "**Therefore this is not recommended to do.**\n",
    "\n",
-    "In order to use your environment in a shell script or on a slurm job, try the following:\n",
+    "This script is used by `conda init bash` to set the configuration, we simply source (load) it and get the same functionality as we would be using `conda init bash`. **(Do not use `conda init bash`)**\n",
    "\n",
-    "Load the shell setup script located in the conda source tree (`${CONDA_ROOT}/etc/profile.d/conda.sh`) and activate your environment:\n",
    "```bash\n",
-    "# Load from the current conda module the BASH functionality (as in activate)\n",
-    "source $(which conda)/../etc/profile.d/conda.sh\n",
+    "source $(dirname $(which conda))/../etc/profile.d/conda.sh\n",
    "# now activating the environment works\n",
    "conda activate myenv\n",
+    "```\n",
+    "\n",
+    "#### bash conda_activate function\n",
+    "optionaly you can add this bash function to your `.bashrc` for convenience:\n",
+    "\n",
+    "```bash\n",
+    "conda_activate(){\n",
+    "    if ! command -v conda &> /dev/null; then\n",
+    "        echo \"Load ana/miniconda module first\"\n",
+    "    else\n",
+    "        if [ $# -ne 1 ]; then\n",
+    "            echo \"usage: conda_activate [env-name]\"\n",
+    "            \n",
+    "        else\n",
+    "            source $(dirname $(which conda))/../etc/profile.d/conda.sh\n",
+    "            conda activate $1\n",
+    "        fi\n",
+    "    fi\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "After your `.bashrc` has been loaded (e.g. logout/login again) you should be able to use\n",
+    "\n",
+    "```bash\n",
+    "module load anaconda3\n",
+    "conda_activate myenv\n",
    "```"
   ]
  },

 %% Cell type:markdown id: tags:

 # Setting up a user environment

 Topics
 * Setup a user environment [using conda](#using-conda)
    - [Add user environment to notebook kernels](#add-the-new-environment-to-your-active-kernels)
    - [Activate your conda environment](#activate-your-conda-environment)
 * Setup a user environment [using venv](#using-virtual-environment)
 * Include `PATH` in [Notebook Kernel](#add-path-to-your-user-environment)

 %% Cell type:code id: tags:

 ``` python
 !module list
 ```

 %% Output

    Currently Loaded Modulefiles:[m
     1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2  [m
    [K[?1l>

 %% Cell type:markdown id: tags:

 ## Using conda

 %% Cell type:code id: tags:

 ``` python
 !conda info
 ```

 %% Output

    
         active environment : None
           user config file : /jetfs/home/mblaschek/.condarc
     populated config files : /jetfs/home/mblaschek/.condarc
              conda version : 4.8.5
        conda-build version : not installed
             python version : 3.8.1.final.0
           virtual packages : __glibc=2.28
           base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv  (read only)
               channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                              https://repo.anaconda.com/pkgs/main/noarch
                              https://repo.anaconda.com/pkgs/r/linux-64
                              https://repo.anaconda.com/pkgs/r/noarch
              package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
                              /jetfs/home/mblaschek/.conda/pkgs
           envs directories : /jetfs/home/mblaschek/.conda/envs
                              /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
                   platform : linux-64
                 user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
                    UID:GID : 54212:100
                 netrc file : None
               offline mode : False
    

 %% Cell type:code id: tags:

 ``` python
 # Create a environment with conda
 !conda create -y -n myenv python=3.7 numpy ipykernel
 ```

 %% Output

    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
    Collecting package metadata (repodata.json): done
    Solving environment: done
    
    
    ==> WARNING: A newer version of conda exists. <==
      current version: 4.8.5
      latest version: 4.9.2
    
    Please update conda by running
    
        $ conda update -n base -c defaults conda
    
    
    
    ## Package Plan ##
    
      environment location: /jetfs/home/mblaschek/.conda/envs/myenv
    
      added / updated specs:
        - ipykernel
        - numpy
        - python=3.7
    
    
    The following packages will be downloaded:
    
        package                    |            build
        ---------------------------|-----------------
        certifi-2020.11.8          |   py37h06a4308_0         148 KB
        ipython-7.19.0             |   py37hb070fc8_0         978 KB
        jupyter_core-4.7.0         |   py37h06a4308_0          68 KB
        numpy-1.19.2               |   py37h54aff64_0          22 KB
        numpy-base-1.19.2          |   py37hfa32c7d_0         4.1 MB
        pexpect-4.8.0              |     pyhd3eb1b0_3          53 KB
        pip-20.2.4                 |   py37h06a4308_0         1.7 MB
        ptyprocess-0.6.0           |     pyhd3eb1b0_2          20 KB
        pygments-2.7.2             |     pyhd3eb1b0_0         674 KB
        setuptools-50.3.1          |   py37h06a4308_1         711 KB
        six-1.15.0                 |   py37h06a4308_0          27 KB
        wheel-0.35.1               |     pyhd3eb1b0_0          38 KB
        ------------------------------------------------------------
                                               Total:         8.5 MB
    
    The following NEW packages will be INSTALLED:
    
      _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
      backcall           pkgs/main/noarch::backcall-0.2.0-py_0
      blas               pkgs/main/linux-64::blas-1.0-mkl
      ca-certificates    pkgs/main/linux-64::ca-certificates-2020.10.14-0
      certifi            pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
      decorator          pkgs/main/noarch::decorator-4.4.2-py_0
      intel-openmp       pkgs/main/linux-64::intel-openmp-2020.2-254
      ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
      ipython            pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
      ipython_genutils   pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
      jedi               pkgs/main/linux-64::jedi-0.17.2-py37_0
      jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
      jupyter_core       pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
      ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
      libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
      libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
      libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
      libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
      libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
      mkl                pkgs/main/linux-64::mkl-2020.2-256
      mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
      mkl_fft            pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
      mkl_random         pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
      ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
      numpy              pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
      numpy-base         pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
      openssl            pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
      parso              pkgs/main/noarch::parso-0.7.0-py_0
      pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
      pickleshare        pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
      pip                pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
      prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
      ptyprocess         pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
      pygments           pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
      python             pkgs/main/linux-64::python-3.7.9-h7579374_0
      python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
      pyzmq              pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
      readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
      setuptools         pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
      six                pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
      sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
      tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
      tornado            pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
      traitlets          pkgs/main/noarch::traitlets-5.0.5-py_0
      wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
      wheel              pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
      xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
      zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
      zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
    
    
    
    Downloading and Extracting Packages
    numpy-1.19.2         | 22 KB     | ##################################### | 100%
    pip-20.2.4           | 1.7 MB    | ##################################### | 100%
    numpy-base-1.19.2    | 4.1 MB    | ##################################### | 100%
    certifi-2020.11.8    | 148 KB    | ##################################### | 100%
    wheel-0.35.1         | 38 KB     | ##################################### | 100%
    ipython-7.19.0       | 978 KB    | ##################################### | 100%
    pexpect-4.8.0        | 53 KB     | ##################################### | 100%
    pygments-2.7.2       | 674 KB    | ##################################### | 100%
    ptyprocess-0.6.0     | 20 KB     | ##################################### | 100%
    jupyter_core-4.7.0   | 68 KB     | ##################################### | 100%
    setuptools-50.3.1    | 711 KB    | ##################################### | 100%
    six-1.15.0           | 27 KB     | ##################################### | 100%
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done
    #
    # To activate this environment, use
    #
    #     $ conda activate myenv
    #
    # To deactivate an active environment, use
    #
    #     $ conda deactivate
    

 %% Cell type:markdown id: tags:

 ### Add the new environment to your active kernels
 This way you can choose the kernel from the Jupyterlab launcher.

 %% Cell type:code id: tags:

 ``` python
 !~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"
 ```

 %% Output

    Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv

 %% Cell type:markdown id: tags:

 ### Activate your conda environment
 Conda Environments need to be added to the `PATH` to work well. So there is an `activate` function for these conda environments.

 However, running
 ```bash
 conda init bash
 ```
 will edit your `.bashrc` and that is not nice, especially when you might move to another module.
 **Therefore this is not recommended to do.**

-In order to use your environment in a shell script or on a slurm job, try the following:
+This script is used by `conda init bash` to set the configuration, we simply source (load) it and get the same functionality as we would be using `conda init bash`. **(Do not use `conda init bash`)**

-Load the shell setup script located in the conda source tree (`${CONDA_ROOT}/etc/profile.d/conda.sh`) and activate your environment:
 ```bash
-# Load from the current conda module the BASH functionality (as in activate)
-source $(which conda)/../etc/profile.d/conda.sh
+source $(dirname $(which conda))/../etc/profile.d/conda.sh
 # now activating the environment works
 conda activate myenv
 ```

+#### bash conda_activate function
+optionaly you can add this bash function to your `.bashrc` for convenience:
+
+```bash
+conda_activate(){
+    if ! command -v conda &> /dev/null; then
+        echo "Load ana/miniconda module first"
+    else
+        if [ $# -ne 1 ]; then
+            echo "usage: conda_activate [env-name]"
+
+        else
+            source $(dirname $(which conda))/../etc/profile.d/conda.sh
+            conda activate $1
+        fi
+    fi
+}
+```
+
+After your `.bashrc` has been loaded (e.g. logout/login again) you should be able to use
+
+```bash
+module load anaconda3
+conda_activate myenv
+```
+
 %% Cell type:code id: tags:

 ``` python
 # List available Kernels
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 ## Using virtual environment

 %% Cell type:code id: tags:

 ``` python
 # Setup a virtual Environment for your Python in the directoy: myfancyenv
 !python -m venv myfancyenv
 ```

 %% Cell type:code id: tags:

 ``` python
 # Creates a directory with new environment
 !ls myfancyenv/
 ```

 %% Output

    bin  include  lib  lib64  pyvenv.cfg

 %% Cell type:code id: tags:

 ``` python
 # Install some package into that environment
 # important is ipykernel to use it within jupyterlab/notebook
 !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
 ```

 %% Output

    [33mWARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.[0m

 %% Cell type:code id: tags:

 ``` python
 # Allow this environment to be available on the LAUNCHERS (+) sign on the left
 !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
 ```

 %% Output

    Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy

 %% Cell type:markdown id: tags:

 Wait a bit and the new python environment should be available on your launchers.

 %% Cell type:code id: tags:

 ``` python
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      fancy      /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 ## Add PATH to your user environment

 In order to get full access to all tools installed in your local environment, add it to your `PATH`.

 Open your `kernel.json` file located `/jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json`:
 ```json
 {
    "argv": [
        "/jetfs/home/mblaschek/.conda/envs/myenv/bin/python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ],
    "env" : {
        "PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}",
        "LD_LIBRARY_PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
    },
    "display_name": "MYENV",
    "language": "python"
 }
 ```

 %% Cell type:markdown id: tags:

 ### Finished

--- a/Python/Your-First-Notebook-onJet_v2.ipynb
+++ b/Python/Your-First-Notebook-onJet_v2.ipynb
@@ -575,13 +575,36 @@
    "\n",
    "Hint: Checkout the guides on [Gitlab](https://gitlab.phaidra.org/imgw/computer-resources/-/blob/master/Python/QA-003-Conda-Environment.ipynb#Activate-your-conda-environment)\n",
    "\n",
-    "### Load from the current conda module the BASH functionality (as in activate)\n",
+    "### Load bash conda configuration\n",
+    "This script is used by `conda init bash` to set the configuration, we simply source (load) it and get the same functionality as we would be using `conda init bash`. **(Do not use `conda init bash`)**\n",
    "\n",
-    "```sh\n",
-    "source $(which conda)/../etc/profile.d/conda.sh\n",
+    "```bash\n",
+    "source $(dirname $(which conda))/../etc/profile.d/conda.sh\n",
    "# now activating the environment works\n",
    "conda activate myenv\n",
    "```\n",
+    "optionaly you can add this bash function to your `.bashrc` for convenience:\n",
+    "```bash\n",
+    "conda_activate(){\n",
+    "    if ! command -v conda &> /dev/null; then\n",
+    "        echo \"Load ana/miniconda module first\"\n",
+    "    else\n",
+    "        if [ $# -ne 1 ]; then\n",
+    "            echo \"usage: conda_activate [env-name]\"\n",
+    "            \n",
+    "        else\n",
+    "            source $(dirname $(which conda))/../etc/profile.d/conda.sh\n",
+    "            conda activate $1\n",
+    "        fi\n",
+    "    fi\n",
+    "}\n",
+    "```\n",
+    "After your `.bashrc` has been loaded (e.g. logout/login again) you should be able to use\n",
+    "\n",
+    "```bash\n",
+    "module load anaconda3\n",
+    "conda_activate myenv\n",
+    "```\n",
    "\n",
    "### Activate your kernel in your notebooks\n",
    "\n",

 %% Cell type:markdown id: tags:

 # Welcome to Jet

 <img src='https://homepage.univie.ac.at/michael.blaschek/openfiles/rocket.png' width='100px'><img src='https://homepage.univie.ac.at/michael.blaschek/openfiles/uni.jpg' width='400px'>

 You have done it! Your Notebook Server is running on the Jet Cluster.

 Welcome, please skip this, if your know where you are, as this is meant as an introduction. However, not complete as it is.
 If you need more infos about Notebooks have a look at [my Homepage](https://homepage.univie.ac.at/michael.blaschek/#python) and there are numerous Tutorials on the Internet for any advice, but feel free to ask me and give me feedback.

 Help can be found here:
 * Goto Help -> JupyterLab Reference
 * Goto Help -> Jupyter Reference
 * [Tour](https://jupyterlab.readthedocs.io/en/stable/user/interface.html)


 In the following I will show you some useful things.

 First: This Notebook is in your Home directory, but also on `/jetfs/scratch` for any reason.

 %% Cell type:markdown id: tags:

 ## Check your current Host

 %% Cell type:code id: tags:

 ``` python
 !hostnamectl
 ```

 %% Output

       Static hostname: localhost.localdomain
    Transient hostname: jet03.jet.local
             Icon name: computer-server
               Chassis: server
      Operating System: ]8;;https://www.centos.org/CentOS Linux 8 (Core)]8;;
           CPE OS Name: cpe:/o:centos:centos:8
                Kernel: Linux 4.18.0-147.el8.x86_64
          Architecture: x86-64

 %% Cell type:markdown id: tags:

 ## Who is running this process

 %% Cell type:code id: tags:

 ``` python
 !whoami
 ```

 %% Output

    mblaschek

 %% Cell type:code id: tags:

 ``` python
 # Where are you
 !pwd
 ```

 %% Output

    /jetfs/home/mblaschek/Documents

 %% Cell type:code id: tags:

 ``` python
 # Show my Current Jobs in the Queue
 !squeue -u $(id -u) -l
 ```

 %% Output

    Fri Oct 16 23:10:58 2020
                 JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
                   235   compute jupyters mblasche  RUNNING    6:29:07   8:00:00      1 jet03

 %% Cell type:markdown id: tags:

 There you can see your `TIME_LIMIT` of 8 hours again and how much time has passed on your current jobs

 %% Cell type:code id: tags:

 ``` python
 # Import sys and check the executable (so what python you are using)
 import sys
 print(sys.executable)
 ```

 %% Output

    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin/python

 %% Cell type:markdown id: tags:

 **Notice the funny path?**

    Yes this is SPACK your nice scientific package manager.
    Allows to build custom libraries and applications tailored to our needs!
    Please not that software currently installed in /opt on JET01 is not available here.
    Software under /opt is deprecated and will be removed after a full transition
    to the current libraries

 More information on [SPACK](https://spack.io/)

 ## Check your environmental PATH Variable, could be useful to know

 %% Cell type:code id: tags:

 ``` python
 %env PATH
 ```

 %% Output

    '/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin:/bin:/jetfs/spack/bin:/jetfs/userservices:/opt/jupyterhub/bin:/opt/slurm/bin:/sbin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/share/Modules/bin::/jetfs/home/mblaschek/bin:/jetfs/home/mblaschek/.local/bin'

 %% Cell type:markdown id: tags:

 ## Check what modules are currently loaded in this Notebook session

 %% Cell type:code id: tags:

 ``` python
 !module list --no-pager
 ```

 %% Output

    Currently Loaded Modulefiles:
     1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2

 %% Cell type:markdown id: tags:

 Put this file `load_modules_into_jupyter.conf` into your Home directory and your Notebook environment will have these modules loaded.
 It is not possible to load these modules via magic (python magic ;).
 **It needs the empty line at the end...**

 %% Cell type:code id: tags:

 ``` python
 %%writefile ~/load_modules_into_jupyter.conf
 eccodes/2.18.0-gcc-8.3.1-s7clum3
 ```

 %% Output

    Writing /jetfs/home/mblaschek/load_modules_into_jupyter.conf

 %% Cell type:code id: tags:

 ``` python
 !cat ~/load_modules_into_jupyter.conf
 ```

 %% Output

    eccodes/2.18.0-gcc-8.3.1-s7clum3

 %% Cell type:markdown id: tags:

 **Restarting the kernel will show now loaded modules**

 %% Cell type:code id: tags:

 ``` python
 # after restarting
 !module list --no-pager
 ```

 %% Output

    Currently Loaded Modulefiles:
     1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
     2) openmpi/4.0.5-gcc-8.3.1-773ztsv
     3) hdf5/1.12.0-gcc-8.3.1-awl4atl
     4) parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn
     5) netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
     6) eccodes/2.18.0-gcc-8.3.1-s7clum3

 %% Cell type:markdown id: tags:

 ## Currently available Modules

 %% Cell type:code id: tags:

 ``` python
 # --no-pager is needed only inside notebook
 !module avail --no-pager
 ```

 %% Output

    --------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
    anaconda2/2019.10-gcc-8.3.1-5pou6ji
    anaconda3/2019.10-gcc-8.3.1-tmy5mgp
    anaconda3/2020.07-gcc-8.3.1-weugqkf
    cdo/1.9.8-gcc-8.3.1-ipgvzeh
    eccodes/2.18.0-gcc-8.3.1-s7clum3
    hdf5/1.12.0-gcc-8.3.1-awl4atl
    intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw
    miniconda2/4.7.12.1-gcc-8.3.1-zduqggv
    miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
    netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
    netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj
    openmpi/3.1.6-gcc-8.3.1-rk5av53
    openmpi/4.0.5-gcc-8.3.1-773ztsv
    parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu
    parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn
    
    --------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
    anaconda2/2019.10-gcc-8.3.1-5pou6ji
    anaconda3/2019.10-gcc-8.3.1-tmy5mgp
    anaconda3/2020.07-gcc-8.3.1-weugqkf
    cdo/1.9.8-gcc-8.3.1-ipgvzeh
    eccodes/2.18.0-gcc-8.3.1-s7clum3
    hdf5/1.12.0-gcc-8.3.1-awl4atl
    intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw
    miniconda2/4.7.12.1-gcc-8.3.1-zduqggv
    miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
    netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
    netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj
    openmpi/3.1.6-gcc-8.3.1-rk5av53
    openmpi/4.0.5-gcc-8.3.1-773ztsv
    parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu
    parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn

 %% Cell type:markdown id: tags:

 # Software on JET
 Currently:
 - [x] Anaconda (2,3)
 - [x] Miniconda (2,3)
 - [x] OPENMPI (4.0.5, 3.6.1) optimized for JET
 - [x] HDF5 (1.12) optimized with OPENMPI
 - [x] NetCDF (Parallel (NC3), NC4->HDF5)
 - [x] Eccodes (2.18)
 - [x] CDO
 - [x] Intel compiled Libraries (License)
 - [x] Emoslib
 - [x] Ecaccess
 - [ ] RTTOV

 %% Cell type:markdown id: tags:

 ## Currently there are a few choices:

 1. Python (2, 3)
    * Anaconda
    * Miniconda
 2. Julia (not yet)
 3. R (not yet)

 %% Cell type:markdown id: tags:

 ## Install Python packages

 %% Cell type:code id: tags:

 ``` python
 # Please use the --user flag to install the packages into .local/lib/python...
 %pip install --user numpy scipy
 ```

 %% Output

    Requirement already satisfied: numpy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.19.2)
    Requirement already satisfied: scipy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.5.2)
    Note: you may need to restart the kernel to use updated packages.

 %% Cell type:code id: tags:

 ``` python
 import os
 import sys
 import site
 # Need to update search Path / only necessary here for the first time
 # sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
 sys.path.append(site.getusersitepackages())
 ```

 %% Cell type:code id: tags:

 ``` python
 import numpy as np
 import scipy as sp

 # WORKS! ;)
 ```

 %% Cell type:markdown id: tags:

 ## Using conda

 %% Cell type:code id: tags:

 ``` python
 !conda info
 ```

 %% Output

    
         active environment : None
           user config file : /jetfs/home/mblaschek/.condarc
     populated config files : /jetfs/home/mblaschek/.condarc
              conda version : 4.8.5
        conda-build version : not installed
             python version : 3.8.1.final.0
           virtual packages : __glibc=2.28
           base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv  (read only)
               channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                              https://repo.anaconda.com/pkgs/main/noarch
                              https://repo.anaconda.com/pkgs/r/linux-64
                              https://repo.anaconda.com/pkgs/r/noarch
              package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
                              /jetfs/home/mblaschek/.conda/pkgs
           envs directories : /jetfs/home/mblaschek/.conda/envs
                              /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
                   platform : linux-64
                 user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
                    UID:GID : 54212:100
                 netrc file : None
               offline mode : False
    

 %% Cell type:code id: tags:

 ``` python
 # Create a environment with conda
 !conda create -y -n myenv python=3.7 numpy ipykernel
 ```

 %% Output

    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
    Collecting package metadata (repodata.json): done
    Solving environment: done
    
    ## Package Plan ##
    
      environment location: /jetfs/home/mblaschek/.conda/envs/myenv
    
      added / updated specs:
        - ipykernel
        - numpy
        - python=3.7
    
    
    The following NEW packages will be INSTALLED:
    
      _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
      backcall           pkgs/main/noarch::backcall-0.2.0-py_0
      blas               pkgs/main/linux-64::blas-1.0-mkl
      ca-certificates    pkgs/main/linux-64::ca-certificates-2020.10.14-0
      certifi            pkgs/main/linux-64::certifi-2020.6.20-py37_0
      decorator          pkgs/main/noarch::decorator-4.4.2-py_0
      intel-openmp       pkgs/main/linux-64::intel-openmp-2020.2-254
      ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
      ipython            pkgs/main/linux-64::ipython-7.18.1-py37h5ca1d4c_0
      ipython_genutils   pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
      jedi               pkgs/main/linux-64::jedi-0.17.2-py37_0
      jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
      jupyter_core       pkgs/main/linux-64::jupyter_core-4.6.3-py37_0
      ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
      libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
      libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
      libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
      libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
      libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
      mkl                pkgs/main/linux-64::mkl-2020.2-256
      mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
      mkl_fft            pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
      mkl_random         pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
      ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
      numpy              pkgs/main/linux-64::numpy-1.19.1-py37hbc911f0_0
      numpy-base         pkgs/main/linux-64::numpy-base-1.19.1-py37hfa32c7d_0
      openssl            pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
      parso              pkgs/main/noarch::parso-0.7.0-py_0
      pexpect            pkgs/main/linux-64::pexpect-4.8.0-py37_1
      pickleshare        pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
      pip                pkgs/main/linux-64::pip-20.2.3-py37_0
      prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
      ptyprocess         pkgs/main/linux-64::ptyprocess-0.6.0-py37_0
      pygments           pkgs/main/noarch::pygments-2.7.1-py_0
      python             pkgs/main/linux-64::python-3.7.9-h7579374_0
      python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
      pyzmq              pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
      readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
      setuptools         pkgs/main/linux-64::setuptools-50.3.0-py37hb0f4dca_1
      six                pkgs/main/noarch::six-1.15.0-py_0
      sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
      tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
      tornado            pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
      traitlets          pkgs/main/noarch::traitlets-5.0.5-py_0
      wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
      wheel              pkgs/main/noarch::wheel-0.35.1-py_0
      xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
      zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
      zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
    
    
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done
    #
    # To activate this environment, use
    #
    #     $ conda activate myenv
    #
    # To deactivate an active environment, use
    #
    #     $ conda deactivate
    

 %% Cell type:markdown id: tags:

 ## Activating a conda environment

 It is not recommended to use `conda init bash`, because it will write that specific version into your `.bashrc` file and will result in errors when you move to another module or newer version.

 Hint: Checkout the guides on [Gitlab](https://gitlab.phaidra.org/imgw/computer-resources/-/blob/master/Python/QA-003-Conda-Environment.ipynb#Activate-your-conda-environment)

-### Load from the current conda module the BASH functionality (as in activate)
+### Load bash conda configuration
+This script is used by `conda init bash` to set the configuration, we simply source (load) it and get the same functionality as we would be using `conda init bash`. **(Do not use `conda init bash`)**

-```sh
-source $(which conda)/../etc/profile.d/conda.sh
+```bash
+source $(dirname $(which conda))/../etc/profile.d/conda.sh
 # now activating the environment works
 conda activate myenv
 ```
+optionaly you can add this bash function to your `.bashrc` for convenience:
+```bash
+conda_activate(){
+    if ! command -v conda &> /dev/null; then
+        echo "Load ana/miniconda module first"
+    else
+        if [ $# -ne 1 ]; then
+            echo "usage: conda_activate [env-name]"
+
+        else
+            source $(dirname $(which conda))/../etc/profile.d/conda.sh
+            conda activate $1
+        fi
+    fi
+}
+```
+After your `.bashrc` has been loaded (e.g. logout/login again) you should be able to use
+
+```bash
+module load anaconda3
+conda_activate myenv
+```

 ### Activate your kernel in your notebooks

 Please run the follwoing command and install the ipykernel, if it is not yet installed:

 %% Cell type:code id: tags:

 ``` python
 !~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"
 ```

 %% Output

    Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv

 %% Cell type:code id: tags:

 ``` python
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 ## Custom Environment Python Environment

 %% Cell type:code id: tags:

 ``` python
 # Setup a virtual Environment for your Python in the directoy: myfancyenv
 !python -m venv myfancyenv
 ```

 %% Cell type:code id: tags:

 ``` python
 # Install some package into that environment
 # important is ipykernel to use it within jupyterlab/notebook
 !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
 ```

 %% Output

    [33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.[0m

 %% Cell type:code id: tags:

 ``` python
 # Allow this environment to be available on the LAUNCHERS (+) sign on the left
 !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
 ```

 %% Output

    Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy

 %% Cell type:markdown id: tags:

 Wait a bit and the new python environment should be available on your launchers.

 %% Cell type:code id: tags:

 ``` python
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      fancy      /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 There you go. Your personal Environment.

 # If you need help. Feel free to ask me. Enjoy!