added profling

d5baffd4 · Michael Blaschek · ed0c1288 · d5baffd4 · d5baffd4 · d5baffd4
Commit d5baffd4 authored 3 years ago by Michael Blaschek
--- a/Python/QA-003-Conda-Environment.ipynb
+++ b/Python/QA-003-Conda-Environment.ipynb
@@ -428,7 +428,7 @@
 ],
 "metadata": {
  "kernelspec": {
-   "display_name": "Anaconda3 (2020.07)",
+   "display_name": "Miniconda3 (4.10.3) - 3.9.5",
   "language": "python",
   "name": "python3"
  },
@@ -442,7 +442,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.8.3"
+   "version": "3.9.5"
  }
 },
 "nbformat": 4,

--- a/Python/QA-009-Memory-Profiling.ipynb
+++ b/Python/QA-009-Memory-Profiling.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7e3efaba-dc8d-4f02-b599-1d8e8d8a12e5",
+   "metadata": {},
+   "source": [
+    "# Memory Profiling in Python\n",
+    "\n",
+    "It is quite important to know what is going wrong in your code, to perform better.\n",
+    "One tool to help you do what you need to do is to use a memory profiler.\n",
+    "\n",
+    "- [PyPI - Memory Profiler](https://pypi.org/project/memory-profiler/)\n",
+    "\n",
+    "for execution speed use:\n",
+    "\n",
+    "- [PyPI - Line Profiler](https://github.com/pyutils/line_profiler)\n",
+    "- [CProfile](https://docs.python.org/3.8/library/profile.html)\n",
+    "\n",
+    "Install memory profile via pip:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "2132773b-1236-482b-8d2d-b1bd5e1a70d5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install --user --quiet memory_profiler"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "58a74d61-e7ab-4216-ba35-6de3f5aced5f",
+   "metadata": {},
+   "source": [
+    "After installing you might need to restart your kernel!\n",
+    "\n",
+    "\n",
+    "Write an example with a function and add the `@profile` decorator to indicate which function shall be profiled."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "e3df4ef0-c97a-49f7-b1a3-36f08d88f7be",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting example.py\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile example.py\n",
+    "\n",
+    "@profile\n",
+    "def my_func():\n",
+    "    a = [1] * (10 ** 6)\n",
+    "    b = [2] * (2 * 10 ** 7)\n",
+    "    del b\n",
+    "    return a\n",
+    "\n",
+    "if __name__ == '__main__':\n",
+    "    my_func()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1de3b0c2-27a1-4182-bbdc-1b67764e4d62",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Filename: example.py\n",
+      "\n",
+      "Line #    Mem usage    Increment  Occurences   Line Contents\n",
+      "============================================================\n",
+      "     2   39.406 MiB   39.406 MiB           1   @profile\n",
+      "     3                                         def my_func():\n",
+      "     4   47.141 MiB    7.734 MiB           1       a = [1] * (10 ** 6)\n",
+      "     5  199.766 MiB  152.625 MiB           1       b = [2] * (2 * 10 ** 7)\n",
+      "     6   47.254 MiB -152.512 MiB           1       del b\n",
+      "     7   47.254 MiB    0.000 MiB           1       return a\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Execute the module with the memory_profiler being loaded:\n",
+    "!python3 -m memory_profiler example.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5bdad5f1-fb41-45b1-8cb0-e0c93cf5834a",
+   "metadata": {},
+   "source": [
+    "Of course you can also add this directly into the file:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "6dda1f01-ae3b-4ccd-af78-6c5c3c227ad6",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting example.py\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile example.py\n",
+    "from memory_profiler import profile\n",
+    "\n",
+    "@profile\n",
+    "def my_func():\n",
+    "    a = [1] * (10 ** 6)\n",
+    "    b = [2] * (2 * 10 ** 7)\n",
+    "    del b\n",
+    "    return a\n",
+    "\n",
+    "if __name__ == '__main__':\n",
+    "    my_func()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "78036b7a-996e-43e5-88ba-c1f9120fc05b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Filename: /home/spack/computer-resources/Python/example.py\n",
+      "\n",
+      "Line #    Mem usage    Increment  Occurences   Line Contents\n",
+      "============================================================\n",
+      "     3     39.4 MiB     39.4 MiB           1   @profile\n",
+      "     4                                         def my_func():\n",
+      "     5     47.1 MiB      7.7 MiB           1       a = [1] * (10 ** 6)\n",
+      "     6    199.7 MiB    152.6 MiB           1       b = [2] * (2 * 10 ** 7)\n",
+      "     7     47.2 MiB   -152.5 MiB           1       del b\n",
+      "     8     47.2 MiB      0.0 MiB           1       return a\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "!python3 example.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5deb5183-9549-4ad3-90c0-2c51c8de783d",
+   "metadata": {},
+   "source": [
+    "It can be loaded into an ipython session with\n",
+    "`%load_ext memory_profiler`\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "ef87b3f1-1d2b-4f3d-a6bb-5458a3cbd4ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext memory_profiler"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "37eecb4f-b8b4-44c2-be4c-8845d3dc00bb",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting example.py\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile example.py\n",
+    "def my_func():\n",
+    "    a = [1] * (10 ** 6)\n",
+    "    b = [2] * (2 * 10 ** 7)\n",
+    "    del b\n",
+    "    return a\n",
+    "\n",
+    "if __name__ == '__main__':\n",
+    "    my_func()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "e6514424-09ff-494a-970d-9f32764ebd93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from example import my_func"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "6b1ed020-633e-4501-a3c3-a984ecab8bc4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "Filename: /home/spack/computer-resources/Python/example.py\n",
+       "\n",
+       "Line #    Mem usage    Increment  Occurences   Line Contents\n",
+       "============================================================\n",
+       "     1     52.2 MiB     52.2 MiB           1   def my_func():\n",
+       "     2     59.8 MiB      7.6 MiB           1       a = [1] * (10 ** 6)\n",
+       "     3    212.4 MiB    152.6 MiB           1       b = [2] * (2 * 10 ** 7)\n",
+       "     4     59.9 MiB   -152.6 MiB           1       del b\n",
+       "     5     59.9 MiB      0.0 MiB           1       return a"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%mprun -f my_func my_func()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "46720507-a09b-4efd-91be-4bf2bc802596",
+   "metadata": {},
+   "source": [
+    "## Line profiler\n",
+    "\n",
+    "Line profiler works similar."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "204afab1-04b8-4c48-b343-becd070371a7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install --user --quiet line_profiler"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "96c33187-40f9-4885-af21-2cc2b6bd4035",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext line_profiler"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d407f5d4-3aaa-4660-a9e5-432613dd5259",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Timer unit: 1e-06 s\n",
+       "\n",
+       "Total time: 0.127 s\n",
+       "File: /home/spack/computer-resources/Python/example.py\n",
+       "Function: my_func at line 1\n",
+       "\n",
+       "Line #      Hits         Time  Per Hit   % Time  Line Contents\n",
+       "==============================================================\n",
+       "     1                                           def my_func():\n",
+       "     2         1       4842.0   4842.0      3.8      a = [1] * (10 ** 6)\n",
+       "     3         1      78616.0  78616.0     61.9      b = [2] * (2 * 10 ** 7)\n",
+       "     4         1      43539.0  43539.0     34.3      del b\n",
+       "     5         1          3.0      3.0      0.0      return a"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%lprun -f my_func my_func()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b242ab95-7c07-4a98-bad4-1cf8693cdb58",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Miniconda3 (4.10.3) - 3.9.5",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
+%% Cell type:markdown id:7e3efaba-dc8d-4f02-b599-1d8e8d8a12e5 tags:
+
+# Memory Profiling in Python
+
+It is quite important to know what is going wrong in your code, to perform better.
+One tool to help you do what you need to do is to use a memory profiler.
+
+- [PyPI - Memory Profiler](https://pypi.org/project/memory-profiler/)
+
+for execution speed use:
+
+- [PyPI - Line Profiler](https://github.com/pyutils/line_profiler)
+- [CProfile](https://docs.python.org/3.8/library/profile.html)
+
+Install memory profile via pip:
+
+%% Cell type:code id:2132773b-1236-482b-8d2d-b1bd5e1a70d5 tags:
+
+``` python
+!pip install --user --quiet memory_profiler
+```
+
+%% Cell type:markdown id:58a74d61-e7ab-4216-ba35-6de3f5aced5f tags:
+
+After installing you might need to restart your kernel!
+
+
+Write an example with a function and add the `@profile` decorator to indicate which function shall be profiled.
+
+%% Cell type:code id:e3df4ef0-c97a-49f7-b1a3-36f08d88f7be tags:
+
+``` python
+%%writefile example.py
+
+@profile
+def my_func():
+    a = [1] * (10 ** 6)
+    b = [2] * (2 * 10 ** 7)
+    del b
+    return a
+
+if __name__ == '__main__':
+    my_func()
+```
+
+%% Output
+
+    Overwriting example.py
+
+%% Cell type:code id:1de3b0c2-27a1-4182-bbdc-1b67764e4d62 tags:
+
+``` python
+# Execute the module with the memory_profiler being loaded:
+!python3 -m memory_profiler example.py
+```
+
+%% Output
+
+    Filename: example.py
+    
+    Line #    Mem usage    Increment  Occurences   Line Contents
+    ============================================================
+         2   39.406 MiB   39.406 MiB           1   @profile
+         3                                         def my_func():
+         4   47.141 MiB    7.734 MiB           1       a = [1] * (10 ** 6)
+         5  199.766 MiB  152.625 MiB           1       b = [2] * (2 * 10 ** 7)
+         6   47.254 MiB -152.512 MiB           1       del b
+         7   47.254 MiB    0.000 MiB           1       return a
+    
+    
+
+%% Cell type:markdown id:5bdad5f1-fb41-45b1-8cb0-e0c93cf5834a tags:
+
+Of course you can also add this directly into the file:
+
+%% Cell type:code id:6dda1f01-ae3b-4ccd-af78-6c5c3c227ad6 tags:
+
+``` python
+%%writefile example.py
+from memory_profiler import profile
+
+@profile
+def my_func():
+    a = [1] * (10 ** 6)
+    b = [2] * (2 * 10 ** 7)
+    del b
+    return a
+
+if __name__ == '__main__':
+    my_func()
+```
+
+%% Output
+
+    Overwriting example.py
+
+%% Cell type:code id:78036b7a-996e-43e5-88ba-c1f9120fc05b tags:
+
+``` python
+!python3 example.py
+```
+
+%% Output
+
+    Filename: /home/spack/computer-resources/Python/example.py
+    
+    Line #    Mem usage    Increment  Occurences   Line Contents
+    ============================================================
+         3     39.4 MiB     39.4 MiB           1   @profile
+         4                                         def my_func():
+         5     47.1 MiB      7.7 MiB           1       a = [1] * (10 ** 6)
+         6    199.7 MiB    152.6 MiB           1       b = [2] * (2 * 10 ** 7)
+         7     47.2 MiB   -152.5 MiB           1       del b
+         8     47.2 MiB      0.0 MiB           1       return a
+    
+    
+
+%% Cell type:markdown id:5deb5183-9549-4ad3-90c0-2c51c8de783d tags:
+
+It can be loaded into an ipython session with
+`%load_ext memory_profiler`
+
+%% Cell type:code id:ef87b3f1-1d2b-4f3d-a6bb-5458a3cbd4ce tags:
+
+``` python
+%load_ext memory_profiler
+```
+
+%% Cell type:code id:37eecb4f-b8b4-44c2-be4c-8845d3dc00bb tags:
+
+``` python
+%%writefile example.py
+def my_func():
+    a = [1] * (10 ** 6)
+    b = [2] * (2 * 10 ** 7)
+    del b
+    return a
+
+if __name__ == '__main__':
+    my_func()
+```
+
+%% Output
+
+    Overwriting example.py
+
+%% Cell type:code id:e6514424-09ff-494a-970d-9f32764ebd93 tags:
+
+``` python
+from example import my_func
+```
+
+%% Cell type:code id:6b1ed020-633e-4501-a3c3-a984ecab8bc4 tags:
+
+``` python
+%mprun -f my_func my_func()
+```
+
+%% Output
+
+    
+
+
+%% Cell type:markdown id:46720507-a09b-4efd-91be-4bf2bc802596 tags:
+
+## Line profiler
+
+Line profiler works similar.
+
+%% Cell type:code id:204afab1-04b8-4c48-b343-becd070371a7 tags:
+
+``` python
+!pip install --user --quiet line_profiler
+```
+
+%% Cell type:code id:96c33187-40f9-4885-af21-2cc2b6bd4035 tags:
+
+``` python
+%load_ext line_profiler
+```
+
+%% Cell type:code id:d407f5d4-3aaa-4660-a9e5-432613dd5259 tags:
+
+``` python
+%lprun -f my_func my_func()
+```
+
+%% Output
+
+
+%% Cell type:code id:b242ab95-7c07-4a98-bad4-1cf8693cdb58 tags:
+
+``` python
+```
--- a/Python/README.md
+++ b/Python/README.md
@@ -60,3 +60,7 @@ Ever needed to backup a conda environment or copy from a colleague or designing

 You just need to add a configuration option and then you can open the Dashboard in any Jupyterhub. Works on SRVX1 and JET.

+## Q: How to profile memory and exeution time of functions?
+[Profile](/QA-009-Memory-Profiling.ipynb)
+
+If you need to get a better understanding of you functions memory and execution time, try these profiling options.
\ No newline at end of file
--- a/Python/Your-First-Notebook-onJet_v2.ipynb
+++ b/Python/Your-First-Notebook-onJet_v2.ipynb
@@ -350,9 +350,9 @@
    " - [x] NetCDF (Parallel (NC3), NC4->HDF5)\n",
    " - [x] Eccodes (2.18)\n",
    " - [x] CDO\n",
-    " - [ ] Intel compiled Libraries (License) \n",
-    " - [ ] Emoslib \n",
-    " - [ ] Ecaccess\n",
+    " - [x] Intel compiled Libraries (License) \n",
+    " - [x] Emoslib \n",
+    " - [x] Ecaccess\n",
    " - [ ] RTTOV"
   ]
  },
@@ -404,8 +404,10 @@
   "source": [
    "import os\n",
    "import sys\n",
+    "import site\n",
    "# Need to update search Path / only necessary here for the first time\n",
-    "sys.path.append(os.path.expandvars(\"$HOME/.local/lib/python3.8/site-packages\"))"
+    "# sys.path.append(os.path.expandvars(\"$HOME/.local/lib/python3.8/site-packages\"))\n",
+    "sys.path.append(site.getusersitepackages())"
   ]
  },
  {
@@ -563,6 +565,29 @@
    "!conda create -y -n myenv python=3.7 numpy ipykernel"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Activating a conda environment\n",
+    "\n",
+    "It is not recommended to use `conda init bash`, because it will write that specific version into your `.bashrc` file and will result in errors when you move to another module or newer version.\n",
+    "\n",
+    "Hint: Checkout the guides on [Gitlab](https://gitlab.phaidra.org/imgw/computer-resources/-/blob/master/Python/QA-003-Conda-Environment.ipynb#Activate-your-conda-environment)\n",
+    "\n",
+    "### Load from the current conda module the BASH functionality (as in activate)\n",
+    "\n",
+    "```sh\n",
+    "source $(which conda)/../etc/profile.d/conda.sh\n",
+    "# now activating the environment works\n",
+    "conda activate myenv\n",
+    "```\n",
+    "\n",
+    "### Activate your kernel in your notebooks\n",
+    "\n",
+    "Please run the follwoing command and install the ipykernel, if it is not yet installed:"
+   ]
+  },
  {
   "cell_type": "code",
   "execution_count": 17,
@@ -693,7 +718,7 @@
 ],
 "metadata": {
  "kernelspec": {
-   "display_name": "Miniconda3 (4.8.2)",
+   "display_name": "Miniconda3 (4.10.3) - 3.9.5",
   "language": "python",
   "name": "python3"
  },
@@ -707,7 +732,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.8.1"
+   "version": "3.9.5"
  }
 },
 "nbformat": 4,

 %% Cell type:markdown id: tags:

 # Welcome to Jet

 <img src='https://homepage.univie.ac.at/michael.blaschek/openfiles/rocket.png' width='100px'><img src='https://homepage.univie.ac.at/michael.blaschek/openfiles/uni.jpg' width='400px'>

 You have done it! Your Notebook Server is running on the Jet Cluster.

 Welcome, please skip this, if your know where you are, as this is meant as an introduction. However, not complete as it is.
 If you need more infos about Notebooks have a look at [my Homepage](https://homepage.univie.ac.at/michael.blaschek/#python) and there are numerous Tutorials on the Internet for any advice, but feel free to ask me and give me feedback.

 Help can be found here:
 * Goto Help -> JupyterLab Reference
 * Goto Help -> Jupyter Reference
 * [Tour](https://jupyterlab.readthedocs.io/en/stable/user/interface.html)


 In the following I will show you some useful things.

 First: This Notebook is in your Home directory, but also on `/jetfs/scratch` for any reason.

 %% Cell type:markdown id: tags:

 ## Check your current Host

 %% Cell type:code id: tags:

 ``` python
 !hostnamectl
 ```

 %% Output

       Static hostname: localhost.localdomain
    Transient hostname: jet03.jet.local
             Icon name: computer-server
               Chassis: server
      Operating System: ]8;;https://www.centos.org/CentOS Linux 8 (Core)]8;;
           CPE OS Name: cpe:/o:centos:centos:8
                Kernel: Linux 4.18.0-147.el8.x86_64
          Architecture: x86-64

 %% Cell type:markdown id: tags:

 ## Who is running this process

 %% Cell type:code id: tags:

 ``` python
 !whoami
 ```

 %% Output

    mblaschek

 %% Cell type:code id: tags:

 ``` python
 # Where are you
 !pwd
 ```

 %% Output

    /jetfs/home/mblaschek/Documents

 %% Cell type:code id: tags:

 ``` python
 # Show my Current Jobs in the Queue
 !squeue -u $(id -u) -l
 ```

 %% Output

    Fri Oct 16 23:10:58 2020
                 JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
                   235   compute jupyters mblasche  RUNNING    6:29:07   8:00:00      1 jet03

 %% Cell type:markdown id: tags:

 There you can see your `TIME_LIMIT` of 8 hours again and how much time has passed on your current jobs

 %% Cell type:code id: tags:

 ``` python
 # Import sys and check the executable (so what python you are using)
 import sys
 print(sys.executable)
 ```

 %% Output

    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin/python

 %% Cell type:markdown id: tags:

 **Notice the funny path?**

    Yes this is SPACK your nice scientific package manager.
    Allows to build custom libraries and applications tailored to our needs!
    Please not that software currently installed in /opt on JET01 is not available here.
    Software under /opt is deprecated and will be removed after a full transition
    to the current libraries

 More information on [SPACK](https://spack.io/)

 ## Check your environmental PATH Variable, could be useful to know

 %% Cell type:code id: tags:

 ``` python
 %env PATH
 ```

 %% Output

    '/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin:/bin:/jetfs/spack/bin:/jetfs/userservices:/opt/jupyterhub/bin:/opt/slurm/bin:/sbin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/share/Modules/bin::/jetfs/home/mblaschek/bin:/jetfs/home/mblaschek/.local/bin'

 %% Cell type:markdown id: tags:

 ## Check what modules are currently loaded in this Notebook session

 %% Cell type:code id: tags:

 ``` python
 !module list --no-pager
 ```

 %% Output

    Currently Loaded Modulefiles:
     1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2

 %% Cell type:markdown id: tags:

 Put this file `load_modules_into_jupyter.conf` into your Home directory and your Notebook environment will have these modules loaded.
 It is not possible to load these modules via magic (python magic ;).
 **It needs the empty line at the end...**

 %% Cell type:code id: tags:

 ``` python
 %%writefile ~/load_modules_into_jupyter.conf
 eccodes/2.18.0-gcc-8.3.1-s7clum3
 ```

 %% Output

    Writing /jetfs/home/mblaschek/load_modules_into_jupyter.conf

 %% Cell type:code id: tags:

 ``` python
 !cat ~/load_modules_into_jupyter.conf
 ```

 %% Output

    eccodes/2.18.0-gcc-8.3.1-s7clum3

 %% Cell type:markdown id: tags:

 **Restarting the kernel will show now loaded modules**

 %% Cell type:code id: tags:

 ``` python
 # after restarting
 !module list --no-pager
 ```

 %% Output

    Currently Loaded Modulefiles:
     1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
     2) openmpi/4.0.5-gcc-8.3.1-773ztsv
     3) hdf5/1.12.0-gcc-8.3.1-awl4atl
     4) parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn
     5) netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
     6) eccodes/2.18.0-gcc-8.3.1-s7clum3

 %% Cell type:markdown id: tags:

 ## Currently available Modules

 %% Cell type:code id: tags:

 ``` python
 # --no-pager is needed only inside notebook
 !module avail --no-pager
 ```

 %% Output

    --------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
    anaconda2/2019.10-gcc-8.3.1-5pou6ji
    anaconda3/2019.10-gcc-8.3.1-tmy5mgp
    anaconda3/2020.07-gcc-8.3.1-weugqkf
    cdo/1.9.8-gcc-8.3.1-ipgvzeh
    eccodes/2.18.0-gcc-8.3.1-s7clum3
    hdf5/1.12.0-gcc-8.3.1-awl4atl
    intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw
    miniconda2/4.7.12.1-gcc-8.3.1-zduqggv
    miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
    netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
    netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj
    openmpi/3.1.6-gcc-8.3.1-rk5av53
    openmpi/4.0.5-gcc-8.3.1-773ztsv
    parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu
    parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn
    
    --------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
    anaconda2/2019.10-gcc-8.3.1-5pou6ji
    anaconda3/2019.10-gcc-8.3.1-tmy5mgp
    anaconda3/2020.07-gcc-8.3.1-weugqkf
    cdo/1.9.8-gcc-8.3.1-ipgvzeh
    eccodes/2.18.0-gcc-8.3.1-s7clum3
    hdf5/1.12.0-gcc-8.3.1-awl4atl
    intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw
    miniconda2/4.7.12.1-gcc-8.3.1-zduqggv
    miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
    netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k
    netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj
    openmpi/3.1.6-gcc-8.3.1-rk5av53
    openmpi/4.0.5-gcc-8.3.1-773ztsv
    parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu
    parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn

 %% Cell type:markdown id: tags:

 # Software on JET
 Currently:
 - [x] Anaconda (2,3)
 - [x] Miniconda (2,3)
 - [x] OPENMPI (4.0.5, 3.6.1) optimized for JET
 - [x] HDF5 (1.12) optimized with OPENMPI
 - [x] NetCDF (Parallel (NC3), NC4->HDF5)
 - [x] Eccodes (2.18)
 - [x] CDO
- - [ ] Intel compiled Libraries (License)
- - [ ] Emoslib
- - [ ] Ecaccess
+ - [x] Intel compiled Libraries (License)
+ - [x] Emoslib
+ - [x] Ecaccess
 - [ ] RTTOV

 %% Cell type:markdown id: tags:

 ## Currently there are a few choices:

 1. Python (2, 3)
    * Anaconda
    * Miniconda
 2. Julia (not yet)
 3. R (not yet)

 %% Cell type:markdown id: tags:

 ## Install Python packages

 %% Cell type:code id: tags:

 ``` python
 # Please use the --user flag to install the packages into .local/lib/python...
 %pip install --user numpy scipy
 ```

 %% Output

    Requirement already satisfied: numpy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.19.2)
    Requirement already satisfied: scipy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.5.2)
    Note: you may need to restart the kernel to use updated packages.

 %% Cell type:code id: tags:

 ``` python
 import os
 import sys
+import site
 # Need to update search Path / only necessary here for the first time
-sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
+# sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
+sys.path.append(site.getusersitepackages())
 ```

 %% Cell type:code id: tags:

 ``` python
 import numpy as np
 import scipy as sp

 # WORKS! ;)
 ```

 %% Cell type:markdown id: tags:

 ## Using conda

 %% Cell type:code id: tags:

 ``` python
 !conda info
 ```

 %% Output

    
         active environment : None
           user config file : /jetfs/home/mblaschek/.condarc
     populated config files : /jetfs/home/mblaschek/.condarc
              conda version : 4.8.5
        conda-build version : not installed
             python version : 3.8.1.final.0
           virtual packages : __glibc=2.28
           base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv  (read only)
               channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                              https://repo.anaconda.com/pkgs/main/noarch
                              https://repo.anaconda.com/pkgs/r/linux-64
                              https://repo.anaconda.com/pkgs/r/noarch
              package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
                              /jetfs/home/mblaschek/.conda/pkgs
           envs directories : /jetfs/home/mblaschek/.conda/envs
                              /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
                   platform : linux-64
                 user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
                    UID:GID : 54212:100
                 netrc file : None
               offline mode : False
    

 %% Cell type:code id: tags:

 ``` python
 # Create a environment with conda
 !conda create -y -n myenv python=3.7 numpy ipykernel
 ```

 %% Output

    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
    Collecting package metadata (repodata.json): done
    Solving environment: done
    
    ## Package Plan ##
    
      environment location: /jetfs/home/mblaschek/.conda/envs/myenv
    
      added / updated specs:
        - ipykernel
        - numpy
        - python=3.7
    
    
    The following NEW packages will be INSTALLED:
    
      _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
      backcall           pkgs/main/noarch::backcall-0.2.0-py_0
      blas               pkgs/main/linux-64::blas-1.0-mkl
      ca-certificates    pkgs/main/linux-64::ca-certificates-2020.10.14-0
      certifi            pkgs/main/linux-64::certifi-2020.6.20-py37_0
      decorator          pkgs/main/noarch::decorator-4.4.2-py_0
      intel-openmp       pkgs/main/linux-64::intel-openmp-2020.2-254
      ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
      ipython            pkgs/main/linux-64::ipython-7.18.1-py37h5ca1d4c_0
      ipython_genutils   pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
      jedi               pkgs/main/linux-64::jedi-0.17.2-py37_0
      jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
      jupyter_core       pkgs/main/linux-64::jupyter_core-4.6.3-py37_0
      ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
      libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
      libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
      libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
      libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
      libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
      mkl                pkgs/main/linux-64::mkl-2020.2-256
      mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
      mkl_fft            pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
      mkl_random         pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
      ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
      numpy              pkgs/main/linux-64::numpy-1.19.1-py37hbc911f0_0
      numpy-base         pkgs/main/linux-64::numpy-base-1.19.1-py37hfa32c7d_0
      openssl            pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
      parso              pkgs/main/noarch::parso-0.7.0-py_0
      pexpect            pkgs/main/linux-64::pexpect-4.8.0-py37_1
      pickleshare        pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
      pip                pkgs/main/linux-64::pip-20.2.3-py37_0
      prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
      ptyprocess         pkgs/main/linux-64::ptyprocess-0.6.0-py37_0
      pygments           pkgs/main/noarch::pygments-2.7.1-py_0
      python             pkgs/main/linux-64::python-3.7.9-h7579374_0
      python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
      pyzmq              pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
      readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
      setuptools         pkgs/main/linux-64::setuptools-50.3.0-py37hb0f4dca_1
      six                pkgs/main/noarch::six-1.15.0-py_0
      sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
      tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
      tornado            pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
      traitlets          pkgs/main/noarch::traitlets-5.0.5-py_0
      wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
      wheel              pkgs/main/noarch::wheel-0.35.1-py_0
      xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
      zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
      zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
    
    
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done
    #
    # To activate this environment, use
    #
    #     $ conda activate myenv
    #
    # To deactivate an active environment, use
    #
    #     $ conda deactivate
    

+%% Cell type:markdown id: tags:
+
+## Activating a conda environment
+
+It is not recommended to use `conda init bash`, because it will write that specific version into your `.bashrc` file and will result in errors when you move to another module or newer version.
+
+Hint: Checkout the guides on [Gitlab](https://gitlab.phaidra.org/imgw/computer-resources/-/blob/master/Python/QA-003-Conda-Environment.ipynb#Activate-your-conda-environment)
+
+### Load from the current conda module the BASH functionality (as in activate)
+
+```sh
+source $(which conda)/../etc/profile.d/conda.sh
+# now activating the environment works
+conda activate myenv
+```
+
+### Activate your kernel in your notebooks
+
+Please run the follwoing command and install the ipykernel, if it is not yet installed:
+
 %% Cell type:code id: tags:

 ``` python
 !~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"
 ```

 %% Output

    Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv

 %% Cell type:code id: tags:

 ``` python
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 ## Custom Environment Python Environment

 %% Cell type:code id: tags:

 ``` python
 # Setup a virtual Environment for your Python in the directoy: myfancyenv
 !python -m venv myfancyenv
 ```

 %% Cell type:code id: tags:

 ``` python
 # Install some package into that environment
 # important is ipykernel to use it within jupyterlab/notebook
 !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
 ```

 %% Output

    [33mWARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.[0m

 %% Cell type:code id: tags:

 ``` python
 # Allow this environment to be available on the LAUNCHERS (+) sign on the left
 !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
 ```

 %% Output

    Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy

 %% Cell type:markdown id: tags:

 Wait a bit and the new python environment should be available on your launchers.

 %% Cell type:code id: tags:

 ``` python
 !jupyter kernelspec list
 ```

 %% Output

    Available kernels:
      fancy      /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
      myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
      python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

 %% Cell type:markdown id: tags:

 There you go. Your personal Environment.

 # If you need help. Feel free to ask me. Enjoy!