Update Python/QA-005-Remote-Kernels.ipynb

289587a5 · Michael Blaschek · 6baab017 · 289587a5
Commit 289587a5 authored 4 years ago by Michael Blaschek
--- a/Python/QA-005-Remote-Kernels.ipynb
+++ b/Python/QA-005-Remote-Kernels.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Connecting to remote IPython Kernels\n",
+    "\n",
+    "In order to connect a few things need to be setup:\n",
+    "1. install Jupyter Console\n",
+    "2. install Remote Ikernel\n",
+    "3. configure the kernel\n",
+    "\n",
+    "These commands will install the required packages into your `~/.local/bin` and `~/.local/lib` directories by default in Linux/Mac. \n",
+    "If you are using the **anaconda** then the packages will be installed there.\n",
+    "\n",
+    "Make sure that the `/bin` path is in your `$PATH` environment variable.\n",
+    "\n",
+    "## Install the required packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip3 -q install jupyter-console"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip3 -q install remote_ikernel"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "env: PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n"
+     ]
+    }
+   ],
+   "source": [
+    "# this might not be necessary for everyone\n",
+    "# adjust the path accordingly to make sure that jupyter-kernelspec, jupyter and remote_ikernel can be found.\n",
+    "%env PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configure remote connection\n",
+    "\n",
+    "Depending on your install, you might have one or two selections here. `jupyter-kernelspec` shows you the kernels that you can use. The one you installed the above packages with should be here. We don't need these anyway. Just so you know about that."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Available kernels:\r\n",
+      "  anaconda3    /home/mblaschek/.local/share/jupyter/kernels/anaconda3\r\n",
+      "  python3      /home/mblaschek/.local/share/jupyter/kernels/python3\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!jupyter-kernelspec list"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      " \"argv\": [\r\n",
+      "  \"/home/mblaschek/anaconda3/bin/python\",\r\n",
+      "  \"-m\",\r\n",
+      "  \"ipykernel_launcher\",\r\n",
+      "  \"-f\",\r\n",
+      "  \"{connection_file}\"\r\n",
+      " ],\r\n",
+      " \"display_name\": \"Python 3(Ana)\",\r\n",
+      " \"language\": \"python\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "# This is the relevant file.\n",
+    "# It contains the path to the executable and some names\n",
+    "!cat /home/mblaschek/.local/share/jupyter/kernels/anaconda3/kernel.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Adding a new kernel\n",
+    "\n",
+    "on SRVX1, SRVX2, SRVX8 it is possible to launch a script called `kernel-launcher`.\n",
+    "\n",
+    "    kernel-launcher [module] [executable] [options]\n",
+    "    \n",
+    "typically we launch this with python processes such as\n",
+    "\n",
+    "    kernel-launcher miniconda3 ipython \n",
+    "    \n",
+    "### Modules available on SRVX1\n",
+    "```bash\n",
+    ">>> module av\n",
+    "\n",
+    "--- /home/opt/spack/share/spack/modules/linux-centos6-sandybridge ---\n",
+    "anaconda3/2020.07-gcc-5.3.0     git/2.29.0-gcc-5.3.0        netcdf-fortran/4.5.3-gcc-5.3.0  \n",
+    "eccodes/2.18.0-gcc-5.3.0        hdf5/1.10.7-gcc-5.3.0       openmpi/3.1.6-gcc-5.3.0         \n",
+    "enstools/2020.11.dev-gcc-5.3.0  miniconda3/4.8.2-gcc-5.3.0  proj/7.1.0-gcc-5.3.0            \n",
+    "gcc/5.3.0-gcc-5.3.0             netcdf-c/4.7.4-gcc-5.3.0    zlib/1.2.11-gcc-5.3.0    \n",
+    "```\n",
+    "\n",
+    "### Modules available on SRVX8\n",
+    "```bash\n",
+    ">>> module av\n",
+    "\n",
+    "--- /opt/spack/share/spack/modules/linux-centos6-haswell/ ---\n",
+    "anaconda3/2020.07-gcc-5.3.0    hdf5/1.10.7-gcc-5.3.0          openmpi/3.1.6-gcc-5.3.0\n",
+    "eccodes/2.18.0-gcc-5.3.0       miniconda3/4.8.2-gcc-5.3.0     proj/7.1.0-gcc-5.3.0\n",
+    "gcc/4.4.7-gcc-5.3.0            netcdf-c/4.7.4-gcc-5.3.0       python/3.8.5-gcc-5.3.0\n",
+    "gcc/5.3.0-gcc-5.3.0            netcdf-fortran/4.5.3-gcc-5.3.0 zlib/1.2.11-gcc-5.3.0\n",
+    "```\n",
+    "\n",
+    "In order to use these different modules one has to specify them in the call:\n",
+    "\n",
+    "1. Module: `miniconda3`\n",
+    "2. Name: `PyS1` for Python on srvx1\n",
+    "3. Host: `mblaschek@srvx1.img.univie.ac.at` username and server\n",
+    "\n",
+    "In order to allow easy access adding the public key to ssh authentication is recommended. Otherwise set `export SSH_ASKPASS=1` and you should be prompted each time the connection is established."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Added kernel ['rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1']: SSH mblaschek@srvx1.img.univie.ac.at PyS1.\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!remote_ikernel manage \\\n",
+    "    --add \\\n",
+    "    --kernel_cmd='kernel-launcher miniconda3 ipython kernel -f {connection_file}' \\\n",
+    "    --name='PyS1' \\\n",
+    "    --interface=ssh \\\n",
+    "    --host=mblaschek@srvx1.img.univie.ac.at"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Available kernels:\r\n",
+      "  anaconda3                                        /home/mblaschek/.local/share/jupyter/kernels/anaconda3\r\n",
+      "  python3                                          /home/mblaschek/.local/share/jupyter/kernels/python3\r\n",
+      "  rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1    /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Here we see the newly added kernel\n",
+    "!jupyter-kernelspec list"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "  \"argv\": [\r\n",
+      "    \"/usr/bin/python3\",\r\n",
+      "    \"-m\",\r\n",
+      "    \"remote_ikernel\",\r\n",
+      "    \"--interface\",\r\n",
+      "    \"ssh\",\r\n",
+      "    \"--host\",\r\n",
+      "    \"mblaschek@srvx1.img.univie.ac.at\",\r\n",
+      "    \"--kernel_cmd\",\r\n",
+      "    \"kernel-launcher miniconda3 ipython kernel -f {host_connection_file}\",\r\n",
+      "    \"{connection_file}\"\r\n",
+      "  ],\r\n",
+      "  \"display_name\": \"SSH mblaschek@srvx1.img.univie.ac.at PyS1\",\r\n",
+      "  \"remote_ikernel_argv\": [\r\n",
+      "    \"/home/mblaschek/.local/bin/remote_ikernel\",\r\n",
+      "    \"manage\",\r\n",
+      "    \"--add\",\r\n",
+      "    \"--kernel_cmd=kernel-launcher miniconda3 ipython kernel -f {connection_file}\",\r\n",
+      "    \"--name=PyS1\",\r\n",
+      "    \"--interface=ssh\",\r\n",
+      "    \"--host=mblaschek@srvx1.img.univie.ac.at\"\r\n",
+      "  ]\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "# Here we can see that the remote_ikernel actually does.\n",
+    "!cat  /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1/kernel.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# this will launch the anaconda3 kernel from the list\n",
+    "!jupyter console --existing=anaconda3"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This will launch the remote kernel in a console\n",
+    "!jupyter console --existing=rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "of course the remote kernel can be used in a local jupyter notebook or lab.\n",
+    "\n",
+    "## Remarks\n",
+    "\n",
+    "When running kernels on remote machines, the notebooks themselves will be saved onto the local filesystem, but the kernel will only have access to filesystem of the remote machine running the kernel. If you need shared directories, set up `sshfs` between your machines."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
+%% Cell type:markdown id: tags:
+
+# Connecting to remote IPython Kernels
+
+In order to connect a few things need to be setup:
+1. install Jupyter Console
+2. install Remote Ikernel
+3. configure the kernel
+
+These commands will install the required packages into your `~/.local/bin` and `~/.local/lib` directories by default in Linux/Mac.
+If you are using the **anaconda** then the packages will be installed there.
+
+Make sure that the `/bin` path is in your `$PATH` environment variable.
+
+## Install the required packages
+
+%% Cell type:code id: tags:
+
+``` python
+!pip3 -q install jupyter-console
+```
+
+%% Cell type:code id: tags:
+
+``` python
+!pip3 -q install remote_ikernel
+```
+
+%% Cell type:code id: tags:
+
+``` python
+# this might not be necessary for everyone
+# adjust the path accordingly to make sure that jupyter-kernelspec, jupyter and remote_ikernel can be found.
+%env PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+```
+
+%% Output
+
+    env: PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+%% Cell type:markdown id: tags:
+
+## Configure remote connection
+
+Depending on your install, you might have one or two selections here. `jupyter-kernelspec` shows you the kernels that you can use. The one you installed the above packages with should be here. We don't need these anyway. Just so you know about that.
+
+%% Cell type:code id: tags:
+
+``` python
+!jupyter-kernelspec list
+```
+
+%% Output
+
+    Available kernels:
+      anaconda3    /home/mblaschek/.local/share/jupyter/kernels/anaconda3
+      python3      /home/mblaschek/.local/share/jupyter/kernels/python3
+
+%% Cell type:code id: tags:
+
+``` python
+# This is the relevant file.
+# It contains the path to the executable and some names
+!cat /home/mblaschek/.local/share/jupyter/kernels/anaconda3/kernel.json
+```
+
+%% Output
+
+    {
+     "argv": [
+      "/home/mblaschek/anaconda3/bin/python",
+      "-m",
+      "ipykernel_launcher",
+      "-f",
+      "{connection_file}"
+     ],
+     "display_name": "Python 3(Ana)",
+     "language": "python"
+    }
+
+%% Cell type:markdown id: tags:
+
+## Adding a new kernel
+
+on SRVX1, SRVX2, SRVX8 it is possible to launch a script called `kernel-launcher`.
+
+    kernel-launcher [module] [executable] [options]
+
+typically we launch this with python processes such as
+
+    kernel-launcher miniconda3 ipython
+
+### Modules available on SRVX1
+```bash
+>>> module av
+
+--- /home/opt/spack/share/spack/modules/linux-centos6-sandybridge ---
+anaconda3/2020.07-gcc-5.3.0     git/2.29.0-gcc-5.3.0        netcdf-fortran/4.5.3-gcc-5.3.0
+eccodes/2.18.0-gcc-5.3.0        hdf5/1.10.7-gcc-5.3.0       openmpi/3.1.6-gcc-5.3.0
+enstools/2020.11.dev-gcc-5.3.0  miniconda3/4.8.2-gcc-5.3.0  proj/7.1.0-gcc-5.3.0
+gcc/5.3.0-gcc-5.3.0             netcdf-c/4.7.4-gcc-5.3.0    zlib/1.2.11-gcc-5.3.0
+```
+
+### Modules available on SRVX8
+```bash
+>>> module av
+
+--- /opt/spack/share/spack/modules/linux-centos6-haswell/ ---
+anaconda3/2020.07-gcc-5.3.0    hdf5/1.10.7-gcc-5.3.0          openmpi/3.1.6-gcc-5.3.0
+eccodes/2.18.0-gcc-5.3.0       miniconda3/4.8.2-gcc-5.3.0     proj/7.1.0-gcc-5.3.0
+gcc/4.4.7-gcc-5.3.0            netcdf-c/4.7.4-gcc-5.3.0       python/3.8.5-gcc-5.3.0
+gcc/5.3.0-gcc-5.3.0            netcdf-fortran/4.5.3-gcc-5.3.0 zlib/1.2.11-gcc-5.3.0
+```
+
+In order to use these different modules one has to specify them in the call:
+
+1. Module: `miniconda3`
+2. Name: `PyS1` for Python on srvx1
+3. Host: `mblaschek@srvx1.img.univie.ac.at` username and server
+
+In order to allow easy access adding the public key to ssh authentication is recommended. Otherwise set `export SSH_ASKPASS=1` and you should be prompted each time the connection is established.
+
+%% Cell type:code id: tags:
+
+``` python
+!remote_ikernel manage \
+    --add \
+    --kernel_cmd='kernel-launcher miniconda3 ipython kernel -f {connection_file}' \
+    --name='PyS1' \
+    --interface=ssh \
+    --host=mblaschek@srvx1.img.univie.ac.at
+```
+
+%% Output
+
+    Added kernel ['rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1']: SSH mblaschek@srvx1.img.univie.ac.at PyS1.
+
+%% Cell type:code id: tags:
+
+``` python
+# Here we see the newly added kernel
+!jupyter-kernelspec list
+```
+
+%% Output
+
+    Available kernels:
+      anaconda3                                        /home/mblaschek/.local/share/jupyter/kernels/anaconda3
+      python3                                          /home/mblaschek/.local/share/jupyter/kernels/python3
+      rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1    /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1
+
+%% Cell type:code id: tags:
+
+``` python
+# Here we can see that the remote_ikernel actually does.
+!cat  /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1/kernel.json
+```
+
+%% Output
+
+    {
+      "argv": [
+        "/usr/bin/python3",
+        "-m",
+        "remote_ikernel",
+        "--interface",
+        "ssh",
+        "--host",
+        "mblaschek@srvx1.img.univie.ac.at",
+        "--kernel_cmd",
+        "kernel-launcher miniconda3 ipython kernel -f {host_connection_file}",
+        "{connection_file}"
+      ],
+      "display_name": "SSH mblaschek@srvx1.img.univie.ac.at PyS1",
+      "remote_ikernel_argv": [
+        "/home/mblaschek/.local/bin/remote_ikernel",
+        "manage",
+        "--add",
+        "--kernel_cmd=kernel-launcher miniconda3 ipython kernel -f {connection_file}",
+        "--name=PyS1",
+        "--interface=ssh",
+        "--host=mblaschek@srvx1.img.univie.ac.at"
+      ]
+    }
+
+%% Cell type:code id: tags:
+
+``` python
+# this will launch the anaconda3 kernel from the list
+!jupyter console --existing=anaconda3
+```
+
+%% Cell type:code id: tags:
+
+``` python
+# This will launch the remote kernel in a console
+!jupyter console --existing=rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1
+```
+
+%% Cell type:markdown id: tags:
+
+of course the remote kernel can be used in a local jupyter notebook or lab.
+
+## Remarks
+
+When running kernels on remote machines, the notebooks themselves will be saved onto the local filesystem, but the kernel will only have access to filesystem of the remote machine running the kernel. If you need shared directories, set up `sshfs` between your machines.
+
+%% Cell type:code id: tags:
+
+``` python
+```