Skip to content
Snippets Groups Projects
Commit eaf3bda9 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

Updated conda environment

parent 1e405b6e
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Setting up a user environment with conda # Setting up a user environment
Topics
* Setup a user environment [using conda](#using-conda)
- [Add user environment to notebook kernels](#add-the-new-environment-to-your-active-kernels)
- [Activate your conda environment](#activate-your-conda-environment)
* Setup a user environment [using venv](#using-virtual-environment)
* Include `PATH` in [Notebook Kernel](#add-path-to-your-user-environment)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!module list !module list
``` ```
%% Output %% Output
Currently Loaded Modulefiles: Currently Loaded Modulefiles:
1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2  1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 
[?1l> [?1l>
%% Cell type:markdown id: tags:
## Using conda
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!conda info !conda info
``` ```
%% Output %% Output
active environment : None active environment : None
user config file : /jetfs/home/mblaschek/.condarc user config file : /jetfs/home/mblaschek/.condarc
populated config files : /jetfs/home/mblaschek/.condarc populated config files : /jetfs/home/mblaschek/.condarc
conda version : 4.8.5 conda version : 4.8.5
conda-build version : not installed conda-build version : not installed
python version : 3.8.1.final.0 python version : 3.8.1.final.0
virtual packages : __glibc=2.28 virtual packages : __glibc=2.28
base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only) base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/r/noarch
package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
/jetfs/home/mblaschek/.conda/pkgs /jetfs/home/mblaschek/.conda/pkgs
envs directories : /jetfs/home/mblaschek/.conda/envs envs directories : /jetfs/home/mblaschek/.conda/envs
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
platform : linux-64 platform : linux-64
user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28 user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
UID:GID : 54212:100 UID:GID : 54212:100
netrc file : None netrc file : None
offline mode : False offline mode : False
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Create a environment with conda # Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel !conda create -y -n myenv python=3.7 numpy ipykernel
``` ```
%% Output %% Output
Collecting package metadata (current_repodata.json): done Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done Collecting package metadata (repodata.json): done
Solving environment: done Solving environment: done
==> WARNING: A newer version of conda exists. <== ==> WARNING: A newer version of conda exists. <==
current version: 4.8.5 current version: 4.8.5
latest version: 4.9.2 latest version: 4.9.2
Please update conda by running Please update conda by running
$ conda update -n base -c defaults conda $ conda update -n base -c defaults conda
## Package Plan ## ## Package Plan ##
environment location: /jetfs/home/mblaschek/.conda/envs/myenv environment location: /jetfs/home/mblaschek/.conda/envs/myenv
added / updated specs: added / updated specs:
- ipykernel - ipykernel
- numpy - numpy
- python=3.7 - python=3.7
The following packages will be downloaded: The following packages will be downloaded:
package | build package | build
---------------------------|----------------- ---------------------------|-----------------
certifi-2020.11.8 | py37h06a4308_0 148 KB certifi-2020.11.8 | py37h06a4308_0 148 KB
ipython-7.19.0 | py37hb070fc8_0 978 KB ipython-7.19.0 | py37hb070fc8_0 978 KB
jupyter_core-4.7.0 | py37h06a4308_0 68 KB jupyter_core-4.7.0 | py37h06a4308_0 68 KB
numpy-1.19.2 | py37h54aff64_0 22 KB numpy-1.19.2 | py37h54aff64_0 22 KB
numpy-base-1.19.2 | py37hfa32c7d_0 4.1 MB numpy-base-1.19.2 | py37hfa32c7d_0 4.1 MB
pexpect-4.8.0 | pyhd3eb1b0_3 53 KB pexpect-4.8.0 | pyhd3eb1b0_3 53 KB
pip-20.2.4 | py37h06a4308_0 1.7 MB pip-20.2.4 | py37h06a4308_0 1.7 MB
ptyprocess-0.6.0 | pyhd3eb1b0_2 20 KB ptyprocess-0.6.0 | pyhd3eb1b0_2 20 KB
pygments-2.7.2 | pyhd3eb1b0_0 674 KB pygments-2.7.2 | pyhd3eb1b0_0 674 KB
setuptools-50.3.1 | py37h06a4308_1 711 KB setuptools-50.3.1 | py37h06a4308_1 711 KB
six-1.15.0 | py37h06a4308_0 27 KB six-1.15.0 | py37h06a4308_0 27 KB
wheel-0.35.1 | pyhd3eb1b0_0 38 KB wheel-0.35.1 | pyhd3eb1b0_0 38 KB
------------------------------------------------------------ ------------------------------------------------------------
Total: 8.5 MB Total: 8.5 MB
The following NEW packages will be INSTALLED: The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main _libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
backcall pkgs/main/noarch::backcall-0.2.0-py_0 backcall pkgs/main/noarch::backcall-0.2.0-py_0
blas pkgs/main/linux-64::blas-1.0-mkl blas pkgs/main/linux-64::blas-1.0-mkl
ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0 ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0
certifi pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0 certifi pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
decorator pkgs/main/noarch::decorator-4.4.2-py_0 decorator pkgs/main/noarch::decorator-4.4.2-py_0
intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254 intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254
ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0 ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
ipython pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0 ipython pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0 ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
jedi pkgs/main/linux-64::jedi-0.17.2-py37_0 jedi pkgs/main/linux-64::jedi-0.17.2-py37_0
jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0 jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0
jupyter_core pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0 jupyter_core pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7 ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1 libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2 libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0 libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0 libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0 libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
mkl pkgs/main/linux-64::mkl-2020.2-256 mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0 mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0 mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0 mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1 ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1
numpy pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0 numpy pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0 numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0 openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
parso pkgs/main/noarch::parso-0.7.0-py_0 parso pkgs/main/noarch::parso-0.7.0-py_0
pexpect pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3 pexpect pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001 pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
pip pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0 pip pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0 prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
ptyprocess pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2 ptyprocess pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
pygments pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0 pygments pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
python pkgs/main/linux-64::python-3.7.9-h7579374_0 python pkgs/main/linux-64::python-3.7.9-h7579374_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0 python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1 pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
readline pkgs/main/linux-64::readline-8.0-h7b6447c_0 readline pkgs/main/linux-64::readline-8.0-h7b6447c_0
setuptools pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1 setuptools pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
six pkgs/main/linux-64::six-1.15.0-py37h06a4308_0 six pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0 sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0 tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0
tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1 tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
traitlets pkgs/main/noarch::traitlets-5.0.5-py_0 traitlets pkgs/main/noarch::traitlets-5.0.5-py_0
wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0 wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0
wheel pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0 wheel pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0 xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3 zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3 zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Downloading and Extracting Packages Downloading and Extracting Packages
numpy-1.19.2 | 22 KB | ##################################### | 100% numpy-1.19.2 | 22 KB | ##################################### | 100%
pip-20.2.4 | 1.7 MB | ##################################### | 100% pip-20.2.4 | 1.7 MB | ##################################### | 100%
numpy-base-1.19.2 | 4.1 MB | ##################################### | 100% numpy-base-1.19.2 | 4.1 MB | ##################################### | 100%
certifi-2020.11.8 | 148 KB | ##################################### | 100% certifi-2020.11.8 | 148 KB | ##################################### | 100%
wheel-0.35.1 | 38 KB | ##################################### | 100% wheel-0.35.1 | 38 KB | ##################################### | 100%
ipython-7.19.0 | 978 KB | ##################################### | 100% ipython-7.19.0 | 978 KB | ##################################### | 100%
pexpect-4.8.0 | 53 KB | ##################################### | 100% pexpect-4.8.0 | 53 KB | ##################################### | 100%
pygments-2.7.2 | 674 KB | ##################################### | 100% pygments-2.7.2 | 674 KB | ##################################### | 100%
ptyprocess-0.6.0 | 20 KB | ##################################### | 100% ptyprocess-0.6.0 | 20 KB | ##################################### | 100%
jupyter_core-4.7.0 | 68 KB | ##################################### | 100% jupyter_core-4.7.0 | 68 KB | ##################################### | 100%
setuptools-50.3.1 | 711 KB | ##################################### | 100% setuptools-50.3.1 | 711 KB | ##################################### | 100%
six-1.15.0 | 27 KB | ##################################### | 100% six-1.15.0 | 27 KB | ##################################### | 100%
Preparing transaction: done Preparing transaction: done
Verifying transaction: done Verifying transaction: done
Executing transaction: done Executing transaction: done
# #
# To activate this environment, use # To activate this environment, use
# #
# $ conda activate myenv # $ conda activate myenv
# #
# To deactivate an active environment, use # To deactivate an active environment, use
# #
# $ conda deactivate # $ conda deactivate
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### Add the new environment to your active kernels ### Add the new environment to your active kernels
This way you can choose the kernel from the Jupyterlab launcher. This way you can choose the kernel from the Jupyterlab launcher.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7" !~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"
``` ```
%% Output %% Output
Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
%% Cell type:markdown id: tags:
### Activate your conda environment
Conda Environments need to be added to the `PATH` to work well. So there is an `activate` function for these conda environments.
However, running
```bash
conda init bash
```
will edit your `.bashrc` and that is not nice, especially when you might move to another module.
**Therefore this is not recommended to do.**
In order to use your environment in a shell script or on a slurm job, try the following:
Load the shell setup script located in the conda source tree (`${CONDA_ROOT}/etc/profile.d/conda.sh`) and activate your environment:
```bash
# Load from the current conda module the BASH functionality (as in activate)
source $(which conda)/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv
```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# List available Kernels # List available Kernels
!jupyter kernelspec list !jupyter kernelspec list
``` ```
%% Output %% Output
Available kernels: Available kernels:
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3 python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Custom Environment Python Environment ## Using virtual environment
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Setup a virtual Environment for your Python in the directoy: myfancyenv # Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv !python -m venv myfancyenv
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Creates a directory with new environment # Creates a directory with new environment
!ls myfancyenv/ !ls myfancyenv/
``` ```
%% Output %% Output
bin include lib lib64 pyvenv.cfg bin include lib lib64 pyvenv.cfg
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Install some package into that environment # Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook # important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
``` ```
%% Output %% Output
WARNING: You are using pip version 19.2.3, however version 20.2.4 is available. WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command. You should consider upgrading via the 'pip install --upgrade pip' command.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# Allow this environment to be available on the LAUNCHERS (+) sign on the left # Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python" !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
``` ```
%% Output %% Output
Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
Wait a bit and the new python environment should be available on your launchers. Wait a bit and the new python environment should be available on your launchers.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
!jupyter kernelspec list !jupyter kernelspec list
``` ```
%% Output %% Output
Available kernels: Available kernels:
fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3 python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Add PATH to your user environment
In order to get full access to all tools installed in your local environment, add it to your `PATH`.
Open your `kernel.json` file located `/jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json`:
```json
{
"argv": [
"/jetfs/home/mblaschek/.conda/envs/myenv/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env" : {
"PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}",
"LD_LIBRARY_PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
},
"display_name": "MYENV",
"language": "python"
}
```
%% Cell type:markdown id: tags:
### Finished ### Finished
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment