Skip to content
Snippets Groups Projects
Commit eaf3bda9 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

Updated conda environment

parent 1e405b6e
Branches
Tags
No related merge requests found
%% Cell type:markdown id: tags:
# Setting up a user environment with conda
# Setting up a user environment
Topics
* Setup a user environment [using conda](#using-conda)
- [Add user environment to notebook kernels](#add-the-new-environment-to-your-active-kernels)
- [Activate your conda environment](#activate-your-conda-environment)
* Setup a user environment [using venv](#using-virtual-environment)
* Include `PATH` in [Notebook Kernel](#add-path-to-your-user-environment)
%% Cell type:code id: tags:
``` python
!module list
```
%% Output
Currently Loaded Modulefiles:
1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 
[?1l>
%% Cell type:markdown id: tags:
## Using conda
%% Cell type:code id: tags:
``` python
!conda info
```
%% Output
active environment : None
user config file : /jetfs/home/mblaschek/.condarc
populated config files : /jetfs/home/mblaschek/.condarc
conda version : 4.8.5
conda-build version : not installed
python version : 3.8.1.final.0
virtual packages : __glibc=2.28
base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
/jetfs/home/mblaschek/.conda/pkgs
envs directories : /jetfs/home/mblaschek/.conda/envs
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
platform : linux-64
user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
UID:GID : 54212:100
netrc file : None
offline mode : False
%% Cell type:code id: tags:
``` python
# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel
```
%% Output
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.5
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /jetfs/home/mblaschek/.conda/envs/myenv
added / updated specs:
- ipykernel
- numpy
- python=3.7
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2020.11.8 | py37h06a4308_0 148 KB
ipython-7.19.0 | py37hb070fc8_0 978 KB
jupyter_core-4.7.0 | py37h06a4308_0 68 KB
numpy-1.19.2 | py37h54aff64_0 22 KB
numpy-base-1.19.2 | py37hfa32c7d_0 4.1 MB
pexpect-4.8.0 | pyhd3eb1b0_3 53 KB
pip-20.2.4 | py37h06a4308_0 1.7 MB
ptyprocess-0.6.0 | pyhd3eb1b0_2 20 KB
pygments-2.7.2 | pyhd3eb1b0_0 674 KB
setuptools-50.3.1 | py37h06a4308_1 711 KB
six-1.15.0 | py37h06a4308_0 27 KB
wheel-0.35.1 | pyhd3eb1b0_0 38 KB
------------------------------------------------------------
Total: 8.5 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
backcall pkgs/main/noarch::backcall-0.2.0-py_0
blas pkgs/main/linux-64::blas-1.0-mkl
ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0
certifi pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
decorator pkgs/main/noarch::decorator-4.4.2-py_0
intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254
ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
ipython pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
jedi pkgs/main/linux-64::jedi-0.17.2-py37_0
jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0
jupyter_core pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1
numpy pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
parso pkgs/main/noarch::parso-0.7.0-py_0
pexpect pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
pip pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
ptyprocess pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
pygments pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
python pkgs/main/linux-64::python-3.7.9-h7579374_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
readline pkgs/main/linux-64::readline-8.0-h7b6447c_0
setuptools pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
six pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0
tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
traitlets pkgs/main/noarch::traitlets-5.0.5-py_0
wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0
wheel pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Downloading and Extracting Packages
numpy-1.19.2 | 22 KB | ##################################### | 100%
pip-20.2.4 | 1.7 MB | ##################################### | 100%
numpy-base-1.19.2 | 4.1 MB | ##################################### | 100%
certifi-2020.11.8 | 148 KB | ##################################### | 100%
wheel-0.35.1 | 38 KB | ##################################### | 100%
ipython-7.19.0 | 978 KB | ##################################### | 100%
pexpect-4.8.0 | 53 KB | ##################################### | 100%
pygments-2.7.2 | 674 KB | ##################################### | 100%
ptyprocess-0.6.0 | 20 KB | ##################################### | 100%
jupyter_core-4.7.0 | 68 KB | ##################################### | 100%
setuptools-50.3.1 | 711 KB | ##################################### | 100%
six-1.15.0 | 27 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
%% Cell type:markdown id: tags:
### Add the new environment to your active kernels
This way you can choose the kernel from the Jupyterlab launcher.
%% Cell type:code id: tags:
``` python
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"
```
%% Output
Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
%% Cell type:markdown id: tags:
### Activate your conda environment
Conda Environments need to be added to the `PATH` to work well. So there is an `activate` function for these conda environments.
However, running
```bash
conda init bash
```
will edit your `.bashrc` and that is not nice, especially when you might move to another module.
**Therefore this is not recommended to do.**
In order to use your environment in a shell script or on a slurm job, try the following:
Load the shell setup script located in the conda source tree (`${CONDA_ROOT}/etc/profile.d/conda.sh`) and activate your environment:
```bash
# Load from the current conda module the BASH functionality (as in activate)
source $(which conda)/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv
```
%% Cell type:code id: tags:
``` python
# List available Kernels
!jupyter kernelspec list
```
%% Output
Available kernels:
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags:
## Custom Environment Python Environment
## Using virtual environment
%% Cell type:code id: tags:
``` python
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv
```
%% Cell type:code id: tags:
``` python
# Creates a directory with new environment
!ls myfancyenv/
```
%% Output
bin include lib lib64 pyvenv.cfg
%% Cell type:code id: tags:
``` python
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
```
%% Output
WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
%% Cell type:code id: tags:
``` python
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
```
%% Output
Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
%% Cell type:markdown id: tags:
Wait a bit and the new python environment should be available on your launchers.
%% Cell type:code id: tags:
``` python
!jupyter kernelspec list
```
%% Output
Available kernels:
fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags:
## Add PATH to your user environment
In order to get full access to all tools installed in your local environment, add it to your `PATH`.
Open your `kernel.json` file located `/jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json`:
```json
{
"argv": [
"/jetfs/home/mblaschek/.conda/envs/myenv/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env" : {
"PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}",
"LD_LIBRARY_PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
},
"display_name": "MYENV",
"language": "python"
}
```
%% Cell type:markdown id: tags:
### Finished
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment