Skip to content
Snippets Groups Projects
Commit c72bc531 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

added conda environments

parent e4309f2b
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id:18953a0d-48d2-454f-9402-40cd0aac6bdb tags:
# Restore / Save Conda Environments
Sometimes it just works and you do not want to change anything.
This is when you can create an environment file and safe your current python installed packges and versions.
What you can do:
- Backup an existing environment
- Restore an environment
- Design an environment
%% Cell type:markdown id:83eb685d-a53c-4816-86ea-de9501fb9665 tags:
# Backup
It is a good choice to create a backup of a working environment. Regardless if you plan on setting it up somewhere else or not.
%% Cell type:code id:e7c0b076-e7d2-48e1-a177-c55a4451040d tags:
``` python
!module list
```
%% Output
Currently Loaded Modulefiles:
1) anaconda3/2020.07-gcc-8.4.1 3) proj/8.1.0-gcc-8.4.1 
2) geos/3.9.1-gcc-8.4.1 
[?1l>
%% Cell type:code id:8932eb21-2025-40ec-8744-2730b19455eb tags:
``` python
# Check What environment are present
# Typically there are some root installed environments
!conda env list
```
%% Output
# conda environments:
#
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/anaconda3-2020.07-l2bohj4adsd6r2oweeytdzrgqmjl64lt
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-5qezliv3bkrc63cv2zk2lyh3xioozd45
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.9.2-htox4jjktvvm7b4d3rfwpwpwa4i2kkhk
/home/spack/envs/test
/home/swd/anaconda3
/home/swd/manual/intelpython/intelpython/latest
/home/swd/manual/intelpython/intelpython/latest/envs/2021.4.0
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda2-2019.10-eaw5h5kknfwa3bsjnjhqfhytzfxgd3nf
base * /home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.07-4onhdcbnvyvwfdxxbelrplxjihjsknex
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.11-v46okvjlotnor7yrmpqnbn63xt3a2qyq
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2021.05-hrg2ocjseksc4af3osaiipzu433xctef
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda2-4.7.12.1-3r6cvxekrhf32f6tdeout3aq7xxsgjit
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda3-4.10.3-l23y36yispi4ue5w7mhyc4mllio4ylfm
%% Cell type:markdown id:6253ef9a-6761-497e-9921-863320f6ae2f tags:
But there is one called **test** in `/home/spack/envs/test` which we are going to backup.
%% Cell type:code id:735145c2-c24d-4006-9751-9c2a5f684387 tags:
``` python
# Check What is installed inside your environment
!conda list -p /home/spack/envs/test
```
%% Output
# packages in environment at /home/spack/envs/test:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
ca-certificates 2021.9.30 h06a4308_1
certifi 2021.10.8 py39h06a4308_0
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libstdcxx-ng 9.3.0 hd4cf53a_17
ncurses 6.2 he6710b0_1
openssl 1.1.1l h7f8727e_0
pip 21.2.4 py39h06a4308_0
python 3.9.7 h12debd9_1
readline 8.1 h27cfd23_0
setuptools 58.0.4 py39h06a4308_0
sqlite 3.36.0 hc218d9a_0
tk 8.6.11 h1ccaba5_0
tzdata 2021a h5d7bf9c_0
wheel 0.37.0 pyhd3eb1b0_1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
%% Cell type:code id:08bcab6f-7056-44ea-8363-af7718c4dd3d tags:
``` python
# Export the definition (versions, packages) to a YAML file
!conda env export -p /home/spack/envs/test > environment.yaml
```
%% Cell type:code id:4ec66832-8a9a-446e-ae4a-7de85fad29d7 tags:
``` python
!cat environment.yaml
```
%% Output
name: null
channels:
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- _openmp_mutex=4.5=1_gnu
- ca-certificates=2021.9.30=h06a4308_1
- certifi=2021.10.8=py39h06a4308_0
- ld_impl_linux-64=2.35.1=h7274673_9
- libffi=3.3=he6710b0_2
- libgcc-ng=9.3.0=h5101ec6_17
- libgomp=9.3.0=h5101ec6_17
- libstdcxx-ng=9.3.0=hd4cf53a_17
- ncurses=6.2=he6710b0_1
- openssl=1.1.1l=h7f8727e_0
- pip=21.2.4=py39h06a4308_0
- python=3.9.7=h12debd9_1
- readline=8.1=h27cfd23_0
- setuptools=58.0.4=py39h06a4308_0
- sqlite=3.36.0=hc218d9a_0
- tk=8.6.11=h1ccaba5_0
- tzdata=2021a=h5d7bf9c_0
- wheel=0.37.0=pyhd3eb1b0_1
- xz=5.2.5=h7b6447c_0
- zlib=1.2.11=h7b6447c_3
prefix: /home/spack/envs/test
%% Cell type:markdown id:4f984479-c901-4876-9ea5-b11918336b8d tags:
**Note: Versions and packages are listed. As well as the path where it was installed**
With that file you will be able to rebuild this environment at anytime.
If you want to use `pip` later to restore your environment, you can use
```bash
./envs/test/bin/pip list format=freeze > requirements.txt
```
However, this is quite different from conda. You need to have the correct version of python available otherwise you can not restore the environment.
%% Cell type:markdown id:5cbfcf62-c832-4a34-9e36-bc0e4244f0c3 tags:
## Restore
Maybe you get a environment file from a colleague or it's your backup, there are ways to restore your environment.
Please note that conda is slow and resolving dependencies can be slow.
%% Cell type:code id:04145eb1-b368-405e-898c-1694f485b57c tags:
``` python
# Create the environment in a special path
!conda env create -p /home/spack/envs/restore -f environment.yaml
```
%% Output
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.3
latest version: 4.10.3
Please update conda by running
$ conda update -n base -c defaults conda
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate /home/spack/envs/restore
#
# To deactivate an active environment, use
#
# $ conda deactivate
%% Cell type:code id:72e85fe7-ed74-4f1c-974b-b328533b52a0 tags:
``` python
!conda env list
```
%% Output
# conda environments:
#
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/anaconda3-2020.07-l2bohj4adsd6r2oweeytdzrgqmjl64lt
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-5qezliv3bkrc63cv2zk2lyh3xioozd45
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.9.2-htox4jjktvvm7b4d3rfwpwpwa4i2kkhk
/home/spack/envs/restore
/home/spack/envs/test
/home/swd/anaconda3
/home/swd/manual/intelpython/intelpython/latest
/home/swd/manual/intelpython/intelpython/latest/envs/2021.4.0
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda2-2019.10-eaw5h5kknfwa3bsjnjhqfhytzfxgd3nf
base * /home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.07-4onhdcbnvyvwfdxxbelrplxjihjsknex
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.11-v46okvjlotnor7yrmpqnbn63xt3a2qyq
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2021.05-hrg2ocjseksc4af3osaiipzu433xctef
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda2-4.7.12.1-3r6cvxekrhf32f6tdeout3aq7xxsgjit
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda3-4.10.3-l23y36yispi4ue5w7mhyc4mllio4ylfm
%% Cell type:markdown id:1ffa23fe-1b8c-45af-9547-6295edcec219 tags:
Please note that we have another environment like this, called restore:
`/home/spack/envs/restore`
%% Cell type:code id:45e09ec6-6670-4dc0-bef1-3dcc1e6504ec tags:
``` python
!conda list -p /home/spack/envs/restore
```
%% Output
# packages in environment at /home/spack/envs/restore:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
ca-certificates 2021.9.30 h06a4308_1
certifi 2021.10.8 py39h06a4308_0
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libstdcxx-ng 9.3.0 hd4cf53a_17
ncurses 6.2 he6710b0_1
openssl 1.1.1l h7f8727e_0
pip 21.2.4 py39h06a4308_0
python 3.9.7 h12debd9_1
readline 8.1 h27cfd23_0
setuptools 58.0.4 py39h06a4308_0
sqlite 3.36.0 hc218d9a_0
tk 8.6.11 h1ccaba5_0
tzdata 2021a h5d7bf9c_0
wheel 0.37.0 pyhd3eb1b0_1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
%% Cell type:markdown id:3c241577-5e41-40d8-9b3e-fd8874118d20 tags:
When you want to use pip to restore the environment:
```bash
# create a virtual environment
python3 -m venv env
# use that to install the packages
./env/bin/pip install -r requirements.txt
```
Here it will use the python3 that you give, meaning that it will not be Python 3.9 as we specified above.
Download the correct python interpreter first. [here](https://www.python.org/downloads/) or any other way.
%% Cell type:markdown id:7b665eb3-d531-4288-9e58-b2e29ec48e74 tags:
# Design
It is possible to write your own environment file and use it for build a special environment.
%% Cell type:markdown id:b23ecff1-a329-42df-b2e6-57cc8d6d480a tags:
There are three things you need:
- name
- channels (mostly conda-forge and defaults)
- dependencies (packages)
An example yaml file
```yaml
name: selfdesigned
channels:
- conda-forge
- defaults
dependencies:
- numpy
- pandas
- xarray
- pip:
- metpy
```
you can specify versions using `numpy=1.0` or `python=3.6`. If no version is given the newest compatible version will be used.
%% Cell type:code id:fe80dc6b-aa2a-40d2-b73e-67acf5b487c2 tags:
``` python
%%writefile design.yaml
name: selfdesigned
channels:
- conda-forge
- defaults
dependencies:
- numpy
- pandas
- xarray
- pip:
- metpy
```
%% Output
Writing design.yaml
%% Cell type:code id:3d821016-50d6-4696-ac47-06755d57e990 tags:
``` python
!conda env create -f design.yaml -p /home/spack/envs/design
```
%% Output
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.3
latest version: 4.10.3
Please update conda by running
$ conda update -n base -c defaults conda
Downloading and Extracting Packages
pandas-1.3.3 | 13.0 MB | ##################################### | 100%
numpy-1.21.2 | 6.2 MB | ##################################### | 100%
ncurses-6.2 | 985 KB | ##################################### | 100%
libcblas-3.9.0 | 11 KB | ##################################### | 100%
pytz-2021.3 | 242 KB | ##################################### | 100%
ld_impl_linux-64-2.3 | 667 KB | ##################################### | 100%
_libgcc_mutex-0.1 | 3 KB | ##################################### | 100%
sqlite-3.36.0 | 1.4 MB | ##################################### | 100%
tk-8.6.11 | 3.3 MB | ##################################### | 100%
pip-21.3 | 1.2 MB | ##################################### | 100%
libblas-3.9.0 | 12 KB | ##################################### | 100%
python_abi-3.9 | 4 KB | ##################################### | 100%
python-3.9.7 | 27.5 MB | ##################################### | 100%
tzdata-2021c | 120 KB | ##################################### | 100%
wheel-0.37.0 | 31 KB | ##################################### | 100%
libgfortran5-11.2.0 | 1.7 MB | ##################################### | 100%
zlib-1.2.11 | 86 KB | ##################################### | 100%
libopenblas-0.3.17 | 9.2 MB | ##################################### | 100%
libstdcxx-ng-11.2.0 | 4.2 MB | ##################################### | 100%
libgfortran-ng-11.2. | 19 KB | ##################################### | 100%
_openmp_mutex-4.5 | 22 KB | ##################################### | 100%
six-1.16.0 | 14 KB | ##################################### | 100%
libffi-3.4.2 | 57 KB | ##################################### | 100%
python-dateutil-2.8. | 240 KB | ##################################### | 100%
ca-certificates-2021 | 139 KB | ##################################### | 100%
libgcc-ng-11.2.0 | 887 KB | ##################################### | 100%
xz-5.2.5 | 343 KB | ##################################### | 100%
libzlib-1.2.11 | 59 KB | ##################################### | 100%
xarray-0.19.0 | 614 KB | ##################################### | 100%
setuptools-58.2.0 | 1011 KB | ##################################### | 100%
openssl-3.0.0 | 2.9 MB | ##################################### | 100%
readline-8.1 | 295 KB | ##################################### | 100%
libgomp-11.2.0 | 427 KB | ##################################### | 100%
liblapack-3.9.0 | 11 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Ran pip subprocess with arguments:
['/home/spack/envs/design/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt']
Pip subprocess output:
Collecting metpy
Using cached MetPy-1.1.0-py3-none-any.whl (361 kB)
Collecting pooch>=0.1
Using cached pooch-1.5.2-py3-none-any.whl (57 kB)
Collecting scipy>=1.2.0
Downloading scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.5 MB)
Collecting pint>=0.10.1
Using cached Pint-0.17-py2.py3-none-any.whl (204 kB)
Requirement already satisfied: pandas>=0.24.0 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.3.3)
Requirement already satisfied: numpy>=1.17.0 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.21.2)
Collecting matplotlib>=3.0.1
Downloading matplotlib-3.4.3-cp39-cp39-manylinux1_x86_64.whl (10.3 MB)
Requirement already satisfied: xarray>=0.14.1 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (0.19.0)
Collecting traitlets>=4.3.0
Downloading traitlets-5.1.0-py3-none-any.whl (101 kB)
Collecting pyproj>=2.3.0
Downloading pyproj-3.2.1-cp39-cp39-manylinux2010_x86_64.whl (6.3 MB)
Collecting pyparsing>=2.2.1
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.3.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Collecting pillow>=6.2.0
Downloading Pillow-8.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Requirement already satisfied: python-dateutil>=2.7 in /home/spack/envs/design/lib/python3.9/site-packages (from matplotlib>=3.0.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (2.8.2)
Collecting cycler>=0.10
Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Requirement already satisfied: pytz>=2017.3 in /home/spack/envs/design/lib/python3.9/site-packages (from pandas>=0.24.0->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (2021.3)
Collecting packaging
Using cached packaging-21.0-py3-none-any.whl (40 kB)
Collecting appdirs
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting requests
Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting certifi
Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Requirement already satisfied: setuptools>=40.4 in /home/spack/envs/design/lib/python3.9/site-packages (from xarray>=0.14.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (58.2.0)
Requirement already satisfied: six in /home/spack/envs/design/lib/python3.9/site-packages (from cycler>=0.10->matplotlib>=3.0.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.16.0)
Collecting idna<4,>=2.5
Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.7-py3-none-any.whl (38 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Installing collected packages: urllib3, pyparsing, idna, charset-normalizer, certifi, requests, pillow, packaging, kiwisolver, cycler, appdirs, traitlets, scipy, pyproj, pooch, pint, matplotlib, metpy
Successfully installed appdirs-1.4.4 certifi-2021.10.8 charset-normalizer-2.0.7 cycler-0.10.0 idna-3.3 kiwisolver-1.3.2 matplotlib-3.4.3 metpy-1.1.0 packaging-21.0 pillow-8.3.2 pint-0.17 pooch-1.5.2 pyparsing-2.4.7 pyproj-3.2.1 requests-2.26.0 scipy-1.7.1 traitlets-5.1.0 urllib3-1.26.7
#
# To activate this environment, use
#
# $ conda activate /home/spack/envs/design
#
# To deactivate an active environment, use
#
# $ conda deactivate
%% Cell type:code id:c1d727e8-a97f-4809-b695-d10be49e1b4a tags:
``` python
!/home/spack/envs/design/bin/pip list
```
%% Output
Package Version
------------------ ---------
appdirs 1.4.4
certifi 2021.10.8
charset-normalizer 2.0.7
cycler 0.10.0
idna 3.3
kiwisolver 1.3.2
matplotlib 3.4.3
MetPy 1.1.0
numpy 1.21.2
packaging 21.0
pandas 1.3.3
Pillow 8.3.2
Pint 0.17
pip 21.3
pooch 1.5.2
pyparsing 2.4.7
pyproj 3.2.1
python-dateutil 2.8.2
pytz 2021.3
requests 2.26.0
scipy 1.7.1
setuptools 58.2.0
six 1.16.0
traitlets 5.1.0
urllib3 1.26.7
wheel 0.37.0
xarray 0.19.0
%% Cell type:markdown id:21a756ab-392b-4238-9272-5494fd8730c8 tags:
FIN
...@@ -36,7 +36,6 @@ The SRVX1 is the central access point to IMG services: ...@@ -36,7 +36,6 @@ The SRVX1 is the central access point to IMG services:
goto: [srvx.img.univie.ac.at](https://srvx1.img.univie.ac.at) goto: [srvx.img.univie.ac.at](https://srvx1.img.univie.ac.at)
Currently running: Currently running:
- TeachingHub - TeachingHub
- ResearchHub
- Webdata - Webdata
## Jupyterhub ## Jupyterhub
...@@ -44,7 +43,8 @@ Currently running: ...@@ -44,7 +43,8 @@ Currently running:
SRVX1 serves a teaching [jupyterhub](https://jupyterhub.readthedocs.io/en/stable/) with a [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/). It allows easy access for students and teachers. SRVX1 serves a teaching [jupyterhub](https://jupyterhub.readthedocs.io/en/stable/) with a [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/). It allows easy access for students and teachers.
Goto: [](https://srvx1.img.univie.ac.at/hub) Goto: [https://srvx1.img.univie.ac.at/hub](https://srvx1.img.univie.ac.at/hub)
Signup is only granted by teachers and requires a srvx1 user account. A new password is needed and a TOTP (time base one-time password) will be created. Signup is only granted by teachers and requires a srvx1 user account. A new password is needed and a TOTP (time base one-time password) will be created.
Download/Use any of these required Authenticator Apps: Download/Use any of these required Authenticator Apps:
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment