Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
Computer Resources
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Deploy
Releases
Package registry
Model registry
Operate
Terraform modules
Monitor
Service Desk
Analyze
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
IMGW
Computer Resources
Commits
eaf3bda9
Commit
eaf3bda9
authored
3 years ago
by
Michael Blaschek
Browse files
Options
Downloads
Patches
Plain Diff
Updated conda environment
parent
1e405b6e
Branches
Branches containing commit
Tags
Tags containing commit
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
Python/QA-003-Conda-Environment.ipynb
+71
-4
71 additions, 4 deletions
Python/QA-003-Conda-Environment.ipynb
with
71 additions
and
4 deletions
Python/QA-003-Conda-Environment.ipynb
+
71
−
4
View file @
eaf3bda9
...
...
@@ -4,7 +4,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Setting up a user environment with conda\n"
"# Setting up a user environment\n",
"\n",
"Topics\n",
"* Setup a user environment [using conda](#using-conda)\n",
" - [Add user environment to notebook kernels](#add-the-new-environment-to-your-active-kernels)\n",
" - [Activate your conda environment](#activate-your-conda-environment)\n",
"* Setup a user environment [using venv](#using-virtual-environment)\n",
"* Include `PATH` in [Notebook Kernel](#add-path-to-your-user-environment)\n"
]
},
{
...
...
@@ -26,6 +33,13 @@
"!module list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using conda"
]
},
{
"cell_type": "code",
"execution_count": 2,
...
...
@@ -231,6 +245,31 @@
"!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name \"Py3.7\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Activate your conda environment\n",
"Conda Environments need to be added to the `PATH` to work well. So there is an `activate` function for these conda environments.\n",
"\n",
"However, running\n",
"```bash\n",
"conda init bash\n",
"```\n",
"will edit your `.bashrc` and that is not nice, especially when you might move to another module. \n",
"**Therefore this is not recommended to do.**\n",
"\n",
"In order to use your environment in a shell script or on a slurm job, try the following:\n",
"\n",
"Load the shell setup script located in the conda source tree (`${CONDA_ROOT}/etc/profile.d/conda.sh`) and activate your environment:\n",
"```bash\n",
"# Load from the current conda module the BASH functionality (as in activate)\n",
"source $(which conda)/../etc/profile.d/conda.sh\n",
"# now activating the environment works\n",
"conda activate myenv\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 5,
...
...
@@ -255,7 +294,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"##
Custom Environment Python E
nvironment"
"##
Using virtual e
nvironment"
]
},
{
...
...
@@ -351,6 +390,34 @@
"!jupyter kernelspec list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Add PATH to your user environment\n",
"\n",
"In order to get full access to all tools installed in your local environment, add it to your `PATH`.\n",
"\n",
"Open your `kernel.json` file located `/jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json`:\n",
"```json\n",
"{\n",
" \"argv\": [\n",
" \"/jetfs/home/mblaschek/.conda/envs/myenv/bin/python\",\n",
" \"-m\",\n",
" \"ipykernel_launcher\",\n",
" \"-f\",\n",
" \"{connection_file}\"\n",
" ],\n",
" \"env\" : {\n",
" \"PATH\": \"/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}\",\n",
" \"LD_LIBRARY_PATH\": \"/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}\"\n",
" },\n",
" \"display_name\": \"MYENV\",\n",
" \"language\": \"python\"\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
...
...
@@ -361,7 +428,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "
Mini
conda3 (
4.8.2
)",
"display_name": "
Ana
conda3 (
2020.07
)",
"language": "python",
"name": "python3"
},
...
...
@@ -375,7 +442,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.
1
"
"version": "3.8.
3
"
}
},
"nbformat": 4,
...
...
%% Cell type:markdown id: tags:
# Setting up a user environment with conda
# Setting up a user environment
Topics
*
Setup a user environment
[
using conda
](
#using-conda
)
-
[
Add user environment to notebook kernels
](
#add-the-new-environment-to-your-active-kernels
)
-
[
Activate your conda environment
](
#activate-your-conda-environment
)
*
Setup a user environment
[
using venv
](
#using-virtual-environment
)
*
Include
`PATH`
in
[
Notebook Kernel
](
#add-path-to-your-user-environment
)
%% Cell type:code id: tags:
```
python
!
module
list
```
%% Output
Currently Loaded Modulefiles:[m
1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 [m
[K[?1l>
%% Cell type:markdown id: tags:
## Using conda
%% Cell type:code id: tags:
```
python
!
conda
info
```
%% Output
active environment : None
user config file : /jetfs/home/mblaschek/.condarc
populated config files : /jetfs/home/mblaschek/.condarc
conda version : 4.8.5
conda-build version : not installed
python version : 3.8.1.final.0
virtual packages : __glibc=2.28
base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
/jetfs/home/mblaschek/.conda/pkgs
envs directories : /jetfs/home/mblaschek/.conda/envs
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
platform : linux-64
user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
UID:GID : 54212:100
netrc file : None
offline mode : False
%% Cell type:code id: tags:
```
python
# Create a environment with conda
!
conda
create
-
y
-
n
myenv
python
=
3.7
numpy
ipykernel
```
%% Output
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.5
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /jetfs/home/mblaschek/.conda/envs/myenv
added / updated specs:
- ipykernel
- numpy
- python=3.7
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2020.11.8 | py37h06a4308_0 148 KB
ipython-7.19.0 | py37hb070fc8_0 978 KB
jupyter_core-4.7.0 | py37h06a4308_0 68 KB
numpy-1.19.2 | py37h54aff64_0 22 KB
numpy-base-1.19.2 | py37hfa32c7d_0 4.1 MB
pexpect-4.8.0 | pyhd3eb1b0_3 53 KB
pip-20.2.4 | py37h06a4308_0 1.7 MB
ptyprocess-0.6.0 | pyhd3eb1b0_2 20 KB
pygments-2.7.2 | pyhd3eb1b0_0 674 KB
setuptools-50.3.1 | py37h06a4308_1 711 KB
six-1.15.0 | py37h06a4308_0 27 KB
wheel-0.35.1 | pyhd3eb1b0_0 38 KB
------------------------------------------------------------
Total: 8.5 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
backcall pkgs/main/noarch::backcall-0.2.0-py_0
blas pkgs/main/linux-64::blas-1.0-mkl
ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0
certifi pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
decorator pkgs/main/noarch::decorator-4.4.2-py_0
intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254
ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
ipython pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
jedi pkgs/main/linux-64::jedi-0.17.2-py37_0
jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0
jupyter_core pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1
numpy pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
parso pkgs/main/noarch::parso-0.7.0-py_0
pexpect pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
pip pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
ptyprocess pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
pygments pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
python pkgs/main/linux-64::python-3.7.9-h7579374_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
readline pkgs/main/linux-64::readline-8.0-h7b6447c_0
setuptools pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
six pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0
tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
traitlets pkgs/main/noarch::traitlets-5.0.5-py_0
wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0
wheel pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Downloading and Extracting Packages
numpy-1.19.2 | 22 KB | ##################################### | 100%
pip-20.2.4 | 1.7 MB | ##################################### | 100%
numpy-base-1.19.2 | 4.1 MB | ##################################### | 100%
certifi-2020.11.8 | 148 KB | ##################################### | 100%
wheel-0.35.1 | 38 KB | ##################################### | 100%
ipython-7.19.0 | 978 KB | ##################################### | 100%
pexpect-4.8.0 | 53 KB | ##################################### | 100%
pygments-2.7.2 | 674 KB | ##################################### | 100%
ptyprocess-0.6.0 | 20 KB | ##################################### | 100%
jupyter_core-4.7.0 | 68 KB | ##################################### | 100%
setuptools-50.3.1 | 711 KB | ##################################### | 100%
six-1.15.0 | 27 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
%% Cell type:markdown id: tags:
### Add the new environment to your active kernels
This way you can choose the kernel from the Jupyterlab launcher.
%% Cell type:code id: tags:
```
python
!
~/
.
conda
/
envs
/
myenv
/
bin
/
python
-
m
ipykernel
install
--
user
--
name
MYENV
--
display
-
name
"
Py3.7
"
```
%% Output
Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
%% Cell type:markdown id: tags:
### Activate your conda environment
Conda Environments need to be added to the
`PATH`
to work well. So there is an
`activate`
function for these conda environments.
However, running
```
bash
conda init bash
```
will edit your
`.bashrc`
and that is not nice, especially when you might move to another module.
**Therefore this is not recommended to do.**
In order to use your environment in a shell script or on a slurm job, try the following:
Load the shell setup script located in the conda source tree (
`${CONDA_ROOT}/etc/profile.d/conda.sh`
) and activate your environment:
```
bash
# Load from the current conda module the BASH functionality (as in activate)
source
$(
which conda
)
/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv
```
%% Cell type:code id: tags:
```
python
# List available Kernels
!
jupyter
kernelspec
list
```
%% Output
Available kernels:
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags:
##
Custom Environment Python E
nvironment
##
Using virtual e
nvironment
%% Cell type:code id: tags:
```
python
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!
python
-
m
venv
myfancyenv
```
%% Cell type:code id: tags:
```
python
# Creates a directory with new environment
!
ls
myfancyenv
/
```
%% Output
bin include lib lib64 pyvenv.cfg
%% Cell type:code id: tags:
```
python
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!
.
/
myfancyenv
/
bin
/
pip
-
q
install
numpy
scipy
pandas
ipykernel
```
%% Output
[33mWARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.[0m
%% Cell type:code id: tags:
```
python
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!
.
/
myfancyenv
/
bin
/
python
-
m
ipykernel
install
--
user
--
name
fancy
--
display
-
name
"
My Python
"
```
%% Output
Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
%% Cell type:markdown id: tags:
Wait a bit and the new python environment should be available on your launchers.
%% Cell type:code id: tags:
```
python
!
jupyter
kernelspec
list
```
%% Output
Available kernels:
fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
%% Cell type:markdown id: tags:
## Add PATH to your user environment
In order to get full access to all tools installed in your local environment, add it to your
`PATH`
.
Open your
`kernel.json`
file located
`/jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json`
:
```
json
{
"argv"
:
[
"/jetfs/home/mblaschek/.conda/envs/myenv/bin/python"
,
"-m"
,
"ipykernel_launcher"
,
"-f"
,
"{connection_file}"
],
"env"
:
{
"PATH"
:
"/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}"
,
"LD_LIBRARY_PATH"
:
"/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
},
"display_name"
:
"MYENV"
,
"language"
:
"python"
}
```
%% Cell type:markdown id: tags:
### Finished
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment