Skip to content
Snippets Groups Projects
Commit 34b6235c authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

updates and changes in 2024

parent 78de83b6
Branches
Tags
No related merge requests found
sort_type: natural
nav: nav:
- README.md - README.md
- Getting Started.md - Getting Started.md
- Students
- SSH-VPN-VNC
- Servers - Servers
- VSC.md - VSC.md
- ECMWF.md - ECMWF
- WRF.md
- TeachingHub.md
- SSH-VPN-VNC
- ... - ...
# Data at ECMWF
## MARS
![MARS request tree](../mkdocs/img/MARS-tree.png)
How to make a mars request
## GRIB
How to handle grib files.
Example using Python and Fortran90.
\ No newline at end of file
# European Center for Medium-Range Weather Forecast # European Center for Medium-Range Weather Forecast
<img src="./mkdocs/img/logo_ecmwf.png" width="400px"> <img src="../mkdocs/img/logo_ecmwf.png" width="400px">
[website](https://www.ecmwf.int) / [service status](https://www.ecmwf.int/en/service-status) / [confluence](https://confluence.ecmwf.int) / [support](https://support.ecmwf.int) / [accounting](https://www.ecmwf.int/user) [website](https://www.ecmwf.int) / [service status](https://www.ecmwf.int/en/service-status) / [confluence](https://confluence.ecmwf.int) / [support](https://support.ecmwf.int) / [accounting](https://www.ecmwf.int/user)
...@@ -8,30 +8,32 @@ If you need access, talk to your supervisor to create an account for you. You wi ...@@ -8,30 +8,32 @@ If you need access, talk to your supervisor to create an account for you. You wi
Available Services @ IMGW: Available Services @ IMGW:
- [ecaccess](https://confluence.ecmwf.int/display/ECAC/ECaccess+Home) - [ecaccess](#connecting-via-ecaccess)
- [srvx1.gateway](https://srvx1.img.univie.ac.at/ecmwf/ecmwf) / [gateway.docs](https://confluence.ecmwf.int/display/ECAC/Releases+-+Gateway+package) / [boaccess](https://boaccess.ecmwf.int) - [ecgateway](#ecaccess-gateway)
## Connecting to ECMWF Services ## Connecting to ECMWF Services
A ECMWF user can connect to the ECS/ATOS using teleport, first load the teleport module and start the ssh-agent: A ECMWF user can connect to the ECS/ATOS using teleport, first load the teleport module and start the ssh-agent:
```bash title="Using teleport" ```shell title="Using teleport"
module load teleport module load teleport
** INFO: Default jumphost now: jump.ecmwf.int ** INFO: Default jumphost now: jump.ecmwf.int
** INFO: Module loaded. SSH Agent required for login, run 'startagent', ** INFO: Module loaded. SSH Agent required for login, run 'ssh-agentstart',
** run ssh-agent -k to kill the agent. ** or 'ssh-agentreconnect' ro reconnect to an existing agent.
Login using: python3 -m teleport.login and your ECMWF credentials. ** run 'ssh-agent -k' to kill the agent.
Login run: 'python3 -m teleport.login' and your ECMWF credentials.
e.g. 'ssh -J <id>@jump.ecmwf.int <id>@ecs-login'
Check certificates, run: 'tsh status'
# Activate the ssh-agent (required to store the key/certificate) # Activate the ssh-agent (required to store the key/certificate)
startagent ssh-agentstart
# Check if it is running # Check if it is running
ssh-add -l ssh-add -l
``` ```
once you have a running ssh-agent, run a browserless login via python once you have a running ssh-agent, run a browserless login via python
```bash title="Connecting to ECMWF" ```shell title="Connecting to ECMWF"
# Login to the default teleport jump host (shell.ecmwf.int) Reading # Login to the default teleport jump host (shell.ecmwf.int) Reading
python3 -m teleport.login python3 -m teleport.login
tsh status tsh status
...@@ -42,6 +44,8 @@ ssh-add -l ...@@ -42,6 +44,8 @@ ssh-add -l
ssh -J [user]@jump.ecmwf.int [user]@ecs-login ssh -J [user]@jump.ecmwf.int [user]@ecs-login
# Login to HPC ATOS # Login to HPC ATOS
ssh -J [user]@jump.ecmwf.int [user]@hpc-login ssh -J [user]@jump.ecmwf.int [user]@hpc-login
# delete current certificates
tsh logout
``` ```
Environment variables configuration: Environment variables configuration:
...@@ -82,8 +86,9 @@ It is required to have an SSH-agent running in order to connect to the ECMWF ser ...@@ -82,8 +86,9 @@ It is required to have an SSH-agent running in order to connect to the ECMWF ser
# load the module # load the module
module load teleport module load teleport
# start a new agent or reconnect # start a new agent or reconnect
startagent ssh-agentstart
# or reconnect
ssh-agentreconnect
# unsure about agents? # unsure about agents?
userservices sshtools -h userservices sshtools -h
# kill all # kill all
...@@ -108,7 +113,7 @@ This will solve some `ecaccess` issues. ...@@ -108,7 +113,7 @@ This will solve some `ecaccess` issues.
## Connecting via ECaccess ## Connecting via ECaccess
using a local installation of ecaccess tools can be used to submit jobs and monitor jobs from a remote location. using a local installation of ecaccess tools can be used to submit jobs and monitor jobs from a remote location. [Documentation @ECMWF](https://confluence.ecmwf.int/display/ECAC/ECaccess+Home)
```bash title="ECAccess module" ```bash title="ECAccess module"
# load the ecaccess module # load the ecaccess module
...@@ -143,3 +148,10 @@ ecaccess-certificate-create ...@@ -143,3 +148,10 @@ ecaccess-certificate-create
## ECaccess Gateway ## ECaccess Gateway
The department is running a member state ecaccess gateway service. The department is running a member state ecaccess gateway service.
There are two gateways:
- [boaccess](https://boaccess.ecmwf.int/)
- [imgw](https://ecaccess.wolke.img.univie.ac.at/)
Please use your ECMWF credentials to login. [Documentation @ECMWF](https://confluence.ecmwf.int/display/ECAC/Releases+-+Gateway+package)
...@@ -49,7 +49,7 @@ clean: ...@@ -49,7 +49,7 @@ clean:
rm *.o triangle rm *.o triangle
``` ```
```sh title="running the Makefile ```sh title="running the Makefile"
$ make $ make
gfortran -c circle.f90 gfortran -c circle.f90
gfortran -c triangle.f90 gfortran -c triangle.f90
......
...@@ -17,24 +17,43 @@ Get some information on Fortran: ...@@ -17,24 +17,43 @@ Get some information on Fortran:
There are a few compilers, but most commonly GNU (Gfortran) and INTEL (ifort) are used on our servers. There are a few compilers, but most commonly GNU (Gfortran) and INTEL (ifort) are used on our servers.
| | GNU Fortran | INTEL Fortran | | | GNU Fortran | INTEL Fortran |
|---|---|---| | ---------------------------------- | ------------------------------------ | ----------------------------------- |
| double precision real| `-fdefault-real-8`| `-r8`| | double Fprecision real | `-fdefault-real-8` | `-r8` |
| check array bounds | `-fbounds-check` | `-check` | | check array bounds | `-fbounds-check` | `-check` |
| call chain traceback | `-fbacktrace` | `-traceback` | | call chain traceback | `-fbacktrace` | `-traceback` |
| convert little/big endian | `-fconvert=big-endian/little-endian` | `-convert big_endian/little_endian` | | convert little/big endian | `-fconvert=big-endian/little-endian` | `-convert big_endian/little_endian` |
| default optimisation | `-O0` | `-O2` | | default optimisation | `-O0` | `-O2` |
| highest recommended optimisation | `-O3` | `-O2` maybe `-O3` or `-fast` | | highest recommended optimisation | `-O3` | `-O2` maybe `-O3` or `-fast` |
| position independent code (shared) | `-fPIC` | `-fPIC` |
Please take a look into the compiler options guidelines from AMD or Intel, which might be very helpful for finding the right compiler flags for certain CPUs Please take a look into the compiler options guidelines from AMD or Intel, which might be very helpful for finding the right compiler flags for certain CPUs
- Servers such as Aurora [AMD EPYC ZEN3](compiler-options-quick-ref-guide-epyc-7xx3-series-processors.pdf) - Servers such as Aurora [AMD EPYC ZEN3 compiler quick reference](compiler-options-quick-ref-guide-epyc-7xx3-series-processors.pdf)
- Servers such as VSC5, LUMI, ... [AMD EPYC ZEN4](compiler-options-quick-ref-guide-amd-epyc-9xx4-series-processors.pdf) - Servers such as VSC5, LUMI, ... [AMD EPYC ZEN4 compiler quick reference](compiler-options-quick-ref-guide-amd-epyc-9xx4-series-processors.pdf)
- Servers such as SRVX1, JET, VSC4 [Intel](quick-reference-guide-intel-compilers-v19-1-final.pdf) - Servers such as SRVX1, JET, VSC4 [Intel compiler quick reference](quick-reference-guide-intel-compilers-v19-1-final.pdf)
more information on how to find good options can be found on the PRACE (Partnership for advanced computing in Europe) [website](https://prace-ri.eu/training-support/best-practice-guides/) more information on how to find good options can be found on the PRACE (Partnership for advanced computing in Europe) [website](https://prace-ri.eu/training-support/best-practice-guides/)
![](https://prace-ri.eu/wp-content/uploads/logo-main.png) ![](https://prace-ri.eu/wp-content/uploads/logo-main.png)
## Generating optimized code for a certain processor
All compiler providers have special options that have optimizations for certain processors, especially intel.
| Processor Type | System | Flag (intel) | Flag (gcc) |
| -------------- | ---------------- | --------------- | -------------------- |
| Cascade Lake | VSC5 | `-xCORE-AVX512` | `-march=cascadelake` |
| Skylake | JET, SRVX1, VSC4 | `-xCORE-AVX512` | `-march=skylake` |
| Broadwell | SRVX8 | `-xCORE-AVX2` | `-march=broadwell` |
| Sandy Bridge | DEVx1 | `-xAVX` | `-march=sandybridge` |
| Milan | VSC5, Aurora | `-xCORE-AVX2` | `-march=znver3` |
| Generic x64 | containers | `-xCORE-AVX2` | `-march=x86-64-v4` |
[GCC/GFortran processor compiler flags](https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html)
## Intel Compiler ## Intel Compiler
from P. Seibert using ifort for the fastest code (srvx1): from P. Seibert using ifort for the fastest code (srvx1):
...@@ -42,6 +61,7 @@ from P. Seibert using ifort for the fastest code (srvx1): ...@@ -42,6 +61,7 @@ from P. Seibert using ifort for the fastest code (srvx1):
# get GRIP_PATH from environment modules # get GRIP_PATH from environment modules
INCPATH = GRIP_API/include INCPATH = GRIP_API/include
LIBPATH = GRIP_API/lib LIBPATH = GRIP_API/lib
# using -xAVX is for old CPUs !!!
FFLAGS = -cpp -xAVX -ipo -O3 -no-prec-div -opt-prefetch -m64 -mcmodel=medium -I$(INCPATH) FFLAGS = -cpp -xAVX -ipo -O3 -no-prec-div -opt-prefetch -m64 -mcmodel=medium -I$(INCPATH)
LDFLAGS = $(FFLAGS) -L$(LIBPATH) -Bstatic -lgrib_api_f90 -lgrib_api -lm -ljasper LDFLAGS = $(FFLAGS) -L$(LIBPATH) -Bstatic -lgrib_api_f90 -lgrib_api -lm -ljasper
``` ```
......
# Getting Started # Getting Started
**Welcome to the Department of Meteorology and Geophysics @ University of Vienna.** ## Welcome to the Department of Meteorology and Geophysics
🥳
Tasks to complete for newcomers, it is recommended that you print this page an tick off your steps: Tasks to complete for newcomers, it is recommended that you print this page an tick off your steps:
- [ ] Request a server account via your supervisor - [ ] Request a server account via your supervisor
- [ ] Receive the inital user account information via mail. - [ ] Receive the inital user account information via mail.
- [ ] Change your initial password, via - [ ] Setup a password manager. [ZID tips](https://zid.univie.ac.at/en/it-worlds/it-security/it-security-tips/password-manager/), e.g. Bitwarden or KeepassXC
- [ ] How to connect using SSH, read the [connection 101](./SSH-VPN-VNC/README.md)
- on Windows install a SSH-Client, e.g. Bitwise, MobaXterm, Putty,...
- [ ] Change your initial password with one of these options:
- browser ([https://wolke.img.univie.ac.at/ipa/ui](https://wolke.img.univie.ac.at/ipa/ui)) - browser ([https://wolke.img.univie.ac.at/ipa/ui](https://wolke.img.univie.ac.at/ipa/ui))
- ssh terminal `ssh [username]@srvx1.img.univie.ac.at`, follow the instructions in the command shell. Windows users install a ssh client - terminal `ssh [username]@srvx1.img.univie.ac.at`
- Optional: Setup a password manager. [tips](https://zid.univie.ac.at/en/it-worlds/it-security/it-security-tips/password-manager/), e.g. Bitwarden - [ ] Optional: [Create a ssh-key](./SSH-VPN-VNC/Questions.md#q-how-to-use-ssh-key-authentication) and add it to your profile on the [IPA](./SSH-VPN-VNC/IPA.md#add-ssh-key)
- [ ] (optional windows) Install a SSH-Client, e.g. Bitwise, MobaXterm, Putty,... - [ ] Apply for your first [VSC Training course](https://vsc.ac.at/research/vsc-research-center/vsc-school-seminar/), e.g. Introduction to Working on the VSC Clusters
- [ ] (optional) setup ssh-key via [IPA](https://wolke.img.univie.ac.at/ipa/ui)
- [ ] Familiarize yourself with the shell environment @SRVX1
- [ ] Apply for your first VSC Training course
- Login in to all servers:
* [ ] srvx1 .img.univie.ac.at
* [ ] srvx8
* [ ] aurora
* [ ] jet01/jet02
* [ ] (optional) connect to vsc5.vsc.ac.at
## Environment ## Environment
Please do the following steps to get a better idea of what is where: When you are new to using servers and linux, these few steps might help to get started. However, it is recommended to make a linux introductionary course, e.g. VSC introduction to linux.
Steps: Steps:
- [ ] login to srvx1 using ssh: `ssh [user]@srvx1.img.univie.ac.at` :earth_africa: - [ ] login to srvx1 using ssh: `ssh [user]@srvx1.img.univie.ac.at` :earth_africa: [More on SRVX1](./Servers/SRVX1.md)
- [ ] run: `userpaths` to understand where different data resides. E.g. - [ ] run: `userpaths` to understand where different data resides. e.g.
- HOME, SCRATCH (personal), DATA, SHARED, WEBDATA, ?_JET - HOME, SCRATCH (personal), DATA, SHARED, WEBDATA, ?_JET
- [ ] check available modules by running: `module av` and load anaconda3 module by running: `module load anaconda3`. This should allow you to run some python programs. - [ ] check available modules by running: `module av` and load anaconda3 module by running: `module load anaconda3`. This should allow you to run some python programs.
- [ ] run: `userservices` to get some IMGW special tools. Maybe check the weather! - [ ] list and unload modules: `module list` and `module purge`. [More on Modules](./Misc/Environment-Modules.md)
- [ ] run: `userservices` to get some IMGW special tools. Maybe check the weather!?
Please find a useful summary of commands [here](./mkdocs/imgw-cheatsheet.pdf) Please find a useful summary of commands in the [IMGW cheatsheet](./mkdocs/imgw-cheatsheet.pdf)
## Summary of Computing Resources ## Summary of Computing Resources
...@@ -43,8 +37,13 @@ The Department of Meteorology and Geophysics has access to the following computi ...@@ -43,8 +37,13 @@ The Department of Meteorology and Geophysics has access to the following computi
- Teaching and Development Server ([SRVX1](Servers/SRVX1.md)) - Teaching and Development Server ([SRVX1](Servers/SRVX1.md))
- Remote Desktop Server ([SRVX8](Servers/SRVX8.md)) - Remote Desktop Server ([SRVX8](Servers/SRVX8.md))
- Development Server ([Aurora](Servers/Aurora.md))
- Computing Cluster ([JET](Servers/JET.md)) - Computing Cluster ([JET](Servers/JET.md))
- Vienna Scientific Cluster ([VSC](VSC.md)) - Vienna Scientific Cluster ([VSC](VSC.md))
- European Center for Medium-Range Weather Forecast ([ECMWF](ECMWF.md))
External resources:
- European Center for Medium-Range Weather Forecast ([ECMWF](./ECMWF/README.md))
- Earth Observation Data Center ([EODC](https://eodc.wolke.img.univie.ac.at), [website](https://eodc.eu))
Please read about access, hardware and quotas at these different resources. A good starting point is [here](./Servers/README.md) Please read about access, hardware and quotas at these different resources. A good starting point is [here](./Servers/README.md)
\ No newline at end of file
# Git Tipps and Tools # Git Tipps and Tools
There is a lot of ways how to use git and sometimes a working solution can be very helpful for everybody. So please share you experience and working solutions.
## HowTo add a specif ssh-key for your git account There are a lot of resources in the web, how to learn git.
Some examples:
- [ZID Introduction to git](https://phaidra.univie.ac.at/detail/o:1403751)
- [Gitlab - Learn Git](https://docs.gitlab.com/ee/tutorials/learn_git.html)
- [Git cheatsheet](./git%20cheat%20sheet.pdf)
Don't be overwhelmed. Everyone learns as they go along, but make sure you have a nice [setup](https://docs.gitlab.com/ee/tutorials/learn_git.html) and understand the [basics](https://docs.gitlab.com/ee/tutorials/learn_git.html).
## HowTo add a specific ssh-key for your git account
Sometimes it might be useful to have different ssh-keys per project or have a special ssh-key just for github/gitlab. Sometimes it might be useful to have different ssh-keys per project or have a special ssh-key just for github/gitlab.
Steps: Steps:
1. Generate a `ssh-keygen` 1. Generate a `ssh-keygen`
2. Add ssh-key to GitHub or GitLab account under settings 2. Add ssh-key to GitHub or GitLab account under settings
3. Write a `~/.ssh/config` file with a content like this 3. Write a `~/.ssh/config` file with a content like this, other examples are in [ECMWF](../ECMWF/README.md).
```
```ini
Host github.com Host github.com
HostName github.com HostName github.com
User git User git
...@@ -20,29 +32,43 @@ Host gitlab.com ...@@ -20,29 +32,43 @@ Host gitlab.com
User git User git
IdentityFile ~/.ssh/id_rsa_for_gitlab IdentityFile ~/.ssh/id_rsa_for_gitlab
``` ```
Make sure that you adjust the `Hostname` accordingly and be sure that you use ssh in your git repo: Make sure that you adjust the `Hostname` accordingly and be sure that you use ssh in your git repo:
```bash ```bash
$ git remote -v $ git remote -v
origin git@github.com:USER/repo.git (fetch) origin git@github.com:USER/repo.git (fetch)
origin git@github.com:USER/repo.git (push) origin git@github.com:USER/repo.git (push)
``` ```
show no `https` in the urls.
These urls should not show `https`, if there is a `https` then you will need to sign in with your user credentials for gitlab (username and password).
Every git repository can be cloned using https, but for ssh-key access you need the `git@...` version (usually there are two options: `ssh`, and `https`).
## HowTo Sync a GitHub and a GitLab repository ## HowTo Sync a GitHub and a GitLab repository
It is easy to import a GitHub repo into GitLab and the otherway around. However, if you want to make sure you can have both repos at the same state, you need to syncronize them. It is easy to import a GitHub repo into GitLab and the otherway around. However, if you want to make sure you can have both repos at the same state, you need to syncronize them.
How to call: `./git-repos-sync [URL1] [URL2] [Branch]` How to call: `./git-repos-sync [URL1] [URL2] [Branch]`
This means: This means:
- `URL1` - Address of the first remote repository - `URL1` - Address of the first remote repository
- `URL2` - Address of the second remote repository - `URL2` - Address of the second remote repository
- The order of `URL1` or `URL2` does not matter. - The order of `URL1` or `URL2` does not matter.
- `Branch` is usually `master` - `Branch` is usually `master`
Different use cases: Different use cases:
1. You have **already** a local copy of either of the repositories (e.g. GitLab or GitHub) 1. You have **already** a local copy of either of the repositories (e.g. GitLab or GitHub)
2. You have no local copy of either repository. 2. You have no local copy of either repository.
??? note "syncronize git repos"
```bash title="git-repos-sync"
--8<-- "Git/git-repos-sync"
```
### Case 1 ### Case 1
```bash ```bash
......
File added
## ICON-NWP # ICON-NWP
This is a short introduction to [using the ICON-NWP model on `jet.imgw`](./ICON-NWP.md) and how you can [run assimilation](./how-to-bacy.md) experiments with [`bacy`](./bacy.md). You can get more information about the model [here](https://code.mpimet.mpg.de/projects/iconpublic). This is a short introduction to using the ICON-NWP model on `jet` ([ICON-NWP](./ICON-NWP.md)) and how you can [run assimilation](./how-to-bacy.md) experiments with [`bacy`](./bacy.md).
\ No newline at end of file
You can get more information about the model [here](https://code.mpimet.mpg.de/projects/iconpublic).
\ No newline at end of file
# Using environment modules # Environment modules
![](https://upload.wikimedia.org/wikipedia/en/thumb/0/0a/Environment_Modules_logo.svg/320px-Environment_Modules_logo.svg.png) ![](https://upload.wikimedia.org/wikipedia/en/thumb/0/0a/Environment_Modules_logo.svg/320px-Environment_Modules_logo.svg.png)
...@@ -16,7 +16,7 @@ Currently Loaded Modulefiles: ...@@ -16,7 +16,7 @@ Currently Loaded Modulefiles:
`module list` shows the currently loaded modules and reports that 6 libraries need to be loaded as dependencies for `ECCODES`. Thus, it is not necessary to load the other libraries manually as they are dependencies of `ECCODES`. However it will be necessary to load the intel compiler suite `intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y` as well for build applications. `module list` shows the currently loaded modules and reports that 6 libraries need to be loaded as dependencies for `ECCODES`. Thus, it is not necessary to load the other libraries manually as they are dependencies of `ECCODES`. However it will be necessary to load the intel compiler suite `intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y` as well for build applications.
```bash ```bash title="using environment modules"
# unload modules # unload modules
$ module unload eccodes/2.18.0-intel-20.0.2-6tadpgr $ module unload eccodes/2.18.0-intel-20.0.2-6tadpgr
...@@ -45,7 +45,9 @@ prepend-path CMAKE_PREFIX_PATH /jetfs/spack/opt/spack/linux-rhel8-skylake_avx ...@@ -45,7 +45,9 @@ prepend-path CMAKE_PREFIX_PATH /jetfs/spack/opt/spack/linux-rhel8-skylake_avx
### GNU ### GNU
gfortran is available on the default command search path, so it is not essential to load a module, unless: (1) You need to set environment variables such as FC and F77; (2) You need to link MPI libraries. In these cases, you should load the gfortran module. This is what the module does to the environment: gfortran is available on the default command search path, so it is not essential to load a module, unless: (1) You need to set environment variables such as FC and F77; (2) You need to link MPI libraries. In these cases, you should load the gfortran module. This is what the module does to the environment:
```
```sh title="module details gcc"
$ module show gcc/8.3.1-gcc-8.3.1-pp3wjou $ module show gcc/8.3.1-gcc-8.3.1-pp3wjou
------------------------------------------------------------------- -------------------------------------------------------------------
/jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512/gcc/8.3.1-gcc-8.3.1-pp3wjou: /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512/gcc/8.3.1-gcc-8.3.1-pp3wjou:
...@@ -69,7 +71,8 @@ setenv F90 /usr/bin/gfortran ...@@ -69,7 +71,8 @@ setenv F90 /usr/bin/gfortran
### INTEL ### INTEL
there are a few version of intel compilers installed and some more might be added as well: there are a few version of intel compilers installed and some more might be added as well:
```bash
```bash title="module details intel"
$ module av intel $ module av intel
----------------------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ------------------------ ----------------------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ------------------------
intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw
...@@ -81,10 +84,12 @@ intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y ...@@ -81,10 +84,12 @@ intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y
--------------------------- /jetfs/spack/share/spack/modules/linux-rhel8-haswell --------------------------- --------------------------- /jetfs/spack/share/spack/modules/linux-rhel8-haswell ---------------------------
intel-parallel-studio/composer.2017.7-intel-17.0.7-disfj2g intel-parallel-studio/composer.2017.7-intel-17.0.7-disfj2g
``` ```
This shows that we have `intel-parallel-studio` with version `20.0.2` and `17.0.7` installed and `intel-oneapi-compilers` at version `2021.2.0`. The first does not come with Intel-MPI, but the second does. This shows that we have `intel-parallel-studio` with version `20.0.2` and `17.0.7` installed and `intel-oneapi-compilers` at version `2021.2.0`. The first does not come with Intel-MPI, but the second does.
Again the module sets a lot of Again the module sets a lot of
```bash
```bash title="module details intel-parallel-studio"
$ module show intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y $ module show intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y
------------------------------------------------------------------- -------------------------------------------------------------------
/jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512/intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y: /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512/intel-parallel-studio/composer.2020.2-intel-20.0.2-zuot22y:
......
# User Shells # User Shells
Almost every linux shell has ways to modify the user experience. There are multiple ways of changing your shell environment.
Here are some topics listed:
- modifying the shell user limits
- virtual terminals
## Limits ## Limits
It is possible to address resource limits in a user shell, which makes sure that programs do not run
It is possible to address resource limits in a user shell, which make sure that programs do not run out of control. This can be useful for a code that might have an infinite recursion. This is usually controlled by the stack size limit.
Check what resource limits you have: Check what resource limits you have:
`ulimit -a`
```sh
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3091355
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
# in an slurm job it might be necessary to use
ulimit -s unlimited
# to allow a larger stack size for e.g. FORTRAN programs.
```
Stack contains all local variables & data, intermediate storage for registers, and function parameters. This setting is used to block infinite recursion.
## Virtual terminals | flags | description |
| ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -H | Specifies that the hard limit for the given resource is set. If you have root user authority, you can increase the hard limit. Anyone can decrease it |
| -S | Specifies that the soft limit for the given resource is set. A soft limit can be increased up to the value of the hard limit. If neither the -H nor -S flags are specified, the limit applies to both |
| -a | Lists all of the current resource limits |
| -b | The maximum socket buffer size |
| -c | The maximum size of core files created |
| -d | The maximum size of a process's data segment |
| -e | The maximum scheduling priority ("nice") |
| -f | The maximum size of files written by the shell and its children |
| -i | The maximum number of pending signals |
| -l | The maximum size that may be locked into memory |
| -m | The maximum resident set size (many systems do not honor this limit) |
| -n | The maximum number of open file descriptors (most systems do not allow this value to be set) |
| -p | The pipe size in 512-byte blocks (this may not be set) |
| -q | The maximum number of bytes in POSIX message queues |
| -r | The maximum real-time scheduling priority |
| -s | The maximum stack size |
| -t | The maximum amount of cpu time in seconds |
| -u | The maximum number of processes available to a single user |
| -v | The maximum amount of virtual memory available to the shell and, on some systems, to its children |
| -x | The maximum number of file locks |
| -T | The maximum number of threads |
...@@ -81,3 +81,21 @@ A full list of these Job State codes can be found in [Slurm’s documentation.]( ...@@ -81,3 +81,21 @@ A full list of these Job State codes can be found in [Slurm’s documentation.](
| `AssociationNodeLimit` | All nodes assigned to your job’s specified association are in use; job will run eventually. | | `AssociationNodeLimit` | All nodes assigned to your job’s specified association are in use; job will run eventually. |
A full list of these Job Reason Codes can be found [in Slurm’s documentation.](https://slurm.schedmd.com/squeue.html#lbAF) A full list of these Job Reason Codes can be found [in Slurm’s documentation.](https://slurm.schedmd.com/squeue.html#lbAF)
# Get information on your jobs
```sh title='Job details'
# get all your jobs since
sacct --start=YY-MM-DD -u $USER -o start,jobid,jobidraw,jobname,partition,maxvmsize,elapsed,state,exitcode
# get more information on one job
sacct -j [jobid]
```
```sh title='Job efficiency'
# get a jobs efficiency report
seff [jobid]
# example
seff
```
\ No newline at end of file
...@@ -134,3 +134,34 @@ You just need to add a configuration option and then you can open the Dashboard ...@@ -134,3 +134,34 @@ You just need to add a configuration option and then you can open the Dashboard
[Profile](QA-009-Memory-Profiling.ipynb) [Profile](QA-009-Memory-Profiling.ipynb)
If you need to get a better understanding of you functions memory and execution time, try these profiling options. If you need to get a better understanding of you functions memory and execution time, try these profiling options.
## Q: How to ignore user site packages?
When using python it is possible to have multiple site, where packages can be installed. the default is to use the path from the python interpreter and a user site.
```sh
$ python -c 'import sys;print("\n".join(sys.path))'
/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/home/USER/.local/lib/python3.8/site-packages
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages
```
this shows that there is a user site in your HOME directory for that python version. Python will always look for a directory for it's own version.
You can disable that by altering the `sys.path` (python type list) manually or using an environment variable (`export PYTHONNOUSERSITE=1`).
```sh
export PYTHONNOUSERSITE=1
$ python -c 'import sys;print("\n".join(sys.path))'
/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages
# unset the variable again
unset PYTHONNOUSERSITE
```
\ No newline at end of file
...@@ -3,24 +3,20 @@ ...@@ -3,24 +3,20 @@
# Welcome to the Department of Meteorology and Geophysics # Welcome to the Department of Meteorology and Geophysics
Find help here with your computer related problems. A mkdocs rendered version is available here: [wolke](https://wolke.img.univie.ac.at/documentation/general/index.html) Here you can find help with your computer/server related problems.
[link to this page on wolke](https://wolke.img.univie.ac.at/documentation/general/index.html)
🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧
Please note that this documentation is under active development and things might shift a bit.
🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧
Search with the top bar or go through the directories: Search with the top bar or go through the directories:
- [Python related Problems](./Python/README.md) - [Python related Problems](./Python/README.md)
- [SSH, VNC, VPN related Problems](./SSH-VPN-VNC/README.md) - [SSH, VNC, VPN related Problems](./SSH-VPN-VNC/README.md)
- [Editors](./Editors/) and [remote connection](./SSH-VPN-VNC/README.md) - [Editors](./Editors/README.md) and [remote connection](./SSH-VPN-VNC/README.md)
- [Data availability and location](./Data/README.md) - [Data availability and location](./Data/README.md)
- [Git related problems](https://gitlab.phaidra.org/imgw/computer-resources/-/issues) - [Git related problems](./Git/README.md) / [Report new issus on gitlab](https://gitlab.phaidra.org/imgw/computer-resources/-/issues)
for new employees or students, you could start with the [Getting Started](./Getting%20Started.md) section or the [Student](./Students.md) section. for new employees or students, you could start with the [Getting Started](./Getting%20Started.md) section or the [Student](./Students/README.md) section.
A useful summary of all resources is given by the [IMGW Cheat sheet](./mkdocs/imgw-cheatsheet.pdf) A useful summary of handy tips is available as [IMGW Cheat sheet (PDF)](./mkdocs/imgw-cheatsheet.pdf)
**If you care to participate please do so:** **If you care to participate please do so:**
......
...@@ -46,15 +46,3 @@ Do not forget to save your changes and finally you should see the fingerprint of ...@@ -46,15 +46,3 @@ Do not forget to save your changes and finally you should see the fingerprint of
![](../mkdocs/img/ipa-add-ssh-final.png) ![](../mkdocs/img/ipa-add-ssh-final.png)
Done. Done.
The IPA needs to sync this to all connected servers and this might take up to 5 minutes. Please be patient. The IPA needs to sync this to all connected servers and this might take up to 5 minutes. Please be patient.
## OTP
This section will be added soon.
:construction:
## Kerberos
:construction:
# Questions and Answers # Questions and Answers
Please feel free to add you questions and anwsers. Please feel free to add you questions and anwsers.
Links:
- [SSH explained](https://blog.teamairship.com/intro-to-ssh-secure-shell-101-exercise)😂
- [SSH agents explained](https://smallstep.com/blog/ssh-agent-explained/)
## Q: How does ssh work?
Secure Shell (SSH) is a popular networking protocol that lets us access a remote computer over an insecure network such as the Internet.
Secure Shell also supports both password and key-based authentication. Password-based authentication let users provide username and password to authenticate to the remote server. A key-based authentication allows users to authenticate through a key-pair. The key pairs are two cryptographically secure keys for authenticating a client to a Secure Shell server.
![ssh key exchange diagram](../mkdocs/img/ssh-key-exchange.png)
Secure Shell has a client-server architecture. Typically, a server administrator installs a server program that accepts or rejects the incoming connections. Besides, a user runs a client program on their system that requests the server. By default, the server listens on HTTP port 22.
## Q: How to use ssh-key authentication? ## Q: How to use ssh-key authentication?
In order to connect passwordless to a remote server a key needs to be generated. In order to connect passwordless to a remote server a secure shell key needs to be generated. This key will be used automatically to login.
```
$ ssh-keygen -b 4096 ???+ danger "SSH KEY Authentication"
Using this **can be a safer way** to connect to our servers. However, if someone gets access to your key, e.g. on your Laptop, that person has access to your data/server. **Secure your ssh-key.** Therefore consider using a phassphrase / ssh-agent.
```sh title="How to create an RSA key"
# Please use at least 4096 bits for generating your key.
$ ssh-keygen -b 4096 -t rsa
Generating public/private rsa key pair. Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): Enter passphrase (empty for no passphrase):
Enter same passphrase again: Enter same passphrase again:
...@@ -27,41 +48,102 @@ The key's randomart image is: ...@@ -27,41 +48,102 @@ The key's randomart image is:
+----[SHA256]-----+ +----[SHA256]-----+
``` ```
```sh title="How to create a secure key"
# this is the newest and securest standard
$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/user/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_ed25519.
Your public key has been saved in /home/user/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:sin8cTe10fOIbSpQW8gETSUKIx27viPolF0w79cBTzU user@srvx1.img.univie.ac.at
The key's randomart image is:
+--[ED25519 256]--+
| ..+..+o.E |
| ..+ .oo . |
| o . oo.. |
| + . ++ .. |
| = S.ooo o |
| + + +...o = + |
| o.+ * o.+ o + .|
| .. .o.= ... o |
| .. .o. .. |
+----[SHA256]-----+
```
It is recommended to use a password to encrpyt the private key `.ssh/id_rsa`. However, this password is then required each time to login. Using an `ssh-agent` can solve that problem. It is recommended to use a password to encrpyt the private key `.ssh/id_rsa`. However, this password is then required each time to login. Using an `ssh-agent` can solve that problem.
???+ warning "Adding ssh keys to our servers"
Remember to use the [IPA for adding your ssh-keys](./IPA.md#add-ssh-key), as keys in `~/.ssh/` on all our servers will be ignored.
## Q: How to use an ssh-agent? ## Q: How to use an ssh-agent?
Using an SSH-Agent will make your connection even safer, as your private key is encrypted with a passphrase. To create a ssh-key goto [How to use ssh-key authentication?](#q-how-to-use-ssh-key-authentication).
Continue with how to use the ssh-agent: Using an SSH-Agent will make your connection even safer, as your private key is encrypted with a passphrase. First create a new ssh-key and **use a passphrase**. Then continue with using the ssh-agent. This agent is installed on all our servers and it is allowed to forward authentication as you go along.
```bash
# Launch the SSH Agent ```bash title="Using an ssh-agent"
# Launch the SSH Agent on your Laptop/Computer
eval `ssh-agent` eval `ssh-agent`
# this will add your ssh-keys from ~/.ssh/ # this will add all your ssh-keys from ~/.ssh/
# You will be asked a passphrase if you have one # You will be asked a passphrase if you have one
# this will add .ssh/id_rsa as default
ssh-add ssh-add
# or any keyfile you created: # or specify the keyfile yourself:
ssh-add .ssh/secret_key ssh-add path/to/secret/keyfile/secret_key
# Copy the ssh-key (pub) to the server
ssh-copy-id [user]@[server] # The you need to add your ssh-key.pub (public) to the IPA
# and connect
# and connect (no password prompt)
ssh [user]@[server] ssh [user]@[server]
# Kill the agent and all store secure information # Kill the agent and all store secure information
ssh-agent -k ssh-agent -k
``` ```
**Tip: configure your `.ssh/config` file to use the correct key a server. Goto [SSH](SSH.md)**
[Nice summary of how an ssh-agent works](https://smallstep.com/blog/ssh-agent-explained/) ???+ warning "ssh-agent on servers"
Please use these commands on our servers: `ssh-agentstart` and `ssh-agentreconnect`. These commands take care that you do not launch too many agents, killing the system.
<img src="https://keepassxc.org/images/keepassxc-logo.svg" width="200px"> [Adding a ssh-key to the IPA](./IPA.md#add-ssh-key).
adding this to your local `.ssh/config` will forward your ssh-agent to be used to hop from server to server.
```sh title="Allow ssh-agent forwarding"
$ vi .ssh/config
Host [host address]
ForwardAgent yes
```
e.g.
```sh title="ssh-agent hopping"
# check a ssh-agent is running on your Laptop
$ ssh-add -l
256 SHA256:3ppnVv7Cw monkey@laptop (ED25519)
# set the option manually or modify the .ssh/config
$ ssh -oForwardAgent=yes srvx1
# on the sever all your ssh-keys should be available as well
[monkey@srvx1] $ ssh-add -l
256 SHA256:3ppnVv7Cw monkey@laptop (ED25519)
# since this key is registered in the IPA, connecting to other servers is easy
[monkey@srvx1] $ ssh jet02
# with no password prompt using your ssh-agent
[monkey@jet02] $
```
[Nice summary of how an ssh-agent works](https://smallstep.com/blog/ssh-agent-explained/)
Keep in mind that you can use the ssh-agent with [KeepassXC](https://keepassxc.org/docs/#faq-ssh-agent-how), find a nice tutorial [here](https://ferrario.me/using-keepassxc-to-manage-ssh-keys/). This is really convenient as it allows you to use all keys in the Keepass as long as it is unlocked. The keys will be automatically removed when the keepass is locked. :) Keep in mind that you can use the ssh-agent with [KeepassXC](https://keepassxc.org/docs/#faq-ssh-agent-how), find a nice tutorial [here](https://ferrario.me/using-keepassxc-to-manage-ssh-keys/). This is really convenient as it allows you to use all keys in the Keepass as long as it is unlocked. The keys will be automatically removed when the keepass is locked. :)
<img src="https://upload.wikimedia.org/wikipedia/commons/c/c1/KeePassXC.svg" width="200px">
## Q: How to transfer files between two VPN networks? ## Q: How to transfer files between two VPN networks?
You should be able to use an SSH tunnel via a gateway server You should be able to use an SSH tunnel via a gateway server
Situation Situation
``` ```
VPN-1 VPN-2 VPN-1 VPN-2
__________ /| ___________ /| __________ __________ /| ___________ /| __________
...@@ -71,12 +153,14 @@ __________ /| ___________ /| __________ ...@@ -71,12 +153,14 @@ __________ /| ___________ /| __________
__________ | | ___________ | | __________ __________ | | ___________ | | __________
|/ |/ |/ |/
``` ```
Assuming you're trying to transfer a file from/to a remote computer ("remote") from/to your local computer ("local"), Assuming you're trying to transfer a file from/to a remote computer ("remote") from/to your local computer ("local"),
establish a tunnel via a third computer ("gateway") by typing this on your local computer: establish a tunnel via a third computer ("gateway") by typing this on your local computer:
```bash ```bash
$ ssh -fNL 12345:remote:22 gatewaylogin@gateway $ ssh -fNL 12345:remote:22 gatewaylogin@gateway
``` ```
Then you can run an unlimited amount of SCP commands on this tunnel (still typing on your local computer): Then you can run an unlimited amount of SCP commands on this tunnel (still typing on your local computer):
```bash ```bash
...@@ -85,10 +169,12 @@ $ scp -P 12345 remotelogin@localhost://path/to/remote/file /local/path/where/you ...@@ -85,10 +169,12 @@ $ scp -P 12345 remotelogin@localhost://path/to/remote/file /local/path/where/you
``` ```
another option is to use `rsync` with a gateway command: another option is to use `rsync` with a gateway command:
```bash ```bash
# rsync [SRC] [DEST] # rsync [SRC] [DEST]
$ rsync -avz -r --stats --progress -e "ssh gateway ssh" remote:/src/documents/ /dest/documents $ rsync -avz -r --stats --progress -e "ssh gateway ssh" remote:/src/documents/ /dest/documents
``` ```
**Note: key-based authentication is required from the gateway to the remote server using rsync.** **Note: key-based authentication is required from the gateway to the remote server using rsync.**
Initial testing showed that using key-based authentication between gateway and remote is required. Initial testing showed that using key-based authentication between gateway and remote is required.
...@@ -100,16 +186,21 @@ Please replace `[USER]` with your username on these servers and adjust the serve ...@@ -100,16 +186,21 @@ Please replace `[USER]` with your username on these servers and adjust the serve
```bash ```bash
$ ssh -t [USER]@srvx1.img.univie.ac.at 'ssh [USER]@jet01.img.univie.ac.at' $ ssh -t [USER]@srvx1.img.univie.ac.at 'ssh [USER]@jet01.img.univie.ac.at'
``` ```
or using the above `~/.ssh/config` you can do: or using the above `~/.ssh/config` you can do:
```bash ```bash
$ ssh -t login ssh jet $ ssh -t login ssh jet
``` ```
or using the connect script or using the connect script
```bash ```bash
$ connect2jet -g login jet $ connect2jet -g login jet
``` ```
## Q: How to mount a remote file system on Linux (MAC)? ## Q: How to mount a remote file system on Linux (MAC)?
You can us programs like [Filezilla](https://filezilla-project.org/) or [Cyberduck](https://cyberduck.io/) (MAC) to transfer files between remote and local host. But sometimes it is much easier to mount a remote file system and work on it like an external drive. You can us programs like [Filezilla](https://filezilla-project.org/) or [Cyberduck](https://cyberduck.io/) (MAC) to transfer files between remote and local host. But sometimes it is much easier to mount a remote file system and work on it like an external drive.
You can use the `mountserver` ([Download: mountserver](mountserver)) script to do so. This requires `sshfs` to be installed, on Linux that is in the standard repositories. A short into and some additional steps can be found [here](https://www.tjansson.dk/2008/01/autofs-and-sshfs-the-perfect-couple/). You can use the `mountserver` ([Download: mountserver](mountserver)) script to do so. This requires `sshfs` to be installed, on Linux that is in the standard repositories. A short into and some additional steps can be found [here](https://www.tjansson.dk/2008/01/autofs-and-sshfs-the-perfect-couple/).
...@@ -127,8 +218,8 @@ This will mount the remote directory to the local directory. The local directory ...@@ -127,8 +218,8 @@ This will mount the remote directory to the local directory. The local directory
--8<-- "SSH-VPN-VNC/mountserver" --8<-- "SSH-VPN-VNC/mountserver"
``` ```
## Q: How to use an SSH tunnel for private browsing? ## Q: How to use an SSH tunnel for private browsing?
based on a tutorial from [Linuxize](https://linuxize.com/post/how-to-setup-ssh-socks-tunnel-for-private-browsing/). based on a tutorial from [Linuxize](https://linuxize.com/post/how-to-setup-ssh-socks-tunnel-for-private-browsing/).
It can be really useful to access resources from inside the IMGW / UNIVIE network without using the VPN from the ZID. This can be done super easily. You need an SSH client (e.g. ssh, Putty) and [Firefox](https://www.mozilla.org/en-US/firefox/new/). It can be really useful to access resources from inside the IMGW / UNIVIE network without using the VPN from the ZID. This can be done super easily. You need an SSH client (e.g. ssh, Putty) and [Firefox](https://www.mozilla.org/en-US/firefox/new/).
...@@ -138,7 +229,9 @@ I'm showing the things here only for Linux, but Windows with Putty should be str ...@@ -138,7 +229,9 @@ I'm showing the things here only for Linux, but Windows with Putty should be str
```bash ```bash
ssh -N -D 8091 [USER]@srvx1.img.univie.ac.at ssh -N -D 8091 [USER]@srvx1.img.univie.ac.at
``` ```
Options: Options:
- `-N` - Tells SSH not to execute a remote command. - `-N` - Tells SSH not to execute a remote command.
- `-D 8091` - Opens a SOCKS tunnel on the specified port number. - `-D 8091` - Opens a SOCKS tunnel on the specified port number.
- To run the command in the background use the `-f` option. - To run the command in the background use the `-f` option.
......
# Connection 101 # Connection 101
Please find here some details on how to connect using SSH or VPN or VNC. And some other linux related terminal based information. Please find here some details on how to connect using SSH or VPN or VNC.
## General Access ## General Access
Please use these methods to access the servers: Please use these methods to access the servers:
1. Connect using [SSH](SSH.md) or [MOSH](https://mosh.org/#) (Mobile Secure Shell) 1. Connect using [SSH](SSH.md) or [MOSH](https://mosh.org/#) (Mobile Secure Shell)
2. Connect using [TeachingHub](../TeachingHub.md) or [ResearchHub](../Servers/JET.md#jupyterhub) 2. Connect using [TeachingHub](../Students/TeachingHub.md) or [ResearchHub](../Servers/JET.md#jupyterhub)
3. Connect using [VNC](VNC.md) 3. Connect using [VNC](VNC.md)
4. Access from outside the UNI, [VPN](VPN.md) 4. Access from outside the UNI via the [VPN](VPN.md)
## Jupyterhub ## Jupyterhub
...@@ -17,50 +17,23 @@ Currently, there are two JupyterHub instances runnning: ...@@ -17,50 +17,23 @@ Currently, there are two JupyterHub instances runnning:
- TeachingHub on SRVX1 - TeachingHub on SRVX1
- ResearchHub on Jet01 - ResearchHub on Jet01
Connect to either : Connect to either via the [wolke](https://wolke.img.univie.ac.at)
- [https://srvx1.img.univie.ac.at](https://srvx1.img.univie.ac.at) from everywhere
- [https://jet01.img.univie.ac.at](https://jet01.img.univie.ac.at) from within the university network.
## Screen
[Screen](https://wiki.ubuntuusers.de/Screen/) is terminal session manager, that allows to start processes and reconnect to these processes after disconnection.
This starts a new session
```bash title="Screen"
# open a interactive screen session
screen -S longjob
```
You can detach from this session with `CTRL + A D` and reconnect again with `screen -r`.
Multiple Sessions can be created and the output saved (`-L` Option).
![](../mkdocs/img/screen-cheatsheet.png)
## Tmux
[Tmux](https://wiki.ubuntuusers.de/tmux/) is a terminal multiplexer, that allows to open more consoles and allows to detach the session. It is much more complex and powerful compared to screen.
``` bash title="Terminal Multiplexer"
tmux
```
Launches a new virtual terminal, with `CTRL + B D` it can bed detached and with `tmux a` it can be reconnected.
![](https://linuxacademy.com/site-content/uploads/2016/08/tmux.png)
## Questions and Answers ## Questions and Answers
- [Q: How does ssh work?](Questions.md#q-how-does-ssh-work)
- [Q: How to use ssh-key authentication?](Questions.md#q-how-to-use-ssh-key-authentication) - [Q: How to use ssh-key authentication?](Questions.md#q-how-to-use-ssh-key-authentication)
- [Q: How to use an ssh-agent?](Questions.md#q-how-to-use-an-ssh-agent) - [Q: How to use an ssh-agent?](Questions.md#q-how-to-use-an-ssh-agent)
- [Q: How to transfer files between two VPN networks?](Questions.md#q-how-to-transfer-files-between-two-vpn-networks) - [Q: How to transfer files between two VPN networks?](Questions.md#q-how-to-transfer-files-between-two-vpn-networks)
- [Q: How to connect to Jet, SRVX8, SRVX2?](Questions.md#q-how-to-connect-to-jet-srvx8-srvx2) - [Q: How to connect to IMGW Servers?](Questions.md#q-how-to-connect-to-jet-srvx8-srvx2)
- [Q: How to mount a remote file system on Linux (MAC)?](Questions.md#q-how-to-mount-a-remote-file-system-on-Linux-mac) - [Q: How to mount a remote file system on Linux (MAC)?](Questions.md#q-how-to-mount-a-remote-file-system-on-Linux-mac)
## Tools ## Tools
Please find some useful tools for connecting to IMGW servers and University of Vienna VPN: Please find some useful tools for connecting to IMGW servers and University of Vienna VPN:
- BASH script using SSH to connect via a gateway, [SSH](SSH.md#connect-script) Download: [connect2jet](connect2jet) - Shell script using SSH to connect via a gateway, [SSH](SSH.md#connect-script) Download: [connect2jet](connect2jet)
- BASH script for 5fpc tools, [VPN](VPN.md#connect-script) Download: [connect2vpn](connect2vpn) - Shell script for F5FPC tools, [VPN](VPN.md#connect-script) Download: [connect2vpn](connect2vpn)
- Change VNC resolution, [VNC](VNC.md#xrandr) Download: [add_xrandr_resolution](add_xrandr_resolution.sh)
- Mount Server directories via sshfs, [SSHFS](SSH.md#sshfs) - Mount Server directories via sshfs, [SSHFS](SSH.md#sshfs)
??? note "connect2jet" ??? note "connect2jet"
...@@ -74,9 +47,3 @@ Please find some useful tools for connecting to IMGW servers and University of V ...@@ -74,9 +47,3 @@ Please find some useful tools for connecting to IMGW servers and University of V
``` bash title="Connect to VPN" ``` bash title="Connect to VPN"
--8<-- "SSH-VPN-VNC/connect2vpn" --8<-- "SSH-VPN-VNC/connect2vpn"
``` ```
??? note "add_xrandr_resolution"
``` bash title="Add a new monitor resolution to VNC"
--8<-- "SSH-VPN-VNC/add_xrandr_resolution.sh"
```
\ No newline at end of file
...@@ -16,36 +16,37 @@ on Linux and Mac, all tools are present. on Windows use one of these: ...@@ -16,36 +16,37 @@ on Linux and Mac, all tools are present. on Windows use one of these:
**Connect from the office** by typing either of the following in a terminal. Replace `[USERNAME]` with your own. **Connect from the office** by typing either of the following in a terminal. Replace `[USERNAME]` with your own.
```bash title="SSH commands" ```bash title="ssh connections"
ssh -X [USERNAME]@srvx1.img.univie.ac.at ssh [USERNAME]@srvx1.img.univie.ac.at
ssh -X [USERNAME]@srvx8.img.univie.ac.at ssh [USERNAME]@srvx8.img.univie.ac.at
ssh -X [USERNAME]@jet01.img.univie.ac.at ssh [USERNAME]@aurora.img.univie.ac.at
ssh -X [USERNAME]@131.130.157.215 ssh [USERNAME]@jet01.img.univie.ac.at
ssh -X [USERNAME]@jet02.img.univie.ac.at ssh [USERNAME]@jet02.img.univie.ac.at
ssh -X [USERNAME]@131.130.157.216
``` ```
There are multiple options for the ssh client, please explore these by yourself if needed. This option is sometimes needed to forward a window to your local computer (Linux, or advanced windows users):
The `-X` option enables X11 forwarding via ssh, i.e., permits opening graphical windows. On Windows you need to enter these details to the ssh client. The `-X` option enables X11 forwarding via ssh, i.e., permits opening graphical windows. On Windows you need to enter these details to the ssh client.
Consider using a `~/.ssh/config` configuration file to allow easier access like this: Consider using a `~/.ssh/config` configuration file to allow easier access like this:
```sh title="./ssh/config" ```sh title="./ssh/config"
Host * Host *
User [USERNAME]
ServerAliveInterval 60 ServerAliveInterval 60
ServerAliveCountMax 2 ServerAliveCountMax 2
Host srvx1 Host srvx1
HostName srvx1.img.univie.ac.at HostName srvx1.img.univie.ac.at
User [USERNAME]
Host srvx8 Host srvx8
HostName srvx8.img.univie.ac.at HostName srvx8.img.univie.ac.at
User [USERNAME]
Host jet Host jet
HostName jet01.img.univie.ac.at HostName jet01.img.univie.ac.at
User [USERNAME]
Host srvx2jet Host srvx2jet
HostName jet01.img.univie.ac.at HostName jet01.img.univie.ac.at
User [USERNAME]
ProxyJump srvx1.img.univie.ac.at ProxyJump srvx1.img.univie.ac.at
Host login Host login
HostName login.univie.ac.at HostName login.univie.ac.at
...@@ -53,6 +54,7 @@ Host login ...@@ -53,6 +54,7 @@ Host login
Host ecaccess Host ecaccess
Host ecaccess.ecmwf.int Host ecaccess.ecmwf.int
HostKeyAlgorithms ssh-dss HostKeyAlgorithms ssh-dss
User [ECMWF USERNAME]
KexAlgorithms diffie-hellman-group1-sha1 KexAlgorithms diffie-hellman-group1-sha1
Ciphers aes256-cbc Ciphers aes256-cbc
Host jump.ecmwf.int shell.ecmwf.int Host jump.ecmwf.int shell.ecmwf.int
...@@ -72,6 +74,7 @@ Host a?-* a??-* hpc-* hpc2020-* ecs-* ...@@ -72,6 +74,7 @@ Host a?-* a??-* hpc-* hpc2020-* ecs-*
User [ECMWF USERNAME] User [ECMWF USERNAME]
ProxyJump jump.ecmwf.int ProxyJump jump.ecmwf.int
``` ```
and replacing `[USERNAME]` and `[u:account USERNAME]` with your usernames. Using such a file allows to connect like this `ssh srvx1` using the correct server adress and specified username. Copy this file as well on `login.univie.ac.at` and you can use commands like this: `ssh -t login ssh jet` to connect directly to `jet` via the `login` gateway. and replacing `[USERNAME]` and `[u:account USERNAME]` with your usernames. Using such a file allows to connect like this `ssh srvx1` using the correct server adress and specified username. Copy this file as well on `login.univie.ac.at` and you can use commands like this: `ssh -t login ssh jet` to connect directly to `jet` via the `login` gateway.
Please note the special algorithms for ecaccess and of course ECMWF uses [teleport](../ECMWF.md#connecting-to-ecmwf-services) now. Please note the special algorithms for ecaccess and of course ECMWF uses [teleport](../ECMWF.md#connecting-to-ecmwf-services) now.
...@@ -84,9 +87,10 @@ If you are a guest, you can apply for a [guest u:account](https://zid.univie.ac. ...@@ -84,9 +87,10 @@ If you are a guest, you can apply for a [guest u:account](https://zid.univie.ac.
## SSH Authentication with keys ## SSH Authentication with keys
If you want to use ssh-keys you can also use different keys in `.ssh/config` per server with `IdentityFile ~/.ssh/id_rsa_for_server`. If you want to use ssh-keys you can also use different keys in `.ssh/config` per server with `IdentityFile ~/.ssh/id_enc_for_server`.
**Note: If you are having trouble with your ssh-key being accepted. You need to check the selinux security policy. Because we use non standard home directories, which are not accepted by default.** Do that by running: **Note: If you are having trouble with your ssh-key being accepted. You need to check the selinux security policy. Because we use non standard home directories, which are not accepted by default.** Do that by running:
```bash ```bash
ls -ldZ ~/.ssh/ ls -ldZ ~/.ssh/
drwx------. 2 monkey users system_u:object_r:ssh_home_t:s0 4.0K Aug 18 21:24 .ssh/ drwx------. 2 monkey users system_u:object_r:ssh_home_t:s0 4.0K Aug 18 21:24 .ssh/
...@@ -94,6 +98,7 @@ drwx------. 2 monkey users system_u:object_r:ssh_home_t:s0 4.0K Aug 18 21:24 .ss ...@@ -94,6 +98,7 @@ drwx------. 2 monkey users system_u:object_r:ssh_home_t:s0 4.0K Aug 18 21:24 .ss
ls -lZ ~/.ssh/ ls -lZ ~/.ssh/
-rw-------. 1 monkey users unconfined_u:object_r:ssh_home_t:s0 1.3K Aug 17 10:58 authorized_keys -rw-------. 1 monkey users unconfined_u:object_r:ssh_home_t:s0 1.3K Aug 17 10:58 authorized_keys
``` ```
If you are not seeing `ssh_home_t` than you need to run `restorecon -Rv ~/.ssh/` which should update the security policy and allow for the SSH-Daemon to accept your keys. If you are not seeing `ssh_home_t` than you need to run `restorecon -Rv ~/.ssh/` which should update the security policy and allow for the SSH-Daemon to accept your keys.
**Please add your ssh-keys via [IPA](IPA.md#add-ssh-key)** **Please add your ssh-keys via [IPA](IPA.md#add-ssh-key)**
......
...@@ -9,12 +9,15 @@ The VNC (Virtual Network Computing) allows to view a graphical user interface (G ...@@ -9,12 +9,15 @@ The VNC (Virtual Network Computing) allows to view a graphical user interface (G
Xvnc is the Unix VNC server. Applications can display themselves on Xvnc as if it were a normal display, but they will appear on any connected VNC viewers rather than on a physical screen. The VNC protocol uses the TCP/IP ports 5900+N, where N is the display number. Xvnc is the Unix VNC server. Applications can display themselves on Xvnc as if it were a normal display, but they will appear on any connected VNC viewers rather than on a physical screen. The VNC protocol uses the TCP/IP ports 5900+N, where N is the display number.
Currently VNC is installed on: Currently VNC is installed on:
- SRVX8, mainly Staff - SRVX8 [staff]
- JET01, mainly Researchers - JET01 [staff]
- AURORA [staff]
## Userservices ## Userservices
It is highly recommended to use the userservices scripts available on all IMGW Servers to make configurations for VNC. It is highly recommended to use the userservices scripts available on all IMGW Servers to make configurations for VNC.
```bash
```bash title="using the userservices"
$ userservices vnc -h $ userservices vnc -h
################################################################################ ################################################################################
User Services - VNC Server Setup/Launcher/Stopper User Services - VNC Server Setup/Launcher/Stopper
...@@ -38,10 +41,12 @@ Path: /home/swd/userservices/userservices.d ...@@ -38,10 +41,12 @@ Path: /home/swd/userservices/userservices.d
Installed Desktops: icewm-session Installed Desktops: icewm-session
################################################################################ ################################################################################
``` ```
Running the script without any options will run all necessary steps. In case of error try removing your `.vnc` directory, as older configurations might be in the way. There shall be at least two desktop options: icewm and xfce. You can specify this directly with the `-w [DESKTOP]` option. Running the script without any options will run all necessary steps. In case of error try removing your `.vnc` directory, as older configurations might be in the way. There shall be at least two desktop options: icewm and xfce. You can specify this directly with the `-w [DESKTOP]` option.
## Setup - Manual ## Setup - Manual
Please consider using the `userservices vnc` script to do this setup. Please consider using the `userservices vnc` script to do this setup.
First of all check if a VNC server is already running or not. Depending on the results you have two options: First of all check if a VNC server is already running or not. Depending on the results you have two options:
...@@ -65,15 +70,19 @@ vncserver ...@@ -65,15 +70,19 @@ vncserver
``` ```
### Jet Cluser ### Jet Cluser
on Jet there are the user services available to you: on Jet there are the user services available to you:
```bash ```bash
# Help information on VNC userservice # Help information on VNC userservice
userservices vnc -h userservices vnc -h
``` ```
if you have never used this script or a `vncserver` just running `userservices vnc` will be enough for you to setup the VNC service. It will tell you which port you have been automatically assigned. if you have never used this script or a `vncserver` just running `userservices vnc` will be enough for you to setup the VNC service. It will tell you which port you have been automatically assigned.
### Connecting, setting the window manager ### Connecting, setting the window manager
Use a VNC client (e.g. [TigerVNC Viewer](https://sourceforge.net/projects/tigervnc/files/stable/) (All OS) or [RealVNC VNC Viewer](https://www.realvnc.com/en/connect/download/viewer/windows/) on Windows, or [Remmina](https://remmina.org/) on any Linux distribution) and connect to `srvx?.img.univie.ac.at:[DISPLAY]`. Use a VNC client (e.g. [TigerVNC Viewer](https://sourceforge.net/projects/tigervnc/files/stable/) (All OS) or [RealVNC VNC Viewer](https://www.realvnc.com/en/connect/download/viewer/windows/) on Windows, or [Remmina](https://remmina.org/) on any Linux distribution) and connect to `srvx?.img.univie.ac.at:[DISPLAY]`.
**Note: Since version 1.10 of tigerVNC it is possible to resize the windows directly** **Note: Since version 1.10 of tigerVNC it is possible to resize the windows directly**
...@@ -85,6 +94,7 @@ Connect with a viewer: ...@@ -85,6 +94,7 @@ Connect with a viewer:
3. VNC passwort (`vncpasswd` allows to set this on the server) 3. VNC passwort (`vncpasswd` allows to set this on the server)
The VNC server will automatically start a graphical window manager/desktop environment. Details are stored in `.vnc/xstartup`. Your default configuration will load GNOME, and `.vnc/xstartup` will be something like this: The VNC server will automatically start a graphical window manager/desktop environment. Details are stored in `.vnc/xstartup`. Your default configuration will load GNOME, and `.vnc/xstartup` will be something like this:
```bash ```bash
#!/bin/sh #!/bin/sh
...@@ -99,7 +109,9 @@ if [ -e /usr/bin/gnome-session ]; then ...@@ -99,7 +109,9 @@ if [ -e /usr/bin/gnome-session ]; then
vncserver -kill $DISPLAY vncserver -kill $DISPLAY
fi fi
``` ```
It is recommended to use IceWM instead. This is a more lightweight alternative to GNOME. To switch to IceWM, first log out from GNOME and disconnect from your VNC viewer, then change your `.vnc/xstartup` to this: It is recommended to use IceWM instead. This is a more lightweight alternative to GNOME. To switch to IceWM, first log out from GNOME and disconnect from your VNC viewer, then change your `.vnc/xstartup` to this:
```bash ```bash
#!/bin/sh #!/bin/sh
...@@ -109,12 +121,14 @@ vncconfig -iconic & ...@@ -109,12 +121,14 @@ vncconfig -iconic &
xterm -geometry -sb -sl 500 -fn 9x15bold -title "$VNCDESKTOP Desktop" & xterm -geometry -sb -sl 500 -fn 9x15bold -title "$VNCDESKTOP Desktop" &
icewm & icewm &
``` ```
Some information on what could be put into `.Xresources` is given [here](https://wiki.archlinux.org/title/x_resources). It might be possible to replace `icewm` here with `startxfce4` to choose XFCE Desktop environment. Some information on what could be put into `.Xresources` is given [here](https://wiki.archlinux.org/title/x_resources). It might be possible to replace `icewm` here with `startxfce4` to choose XFCE Desktop environment.
### VNC as a Service ### VNC as a Service
This is only here for reference, on SRVX2 and Jet use the `userservices vnc`. This is only here for reference, on SRVX2 and Jet use the `userservices vnc`.
Setup, replace `[DISPLAY]` with an appropriate number, e.g. `3`: Setup, replace `[DISPLAY]` with an appropriate number, e.g. `3`:
```bash ```bash
mkdir -p ~/.config/systemd/user mkdir -p ~/.config/systemd/user
cp /usr/lib/systemd/user/vncserver@.service ~/.config/systemd/user/ cp /usr/lib/systemd/user/vncserver@.service ~/.config/systemd/user/
...@@ -125,6 +139,7 @@ loginctl enable-linger ...@@ -125,6 +139,7 @@ loginctl enable-linger
``` ```
Finally stop/restart the VNC server. Finally stop/restart the VNC server.
```bash ```bash
# Restart # Restart
systemctl --user restart vncserver@:[DISPLAY].service --now systemctl --user restart vncserver@:[DISPLAY].service --now
...@@ -137,6 +152,7 @@ systemctl --user enable vncserver@:[DISPLAY].service --now ...@@ -137,6 +152,7 @@ systemctl --user enable vncserver@:[DISPLAY].service --now
``` ```
Monitor the status: Monitor the status:
```bash ```bash
# Like this # Like this
systemctl --user status vncserver@:[DISPLAY].service systemctl --user status vncserver@:[DISPLAY].service
...@@ -169,10 +185,20 @@ Change the resolution to e.g. 1920x1080 (HD): ...@@ -169,10 +185,20 @@ Change the resolution to e.g. 1920x1080 (HD):
xrandr -s 1920x1080 -d $DISPLAY xrandr -s 1920x1080 -d $DISPLAY
``` ```
Adding resolutions according to your display's resolution have a look here: [add_xrandr_resolution.sh](add_xrandr_resolution.sh) Adding resolutions according to your display's resolution have a look at
```bash
??? note "add_xrandr_resolution"
``` bash title="Add a new monitor resolution to VNC"
--8<-- "SSH-VPN-VNC/add_xrandr_resolution.sh"
```
and run it like this:
```bash title="Add a custom resolution to your virtual display"
# running the script and adding a resolution you require, in pixel # running the script and adding a resolution you require, in pixel
$ add_xrandr_resolution [width] [height] $ add_xrandr_resolution [width] [height]
``` ```
Note: `$DISPLAY` is an environment variable that is usually set to your VNC server port. Note: `$DISPLAY` is an environment variable that is usually set to your VNC server port.
...@@ -12,7 +12,7 @@ Links: ...@@ -12,7 +12,7 @@ Links:
* [ZID-VPN](https://vpn.univie.ac.at/f5-w-68747470733a2f2f7a69642e756e697669652e61632e6174$$/vpn/) * [ZID-VPN](https://vpn.univie.ac.at/f5-w-68747470733a2f2f7a69642e756e697669652e61632e6174$$/vpn/)
* Linux (Ubuntu, Generic), Windows, Mac: [VPN user guides](https://vpn.univie.ac.at/f5-w-68747470733a2f2f7a69642e756e697669652e61632e6174$$/vpn/anleitungen/) * Linux (Ubuntu, Generic), Windows, Mac: [VPN user guides](https://vpn.univie.ac.at/f5-w-68747470733a2f2f7a69642e756e697669652e61632e6174$$/vpn/anleitungen/)
* ~~Arch based AUR package [AUR f5fpc](https://aur.archlinux.org/packages/f5fpc/)~~ broken. Download via `yay`, replace `.deb` with `.deb` from ZID. `makepkg --skipinteg -i`. * Arch based AUR package [AUR f5fpc](https://aur.archlinux.org/packages/f5fpc/) does not contain the correct f5fpc. You can download via `yay -Sw f5fpc`, replace `.deb` with `.deb` from ZID. Then build the package yourself: `makepkg --skipinteg -i`.
Follow the install instructions for Windows, Mac and Linux and make sure the software works. Follow the install instructions for Windows, Mac and Linux and make sure the software works.
...@@ -48,4 +48,4 @@ $ connect2vpn [u:account username] ...@@ -48,4 +48,4 @@ $ connect2vpn [u:account username]
[VPN] Full (1) or split (None) tunnel? (1/None): [VPN] Full (1) or split (None) tunnel? (1/None):
``` ```
Continue and wait until you get a response that it's connected. Continue and wait until you get a response that it's connected.
The status stays visible. The status stays visible. This will block your current terminal window.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment