Skip to content
Snippets Groups Projects
Commit 32ccb851 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

updates for srvx1,aurora, connection

parent 94d2d993
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 58 additions and 309 deletions
Servers/Aurora.md
+ 13
− 0
View file @ 32ccb851
......@@ -47,6 +47,19 @@ Steps:
-----------------------------------------------
```
The SRV Storage System (2x 400 TB raid protected storage array) is connected to Aurora, as well as the SSD Storage system (11 TB raid protected SSD storage array).
Paths:
- `/srvfs/home/[username]`
- `/srvfs/scratch/[username]`
- `/srvfs/tmp/[username]`
- `/srvfs/data`
- `/srvfs/shared`
- `/srvfs/webdata`
- `/srvfs/fastscratch/[username]` (SSD)
## Software
The typcial installation of a server has the INTEL Compiler suite (`intel-oneapi`), the open source GNU Compilers and AMD compilers (`aocc`) installed. Based on these three different compilers (`intel`, `gnu`, `aocc`), there are usually two version of each scientific software.
......
Servers/README.md
+ 22
− 11
View file @ 32ccb851
......@@ -2,17 +2,18 @@
The Department of Meteorology and Geophysics has access to the following computing resources:
- Teaching and Development Server ([SRVX1](SRVX1.md))
- Remote Desktop Server ([SRVX8](SRVX8.md))
- Computing Cluster ([JET](JET.md))
- Vienna Scientific Cluster ([VSC](../VSC.md))
- Teaching and Development Server [SRVX1](SRVX1.md)
- Development Server [Aurora](Aurora.md)
- Services Server [SRVX8](SRVX8.md)
- Computing Cluster [JET](JET.md)
- Vienna Scientific Cluster [VSC](../VSC.md)
## Available Work Environments
Locations:
- Staff + Students [SRVX1](SRVX1.md), [TeachingHub](../Students/TeachingHub.md)
- Staff + Remote Desktop [SRVX8](SRVX8.md)
- Students [SRVX1](SRVX1.md), [TeachingHub + MasterHub](../Students/TeachingHub.md)
- Staff Development [Aurora](Aurora.md)
- Staff + Remote Desktop + Jupyterhub [Jet Cluster](JET.md)
- Staff [Vienna Scientific Cluster (VSC)](../VSC.md)
- [VSC Training](https://vsc.ac.at/training)
......@@ -30,17 +31,27 @@ _Note: Please take a look at the training course @VSC. There might be a beginner
## Services
Summary of services available @IMGW.
Summary of services available @IMGW for staff and students. Some of these services require different accounts ([info](https://wolke.img.univie.ac.at/accounts.html)) or nework access.
| name | description | url/path | access | host |
| ----------- | ------------------------------------- | -------------------------------------------------------------------------- | --- | ----- |
| teachinghub | Jupyterlab for Teaching & Development | [teaching.wolke.img.univie.ac.at](https://teaching.wolke.img.univie.ac.at) | web | srvx1 |
| researchhub | Jupyterlab for Development | [jupyter.wolke.img.univie.ac.at](https://jupyter.wolke.img.univie.ac.at) | img/vpn | jet02 |
| name | description | url/path | access | host |
| ----------- | ------------------------------------- | ------------------------------------------------------------------------------- | ------- | ----- |
| teachinghub | Jupyterlab for Teaching & Development | [teaching.wolke.img.univie.ac.at](https://teaching.wolke.img.univie.ac.at) | web | SRVX1 |
| MasterHub | Jupyterlabfor Master Students | [quell.wolke.img.univie.ac.at](https://quell.wolke.img.univie.ac.at) | web | SRVX1 |
| ecgateway | ECMWF Member State Gateway | [ecaccess.wolke.img.univie.ac.at](https://ecaccess.wolke.img.univie.ac.at/) | img/vpn | JET02 |
| researchhub | Jupyterlab for Development | [jupyter.wolke.img.univie.ac.at](https://jupyter.wolke.img.univie.ac.at) | img/vpn | JET02 |
| webdata | File server (/srvfs/webdata) | [webdata.wolke.img.univie.ac.at](https://webdata.wolke.img.univie.ac.at) | web | SRVX8 |
| YoPass | Secure Message Service (YoPass) | [secure.wolke.img.univie.ac.at](https://secure.wolke.img.univie.ac.at/) | web | SRVX8 |
| TransferSH | Easy commandline file transfer/send | [transfersh.wolke.img.univie.ac.at](https://transfersh.wolke.img.univie.ac.at/) | web | SRVX8 |
| Uptime | Server Monitoring | [uptime.wolke.img.univie.ac.at](https://uptime.wolke.img.univie.ac.at/) | web | SRVX8 |
| IPA | User Management | [wolke.img.univie.ac.at/ipa/ui](https://wolke.img.univie.ac.at/ipa/ui) | img/vpn | WOLKE |
## Containers
🚧🚧🚧🚧🚧🚧(ongoing)🚧🚧🚧🚧🚧
Summary of containerized applications available @IMGW.
Containerization is done via singularity/apptainer. A some what incomplete introduction is given [here](https://gitlab.phaidra.org/imgw/singularity).
Servers/SRVX1.md
+ 4
− 95
View file @ 32ccb851
# S R V X 1
> Teaching Hub and Access Point from the web
> Teaching Hub and Master Hub from the web
>
## Getting started
Steps:
1. Request access / will be done for you by your supervisor.
2. As Staff, access using SSH - [How to SSH / VNC / VPN](../SSH-VPN-VNC/README.md)
3. As Student, access using Teaching Hub - [How to connect using the TeachingHub](../Students/TeachingHub.md)
1. Students access via Moodle - [How to connect using the TeachingHub](../Students/TeachingHub.md)
2. Master students via [MasterHub](https://quell.wolke.img.univie.ac.at). Supervisor requests access.
## System information
| Name | Value |
......@@ -36,23 +36,6 @@ Steps:
----------------------------------------------
```
## Jupyterhub
<img src="../mkdocs/img/jupyterhub-logo.svg" width="150px">
SRVX1 serves a teaching [jupyterhub](https://jupyterhub.readthedocs.io/en/stable/) with a [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/). It allows easy access for students and teachers. Access: [https://teachinghub.wolke.img.univie.ac.at](https://teachinghub.wolke.img.univie.ac.at)
Signup is only granted by teachers and requires a srvx1 user account. A new password is needed and a TOTP (time base one-time password) will be created.
Download/Use any of these required Authenticator Apps:
- [2FAS (Mobile, recommended)](https://2fas.com/)
- [KeepassX (Desktop)](https://www.keepassx.org/)
- [FreeOTP (Mobile)](https://freeotp.github.io/)
- [Google Auth (Mobile)](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2)
**Please note that having a synchronized time on the TOTP device is required. Otherwise the time-based passwords will be out of sync and the authentication fails.**
After registering the teacher/admin has to grant you access and you can login.
## Software
The typcial installation of a intel-server has the INTEL Compiler suite (`intel-parallel-studio`, `intel-oneapi`) and the open source GNU Compilers installed. Based on these two different compilers (`intel`, `gnu`), there are usually two version of each scientific software.
......@@ -145,77 +128,3 @@ enstools/v2021.11 micromamba/0.27.0 teleport/10.3.3
idl/8.2-sp1 ncl/6.6.2 xconv/1.94
```
on how to use environment modules go to [Using Environment Modules](../Misc/Environment-Modules.md)
## User services
There is a script collection that is accessible via the `userservices` command. e.g. running
```bash
$ userservices
Usage: userservices [service] [Options]
Available Services:
------------------------------------------------------------------------
archive --- Submit files/folders to ZID Archive
fetch-sysinfo --- Display system information
filesender --- Transfer files to ACONET filesender (requires account)
fix-permissions --- fix file/directory permissions
home-dir-check --- Check home directory/configuration
modules --- Pretty print environment modules
transfersh --- Transfer files/directories (IMGW subnet)
weather --- Retrieve weather information
yopass --- Send messages/small files to YoPass (encrypted)
------------------------------------------------------------------------
These scripts are intended to help with certain known problems.
Report problems to: michael.blaschek@univie.ac.at
```
These are scripts in a common directory. Feel free to copy or edit as you like. Note that some services like `filesender` require an [ACONET](https://filesender.aco.net/) account (accessible via your [u:account](https://zid.univie.ac.at/my-uaccount/)).
## Container Hub
Currently there is the possibility to run [singularity/apptainer](https://singularity.hpcng.org/) containers on all our Servers. This is really similar to docker, but much more secure for multi-user servers. Almost every docker container can be converted into a singularity container. Some of the build recipes use docker.
There are a number of prepared containers but more can be added. If you have a wish or an existing container useful for others please share.
```yaml
containers:
- root: /home/swd/containers
- available:
- RTTOV:
- RTTOV: 12.3
- compiler: gcc:7.3.0 (anaconda)
- path: /home/swd/containers/rttov-jupyter/jup3rttov.sif
- os: centos:6.10
- python: 3.7.4
- singularity: 3.5.2
- packages:
- anaconda3
- jupyter jupyterlab numpy matplotlib pandas xarray bottleneck dask numba scipy netcdf4 cartopy h5netcdf nc-time-axis cfgrib eccodes nodejs
- apps:
- atlas
- lab
- notebook
- rtcoef
- rthelp
- rttest
- description: Use for running RTTOV simulations with Python.
- LIBEMOS:
- LIBEMOS: 4.5.1
- compiler: gcc:7.5.0
- path: /home/swd/containers/libemos-dev/libemos-dev.sif
- os: ubuntu:18.04
- singularity: 3.8.6
- packages:
- openmpi:2.1.1-8
- fftw3:3.3.7-1
- eccodes:2.6.0
- libemos
- description: Use for building flexextract with working libemos dependencies.
- TEXLIVE:
- TEXLIVE: TeX Live 2022
- compiler: gcc:12.2.0
- path: /home/swd/containers/texlive/texlive-2022.sif
- os: debian 12
- singularity: 3.8.7
- packages:
- texlive from gitlab
```
Servers/SRVX8.md
+ 3
− 106
View file @ 32ccb851
# S R V X 8
> Remote Desktop and Virtual Machines
> Services and Virtual Machines
>
## Getting Started
Steps:
1. Request access
2. Access using SSH - [How to SSH / VNC / VPN](../SSH-VPN-VNC/README.md)
## System Information
| Name | Value |
......@@ -39,102 +32,6 @@ Steps:
----------------------------------------------
```
## Software
The typcial installation of a intel-server has the INTEL Compiler suite (`intel-parallel-studio`, `intel-oneapi`) and the open source GNU Compilers installed. Based on these two different compilers (`intel`, `gnu`), there are usually two version of each scientific software.
Major Libraries:
- OpenMPI (3.1.6, 4.0.5)
- HDF5
- NetCDF (C, Fortran)
- ECCODES from [ECMWF](https://confluence.ecmwf.int/display/ECC)
- Math libraries e.g. intel-mkl, lapack,scalapack
- Interpreters: Python, Julia
- Tools: cdo, ncl, nco, ncview
These software libraries are usually handled by environment modules.
![](../mkdocs/img/envmodules.png)
## Currently installed modules
on how to use environment modules go to [Using Environment Modules](../Misc/Environment-Modules.md)
```sh title="Environment Modules"
module av
--- /home/swd/spack/share/spack/modules/linux-rhel7-haswell
anaconda2/2019.10-gcc-8.4.0
anaconda3/2021.05-gcc-8.4.0
eccodes/2.21.0-gcc-8.4.0
gcc/8.4.0-gcc-4.8.5
git/1.8.3.1-gcc-8.4.0
git/2.31.1-gcc-8.4.0
hdf5/1.10.7-gcc-8.4.0
hdf5/1.12.0-gcc-8.4.0
intel-oneapi-compilers/2021.3.0-oneapi-2021.3.0
intel-oneapi-mkl/2021.3.0-oneapi-2021.3.0
intel-oneapi-mpi/2021.3.0-oneapi-2021.3.0
miniconda2/4.7.12.1-gcc-8.4.0
miniconda3/4.10.3-gcc-8.4.0
ncl/6.5.0-gcc-8.4.0
ncl/6.6.2-gcc-8.4.0
nco/4.9.3-gcc-8.4.0
ncview/2.1.8-gcc-8.4.0
netcdf-c/4.6.3-gcc-8.4.0
netcdf-c/4.7.4-gcc-8.4.0
netcdf-fortran/4.5.2-gcc-8.4.0
netcdf-fortran/4.5.3-gcc-8.4.0
netlib-lapack/3.9.1-gcc-8.4.0
netlib-scalapack/2.1.0-gcc-8.4.0
openblas/0.3.17-gcc-8.4.0
openmpi/3.1.6-gcc-8.4.0
openmpi/4.0.5-gcc-8.4.0
openmpi/4.1.1-gcc-8.4.0
proj/8.1.0-gcc-8.4.0
python/3.8.9-gcc-4.8.5
---/home/swd/modules
micromamba/latest
```
## User services
There is a script collection that is accessible via the `userservices` command. e.g. running
```bash title="userservices"
userservices
Usage: userservices [service] [Options]
Available Services:
------------------------------------------------------------------------
archive --- Submit files/folders to ZID Archive
fetch-sysinfo --- Display system information
filesender --- Transfer files to ACONET filesender (requires account)
fix-permissions --- fix file/directory permissions
home-dir-check --- Check home directory/configuration
modules --- Pretty print environment modules
sysinfo --- Display system information
transfersh --- Transfer files/directories (IMGW subnet)
vnc --- VNC Server Setup/Launcher/Stopper
vnc-geometry --- Change geometry of VNC display
weather --- Retrieve weather information
------------------------------------------------------------------------
These scripts are intended to help with certain known problems.
Report problems to: michael.blaschek@univie.ac.at
```
These are scripts in a common directory. Feel free to copy or edit as you like. Note that some services like filesender require an **ACONET account** (accessible via your u:account). **Please note the available VNC services** for the remote desktop environment.
For VNC the user can choose between installed desktops:
- xfce
- icewm (default)
## Virtual Machine Hub
Currently the system acts as a virtual machine host.
Active:
## Services
- VERA
- Ubuntu Geographic
There are a number of services running that are just forwarded to the WOLKE. Find some more information in [services](README.md#services)
Students/README.md
+ 0
− 3
View file @ 32ccb851
......@@ -2,9 +2,6 @@
To make you life easier we have listed a few things here that you should know to make good use of the IT services at our department.
???+ warning "since winter semester 2023"
There is the [old teachinghub](./TeachingHub.md) (will retire 31.3.2024) and the new teachinghub, which is handled via Moodle. Your lecturerer will give you access.
## Guidelines
Access to computational resources at IMGW is granted to the user by employment at the university or by a IMGW sponsor. Access is disabled on the termination date without further notice. An extension of the allocation may be negotiated with the responsible Admin or IMGW sponsor. The user accepts the following responsibilities:
......
Students/TeachingHub.md
+ 15
− 90
View file @ 32ccb851
# Teaching Hub
???+ warning inline end "Update"
There is an old and a new TeachingHub.
<img src="../mkdocs/img/jupyterhub-logo.svg" width="100px">
# Jupyter Notebook Lab
<img src="../mkdocs/img/jupyter-logo.png" width="300px">
Please notice that only the new teaching hub is available in the future.
Migration for the old teaching hub will finish at the end of WS2023/24.
Access: [teaching.wolke.img.univie.ac.at](https://teaching.wolke.img.univie.ac.at) or via your [Moodle lecture](https://moodle.univie.ac.at/my/).
<img src="../mkdocs/img/moodlelogo_small.png">
<img align="right" width="150px" src="../mkdocs/img/Masterhub_logo.png">
## MasterHub
Students need to get a server account for accessing this resource. However, it is intended to help students complete their Masters and have an easy accessible Jupyterhub. Your Supervisor can ask for an account for you.
[Master Hub](https://quell.wolke.img.univie.ac.at)
## TeachingHub
<img width="200px" src="../mkdocs/img/moodlelogo_small.png">
???+ warning "Purge"
It is necessary for a smooth operation to purge the contents of the TeachingHub on an annual basis. Every year before the winter semester starts there will be a purge of students home directories.
What is available?
Authentication is now enabled via moodle. Every student can use their u:account to login into moodle and be forwarded to the new teaching hub. no more login required.
Authentication is now enabled via moodle. Every student can use their u:account to login into moodle and be forwarded to the new teaching hub. no more login required.
![TeachingHub access via Moodle](../mkdocs/img/teachinghub-new-moodle.png)
......@@ -31,86 +39,3 @@ It is also possible to use a VNC (Virtual Desktop) to open installed gui applica
![TeachingHub vnc desktop](../mkdocs/img/teachinghub-new-vnc.png)
## OLD TeachingHub
OLD: [teachinghub.wolke.img.univie.ac.at](https://teachinghub.wolke.img.univie.ac.at)
### How to get access?
#### Students
Usually **your teacher** is going to **sign you up** and you will receive one or two emails containing information on how to login.
You will receive access to the Teaching Hub usually for 6 month. Can be extended for good reasons or until a teacher signs you up again.
#### Staff
Request an account by writing to the [IT](mailto:it.img-wien@univie.ac.at), which will extend your existing server accout.
The email you received contains credentials for the Teaching Hub. This email contains your username, a new password (only for the Teaching Hub) and a secret (16 digit code). They look like this:
```
Dear TeachingHub User Monkey
Some of your user information has been updated.
TeachingHub Account Specification:
servers: https://srvx1.img.univie.ac.at
username: monkey
password: ABCDEFGHIJK (signed in, click on user name, upper right corner to change)
Email: a000000@univie.ac.at
Secret: 1A2B3C4D5E6F7G8H
Please note that the TeachingHub is open from the web.
Two-factor Authentication is required. Download any TOTP (Time based One Time Passwords) app according to your OS.
...
```
As stated in the email you received, you need to download one of these TOTP apps:
- [2FA Authenticator](https://2fas.com) :rocket: (android, ios)
- [FreeOTP](https://freeotp.github.io/)
- [Google Auth](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2)
- or any other (search `2fa apps` and you will find plenty)
I recommend using your phone and an app there, as this is the most secure way. And with you phone you can scan the qr code, which looks like this:
<img src="../mkdocs/img/teachinghub-qrcode.png" width="300px">
When you scan to your Authenticator App, it will automatically get the details (username, Secret) required to generate every 30 seconds a Time-based One Time Password (TOTP). Like this:
![](../mkdocs/img/teachinghub-authenticator.png)
If you want to use a desktop or other app that might not scan QR codes, you can fill in the details manually like this:
- Provider: `TeachingHub`
- Account: `[user]` your username, here `monkey`
- Token or Secret or Hash: `1A2B3C4D5E6F7G8H` is the secret from your email.
![](../mkdocs/img/teachinghub-authenticator-manual.png)
The combination of username, password and TOTP needs to be used to sign in:
![](../mkdocs/img/teachinghublogin.png)
This sounds all really complicated, but I promise this will be easier from time to time. Don't forget that this for our all security and your contribution is appreciated! Thanks.
If you have trouble with login, please check if you device has time synchronization activated, otherwise you might just get wrong codes. (Nice for checking: [totp.app](https://totp.app) *However, be warned this is an online service and your data might not be as secure as you think!*)
Now you are ready to go to the teaching hub and sign in:
[Teaching Hub](https://srvx1.img.univie.ac.at)
### Teaching Hub Notebooks and Kernels
Currently there are several Notebooks with different Kernels (Python kernels) installed at SRVX1. Look at the logos, if SRVX1 is in there then you will have access to these notebooks and kernels. If SRVX2 or SRVX8 are in the logo you might not have access as this is for researchers.
![](../mkdocs/img/teachinghubnotebooks.png)
As you can see it is also possible to launch a terminal inside the TeachingHub and you will find you files from your home directory there.
It might be that your course has a separate Notebook/Kernel for you, with installed packages and dependencies required for this specific course. However, these Notebooks/Kernels are only available during these lectures and might be removed in the future. So be aware of that. Note that you can easily change a kernel in your notebook. So no need to delete, copy or do anything. Your data/notebook information is safe.
![](../mkdocs/img/teachinghubnotebooks_kernels.png)
mkdocs/img/IMGW-connection-diagram.dot
+ 1
− 4
View file @ 32ccb851
......@@ -3,8 +3,7 @@ digraph Servers {
node [shape=none labelloc="b" height="1" imagepos="tc"]
users [image="student_sm.png" label="User"]
login [image="server_sm.png" label="login"]
aurora [image="server_sm.png" label="AURORA"]
login [image="server_sm.png" label="login/AURORA"]
jet [image="server_sm.png" label="JET"]
VPN [image="local-area-network_sm.png" label="VPN"]
vsc [image="logo_vsc_sm.png" label="VSC" height=".8"]
......@@ -12,9 +11,7 @@ digraph Servers {
users -> login
users -> VPN
login -> jet
login -> aurora
login -> vsc
VPN -> jet
VPN -> aurora
VPN -> vsc
}
\ No newline at end of file
mkdocs/img/IMGW-connection-diagram.png
+ 0
− 0
View replaced file @ 94d2d993 View file @ 32ccb851
mkdocs/img/IMGW-connection-diagram.png

32.7 KiB | W: | H:

mkdocs/img/IMGW-connection-diagram.png

28.6 KiB | W: | H:

mkdocs/img/IMGW-connection-diagram.png
mkdocs/img/IMGW-connection-diagram.png
mkdocs/img/IMGW-connection-diagram.png
mkdocs/img/IMGW-connection-diagram.png
  • 2-up
  • Swipe
  • Onion skin
mkdocs/img/IMGW-connection-diagram2.png
+ 0
− 0
View replaced file @ 94d2d993 View file @ 32ccb851
mkdocs/img/IMGW-connection-diagram2.png

21.5 KiB | W: | H:

mkdocs/img/IMGW-connection-diagram2.png

21.4 KiB | W: | H:

mkdocs/img/IMGW-connection-diagram2.png
mkdocs/img/IMGW-connection-diagram2.png
mkdocs/img/IMGW-connection-diagram2.png
mkdocs/img/IMGW-connection-diagram2.png
  • 2-up
  • Swipe
  • Onion skin
mkdocs/img/Masterhub_logo.png 0 → 100644
+ 0
− 0
View file @ 32ccb851
mkdocs/img/Masterhub_logo.png

21.5 KiB

mkdocs/img/jupyter-logo.png 0 → 100644
+ 0
− 0
View file @ 32ccb851
mkdocs/img/jupyter-logo.png

69.6 KiB

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment