Skip to content
Snippets Groups Projects
Commit 0e460638 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

Merged data update

parents 08fa2757 f0e4d868
No related branches found
No related tags found
No related merge requests found
Showing
with 592 additions and 521 deletions
# Teaching Hub @ IMGW
# Teaching Hub
<img src="https://jupyter.org/assets/hublogo.svg" width="300px">
<img src="./mkdocs/img/jupyterhub-logo.svg" width="100px">
- Address: [Teaching Hub](https://srvx1.img.univie.ac.at)
- Access to the Teaching Hub can be done only via a browser.
- Access to SRVX1 via a terminal (Putty,...)
## Getting Started
Address: [https://srvx1.img.univie.ac.at/hub](https://srvx1.img.univie.ac.at/hub)
[[_TOC_]]
Access to the Teaching Hub in a browser (mobile or desktop)
## How to get access?
Usually **your teacher** is going to **sign you up** and you will recieve one or two emails containing information on how to login.
You will recieve access to the Teaching Hub usually for 6 month. Can be extended for good reasons or until a teacher signs you up again.
## You recieved two Emails
This means you are new to the SERVERS and the Teaching Hub.
The first email contains your login details for SRVX1, you get a username (Matrikelnumber) and a password. It looks like this:
```
Dear TeachingHub User Monkey
Some of your user information has been updated.
Server Account Specification:
servers: srvx1.img.univie.ac.at
username: monkey
password: ABCDEFGHIJK (Change as soon as possible on all servers)
uid: 40XXX
gid: 59000
Name: Monkey Banana
Email: a000000@unet.univie.ac.at
GECOS: Monkey Banana (a000000@unet.univie.ac.at)
home: /home/srvx11/lehre/users/a000000
shell: /bin/bash
Account expires: 2021-08-30
...
```
Use these to login to the server and change the password to something better and still something that is safe:
```bash
$ ssh [user]@srvx1.img.univie.ac.at
----------------------------------------------
_
(` ).
( ).
) _( SRVX1 '`.
.=(`( . ) .--
(( (..__.:'-' .+( )
`. `( ) ) ( . )
) ` __.:' ) ( ( ))
) ) ( ) --' `- __.'
.-' (_.' .')
(_ )
131.130.157.5
--..,___.--,--'`,---..-.--+--.,,-,,.-..-._.-.-
----------------------------------------------
Teaching Server @IMGW
Environmental Modules: module av
Jupyterhub: https://srvx1.img.univie.ac.at
Usage Profile: Students
----------------------------------------------
$ passwd
Changing password for user [user].
Changing password for [user].
(current) UNIX password:
```
After you completed these steps you are free to login anytime and use the server.
The second email contains your Teaching Hub credentials. See below.
Usually **your teacher** is going to **sign you up** and you will receive one or two emails containing information on how to login.
You will receive access to the Teaching Hub usually for 6 month. Can be extended for good reasons or until a teacher signs you up again.
## You recieved one Email
the email you recieved contains credentials for the Teaching Hub. This email contains your username (the same as on SRVX1 for login), a new password (only for the Teaching Hub) and a secret (16 digit code). They look like this:
The email you received contains credentials for the Teaching Hub. This email contains your username, a new password (only for the Teaching Hub) and a secret (16 digit code). They look like this:
```
Dear TeachingHub User Monkey
Some of your user information has been updated.
......@@ -76,61 +19,61 @@ Some of your user information has been updated.
TeachingHub Account Specification:
servers: https://srvx1.img.univie.ac.at
username: monkey
password: ABCDEFGHIJK (signed in, click on user name, upper right corner)
password: ABCDEFGHIJK (signed in, click on user name, upper right corner to change)
Email: a000000@univie.ac.at
Secret: 1A2B3C4D5E6F7G8H
Please note that the TeachingHub is open from the web.
Two-factor Authentication is required. Download one of these apps and use your secret to generate TOTP (Time based One Time Passwords).
Authy (https://authy.com/download/)
FreeOTP (https://freeotp.github.io/)
Google Auth (https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2)
Two-factor Authentication is required. Download any TOTP (Time based One Time Passwords) app according to your OS.
...
```
You need to download one of these apps:
As stated in the email you received, you need to download one of these TOTP apps:
- [2FA Authenticator](https://2fas.com) :rocket: (android, ios)
- [Authy](https://authy.com/download/)
- [FreeOTP](https://freeotp.github.io/)
- [Google Auth](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2)
- or any other (search `2fa apps` and you will find plenty)
I recommend using your phone and an app there, as this is the most secure way. And with you phone you can scan the qr code, which looks like this:
<img src="Documentation/teachinghub-qrcode.png" width="300px">
<img src="./mkdocs/img/teachinghub-qrcode.png" width="300px">
When you scan to your Authenticator App, it will automatically get the details (username, Secret) required to generate every 30 seconds a Time-based One Time Password (TOTP). Like this:
![](Documentation/teachinghub-authenticator.png)
![](./mkdocs/img/teachinghub-authenticator.png)
If you want to use a desktop or other app that might not scan QR codes, you can fill in the details manually like this:
- Provider: `TeachingHub`
- Account: `[user]` your username, here `monkey`
- Token or Secret or Hash: `1A2B3C4D5E6F7G8H` is the secret from your email.
![](Documentation/teachinghub-authenticator-manual.png)
![](./mkdocs/img/teachinghub-authenticator-manual.png)
The combination of username, password and TOTP needs to be used to sign in:
![](Documentation/teachinghublogin.png)
![](./mkdocs/img/teachinghublogin.png)
This sounds all really complicated, but I promise this will be easier from time to time. Don't forget that this for our all security and your contribution is appreciated! Thanks.
If you have trouble with login, please check if you device has time synchronization activated, otherwise you might just get wrong codes. (Nice for checking: [totp.app](https://totp.app) *However, be warned this is an online service and your data might not be as secure as you think!*)
Now you are ready to go to the teaching hub and sign in:
[Teaching Hub](https://srvx1.img.univie.ac.at)
# Teaching Hub Notebooks and Kernels
## Teaching Hub Notebooks and Kernels
Currently there are several Notebooks with different Kernels (Python kernels) installed at SRVX1. Look at the logos, if SRVX1 is in there then you will have access to these notebooks and kernels. If SRVX2 or SRVX8 are in the logo you might not have access as this is for researchers.
![](Documentation/teachinghubnotebooks.png)
![](./mkdocs/img/teachinghubnotebooks.png)
As you can see it is also possible to launch a terminal inside the TeachingHub and you will find you files from your home directory there.
It might be that your course has a separate Notebook/Kernel for you, with installed packages and dependencies required for this specific course. However, these Notebooks/Kernels are only available during these lectures and might be removed in the future. So be aware of that. Note that you can easily change a kernel in your notebook. So no need to delete, copy or do anything. Your data/notebook information is safe.
![](Documentation/teachinghubnotebooks_kernels.png)
![](./mkdocs/img/teachinghubnotebooks_kernels.png)
......
# Vienna Scientific Cluster (VSC)
# Vienna Scientific Cluster
[VSC](https://vsc.ac.at/home/)
[VSC-Wiki](https://wiki.vsc.ac.at)
> High Performance Computing available to Staff
> Austrian HPC effort
> part of EuroCC
>
[[_TOC_]]
Links:
We have the privilege to be part of the VSC and have private nodes at VSC-4 (since 2020) and VSC-3 (since 2014)
- [VSC](https://vsc.ac.at/home/)
- [VSC-Wiki](https://wiki.vsc.ac.at)
- [EuroCC - Austria](https://eurocc-austria.at)
We have the privilege to be part of the VSC and have private nodes at VSC-5 (since 2022), VSC-4 (since 2020) and VSC-3 (since 2014), which is retired by 2022.
Access is primarily via SSH:
``` bash
$ ssh user@vsc5.vsc.ac.at
$ ssh user@vsc4.vsc.ac.at
$ ssh user@vsc3.vsc.ac.at
$ ssh user@vsc3.vsc.ac.at (old does not work anymore)
```
Please follow some connection instruction on the [wiki](https://wiki.vsc.ac.at) which is similar to all other servers (e.g. [SRVX1](SRVX1.md)).
The VSC is only available from within the UNINET (VPN, ...).
The VSC is only available from within the UNINET (VPN, ...). Authentication requires a mobile phone.
We have private nodes at our disposal and in order for you to use these you need to specify the correct account in the jobs you submit to the queueing system (SLURM). The correct information will be given to you in the registration email.
## IMGW customizations in the shell
If you want you can use some shared shell scripts that provide information for users about the VSC system.
```bash
# run the install script, that just appends to your PATH variable.
/gpfs/data/fs71386/imgw/install_imgw.sh
```
Please find the following commands available:
- `imgw-quota` shows the current quota on VSC for both HOME and DATA
- `imgw-container` singularity/apptainer container run script, see [below](#containers)
- `imgw-transfersh` Transfer-sh service on [srvx1](https://srvx1.img.univie.ac.at/filetransfer), easily share small files.
- `imgw-cpuinfo` Show CPU information
Please find a shared folder in `/gpfs/data/fs71386/imgw/shared` and add data there that needs to be used by multiple people. Please make sure that things are removed again as soon as possible. Thanks.
## Node Informaiton VSC-5
```
CPU model: AMD EPYC 7713 64-Core Processor
1 CPU, 64 physical cores per CPU, total 128 logical CPU units
512 GB Memory
```
We have access to 11 private Nodes of that kind. We also have access to 1 GPU node with Nvidia A100 accelerators. Find the partition information with:
```bash
$ sqos
qos_name total used free walltime priority partitions
=========================================================================
p71386_0512 11 0 11 10-00:00:00 100000 zen3_0512
p71386_a100dual 1 0 0 10-00:00:00 100000 gpu_a100_dual
```
## Storage on VSC-5
the HOME and DATA partition are the same as on [VSC-4](#storage-on-vsc-4).
## Node Information VSC-4
```
CPU model: Intel(R) Xeon(R) Platinum 8174 CPU @ 3.10GHz
......@@ -27,25 +74,31 @@ CPU model: Intel(R) Xeon(R) Platinum 8174 CPU @ 3.10GHz
378 GB Memory
```
We have access to 5 private Nodes of that kind. Check with
We have access to 5 private Nodes of that kind. We also have access to the jupyterhub on VSC. Check with
```bash
$ sqos
qos_name total used free walltime priority partitions
=========================================================================
p71386_0384 5 1 4 10-00:00:00 100000 mem_0384
jupyter 20 1 19 3-00:00:00 1000 jupyter
p71386_0384 5 0 5 10-00:00:00 100000 mem_0384
```
## Storage on VSC-4
All quotas are shared between users:
All quotas are **shared between all** IMGW/Project users:
- `$HOME` (up to 100 GB, all home directories)
- `$HOME` (up to 100 GB, **all home directories**)
- `$DATA` (up to 10 TB, backed up)
- `$BINFL` (up to 1TB, fast scratch)
- `$BINFS` (up to 2GB, SSD fast)
- `$BINFL` (up to 1TB, fast scratch), **will be retired**
- `$BINFS` (up to 2GB, SSD fast), **will be retired**
- `$TMPDIR` (50% of main memory, deletes after job finishes)
- `/local` (Compute Nodes, 480 GB SSD, deletes after Job finishes)
![VSC Storage Performance](https://service.vsc.ac.at/slides/introduction-to-vsc/08_storage_infrastructure/pictures/vsc_write_performance.png)
Check quotas running the following commands yourself, including your PROJECTID or use the `imgw-quota` command as from the [imgw shell extensions](#imgw-customizations-in-the-shell)
Check quotas:
```bash
$ mmlsquota --block-size auto -j data_fs71386 data
Block Limits | File Limits
......@@ -59,6 +112,15 @@ home FILESET 62.17G 100G 100G 207.8M none | 631
```
## Other Storage
We have access to the Earth Observation Data Center [EODC](https://eodc.eu/data/), where one can find primarily the following data sets:
- Sentinel-1, 2, 3
- Wegener Center GPS RO
These datasets can be found directly via `/eodc/products/`.
We are given a private data storage location (`/eodc/private/uniwien`), where we can store up to 22 TB on VSC-4. However, that might change in the future.
## Node Information VSC-3
```
CPU model: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
......@@ -85,13 +147,12 @@ $ beegfs-ctl --getquota --cfgFile=/etc/beegfs/global3.d/beegfs-client.conf --gid
p70653| 70653|| 5.62 TiB| 12.00 TiB|| 175886| 1000000
```
**However, it seems that most filesystems are quite full.**
## Run time limits
On VSC-3 we have a max runtime of 10 days for the private Queue. The normal queues have 3 days. the devel only 10 min (for testing)
```bash
<pre>
$ sacctmgr show qos format=name%20s,priority,grpnodes,maxwall,description%40s
Name Priority GrpNodes MaxWall Descr
......@@ -101,13 +162,12 @@ $ sacctmgr show qos format=name%20s,priority,grpnodes,maxwall,description%40s
normal_0256 2000 6 3-00:00:00 all user
normal_0128 2000 11 3-00:00:00 all user
devel_0128 5000000 10 00:10:00 for developing and testing codes
</pre>
```
on VSC-4 accordingly:
on VSC-4 accordingly.
```bash
<pre>
$ sacctmgr show qos format=name%20s,priority,grpnodes,maxwall,description%40s
Name Priority GrpNodes MaxWall Descr
-------------------- ---------- -------- ----------- ----------------------------------------
p71386_0384 100000 10-00:00:00 private nodes haimberger
......@@ -119,9 +179,10 @@ $ sacctmgr show qos format=name%20s,priority,grpnodes,maxwall,description%40s
mem_0096 1000 3-00:00:00 vsc-4 regular nodes with 96 gb of memory
mem_0384 1000 3-00:00:00 vsc-4 regular nodes with 384 gb of memo+
mem_0768 1000 3-00:00:00 vsc-4 regular nodes with 768 gb of memo+
```
</pre>
SLURM allows for setting a run time limit below the default QOS's run time limit. After the specified time is elapsed, the job is killed:
```bash
#SBATCH --time=<time>
```
......@@ -143,14 +204,14 @@ We have 16 CPUs per Node. In order to fill:
The core hours will be charged to the specified account. If not specified, the default account will be used.
on VSC-3 our account is called:
```bash
<pre>
$ sacctmgr show user `id -u` withassoc format=user,defaultaccount,account,qos%40s,defaultqos%20s
User Def Acct Account QOS Def QOS
---------- ---------- ---------- ---------------------------------------- --------------------
71633 p70653 p70653 devel_0128,p70653_0128 p70653_0128
```
</pre>
Put this in the Job file:
......@@ -200,14 +261,14 @@ We have 48 CPUs per Node. In order to fill:
The core hours will be charged to the specified account. If not specified, the default account will be used.
on VSC-4 our account is called:
```bash
<pre>
$ sacctmgr show user `id -u` withassoc format=user,defaultaccount,account,qos%40s,defaultqos%20s
User Def Acct Account QOS Def QOS
---------- ---------- ---------- ---------------------------------------- --------------------
74101 p71386 p71386 p71386_0384 p71386_0384
</pre>
```
Put this in the Job file:
```bash
......@@ -342,6 +403,8 @@ py3centos7anaconda3-2020-07-dev.sif conda list
which shows something like this
??? note "anaconda environment list"
```
# packages in environment at /opt/software/linux-centos7-haswell/gcc-4.8.5/anaconda3-2020.07-xl53rxqkccbjdufemaupvtuhs3wsj5d2:
#
......
site_name: Computer Resources @ IMGW
# default is ./site/
# could change this to /var/www/html/docs
site_url: "https://wolke.img.univie.ac.at/documentation/general/"
site_dir: "./site/"
# site_dir: "/var/www/html/documentation/general/"
docs_dir: "."
repo_url: https://gitlab.phaidra.org/imgw/computer-resources
repo_name: IMGW/Computer-Resources
# this makes relative links valid
use_directory_urls: false
# this adds the feature to directly edit the file on gitlab
edit_uri: edit/master/
copyright: Copyright &copy; 2022 - 2022 Michael Blaschek
theme:
name: material
palette:
scheme: uniwien
features:
- navigation.indexes
- navigation.top
logo: mkdocs/img/mkdocs.png
favicon: mkdocs/img/favicon.ico
custom_dir: mkdocs/overrides
plugins:
- same-dir
- search
- mkdocs-jupyter:
include_source: True
- git-revision-date-localized:
enable_creation_date: true
- awesome-pages
# collapse_single_pages: true
- tags
- exclude:
glob:
- testing/*
- site/*
extra:
social:
- icon: fontawesome/solid/compass
link: https://univie.ac.at
name: Universität Wien
- icon: fontawesome/solid/cloud
link: https://img.univie.ac.at
name: Institut für Meteorologie und Geophysik
extra_css:
- mkdocs/stylesheets/extra.css
markdown_extensions:
- pymdownx.highlight:
use_pygments: true
linenums: true
pygments_style: xcode
- pymdownx.inlinehilite
- pymdownx.snippets:
check_paths: true
- pymdownx.superfences
- pymdownx.tasklist
- pymdownx.details
- pymdownx.emoji
mkdocs/img/GPFS-jet.png

48.3 KiB

mkdocs/img/envmodules.png

12 KiB

mkdocs/img/favicon.ico

168 KiB

mkdocs/img/ipa-add-otp.png

25.9 KiB

mkdocs/img/ipa-add-ssh-final.png

88.5 KiB

mkdocs/img/ipa-add-ssh-pub.png

61.1 KiB

mkdocs/img/ipa-add-ssh-save.png

29.7 KiB

mkdocs/img/ipa-add-ssh.png

76.9 KiB

mkdocs/img/ipa-login.png

474 KiB

mkdocs/img/ipa-otp-final.png

33.6 KiB

mkdocs/img/ipa-otp-qr.png

25.6 KiB

mkdocs/img/ipa-otp-totp.png

16.4 KiB

mkdocs/img/ipa-ssh-login.png

61.6 KiB

File moved
File moved
File moved
File moved
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment