Skip to content
Snippets Groups Projects
Commit 763788b4 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

updated srvx1 removed, jet, vscode

parent 32ccb851
No related branches found
No related tags found
No related merge requests found
# European Center for Medium-Range Weather Forecast # European Center for Medium-Range Weather Forecast
<img src="../mkdocs/img/logo_ecmwf.png" width="400px"> ![ECMWF](../mkdocs/img/logo_ecmwf.png){: width="400px"}
[website](https://www.ecmwf.int) / [service status](https://www.ecmwf.int/en/service-status) / [confluence](https://confluence.ecmwf.int) / [support](https://support.ecmwf.int) / [accounting](https://www.ecmwf.int/user) [website](https://www.ecmwf.int) / [service status](https://www.ecmwf.int/en/service-status) / [confluence](https://confluence.ecmwf.int) / [support](https://support.ecmwf.int) / [accounting](https://www.ecmwf.int/user)
......
...@@ -8,7 +8,7 @@ Thanks. :elephant: ...@@ -8,7 +8,7 @@ Thanks. :elephant:
## Vim ## Vim
![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9f/Vimlogo.svg/64px-Vimlogo.svg.png) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9f/Vimlogo.svg/64px-Vimlogo.svg.png){: width="100px"}
Vim (a contraction of Vi IMproved) is a clone, with additions, of Bill Joy's vi text editor program for Unix. Vim (a contraction of Vi IMproved) is a clone, with additions, of Bill Joy's vi text editor program for Unix.
...@@ -50,7 +50,7 @@ zc - close folded text ...@@ -50,7 +50,7 @@ zc - close folded text
``` ```
## Emacs ## Emacs
![](https://www.gnu.org/software/emacs/images/emacs.png) ![](https://www.gnu.org/software/emacs/images/emacs.png){: width="100px"}
This is another major editor that is available on all unix (linux) machines as well. It has a commandline interface and a gui. It is much easier than VIM, but again has a lot of [shortcuts](Emacs-Reference.pdf) to learn. This is another major editor that is available on all unix (linux) machines as well. It has a commandline interface and a gui. It is much easier than VIM, but again has a lot of [shortcuts](Emacs-Reference.pdf) to learn.
...@@ -62,7 +62,7 @@ There are much more things that can be added to `~/.emacs` if you want. Have a l ...@@ -62,7 +62,7 @@ There are much more things that can be added to `~/.emacs` if you want. Have a l
## IDEs ## IDEs
### Atom ### Atom
![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/Atom_icon.svg/150px-Atom_icon.svg.png) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/Atom_icon.svg/150px-Atom_icon.svg.png){: width="100px"}
Atom is a free and open-source text and source code editor for macOS, Linux, and Microsoft Windows with support for plug-ins written in JavaScript, and embedded Git Control, developed by GitHub. Atom is a desktop application built using web technologies. Atom is a free and open-source text and source code editor for macOS, Linux, and Microsoft Windows with support for plug-ins written in JavaScript, and embedded Git Control, developed by GitHub. Atom is a desktop application built using web technologies.
...@@ -71,7 +71,7 @@ Install: [Atom](https://atom.io/) ...@@ -71,7 +71,7 @@ Install: [Atom](https://atom.io/)
This editor can do everything, as it is open source and solutions on common problems exist on github. However, it is for more advanced users, as sometimes you need to change things and fix small bugs. This editor can do everything, as it is open source and solutions on common problems exist on github. However, it is for more advanced users, as sometimes you need to change things and fix small bugs.
### VS-Code ### VS-Code
![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Visual_Studio_Code_1.35_icon.svg/100px-Visual_Studio_Code_1.35_icon.svg.png) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Visual_Studio_Code_1.35_icon.svg/100px-Visual_Studio_Code_1.35_icon.svg.png){: width="100px"}
Visual Studio Code is a freeware source-code editor made by Microsoft for Windows, Linux and macOS. Features include support for debugging, syntax highlighting, intelligent code completion, snippets, code refactoring, and embedded Git. Microsoft has released Visual Studio Code's source code on the `microsoft/vscode` (Code - OSS) repository of GitHub, under the permissive MIT License, while the releases by Microsoft are freeware. Visual Studio Code is a freeware source-code editor made by Microsoft for Windows, Linux and macOS. Features include support for debugging, syntax highlighting, intelligent code completion, snippets, code refactoring, and embedded Git. Microsoft has released Visual Studio Code's source code on the `microsoft/vscode` (Code - OSS) repository of GitHub, under the permissive MIT License, while the releases by Microsoft are freeware.
...@@ -81,7 +81,7 @@ Open-Source: [OSS](https://github.com/microsoft/vscode) ...@@ -81,7 +81,7 @@ Open-Source: [OSS](https://github.com/microsoft/vscode)
This is a great editor and has some nice features for Python, however PyCharm is much better for Python. A great feature is remote development. Using VS-code with remote development on SRVX1 or SRVX8 can be done using [instructions](vscode.md) and using a [singularity](https://sylabs.io/guides/3.7/user-guide/) container. This is a great editor and has some nice features for Python, however PyCharm is much better for Python. A great feature is remote development. Using VS-code with remote development on SRVX1 or SRVX8 can be done using [instructions](vscode.md) and using a [singularity](https://sylabs.io/guides/3.7/user-guide/) container.
### Pycharm ### Pycharm
![](https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/PyCharm_Icon.svg/150px-PyCharm_Icon.svg.png) ![](https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/PyCharm_Icon.svg/150px-PyCharm_Icon.svg.png){: width="100px"}
PyCharm is an integrated development environment (IDE) used in computer programming, specifically for the Python language. It is developed by the Czech company JetBrains. It provides code analysis, a graphical debugger, an integrated unit tester, integration with version control systems (VCSes), and supports web development with Django as well as data science with Anaconda. PyCharm is an integrated development environment (IDE) used in computer programming, specifically for the Python language. It is developed by the Czech company JetBrains. It provides code analysis, a graphical debugger, an integrated unit tester, integration with version control systems (VCSes), and supports web development with Django as well as data science with Anaconda.
...@@ -92,14 +92,14 @@ Install: [Pycharm Community](https://www.jetbrains.com/de-de/pycharm/) ...@@ -92,14 +92,14 @@ Install: [Pycharm Community](https://www.jetbrains.com/de-de/pycharm/)
This is perfect for python development. However, remote development is not supported by the community version and running it via VNC works. This is perfect for python development. However, remote development is not supported by the community version and running it via VNC works.
## Meld ## Meld
![](https://media-cdn.ubuntu-de.org/wiki/attachments/19/03/meld_logo.png) ![](https://media-cdn.ubuntu-de.org/wiki/attachments/19/03/meld_logo.png){: width="100px"}
Download: [Meld](https://meldmerge.org/) Download: [Meld](https://meldmerge.org/)
Meld is a visual diff and merge tool targeted at developers. Meld helps you compare files, directories, and version controlled projects. It provides **two- and three-way** comparison of both files and directories, and has support for many popular version control systems. Meld is a visual diff and merge tool targeted at developers. Meld helps you compare files, directories, and version controlled projects. It provides **two- and three-way** comparison of both files and directories, and has support for many popular version control systems.
## Joplin ## Joplin
![](https://upload.wikimedia.org/wikipedia/en/thumb/0/08/Joplin-icon.svg/240px-Joplin-icon.svg.png) ![](https://upload.wikimedia.org/wikipedia/en/thumb/0/08/Joplin-icon.svg/240px-Joplin-icon.svg.png){: width="100px"}
Download: [Joplin](https://joplinapp.org) Download: [Joplin](https://joplinapp.org)
......
...@@ -7,16 +7,31 @@ ...@@ -7,16 +7,31 @@
## Remote SSH ## Remote SSH
There is package called `Remote Development` in VSCode which includes There is package called `Remote Development` in VSCode which includes
- Remote SSH - Remote SSH
- Remote WSL - Remote WSL
- Remote Containers - Remote Containers
- Remote Explorer
Although all might be useful, we try to make use of the first one Remote-SSH. This allows to connect with your locally installed VSCode to connect to a remote environment, offering a shell and access to the file system on that host.
![](../mkdocs/vids/vscode-remote-ssh.gif)
Information on:
- [Remote Development with SSH](https://code.visualstudio.com/docs/remote/ssh)
- [Remote Development with SSH and JumpHost](https://support.cs.wwu.edu/home/survival_guide/tools/VSCode_Jump.html)
Although all might be useful, we try to make use of the first one Remote-SSH.
### Kernel version is too old? ### Kernel version is too old?
We are going to use a [singularity container](https://sylabs.io/guides/3.7/user-guide/) to run a more recent version of `node` and allow VSCode to use `srvx8` as well. It will complain a bit, but work.
???+ bug "Update 2023"
Not relevant anymore. All our servers have been upgraded.
What happens if the server has a too old kernel. We are going to use a [singularity container](https://sylabs.io/guides/3.7/user-guide/) to run a more recent version of `node` and allow VSCode to use `srvx8` as well. It will complain a bit, but work.
Here is how: Here is how:
1. Login to `srvx8` 1. Login to `srvx8`
2. run `/opt/containers/vscode-server/setup.sh` 2. run `/opt/containers/vscode-server/setup.sh`
3. Look inside `$HOME/.vscode-server/singularity.bash` and make sure that path that you require are available in `SINGULARITY_BIND` variable. 3. Look inside `$HOME/.vscode-server/singularity.bash` and make sure that path that you require are available in `SINGULARITY_BIND` variable.
......
...@@ -66,5 +66,5 @@ Links: ...@@ -66,5 +66,5 @@ Links:
- [Herbert Voß - Latex Referenz der Umgebungen (Makros, Längen, Zähler)](http://www.lehmanns.de/pdf/latexreferenz.pdf) - [Herbert Voß - Latex Referenz der Umgebungen (Makros, Längen, Zähler)](http://www.lehmanns.de/pdf/latexreferenz.pdf)
- [Table Converter Online (csv, xml, json, excel, md, tex, ...)](https://tableconvert.com) - [Table Converter Online (csv, xml, json, excel, md, tex, ...)](https://tableconvert.com)
- [Pandoc Demos](https://pandoc.org/demos.html) - [Pandoc Demos](https://pandoc.org/demos.html)
- [Academic Pandoc template](https://maehr.github.io/academic-pandoc-template/) <img src="https://maehr.github.io/academic-pandoc-template/android-chrome-512x512.png" width="30px"> - [Academic Pandoc template](https://maehr.github.io/academic-pandoc-template/) ![](https://maehr.github.io/academic-pandoc-template/android-chrome-512x512.png){: width="30px"}
- [Eisvogel Pandoc Template](https://github.com/Wandmalfarbe/pandoc-latex-template) <img src="https://github.com/Wandmalfarbe/pandoc-latex-template/blob/master/icon.png?raw=true" width="30px"> - [Eisvogel Pandoc Template](https://github.com/Wandmalfarbe/pandoc-latex-template) ![](https://github.com/Wandmalfarbe/pandoc-latex-template/blob/master/icon.png?raw=true){: width="30px"}
<img src="./mkdocs/img/logo_uniwien.jpg" width="300px"> ![](./mkdocs/img/logo_uniwien.jpg){: width="300px"}
<img src="./mkdocs/img/logo_img2_color.png" width="300px"> ![](./mkdocs/img/logo_img2_color.png){: width="300px"}
# Welcome to the Department of Meteorology and Geophysics # Welcome to the Department of Meteorology and Geophysics
......
...@@ -135,7 +135,7 @@ $ ssh -oForwardAgent=yes srvx1 ...@@ -135,7 +135,7 @@ $ ssh -oForwardAgent=yes srvx1
Keep in mind that you can use the ssh-agent with [KeepassXC](https://keepassxc.org/docs/#faq-ssh-agent-how), find a nice tutorial [here](https://ferrario.me/using-keepassxc-to-manage-ssh-keys/). This is really convenient as it allows you to use all keys in the Keepass as long as it is unlocked. The keys will be automatically removed when the keepass is locked. :) Keep in mind that you can use the ssh-agent with [KeepassXC](https://keepassxc.org/docs/#faq-ssh-agent-how), find a nice tutorial [here](https://ferrario.me/using-keepassxc-to-manage-ssh-keys/). This is really convenient as it allows you to use all keys in the Keepass as long as it is unlocked. The keys will be automatically removed when the keepass is locked. :)
<img src="https://upload.wikimedia.org/wikipedia/commons/c/c1/KeePassXC.svg" width="200px"> [![KeepassXC](../mkdocs/img/logo-keepassxc.png){: width="200px"}](https://keepassxc.org/)
## Q: How to transfer files between two VPN networks? ## Q: How to transfer files between two VPN networks?
......
![](../mkdocs/img/rocket.png){: width="100px" align=right}
# J E T # J E T
> Research Cluster for Staff > Research Cluster for Staff
> :rocket: 🔒 > :rocket: 🔒
<img src="../mkdocs/img/rocket.png" width="100px">
## Getting Started ## Getting Started
...@@ -26,12 +26,13 @@ Last Update: 16.07.2024 ...@@ -26,12 +26,13 @@ Last Update: 16.07.2024
Node Setup: Node Setup:
- 2x Login Nodes (jet01, jet02) - 2x Login Nodes (jet01, jet02)
- 17x Compute Nodes (jet03-19) - 7x Compute Nodes INTEL (jet03-09)
- 10x Compute Nodes AMD (jet10-jet19)
- 5x Storage Nodes - 5x Storage Nodes
![GPFS](../mkdocs/img/jet-gpfs-setup.png) ![GPFS](../mkdocs/img/jet-gpfs-setup.png)
### Example of one Compute Node ### Example INTEL Node
| Type | Detail | | Type | Detail |
| --- | --- | | --- | --- |
...@@ -43,12 +44,27 @@ Node Setup: ...@@ -43,12 +44,27 @@ Node Setup:
| Memory/Core | 18.9 GB | | Memory/Core | 18.9 GB |
| Network | 100 Gbit/s (Infiniband) | | Network | 100 Gbit/s (Infiniband) |
### Example AMD Node
| Type | Detail |
| --- | --- |
| Product | ThinkSystem SR635 V3 |
| Processor | AMD EPYC 9454P 48-Core Processor |
| Cores | 1 CPU, 48 physical cores per CPU, total 96 logical CPU units |
| CPU Time | 420 kh |
| Memory | 1132 GB Total |
| Memory/Core | 23.5 GB |
| Network | 200 Gbit/s (Infiniband) |
### Storage
All nodes are connected to a global file system (GPFS) with about 3.5 PB (~3500 TB) of storage. There is no need to copy files to the compute nodes, your HOME and SCRATCH directories will be available under the same path as on the login nodes. All nodes are connected to a global file system (GPFS) with about 3.5 PB (~3500 TB) of storage. There is no need to copy files to the compute nodes, your HOME and SCRATCH directories will be available under the same path as on the login nodes.
Paths: Paths:
- `/jetfs/home/[username]` - `/jetfs/home/[username]`
- `/jetfs/scratch/[username]` - `/jetfs/scratch/[username]`
- `/jetfs/shared-data`
## Software ## Software
...@@ -153,7 +169,7 @@ on how to use environment modules go to [Using Environment Modules](../Misc/Envi ...@@ -153,7 +169,7 @@ on how to use environment modules go to [Using Environment Modules](../Misc/Envi
## Jupyterhub ## Jupyterhub
<img src="../mkdocs/img/jupyterhub-logo.svg" width="100px"> ![](../mkdocs/img/jupyterhub-logo.svg){: width="100px"}
The Jet Cluster serves a [jupyterhub](https://jupyterhub.readthedocs.io/en/stable/) with a [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/) that launches on the JET cluster compute nodes and allows users to work directly on the cluster as well as submit jobs. The Jet Cluster serves a [jupyterhub](https://jupyterhub.readthedocs.io/en/stable/) with a [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/) that launches on the JET cluster compute nodes and allows users to work directly on the cluster as well as submit jobs.
...@@ -166,11 +182,11 @@ Steps: ...@@ -166,11 +182,11 @@ Steps:
**Please use the resources responsible. We trust that you apply a fair-share policy and collaborate with your colleagues.** **Please use the resources responsible. We trust that you apply a fair-share policy and collaborate with your colleagues.**
<img src="../mkdocs/img/jet-login.png" width="500px"> ![](../mkdocs/img/jet-login.png){: width="500px"}
<img src="../mkdocs/img/jet-job1.png" width="500px"> ![](../mkdocs/img/jet-job1.png){: width="500px"}
<img src="../mkdocs/img/jet-job2.png" width="500px"> ![](../mkdocs/img/jet-job2.png){: width="500px"}
There are several kernels available as modules and how to use other kernels can be found here: There are several kernels available as modules and how to use other kernels can be found here:
...@@ -227,12 +243,19 @@ How to use a vnc server, go to [VNC](../SSH-VPN-VNC/VNC.md). ...@@ -227,12 +243,19 @@ How to use a vnc server, go to [VNC](../SSH-VPN-VNC/VNC.md).
## Network drives ## Network drives
Transfer of files between SRVX1 and JET is **not** necessary. The file system from SRVX1 is mounted on the JET cluster and vice versa. These mounted drives need to transfer the data via the network and latencies might be higher. However, since there is a strong connection between SRVX1 and JET, it might be applicable for you to just read the data over the network. Transfer of files between SRV and JET is **not** necessary. The file system is mounted on JET Nodes JET01/JET02 and vice versa. These mounted drives need to transfer the data via the network and latencies might be higher.
```bash title="Mounted files systems" ```bash title="Mounted files systems"
$ df -h $ df -h
131.130.157.11:/mnt/users 319T 250T 70T 79% /mnt/users
131.130.157.11:/mnt/scratch 400T 350T 51T 88% /mnt/scratch 131.130.157.5:/mnt/users 319T 300T 20T 95% /mnt/users
131.130.157.5:/mnt/scratch 400T 321T 80T 81% /mnt/scratch
131.130.157.5:/mnt/users/staff 319T 300T 20T 95% /srvfs/home
131.130.157.5:/mnt/users/scratch 319T 300T 20T 95% /srvfs/tmp
131.130.157.5:/mnt/users/data 319T 300T 20T 95% /srvfs/data
131.130.157.5:/mnt/scratch/scratch 400T 321T 80T 81% /srvfs/scratch
131.130.157.5:/mnt/scratch/shared 400T 321T 80T 81% /srvfs/shared
131.130.157.5:/mnt/scratch/webdata 400T 321T 80T 81% /srvfs/webdata
remjetfs 3.6P 1.6P 2.0P 44% /jetfs remjetfs 3.6P 1.6P 2.0P 44% /jetfs
``` ```
......
# Jupyter Notebook Lab # Jupyter Notebook Lab
<img src="../mkdocs/img/jupyter-logo.png" width="300px"> ![](../mkdocs/img/jupyter-logo.png){: width="300px"}
Please notice that only the new teaching hub is available in the future. Please notice that only the new teaching hub is available in the future.
Migration for the old teaching hub will finish at the end of WS2023/24. Migration for the old teaching hub will finish at the end of WS2023/24.
Access: [teaching.wolke.img.univie.ac.at](https://teaching.wolke.img.univie.ac.at) or via your [Moodle lecture](https://moodle.univie.ac.at/my/). Access: [teaching.wolke.img.univie.ac.at](https://teaching.wolke.img.univie.ac.at) or via your [Moodle lecture](https://moodle.univie.ac.at/my/).
<img align="right" width="150px" src="../mkdocs/img/Masterhub_logo.png"> ![](../mkdocs/img/Masterhub_logo.png){: width="200px" align=right}
## MasterHub ## MasterHub
...@@ -16,9 +17,9 @@ Students need to get a server account for accessing this resource. However, it i ...@@ -16,9 +17,9 @@ Students need to get a server account for accessing this resource. However, it i
## TeachingHub ## TeachingHub
<img width="200px" src="../mkdocs/img/moodlelogo_small.png"> ![](../mkdocs/img/moodlelogo_small.png"){: width="200px"}
???+ warning "Purge" ???+ warning "Annual purge of HOME"
It is necessary for a smooth operation to purge the contents of the TeachingHub on an annual basis. Every year before the winter semester starts there will be a purge of students home directories. It is necessary for a smooth operation to purge the contents of the TeachingHub on an annual basis. Every year before the winter semester starts there will be a purge of students home directories.
What is available? What is available?
......
...@@ -79,3 +79,4 @@ markdown_extensions: ...@@ -79,3 +79,4 @@ markdown_extensions:
- pymdownx.details - pymdownx.details
- pymdownx.emoji - pymdownx.emoji
- mkdocs_graphviz - mkdocs_graphviz
- attr_list
mkdocs/img/logo-keepassxc.png

17.4 KiB

Image diff could not be displayed: it is too large. Options to address this: view the blob.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment