Skip to content
Snippets Groups Projects
Commit 78de83b6 authored by Michael Blaschek's avatar Michael Blaschek :bicyclist:
Browse files

documentation update and fixes

parent 9310a2d7
No related branches found
No related tags found
No related merge requests found
Showing with 3990 additions and 1 deletion
digraph Servers {
bgcolor="transparent"
users [shape=none image="student_sm.png" label="User" labelloc="b" height="1" imagepos="tc"]
srvx1 [shape=none image="server_sm.png" label="SRVX1" labelloc="b" height="1" imagepos="tc"]
srvx8 [shape=none image="server_sm.png" label="SRVX8" labelloc="b" height="1" imagepos="tc"]
aurora [shape=none image="server_sm.png" label="AURORA" labelloc="b" height="1" imagepos="tc"]
jet [shape=none image="server_sm.png" label="JET" labelloc="b" height="1" imagepos="tc"]
VPN [shape=none image="local-area-network_sm.png" label="VPN" labelloc="b" height="1" imagepos="tc"]
vsc [shape=none image="logo_vsc_sm.png" label="VSC" labelloc="b" height=".8" imagepos="tc"]
users -> srvx1
users -> VPN
srvx1 -> srvx8
srvx1 -> jet
srvx1 -> aurora
srvx1 -> vsc
VPN -> jet
VPN -> srvx8
VPN -> aurora
VPN -> vsc
}
\ No newline at end of file
mkdocs/img/IMGW-connection-diagram.png

36.5 KiB

digraph Servers {
bgcolor="transparent"
users [shape=none image="student_sm.png" label="User" labelloc="b" height="1" imagepos="tc"]
srvx1 [shape=none image="server_sm.png" label="SRVX1" labelloc="b" height="1" imagepos="tc"]
srvx8 [shape=none image="server_sm.png" label="SRVX8" labelloc="b" height="1" imagepos="tc"]
jet [shape=none image="server_sm.png" label="JET" labelloc="b" height="1" imagepos="tc"]
aurora [shape=none image="server_sm.png" label="AURORA" labelloc="b" height="1" imagepos="tc"]
vsc [shape=none image="logo_vsc_sm.png" label="VSC" labelloc="b" height=".8" imagepos="tc"]
users -> srvx1
users -> srvx8
users -> jet
users -> vsc
users -> aurora
}
\ No newline at end of file
mkdocs/img/IMGW-connection-diagram2.png

25.4 KiB

dot -Tpng IMGW-connection-diagram.dot > IMGW-connection-diagram.png
dot -Tpng IMGW-connection-diagram2.dot > IMGW-connection-diagram2.png
mkdocs/img/cmake-explanation.png

143 KiB

This diff is collapsed.
This diff is collapsed.
No preview for this file type
class: inverse
layout: true
name: Introduction to IT for new employees
---
class: title, center, middle
# Bare metal, shells and clouds ​- IT2Research​
![](./img/logo_imgw.svg)
![](../img/logo_fgga2.svg)
9.10.2023 - M. Blaschek - PhD Seminar @ IMGW
---
# What can you expect from this presentation?
- Asking Questions (anytime, please interrupt)
- Hardware
- Data and Storage
- Services
- Rules
- Updates
---
# Important Things @UNIVIE
The ZID offers a lot of information on its services, most importantly your **u:account**.
Services:
- [u:account](https://zid.univie.ac.at/uaccount/) everything depends on it
- [vpn](https://zid.univie.ac.at/vpn/) access to only university wide things (our servers)
- [u:cloud](https://ucloud.univie.ac.at) University cloud
- [u:access](https://bibliothek.univie.ac.at/en/uaccess.html) library access for publications
- [gitlab/mattermost](https://gitlab.phaidra.org/imgw) code development / socializing
- [u:wiki](https://wiki.univie.ac.at/display/theomet) documentation of workflows, group communication, tutorials, intranet, ...
* Lots of tutorials for specific problems (travel, expenses, new employees, printer, ...)
* HPC / Computing information and guidelines
* Please login once. More access can be granted afterwards.
- [wolke.img.univie.ac.at](https://wolke.img.univie.ac.at)
* Landing Page of IT service @ IMGW
* User Management [IPA](https://wolke.img.univie.ac.at/ipa/ui)
---
# Hardware @ IMGW
Documentation: [Computing@u:wiki](https://wiki.univie.ac.at/x/6kW0Bw) or [Computing@gitlab](https://gitlab.phaidra.org/imgw/computer-resources) (public)
Currently the department has:
- SRVX1 (Teaching and development node) access: open
* All Students
* All Staff
* Teaching
* Development Node for interactive computing
- SRVX8 (Staff, graphical node, VNC, GPU) access: univie
* special use cases
* used for VNC
- JET Cluster (2+7, Staff, HPC) access: univie
* default access for staff
* some master students
* slurm scheduler
- [VSC](https://vsc.ac.at) 4/5 (Staff, HPC), managed by VSC. access: univie
* Our department has private Nodes on VSC4 and VSC5
* Request an account with IT
* slurm scheduler
---
class: center, middle
# How to connect? - HOME / UNI
![](../img/IMGW-connection-diagram.png)
![](../img/IMGW-connection-diagram2.png)
---
# Tools - How to connect?
Depending on your operating system (OS) you should find the right tool to connect.
Options:
- Web browser - Jupyterhub running Jupyter Notebooks
* Teaching Hub on [SRVX1](https://srvx1.img.univie.ac.at/hub)
* Research Hub on [JET01](https://wolke.img.univie.ac.at/researchhub)
- SSH (Windows: Putty, ...)
- SFTP (Windows: Mobaxterm, winscp, ...)
- VNC to SRVX8
* install a vnc client on your OS
* launch a vnc server on SRVX8, run: `userservices vnc -h`
* more information on @gitlab @VNC
---
# Login and Passwords
When you login the first time, you will need to change your password. **Please be advised that you should choose a secure one.** *A password manager is recommended.*
```bash
$ ssh [user]@srvx1.img.univie.ac.at
$ ssh [user]@jet01.img.univie.ac.at
# from home to jet directly
$ ssh -J [user]@srvx1.img.univie.ac.at [user]@jet01.img.univie.ac.at
# or using the ZID server
$ ssh -J [u:account]@login.univie.ac.at [user]@jet01.img.univie.ac.at
```
Each of these require a different password:
- IMGW Servers (srvx1, srvx8, jet)
- u:account
- eduroam
- TeachingHub + OTP
- (optional) iLibrarian
---
# Services @ Servers
All servers try to be up to date and will be updated regularly, the base OS is Red Hat OS 8/AlmaLinux 8.
All servers handle software like this:
- environment modules, run: `modules av`
* different compilers: gnu, intel
* libraries: math, eccodes, netcdf ,...
* different MPI
* tools: teleport, ecaccess, ...
- userservices, run: `userservices`
* **quota**
* server-schedule
* weather
* containers
* yopass
* filetransfer
* transfersh
- singularity containers, advanced topic.
* building singularity containers for HPC applications
---
# Services @ Servers 2
Locations on Servers are very important. Where to store and find what?
| name | Server | Path | Quota | comment |
| --- | --- | --- | --- | --- |
| HOME | srvx1 | `/users/staff/[user]` | 100 GB | source code, daily backup |
| HOME | jet01 | `/jetfs/home/[user]` | 100 GB | source code, daily backup |
| SCRATCH | srvx1 | `/scratch/das/[user]` | 1 TB | data, no backup |
| SCRATCH | srvx1 | `/scratch/mas/[user]` | 1 TB | data, no backup |
| SCRATCH | jet01 | `/jetfs/scratch/[user]` | no | data, no backup |
| DATA | srvx1 | `/data/[dataset]` | no | data, daily backup, constant |
| SHARED-DATA | srvx1 | `/scratch/shared/[dataset]` | no| data, no backup, shared |
| SHARED-DATA | jet01 | `/jetfs/shared-data/[dataset]` | no| data, no backup, shared |
All these paths can be remembered by running `userpaths`
---
# DATA Management
Science is about computing, but mostly about data and storaging that data.
| name | Server | Path |
| --- | --- | --- |
| DATA | srvx1 | `/data/[dataset]` |
| SHARED-DATA | srvx1 | `/scratch/shared/[dataset]` |
| SHARED-DATA | jet01 | `/jetfs/shared-data/[dataset]` |
Please fill in the **Database** and follow the **Data Guidelines**, that can be found in these directories.
**Add a short decsription of your data and whom to contact for more information.**
---
# Support & Contacts
In case you need help, whom to contact?
- Servers > M. Blaschek
- VSC > M. Meindl / M. Blaschek
- Gitlab > M. Meindl / M. Blaschek
- Everything else > M. Ristic
Use Mattermost or send mail to [IT](mailto:it.img-wien@univie.ac.at) or [Gitlab Issues](https://gitlab.phaidra.org/imgw/computer-resources/-/issues)
---
class: center, title
# Questions
```
???????
?????
???
??
?
```
<!DOCTYPE html>
<html>
<head>
<title>Markdown Presenter</title>
<meta charset="utf-8">
<link rel="icon" href="./img/logo_imgw.svg" sizes="any" type="image/svg">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<!-- <link href="https://fonts.googleapis.com/css2?family=Roboto&display=swap" rel="stylesheet"> -->
<link href="https://fonts.cdnfonts.com/css/source-sans-pro" rel="stylesheet">
<!-- <link rel="stylesheet" href="src/fonts.css"/> -->
<link rel="stylesheet" type="text/css" href="./css/remark_theme.css"/>
</head>
<body>
<!-- Always download the newest version -->
<script src="https://remarkjs.com/downloads/remark-latest.min.js">
</script>
<script>
function doesFileExist(urlToFile) {
var xhr = new XMLHttpRequest();
xhr.open('HEAD', urlToFile, false);
xhr.send();
if (xhr.status == "404") {
return false;
} else {
return true;
}
}
const queryString = window.location.search;
console.log(queryString);
const urlParams = new URLSearchParams(queryString);
var markdownfile = urlParams.get('file');
// default file for testing
if (markdownfile == null) var markdownfile = 'README.md';
// add .md as extensions
if (markdownfile.slice(-3) != ".md") markdownfile += ".md";
if (doesFileExist(markdownfile)) {
var slideshow = remark.create({
sourceUrl: markdownfile,
ratio: '16:9'
});
} else {
var slideshow = remark.create({
source: '# 404 Error \nYour Presentation file cannot be found: <pre>' + markdownfile + '</pre> \n Please check instructions again [here](/).',
ratio: '16:9'
});
}
console.log(markdownfile);
</script>
</body>
</html>
class: inverse
layout: true
---
# Compiling code
**author: Jan Zabloudil**
![](https://vsc.ac.at//fileadmin/templates/vsc/logo_vsc.gif)
[VSC Gilab](https://gitlab.tuwien.ac.at/vsc-public/training/vsc-intro/-/blob/main/compiling/compiling.md?plain=0)
[open online](https://webdata.wolke.img.univie.ac.at/imgw/Presentation.html?file=compiling.md)
---
# Compiling code
## CPU architectures Skylake and Cascadelake:
These architectures are available in the following nodes/partitions:
* VSC-4 Login nodes l4[0-9]
* skylake_0096
* skylake_0384
* skylake_0768
* cascadelake_0384
You can choose between these compilers:
* Intel
* GNU
Usually on these architectures the Intel compiler delivers the best performance.
---
# Compiling code
## CPU architectures Zen2 and Zen3:
These architectures are available in the following nodes/partitions:
* VSC-5 Login nodes l5[0-6]
* zen3_0512
* zen3_1024
* zen3_2048
* gpu_a100_dual
* zen2_0256_a40x2
You can choose between these compilers:
* Intel
* GNU
* AOCC
Usually on these architectures the AOCC compiler delivers the best performance.
---
# Compiling code
## Skylake environment
Modules:
Intel compilers:
```shell
(skylake) [VSC-4]$ module load intel/19
(skylake) [VSC-4]$ module load compiler/latest
```
GNU compilers:
```shell
(skylake) [VSC-4]$ module load gcc/12.2.0-gcc-8.5.0-aal4zp2
(skylake) [VSC-4]$ module load gcc/9.1.0-gcc-8.5.0-mj7s6dg
```
---
# Compiling code
## Zen3 environment
Modules:
Intel compilers:
```shell
(zen3) [VSC-5]$ module load intel/19
(zen3) [VSC-5]$ module load compiler/latest
```
GNU compilers:
```shell
(zen3) [VSC-5]$ module load gcc/11.2.0-gcc-11.2.0-5i4t2bo
(zen3) [VSC-5]$ module load gcc/10.3.0-gcc-11.2.0-3k4sof7
```
AOCC compiler:
```shell
(zen3) [VSC-5]$ module load aocc/3.2.0-gcc-11.2.0-y53mdzy
```
---
# Compiling code
## Compiler names
Intel:
```shell
icc # C compiler
icpc # C++ compiler
ifort # Fortran compiler
```
GNU:
```shell
gcc # C compiler
g++/c++ # C++ compiler
gfortran # Fortran compiler
```
AOCC:
```shell
clang # C compiler
clang++ # C++ compiler
flang # Fortran compiler
```
---
# Compiling code
## Examples: compiling serial code
* Compiler: Intel
* Language: C
* *~training/examples/15_compiling/hello.c*
```shell
(skylake/zen3) [VSC-4/5]$ module load compiler/latest
```
Compile without optimization:
```shell
(skylake/zen3) [VSC-4/5]$ icc -O0 hello.c -o hello_c
(skylake/zen3) [VSC-4/5]$ ./hello_c
Hello World
```
Compile with optimization:
```shell
(skylake/zen3) [VSC-4/5]$ icc -O3 -xHost hello.c -o hello_c
(skylake/zen3) [VSC-4/5]$ ./hello_c
Hello World
```
---
# Compiling code
## Examples: compiling serial code
* Compiler: GNU
* Language: C
* *~training/examples/15_compiling/hello.c*
```shell
(skylake) [VSC-4]$ module load gcc/12.2.0-gcc-8.5.0-aal4zp2
(zen3) [VSC-5]$ module load gcc/11.2.0-gcc-11.2.0-5i4t2bo
```
Compile without optimization:
```shell
(skylake/zen3) [VSC-4/5]$ gcc -O0 hello.c -o hello_c
(skylake/zen3) [VSC-4/5]$ ./hello_c
Hello World
```
---
# Compiling code
## Examples: compiling serial code
* Compiler: GNU
* Language: C
* *~training/examples/15_compiling/hello.c*
Compile with optimization:
```shell
(skylake) [VSC-4]$ gcc -O2 -march=skylake hello.c -o hello_c
(zen3) [VSC-5]$ gcc -O2 -march=znver3 hello.c -o hello_c
```
```shell
(skylake/zen3) [VSC-4/5]$ ./hello_c
Hello World
```
---
# Compiling code
## Examples: compiling serial code
* Compiler: AOCC
* Language: C
* *~training/examples/15_compiling/hello.c*
```shell
(zen3) [VSC-5]$ module load aocc/3.2.0-gcc-11.2.0-y53mdzy
```
Compile without optimization:
```shell
(zen3) [VSC-5]$ clang -O0 hello.c -o hello_c
(zen3) [VSC-5]$ ./hello_c
Hello World
```
---
# Compiling code
## Examples: compiling serial code
* Compiler: AOCC
* Language: C
* *~training/examples/15_compiling/hello.c*
Compile with optimization:
```shell
(zen3) [VSC-5]$ clang -O3 -march=znver3 hello.c -o hello_c
```
```shell
(zen3) [VSC-5]$ ./hello_c
Hello World
```
---
# Compiling code
## Examples: compiling OPENMP code:
* Compiler: Intel
* Language: C
* *~training/examples/15_compiling/hello-openmp.c*
```shell
(skylake/zen3) [VSC-4/5]$ icc -qopenmp hello_openmp.c -o hello_openmp_c
(skylake/zen3) [VSC-4/5]$ export OMP_NUM_THREADS=2
(skylake/zen3) [VSC-4/5]$ ./hello_openmp_c
Hello World... from thread = 0
Hello World... from thread = 1
```
---
# Compiling code
## Examples: compiling OPENMP code:
* Compiler: GNU
* Language: C
* *~training/examples/15_compiling/hello_openmp.c*
```shell
(skylake/zen3) [VSC-4/5]$ gcc -fopenmp hello_openmp.c -o hello_openmp_c
(skylake/zen3) [VSC-4/5]$ export OMP_NUM_THREADS=2
(skylake/zen3) [VSC-4/5]$ ./hello_openmp_c
Hello World... from thread = 0
Hello World... from thread = 1
```
---
# Compiling code
## Examples: compiling OPENMP code:
* Compiler: AOCC
* Language: C
* *~training/examples/15_compiling/hello_openmp.c*
```shell
(skylake/zen3) [VSC-4/5]$ clang -fopenmp hello_openmp.c hello_openmp_c
(skylake/zen3) [VSC-4/5]$ export OMP_NUM_THREADS=2
(skylake/zen3) [VSC-4/5]$ ./hello_openmp_c
Hello World... from thread = 0
Hello World... from thread = 1
```
---
# Compiling code
## Compiling MPI code:
* Use compiler wrappers
* Various MPI implementations available:
* Intel MPI
* OpenMPI
---
# Compiling code
## Compiling MPI code: Intel MPI
Compiler wrappers for Intel compiler:
```shell
mpiicc # C
mpiicpc # C++
mpiifort # Fortran
```
Compiler wrappers for GNU compiler:
```shell
mpicc/mpigcc # C
mpicxx/mpigxx # C++
mpif90/mpif77/mpifc # Fortran
```
---
# Compiling code
## Compiling MPI code: Intel MPI
```shell
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load compiler/latest intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
(skylake) [VSC-4]$ mpiicc -show
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load gcc/12.2.0-gcc-8.5.0-aal4zp2 intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
(skylake) [VSC-4]$ mpicc -show
```
---
# Compiling code
## Compiling MPI code: OpenMPI
Compiler specific installations, e.g.: <!-- .element: class="fragment highlight-current-red" -->
```shell
(skylake) [VSC-4]$ module load openmpi/4.1.4-intel-19.1.3.304-6xqnw75
(skylake) [VSC-4]$ module load openmpi/4.1.4-gcc-12.2.0-owirnm6
```
```shell
(zen3) [VSC-5]$ module load openmpi/4.1.4-gcc-11.2.0-ub765vm
(zen3) [VSC-5]$ module load openmpi/4.1.4-aocc-3.2.0-7abidzt
```
Compiler wrappers: <!-- .element: class="fragment highlight-current-red" -->
```shell
mpicc # C
mpic++/mpiCC/mpicxx # C++
mpif77/mpif90/mpifort # Fortran
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: Intel
* MPI: Intel
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load compiler/latest intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
```
Compile without optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(skylake) [VSC-4]$ mpiicc -O0 hello-mpi.c -o hello-mpi_c
(skylake) [VSC-4]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l44, rank 1 out of 2 processors
Hello world from processor l44, rank 0 out of 2 processors
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: Intel
* MPI: Intel
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load compiler/latest intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
```
Compile with optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(skylake) [VSC-4]$ mpiicc -O3 -xHost hello-mpi.c -o hello-mpi_c
(skylake) [VSC-4]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l44, rank 1 out of 2 processors
Hello world from processor l44, rank 0 out of 2 processors
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: GNU
* MPI: Intel
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load gcc/12.2.0-gcc-8.5.0-aal4zp2 intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
```
Compile without optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(skylake) [VSC-4]$ mpicc -O0 hello-mpi.c -o hello-mpi_c
(skylake) [VSC-4]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l44, rank 1 out of 2 processors
Hello world from processor l44, rank 0 out of 2 processors
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: GNU
* MPI: Intel
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(skylake) [VSC-4]$ module purge
(skylake) [VSC-4]$ module load gcc/12.2.0-gcc-8.5.0-aal4zp2 intel-oneapi-mpi/2021.6.0-intel-2021.5.0-wpt4y32
```
Compile with optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(skylake) [VSC-4]$ mpicc -O2 -march=skylake hello-mpi.c -o hello-mpi_c
(skylake) [VSC-4]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l44, rank 1 out of 2 processors
Hello world from processor l44, rank 0 out of 2 processors
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: AOCC
* MPI: Openmpi
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(zen3) [VSC-5]$ module purge
(zen3) [VSC-5]$ module load aocc/3.2.0-gcc-11.2.0-y53mdzyr openmpi/4.1.4-aocc-3.2.0-7abidzt
```
Compile without optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(zen3) [VSC-5]$ mpicc -O0 hello-mpi.c -o hello-mpi_c
(zen3) [VSC-5]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l54.vsc.xcat, rank 1 out of 2 processors
Hello world from processor l54.vsc.xcat, rank 0 out of 2 processors
```
---
# Compiling code
## Examples: compiling MPI code
* Compiler: AOCC
* MPI: Openmpi
* Language: C
* *~training/examples/15_compiling/hello_mpi.c*
```shell
(zen3) [VSC-5]$ module purge
(zen3) [VSC-5]$ module load aocc/3.2.0-gcc-11.2.0-y53mdzyr openmpi/4.1.4-aocc-3.2.0-7abidzt
```
Compile with optimization: <!-- .element: class="fragment highlight-current-red" -->
```shell
(zen3) [VSC-5]$ mpicc -O2 -march=znver3 hello-mpi.c -o hello-mpi_c
(zen3) [VSC-5]$ mpirun -np 2 ./hello-mpi_c
Hello world from processor l54.vsc.xcat, rank 1 out of 2 processors
Hello world from processor l54.vsc.xcat, rank 0 out of 2 processors
```
---
This diff is collapsed.
This diff is collapsed.
...@@ -4,6 +4,11 @@ ...@@ -4,6 +4,11 @@
--md-primary-fg-color--dark: #004675; --md-primary-fg-color--dark: #004675;
} }
/* Maximum space for text block */ /* Maximum space for text block */
/* or 100%, if you want to stretch to full-width */
.md-grid { .md-grid {
max-width: 100%; /* or 100%, if you want to stretch to full-width */ max-width: 100%;
}
/* Like larger fonts */
@font-face {
font-size: .9rem;
} }
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment