diff --git a/ICON-NWP/.gitkeep b/ICON-NWP/.gitkeep new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/ICON-NWP.md b/ICON-NWP/ICON-NWP.md similarity index 100% rename from ICON-NWP.md rename to ICON-NWP/ICON-NWP.md diff --git a/ICON-NWP/README.md b/ICON-NWP/README.md new file mode 100644 index 0000000000000000000000000000000000000000..471d3915ddff332cb575d18888d3368de5fa1888 --- /dev/null +++ b/ICON-NWP/README.md @@ -0,0 +1,2 @@ +## ICON-NWP +This is a short introduction to [using the ICON-NWP model on `jet.imgw`](./ICON-NWP.md) and how you can [run assimilation](./how-to-bacy.md) experiments with [`bacy`](./bacy.md). You can get more information about the model [here](https://code.mpimet.mpg.de/projects/iconpublic). \ No newline at end of file diff --git a/ICON-NWP/bacy.md b/ICON-NWP/bacy.md new file mode 100644 index 0000000000000000000000000000000000000000..63bddb4a39e73bd137f8f402696d4dc6f3dfb322 --- /dev/null +++ b/ICON-NWP/bacy.md @@ -0,0 +1,107 @@ +bacy (short for **ba**sic **cy**cling) is a collection of scripts that run assimilation +cycles with ICON. It is modular in structure with each module performing steps +like computing the dynamical model evolution (more), data assimilation (core), +surface, snow and sea surface analysis. + + +# BACY_1.0 + +This document gives you a short introduction into the usage of BACY_1.0. +A detailed userguide and further documentation is available at +`git@gitlab.dkrz.de:dace/dace_doc.git` under dace_doc/org/dace_bacy.git +and is continuously being updated. + +## How to get BACY_1.0? + +BACY_1.0 is under version control using `git`. To obtain BACY_1.0 from the +remote repository `git@gitlab.dkrz.de:dace/dace_bacy.git` the following steps on +your local machine are necessary: +~~~ +git clone git@gitlab.dkrz.de:dace/dace_bacy.git <my_bacy_name> +cd <my_bacy_name>/modules +./copy_defaults.sh +~~~ +*If you want to get access to the DKRZ-gitlab, someone at DWD or DKRZ needs to approve, maybe someone from the DA or Climate group can help you.* + +## How to use BACY_1.0? (Overview) + +### Modules + +BACY_1.0 pursues a MODULAR CONCEPT. Each MODULE is an self-sufficient unit. +The only contact points to its environment are realized by INTERFACES. +These are the directories `iodir/input` and `iodir/output` which are +part of each module. + +The `prep_<module>` and `save_<module>` scripts take care of PROVIDING the +module with input data from and of STORING its output data to a central place, +respectively. In this way the COMMUNICATION between modules is realized. +Unless the `input` directory is not filled with input data, the module itself +cannot be run. Otherwise, the module can start its work and it uses its working +directory `run`. + +The behaviour of a module can be controlled via its INPUT ARGUMENTS during the +`<module>` script call and via its PARAMETERS defined in the configuration file +`<module>_conf.sh`. Parameters which are needed by more than one module are +outsourced to the COMMON configuration file `common/bacy_conf.sh`. + +For a more detailed module-setup please read `modules/module/README`. This +module is a dummy module and serves as TEMPLATE to help you to create your own +modules. In particular, in its README file you can find some valuable hints +which rely on NAMING CONVENTIONS, classification of variables, etc., which are +IMPORTANT to maintain the modular concept. + +In summary: to run a module stand-alone, the following steps are necessary. +~~~ +prep_<module> [-h] [-d] arg1 ... argn +<module> [-h] [-d] arg1 ... argn +save_<module> [-h] [-d] arg1 ... argn +~~~ + +The option `[-h]` gives you an explanation of the usage of the script and the +option `[-d]` calls the script in a dry-run, meaning that no jobs will be +submitted. + +### A cycle + +The module `cycle` plays a special role and cannot be understood as a module in +the usual sense (see above). The main difference is the absence of the `input`, +`output` and `run` directories. The purpose of `cycle` is the provision of an +environment in which the order of individual module calls can be controlled. +Thus, this is the central place to arrange all modules to set up an experiment. + +In summary: to start an experiment, the following steps are necessary. + + prep_cycle [-h] arg1 ... argn + cycle [-h] [-d] arg1 ... argn + +For the options `[-h]` and `[-d]` see the explanations above. + +### Default files + +The configuration files `<module>_conf.sh` as well as namelists and job +scripts are not under version control. Instead we use default-files, e.g., +`<module>_conf.sh.default`. This has the advantage that subsequent calls of +`git pull` will not change the current configuration. + +The script `copy_defaults.sh` creates the normal files from the default +versions. This must be done whenever you clone a fresh BACY from the +repository! + +## How to make your own developments on BACY_1.0? + +The BACY_1.0 MAIN DEVELOPMENT BRANCH is called `bacy-dev`. This branch provides +the BACY_1.0 REFERENCE which has to be the BASIS for ALL users! Only a few +people can push to `bacy-dev` to keep this branch away from accidental or not +quality proven commits. + +If you want to contribute to a NEW FEATURE for BACY_1.0, first of all create a +new branch `<project>-<feature>-<team>`, which you can setup as `remote` +to share results inside your team. + +During your development it is IMPORTANT to regularly pull from the `bacy-dev` +branch to keep your `basic` BACY_1.0 up to date. + +Once you have FINISHED your contribution and you have tested, tested and again +tested your work and you are sure that you match all naming conventions for +BACY_1.0 (see above), then you are ready to contact one person responsible +which merges (together with you) your contribution into `bacy-dev`. diff --git a/ICON-NWP/how-to-bacy.md b/ICON-NWP/how-to-bacy.md new file mode 100644 index 0000000000000000000000000000000000000000..cb8d5e153b35bf72c8d028c7dfa4d7d89d2e1319 --- /dev/null +++ b/ICON-NWP/how-to-bacy.md @@ -0,0 +1,114 @@ +This is a small guide on how to use bacy to run assimilation cycles in practice. + +## Prerequisites + +Before running you need: +* a path to a directory containing a bacy with initial conditions for your experiment +* a icon gridfile, a radiation gridfile and extpar-Data. You can create these using the `icongridgen` +or via the [web](https://webservice.dwd.de/cgi-bin/spp1167/webservice.cgi) interface at DWD (user icon-web, pw: icon@dwd). +* a working bacy, cloned eihter from [`git@gitlab.dkrz.de:dace/dace_bacy.git`](./bacy.md#how-to-get-bacy_10) or from another bacy using `bacy/bacy_clone` + +## Running bacy + +To run cycles you need to go to to the `cycle` module: `bacy/modules/cycle/`. There you can find the scripts `prep_cycle` and `cycle`. + +### Running `prep_cycle` +To initialize bacy run `prep_cycle` in the `cycle` directory. The script describes itself like this: +~~~ +SCRIPT for preparing a data assimilation cycle + +USAGE: ./prep_cycle [-h] model mode frombacypath startdate + + - '-h': help (optional) + - 'model': ICON|COSMO|ICON-LAM + - 'mode': ASS|MAIN|SUPP|URDA|CLIMA + - 'frombacypath': path from which bacy experiment to start the cycle + - 'startdate': format yyyymmddhhmmss|yyyymmddhh + - 'enddate': format yyyymmddhhmmss|yyyymmddhh + (only for mode MAIN|SUPP|URDA)" + +DESCRIPTION: serves as preparation of script 'cycle'; + links all module relevant input data for a given start date (as specified by + 'startdate') from an existing bacy experiment (as specified by 'frombacypath') + into its own cycling directory tree (also containing 'data', 'feedback' and + 'veri' subdirectories, but only with initial time-stamp subfolders); +~~~ +So you can call it like: +~~~ +./prep_cycle ICON-LAM ASS /hpc/path/to/initial/conditons/bacy/ 2023112812 +~~~ +This will prepare a cycle with the ICON large area model in assimilation mode with initial condition from `/hpc/path/to/initial/conditons/bacy/` starting at the 28.11.2023 - 12 UTC. + +### Running `cycle` +After prearation you can run the cycle by calling `cycle` with similar arguments. The script describes itself like this: +~~~ +SCRIPT for running a data assimilation cycle + +USAGE: ./cycle [-h] [-d] \ + [--cycle[=][ ]value] [--task[=][ ]value] [--subtask[=][ ]value] \ + model mode startdate enddate + + options: + - '-h': help (shows usage) + - '-d': dry run (does not submit any long-time jobs) + - '--cycle' : from which cycle to resume work (requires a number) + - '--task' : from which task to resume work (requires a number) + - '--subtask': from which subtask to resume work (requires a number) + arguments: + - 'model': ICON|COSMO|ICON-LAM + - 'mode': ASS|MAIN|VERI|VERIENS|SUPP|URDA|CLIMA + - 'startdate': start of bacy experiment (format yyyymmddhhmmss|yyyymmddhh) + - 'enddate' : end of bacy experiment (format yyyymmddhhmmss|yyyymmddhh) + + Creating the file 'STOP_CYCLE' in the 'cycle' directory will interrupt + all running cycles. To continue the experiment delete the file and restart + the cycle(s) or run the 'RESTART_CYCLE_*' script(s)'. + +DESCRIPTION: Depending on mode, the scripts runs specific cycles: + ASS: a data assimilation cycle by calling the modules 'core', + 'more', 'sst', 'sma', 'snow', and 'mec'(COSMO) in the correct order + MAIN: long forecasts (building upon an ASS cycle) + VERI: mec for deterministic verification against observations + (building upon a MAIN cycle) + VERIENS: mec for ensemble verification against observations + (building upon a MAIN cycle) + SUPP: supplemental assimilation of a previous cycle + (building upon an ASS cycle) + URDA: an ultra rapid data assimilation cycle + CLIMA: climate data assimilation cycle with ICON-SML +~~~ +So you can call it like: +~~~ +./cycle ICON-LAM ASS 2023112812 2023112912 +~~~ +This will start a cycle prepared above going from the 28.11.2023 - 12 UTC until the 29.11.2023 - 12 UTC. + +#### Using `nohup` +For longer experiments you wankt to be able to leave the command shell and let the cycles take care of its own. To do so, you can start the process using [`nohup`](https://linux.die.net/man/1/nohup): +~~~ +nohup ./cycle ICON-LAM ASS 2023112812 2023112912 &> cycle.log & +~~~ +This will start the process in the background, writing its output to the file `cycle.log`. + + +*Pro Tip:* You can read the output using `less`. While in `less` press `SHIFT + F` to read it like a feed. To exit this mode press `CTRL + C` and then `q` to exit `less`. + +### Interrupting and Restarting the cycle +To interrupt a cycle you can create a file named `STOP_CYCLE` (eg. `touch STOP_CYCLE`) in the `cycle` directory. After completion of the current step, the cycle will break. + +To restart a interrupted cycle at any of the past times, delete `STOP_CYCLE` and use the flags `--cycle` and `--task` of the `cycle` script. For example +~~~ +./cycle --cycle=5 --task=7 ICON-LAM ASS 2023112812 2023112912 +~~~ +restarts the former cycle at 28.11.2023 - 16 UTC, at the 7th task, which is the `core` module (assimilation). This of course also works with `nohup`. + +You can also change the `enddate` when resarting a cycle. Just make sure you also change the name of a logfile `bacy/data/cycle_icon-lam_ass.log_20230928120000_20231129120000` so the last number matches your new `endtime`. + +## Adapting bacy +Bacy is modular and quite flexible. The challenge is to know where to find the settings you want to change. A few hints: +* "global" settings can be changed in `modules/common/bacy_conf.sh` This includes parameters like the ensemble size (`BA_ENSIZE_ASS`), on-off switches for snow, surface and sst analysis (`BA_RUN_SNW`, `BA_RUN_SMA`, `BA_RUN_SST`) and several settings regarding the icon grid (serach for `GRID`) +* each module has + - namelists, which can be found in the `modules/module-name/const/` folder. Only the namelists without the `.default` ending affect the run. Inside there probably all paramters of the said module can be set. + - the file `module_name_conf.sh` containing high-level configurations of the module + - the scripts `prep_module-name`, `module-name`, `save_module-name` which make use of `prep_module-name_fcns.sh`, `module-name_fcns.sh`, `save_module-name_fcns.sh`. Inside said functions the real work of the module (file logistics, putting the namelists together and filling them with values, starting and monitoring the jobs and saving the output) is done. You can insert `ksh` prompts inside the functions to get your code executed at the desired time. Be aware that many functions differentiate between cycle and model modes, so make sure your code is actually executed where you put it. + - also see [here](./bacy.md#modules) \ No newline at end of file