From fc5538fa38b5bdff079702e00613e3db6d34f79c Mon Sep 17 00:00:00 2001
From: lkugler <lukas.kugler@gmail.com>
Date: Tue, 14 Mar 2023 16:55:16 +0100
Subject: [PATCH] add real docs
---
docs/source/index.rst | 4 +-
docs/source/notebooks/test.ipynb | 51 --------
docs/source/notebooks/tutorial1.ipynb | 169 ++++++++++++++++++++++++++
docs/source/notebooks/tutorial2.ipynb | 122 +++++++++++++++++++
docs/source/notebooks/tutorial3.ipynb | 129 ++++++++++++++++++++
5 files changed, 423 insertions(+), 52 deletions(-)
delete mode 100644 docs/source/notebooks/test.ipynb
create mode 100644 docs/source/notebooks/tutorial1.ipynb
create mode 100644 docs/source/notebooks/tutorial2.ipynb
create mode 100644 docs/source/notebooks/tutorial3.ipynb
diff --git a/docs/source/index.rst b/docs/source/index.rst
index bb84e17..282d53b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -31,7 +31,9 @@ Other helpful resources
:maxdepth: 2
:caption: Tutorials:
- notebooks/test
+ notebooks/tutorial1
+ notebooks/tutorial2
+ notebooks/tutorial3
API
diff --git a/docs/source/notebooks/test.ipynb b/docs/source/notebooks/test.ipynb
deleted file mode 100644
index 130030b..0000000
--- a/docs/source/notebooks/test.ipynb
+++ /dev/null
@@ -1,51 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "aed6f7a5-6afe-4e11-ab8b-02d8557ad457",
- "metadata": {},
- "source": [
- "# Hello World"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "f66039b3-717f-48ff-b4af-19c93cccb911",
- "metadata": {},
- "outputs": [],
- "source": [
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5799568d-2ffe-4e67-88c1-87113b252a74",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "JHubPy - 3.9.6",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.6"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/docs/source/notebooks/tutorial1.ipynb b/docs/source/notebooks/tutorial1.ipynb
new file mode 100644
index 0000000..798d6b9
--- /dev/null
+++ b/docs/source/notebooks/tutorial1.ipynb
@@ -0,0 +1,169 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "fd5c3005-f237-4495-9185-2d4d474cafd5",
+ "metadata": {},
+ "source": [
+ "# Tutorial 1: The assimilation step\n",
+ "DART-WRF is a python package which automates many things like configuration, saving configuration and output, handling computing resources, etc.\n",
+ "\n",
+ "The data for this experiment is accessible for students on the server srvx1.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "93d59d4d-c514-414e-81fa-4ff390290811",
+ "metadata": {},
+ "source": [
+ "### Configuring the experiment\n",
+ "Firstly, you need to configure the experiment in `config/cfg.py`.\n",
+ "\n",
+ "Let's go through the most important settings:\n",
+ "\n",
+ "- expname should be a unique identifier and will be used as folder name\n",
+ "- model_dx is the model resolution in meters\n",
+ "- n_ens is the ensemble size\n",
+ "- update_vars are the WRF variables which shall be updated by the assimilation\n",
+ "- filter_kind is 1 for the EAKF (see the DART documentation for more)\n",
+ "- prior and post_inflation defines what inflation we want (see the DART docs)\n",
+ "- sec is the statistical sampling error correction from Anderson (2012)\n",
+ "\n",
+ "```python\n",
+ "exp = utils.Experiment()\n",
+ "exp.expname = \"test_newcode\"\n",
+ "exp.model_dx = 2000\n",
+ "exp.n_ens = 10\n",
+ "exp.update_vars = ['U', 'V', 'W', 'THM', 'PH', 'MU', 'QVAPOR', 'QCLOUD', 'QICE', 'PSFC']\n",
+ "exp.filter_kind = 1\n",
+ "exp.prior_inflation = 0\n",
+ "exp.post_inflation = 4\n",
+ "exp.sec = True\n",
+ "\n",
+ "```\n",
+ "In case you want to generate new observations like for an observing system simulations experiment, OSSE), set \n",
+ "```python\n",
+ "exp.use_existing_obsseq = False`.\n",
+ "```\n",
+ "\n",
+ "`exp.nature` defines which WRF files will be used to draw observations from, e.g.: \n",
+ "```python\n",
+ "exp.nature = '/users/students/lehre/advDA_s2023/data/sample_nature/'\n",
+ "```\n",
+ "\n",
+ "`exp.input_profile` is used, if you create initial conditions from a so called wrf_profile (see WRF guide).\n",
+ "```python\n",
+ "exp.input_profile = '/doesnt_exist/initial_profiles/wrf/ens/raso.fc.<iens>.wrfprof'\n",
+ "```\n",
+ "\n",
+ "\n",
+ "For horizontal localization half-width of 20 km and 3 km vertically, set\n",
+ "```python\n",
+ "exp.cov_loc_vert_km_horiz_km = (3, 20)\n",
+ "```\n",
+ "You can also set it to False for no vertical localization.\n",
+ "\n",
+ "#### Single observation\n",
+ "Set your desired observations like this. \n",
+ "```python\n",
+ "t = dict(plotname='Temperature', plotunits='[K]',\n",
+ " kind='RADIOSONDE_TEMPERATURE', \n",
+ " n_obs=1, # number of observations\n",
+ " obs_locations=[(45., 0.)], # location of observations\n",
+ " error_generate=0.2, # observation error used to generate observations\n",
+ " error_assimilate=0.2, # observation error used for assimilation\n",
+ " heights=[1000,], # for radiosondes, use range(1000, 17001, 2000)\n",
+ " cov_loc_radius_km=50) # horizontal localization half-width\n",
+ "\n",
+ "exp.observations = [t,] # select observations for assimilation\n",
+ "```\n",
+ "\n",
+ "#### Multiple observations\n",
+ "To generate a grid of observations, use\n",
+ "```python\n",
+ "vis = dict(plotname='VIS 0.6µm', plotunits='[1]',\n",
+ " kind='MSG_4_SEVIRI_BDRF', sat_channel=1, \n",
+ " n_obs=961, obs_locations='square_array_evenly_on_grid',\n",
+ " error_generate=0.03, error_assimilate=0.03,\n",
+ " cov_loc_radius_km=20)\n",
+ "```\n",
+ "\n",
+ "But caution, n_obs should only be one of the following:\n",
+ "\n",
+ "- 22500 for 2km observation density/resolution \n",
+ "- 5776 for 4km; \n",
+ "- 961 for 10km; \n",
+ "- 256 for 20km; \n",
+ "- 121 for 30km\n",
+ "\n",
+ "For vertically resolved data, like radar, n_obs is the number of observations at each observation height level."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "16bd3521-f98f-4c4f-8019-31029fd678ae",
+ "metadata": {},
+ "source": [
+ "### Configuring the hardware\n",
+ "In case you use a cluster which is not supported, configure paths inside `config/clusters.py`.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "### Assimilate observations\n",
+ "We start by importing some modules:\n",
+ "```python\n",
+ "import datetime as dt\n",
+ "from dartwrf.workflows import WorkFlows\n",
+ "```\n",
+ "\n",
+ "To assimilate observations at dt.datetime `time` we set the directory paths and times of the prior ensemble forecasts:\n",
+ "\n",
+ "```python\n",
+ "prior_path_exp = '/users/students/lehre/advDA_s2023/data/sample_ensemble/'\n",
+ "prior_init_time = dt.datetime(2008,7,30,12)\n",
+ "prior_valid_time = dt.datetime(2008,7,30,12,30)\n",
+ "assim_time = prior_valid_time\n",
+ "```\n",
+ "\n",
+ "Finally, we run the data assimilation by calling\n",
+ "```python\n",
+ "w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py')\n",
+ "\n",
+ "w.assimilate(assim_time, prior_init_time, prior_valid_time, prior_path_exp)\n",
+ "```\n",
+ "\n",
+ "Congratulations! You're done!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "82e809a8-5972-47f3-ad78-6290afe4ae17",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "nwp 2023.1 - 3.10",
+ "language": "python",
+ "name": "nwp2023.1"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/source/notebooks/tutorial2.ipynb b/docs/source/notebooks/tutorial2.ipynb
new file mode 100644
index 0000000..df58b78
--- /dev/null
+++ b/docs/source/notebooks/tutorial2.ipynb
@@ -0,0 +1,122 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "fd5c3005-f237-4495-9185-2d4d474cafd5",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Tutorial 2: Forecast after DA\n",
+ "\n",
+ "\n",
+ "**Goal**: To run a cycled data assimilation experiment.\n",
+ "[`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here:\n",
+ "\n",
+ "Now there are two options:\n",
+ "1) To start a forecast from an existing forecast, i.e. from WRF restart files\n",
+ "2) To start a forecast from defined thermodynamic profiles, i.e. from a `wrf_profile`\n",
+ "\n",
+ "\n",
+ "### Restart a forecast\n",
+ "To run a forecast from initial conditions of a previous forecasts, we import these modules\n",
+ "```python\n",
+ "import datetime as dt\n",
+ "from dartwrf.workflows import WorkFlows\n",
+ "```\n",
+ "\n",
+ "Let's say you want to run a forecast starting at 9 UTC until 12 UTC.\n",
+ "Initial conditions shall be taken from a previous experiment in `/user/test/data/sim_archive/exp_abc` which was initialized at 6 UTC and there are WRF restart files for 9 UTC.\n",
+ "Then the code would be\n",
+ "\n",
+ "```python\n",
+ "prior_path_exp = '/user/test/data/sim_archive/exp_abc'\n",
+ "prior_init_time = dt.datetime(2008,7,30,6)\n",
+ "prior_valid_time = dt.datetime(2008,7,30,9)\n",
+ "\n",
+ "w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py')\n",
+ "\n",
+ "begin = dt.datetime(2008, 7, 30, 9)\n",
+ "end = dt.datetime(2008, 7, 30, 12)\n",
+ "\n",
+ "w.prepare_WRFrundir(begin)\n",
+ "\n",
+ "w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time)\n",
+ "\n",
+ "w.run_ENS(begin=begin, # start integration from here\n",
+ " end=end, # integrate until here\n",
+ " output_restart_interval=9999, # do not write WRF restart files\n",
+ " )\n",
+ "```\n",
+ "Note that we use predefined workflow functions like `run_ENS`.\n",
+ "\n",
+ "\n",
+ "### Forecast run after Data Assimilation\n",
+ "In order to continue after assimilation you need the posterior = prior (1) + increments (2)\n",
+ "\n",
+ "1. Set posterior = prior\n",
+ "```python\n",
+ "id = w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id)\n",
+ "```\n",
+ "\n",
+ "2) Update posterior with increments from assimilation\n",
+ "After this, the wrfrst files are updated with assimilation increments from DART output and copied to the WRF's run directories so you can continue to run the forecast ensemble.\n",
+ "```python\n",
+ "id = w.update_IC_from_DA(time, depends_on=id)\n",
+ "```\n",
+ "\n",
+ "3) Define how long you want to run the forecast and when you want WRF restart files. Since they take a lot of space, we want as few as possible.\n",
+ "\n",
+ "```python\n",
+ "timedelta_integrate = dt.timedelta(hours=5)\n",
+ "output_restart_interval = 9999 # any value larger than the forecast duration\n",
+ "```\n",
+ "\n",
+ "If you run DA in cycles of 15 minutes, it will be\n",
+ "```python\n",
+ "timedelta_integrate = dt.timedelta(hours=5)\n",
+ "timedelta_btw_assim = dt.timedelta(minutes=15)\n",
+ "output_restart_interval = timedelta_btw_assim.total_seconds()/60\n",
+ "```\n",
+ "\n",
+ "\n",
+ "3) Run WRF ensemble\n",
+ "```python\n",
+ "id = w.run_ENS(begin=time, # start integration from here\n",
+ " end=time + timedelta_integrate, # integrate until here\n",
+ " output_restart_interval=output_restart_interval,\n",
+ " depends_on=id)\n",
+ "```\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "400244f1-098b-46ea-b29d-2226c7cbc827",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "nwp 2023.1 - 3.10",
+ "language": "python",
+ "name": "nwp2023.1"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/source/notebooks/tutorial3.ipynb b/docs/source/notebooks/tutorial3.ipynb
new file mode 100644
index 0000000..d3018ab
--- /dev/null
+++ b/docs/source/notebooks/tutorial3.ipynb
@@ -0,0 +1,129 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "fd5c3005-f237-4495-9185-2d4d474cafd5",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Tutorial 3: Cycle forecast and assimilation\n",
+ "\n",
+ "\n",
+ "**Goal**: To run a cycled data assimilation experiment.\n",
+ "[`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here:\n",
+ "\n",
+ "For example, your experiment can look like this\n",
+ "\n",
+ "```python\n",
+ "prior_path_exp = '/jetfs/home/lkugler/data/sim_archive/exp_v1.19_P2_noDA'\n",
+ "\n",
+ "init_time = dt.datetime(2008, 7, 30, 13)\n",
+ "time = dt.datetime(2008, 7, 30, 14)\n",
+ "last_assim_time = dt.datetime(2008, 7, 30, 14)\n",
+ "forecast_until = dt.datetime(2008, 7, 30, 14, 15)\n",
+ "\n",
+ "w.prepare_WRFrundir(init_time)\n",
+ "id = w.run_ideal(depends_on=id)\n",
+ "\n",
+ "prior_init_time = init_time\n",
+ "prior_valid_time = time\n",
+ "\n",
+ "while time <= last_assim_time:\n",
+ "\n",
+ " # usually we take the prior from the current time\n",
+ " # but one could use a prior from a different time from another run\n",
+ " # i.e. 13z as a prior to assimilate 12z observations\n",
+ " prior_valid_time = time\n",
+ "\n",
+ " id = w.assimilate(time, prior_init_time, prior_valid_time, prior_path_exp, depends_on=id)\n",
+ "\n",
+ " # 1) Set posterior = prior\n",
+ " id = w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id)\n",
+ "\n",
+ " # 2) Update posterior += updates from assimilation\n",
+ " id = w.update_IC_from_DA(time, depends_on=id)\n",
+ "\n",
+ " # How long shall we integrate?\n",
+ " timedelta_integrate = timedelta_btw_assim\n",
+ " output_restart_interval = timedelta_btw_assim.total_seconds()/60\n",
+ " if time == last_assim_time: #this_forecast_init.minute in [0,]: # longer forecast every full hour\n",
+ " timedelta_integrate = forecast_until - last_assim_time # dt.timedelta(hours=4)\n",
+ " output_restart_interval = 9999 # no restart file after last assim\n",
+ "\n",
+ " # 3) Run WRF ensemble\n",
+ " id = w.run_ENS(begin=time, # start integration from here\n",
+ " end=time + timedelta_integrate, # integrate until here\n",
+ " output_restart_interval=output_restart_interval,\n",
+ " depends_on=id)\n",
+ "\n",
+ " # as we have WRF output, we can use own exp path as prior\n",
+ " prior_path_exp = cluster.archivedir \n",
+ "\n",
+ " id_sat = w.create_satimages(time, depends_on=id)\n",
+ "\n",
+ " # increment time\n",
+ " time += timedelta_btw_assim\n",
+ "\n",
+ " # update time variables\n",
+ " prior_init_time = time - timedelta_btw_assim\n",
+ " \n",
+ "w.verify_sat(id_sat)\n",
+ "w.verify_wrf(id)\n",
+ "w.verify_fast(id)\n",
+ "```\n",
+ "\n",
+ "#### Job scheduling status\n",
+ "If you work on a server with a queueing system, the script submits jobs into the SLURM queue with dependencies so that SLURM starts the jobs itself as soon as resources are available. Most jobs need only a few cores, but model integration is done across many nodes. You can look at the status with\n",
+ "```bash\n",
+ "$ squeue -u `whoami` --sort=i\n",
+ " JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)\n",
+ " 1710274 mem_0384 prepwrfr lkugler PD 0:00 1 (Priority)\n",
+ " 1710275 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)\n",
+ " 1710276 mem_0384 Assim-42 lkugler PD 0:00 1 (Dependency)\n",
+ " 1710277 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)\n",
+ " 1710278 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency)\n",
+ " 1710279 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency)\n",
+ " 1710280_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency)\n",
+ " 1710281 mem_0384 pRTTOV-6 lkugler PD 0:00 1 (Dependency)\n",
+ " 1710282 mem_0384 Assim-3a lkugler PD 0:00 1 (Dependency)\n",
+ " 1710283 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)\n",
+ " 1710284 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency)\n",
+ " 1710285 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency)\n",
+ " 1710286_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency)\n",
+ " 1710287 mem_0384 pRTTOV-7 lkugler PD 0:00 1 (Dependency)\n",
+ "```\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "400244f1-098b-46ea-b29d-2226c7cbc827",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "nwp 2023.1 - 3.10",
+ "language": "python",
+ "name": "nwp2023.1"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
--
GitLab