Skip to content
Snippets Groups Projects
Commit 31c34397 authored by lkugler's avatar lkugler
Browse files

docs

parent ba58e361
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
analysis_only.py
+ 2
− 5
View file @ 31c34397
...@@ -5,7 +5,6 @@ running the forecast model without assimilation ...@@ -5,7 +5,6 @@ running the forecast model without assimilation
import os, sys, shutil import os, sys, shutil
import datetime as dt import datetime as dt
from dartwrf import utils
from dartwrf.workflows import WorkFlows from dartwrf.workflows import WorkFlows
...@@ -16,8 +15,6 @@ assim_time = prior_valid_time ...@@ -16,8 +15,6 @@ assim_time = prior_valid_time
w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py') w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py')
w.assimilate(assim_time, prior_init_time, prior_valid_time, prior_path_exp) id = w.assimilate(assim_time, prior_init_time, prior_valid_time, prior_path_exp)
# w.create_satimages(time, depends_on=id)
# id_sat = create_satimages(time, depends_on=id)
docs/source/index.rst
+ 3
− 1
View file @ 31c34397
Welcome to the DART-WRF documentation! Welcome to the DART-WRF documentation!
====================================== ======================================
DART-WRF is a python package to run an Ensemble Data Assimilation system using the data assimilation suite `DART <https://docs.dart.ucar.edu/en/latest/README.html>`_ and the weather research and forecast model `WRF <https://www2.mmm.ucar.edu/wrf/users/docs/docs_and_pubs.html>`_. **DART-WRF** is a python package to run an Ensemble Data Assimilation system using the data assimilation suite `DART <https://docs.dart.ucar.edu/en/latest/README.html>`_ and the weather research and forecast model `WRF <https://www2.mmm.ucar.edu/wrf/users/docs/docs_and_pubs.html>`_.
DART-WRF is available at `github.com/lkugler/DART-WRF <https://github.com/lkugler/DART-WRF>`.
Installation Installation
------------ ------------
......
docs/source/tutorial1.ipynb
+ 77
− 45
View file @ 31c34397
%% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags: %% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags:
# Tutorial 1: The assimilation step # Tutorial 1: The assimilation step
DART-WRF is a python package which automates many things like configuration, saving configuration and output, handling computing resources, etc. DART-WRF is a python package which automates many things like configuration, saving configuration and output, handling computing resources, etc.
**Goal**: Using a predefined configuration file, run an example of Data Assimilation.
**What you need to know**: %% Cell type:markdown id:93d59d4d-c514-414e-81fa-4ff390290811 tags:
- There is a config/ folder with experimental configuration in config/cfg.py. ### Configuring the experiment
- There is an example script analysis_only.py which handles the DA programs. Firstly, you need to configure the experiment in `config/cfg.py`.
**To-Do**: Let's go through the most important settings:
```python
exp = utils.Experiment()
exp.expname = "test_newcode"
exp.model_dx = 2000
exp.n_ens = 10
exp.update_vars = ['U', 'V', 'W', 'THM', 'PH', 'MU', 'QVAPOR', 'QCLOUD', 'QICE', 'PSFC']
exp.filter_kind = 1
exp.prior_inflation = 0
exp.post_inflation = 4
exp.sec = True
exp.cov_loc_vert_km_horiz_km = (3, 20)
```
In case you want to generate new observations (observing system simulations experiment, OSSE), set `sxp.use_existing_obsseq = False`.
`exp.nature` defines where observations will be drawn from, e.g.:
```python
exp.nature = '/mnt/jetfs/scratch/lkugler/data/sim_archive/exp_v1.18_P1_nature/2008-07-30_06:00/1'
```
`exp.input_profile` is used, if you create initial conditions from a so called wrf_profile (see WRF guide).
```python
exp.input_profile = '/mnt/jetfs/home/lkugler/data/initial_profiles/wrf/ens/2022-03-31/raso.fc.<iens>.wrfprof'
```
Vertical localization is tricky to set.
For horizontal localization half-width of 20 km and 3 km vertically, set
`exp.cov_loc_vert_km_horiz_km = (3, 20)`
You can also set it to zero for no vertical localization.
Set you desired observations like this.
```python
t = dict(plotname='Temperature', plotunits='[K]',
kind='RADIOSONDE_TEMPERATURE',
n_obs=1, obs_locations=[(45., 0.)],
error_generate=0.2, error_assimilate=0.2,
heights=[1000,], # range(1000, 17001, 2000),
cov_loc_radius_km=50)
exp.observations = [t,]
```
To generate a grid of observations, use
```python
vis = dict(plotname='VIS 0.6µm', plotunits='[1]',
kind='MSG_4_SEVIRI_BDRF', sat_channel=1,
n_obs=961, obs_locations='square_array_evenly_on_grid',
error_generate=0.03, error_assimilate=0.03,
cov_loc_radius_km=20)
```
But caution, n_obs should only be one of the following:
- 22500 for 2km observation density/resolution
- 5776 for 4km;
- 961 for 10km;
- 256 for 20km;
- 121 for 30km
For vertically resolved data, like radar, n_obs is the number of observations at each observation height level.
%% Cell type:markdown id:16bd3521-f98f-4c4f-8019-31029fd678ae tags:
### Configuring the hardware
In case you use a cluster which is not supported, configure paths inside `config/clusters.py`.
- Go into the DART-WRF folder that we created and installed last time.
- Copy the configuration file from `` to your config/ folder.
%% Cell type:markdown id:24b23d8c-29e6-484c-ad1f-f2bc07e66c66 tags:
### Assimilate observations with a prior given by previous forecasts
We start by importing some modules: We start by importing some modules:
```python ```python
import os, sys, shutil
import datetime as dt import datetime as dt
from dartwrf.workflows import WorkFlows
from dartwrf import utils
from config.cfg import exp
from config.clusters import cluster
``` ```
Then, we set the directory paths and times of the prior ensemble forecasts: To assimilate observations at dt.datetime `time` we set the directory paths and times of the prior ensemble forecasts:
```python ```python
prior_path_exp = '/mnt/jetfs/scratch/lkugler/data/sim_archive/exp_v1.19_P3_wbub7_noDA' prior_path_exp = '/mnt/jetfs/scratch/lkugler/data/sim_archive/exp_v1.19_P3_wbub7_noDA'
prior_init_time = dt.datetime(2008,7,30,12) prior_init_time = dt.datetime(2008,7,30,12)
prior_valid_time = dt.datetime(2008,7,30,12,30) prior_valid_time = dt.datetime(2008,7,30,12,30)
assim_time = prior_valid_time assim_time = prior_valid_time
``` ```
Finally, we run the data assimilation by calling Finally, we run the data assimilation by calling
```python ```python
cluster.setup() w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py')
os.system(
cluster.python+' '+cluster.scripts_rundir+'/assim_synth_obs.py '
+assim_time.strftime('%Y-%m-%d_%H:%M ')
+prior_init_time.strftime('%Y-%m-%d_%H:%M ')
+prior_valid_time.strftime('%Y-%m-%d_%H:%M ')
+prior_path_exp
)
create_satimages(time) w.assimilate(assim_time, prior_init_time, prior_valid_time, prior_path_exp)
``` ```
Congratulations! You're done! Congratulations! You're done!
%% Cell type:markdown id:31b23faf-0986-407f-b07f-d635a71ec2c6 tags:
---
#### Queueing systems
Note: In case you have to use a queueing system, use the builtin job scheduler, e.g.:
```python
id = cluster.create_job("Assim",
cfg_update={"ntasks": "12", "time": "60", "mem": "200G",
"ntasks-per-node": "12", "ntasks-per-core": "2"}
).run(cluster.python+' '+cluster.scripts_rundir+'/assim_synth_obs.py '
+assim_time.strftime('%Y-%m-%d_%H:%M ')
+prior_init_time.strftime('%Y-%m-%d_%H:%M ')
+prior_valid_time.strftime('%Y-%m-%d_%H:%M ')
+prior_path_exp, depends_on=[depends_on])
```
where `depends_on` is either `None` or `int` (a previous job's SLURM id).
%% Cell type:code id:82e809a8-5972-47f3-ad78-6290afe4ae17 tags: %% Cell type:code id:82e809a8-5972-47f3-ad78-6290afe4ae17 tags:
``` python ``` python
``` ```
......
docs/source/tutorial2.ipynb
+ 41
− 44
View file @ 31c34397
%% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags: %% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags:
# Tutorial 2: Cycled experiment # Tutorial 2: Forecast after DA
**Goal**: To run a cycled data assimilation experiment. **Goal**: To run a cycled data assimilation experiment.
[`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here: [`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here:
#### Configure your experiment Now there are two options:
We start again by configuring `config/cfg.py`. 1) To start a forecast from an existing forecast, i.e. from WRF restart files
2) To start a forecast from defined thermodynamic profiles, i.e. from a `wrf_profile`
Then we write a script (or edit an existing one) in the main directory `DART-WRF/`.
`nano new_experiment.py`
--- ### Restart a forecast
Any script needs to import some modules: To run a forecast from initial conditions of a previous forecasts, we import these modules
```python ```python
import os, sys, shutil
import datetime as dt import datetime as dt
from dartwrf.workflows import WorkFlows
from dartwrf import utils
from config.cfg import exp
from config.clusters import cluster
``` ```
---
Now there are two options:
- To start a forecast from an existing forecast, i.e. from WRF restart files
- To start a forecast from defined thermodynamic profiles, i.e. from a `wrf_profile`
#### Run a forecast from initial conditions of a previous forecasts
Let's say you want to run a forecast starting at 9 UTC until 12 UTC. Let's say you want to run a forecast starting at 9 UTC until 12 UTC.
Initial conditions shall be taken from a previous experiment in `/user/test/data/sim_archive/exp_abc` which was initialized at 6 UTC and there are WRF restart files for 9 UTC. Initial conditions shall be taken from a previous experiment in `/user/test/data/sim_archive/exp_abc` which was initialized at 6 UTC and there are WRF restart files for 9 UTC.
Then the code would be Then the code would be
```python ```python
prior_path_exp = '/user/test/data/sim_archive/exp_abc' prior_path_exp = '/user/test/data/sim_archive/exp_abc'
prior_init_time = dt.datetime(2008,7,30,6) prior_init_time = dt.datetime(2008,7,30,6)
prior_valid_time = dt.datetime(2008,7,30,9) prior_valid_time = dt.datetime(2008,7,30,9)
cluster.setup() w = WorkFlows(exp_config='cfg.py', server_config='srvx1.py')
begin = dt.datetime(2008, 7, 30, 9) begin = dt.datetime(2008, 7, 30, 9)
end = dt.datetime(2008, 7, 30, 12) end = dt.datetime(2008, 7, 30, 12)
prepare_WRFrundir(begin) w.prepare_WRFrundir(begin)
prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time) w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time)
run_ENS(begin=begin, # start integration from here w.run_ENS(begin=begin, # start integration from here
end=end, # integrate until here end=end, # integrate until here
output_restart_interval=9999, # do not write WRF restart files output_restart_interval=9999, # do not write WRF restart files
) )
``` ```
Note that we use predefined workflow functions like `run_ENS`. Note that we use predefined workflow functions like `run_ENS`.
#### Assimilate observations with a prior given by previous forecasts ### Forecast run after Data Assimilation
To assimilate observations at dt.datetime `time` use this command: In order to continue after assimilation you need the posterior = prior (1) + increments (2)
1. Set posterior = prior
```python ```python
prior_path_exp = '/user/test/data/sim_archive/exp_abc' id = w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id)
prior_init_time = dt.datetime(2008,7,30,6)
prior_valid_time = dt.datetime(2008,7,30,9)
time = dt.datetime(2008,7,30,9) # time of assimilation
assimilate(time, prior_init_time, prior_valid_time, prior_path_exp)
``` ```
2) Update posterior with increments from assimilation
After this, the wrfrst files are updated with assimilation increments from DART output and copied to the WRF's run directories so you can continue to run the forecast ensemble.
```python
id = w.update_IC_from_DA(time, depends_on=id)
```
#### Update initial conditions from Data Assimilation 3) Define how long you want to run the forecast and when you want WRF restart files. Since they take a lot of space, we want as few as possible.
In order to continue after assimilation you need the posterior = prior (1) + increments (2)
1. Set prior with this function:
`id = prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id)`
where path is `str`, times are `dt.datetime`. ```python
timedelta_integrate = dt.timedelta(hours=5)
output_restart_interval = 9999 # any value larger than the forecast duration
```
2. To update the model state with assimilation increments, you need to update the WRF restart files by running If you run DA in cycles of 15 minutes, it will be
```python
timedelta_integrate = dt.timedelta(hours=5)
timedelta_btw_assim = dt.timedelta(minutes=15)
output_restart_interval = timedelta_btw_assim.total_seconds()/60
```
`id = update_IC_from_DA(time, depends_on=id)`
After this, the wrfrst files are updated with assimilation increments (filter_restart) and copied to the WRF's run directories so you can continue to run the ENS after assimilation using function `run_ENS()`. 3) Run WRF ensemble
```python
id = w.run_ENS(begin=time, # start integration from here
end=time + timedelta_integrate, # integrate until here
output_restart_interval=output_restart_interval,
depends_on=id)
```
%% Cell type:code id:400244f1-098b-46ea-b29d-2226c7cbc827 tags: %% Cell type:code id:400244f1-098b-46ea-b29d-2226c7cbc827 tags:
``` python ``` python
``` ```
......
docs/source/tutorial3.ipynb
+ 16
− 17
View file @ 31c34397
%% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags: %% Cell type:markdown id:fd5c3005-f237-4495-9185-2d4d474cafd5 tags:
# Tutorial 3: Cycled experiment # Tutorial 3: Cycle forecast and assimilation
**Goal**: To run a cycled data assimilation experiment. **Goal**: To run a cycled data assimilation experiment.
[`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here: [`cycled_exp.py`](https://github.com/lkugler/DART-WRF/blob/master/generate_free.py) contains an example which will be explained here:
After configuring your experiment the loop looks like For example, your experiment can look like this
```python ```python
prior_path_exp = '/jetfs/home/lkugler/data/sim_archive/exp_v1.19_P2_noDA' prior_path_exp = '/jetfs/home/lkugler/data/sim_archive/exp_v1.19_P2_noDA'
init_time = dt.datetime(2008, 7, 30, 13) init_time = dt.datetime(2008, 7, 30, 13)
time = dt.datetime(2008, 7, 30, 14) time = dt.datetime(2008, 7, 30, 14)
last_assim_time = dt.datetime(2008, 7, 30, 14) last_assim_time = dt.datetime(2008, 7, 30, 14)
forecast_until = dt.datetime(2008, 7, 30, 14, 15) forecast_until = dt.datetime(2008, 7, 30, 14, 15)
prepare_WRFrundir(init_time) w.prepare_WRFrundir(init_time)
id = run_ideal(depends_on=id) id = w.run_ideal(depends_on=id)
prior_init_time = init_time prior_init_time = init_time
prior_valid_time = time prior_valid_time = time
while time <= last_assim_time: while time <= last_assim_time:
# usually we take the prior from the current time # usually we take the prior from the current time
# but one could use a prior from a different time from another run # but one could use a prior from a different time from another run
# i.e. 13z as a prior to assimilate 12z observations # i.e. 13z as a prior to assimilate 12z observations
prior_valid_time = time prior_valid_time = time
id = assimilate(time, prior_init_time, prior_valid_time, prior_path_exp, depends_on=id) id = w.assimilate(time, prior_init_time, prior_valid_time, prior_path_exp, depends_on=id)
# 1) Set posterior = prior # 1) Set posterior = prior
id = prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id) id = w.prepare_IC_from_prior(prior_path_exp, prior_init_time, prior_valid_time, depends_on=id)
# 2) Update posterior += updates from assimilation # 2) Update posterior += updates from assimilation
id = update_IC_from_DA(time, depends_on=id) id = w.update_IC_from_DA(time, depends_on=id)
# How long shall we integrate? # How long shall we integrate?
timedelta_integrate = timedelta_btw_assim timedelta_integrate = timedelta_btw_assim
output_restart_interval = timedelta_btw_assim.total_seconds()/60 output_restart_interval = timedelta_btw_assim.total_seconds()/60
if time == last_assim_time: #this_forecast_init.minute in [0,]: # longer forecast every full hour if time == last_assim_time: #this_forecast_init.minute in [0,]: # longer forecast every full hour
timedelta_integrate = forecast_until - last_assim_time # dt.timedelta(hours=4) timedelta_integrate = forecast_until - last_assim_time # dt.timedelta(hours=4)
output_restart_interval = 9999 # no restart file after last assim output_restart_interval = 9999 # no restart file after last assim
# 3) Run WRF ensemble # 3) Run WRF ensemble
id = run_ENS(begin=time, # start integration from here id = w.run_ENS(begin=time, # start integration from here
end=time + timedelta_integrate, # integrate until here end=time + timedelta_integrate, # integrate until here
output_restart_interval=output_restart_interval, output_restart_interval=output_restart_interval,
depends_on=id) depends_on=id)
# as we have WRF output, we can use own exp path as prior # as we have WRF output, we can use own exp path as prior
prior_path_exp = cluster.archivedir prior_path_exp = cluster.archivedir
id_sat = create_satimages(time, depends_on=id) id_sat = w.create_satimages(time, depends_on=id)
# increment time # increment time
time += timedelta_btw_assim time += timedelta_btw_assim
# update time variables # update time variables
prior_init_time = time - timedelta_btw_assim prior_init_time = time - timedelta_btw_assim
verify_sat(id_sat) w.verify_sat(id_sat)
verify_wrf(id) w.verify_wrf(id)
verify_fast(id) w.verify_fast(id)
``` ```
#### Job scheduling status #### Job scheduling status
The script submits jobs into the SLURM queue with dependencies so that SLURM starts the jobs itself as soon as resources are available. Most jobs need only a few cores, but model integration is done across many nodes: If you work on a server with a queueing system, the script submits jobs into the SLURM queue with dependencies so that SLURM starts the jobs itself as soon as resources are available. Most jobs need only a few cores, but model integration is done across many nodes. You can look at the status with
```bash ```bash
$ squeue -u `whoami` --sort=i $ squeue -u `whoami` --sort=i
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1710274 mem_0384 prepwrfr lkugler PD 0:00 1 (Priority) 1710274 mem_0384 prepwrfr lkugler PD 0:00 1 (Priority)
1710275 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency) 1710275 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)
1710276 mem_0384 Assim-42 lkugler PD 0:00 1 (Dependency) 1710276 mem_0384 Assim-42 lkugler PD 0:00 1 (Dependency)
1710277 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency) 1710277 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)
1710278 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency) 1710278 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency)
1710279 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency) 1710279 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency)
1710280_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency) 1710280_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency)
1710281 mem_0384 pRTTOV-6 lkugler PD 0:00 1 (Dependency) 1710281 mem_0384 pRTTOV-6 lkugler PD 0:00 1 (Dependency)
1710282 mem_0384 Assim-3a lkugler PD 0:00 1 (Dependency) 1710282 mem_0384 Assim-3a lkugler PD 0:00 1 (Dependency)
1710283 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency) 1710283 mem_0384 IC-prior lkugler PD 0:00 1 (Dependency)
1710284 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency) 1710284 mem_0384 IC-updat lkugler PD 0:00 1 (Dependency)
1710285 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency) 1710285 mem_0384 preWRF2- lkugler PD 0:00 1 (Dependency)
1710286_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency) 1710286_[1-10] mem_0384 runWRF2- lkugler PD 0:00 1 (Dependency)
1710287 mem_0384 pRTTOV-7 lkugler PD 0:00 1 (Dependency) 1710287 mem_0384 pRTTOV-7 lkugler PD 0:00 1 (Dependency)
``` ```
%% Cell type:code id:400244f1-098b-46ea-b29d-2226c7cbc827 tags: %% Cell type:code id:400244f1-098b-46ea-b29d-2226c7cbc827 tags:
``` python ``` python
``` ```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment