scripts now separated from rundir

LICENSE

+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

+# DART-WRF
+
+This code runs an OSSE with DART and WRF on a cluster with SLURM.
+
+- readable: see what it does at first glance
+- flexible: schedule jobs in python or any other language
+- Slurm-aware: functions return a SLURM ID `id1 = function()`, to start another function as soon as the one before returned `function2(depends_on=id1)` thanks to ![brentp/slurmpy](https://github.com/brentp/slurmpy).
+
+
+### A possible workflow:
+`scheduler.py`
+```python
+
+# create initial conditions
+id = prep_osse()  
+
+# spin up the ensemble
+background_init_time = dt.datetime(2008, 7, 30, 6, 0)
+integration_end_time = dt.datetime(2008, 7, 30, 11, 0)
+id = run_ENS(begin=background_init_time,
+             end=integration_end_time,
+             depends_on=id)
+             
+time = integration_end_time  # time now
+
+# now, start the ensemble data assimilation cycles
+timedelta_integrate = dt.timedelta(minutes=15)
+timedelta_btw_assim = dt.timedelta(minutes=15)
+
+while time < dt.datetime(2008, 7, 30, 16, 15):
+     assim_time = time
+     id = gen_synth_obs(assim_time, depends_on=id)
+     id = assimilate(assim_time,
+                     background_init_time,
+                     depends_on=id)
+
+     background_init_time = assim_time  # start integration now
+     integration_end_time = assim_time + timedelta_integrate
+     id = run_ENS(begin=background_init_time,
+                  end=integration_end_time,
+                  depends_on=id)
+
+     time += timedelta_btw_assim
+```
+
+### SLURM submissions
+`scheduler.py` submits jobs into the SLURM queue with dependencies, so that SLURM starts the jobs itself as soon as resources are available. Most jobs need only one node, but model integration is done in a SLURM job array across e.g. 5 nodes:
+```
+$ squeue -u `whoami` --sort=$E
+            308377  mem_0384 ideal-01  lkugler PD       0:00      1 (Resources)
+            308378  mem_0384 prerun-a  lkugler PD       0:00      1 (Priority)
+      308379_[1-5]  mem_0384 EnsWRF-3  lkugler PD       0:00      1 (Dependency)
+            308380  mem_0384 pregensy  lkugler PD       0:00      1 (Dependency)
+            308381  mem_0384 gensynth  lkugler PD       0:00      1 (Dependency)
+            308382  mem_0384 preassim  lkugler PD       0:00      1 (Dependency)
+            308383  mem_0384 assim-37  lkugler PD       0:00      1 (Dependency)
+            308384  mem_0384 postassi  lkugler PD       0:00      1 (Dependency)
+            308385  mem_0384 prerun-e  lkugler PD       0:00      1 (Dependency)
+      308386_[1-5]  mem_0384 EnsWRF-3  lkugler PD       0:00      1 (Dependency)
+            308387  mem_0384 pregensy  lkugler PD       0:00      1 (Dependency)
+            308388  mem_0384 gensynth  lkugler PD       0:00      1 (Dependency)
+            308389  mem_0384 preassim  lkugler PD       0:00      1 (Dependency)
+            308390  mem_0384 assim-37  lkugler PD       0:00      1 (Dependency)
+            308391  mem_0384 postassi  lkugler PD       0:00      1 (Dependency)
+            308392  mem_0384 prerun-6  lkugler PD       0:00      1 (Dependency)
+      308393_[1-5]  mem_0384 EnsWRF-3  lkugler PD       0:00      1 (Dependency)
+```
+
+### Easily switch between clusters
+
+`config/clusters.py `
+```python
+
+clusterA = ClusterConfig()
+clusterA.name = 'vsc'
+clusterA.userdir = '/home/pathA/myuser/'
+...
+clusterB = ClusterConfig()
+clusterB.name = 'jet'
+clusterB.userdir = '/home/pathB/myuser/'
+```
+
+`config/cfg.py`
+```python
+
+from . import clusters
+cluster = clusters.clusterA  # change cluster configuration here
+```
+
+
+### References
+This repo is a simplified python version of the code provided in the [DART-WRF Tutorial](http://www.image.ucar.edu/wrfdart/tutorial/).
+DART itself is available at github: [@NCAR/DART](https://github.com/NCAR/DART)
+
+### License
+DART is licensed under the Apache License, Version 2.0
+Copyright 2019 University Corporation for Atmospheric Research

config/cfg.py

+
+from . import clusters
+cluster = clusters.vsc  # change cluster configuration here
+
+class ExperimentConfiguration(object):
+    def __init__(self):
+        pass
+
+exp = ExperimentConfiguration()
+exp.expname = "exp_v1.10_LMU+shear_filter"
+exp.model_dx = 2000
+exp.timestep = 10
+exp.n_ens = 20
+exp.n_nodes = 5
+exp.n_obs = 100
+exp.error_variance = 0.001
+
+
+# directory paths depend on the name of the experiment
+cluster.expname = exp.expname

config/clusters.py

+import os, sys
+import datetime as dt
+
+class ClusterConfig(object):
+    """Helper class"""
+    def __init__(self):
+        pass
+
+    def archivedir(self):
+        return '/gpfs/data/fs71386/lkugler/sim_archive/'+self.expname  # #return '/raid61/scratch/lkugler/VSC/'+self.expname
+
+    def wrf_rundir(self, iens):
+        return self.userdir+'/run_WRF/'+self.expname+'/'+str(iens)
+
+#######################################################################################
+
+
+vsc = ClusterConfig()
+vsc.name = 'vsc'
+vsc.python = '/home/fs71386/lkugler/miniconda3/bin/python'
+vsc.ncks = '/home/fs71386/lkugler/miniconda3/envs/DART/bin/ncks'
+vsc.userdir = '/home/fs71386/lkugler'
+vsc.srcdir = '/home/fs71386/lkugler/compile/WRF/WRF-4.1.5/run'
+vsc.dartrundir = '/home/fs71386/lkugler/run_DART'
+vsc.scriptsdir = '/home/fs71386/lkugler/DART-WRF/scripts'
+vsc.nature_wrfout = '/home/fs71386/lkugler/data/sim_archive/exp_v1.10_LMU+shear_nature/2008-07-30_06:00/2/wrfout_d01_%Y-%m-%d_%H:%M:%S'
+
+vsc.ideal = vsc.userdir+'/compile/bin/ideal-v4.1.5_v1.10.exe'
+vsc.wrfexe = vsc.userdir+'/compile/bin/wrf-v4.1.5_v1.10.exe'
+vsc.namelist = vsc.scriptsdir+'/templates/namelist.input'
+vsc.run_WRF = '/gpfs/data/fs71386/lkugler/DART-WRF/scripts/osse/run_ens.vsc.sh'
+
+vsc.slurm_cfg = {"account": "p71386", "partition": "mem_0384", "qos": "p71386_0384",
+                 "ntasks-per-node": "48", "ntasks-per-core": 1}
+
+
+jet = ClusterConfig()
+jet.name = 'jet'
+jet.python = '/jetfs/home/lkugler/miniconda3/bin/python'
+jet.ncks = 'ncks'
+jet.userdir = '/jetfs/home/lkugler'
+jet.srcdir = '/jetfs/home/lkugler/compile/WRF/WRF-4.1.5/run'
+jet.dartrundir = '/jetfs/home/lkugler/DART-WRF/rundir'
+jet.scriptsdir = '/jetfs/home/lkugler/DART-WRF/scripts/osse'
+jet.nature_wrfout = '/raid61/scratch/lkugler/VSC/sim_archive/OSSE_v1.10_LMU+shear/2/single/wrfout_d01_%Y-%m-%d_%H:%M:%S'
+
+jet.ideal = jet.userdir+'/compile/bin/ideal.exe'
+jet.wrfexe = jet.userdir+'/compile/bin/wrf-v4.2_v1.10.dmpar.exe'
+jet.namelist = jet.userdir+'/config_files/namelist.input'
+jet.run_WRF = '/jetfs/home/lkugler/DART-WRF/scripts/osse/run_ens.jet.sh'
+
+jet.slurm_cfg = {"account": "p71386", "partition": "mem_0384", "qos": "p71386_0384",
+                 "mem-per-cpu": "2GB",
+                 "ntasks-per-node": "48", "ntasks-per-core": 1, "gres": "none"}

scheduler.py

+#!/usr/bin/python3
+"""
+high level control script
+submitting jobs into SLURM queue
+"""
+import os, sys, shutil
+import datetime as dt
+from slurmpy import Slurm
+
+# necessary to find modules in folder, since SLURM runs the script elsewhere
+sys.path.append(os.getcwd())
+
+from config.cfg import exp, cluster
+from utils import script_to_str
+
+
+def my_Slurm(*args, cfg_update=dict(), **kwargs):
+    """Shortcut to slurmpy's class; keep default kwargs
+    and only update according to `cfg_update`
+    see https://github.com/brentp/slurmpy
+    """
+    return Slurm(*args, slurm_kwargs=dict(cluster.slurm_cfg, **cfg_update), **kwargs)
+
+
+def clear_logs(backup_existing_to_archive=True):
+    dirs = ['/logs/', '/slurm-scripts/']
+    for d in dirs:
+        archdir = cluster.archivedir()+d
+        if backup_existing_to_archive: 
+            os.makedirs(archdir, exist_ok=True)
+        for f in os.listdir(cluster.scriptsdir+d):
+            if backup_existing_to_archive: 
+                shutil.move(cluster.scriptsdir+d+f, archdir+f)
+            else:
+                os.remove(cluster.scriptsdir+d+f)
+
+def prep_osse():
+    s = my_Slurm("pre_osse", cfg_update={"nodes": "1", "ntasks": "1", "time": "5",
+                 "mail-type": "BEGIN", "mail-user": "lukas.kugler@univie.ac.at"})
+    id = s.run(cluster.python+' '+cluster.scriptsdir+'/prep_osse.py')
+
+    s = my_Slurm("ideal", cfg_update={"nodes": "1", "time": "10", "mem-per-cpu": "2G"})
+    cmd = """# run ideal.exe in parallel, then add geodata
+export SLURM_STEP_GRES=none
+for ((n=1; n<="""+str(exp.n_ens)+"""; n++))
+do
+    rundir="""+cluster.userdir+'/run_WRF/'+exp.expname+"""/$n
+    echo $rundir
+    cd $rundir
+    mpirun -np 1 ./ideal.exe &
+done
+wait
+for ((n=1; n<="""+str(exp.n_ens)+"""; n++))
+do
+    rundir="""+cluster.userdir+'/run_WRF/'+exp.expname+"""/$n
+    mv $rundir/rsl.out.0000 $rundir/rsl.out.input
+    mkdir -p """+cluster.archivedir()+"""/wrfinput/$n
+    cp $rundir/wrfinput_d01 """+cluster.archivedir()+"""/wrfinput/$n/wrfinput_d01
+done
+"""
+    id = s.run(cmd, depends_on=[id])
+    return id
+
+
+def run_ENS(begin, end, depends_on=None, **kwargs):
+    prev_id = depends_on
+
+    s = my_Slurm("preWRF", cfg_update=dict(nodes="1", ntasks="1", time="2"))
+    id = s.run(cluster.python+' '+cluster.scriptsdir+'/prepare_namelist.py '
+               + begin.strftime('%Y-%m-%d_%H:%M')+' '
+               + end.strftime('%Y-%m-%d_%H:%M'),
+               depends_on=[prev_id])
+
+    runtime_real_hours = (end-begin).total_seconds()/3600
+    runtime_wallclock_mins_expected = int(11+runtime_real_hours*10)  # usually below 8 min/hour
+    s = my_Slurm("runWRF", cfg_update={"nodes": "1", "array": "1-"+str(exp.n_nodes),
+                 "time": str(runtime_wallclock_mins_expected), "mem-per-cpu": "2G"})
+    cmd = script_to_str(cluster.run_WRF).replace('<expname>', exp.expname)
+    id2 = s.run(cmd, depends_on=[id])
+
+    # not needed, since wrf.exe writes directly to archive folder
+    #s = my_Slurm("archiveWRF", cfg_update=dict(nodes="1", ntasks="1", time="10"))
+    #id3 = s.run(cluster.python+' '+cluster.scriptsdir+'/archive_wrf.py '
+    #           + begin.strftime('%Y-%m-%d_%H:%M'), depends_on=[id2])
+    return id2
+
+
+def gen_synth_obs(time, depends_on=None):
+    prev_id = depends_on
+
+    s = my_Slurm("pre_gensynthobs", cfg_update=dict(nodes="1", ntasks="1", time="2"))
+    id = s.run(cluster.python+' '+cluster.scriptsdir+'/pre_gen_synth_obs.py '+time.strftime('%Y-%m-%d_%H:%M'),
+               depends_on=[prev_id])
+
+    s = my_Slurm("gensynth", cfg_update=dict(nodes="1", time="20"))
+    cmd = 'cd '+cluster.dartrundir+'; mpirun -np 24 ./perfect_model_obs'
+    id2 = s.run(cmd, depends_on=[id])
+    return id2
+
+
+def assimilate(assim_time, background_init_time, depends_on=None, **kwargs):
+    prev_id = depends_on
+
+    s = my_Slurm("preAssim", cfg_update=dict(nodes="1", ntasks="1", time="2"))
+    id = s.run(cluster.python+' '+cluster.scriptsdir+'/pre_assim.py ' \
+               +assim_time.strftime('%Y-%m-%d_%H:%M ')  \
+               +background_init_time.strftime('%Y-%m-%d_%H:%M'),
+               depends_on=[prev_id])
+
+    s = my_Slurm("Assim", cfg_update=dict(nodes="1", time="50", mem="200G"))
+    cmd = 'cd '+cluster.dartrundir+'; mpirun -genv I_MPI_PIN_PROCESSOR_LIST=0-47 -np 48 ./filter'
+    id2 = s.run(cmd, depends_on=[id])
+
+    s = my_Slurm("archiveAssim", cfg_update=dict(nodes="1", ntasks="1", time="10"))
+    id3 = s.run(cluster.python+' '+cluster.scriptsdir+'/archive_assim.py '
+               + time.strftime('%Y-%m-%d_%H:%M'), depends_on=[id2])
+
+    s = my_Slurm("updateIC", cfg_update=dict(nodes="1", ntasks="1", time="3"))
+    id4 = s.run(cluster.python+' '+cluster.scriptsdir+'/update_wrfinput_from_filteroutput.py '
+                +time.strftime('%Y-%m-%d_%H:%M'), depends_on=[id3])
+    return id4
+
+def prep_initials_from_archive(time):
+    s = my_Slurm("pre_initial", cfg_update={"nodes": "1", "ntasks": "1", "time": "5",
+                 "mail-type": "BEGIN", "mail-user": "lukas.kugler@univie.ac.at"})
+    id = s.run(cluster.python+' '+cluster.scriptsdir+'/prep_osse.py')
+
+    s = my_Slurm("ideal", cfg_update={"nodes": "1", "time": "10", "mem-per-cpu": "2G"})
+    cmd = """# run ideal.exe in parallel, then add geodata
+export SLURM_STEP_GRES=none
+for ((n=1; n<="""+str(exp.n_ens)+"""; n++))
+do
+    rundir="""+cluster.userdir+'/run_WRF/'+exp.expname+"""/$n
+    echo $rundir
+    cd $rundir
+    mpirun -np 1 ./ideal.exe &
+done
+wait
+for ((n=1; n<="""+str(exp.n_ens)+"""; n++))
+do
+    rundir="""+cluster.userdir+'/run_WRF/'+exp.expname+"""/$n
+    mv $rundir/rsl.out.0000 $rundir/rsl.out.input
+done
+"""
+    id2 = s.run(cmd, depends_on=[id])
+    s = my_Slurm("upd_wrfinput", cfg_update={"nodes": "1", "ntasks": "1", "time": "5",
+                 "mail-type": "BEGIN", "mail-user": "lukas.kugler@univie.ac.at"})
+    id3 = s.run(cluster.python+' '+cluster.scriptsdir+'/update_wrfinput_from_wrfout.py '
+                +time.strftime('%Y-%m-%d_%H:%M'), depends_on=[id2])
+    return id3
+
+def mailme(depends_on=None):
+    id = depends_on
+    if id:
+        s = my_Slurm("AllFinished", cfg_update={"nodes": "1", "ntasks": "1", "time": "1",
+                     "mail-type": "BEGIN", "mail-user": "lukas.kugler@univie.ac.at"})
+        s.run('sleep 1', depends_on=[id])
+
+
+################################
+print('starting osse')
+
+clear_logs(backup_existing_to_archive=True)
+
+#id = prep_osse()  # create initial conditions
+id = None
+# spin up the ensemble
+#background_init_time = dt.datetime(2008, 7, 30, 6, 0)
+#integration_end_time = dt.datetime(2008, 7, 30, 11, 0)
+#id = run_ENS(begin=background_init_time,
+#             end=integration_end_time,
+#             depends_on=id)
+
+background_init_time = dt.datetime(2008, 7, 30, 13, 0)
+time = dt.datetime(2008, 7, 30, 13, 15) #integration_end_time
+
+#id = prep_initials_from_archive(time)
+
+# now, start the ensemble data assimilation cycles
+timedelta_integrate = dt.timedelta(minutes=15)
+timedelta_btw_assim = dt.timedelta(minutes=15)
+
+while time < dt.datetime(2008, 7, 30, 16, 15):
+
+     assim_time = time
+     id = gen_synth_obs(assim_time, depends_on=id)
+     id = assimilate(assim_time, 
+                     background_init_time,
+                     depends_on=id)
+
+     background_init_time = assim_time  # start integration from here
+     integration_end_time = assim_time + timedelta_integrate
+     id = run_ENS(begin=background_init_time, 
+                  end=integration_end_time, 
+                  depends_on=id)
+     
+     time += timedelta_btw_assim
+
+mailme(id)

scripts/archive_assim.py

+import os, sys, warnings, glob
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_scp_srvx8, copy, mkdir, mkdir_srvx8, clean_wrfdir
+
+# if cluster.name != 'srvx8':
+#     copy = copy_scp_srvx8
+#     mkdir = mkdir_srvx8
+
+time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+
+print('archive obs space diagnostics')
+savedir = cluster.archivedir()+'/obs_seq_final/'
+mkdir(savedir)
+copy(cluster.dartrundir+'/obs_seq.final', savedir+time.strftime('/%Y-%m-%d_%H:%M_obs_seq.final'))
+
+print('archive model state')
+try:
+    mkdir(cluster.archivedir())
+
+    # copy mean and sd to archive
+    for f in ['preassim_mean.nc', 'preassim_sd.nc',
+              'output_mean.nc', 'output_sd.nc']:
+        copy(cluster.dartrundir+'/'+f,
+             cluster.archivedir()+'/'+f[:-3]+time.strftime('_%Y-%m-%d_%H:%M:%S'))
+
+    print('copy members to archive')
+    for iens in range(1, exp.n_ens+1):
+        savedir = cluster.archivedir()+'/'+time.strftime('/%Y-%m-%d_%H:%M/')+str(iens)
+        mkdir(savedir)
+
+        filter_in = cluster.dartrundir+'/preassim_member_'+str(iens).zfill(4)+'.nc'
+        filter_out = cluster.dartrundir+'/filter_restart_d01.'+str(iens).zfill(4)
+
+        copy(filter_in, savedir+time.strftime('/%Y-%m-%d_%H:%M_prior'))
+        copy(filter_out, savedir+time.strftime('/%Y-%m-%d_%H:%M_posterior'))
+
+except Exception as e:
+    warnings.warn(str(e))

scripts/archive_wrf.py

+import os, sys, warnings, glob
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_scp_srvx8, copy, mkdir, mkdir_srvx8, clean_wrfdir
+
+# if cluster.name != 'srvx8':
+#     copy = copy_scp_srvx8
+#     mkdir = mkdir_srvx8
+
+time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+
+print('archive forecasts')
+try:
+    start = time.strftime('%Y-%m-%d_%H:%M')
+    for iens in range(1, exp.n_ens+1):
+        savedir = cluster.archivedir()+'/'+start+'/'+str(iens)
+        mkdir(savedir)
+
+        wrfout_files = glob.glob(cluster.wrf_rundir(iens)+'/wrfout_d01_*')
+        wrfout_files.sort()
+        for f in wrfout_files:
+            copy(f, savedir+'/'+os.path.basename(f))
+            print(savedir+'/'+os.path.basename(f), 'saved.')
+
+except Exception as e:
+    warnings.warn(str(e))

scripts/create_obs_sat.py

+"""Create obs_seq.in
+
+"""
+import os, sys
+import numpy as np
+import datetime as dt
+from pysolar.solar import get_altitude, get_azimuth
+
+def degr_to_rad(degr):
+    if degr < 0:
+        degr += 360
+    return degr/360*2*np.pi
+
+def round_to_day(dtobj):
+    return dtobj.replace(second=0, minute=0, hour=0)
+
+def add_timezone_UTC(t):
+    return dt.datetime(t.year, t.month, t.day, t.hour, t.minute, tzinfo=dt.timezone.utc)
+
+
+def run(time_dt, n_obs, error_variance, output_path='./'):
+    """Create obs_seq.in
+
+    implemented for
+        - reflectivity of VIS 0.6
+    """
+
+
+    time_dt = add_timezone_UTC(time_dt)
+    # time_dt = dt.datetime(2008, 7, 30, 15, 30, tzinfo=dt.timezone.utc)
+    assert n_obs == int(n_obs)
+    n_obs_str = str(int(n_obs))
+    error_variance = str(error_variance)
+
+    # solar angles
+    lat0 = 45.
+    lon0 = 0.
+
+    sat_az = "180.0"
+    sat_zen = "45.0"
+    radius_earth = 6.371*1E6
+
+    # equally spaced grid for satellite observations every 4 km
+    nx, ny = int(np.sqrt(n_obs)), int(np.sqrt(n_obs))
+    coords = []
+
+    km_per_degree = 2*np.pi*radius_earth/360
+    dy_4km_in_degree = 4000/km_per_degree
+    #print(dy_4km_in_degree)
+
+    for iy in range(int(-ny/2), int(ny/2+1)):
+        for ix in range(int(-nx/2), int(nx/2+1)):
+
+            lat = lat0 + iy*dy_4km_in_degree
+            km_per_degree_x = 2*np.pi*radius_earth*np.sin(lat/180*np.pi)/360
+            dx_4km_in_degree = 4000/km_per_degree_x
+            lon = lon0 + ix*dx_4km_in_degree
+
+            coords.append((lat, lon))
+
+    if False:
+        import pickle
+        with open('obs_coords.pkl', 'wb') as f:
+            pickle.dump(coords, f)
+
+    sun_az = str(get_azimuth(lat0, lon0, time_dt))
+    sun_zen = str(90. - get_altitude(lat0, lon0, time_dt))
+    print('sunzen', sun_zen, 'sunazi', sun_az)
+
+    days_since_010120 = 145731
+    ref_day = dt.datetime(2000,1,1, tzinfo=dt.timezone.utc)
+    dart_date_day = str((time_dt - ref_day).days + days_since_010120)
+    secs_thatday = str(int((time_dt - round_to_day(time_dt)).total_seconds()))
+    print('secs, days:', secs_thatday, dart_date_day)
+
+    msg = """
+ obs_sequence
+obs_kind_definitions
+           1
+         256 MSG_4_SEVIRI_BDRF
+  num_copies:            0  num_qc:            0
+  num_obs:            """+n_obs_str+"  max_num_obs:            "+n_obs_str+"""
+  first:            1  last:            """+n_obs_str
+
+    for i_obs in range(1, int(n_obs)+1):
+        # data
+
+        lon = coords[i_obs][1]
+        lat = coords[i_obs][0]
+
+        lon_rad = str(degr_to_rad(lon))
+        lat_rad = str(degr_to_rad(lat))
+
+
+        # compile text
+        if i_obs < int(n_obs):
+            msg += """
+ OBS            """+str(i_obs)+"""
+          -1           """+str(i_obs+1)+"""          -1
+obdef
+loc3d
+     """+lon_rad+"""        """+lat_rad+"""        -888888.0000000000     -2
+kind
+         256
+ visir
+   """+sat_az+"""        """+sat_zen+"""        """+sun_az+"""
+   """+sun_zen+"""
+          12           4          21           1
+  -888888.000000000
+           1
+ """+secs_thatday+"""     """+dart_date_day+"""
+  """+error_variance
+        if i_obs == int(n_obs):  # last_observation
+            # compile text
+            msg += """
+ OBS            """+str(i_obs)+"""
+          """+str(i_obs-1)+"""           -1          -1
+obdef
+loc3d
+     """+lon_rad+"""        """+lat_rad+"""        -888888.0000000000     -2
+kind
+         256
+ visir
+   """+sat_az+"""        """+sat_zen+"""        """+sun_az+"""
+   """+sun_zen+"""
+          12           4          21           1
+  -888888.000000000
+           1
+ """+secs_thatday+"""     """+dart_date_day+"""
+  """+error_variance
+
+    #print(msg)
+    os.remove(output_path+'/obs_seq.in')
+    with open(output_path+'/obs_seq.in', 'w') as f:
+        f.write(msg)
+
+if __name__ == '__main__':
+    time_dt = dt.datetime(2008, 7, 30, 15, 30, tzinfo=dt.timezone.utc)
+    n_obs = 100
+    error_variance = str(.001)
+    run(time_dt, n_obs, error_variance)

scripts/pre_assim.py

+
+import os, sys, shutil
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_scp_srvx8, copy
+
+assim_time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+background_init_time = dt.datetime.strptime(sys.argv[2], '%Y-%m-%d_%H:%M')
+
+#if cluster.name != 'srvx8':
+#    copy = copy_scp_srvx8  # use scp
+
+print('prepare prior state estimate')
+for iens in range(1, exp.n_ens+1):
+    #wrfout_run = cluster.wrf_rundir(iens) + time.strftime('/wrfout_d01_%Y-%m-%d_%H:%M:%S')
+
+    print('link wrfout file to DART background file')
+    wrfout_run = cluster.archivedir()+background_init_time.strftime('/%Y-%m-%d_%H:%M/')  \
+                 +str(iens)+'/'+assim_time.strftime('/wrfout_d01_%Y-%m-%d_%H:%M:%S')
+    dart_ensdir = cluster.dartrundir+'/advance_temp'+str(iens)
+    wrfout_dart = dart_ensdir+'/wrfout_d01'
+    
+    os.makedirs(dart_ensdir, exist_ok=True)
+    print('linking', wrfout_run, 'to', wrfout_dart)
+    symlink(wrfout_run, wrfout_dart)
+    symlink(wrfout_dart, dart_ensdir+'/wrfinput_d01')
+
+fpath = cluster.dartrundir+'/input_list.txt'
+print('writing', fpath)
+os.remove(fpath)
+with open(fpath, 'w') as f:
+    for iens in range(1, exp.n_ens+1):
+        f.write('./advance_temp'+str(iens)+'/wrfout_d01')
+        f.write('\n')
+
+fpath = cluster.dartrundir+'/output_list.txt'
+print('writing', fpath)
+os.remove(fpath)
+with open(fpath, 'w') as f:
+    for iens in range(1, exp.n_ens+1):
+        f.write('./filter_restart_d01.'+str(iens).zfill(4))
+        f.write('\n')
+
+
+print('removing preassim and filter_restart')
+os.system('rm -rf '+cluster.dartrundir+'/preassim_*')
+os.system('rm -rf '+cluster.dartrundir+'/filter_restart*')
+os.system('rm -rf '+cluster.dartrundir+'/output_mean*')
+os.system('rm -rf '+cluster.dartrundir+'/output_sd*')
+os.system('rm -rf '+cluster.dartrundir+'/perfect_output_*')
+

scripts/pre_gen_synth_obs.py

+
+import os, sys, shutil
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy, sed_inplace, append_file
+
+time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+
+# ensure correct input.nml
+copy(cluster.scriptsdir+'/templates/input.nml',
+     cluster.dartrundir+'/input.nml')
+sed_inplace(cluster.dartrundir+'/input.nml', '<n_ens>', str(int(exp.n_ens)))
+append_file(cluster.dartrundir+'/input.nml', cluster.scriptsdir+'/templates/obs_def_rttov.VIS.nml')
+
+# prepare observation file
+import create_obs_sat
+create_obs_sat.run(time, exp.n_obs, exp.error_variance, output_path=cluster.dartrundir)
+
+if not os.path.exists(cluster.dartrundir+'/obs_seq.in'):
+    raise RuntimeError('obs_seq.in does not exist in '+cluster.dartrundir)
+
+# get wrfout_d01 from nature run
+shutil.copy(time.strftime(cluster.nature_wrfout),
+            cluster.dartrundir+'/wrfout_d01')
+
+import wrfout_add_geo
+wrfout_add_geo.run(cluster.dartrundir+'/geo_em.d01.nc', cluster.dartrundir+'/wrfout_d01')
+
+# prepare wrfinput_d01 for DART (for dimensions of variable)
+symlink(cluster.dartrundir+'/wrfout_d01', cluster.dartrundir+'/wrfinput_d01')

scripts/prep_osse.py

+
+import os, sys, shutil
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_contents, copy
+import prepare_namelist
+
+# archive configuration
+os.makedirs(cluster.archivedir(), exist_ok=True)
+shutil.copy(cluster.scriptsdir+'/config/clusters.py', cluster.archivedir()+'/clusters.py')
+shutil.copy(cluster.scriptsdir+'/config/cfg.py', cluster.archivedir()+'/cfg.py')
+
+for iens in range(1, exp.n_ens+1):
+    print('preparing ens', iens)
+    #input_prof=$USERDIR"/wrf_sounding/data/wrf/ens/from_uwyo/06610_2008073000_uwyo."$(printf "%.3d" $IENS)".wrfprof"
+    input_prof=cluster.userdir+"/wrf_sounding/data/wrf/ens/LMU+shear/raso.raso."+str(iens).zfill(3)+".wrfprof"
+
+    rundir = cluster.wrf_rundir(iens)
+    os.makedirs(rundir, exist_ok=True)
+    copy_contents(cluster.srcdir, rundir)
+    print('linking ideal and wrf.exe:')
+    symlink(cluster.ideal, rundir+'/ideal.exe')
+    symlink(cluster.wrfexe, rundir+'/wrf.exe')
+
+    prepare_namelist.run(cluster, iens, begin=dt.datetime(2008, 7, 30, 6, 0),
+                                        end=dt.datetime(2008, 7, 30, 6, 30))
+
+    symlink(input_prof, rundir+'/input_sounding')
+print('finished.')

scripts/prepare_dart.py

+
+import os, sys
+from config.cfg import exp, cluster
+from utils import symlink
+joinp = os.path.join
+
+dart_bin_dir = '/home/fs71386/lkugler/DART/DART_WRF_RTTOV_early_access/models/wrf/work/'
+rttov_dir = '/home/fs71386/lkugler/RTTOV/'
+
+# DART executables
+bins = ['perfect_model_obs', 'filter', 'obs_diag']
+for b in bins:
+    symlink(joinp(dart_bin_dir, b),
+            joinp(cluster.dartrundir, b))
+
+# DART RTTOV capability
+symlink('/home/fs71386/lkugler/DART/DART_WRF_RTTOV_early_access/'
+        +'observations/forward_operators/rttov_sensor_db.csv',
+        joinp(cluster.dartrundir, 'rttov_sensor_db.csv'))
+
+# Basic MSG4 Seviri
+files = ['rtcoef_rttov12/rttov9pred54L/rtcoef_msg_4_seviri.dat',
+         'rtcoef_rttov12/cldaer_visir/sccldcoef_msg_4_seviri.dat',]
+for f in files:
+    symlink(joinp(rttov_dir, f),
+            joinp(cluster.dartrundir, os.path.basename(f)))
+
+# MFASIS
+deff = True
+if deff:
+    mfasis_tbl = 'rtcoef_rttov12/mfasis_lut/rttov_mfasis_cld_msg_4_seviri_deff.H5'
+else:
+    mfasis_tbl = 'rtcoef_rttov12/mfasis_lut/rttov_mfasis_cld_msg_4_seviri_opac.H5'
+
+symlink(joinp(rttov_dir, mfasis_tbl),
+        joinp(cluster.dartrundir, 'rttov_mfasis_cld_msg_4_seviri.dat'))

scripts/prepare_namelist.py

+
+import os, sys, shutil
+import datetime as dt
+
+sys.path.append(os.getcwd())
+from config.cfg import exp, cluster
+from utils import sed_inplace, copy, symlink
+
+def run(cluster, iens, begin, end):
+    rundir = cluster.wrf_rundir(iens)
+
+    copy(cluster.namelist, rundir+'/namelist.input')
+
+    sed_inplace(rundir+'/namelist.input', '<dx>', str(int(exp.model_dx)))
+    sed_inplace(rundir+'/namelist.input', '<timestep>', str(int(exp.timestep)))
+    
+    archdir = cluster.archivedir()+begin.strftime('/%Y-%m-%d_%H:%M/'+str(iens)+'/')
+    sed_inplace(rundir+'/namelist.input', '<archivedir>', archdir)
+    os.makedirs(archdir, exist_ok=True)
+
+    # set times
+    for k, v in {'<y1>': '%Y', '<m1>': '%m', '<d1>': '%d',
+                 '<HH1>': '%H', '<MM1>': '%M'}.items():
+        sed_inplace(rundir+'/namelist.input', k, begin.strftime(v))
+    for k, v in {'<y2>': '%Y', '<m2>': '%m', '<d2>': '%d',
+                 '<HH2>': '%H', '<MM2>': '%M'}.items():
+        sed_inplace(rundir+'/namelist.input', k, end.strftime(v))
+
+
+if __name__ == '__main__':
+    begin = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+    end = dt.datetime.strptime(sys.argv[2], '%Y-%m-%d_%H:%M')
+
+    print('prepare namelists for all ens members')
+    print('begin', begin, 'end', end)
+    for iens in range(1, exp.n_ens+1):
+        run(cluster, iens, begin, end)

scripts/run_ens.jet.sh

+export SLURM_STEP_GRES=none
+
+##  $SLURM_ARRAY_TASK_ID
+echo "SLURM_ARRAY_TASK_ID:"$SLURM_ARRAY_TASK_ID
+
+EXPNAME="OSSE_v1.10_test"
+NAMELIST="namelist.input"
+
+USERDIR=/jetfs/home/lkugler/
+DATADIR=$USERDIR
+SRC_DIR=/jetfs/home/lkugler/compile/WRF/WRF-v4.2/run/
+IDEAL_EXE=/jetfs/home/lkugler/compile/bin/ideal.exe
+WRF_EXE=/jetfs/home/lkugler/compile/bin/wrf-v4.2_v1.10.dmpar.exe
+
+
+# VSC support: be careful with correct pinning !
+pinning=(0-11 12-23 24-35 36-47)
+for ((n=1; n<=4; n++))
+do
+    RUNDIR=$USERDIR/run_WRF/$EXPNAME/$IENS
+    cd $RUNDIR
+    rm -r wrfout_d01_*
+    echo 'mpirun -genv I_MPI_PIN_PROCESSOR_LIST=${pinning[$n]} -np 12 ./wrf.exe >/dev/null 2>&1'
+    mpirun -genv I_MPI_PIN_PROCESSOR_LIST=${pinning[$n]} -np 12 ./wrf.exe >/dev/null 2>&1
+    cd ../
+done
+wait

scripts/run_ens.vsc.sh

+
+module purge
+module load intel-mpi/2019.6 intel/19.1.0 netcdf-fortran/4.4.5-intel-19.0.5.281-qye4cqn zlib/1.2.11-intel-19.1.0.166-hs6m2qh  hdf5/1.10.5-intel-19.0.5.281-qyzojtm netcdf/4.7.0-intel-19.0.5.281-75t52g6
+export SLURM_STEP_GRES=none
+
+##  $SLURM_ARRAY_TASK_ID
+echo "SLURM_ARRAY_TASK_ID:"$SLURM_ARRAY_TASK_ID
+EXPNAME=<expname>
+
+USERDIR=/home/fs71386/lkugler
+pinning=(0-11 12-23 24-35 36-47)
+
+for ((n=1; n<=4; n++))
+do
+   IENS="$(((($SLURM_ARRAY_TASK_ID - 1)* 4) + $n))"
+   RUNDIR=$USERDIR/run_WRF/$EXPNAME/$IENS
+   echo "ENSEMBLE NR: "$IENS" in "$RUNDIR
+   cd $RUNDIR
+   rm -rf wrfout_d01_* rsl.out.0* 
+   echo "mpirun -genv I_MPI_PIN_PROCESSOR_LIST="${pinning[$n-1]}" -np 12 ./wrf.exe"
+   mpirun -genv I_MPI_PIN_PROCESSOR_LIST=${pinning[$n-1]} -np 12 ./wrf.exe &
+   cd ../
+done
+wait
+
+# error checking
+for ((n=1; n<=4; n++))
+do
+   IENS="$(((($SLURM_ARRAY_TASK_ID - 1)* 4) + $n))"
+   RUNDIR=$USERDIR/run_WRF/$EXPNAME/$IENS
+   cd $RUNDIR
+   line=`tail -n 1 rsl.out.0000`
+   if [[ $line == *"SUCCESS COMPLETE WRF"* ]]; 
+   then 
+      echo $RUNDIR 'SUCCESS COMPLETE WRF'
+   else  
+      echo $RUNDIR $line
+      exit 1
+   fi
+done

scripts/run_obs_diag.py

+import os, sys, shutil, glob
+from config.cfg import exp, cluster
+from utils import symlink, copy, sed_inplace, append_file
+
+folder_obs_seq_final = '/home/fs71386/lkugler/data/sim_archive/exp_v1.10_LMU+shear_filter/obs_seq_final/'
+files = sorted(glob.glob(folder_obs_seq_final+'/*.final'))  # input for obs_diag program 
+
+rundir_program = '/home/fs71386/lkugler/DART/rundir-diagnostics/'
+fpath = rundir_program+'/obsdiag_inputlist.txt'
+print('writing', fpath)
+
+if os.path.exists(fpath):
+    os.remove(fpath)
+
+with open(fpath, 'w') as f:
+    for fin in files:
+        f.write(fin)
+        f.write('\n')
+
+
+print('ensure correct input.nml')
+copy(cluster.scriptsdir+'/templates/input.nml',
+     rundir_program+'/input.nml') #cluster.dartrundir+'/input.nml')
+sed_inplace(rundir_program+'/input.nml', '<n_ens>', str(int(exp.n_ens)))
+append_file(rundir_program+'/input.nml', cluster.scriptsdir+'/templates/obs_def_rttov.VIS.nml')
+
+# run obs_diag
+print('running obs_diag program')
+os.chdir(rundir_program)
+symlink(cluster.dartrundir+'/obs_diag', rundir_program+'/obs_diag')
+os.system('./obs_diag >& obs_diag.log') 
+
+print('moving output to', cluster.archivedir()+'/obs_diag_output.nc') 
+copy(rundir_program+'/obs_diag_output.nc', cluster.archivedir()+'/obs_diag_output.nc')
+
+print('running obs_seq_to_netcdf program')
+symlink(cluster.dartrundir+'/obs_seq_to_netcdf', rundir_program+'/obs_diag_output.nc')
+os.system('./obs_seq_to_netcdf  >& obs_seq_to_netcdf.log')
+print('moving output to', cluster.archivedir()+'/obs_seq_output.nc')
+copy(rundir_program+'/obs_diag_output.nc', cluster.archivedir()+'/obs_seq_output.nc')
+
+os.system('rm obs_seq_to_netcdf obs_diag')

scripts/update_wrfinput_from_filteroutput.py

+import os, sys, warnings
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_scp_srvx8, copy, mkdir, mkdir_srvx8, clean_wrfdir
+
+time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+
+#if cluster.name != 'srvx8':
+#    copy = copy_scp_srvx8
+#    mkdir = mkdir_srvx8
+
+update_vars = ['Times', 'U', 'V', 'PH', 'T', 'MU', 'QVAPOR', 'QCLOUD', 'QICE', 'PSFC', 'TSK']
+
+print('move output to WRF dir as new initial conditions')
+for iens in range(1, exp.n_ens+1):
+    clean_wrfdir(cluster.wrf_rundir(iens))
+    filter_out = cluster.dartrundir+'/filter_restart_d01.'+str(iens).zfill(4)
+    wrf_ic = cluster.wrf_rundir(iens) + '/wrfinput_d01'
+
+    # overwrite variables in wrfinput file
+    vars = ','.join(update_vars)
+    print('updating', vars, 'in', wrf_ic)
+    os.system(cluster.ncks+' -A -v '+vars+' '+filter_out+' '+wrf_ic)
+
+    # clean up
+    #try:
+    #    os.remove(cluster.dartrundir+'/advance_temp'+str(iens)+'/wrfout_d01')
+    #except:
+    #    pass

scripts/update_wrfinput_from_wrfout.py

+
+import os, sys, shutil
+import datetime as dt
+from config.cfg import exp, cluster
+from utils import symlink, copy_scp_srvx8, copy
+import prepare_namelist
+
+time = dt.datetime.strptime(sys.argv[1], '%Y-%m-%d_%H:%M')
+
+update_vars = ['Times', 'U', 'V', 'PH', 'T', 'MU', 'QVAPOR', 'QCLOUD', 'QICE', 'PSFC', 'TSK']
+vars = ','.join(update_vars)
+
+for iens in range(1, exp.n_ens+1):
+    print('update state in wrfinput  wrfout file to DART background file')
+    wrfout = cluster.archivedir()+background_init_time.strftime('/%Y-%m-%d_%H:%M/')  \
+                 +str(iens)+'/'+time.strftime('/wrfout_d01_%Y-%m-%d_%H:%M:%S')
+    wrfin = cluster.wrf_rundir(iens)+'/wrfinput_d01'
+
+    print('updating', wrfin, 'to state in', wrfout)
+    # overwrite variables in wrfinput file
+    os.system(cluster.ncks+' -A -v '+vars+' '+wrfout+' '+wrfin)

scripts/utils.py

+
+import os, sys, shutil, glob
+copy = shutil.copy
+
+def mkdir(path):
+    os.system('mkdir -p '+path)
+
+def mkdir_srvx8(path):
+    os.system('ssh a1254888@srvx8.img.univie.ac.at mkdir -p '+path)
+
+def script_to_str(path):
+    return open(path, 'r').read()
+
+def copy_contents(src, dst):
+    os.system('cp -rf '+src+'/* '+dst+'/')
+
+def clean_wrfdir(dir):
+    for s in ['wrfout_*', 'rsl.*', 'wrfrst_*']:
+        for f in glob.glob(dir+'/'+s):
+            os.remove(f)
+
+def symlink(src, dst):
+    try:
+        os.remove(dst)
+    except Exception as e:
+        pass
+    os.symlink(src, dst)
+
+def copy_scp_srvx8(src, dst):
+    os.system('scp '+src+' a1254888@srvx8.img.univie.ac.at:'+dst)
+
+def sed_inplace(filename, pattern, repl):
+    import re, tempfile
+    '''
+    Perform the pure-Python equivalent of in-place `sed` substitution: e.g.,
+    `sed -i -e 's/'${pattern}'/'${repl}' "${filename}"`.
+    '''
+    # For efficiency, precompile the passed regular expression.
+    pattern_compiled = re.compile(pattern)
+
+    # For portability, NamedTemporaryFile() defaults to mode "w+b" (i.e., binary
+    # writing with updating). This is usually a good thing. In this case,
+    # however, binary writing imposes non-trivial encoding constraints trivially
+    # resolved by switching to text writing. Let's do that.
+    with tempfile.NamedTemporaryFile(mode='w', delete=False) as tmp_file:
+        with open(filename) as src_file:
+            for line in src_file:
+                tmp_file.write(pattern_compiled.sub(repl, line))
+
+    # Overwrite the original file with the munged temporary file in a
+    # manner preserving file attributes (e.g., permissions).
+    shutil.copystat(filename, tmp_file.name)
+    shutil.move(tmp_file.name, filename)
+
+def append_file(f_main, f_gets_appended):
+    os.system('cat '+f_gets_appended+' >> '+f_main)
+
+

scripts/wrfinput_add_geo.py

+#!/home/fs71386/lkugler/miniconda3/envs/DART/bin/python
+#!/jetfs/home/lkugler/miniconda3/envs/DART/bin/python
+
+"""Add geogrid data to wrfinput
+this is needed for DART, but not provided by ideal.exe
+
+take LAT,LON, mapfac from geogrid, so that they are consistent.
+do not change E, F, HGT_M as they would alter the dynamics and have no impact on assimilation
+
+example call:
+    ./wrfinput_add_geo.py geo_em.d01.nc wrfinput_d01
+
+"""
+import os, sys
+import netCDF4 as nc
+from config.cfg import exp, cluster
+
+def run(geo_data_file, wrfinput_file):
+    geo_ds = nc.Dataset(geo_data_file, 'r')
+    wrfinp_ds = nc.Dataset(wrfinput_file, 'r+')
+
+    fields_old = ["XLAT_M",   "XLONG_M",      "CLAT",
+                  "MAPFAC_M",   "MAPFAC_U",  "MAPFAC_V",
+                    "MAPFAC_MX", "MAPFAC_MY", "MAPFAC_UX", "MAPFAC_UY",
+                    "MAPFAC_VX", "MAPFAC_VY",  "SINALPHA",  "COSALPHA",
+                    "XLONG_U",  "XLONG_V",     "XLAT_U",    "XLAT_V"]
+    fields_new = ["XLAT",     "XLONG",      "CLAT",
+                   "MAPFAC_M",  "MAPFAC_U",  "MAPFAC_V",
+                    "MAPFAC_MX", "MAPFAC_MY", "MAPFAC_UX", "MAPFAC_UY",
+                    "MAPFAC_VX", "MAPFAC_VY",  "SINALPHA",  "COSALPHA",
+                    "XLONG_U",  "XLONG_V",     "XLAT_U",    "XLAT_V"]
+
+    for old, new in zip(fields_old, fields_new):
+        print('moving old field', old, 'into new field', new)
+        print(geo_ds.variables[old][:].shape, wrfinp_ds.variables[new][:].shape)
+        wrfinp_ds.variables[new][:] = geo_ds.variables[old][:]
+        print(wrfinp_ds.variables[new][:])
+
+    wrfinp_ds.close()
+    geo_ds.close()
+
+    # overwrite attributes
+    os.system(cluster.ncks+' -A -x '+geo_data_file+' '+wrfinput_file)
+
+
+if __name__ == '__main__':
+    geo_data_file = sys.argv[1]  # '/home/fs71386/lkugler/compile_WRF/WPS-release-v4.2/geo_em.d01.nc'
+    wrfinput_file = sys.argv[2]  # '/home/fs71386/lkugler/DART/wrfinput_d01'
+    run(geo_data_file, wrfinput_file)