added mars requests and era5 plotting

ECMWF/Data.md

 # Data at ECMWF
 
-## MARS
+ECMWF's Meteorological Archival and Retrieval System (MARS) enables you to explore and retrieve meteorological data in GRIB or NetCDF format. GRIB (General Regularly distributed Information in Binary form) is the WMO's format for binary gridded data and is designed for storing and distributing weather data. GRIB files are widely used in meteorological applications. NetCDF (Network Common Data Form) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
 
+## Meteorological Archival and Retrieval System (MARS)
 
 ![MARS request tree](../mkdocs/img/MARS-tree.png)
 
+How to make a MARS request?
 
-How to make a mars request
+What ECMWF says: [here](https://www.ecmwf.int/en/forecasts/access-forecasts/access-archive-datasets)
 
+- using ECMWF's servers
+- using [MARS web api](../Python/QA-012-Mars-Requests.ipynb) in Python
+- [using ECMWF mars web interface](https://apps.ecmwf.int/archive-catalogue/?class=od) using the archive catalogue.
 
-## GRIB
 
-How to handle grib files.
+## General Regularly distributed Information in Binary form (GRIB)
 
-Example using Python and Fortran90.
\ No newline at end of file
+GRIB is a binary format, and the data is packed to increase storage efficiency. GRIB messages are often concatenated together to form a GRIB file. GRIB files usually have the extension .grib, .grb or .gb.
+
+Currently there are two different coding standards: GRIB edition 1 (commonly referred to as GRIB1) and GRIB edition 2 (GRIB2). The major differences are in the structure of the messages; in GRIB2, several variables are defined with more precision (e.g. in GRIB1, latitudes and longitudes are in milli-degrees while in GRIB2, they are in micro-degrees). Also in GRIB2, longitude values must lie between 0 and 360 degrees), the encoding of the parameter is very different, and in GRIB2 the description of the data is template/table based. Note that a GRIB file can contain a mix of GRIB1 and GRIB2 messages.
+
+### How to read GRIB files?
+
+ECMWF provides and supports ecCodes. This software package has an Application Program Interface which makes ECMWF GRIB1 and GRIB2 files accessible from C, FORTRAN and Python programmes. ecCodes also provides a useful set of command line tools to give quick access to GRIB messages within the files.
+
+Commonly used programs that can handle grib files:
+
+- [Panoply](http://www.giss.nasa.gov/tools/panoply/) by NASA
+- [Metview](https://confluence.ecmwf.int/display/METV) by ECMWF
+- ecCodes by ECMWF
+- magics / metview by ECMWF
+- CDO
+- NCO
+- GrADS
+- idl
+
+We have some examples of reading grib files:
+
+- [Reading grib using cfgrib into xarray](../Python/QA-012-Mars-Requests.ipynb)
+- [Reading grib using magics](../Python/QA-013-Retrieving-ERA5-Magics.ipynb)
+
+it is also possible to convert grib files into netcdf files:
+
+```sh title="Convert GRIB to NetCDF"
+# load eccodes module or make grib_to_netcdf available to our PATH
+module load eccodes
+# convert to netcdf
+grib_to_netcdf input.grb -o output.nc
+```
+
+`grib_to_netcdf` only works correctly when a single level coordinate is present in the GRIB file. Often the model output files have fields on multiple level types (ie. hybrid model levels and pressure levels).
+
+```sh title="Split Model level types"
+# check the contents of the grib file
+grib_ls input.grb
+# split by model lvel type
+grib_copy input.grb output_[typeOfLevel].grb
+```
+more information can be found at [ECMWF](https://confluence.ecmwf.int/display/OIFS/How+to+convert+GRIB+to+netCDF)
\ No newline at end of file

Python/QA-012-Mars-Requests.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7d86c6c1-140e-4c0c-9cbb-98355f11da04",
+   "metadata": {},
+   "source": [
+    "# Access to MARS\n",
+    "\n",
+    "This Service will allow authorised users to retrieve and list MARS data from outside the ECMWF facilities. Users within ECMWF Member and Co-operating States may contact their Computing Representative to obtain access to MARS. All other users may request a username and password, following these [instructions](https://confluence.ecmwf.int/display/WEBAPI/Access+MARS#AccessMARS-downloadmars).\n",
+    "\n",
+    "Accessing and retrieving files from MARS got a lot easier, with this package.\n",
+    "\n",
+    "We need to install the web client for MARS and get credentials from ECMWf to allow the API to call MARS."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "543bfff5-ce4e-4bcc-af5f-5013a0470f9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip -q install --user ecmwf-api-client"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1257ed6b-f5fc-4657-b4b9-43eb28f13d6c",
+   "metadata": {},
+   "source": [
+    "You need an ECMWF account and login to get the login credentials for the web api of MARS.\n",
+    "Go to https://api.ecmwf.int/v1/key ( click login ) and then same the information to a file called `$HOME/.ecmwfapirc`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "0fce3a30-dca5-40e1-94ac-f7189bc92154",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting /mnt/users/staff/mblaschek/.ecmwfapirc\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile /mnt/users/staff/mmustermann/.ecmwfapirc\n",
+    "{\n",
+    "    \"url\"   : \"https://api.ecmwf.int/v1\",\n",
+    "    \"key\"   : \"1234567890???????\",\n",
+    "    \"email\" : \"max.mustermann@univie.ac.at\"\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2dddf15a-d542-4e03-9155-2dcb57b9637e",
+   "metadata": {},
+   "source": [
+    "This is an example of a request. If you want to figure more exactly what you need it is best to browse the online catalog of MARS and then get exactly what you want.\n",
+    "\n",
+    "Look here: [Mars Catalogue](https://apps.ecmwf.int/mars-catalogue/) or the [Archive Catalogue (freely)](http://apps.ecmwf.int/archive-catalogue/)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ba4b42e5-5218-4948-ab43-c64c3606734c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2024-02-29 15:49:59 ECMWF API python library 1.6.3\n",
+      "2024-02-29 15:49:59 ECMWF API at https://api.ecmwf.int/v1\n",
+      "2024-02-29 15:49:59 Welcome Max Mustermann\n",
+      "2024-02-29 15:50:00 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int\n",
+      "2024-02-29 15:50:00 Request submitted\n",
+      "2024-02-29 15:50:00 Request id: 65e09998778174dbb7b3479f\n",
+      "2024-02-29 15:50:00 Request is submitted\n",
+      "2024-02-29 15:50:01 Request is queued\n",
+      "2024-02-29 15:50:07 Calling 'nice mars /tmp/20240229-1450/ff/tmp-_mars-BMsk2q.req'\n",
+      "2024-02-29 15:50:07 Forcing MIR_CACHE_PATH=/data/ec_coeff\n",
+      "2024-02-29 15:50:07 mars - WARN -\n",
+      "2024-02-29 15:50:07 mars - WARN -\n",
+      "2024-02-29 15:50:07 MIR environment variables:\n",
+      "2024-02-29 15:50:07 MIR_CACHE_PATH=/data/ec_coeff\n",
+      "2024-02-29 15:50:07 MIR_LSM_NAMED=1km.climate.v013\n",
+      "2024-02-29 15:50:07 Using MARS binary: /usr/local/apps/mars/versions/6.33.16.2/bin/mars.bin\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Welcome to MARS\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - MARS Client build stamp: 20231005152332\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - MARS Client bundle version: 6.33.16.2\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package mars-client version: 6.33.16\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package mir version: 1.18.0\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package odc version: 1.4.6\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package fdb version: 5.11.23\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package metkit version: 1.10.15\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package eckit version: 1.24.4\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - package eccodes version: 2.32.0\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Maximum retrieval size is 75.00 G\n",
+      "2024-02-29 15:50:07 retrieve,stream=oper,levtype=sfc,param=167.128,padding=0,step=0/to/240/by/12,expver=1,time=00,date=20150101,type=fc,class=odmars - INFO   - 20240229.145002 - Automatic split on dates is on\n",
+      "2024-02-29 15:50:07 \n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Processing request 1\n",
+      "2024-02-29 15:50:07 \n",
+      "2024-02-29 15:50:07 RETRIEVE,\n",
+      "2024-02-29 15:50:07     CLASS      = OD,\n",
+      "2024-02-29 15:50:07     TYPE       = FC,\n",
+      "2024-02-29 15:50:07     STREAM     = OPER,\n",
+      "2024-02-29 15:50:07     EXPVER     = 0001,\n",
+      "2024-02-29 15:50:07     REPRES     = GG,\n",
+      "2024-02-29 15:50:07     LEVTYPE    = SFC,\n",
+      "2024-02-29 15:50:07     PARAM      = 167.128,\n",
+      "2024-02-29 15:50:07     TIME       = 0000,\n",
+      "2024-02-29 15:50:07     STEP       = 0/12/24/36/48/60/72/84/96/108/120/132/144/156/168/180/192/204/216/228/240,\n",
+      "2024-02-29 15:50:07     DOMAIN     = G,\n",
+      "2024-02-29 15:50:07     PADDING    = 0,\n",
+      "2024-02-29 15:50:07     DATE       = 20150101\n",
+      "2024-02-29 15:50:07 \n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Web API request id: 65e09998778174dbb7b3479f\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Requesting 21 fields\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Setting SO_SNDBUF to 33554432 (32.00 M)\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Current value is 8192 (8.00 K)\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Setting SO_RCVBUF to 33554432 (32.00 M)\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Current value is 43690 (42.67 K)\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Calling mars on 'fdbprod', local port is 57169\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Server task is 447 [ATOS FDB]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Retrieving from FDB [ATOS FDB]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Looking up FDB indexes: 0.00488 second elapsed, 0.003801 second cpu [ATOS FDB]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Calling mars on 'marsod-core', local port is 52683\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Server task is 390 [marsod]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Request cost: 21 fields, 85.7978 Mbytes online, nodes: mvr004 mvr006 [marsod]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - The efficiency of your requests in the last 12 hours is 100% [marsod]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145002 - Transfering 89965512 bytes\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 - 21 fields retrieved from 'marsod'\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 - Request time:  wall: 2 sec\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 -   Visiting marsod: wall: 2 sec\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 -   Read from network: 85.80 Mbyte(s) in 2 sec  [53.45 Mbyte/sec]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 -   Writing to target file: 85.80 Mbyte(s) in < 1 sec [220.00 Mbyte/sec]\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 - Memory used: 40.92 Mbyte(s)\n",
+      "2024-02-29 15:50:07 mars - INFO   - 20240229.145004 - No errors reported\n",
+      "2024-02-29 15:50:07 Process '['nice', 'mars', '/tmp/20240229-1450/ff/tmp-_mars-BMsk2q.req']' finished\n",
+      "2024-02-29 15:50:07 Request is complete\n",
+      "2024-02-29 15:50:07 Transfering 85.7978 Mbytes into target.grib\n",
+      "2024-02-29 15:50:07 From https://apps.ecmwf.int/api/streaming/private/blue/02/20240229-1450/d9/_mars-bol-webmars-private-svc-blue-003-4a73a881a8d5eead47db9eff2f9935a4-rDzHit.grib\n",
+      "2024-02-29 15:50:13 Transfer rate 14.1941 Mbytes/s\n",
+      "2024-02-29 15:50:13 Done\n"
+     ]
+    }
+   ],
+   "source": [
+    "from ecmwfapi import ECMWFService\n",
+    "  \n",
+    "server = ECMWFService(\"mars\")\n",
+    "server.execute(\n",
+    "    {\n",
+    "    \"class\": \"od\",\n",
+    "    \"date\": \"20150101\",\n",
+    "    \"expver\": \"1\",\n",
+    "    \"levtype\": \"sfc\",\n",
+    "    \"param\": \"167.128\",\n",
+    "    \"step\": \"0/to/240/by/12\",\n",
+    "    \"stream\": \"oper\",\n",
+    "    \"time\": \"00\",\n",
+    "    \"type\": \"fc\"\n",
+    "    },\n",
+    "    \"target.grib\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "25cd62c4-a3da-4411-bec0-c431763571d2",
+   "metadata": {},
+   "source": [
+    "## Reading the grib file\n",
+    "\n",
+    "we can use the [cfgrib](https://github.com/ecmwf/cfgrib) package, which allows xarray to read grib files."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "38ba97fe-3fb8-4c36-a802-0282fcc7c860",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip -q install --user cfgrib"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "3debef90-b0e0-4ac2-a519-a89323f3652b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import xarray as xr\n",
+    "ds = xr.open_dataset('target.grib', engine='cfgrib')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8b1bc0e1-3ec3-4e62-b9e4-b9279435b508",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "<xarray.Dataset>\n",
+      "Dimensions:     (step: 21, values: 2140702)\n",
+      "Coordinates:\n",
+      "    number      int64 ...\n",
+      "    time        datetime64[ns] ...\n",
+      "  * step        (step) timedelta64[ns] 0 days 00:00:00 ... 10 days 00:00:00\n",
+      "    surface     float64 ...\n",
+      "    latitude    (values) float64 ...\n",
+      "    longitude   (values) float64 ...\n",
+      "    valid_time  (step) datetime64[ns] ...\n",
+      "Dimensions without coordinates: values\n",
+      "Data variables:\n",
+      "    t2m         (step, values) float32 ...\n",
+      "Attributes:\n",
+      "    GRIB_edition:            1\n",
+      "    GRIB_centre:             ecmf\n",
+      "    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts\n",
+      "    GRIB_subCentre:          0\n",
+      "    Conventions:             CF-1.7\n",
+      "    institution:             European Centre for Medium-Range Weather Forecasts\n",
+      "    history:                 2024-02-29T15:55 GRIB to CDM+CF via cfgrib-0.9.1...\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(ds)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a207cddd-ddba-4bc5-a888-33e66b721006",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "nwp 2023.1 - 3.10",
+   "language": "python",
+   "name": "nwp2023.1"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

Python/README.md

@@ -145,6 +145,15 @@ ECMWF created a python package that helps to read BUFR messages via ECCODES into
 
 ECMWF created a python package that helps to read ODB messages via ODC or pure Python into pandas DataFrame.
 
+## Q: How to retrieve data from MARS in python?
+[MARS](QA-012-Mars-Requests.ipynb)
+
+ECMWF has created a package that allows to retrieve data from MARS, easily.
+
+## Q: How to retrieve ERA5 from CDS and plot in Magics?
+[Magics](QA-013-Retrieving-ERA5-Magics.ipynb)
+
+It is very easy to download and plot ERA5 data, using cdsapi and Magics.
 
 ## Q: How to ignore user site packages?