Skip to content
Snippets Groups Projects
Select Git revision
  • d143246f940651a90f87bdeb7c9c22f07ef01342
  • master default protected
  • kuglerl93-master-patch-83288
  • fritzm97-master-patch-31221
  • 03-2025
5 results

Data.md

Blame
  • Data at ECMWF

    ECMWF's Meteorological Archival and Retrieval System (MARS) enables you to explore and retrieve meteorological data in GRIB or NetCDF format. GRIB (General Regularly distributed Information in Binary form) is the WMO's format for binary gridded data and is designed for storing and distributing weather data. GRIB files are widely used in meteorological applications. NetCDF (Network Common Data Form) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

    Meteorological Archival and Retrieval System (MARS)

    MARS request tree

    How to make a MARS request?

    What ECMWF says: here

    General Regularly distributed Information in Binary form (GRIB)

    GRIB is a binary format, and the data is packed to increase storage efficiency. GRIB messages are often concatenated together to form a GRIB file. GRIB files usually have the extension .grib, .grb or .gb.

    Currently there are two different coding standards: GRIB edition 1 (commonly referred to as GRIB1) and GRIB edition 2 (GRIB2). The major differences are in the structure of the messages; in GRIB2, several variables are defined with more precision (e.g. in GRIB1, latitudes and longitudes are in milli-degrees while in GRIB2, they are in micro-degrees). Also in GRIB2, longitude values must lie between 0 and 360 degrees), the encoding of the parameter is very different, and in GRIB2 the description of the data is template/table based. Note that a GRIB file can contain a mix of GRIB1 and GRIB2 messages.

    How to read GRIB files?

    ECMWF provides and supports ecCodes. This software package has an Application Program Interface which makes ECMWF GRIB1 and GRIB2 files accessible from C, FORTRAN and Python programmes. ecCodes also provides a useful set of command line tools to give quick access to GRIB messages within the files.

    Commonly used programs that can handle grib files:

    • Panoply by NASA
    • Metview by ECMWF
    • ecCodes by ECMWF
    • magics / metview by ECMWF
    • CDO
    • NCO
    • GrADS
    • idl

    We have some examples of reading grib files:

    it is also possible to convert grib files into netcdf files:

    # load eccodes module or make grib_to_netcdf available to our PATH
    module load eccodes
    # convert to netcdf
    grib_to_netcdf input.grb -o output.nc

    grib_to_netcdf only works correctly when a single level coordinate is present in the GRIB file. Often the model output files have fields on multiple level types (ie. hybrid model levels and pressure levels).

    # check the contents of the grib file
    grib_ls input.grb
    # split by model lvel type
    grib_copy input.grb output_[typeOfLevel].grb

    more information can be found at ECMWF

    Example of an efficient MARS request

    It is not easy to write good MARS requests, because there are so many parameters.

    A few things are important:

    1. NetCDF does not handle well different times and steps.
    2. Retrieving should loop by experiment, date, time and retrieve as much as possible
    3. Check the catalogue first and do a "estimate download size" check. Look for the number of tapes. If it is one tape, then you are fine.

    MARS keywords HRES Guide

    #!/bin/bash
    
    # this example will filter the area of Europe (N/W/S/E) and interpolate the final fields to
    # a 0.5x0.5 regular lat-lon grid (GRID=0.5/0.5)
    AREA="73.5/-27/33/45"
    GRID="0.5/0.5"
    
    # fixed selection from the same block
    PARAMS="130/131/132"
    LEVELIST="127/128/129/130/131/132/133/134/135/136/137"
    STEP="0/to/90/by/1"
    
    TIMES="0000 1200"
    YEAR="2017"
    MONTH="04"
    
    #date loop
    for y in ${YEAR}; do
        for m in ${MONTH}; do
            #get the number of days for this particular month/year
            days_per_month=$(cal ${m} ${y} | awk 'NF {DAYS = $NF}; END {print DAYS}')
    
            for my_date in $(seq -w 1 ${days_per_month}); do
                my_date=${YEAR}${m}${my_date}
    
                #time loop
                for my_time in ${TIMES}; do
                    cat << EOF > my_request_${my_date}_${my_time}.mars
    RETRIEVE,
        CLASS      = OD,
        TYPE       = FC,
        STREAM     = OPER,
        EXPVER     = 0001,
        LEVTYPE    = ML,
        GRID       = ${GRID},
        AREA       = ${AREA},
        LEVELIST   = ${LEVELIST},
        PARAM      = ${PARAMS},
        DATE       = ${my_date},
        TIME       = ${my_time},
        STEP       = ${STEP},
        TARGET     = "oper_ml_${my_date}_${my_time}.grib"
    EOF
    
                # request all times together
                mars my_request_${my_date}_${my_time}.mars
                if [ $? -eq 0 ]; then
                    rm -f my_request_${my_date}_${my_time}.mars
                fi
                done
            done
        done
    done