flex_extract fails with gribapi.errors.ConceptNoMatchError; ECCODES ERROR; no match for paramId=201031 (specific cloud water content) when trying to download ERA5

after not using flex_extract for a while, it now fails with the error below. No change to flex_extract setup. Tried to download same files I already downloaded a few months ago. Probably related to a changes to ERA5 files?

Error occurs when flex_extract is summing cloud liquid water and ice. Also see corresponding discussion here: https://forum.ecmwf.int/t/eccodes-error-concept-no-match-for-paramid-201031-when-using-flex-extract-to-prepare-era5/7240/2

Error message:

ECCODES ERROR   :  concept: no match for paramId=201031
ECCODES ERROR   :  concept: input handle edition=2, centre=ecmf
ECCODES ERROR   :  concept: input handle dataset=era
ECCODES ERROR   :  Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=201031'

Traceback (most recent call last):
  File "/home/km4f/flex_extract_v7.1.3/Source/Python/submit.py", line 268, in <module>
    main()
  File "/home/km4f/flex_extract_v7.1.3/Source/Python/submit.py", line 109, in main
    prepare_flexpart(ppid, c)
  File "/etc/ecmwf/nfs/dh1_home_b/km4f/flex_extract_v7.1.3/Source/Python/Mods/prepare_flexpart.py", line 169, in prepare_flexpart
    flexpart.create(inputfiles, c)
  File "/etc/ecmwf/nfs/dh1_home_b/km4f/flex_extract_v7.1.3/Source/Python/Classes/EcFlexpart.py", line 1655, in create
    codes_set(gid, 'paramId', 201031)
  File "/usr/local/apps/python3/3.11.10-01/lib/python3.11/site-packages/gribapi/gribapi.py", line 2146, in grib_set
    grib_set_long(msgid, key, value)
  File "/usr/local/apps/python3/3.11.10-01/lib/python3.11/site-packages/gribapi/gribapi.py", line 1012, in grib_set_long
    GRIB_CHECK(lib.grib_set_long(h, key.encode(ENC), value))
  File "/usr/local/apps/python3/3.11.10-01/lib/python3.11/site-packages/gribapi/gribapi.py", line 232, in GRIB_CHECK
    errors.raise_grib_error(errid)
  File "/usr/local/apps/python3/3.11.10-01/lib/python3.11/site-packages/gribapi/errors.py", line 381, in raise_grib_error
    raise ERROR_MAP[errid](errid)
gribapi.errors.ConceptNoMatchError: Concept no match

changed the description

assigned to @placha84

Found the bug in Source/Python/Classes/EcFlexpart.py

codes_set(gid, 'paramId', 201031)

There is no such code in the ECMWF reanalysis table. Write clwc and ciwc separately. FLEXPART can read it.

Create a new branch: cloud_water_contents

Running final test with FLEXPART. Will merge it back.

there is one catch, the files are larger because of two instead of one variable... it seems that the split into different cloud water/ice content is only calculated due to temperature. Need to check that. Flexpart splits it again by temperature into water and ice.

@duetschm75 reported that it works with this fix.

mentioned in issue #6 (closed)

This is related to the recent update of eccodes to version 2.38.0 or later.

I can confirm that using eccodes 2.35.0 does not have this issue. See closed issue #6 (closed) . A fix will be submitted.

closed

I opened a ticket on ECMWF support site to get a better understanding on how to deal with this issue in the future. We will wnat to have ERA6 data as well and can not be limited by this little bug....

I have been in conteact with ECMWF's Support and the solution is to use the WMO definition:

codes_set(gid, 'paramId', 246) 

this is the clwc parameter.

According to support, all local ECMWF encodings will be deprecated soon. Need to check the other parameters too.