Skip to content
Snippets Groups Projects

Issues with running flex_extract

    • Closed Issue created by ke*****@u*****.c** via GitLab Support Bot @support-bot

    Hello,

    I am having trouble getting flex_extract to run properly. In following the online guide, I have successfully been able to install flex_extract to my local computer. However, I have been having trouble executing the command run_local.sh

    Here is some information:

    -The control file I am testing is CONTROL_EA5

    -The script run_local.sh has been modified to the following:

    QUEUE=''
START_DATE=None
END_DATE=None
DATE_CHUNK=None
JOB_CHUNK=None
BASETIME=None
STEP=None
LEVELIST=None
AREA=None
INPUTDIR='./Workspace/EA5'
OUTPUTDIR=None
PP_ID=None
JOB_TEMPLATE=''
CONTROLFILE='CONTROL_EA5'
DEBUG=0
REQUEST=0
PUBLIC=1

After typing $ ./run_local.sh, I get the following:

Retrieving ECMWF data!
start date 20180809
end date 20180809
Using ECMWF WebAPI: False
Using CDS API: False
... removing old files in /usr/local/src/flex_extract/Run/./Workspace/EA5
... retrieve 20180808/to/20180810 in dir /usr/local/src/flex_extract/Run/./Workspace/EA5
marsclass: EA
dataset: None
type: FC
levtype: SFC
levelist: 1
repres:
date: 20180808/to/20180810
resol: 799
stream: OPER
area: 74.0/-24.0/9.875/60.375
time: 06/18
step: 1/to/12/by/1
expver: 1
number: OFF
accuracy: 24
grid: 0.28125/0.28125
gaussian:
target: /usr/local/src/flex_extract/Run/./Workspace/EA5/FCOG_acc_SL.20180808.3308.3309.grb
param: 142.128/143.128/146.128/180.128/181.128/176.128
target: /usr/local/src/flex_extract/Run/./Workspace/EA5/FCOG_acc_SL.20180808.3308.3309.grb
/usr/lib/python3.10/subprocess.py:955: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdin = io.open(p2cwrite, 'wb', bufsize)
/usr/lib/python3.10/subprocess.py:961: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
/usr/lib/python3.10/subprocess.py:966: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stderr = io.open(errread, 'rb', bufsize)
MARS request failed

  File "/usr/local/src/flex_extract/Run/../Source/Python/submit.py", line 268, in <module>
    main()

  File "/usr/local/src/flex_extract/Run/../Source/Python/submit.py", line 107, in main
    get_mars_data(c)

  File "/usr/local/src/flex_extract/Source/Python/Mods/get_mars_data.py", line 158, in get_mars_data
    do_retrievement(c, server, start, end, datechunk, fluxes=True)

  File "/usr/local/src/flex_extract/Source/Python/Mods/get_mars_data.py", line 386, in do_retrievement
    my_error('MARS request failed')

  File "/usr/local/src/flex_extract/Source/Python/Mods/tools.py", line 335, in my_error
    trace = '\n'.join(traceback.format_stack())

    I have been struggling to pinpoint what the exact problem is. If you have any thoughts or suggestions, I would greatly appreciate them.

    Thank you very much,

    Kevin Bloxam (He/Him) Postdoctoral Fellow Department of Physics | Faculty of Arts and Science | University of Toronto kevin.bloxam@utoronto.ca

    Edited

    Designs

      Child items ...

      2. Activity

      • Michael Blaschek assigned to @blaschm7

        assigned to @blaschm7

      • Marina Dütsch assigned to @blaschm7

        assigned to @blaschm7

      • Michael Blaschek changed the description

        changed the description

      • Michael Blaschek made the issue visible to everyone

        made the issue visible to everyone

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        Sorry for the long wait. I'm not really an flex extract expert, but I will try to help you. I'm using flex_extract mainly on ECMWF's servers, that's why im not too familiar with the local installation and retrieval via CDS, but that should work too.

        I will try to recreate your issue on my setup and see if that helps.

        You are using the dev branch right?

      • Michael Blaschek moved from flexpart#32 (moved)

        moved from flexpart#32 (moved)

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        What i did:

        1. get flex extract
        2. create micromamba env
        3. test download from CDS (accepting all terms!)
        4. Modify setup_local.sh with lots of None
        5. Modify Source/Fortran/makefile_local_gfortran, ECCODES_INCLUDE_DIR=/path/to/flex_extract/pyflexextract/include 
        # clone the source code
git clone --single-branch --branch dev git@gitlab.phaidra.org:flexpart/flex_extract.git
cd flex_extract
# setup micromamba plus gfortran in a local subdir
micromamba install -c conda-forge -p ./pyflexextract python==3.10 numpy genshi eccodes==2.35 ecmwf-api-client cdsapi fortran-compiler
# fix and install eccodes
pip3 install eccodes

# run the CDS Example:
python3 

import cdsapi
c = cdsapi.Client()
c.retrieve('reanalysis-era5-complete',
{
    'class'   : 'ea',
    'expver'  : '1',
    'stream'  : 'oper',
    'type'    : 'fc',
    'step'    : '3/to/12/by/3',
    'param'   : '130.128',
    'levtype' : 'ml',
    'levelist': '135/to/137',
    'date'    : '2013-01-01',
    'time'    : '06/18',
    'area'    : '50/-5/40/5',
    'grid'    : '1.0/1.0',
    'format'  : 'grib',
}, 'download_era5_cdsapi.grib')

# which works after accepting the terms in ERA5 data set. (complicated)

# modify the Source/Fortran/makefile_local_gfortran
./setup_local.sh
WARNING: installdir has not been specified
flex_extract will be installed in current dir by compiling the Fortran source in /srvfs/home/mblaschek/workspace/flex_extract/Source/Fortran
Install flex_extract_v7.1.3 software at local in directory /srvfs/home/mblaschek/workspace/flex_extract

Using makefile: makefile_local_gfortran
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./rwgrib2.f90
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./fft99.f90
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./phgrreal.f90
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./grphreal.f90
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./ftrafo.f90                                                                                      
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./jsppole.f90                                                                                     
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./set99.f90  
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./calc_etadot.f90                                                                                 
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./posnam.f90                                                                                      
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./qpassm.f90                                                                                      
gfortran   -O3  -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm   -I. -I/srvfs/home/mblaschek/workspace/flex_extract/pyflexextract/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c    ./rpassm.f90
ln -sf calc_etadot_fast.out calc_etadot
lrwxrwxrwx. 1 mblaschek vscusers 20 Jan 29 12:04 ./calc_etadot -> calc_etadot_fast.out
SUCCESS: INSTALLATION FINISHED!
      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        ok. Finally there. Went to Run and modified the run_local.sh

        QUEUE=None
START_DATE=None
END_DATE=None
DATE_CHUNK=None
JOB_CHUNK=3
BASETIME=None
STEP=None
LEVELIST=None
AREA=None
INPUTDIR='/some/path/flexextract'
OUTPUTDIR='/some/path/flexextract_out'
PP_ID=None
JOB_TEMPLATE=None
CONTROLFILE='CONTROL_EA5'
DEBUG=0
REQUEST=2
PUBLIC=0

        and then I get the same error:

        ./run_local.sh                                    
Retrieving ECMWF data and printing MARS request!
start date 20180809                                                                              
end date 20180809                                                                                
... removing old files in /some/path/flexextract                                   
... retrieve 20180808/to/20180810 in dir /some/path/flexextract                    
marsclass: EA                                                                                    
dataset: None                                                                                    
type: FC                                                                                         
levtype: SFC                                                                                     
levelist: 1                                                                                      
repres:                                                                                          
date: 20180808/to/20180810                                                                       
resol: 799                                                                                       
stream: OPER                                                                                     
area: 74.0/-24.0/9.875/60.375                                                                    
time: 06/18                                                                                      
step: 1/to/12/by/1                                                                               
expver: 1                                                                                        
number: OFF                                                                                      
accuracy: 24                                                                                     
grid: 0.28125/0.28125                                                                            
gaussian:                                                                                        
target: /some/path/flexextract/FCOG_acc_SL.20180808.1886661.1886662.grb            
param: 142.128/143.128/146.128/180.128/181.128/176.128                                           
target: /some/path/flexextract/FCOG_acc_SL.20180808.1886661.1886662.grb
MARS request failed

  File "/some/path/workspace/flex_extract/Run/../Source/Python/submit.py", line 272, in <module>
    main()

  File "/some/path/workspace/flex_extract/Run/../Source/Python/submit.py", line 108, in main
    get_mars_data(c)

  File "/some/path/workspace/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data
    do_retrievement(c, server, start, end, datechunk, fluxes=True)

  File "/some/path/workspace/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement
    my_error('MARS request failed')

  File "/some/path/workspace/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error
    trace = '\n'.join(traceback.format_stack())
      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        running it in debugger mode, shows that it tryies to run mars.

        param: 142.128/143.128/146.128/180.128/181.128/176.128
target: /some/path/flexextract/FCOG_acc_SL.20180808.1894481.1894482.grb
FileNotFoundError: [Errno 2] No such file or directory: 'mars'
> /some/path/workspace/flex_extract/Source/Python/Classes/EcFlexpart.py(758)retrieve()
-> self._start_retrievement(request, retr_param_dict)
-> MR.data_retrieve()
/some/path/workspace/flex_extract/Source/Python/Classes/MarsRetrieval.py(616)data_retrieve()
-> p = subprocess.Popen(['mars'], #'-e'],  
/some/path/workspace/flex_extract/pyflexextract/lib/python3.10/subprocess.py(966)__init__()
-> self._execute_child(args, executable, preexec_fn, close_fds,
  /some/path/workspace/flex_extract/pyflexextract/lib/python3.10/subprocess.py(1842)_execute_child()
-> raise child_exception_type(errno_num, err_msg, err_filename)
      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        I switched the PUBLIC=1 or using the commandline --public and i get past this error. Not sure yet if it does the right thing....

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        it managed to run:

        MARS retrieve done ...  
Prepare 20180809/to/20180809
... index will be done
Inputfile: /srvfs/scratch/mblaschek/flexextract/FCOG_acc_SL.20180808.1979117.1979118.grb 
... index done
CURRENT PRODUCT:  ('20180807', '600', '1')
CURRENT PRODUCT:  ('20180807', '600', '2')
CURRENT PRODUCT:  ('20180807', '600', '3')
CURRENT PRODUCT:  ('20180807', '600', '4')
CURRENT PRODUCT:  ('20180807', '600', '5')
CURRENT PRODUCT:  ('20180807', '600', '6')
CURRENT PRODUCT:  ('20180807', '600', '7')
CURRENT PRODUCT:  ('20180807', '600', '8')
CURRENT PRODUCT:  ('20180807', '600', '9')
CURRENT PRODUCT:  ('20180807', '600', '10')
CURRENT PRODUCT:  ('20180807', '600', '11')
CURRENT PRODUCT:  ('20180807', '600', '12')
CURRENT PRODUCT:  ('20180807', '1800', '1')
CURRENT PRODUCT:  ('20180807', '1800', '2')
CURRENT PRODUCT:  ('20180807', '1800', '3')
CURRENT PRODUCT:  ('20180807', '1800', '4')
CURRENT PRODUCT:  ('20180807', '1800', '5')
CURRENT PRODUCT:  ('20180807', '1800', '6')
outputfile = /srvfs/scratch/mblaschek/flexextract/flux2018080722
142 1800 6 68929 0.0 0.1838065644320541
143 1800 6 68929 0.0 0.19190578746866577
146 1800 6 68929 3.4062847222222223 14.69085441062828
180 1800 6 68929 0.17919053819444444 0.10045673846519904
181 1800 6 68929 -0.19189127604166667 0.11422989743097448
176 1800 6 68929 4.622222222222224 1.9405748073874236
CURRENT PRODUCT:  ('20180807', '1800', '7')
outputfile = /srvfs/scratch/mblaschek/flexextract/flux2018080723
142 1800 7 68929 0.0 0.1765682024170976
143 1800 7 68929 0.0 0.19041478638948497
146 1800 7 68929 3.8561111111111113 14.714655198430686
180 1800 7 68929 0.19801215277777778 0.10051688865105889
181 1800 7 68929 -0.2265212673611111 0.11281156238008543
176 1800 7 68929 1.4222222222222232 4.8781621888761375
CURRENT PRODUCT:  ('20180807', '1800', '8')
CURRENT PRODUCT:  ('20180810', '1800', '5')
outputfile = /srvfs/scratch/mblaschek/flexextract/flux2018081021
142 1800 5 68929 0.0 0.19899775930672534
143 1800 5 68929 0.0 0.2433926770127348
146 1800 5 68929 2.776527777777778 17.599957986374108
180 1800 5 68929 0.14386176215277777 0.1785751845333336
181 1800 5 68929 -0.10228949652777777 0.1727327772492998
176 1800 5 68929 7.92888888888889 1.0640616715480276
CURRENT PRODUCT:  ('20180810', '1800', '6')
CURRENT PRODUCT:  ('20180810', '1800', '7')
CURRENT PRODUCT:  ('20180810', '1800', '8')
CURRENT PRODUCT:  ('20180810', '1800', '9')
CURRENT PRODUCT:  ('20180810', '1800', '10')
CURRENT PRODUCT:  ('20180810', '1800', '11')
CURRENT PRODUCT:  ('20180810', '1800', '12')
... index will be done
Inputfile: /srvfs/scratch/mblaschek/flexextract/ANOG__ML.20180809.1979117.1979118.grb 
Inputfile: /srvfs/scratch/mblaschek/flexextract/ANOG__SL.20180809.1979117.1979118.grb 
Inputfile: /srvfs/scratch/mblaschek/flexextract/ANSH__SL.20180809.1979117.1979118.grb 
... index done
current product:  ('20180809', '0', '0')
ECCODES ERROR   :  concept: no match for paramId=201031
ECCODES ERROR   :  concept: input handle edition=2, centre=ecmf
ECCODES ERROR   :  concept: input handle dataset=era
ECCODES ERROR   :  Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=201031'

        and then it runs into the known problem with the changed code table.

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        I think using pip3 install eccodes --no-binary eccodes instead of pip3 install eccodes should fix that. then the version of the library should be 2.35 and not 2.39.2, which causes this error. Rerunning...

      • S
        GitLab Support Bot @support-bot ·
        Author

        Hi Michael,

        Thank you for looking into this. Since submitting this ticket, I have been able to get flex_extract installed. The problem ended up being that I needed to change the shebang hander in the some of the files so that the proper python environment was being loaded. However, I did get those same warnings regarding the binary, so I will try reinstalling it using the $ pip3 install eccodes --no-binary eccodes line of code that you mentioned.

        My next problem is one that I have seen previously documented (see https://forum.ecmwf.int/t/eccodes-error-concept-no-match-for-paramid-201031-when-using-flex-extract-to-prepare-era5/7240 ), but it doesn't look like there is a solution for this problem yet.

        Thank you again for your assistance,

        Kevin

        ...

        From: Michael Blaschek (@blaschm7) gitlab.phaidra@univie.ac.at Sent: January 29, 2025 8:30 AM To: Kevin Bloxam kevin.bloxam@utoronto.ca Subject: Re: Issues with running flex_extract (#7 (closed))

        I think using pip3 install eccodes --no-binary eccodes instead of pip3 install eccodes should fix that. then the version of the library should be 2.35 and not 2.39.2, which causes this error. Rerunning...

        — Unsubscribehttps://gitlab.phaidra.org/-/sent_notifications/REDACTED/unsubscribe

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        reinstalling does the trick.

        Inputfile: /some/path/flexextract/ANOG__SL.20180809.2009050.2009051.grb            
... index done                                                                                   
current product:  ('20180809', '0', '0')                                                         
STATISTICS:          98204.5678  98105.2958   4414.5296                                          
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL  
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS                                          
outputfile = /some/path/flexextract/EA18080900                                     
current product:  ('20180809', '100', '0')                                                       
STATISTICS:          98202.9809  98103.6563   4415.6630                                          
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL  
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS                                          
outputfile = /some/path/flexextract/EA18080901                                     
current product:  ('20180809', '200', '0')                                                       
STATISTICS:          98204.9780  98105.7558   4413.4316                                          
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL  
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS                                          
outputfile = /some/path/flexextract/EA18080902                                     
current product:  ('20180809', '300', '0')                                                       
STATISTICS:          98219.5501  98120.5240   4409.3995                                          
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL  
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS                                          
outputfile = /some/path/flexextract/EA18080903                                     
current product:  ('20180809', '400', '0')                                                       
STATISTICS:          98239.3662  98140.5783   4404.5394                                          
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL  
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS                                          
outputfile = /some/path/flexextract/EA18080904
current product:  ('20180809', '500', '0')                                                       
STATISTICS:          98264.0184  98165.4277   4400.6967                                        
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS     
outputfile = /some/path/flexextract/EA18080905
current product:  ('20180809', '600', '0')
...
current product:  ('20180809', '2300', '0')
STATISTICS:          98424.9895  98324.5547   4445.2783
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS
outputfile = /some/path/flexextract/EA18080923


Postprocessing:
 Format: GRIB2



Grib compression type:
 packingType: grid_simple

Output filelist: 
['EA18080900', 'EA18080901', 'EA18080902', 'EA18080903', 'EA18080904', 'EA18080905', 'EA18080906', 'EA18080907', 'EA18080908', 'EA18080909', 'EA18080910', 'EA18080911', 'EA18080912', 'EA18080913', 'EA18080914', 'EA18080915', 'EA18080916', 'EA18080917', 'EA18080918', 'EA18080919', 'EA18080920', 'EA18080921', 'EA18080922', 'EA18080923']
... clean inputdir!
... done!
FLEX_EXTRACT IS DONE!


ll /some/path/flexextract_out/
total 3.7G
-rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080900
-rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080901
-rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080902
-rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080903
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080904
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080905
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080906
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080907
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080908
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080909
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080910
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080911
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080912
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080913
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080914
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080915
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080916
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080917
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080918
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080919
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080920
-rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080921
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080922
-rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080923

        Then everything works. Thanks for letting me know aout the forum from ecmwf. I will write something there explaining the details.

      • Michael Blaschek closed

        closed

      • S
        GitLab Support Bot @support-bot ·
        Author

        Awesome! I will try it out and let you know if I have any troubles.

        Thank you again for your time.

        Cheers,

        Kevin

        ...

        From: Michael Blaschek (@blaschm7) gitlab.phaidra@univie.ac.at Sent: January 29, 2025 3:13 PM To: Kevin Bloxam kevin.bloxam@utoronto.ca Subject: Re: Issues with running flex_extract (#7 (closed))

        reinstalling does the trick.

        Inputfile: /some/path/flexextract/ANOG__SL.20180809.2009050.2009051.grb ... index done current product: ('20180809', '0', '0') STATISTICS: 98204.5678 98105.2958 4414.5296 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080900 current product: ('20180809', '100', '0') STATISTICS: 98202.9809 98103.6563 4415.6630 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080901 current product: ('20180809', '200', '0') STATISTICS: 98204.9780 98105.7558 4413.4316 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080902 current product: ('20180809', '300', '0') STATISTICS: 98219.5501 98120.5240 4409.3995 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080903 current product: ('20180809', '400', '0') STATISTICS: 98239.3662 98140.5783 4404.5394 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080904 current product: ('20180809', '500', '0') STATISTICS: 98264.0184 98165.4277 4400.6967 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080905 current product: ('20180809', '600', '0') ... current product: ('20180809', '2300', '0') STATISTICS: 98424.9895 98324.5547 4445.2783 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080923

        Postprocessing: Format: GRIB2

        Grib compression type: packingType: grid_simple

        Output filelist: ['EA18080900', 'EA18080901', 'EA18080902', 'EA18080903', 'EA18080904', 'EA18080905', 'EA18080906', 'EA18080907', 'EA18080908', 'EA18080909', 'EA18080910', 'EA18080911', 'EA18080912', 'EA18080913', 'EA18080914', 'EA18080915', 'EA18080916', 'EA18080917', 'EA18080918', 'EA18080919', 'EA18080920', 'EA18080921', 'EA18080922', 'EA18080923'] ... clean inputdir! ... done! FLEX_EXTRACT IS DONE!

        ll /some/path/flexextract_out/ total 3.7G -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080900 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080901 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080902 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080903 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080904 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080905 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080906 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080907 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080908 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080909 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080910 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080911 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080912 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080913 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080914 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080915 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080916 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080917 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080918 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080919 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080920 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080921 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080922 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080923

        Then everything works. Thanks for letting me know aout the forum from ecmwf. I will write something there explaining the details.

        — Unsubscribehttps://gitlab.phaidra.org/-/sent_notifications/REDACTED/unsubscribe

      • S
        GitLab Support Bot @support-bot ·
        Author

        Hi Michael,

        Sorry for another email. I have some more questions. I have been having some trouble with micromamba (I haven't used it before). After installing micromamba, I tried then line of code you had,

        $ micromamba install -c conda-forge -p ./pyflexextract python==3.10 numpy genshi eccodes==2.35 ecmwf-api-client cdsapi fortran-compiler

        but didn't work. Should "install" instead be set to "create"? In running with create, I don't get any error messages and it appears to be okay.

        The next question is, do I then need to activate the environment? I.e.

        $ micromamba activate /path/to/pyflexextract

        My last question (for now) is after activating (?) the environment, I do

        $ pip3 install eccodes --no-binary eccodes

        but in doing so it uses version 2.39.2. I checked the other versions available, but I don't see version 2.35 that you mentioned. Is 2.35 the correct version?

        Thank you again,

        Kevin

        ...

        From: Kevin Bloxam kevin.bloxam@utoronto.ca Sent: January 29, 2025 3:16 PM To: Flexpart / Flex Extract gitlab.phaidra+9c134b312b59ca4946f079ed005e53a9@univie.ac.at Subject: Re: Issues with running flex_extract (#7 (closed))

        Awesome! I will try it out and let you know if I have any troubles.

        Thank you again for your time.

        Cheers,

        Kevin

        From: Michael Blaschek (@blaschm7) gitlab.phaidra@univie.ac.at Sent: January 29, 2025 3:13 PM To: Kevin Bloxam kevin.bloxam@utoronto.ca Subject: Re: Issues with running flex_extract (#7 (closed))

        reinstalling does the trick.

        Inputfile: /some/path/flexextract/ANOG__SL.20180809.2009050.2009051.grb ... index done current product: ('20180809', '0', '0') STATISTICS: 98204.5678 98105.2958 4414.5296 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080900 current product: ('20180809', '100', '0') STATISTICS: 98202.9809 98103.6563 4415.6630 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080901 current product: ('20180809', '200', '0') STATISTICS: 98204.9780 98105.7558 4413.4316 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080902 current product: ('20180809', '300', '0') STATISTICS: 98219.5501 98120.5240 4409.3995 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080903 current product: ('20180809', '400', '0') STATISTICS: 98239.3662 98140.5783 4404.5394 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080904 current product: ('20180809', '500', '0') STATISTICS: 98264.0184 98165.4277 4400.6967 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080905 current product: ('20180809', '600', '0') ... current product: ('20180809', '2300', '0') STATISTICS: 98424.9895 98324.5547 4445.2783 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080923

        Postprocessing: Format: GRIB2

        Grib compression type: packingType: grid_simple

        Output filelist: ['EA18080900', 'EA18080901', 'EA18080902', 'EA18080903', 'EA18080904', 'EA18080905', 'EA18080906', 'EA18080907', 'EA18080908', 'EA18080909', 'EA18080910', 'EA18080911', 'EA18080912', 'EA18080913', 'EA18080914', 'EA18080915', 'EA18080916', 'EA18080917', 'EA18080918', 'EA18080919', 'EA18080920', 'EA18080921', 'EA18080922', 'EA18080923'] ... clean inputdir! ... done! FLEX_EXTRACT IS DONE!

        ll /some/path/flexextract_out/ total 3.7G -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080900 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080901 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080902 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080903 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080904 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080905 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080906 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080907 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080908 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080909 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080910 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080911 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080912 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080913 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080914 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080915 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080916 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080917 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080918 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080919 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080920 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080921 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080922 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080923

        Then everything works. Thanks for letting me know aout the forum from ecmwf. I will write something there explaining the details.

        — Unsubscribehttps://gitlab.phaidra.org/-/sent_notifications/REDACTED/unsubscribe

      • S
        GitLab Support Bot @support-bot ·
        Author

        Hi Michael,

        Since my last email, I continued to move forward with trying to get flex_extract to work. It seems to be working better, and I have moved past my last problem, however I have encountered the following error message:

        ... STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /home/kbloxam/flex_extract/Run/Workspace/EA5/EA18080922 current product: ('20180809', '2300', '0') STATISTICS: 98424.9895 98324.5547 4445.2783 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /home/kbloxam/flex_extract/Run/Workspace/EA5/EA18080923

        Postprocessing: Format: GRIB2

        Output filelist: ['EA18080900', 'EA18080901', 'EA18080902', 'EA18080903', 'EA18080904', 'EA18080905', 'EA18080906', 'EA18080907', 'EA18080908', 'EA18080909', 'EA18080910', 'EA18080911', 'EA18080912', 'EA18080913', 'EA18080914', 'EA18080915', 'EA18080916', 'EA18080917', 'EA18080918', 'EA18080919', 'EA18080920', 'EA18080921', 'EA18080922', 'EA18080923'] ECCODES ERROR : concept: no match for paramId=142 ECCODES ERROR : concept: input handle edition=2, centre=ecmf ECCODES ERROR : concept: input handle dataset=era ECCODES ERROR : Please check the Parameter Database 'https://codes.ecmwf.int/grib/param-db/?id=142' ECCODES ERROR : grib_set_values[0] edition (type=long) failed: Concept no match (message 824) ECCODES ERROR : grib_set_values[1] productDefinitionTemplateNumber (type=long) failed: Value is read only (message 824) ... ERROR CODE: 220 ... ERROR MESSAGE: Command '['grib_set', '-s', 'edition=2,productDefinitionTemplateNumber=8', '/home/kbloxam/flex_extract/Run/Workspace/EA5/EA18080900', '/home/kbloxam/flex_extract/Run/Workspace/EA5/EA18080900_2']' returned non-zero exit status 220. ... GRIB2 CONVERSION FAILED!

        Do you have any idea what do to here?

        Thank you,

        Kevin

        ...

        From: Kevin Bloxam kevin.bloxam@utoronto.ca Sent: January 30, 2025 3:04 PM To: Flexpart / Flex Extract gitlab.phaidra+9c134b312b59ca4946f079ed005e53a9@univie.ac.at Subject: Re: Issues with running flex_extract (#7 (closed))

        Hi Michael,

        Sorry for another email. I have some more questions. I have been having some trouble with micromamba (I haven't used it before). After installing micromamba, I tried then line of code you had,

        $ micromamba install -c conda-forge -p ./pyflexextract python==3.10 numpy genshi eccodes==2.35 ecmwf-api-client cdsapi fortran-compiler

        but didn't work. Should "install" instead be set to "create"? In running with create, I don't get any error messages and it appears to be okay.

        The next question is, do I then need to activate the environment? I.e.

        $ micromamba activate /path/to/pyflexextract

        My last question (for now) is after activating (?) the environment, I do

        $ pip3 install eccodes --no-binary eccodes

        but in doing so it uses version 2.39.2. I checked the other versions available, but I don't see version 2.35 that you mentioned. Is 2.35 the correct version?

        Thank you again,

        Kevin

        From: Kevin Bloxam kevin.bloxam@utoronto.ca Sent: January 29, 2025 3:16 PM To: Flexpart / Flex Extract gitlab.phaidra+9c134b312b59ca4946f079ed005e53a9@univie.ac.at Subject: Re: Issues with running flex_extract (#7 (closed))

        Awesome! I will try it out and let you know if I have any troubles.

        Thank you again for your time.

        Cheers,

        Kevin

        From: Michael Blaschek (@blaschm7) gitlab.phaidra@univie.ac.at Sent: January 29, 2025 3:13 PM To: Kevin Bloxam kevin.bloxam@utoronto.ca Subject: Re: Issues with running flex_extract (#7 (closed))

        reinstalling does the trick.

        Inputfile: /some/path/flexextract/ANOG__SL.20180809.2009050.2009051.grb ... index done current product: ('20180809', '0', '0') STATISTICS: 98204.5678 98105.2958 4414.5296 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080900 current product: ('20180809', '100', '0') STATISTICS: 98202.9809 98103.6563 4415.6630 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080901 current product: ('20180809', '200', '0') STATISTICS: 98204.9780 98105.7558 4413.4316 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080902 current product: ('20180809', '300', '0') STATISTICS: 98219.5501 98120.5240 4409.3995 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080903 current product: ('20180809', '400', '0') STATISTICS: 98239.3662 98140.5783 4404.5394 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080904 current product: ('20180809', '500', '0') STATISTICS: 98264.0184 98165.4277 4400.6967 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080905 current product: ('20180809', '600', '0') ... current product: ('20180809', '2300', '0') STATISTICS: 98424.9895 98324.5547 4445.2783 Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS outputfile = /some/path/flexextract/EA18080923

        Postprocessing: Format: GRIB2

        Grib compression type: packingType: grid_simple

        Output filelist: ['EA18080900', 'EA18080901', 'EA18080902', 'EA18080903', 'EA18080904', 'EA18080905', 'EA18080906', 'EA18080907', 'EA18080908', 'EA18080909', 'EA18080910', 'EA18080911', 'EA18080912', 'EA18080913', 'EA18080914', 'EA18080915', 'EA18080916', 'EA18080917', 'EA18080918', 'EA18080919', 'EA18080920', 'EA18080921', 'EA18080922', 'EA18080923'] ... clean inputdir! ... done! FLEX_EXTRACT IS DONE!

        ll /some/path/flexextract_out/ total 3.7G -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080900 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080901 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080902 -rw-r--r--. 1 user users 156M Jan 29 16:21 EA18080903 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080904 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080905 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080906 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080907 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080908 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080909 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080910 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080911 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080912 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080913 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080914 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080915 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080916 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080917 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080918 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080919 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080920 -rw-r--r--. 1 user users 156M Jan 29 16:22 EA18080921 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080922 -rw-r--r--. 1 user users 157M Jan 29 16:22 EA18080923

        Then everything works. Thanks for letting me know aout the forum from ecmwf. I will write something there explaining the details.

        — Unsubscribehttps://gitlab.phaidra.org/-/sent_notifications/REDACTED/unsubscribe

      • Michael Blaschek
        Michael Blaschek @blaschm7 ·
        Owner

        Hi, So this is issue #6 (closed) with eccodes 2.38. It works with eccodes 2.35. I'm in contact with support to change this too. Best wishes Michael

      Please register or sign in to reply