The ``flex_extract`` software package contains python and shell scripts as well as a Fortran program. These components rely on several libraries which need to be available before starting the installation process. Currently, the software is only tested for a GNU/Linux environment. Feel free to try it out on other platforms.
At first, go to the `user group <Ecmwf/access.html>`_ section and decide which group you belong to and follow the instructions at :ref:`ref-registration` to obtain an account at ECMWF (if you don't have it already). Depending on the user group and the way to access the ECWMF MARS archive, there are four possible :doc:`Documentation/Overview/app_modes`:
At first, find out to which `user group <Ecmwf/access.html>`_ you belong and follow the instructions at :ref:`ref-registration` to obtain an account at ECMWF (if you don't have it already). Depending on the user group and the way to access the ECWMF MARS archive, there are four possible :doc:`Documentation/Overview/app_modes`:
- Local, member-state user :ref:`[installation]<ref-local-mode>`
- Local, public user :ref:`[installation]<ref-local-mode>`
Please follow the link :doc:`Documentation/Overview/app_modes` to get a description of the modes.
More information can be found in :doc:`Documentation/Overview/app_modes`.
.. note::
If you encounter any problems in the installation process, you can ask for :doc:`support`.
.. _ref-registration:
...
...
@@ -91,10 +91,10 @@ Registration options:
.. _ref-licence:
Licence agreement for public datasets
Accept licences for public datasets
=====================================
Each ECMWF public dataset has its own licence which has to be accepted, regardless of the user group.
Each ECMWF :underline:`public` dataset has its own licence whose acceptance requires user activity, regardless of the user group.
For the *ERA-Interim* and *CERA-20C* datasets this can be done at the ECMWF website `Available ECMWF Public Datasets <https://confluence.ecmwf.int/display/WEBAPI/Available+ECMWF+Public+Datasets>`_. Log in and follow the licence links on the right side for each dataset and accept it.
``Flex_extract`` by itself is a command-line tool. With version 7.1 an upstream shell script was implemented which calls ``flex_extract`` with the command-line parameters.
``flex_extract`` is a command-line tool. In the first versions, it was started via a korn shell script and since version 6, the entry point was a python script. From version 7.1, a bash shell script was implemented to call ``flex_extract`` with the command-line parameters.
To submit a job with ``flex_extract`` change the working directory to the ``Run`` directory which is placed directly under the ``flex_extract_vX.X`` root directory (``X.X`` is the version number):
To submit an extraction job, change the working directory to the subdirectory ``Run`` (directly under the ``flex_extract_vX.X`` root directory, where ``X.X`` is the version number):
.. code-block:: bash
cd <path-to-flex_extract_vX.X>/Run
Within this directory you can find everything you need to modify and run ``flex_extract``. The following tree shows a shortened list of directories and important files. The ``*`` serves as a wildcard. The brackets ``[]`` means that the file appearance depends on the mode of application and night not be present.
Within this directory you can find everything you need to modify and run ``flex_extract``. The following tree shows a shortened list of directories and important files. The ``*`` serves as a wildcard. The brackets ``[]`` indicate that the file is present only in certain modes of application.
.. code-block:: bash
...
...
@@ -28,30 +28,30 @@ Within this directory you can find everything you need to modify and run ``flex_
├── run_local.sh
└── run.sh
The ``Jobscripts`` directory is used to store the Korn shell job scripts generated by a ``flex_extract`` run in the **Remote** or **Gateway** mode. They are used to submit the setup information to the ECMWF server and start the jobs in ECMWF's batch mode. The typical user must not touch these files. They will be generated from template files which are stored in the ``Templates`` directory under ``flex_extract_vX.X``. Usually there will be a ``compilejob.ksh`` and a ``job.ksh`` script which are explained in the section :doc:`Documentation/input`. In the rare case of operational data extraction there will be a ``joboper.ksh`` which is designed to get the time parameters from environment variables at the ECMWF servers.
The ``Jobscripts`` directory is used to store the Korn shell job scripts generated by a ``flex_extract`` run in the **Remote** or **Gateway** mode. They are used to submit the setup information to the ECMWF server and start the jobs in ECMWF's batch mode. The typical user must not touch these files. They are generated from template files which are stored in the ``Templates`` directory under ``flex_extract_vX.X``. Usually there will be a ``compilejob.ksh`` and a ``job.ksh`` script which are explained in the section :doc:`Documentation/input`. In the rare case of operational data extraction there will be a ``joboper.ksh`` which reads time information from environment variables at the ECMWF servers.
The ``Controls`` directory contains a number of example ``CONTROL`` files. These ``CONTROL`` files represent the current range of possible dataset retrievals with ``flex_extract``. Some parameters in the ``CONTROL`` files can be adapted and some others should not be changed. In this :doc:`quick_start` guide we explain how an extraction with ``flex_extract`` can be started in the different :doc:`Documentation/Overview/app_modes` and point out some specifics of each dataset and ``CONTROL`` file.
The ``Controls`` directory contains a number of sample ``CONTROL`` files. They cover the current range of possible kinds of extractions. Some parameters in the ``CONTROL`` files can be adapted and some others should not be changed. In this :doc:`quick_start` guide we explain how an extraction with ``flex_extract`` can be started in the different :doc:`Documentation/Overview/app_modes` and point out some specifics of each dataset and ``CONTROL`` file.
Directly under ``Run`` you find the files ``run.sh`` and ``run_local.sh`` and according to your selected :doc:`Documentation/Overview/app_modes` there might also be a file named ``ECMWF_ENV`` for the user credentials to quickly and automatically access ECMWF servers.
From version 7.1 on, the ``run.sh`` (or ``run_local.sh``) script is the main access point to ``flex_extract``.
From version 7.1 on, the ``run.sh`` (or ``run_local.sh``) script is the main entry point to ``flex_extract``.
.. note::
Note that, for experienced users (or users of older versions), it is still possible to access ``flex_extract`` directly via the ``submit.py`` script which can be found in the directory ``flex_extract_vX.X/Source/Python``.
Note that for experienced users (or users of older versions), it is still possible to start ``flex_extract`` directly via the ``submit.py`` script in directory ``flex_extract_vX.X/Source/Python``.
Job preparation
===============
To actually start a job with ``flex_extract`` it is only necessary to submit the ``run.sh`` or ``run_local.sh`` script on the command-line. For the specification of dataset and the selection of the access mode it is necessary to select and modify a ``CONTROL`` file and change the parameters in the user section of the ``run`` scripts. The following sections describes the differences in the application modes and where the results will be stored.
To actually start a job with ``flex_extract`` it is sufficient to start either ``run.sh`` or ``run_local.sh``. Data sets and access modes are selected in ``CONTROL`` files and within the user section of the ``run`` scripts. One should select one of the sample ``CONTROL`` files. The following sections describes the differences in the application modes and where the results will be stored.
Remote and gateway modes
------------------------
For memberstate users it is recommended to use the *remote* or *gateway* mode, especially for more demanding tasks, to retrieve and convert data on ECMWF machines and to transfer only the final output files to the local host.
For member-state users it is recommended to use the *remote* or *gateway* mode, especially for more demanding tasks, which retrieve and convert the data on ECMWF machines; only the final output files are transferrred to the local host.
Remote mode
The only difference between both modes is the users working location. In the *remote* mode you have to login to the ECMWF server and then go to the ``Run`` directory as shown above. At ECMWF servers ``flex_extract`` is installed in the ``$HOME`` directory. However, to be able to start the program you have to load the ``Python3`` environment with the module system first.
