Testing HighResCut 0.0.1 version with precipitation ("pr") variable

The tests performed using the variable "temperature", after correcting previously identified errors, worked as expected with both methods (the highrescut library and the "old way" method). The only issue encountered was the need to install the matplotlib and numpy libraries manually.

However, the tests conducted with the variable "precipitation" ("pr") presented the following problems:

1. Spartacus Dataset

It was not possible to proceed with downloading the Spartacus dataset. The following error was returned:

FileNotFoundError Traceback (most recent call last) Cell In[7], line 2 1 # Load the dataset for 'pr' variable of 'spartacus' type ----> 2 p_spartacus = hr.load(type='spartacus', model_global=None, model_regional=None, variable='pr', experiment=None, ens=None, start='2020', end='2020', output_path=output_path_spartacus)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/init.py:37, in load(type, model_global, model_regional, variable, experiment, ens, start, end, output_path) 33 return file_path_oeks15, dataset_oeks15 35 if type == 'spartacus': 36 # Call the load_spartacus function to get the file path ---> 37 file_path_spartacus = load_spartacus(variable, start, end, output_path) 39 # Use xarray to load the dataset from the downloaded netCDF file 40 dataset_spartacus = xr.open_dataset(file_path_spartacus)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/spartacus_download.py:125, in load_spartacus(variable, start, end, output_path) 122 print(f"\nAn error occured: {e}") 124 # Path to the directory you want to remove --> 125 shutil.rmtree(os.path.join(output_path, "tmp")) 127 return output_file

File ~/.conda/envs/highrescut/lib/python3.10/shutil.py:715, in rmtree(path, ignore_errors, onerror) 713 orig_st = os.lstat(path) 714 except Exception: --> 715 onerror(os.lstat, path, sys.exc_info()) 716 return 717 try:

File ~/.conda/envs/highrescut/lib/python3.10/shutil.py:713, in rmtree(path, ignore_errors, onerror) 710 # Note: To guard against symlink races, we use the standard 711 # lstat()/open()/fstat() trick. 712 try: --> 713 orig_st = os.lstat(path) 714 except Exception: 715 onerror(os.lstat, path, sys.exc_info())

FileNotFoundError: [Errno 2] No such file or directory: '/jetfs/scratch/lsabchuk/highreslearn/download/SPARTACUS/tmp'

2. Destination Earth Dataset ("destine")

An error occurred during the attempt to download the "pr" dataset from Destination Earth. This seems to be related to authentication via polytope:

HTTPResponseError Traceback (most recent call last) Cell In[12], line 2 1 # Load the dataset for 'pr' variable of 'destine' type ----> 2 p_destine = hr.load(type='destine', model_global='IFS-NEMO', model_regional=None, variable='pr', experiment='SSP3-7.0', ens=None, start='20200901', end='20200930', output_path=output_path_destine)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/init.py:61, in load(type, model_global, model_regional, variable, experiment, ens, start, end, output_path) 57 return file_path_eobs, dataset_eobs 59 if type == 'destine': 60 # Call the download function to get the file path ---> 61 file_path_destine = load_destine(model_global, variable, experiment, start, end, output_path) 63 # Use xarray to load the dataset from the downloaded nectCDF file 64 dataset_destine = xr.open_dataset(file_path_destine)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/destine_download.py:76, in load_destine(model_global, variable, experiment, start, end, output_path) 71 client = Client( 72 address="polytope.lumi.apps.dte.destination-earth.eu", 73 ) 75 # Optionally revoke previous requests ---> 76 client.revoke("all") 78 # Check experiment/activity consistency for scenario simulation 79 if experiment == 'SSP3-7.0':

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/helpers.py:197, in wraps..wrapper..decorated(self, *args, **kwargs) 196 def decorated(self, *args, **kwargs): --> 197 return fun(self, *args, **kwargs)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/helpers.py:229, in authenticated..decorated(self, *args, **kwargs) 227 raise e 228 # try: --> 229 return method(self, *args, **kwargs)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/Client.py:222, in Client.revoke(self, *args, **kwargs) 219 @authenticated 220 @wraps(RequestManager.revoke) 221 def revoke(self, *args, **kwargs): --> 222 return self.request_manager.revoke(*args, **kwargs)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/RequestManager.py:153, in RequestManager.revoke(self, request_id) 150 headers = {"Authorization": ", ".join(self.auth.get_auth_headers())} 152 if request_id == "all": --> 153 request_ids = self.list() 154 else: 155 request_ids = [request_id]

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/RequestManager.py:69, in RequestManager.list(self, collection_id) 67 method = "get" 68 expected_responses = [requests.codes.ok] ---> 69 response, _ = helpers.try_request( 70 method, 71 situation=situation, 72 expected=expected_responses, 73 logger=self._logger, 74 url=url, 75 headers=headers, 76 skip_tls=self.config.get()["skip_tls"], 77 ) 78 found_requests = response.json()["message"] 79 message = "The authenticated user has " + str(len(found_requests)) + " active requests"

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/helpers.py:593, in try_request(method, situation, expected, logger, stream, skip_tls, **kwargs) 590 logger.warning("Maximum HTTP request retries reached") 591 raise e --> 593 response_title, response_messages = process_response( 594 response, situation, url, method, stream, request_content, expected 595 ) 597 logger.debug("Polytope client received HTTP " + response_title) 599 return response, response_messages

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/polytope/api/helpers.py:523, in process_response(response, situation, url, method, stream, request_content, expected) 521 if response.status_code == 401 and "expired" in response_messages[0]: 522 e.class = "ExpiredCredentialsError" --> 523 raise e 524 elif response.status_code not in expected: 525 e.description = "Unexpected HTTP response from the server."

HTTPResponseError: Polytope error Situation: trying to list requests Description: HTTP CLIENT ERROR (401) URL: https://polytope.lumi.apps.dte.destination-earth.eu:443/api/v1/requests HTTP method: GET Request header/body contents: {'headers': {'Authorization': 'Bearer **********HhXw'}, 'json': None} Expected responses: 200 Received response: CLIENT ERROR (401) Details: Invalid credentials

3. ERA5 Dataset Although the download process reported success, the code returned the following error when trying to load the dataset:

Downloaded data successfully.

NameError Traceback (most recent call last) Cell In[13], line 2 1 # Load the dataset for 'pr' variable of 'era5' type ----> 2 p_era5 = hr.load(type='era5', model_global='reanalysis-era5-land', model_regional=None, variable='pr', experiment=None, ens=None, start='20200901', end='20200930', output_path=output_path_era5)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/init.py:69, in load(type, model_global, model_regional, variable, experiment, ens, start, end, output_path) 65 return file_path_destine, dataset_destine 67 if type == 'era5': 68 # Call the download function to get the file path ---> 69 file_path_era5 = load_era5(model_global, variable, start, end, output_path) 71 # Use xarray to load the dataset from the downloaded netCDF file 72 dataset_era5 = xr.open_dataset(file_path_era5)

File ~/.conda/envs/highrescut/lib/python3.10/site-packages/highrescut/era5_download.py:141, in load_era5(model_global, variable, start, end, output_path) 138 daily_sum = dataset.sel(valid_time=dataset.valid_time.dt.hour == 0) 140 # Shift timesteps back by one day --> 141 daily_sum['valid_time'] = daily_sum.valid_time - np.timedelta64(1, 'D') 143 # Rename the time coordinate 144 daily_sum = daily_sum.rename({'valid_time': 'time'})

NameError: name 'np' is not defined

Remapping step:

In this step, the datasets that were downloaded returned the following error (the following example is from the Cordex dataset, but all those that did not return an error when downloading returned the same error):

TypeError Traceback (most recent call last) Cell In[17], line 2 1 # regrid the downloaded cordex data based on the grid of the ÖKS15 data ----> 2 p_regridded_cordex = hr.regrid(type='conservative', input_file = p_cordex[0], output_path_remapped = output_path_remapped)

TypeError: regrid() missing 1 required positional argument: 'target_file'

The datasets that failed during the download stage presented the following error during remapping:

NameError Traceback (most recent call last) Cell In[16], line 2 1 # regrid the downloaded spartacus data based on the grid of the ÖKS15 data ----> 2 p_regridded_spartacus = hr.regrid(type='conservative', input_file = p_spartacus[0], output_path_remapped = output_path_remapped)

NameError: name 'p_spartacus' is not defined

Consequence:

As a result, none of the subsequent steps in the HighResCut processing pipeline could be completed.