Awesome

Atmospheric Datasets Comparison (ADC) Toolbox is aimed to have a set of tools that allows users to compare atmospheric composition datasets from different sources. Currently, it is possible to compare:

• Forecast data from CAMS model vs. L2 near real-time and offline data from TROPOMI (NO₂, O₃, CO, SO₂, HCHO), IASI (O₃, CO, SO₂) and GOME-2 (NO₂, O₃, SO₂, HCHO) sensors.
• Reanalysis monthly data from CAMS model vs. L3 monthly data from TROPOMI (NO₂), IASI (O₃, CO) and GOME-2 (NO₂) sensors.

A full explanation of the initial version of this software and its theoretical background can be read in <a href = "https://www.researchgate.net/publication/359342884_Development_of_a_Toolbox_to_Compare_Atmospheric_Composition_Datasets_Long-term_trends_in_urban_NO2_concentrations_in_Spain_derived_from_CAMS_reanalysis_and_GOME-2_data" target = "_blank">Research Gate</a>.

Users can install the software in their desktop by following the steps described in <a href = "#requirements">Requirements</a>. A <a href = "https://nbviewer.org/github/esowc/ADC Toolbox/blob/main/thesis/case_study.ipynb" target = "_blank">case study</a> was created to show the usability of the toolbox for deriving long-term trends in air pollution.

ADC Toolbox facilitates the retrieval of all the datasets presented in Table 1, since the dates they became available to the public. As an exception, the retrieval of IASI L2 data is currently available only since May 14, 2019.

The temporal availability of each dataset differs from others. CAMS forecasts became available in 2015 and the reanalysis datasets are accessible since 2003. Besides, the satellites that carry the instruments were launched in different years. Sentinel 5-P was launched in 2017, whereas Metop-A, the first European polar-orbiting satellite, was launched in 2006 and de-orbited in 2021. Metop-B and Metop-C have been operational since 2012 and 2018, respectively.

The needed packages to run this software can be found in <em>requirements.txt</em>. The virtual environment <em>environment.yml</em> was generated to simplify the installation process, users just need to clone the project, create the environment and activate it with:

$ conda config --set channel_priority false
$ conda env create -f environment.yml
$ conda activate ADC Toolbox

If this takes too long, they can manually create it instead <strong>(recommended)</strong>:

$ conda create --name ADC Toolbox
$ conda activate ADC Toolbox
$ conda install -c conda-forge/label/cartopy_dev cartopy
$ pip install -r requirements.txt

To download data from CAMS, users will need to create an account, <a href = "https://ads.atmosphere.copernicus.eu/api-how-to" target = "_blank">get the ADS API key</a>, create a file with the name <em>keys.txt</em> under the folder <em>data</em>, and write their API key in one line.

It is important to know that this code only runs in Linux operative systems due to the fact that CAMS model data must be downloaded in GRIB format. This can only be done using <em>ecCodes</em>, a package developed by ECMWF that is only available for Linux. In order to install this library, refer to this <a href = "https://gist.github.com/MHBalsmeier/a01ad4e07ecf467c90fad2ac7719844a" target = "_blank">installation guide</a> for clear instructions.

Lately, there have been problems retrieving the data from NACIS Natural Earth to generate the maps. It has also been found out that there are compatibility issues between the libraries <em>Cartopy</em> and <em>Shapely</em>. It is recommended that <em>Shapely</em> is reinstalled after installing <em>Cartopy</em> and that the maps data are downloaded manually with the following commands:

!pip install cartopy
!pip uninstall -y shapely
!pip install shapely --no-binary shapely
!wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
!python cartopy_feature_download.py physical
!python cartopy_feature_download.py cultural

In case users want to see the distribution of the components by countries, they will need a Google API Key, Client User ID and Secret. The library Geocoder is used to get the data from Google, but it also allows the users of the ADC Toolbox to use other APIs to do reverse geocoding (retrieving location information by coordinates). In the function scatter_plot, they will find the following line. You can rewrite as they wish. The list of providers can be found in <a href = "https://github.com/DenisCarriere/geocoder" target = "_blank"> <em>Geocoder</em>'s Github repository</a>.

>>> merge_df['Country'] = merge_df.apply(lambda row: geocoder.google([row['latitude'], row['longitude']], 
                                                     method = 'reverse', key = google_api_key).country_long, 
                                                     axis = 1)

If they do not want to edit anything and prefer to run the code using Google API, then they should edit the file <em>keys.txt</em> under the folder <em>data</em>, and write three lines, under their ADS API key. They should contain, in this order, the Google API Key, Client User ID and Secret. They can get their credentials in <a href = "https://console.cloud.google.com/projectselector/google/maps-apis/credentials" target = "_blank">Google Cloud Platform</a>.

├── README.md                               <- Details about project
├── LICENSE                                 <- Details about license
├── apt.txt                                 <- Details about requirements (apt-get)
├── requirements.txt                        <- Details about requirements (pip)
├── environment.yml                         <- Details about virtual environment
├── data                                    <- Folder where datasets are stored
│   │
│   └── cams                                <- Folder where CAMS datasets are stored
│       │
│       ├── 137-levels-definition.csv       <- Details about CAMS L137 levels
│       └── 60-levels-definition.csv        <- Details about CAMS L60 levels
│
├── functions                               <- Folder that contains all functions
│   │
│   ├── functions_general.ipynb             <- Functions that apply to all datasets
│   ├── functions_cams.ipynb                <- Functions that apply to CAMS datasets
│   ├── functions_iasi.ipynb                <- Functions that apply to IASI datasets
│   ├── functions_tropomi.ipynb             <- Functions that apply to TROPOMI datasets
│   └── functions_gome.ipynb                <- Functions that apply to GOME-2 datasets
│
├── report                                  <- Folder where the images of the README.md are stored
│
├── thesis                                  <- Folder for case study (master's thesis)
│
├── trainings                               <- Folder for trainings material 
│   │
│   └── 2021-12-atmospheric-composition     <- 3rd EUMETSAT/ESA/ECMWF Joint Training in AC
│     │
│     ├── images                            <- Folder where the images of the training are stored
│     ├── data-comparison-practical.ipynb   <- Data comparison practical
│     └── keys-example.txt                  <- File format example of keys.txt (hidden)
│
├── main_cams_gome_L2.ipynb                 <- Comparison between CAMS and GOME-2 L2 datasets
├── main_cams_gome_L3.ipynb                 <- Comparison between CAMS and GOME-2 L3 datasets
├── main_cams_iasi_L2.ipynb                 <- Comparison between CAMS and IASI L2 datasets
├── main_cams_iasi_L3.ipynb                 <- Comparison between CAMS and IASI L3 datasets
├── main_cams_tropomi_L2.ipynb              <- Comparison between CAMS and TROPOMI L2 datasets
└── main_cams_tropomi_L3.ipynb              <- Comparison between CAMS and TROPOMI L3 datasets

If you are interested in the ADC toolbox and would like it to have other functionalities or datasets, you can <a href = "https://docs.google.com/forms/d/e/1FAIpQLSd2DYumvDcIGIz1cQrbfezOOEdGBFli5q76uZvmLva6mZ0E7w/viewform?usp=sf_link" target = "_blank">write your feedback in this form</a>. <strong>Thank you very much!</strong>