Awesome

AiTLAS: Benchmark Arena

We present AiTLAS: Benchmark Arena -- an open-source benchmark suite for evaluating state-of-the-art deep learning approaches for image classification in Earth Observation (EO). To this end, we present a comprehensive comparative analysis of more than 500 models derived from ten different state-of-the-art architectures and compare them to a variety of multi-class and multi-label classification tasks from 22 datasets with different sizes and properties. In addition to models trained entirely on these datasets, we benchmark models trained in the context of transfer learning, leveraging pre-trained model variants, as it is typically performed in practice. All presented approaches are general and can be easily extended to many other remote sensing image classification tasks not considered in this study. To ensure reproducibility and facilitate better usability and further developments, all of the experimental resources including the trained models, model configurations and processing details of the datasets (with their corresponding splits used for training and evaluating the models) are available on this repository.

For further details, please refer to our paper Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification, ISPRS Journal of Photogrammetry and Remote Sensing, Vol.197, pp 18-35

AiTLAS: Benchmark Arena is part of the AiTLAS ecosystem, an open-source library for exploratory and predictive analysis of satellite imaginary pertaining to different remote-sensing tasks.

Citation

For attribution in academic contexts, please cite this work as

@article{aitlas_arena2022,
      title={{Current Trends in Deep Learning for Earth Observation:An Open-source Benchmark Arena for Image Classification}}, 
      author={Ivica Dimitrovski and Ivan Kitanovski and Dragi Kocev and Nikola Simidjievski},
      journal = {ISPRS Journal of Photogrammetry and Remote Sensing},
      volume = {197},
      pages = {18-35},
      year = {2023},
      issn = {0924-2716},
}

Datasets

You can obtain each dataset (with the respective splits) on the links below. If you use these datasets please cite the original authors accordingly. The references are provided in Tables 1 and 2 in our paper. You can find more details about each dataset in the supplementary material of the paper as well as on our AiTLAS Semantic Data Catalog.

Multi-class datasets

Multi-label datasets

Performance

Pretrained [ImageNet-1K]

Multi-class datasets

*Top-1 Accuracy

Multi-label datasets

*Mean Average Precision mAP

Models

All trained models are available here.

Model list