Awesome

Sentinel-1 & Sentinel-2 ARD code list

logos

A curated list supporting the use of Sentinel-1 and Sentinel-2 analysis-ready data (ARD) in the UK

Project Background

JNCC are leading a project to help people use Sentinel-1 and Sentinel-2 analysis-ready data (ARD) provided by the Defra EO Data Service and the CEDA Archive, with a particular focus on accessing data via API (application programming interfaces). As part of this project, we have set up a public repository for sharing code in any language for processing or analysing Sentinel-1 and Sentinel-2 ARD. We conducted a survey in July 2020 to gain a better understanding of user needs and inform the choice of code-sharing platform. Based on the survey results and subsequent discussion with the project team and partners, it was agreed that the best option was to set up this curated list on GitHub. We have also set up a Slack workspace to enable discussion and messaging between users of Sentinel-1 and Sentinel-2 ARD; you can join the group here if you have a Slack account, or sign up here if you do not yet have a Slack account.

How to contribute

This list is a work in progress and we need your help to keep it up-to-date! If you have examples of code for selecting, downloading, manipulating or analysing Sentinel-1 or Sentinel-2 ARD, please add them to the list by making a pull request. See our contribution guidelines for details on how to do this.

Acknowledgements

Thank you very much to all the authors who have contributed their code to this list, and to Andrew Cutts for help with setting up the list. This project is funded by the Caroline Herschel Framework Partnership Agreement on Copernicus User Uptake.

| Tutorials and functions | Defra EO Data Service API | CEDA Data Access | Other functions

| Marine and Coastal Applications | Maerl monitoring | Satellite-derived bathymetry | Intertidal extent mapping

| GitHub accounts of relevant organisations | Other useful Earth Observation GitHub links |

Start Here

Tutorials and functions

Defra EO Data Service API

EODS API training materials

These Jupyter Notebooks were created as training materials to demonstrate use of the EO Data Service API. They were produced before the EODS API Python Library was developed. Many of the processes in these Notebooks can now be carried out more efficiently using the functions in the EODS API Python Library.

'eods-api-example-simple-query'. Introduction to the Defra EO Data Service API and creating a simple query. Python Defra EODS-API
'eods-api-example-generate-cloudless-mosaic'. Generate a list of Sentinel-2 granules with least cloud per granule per orbit for a given date range and geographic area. Python Defra EODS-API
'eods-api-example-wps'. Use web processing services (WPS) to clip raster, select single band from raster, re-project raster, create points from raster, and generate zonal statistics Python Defra EODS-API WPS
'EODS_API' Download Sentinel-2 data via API from the Defra EO Data Service, then create a mosaic, clip to area of interest and create NDVI. Python JNCC EODS-API WPS copernicus-user-uptake
'EODS_API_Best_pixel'. Download Sentinel-2 data via API from the Defra EO Data Service, then create best-pixel composite using the pixels with least cloud in a stack of imagery from different dates RSGISLib Python JNCC EODS-API WPS copernicus-user-uptake
config.py. Config file for use with the above Jupyter Notebooks, e.g. to input your authentication token.
environment.yml. Environment file for use with the Jupyter Notebooks in this section and the section below.

EODS API Python library

A module of library functions for programmatic interaction with the EO Data Service developed by Sam Franklin at CGI, together with four Jupyter Notebooks demonstrating applications of the library for filtering, downloading and manipulating Sentinel-2 data.

eodslib.py. Functions for interacting with the EO Data Service. Examples include: keyword arguments for filtering data; finding the least cloudy Sentinel-2 granules; creating XML files for use in Web Processing Service (WPS) requests; submitting WPS requests; removing split Sentinel-2 granules from a dataframe; processing data downloaded via WPS (unzipping and renaming files, deleting zip files). Python Defra EODS-API WPS
eodslib.py example 1: simple query. Apply filter parameters and return results as a dataframe. Python Defra EODS-API
eodslib.py example 2: query and download. Apply filter parameters and download returned data. Python Defra EODS-API WPS
eodslib.py example 3: query, download, optimise and mosaic. Apply filter parameters, download results, convert downloaded files to 'optimised' GEOTIFFs and mosaic the imagery by creating a virtual raster. Python Defra EODS-API WPS
eodslib.py example 4: query, download clipped area, calculate ndvi. Apply filter parameters, download the returned data clipped to an area of interest, calculate and plot NDVI from the downloaded data. Python Defra EODS-API WPS
eodslib.py example 5: query EODS and create a layer group. Apply filter parameters and create a layer group (mosaic) from selected Sentinel-2 granules. Python Defra EODS-API
eodslib.py example 6: query EODS and update a layer group. Apply filter parameters and update an existing layer group (mosaic), e.g. to use more recently acquired imagery. Python Defra EODS-API

CEDA Data Access

Accessing ARD over HTTP using geotiff.js This example shows the process of creating true colour Sentinel-2 thumbnails using ARD hosted in the CEDA Archive.

Other functions

'habitat-condition-monitoring'. A package of various functions involved in preparation, statistical analysis and modelling with Sentinel-1 and Sentinel-2 data, including cloud and shadow masking, calculating indices, creating thumbnails, generating zonal statistics and summarising these per polygon. R JNCC copernicus-user-uptake
Generate coherence image from Sentinel-1. Please note this uses Sentinel-1 Single Look Complex (SLC) data, not the analysis-ready data processed by JNCC and Defra. XML script and supporting command line scripts to produce a coherence image from two SLC input datasets, enabling detection of ground surface changes over time. JNCC SEPA copernicus-user-uptake

Marine and Coastal Applications

Maerl monitoring

Live maerl classification Jupyter notebooks used to train and apply a mask R-cnn model to classify live maerl in imagery of the sea-bed obtained from drop-down cameras. This example does not use Sentinel-1 or 2 ARD but the method could be applied to satellite imagery. Python NatureScot

Satellite-derived bathymetry

IMECsentinel Bathymetry from Sentinel-2 imagery to detect sandbanks in the southern North Sea. R Newcastle-University

Mapping intertidal extent

Coast X-Ray generates Normalised Difference Water Index for a time-series of Sentinel-2 images to calculate the frequency of water occurrence across the intertidal zone. This approach was developed by the Scottish Dynamic Coast project. The full method is published in Remote Sensing Applications Python

Terrestrial Applications

Habitat Change Detection

Workflow for processing Sentinel-1 and Sentinel-2 with habitat map shapefiles to produce input data for change detection RShiny app, using functions provided in the 'habitat-condition-monitoring' package. There are separate workflows for English sites, Welsh sites and Scottish sites. R JNCC copernicus-user-uptake
'change-analysis-examples' A short analysis of NDVI statistics generated from Sentinel-2 and how they can be used in conjunction with a habitat map shapefile to identify polygons which deviate from mean values by more than set thresholds. R JNCC copernicus-user-uptake
'change-statistics-analysis'. An interactive document demonstrating use of NDVI derived from Sentinel-2 in conjunction with a habitat map shapefile to identify polygons which deviate from mean values by more than set thresholds. R JNCC copernicus-user-uptake
Change detection RShiny App 2020 pilot - coming soon R JNCC copernicus-user-uptake

Peatland Mapping

Nature.Scot bare peat mapping

Bare peat mapping - creating indices. Calculates various indices for bare peat mapping vectors too large to apply in one script, therefore carried out with single raster at a time, controlled input by moving rasters between folders on remote sensing drive. R NatureScot
Bare peat mapping - creating indices. Creates indices from ARD of 10 bands from Sentinel-2. Creates 23 band raster (float 32). Python NatureScot
Bare peat mapping – random forest classification. The script reads an ESRI Shapefile (defined by the "shapefile" variable) with training polygons and then selects a user-determined number of samples from each land cover type. A multilayer image that contains spectral, other continuous data or categorical data is also input. For each randomly selected sample the data values for that pixel are determined and these data are used to run the Random Forest model. After building the model the multilayer image is read, and up to three output layers can be selected for the output image. R NatureScot
Bare peat mapping – random forest classification. Script to perform supervised classification. Applied to bare peat classification on nationwide scale using Sentinel-2. Python NatureScot

JNCC Bare Peat Mapper app

Bare peat mapping functions. A set of R functions developed for a JNCC pilot study on using EO to monitor peatland condition. Functions include detecting bare peat in Sentinel-2 imagery using Random Forest classification and regression modelling. R JNCC copernicus-user-uptake
'CUUPeatApp_map_prep'. Prepares Sentinel-2 and bare peat layers for use in the Bare Peat Mapper app, using functions provided in the 'habitat-condition-monitoring' package. R JNCC copernicus-user-uptake
Markdown documentation detailing the process of using Sentinel-2 imagery and a spatial framework of polygons to study change in bare peat at the following sites:
- Caithness and East Sutherland, Scotland fine-scale mapping and time-series modelling
- Dark Peak, England fine-scale mapping and time-series modelling
- Humberhead NNR, England fine-scale mapping and time-series modelling
- Brecon Beacons, Wales fine-scale mapping and time-series modelling.

Burn mapping

Code developed by Nature.Scot as part of the Copernicus User Uptake project on upland burn mapping from Sentinel 2. The code has been developed in Python in Jupyter notebooks designed to work in a Google Colabs environment, though it should only need minor changes to work in other environments. Project still in progress so code may change.

Burn extent indices - Script to create indices from ARD of 10 bands from Sentinel 2. Creates rasters float 32. copernicus-user-uptake Python Nature.Scot
Burn extent box plots of pixel values - Script to extract pixel values from a raster relating to a set of training polygons and then generate box plots using seaborn python package. copernicus-user-uptake Python Nature.Scot
Burn extent image thresholding - Script to threshold an input image based on one or more spectral index thresholds copernicus-user-uptake Python Nature.Scot

Code developed by Pixalytics and EnviroSAR for the Defra EO Centre of Excellence R&D project: 'Upland Burn Detection with Radar' A set of Jupyter notebooks and example data to:

download Sentinel-1 ARD for a user-defined area and timeframe Python Defra
process Sentinel-1 coherence data for a user-defined area and timeframe Python Defra
analyse the ARD and coherence data and generate an interactive map of burns Python Defra
automate detection of burns from Sentinel-1 data based on threshold values Python Defra

Habitat mapping

The Living England project, led by Natural England, is a multi-year programme delivering a satellite-derived national habitat layer in support of the Environmental Land Management System (ELMS) and the National Capital and Ecosystem Assessment (NCEA) Pilot. The project uses European Space Agency Sentinel-1 and Senetinel-2 imagery, alongside additional open source datasets, within a Random Forest machine learning algorithm to classify land segments into one of a defined list of habitat classes (currently UKBAP), achieving an accuracy of 84% for the latest iteration (2019/20). For further information please contact earth.observation@naturalengland.org.uk quoting "Living England" in the subject line. R Natural England

Water body monitoring

Monitoring extent of water bodies using Sentinel-1 scripts developed by NatureScot can be used for classification of Sentinel-1 on JASMIN using the ARD held on CEDA. Delineating land/water boundaries has many applications including flood mapping and monitoring habitat for aquatic species. Python NatureScot

GitHub accounts of relevant organisations

Nature.Scot - Scotland's Nature Agency homepage
defra - UK government department responsible for safeguarding our natural environment, supporting our food & farming industry, and sustaining a thriving rural economy. homepage
CefasRepRes - Cefas is a world leader in marine science and technology, providing innovative solutions for the aquatic environment, biodiversity and food security homepage
cedadev - Centre for Environmental Data Analysis Developers homepage