Home

Awesome

stac-sentinel

This repository is used for the creating STAC Items for Sentinel remote sensing data from their original metadata. Currently Sentinel-1 and Sentinel-2 data is supported.

The library includes:

Table of Contents

SNS Topics

There is a publicly deployed version of the stac-sentinel Lambda function along with a publicly available SNS topic. Anyone can subscribe to the SNS topic from resources in their AWS account (use the ARNs provided below) in order to get metadata for the latest Sentinel scenes. The SNS message is JSON that contains the STAC Item at:

stac_item = message['Records'][0]['Sns']['Message']

The published message also uses SNS Message attributes that can be used to filter SNS messages using SNS Message Filter. The published STAC SNS messages can be filtered using these attributes:

For example, an SNS filter policy that only notifies the subscriber when an Item is over Rio de Janeiro (22.9068° S, 43.1729° W) looks like this keeping-a-spatiotemporal-asset-catalog-stac-up-to-date-with-sns-sqs/)):

{
	"bbox.ll_lon": [{"numeric":["<=",-43.1729]}],
	"bbox.ur_lon": [{"numeric":[">=",-43.1729]}],
	"bbox.ll_lat": [{"numeric":["<=",-22.9068]}],
	"bbox.ur_lat": [{"numeric":[">=",-22.9068]}]
}

Thanks to Frederico Liporace for his article Keeping a SpatioTemporal Asset Catalog (STAC) Up To Date with SNS/SQS.

SNS ARNs

Sentinel-1 L1C

STAC VersionSNS ARN
0.9.0arn:aws:sns:eu-central-1:608149789419:stac-0-9-0_sentinel-s1-l1c

Sentinel-2 L1C

STAC VersionSNS ARN
0.6.0arn:aws:sns:eu-central-1:552188055668:sentinel-stac
0.9.0arn:aws:sns:eu-central-1:608149789419:stac-0-9-0_sentinel-s2-l1c

Sentinel-2 L2A

STAC VersionSNS ARN
0.9.0arn:aws:sns:eu-central-1:608149789419:stac-0-9-0_sentinel-s2-l2a

Public Catalogs

An earlier version of the repository was used to create the Sentinel-2 STAC 0.6.0 catalog, which is indexed in Earth-Search and Development Seed's sat-api.

This section will be updated with newer catalogs as they become available.

Installation

If you are interested in using the library to create STAC Items from the historical archive rather than an existing catalog or new scene notifications, you will need to install this library from GitHub.

Because stac-sentinel uses PyProj, the PROJ system libraries will be needed as well and needs to be installed as per your system. You could also consider using the GeoLambda base Docker image, which includes PROJ.

Then, to install the latest released version (from the master branch).

$ pip install git+https://github.com/sat-utils/sat-stac-sentinel

To install a specific versions of stac-sentinel, install the matching version of stac-sentinel.

pip install git+https://github.com/sat-utils/sat-stac-sentinel@0.2.0

The table below shows the corresponding versions between stac-sentinel and STAC:

stac-sentinelSTAC
0.1.00.6.0
0.2.00.9.0

Usage

Accessing Sentinel archive on AWS

The most common way of using this library will be to access the historical archive and generate STAC records from it in order to add to a catalog, index, or use for processing.

While the AWS Sentinel buckets are all requester-pays (meaning the requester pays egress costs), there are public buckets containing the S3 inventories, which is what stac-sentinel uses. This is a quick and efficient way to get a listing of the contents of a bucket.

from stac_sentinel import SentinelSTAC

for item in SentinelSTAC.get_aws_archive('sentinel-s2-l1c'):
    print(item['id'])

The get_aws_archive function finds the latest inventory (within 1 day) for the proper Sentinel collection, it then reads the files looking for the correct metadata file it needs (productInfo.json or tileInfo.json). For each found metadata file it creates the STAC Item and returns it. The get_aws_archive function is a Python generator, meaning that it is not processing all of the files before it returns. It will return the first STAC Item, then processes the next one on the subsequent loop.

There are a few keyword arguments supported to the get_aws_archive that allow some basic filtering.

Command Line Interface

A command line tool is available for accessing the AWS archive in the same manner as using get_aws_archive.

$ stac-sentinel -h

stac-sentinel (v0.2.0b1)

positional arguments:
  collection            Collection ID

optional arguments:
  -h, --help            show this help message and exit
  --version             Print version and exit
  --log LOG             0:all, 1:debug, 2:info, 3:warning, 4:error, 5:critical
                        (default: 2)
  --prefix PREFIX       Only ingest scenes with a path starting with prefix
                        (default: None)
  --start_date START_DATE
                        Only ingest scenes with a Last Modified Date past
                        provided start date (default: None)
  --end_date END_DATE   Only ingest scenes with a Last Modified Date before
                        provided end date (default: None)
  --save SAVE           Save fetch Items as <id>.json files to this folder
                        (default: None)

The difference between using the CLI and using the library function is that the CLI provides a few options for what to do with the STAC Item once it is returned.

These two options are not yet implemented, but are in the roadmap:

Transforming individual scenes

Transforming a single scene is not useful for most users, but is included here for clarity. It may also be useful to look at the SentinelSTAC.get_aws_archive() function

The class SentinelSTAC allows one to transform the original metadata provided in productInfo.json (Sentinel-1) or tileInfo.json (Sentinel-2) into a STAC Item.

# transform a Sentinel-1 L1C scene

import json
from stac_sentinel import SentinelSTAC

metadata = json.loads(open('productInfo.json').read())

# create an instance of SentinelSTAC for this scene
scene = SentinelSTAC('sentinel-s1-l1c', metadata)

# get the STAC Item
item = scene.to_stac(base_url='')

The base_url is the location, either local or remote, where the data for this scene can be found. In the case above, the data is staged locally, so the base_url is given as just the current directory, where it will look for the asset data files expected (as provided in the original metadata).

If the data is remote, then reading it for a scene would look different:

# transform a Sentinel-2 L1C scene

import requests
import json
from stac_sentinel import SentinelSTAC

# get tileInfo
url = 'https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/17/T/KE/2015/10/23/0/tileInfo.json'
r = requests.get(url, stream=True)
metadata = json.loads(r.text)

# create an instance of SentinelSTAC for this scene
scene = SentinelSTAC('sentinel-s2-l1c', metadata)

# get the STAC Item
base_url = 's3://sentinel-s2-l1c/tiles/17/T/KE/2015/10/23/0'
item = scene.to_stac(base_url=base_url)

Note however that in this example, the base_url of s3://sentinel-s2-l1c, is a requester-pays bucket. It will be used to generate the links to the assets as seen in the Sentinel-2 L1C example, but none of these files are accessed directly.

However, for Sentinel-1, there is an additional metadata file that is needed that is only available in the bucket. If you have credentials defined when running this code, it will automatically use requester-pays and you will be charged!

STAC conversion notes

Sentinel-1

The data that is ingested by the sat-stac-sentinel CLI starts with bucket inventory files that are retrieved and used to find all of the productInfo.json files (this is the metadata for one Sentinel scene). This does not contain all of the data needed to create a STAC item however. For each Sentinel-1 asset in the scene there is an XML annotation file (in the annotation/ subdirectory). Most of the data in the annotation file is the same across all annotation files within the scene, so only a single one is needed (there is asset specific data including statistics and noise estimates that are not needed). This annotation file is fetched (from a requester pays bucket), and usedto fill in the additional STAC metadata.

Sentinel-2

The data that is ingested by the sat-stac-sentinel CLI starts with bucket inventory files that are retrieved and used to find all of the tileInfo.json files (this is the metadata for one Sentinel scene). In addition to the inventories, an SNS message is published (arn:aws:sns:us-west-2:274514004127:NewSceneHTML) whenever a new index.html appears in the bucket. The sat-stac-sentinel Lambda function listens for this message to get the link of the s3 path with the new scene.

In either case the tileInfo.json contains all of the data needed to create a STAC Item, however the given geometry is in native coordinates rather than lat/lon. The library reprojects the geometry using EPSG:4326. Additionally, the convex hull is calculated for the geometry. In most cases this doesn't make a difference, however some of the tile geometries can be long and complex. Taking the convex hull is a way to simplify the geometry without impacting search and discovery.

Development

The master branch is the latest versioned release, while the develop branch is the latest development version. When making a new release:

Currently, the Lambda function needs to be deployed manually. It is stac-sentinel-v0 located in eu-central-1.

$ pip install -r requirements-lambda.txt -t lambda/
$ pip install . -t lambda/ --no-deps
$ cd lambda; zip -ru ../lambda-deploy.zip .; cd ..
$ aws lambda update-function-code --function-name stac-sentinel-v0 --region eu-central-1 --zip-file fileb://lambda-deploy.zip

About

stac_sentinel leverages the use of Spatio-Temporal Asset Catalogs