Home

Awesome

planet_tools

This repository utilizes the Planet API to facilitate selecting and ordering imagery, including identfying scenes that are candidates for multilook stereo DEM production.

Primary workflow tools

  1. Locate footprints matching specified attributes in the Planet archives (search4footprints.py)
  2. Ordering and downloading imagery. (order_and_download.py)
  3. Shelving and indexing imagery. (shelve_scenes.py)
  4. Retrieve on hand imagery using either a list of IDs or a footprint of the index table (scene_retreiver.py)

Additional tools:

Installation

Using conda:

conda env create -f environment.yml
conda activate planet_tools

Setup

Planet API Key

An environmental variable PL_API_KEY must exist with your Planet API key in order to query the Planet Data API and place orders.
On Windows:

setx PL_API_KEY [your API key]

On Linux add the following to your .bashrc file:

export PL_API_KEY=[your API key]

Configuration file

A number of settings, including database and AWS credentials are set through the use of a configuration file at: config/config.json. See config/config_example.json for an example.

Usage

Search Planet archives for footprints (search4footprints.py)

This tool searches the Planet Data API for footprints matching the arguments provided. The API returns the footprints and metadata in pages of 250 records, which can take a while to parse. Note: There is a limit to the payload size that can get POSTed when performing a search, making it impossible to remove all onhand or previously searched for records during the search itself. Therefore, it would be a best practice to track what searches (and subsequent matching footprints) have been performed and written to the scenes table and keep search parameters narrow to avoid repeated parsing of records. Duplicates will however be removed before writing to the 'scenes' table.

The footprint written out as --out_path can be passed to order_and_download.py for ordering and downloading.

Search using attribute arguments:

python search4footprints.py --name my_search \ 
    --item_types PSScene4Band \
    --asset_filter basic_analytic \
    --min_date 2019-01-27 \
    --instrument PS2 \
    --aoi vectorfile.shp 

Search using raw filters:

python search4footprints.py --name my_search \ 
    --item_types PSScene4Band \
    --asset_filter basic_analytic \
    -f DateRangeFilter acquired gte 2019-01-27 \
    -f StringInFilter instrument PS2 \
    -f GeometryFilter vectorfile.shp

where the general syntax is:

[type of filter] [field to filter] [comparison (gte, lte, gt, lt)] [value]

The above two searches are identical. The footprints returned by the search criteria can be saved using any of the following arguments:

To get the count for a search without saving the search to your Planet account:

python search4footprints.py 
    --name my_search \ 
    --item_types PSScene4Band \
    --asset_filter basic_analytic \
    --min_date 2019-01-27 \
    --instrument PS2 \
    --aoi vectorfile.shp \
    --get_count \
    --dryrun

Select footprints (select_footprints.py)

This tool searches the scenes table for scenes matching the arguments provided and is thus used when the user knows the records they are searching for have already been parsed from the Planet Data API and written to the scenes table via search4footprints.py. Records can then be retrieved much more quickly than would be from parsing responses from the Data API. This could potentially be combined into search4footprints.py as an argument flag, denoting that scenes should be parsed rather than the Data API.

The footprint written out as --out_selection can be passed to order_and_download.py for ordering and downloading.

python select_footprints.py 
    --aoi vectorfile.shp \
    --min_date 2019-01-27 \
    --instrument PS2 \
    --max_cc 20 \
    --out_selection selection.shp

Order and download imagery (order_and_download.py)

A selected footprint (or list of IDs) can be used to order and download imagery. Orders can only contain 500 IDs each, so this will split the input into groups of 500 and submit them as separate orders. Note: When writing this script (and using AWS) orders were completing in roughly 10 minutes, so it made sense to link ordering and downloading. Recently (using deliver via ZIP, though that may not be the reason) orders have been taking multiple hours to complete. As such, it may make sense to entirely decouple ordering and downloading. Currently, orders can just be placed and downloads not attempted using the --order_only flag. This should be combined with writing the order IDs to a text file using --orders. That text file can then be passed to --download_orders on a subsequent run to skip directly to downloading. Additional functionality can be developed to:

  1. notify when orders are ready via email (this exists in the Orders API).
  2. download all undownloaded orders (order status can be checked via API and compared to a destination directory to determine which are new)
python order_and_download.py 
    --name my_order \
    --selection selection.shp \
    --product_bundle basic_analytic \
    --orders ordered_ids.txt 
    --destination_parent_directory orders/
    --delivery zip

Delivery via AWS is also possible, using --delivery aws, but requires that AWS account credentials are present in config/config.json

Shelving and Indexing (shelve_and_index.py)

Once an order has been downloaded, it can be shelved and indexed. This is a routine linking shelving and indexing. In a standard run, a directory is parsed for order manifests. The order manifest's are parsed for sections corresponding to the actual imagery (as opposed to the metadata files, masks, etc.), and these sections are written to new 'scene manifest' files following the naming convention of the scenes. Next, all scenes are located based on the presence of these scene manifest files, and their metadata (XML and scene manifest) is parsed in order to create both the shelved path and the row to add to the index table. Next the copy to the shelving location is performed, with options to remove the source files after copying and/or to move any unshelveable scenes to an alternate location. Finally, the new records are written to the index table: planet.scenes

python shelve_and_index.py -i orders/

Multilook Stereo Selection (multilook_selection.py)

Select multilook 'pairs' (really groups of scenes) from multilook_candidates table that meet minimum number of scene and minimum area (in m<sup>2</sup>) arguments. Note this can take a long time and it is recommended to use an AOI to reduce the selection.

python multilook_selection.py --out_ids multilook_ids.txt --out_overlaps multilook_overlaps.shp 
--aoi my_aoi.shp --min_pairs 3 --min_area 32_000_000

Footprint a directory of scenes

fp_planet.py
Footprint a directory containing Planet imagery. A field is added to the footprint, specifying the path to each scene, relative to the directory provided to --relative_directory. If not directory is provided, the paths will be relative to the --input_directory.
Note: Imagery is identified by locating scene-level manifest files, like [scene]_manifest.json.

Miscellaneous

lib
A number of submodules are included to group functions that share a common purpose.

ingest_off_nadir.py
Parse off-nadir metadata .csv's from Planet into database table off_nadir. These csv's contain Planet's internal metadata values for each scene's off-nadir and azimuth

manage_searches.py
List or delete saved searches using the Planet Data API.

scene_retriever.py
Copy scenes from shelved locations to a destination directory. (in progress)

sort_scenes_by_date.py
Simple script to sort scenes by date, using only the filenames.

submit_order.py
Wrapper around lib.order to provide a stand-alone method of ordering a list of IDs or selection footprint. Can likely be removed as order_and_download.py can easily be modified to just submit an order.

SQL

sql\table_views_generation.sql: Contains the SQL statements to create all tables and views on sandwich-pool.planet. Not meant to be run as a standalone script.

TODO: