Home

Awesome

Planet GEE Pipeline CLI

PyPI version Build Status DOI Planet Say Thanks!

This tool is designed to facilitate moving data from Planet's API into Google Earth Engine and using a metadata library. The tool downloads data into a local storage and allows you to process the metadata before uploading into Google Earth Engine. This tool has been updated to account for metadata property type by going through each metadata column and then uses the upload manifest function to upload images for ingestion to EE. This tool also draws from an additional tool I created with is the Google Earth Engine Asset Manager Addon This includes the batch upload feature, but now additional tools such as generating reports of Earth Engine assets and querying quota to name just a few. The ambition is apart from helping users with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as the need arises. I have now released this as a PyPI package for easy installation and this will be updated along with the the GitHub package.

Though this tool is designed to download planet imagery, use porder to use the new ordersv2 API to download planet imagery. This includes additional filter capabilities along with capability to clip and download.

Please note: This tool is in no way an official tool or Planet offering, but is a personal project created and maintained by Samapriya Roy

If you use this tool to download data for your research, and find this tool useful, star and cite it as below

Samapriya Roy. (2019, October 10). samapriya/Planet-GEE-Pipeline-CLI: Planet-GEE-Pipeline-CLI (Version 0.4.8). Zenodo.
http://doi.org/10.5281/zenodo.3479525

Table of contents

Installation

This assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying

python and then pip list

If you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have released this as a python 2.7 but can be easily modified for python 3.

Make sure you install OpenSSL from this page

This toolbox also uses some functionality from GDAL For installing GDAL in Ubuntu

sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install gdal-bin
sudo apt-get install python-gdal

For Windows I found this guide from UCLA

To install Planet-GEE-Pipeline-CLI: You can install using two methods

pip install ppipe

or you can also try

git clone https://github.com/samapriya/Planet-GEE-Pipeline-CLI.git
cd Planet-GEE-Pipeline-CLI
python setup.py install

For linux use sudo. This release also contains a windows installer which bypasses the need for you to have admin permission, it does however require you to have python in the system path meaning when you open up command prompt you should be able to type python and start it within the command prompt window. Post installation using the installer you can just call ppipe using the command prompt similar to calling python. Give it a go post installation type

ppipe -h

Installation is an optional step; the application can be also run directly by executing ppipe.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment. If you don't want to install, browse into the ppipe folder and try python ppipe.py to get to the same result.

Getting started

As usual, to print help:

> ppipe -h
usage: ppipe [-h]
             {
             ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,ca
             ...

Planet Pipeline with Google Earth Engine Batch Addons

positional arguments:
  { ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
                        ---------------------------------------
                        -----Choose from Planet Tools Below-----
                        ---------------------------------------
    planetkey           Enter your planet API Key
    pquota              Prints your Planet Quota Details
    dasync              Uses the Planet Client Async Downloader to download
                        Planet Assets: Does not require activation
    savedsearch         Tool to download saved searches from Planet Explorer
    metadata            Tool to tabulate and convert all metadata files from
                        Planet or Digital Globe Assets
                        -------------------------------------------
                        ----Choose from Earth Engine Tools Below----
                        -------------------------------------------
    update              Updates Selenium drivers for firefox
    selsetup            Non headless setup for new google account, use if
                        upload throws errors
    ee_user             Get Earth Engine API Key & Paste it back to Command
                        line/shell to change user
    quota               Print Earth Engine total quota and used quota
    create              Allows the user to create an asset collection or
                        folder in Google Earth Engine
    selupload           Batch Asset Uploader for Planet Items & Assets using
                        Selenium
    lst                 List assets in a folder/collection or write as text
                        file
    assetsize           Prints collection size in Human Readable form & Number
                        of assets
    tasks               Queries current task status
                        [completed,running,ready,failed,cancelled]
    access              Sets Permissions for items in folder
    cancel              Cancel all running tasks

optional arguments:
  -h, --help            show this help message and exit

To obtain help for a specific functionality, simply call it with help switch, e.g.: ppipe upload -h. If you didn't install ppipe, then you can run it just by going to ppipe directory and running python ppipe.py [arguments go here]

Usage examples

Usage examples have been segmented into two parts focusing on both planet tools as well as earth engine tools, earth engine tools include additional developments in CLI which allows you to recursively interact with their python API

Planet Tools

The Planet Toolsets consists of tools required to access control and download planet labs assets (PlanetScope and RapidEye OrthoTiles) as well as parse metadata in a tabular form which maybe required by other applications.

Planet Key

This tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools. This tool now includes an option for a quiet authentication using the API key incase it is unable to invoke an interactive environment such as in Google colaboratory. You can also perform planet init to set the key as the tool can read from it directly.

usage: ppipe planetkey [-h] [--type TYPE] [--key KEY]

optional arguments:
  -h, --help   show this help message and exit

Optional named arguments:
  --type TYPE  For direct key entry type --type quiet
  --key KEY    Your Planet API Key

If using on a private machine the Key is saved as a csv file for all future runs of the tool.

Planet Quota

This tool prints your Planet quota including allocation name, Total quota, quota used and quota remaining. Calling it is simple

ppipe pquota

Download Async

This tool is built as a wrapper around the Planet Client's own download tool. The tool included in the planet client is multithreaded and allows the user to activate, poll and download at the same time. This will allow you to pass geometry in terms of a geojson files or all filter like start and end date and time using the .json files you created using the aoijson tool earlier.

usage: ppipe dasync [-h] [--infile INFILE] [--item ITEM] [--asset ASSET]
                       [--local LOCAL] [--start START] [--end END]
                       [--cmin CMIN] [--cmax CMAX]

optional arguments:
  -h, --help       show this help message and exit
  --infile INFILE  Choose a geojson from geojson.io or the aoi-json you
                   created earlier using ppipe aoijson
  --item ITEM      Choose from Planet Item types Example: PSScene4Band,
                   PSOrthoTile, REOrthoTile etc
  --asset ASSET    Choose an asset type example: anlaytic,
                   analytic_dn,analytic_sr,analytic_xml etc
  --local LOCAL    Local Path where Planet Item and asset types are saved

Optional named arguments:
  --start START    Start date filter format YYYY-MM-DD
  --end END        End date filter format YYYY-MM-DD
  --cmin CMIN      Cloud cover minimum between 0-1
  --cmax CMAX      Cloud cover maximum between 0-1

A setup using geojson needs to include other filters too and a typical setup would be

ppipe dasync --infile "C:\Users\johndoe\geometry.geojson" --item "PSScene4Band" --asset "analytic" --local "C:\planet" --start "2018-06-01" --end "2018-08-01" --cmin 0 --cmax 0.4

Using a stuctured json file that you might have created earlier means you don't have to pass additional filters everytime

python ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo"

However, you can still decide to pass the filters and the filters you pass will overwrite existing filters

python ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo" --start "2018-06-01" --end "2018-08-01" --cmin "0" --cmax 0.4

Download Saved Searches

Download assets from saved searches which are saved in your planet explorer.

usage: ppipe savedsearch [-h] [--name NAME] [--asset ASSET] [--local LOCAL]
                         [--limit LIMIT]

optional arguments:
  -h, --help     show this help message and exit
  --name NAME    Name of your saved search(It is case sensitive)
  --asset ASSET  Choose asset type analytic, analytic_xml, analytic_sr,
                 analytic_dn etc
  --local LOCAL  Local Path (full path address) where PlanetAssets are saved

Optional named arguments:
  --limit LIMIT  Choose number of assets you want to download

Metadata Parser

The metadata tab is a more powerful tool and consists of metadata parsing for All PlanetScope and RapiEye Assets along with Digital Globe MultiSpectral and DigitalGlobe PanChromatic datasets. This was developed as a standalone to process xml metadata files from multiple sources and is important step is the user plans to upload these assets to Google Earth Engine.

usage: ppipe metadata [-h] [--asset ASSET] [--mf MF] [--mfile MFILE]
                      [--errorlog ERRORLOG] [--dir DIR]

optional arguments:
  -h, --help           show this help message and exit
  --asset ASSET        Choose PS OrthoTile(PSO)|PS OrthoTile DN(PSO_DN)|PS
                       OrthoTile Visual(PSO_V)|PS4Band Analytic(PS4B)|PS4Band
                       DN(PS4B_DN)|PS4Band SR(PS4B_SR)|PS3Band
                       Analytic(PS3B)|PS3Band DN(PS3B_DN)|PS3Band
                       Visual(PS3B_V)|RE OrthoTile (REO)|RE OrthoTile
                       Visual(REO_V)|DigitalGlobe
                       MultiSpectral(DGMS)|DigitalGlobe
                       Panchromatic(DGP)|PolarGeospatial CenterDEM
                       Strip(PGCDEM)?
  --mf MF              Metadata folder?
  --mfile MFILE        Metadata filename to be exported along with Path.csv
  --errorlog ERRORLOG  Errorlog to be exported along with Path.csv

Optional named arguments:
  --dir DIR            Path to Image Directory to be used to get ImageTags
                       with metadata. use only with PS4B_SR

Earth Engine Tools

The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. This is also a seperate package for earth engine users to use and can be downloaded here

selenium update

This is a key step since all upload function depends on this step, so make sure you run this. This downloads selenium driver and places to your local directory for windows and Linux subsystems. This is the first step to use selenium supported upload.

ppipe update

gee selsetup

Once in a while the geckodriver requires manual input before signing into the google earth engine, this tool will allow you to interact with the initialization of Google Earth Engine code editor window. It allows the user to specify the account they want to use, and should only be needed once.

geeup selsetup

EE User

This tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back. This command takes in no arguments.

ppipe ee_user

EE Quota

This tool prints earthengine quota tools, used and remaining quota. No argument is required

ppipe quota

Create

This tool allows you to create a collection or folder in your earth engine root directory. The tool uses the system cli to achieve this and this has been included so as to reduce the need to switch between multiple tools and CLI.

usage: ppipe.py create [-h] --typ TYP --path PATH

optional arguments:
  -h, --help   show this help message and exit
  --typ TYP    Specify type: collection or folder
  --path PATH  This is the path for the earth engine asset to be created full
               path is needsed eg: users/johndoe/collection

Batch uploader

The script creates an Image Collection from GeoTIFFs in your local directory. By default, the image name in the collection is the same as the local directory name; with the optional parameter you can provide a different name.You have to process the metadata for images, which is covered in the next section along with a manifest type for Planet image and asset: Metadata parser.

usage: ppipe selupload [-h] --source SOURCE --dest DEST [-m METADATA]
                          [--large] [--nodata NODATA] [--bands BANDS]
                          [-u USER] [-b BUCKET]

optional arguments:
  -h, --help            show this help message and exit

Required named arguments.:
  --source SOURCE       Path to the directory with images for upload.
  --dest DEST           Destination. Full path for upload to Google Earth
                        Engine, e.g. users/pinkiepie/myponycollection
  -u USER, --user USER  Google account name (gmail address).

Optional named arguments:
  -m METADATA, --metadata METADATA
                        Path to CSV with metadata.
  --large               (Advanced) Use multipart upload. Might help if upload
                        of large files is failing on some systems. Might cause
                        other issues.
  --nodata NODATA       The value to burn into the raster as NoData (missing
                        data)
  --bands BANDS         Comma-separated list of names to use for the image
                        bands. Spacesor other special characters are not
                        allowed.
  -b BUCKET, --bucket BUCKET
                        Google Cloud Storage bucket name.

Asset List

This tool is designed to either print or output asset lists within folders or collections using earthengine ls tool functions.

usage: ppipe lst [-h] --location LOCATION --typ TYP [--items ITEMS]
                 [--output OUTPUT]

optional arguments:
  -h, --help           show this help message and exit

Required named arguments.:
  --location LOCATION  This it the location of your folder/collection
  --typ TYP            Whether you want the list to be printed or output as
                       text[print/report]

Optional named arguments:
  --items ITEMS        Number of items to list
  --output OUTPUT      Folder location for report to be exported

Asset Size

This tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size.

usage: ppipe assetsize [-h] --asset ASSET

optional arguments:
  -h, --help     show this help message and exit
  --asset ASSET  Earth Engine Asset for which to get size properties

Task Query

This script counts all currently running,ready,completed,failed and cancelled tasks along with failed tasks. This is linked to the account you initialized with your google earth engine account. This takes no argument.

usage: ppipe.py tasks [-h]

optional arguments:
  -h, --help  show this help message and exit

ppipe.py tasks

Assets Access

This tool allows you to set asset acess for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time.

usage: ppipe.py access [-h] --asset ASSET --user USER --role ROLE

optional arguments:
  -h, --help     show this help message and exit
  --asset ASSET  This is the path to the earth engine asset whose permission
                 you are changing folder/collection/image
  --user USER    Full email address of the user, try using "AllUsers" to make
                 it public
  --role ROLE    Choose between reader, writer or delete

Cancel all tasks

This is a simpler tool, can be called directly from the earthengine cli as well

earthengine cli command
earthengine task cancel all

Usage using this tool

usage: ppipe cancel [-h]

optional arguments:
  -h, --help  show this help message and exit

Credits

JetStream A portion of the work is suported by JetStream Grant TG-GEO160014.

Also supported by Planet Labs Ambassador Program

Original upload function adapted from Lukasz's asset manager tool

Changelog

v0.4.9

v0.4.8

v0.4.7

v0.4.6

v0.4.4

v0.4.3

v0.4.1

v0.3.8

v0.3.7

v0.3.6

v0.3.5

v0.3.2

v0.3.0

v0.2.91

v0.2.9

v0.2.2

v0.2.1

v0.2.0

v0.1.9

v0.1.8