Awesome

porder: Simple CLI for Planet ordersV2 API  

Disclaimer: This is an unofficial tool. Is not licensed or endorsed by Planet Labs. It was created by Samapriya Roy and is owned and maintained by Tyson Swetnam

Orders v2 is the next iteration of Planet's API for getting Analysis Ready Data (ARD) delivered to you. Orders v2 has the capability to submit any number of images in a batch order and perform operations such as: top of atmospheric (TOA) reflectance, compression, coregistration, and supports enhanced notifications such as email and webhooks. Based on your personal license access you can use the porder to place orders, download orders, or chain together a sequence of operations from the Orders v2 API. The porder tool runs on the command line interface only. For exporting to cloud storage porder uses a configuration folder with config.yml structures.

If you use this tool to download data and find it useful please star us on GitHub (top right corner of this page).

If you publish research which used this tool, please use this citation:

Samapriya Roy, & Tyson L. Swetnam. (2020, August 23). tyson-swetnam/porder: porder: Simple CLI for Planet ordersV2 API (Version 0.7.8). Zenodo.
http://doi.org/10.5281/zenodo.3996650

Read the Guided Docs here

Table of contents

• Prerequisites
• Installing porder
• Windows Setup
• Getting started
• porder Simple CLI for Planet ordersv2 API
  • porder readme
  • porder quota
  • base64
  • convert to geojson
  • simplify
  • idlist
  • difflist
  • idsplit
  • idcheck
  • bundles
  • order
  • reorder
  • cancel
  • orderstate list
  • ordersize
  • stats
  • download
  • multipart download
  • multiprocessing download

Prerequisites

You must have native python & pip installed. You can test this by going to the terminal (or windows command prompt) and trying.

python --version and then pip list

If you have no errors and you have Python 2.7.14 or higher you should be good to go.

We also recommend installation within virtual environment. If you use Conda

$ conda create -n planet_orders python
$ conda activate planet_orders

The porder tool is dependent on shapely and fiona which depend on GDAL

QGIS is a useful open-source GIS platform which includes all of the major required dependencies for running porder. Installation of QGIS will ensure all porder requirements are met.

Linux

Installing Python GDAL in Ubuntu

$ sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
$ sudo apt-get install gdal-bin libcurl4-openssl-dev libssl-dev python3-gdal

Windows

We also recommend using Windows Subsystem for Linux 2 and following the linux instructions.

For Windows this guide from UCLA is useful.

MacOS X

KyngChaos maintains a MacOS X installation for GDAL

Install GDAL using [brew]

Install brew:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install GDAL:

% brew install gdal --HEAD
% brew install gdal
% gdal-config --version 

Install GDAL Python binding:

% pip3 install —-upgrade pip
% pip3 install numpy
% python3 -c ‘import gdal; print(gdal.VersionInfo())’

Docker

You can also run porder with Docker

$ docker pull tswetnam/porder:latest

If you haven't run planet init to create your token, do so first. Make sure to run docker with the volume flag mounted back to your home ${HOME} or current working directory ${PWD}:

$ docker run -it --rm -v ${HOME}:/root tswetnam/porder:latest planet init

check if you now have a private .planet.json file in your home or current working directory.

Run porder as a test:

$ docker run -it --rm -v ${HOME}:/root tswetnam/porder:latest porder quota

This should return your user information and the remaining available km of data:

Subscription ID: XXXXX
Plan ID: XXXXX
Allocation Name: your@email.address
Allocation active from: YEAR-MM-DD
Quota Enabled: True
Total Quota in SqKm: 10000
Total Quota used: 0.0
Remaining Quota in SqKm: 10000.0

Installing porder

Once you have the dependency libraries configured you're ready to install

From pip:

pip install porder

For linux it might help to specify the pip type and use --user

pip3 install porder --user

From GitHub source:

git clone https://github.com/tyson-swetnam/porder.git
cd porder
python setup.py install

Type porder -h to view the help

Windows Setup

Shapely and a few other libraries are notoriously difficult to install on windows machines so follow the steps mentioned here before installing porder. You can download and install shapely and other libraries from the Unofficial Wheel files from here download depending on the python version you have. Do this only once you have install GDAL. I would recommend the steps mentioned above to get the GDAL properly installed. However I am including instructions to using a precompiled version of GDAL similar to the other libraries on windows. You can test to see if you have gdal by simply running

gdalinfo

in your command prompt. If you get a read out and not an error message you are good to go. If you don't have gdal try Option 1,2 or 3 in that order and that will install gdal along with the other libraries

Option 1:

Starting from porder v0.4.5 onwards:

Simply run porder -h after installation. This should go fetch the extra libraries you need and install them. Once installation is complete, the porder help page will show up. This should save you from the few steps below.

Option 2:

If this does not work or you get an unexpected error try the following commands. You can also use these commands if you simply want to update these libraries.

pipwin refresh
pipwin install gdal
pipwin install pyproj
pipwin install shapely
pipwin install fiona
pipwin install geopandas

Option 3

For windows first thing you need to figure out is your Python version and whether it is 32 bit or 64 bit. You can do this by going to your command prompt and typing python.

For my windows machine, I have both 32-bit python 2.7.16 and 64-bit Python 3.6.6. You can get the python version at the beginning of the highlighted lines and the 32 or 64 bit within the Intel or AMD64 within the square brackets. Your default python is the one you get by just typing python in the command line. Then download the following packages based on the information we collect about our python type in the earlier step. We use unofficial binaries to install these. This step is only needed if you are on a windows machine if you are using a setup manager like anaconda you might be able to avoid this setup completely

At this stage if you were unable to install gdal then download the gdal binaries first, install that before everything else

gdal: https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal

Then follow along the following libraries

• pyproj: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyproj
• shapely: https://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely
• fiona: https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona
• geopandas: https://www.lfd.uci.edu/~gohlke/pythonlibs/#geopandas

To choose the version that is correct for you use the python information you collected earlier For example for my python 3.6.6 and AMD 64 if I was installing shapely I would choose the following, here 36 means python 3.6 and amd64 refers to the 64bit we were talking about.

Shapely‑1.6.4.post2‑cp36‑cp36m‑win_amd64.whl

You will get a wheel file or a file ending with .whl. You can now simply browse to the folder or migrate to it in your command prompt. Once there if I am installing for my python 3.6 the command was. At this point we will make use of our trusted package installer that comes with python called pip. Note the choice of pip or pip3 depends on your python version usually you can get the pip to use with your python by typing

pip3 -V

you get a readout like this

pip 18.1 from c:\python3\lib\site-packages\pip (python 3.6)

if you have pip just replace that with pip -V

Then simply install the wheel files you downloaded using the following setup

pip3 install full path to Shapely‑1.6.4.post2‑cp36‑cp36m‑win_amd64.whl

in my case that would be

pip3 install "C:\Users\samapriya\Downloads\Shapely‑1.6.4.post2‑cp36‑cp36m‑win_amd64.whl"

Or you can use anaconda to install. Again, both of these options are mentioned on Shapely’s Official PyPI page.

Getting started

Make sure you initialized planet client by typing planet init. As usual, to print help:

To obtain help for a specific functionality, simply call it with help switch, e.g.: porder idlist -h. If you didn't install porder, then you can run it just by going to porder directory and running python porder.py [arguments go here]

porder Simple CLI for Planet ordersv2 API

The tool is designed to simplify using the ordersv2 API and allows the user to chain together tools and operations for multiple item and asset types and perform these operations and download the assets locally.

porder readme

Now open the readme webpage in your browser using

porder readme

porder quota

Just a simple tool to print your planet subscription quota quickly.

base64

This does exactly as it sounds, it encodes your credential files to base64 for use with gcs.

convert to geojson

This tool allows you to convert from a folder with multiple shapefiles or kml files to a folder with geojson that can then be used with the tool. It makes use of geopandas and reprojects your shapefile to make it compatible while passing onto the API for search and download. KML conversions are handled seperately.

simplify

This reduces the number of vertices for a multi vertex and complex GeoJSON. Extremely high vertex count (over 500) seem to fail and hence this tool allows you to export a new geojson with reduced vertices. It uses an implementation of the Visvalingam-Wyatt line simplification algorithm. This tool does work with and without Fiona, but Fiona installation is recommended.

idlist

Create an idlist for your geometry based on some basic filters,including geometry, start and end date and cloud cover. If no cloud cover is specified everything form 0 to 100% cloud cover is included. For now the tool can handle geojson,json and kml files. The output is a csv file with ids. The tool also allows you to make sure you get percentage overlap, when selecting image, for clip operations adjust it accordingly (usally --ovp 1 for orders not to fail during clip). The tool now also prints estimated area in Square kilometes for the download and estimated area if you clipped your area with the geometry you are searching (just estimates). You can now pass multiple assetswith the idlist tool pass assets as analytic_sr,udm2 for example to get items with both these assets ``` --asset analytic_sr,udm2.

I have changed the setup to now do the following two things

• The number option is optional, so it can look for all images in the time range, but be careful if the area is too large, use at own risk. A better option is to supply the number.

• It is possible to often forget about the different asset types, so you can now not pass an item and the script will return every possible type of asset for each item type depending on the bundle.

A simple setup would be

To run an experiment to add additional filter, you can now pass an additional string or range filter or both flag for string and range filters, a setup would be. The additional filters are optional

porder idlist --input "Path to geojson file" --start "YYYY-MM-DD" --end "YYYY-MM-DD" --item "PSScene4Band" --asset "analytic" --outfile "Path to idlist.csv" --filters range:clear_percent:55:100 --number 20

porder idlist --input "Path to geojson file" --start "YYYY-MM-DD" --end "YYYY-MM-DD" --item "PSScene4Band" --asset "analytic" --outfile "Path to idlist.csv" --filters string:satellite_id:"1003,1006,1012,1020,1038" --number 20

porder idlist --input "Path to geojson file" --start "YYYY-MM-DD" --end "YYYY-MM-DD" --item "PSScene4Band" --asset "analytic" --outfile "Path to idlist.csv" --filters string:satellite_id:"1003,1006,1012,1020,1038" range:clear_percent:55:100 --number 20

The idlist tool can now use a multipolygon and iteratively look for scenes.

difflist

It is possible you already downloaded some images or metadata files, and your you want a difference idlist to create orders for only assets and item types you do not have. It takes in your local folder path, type image or metadata and some basic filters,including geometry, start and end date and cloud cover. If no cloud cover is specified everything form 0 to 100% cloud cover is included. For now the tool can handle geojson,json and kml files. The output is a csv file with ids.

A simple setup would be

or without the cloud filter

idsplit

This allows you to split your idlist into small csv files incase you wanted to created batches of orders.

A simple setup would be

idcheck

It is possible for you to modify the idlist, add or remove ids. Once done, this tool allows you to estimate the total area of images and area that intersect with your geometry or area if clipped.

A simple setup would be

bundles

Ordering using ordersv2 uses the concept of bundles. A bundle is a combination of multiple assets for an item that come together and are delivered as part of the overall fulfillment of the order. For example an analytic asset for PSScene4Band is a single tif file, however the analytic bundle for PSScene4Band includes analytic tiff file, the analytic_xml metadata and the udm data mask file as part of the bundle. You can find more information about bundles here. Thus the concept of bundles bring together single function to order and download multiple related assets. Since the list of bundles is long, this tool simply allows you to get every bundle type based on item type. The setup is simple

A simple setup would be

porder bundles --item "PSScene4Band"

order

This tool allows you to actually place the order using the idlist that you created earlier. the --op argument allows you to take operations, delivery and notifications in a sequence for example --op toar clip email performs Top of Atmospheric reflectance, followed by clipping to your geometry and send you an email notification once the order has completed, failed or had any any change of status. An important changes is the concept of passing bundles instead of using assets. Bundles are predefined meaning all assets in a bundle are not available for an item your attempt at downloading that attempt will fail.

For example if an item id '20181227_125554_0f4c' does not have surface reflectance asset type. So if you try to download this using bundle type analytic_sr_udm2 it will not work, similary if you order an item where a specific operation cannot be performed for example if you order visual and then try to do bandmath with four bands. These examples and more are where fallback bundles come in handy. Think of this as providing a list of bundles to keep trying if one bundle type fails. The priority goes left to right. You can provide comma seperated fallback bundles for example as

analytic_sr_udm2,analytic instead of analytic_sr_udm2 to avoid certain items from failing to download.

The list of operations for the --op are below and ** the order of these operations is important**

clip|toar|comp osite|zip|zipall|compression|projection|kernel|aws|azu re|gcs|email <Choose indices from>: ndvi|gndvi|bndvi|ndwi|tvi|osavi|evi2|msavi2|sr

You can now add some predefined indices for PlanetScope 4 band items with a maximum of 5 indices for a single setup . This is experimental. The list of indices include

A simple setup with image clip with email notification would be

The same setup with delivery of each image, its metadata as a zip file would be. Note how we only added zip to the op list

A simple setup with Top of Atmospher reflectance and a few indices along with email notification would be

reorder

This tool allows you to reorder an existing order with the same item ids and tools. This is trying to look into the overall structures of existing orders and maybe useful to reorder if an order fails for example.

usage: porder reorder [-h] --url URL [--notification NOTIFICATION]

optional arguments:
  -h, --help            show this help message and exit

Required named arguments.:
  --url URL             Order url to be ordered

Optional named arguments:
  --notification NOTIFICATION
                        Use "email" to get an email notification

cancel

You can cancel a queued order or cancel all queued orders, before the status changes to running. Simply put you can cancel a specific order before it starts running or cancel all of your queued orders.

usage: porder cancel [-h] [--id ID]

optional arguments:
  -h, --help  show this help message and exit
  --id ID     order id you want to cancel use "all" to cancel all

Setup to cancel an order

porder cancel --id "orderid goes here"

to cancel all queued orders

porder cancel --id "all"

orderstate list

This tool allows you to get the list of orders based on the states and based on the start and end dates of orders. For example if you want to find out all orders that failed within the week you can use this tool to check that.

usage: porder ostate [-h] [--state STATE] [--start START] [--end END]
                     [--limit LIMIT]

optional arguments:
  -h, --help     show this help message and exit
  --state STATE  choose state between queued| running | success | failed |
                 partial
  --start START  start date in format YYYY-MM-DD
  --end END      end date in format YYYY-MM-DD

Optional named arguments:
  --limit LIMIT  Limit the maximum table size

The setup to check failed orders would be for example the following, You can place a limit on the number of orders to get by using --limit

porder ostate --state failed --start 2019-11-01 --end 2019-11-20

ordersize

The tool now allows you to estimate the total download size for a specific order.

An example setup look like the following

stats

The tool allows you to check on number of running and queued orders for both organization and user level. Using this is simple

porder stats

output should look like this:

Checking on all running orders...
Total queued order for organization: 0
Total running orders for organization: 1

Total queued orders for user: 0
Total running orders for user: 0

download

The allows you to download the files in your order, to a local folder. It uses the order url generated using the orders tool to access and download the files.

multipart download

The allows you to multipart download the files in your order, this uses a multiprocessing downloader to quickly download your files to a local folder. It uses the order url generated using the orders tool to access and download the files.

multiprocessing download

The uses the multiprocessing library to quickly download your files to a local folder. It uses the order url generated using the orders tool to access and download the files and includes an expotential rate limiting function to handle too many requests. To save on time it uses an extension filter so for example if you are using the zip operation you can use ".zip" and if you are downloading only images, udm and xml you can use ".tif" or ".xml" accordingly. For python 3.4 or higher, this switches to using an true async downloader instead of using multiprocessing.

A simple setup would be

Changelog

v0.7.8

• Added support for multipolygon search and clip.
• Added coregistration and format tools to order
• Removed persistent version print
• Overall cleanup and general Improvements

v0.7.7

• Removed dependency on pendulum and general cleanup.

v0.7.6

• Now use multiple assets with the idlist tool pass assets as analytic_sr,udm2 for example to get items with both these assets.
• Updated readme.

v0.7.5

• Fixed pipwin python path for installer.
• idlist tool now only returns standard and excludes beta and test quality.

v0.7.4

• added smart check on partially completed order for reordering.
• completed orders can be reordered as well

v0.7.3

• Now constantly checks for updated version incase your porder version is not updated.
• Added a reorder tool for users to reorder an exiting order or failed order.

v0.7.1

• ID check tool now works with/without geometry, pass an idlist, and item and asset type to check.
• Order size tool is optimized for speed.
• pipwin tool uploaded to v0.5.0 to handle windows packages.
• Overall general improvements.

v0.7.0

• Now estimates total download size before downloading.
• Fixed issue with downloading single archive zip files.

v0.6.9

• Updated readme to do bulk conversion from Shapefile and KML to geojson.
• Fixed issues with bundles read and update.

v0.6.8

• Fixed issues with direct usage of kml to get idlist.
• Combined convert tool to convert folders with shapefiles and kml to geojsons.

v0.6.6-v0.6.7

• Fixed downloader for pipwin for release >= 0.4.8
• Improved overall package installation for windows
• Check pipwin import version to get release 0.4.9

v0.6.5

• Added bundles check function to get latest bundles from developers page.

v0.6.4

• Pull request fixed issue with pipwin installation.

v0.6.2

• Fixed issue with werkzeug and pipwin implementation.
• Changed default overlap to 0.01
• Fixed issue with YAML loader implementation.
• General improvements.

v0.6.1

• Added cancel order and bulk cancel orders tools

v0.6.0

• Fixed base64 encoding for GCS credentials for python 3.
• Re-release of python 3 only supported version. Use upto version 0.5.7 only for Python 2.

v0.5.9

• No more deperecation or future warnings from pyproj Issue 40
• porder search is faster
• Python 2.7 will reach the end of its life on January 1st, 2020. No more python 2 support from v0.5.9

v0.5.8

• Handles python version while installing pipwin
• Now open readme page in browser using porder readme

v0.5.7

• Fixed arg readout if no argument is passed to CLI. Issue 39.
• Updated requirements to include DateTimeRange
• Fixed issue with GDAL~=3 requirement for fiona 1.8.11 release.

v0.5.6

• Merged pull request 38 to allow for nested delivery of zip files to cloud storage.
• Added an order state list tool.

v0.5.5

• Better file check for skipping download requests.
• Improvements to multiprocessing and async downloader.
• General improvements to stability and performance.

v0.5.4

• Updated bundles.json.
• Merged pull request to update bundles.

v0.5.3

• Updated order status.
• Check existing files before attempting redirect url and download.
• Overall optimization for faster check and updated readme for fallback bundles.

v0.5.2

• Added harmonization tool to harmonize PS2.SD to PS2.
• Improvements and error handling to quota tool
• Merged pull request 35 to keep download progress via enumerate.

v0.5.1

• Added utf-8 encoding for shapefile to geojson conversion
• Merged pull request 34 to refresh url once expired.

v0.4.9

• Fixed issue with gdal import for pipwin windows.
• Fixed import issue with stats endpoint.

v0.4.8

• Replaced concurrency check with stats endpoint to get queued and running orders.
• Change pipwin cache refresh time to two weeks.

v0.4.7

• Fixed issue with queuing state for orders and downloads.

v0.4.6

• Handles refreshing pipwin cache and better error handling
• Fixed issue with downloading unique manifest ID for zip files.
• Updated ReadMe with improved documentation.

v0.4.5

• Handles installation of windows specific libraries using pipwin.
• General improvements

v0.4.4

• Manifest files for each asset is now written in format ItemID_manifest.json to avoid skipping manifest.json common file name.
• Simple and multipart downloader now show number of items remaining during the download.
• General improvements, bundles tool now prints Bundles:Name followed by assets included in the bundle

v0.4.3

• Fixed issues with setup.py and pyproj version.
• Improved ReadMe instructions.

v0.4.2

• Added geometry check functionality to multipolygon with shapely self intersection Issue 30.
• For multipolygons this also performs a vertex count check and simplifies polygon to fit under 500 vertices.
• General improvements

v0.4.1

• Fixed issue with shapely self intersection using buffer(0).
• General improvements

v0.4.0

• Fixed issue with placing reprojection request.
• Downloader can now download partial as well as completely successful orders.
• Added retry method for rate limit during downloading
• General improvements

v0.3.9

• Removed deprecated bundles from bundles list.
• Improved parameter description

v0.3.7

• Added capability to pass subscription id when submitting order.

v0.3.6

• Replaced asset in order tool with bundles.
• Created a new bundles tool to generate bundle list for an item type
• Improvements to the idlist tool now prints output as it makes progress.

v0.3.5

• Better integration for quota tool
• Updates information while waiting for idlist
• Updated requirements

v0.3.4

• Added async downloader for python 3.4
• Checks for existing files before spawning processes
• Better handling of multiprocessing output
• Added a quick version tool

v0.3.3

• Fixed issue with order name when no ops are used.
• Used file basename for splitting the idlist.

v0.3.2

• idlist tool can no use a multipolygon and iteratively look for scenes
• Orders clip tool can now handle multipolygon clip
• Added new tool zipall to handle single archive download in format ordername_date.zip

v0.3.1

• Can now support an additional string and range filter
• Check total area and clip area estimates from any idlist using idcheck.
• General improvements to the tool.

v0.3.0

• Enhances idlist to execute faster search and return using Planet CLI
• Included better error handling while placing order.

v0.2.8

• Added tool to convert folder with shapefiles to GeoJSONs

v0.2.7

• Improved overlap calculations for larger geometries
• Added a geometry simplification tool to reduce number of vertices

v0.2.6

• Skysat area are calculated using EPSG:3857 to resolve metadata EPSG issue
• General improvements

v0.2.5

• Fixed issue with area calculation estimates
• General improvements

v0.2.4

• Now functions without limit on the number of assets in the idlist
• Parses possible asset types if only item type is supplied for idlist

v0.2.3

• Now estimates area before and after clip when you run idlist
• General improvements

v0.2.1

• Now exports only csv idlist
• Fixed count with concurrency check

v0.2.0

• Fixed pysmartdl install issues
• Added concurrent orders check
• version and os resolve for shapely

v0.1.9

• Added msavi
• Fixed issues with GeoJSON read

v0.1.8

• Fixed issues with empty JSON append
• General improvements to the tool

v0.1.7

• Added band math indices for PlanetScope item
• Fixed issues with retry for downloader
• General improvements to the tool

v0.1.6

• Made fixes to have python 3.X compatability

v0.1.5

• General improvements and bug fixes

v0.1.4

• Fixed issue with Python 3 CSV write compatability
• Fixed issues with Shapely instance issue

v0.1.3

• Fixed issue with clipboard access in headless setup

v0.1.2

• Fixed issue and extension for multiprocessing downloader
• Overall general improvements to the tool

v0.1.0

• Fixed issue and improved idlist and sort
• Fixed issue with clip tool
• Overall general improvements to the tool

v0.0.8

• Improvements to operations in order tool
• Now supports export to gcs/azure/aws along with kernel, projection and compression
• base64 encoding tool for encoding gcs credentials
• Overall general improvements to the tool

v0.0.7

• Now allows for all downloads or download using extension
• Polling for order to complete and automatically download
• General improvements

v0.0.6

• Merged contribution by David Shean
• Fixed issues with op equals None
• Fixed issues with relative import
• Improved Py3 compatability
• General improvements

v0.0.5

• Added exponential backoff for pydl
• Fixed issues with dependency
• General overall improvements

v0.0.4

• Created strict geoinstersection to avoid orders to fail
• Improvements to overlap function
• General overall improvements

v0.0.3

• Added overlap function to idlist
• Added multiprocessing downloader with rate limit and extension filter
• General overall improvements

v0.0.2

• Fixed issues with import modules