Order Up: Using and Building with Planet ’s new Orders v2 API
Tim Berners-Lee wrote
“It’s difficult to imagine the power that you’re going to have when so many different sorts of data are available.”
Tell me a story about why we need better tools and I will tell you while some good research starts with good data, almost all research must start with getting the data first. It doesn’t matter if you use remote sensing to solve puzzles of the earth system, or simply likes a good time-lapse or even stitches endless pieces of datasets together, “all data must start with gathering”. Imagine if satellite data could be structured, if it could be delivered to you, already clipped and reprojected and trimmed and made analysis ready for you. Imagine your order could speak to you, tell you in an email when it has completed as you work on something else. I think combining these and many more functionalities went into orders v2 API and as an early user, I got to play around with it and see it evolve over a period of time. As per me, their message was simple from gathering we went to
All data must start with ordering
This tutorial is in no way an overall synopsis of the ordersv2 API for that you can head here. But this was my attempt at using the API and also introducing the new command line tool I built around it to build toolchains. If you are confused about tools and toolchains, don’t worry I will introduce them later or you can head over here to read about it here
Setting the Stage
No, the Data API is not going away, it is still one of the best ways to search for and query data, and you can still use it to activate and download data. But if you are doing this for hundreds of datasets, if you would like to have datasets in bundles (a combination of all assets belonging to an item) you can now use the new Orders v2 API to run this.
For those who have used the Clips API from Planet and enjoyed it might also be aware that Clips API was deprecated in April 2018. So this includes your favorite tools to clip and ship as well along with some premium tools and features. In the next couple of paragraphs, I will highlight some of the tools and toolchains that I found useful for my own work but then again you can read the reference for me.
Building with the ordersv2 API
Now to work with this atleast in a programmatic interface you need to make yourself aware of an Application Programming Interfaces or API. In general you can get information about API request types here and read a detailed introduction to an API from restful here.
At this point let me tell you where we are headed
- You need to have python installed on your system so you can pull and use the command line interface I made and for debugging if you need it.
- Read through API and get a general idea about what it is if you want to understand how the tool works.
- That being said I have already built the tool so you don’t really need to build much yourself but you still need python and it should be accessible in Command prompt in your windows environment. Linux and Mac come prebuilt with python.
- I have built the tool using Python 2.7.15 and I had no major issues upto Python 3.6.6.
- Throughout the tutorial I will introduce some terms about the ordersv2 API and talk more about the Command Line Interface (CLI) I built to use it.
Setup and prerequsities
Things you will need to get setup includes a few things
- Make sure that Python is installed and accessible in your command prompt or terminal (Windows users note we are using command prompt and not the powershell if in doubt
use windows key+r/Run, followed by cmd and hit return) - Planet’s own CLI which allows you to interact with the data API
- This is important Initialize Planet CLI, type
planet initin your command prompt or terminal and enter your email and password to initialize.
- Shapely is notoriously difficult as a library to install on windows machines so follow the steps mentioned from Shapely’s PyPI package page
- You can download and install from the Unofficial Wheel files from here download depending on the python version you have. You will get a wheel file or a file ending with .whl. You can now simply browse to the folder or migrate to it in your command prompt , for example in my case I was running Python 2.7.15 and win32 version so the command was
pip install Shapely-1.6.4.post1-cp27-cp27m-win32.whl - Or you can use anaconda to install. Again, both of these options are mentioned on Shapely’s Official PyPI page
Product Bundles
Product bundles are a combination of items and asset types together, designed to make the experience of downloading, images and associates files seamless. To give you an example when you download a PSScene4Band analytic image, you often also download the metadata file which is an xml file and the associated unusable data mask (udm) file and the product bundle configures this for you and allow you to order them all together.
For example for Item type PSScene4Band and asset type analytic the product bundle includes
"PSScene4Band": ["analytic","analytic_xml","udm"]For PSScene4Band and asset type analytic_sr (surface reflectance) the product bundle includes
"PSScene4Band": ["analytic_sr","udm","analytic_xml"]
To find out about all product bundles you can head over and find them here
Toolbox and Toolchains
The toolbox comes with a few tools that everyone has access to, apart from being able to simply order the datasets, you can choose to make it analysis ready or preprocess it to some degree. One of the most sought after tool by many and one that is included by default for all users is the Clip tool which would effectively replace the deprecated Clips API and performs this on all your images before you download them saving you on your quota. There is an entire toolbox that is included with Ordersv2 API, apart from the default tools that are beyond the Education and Research License users. Some of the additional tools include
- Top of Atmosphere reflectance( TOAR) from the radiance imagery
- Composite as the name suggested is to composite multiple images into one
- Reproject is another useful one to do just the same reproject before downloading them and one of my favorite and powerful ones is
- Band Math where you can perform simple band math operations on all images in the stack of images you want to download. So you may be able to download indices such as NDVI directly instead of raw imagery.
The choice of tools are important but more important is the order in which these tools should be tied together. Read the instructions here about creating a valid tool chain.
The idea of building something around ordersv2 API came with the potential to explore, tie and work with complex orders using a simple command line interface (CLI). You can explore the simple setup using the payloads in the readme and go through the API and place post calls and get requests using something like Postman. This tool was created to tie multiple operations together within the framework of what tools might be available to Education and Research users (at the time of writing this story you should have access to clip,zip and email notification) as well as users with additional tools based on their license type. You can read about and find the tool on GitHub here.
You can install using two methods
pip install porder
on Ubuntu I found it helps to specify the pip type and use sudo
sudo pip2 install porder or sudo pip3 install porderor you can also try the following (It might require sudo privileges for Linux users)
git clone https://github.com/samapriya/porder.git
cd porder
python setup.py installOnce the setup is completed you should be able to access porder in your terminal by typing porder -h
Get your quota straight
To make sure you have enough quota left before placing an order, use this to estimate the area left. The API will run a check against your quota as well before order is complete. The tool will print the following information by using porder quota
Allocation Name:
Allocation active from:
Quota Enabled:
Total Quota in SqKm:
Total Quota used:
Remaining Quota in SqKm:Next up: Get your ID list or Difference ID List
The next step is to interact with Data API and get an idlist to be submitted in the order, there are two tools that I have included here. The first one is idlist which as the name suggests is a tool to generate the IDlist for a given geometry in combination with date and cloud filters. This is the first step and constitutes the core of the order you want to place.
A simple setup would be
porder idlist --input "C:\johndoe\geometry.geojson" --start "2017-01-01" --end "2018-12-31" --item "PSScene4Band" --asset "analytic_sr" --number 800 --outfile "C:\johndoe\orderlist.csv"But maybe you have already downloaded some images to your computer or a local system, and though the downloader can skip re-downloading these images, it is better to skip these while placing the order all together. The difflist tool allows you to check with files you have already downloaded and creates a difference ID list with only those ID(s) that you do not already have. This tool has been designed to work with complete images only and not clips for now.
A simple setup would be
porder difflist --folder "F:\johndoe\ps4b_xml" --typ "metadata" --input "F:\johndoe\grid.geojson" --item "PSScene4Band" --asset "analytic_sr" --start "2018-06-01" --end "2018-08-01" --cmin 0 --cmax 0.9 --outfile "F:\johndoe\diff.txt"Split the List: Use if Needed
Sometimes it is possible that your list is too dry and there are limits on the maximum number of items IDs that could be included in a list. I am hoping these limits will be included in the documentation somewhere, for now you can choose to create smaller orders or split the IDlist into equal parts.
A simple setup would be
porder idsplit --idlist "C:\johndone\orderlist.csv" --lines "100" --local "C:\johndoe\split"Ordering Now
This tool is designed to take in multiple arguments and chain together tools. The --op argument allows you to take operations, delivery and notifications in a sequence for example --op clip email performs clip and sends you an email notification once the order has completed, failed or had any any change of status. For now clip, Top of Atmospheric Reflectance (toar), composite, zip and email tools are included from the tool list. Which tool is available to you, depends on your license type
Once you place the order the order url is generated and copied to your clipboard so you can paste it in your browser if you would like. A simple setup to clip and ship with email notification for example would be
porder order --name "testorder" --idlist "idlistfile.txt" --item "PSScene4Band" --asset "analytic" --boundary "clip_boundary.geojon" --op clip email The link would look something like this(not a real link) and use your API key as the username and leave the password blank
https://api.planet.com/compute/ops/orders/v2/22015422-8608-48e7-9456-b84a439da91e
Save it since you need it to download the files using the download or multipart download tools. Incase you placed your order and lost your download url, you can search based on your download name. All your orders are listed at the API endpoint.
https://api.planet.com/compute/ops/orders/v2Download and Multipart Download
These two tools allows you to download the files to your local drive, the first tool which is download only does this sequentially and uses chunks to download your order, the multipart tool uses a library to split each file into parts for faster download.
The multipart download setup is the same as download setup
There you have it, you just placed and order, and downloaded it. Depending on the size of your order the time required to download can vary, but you can also write your own version of a downloader.
Next iterations of the tool, I am hoping to include a downloader that employs multiprocessing to download quicker and also create an overlap tool that checks the minimum percentage overlap an image has with your geometry before creating the orderlist.
Please cite the tool, or star it on GitHub if you find it useful and use it as it allows me to understand usage and develop better tools.
Samapriya Roy. (2018, November 26). samapriya/porder: porder: Simple CLI for Planet ordersV2 API (Version 0.0.1). Zenodo. http://doi.org/10.5281/zenodo.1534692If you find this guide useful please click the clap 👏 button a couple of times below to show your support (You can click more than once :D ) ! and if you have used it to recreate something or to create better workflow, let me know.
