(Command line) interface to download data batches directly from the VanderSat API
Using this module, one can get data from the VanderSat API using either:
Compatible for Linux, Mac and Windows
Python >= 3.6
This package offers an easy interface to the asynchronous endpoints offered by the VanderSat API. However, not all available endpoints can be accessed through this package.
- click
- requests
- pandas
- click_datetime
- joblib
If you don't have an environment yet or would like a new one, use the following line to make a new one using conda
$ conda create -n vds_api -c conda-forge python=3 requests "click>=7" pandas joblib pip
activate it
$ conda activate vds_api
and follow the installation steps
The package can be installed directly from PyPI. Activate your environment and then install with
$ pip install vds-api-client
With this activated environment one can access the vds cli with
$ vds-api
(If not, your installation did not succeed)
$ vds-api
will show all available commands which should include:
grid- download gridded datainfo- Show info for this accounttest- test connection, credentials and if api is operationalts- download time-series as csv over points or rois
Calling any of these commands should be done after suppliying credentials:
$ vds api -u [username] -p [password] [command]
And it is always a good idea to start with a test:
$ vds-api -u [username] -p [password] test
For each api call using the cli, the credentials need to be supplied. These can be parsed along with the call by typing them explicitly like:
$ vds-api -u [username] -p [password] [command]
However, it is also convenient to store the credentials so they don't have to be
typed each time. Set the environment variables
$VDS_USER and $VDS_PASS
with the correct values to automatically fill in your credentials.
To use Planet SSO get a token using the following procedure
Install:
pip install planet-auth-utils
Login:
Default login: opens browser for authorization and listens locally for callback.
plauth oauth login
Get Token:
plauth oauth print-access-token
For each api call using the cli, the credentials need to be supplied. These can be parsed along with the call by typing them explicitly like:
$ vds-api -t $(plauth oauth print-access-token) [command]
However, it is also convenient to store the credentials so they don't have to be
typed each time. Set the environment variable
$PL_VDS_OAUTH_TOKEN with the correct value to automatically fill in your credentials.
This can be done in one step using this command:
export PL_VDS_OAUTH_TOKEN=$(plauth oauth print-access-token)
Note
The Token will expire every 3600 seconds. If that happens during a session logging in again and re-setting the environment variable will fix it.
Note
With the envvars set, the credentials don't have to be parsed explicitly anymore thus the syntax reduces to:
$ vds-api [command]
If a user manages another VanderSat API user account, it can impersonate this user.
Through the CLI this can also be done using the --impersonate flag. e.g.
$ vds-api -u [email protected] -p password --impersonate "[email protected]" [command]
or when credentials were stored already
$ vds-api --impersonate "[email protected]" [command]
Use the help function to retrieve all options for the command line interface.
$ vds-api [command] --help
Get L-band for one month over NL in geotiff with 8 threads
$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2015-04-30 -lo 3 8 -la 50 54 -o SM_L_Data -n 8 -v
Get L+C+X-band for two dates over NL in netcdf
$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -p SM-AMSR2-C1N-DESC_V003_100 -p SM-AMSR2-XN_V003_100 -f netcdf4 -dr 2016-07-01 2016-07-02 -lo 3.0 8.0 -la 50.0 54.0 -o NCData -v
Get L-band time-series for a region-of-interest (roi) and a lat-lon pair
$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-05-01 2020-01-01 -ll 52 4.5 -r 3249 -o tsfold -v
Get time-series with all additional columns
$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2019-01-01 -ll 52 4.5 -o tsfold --masked --av_win 35 --backward --clim -t 20 -cov -v
Asynchronous requests can easily be downloaded using the VdsApiV2 class.
For downloading of the desired files, the following steps need to be taken:
- For the version 2 api, three steps have to be taken to download data from the api which are all methods of the
VdsApiV2class: - Generate a request
- Configure gridded data download or time-series download
through one of
gen_time_series_requests()orgen_gridded_data_request()
- Submit request
- After generating all desired URIs, submit these with
submit_async_requests()to start the processing of these jobs
- Download files
- Get all data using
download_async_files()
Initialize class
from vds_api_client import VdsApiV2
# Choose one of the following options to initialize
vds = VdsApiV2('username', 'password')
vds = VdsApiV2() # extract login from $VDS_USER and $VDS_PASSImpersonate user
When a user manages another account, it can impersonate this managed acount which means that all requests will be done as if the impersonated user has made them
vds = VdsApiV2('[email protected]', 'password')
# Start impersonation
vds.impersonate('[email protected]')
# do_requests
# End impersonation
vds.forget()Gridded data example [asynchronous]
Request raster data using the products/<api_name>/gridded-data endpoint
from vds_api_client import VdsApiV2
vds = VdsApiV2()
vds.set_outfold('testdata/tiff') # Created if it does not exist
vds.gen_gridded_data_request(products=['SM-SMAP-LN-DESC_V003_100', 'SM-AMSR2-XN-DESC_V003_100'],
start_date='2015-10-01', end_date='2016-09-30',
lat_min=-3.15, lat_max=-1.5, lon_min=105, lon_max=107,
nrequests=4)
vds.submit_async_requests()
vds.download_async_files()
# Get information on the downloaded files
vds.summary()Time-series example [asynchronous]
Request time-series data using the products/<api_name>/[point|roi]-time-series endpoints.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
vds.set_outfold('testdata/csv') # Created if it does not exist
vds.gen_time_series_requests(products=['SM-XN_V001_100'],
start_time='2018-01-01', end_time='2018-01-03',
lons=[6.5], lats=[41.5], rois=[527, 811])
vds.submit_async_requests()
vds.download_async_files()
# Get information on the downloaded files
vds.summary()Notice that the lons and lats are given in a list. When multiple points are defined, the latitude and longitude pairs can be added to the single lists like this:
lons=[6.5, 7.5], lats=[41.5, 45]and they will be processed in parallel.
Re-download previous requests
Re-download data using previously generated uuids. Note that data is not stored indefinitely, but within 7 days you should be able to re-download your data.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Choose from
vds.uuids.append('5742540a-cf87-49dd-a6e7-d484de137324')
vds.queue_uuids_files()
# or
vds.queue_uuids_files(uuids=['57f9950a-4e41-49dd-a6e7-d484de137324'])Get a single point value
Extract a single value based on a product-coordinate using the products/<api-name>/point-value endpoint
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Load using the roi-id
val = vds.get_value('SM-XN_V001_100', '2020-04-01', lon=20.6, lat=40.4)Load Roi time-series as pandas dataframe [synchronous]
Request roi time-series data using the products/<api_name>/roi-time-series-sync endpoint and load the result as a pandas.DataFrame
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Load using the roi-id
df1 = vds.get_roi_df('SM-XN_V001_100', 2464, '2016-01-01', '2018-12-31')
# Load using the roi-name
df2 = vds.get_roi_df('SM-XN_V001_100', 'MyArea', '2016-01-01', '2018-12-31')Knowing and using the regions of interest (rois) attached to your account is now easier using the client methods that allow you to filter the rois.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
print(vds.rois)# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 25009 / [X] | Center | 1.063e+05 ha | 2020-08-16 12:49 | Center pixels 25010 / [X] | Right | 9.949e+04 ha | 2020-08-16 12:58 | Right side pixels 25011 / [X] | Bottom | 6.616e+04 ha | 2020-08-16 12:59 | Bottom side pixels 30596 / [ ] | NewName | 9.140e+03 ha | 2020-09-18 07:19 | Same rectangle
Filters
But now, also filters can be applied to select Rois based on a criterium, and give the corresponding ids:
rois_filtered = vds.rois.filter(
min_id=25000, max_id=25020,
area_min=1e8, area_max=1e9,
name_regex='Right|Bottom', description_regex='pixels',
created_before=dt.datetime(2020, 8, 16, 13, 0),
created_after=dt.datetime(2020, 8, 16, 12, 57),
display=True)
print(rois_filtered)
print(rois_filtered.ids_to_list())# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 25010 / [X] | Right | 9.949e+04 ha | 2020-08-16 12:58 | Right side pixels 25011 / [X] | Bottom | 6.616e+04 ha | 2020-08-16 12:59 | Bottom side pixels [25010, 25011]
Geometry
Accessing the geometry is now supported through the geojson property:
roi = vds.rois[25010]
geojson = roi.geojson # Loads geometry from api
print(geojson)
{'type': 'MultiPolygon', 'coordinates': [[[[-5.237732, 66.044796], [-5.237732, 66.956952], [-5.018005, 66.956952], [-5.018005, 66.044796], [-5.237732, 66.044796]]]]}Updating
Updating an Roi's metadata is supported through the roi.update method:
roi = vds.rois[30596]
roi.update(name='New name', description='New description', display=False)
print(vds.rois.filter(name_regex='New name'))# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 30596 / [ ] | New name | 9.140e+03 ha | 2020-09-18 07:19 | New description
Deleting
Deleting ROIS from your account is supported through the delete_rois_from_account() method. It expects a list of integers, or a filtered Rois instance. Now we can delete our Rois quite easily like:
vds.delete_rois_from_account(vds.rois.filter(description_regex='Selection to Delete'))