From 7ccde28152f2c52b252e1164811a91dc3c57ef87 Mon Sep 17 00:00:00 2001 From: aversnik Date: Fri, 17 Jan 2025 11:28:06 +0100 Subject: [PATCH] Add byoc processing example page. Sync documentation --- APIs/SentinelHub/BatchStatistical.qmd | 2 +- APIs/SentinelHub/BatchV2.qmd | 118 ++++++++++------- APIs/SentinelHub/Data/Byoc.qmd | 7 +- APIs/SentinelHub/Evalscript/Functions.qmd | 22 ++-- APIs/SentinelHub/Overview/ProcessingUnit.qmd | 24 +++- APIs/SentinelHub/Process/Examples/BYOC.qmd | 128 +++++++++++++++++++ _quarto.yml | 2 + 7 files changed, 234 insertions(+), 69 deletions(-) create mode 100644 APIs/SentinelHub/Process/Examples/BYOC.qmd diff --git a/APIs/SentinelHub/BatchStatistical.qmd b/APIs/SentinelHub/BatchStatistical.qmd index 00c3a65df..ddc3c63d5 100644 --- a/APIs/SentinelHub/BatchStatistical.qmd +++ b/APIs/SentinelHub/BatchStatistical.qmd @@ -142,7 +142,7 @@ request, the input GeoPackage is specified by setting the path to the All features (polygons) in an input GeoPackage must be in the same CRS [supported by Sentinel Hub](/APIs/SentinelHub/Process/Crs.qmd). There -can be a maximum of 700,000 features in the GeoPackage. +can be a maximum of 700.000 features in the GeoPackage. An example of a GeoPackage file can be downloaded [here](/APIs/SentinelHub/BatchStatistical/resources/gpkg/geopackage-example.gpkg). diff --git a/APIs/SentinelHub/BatchV2.qmd b/APIs/SentinelHub/BatchV2.qmd index 9f601d2b7..2002b9f44 100644 --- a/APIs/SentinelHub/BatchV2.qmd +++ b/APIs/SentinelHub/BatchV2.qmd @@ -122,13 +122,13 @@ batch task: 2. User-defined [GeoPackage](/APIs/SentinelHub/BatchV2.qmd#2-geopackage) -### 1. Tiling grid {#1-tiling-grid} +### 1. Tiling Grid {#1-tiling-grid} For more effective processing we divide the area of interest into tiles and process each tile separately. While `process` API uses grids which come together with each datasource for processing of the data, the -`batch` API uses one of the predefined tiling grids. The predefined -tiling grids 0-2 are based on the [Sentinel-2 +`batch` API uses one of the predefined tiling grids. The tiling grids +0-2 are based on the [Sentinel-2 tiling](https://sentinel.esa.int/web/sentinel/missions/sentinel-2/data-products){target="_blank"} in WGS84/UTM projection with some adjustments: @@ -175,7 +175,8 @@ as: ``` JSON { ... - "tilingGrid": { + "input": { + "type" : "tiling-grid", "id": 0, "resolution": 60.0 }, @@ -185,7 +186,7 @@ as: ### 2. GeoPackage {#2-geopackage} -In addition to the predefined tiling grids, BatchV2 API now also support +In addition to the tiling grids, BatchV2 API now also support user-defined features through [GeoPackages](https://www.geopackage.org/spec/){target="_blank"}. This allows you to specify features of any shape as long as the underlying @@ -202,14 +203,14 @@ arbitrarily, but it must be listed as the geometry column in the `gpkg_geometry_columns` table. The table schema should include the following columns: - Column Type Example - ------------ ------------------------- ---------------------------------------------------------- - id INTEGER **(UNIQUE)** 1000 - identifier TEXT **(UNIQUE)** FEATURE_NAME - geometry POLYGON or MULTIPOLYGON Feature geometry representation in GeoPackage WKB format - width INTEGER 1000 - height INTEGER 1000 - resolution REAL 0.005 + Column Type Example + ------------------ ------------------------- ---------------------------------------------------------- + id - primary key INTEGER **(UNIQUE)** 1000 + identifier TEXT **(UNIQUE)** FEATURE_NAME + geometry POLYGON or MULTIPOLYGON Feature geometry representation in GeoPackage WKB format + width INTEGER 1000 + height INTEGER 1000 + resolution REAL 0.005 #### Caveats @@ -217,13 +218,57 @@ following columns: specify resolution. If both values are provided, width and height will be used, and resolution will be ignored. - The feature table must use a CRS that is **EPSG compliant**. -- Both `id` and `identifier` values must not be null and unique across - all feature tables. -- There can be a maximum of 700,000 features in the GeoPackage. +- `identifier` values must not be null and unique across all feature + tables. +- There can be a maximum of 700.000 features in the GeoPackage. +- The feature output width and height cannot exceed 3500 by 3500 + pixels or the equivalent in resolution. + +Below you will find a list of example GeoPackages that serve as a +showcase of how a GeoPackage file should be structured. Please note that +these examples do not serve as production-ready GeoPackages and should +only be used for testing purposes. If you\'d like to use these tiling +grids for processing, use the equivalent tiling grid with the tiling +grid input instead. + + name id output CRS geopackage + --------------------- ---- ------------ ----------------------------------------------------------------------------------------------------------------- + UTM 20km grid 0 UTM [UTM 20km grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-0.gpkg){target="_blank"} + UTM 10km grid 1 UTM [UTM 10km grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-1.gpkg){target="_blank"} + UTM 100km grid 2 UTM [UTM 100km grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-2.gpkg){target="_blank"} + WGS84 1 degree grid 3 WGS84 [WGS84 1 degree grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-3.gpkg){target="_blank"} + LAEA 100km grid 6 EPSG:3035 [LAEA 100km grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-6.gpkg){target="_blank"} + LAEA 20km grid 7 EPSG:3035 [LAEA 20km grid](https://s3.eu-central-1.amazonaws.com/sh-batch-grids/tiling-grid-7.gpkg){target="_blank"} An example of a batch task with GeoPackage input is available [here](/APIs/SentinelHub/BatchV2/Examples.qmd#option-3-geopackage-input-and-geotiff-output). +### Area of Interest and PUs + +When using either [Tiling +Grid](/APIs/SentinelHub/BatchV2.qmd#1-tiling-grid) or +[GeoPackage](/APIs/SentinelHub/BatchV2.qmd#2-geopackage) as input, the +features that end up being processed are determined by the +`processRequest.input.bounds` parameter specified in the request, called +Area of Interest or AOI. + +The way the AOI parameter is used and its effect depend on the input +type used: + +- Tiling grid: The AOI **must** be specified in the request. Only the + tiles (features) that intersect with the AOI will be processed. +- GeoPackage: The AOI can optionally be omitted. If the AOI is + omitted, all the features inside your GeoPackage will be processed. + Conversely, if AOI is specified, only the features that intersect + with the AOI will be processed. + +Please note that in both cases of input types, if the feature is only +**partially** covered by the AOI, the feature will be processed in its +**entirety**. + +You are only charged PUs for the features that are processed. If a +feature does not intersect with the AOI, it will not be charged for. + ------------------------------------------------------------------------ ## Processing results @@ -356,32 +401,11 @@ When creating a new batch collection, one has to be careful to: #### Mandatory bucket settings -Regardless of the credentials provided on the request, your bucket needs -to be configured to allow full access to Sentinel Hub. To do this, -update your bucket policy to include the following statement (don\'t -forget to replace \ with your actual bucket name): - -``` JSON -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "Sentinel Hub permissions", - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam::ddf4c98b5e6647f0a246f0624c8341d9:root" - }, - "Action": [ - "s3:*" - ], - "Resource": [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" - ] - } - ] -} -``` +Regardless of the credentials provided in the request, you still need to +set a bucket policy to allow Sentinel Hub services to access the data. +For detailed instructions on how to configure your bucket policy, please +refer to the [BYOC bucket settings +documentation](/APIs/SentinelHub/Byoc.qmd#aws-bucket-settings). ## Feature Manifest @@ -421,11 +445,13 @@ e.g. `feature_4326`. `
`{=html}The schema of feature tables inside the database is currently the following: Name Type Description - ------------ ---------- ------------------------------------------------------------------- + ------------ ---------- ---------------------------------------------------------------------------------- fid INTEGER Auto-incrementing ID - id INTEGER Numerical ID of the feature - identifier TEXT Textual ID of the feature - path TEXT The object storage path URI where the feature will be uploaded to + outputId TEXT Output identifier defined in the `processRequest` + identifier TEXT ID of the feature + path TEXT The object storage path URI where the output of this feature will be uploaded to + width INTEGER Width of the feature in pixels + height INTEGER Height of the feature in pixels geometry GEOMETRY Feature geometry representation in GeoPackage WKB format ------------------------------------------------------------------------ diff --git a/APIs/SentinelHub/Data/Byoc.qmd b/APIs/SentinelHub/Data/Byoc.qmd index f78a8c292..e50c711b3 100644 --- a/APIs/SentinelHub/Data/Byoc.qmd +++ b/APIs/SentinelHub/Data/Byoc.qmd @@ -112,10 +112,9 @@ Here is an example of a WFS `GetFeature` request: ## Catalog API Capabilities -To access Landsat BYOC or Batch product metadata you need to send search -request to our [Catalog API](/APIs/SentinelHub/Catalog.qmd). The -requested metadata will be returned as JSON formatted response to your -request. +To access BYOC product metadata you need to send search request to our +[Catalog API](/APIs/SentinelHub/Catalog.qmd). The requested metadata +will be returned as JSON formatted response to your request. ### Collection identifier: \ diff --git a/APIs/SentinelHub/Evalscript/Functions.qmd b/APIs/SentinelHub/Evalscript/Functions.qmd index baea13e07..fc305d39f 100644 --- a/APIs/SentinelHub/Evalscript/Functions.qmd +++ b/APIs/SentinelHub/Evalscript/Functions.qmd @@ -108,9 +108,9 @@ const ramps = [ const visualizer = new ColorRampVisualizer(ramps); visualizer.process(199); // [ 1, 0, 0 ] visualizer.process(200); // [ 1, 0, 0 ] -visualizer.process(250); // [ 0.5019607843137255, 0, 0.5019607843137255 ] -visualizer.process(299); // [ 0.011764705882352941, 0, 0.9882352941176471 ] -visualizer.process(300); // [ 0, 1, 0 ] +visualizer.process(250); // [ 0.4980392156862745, 0, 0.4980392156862745 ] +visualizer.process(299); // [ 0.00784313725490196, 0, 0.9882352941176471 ] +visualizer.process(300); // [ 0, 0, 1 ] ``` #### inverse @@ -152,9 +152,9 @@ Creates ColorRampVisualizer with valColPairs ``` javascript const visualizer = ColorRampVisualizer.createRedTemperature(0.0, 1.0); visualizer.process(0.0); // returns [ 0, 0, 0 ] -visualizer.process(0.3); // returns [ 0.43137254901960786, 0, 0 ] -visualizer.process(0.5); // returns [ 0.7176470588235294, 0.047058823529411764, 0 ] -visualizer.process(0.8); // returns [ 1, 0.6196078431372549, 0.2 ] +visualizer.process(0.3); // returns [ 1, 0.43137254901960786, 0 ] +visualizer.process(0.5); // returns [ 1, 0.5137254901960784, 0 ] +visualizer.process(0.8); // returns [ 1, 0.8, 0.5764705882352941 ] visualizer.process(1.0); // returns [ 1, 1, 1 ] ``` @@ -178,9 +178,9 @@ Creates ColorRampVisualizer with valColPairs [greenWhite](#greenwhite) ``` javascript const visualizer = ColorRampVisualizer.createWhiteGreen(0.0, 1.0); visualizer.process(0.0); // returns [ 0, 0, 0 ] -visualizer.process(0.3); // returns [ 0, 0.2980392156862745, 0 ] -visualizer.process(0.5); // returns [ 0.16862745098039217, 0.5019607843137255, 0 ] -visualizer.process(0.8); // returns [ 0.6666666666666666, 0.8, 0.3333333333333333 ] +visualizer.process(0.3); // returns [ 0, 0.4, 0 ] +visualizer.process(0.5); // returns [ 0.3333333333333333, 0.6, 0 ] +visualizer.process(0.8); // returns [ 0.7490196078431373, 0.8509803921568627, 0.4980392156862745 ] visualizer.process(1.0); // returns [ 1, 1, 1 ] ``` @@ -204,8 +204,8 @@ Creates ColorRampVisualizer with valColPairs [blueRed](#bluered) ``` javascript const visualizer = ColorRampVisualizer.createBlueRed(0.0, 1.0); visualizer.process(0.0); // returns [ 0, 0, 0.5019607843137255 ] -visualizer.process(0.3); // returns [ 0, 0.7019607843137254, 1 ] -visualizer.process(0.5); // returns [ 0.5019607843137255, 1, 0.5019607843137255 ] +visualizer.process(0.3); // returns [ 0, 0.6980392156862745, 1 ] +visualizer.process(0.5); // returns [ 0.4980392156862745, 1, 0.4980392156862745 ] visualizer.process(0.8); // returns [ 1, 0.2980392156862745, 0 ] visualizer.process(1.0); // returns [ 0.5019607843137255, 0, 0 ] ``` diff --git a/APIs/SentinelHub/Overview/ProcessingUnit.qmd b/APIs/SentinelHub/Overview/ProcessingUnit.qmd index 333589035..d8f4bb93a 100644 --- a/APIs/SentinelHub/Overview/ProcessingUnit.qmd +++ b/APIs/SentinelHub/Overview/ProcessingUnit.qmd @@ -135,13 +135,23 @@ apply with the following exceptions: same system (i.e. Copernicus Data Space Ecosystem, AWS) there is an additional cost of 0.03 PU per MB of data. -## Third party data order - applicable to Third Party Data Import API - -- \*\* Each search request costs 1 PU. -- \*\* Each thumbnail request costs 1 PU. -- \*\* Each created order/subscription costs 5 PU. -- \*\* Each processed order delivery costs 5 PU. -- \*\* Each processed subscription delivery costs 2 PU. +## Data ordering and delivery - applicable to Orders API, Subscriptions API and Third Party Data Import API + +- \*\* Each image requested costs 20 PUs. If an image covers multiple + of your areas of interest, then it is activated only once within a + 24-hour time window. For example, if you have numerous small + agriculture fields which are close to each other and therefore + covered by one satellite image. +- \*\* Using the clip tool costs 1 PU per data asset. +- \*\* Each tool used, per data asset, costs 2 PUs. Applies to the + following tools: Band Math, Composite, Coregister, Harmonization, + Reproject, Tile, TOAR. To learn more, see documentation for [Planet + Orders + API](https://developers.planet.com/apis/orders/tools/){target="_blank"} + and [Planet Subscriptions + API](https://developers.planet.com/docs/subscriptions/tools/#supported-tools){target="_blank"}. +- \*\* Each GB egressed off the platform (i.e. downloaded or delivered + to your cloud) costs 200 PUs. ## Data ingestion - applicable to Bring your own COG API and Zarr API diff --git a/APIs/SentinelHub/Process/Examples/BYOC.qmd b/APIs/SentinelHub/Process/Examples/BYOC.qmd new file mode 100644 index 000000000..de514ed7b --- /dev/null +++ b/APIs/SentinelHub/Process/Examples/BYOC.qmd @@ -0,0 +1,128 @@ +--- +title: Examples for BYOC +--- + +The requests below are written in python. To execute them you need to +create an OAuth client as is explained +[here](/APIs/SentinelHub/Overview/Authentication.qmd#python). It is +named `oauth` in these examples. + +### Check collection bands in catalog + +This example uses the publicly available [Sentinel-2 Level 3 Quarterly +Mosaics](/Data/SentinelMissions/Sentinel2.html#sentinel-2-level-3-quarterly-mosaics) +collection. To use your own collection, replace the ID used in the URL +with `byoc-`. + +``` python +url = "https://sh.dataspace.copernicus.eu/api/v1/catalog/1.0.0/collections/byoc-5460de54-082e-473a-b6ea-d5cbe3c17cca" +response = oauth.get(url) +``` + +Bands can be found in the returned json object under +`summaries.eo:bands`. An example of the returned `summaries` object that +can be found in the above request\'s response: + +``` json +{ + "eo:bands": [ + { + "name": "B02" + }, + { + "name": "B03" + }, + { + "name": "B04" + }, + { + "name": "B08" + }, + { + "name": "observations" + } + ], + "raster:bands": [ + { + "data_type": "int16", + "nodata": -32768 + }, + { + "data_type": "int16", + "nodata": -32768 + }, + { + "data_type": "int16", + "nodata": -32768 + }, + { + "data_type": "int16", + "nodata": -32768 + }, + { + "data_type": "uint16", + "nodata": 0 + } + ] +} +``` + +### Create a True Color image from the gathered bands + +As mentioned above this example uses an [Sentinel-2 Level 3 Quarterly +Mosaics](/Data/SentinelMissions/Sentinel2.html#sentinel-2-level-3-quarterly-mosaics) +collection. To use your own collection, replace the collection ID in +`input.data.type`, list the desired input and output bands in the +[`setup function`](/APIs/SentinelHub/Evalscript/V3.qmd#setup-function) +(you can use any subset of the input bands found in the `summaries` +object for your collection), and then map the input bands to the output +raster in the +[`evaluatePixel function`](/APIs/SentinelHub/Evalscript/V3.qmd#evaluatepixel-function). + +``` python +curl -X POST https://sh.dataspace.copernicus.eu/api/v1/process -H 'Authorization: Bearer ' -F 'request={ + "input": { + "bounds": { + "bbox": [ + 12.44693, + 41.870072, + 12.541001, + 41.917096 + ] + }, + "data": [ + { + "type": "byoc-5460de54-082e-473a-b6ea-d5cbe3c17cca", + "dataFilter": { + "timeRange": { + "from": "2023-01-01T00:00:00Z", + "to": "2023-01-02T23:59:59Z" + } + } + } + ] + }, + "output": { + "width": 780, + "height": 523, + "responses": [ + { + "identifier": "default", + "format": { + "type": "image/jpeg" + } + } + ] + } +}' -F 'evalscript=//VERSION=3 +function setup() { + return { + input: ["B02", "B03", "B04"], + output: { bands: 3 } + }; +} + +function evaluatePixel(sample) { + return [2.5 * sample.B04/10000, 2.5 * sample.B03/10000, 2.5 * sample.B02/10000]; +}' +``` diff --git a/_quarto.yml b/_quarto.yml index 20b58d2b3..0e9873050 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -241,6 +241,8 @@ website: text: Examples for DEM - href: "APIs/SentinelHub/Process/Examples/DataFusion.qmd" text: Examples for DataFusion + - href: "APIs/SentinelHub/Process/Examples/BYOC.qmd" + text: Examples for BYOC - section: "Asynchronous Processing API" href: "APIs/SentinelHub/AsyncProcess.qmd" contents: