Skip to content

Commit

Permalink
Nsidc access (#124)
Browse files Browse the repository at this point in the history
  • Loading branch information
@asteiker
asteiker authored Jul 13, 2021
1 parent 85cc73d commit 6a72059
Show file tree
Hide file tree
Showing 3 changed files with 499 additions and 70 deletions.
42 changes: 21 additions & 21 deletions book/_toc.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,29 +27,29 @@
sections:
- file: tutorials/day_1
sections:
- file: tutorials/jupyter
- file: tutorials/core-datasets/index
sections:
- file: tutorials/core-datasets/01_introduction
- file: tutorials/core-datasets/02_data-package
- file: tutorials/core-datasets/03_practice-querying
- file: tutorials/jupyter
- file: tutorials/core-datasets/index
sections:
- file: tutorials/core-datasets/01_introduction
- file: tutorials/core-datasets/02_data-package
- file: tutorials/core-datasets/03_practice-querying
- file: tutorials/day_2
sections:
- file: tutorials/geospatial/index
sections:
- file: tutorials/geospatial/raster
- file: tutorials/geospatial/vector
- file: tutorials/geospatial/SNOTEL_query
- file: tutorials/nsidc-access/nsidc-data-access
- file: tutorials/database/index
sections:
- file: tutorials/database/1_getting_started_example
- file: tutorials/database/2_database_structure
- file: tutorials/database/3_forming_queries
- file: tutorials/database/4_get_spiral_example
- file: tutorials/database/5_plot_raster_example
- file: tutorials/database/6_exporting_data
- file: tutorials/database/7_wrap_up
- file: tutorials/geospatial/index
sections:
- file: tutorials/geospatial/raster
- file: tutorials/geospatial/vector
- file: tutorials/geospatial/SNOTEL_query
- file: tutorials/nsidc-access/nsidc-data-access
- file: tutorials/database/index
sections:
- file: tutorials/database/1_getting_started_example
- file: tutorials/database/2_database_structure
- file: tutorials/database/3_forming_queries
- file: tutorials/database/4_get_spiral_example
- file: tutorials/database/5_plot_raster_example
- file: tutorials/database/6_exporting_data
- file: tutorials/database/7_wrap_up
- file: tutorials/day_3
sections:
- file: tutorials/lidar/index
Expand Down
123 changes: 74 additions & 49 deletions book/tutorials/nsidc-access/nsidc-data-access.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@
"Author: Amy Steiker, NSIDC DAAC, CIRES, University of Colorado\n",
"\n",
"\n",
"This tutorial provides a brief overview of the resources provided by the NASA National Snow and Ice Data Center Distributed Active Archive Center, or NSIDC DAAC, and demonstrates how to access SnowEx data across several methods. \n",
"This tutorial provides a brief overview of the resources provided by the NASA National Snow and Ice Data Center Distributed Active Archive Center, or NSIDC DAAC, and demonstrates how to disscovery and access SnowEx data programmatically. \n",
"\n",
"## Learning Objectives:\n",
"### Learning Objectives:\n",
"\n",
"1. Identify resources available at the NSIDC DAAC to help you work with SnowEx data. \n",
"2. Search and discover file size and coverage of SnowEx data over a time and geojson region of interest.\n",
"3. Set up Earthdata Login authentication.\n",
"4. Download SnowEx data programmatically using the NSIDC DAAC API."
"3. Set up Earthdata Login authentication needed to access NASA Earthdata.\n",
"4. Download SnowEx data programmatically using the NSIDC DAAC Application Programming Interface (API)."
]
},
{
Expand All @@ -41,6 +41,8 @@
"\n",
"#### Select Data Resources\n",
"\n",
"* [NSIDC Data Tutorials](https://github.com/nsidc/NSIDC-Data-Tutorials)\n",
" * Jupyter Notebook tutorials providing guidance on working with various NSIDC DAAC data products, including how to access, subset, transform, and visualize data.\n",
"* [NSIDC Data Search](https://nsidc.org/data/search/#keywords=snow)\n",
" * Search NSIDC snow data\n",
"* [NSIDC Data Update Announcements](https://nsidc.org/the-drift/data-update/) \n",
Expand All @@ -56,7 +58,7 @@
"\n",
"[Snow Today](https://nsidc.org/snow-today), a collaboration with the University of Colorado's Institute of Alpine and Arctic Research (INSTAAR), provides near-real-time snow analysis for the western United States and regular reports on conditions during the winter season. Snow Today is funded by NASA Hydrological Sciences Program and utilizes data from the Moderate Resolution Imaging Spectroradiometer (MODIS) instrument and snow station data from the Snow Telemetry (SNOTEL) network by the Natural Resources Conservation Service (NRCS), United States Department of Agriculture (USDA) and the California Department of Water Resources: www.wcc.nrcs.usda.gov/snow.\n",
"\n",
"#### NSIDC's SnowEx pages\n",
"### NSIDC's SnowEx pages\n",
"\n",
"* [SnowEx](https://nsidc.org/data/snowex)\n",
" * We will be using the 'SnowEx17 Ground Penetrating Radar, Version 2' dataset in the following steps: https://doi.org/10.5067/G21LGCNLFSC5\n",
Expand All @@ -81,7 +83,7 @@
"metadata": {},
"source": [
"___\n",
"## Import Packages\n",
"### Import Packages\n",
"\n",
"Get started by importing packages needed to run the following code blocks, including the `tutorial_helper_functions` module provided within this repository."
]
Expand Down Expand Up @@ -124,10 +126,7 @@
"\n",
"## Data Discovery\n",
"\n",
"Start by identifying your study area and exploring coincident data over the same time and area. \n",
"\n",
"NASA Earthdata Search can be used to visualize file coverage over multiple data sets of interest: \n",
"https://search.earthdata.nasa.gov/projects?projectId=5366449248\n"
"Start by identifying your study area and exploring coincident data over the same time and area. "
]
},
{
Expand Down Expand Up @@ -170,6 +169,26 @@
"poly"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Alternatively: Specify bounding box region of interest\n",
"\n",
"Instead of using a vector shape file to specify a region of interest, you can simply use a bounding box. The following cell is commented out below, which can be used instead of the `polygon` search parameter."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# bounds = poly.bounds # Get polygon bounds to be used as bounding box input\n",
"# bounding_box = ','.join(map(str, list(bounds))) # Bounding Box spatial parameter in decimal degree 'W,S,E,N' format. \n",
"# print(bounding_box)"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -205,7 +224,9 @@
"source": [
"param_dict = {\n",
" 'short_name': 'SNEX17_GPR',\n",
" 'version': '2','polygon': polygon,\n",
" 'version': '2',\n",
" 'polygon': polygon,\n",
"# 'bounding_box': bounding_box, #optional alternative to polygon search parameter; if using, remove or comment out polygon search parameter\n",
" 'temporal':temporal,\n",
"}"
]
Expand Down Expand Up @@ -283,7 +304,32 @@
"## Data Access\n",
"\n",
"### Option 1: Python script from the data set landing page\n",
"The [Depths and Snow Pit Data Package Contents](https://snowex-hackweek.github.io/website/tutorials/core-datasets/02_data-package.html) tutorial demonstrates how to access data using the script provided under the \"Download Data\" tab of each NSIDC DAAC data set landing page. Note that you must create a `.netrc` file within the JupyterHub environment, as well ensure you have a Earthdata Login profile, as described in the preliminary work article [here](https://snowex-hackweek.github.io/website/preliminary/earthdata.html#configure-programmatic-access-to-nasa-servers). "
"The [Depths and Snow Pit Data Package Contents](https://snowex-hackweek.github.io/website/tutorials/core-datasets/02_data-package.html) tutorial demonstrates how to access data using the script provided under the \"Download Data\" tab of each NSIDC DAAC data set landing page. \n",
"\n",
"#### Earthdata Login authentication setup notes\n",
"\n",
"1. All data with the NASA Earthdata system are freely available to the public, but requires an Earthdata Login profile to access. If you have not already done so, visit http://urs.earthdata.nasa.gov to register (it just takes a minute to set up). \n",
"\n",
"2. Create a `.netrc` file within the JupyterHub environment. This is a hidden file typically stored in your home directory that contains your Earthdata login username and password. This is a secure and easy way to ensure that any data download requests are authenticated against your profile. You can set up your `.netrc` within the JupyterHub environment as copied from the preliminary work article [here](https://snowex-hackweek.github.io/website/preliminary/earthdata.html#configure-programmatic-access-to-nasa-servers):\n",
"\n",
"Run the following commands on the JupyterHub in a terminal replacing your Earthdata login username and password:\n",
"\n",
"```\n",
"echo \"machine urs.earthdata.nasa.gov login EARTHDATA_LOGIN password EARTHDATA_PASSWORD\" > ~/.netrc\n",
"chmod 0600 .netrc\n",
"```\n",
"\n",
"Note that the script below should prompt you with your Earthdata Login username and password if a `.netrc` file does not exist. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%run './scripts/nsidc-download_SNEX20_SD.001.py' \n",
"print('Grand Mesa 2020 Snow Depth data download complete') "
]
},
{
Expand All @@ -305,7 +351,9 @@
"metadata": {},
"source": [
"#### Create the data request API endpoint\n",
"Programmatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above. We will first create a string of key-value-pairs from our data dictionary and we'll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below."
"Programmatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above. \n",
"\n",
"The cell below sets up the API request URL using our data search parameters as well as a few other configuration parameters. We will first create a string of key-value-pairs from our data dictionary and we'll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below."
]
},
{
Expand All @@ -317,10 +365,11 @@
"#Set NSIDC data access base URL\n",
"base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'\n",
"\n",
"#Set the request mode to asynchronous and set page number\n",
"#Set the request mode to asynchronous, \"no\" processing agent (no subsetting or reformatting services available), and optionally removing metadata delivery\n",
"\n",
"param_dict['request_mode'] = 'async'\n",
"param_dict['agent'] = 'NO'\n",
"param_dict['INCLUDE_META'] ='N' #optional if you do not wish to receive the associated metadata files with each science file. \n",
"\n",
"param_string = '&'.join(\"{!s}={!r}\".format(k,v) for (k,v) in param_dict.items()) # Convert param_dict to string\n",
"param_string = param_string.replace(\"'\",\"\") # Remove quotes\n",
Expand All @@ -336,7 +385,7 @@
"source": [
"#### Input Earthdata Login credentials\n",
"\n",
"An Earthdata Login account is required to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register."
"For our API access option, An Earthdata Login account is required to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register."
]
},
{
Expand All @@ -345,7 +394,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Start authenticated session with Earthdata Loging to allow for data downloads:\n",
"# Start authenticated session with Earthdata Login to allow for data downloads:\n",
"\n",
"def setup_earthdata_login_auth(endpoint: str='urs.earthdata.nasa.gov'):\n",
" netrc_name = \"_netrc\" if system()==\"Windows\" else \".netrc\"\n",
Expand Down Expand Up @@ -461,9 +510,9 @@
"metadata": {},
"source": [
"___\n",
"## Read in SnowEx data and buffer points around Snotel location\n",
"## Read in SnowEx data\n",
"\n",
"This SnowEx data set is provided in CSV. A [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html) is used to easily read in data. For these next steps, just one day's worth of data will be selected from this file and the coincident ASO and MODIS data will be selected.\n"
"This SnowEx data set is provided in CSV. A [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html) is used to easily read in data."
]
},
{
Expand All @@ -481,14 +530,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Convert to time values and extract a single day"
"### Extract date values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The collection date needs to be extracted from the `collection` value and a new dataframe will be generated as a subset of the original based on a single day:"
"The collection date needs to be extracted from the `collection` parameter:"
]
},
{
Expand All @@ -500,7 +549,6 @@
"df['date'] = df.collection.str.rsplit('_').str[-1].astype(str)\n",
"df.date = pd.to_datetime(df.date, format=\"%m%d%y\")\n",
"df = df.sort_values(['date'])\n",
"df_subset = df[df['date'] == '2017-02-08'] # subset original dataframe and only select this date\n",
"df.head()"
]
},
Expand All @@ -510,7 +558,7 @@
"source": [
"#### Convert to Geopandas dataframe to provide point geometry\n",
"\n",
"According to the SnowEx documentation, the data are available in UTM Zone 12N so we'll set to this projection so that we can buffer in meters in the next step:"
"According to the SnowEx documentation, the data are available in UTM Zone 12N so we'll set to this projection to allow for geospatial analysis:"
]
},
{
Expand All @@ -519,49 +567,26 @@
"metadata": {},
"outputs": [],
"source": [
"gdf_utm= gpd.GeoDataFrame(df_subset, geometry=gpd.points_from_xy(df_subset.x, df_subset.y), crs='EPSG:32612')\n",
"gdf_utm= gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.x, df.y), crs='EPSG:32612')\n",
"gdf_utm.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Additional data imagery services\n",
"___\n",
"## Additional data imagery services\n",
"\n",
"#### NASA Worldview and the Global Browse Imagery Service\n",
"\n",
"NASA’s EOSDIS Worldview mapping application provides the capability to interactively browse over 900 global, full-resolution satellite imagery layers and then download the underlying data. Many of the available imagery layers are updated within three hours of observation, essentially showing the entire Earth as it looks “right now.\"\n",
"\n",
"According to the [MOD10A1 landing page](https://nsidc.org/data/mod10a1), snow cover imagery layers from this data set are available through NASA Worldview. This layer can be downloaded as various image files including GeoTIFF using the snapshot feature at the top right of the page. This link presents the MOD10A1 NDSI layer over our time and area of interest: https://go.nasa.gov/35CgYMd. \n",
"Several MODIS snow data products from NSIDC include high-resolution browse imagery available through NASA Worldview, including [\"MODIS/Terra Snow Cover Daily L3 Global 500m SIN Grid, Version 61\"](https://nsidc.org/data/mod10a1). This layer can be downloaded as various image files including GeoTIFF using the snapshot feature at the top right of the page. This link presents the MOD10A1 NDSI layer over our time and area of interest: https://go.nasa.gov/35CgYMd. \n",
"\n",
"Additionally, the NASA Global Browse Imagery Service provides up to date, full resolution imagery for select NSIDC DAAC data sets as web services including WMTS, WMS, KML, and more. These layers can be accessed in GIS applications following guidance on the [GIBS documentation pages](https://wiki.earthdata.nasa.gov/display/GIBS/Geographic+Information+System+%28GIS%29+Usage). "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Export dataframe to Shapefile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, the dataframe can be exported as an Esri shapefile for further analysis in GIS:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_buffer = gdf_buffer.drop(columns=['date'])\n",
"gdf_buffer.to_file('snow-data-20170208.shp')"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down
Loading

0 comments on commit 6a72059

Please sign in to comment.