Skip to content

Commit

Permalink
update existing TESS (CAOM) and PanSTARRS notebooks to pyvo
Browse files Browse the repository at this point in the history
  • Loading branch information
@tsdower
tsdower committed Jan 9, 2020
1 parent 2e55b35 commit 617b2a4
Show file tree
Hide file tree
Showing 2 changed files with 80 additions and 46 deletions.
64 changes: 37 additions & 27 deletions notebooks/MAST/PanSTARRS/PS1_DR2_TAP/PS1_DR2_TAP.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"source": [
"Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic / spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns returned from cone, image, and spectral services.\n",
"\n",
"For this example, we'll be using the astroquery TAP/TAP+ client, which was developed by the ESAC Space Data Centre for working with the GAIA catalog, but is interoperable with other valid TAP services, including those at MAST. As an astroquery project, TAP+ documentation is available at ReadTheDocs: http://astroquery.readthedocs.io/en/latest/utils/tap.html\n",
"For this example, we'll be using the astropy affiliated PyVO client, which is interoperable with other valid TAP services, including those at MAST. PyVO documentation is available at ReadTheDocs: https://readthedocs.org/projects/pyvo/\n",
"\n",
"We'll be using TAP+ to call the TAP service at MAST serving PanSTARRS 1 Data Release 2, now with individual detection information. The schema is described within the service, and we'll show how to inspect it. The schema is also the same as the one available via the CasJobs interface, with an additional view added for the most common positional queries. CasJobs has its own copy of the schema documentation, which can be accessed through its own site: http://mastweb.stsci.edu/ps1casjobs/\n",
"\n"
Expand All @@ -58,8 +58,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Use the astroquery TapPlus library as our client to the data service.\n",
"from astroquery.utils.tap.core import TapPlus\n",
"# Use the pyvo library as our client to the data service.\n",
"import pyvo as vo\n",
"\n",
"# For resolving objects with tools from MAST\n",
"from astroquery.mast import Mast\n",
Expand All @@ -85,7 +85,12 @@
"\n",
"# To allow display tweaks for wider response tables\n",
"from IPython.core.display import display\n",
"from IPython.core.display import HTML"
"from IPython.core.display import HTML\n",
"\n",
"# suppress unimportant unit warnings from many TAP services\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\", module=\"astropy.io.votable.*\")\n",
"warnings.filterwarnings(\"ignore\", module=\"pyvo.utils.xml.elements\")"
]
},
{
Expand All @@ -109,7 +114,8 @@
"metadata": {},
"outputs": [],
"source": [
"TAP_service = TapPlus(url=\"http://vao.stsci.edu/PS1DR2/tapservice.aspx\")"
"TAP_service = vo.dal.TAPService(\"http://vao.stsci.edu/PS1DR2/tapservice.aspx\")\n",
"TAP_service.describe()"
]
},
{
Expand All @@ -125,9 +131,12 @@
"metadata": {},
"outputs": [],
"source": [
"tables = TAP_service.load_tables(only_names=True)\n",
"for table in tables:\n",
" print(table.name)"
"TAP_tables = TAP_service.tables\n",
"for tablename in TAP_tables.keys():\n",
" if not \"tap_schema\" in tablename: \n",
" TAP_tables[tablename].describe()\n",
" print(\"Columns={}\".format(sorted([k.name for k in TAP_tables[tablename].columns ])))\n",
" print(\"----\")"
]
},
{
Expand Down Expand Up @@ -158,14 +167,14 @@
},
"outputs": [],
"source": [
"job = TAP_service.launch_job_async(\"\"\"\n",
"job = TAP_service.run_async(\"\"\"\n",
"SELECT objID, RAMean, DecMean, nDetections, ng, nr, ni, nz, ny, gMeanPSFMag, rMeanPSFMag, iMeanPSFMag, zMeanPSFMag, yMeanPSFMag\n",
"FROM dbo.MeanObjectView\n",
"WHERE\n",
"CONTAINS(POINT('ICRS', RAMean, DecMean),CIRCLE('ICRS',187.706,12.391,.2))=1\n",
"AND nDetections > 1\n",
" \"\"\")\n",
"TAP_results = job.get_results()\n",
"TAP_results = job.to_table()\n",
"TAP_results"
]
},
Expand Down Expand Up @@ -199,8 +208,8 @@
"\"\"\".format(ra,dec,radius)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"TAP_results = job.to_table()\n",
"TAP_results"
]
},
Expand Down Expand Up @@ -234,8 +243,8 @@
"\"\"\".format(objid)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"detection_TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"detection_TAP_results = job.to_table()\n",
"detection_TAP_results"
]
},
Expand Down Expand Up @@ -491,8 +500,8 @@
"\"\"\".format(ra,dec,radius)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"TAP_results = job.to_table()\n",
"TAP_results"
]
},
Expand Down Expand Up @@ -526,8 +535,8 @@
"\"\"\".format(objid)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"detection_TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"detection_TAP_results = job.to_table()\n",
"\n",
"# add magnitude and difference from mean\n",
"detection_TAP_results['magmean'] = 0.0\n",
Expand Down Expand Up @@ -635,8 +644,8 @@
"\"\"\".format(ra,dec,radius)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"TAP_results = job.to_table()\n",
"TAP_results"
]
},
Expand All @@ -662,8 +671,8 @@
"\"\"\".format(objid)\n",
"print(query)\n",
"\n",
"job = TAP_service.launch_job_async(query)\n",
"detection_TAP_results = job.get_results()\n",
"job = TAP_service.run_async(query)\n",
"detection_TAP_results = job.to_table()\n",
"\n",
"# add magnitude and difference from mean\n",
"detection_TAP_results['magmean'] = 0.0\n",
Expand Down Expand Up @@ -772,9 +781,10 @@
"* IVOA standard for querying astronomical data in tabular format, with geometric search support\n",
"* http://www.ivoa.net/documents/latest/ADQL.html\n",
"\n",
"## TapPlus \n",
"* Module created by ESAC Space Data Centre\n",
"* http://astroquery.readthedocs.io/en/latest/utils/tap.html"
"## PyVO\n",
"* an affiliated package for astropy\n",
"* find and retrieve astronomical data available from archives that support standard IVOA virtual observatory service protocols.\n",
"* https://pyvo.readthedocs.io/en/latest/index.html"
]
},
{
Expand All @@ -790,7 +800,7 @@
"source": [
"## About this Notebook\n",
"**Authors:** Rick White & Theresa Dower, STScI Archive Scientist & Software Engineer\n",
"**Updated On:** 02/27/2019"
"**Updated On:** 01/09/2020"
]
},
{
Expand Down Expand Up @@ -824,7 +834,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
"version": "3.7.3"
},
"widgets": {
"state": {},
Expand Down
62 changes: 43 additions & 19 deletions notebooks/MAST/TESS/beginner_tess_tap_search/beginner_tess_tap_search.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,9 @@
"source": [
"Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic / spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns retunred from cone, image, and spectral services.\n",
"\n",
"For this example, we'll be using the astroquery TAP/TAP+ client, which was developed by the ESAC Space Data Centre for working with the GAIA catalog, but is interoperable with other valid TAP services, including those at MAST. As an astroquery project, TAP+ documentation is available at ReadTheDocs: http://astroquery.readthedocs.io/en/latest/utils/tap.html\n",
"For this example, we'll be using the astropy affiliated PyVO client, which is interoperable with other valid TAP services, including those at MAST. PyVO documentation is available at ReadTheDocs: https://readthedocs.org/projects/pyvo/\n",
"\n",
"We'll be using TAP+ to call the CAOM Catalog TAP service at MAST and filter the results for TESS-related information. The schema for this catalog is an IVOA standard, and is also described within the service itself.\n",
"\n"
"We'll be using TAP+ to call the CAOM Catalog TAP service at MAST and filter the results for TESS-related information. The schema for this catalog is an IVOA standard, and is also described within the service itself."
]
},
{
Expand All @@ -61,8 +60,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Use the astroquery TapPlus library as our client to the data service.\n",
"from astroquery.utils.tap.core import TapPlus\n",
"# Use the pyvo library as our client to the data service.\n",
"import pyvo as vo\n",
"\n",
"# For handling ordinary astropy Tables in responses\n",
"from astropy.table import Table\n",
Expand All @@ -74,7 +73,13 @@
"\n",
"# To allow display tweaks for wider response tables\n",
"from IPython.core.display import display\n",
"from IPython.core.display import HTML"
"from IPython.core.display import HTML\n",
"\n",
"# suppress unimportant unit warnings from many TAP services\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\", module=\"astropy.io.votable.*\")\n",
"warnings.filterwarnings(\"ignore\", module=\"pyvo.utils.xml.elements\")\n",
"warnings.filterwarnings(\"ignore\", module=\"pyvo.io.vosi.vodataservice\")"
]
},
{
Expand Down Expand Up @@ -110,7 +115,29 @@
"metadata": {},
"outputs": [],
"source": [
"TAP_service = TapPlus(url=TAP_URL)"
"TAP_service = vo.dal.TAPService(TAP_URL)\n",
"TAP_service.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### List available tables"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"TAP_tables = TAP_service.tables\n",
"for tablename in TAP_tables.keys():\n",
" if not \"tap_schema\" in tablename: \n",
" TAP_tables[tablename].describe()\n",
" print(\"Columns={}\".format(sorted([k.name for k in TAP_tables[tablename].columns ])))\n",
" print(\"----\")"
]
},
{
Expand Down Expand Up @@ -156,17 +183,15 @@
"metadata": {},
"outputs": [],
"source": [
"job = TAP_service.launch_job_async(\"\"\"\n",
"job = TAP_service.run_async(\"\"\"\n",
" SELECT top 1 obs_id, s_region \n",
" FROM dbo.caomobservation JOIN ivoa.obscore on dbo.caomobservation.observationID = ivoa.obscore.obs_id \n",
" WHERE collection = 'TESS' and trgType = 'FIELD' and \n",
" sequenceNumber = {} and\n",
" observationID like '{}'\n",
" \"\"\".format(sector, observationIDwildcard))\n",
"\n",
"# Uncomment the line below to see the whole query with wildcards replaced\n",
"#print(job.parameters)\n",
"footprint_results = job.get_results()\n",
"footprint_results = job.to_table()\n",
"footprint_results"
]
},
Expand Down Expand Up @@ -215,7 +240,7 @@
},
"outputs": [],
"source": [
"job = TAP_service.launch_job_async(\"\"\"\n",
"job = TAP_service.run_async(\"\"\"\n",
" SELECT target_name, sequenceNumber as sector, s_ra, s_dec, access_url, access_estsize, obs_id \n",
" FROM dbo.caomobservation JOIN ivoa.obscore on dbo.caomobservation.observationID = ivoa.obscore.obs_id \n",
" WHERE \n",
Expand All @@ -225,9 +250,7 @@
" ORDER BY obs_id\n",
" \"\"\".format(footprintShape, footprintVertices))\n",
"\n",
"# Uncomment the line below to see the whole query with wildcards replaced\n",
"#print(job.parameters)\n",
"TAP_results = job.get_results()\n",
"TAP_results = job.to_table()\n",
"TAP_results "
]
},
Expand Down Expand Up @@ -314,9 +337,10 @@
"* http://www.opencadc.org/caom2/\n",
"\n",
"\n",
"## TapPlus \n",
"* Module created by ESAC Space Data Centre\n",
"* http://astroquery.readthedocs.io/en/latest/utils/tap.html"
"## PyVO\n",
"* an affiliated package for astropy\n",
"* find and retrieve astronomical data available from archives that support standard IVOA virtual observatory service protocols.\n",
"* https://pyvo.readthedocs.io/en/latest/index.html"
]
},
{
Expand All @@ -332,7 +356,7 @@
"source": [
"## About this Notebook\n",
"**Authors:** Scott Fleming & Theresa Dower, STScI Archive Scientists & Software Engineer\n",
"**Updated On:** 07/09/2019"
"**Updated On:** 01/09/2020"
]
},
{
Expand Down

0 comments on commit 617b2a4

Please sign in to comment.