Add functionality for multiple sensor / parameter code combinations

[ajb474]

• HydroFunctions version: 0.2.0
• Python version: 3.8
• Operating System: Linux Mint 20

Description

When making a general data request using the 'NWIS' function, I could not differentiate between sensor/parameter code combinations and ultimately unable to retrieve a full dataset for a location. More specifically, I am looking at a site with multiple instances of a parameter code and the NWIS request cannot specify between these. For example, if you refer to USGS site 444306122144600 the measurements of a parameter (say Turbidity / 63680) are taken from different sensors. When querying using the parameter code '63680' there is no way to get either or both instances.

What I Did

Here is a basic query using the above mentioned specifications:

data = hf.NWIS(site = '444306122144600', service = 'iv', parameterCd = '63680', start_date = '2020-03-01', end_date = '2020-10-15') 

Here is a link to the same query via the web interface:

https://nwis.waterdata.usgs.gov/nwis/uv?cb_32295=on&cb_32316=on&cb_32319=on&cb_63680=on&format=gif_stats&site_no=444306122144600&period=&begin_date=2020-03-01&end_date=2020-07-20

Note the multiple instances of the WQ parameters.

[mroberge]

This is interesting!

I never knew it was possible for one site to have two versions of the same parameter code! NWIS is so complex and has so many different applications that it is hard to cover them all.

I'm going to look into this and see if I can come up with a way of dealing with it.

In the meantime, I think you should be able to go data.df('all') to get the full dataframe. Or, you can go data.json to see the raw json response.

I'll take a look into this! Thank you!

[ajb474]

Thank you for being so responsive.

I just tried to view the full dataframe and the same level of data is available for the parameter code (with the addition of the '_qualifiers' column).

Thanks again and let me know if I can clarify anything!

[ajb474]

Hello again, I wanted to check back and see if you were able to understand my issue or if you have had the chance to look into this at all.

Thank you!

[mroberge]

Hi Adam! @ajb474
Thank you for your interest in this! So the issue here is that this goes right to the heart of my algorithm for decoding the NWIS response. It is an overly complex algorithm as it is, and I never anticipated that the WaterML format could have more than one sensor with the same parameter at a site... but now that I've made my perfunctory excuses, I think I've got a solution!

It seems too easy, but I just look for multiple 'method'-series inside of a series, and then treat it like another series. It is basically just an extra for loop. You can see my progress on this branch: https://github.com/mroberge/hydrofunctions/tree/bugfix-multiple-methods-in-NWIS-json
The problem now (and reason why I'm a little slow) is that I need to come up with some tests to make sure that I'm not breaking anything.

Would you be interested in helping out? I need to:

• find more examples of queries and sites that return multiple methods per parameter, or other edge cases that I haven't thought about before.
• is there any situation where two timeseries should be combined? Like maybe one parameter code has multiple sensors because the first sensor broke so they put in a new sensor, and so really the timeseries should be combined, not kept separate.

BTW, I just noticed that you were NAU! I got my PhD from ASU Geography!

Anyway, thank you, and I hope I can get something working soon. You can try using the new branch, if you want.
-Marty

[mroberge]

I think I've got this one solved. I'm closing the issue, but feel free to comment if you find another site or situation that has multiple methods for a single parameter.

[ajb474]

Thank you so much for taking this on! Believe it or not providing you some edge cases was on my to do list this week, I've just been a bit behind. Actually, I was working with the R version (dataRetrieval) last week and found some good examples of the issue. It was the same deal, for a single site and WQ parameter multiple instruments were used for data collection at different times. Thanks again and if I can be of any help in the future please let me know.

…

On Tue, Nov 17, 2020, 21:36 Martin Roberge ***@***.***> wrote: I think I've got this one solved. I'm closing the issue, but feel free to comment if you find another site or situation that has multiple methods for a single parameter. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGBCO6QPLSTSHZ56BIOG5F3SQNFOHANCNFSM4TBFQCZQ> .