Upload reliability improvements #86
Conversation
scope/fritz.py
Outdated
|
|
||
| # post photometry to obj_id; drop flagged data | ||
| df_photometry = make_photometry(light_curves, drop_flagged=True) | ||
| for attempt in range(10): |
There was a problem hiding this comment.
Maybe make the number of attempts a variable like
MAX_ATTEMPTS = 10
Defined at the top of the script?
There was a problem hiding this comment.
Agreed! I added this variable for both scripts and replaced all range(10) with range(MAX_ATTEMPTS).
scope/fritz.py
Outdated
| print(f'Error - Retrying (attempt {attempt+1}).') | ||
|
|
||
| # start by checking for existing photometry | ||
| for attempt in range(10): |
There was a problem hiding this comment.
Then you can reuse it here.
scope/fritz.py
Outdated
| if response.json()["status"] == "error": | ||
| print('Failed to post to Fritz') | ||
| return None | ||
| for attempt in range(10): |
scope/fritz.py
Outdated
| @@ -334,7 +354,14 @@ def save_newsource( | |||
| "group_ids": group_ids, | |||
| "data": {'period': period}, | |||
| } | |||
| response = api("POST", "api/sources/%s/annotations" % obj_id, token, data=data) | |||
| for attempt in range(10): | |||
| @@ -35,6 +39,12 @@ def upload_classification( | |||
| sources = pd.read_csv(file) | |||
| columns = sources.columns | |||
|
|
|||
| if start is not None: | |||
| if stop is not None: | |||
| sources = sources.loc[start:stop] | |||
There was a problem hiding this comment.
Should this be start+1?
There was a problem hiding this comment.
Since the code currently prints the zero-based index of each row in the dataframe, I wanted to stick with that convention for the start argument. I've noted this convention in usage.md and the help info of the scripts.
tools/scope_upload_classification.py
Outdated
| response = api("GET", f"/api/sources?&ra={ra}&dec={dec}&radius={2/3600}", token) | ||
| sleep(0.9) | ||
| data = response.json().get("data") | ||
| for attempt in range(10): |
|
Thanks Michael! I just pushed the changes you requested. |
|
I also just added handling for a ConnectionError, which also should prompt the MAX_ATTEMPTS loop. |
|
LGTM @bfhealy ! |
|
Thanks, and sorry for the last-minute changes - I added handling for a couple more connection-related errors. |
|
Which is great. Feel free to merge when ready @bfhealy. |
This PR improves the stability of source uploads using scope_upload_classification.py: