Address instability from report download caching

[ecayer]

In order to improve report performance, the team initially implemented a cache, but this isn't working. We are going to replace the cache with an approach that separates out report generation from downloading in the UI.

For more details on the problem and proposed approach, see here the ADR to Improve API stability when processing job data and generating reports

A full fix for this includes the following fixes and improvements and investigations:

• Limit Celery worker lifetime notifications-api#1413
• Increase max pool connections all the way to 10 notifications-api#1411
• Investigate builtins:ConnectionRefusedError when attempting to pull information from S3 notifications-api#1395
• Investigate builtins:BrokenPipeError when attempting to connect to S3 notifications-api#1393
• Review and improve/correct our Celery task retry logic notifications-api#1391
• Research the proper amount of workers and RAM (memory) per worker for Celery notifications-api#1390

Provisional decision: remove phone numbers. Needs socializing with partners.

[ecayer]

@ccostino Do you have a preference whether we group the work to decouple report generation/download with this or track it in a separate epic? And there isn't yet an issue that includes the UI for that, right?

[ccostino]

@ecayer I think this does need to be split up now at this point... how does this sound?

• This epic should probably be focused on removing the job cache only.
• A new epic needs to be created for the report generation itself
• The UI work... could be a part of the report generation, or it could be its own epic.

For the UI side of all of this you're right, there is no issue yet per se but it's been loosely captured in the ADR. There's been some conversation that's mostly led to more questions and the need for at least some user research.

To me that sounds like a new issue entirely for that, but I figured I'd gut check to be sure! Regardless, if this makes sense, I can help write up some more of the details to help get things more organized with this!

[ecayer]

Considering the original epic done, now working on removing phone numbers.