E2E Test Improvement: leverage the live staging API instead of deploying a local instance in CI/CD

[ccostino]

E2E tests are now working in the CI/CD pipeline for pull requests! 🎉 However, to get them to work we had to include the API repo and stand up a local API server instance in the CI/CD job in addition to the other actions. This adds complexity and time to the tests, when ideally we'd be able to just use the staging API directly.

What currently happens

When we leverage the staging API directly in the E2E test job right now, the E2E user cannot be authenticated despite what appears to be he proper configuration being in place. For E2E tests to work, several environment variables need to be shared and set in both the API and Admin sites (details are in the E2E documentation). The staging API has these configured (you can verify in cloud.gov by running cf env notify-staging-api), and the Admin application spun up in the CI/CD pipeline also has the environment variables set that it needs (take a look at the PR to see details).

Despite these environment variables being set and the environment configuration set to match with the new E2ETest environment config object, any attempt at signing in fails and all subsequent pages will instead show the message "You are not authorized..." - the API response will be a 403, which you can see in the staging API logs, and the tests will fail.

Possible causes and things to look at

There are not a lot of leads on this at the moment, other than continuing to check for differences of what's set locally vs. in the CI/CD environment. Mirroring the set up locally and connecting directly to the staging API works.

I have wondered if setting FLASK_DEBUG=true would fix this, but I don't want to enable that outside of a local environment, and not setting it locally doesn't seem to affect the ability to authenticate.

The only thing I can hazard a guess at right now is that the API token is ultimately still not present or being malformed somehow.

Implementation Sketch and Acceptance Criteria

• Figure out why connecting directly to the staging API doesn't work in the CI/CD pipeline
• Fix the issue(s)
• Remove the extra API setup steps in the E2E tests job and point to the staging API instead
• Update documentation as necessary

Security Considerations

• We need to be mindful of not exposing debug or log information that would compromise any credentials as we debug this issue.
• We do not want to open any additional connection points that we don't need to; we should be able to connect to the staging API as is.

[ccostino]

Putting this back in the backlog for now; we'll revisit once we tackle some other things and get the E2Es working with Dependabot at least!

[terrazoon]

This ticket is done in the sense that a 'local' admin server is now talking to staging API in CI/CD. There are some remaining snacks around accessing S3, but parts of the tests not using S3 are working. S3 work is on a separate ticket.

[ccostino]

Mmmmm, snacks... 😆

Yes, thank you @terrazoon! If you'd like to get a new ticket written for the S3 work here, link it here once finished and then we can close this one out!