Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Fix flaky tests #841

Merged
merged 3 commits into from
Dec 18, 2024
Merged

chore: Fix flaky tests #841

merged 3 commits into from
Dec 18, 2024

Conversation

cmaddox5
Copy link
Contributor

@cmaddox5 cmaddox5 commented Dec 16, 2024

Summary of changes

Asana Ticket: [🍎💳] Fix intermittent API test failures

I believe the root cause of the flaky tests is the fact that we reuse trip_ids across test modules. This causes some logic to include the wrong model in the returned map. In each test module, I changed the trip_ids to be unique to the module.

Even after the above change, tests were still intermittently failing. Decided to explore ways to ensure fresh state before each test. Found that stopping and starting the State app before each test allows for more predictable state in each process.

@cmaddox5 cmaddox5 requested review from a team and lemald and removed request for a team December 16, 2024 16:41
@cmaddox5 cmaddox5 marked this pull request as ready for review December 16, 2024 16:41
@Whoops
Copy link
Collaborator

Whoops commented Dec 16, 2024

I checked this out and did a little loop to run tests until they failed, but I actually got a failure on the TripController on my first run.
test.txt

@Whoops
Copy link
Collaborator

Whoops commented Dec 16, 2024

Here's another that includes some failures on our authentication plug
test.txt
(To be clear if the intent is to do multiple PRs to clean this up then I'm all for going ahead and merging this one)

@cmaddox5
Copy link
Contributor Author

Dang, thanks for letting me know. Believe it or not, I didn't see failures there in my testing.

@cmaddox5 cmaddox5 marked this pull request as draft December 17, 2024 15:14
@cmaddox5
Copy link
Contributor Author

Revisited this with @jzimbel-mbta and we think we found a better solution. After making the change, the plug test failures seemed to go away but I am not confident that my changes would be the reason. Because I am reaching the end of the timebox of 3 days, it might be best to go with these changes and revisit if the plug failures show up again.

@cmaddox5 cmaddox5 marked this pull request as ready for review December 17, 2024 16:45
@cmaddox5 cmaddox5 merged commit 824e0f9 into master Dec 18, 2024
24 checks passed
@cmaddox5 cmaddox5 deleted the cm/fix-flaky-tests branch December 18, 2024 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants