fix(wpt): run async tests and collect results #724
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Closes JSTZ-256.
JSTZ-256
Description
Fixes the wpt runner in jstz_api so that it covers async test cases. Asynchronous tests, including those wrapped in promises, were not properly executed because
eval
only runs the synchronous tests and puts async ones in the task queue. The runtime task queue was never touched afterwards and thus the async tests were simply abandoned. With changes in this PR, async test cases can be executed as expected, and also because of this, we don't need theNull
test status because at the end of the async task chain, test suites are completed and therefore there is always a meaningful test status returned.Note that the task queue must be executed after all synchronous tests finish, i.e. after all
eval
calls are made. I haven't fully figured out why, but my guess is that the test harness API has some assumptions about the test flow for each test suite, e.g. after async tests are done, other things are ignored, such that some test cases are dropped ifrun_jobs
is called right after aneval
call.Manually testing the PR
env UPDATE_EXPECT=1 cargo test --test wpt
and more tests show up in
wptreport.json
. These tests are already defined in the test suites but they were never executed.