Test cases not executed: switch to enforcement mode #9593
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gilles-peskine-arm
added
needs-ci
gilles-peskine-arm
changed the title
gilles-peskine-arm
force-pushed
the
test-cases-not-executed-enforce-development
branch
from
aeecc77 to
7238e19
Compare
5 tasks
gilles-peskine-arm
added
needs-preceding-pr
This was referenced Sep 20, 2024
|
This PR will make 3.6 and development diverge because they'll have slightly different ignore lists for coverage. With the repo split, we'll want to have the common code in the framework, and the data (list of driver components, ignore lists) in each consuming branch. Having discussed with Manuel who's handling the repo split side of things (private link), we're going to join efforts: I'll rework this PR to first move the common code into the framework repository. It's easier overall to schedule things this way. |
gilles-peskine-arm
force-pushed
the
test-cases-not-executed-enforce-development
branch
from
5044a27 to
fa5eba7
Compare
This was referenced Oct 4, 2024
gilles-peskine-arm
force-pushed
the
test-cases-not-executed-enforce-development
branch
2 times, most recently
from
0ba6368 to
b825f10
Compare
This clears more than half of the test cases that are not executed. This also captures a few negative test cases that are executed. Subsequent commits will refine the filtering. Signed-off-by: Gilles Peskine <[email protected]>
… tests Some negative tests involving unsupported mechanisms are executed, because they're testing what happens if the mechanism is unsupported. Refine the ignore list for `test_suite_psa_crypto_generate_key.generated` and `test_suite_psa_crypto_op_fail.generated` accordingly. Signed-off-by: Gilles Peskine <[email protected]>
…aised Ignore certain test cases which either should not be generated or should be executed. For each ignore list entry, link to a GitHub issue whose definition of done includes removing the entry. Signed-off-by: Gilles Peskine <[email protected]>
Signed-off-by: Gilles Peskine <[email protected]>
Our PSA crypto implementation does not and will not support SECP224K1. Signed-off-by: Gilles Peskine <[email protected]>
For each ignore list entry, link to a GitHub issue for its resolution, except for ssl-opt Valgrind tests which we never intend to run on the CI. Signed-off-by: Gilles Peskine <[email protected]>
For each ignore list entry, link to a GitHub issue for its resolution. Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
added
approved
Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
dismissed stale reviews from paul-elliott-arm and yanesca
via
86421f3
gilles-peskine-arm
added
needs-review
gilles-peskine-arm
removed
the
needs-preceding-pr
paul-elliott-arm
previously approved these changes
Oct 17, 2024
|
The failure of |
…ed-enforce-development Conflicts: * framework: update to the head of 'main'.
gilles-peskine-arm
dismissed stale reviews from paul-elliott-arm and yanesca
via
f5025e2
|
There was another framework update happening concurrently, which caused a CI failure when this PR had an updated framework without a necessary change in the consuming branch. I've pushed a merge commit where the one conflict is in the framework, resolved by taking the head of the |
yanesca
approved these changes
Oct 18, 2024
paul-elliott-arm
added
approved
Merged
via the queue into
Mbed-TLS:development
with commit Oct 18, 2024
6707584
4 of 5 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch the test coverage analysis in
analyze_outcomes.pyto enforcement mode. In other words, the CI will now complain if there is a test case that is not executed in any configuration. Legitimate exceptions can be recorded in the script.The goal of this pull request is only to switch to enforcement mode, so that if we stop running a test case in the future, the CI will fail. Resolving the test cases that are already reported as not executed is out of scope here. Therefore this pull requests creates an ignore list that currently catches 9953 test cases (a vast majority from test cases systematically generated by
generate_psa_tests.py). Apart from a very small number of legitimate never-executed test cases which I document as such, I have filed issues to resolve the existing problems, linked from comments in the ignore list.Resolves #5390 (everything is analyzed or at least sorted into issues for further analysis), #2691 (finally!), #3417 (resolved by
test_suite_configplus enforcement plus issues filed for the reported missing coverage).See an example of a failed run with an earlier version of this PR in #9610, specifically https://mbedtls.trustedfirmware.org/job/mbed-tls-pr-head/job/PR-9610-head/3/display/redirect . An example with a spurious coverage exemption: https://mbedtls.trustedfirmware.org/job/mbed-tls-pr-head/job/PR-9614-head/3/ .
Content:
PR checklist