Skip to content

Commit

Permalink
Add docs on handling alternative test outputs (citusdata#6469)
Browse files Browse the repository at this point in the history
I recently cleaned up our test suite from redundant test outputs: citusdata#6111
citusdata#6140 citusdata#6214 citusdata#6140 citusdata#6434

I had to check many files manually, as they didn't have any
documentation on why the alternative test output existed in the first
place.

Adding a section in our test docs to remind developers to add
alternative test outputs with enough information/keywords.
  • Loading branch information
naisila authored Nov 3, 2022
1 parent 1af28b3 commit b8c7a98
Showing 1 changed file with 32 additions and 0 deletions.
32 changes: 32 additions & 0 deletions src/test/regress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,38 @@ To automatically setup a citus cluster in tests we use our
then starts the standard postgres test tooling. You almost never have to change
this file.

## Handling different test outputs

Sometimes the test output changes because we run tests in different configurations.
The most common example is an output that changes in different Postgres versions.
We highly encourage to find a way to avoid these test outputs.
You can try the following, if applicable to the changing output:
- Change the test such that you still test what you want, but you avoid the different test outputs.
- Reduce the test verbosity via: `\set VERBOSITY terse`, `SET client_min_messages TO error`, etc
- Drop the specific test lines altogether, if the test is not critical.
- Use utility functions that modify the output to your preference,
like [coordinator_plan](https://github.com/citusdata/citus/blob/main/src/test/regress/sql/multi_test_helpers.sql#L23),
which modifies EXPLAIN output
- Add [a normalization rule](https://github.com/citusdata/citus/blob/main/ci/README.md#normalize_expectedsh)

Alternative test output files are highly discouraged, so only add one when strictly necessary.
In order to maintain a clean test suite, make sure to explain why it has an alternative
output in the test header, and when we can drop the alternative output file in the future.

For example:

```sql
--
-- MULTI_INSERT_SELECT
--
-- This test file has an alternative output because of the change in the
-- display of SQL-standard function's arguments in INSERT/SELECT in PG15.
-- The alternative output can be deleted when we drop support for PG14
--
```
Including important keywords, like "PG14", "PG15", "alternative output" will
help cleaning up in the future.

## Randomly failing tests

In CI sometimes a test fails randomly, we call these tests "flaky". To fix these
Expand Down

0 comments on commit b8c7a98

Please sign in to comment.