Flink End-to-End Tests

This module contains tests that verify end-to-end behaviour of Flink.

The tests defined in run-nightly-tests.sh are run by the CI system on every pull request and push to master.

Running Tests

You can run all tests by executing

$ FLINK_DIR=<flink dir> flink-end-to-end-tests/run-nightly-tests.sh

where <flink dir> is a Flink distribution directory.

You can also run tests individually via

$ FLINK_DIR=<flink dir> flink-end-to-end-tests/run-single-test.sh your_test.sh arg1 arg2

NOTICE: Please DON'T run the scripts with explicit command like sh run-nightly-tests.sh since #!/usr/bin/env bash is specified as the header of the scripts to assure flexibility on different systems.

NOTICE: We do not recommend executing the nightly test script on production or personal desktop systems, as tests contained there might modify the environment (leftover processes, modification of system files, request for root permissions via sudo, ...).

Kubernetes test

Kubernetes test (test_kubernetes_embedded_job.sh) assumes a running minikube cluster.

Writing Tests

As of November 2020, Flink has two broad types of end-to-end tests: Bash-based end-to-end tests, located in the test-scripts/ directory and Java-based end-to-end tests, such as the PrometheusReporterEndToEndITCase. The community recommends writing new tests as Java tests, as we are planning to deprecate the bash-based tests in the long run.

Examples

Have a look at test_batch_wordcount.sh for a very basic test and test_streaming_kafka010.sh for a more involved example. Whenever possible, try to put new functionality in common.sh so that it can be reused by other tests.

Adding a test case

In order to add a new test case you need add it to test-scripts/run-nightly-tests.sh. Templates on how to add tests can be found in those respective files.

Note: If you want to parameterize your tests please do so by adding multiple test cases with parameters as arguments to the nightly / pre-commit test suites. This allows the test runner to do a cleanup in between each individual test and also to fail those tests individually.

Note: While developing a new test case make sure to enable bash's error handling in test-scripts/common.sh by uncommenting set -Eexuo pipefail and commenting the current default set call. Once your test is implemented properly, add set -Eeuo pipefail on the very top of your test script (before any common script).

Passing your test

A test is considered to have passed if it:

• has exit code 0
• there are no non-empty .out files (nothing was written to stdout / stderr by your Flink program)
• there are no exceptions in the log files
• there are no errors in the log files

Note: There is a whitelist for exceptions and errors that do not lead to failure, which can be found in the check_logs_for_errors and check_logs_for_exceptions in test-scripts/common.sh.

Please note that a previously supported pattern where you could assign a value the global variable PASS to have your tests fail is not supported anymore.

Cleanup

The test runner performs a cleanup after each test case, which includes:

• Stopping the cluster
• Killing all task and job managers
• Reverting conf and lib dirs to default
• Cleaning up log and temp directories

In some cases your test is required to do some additional cleanup, for example shutting down external systems like Kafka or Elasticsearch. In this case you can register a function that will be called on test exit like this:

function test_cleanup {
    # do your custom cleanup here
}

on_exit test_cleanup