Skip to content

Latest commit

 

History

History

testsuite

README.md

SIMD Tests

To execute the simd testsuite, call make check-simd, typically with -j N argument.

For more control over verbosity, compiler flags, and use of a simulator, use the environment variables documented below.

Environment variables

target_list

Similar to dejagnu target lists: E.g. target_list="unix{-march=sandybridge,-march=native/-ffast-math,-march=native/-ffinite-math-only}" would create three subdirs in testsuite/simd/ to run the complete simd testsuite first with -march=sandybridge, then with -march=native -ffast-math, and finally with -march=native -ffinite-math-only.

CHECK_SIMD_CONFIG

This variable can be set to a path to a file which is equivalent to a dejagnu board. The file needs to be a valid sh script since it is sourced from the scripts/check_simd script. Its purpose is to set the target_list variable depending on $target_triplet (or whatever else makes sense for you). Example:

case "$target_triplet" in
x86_64-*)
  target_list="unix{-march=sandybridge,-march=skylake-avx512,-march=native/-ffast-math,-march=athlon64,-march=core2,-march=nehalem,-march=skylake,-march=native/-ffinite-math-only,-march=knl}"
  ;;

powerpc64le-*)
  define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112"
  define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc112"
  define_target power9 "-mcpu=power9 -static" "$HOME/bin/run_on_gccfarm gcc135"
  target_list="power7 power8 power9{,-ffast-math}"
  ;;

powerpc64-*)
  define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc110"
  define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc110"
  target_list="power7 power8{,-ffast-math}"
  ;;
esac

The unix target is pre-defined to have no initial flags and no simulator. Use the define_target(name, flags, sim) function to define your own targets for the target_list variable. In the example above define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112" defines the target power7 which always uses the flags -mcpu=power7 and -static when compiling tests and prepends $HOME/bin/run_on_gccfarm gcc112 to test executables. In target_list you can now use the name power7. E.g. target_list="power7 power7/-ffast-math" or its shorthand target_list="power7{,-ffast-math}".

DRIVEROPTS

This variable affects the Makefiles generated per target (as defined above). It's a string of flags that are prepended to the driver.sh invocation which builds and runs the tests. You cd into a simd test subdir and use make help to see possible options and a list of all valid targets.

use DRIVEROPTS=<options> to pass the following options:
-q, --quiet         Disable same-line progress output (default if stdout is
                    not a tty).
-p, --percentage    Add percentage to default same-line progress output.
-v, --verbose       Print one line per test and minimal extra information on
                    failure.
-vv                 Print all compiler and test output.
-k, --keep-failed   Keep executables of failed tests.
--sim <executable>  Path to an executable that is prepended to the test
                    execution binary (default: the value of
                    GCC_TEST_SIMULATOR).
--timeout-factor <x>
                    Multiply the default timeout with x.
-x, --run-expensive Compile and run tests marked as expensive (default:
                    true if GCC_TEST_RUN_EXPENSIVE is set, false otherwise).
-o <pattern>, --only <pattern>
                    Compile and run only tests matching the given pattern.

TESTFLAGS

This variable also affects the Makefiles generated per target. It's a list of compiler flags that are appended to CXXFLAGS.

GCC_TEST_SIMULATOR

If --sim is not passed via DRIVEROPTS, then this variable is prepended to test invocations. If a simulator was defined via the CHECK_SIMD_CONFIG script, then then generated Makefile sets the GCC_TEST_SIMULATOR variable.

GCC_TEST_RUN_EXPENSIVE

If set to any non-empty string, run tests marked as expensive, otherwise treat these tests as UNSUPPORTED.

Writing new tests

A test starts with the copyright header, directly followed by directives influencing the set of tests to generate and whether the test driver should expect a failure.

Then the test must at least #include "bits/verify.h", which provides main and declares a template <typename V> void test() function, which the test has to define. The template parameter is set to simd<T, Abi> type where T and Abi are determined by the type and ABI subset dimensions.

The test() functions are typically implemented using the COMPARE(x, reference), VERIFY(boolean), and ULP_COMPARE(x, reference, allowed_distance) macros.

Directives

In the following, the optional <shell code> is considered as "matching" if it either isn't given or if evaluation (in the build directory) returns 0.

  • // skip: <type pattern> <ABI subset pattern> <target triplet pattern> <CXXFLAGS pattern> ['<shell code>'] If all patterns match, the test is silently skipped.

  • // only: <type pattern> <ABI subset pattern> <target triplet pattern> <CXXFLAGS pattern> ['<shell code>'] If any pattern doesn't match, the test is silently skipped.

  • // expensive: <type pattern> <ABI subset pattern> <target triplet pattern> <CXXFLAGS pattern> ['<shell code>'] If all patterns match, the test is UNSUPPORTED unless expensive tests are enabled.

  • // xfail: run|compile <type pattern> <ABI subset pattern> <target triplet pattern> <CXXFLAGS pattern> ['<shell code>'] If all patterns match, test compilation or execution is expected to fail. The test then shows as "XFAIL: ...". If the test passes, the test shows "XPASS: ...".

  • // timeout: <seconds> Set a fixed timeout in seconds for this test. This overrides the default and command line argument.

  • // timeout-factor: <scale factor> Multiply the timeout, after scaling with the --timeout-factor command line argument, with the specified factor.

All patterns are matched via

case '<test context>' in
  <pattern>)
  # treat as match
  ;;
esac

The <CXXFLAGS pattern> implicitly adds a * wildcard before and after the pattern. Thus, the CXXFLAGS pattern matches a substring and all other patterns require a full match.

Examples:

// The test is only valid for floating-point types:
// only: float|double|ldouble * * *

// Skip the test for long double for all powerpc64* targets:
// skip: ldouble * powerpc64* *

// The test is expected to unconditionally fail on execution:
// xfail: run * * * *

// ABI subsets 1-9 are considered expensive:
// expensive: * [1-9] * *

Implementation sketch

  • scripts/create_testsuite_files collects all *.c and *.cc files with simd/tests/ in their path into the file testsuite_file_simd (and at the same time removes them from testsuite_files.

  • The check-simd target in testsuite/Makefile.am calls scripts/check_simd. This script calls testsuite/experimental/simd/generate_makefile.sh to generate Makefiles in all requested subdirectories. The subdirectories are communicated back to the make target via a stdout pipe. The check-simd rule then spawns sub-make in these subdirectories. Finally it collects all summaries (simd_testsuite.sum) to present them at the end of the rule.

  • The generated Makefiles define targets for each file in testsuite_file_simd (you can edit this file after it was generated, though that's not recommended) while adding two test dimensions: type and ABI subset. The type is a list of all arithmetic types, potentially reduced via only and/or skip directives in the test's source file. The ABI subset is a number between 0 and 9 (inclusive) mapping to a set of simd_abis in testsuite/experimental/simd/tests/bits/verify.h (iterate_abis()). The tests are thus potentially compiled 170 (17 arithmetic types * 10 ABI subsets) times. This is necessary to limit the memory usage of GCC to reasonable numbers and keep the compile time below 1 minute (per compiler invocation).

  • When make executes in the generated subdir, the all target depends on building and running all tests via testsuite/experimental/simd/driver.sh and collecting their logs into a simd_testsuite.log and then extracting simd_testsuite.sum from it.

  • The driver.sh script builds and runs the test, parses the compiler and test output, and prints progress information to the terminal.

Appendix

run_on_gccfarm script

#!/bin/sh
usage() {
  cat <<EOF
Usage $0 <hostname> <executable> [arguments]

Copies <executable> to $host, executes it and cleans up again.
EOF
}

[ $# -lt 2 ] && usage && exit 1
case "$1" in
  -h|--help)
    usage
    exit
    ;;
esac

host="$1"
exe="$2"
shift 2

# Copy executable locally to strip it before scp to remote host
local_tmpdir=$(mktemp -d)
cp "$exe" $local_tmpdir
cd $local_tmpdir
exe="${exe##*/}"
powerpc64le-linux-gnu-strip "$exe"

ssh_controlpath=~/.local/run_on_gccfarm/$host
if [ ! -S $ssh_controlpath ]; then
  mkdir -p ~/.local/run_on_gccfarm
  (
    flock -n 9
    if [ ! -S $ssh_controlpath ]; then
      ssh -o ControlMaster=yes -o ControlPath=$ssh_controlpath -o ControlPersist=10m $host.fsffrance.org true
    fi
  ) 9> ~/.local/run_on_gccfarm/lockfile
fi
opts="-o ControlPath=$ssh_controlpath"

remote_tmpdir=$(ssh $opts $host.fsffrance.org mktemp -d -p .)
scp $opts -C -q "$exe" $host.fsffrance.org:$remote_tmpdir/
cd
rm -r "$local_tmpdir" &
ssh $opts $host.fsffrance.org $remote_tmpdir/$exe "$@"
ret=$?
ssh $opts $host.fsffrance.org rm -r $remote_tmpdir &
exit $ret