Skip to content

Commit

Permalink
mk: Don't run benchmarks with make check
Browse files Browse the repository at this point in the history
The current suite of benchmarks for the standard distribution take a significant
amount of time to run, but it's unclear whether we're gaining any benefit from
running them. Some specific pain points:

* No one is looking at the data generated by the benchmarks. We have no graphs
  or analysis of what's happening, so all the data is largely being cast into
  the void.

* No benchmark has ever uncovered a bug, they have always run successfully.

* Benchmarks not only take a significant amount of time to run, but also take a
  significant amount of time to compile. I don't think we should mitigate this
  for now because it's useful to ensure that they do indeed still compile.

This commit disables --bench test runs by default as part of `make check`,
flipping the NO_BENCH environment variable to a PLEASE_BENCH variable which will
manually enable benchmarking. If and when a dedicated bot is set up for
benchmarking, this flag can be enabled on that bot.
  • Loading branch information
alexcrichton committed May 15, 2014
1 parent ba5f530 commit bfbd732
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion Makefile.in
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
#
# * `TESTNAME=...` - Specify the name of tests to run
# * `CHECK_IGNORED=1` - Run normally-ignored tests
# * `NO_BENCH=1` - Don't run crate benchmarks (disable `--bench` flag)
# * `PLEASE_BENCH=1` - Run crate benchmarks (enable `--bench` flag)
#
# * `CFG_ENABLE_VALGRIND=1` - Run tests under valgrind
# * `VALGRIND_COMPILE=1` - Run the compiler itself under valgrind
Expand Down
6 changes: 3 additions & 3 deletions mk/tests.mk
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,16 @@ ifdef CHECK_IGNORED
TESTARGS += --ignored
endif

TEST_BENCH = --bench
TEST_BENCH =

# Arguments to the cfail/rfail/rpass/bench tests
ifdef CFG_VALGRIND
CTEST_RUNTOOL = --runtool "$(CFG_VALGRIND)"
TEST_BENCH =
endif

ifdef NO_BENCH
TEST_BENCH =
ifdef PLEASE_BENCH
TEST_BENCH = --bench
endif

# Arguments to the perf tests
Expand Down

0 comments on commit bfbd732

Please sign in to comment.