ARROW-5269: [C++][Archery] Mark relevant benchmarks as regression

The goal of this change is to mark benchmarks candidate for automated regression checks. Some benchmarks were refactored for various reasons: - Code deduplication where needed - Uniform input size for benchmarks in the same suite, usually to minimize the effect of cache hierarchy. - Favor external repetitions over manual repetitions and mintime when possible. - Add cmake ARROW_BUILD_BENCHMARKS_REFERENCE to toggle reference benchmarks. - Remove default benchmark filter of `^Regression`. - Remove `BM_` prefix from benchmark names - Fixes and addons to `archery` to support the `--pdb` option for debugging and add support for benchmarks reported as `items_per_seconds`. - Adds `archery benchmark list` sub-command to list suites and benchmarks Author: François Saint-Jacques <[email protected]> Closes apache#4285 from fsaintjacques/ARROW-5269-regression-candidates and squashes the following commits: 8378030 <François Saint-Jacques> Address comments 4a570ab <François Saint-Jacques> Reformat a9ecffd <François Saint-Jacques> address comments d8e779a <François Saint-Jacques> Reformat 8334c68 <François Saint-Jacques> Remove BM_ benchmark prefix 5cebe2c <François Saint-Jacques> Normalize benchmarks 908f05a <François Saint-Jacques> Change default repetitions to 10 instead of 20 ffe723f <François Saint-Jacques> Add `benchmark list` sub-command. e087647 <François Saint-Jacques> Refactor JSON benchmarks a7b4f5f <François Saint-Jacques> Improve archery benchmark support bea76ed <François Saint-Jacques> Refactor csv benchmarks f207974 <François Saint-Jacques> Refactor CompareFilter benchmarks b7911df <François Saint-Jacques> Refactor Bitmap benchmarks 1b5135e <François Saint-Jacques> Refactor Builder benchmarks
fanzhen · May 30, 2019 · 20961b7 · 20961b7
1 parent c8466cd
commit 20961b7
Show file tree

Hide file tree

Showing 29 changed files with 698 additions and 765 deletions.
diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt
@@ -269,6 +269,9 @@ else()
   add_custom_target(all-benchmarks)
   add_custom_target(benchmark ctest -L benchmark)
   add_dependencies(benchmark all-benchmarks)
+  if(ARROW_BUILD_BENCHMARKS_REFERENCE)
+    add_definitions(-DARROW_WITH_BENCHMARKS_REFERENCE)
+  endif()
 endif()
 
 if(NOT ARROW_BUILD_EXAMPLES)

diff --git a/cpp/cmake_modules/DefineOptions.cmake b/cpp/cmake_modules/DefineOptions.cmake
@@ -96,6 +96,11 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
   define_option(ARROW_BUILD_BENCHMARKS "Build the Arrow micro benchmarks, default OFF"
                 OFF)
 
+  # Reference benchmarks are used to compare to naive implementation, or
+  # discover various hardware limits.
+  define_option(ARROW_BUILD_BENCHMARKS_REFERENCE
+                "Build the Arrow micro reference benchmarks, default OFF." OFF)
+
   define_option_string(ARROW_TEST_LINKAGE
                        "Linkage of Arrow libraries with unit tests executables."
                        "shared"