Skip to content

Latest commit

 

History

History

python

README.md

Python scripts to benchmark a program / the test suite

This folder consists of performance study and utility Python scripts. Each of these programs has an integrated help with -h.

Studying a single .c source file

To study the performance of a single .c file and run multiple benchmarks on it, single_source_full_study.py can be used.

This script makes 4 kinds of measures:

  • The execution time using a wrapper script
  • The execution time using basic python functions
  • The maximum used stack size, using the massif tool from valgrind
  • The size of the .text section of the executable, using readelf

The measures are then either reported on the standard output or appended to a CSV file depending on the options.

Usage

% ./single_source_full_study <file> <opt>

This will compile the provided C source file with the specified optimization sequence (eg. -O3) and measure its performance.

% ./single_source_full_study.py ../exclude/main.c -O3
Time (wr) : 0.053807 ms
Time (py) : 0.802279 ms
Stack size : 7176
.text size : 402

You can also manually specify passes manually. Each argument starting with a dash is taken to be a pass; all others are considered filenames from which to read pass sequences.

The -n option controls the number of benchmarks for a program.

% ./single_source_full_study.py ../exclude/main.c -opt="o3args.txt -licm" -n 15
Time (wr) : 0.000090 ms
Time (py) : 0.711679 ms
Stack size : 7176
.text size : 434

Note: The minimal time is displayed, since in the optimal case a deterministic program is never interrupted and should have the same execution time.

About the wrapper file

The tools/wrapper.c file is used to wrap the given C file.

The motivation is if we have a program that is small enough, the execution time will mainly be due to the use of fork() and exec() to start the program, and not running the program itself.

To remove this time from our measures, we place a wrapper around the main function of the benchmarked program, which is renamed old_main(). The wrapper essentially does a loop of start_time(); old_main(); end_time();

This process also enables us to run multiple benchmarks during the same execution.

Studying the pass order on the test-suite

LLVM provides a test-suite that is composed of an already built architecture and a lot of already existing programs.

The other scripts are built to reuse the test-suite.

The basic idea of measuring a pass order with the test suite is:

  • Configure the test-suite and rewrite the compilation rules
  • Build all programs with rewritten rules
  • Run the compiled test-suite one or several times
  • Draw plots with the results and try to find a meaning to our existence

Test suite builder

Configures the test suite. Use rule rewriting to transform the clang calls to use a custom script that can execute any pass sequence.

Usage

% ./test-suite-builder.py -d <output directory> -s <script name>

This will compile a test suite in the specified output directory. When run, the new test suite will execute the given script from /testsuite-scripts for all test programs.

This script can be used to perform any tests on the program but must compile it into an object file as a side effect.

Pre-written scripts are provided, for instance 03.sh to build with -O3. Other scripts, such as assoc.sh, compile the input program twice to compare the results.

A script /testsuite-scripts/tools/ts.sh provides utitily functions to implement test suite scripts easily.

test-site-builder.sh used to build a test suite that always executes clang, opt, clang and pass all options starting with a '-' to opt. Currently such options are ignored.

With -f instead of -d, the output directory is prepended with the path to /test-site-builder in the repository.

Notes

  • Rule rewriting process is implemented in tools/rules_rewriter.py.
  • perf can be used instead of timeit with -p option. This might need root privileges and only changes the compile time measurements.

Run test suites with llvm-lit

run_multiple_test_suites.py can be used to run the given compiled test suites with llvm-lit and report their result in JSON files.

Usage

% ./run_multiple_test_suites.py <test suite folders...>

The basic manual equivalent is running llvm-lit in each folder.

Plotter

plotter.py draws plots from the files generated by llvm-lit (in most cases, these files will be generated throught the run multiple test suites script).

Usage

% cd notes/data/
% ls
O0    O1    O2    O3    D3    SomeOtherFolderWithJsonResults
% ../../python/plotter.py -t plot O0 O1 O2 O3 -r O1 -s O2

Draws a basic scatter plot with a comparison of opt -O0 to opt -O3, sorted by speedup of -O2, the reference time being that of -O1.

Configuration file

A configuration file can be passed to plotter.py with -c. It must be in the following format :

OPT_NAME = /path/to/a/folter/with/json/files
* = /path/to/a/folder/with/a/*/

In the first example, the script will get every files whose match /path/to/a/ folter/with/json/files/*.json and register the results in these files for the pass sequence named OPT_NAME.

In the second example, the script will match every files with /path/to/a/ folder/with/a/*/*.json, and each files will be register as a result for the pass sequence named after the first star matching. (for example /path/to/a/ folder/with/a/O3/some_json_file.json will be registered as a result of the pass sequence O3).

JSON file simplifier

The JSON files resulting from running the test suite contain a lot of information, not all of which is relevant to draw plots. You might want to share only subsections of the JSON files.

This script is a simple JSON file rewriter which removes any unnecessary data in our context.

You can either:

  • call json_simplifier.py on a list of folders to generated *_short.json files from the originals;
  • or call json_simplifier.py with -s <X> and -d <Y> to reproduce in Y the file tree of X with short versions of all JSON files in X.

An example of the second method would be:

% ./json_simplifier.py -d extracted_times -s test-suite-builder

This copies the relevant JSON from test-suite-builder into extracted_times and simplifies them. It also copies the chosen_opt.txt file if any was generated by the legacy test suite builder.

Step-by-step optimization

tools/opt_step_by_step.py can optimize a program with opt and print the resulting IR after each step. It is similar to

% opt <source> -print-after-all <passes> -o <source> -print-regusage

Its purpose is to manually study the effect of some passes on an IR, and print the differences every time it changes.

Usage example:

% opt_step_by_step.py /path/to/a/program.c -some -pass -some -other -passes

This will be equivalent to the following script.

clang -emit-llvm -S /path/to/a/program.c -o temp/src.bc

for pass in -some -pass -some -other -passes; do
  opt -pass temp/src.bc -o temp/dst.bc
  diff temp/src.bc temp/dst.bc
  mv temp/dst.bc temp/src.bc
done

The source file is the first argument; everything else is passed verbatim to opt. You cannot specify a file name such as o3args.txt.