File Format Benchmarks

These big data file format benchmarks, compare:

• Avro
• Json
• ORC
• Parquet

There are three sub-modules to try to mitigate dependency hell:

• core - the shared part of the benchmarks
• hive - the Hive benchmarks
• spark - the Spark benchmarks

To build this library:

% mvn clean package

To fetch the source data:

% ./fetch-data.sh

To generate the derived data:

% java -jar core/target/orc-benchmarks-core-*-uber.jar generate data

To run a scan of all of the data:

% java -jar core/target/orc-benchmarks-core-*-uber.jar scan data

To run full read benchmark:

% java -jar hive/target/orc-benchmarks-hive-*-uber.jar read-all data

To run column projection benchmark:

% java -jar hive/target/orc-benchmarks-hive-*-uber.jar read-some data

To run decimal/decimal64 benchmark:

% java -jar hive/target/orc-benchmarks-hive-*-uber.jar decimal data

To run spark benchmark:

% java -jar spark/target/orc-benchmarks-spark-*.jar spark data