Skip to content

Commit

Permalink
Fixing the archiving of benchmark results (apache#7000)
Browse files Browse the repository at this point in the history
We were using the wrong directory name when trying to create
a benchmark archive. It was looking for a single directory in the format
benchmarks-apache-develop-main-benchmark-base-277-89433-base, but there
are actually multiple directories in the format
benchmarks-apache-develop-main-benchmark-base-277-89433-base-output-10-13-2021-18-26-22-apache-develop-main-benchmark-base-277-89433-base
as well as a text file.

Archiving the whole results directory. Also, including a tabular
dump of the results in the archive for potentially easier parsing.
  • Loading branch information
upthewaterspout authored Oct 14, 2021
1 parent fa82bc8 commit aa93f93
Showing 1 changed file with 9 additions and 13 deletions.
22 changes: 9 additions & 13 deletions ci/scripts/cleanup_benchmarks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,9 @@ popd

source concourse-metadata-resource/concourse_metadata
CLUSTER_TAG="${BUILD_PIPELINE_NAME}-${BUILD_JOB_NAME}-${BUILD_NAME}-${BUILD_ID}${TAG_POSTFIX}"
RESULTS_BASE_DIR=$(pwd)/results
BENCHMARKS_DIR=benchmarks-${CLUSTER_TAG}
RESULTS_DIR=${RESULTS_BASE_DIR}/benchmarks-${CLUSTER_TAG}
BENCHMARKS_ARCHIVE_FILENAME=${BENCHMARKS_DIR}.tgz
BENCHMARKS_ARCHIVE_FILE=${RESULTS_BASE_DIR}/${BENCHMARKS_ARCHIVE_FILENAME}
BENCHMARKS_PREFIX=benchmarks-${CLUSTER_TAG}
BENCHMARKS_ARCHIVE_FILENAME=${BENCHMARKS_PREFIX}.tgz
BENCHMARKS_ARCHIVE_FILE=${RESULTS_DIR}/${BENCHMARKS_ARCHIVE_FILENAME}
BENCHMARKS_ARTIFACTS_DESTINATION="${ARTIFACT_BUCKET}/benchmarks/${BUILD_PIPELINE_NAME}/${GEODE_SHA}"

if [[ "${ARTIFACT_BUCKET}" =~ \. ]]; then
Expand All @@ -54,10 +52,13 @@ pushd geode-benchmarks/infrastructure/scripts/aws/
./destroy_cluster.sh -t ${CLUSTER_TAG} --ci
popd

pushd ${RESULTS_BASE_DIR}
if [[ -d ${BENCHMARKS_DIR} ]]; then
pushd geode-benchmarks
./infrastructure/scripts/aws/dump_results.sh ${RESULTS_DIR}/benchmarks-*/* | tee ${RESULTS_DIR}/results.txt
popd

pushd ${RESULTS_DIR}
echo "***** Creating benchmarks archive"
tar zcf ${BENCHMARKS_ARCHIVE_FILE} ${BENCHMARKS_DIR}
tar zcf ${BENCHMARKS_ARCHIVE_FILE} *
echo "***** Copying benchmarks archive to storage"
gsutil cp ${BENCHMARKS_ARCHIVE_FILE} gs://${BENCHMARKS_ARTIFACTS_DESTINATION}/${BENCHMARKS_ARCHIVE_FILENAME}
printf "\n"
Expand All @@ -69,9 +70,4 @@ pushd ${RESULTS_BASE_DIR}
printf "\033[92m${ARTIFACT_SCHEME}://${BENCHMARKS_ARTIFACTS_DESTINATION}/${BENCHMARKS_ARCHIVE_FILENAME}\033[0m\n"
printf "\033[92m=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\033[0m\n"
printf "\n"
else
echo "***************************"
echo "No benchmark results found!"
echo "***************************"
fi
popd

0 comments on commit aa93f93

Please sign in to comment.