Skip to content

Commit

Permalink
Move gcs-connector from lib to hadoop-dependencies for integration te…
Browse files Browse the repository at this point in the history
…st (apache#12144)
  • Loading branch information
jihoonson authored Jan 13, 2022
1 parent 168187e commit 58378aa
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions integration-tests/script/copy_resources_template.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,13 @@ if [ -n "$DRUID_INTEGRATION_TEST_START_HADOOP_DOCKER" ] && [ "$DRUID_INTEGRATION
then
## We put same version in both commands but as we have an if, correct code path will always be executed as this is generated script.
## <TODO> Remove if
mkdir $SHARED_DIR/hadoop-dependencies/hadoop-gcs
if [ -n "${HADOOP_VERSION}" ] && [ "${HADOOP_VERSION:0:1}" == "3" ]; then
java -cp "$SHARED_DIR/docker/lib/*" -Ddruid.extensions.hadoopDependenciesDir="$SHARED_DIR/hadoop-dependencies" org.apache.druid.cli.Main tools pull-deps -h org.apache.hadoop:hadoop-client-api:${hadoop.compile.version} -h org.apache.hadoop:hadoop-client-runtime:${hadoop.compile.version} -h org.apache.hadoop:hadoop-aws:${hadoop.compile.version} -h org.apache.hadoop:hadoop-azure:${hadoop.compile.version}
curl https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar --output $SHARED_DIR/docker/lib/gcs-connector-hadoop3-latest.jar
curl https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar --output $SHARED_DIR/hadoop-dependencies/hadoop-gcs/gcs-connector-hadoop3-latest.jar
else
java -cp "$SHARED_DIR/docker/lib/*" -Ddruid.extensions.hadoopDependenciesDir="$SHARED_DIR/hadoop-dependencies" org.apache.druid.cli.Main tools pull-deps -h org.apache.hadoop:hadoop-client:${hadoop.compile.version} -h org.apache.hadoop:hadoop-aws:${hadoop.compile.version} -h org.apache.hadoop:hadoop-azure:${hadoop.compile.version}
curl https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar --output $SHARED_DIR/docker/lib/gcs-connector-hadoop2-latest.jar
curl https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar --output $SHARED_DIR/hadoop-dependencies/hadoop-gcs/gcs-connector-hadoop2-latest.jar
fi
fi

Expand Down

0 comments on commit 58378aa

Please sign in to comment.