forked from apache/pulsar
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Testing] Add slim java-test-image which can run most integration tes…
…ts (apache#9625) ### Motivation The feedback loop in fixing and debugging integration tests is extremely slow at the moment because of the long time to build the pulsar-test-latest-version docker image and the images pulsar and pulsar-all that it depends on. Introduce a slim docker image which can run most integration tests. ### Modifications Adds a maven module for building `java-test-image`. Produces a relatively slim 750MB image (+461MB to base image) which doesn't depend on building `pulsar` and `pulsar-all` images like the current `pulsar-test-latest-version image`. Image doesn't include CPP client, Python client, Pulsar SQL or Pulsar Connectors since most integration tests don't require those. Enables a lot quicker feedback loop when debugging and fixing integration tests locally since building the image takes less than 1.5 minutes on a modern laptop. Adding this image is also a preparation for speeding up CI where this slim image would be used to run the "core" integration tests which don't require CPP client, Python client, Pulsar SQL or Pulsar Connectors. Usage example to run single CLI integration test method against java-test-image: ``` ./build/build_java_test_image.sh export PULSAR_TEST_IMAGE_NAME=apachepulsar/java-test-image:latest mvn -B -f tests/pom.xml test -DintegrationTests -DredirectTestOutputToFile=false -DtestRetryCount=0 -DfailIfNoTests=false -Dtest=CLITest#testCreateSubscriptionCommand ``` Adds changes to the integration test infrastructure so that the name of the docker image to use can be passed in `PULSAR_TEST_IMAGE_NAME` environment variable. In addition, adds `skipSourceReleaseAssembly` maven property which can be used to skip building the `apache-pulsar-*-src.tar.gz` when it's not needed. This is to speed up building of the docker image. It's used in the provided `build/build_java_test_image.sh` script.
- Loading branch information
Showing
12 changed files
with
286 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
|
|
||
| SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" | ||
| cd "$SCRIPT_DIR/.." | ||
| SQUASH_PARAM="" | ||
| # check if docker experimental mode is enabled which is required for | ||
| # using "docker build --squash" for squashing all intermediate layers of the build to a single layer | ||
| if [[ "$(docker version -f '{{.Server.Experimental}}' 2>/dev/null)" == "true" ]]; then | ||
| SQUASH_PARAM="-Ddockerfile.build.squash=true" | ||
| fi | ||
| mvn -am -pl tests/docker-images/java-test-image install -Pcore-modules,integrationTests,docker \ | ||
| -Dmaven.test.skip=true -DskipSourceReleaseAssembly=true -Dspotbugs.skip=true -Dlicense.skip=true $SQUASH_PARAM "$@" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| # | ||
|
|
||
| FROM openjdk:8-jdk-slim | ||
|
|
||
| RUN groupadd -g 10001 pulsar | ||
| RUN adduser -u 10000 --gid 10001 --disabled-login --disabled-password --gecos '' pulsar | ||
|
|
||
| ARG PULSAR_TARBALL=target/pulsar-server-distribution-bin.tar.gz | ||
| ADD ${PULSAR_TARBALL} / | ||
| RUN mv /apache-pulsar-* /pulsar | ||
| RUN chown -R root:root /pulsar | ||
|
|
||
| COPY target/scripts /pulsar/bin | ||
| RUN chmod a+rx /pulsar/bin/* | ||
|
|
||
| RUN echo networkaddress.cache.ttl=1 >> $JAVA_HOME/jre/lib/security/java.security | ||
|
|
||
| WORKDIR /pulsar | ||
|
|
||
| RUN apt-get update | ||
|
|
||
| # /pulsar/bin/watch-znode.py requires python3-kazoo | ||
| # /pulsar/bin/pulsar-managed-ledger-admin requires python3-protobuf | ||
| RUN apt-get install -y python3-kazoo python3-protobuf | ||
| # use python3 for watch-znode.py and pulsar-managed-ledger-admin | ||
| RUN sed -i '1 s/.*/#!\/usr\/bin\/python3/' /pulsar/bin/watch-znode.py /pulsar/bin/pulsar-managed-ledger-admin | ||
|
|
||
| # required by gen-yml-from-env.py | ||
| RUN apt-get install -y python-yaml | ||
|
|
||
| RUN apt-get install -y supervisor procps curl less | ||
|
|
||
|
|
||
| RUN mkdir -p /var/log/pulsar \ | ||
| && mkdir -p /var/run/supervisor/ \ | ||
| && mkdir -p /pulsar/ssl | ||
|
|
||
| COPY target/conf /etc/supervisord/conf.d/ | ||
| RUN mv /etc/supervisord/conf.d/supervisord.conf /etc/supervisord.conf | ||
|
|
||
| COPY target/ssl /pulsar/ssl/ | ||
|
|
||
| COPY target/java-test-functions.jar /pulsar/examples/ | ||
|
|
||
| ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE | ||
|
|
||
| RUN chown -R pulsar:0 /pulsar && chmod -R g=u /pulsar | ||
|
|
||
| # cleanup | ||
| RUN apt-get clean \ | ||
| && rm -rf /var/lib/apt/lists/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
| --> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <parent> | ||
| <groupId>org.apache.pulsar.tests</groupId> | ||
| <artifactId>docker-images</artifactId> | ||
| <version>2.8.0-SNAPSHOT</version> | ||
| </parent> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <artifactId>java-test-image</artifactId> | ||
| <name>Apache Pulsar :: Tests :: Docker Images :: Java Test Image</name> | ||
| <packaging>pom</packaging> | ||
|
|
||
| <profiles> | ||
| <profile> | ||
| <id>docker</id> | ||
| <activation> | ||
| <property> | ||
| <name>integrationTests</name> | ||
| </property> | ||
| </activation> | ||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.pulsar.tests</groupId> | ||
| <artifactId>java-test-functions</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| </dependency> | ||
| </dependencies> | ||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <artifactId>maven-dependency-plugin</artifactId> | ||
| <executions> | ||
| <execution> | ||
| <id>copy-installed</id> | ||
| <phase>package</phase> | ||
| <goals> | ||
| <goal>copy</goal> | ||
| </goals> | ||
| <configuration> | ||
| <artifactItems> | ||
| <artifactItem> | ||
| <groupId>org.apache.pulsar.tests</groupId> | ||
| <artifactId>java-test-functions</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <type>jar</type> | ||
| <overWrite>true</overWrite> | ||
| <outputDirectory>${project.build.directory}</outputDirectory> | ||
| <destFileName>java-test-functions.jar</destFileName> | ||
| </artifactItem> | ||
| <artifactItem> | ||
| <groupId>org.apache.pulsar</groupId> | ||
| <artifactId>pulsar-server-distribution</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <classifier>bin</classifier> | ||
| <type>tar.gz</type> | ||
| <overWrite>true</overWrite> | ||
| <outputDirectory>${project.build.directory}</outputDirectory> | ||
| <destFileName>pulsar-server-distribution-bin.tar.gz</destFileName> | ||
| </artifactItem> | ||
| </artifactItems> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <artifactId>maven-resources-plugin</artifactId> | ||
| <executions> | ||
| <execution> | ||
| <id>copy-files</id> | ||
| <phase>generate-resources</phase> | ||
| <goals> | ||
| <goal>copy-resources</goal> | ||
| </goals> | ||
| <configuration> | ||
| <outputDirectory>${project.build.directory}</outputDirectory> | ||
| <overwrite>true</overwrite> | ||
| <resources> | ||
| <resource> | ||
| <directory>${pulsar.basedir}/docker/pulsar/scripts</directory> | ||
| <targetPath>scripts</targetPath> | ||
| <filtering>false</filtering> | ||
| </resource> | ||
| <resource> | ||
| <directory>${project.basedir}/../latest-version-image/scripts</directory> | ||
| <targetPath>scripts</targetPath> | ||
| <filtering>false</filtering> | ||
| </resource> | ||
| <resource> | ||
| <directory>${project.basedir}/../latest-version-image/ssl</directory> | ||
| <targetPath>ssl</targetPath> | ||
| <filtering>false</filtering> | ||
| </resource> | ||
| <resource> | ||
| <directory>${project.basedir}/../latest-version-image/conf</directory> | ||
| <targetPath>conf</targetPath> | ||
| <filtering>false</filtering> | ||
| </resource> | ||
| </resources> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>com.spotify</groupId> | ||
| <artifactId>dockerfile-maven-plugin</artifactId> | ||
| <version>${dockerfile-maven.version}</version> | ||
| <executions> | ||
| <execution> | ||
| <id>default</id> | ||
| <phase>package</phase> | ||
| <goals> | ||
| <goal>build</goal> | ||
| </goals> | ||
| </execution> | ||
| <execution> | ||
| <id>add-latest-tag</id> | ||
| <phase>package</phase> | ||
| <goals> | ||
| <goal>tag</goal> | ||
| </goals> | ||
| <configuration> | ||
| <repository>${docker.organization}/java-test-image</repository> | ||
| <tag>latest</tag> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| <configuration> | ||
| <repository>${docker.organization}/java-test-image</repository> | ||
| <tag>${project.version}</tag> | ||
| <pullNewerImage>false</pullNewerImage> | ||
| <noCache>true</noCache> | ||
| <buildArgs> | ||
| <PULSAR_TARBALL>target/pulsar-server-distribution-bin.tar.gz</PULSAR_TARBALL> | ||
| </buildArgs> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </profile> | ||
| </profiles> | ||
| </project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters