Skip to content

Commit

Permalink
Upgrade k8s java-client to fix dependency resolution (apache#10260)
Browse files Browse the repository at this point in the history 
### Motivation 

- sundr library has been upgraded to 0.22.1 version which contains a fix a dependency resolution issue, sundrio/sundrio#168
  - various workarounds for sundr library dependency issues can be removed

- newer version of k8s java-client has fewer dependencies

### Modifications

- upgrade io.kubernetes:java-client version from 10.0.0 to 12.0.0
- add dependency management rules for com.google.errorprone:error_prone_annotations and com.google.j2objc:j2objc-annotations so that the dependency versions are consistent with the other dependencies.
- adapt license files to the modifications
- adapt usage of `coreClient.listNamespacedPod` since the api has changed (1 new parameter, passes as null)
- remove workarounds that have been in place to address issues with sundr library dependency resolution issue
- reformat `pulsar-sql/presto-distribution/pom.xml`
  • Loading branch information
@lhotari
lhotari authored Apr 20, 2021
1 parent 762fea3 commit 715e588
Show file tree
Hide file tree
Showing 8 changed files with 343 additions and 378 deletions.
2 changes: 0 additions & 2 deletions .github/workflows/ci-maven-cache-update.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,5 @@ jobs:
- name: Download dependencies
if: ${{ (github.event_name == 'schedule' || steps.changes.outputs.poms == 'true') && steps.cache.outputs.cache-hit != 'true' }}
run: |
# workaround issue https://github.com/sundrio/sundrio/issues/168
perl -0777 -i.original -pe 's/<dependency>\s+<groupId>io.sundr<\/groupId>.*?<\/dependency>//sg' pulsar-functions/runtime/pom.xml pulsar-functions/secrets/pom.xml || true
# download dependencies, ignore errors
mvn -B -fn -ntp ${{ matrix.mvn_arguments }} dependency:go-offline
16 changes: 6 additions & 10 deletions distribution/server/src/assemble/LICENSE.bin.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -323,11 +323,10 @@ The Apache Software License, Version 2.0
* Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
* Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.1.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.17.0.jar
* Joda -- org.joda-joda-convert-2.2.1.jar
* Bitbucket -- org.bitbucket.b_c-jose4j-0.7.2.jar
* Bitbucket -- org.bitbucket.b_c-jose4j-0.7.6.jar
* Gson
- com.google.code.gson-gson-2.8.6.jar
- io.gsonfire-gson-fire-1.8.4.jar
- io.gsonfire-gson-fire-1.8.5.jar
* Guava
- com.google.guava-guava-30.1-jre.jar
- com.google.guava-failureaccess-1.0.1.jar
Expand Down Expand Up @@ -445,7 +444,7 @@ The Apache Software License, Version 2.0
- org.eclipse.jetty.websocket-websocket-servlet-9.4.39.v20210325.jar
* SnakeYaml -- org.yaml-snakeyaml-1.26.jar
* RocksDB - org.rocksdb-rocksdbjni-6.10.2.jar
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.3.4.jar
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.5.1.jar
* Apache Thrifth - org.apache.thrift-libthrift-0.12.0.jar
* OkHttp and OkHttp3
- com.squareup.okhttp-okhttp-2.7.4.jar
Expand Down Expand Up @@ -489,12 +488,9 @@ The Apache Software License, Version 2.0
* @FreeBuilder
- org.inferred-freebuilder-1.14.9.jar
* Kubernetes Client
- io.kubernetes-client-java-10.0.0.jar
- io.kubernetes-client-java-api-10.0.0.jar
- io.kubernetes-client-java-proto-10.0.0.jar
* Joda Time
- joda-time-2.10.1.jar
- joda-time-joda-time-2.10.1.jar
- io.kubernetes-client-java-12.0.0.jar
- io.kubernetes-client-java-api-12.0.0.jar
- io.kubernetes-client-java-proto-12.0.0.jar
* Dropwizard
- io.dropwizard.metrics-metrics-core-3.2.5.jar
- io.dropwizard.metrics-metrics-graphite-3.2.5.jar
Expand Down
16 changes: 14 additions & 2 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,7 @@ flexible messaging model and an intuitive client API.</description>
<jaxb-api>2.3.1</jaxb-api>
<javax.activation.version>1.2.0</javax.activation.version>
<jna.version>4.2.0</jna.version>
<kubernetesclient.version>10.0.0</kubernetesclient.version>
<sundr.version>0.21.0</sundr.version>
<kubernetesclient.version>12.0.0</kubernetesclient.version>
<nsq-client.version>1.0</nsq-client.version>
<cron-utils.version>9.1.3</cron-utils.version>
<spring-context.version>5.3.1</spring-context.version>
Expand Down Expand Up @@ -235,6 +234,7 @@ flexible messaging model and an intuitive client API.</description>
<errorprone.version>2.5.1</errorprone.version>
<errorprone.javac.version>9+181-r4173-1</errorprone.javac.version>
<errorprone-slf4j.version>0.1.4</errorprone-slf4j.version>
<j2objc-annotations.version>1.3</j2objc-annotations.version>
<lightproto-maven-plugin.version>0.4</lightproto-maven-plugin.version>

<!-- Used to configure rename.netty.native. Libs -->
Expand Down Expand Up @@ -1061,6 +1061,18 @@ flexible messaging model and an intuitive client API.</description>
<optional>true</optional>
</dependency>

<dependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
<version>${errorprone.version}</version>
</dependency>

<dependency>
<groupId>com.google.j2objc</groupId>
<artifactId>j2objc-annotations</artifactId>
<version>${j2objc-annotations.version}</version>
</dependency>

</dependencies>
</dependencyManagement>

Expand Down
62 changes: 19 additions & 43 deletions pulsar-functions/runtime/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
Expand Down Expand Up @@ -61,39 +61,15 @@
</dependency>

<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>${kubernetesclient.version}</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>io.sundr</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.sundr</groupId>
<artifactId>sundr-codegen</artifactId>
<version>${sundr.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.sundr</groupId>
<artifactId>sundr-core</artifactId>
<version>${sundr.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-broker-common</artifactId>
<version>${project.version}</version>
</dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>${kubernetesclient.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-broker-common</artifactId>
<version>${project.version}</version>
</dependency>

</dependencies>

Expand Down Expand Up @@ -130,15 +106,15 @@
<configuration>
<excludeFilterFile>${basedir}/src/main/resources/findbugsExclude.xml</excludeFilterFile>
</configuration>
<!-- <executions>-->
<!-- <execution>-->
<!-- <id>check</id>-->
<!-- <phase>verify</phase>-->
<!-- <goals>-->
<!-- <goal>check</goal>-->
<!-- </goals>-->
<!-- </execution>-->
<!-- </executions>-->
<!-- <executions>-->
<!-- <execution>-->
<!-- <id>check</id>-->
<!-- <phase>verify</phase>-->
<!-- <goals>-->
<!-- <goal>check</goal>-->
<!-- </goals>-->
<!-- </execution>-->
<!-- </executions>-->
</plugin>
</plugins>
</build>
Expand Down
10 changes: 5 additions & 5 deletions ...ntime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,7 @@ public void deleteStatefulSet() throws InterruptedException {
try {
response = coreClient.listNamespacedPod(jobNamespace, null, null,
null, null, labels,
null, null, null, null);
null, null, null, null, null);
} catch (ApiException e) {

String errorMsg = e.getResponseBody() != null ? e.getResponseBody() : e.getMessage();
Expand Down Expand Up @@ -1145,7 +1145,7 @@ private List<V1ContainerPort> getPrometheusContainerPorts() {

public static String createJobName(Function.FunctionDetails functionDetails, String jobName) {
return jobName == null ? createJobName(functionDetails.getTenant(),
functionDetails.getNamespace(), functionDetails.getName()) :
functionDetails.getNamespace(), functionDetails.getName()) :
createJobName(jobName, functionDetails.getTenant(),
functionDetails.getNamespace(), functionDetails.getName());
}
Expand All @@ -1157,16 +1157,16 @@ private static String toValidPodName(String ori) {
private static String toValidLabelName(String ori) {
return left(ori.toLowerCase().replaceAll("[^a-zA-Z0-9-_\\.]", "-").replaceAll("^[^a-zA-Z0-9]", "0").replaceAll("[^a-zA-Z0-9]$", "0"), maxLabelSize);
}

private static String createJobName(String jobName, String tenant, String namespace, String functionName) {
final String convertedJobName = toValidPodName(jobName);
// use of customRuntimeOptions 'jobName' may cause naming collisions,
// use of customRuntimeOptions 'jobName' may cause naming collisions,
// add a short hash here to avoid it
final String hashName = String.format("%s-%s-%s-%s", jobName, tenant, namespace, functionName);
final String shortHash = DigestUtils.sha1Hex(hashName).toLowerCase().substring(0, 8);
return convertedJobName + "-" + shortHash;
}

private static String createJobName(String tenant, String namespace, String functionName) {
final String jobNameBase = String.format("%s-%s-%s", tenant, namespace, functionName);
final String jobName = "pf-" + jobNameBase;
Expand Down
35 changes: 6 additions & 29 deletions pulsar-functions/secrets/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.pulsar</groupId>
Expand All @@ -31,34 +31,11 @@
<name>Pulsar Functions :: Secrets</name>

<dependencies>
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>${kubernetesclient.version}</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>io.sundr</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.sundr</groupId>
<artifactId>sundr-codegen</artifactId>
<version>${sundr.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.sundr</groupId>
<artifactId>sundr-core</artifactId>
<version>${sundr.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>${kubernetesclient.version}</version>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
Expand Down
2 changes: 1 addition & 1 deletion pulsar-sql/presto-distribution/LICENSE
View file
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ The Apache Software License, Version 2.0
- httpclient-4.5.13.jar
- httpcore-4.4.13.jar
* Error Prone Annotations
- error_prone_annotations-2.3.4.jar
- error_prone_annotations-2.5.1.jar
* Esri Geometry API For Java
- esri-geometry-api-2.2.2.jar
* Failsafe
Expand Down
Loading

0 comments on commit 715e588

Please sign in to comment.