Skip to content

Commit

Permalink
[CELEBORN-1477] Using openapi-generator apache-httpclient library i…
Browse files Browse the repository at this point in the history
…nstead of `jersey2`

### What changes were proposed in this pull request?
We used `jersey2` library for celeborn-openapi-client before, and I found that there is dependencies lack issue for shaded celeborn-openapi-client.
I tried to raise a [PR apache#2640] to fix it, but seems It is difficult to maintain the dependencies transition from jersey dependencies.

And I received the suggestion from pan to migrate the library from jersey2 to `apache-httpclient`.

FYI: for https://openapi-generator.tech/docs/generators/java/

<img width="500" alt="image" src="https://github.com/user-attachments/assets/d102a7c9-46cd-4fd7-a2a0-7396a815776d">

To leverage the latest openapi-generator plugin, I upgrade the openapi-generator version to latest 7.7.0 and it requires JDK11+.
Due celeborn does not drop the Java8 support so far, so I include the generated code into repo and add user guide for re-generation.

### Why are the changes needed?

To fix dependencies leak issue and maintain the dependencies easily.

### Does this PR introduce _any_ user-facing change?

No, this SDK has not been released, so no user-facing change.

### How was this patch tested?

Testing with sample maven project.

pom.xml:
```
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>test_openapi</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.celeborn</groupId>
            <artifactId>celeborn-openapi-client_2.12</artifactId>
            <version>0.6.0-SNAPSHOT</version>
        </dependency>
    </dependencies>
</project>
```

Testing code:
```
package org.example;

import org.apache.celeborn.rest.v1.master.MasterApi;
import org.apache.celeborn.rest.v1.master.WorkerApi;
import org.apache.celeborn.rest.v1.master.invoker.ApiClient;

public class Main {
    public static void main(String[] args) throws Exception {

        String cmUrl = "http://***:9098";
        MasterApi masterApi  = new MasterApi(new ApiClient().setBasePath(cmUrl));
        System.out.println(masterApi.getMasterGroupInfo().getLeader().getAddress().split(":")[0]);
        WorkerApi workerApi = new WorkerApi(new ApiClient().setBasePath(cmUrl));
        System.out.println(workerApi.getWorkers());
        System.out.println(workerApi.getWorkerEvents());
    }
}
```

```
java -Dfile.encoding=UTF-8 -classpath /Users/fwang12/todo/test_openapi/target/classes:/Users/fwang12/todo/celeborn/openapi/openapi-client/target/celeborn-openapi-client_2.12-0.6.0-SNAPSHOT.jar org.example.Main
```

<img width="1727" alt="image" src="https://github.com/user-attachments/assets/2da8b126-be96-4c37-9a33-ba196024f2ba">

Closes apache#2641 from turboFei/appache_httpclient.

Lead-authored-by: Wang, Fei <[email protected]>
Co-authored-by: Fei Wang <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Signed-off-by: Shuang <[email protected]>
  • Loading branch information
3 people authored and RexXiong committed Jul 31, 2024
1 parent 41a0f7a commit 1515ed3
Show file tree
Hide file tree
Showing 84 changed files with 12,986 additions and 484 deletions.
2 changes: 2 additions & 0 deletions .rat-excludes
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,5 @@ build/scala-*/**
build/sbt-config/**
**/benchmarks/**
**/node_modules/**
**/src/main/java/org/apache/celeborn/rest/v1/**/ServerConfiguration.java
**/src/main/java/org/apache/celeborn/rest/v1/**/ServerVariable.java
5 changes: 3 additions & 2 deletions dev/deps/dependencies-server
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ hadoop-client-runtime/3.3.6//hadoop-client-runtime-3.3.6.jar
hk2-api/2.6.1//hk2-api-2.6.1.jar
hk2-locator/2.6.1//hk2-locator-2.6.1.jar
hk2-utils/2.6.1//hk2-utils-2.6.1.jar
httpclient5/5.3.1//httpclient5-5.3.1.jar
httpcore5-h2/5.2.4//httpcore5-h2-5.2.4.jar
httpcore5/5.2.4//httpcore5-5.2.4.jar
jackson-annotations/2.15.3//jackson-annotations-2.15.3.jar
jackson-core/2.15.3//jackson-core-2.15.3.jar
jackson-databind-nullable/0.2.6//jackson-databind-nullable-0.2.6.jar
Expand Down Expand Up @@ -137,12 +140,10 @@ scala-reflect/2.12.18//scala-reflect-2.12.18.jar
slf4j-api/1.7.36//slf4j-api-1.7.36.jar
snakeyaml/2.2//snakeyaml-2.2.jar
snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar
swagger-annotations/1.6.11//swagger-annotations-1.6.11.jar
swagger-annotations/2.2.1//swagger-annotations-2.2.1.jar
swagger-core/2.2.1//swagger-core-2.2.1.jar
swagger-integration/2.2.1//swagger-integration-2.2.1.jar
swagger-jaxrs2/2.2.1//swagger-jaxrs2-2.2.1.jar
swagger-models/1.6.11//swagger-models-1.6.11.jar
swagger-models/2.2.1//swagger-models-2.2.1.jar
swagger-ui/4.9.1//swagger-ui-4.9.1.jar
zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
14 changes: 14 additions & 0 deletions openapi/openapi-client/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Celeborn OpenAPI Client

To update the OpenAPI specification
- just update the specification under `openapi/openapi-client/src/main/openapi3/` and keep the schema definitions consistent between master and worker.
- Install JDK 11 or above by whatever mechanism is appropriate for your system, and set that version to be the default Java version (e.g., by setting env variable `JAVA_HOME`)
- run the following:
```sh
build/mvn -pl openapi/openapi-client clean package -Pgenerate
```
or
```sh
build/sbt "clean;celeborn-openapi-client/generate"
```
This will regenerate the OpenAPI data models + APIs in the celeborn-openapi-client SDK.
Loading

0 comments on commit 1515ed3

Please sign in to comment.