Skip to content

Commit

Permalink
[FLINK-20043][kinesis] Add flink-sql-connector-kinesis module for SQL…
Browse files Browse the repository at this point in the history
… client

This closes apache#14049.
  • Loading branch information
dannycranmer authored and twalthr committed Nov 16, 2020
1 parent caa5104 commit 05b6124
Show file tree
Hide file tree
Showing 11 changed files with 189 additions and 4 deletions.
1 change: 1 addition & 0 deletions docs/_data/sql-connectors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,4 +121,5 @@ kinesis:
name: Kinesis
category: connector
maven: flink-connector-kinesis{{ site.scala_version_suffix }}
sql-url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kinesis{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kinesis{{site.scala_version_suffix}}-{{site.version}}.jar

8 changes: 7 additions & 1 deletion docs/dev/table/connectors/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ Flink natively support various connectors. The following tables list all availab
<td>Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{% link dev/table/connectors/kinesis.md %}">Amazon Kinesis Data Streams</a></td>
<td></td>
<td>Unbounded Scan</td>
<td>Streaming Sink</td>
</tr>
<tr>
<td><a href="{% link dev/table/connectors/jdbc.md %}">JDBC</a></td>
<td></td>
Expand Down Expand Up @@ -265,4 +271,4 @@ Make sure to always declare both timestamps and watermarks. Watermarks are requi

Please see the [Data Types]({% link dev/table/types.md %}) page about how to declare a type in SQL.

{% top %}
{% top %}
6 changes: 6 additions & 0 deletions docs/dev/table/connectors/index.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ Flink natively support various connectors. The following tables list all availab
<td>Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{% link dev/table/connectors/kinesis.md %}">Amazon Kinesis Data Streams</a></td>
<td></td>
<td>Unbounded Scan</td>
<td>Streaming Sink</td>
</tr>
<tr>
<td><a href="{% link dev/table/connectors/jdbc.zh.md %}">JDBC</a></td>
<td></td>
Expand Down
5 changes: 5 additions & 0 deletions docs/dev/table/connectors/kinesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ Dependencies
connector=connector
%}

{% assign connector = site.data.sql-connectors['kinesis'] %}
{% include sql-connector-download-table.html
connector=connector
%}

How to create a Kinesis data stream table
-----------------------------------------

Expand Down
5 changes: 5 additions & 0 deletions docs/dev/table/connectors/kinesis.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ To use the connector, add the following Maven dependency to your project:
</dependency>
{% endhighlight %}

{% assign connector = site.data.sql-connectors['kinesis'] %}
{% include sql-connector-download-table.html
connector=connector
%}

How to create a Kinesis data stream table
-----------------------------------------

Expand Down
6 changes: 6 additions & 0 deletions flink-connectors/flink-connector-kinesis/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,12 @@ under the License.
</relocation>
</relocations>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>.gitkeep</exclude>
</excludes>
</filter>
<filter>
<artifact>com.amazonaws:amazon-kinesis-producer</artifact>
<excludes>
Expand Down
111 changes: 111 additions & 0 deletions flink-connectors/flink-sql-connector-kinesis/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<artifactId>flink-connectors</artifactId>
<groupId>org.apache.flink</groupId>
<version>1.12-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>

<artifactId>flink-sql-connector-kinesis_${scala.binary.version}</artifactId>
<name>Flink : Connectors : SQL : Kinesis</name>

<packaging>jar</packaging>

<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>shade-flink</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeTestJar>false</shadeTestJar>
<artifactSet>
<includes>
<include>org.apache.flink:flink-connector-kinesis_${scala.binary.version}</include>
<include>com.fasterxml.jackson.core:jackson-core</include>
<include>com.fasterxml.jackson.core:jackson-databind</include>
<include>com.fasterxml.jackson.core:jackson-annotations</include>
<include>com.fasterxml.jackson.dataformat:jackson-dataformat-cbor</include>
<include>joda-time:joda-time</include>
<include>commons-codec:commons-codec</include>
<include>commons-io:commons-io</include>
<include>commons-lang:commons-lang</include>
<include>commons-logging:commons-logging</include>
<include>org.apache.commons:commons-lang3</include>
<include>com.google.guava:guava</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>codegen-resources/**</exclude>
<exclude>mozilla/public-suffix-list.txt</exclude>
<exclude>VersionInfo.java</exclude>
<exclude>mime.types</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.fasterxml.jackson</pattern>
<shadedPattern>org.apache.flink.kinesis.shaded.com.fasterxml.jackson</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.commons</pattern>
<shadedPattern>org.apache.flink.kinesis.shaded.org.apache.commons</shadedPattern>
</relocation>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>org.apache.flink.kinesis.shaded.com.google</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda.time</pattern>
<shadedPattern>org.apache.flink.kinesis.shaded.org.joda.time</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
flink-sql-connector-kinesis
Copyright 2014-2020 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

This project bundles the following dependencies under the Apache Software License 2.0. (http://www.apache.org/licenses/LICENSE-2.0.txt)

- joda-time:joda-time:2.5
- commons-io:commons-io:2.7
- commons-lang:commons-lang:2.6
- commons-logging:commons-logging:1.1.3
- commons-codec:commons-codec:1.13
- org.apache.commons:commons-lang3:3.3.2
- com.google.guava:guava:18.0
- com.fasterxml.jackson.core:jackson-annotations:2.10.1
- com.fasterxml.jackson.core:jackson-databind:2.10.1
- com.fasterxml.jackson.core:jackson-core:2.10.1
- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.10.1
1 change: 1 addition & 0 deletions flink-connectors/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ under the License.
<module>flink-sql-connector-hive-2.3.6</module>
<module>flink-sql-connector-hive-3.1.2</module>
<module>flink-sql-connector-kafka</module>
<module>flink-sql-connector-kinesis</module>
</modules>
</profile>
</profiles>
Expand Down
13 changes: 13 additions & 0 deletions flink-end-to-end-tests/flink-sql-client-test/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,13 @@ under the License.
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<!-- Used by maven-dependency-plugin -->
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-kinesis_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<!-- Used by maven-dependency-plugin -->
<groupId>org.apache.flink</groupId>
Expand Down Expand Up @@ -150,6 +157,12 @@ under the License.
<version>${project.version}</version>
<type>jar</type>
</artifactItem>
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-kinesis_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>jar</type>
</artifactItem>
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-elasticsearch6_${scala.binary.version}</artifactId>
Expand Down
18 changes: 15 additions & 3 deletions flink-end-to-end-tests/test-scripts/test_sql_client.sh
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,26 @@ for SQL_JAR in $SQL_JARS_DIR/*.jar; do
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/META-INF"* ]] && \
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/LICENSE"* ]] && \
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/NOTICE"* ]] && \
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/org/apache/avro"* ]] ; then
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/org/apache/avro"* ]] && \
# Following required by amazon-kinesis-producer in flink-connector-kinesis
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/amazon-kinesis-producer-native-binaries"* ]] && \
! [[ $EXTRACTED_FILE = "$EXTRACTED_JAR/cacerts"* ]] ; then
echo "Bad file in JAR: $EXTRACTED_FILE"
exit 1
fi
done

# check for proper factory
if [ ! -f $EXTRACTED_JAR/META-INF/services/org.apache.flink.table.factories.TableFactory ]; then
# check for proper legacy table factory
# Kinesis connector does not support legacy Table API
if [[ $SQL_JAR == *"flink-sql-connector-kinesis"* ]]; then
echo "Skipping Legacy Table API for: $SQL_JAR"
elif [ ! -f $EXTRACTED_JAR/META-INF/services/org.apache.flink.table.factories.TableFactory ]; then
echo "No legacy table factory found in JAR: $SQL_JAR"
exit 1
fi

# check for table factory
if [ ! -f $EXTRACTED_JAR/META-INF/services/org.apache.flink.table.factories.Factory ]; then
echo "No table factory found in JAR: $SQL_JAR"
exit 1
fi
Expand Down

0 comments on commit 05b6124

Please sign in to comment.