Skip to content

Commit

Permalink
[Doc] Re-organize built-in connector guide (apache#5670)
Browse files Browse the repository at this point in the history
  • Loading branch information
Anonymitaet authored and sijie committed Nov 19, 2019
1 parent 56517d5 commit 9dd8dfd
Show file tree
Hide file tree
Showing 5 changed files with 194 additions and 49 deletions.
2 changes: 1 addition & 1 deletion site2/docs/io-cdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Currently, Pulsar has the following CDC connectors.

Name|Java Class
|---|---
[Canal source connector](io-cdc-canal.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java)
[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java)
[Debezium source connector](io-cdc-debezium.md)|<li>[org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)<br/><li>[org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)<br/><li>[org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)

For more information about Canal and Debezium, see the information below.
Expand Down
198 changes: 169 additions & 29 deletions site2/docs/io-connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,37 +12,177 @@ Using any of these connectors is as easy as writing a simple connector and runni

Pulsar has various source connectors, which are sorted alphabetically as below.

Name|Java class
|---|---
[Canal](io-canal-source.md) |[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java)
[Debezium MySQL](io-debezium-source.md#example-of-mysql)|[org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
[Debezium PostgreSQL](io-debezium-source.md#example-of-postgresql)|[org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
[File](io-file-source.md)|[org.apache.pulsar.io.file.FileSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSource.java)
[Flume](io-flume-source.md)|[org.apache.pulsar.io.flume.FlumeConnector.java](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/FlumeConnector.java)
[Twitter firehose](io-twitter-source.md)|[org.apache.pulsar.io.twitter.TwitterFireHose.java](https://github.com/apache/pulsar/blob/master/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java)
[Kafka](io-kafka-source.md)|[org.apache.pulsar.io.kafka.KafkaAbstractSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java)
[Kinesis](io-kinesis-source.md)|[org.apache.pulsar.io.kinesis.KinesisSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSource.java)
[Netty](io-netty-source.md)|[org.apache.pulsar.io.netty.NettySource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java)
[RabbitMQ](io-rabbitmq-source.md)|[org.apache.pulsar.io.rabbitmq.RabbitMQSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java)
### Canal

* [Configuration](io-canal-source.md#configuration)

* [Example](io-canal-source.md#usage)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java)


### Debezium MySQL

* [Configuration](io-debezium-source.md#configuration)

* [Example](io-debezium-source.md#example-of-mysql)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)

### Debezium PostgreSQL

* [Configuration](io-debezium-source.md#configuration)

* [Example](io-debezium-source.md#example-of-postgresql)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)


### File

* [Configuration](io-file-source.md#configuration)

* [Example](io-file-source.md#usage)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSource.java)

### Flume

* [Configuration](io-flume-source.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/FlumeConnector.java)

### Twitter firehose

* [Configuration](io-twitter-source.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java)

### Kafka

* [Configuration](io-kafka-source.md#configuration)

* [Example](io-kafka-source.md#usage)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java)

### Kinesis

* [Configuration](io-kinesis-source.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSource.java)

### Netty

* [Configuration](io-netty-source.md#configuration)

* [Example of TCP](io-netty-source.md#tcp)

* [Example of HTTP](io-netty-source.md#http)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java)

### RabbitMQ

* [Configuration](io-rabbitmq-source.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java)

## Sink connector

Pulsar has various sink connectors, which are sorted alphabetically as below.

Name|Java class
|---|---
[Aerospike](io-aerospike-sink.md)|[org.apache.pulsar.io.aerospike.AerospikeStringSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)
[Cassandra](io-cassandra-sink.md)|[org.apache.pulsar.io.cassandra.CassandraStringSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)
[ElasticSearch](io-elasticsearch-sink.md)|[org.apache.pulsar.io.elasticsearch.ElasticSearchSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java)
[Flume](io-flume-sink.md)|[org.apache.pulsar.io.flume.sink.StringSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/sink/StringSink.java)
[HBase](io-hbase.md)|[org.apache.pulsar.io.hbase.HbaseAbstractConfig.java](https://github.com/apache/pulsar/blob/master/pulsar-io/hbase/src/main/java/org/apache/pulsar/io/hbase/HbaseAbstractConfig.java)
[HDFS2](io-hdfs2-sink.md)|[org.apache.pulsar.io.hdfs2.AbstractHdfsConnector.java](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/AbstractHdfsConnector.java)
[HDFS3](io-hdfs3-sink.md)|[org.apache.pulsar.io.hdfs3.AbstractHdfsConnector.java](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs3/src/main/java/org/apache/pulsar/io/hdfs3/AbstractHdfsConnector.java)
[InfluxDB](io-influxdb-sink.md)|[org.apache.pulsar.io.influxdb.InfluxDBAbstractSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/influxdb/src/main/java/org/apache/pulsar/io/influxdb/InfluxDBAbstractSink.java)
[JDBC](io-jdbc-sink.md)|[org.apache.pulsar.io.jdbc.JdbcAbstractSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/src/main/java/org/apache/pulsar/io/jdbc/JdbcAbstractSink.java)
[Kafka](io-kafka-sink.md)|[org.apache.pulsar.io.kafka.KafkaAbstractSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java)
[Kinesis](io-kinesis-sink.md)|[org.apache.pulsar.io.kinesis.KinesisSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java)
[MongoDB](io-mongo-sink.md)|[org.apache.pulsar.io.mongodb.MongoSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/mongo/src/main/java/org/apache/pulsar/io/mongodb/MongoSink.java)
[RabbitMQ](io-rabbitmq-sink.md)|[org.apache.pulsar.io.rabbitmq.RabbitMQSink.java](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSink.java)
[Redis](io-redis-sink.md)|[org.apache.pulsar.io.redis.RedisAbstractConfig.java](https://github.com/apache/pulsar/blob/master/pulsar-io/redis/src/main/java/org/apache/pulsar/io/redis/RedisAbstractConfig.java)
[Solr](io-solr-sink.md)|[org.apache.pulsar.io.solr.SolrSinkConfig.java](https://github.com/apache/pulsar/blob/master/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrSinkConfig.java)
### Aerospike

* [Configuration](io-aerospike-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)

### Cassandra

* [Configuration](io-cassandra-sink.md#configuration)

* [Example](io-cassandra-sink.md#usage)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)

### ElasticSearch

* [Configuration](io-elasticsearch-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java)

### Flume

* [Configuration](io-flume-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/java/org/apache/pulsar/io/flume/sink/StringSink.java)

### HBase

* [Configuration](io-hbase.md)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hbase/src/main/java/org/apache/pulsar/io/hbase/HbaseAbstractConfig.java)

### HDFS2

* [Configuration](io-hdfs2-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/AbstractHdfsConnector.java)

### HDFS3

* [Configuration](io-hdfs3-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/hdfs3/src/main/java/org/apache/pulsar/io/hdfs3/AbstractHdfsConnector.java)

### InfluxDB

* [Configuration](io-influxdb-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/influxdb/src/main/java/org/apache/pulsar/io/influxdb/InfluxDBAbstractSink.java)

### JDBC

* [Configuration](io-jdbc-sink.md#configuration)

* [Example](io-jdbc-sink.md#usage)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/jdbc/src/main/java/org/apache/pulsar/io/jdbc/JdbcAbstractSink.java)

### Kafka

* [Configuration](io-kafka-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java)

### Kinesis

* [Configuration](io-kinesis-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java)

### MongoDB

* [Configuration](io-mongo-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/mongo/src/main/java/org/apache/pulsar/io/mongodb/MongoSink.java)

### RabbitMQ

* [Configuration](io-rabbitmq-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSink.java)

### Redis

* [Configuration](io-redis-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/redis/src/main/java/org/apache/pulsar/io/redis/RedisAbstractConfig.java)

### Solr

* [Configuration](io-solr-sink.md#configuration)

* [Java class](https://github.com/apache/pulsar/blob/master/pulsar-io/solr/src/main/java/org/apache/pulsar/io/solr/SolrSinkConfig.java)

2 changes: 2 additions & 0 deletions site2/docs/io-kafka-sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ This guide explains how to configure and use the Kafka sink connector.

The configuration of the Kafka sink connector has the following parameters.

### Property

| Name | Type| Required | Default | Description
|------|----------|---------|-------------|-------------|
| `bootstrapServers` |String| true | " " (empty string) | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster. |
Expand Down
37 changes: 22 additions & 15 deletions site2/docs/io-kinesis-sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ The Kinesis sink connector pulls data from Pulsar and persists data into Amazon

## Configuration

The configuration of the Kinesis sink connector has the following parameters.
The configuration of the Kinesis sink connector has the following property.

### Parameter
### Property

| Name | Type|Required | Default | Description
|------|----------|----------|---------|-------------|
Expand All @@ -22,6 +22,26 @@ The configuration of the Kinesis sink connector has the following parameters.
`awsCredentialPluginName`|String|false|" " (empty string)|The fully-qualified class name of implementation of {@inject: github:`AwsCredentialProviderPlugin`:/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/AwsCredentialProviderPlugin.java}. <br/><br/>It is a factory class which creates an AWSCredentialsProvider that is used by Kinesis sink. <br/><br/>If it is empty, the Kinesis sink creates a default AWSCredentialsProvider which accepts json-map of credentials in `awsCredentialPluginParam`.
`awsCredentialPluginParam`|String |false|" " (empty string)|The JSON parameter to initialize `awsCredentialsProviderPlugin`.

### Built-in plugins

The following are built-in `AwsCredentialProviderPlugin` plugins:

* `org.apache.pulsar.io.kinesis.AwsDefaultProviderChainPlugin`

This plugin takes no configuration, it uses the default AWS provider chain.

For more information, see [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

* `org.apache.pulsar.io.kinesis.STSAssumeRoleProviderPlugin`

This plugin takes a configuration (via the `awsCredentialPluginParam`) that describes a role to assume when running the KCL.

This configuration takes the form of a small json document like:

```json
{"roleArn": "arn...", "roleSessionName": "name"}
```

### Example

Before using the Kinesis sink connector, you need to create a configuration file through one of the following methods.
Expand Down Expand Up @@ -50,16 +70,3 @@ Before using the Kinesis sink connector, you need to create a configuration file
messageFormat: "ONLY_RAW_PAYLOAD"
retainOrdering: "true"
```

### Built-in `AwsCredentialProviderPlugin` plugins

#### `org.apache.pulsar.io.kinesis.AwsDefaultProviderChainPlugin`
This plugin takes no configuration, it uses the default AWS provider chain. See the [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) for more details

#### `org.apache.pulsar.io.kinesis.STSAssumeRoleProviderPlugin`
This plugin takes a configuration (via the `awsCredentialPluginParam`) that describes a role to assume when running the KCL.

This configuration takes the form of a small json document like:
```Json
{"roleArn": "arn...", "roleSessionName": "name"}
```
4 changes: 0 additions & 4 deletions site2/docs/io-kinesis-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,6 @@ The configuration of the Kinesis source connector has the following properties.

Before using the Kinesis source connector, you need to create a configuration file through one of the following methods.


### Property


* JSON

```json
Expand Down

0 comments on commit 9dd8dfd

Please sign in to comment.