This is an implementation of the kafka stream for kafka versions 2.x.
A stream plugin for another version of kafka, or another stream, can be added in a similar fashion. Refer to documentation on Stream Ingestion Plugin for the specific interfaces to implement.
- How to build and release Pinot package with Kafka 2.x connector
./mvnw clean package -DskipTests -Pbin-dist
- How to use Kafka 2.x connector
Below is a sample
streamConfigs
used to create a realtime table with Kafka Stream(High) level consumer:
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "highLevel",
"stream.kafka.topic.name": "meetupRSVPEvents",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.hlc.zk.connect.string": "localhost:2191/kafka",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.zk.broker.url": "localhost:2191/kafka",
"stream.kafka.hlc.bootstrap.server": "localhost:19092"
}