Skip to content

Files

pinot-kafka-2.0

Pinot connector for kafka 2.x

This is an implementation of the kafka stream for kafka versions 2.x.

A stream plugin for another version of kafka, or another stream, can be added in a similar fashion. Refer to documentation on Stream Ingestion Plugin for the specific interfaces to implement.

  • How to build and release Pinot package with Kafka 2.x connector
./mvnw clean package -DskipTests -Pbin-dist
  • How to use Kafka 2.x connector Below is a sample streamConfigs used to create a realtime table with Kafka Stream(High) level consumer:
"streamConfigs": {
  "streamType": "kafka",
  "stream.kafka.consumer.type": "highLevel",
  "stream.kafka.topic.name": "meetupRSVPEvents",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
  "stream.kafka.hlc.zk.connect.string": "localhost:2191/kafka",
  "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
  "stream.kafka.zk.broker.url": "localhost:2191/kafka",
  "stream.kafka.hlc.bootstrap.server": "localhost:19092"
}