Skip to content

Latest commit

 

History

History
86 lines (66 loc) · 3.29 KB

jdbc-driver.adoc

File metadata and controls

86 lines (66 loc) · 3.29 KB

Kafka Connect JDBC connector and JDBC Drivers

Introduction

This is a very common error that people encounter when using the Kafka Connect JDBC connector:

Caused by: java.sql.SQLException: No suitable driver found for…

e.g.

Caused by: java.sql.SQLException: No suitable driver found for jdbc:oracle:thin:@10.10.10.10:1954/MYSERVICE

It means that either:

  1. You have specific the JDBC URL in the wrong format. Check against this useful list to make sure it’s correct.

  2. The JDBC driver for your database has not been correctly made available to Kafka Connect

🎥Video tutorial

Check out the recording at https://rmoff.dev/fix-jdbc-driver-video

  • 00:00 Introduction and background

  • 01:19 Installing JDBC driver on local install

  • 05:00 Determining the Kafka Connect JDBC plugin folder

  • 07:16 How to find the Kafka Connect worker log

  • 09:23 Viewing the data in Confluent Control Center

  • 11:05 Installing JDBC driver for Kafka Connect running on Docker

  • 11:50 Installing the JDBC driver at runtime vs baking an image

  • 14:29 Listing installed Kafka Connect plugins

  • 15:00 Creating the connector and figuring out network addresses for the database

  • 16:22 Using kafkacat to view the topics and data

Making the JDBC connector available to Kafka Connect

Follow this one weird trick to make it work:

  1. Put the JDBC driver in the same folder as the Kafka Connect JDBC plugin.

  2. Restart the Kafka Connect worker

Don’t muck about with CLASSPATH, or plugin.path, or anything else that the Interwebs might throw up. So long as the correct JDBC driver is within the same folder as the Kafka Connect plugin JAR (kafka-connect-jdbc-5.4.1.jar), you’re sorted.

Locating the Kafka Connect JDBC plugin

  1. Search for it

    find / -name kafka-connect-jdbc\*.jar

    If you find more than one folder, you need to determine which plugin Kafka Connect is loading.

  2. Check the Kafka Connect worker log (grep for jdbc and pluginLocation)

    INFO Loading plugin from: /Users/rmoff/confluent-platform/share/java/kafka-connect-jdbc (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/rmoff/confluent-platform/share/java/kafka-connect-jdbc/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
INFO Added plugin 'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
INFO Added plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)

Sample JDBC Source connector config

curl -i -X PUT http://localhost:8083/connectors/jdbc_source_oracle_01/config \
     -H "Content-Type: application/json" \
     -d '{
            "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
            "connection.url": "jdbc:oracle:thin:@oracle:1521/ORCLPDB1",
            "connection.user": "rmoff",
            "connection.password": "asgard",
            "mode":"bulk",
            "topic.prefix":"oracle_",
            "table.whitelist":"CUSTOMERS",
            "poll.interval.ms" : 10000
        }'