migrator-with-kafka.txt

.. _kafka-intro:

======================
Kafka Deployment Guide
======================

.. default-domain:: mongodb

.. facet::
   :name: genre
   :values: reference

.. contents:: On this page
   :local:
   :backlinks: none
   :depth: 1
   :class: singlecol

{+rel-mig+} uses `Apache Kafka 
<https://www.mongodb.com/docs/kafka-connector/current/introduction/kafka-connect/#apache-kafka>`__ 
as a robust transport layer to migrate data from a source relational database 
to MongoDB. {+rel-mig+} can be used as a Kafka Connect plugin that 
improves resilience and scalability of big data migration jobs.

Use Cases
---------

{+rel-mig+} with Kafka is intended for large and long running jobs.
Kafka's built-in resilience ensures the migration job can recover and continue 
running if any component becomes temporarily unavailable. 

Behavior
--------

Architecturally, {+rel-mig+} is deployed within Kafka as a Kafka 
Connect sink connector. Database events are captured using the open-source 
`Debezium tool <https://debezium.io/documentation/reference/stable/connectors/index.html>`__ 
for database change data capture. A Debezium Connector captures these events 
from your source database. 

.. image:: /img/kafka/kafka-behavior.png
   :alt: {+rel-mig+} with Kafka

Tasks
-----

You can use the following deployment methods to install {+rel-mig+} 
with Kafka:

.. list-table::
   :header-rows: 1

   * - Deployment Method
     - Description

   * - :ref:`<install-kafka-cluster>`

     - This deployment method is ideal if you are already running Kafka or 
       have your own approach to installing and configuring Kafka.

   * - :ref:`<install-confluent-kafka>` 

     - Confluent Cloud is an official MongoDB partner providing a fully-managed 
       cloud-based Kafka solution. This deployment method is ideal if you want the 
       reliability of Kafka without having to manage your own cluster.

Learn More
----------

- `Kafka and Kafka Connect <https://www.mongodb.com/docs/kafka-connector/current/introduction/kafka-connect/>`__
- :ref:`<advanced-settings>`
- `Debezium <https://debezium.io/documentation/reference/stable/install.html>`__

.. toctree::
   :hidden:
   :titlesonly:

   Self-Managed Kafka </installation/kafka-deployments/install-kafka-cluster>
   Confluent Cloud </installation/kafka-deployments/install-confluent-kafka>
   Advanced Kafka Settings </installation/kafka-deployments/advanced-settings.txt>