advanced-settings.txt

.. _advanced-settings:

=======================
Advanced Kafka Settings
=======================

.. default-domain:: mongodb

.. facet::
   :name: genre
   :values: reference

.. contents:: On this page
   :local:
   :backlinks: none
   :depth: 2
   :class: singlecol

When you integrate {+rel-mig+} with Kafka, you can configure 
more advanced settings by updating your ``user.properties`` file.
Your ``user.properties`` file is located in an OS specific location. For more 
information, see :ref:`{+rel-mig+} File Locations <file-location>`.

.. _rm-user-properties:

User Properties
---------------

Provide optional configuration settings by updating your ``user.properties`` file. 

.. glossary::

   ``spring.profiles.active``
    *Default*: local

    {+rel-mig+} can be run in the following profiles: 

    - ``kafka``: Remote server using embedded server deployment.
    - ``confluent``: Remote server using Confluent Cloud for managed Kafka.

   ``server.port``
    *Default*: 8278

    The port the {+rel-mig+} application runs on.

   ``migrator.kafka.bootstrap.servers``
    A list of ``host:port`` pairs of your pre-existing kafka cluster.

    For example: ``kafka-1:9092, kafka-2:9093``. 

    For more information, see `Worker Configuration Properties 
    <https://docs.confluent.io/platform/current/connect/userguide.html#worker-configuration-properties-file>`__. 

   ``migrator.kafka.connect.url``
    The Kafka Connect host url using the ``host:port`` format.

    For example: ``https://localhost:8083``. 

   ``migrator.kafka.connect.metrics.jmx.service.url``
    The Kafka Connect Java Management Extensions (JMX) url. 

   ``migrator.kafka.connect.metrics.jmx.ssl``
    *Default*: false
    
    Set to ``true`` if SSL is enabled.

    You can add additonal remote Java Management Extensions (JMX) for SSL 
    configurations by prefixing them with ``migrator.kafka.connect.metrics.jmx.properties``.

   ``migrator.kafka.enable.topic.cleanup``
    *Default*: false
    
    Cleans up the topics created during the last migration job.

   ``migrator.connector.source.common.errors.max.retries``
    *Default*: 5

    For the source connector, it specifies the maximum number of retries on a 
    retriable failure. {+rel-mig+} attempts recovery from source database 
    retriable failures up to a set limit before the job fails.

   ``migrator.connector.source.common.errors.retry.initial.max.ms``
    *Default*: 30000 ms

    For the source connector, it specifies the delay in milliseconds to start 
    again after a retriable failure. The value is doubled after every retry but 
    does not exceed ``migrator.connector.source.common.errors.retry.delay.max.ms``.

   ``migrator.connector.source.common.errors.retry.delay.max.ms``
    *Default*: 60000 ms

    For the source connector, it specifies the maximum delay in milliseconds 
    between retries after a retriable failure.

   ``migrator.connector.sink.common.errors.max.retries``
    *Default*: 5
    
    For the sink connector, it specifies the maximum number of retries on a 
    retriable failure. {+rel-mig+} attempts recovery from source database 
    retriable failures up to a set limit before the job fails.

   ``migrator.connector.sink.common.errors.retry.initial.max.ms``
    *Default*: 30000 ms

    For the sink connector, it specifies the delay in milliseconds to start again 
    after a retriable failure. The value is doubled after every retry but does not exceed 
    ``migrator.connector.sink.common.errors.retry.delay.max.ms``.

   ``migrator.connector.sink.common.errors.retry.delay.max.ms``
    *Default*: 60000 ms

    For the sink connector, it specifies the maximum delay in milliseconds 
    between retries after a retriable failure.

Schema Registry Properties
--------------------------

Use {+rel-mig+} with any schema registry by adding properties to the ``user.properties`` 
file with the prefix ``migrator.connector.properties.common``.

.. glossary::

   ``migrator.connector.properties.common.key.converter``
    *Default*: ``migrator.connector.properties.common.key.converter``
    
    The convertor class for the message key.

   ``migrator.connector.properties.common.value.converter``
    *Default*: ``org.apache.kafka.connect.json.JsonConverter``
    
    The convertor class for the message value.

   ``migrator.connector.properties.common.key.converter.apicurio.registry.url``
    This specifies the URL of the Apicurio Registry for the key converter.

    It uses the ``http://{host}:{port}`` or ``https://{host}:{port}`` format.

   ``migrator.connector.properties.common.value.converter.apicurio.registry.url``
    This specifies the URL of the Apicurio Registry for the value converter.

    It uses the ``http://{host}:{port}`` or ``https://{host}:{port}`` format.

Source Connector Properties
---------------------------

{+rel-mig+} sets default `Debezium source connector properties 
<https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-connector-properties>`__
for each supported source database. The connector properties can be 
overridden by adding them in the ``user.properties`` file with the prefix 
``migrator.connector.source.common``. For example: 

.. code-block::

   migrator.connector.source.common.errors.max.retries: 0

In this example, the default value is five but it has now been set to zero. This means
there is no retries after a retriable failure.

Sink Connector Properties
-------------------------

{+rel-mig+} sets default Migrator `sink connector properties 
<https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html>`__
for each supported source database. The connector properties can be 
overridden by adding them in the ``user.properties`` file with the prefix 
``migrator.connector.sink.common``. For example:

.. code-block::

   migrator.connector.sink.common.errors.max.retries: 0

In this example, the default value is five but it has now been set to zero. This means
there are no retries.