Skip to content

Latest commit

 

History

History
135 lines (78 loc) · 4.76 KB

io-overview.md

File metadata and controls

135 lines (78 loc) · 4.76 KB
Raw
id title sidebar_label
io-overview
Pulsar connector overview
Overview

Messaging systems are most powerful when you can easily use them with external systems like databases and other messaging systems.

Pulsar IO connectors enable you to easily create, deploy, and manage connectors that interact with external systems, such as Apache Cassandra, Aerospike, and many others.

Concept

Pulsar IO connectors come in two types: source and sink.

This diagram illustrates the relationship between source, Pulsar, and sink:

Pulsar IO diagram

Source

Sources feed data from external systems into Pulsar.

Common sources include other messaging systems and firehose-style data pipeline APIs.

For the complete list of Pulsar built-in source connectors, see source connector.

Sink

Sinks feed data from Pulsar into external systems.

Common sinks include other messaging systems and SQL and NoSQL databases.

For the complete list of Pulsar built-in sink connectors, see sink connector.

Processing guarantee

Processing guarantees are used to handle errors when writing messages to Pulsar topics.

Pulsar connectors and Functions use the same processing guarantees as below.

Delivery semantic Description
at-most-once Each message sent to a connector is to be processed once or not to be processed.
at-least-once Each message sent to a connector is to be processed once or more than once.
effectively-once Each message sent to a connector has one output associated with it.

Processing guarantees for connectors not just rely on Pulsar guarantee but also relate to external systems, that is, the implementation of source and sink.

  • Source: Pulsar ensures that writing messages to Pulsar topics respects to the processing guarantees. It is within Pulsar's control.

  • Sink: the processing guarantees rely on the sink implementation. If the sink implementation does not handle retries in an idempotent way, the sink does not respect to the processing guarantees.

Set

When creating a connector, you can set the processing guarantee with the following semantics:

  • ATLEAST_ONCE

  • ATMOST_ONCE

  • EFFECTIVELY_ONCE

If --processing-guarantees is not specified when creating a connector, the default semantic is ATLEAST_ONCE.

Here takes Admin CLI as an example. For more information about REST API or JAVA Admin API, see here.

$ bin/pulsar-admin sources create \
  --processing-guarantees ATMOST_ONCE \
  # Other source configs

For more information about the options of pulsar-admin sources create, see here.

$ bin/pulsar-admin sinks create \
  --processing-guarantees EFFECTIVELY_ONCE \
  # Other sink configs

For more information about the options of pulsar-admin sinks create, see here.

Update

After creating a connector, you can update the processing guarantee with the following semantics:

  • ATLEAST_ONCE

  • ATMOST_ONCE

  • EFFECTIVELY_ONCE

Here takes Admin CLI as an example. For more information about REST API or JAVA Admin API, see here.

$ bin/pulsar-admin sources update \
  --processing-guarantees EFFECTIVELY_ONCE \
  # Other source configs

For more information about the options of pulsar-admin sources update, see here.

$ bin/pulsar-admin sinks update \
  --processing-guarantees ATMOST_ONCE \
  # Other sink configs

For more information about the options of pulsar-admin sinks update, see here.

Work with connector

You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the Connector Admin CLI with sources and sinks subcommands.

Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via Connector Admin CLI or Functions Admin CLI, an instance is started on a worker. For more information, see Functions worker.