From 85a1c6c49e8a602df3f9326eb97cfec44540169e Mon Sep 17 00:00:00 2001 From: Rajan Dhabalia Date: Fri, 28 Jul 2017 17:24:46 -0700 Subject: [PATCH] add non-persistent topic documentation (#532) --- .../explanations/non-persistent-topics.md | 61 +++++++++++++++++++ .../ConceptsAndArchitecture.md | 6 +- 2 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 site/_includes/explanations/non-persistent-topics.md diff --git a/site/_includes/explanations/non-persistent-topics.md b/site/_includes/explanations/non-persistent-topics.md new file mode 100644 index 0000000000000..6e383c2135bc4 --- /dev/null +++ b/site/_includes/explanations/non-persistent-topics.md @@ -0,0 +1,61 @@ +{% include admonition.html type="success" title='Notice' content=" +This feature is still in experimental mode and implementation details may change in future release. +" %} + +As name suggests, non-persist topic does not persist messages into any durable storage disk unlike persistent topic where messages are durably persisted on multiple disks. + +Therefore, if you are using persistent delivery, messages are persisted to disk/database so that they will survive a broker restart or subscriber failover. While using non-persistent delivery, if you kill a broker or subscriber is disconnected then subscriber will lose all in-transit messages. So, client may see message loss with non-persistent topic. + +- In non-persistent topic, as soon as broker receives published message, it immediately delivers this message to all connected subscribers without persisting them into any storage. So, if subscriber gets disconnected with broker then broker will not be able to deliver those in-transit messages and subscribers will never be able to receive those messages again. Broker also drops a message for the consumer, if consumer does not have enough permit to consume message, or consumer TCP channel is not writable. Therefore, consumer receiver queue size (to accommodate enough permits) and TCP-receiver window size (to keep channel writable) should be configured properly to avoid message drop for that consumer. +- Broker only allows configured number of in-flight messages per client connection. So, if producer tries to publish messages higher than this rate, then broker silently drops those new incoming messages without processing and delivering them to the subscribers. However, broker acknowledges with special message-id (`msg-id: -1:-1`) for those dropped messages to signal producer about the message drop. + +### Performance + +Non-persistent messaging is usually faster than persistent messaging because broker does not persist messages and immediately sends ack back to producer as soon as that message deliver to all connected subscribers. Therefore, producer sees comparatively low publish latency with non-persistent topic. + + +## Client API + + +A topic name will look like: + +``` +non-persistent://my-property/us-west/my-namespace/my-topic +``` + +Producer and consumer can connect to non-persistent topic in a similar way, as persistent topic except topic name must start with `non-persistent`. + +Non-persistent topic supports all 3 different subscription-modes: **Exclusive**, **Shared**, **Failover** which are already explained in details at [GettingStarted](../../getting-started/ConceptsAndArchitecture.md). + + +### Consumer API + +```java +PulsarClient client = PulsarClient.create("pulsar://localhost:6650"); + +Consumer consumer = client.subscribe( + "non-persistent://sample/standalone/ns1/my-topic", + "my-subscribtion-name"); +``` + +### Producer API + +```java +PulsarClient client = PulsarClient.create("pulsar://localhost:6650"); + +Producer producer = client.createProducer( + "non-persistent://sample/standalone/ns1/my-topic"); +``` + +### Broker configuration + +Sometimes, there would be a need to configure few dedicated brokers in a cluster, to just serve non-persistent topics. + +Broker configuration for enabling broker to own only configured type of topics + +``` +# It disables broker to load persistent topics +enablePersistentTopics=false +# It enables broker to load non-persistent topics +enableNonPersistentTopics=true +``` diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md b/site/docs/latest/getting-started/ConceptsAndArchitecture.md index 9be53bdf94e3f..e44130097c359 100644 --- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md +++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md @@ -79,7 +79,7 @@ As in other pub-sub systems, topics in Pulsar are named channels for transmittin | Topic name component | Description | |:---------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `persistent` | Identifies a topic for which all messages are durably [persisted](#persistent-storage) on disk (that means on multiple disks unless the {% popover broker %} is {% popover standalone %}). Pulsar currently only supports persistent topics. | +| `persistent` | It identifies type of topic. Pulsar supports two kind of topics: persistent and non-persistent. In persistent topic, all messages are durably [persisted](#persistent-storage) on disk (that means on multiple disks unless the {% popover broker %} is {% popover standalone %}), whereas [non-persistent](#non-persistent-topics) topic does not persist message into storage disk. | | `property` | The topic's {% popover tenant %} within the instance. Tenants are essential to {% popover multi-tenancy %} in Pulsar and can be spread across clusters. | | `cluster` | Where the topic is located. Typically there will be one {% popover cluster %} for each geographical region or data center. | | `namespace` | The administrative unit of the topic, which acts as a grouping mechanism for related topics. Most topic configuration is performed at the namespace level. Each property (tenant) can have multiple namespaces. | @@ -119,6 +119,10 @@ In the diagram above, Consumer-C-1 is the master consumer while Consumer-C-2 wou {% include explanations/partitioned-topics.md %} +### Non-persistent topics + +{% include explanations/non-persistent-topics.md %} + ## Architecture overview At the highest level, a Pulsar {% popover instance %} is composed of one or more Pulsar {% popover clusters %}. Clusters within an instance can [replicate](#replicate) data amongst themselves.