Skip to content

Commit

Permalink
add non-persistent topic documentation (apache#532)
Browse files Browse the repository at this point in the history
  • Loading branch information
rdhabalia authored and merlimat committed Jul 29, 2017
1 parent 875767f commit 85a1c6c
Show file tree
Hide file tree
Showing 2 changed files with 66 additions and 1 deletion.
61 changes: 61 additions & 0 deletions site/_includes/explanations/non-persistent-topics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{% include admonition.html type="success" title='Notice' content="
This feature is still in experimental mode and implementation details may change in future release.
" %}

As name suggests, non-persist topic does not persist messages into any durable storage disk unlike persistent topic where messages are durably persisted on multiple disks.

Therefore, if you are using persistent delivery, messages are persisted to disk/database so that they will survive a broker restart or subscriber failover. While using non-persistent delivery, if you kill a broker or subscriber is disconnected then subscriber will lose all in-transit messages. So, client may see message loss with non-persistent topic.

- In non-persistent topic, as soon as broker receives published message, it immediately delivers this message to all connected subscribers without persisting them into any storage. So, if subscriber gets disconnected with broker then broker will not be able to deliver those in-transit messages and subscribers will never be able to receive those messages again. Broker also drops a message for the consumer, if consumer does not have enough permit to consume message, or consumer TCP channel is not writable. Therefore, consumer receiver queue size (to accommodate enough permits) and TCP-receiver window size (to keep channel writable) should be configured properly to avoid message drop for that consumer.
- Broker only allows configured number of in-flight messages per client connection. So, if producer tries to publish messages higher than this rate, then broker silently drops those new incoming messages without processing and delivering them to the subscribers. However, broker acknowledges with special message-id (`msg-id: -1:-1`) for those dropped messages to signal producer about the message drop.

### Performance

Non-persistent messaging is usually faster than persistent messaging because broker does not persist messages and immediately sends ack back to producer as soon as that message deliver to all connected subscribers. Therefore, producer sees comparatively low publish latency with non-persistent topic.


## Client API


A topic name will look like:

```
non-persistent://my-property/us-west/my-namespace/my-topic
```

Producer and consumer can connect to non-persistent topic in a similar way, as persistent topic except topic name must start with `non-persistent`.

Non-persistent topic supports all 3 different subscription-modes: **Exclusive**, **Shared**, **Failover** which are already explained in details at [GettingStarted](../../getting-started/ConceptsAndArchitecture.md).


### Consumer API

```java
PulsarClient client = PulsarClient.create("pulsar://localhost:6650");

Consumer consumer = client.subscribe(
"non-persistent://sample/standalone/ns1/my-topic",
"my-subscribtion-name");
```

### Producer API

```java
PulsarClient client = PulsarClient.create("pulsar://localhost:6650");

Producer producer = client.createProducer(
"non-persistent://sample/standalone/ns1/my-topic");
```

### Broker configuration

Sometimes, there would be a need to configure few dedicated brokers in a cluster, to just serve non-persistent topics.

Broker configuration for enabling broker to own only configured type of topics

```
# It disables broker to load persistent topics
enablePersistentTopics=false
# It enables broker to load non-persistent topics
enableNonPersistentTopics=true
```
6 changes: 5 additions & 1 deletion site/docs/latest/getting-started/ConceptsAndArchitecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ As in other pub-sub systems, topics in Pulsar are named channels for transmittin

| Topic name component | Description |
|:---------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `persistent` | Identifies a topic for which all messages are durably [persisted](#persistent-storage) on disk (that means on multiple disks unless the {% popover broker %} is {% popover standalone %}). Pulsar currently only supports persistent topics. |
| `persistent` | It identifies type of topic. Pulsar supports two kind of topics: persistent and non-persistent. In persistent topic, all messages are durably [persisted](#persistent-storage) on disk (that means on multiple disks unless the {% popover broker %} is {% popover standalone %}), whereas [non-persistent](#non-persistent-topics) topic does not persist message into storage disk. |
| `property` | The topic's {% popover tenant %} within the instance. Tenants are essential to {% popover multi-tenancy %} in Pulsar and can be spread across clusters. |
| `cluster` | Where the topic is located. Typically there will be one {% popover cluster %} for each geographical region or data center. |
| `namespace` | The administrative unit of the topic, which acts as a grouping mechanism for related topics. Most topic configuration is performed at the namespace level. Each property (tenant) can have multiple namespaces. |
Expand Down Expand Up @@ -119,6 +119,10 @@ In the diagram above, Consumer-C-1 is the master consumer while Consumer-C-2 wou

{% include explanations/partitioned-topics.md %}

### Non-persistent topics

{% include explanations/non-persistent-topics.md %}

## Architecture overview

At the highest level, a Pulsar {% popover instance %} is composed of one or more Pulsar {% popover clusters %}. Clusters within an instance can [replicate](#replicate) data amongst themselves.
Expand Down

0 comments on commit 85a1c6c

Please sign in to comment.