[Doc][SQL] add explanations for how to get last message in topics (ap…

…ache#14412)

site2/docs/sql-deployment-configurations.md

@@ -110,20 +110,15 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
-A frequently asked question is why my latest message not showing up when querying with Pulsar SQL.
-It's not a bug but controlled by a setting, by default BookKeeper LAC only advanced when subsequent entries are added.
-If there is no subsequent entries added, the last entry written will not be visible to readers until the ledger is closed.
-This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly read from BookKeeper ledger.
-We can add following setting to change the behavior:
-In Broker config, set
-bookkeeperExplicitLacIntervalInMills > 0
-bookkeeperUseV2WireProtocol=false
-
-And in Presto config, set
-pulsar.bookkeeper-explicit-interval > 0
-pulsar.bookkeeper-use-v2-protocol=false
-
-However,keep in mind that using bk V3 protocol will introduce additional GC overhead to BK as it uses Protobuf.
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
 
 ## Query data from existing Presto clusters
 

site2/website/versioned_docs/version-2.7.0/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.broker-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.7.1/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.broker-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.7.2/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.broker-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.7.3/sql-deployment-configurations.md

@@ -34,14 +34,23 @@ pulsar.broker-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.
 
 ```bash
 $ wget pulsar:binary_release_url
 ```
-
 ## Deploy a new cluster
 
 Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration for deployment is the same for the Pulsar SQL worker. 

site2/website/versioned_docs/version-2.7.4/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.broker-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.8.0/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.8.1/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.8.2/sql-deployment-configurations.md

@@ -34,14 +34,23 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.
 
 ```bash
 $ wget pulsar:binary_release_url
 ```
-
 ## Deploy a new cluster
 
 Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), the configuration for deployment is the same for the Pulsar SQL worker. 

site2/website/versioned_docs/version-2.9.0/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.

site2/website/versioned_docs/version-2.9.1/sql-deployment-configurations.md

@@ -34,6 +34,16 @@ pulsar.web-service-url=http://localhost:8080,localhost:8081,localhost:8082
 pulsar.zookeeper-uri=localhost1,localhost2:2181
 ```
 
+**Note: by default, Pulsar SQL does not get the last message in a topic**. It is by design and controlled by settings. By default, BookKeeper LAC only advances when subsequent entries are added. If there is no subsequent entry added, the last written entry is not visible to readers until the ledger is closed. This is not a problem for Pulsar which uses managed ledger, but Pulsar SQL directly reads from BookKeeper ledger. 
+
+If you want to get the last message in a topic, set the following configurations:
+
+1. For the broker configuration, set `bookkeeperExplicitLacIntervalInMills` > 0 in `broker.conf` or `standalone.conf`.
+
+2. For the Presto configuration, set `pulsar.bookkeeper-explicit-interval` > 0 and `pulsar.bookkeeper-use-v2-protocol=false`.
+
+However, using BookKeeper V3 protocol introduces additional GC overhead to BK as it uses Protobuf.
+
 ## Query data from existing Presto clusters
 
 If you already have a Presto cluster, you can copy the Presto Pulsar connector plugin to your existing cluster. Download the archived plugin package with the following command.