Documentation changes (apache#574)

site/_data/codebase.yaml

@@ -20,24 +20,39 @@
 directories:
 - name: pulsar-broker
   language: Java
-  description: Code related to the Pulsar broker
-  key_files:
-  - path:
-    purpose:
+  description: Code related to the Pulsar [broker](../../getting-started/ConceptsAndArchitecture#brokers).
 - name: pulsar-common
   language: Java, Protocol Buffers
-  description: Common assets
+  description: Assets related to Pulsar's message protocol.
 - name: dashboard
   language: python
+  description: The web client powering Pulsar's [dashboard](../../admin/Dashboard).
 - name: kubernetes
   language: YAML
-  description: Stuff for [Kubernetes deployment](../../deployment/Kubernetes)
+  description: "[Kubernetes](../../deployment/Kubernetes) configuration for Pulsar."
+- name: docker
+  language: Dockerfile, shell, Python
+  description: Assets used to build Pulsar's [Docker image](https://hub.docker.com)
 - name: managed-ledger
   language: Java
+  description: 
 - name: pulsar-broker-auth-athenz
 - name: pulsar-client
 - name: pulsar-spark
 - name: pulsar-storm
 - name: pulsar-testclient
 - name: src
   language: XML
+- name: protobuf
+  description: Some helpers for generating code from Protocol Buffer definitions.
+- name: all
+  description: Configuration for Maven's assembly plugin.
+  language: xml
+- name: bin
+  description: Executable scripts used for Pulsar administration
+  language: Bash
+- name: site
+  description: Assets used to generate this website
+  language: Markdown, Sass, HTML, JavaScript
+- name: .travis
+  description: Settings for [Travis CI](https://travis-ci.org), the continuous integration system used by Pulsar

site/_data/config/client.yaml

@@ -19,11 +19,17 @@
 
 configs:
 - name: webServiceUrl
+  description: The web URL for the cluster.
   default: http://localhost:8080/
 - name: brokerServiceUrl
+  description: The Pulsar protocol URL for the cluster.
   default: pulsar://localhost:6650/
 - name: authPlugin
+  description: The authentication plugin.
 - name: authParams
+  description: The authentication parameters for the cluster, as a comma-separated string.
 - name: useTls
+  description: Whether or not TLS authentication will be enforced in the cluster.
+  default: "false"
 - name: tlsAllowInsecureConnection
 - name: tlsTrustCertsFilePath

site/_data/features.yaml

@@ -17,9 +17,13 @@
 # under the License.
 #
 
+- title: Proven in production
+  content: Pulsar has run in production at Yahoo scale for over 3 years, with millions of messages per second across millions of topics.
+  endpoint: getting-started/ConceptsAndArchitecture
+
 - title: Horizontally scalable
-  content: Millions of independent topics and seamless capacity expansion by simply adding machines
-  endpoint: getting-started/ConceptsAndArchitecture/#Architectureoverview
+  content: Seamlessly expand capacity to hundreds of nodes
+  endpoint: getting-started/ConceptsAndArchitecture
 
 - title: Low latency with durability
   content: Designed for low publish latency (< 5ms) at scale with strong durabilty guarantees

site/_data/popovers.yaml

@@ -97,7 +97,7 @@ standalone:
 subscription:
   q: What is a subscription?
   def: |
-    "A lease on a topic established by a consumer. Pulsar has three subscription modes: exclusive, shared, and failover."
+    A lease on a topic established by a consumer. Pulsar has three subscription modes (exclusive, shared, and failover).
 tenant:
   q: What is a tenant?
   def: An administrative unit for allocating capacity and enforcing an authentication/authorization scheme. Tenants in Pulsar are managed at the property level.
@@ -107,6 +107,9 @@ topic:
 topic-lookup:
   q: What is topic lookup?
   def: A service provided by Pulsar brokers that enables connecting clients to automatically determine which Pulsar cluster is responsible for a topic (and thus where message traffic for the topic needs to be routed).
+unacknowledged:
+  q: What is an unacknowledged message?
+  def: A message that has been delivered to a consumer for processing but not yet confirmed as processed by the consumer.
 zookeeper:
   q: What is ZooKeeper?
   def: |

site/_data/sidebar.yaml

@@ -31,18 +31,15 @@ groups:
 - title: Deployment
   dir: deployment
   docs:
-  - title: Deploying a Pulsar instance
+  - title: Deploying on bare metal
     endpoint: InstanceSetup
   - title: Pulsar on Kubernetes
     endpoint: Kubernetes
+  - title: Pulsar on Google Container Engine
+    endpoint: Kubernetes/#pulsar-on-google-container-engine
+  - title: Pulsar on AWS
+    endpoint: Kubernetes/#pulsar-on-amazon-web-services
 
-- title: Concerns
-  dir: concerns
-  docs:
-  - title: Partitioned topics
-    endpoint: PartitionedTopics
-  - title: Retention and expiry
-    endpoint: RetentionExpiry
 - title: Pulsar administration
   dir: admin
   docs:
@@ -65,27 +62,39 @@ groups:
   - title: Modular load manager
     endpoint: ModularLoadManager
 
-- title: Pulsar applications
-  dir: applications
+- title: Client libraries
+  dir: clients
   docs:
-  - title: The Pulsar Java client
-    endpoint: JavaClient
-  - title: The Pulsar C++ client
-    endpoint: CppClient
-  - title: The Pulsar Python client
-    endpoint: PythonClient
+  - title: Java client
+    endpoint: Java
+  - title: C++ client
+    endpoint: Cpp
+  - title: Python client
+    endpoint: Python
   - title: WebSocket API
     endpoint: WebSocket
-  - title: Spark Streaming receiver
+
+- title: Adaptors
+  dir: adaptors
+  docs:
+  - title: Spark Streaming
     endpoint: PulsarSpark
-  - title: Apache Storm adaptor
+  - title: Apache Storm
     endpoint: PulsarStorm
-  - title: Simulation tools
-    endpoint: SimulationTools
+
+- title: Advanced
+  dir: advanced
+  docs:
+  - title: Partitioned topics
+    endpoint: PartitionedTopics
+  - title: Retention and expiry
+    endpoint: RetentionExpiry
 
 - title: Developing Pulsar
   dir: project
   docs:
+  - title: Simulation tools
+    endpoint: SimulationTools
   - title: Pulsar binary protocol
     endpoint: BinaryProtocol
   - title: Codebase

site/_includes/codebase.html

@@ -18,14 +18,17 @@
     under the License.
 
 -->
-{% for dir in site.data.codebase.directories %}
-<div class="card">
-  <div class="card-block">
-    <h2 class="card-title">{{ dir.name }}</h2>  <a href="{{ site.pulsar_repo }}/{{ dir.name }}"><i class="fa fa-github fa-lg"></i></a>
-    <p>{{ dir.language }}</p>
-    <div class="card-text">
+
+<section class="codebase">
+  {% for dir in site.data.codebase.directories %}
+  <div class="card">
+    <div class="card-block">
+      <h2 class="card-title">{{ dir.name }} <a href="{{ site.pulsar_repo }}/{{ dir.name }}"><i class="fa fa-github"></i></a></h2>
+      <section class="card-text">
       {{ dir.description | markdownify }}
+      {% if dir.language %}<span>Languages: {{ dir.language }}</span>{% endif %}
+      </section>
     </div>
   </div>
-</div>
-{% endfor %}
+  {% endfor %}
+</section>

site/_includes/explanations/admin-setup.md

@@ -1,10 +1,8 @@
-If you have [authentication](../../admin/Authz#authentication-providers) enabled in your Pulsar {% popover instance %}, then you will need to perform a few authentication-related setup steps to use the Pulsar [admin interface](../../admin/AdminInterface). Those steps will vary depending on if you're using the [`pulsar-admin` command-line interface](#the-pulsar-admin-command-line-tool), the [REST API](#rest-api), or the [Java admin client](#java-admin-client).
+Each of Pulsar's three admin interfaces---the [`pulsar-admin`](../../reference/CliTools#pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the [REST API](../../reference/RestApi)---requires some special setup if you have [authentication](../../admin/Authz#authentication-providers) enabled in your Pulsar {% popover instance %}.
 
-Before you get started using Pulsar's admin interface, you will need to complete a few setup steps that vary depending on whether you're using the [`pulsar-admin` CLI tool](#pulsar-admin-cli-tool), [Java API](#java-api), or [REST API](#rest-api) directly.
+### pulsar-admin
 
-### The `pulsar-admin` command-line tool
-
-If a Pulsar {% popover broker %} has [authentication](../../admin/Authz#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](../../reference/CliTools#pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](../../reference/Configuration#client) file. Here are the available parameters:
+If you have [authentication](../../admin/Authz#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](../../reference/CliTools#pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](../../reference/Configuration#client) file. Here are the available parameters:
 
 {% include config.html id="client" %}
 

site/_includes/explanations/broker-admin.md

@@ -7,7 +7,7 @@ Pulsar brokers consist of two components:
 
 * The [`brokers`](../../reference/CliTools#pulsar-admin-brokers) command of the [`pulsar-admin`](../../reference/CliTools#pulsar-admin) tool
 * The `/admin/brokers` endpoint of the admin [REST API](../../reference/RestApi)
-* The `brokers` method of the {% javadoc PulsarAdmin admin org.apache.pulsar.client.admin.PulsarAdmin %} object in the [Java API](../../applications/JavaClient)
+* The `brokers` method of the {% javadoc PulsarAdmin admin com.yahoo.pulsar.client.admin.PulsarAdmin %} object in the [Java API](../../clients/Java)
 
 In addition to being configurable when you start them up, brokers can also be [dynamically configured](#dynamic-broker-configuration).
 
@@ -42,7 +42,6 @@ broker1.use.org.com:8080
 admin.brokers().getActiveBrokers(clusterName)
 ```
 
-
 #### list of namespaces owned by a given broker
 
 It finds all namespaces which are owned and served by a given broker.  

site/_includes/explanations/cluster-admin.md

@@ -40,6 +40,39 @@ ClusterData clusterData = new ClusterData(
 admin.clusters().createCluster(clusterName, clusterData);
 ```
 
+### Initialize cluster metadata
+
+When provision a new cluster, you need to initialize that cluster's [metadata](../../getting-started/ConceptsAndArchitecture#metadata-store). When initializing cluster metadata, you need to specify all of the following:
+
+* The name of the cluster
+* The local ZooKeeper connection string for the cluster
+* The global ZooKeeper connection string for the entire instance
+* The web service URL for the cluster
+* A broker service URL enabling interaction with the {% popover brokers %} in the cluster
+
+You must initialize cluster metadata *before* starting up any [brokers](#managing-brokers) that will belong to the cluster.
+
+{% include admonition.html type="warning" title="No cluster metadata initialization through the REST API or the Java admin API" content='
+Unlike most other admin functions in Pulsar, cluster metadata initialization cannot be performed via the admin REST API or the admin Java client, as metadata initialization involves communicating with ZooKeeper directly. Instead, you can use the [`pulsar`](../../reference/CliTools#pulsar) CLI tool, in particular the [`initialize-cluster-metadata`](../../reference/CliTools#pulsar-initialize-cluster-metadata) command.
+' %}
+
+Here's an example cluster metadata initialization command:
+
+```shell
+bin/pulsar initialize-cluster-metadata \
+  --cluster us-west \
+  --zookeeper zk1.us-west.example.com:2181 \
+  --global-zookeeper zk1.us-west.example.com:2184 \
+  --web-service-url http://pulsar.us-west.example.com:8080/ \
+  --web-service-url-tls https://pulsar.us-west.example.com:8443/ \
+  --broker-service-url pulsar://pulsar.us-west.example.com:6650/ \
+  --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651/
+```
+
+You'll need to use `--*-tls` flags only if you're using [TLS authentication](../../admin/Authz#tls-client-auth) in your instance.
+
+Make sure to initialize
+
 ### Get configuration
 
 You can fetch the [configuration](../../reference/Configuration) for an existing cluster at any time.

site/_includes/explanations/install-package.md

@@ -9,7 +9,7 @@ Pulsar is currently available for **MacOS** and **Linux**. In order to use Pulsa
 
 To get started running Pulsar, download a binary tarball release in one of the following ways:
 
-* by clicking one of these two buttons:
+* by clicking one of these friendly buttons:
 
   <a href="{{ source_release_url }}" class="download-btn btn btn-lg" role="button" aria-pressed="true">Pulsar {{ site.current_version }} source release</a>
   <a href="{{ binary_release_url }}" class="download-btn btn btn-lg" role="button" aria-pressed="true">Pulsar {{ site.current_version }} binary release</a>
@@ -40,10 +40,14 @@ $ cd pulsar-{{ site.latest }}
 
 ## What your package contains
 
-| Directory | Contains                                                                                                                                                                                 |
-|:----------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `bin`     | Pulsar's [command-line tools](../../reference/CliTools), such as [`pulsar`](../../reference/CliTools#pulsar) and [`pulsar-admin`](../../reference/CliTools#pulsar-admin)                 |
-| `conf`    | Configuration files for Pulsar, including for [broker configuration](../../reference/Configuration#broker), [ZooKeeper configuration](../../reference/Configuration#zookeeper), and more |
-| `data`    | The data storage directory used by {% popover ZooKeeper %} and {% popover BookKeeper %}.                                                                                                 |
-| `lib`     | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar.                                                                                                         |
-| `logs`    | Logs created by the installation.                                                                                                                                                        |
+Both the source and binary packages contain the following directories:
+
+Directory | Contains
+:---------|:--------
+`bin` | Pulsar's [command-line tools](../../reference/CliTools), such as [`pulsar`](../../reference/CliTools#pulsar) and [`pulsar-admin`](../../reference/CliTools#pulsar-admin)
+`conf` | Configuration files for Pulsar, including for [broker configuration](../../reference/Configuration#broker), [ZooKeeper configuration](../../reference/Configuration#zookeeper), and more
+`data` | The data storage directory used by {% popover ZooKeeper %} and {% popover BookKeeper %}.
+`lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar.
+`logs` | Logs created by the installation.
+
+The source package contains all of the assets, specific to version {{ site.current_version}}, from the [Pulsar repository]({{ site.pulsar_repo }}).

site/_includes/explanations/partitioned-topic-admin.md

@@ -1,9 +1,17 @@
+You can use Pulsar's [admin API](../../admin/AdminInterface) to create and manage partitioned topics.
+
 In all of the instructions and commands below, the topic name structure is:
 
 {% include topic.html p="property" c="cluster" n="namespace" t="topic" %}
 
 ### Create
 
+Partitioned topics in Pulsar must be explicitly created. When creating a new partitioned topic you need to provide a name for the topic as well as the desired number of partitions.
+
+{% include admonition.html type="info" title="Global partitioned topics" content="
+If you'd like to create a [global]() partitioned topic, you need to create a partitioned topic using the instructions here and specify `global` as the cluster in the topic name.
+" %}
+
 #### pulsar-admin
 
 You can create partitioned topics using the [`create-partitioned-topic`](../../reference/CliTools#pulsar-admin-persistent-create-partitioned-topic) command and specifying the topic name as an argument and the number of partitions using the `-p` or `--partitions` flag. Here's an example:
@@ -23,14 +31,22 @@ $ bin/pulsar-admin persistent create-partitioned-topic \
 #### Java
 
 ```java
-admin.persistentTopics().createPartitionedTopic(persistentTopic, numPartitions);
+String topicName = "persistent://my-property/my-cluster-my-namespace/my-topic";
+int numPartitions = 4;
+admin.persistentTopics().createPartitionedTopic(topicName, numPartitions);
 ```
 
 ### Get metadata
 
+Partitioned topics have metadata associated with them that you can fetch as a JSON object. The following metadata fields are currently available:
+
+Field | Meaning
+:-----|:-------
+`partitions` | The number of partitions into which the topic is divided
+
 #### pulsar-admin
 
-You can see the see number of partitions (as a JSON object) in a partitioned topic using the [`get-partitioned-topic-metadata`](../../reference/CliTools#pulsar-admin-persistent-get-partitioned-topic) subcommand. Here's an example:
+You can see the see number of partitions in a partitioned topic using the [`get-partitioned-topic-metadata`](../../reference/CliTools#pulsar-admin-persistent-get-partitioned-topic) subcommand. Here's an example:
 
 ```shell
 $ pulsar-admin persistent get-partitioned-topic-metadata \
@@ -49,7 +65,8 @@ $ pulsar-admin persistent get-partitioned-topic-metadata \
 #### Java
 
 ```java
-admin.persistentTopics().getPartitionedTopicMetadata(persistentTopic);
+String topicName = "persistent://my-property/my-cluster-my-namespace/my-topic";
+admin.persistentTopics().getPartitionedTopicMetadata(topicName);
 ```
 
 ### Update