Pulsar website using docusaurus (apache#2206)

### Motivation

Improve the documentation and usability of the pulsar website. This moves the website and documentation to a new framework (https://docusaurus.io/)  which will make it easier to maintain going forward.

### Modifications

A new version of the website in site2 directory. Also updates the pulsar build docker to add the new website build dependencies.

### Result

A more usable website and documentation.

A preview of the site can be seen here: https://cckellogg.github.io/incubator-pulsar
*All the links and images might not work on this site since it's a test only site*

build/docker/Dockerfile

@@ -50,6 +50,13 @@ RUN gpg2 --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A17031138
 ENV PATH "$PATH:/usr/local/rvm/bin"
 RUN rvm install 2.4.1
 
+# Install nodejs and yarn
+RUN curl -sL https://deb.nodesource.com/setup_10.x | bash -
+RUN apt-get install -y nodejs
+RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
+RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
+RUN apt-get update && apt-get install yarn
+
 # Install PIP and PDoc
 RUN wget https://bootstrap.pypa.io/get-pip.py && python get-pip.py
 RUN pip install pdoc

pom.xml

@@ -1005,6 +1005,7 @@ flexible messaging model and an intuitive client API.</description>
             <exclude>site/_sass/font-awesome/**</exclude>
             <exclude>site/fonts/**</exclude>
             <exclude>site/img/**</exclude>
+            <exclude>site2/**</exclude>
             <exclude>generated-site/**</exclude>
             <exclude>.github/*.md</exclude>
             <exclude>**/.idea/*</exclude>

site2/.gitignore

@@ -0,0 +1,12 @@
+.DS_Store
+
+node_modules
+
+lib/core/metadata.js
+lib/core/MetadataBlog.js
+
+website/translated_docs
+website/build/
+website/yarn.lock
+website/node_modules
+website/i18n/*

site2/README.md

@@ -0,0 +1,24 @@
+
+
+# The Pulsar website and documentation
+
+This `README` is basically the meta-documentation for the Pulsar website and documentation. Here you'll find instructions on running the site locally
+
+## Tools
+
+Framework [Docusaurus](https://docusaurus.io/).
+
+Ensure you have the latest version of [Node](https://nodejs.org/en/download/) installed. We also recommend you install [Yarn](https://yarnpkg.com/en/docs/install) as well.
+
+> You have to be on Node >= 8.x and Yarn >= 1.5.
+
+
+## Running the site locally
+
+To run the site locally:
+
+```bash
+cd website
+yarn install
+yarn start
+```

site2/docs/adaptors-spark.md

@@ -0,0 +1,66 @@
+---
+id: adaptors-spark
+title: Pulsar adaptor for Apache Spark
+sidebar_label: Apache Spark
+---
+
+The Spark Streaming receiver for Pulsar is a custom receiver that enables Apache [Spark Streaming](https://spark.apache.org/streaming/) to receive data from Pulsar.
+
+An application can receive data in [Resilient Distributed Dataset](https://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds) (RDD) format via the Spark Streaming Pulsar receiver and can process it in a variety of ways.
+
+## Prerequisites
+
+To use the receiver, include a dependency for the `pulsar-spark` library in your Java configuration.
+
+### Maven
+
+If you're using Maven, add this to your `pom.xml`:
+
+```xml
+<!-- in your <properties> block -->
+<pulsar.version>pulsar:version</pulsar.version>
+
+<!-- in your <dependencies> block -->
+<dependency>
+  <groupId>org.apache.pulsar</groupId>
+  <artifactId>pulsar-spark</artifactId>
+  <version>${pulsar.version}</version>
+</dependency>
+```
+
+### Gradle
+
+If you're using Gradle, add this to your `build.gradle` file:
+
+```groovy
+def pulsarVersion = "pulsar:version"
+
+dependencies {
+    compile group: 'org.apache.pulsar', name: 'pulsar-spark', version: pulsarVersion
+}
+```
+
+## Usage
+
+Pass an instance of `SparkStreamingPulsarReceiver` to the `receiverStream` method in `JavaStreamingContext`:
+
+```java
+SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("pulsar-spark");
+JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(5));
+
+ClientConfiguration clientConf = new ClientConfiguration();
+ConsumerConfiguration consConf = new ConsumerConfiguration();
+String url = "pulsar://localhost:6650/";
+String topic = "persistent://public/default/topic1";
+String subs = "sub1";
+
+JavaReceiverInputDStream<byte[]> msgs = jssc
+        .receiverStream(new SparkStreamingPulsarReceiver(clientConf, consConf, url, topic, subs));
+```
+
+
+## Example
+
+You can find a complete example [here](https://github.com/apache/incubator-pulsar/tree/master/pulsar-spark/src/test/java/org/apache/pulsar/spark/example/SparkStreamingPulsarReceiverExample.java).
+In this example, the number of messages which contain the string "Pulsar" in received messages is counted.
+

site2/docs/adaptors-storm.md

@@ -0,0 +1,104 @@
+---
+id: adaptors-storm
+title: Pulsar adaptor for Apache Storm
+sidebar_label: Apache Storm
+---
+
+Pulsar Storm is an adaptor for integrating with [Apache Storm](http://storm.apache.org/) topologies. It provides core Storm implementations for sending and receiving data.
+
+An application can inject data into a Storm topology via a generic Pulsar spout, as well as consume data from a Storm topology via a generic Pulsar bolt.
+
+## Using the Pulsar Storm Adaptor
+
+Include dependency for Pulsar Storm Adaptor:
+
+```xml
+<dependency>
+  <groupId>org.apache.pulsar</groupId>
+  <artifactId>pulsar-storm</artifactId>
+  <version>${pulsar.version}</version>
+</dependency>
+```
+
+## Pulsar Spout
+
+The Pulsar Spout allows for the data published on a topic to be consumed by a Storm topology. It emits a Storm tuple based on the message received and the `MessageToValuesMapper` provided by the client.
+
+The tuples that fail to be processed by the downstream bolts will be re-injected by the spout with an exponential backoff, within a configurable timeout (the default is 60 seconds) or a configurable number of retries, whichever comes first, after which it is acknowledged by the consumer. Here's an example construction of a spout:
+
+```java
+// Configure a Pulsar Client
+ClientConfiguration clientConf = new ClientConfiguration();
+
+// Configure a Pulsar Consumer
+ConsumerConfiguration consumerConf = new ConsumerConfiguration();  
+
+@SuppressWarnings("serial")
+MessageToValuesMapper messageToValuesMapper = new MessageToValuesMapper() {
+
+    @Override
+    public Values toValues(Message msg) {
+        return new Values(new String(msg.getData()));
+    }
+
+    @Override
+    public void declareOutputFields(OutputFieldsDeclarer declarer) {
+        // declare the output fields
+        declarer.declare(new Fields("string"));
+    }
+};
+
+// Configure a Pulsar Spout
+PulsarSpoutConfiguration spoutConf = new PulsarSpoutConfiguration();
+spoutConf.setServiceUrl("pulsar://broker.messaging.usw.example.com:6650");
+spoutConf.setTopic("persistent://my-property/usw/my-ns/my-topic1");
+spoutConf.setSubscriptionName("my-subscriber-name1");
+spoutConf.setMessageToValuesMapper(messageToValuesMapper);
+
+// Create a Pulsar Spout
+PulsarSpout spout = new PulsarSpout(spoutConf, clientConf, consumerConf);
+```
+
+## Pulsar Bolt
+
+The Pulsar bolt allows data in a Storm topology to be published on a topic. It publishes messages based on the Storm tuple received and the `TupleToMessageMapper` provided by the client.
+
+A partitioned topic can also be used to publish messages on different topics. In the implementation of the `TupleToMessageMapper`, a "key" will need to be provided in the message which will send the messages with the same key to the same topic. Here's an example bolt:
+
+```java
+// Configure a Pulsar Client
+ClientConfiguration clientConf = new ClientConfiguration();
+
+// Configure a Pulsar Producer  
+ProducerConfiguration producerConf = new ProducerConfiguration();
+
+@SuppressWarnings("serial")
+TupleToMessageMapper tupleToMessageMapper = new TupleToMessageMapper() {
+
+    @Override
+    public Message toMessage(Tuple tuple) {
+        String receivedMessage = tuple.getString(0);
+        // message processing
+        String processedMsg = receivedMessage + "-processed";
+        return MessageBuilder.create().setContent(processedMsg.getBytes()).build();
+    }
+
+    @Override
+    public void declareOutputFields(OutputFieldsDeclarer declarer) {
+        // declare the output fields
+    }
+};
+
+// Configure a Pulsar Bolt
+PulsarBoltConfiguration boltConf = new PulsarBoltConfiguration();
+boltConf.setServiceUrl("pulsar://broker.messaging.usw.example.com:6650");
+boltConf.setTopic("persistent://my-property/usw/my-ns/my-topic2");
+boltConf.setTupleToMessageMapper(tupleToMessageMapper);
+
+// Create a Pulsar Bolt
+PulsarBolt bolt = new PulsarBolt(boltConf, clientConf);
+```
+
+## Example
+
+You can find a complete example [here](https://github.com/apache/incubator-pulsar/tree/master/pulsar-storm/src/test/java/org/apache/pulsar/storm/example/StormExample.java).

site2/docs/admin-api-brokers.md

@@ -0,0 +1,158 @@
+---
+id: admin-api-brokers
+title: Managing Brokers
+sidebar_label: Brokers
+---
+
+Pulsar brokers consist of two components:
+
+1. An HTTP server exposing a [REST interface](reference-rest-api.md) administration and {% popover topic %} lookup.
+2. A dispatcher that handles all Pulsar {% popover message %} transfers.
+
+{% popover Brokers %} can be managed via:
+
+* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool
+* The `/admin/brokers` endpoint of the admin [REST API](reference-rest-api.md)
+* The `brokers` method of the {% javadoc PulsarAdmin admin org.apache.pulsar.client.admin.PulsarAdmin %} object in the [Java API](client-libraries-java.md)
+
+In addition to being configurable when you start them up, brokers can also be [dynamically configured](#dynamic-broker-configuration).
+
+{% include admonition.html type="info" content="
+See the [Configuration](reference-configuration.md#broker) page for a full listing of broker-specific configuration parameters.
+" %}
+
+## Brokers resources
+
+### List active brokers
+
+Fetch all available active brokers that are serving traffic.  
+
+#### pulsar-admin
+
+
+```shell
+$ pulsar-admin brokers list use
+```
+
+```
+broker1.use.org.com:8080
+```
+
+###### REST
+
+{% endpoint GET /admin/brokers/:cluster %}
+
+[More info](reference-rest-api.md#/admin/brokers/:cluster)
+
+###### Java
+
+```java
+admin.brokers().getActiveBrokers(clusterName)
+```
+
+#### list of namespaces owned by a given broker
+
+It finds all namespaces which are owned and served by a given broker.  
+
+###### CLI
+
+```shell
+$ pulsar-admin brokers namespaces use \
+  --url broker1.use.org.com:8080
+```
+
+```json
+{
+  "my-property/use/my-ns/0x00000000_0xffffffff": {
+    "broker_assignment": "shared",
+    "is_controlled": false,
+    "is_active": true
+  }
+}
+```
+###### REST
+
+{% endpoint GET /admin/brokers/:cluster/:broker:/ownedNamespaces %}
+
+###### Java
+
+```java
+admin.brokers().getOwnedNamespaces(cluster,brokerUrl);
+```
+
+### Dynamic broker configuration
+
+One way to configure a Pulsar {% popover broker %} is to supply a [configuration](reference-configuration.md#broker) when the broker is [started up](reference-cli-tools.md#pulsar-broker).
+
+But since all broker configuration in Pulsar is stored in {% popover ZooKeeper %}, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values.
+
+* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more.
+* In the Pulsar admin [REST API](reference-rest-api.md), dynamic configuration is managed through the `/admin/brokers/configuration` endpoint.
+
+### Update dynamic configuration
+
+#### pulsar-admin
+
+The [`update-dynamic-config`](reference-pulsar-admin.md#brokers-update-dynamic-config) subcommand will update existing configuration. It takes two arguments: the name of the parameter and the new value. Here's an example for the [`brokerShutdownTimeoutMs`](reference-configuration.md#broker-brokerShutdownTimeoutMs) parameter:
+
+```shell
+$ pulsar-admin brokers update-dynamic-config brokerShutdownTimeoutMs 100
+```
+
+#### REST API
+
+{% endpoint POST /admin/brokers/configuration/:configName/:configValue %}
+
+[More info](reference-rest-api.md#/admin/brokers/configuration/:configName/:configValue)
+
+#### Java
+
+```java
+admin.brokers().updateDynamicConfiguration(configName, configValue);
+```
+
+### List updated values
+
+Fetch a list of all potentially updatable configuration parameters.
+
+#### pulsar-admin
+
+```shell
+$ pulsar-admin brokers list-dynamic-config
+brokerShutdownTimeoutMs
+```
+
+#### REST API
+
+{% endpoint GET /admin/brokers/configuration %}
+
+[More info](reference-rest-api.md#/admin/brokers/configuration)
+
+#### Java
+
+```java
+admin.brokers().getDynamicConfigurationNames();
+```
+
+### List all
+
+Fetch a list of all parameters that have been dynamically updated.
+
+#### pulsar-admin
+
+```shell
+$ pulsar-admin brokers get-all-dynamic-config
+brokerShutdownTimeoutMs:100
+```
+
+#### REST API
+
+{% endpoint GET /admin/brokers/configuration/values %}
+
+[More info](reference-rest-api.md#/admin/brokers/configuration/values)
+
+#### Java
+
+```java
+admin.brokers().getAllDynamicConfigurations();
+```