Skip to content

Commit

Permalink
Edits to DC/OS docs (apache#1013)
Browse files Browse the repository at this point in the history
  • Loading branch information
lucperkins authored and merlimat committed Dec 27, 2017
1 parent 3e7cd2c commit fcec34b
Show file tree
Hide file tree
Showing 3 changed files with 66 additions and 60 deletions.
6 changes: 6 additions & 0 deletions site/_sass/_docs.scss
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@


.rest-api-container, .docs-container {
blockquote {
border: 1px solid $sx-light-gray;
background-color: lighten($sx-light-gray, 20%);
padding: 0.75rem;
}

margin-top: $top-margin;

h4 + ul {
Expand Down
2 changes: 0 additions & 2 deletions site/docs/latest/advanced/Encryption.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ tags:
-->

# Pulsar Encryption

Pulsar encryption allows applications to encrypt messages at the producer and decrypt at the consumer. Encryption is performed using the public/private key pair configured by the application. Encrypted messages can only be decrypted by consumers with a valid key.

## Asymmetric and symmetric encryption
Expand Down
118 changes: 60 additions & 58 deletions site/docs/latest/deployment/dcos.md
Original file line number Diff line number Diff line change
@@ -1,175 +1,177 @@
---
title: Pulsar on DC/OS
subtitle: Get up and running Pulsar with one command on an Apache Mesos DC/OS cluster
subtitle: Get Pulsar up and running on an Apache Mesos DC/OS cluster with one command
logo: img/dcos/dcos-logo.png
---

[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system used for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool created and maintained by [Mesosphere](https://mesosphere.com/).

Apache Pulsar is available as a [Marathon Application Groups](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets.
Apache Pulsar is available as a [Marathon Application Group](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets.

## Prerequisites

In order to run Pulsar on DC/OS, we will need:
In order to run Pulsar on DC/OS, you will need the following:

* DC/OS version [1.9](https://dcos.io/docs/1.9/) or higher
* A [DC/OS cluster](https://dcos.io/install/) with at least three agent nodes
* The [DC/OS CLI tool](https://dcos.io/docs/1.9/usage/cli/install/) installed
* Download and save `PulsarGroups.json` from Pulsar github repo.
* The [`PulsarGroups.json`](https://github.com/apache/incubator-pulsar/blob/master/deploy/dcos/PulsarGroups.json) configuration file from the Pulsar GitHub repo.

```bash
$ curl -O https://raw.githubusercontent.com/apache/incubator-pulsar/master/deploy/dcos/PulsarGroups.json
```

Each node in the DC/OS-managed Mesos cluster must have at least:

* 4 CPU
* 4 GB of memory
* 60 GB of total persistent disk
* 60 GB of total persistent disk

Alternatively, you can change the configuration in `PulsarGroups.json` according to match your DC/OS cluster's resources.

Or, we could change the configuration in `PulsarGroups.json` according to the current cluster resources.
## Deploy Pulsar using the DC/OS command interface

## Deploy Pulsar use dcos command interface
You can deploy Pulsar on DC/OS using this command:

```shell
```bash
$ dcos marathon group add PulsarGroups.json
```

This command will deploy some docker container instances in 3 groups, which consist a Pulsar cluster:
This command will deploy Docker container instances in three groups, which together comprise a Pulsar cluster:

* 3 bookies (1 {% popover bookie %} on each agent node and 1 [bookie recovery](http://bookkeeper.apache.org/docs/latest/admin/autorecovery/) instance)
* 3 Pulsar {% popover brokers %} (1 broker on each node and 1 admin instance)
* 1 [Prometheus](http://prometheus.io/) instance and 1 [Grafana](https://grafana.com/) instance

* deploy 3 bookies, 1 `bookie` on each agent node; and 1 `bookie_recovery` instance.
* deploy 3 Pulsar brokers, 1 `broker` on each node; and 1 `broker_admin` instance.
* deploy 1 `prometheus` instance and 1 `grafana` instance.
{% include admonition.html type="success"
content="When running DC/OS, a ZooKeeper cluster is already running at `master.mesos:2181`, thus there's no need to install or start up ZooKeeper separately." %}

Since DC/OS is already deployed a zookeeper cluster serving at `master.mesos:2181`, we leverage it, and no need to deploy it ourselves.
After execute the command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), whose link is `http://m1.dcos` in this example, and we should see applications deploying.
After executing the `dcos` command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications in the process of deploying.

![DC/OS command executed](/img/dcos/command_execute.png)

![DC/OS command executed2](/img/dcos/command_execute2.png)

## The BookKeeper group

## BookKeeper Group

To watch the status of `BookKeeper` Cluster deployment, we could click into the 'bookkeeper' group in parent pulsar group picture.
To monitor the status of the BookKeeper cluster deployment, click on the **bookkeeper** group in the parent **pulsar** group.

![DC/OS bookkeeper status](/img/dcos/bookkeeper_status.png)

Now 3 bookies is shown as green colour, which means they are already deployed successfully and running.
At this point, 3 {% popover bookies %} should be shown as green, which means that they have been deployed successfully and are now running.

![DC/OS bookkeeper running](/img/dcos/bookkeeper_run.png)

We could also click into each Instance of bookie, to get more detailed info, such as the bookie running log.
You can also click into each bookie instance to get more detailed info, such as the bookie running log.

![DC/OS bookie log](/img/dcos/bookie_log.png)

And we could also get the BookKeeper cluster information in Zookeeper, whose link is `http://m1.dcos/exhibitor`, and could see 3 bookies under `available` directory.
To display information about the BookKeeper in ZooKeeper, you can visit [http://m1.dcos/exhibitor](http://m1.dcos/exhibitor). In this example, there are 3 bookies under the `available` directory.

![DC/OS bookkeeper in zk](/img/dcos/bookkeeper_in_zookeeper.png)

## The Pulsar broker Group

## Pulsar Broker Group

Similar to BookKeeper Group above, click into `brokers` group, and we will see brokers status.
Similar to the BookKeeper group above, click into the **brokers** to check the status of the Pulsar brokers.

![DC/OS broker status](/img/dcos/broker_status.png)

![DC/OS broker running](/img/dcos/broker_run.png)

We could also click into each Instance of broker, to get more detailed info, such as the broker running log.
You can also click into each broker instance to get more detailed info, such as the broker running log.

![DC/OS broker log](/img/dcos/broker_log.png)

And also get Broker cluster information in Zookeeper, which created directory `loadbalance` and `managed-ledgers`.
Broker cluster information in Zookeeper is also available through the web UI. In this example, you can see that that the `loadbalance` and `managed-ledgers` directories have been created.

![DC/OS broker in zk](/img/dcos/broker_in_zookeeper.png)


## Monitor Group

Monitor Group consist of Prometheus and Grafana.
The **monitory** group consists of Prometheus and Grafana.

![DC/OS monitor status](/img/dcos/monitor_status.png)

### Prometheus

We could click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.
Click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example.

![DC/OS prom endpoint](/img/dcos/prom_endpoint.png)

While click this endpoint above, and we could get into Prometheus dashboard, and the targets `http://192.168.65.121:9090/targets` tab, should contains all bookies and brokers.
If you click that endpoint, you'll see the Prometheus dashboard. The [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets) URL will display all the bookies and brokers.

![DC/OS prom targets](/img/dcos/prom_targets.png)

### Grafana

We could click into the instance of `grafana` to get the endpoint of Grafana, which is `192.168.65.121:3000` in this example.
Click into `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example.

![DC/OS grafana endpoint](/img/dcos/grafana_endpoint.png)

While click this endpoint above, and we could get into Grafana dashboard.
If you click that endpoint, you can access the Grafana dashbaord.

![DC/OS grafana targets](/img/dcos/grafana_dashboard.png)

## Run a simple Pulsar consumer and producer on DC/OS

## Run simple consumer and producer on DC/OS deployment.

At present we could get a deployed Pulsar cluster. and we can run simple consumer and producer on it now.
Now that we have a fully deployed Pulsar cluster, we can run a simple consumer and producer to show Pulsar on DC/OS in action.

### Download and prepare pulsar java tutorial
### Download and prepare the Pulsar Java tutorial

This code contains a simple Pulsar Consumer and Producer, and the `README.md` contains more information about it.
There's a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo that you can clone. This repo contains a simple Pulsar consumer and producer (more info can be found in the repo's `README` file).

```shell
$ git clone https://github.com/streamlio/pulsar-java-tutorial.git
```bash
$ git clone https://github.com/streamlio/pulsar-java-tutorial
```

Change `SERVICE_URL` from "pulsar://localhost:6650" to "pulsar://a1.dcos:6650" in both [ConsumerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [ProducerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java).
"pulsar://a1.dcos:6650" is the endpoint of broker service, we could get the endpoint details from DC/OS GUI of each broker instance. `a1.dcos` is one of DC/OS client agent, which runs a broker, it can also be replaced by the client agent IP address.
Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java).
The `pulsar://a1.dcos:6650` endpoint is for the broker service. Endpoint details for each broker instance can be fetched from the DC/OS GUI. `a1.dcos` is a DC/OS client agent, which runs a broker. This can also be replaced by the client agent IP address.

Change the message number from 10 to 10000000 in main method of [ProducerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java), so it will produce more message.
Now, change the message number from 10 to 10000000 in main method of [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) so that it will produce more messages.

Now compile the project code using command:

```shell
```bash
$ mvn clean package
```

### Run consumer and producer
### Run the consumer and producer

Execute command to run consumer:
Execute this command to run the consumer:

```shell
```bash
$ mvn exec:java -Dexec.mainClass="tutorial.ConsumerTutorial"
```

Execute command to run producer:
Execute this command to run the producer:

```shell
```bash
$ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"
```

And we could see the producer producing messages and consumer consuming messages.
You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI.

![DC/OS pulsar producer](/img/dcos/producer.png)

![DC/OS pulsar consumer](/img/dcos/consumer.png)


### View Grafana metric output

While producer and consumer is running, we could get some running metrics information from Grafana
While the producer and consumer are running, you can access running metrics information from Grafana.

![DC/OS pulsar dashboard](/img/dcos/metrics.png)


## Uninstall Pulsar

We can shut down and uninstall the `pulsar` from DC/OS at any time by 2 ways.
You can shut down and uninstall the `pulsar` application from DC/OS at any time in two ways:

Using DC/OS GUI, we could choose `Delete` at the right end of Pulsar group.
1. Using the DC/OS GUI, you can choose **Delete** at the right end of Pulsar group.

![DC/OS pulsar uninstall](/img/dcos/uninstall.png)
![DC/OS pulsar uninstall](/img/dcos/uninstall.png)

2. You can use the following command:

Or we could use command to do the uninstall:

```shell
$ dcos marathon group remove /pulsar
```
```bash
$ dcos marathon group remove /pulsar
```

0 comments on commit fcec34b

Please sign in to comment.