forked from apache/pulsar
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
3e7cd2c
commit fcec34b
Showing
3 changed files
with
66 additions
and
60 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,175 +1,177 @@ | ||
--- | ||
title: Pulsar on DC/OS | ||
subtitle: Get up and running Pulsar with one command on an Apache Mesos DC/OS cluster | ||
subtitle: Get Pulsar up and running on an Apache Mesos DC/OS cluster with one command | ||
logo: img/dcos/dcos-logo.png | ||
--- | ||
|
||
[DC/OS](https://dcos.io/) (the <strong>D</strong>ata<strong>C</strong>enter <strong>O</strong>perating <strong>S</strong>ystem) is a distributed operating system used for deploying and managing applications and systems on [Apache Mesos](http://mesos.apache.org/). DC/OS is an open-source tool created and maintained by [Mesosphere](https://mesosphere.com/). | ||
|
||
Apache Pulsar is available as a [Marathon Application Groups](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets. | ||
Apache Pulsar is available as a [Marathon Application Group](https://mesosphere.github.io/marathon/docs/application-groups.html), which runs multiple applications as manageable sets. | ||
|
||
## Prerequisites | ||
|
||
In order to run Pulsar on DC/OS, we will need: | ||
In order to run Pulsar on DC/OS, you will need the following: | ||
|
||
* DC/OS version [1.9](https://dcos.io/docs/1.9/) or higher | ||
* A [DC/OS cluster](https://dcos.io/install/) with at least three agent nodes | ||
* The [DC/OS CLI tool](https://dcos.io/docs/1.9/usage/cli/install/) installed | ||
* Download and save `PulsarGroups.json` from Pulsar github repo. | ||
* The [`PulsarGroups.json`](https://github.com/apache/incubator-pulsar/blob/master/deploy/dcos/PulsarGroups.json) configuration file from the Pulsar GitHub repo. | ||
|
||
```bash | ||
$ curl -O https://raw.githubusercontent.com/apache/incubator-pulsar/master/deploy/dcos/PulsarGroups.json | ||
``` | ||
|
||
Each node in the DC/OS-managed Mesos cluster must have at least: | ||
|
||
* 4 CPU | ||
* 4 GB of memory | ||
* 60 GB of total persistent disk | ||
* 60 GB of total persistent disk | ||
|
||
Alternatively, you can change the configuration in `PulsarGroups.json` according to match your DC/OS cluster's resources. | ||
|
||
Or, we could change the configuration in `PulsarGroups.json` according to the current cluster resources. | ||
## Deploy Pulsar using the DC/OS command interface | ||
|
||
## Deploy Pulsar use dcos command interface | ||
You can deploy Pulsar on DC/OS using this command: | ||
|
||
```shell | ||
```bash | ||
$ dcos marathon group add PulsarGroups.json | ||
``` | ||
|
||
This command will deploy some docker container instances in 3 groups, which consist a Pulsar cluster: | ||
This command will deploy Docker container instances in three groups, which together comprise a Pulsar cluster: | ||
|
||
* 3 bookies (1 {% popover bookie %} on each agent node and 1 [bookie recovery](http://bookkeeper.apache.org/docs/latest/admin/autorecovery/) instance) | ||
* 3 Pulsar {% popover brokers %} (1 broker on each node and 1 admin instance) | ||
* 1 [Prometheus](http://prometheus.io/) instance and 1 [Grafana](https://grafana.com/) instance | ||
|
||
* deploy 3 bookies, 1 `bookie` on each agent node; and 1 `bookie_recovery` instance. | ||
* deploy 3 Pulsar brokers, 1 `broker` on each node; and 1 `broker_admin` instance. | ||
* deploy 1 `prometheus` instance and 1 `grafana` instance. | ||
{% include admonition.html type="success" | ||
content="When running DC/OS, a ZooKeeper cluster is already running at `master.mesos:2181`, thus there's no need to install or start up ZooKeeper separately." %} | ||
|
||
Since DC/OS is already deployed a zookeeper cluster serving at `master.mesos:2181`, we leverage it, and no need to deploy it ourselves. | ||
After execute the command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), whose link is `http://m1.dcos` in this example, and we should see applications deploying. | ||
After executing the `dcos` command above, click on the **Services** tab in the DC/OS [GUI interface](https://docs.mesosphere.com/latest/gui/), which you can access at [http://m1.dcos](http://m1.dcos) in this example. You should see several applications in the process of deploying. | ||
|
||
![DC/OS command executed](/img/dcos/command_execute.png) | ||
|
||
![DC/OS command executed2](/img/dcos/command_execute2.png) | ||
|
||
## The BookKeeper group | ||
|
||
## BookKeeper Group | ||
|
||
To watch the status of `BookKeeper` Cluster deployment, we could click into the 'bookkeeper' group in parent pulsar group picture. | ||
To monitor the status of the BookKeeper cluster deployment, click on the **bookkeeper** group in the parent **pulsar** group. | ||
|
||
![DC/OS bookkeeper status](/img/dcos/bookkeeper_status.png) | ||
|
||
Now 3 bookies is shown as green colour, which means they are already deployed successfully and running. | ||
At this point, 3 {% popover bookies %} should be shown as green, which means that they have been deployed successfully and are now running. | ||
|
||
![DC/OS bookkeeper running](/img/dcos/bookkeeper_run.png) | ||
|
||
We could also click into each Instance of bookie, to get more detailed info, such as the bookie running log. | ||
You can also click into each bookie instance to get more detailed info, such as the bookie running log. | ||
|
||
![DC/OS bookie log](/img/dcos/bookie_log.png) | ||
|
||
And we could also get the BookKeeper cluster information in Zookeeper, whose link is `http://m1.dcos/exhibitor`, and could see 3 bookies under `available` directory. | ||
To display information about the BookKeeper in ZooKeeper, you can visit [http://m1.dcos/exhibitor](http://m1.dcos/exhibitor). In this example, there are 3 bookies under the `available` directory. | ||
|
||
![DC/OS bookkeeper in zk](/img/dcos/bookkeeper_in_zookeeper.png) | ||
|
||
## The Pulsar broker Group | ||
|
||
## Pulsar Broker Group | ||
|
||
Similar to BookKeeper Group above, click into `brokers` group, and we will see brokers status. | ||
Similar to the BookKeeper group above, click into the **brokers** to check the status of the Pulsar brokers. | ||
|
||
![DC/OS broker status](/img/dcos/broker_status.png) | ||
|
||
![DC/OS broker running](/img/dcos/broker_run.png) | ||
|
||
We could also click into each Instance of broker, to get more detailed info, such as the broker running log. | ||
You can also click into each broker instance to get more detailed info, such as the broker running log. | ||
|
||
![DC/OS broker log](/img/dcos/broker_log.png) | ||
|
||
And also get Broker cluster information in Zookeeper, which created directory `loadbalance` and `managed-ledgers`. | ||
Broker cluster information in Zookeeper is also available through the web UI. In this example, you can see that that the `loadbalance` and `managed-ledgers` directories have been created. | ||
|
||
![DC/OS broker in zk](/img/dcos/broker_in_zookeeper.png) | ||
|
||
|
||
## Monitor Group | ||
|
||
Monitor Group consist of Prometheus and Grafana. | ||
The **monitory** group consists of Prometheus and Grafana. | ||
|
||
![DC/OS monitor status](/img/dcos/monitor_status.png) | ||
|
||
### Prometheus | ||
|
||
We could click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example. | ||
Click into the instance of `prom` to get the endpoint of Prometheus, which is `192.168.65.121:9090` in this example. | ||
|
||
![DC/OS prom endpoint](/img/dcos/prom_endpoint.png) | ||
|
||
While click this endpoint above, and we could get into Prometheus dashboard, and the targets `http://192.168.65.121:9090/targets` tab, should contains all bookies and brokers. | ||
If you click that endpoint, you'll see the Prometheus dashboard. The [http://192.168.65.121:9090/targets](http://192.168.65.121:9090/targets) URL will display all the bookies and brokers. | ||
|
||
![DC/OS prom targets](/img/dcos/prom_targets.png) | ||
|
||
### Grafana | ||
|
||
We could click into the instance of `grafana` to get the endpoint of Grafana, which is `192.168.65.121:3000` in this example. | ||
Click into `grafana` to get the endpoint for Grafana, which is `192.168.65.121:3000` in this example. | ||
|
||
![DC/OS grafana endpoint](/img/dcos/grafana_endpoint.png) | ||
|
||
While click this endpoint above, and we could get into Grafana dashboard. | ||
If you click that endpoint, you can access the Grafana dashbaord. | ||
|
||
![DC/OS grafana targets](/img/dcos/grafana_dashboard.png) | ||
|
||
## Run a simple Pulsar consumer and producer on DC/OS | ||
|
||
## Run simple consumer and producer on DC/OS deployment. | ||
|
||
At present we could get a deployed Pulsar cluster. and we can run simple consumer and producer on it now. | ||
Now that we have a fully deployed Pulsar cluster, we can run a simple consumer and producer to show Pulsar on DC/OS in action. | ||
|
||
### Download and prepare pulsar java tutorial | ||
### Download and prepare the Pulsar Java tutorial | ||
|
||
This code contains a simple Pulsar Consumer and Producer, and the `README.md` contains more information about it. | ||
There's a [Pulsar Java tutorial](https://github.com/streamlio/pulsar-java-tutorial) repo that you can clone. This repo contains a simple Pulsar consumer and producer (more info can be found in the repo's `README` file). | ||
|
||
```shell | ||
$ git clone https://github.com/streamlio/pulsar-java-tutorial.git | ||
```bash | ||
$ git clone https://github.com/streamlio/pulsar-java-tutorial | ||
``` | ||
|
||
Change `SERVICE_URL` from "pulsar://localhost:6650" to "pulsar://a1.dcos:6650" in both [ConsumerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [ProducerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java). | ||
"pulsar://a1.dcos:6650" is the endpoint of broker service, we could get the endpoint details from DC/OS GUI of each broker instance. `a1.dcos` is one of DC/OS client agent, which runs a broker, it can also be replaced by the client agent IP address. | ||
Change the `SERVICE_URL` from `pulsar://localhost:6650` to `pulsar://a1.dcos:6650` in both [`ConsumerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ConsumerTutorial.java) and [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java). | ||
The `pulsar://a1.dcos:6650` endpoint is for the broker service. Endpoint details for each broker instance can be fetched from the DC/OS GUI. `a1.dcos` is a DC/OS client agent, which runs a broker. This can also be replaced by the client agent IP address. | ||
|
||
Change the message number from 10 to 10000000 in main method of [ProducerTutorial.java](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java), so it will produce more message. | ||
Now, change the message number from 10 to 10000000 in main method of [`ProducerTutorial.java`](https://github.com/streamlio/pulsar-java-tutorial/blob/master/src/main/java/tutorial/ProducerTutorial.java) so that it will produce more messages. | ||
|
||
Now compile the project code using command: | ||
|
||
```shell | ||
```bash | ||
$ mvn clean package | ||
``` | ||
|
||
### Run consumer and producer | ||
### Run the consumer and producer | ||
|
||
Execute command to run consumer: | ||
Execute this command to run the consumer: | ||
|
||
```shell | ||
```bash | ||
$ mvn exec:java -Dexec.mainClass="tutorial.ConsumerTutorial" | ||
``` | ||
|
||
Execute command to run producer: | ||
Execute this command to run the producer: | ||
|
||
```shell | ||
```bash | ||
$ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial" | ||
``` | ||
|
||
And we could see the producer producing messages and consumer consuming messages. | ||
You can see the producer producing messages and the consumer consuming messages through the DC/OS GUI. | ||
|
||
![DC/OS pulsar producer](/img/dcos/producer.png) | ||
|
||
![DC/OS pulsar consumer](/img/dcos/consumer.png) | ||
|
||
|
||
### View Grafana metric output | ||
|
||
While producer and consumer is running, we could get some running metrics information from Grafana | ||
While the producer and consumer are running, you can access running metrics information from Grafana. | ||
|
||
![DC/OS pulsar dashboard](/img/dcos/metrics.png) | ||
|
||
|
||
## Uninstall Pulsar | ||
|
||
We can shut down and uninstall the `pulsar` from DC/OS at any time by 2 ways. | ||
You can shut down and uninstall the `pulsar` application from DC/OS at any time in two ways: | ||
|
||
Using DC/OS GUI, we could choose `Delete` at the right end of Pulsar group. | ||
1. Using the DC/OS GUI, you can choose **Delete** at the right end of Pulsar group. | ||
|
||
![DC/OS pulsar uninstall](/img/dcos/uninstall.png) | ||
![DC/OS pulsar uninstall](/img/dcos/uninstall.png) | ||
|
||
2. You can use the following command: | ||
|
||
Or we could use command to do the uninstall: | ||
|
||
```shell | ||
$ dcos marathon group remove /pulsar | ||
``` | ||
```bash | ||
$ dcos marathon group remove /pulsar | ||
``` |