Instructions for deployment of simple monitoring stack

[greenstatic]

@sbellm .. I'm a novice in coding.
Can you be kind enough to spell in easy steps on how to install BBB exporter(and the accompaniments) and how much cpu does it eat?
It seems we need to to install Grafana, Prometheus and Netdata .. but in what sequence and in which directories etc?

Thanks for the help.

Originally posted by @tarunb999 in #18 (comment)

[greenstatic]

Next time please open a separate issue and not hijack other issues.

Correct. You will need to have Grafana, Prometheus and optionally Netdata to get the fancy dashboard (in the next release it will be possible to swap Netdata with node_exporter as per #7).

Would you like to have your monitoring stack (Grafana & Prometheus with the optional Alertmanager to trigger alerts on certain metrics) on a different server (recommended) or on the same machine that you host BBB (not recommended)?

[tarunb999]

oops.. I didn't know I'm hijacking someone..just asked him for some help as he'd setup the working BBB exporter. My apologies !!!

[tarunb999]

I would prefer to install on the BBB server as of now please.
Also if I can just use Grafana & Prometheus as I don't want to overload my running BBB server with Netdata.

[greenstatic]

@tarunb999 no worries, now you know :)

I can give you a working docker-compose file with which you can install Grafana, Prometheus and bbb-exporter. Are you familiar with docker and docker-compose?

[tarunb999]

Yes I'm a bit familiar but not too much :-/ .. but I can always give it a try till it doesn't break my bbb server !!

[greenstatic]

Update: This information is a bit outdated, please refer to the documentation section installing All-in-One Monitoring Stack

Original:

If you wish to have a simple monitoring stack (Grafana, Prometheus & BigBlueButton-exporter) on the same host as your BBB installation, here is how you can do it using docker-compose.

Get your BBB secret ready (bbb-conf --secret) for the file bbb_exporter_secrets.env and anywhere you see example.com change it to your BBB hosts domain.

mkdir ~/bbb-monitoring
touch docker-compose.yaml prometheus.yaml bbb_exporter-secrets.env

2. Copy the following contents into the appropriate files:

# docker-compose.yaml
version: '3.2'
services:
  bbb-exporter:
    container_name: "bbb-exporter"
    image: greenstatic/bigbluebutton-exporter:v0.2.0
    expose:
     - 9688
    env_file:
     - bbb_exporter_secrets.env
    restart: unless-stopped

  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.17.2
    command:
     # Default commands from Dockerfile
     - "--config.file=/etc/prometheus/prometheus.yml"
     - "--storage.tsdb.path=/prometheus"
     - "--web.console.libraries=/usr/share/prometheus/console_libraries"
     - "--web.console.templates=/usr/share/prometheus/consoles"
     # Our custom commands - metrics will be stored for 400 days
     - "--storage.tsdb.retention.time=400d"
    expose:
     - 9090
    volumes:
     - "./prometheus.yaml:/etc/prometheus/prometheus.yml"
     - "prometheus_data:/prometheus"
    restart: unless-stopped

  grafana:
    container_name: grafana
    image: grafana/grafana:6.7.1
    ports:
    # Bind to 127.0.0.1 when setting up a Nginx SSL reverse proxy so that
    # we do not leak Grafana directly
    # - "127.0.0.1:3001:3000"
      - "3001:3000"
    volumes:
      - "grafana_data:/var/lib/grafana"
    environment:
      GF_SERVER_ROOT_URL: "https://example.com/monitoring"
    restart: unless-stopped
  
volumes:
  prometheus_data:
  grafana_data:

# prometheus.yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'bbb'
    static_configs:
      - targets: ['bbb-exporter:9688']

# bbb_exporter_secrets.env
API_BASE_URL=https://example.com/bigbluebutton/api/
API_SECRET=<SECRET>

3. Start the compose file:

cd ~/bbb-monitoring
sudo docker-compose up -d

4. Login to grafana (http://example.com:3000) in your web browser (admin:admin) and change the password

5. Add data source -> Prometheus:

URL: http://prometheus:9090

6. Left menu -> + icon -> import -> copy the json of the desired dashboard (i.e. grafana_dashboard_all_servers.json and/or grafana_dashboard_server_instance.json).
   Since you only have 1 BBB instance the grafana_dashboard_all_servers.json dashboard probably won't be useful, so just import the grafana_dashboard_server_instance.json dashboard.

Since you do not have netdata the CPU Utilization vs. Participants and Bandwidth vs. Participants dashboards won't work.

You can edit your dashboard as much as you like, but note that you will lose your changes if you will reimport updated versions of the dashboard. So you will have to manually patch the dashboard when we update the dashboard json files in the repository.

Dashboards and instructions for node_exporter will be soon released (replacing netdata) with which the aforementioned (CPU and bandwidth) dashboards will become useful. So keep up-to-date with the development of this repo.

I strongly recommend that you setup HTTPS for your Grafana installation. You can use the Nginx installation that is on the BBB host. You can piggyback on the existing Nginx config (/etc/nginx/sites-available/bigbluebutton) by having a URL reverse proxy rule, e.g.

# /etc/nginx/sites-available/bigbluebutton

# Grafana
location /metrics/ {
  proxy_pass         http://127.0.0.1:3001/;
  proxy_redirect     default;
  proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
  client_max_body_size       10m;
  client_body_buffer_size    128k;
  proxy_connect_timeout      90;
  proxy_send_timeout         90;
  proxy_read_timeout         90;
  proxy_buffer_size          4k;
  proxy_buffers              4 32k;
  proxy_busy_buffers_size    64k;
  proxy_temp_file_write_size 64k;
  include    fastcgi_params;
}

Or you could do virtual hosting - create a separate nginx site configuration and have a separate domain for the Grafana dashboard and get a free Let's encrypt certificate (this is more proper).

If you are going to piggyback on BBB's Nginx site config be warned that during a BBB upgrade the BBB's upgrade script overwrites the Nginx site config and you will have to copy the Grafana reverse proxy configuration again.

Once you use Nginx as a SSL reverse proxy you can bind Grafana to 127.0.0.1 in the docker-compose.yaml file. This will make Grafana accessible only through Nginx and not directly. If you change your docker-compose.yaml file you will need to issue another docker-compose up -d in the ~/bbb-monitoring directory to apply changes.

Hope this helps.
I'll add this to the repos readme if everything works out.

EDIT: YAML indentation
EDIT 2: Grafana port 3000 -> 3001 (due to https://github.com/greenstatic/bigbluebutton-exporter/#note-on-installing-grafana-on-the-bbb-host)
EDIT 3: add Grafana ROOT_URL ENV variable

[tarunb999]

Many thanks Gregor for this fantastic piece of information. I'm sure it'll of much use to many more users like me.
I'll give it a go tonight once all our users have logged out and revert with the status.

Thanks... much appreciated !!!

[tarunb999]

After Step 3, I got the following error:

root@e2e-67-176:~/bbb-monitoring# sudo docker-compose up -d
ERROR: In file './../docker-compose.yaml', service must be a mapping, not a NoneType.

[greenstatic]

Next time please use either a blockquote or enclose the error message in backticks (see GitHub Markdown reference). It makes it easier to read error messages and separate logs/code/errors from the content.

The issue is because of YAML indentation. It's my fault since in the snippet I gave you the indentation got messed up. I'll edit the original message and let you know when it's fixed.

[greenstatic]

@tarunb999 give it a go now, replace all the files with the updated content. The indentation should be okay now, but I haven't test it. Please report back if it works or if you have any other errors.

[greenstatic]

I just remembered that you might have issues if you try to setup Nginx to do TLS proxy termination for Grafana since port 3000 is already taken by BigBlueButton's html5-client (https://github.com/greenstatic/bigbluebutton-exporter/#note-on-installing-grafana-on-the-bbb-host).
Change Grafana's port from 3000 to something else like 3001 in the docker-compose and additional Nginx location configuration.
I have updated the YAML files from above to reflect this.

[greenstatic]

Updated documentation: https://bigbluebutton-exporter.greenstatic.dev/installation/all_in_one_monitoring_stack/

[tarunb999]

Thanks a lot Gregor to making it so elaborate.
I've done the groundwork..just waiting for users to log off so that I can restart Nginx after adding /monitoring directive to it for it to take effect.

[tarunb999]

Ok, so when I complete the location directive to Nginx, restart Nginx and goto the Base_URL/monitoring link, Grafana takes me to Base_URL/login page but this page gives a 404 error.

[greenstatic]

Ah, minor oversight. You will need to configure root_url in Grafana.

Add to your docker-compose file the following env variable to grafana:

GF_SERVER_ROOT_URL: "https://example.com/monitoring"

[tarunb999]

All's fine Gregor ... yes Grafana is up and running (no users online so empty graphs as yet ;-)) !!!

Thanks a lot for all your help..can't express my gratitude for the pains you've taken to break it down into simple steps and uploading for future users. Many thanks indeed.

[greenstatic]

Great,I'll add it to the documentation. Thanks for your kind words.

[tarunb999]

I'm sure it'll benefit many people like me in coming times..

just a quick one ..I just started my meeting ..but Grafana is still not showing any Participants !! I think its at 5 sec refresh.

[greenstatic]

This depends on the scrape interval - which is set by default to 15 seconds (check prometheus.yaml).
Or are you not seeing any metrics in your Grafana dashboard?

[tarunb999]

Yup, all the panels are just blank. All of them say.. NO DATA.
(scrape interval is 15 sec.)

[greenstatic]

Check if your Prometheus is able to fetch the exporters metrics successfully.

You will require a bit of network magic in order to access Prometheus.

If you followed the All-In-One Monitoring Stack instructions you Prometheus will be binded to the localhost interface (you cannot access it externally). However since your Prometheus container has host network permissions (meaning you are using the host's localhost interface and not docker's network local interface - yay networking confusion) you can simply use SSH to do a local forward.

ssh -NL 9090:127.0.0.1:9090 <BBB_HOST>

This will make accessible on your local machine on port 9090 the application which was binded on the remote machine (bbb host) on 127.0.0.1:9090 (Prometheus).

So open up a web browser and go to http://localhost:9090, then check status->targets and see if the exporter is in the state Up.

If you did NOT follow the All-In-One Monitoring Stack instructions (i.e. used the outdated compose file in this thread) then I suggest you upgrade first. You will even get node_exporter for free bundled together.

[tarunb999]

Many thanks again !!

[greenstatic]

Did you manage to find your issue?

[fl4t23]

Hi,
First of all, thank you for your work.
I followed the current All-In-One Monitoring Stack.
I have two problems.

add prometheus data sources in grafana:
URL: http://127.0.0.1:9090
or
URL: http://localhost:9090
always show
'HTTP Error Bad Gateway'
prometheus part in docker-compose.yaml:

...
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.17.2
    command:
      # Default commands from Dockerfile
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      # Our custom commands - metrics will be stored for 400 days
      - "--storage.tsdb.retention.time=400d"
      - "--web.listen-address=127.0.0.1:9090"
    volumes:
      - "./prometheus.yaml:/etc/prometheus/prometheus.yml"
      - "prometheus_data:/prometheus"
    restart: unless-stopped
...

To successfully add the prometheus data sources in Grafana, I had to change the following in docker-compose.yaml:
(in prometheus part)

...
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.17.2
    command:
      # Default commands from Dockerfile
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      # Our custom commands - metrics will be stored for 400 days
      - "--storage.tsdb.retention.time=400d"
#      - "--web.listen-address=127.0.0.1:9090"
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus.yaml:/etc/prometheus/prometheus.yml"
      - "prometheus_data:/prometheus"
    restart: unless-stopped
...

than the data source connection is ok.
and i can visit the prometheus page
http://localhost:9090

And then I have the problem.
All panels in the Dashboard (BigBlueButton Server Instance) show
'No Data'

Instance, top left in grafana show
'None'

scrape interval in prometheus.yaml = 15s

netstat -tulpen |grep 9090
tcp6       0      0 :::9090                 :::*                    LISTEN      0          359266      6602/docker-proxy

docker container ls -a
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                      NAMES
5170acb80e1b        prom/prometheus:v2.17.2                     "/bin/prometheus --c…"   4 minutes ago       Up 4 minutes        0.0.0.0:9090->9090/tcp     prometheus
b4626b9f42c3        greenstatic/bigbluebutton-exporter:latest   "python server.py"       34 minutes ago      Up 34 minutes                                  bbb-exporter
95eb62f3aa8a        prom/node-exporter:v0.18.1                  "/bin/node_exporter …"   34 minutes ago      Up 34 minutes                                  node_exporter
b16685b05bc6        grafana/grafana:6.7.1                       "/run.sh"                34 minutes ago      Up 34 minutes                                  grafana
90493e0ab293        bigbluebutton/greenlight:v2                 "bin/start"              40 hours ago        Up 18 hours         127.0.0.1:5000->80/tcp     greenlight-v2
b92f7dc1488f        postgres:9.5                                "docker-entrypoint.s…"   40 hours ago        Up 18 hours         127.0.0.1:5432->5432/tcp   greenlight_db_1

docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
fc3afc3a9897        bbb-monitoring_default   bridge              local
80fe1ff9171f        bridge                   bridge              local
19e878e831c6        greenlight_default       bridge              local
93ddbd9329c6        host                     host                local
ea1f2b733862        none                     null                local

When i change
in docker-compose.yaml
(in prometheus part):

...
      # Our custom commands - metrics will be stored for 400 days
      - "--storage.tsdb.retention.time=400d"
#      - "--web.listen-address=127.0.0.1:9090"
    ports:
      - "127.0.0.1:9090:9090"

Then netstat shows me the following:

netstat -tulpen |grep 9090                                          
tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      0          3471
44      5342/docker-proxy      

but the problem remains
'No Data'

The prometheus panel under status -> targets show:

Maybe you can see my problem?
best regards

[greenstatic]

You are missing the field network_mode: host. This allows the container to use the host's networking stack. It is added to all containers so they can communicate with each other on the same loopback interface (localhost) of the host. Albeit it's better practice to have a docker network and use that for the containers to communicate, but in this case it was necessary because afaik node_exporter needs to access the entire hosts networking stack in order to gain metrics (I could be wrong).

Since you forgot the network_mode field, all the containers can communicate with one another since they are in the same docker network (you can verify this using docker-compose network ls) but not on localhost. By adding the lines:

ports:
      - "9090:9090"

You inadvertently exposed Prometheus on all your interfaces (including the host's localhost) and made it world accessible (very bad). And so you managed to access Prometheus by accident with the wrong configuration but all the rest of the containers cannot since they cannot use the hosts localhost.

[fl4t23]

ah ok, i get that. thx
my problem is solved.

I have reversed my changes and entered the missing

network_mode: host

And it works now.

  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.17.2
    network_mode: host
    command:
      # Default commands from Dockerfile
      - "--config.file=/etc/prometheus/prometheus.yml"                   
      
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
   
      - "--web.console.templates=/usr/share/prometheus/consoles"         

      # Our custom commands - metrics will be stored for 400 days        
      
      - "--storage.tsdb.retention.time=400d"
      - "--web.listen-address=localhost:9090"
    volumes:
      - "./prometheus.yaml:/etc/prometheus/prometheus.yml"               
      
      - "prometheus_data:/prometheus"
    restart: unless-stopped

Many thanks for your hint and your quick answer.

[greenstatic]

Great, if everything is up and running you can close the issue.

[fl4t23]

From my side the issue can be closed. I can't close this.

[tarunb999]

Sorry Gregor, I was indisposed for a few days , hence was unable to respond. Yes, all went well, thanks a lot.
Unfortunately the school decided to move to GSuite as hosting a BBB server costs a bit more, so had to close that installation.
All the best to all the adventurers !!!

[ethpi]

Hi,

I tried to do the steps explained here (or in https://bigbluebutton-exporter.greenstatic.dev/installation/all_in_one_monitoring_stack/). But when executing the command fourth step "Start the services"

cd ~/bbb-monitoring
sudo docker-compose up -d

I get the error message

ERROR: yaml.scanner.ScannerError: mapping values are not allowed here
  in "./docker-compose.yaml", line 90, column 65

What am i doing wrong?

[greenstatic]

Not sure, since the docker-compose for the All In One Monitoring stack is 65 lines long not 90. I suggest you review if you have the correct docker-compose.yaml file.

If you still can't figure it out, post your YAML here and make sure you censor any sensitive information (if you have added any).

[ethpi]

God point. My docker-compose.yaml has 1608 lines ...

I tried to load the three files

bbb_exporter_secrets.env
docker-compose.yaml
prometheus.yaml 

as discribed in the "All-In-One ..."-manual at this location: https://github.com/greenstatic/bigbluebutton-exporter/tree/master/extras/all_in_one_monitoring.

But ... very embarrassing, I loaded the html-file down that shows docker-compose.yaml in the browser.

Now I loaded the real files and everything seems to be ok. I can login at https://my.do.main/monitoring

Thanks for your superfast help.

[greenstatic]

Happens to everyone once in while 😅 .

[ethpi]

Every things works fine now.

There is only one point:

At the imported dashboard https://github.com/greenstatic/bigbluebutton-exporter/tree/master/extras/dashboards/server_instance_node_exporter.json I had to change the datasource from "Prometheus" to "default".

[greenstatic]

This should have worked if you entered the name of the Prometheus data source as Prometheus. But I think it's not explicitly mentioned in the docs.

As a workaround (as you probably figured), you just need to change the datasource variable (top left corner of the dashboard) and click save dashboard (top right corner) with the checkbox that says something along the lines of "use as default values".

[abraxaswd]

Every things works fine now.

There is only one point:

At the imported dashboard https://github.com/greenstatic/bigbluebutton-exporter/tree/master/extras/dashboards/server_instance_node_exporter.json I had to change the datasource from "Prometheus" to "default".

I had exactly the same issue today and I named the source Prometheus as mentioned by @greenstatic. It did not work. So, thanks for pointing to the default setting. And many thanks @greenstatic for this wonderful monitoring stack.

[rmonterr]

@greenstatic

Hello, I am trying to get some help with a couple things. I have installed the monitoring stack into my BBB servers, however I can't setup the alerts because:

1. they refer to template variables and they are not supported in the alerts (what's the best way to work around this?). Basically I care about a)cpu usage b)storage (not crucial) c) bandwidth(transmit) d) BBB API = online (true/false) (if possible).

2. I installed the stack into each server which is not an ideal alerting scenario in case the server goes down (at least it would be still be helpful to alert high cpu usage, disk space, perhaps api response and any other variable which might impact user experience other than the server being down haha). So the second question is clearly, how can I scrape different servers from one place? I don't mind about recordings which I read in the documentation, would be a problem in case one does this.

Thanks!