ChiaMon

Example Chia monitoring stack, using:

• mtail to collect metrics from Chia logs
• chia_exporter to collect metrics from the Chia node
• node_exporter or windows exporter to collect system metrics
• prometheus to store metrics
• promtail and loki to collect and store logs from the Chia node and plotters (and system too if desired)
• grafana to display everything

This includes a docker-compose configuration to run everything, but this is primarily intended for development and testing.

WARNING this is NOT a one-click install, expect to need to do some work setting everything up for your machine. PLEASE read the notes below and understand what all the services are, what they do, and how they work together.

mtail program

The mtail program is in mtail/chialog.mtail. Currently it only collects harvester metrics:

• chia_harvester_blocks_total: cumulative number of block challenges attempted
• chia_harvester_plots_total: current number of plots
• chia_harvester_plots_eligible: cumulative number of plots that passed filter
• chia_harvester_proofs_total: cumulative number of proofs won
• chia_harvester_search_time: histogram of proof search times

NOTE you need to set log_level to INFO in your Chia config.yaml to get harvester metrics.

chia_exporter

The chia_exporter is used to collect metrics from the Chia node RPC API.

Grafana dashboard

The example Grafana dashboard is in grafana/dashboards/Chia.json. It defines a number of variables that will be auto-populated from the node metrics. Grafana dashboards are easily customized to show what you're interested in seeing, in the way you find best; this dashboard is just meant to demonstrate what can be done.

Running on Linux/Mac

The docker-compose file will mount the Chia log from $HOME/.chia/mainnet/log/debug.log, verify that this location is correct and set the log level to INFO in the Chia configuration (usually at $HOME/.chia/mainnet/config/config.yaml).

Run:

docker-compose up -d

This will do the following:

• Build container image with configuration for mtail from source
• Build container image for chia_exporter from source
• Download other images from docker hub
• Run containers in the background, attached to the host network (this makes it easy to communicate with native services, but has some trade-offs. See notes.)

The grafana service provisions the prometheus and loki datasources and a basic dashboard that displays harvester and node metrics.

Access Grafana at http://localhost:3000 and login with the default admin/admin username and password (you'll be prompted to change the password).

Notes

• It's highly encouraged to run the node exporter natively rather than in docker - see the discussion in the node_exporter docs. If you do run it in Docker, you'll need to bind-mount in any other volumes you want to monitor (add them to the volumes list in docker-compose.yml, e.g. - '/scratch:/scratch'). See issue #3.

• On Mac you'll need to run node_exporter natively, not under Docker: brew install node_exporter. You'll probably need to change the networking setup too, since Docker on Mac runs in a VM. See the windows docker-compose and prometheus configs.

Running on Windows

The node exporter does not work on Windows; instead you need to use the Windows exporter for system metrics. Modified config and example dashboard are in the windows branch. You may also want to review the discussion in issue #2.

These steps will get you to a working setup (but aren't the only way):

• Install Docker Desktop
• Install Visual Studio Code
• Install git
• Install Windows exporter
• Clone the chiamon repository with VSCode
• Modify docker-compose.yml:
  • Change volume paths to point to your home directory.
• Run services. In VSCode with docker extension you can just right-click on docker-compose.yml and select "Compose Up"
• Check target status in Prometheus at http://localhost:9090/targets
• Access Grafana at http://localhost:3000 (admin/admin).