Send Webhook-based Grafana Alerts via Telegram

Contents

• Features
• Demo
• Build From Source
• Configure
• Run
  • Environment
• Reload Config
  • Example 1
  • Example 2
• Connect Grafana
• Sub/unsub to Alerts
  • Subscribe
  • Unsubscribe
• License
• To Do

Warning

This is an experimental prototype, a proof of concept. Don’t use in production. A lot of things don’t work as expected, and extensive testing is still needed.

Note	The webhook endpoint doesn’t support auth and encryption. If you need HTTPS and auth, use a reverse proxy.

Prerequisites

Note	If you are planning to run it in Docker containers, ignore this list. See Dockerfile and compose.yml.

• Node.js latest

• Redis-compatible in-memory database (I prefer Valkey)

• pnpm

Features

• Access control based on Telegram UIDs

• Multiple chats can be subscribed simultaneously

• Both private chats and group chats can be subscribed to alerts

  Note	The user interacting with the bot in a group chat must be added to the ACL, but not the group chat itself.

• Unauthorized users are ignored

• Arbitrary messages and unrecognized commands are ignored

• Unauthorized access to valid bot commands is logged

• Grafana interacts with the bot via a webhook endpoint

• Style messages with Telegram-supported HTML tags (the bot doesn’t support Markdown)

Demo

Suppose, you use ClickHouse as a data warehouse for your logs, and it’s connected to your Grafana instance as a data source.

You have created an alert rule with the following SQL query:

And annotations look like the following:

When the alert rule is triggered, all subscribed Telegram chats will get messages similar to this:

Another Example for Sudo Alerts

Example Video

assets/screencast.mp4

Build From Source

$ pnpm build

Configure

1. Copy the example configuration file botconfig.toml to a new directory ./config.

2. Edit config.

Run

Note	Don’t run using pnpm. It breaks signal handling.

$ node build/main.js

Environment

You can set environment variables to change the program’s behavior.

Variable	Used for	Default
BOTCONFIG	Change config file path.	`${getProjectRoot()}/config/botconfig.toml`

Reload Config

To reload configuration, send the SIGHUP signal to the process.

Example 1

$ pnpm reload

Example 2

$ kill -HUP <PID here> # You can use something like $(pidof node) if there's only one Node.js process.

Connect Grafana

1. Create a webhook contact point pointing to your bot instance

2. Make sure it uses the POST method

Sub/unsub to Alerts

Subscribe

Group Chat

Add the bot to a chat just like a regular user, and issue the /start@bot_name command.

Personal Chat

Open a chat with the bot, and press the start button.

Unsubscribe

Same as above, but the command is /stop@bot_name or /stop depending on the chat type.

License

0BSD

To Do

• Catch Grammy errors

• Make queues reliable

• Send out messages with time represented in a local timezone instead of UTC

• Improve cache error handling, fix infinite frequent retries on wrong user/password

• Improve logging, revert cache connect and disconnect logging

• Improve config validation