readme

README.md

@@ -1,46 +1,109 @@
-# Blog: Opentelemetry: Building decoupled monitoring
+# Blog "Building decoupled monitoring with OpenTelemetry"
 
-## Blog
-This repository contains source code for "@Opentelemetry: Building decoupled monitoring" blog post.
+This repository contains source code for "Building decoupled monitoring with OpenTelemetry" blog post.
 
 ## Layout
+The repository is organized as follows:
+```
+./blog - directory containing blog post content
+./docker-compose - dirrectory containing docker-compose files for running various system setups
+./docker-compose/mount - directory containing files to be mounted to docker containers.
+./java - directory containing java source code for the example application
+.requests.http - file containing HTTP requests as an example of how to interact with the example application
+.traffic.sh - shell script to simulate user traffic
+```
+
+## How to run
 
-### Subprojects
-Main source code divided on following subprojects:
+## Prerequisites
+Before running any system setup, you need to build the example application:
 ```
-TODO
-/ folder - description
+cd java
+docker build -t products-service .
 ```
 
-### Misc
-Apart from source code current repo has the following folders:
+## Run direct publishing
+This is the simplest monitoring setup. The idea behind this setup is to demonstrate how to send telemetry directly from the application to the monitoring system backend using OpenTelemetry.
+In particular, the application sends traces to Jaeger and metrics to Prometheus.
+To run it, use the following commands:
 ```
-blog - blog post
-docker-compose - various docker compose files to setup  
+cd docker-compose
+docker-compose -f docker-compose/setup-direct-publishing.yaml up -d
 ```
+You can check whether application is running by sending a request `GET localhost:8080/health` or by checking status of the docker container `products-service`.
+After the system is up and running, run [traffic.sh](traffic.sh) to simulate user traffic.
 
-## How to run
-To run the system.
+After that, you can check telemetry data in:
+- Prometheus. Open http://localhost:9090. You can use the following query to see application average latency: `sum(products_service_http_server_request_duration_seconds_sum) / sum(products_service_http_server_request_duration_seconds_count)` 
+- Jaeger. Open http://localhost:16686. You can choose `products-service` from the list of services and see traces.
 
-Build necessary docker containers:
+Stop environment using docker compose:
 ```
-TODO
+docker-compose -f docker-compose/setup-direct-publishing.yaml down
 ```
 
-Run system setup using docker compose. For instance :
+## Run publishing though OpenTelemetry Collector
+This setup demonstrates how to send telemetry data to the monitoring system backend using OpenTelemetry Collector.
+It still sends traces to Jaeger and metrics to Prometheus, but the application sends telemetry data to OpenTelemetry Collector, which then forwards it to the monitoring system backend.
+To run it, use the following commands:
 ```
-docker-compose -f docker-TODO
+cd docker-compose
+docker-compose -f docker-compose/setup-otel-collector-publishing.yaml up -d
 ```
 
-After, run load testing to simulate user traffic
+You can check application status, send traffic and check telemetry data as described in the previous section.
+
+Stop environment using docker compose:
 ```
-TODO
+docker-compose -f docker-compose/setup-otel-collector-publishing.yaml down
 ```
 
-Check target APM or any monitoring tool and verify telemetry has been sent.
+## Run publishing through OpenTelemetry Collector to multiple backends
+This setup demonstrates how to send telemetry data to multiple monitoring system backends using OpenTelemetry Collector.
+It sends the following telemetry data to the monitoring system backends:
+- Traces to Jaeger and Grafana Agent, which in its turn sends traces to Tempo.
+- Metrics to Prometheus and Grafana Agent, which in its turn sends metrics to Mimir.
+- Logs to Grafana Agent, which in its turn sends logs to Loki.
+
+To run it, use the following commands:
+```
+cd docker-compose
+docker-compose -f docker-compose/setup-otel-collector-publishing-extended.yaml up -d
+```
+
+You can check application status, send traffic as described in the previous section.
+Check Grafana stack health in Grafana Agent UI: http://localhost:12345. All components should show "Healthy" status.
+
+Additionally, to the previous setups, you can check telemetry data in:
+- Tempo. Open Grana UI http://localhost:3000 and go to Explore. Choose "Tempo" as a data source, click on "Run Query" see traces.
+- Mimir. Open Grana UI http://localhost:3000 and go to Explore. Choose "Mimir" as a data source, use Prometheus like query to find metrics. For instance: `http_server_request_duration_seconds_bucket{http_route="/api/product"}`
+- Loki. Open Grana UI http://localhost:3000 and go to Explore. Choose "Loki" as a data source, use Loki like query to find logs. For instance: `{job="products_service"}`
 
 Stop environment using docker compose:
 ```
-docker-compose -f TODO down
+docker-compose -f docker-compose/setup-otel-collector-publishing-extended.yaml down
+```
+
+## Run publishing through OpenTelemetry Collector to multiple backends with telemetry data processing.
+This setup demonstrates how to send telemetry data to multiple monitoring system backends using OpenTelemetry Collector and process it before sending.
+It sends telemetry data to the same backends as the previous setup, but before sending it, it processes telemetry data using OpenTelemetry Collector processors:
+- It filters out all traces for `GET /health` requests.
+- It filters out all logs containing `Health check API invoked!` line in the log line body.
+- It adds a new attribute `environment` to all telemetry data with the value `development`.
+
+To run it, use the following commands:
+```
+cd docker-compose
+docker-compose -f docker-compose/setup-otel-collector-publishing-extended-processed.yaml up -d
 ```
 
+You can check application status, send traffic as described in the previous section.
+You can check telemetry data in the same way as in the previous setup.
+
+Make sure there is no logs containing `Health check API invoked!` line in the log line body in Loki using the following query: `{job="products_service"} |~ "Health check API invoked!.*"`
+Make sure there is no traces for `GET /health` requests in Tempo.
+
+Stop environment using docker compose:
+```
+docker-compose -f docker-compose/setup-otel-collector-publishing-extended-processed.yaml down
+```

blog/BLOG.md

@@ -87,6 +87,8 @@ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/proc
 Add attribute - environment=prod for all signals.
   https://opentelemetry.io/docs/collector/transforming-telemetry/#adding-or-deleting-attributes
 
+NOTE - PROCESSORS AT THE MOMENT OF WRITING ARE ALPHA VERSION AND SUBJECT TO CHANGE!
+
 ## Links and further reading
 Code source for this blog post can be found [here](TODO)
 OpenTelemetry links:

traffic.sh

@@ -1,5 +1,10 @@
 #!/bin/bash
 
+# This script is used to generate traffic to the API server.
+# It creates, retrieves, searches, and deletes 100 product items.
+# It expects the API server to be running on localhost:8080.
+# API server host and port can be overridden by setting the HOST and PORT environment variables.
+
 HOST=${HOST:-localhost}
 PORT=${PORT:-8080}
 BASE_URL="http://$HOST:$PORT/api"