Observability Collector for Kubernetes

o11y-collector is a Helm chart that creates a kubernetes daemonset along with other kubernetes objects in a kubernetes cluster to collect the cluster's logs, traces and metrics send them to Signalfx.

Log Collection

The deamonset runs fluentd with the Splunk HEC output plugin to collect logs and send them over Splunk HEC.

It does not only collects logs for applications which are running in the kubernetes cluster, but also the logs for kubernetes itself (i.e. logs from kubelet, apiserver, etc.). It reads logs from both file system with the fluentd tail plugin and systemd journal with fluent-plugin-systemd.

Trace Collection

The deamonset runs OpenTelemetry Collector with the Splunk SAPM Exporter to collect traces and send them to Splunk SignalFx Microservices APM.

Metric Collection

OpenTelemetry Collector also collects kubernetes and host metrics using the following components enabled by default:

• Kubeletstats receiver to collect metrics from Kubelet API.
• K8s cluster receiver to collect metrics from Kubernetes API.
• Host metrics receiver to collect host metrics from kubernetes node.

The metrics are sent to Splunk SignalFx Infrastructure Monitoring.

Usage

Prerequisites

The following components required to use the helm chart:

• Helm client
• Kubernetes cluster

How to install

To install o11y-collector in k8s cluster at least three parameters must be provided:

• splunkRealm (default us0): SignalFx realm to send telemetry data to.
• splunkAccessToken: Your SignalFx org access token.
• clusterName: arbitrary value that will identify your kubernetes cluster in SignalFx environment.

The repository is not publicly available yet. In order to install the helm chart you need to clone the repo first and use it locally.

$ git clone [email protected]:signalfx/o11y-collector-for-kubernetes.git
$ cd ./o11y-collector-for-kubernetes
$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" ./helm-charts/o11y-collector

Once the repository is public, it will be possible to use helm repository to install the chart.

$ helm repo add o11y-collector-for-kubernetes https://signalfx.github.io/o11y-collector-for-kubernetes
$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" o11y-collector-for-kubernetes/o11y-collector

Instead of setting helm values as arguments a yaml file can be provided:

$ helm install my-o11y-collector --values my_values.yaml ./helm-charts/o11y-collector

How to uninstall

To uninstall/delete a deployment with name my-o11y-collector:

$ helm delete my-o11y-collector

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The values.yaml lists all supported configurable parameters for this chart, along with detailed explanation. Read through it to understand how to configure this chart.

At the minimum you need to configure the following values.

clusterName: my-k8s-cluster
splunkAccessToken: xxxxxx
splunkRealm: us0

Kubernetes platform

Use platform parameter to provide information about underlying kubernetes platform. It'll allow the o11y collector to automatically scrape additional cloud metadata. Supported options:

• aws - Amazon EKS or self-managed k8s cluster in AWS environment.
• gcp - Google GKE or self-managed k8s cluster in GCP environment.
• default - default configuration for other platforms.

Disable particular types of telemetry

By default all telemetry data (metrics, traces and logs) is collected from the k8s cluster. It's possible to disable any kind of telemetry with the following parameters:

• metricsEnabled: false
• tracesEnabled: false
• logsEnabled: false

For example, to install o11y collector only for logs:

$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster,metricsEnabled=false,tracesEnabled=false" ./helm-charts/o11y-collector

License

See LICENSE.