Skip to content

Commit

Permalink
Renamed o11y-collector to splunk-otel-collector (signalfx#42)
Browse files Browse the repository at this point in the history
* Renamed o11y-collector to splunk-otel-collector
  • Loading branch information
flands authored Feb 5, 2021
1 parent 5f2bfcd commit c311e8e
Show file tree
Hide file tree
Showing 22 changed files with 176 additions and 204 deletions.
11 changes: 11 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
## Running locally

It's recommended to use [minikube](https://github.com/kubernetes/minikube) with
[calico networking](https://docs.projectcalico.org/getting-started/kubernetes/) to run splunk-otel-collector locally.

If you run it on Windows or MacOS, use a linux VM driver, e.g. virtualbox.
In that case use the following arguments to start minikube cluster:

```bash
minikube start --cni calico --vm-driver=virtualbox
```
120 changes: 40 additions & 80 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,20 @@
# Observability Collector for Kubernetes
# Splunk OpenTelemetry Connector for Kubernetes

`o11y-collector` is a [Helm](https://github.com/kubernetes/helm) chart that
creates a kubernetes daemonset along with other kubernetes objects in a
kubernetes cluster to collect the cluster's logs, traces and metrics send them to
[Signalfx](https://www.signalfx.com/).
The Splunk OpenTelemetry Connector for Kubernetes is a
[Helm](https://github.com/kubernetes/helm) chart for the [Splunk Distribution
of OpenTelemetry Collector](https://github.com/signalfx/splunk-otel-collector).
This chart creates a Kubernetes DaemonSet along with other Kubernetes objects
in a Kubernetes cluster to collect the cluster's:

### Log Collection
- Metrics for [Splunk Infrastructure
Monitoring](https://www.splunk.com/en_us/software/infrastructure-monitoring.html)
- Traces for [Splunk
APM](https://www.splunk.com/en_us/software/microservices-apm.html)
- Logs for Splunk Log Observer

The deamonset runs [fluentd](https://www.fluentd.org/) with the
[Splunk HEC output plugin](https://github.com/splunk/fluent-plugin-splunk-hec)
to collect logs and send them over
[Splunk HEC](http://docs.splunk.com/Documentation/Splunk/7.1.0/Data/AboutHEC).
> :construction: This project is currently in **BETA**.
It does not only collects logs for applications which are running in the
kubernetes cluster, but also the logs for kubernetes itself (i.e. logs from
`kubelet`, `apiserver`, etc.). It reads logs from both file system with the
[fluentd tail plugin](https://docs.fluentd.org/v1.0/articles/in_tail) and
[systemd journal](http://0pointer.de/blog/projects/journalctl.html) with
[`fluent-plugin-systemd`](https://github.com/reevoo/fluent-plugin-systemd).

### Trace Collection

The deamonset runs [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector/) with the
[Splunk SAPM Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/sapmexporter)
to collect traces and send them to
[Splunk SignalFx Microservices APM](https://www.splunk.com/en_us/software/microservices-apm.html).

### Metric Collection

[OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector/) also collects kubernetes and host
metrics using the following components enabled by default:
- [Kubeletstats receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver)
to collect metrics from Kubelet API.
- [K8s cluster receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver)
to collect metrics from Kubernetes API.
- [Host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/hostmetricsreceiver)
to collect host metrics from kubernetes node.

The metrics are sent to
[Splunk SignalFx Infrastructure Monitoring](https://www.splunk.com/en_us/software/infrastructure-monitoring.html).

## Usage
## Getting Started

### Prerequisites

Expand All @@ -51,49 +25,43 @@ The following components required to use the helm chart:

### How to install

To install o11y-collector in k8s cluster at least three parameters must be provided:
- `splunkRealm` (default `us0`): SignalFx realm to send telemetry data to.
- `splunkAccessToken`: Your SignalFx org access token.
- `clusterName`: arbitrary value that will identify your kubernetes cluster in SignalFx environment.
To install splunk-otel-collector in k8s cluster at least three parameters must be provided:

The repository is not publicly available yet.
In order to install the helm chart you need to clone the repo first and use it locally.

```bash
$ git clone [email protected]:signalfx/o11y-collector-for-kubernetes.git
$ cd ./o11y-collector-for-kubernetes
$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" ./helm-charts/o11y-collector
```
- `splunkRealm` (default `us0`): Splunk realm to send telemetry data to.
- `splunkAccessToken`: Your Splunk org access token.
- `clusterName`: arbitrary value that will identify your Kubernetes cluster in Splunk.

Once the repository is public, it will be possible to use helm repository to install the chart.
To deploy the chart run the following commands replacing the parameters above
with their appropriate values.

```bash
$ helm repo add o11y-collector-for-kubernetes https://signalfx.github.io/o11y-collector-for-kubernetes
$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" o11y-collector-for-kubernetes/o11y-collector
$ helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
$ helm install my-splunk-otel-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector
```

Instead of setting helm values as arguments a yaml file can be provided:

```bash
$ helm install my-o11y-collector --values my_values.yaml ./helm-charts/o11y-collector
$ helm install my-splunk-otel-collector --values my_values.yaml ./helm-charts/splunk-otel-collector
```

### How to uninstall

To uninstall/delete a deployment with name `my-o11y-collector`:
To uninstall/delete a deployment with name `my-splunk-otel-collector`:

```bash
$ helm delete my-o11y-collector
$ helm delete my-splunk-otel-collector
```

The command removes all the Kubernetes components associated with the chart and
deletes the release.

## Configuration
## Advanced Configuration

The [values.yaml](https://github.com/signalfx/o11y-collector-for-kubernetes/blob/main/helm-charts/o11y-collector/values.yaml)
lists all supported configurable parameters for this chart, along with detailed explanation.
Read through it to understand how to configure this chart.
The
[values.yaml](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml)
lists all supported configurable parameters for this chart, along with detailed
explanation. Read through it to understand how to configure this chart.

At the minimum you need to configure the following values.

Expand All @@ -105,39 +73,31 @@ splunkRealm: us0
### Kubernetes platform
Use `platform` parameter to provide information about underlying kubernetes platform.
It'll allow the o11y collector to automatically scrape additional cloud metadata. Supported options:
Use the `platform` parameter to provide information about underlying Kubernetes
platform. This parameter allows the connector to automatically scrape
additional cloud metadata. The supported options are:

- `aws` - Amazon EKS or self-managed k8s cluster in AWS environment.
- `gcp` - Google GKE or self-managed k8s cluster in GCP environment.
- `default` - default configuration for other platforms.

### Disable particular types of telemetry

By default all telemetry data (metrics, traces and logs) is collected from the k8s cluster.
By default all telemetry data (metrics, traces and logs) is collected from the Kubernetes cluster.
It's possible to disable any kind of telemetry with the following parameters:

- `metricsEnabled`: `false`
- `tracesEnabled`: `false`
- `logsEnabled`: `false`

For example, to install o11y collector only for logs:

```bash
$ helm install my-o11y-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster,metricsEnabled=false,tracesEnabled=false" ./helm-charts/o11y-collector
```

## Running locally

It's recommended to use [minikube](https://github.com/kubernetes/minikube) with
[calico networking](https://docs.projectcalico.org/getting-started/kubernetes/) to run o11y-collector locally.

If you run it on Windows or MacOS, use a linux VM driver, e.g. virtualbox.
In that case use the following arguments to start minikube cluster:
For example, to install the connector only for logs:

```bash
minikube start --cni calico --vm-driver=virtualbox
$ helm install my-splunk-otel-collector \
--set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster,metricsEnabled=false,tracesEnabled=false" \
./helm-charts/splunk-otel-collector
```

## License ##
## License

See [LICENSE](LICENSE).
[Apache Software License version 2.0](LICENSE).
5 changes: 3 additions & 2 deletions helm-charts/o11y-collector/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
apiVersion: v2
name: o11y-collector
name: splunk-otel-collector
version: 0.3.0
description: Splunk Observability Collector for Kubernetes Helm Chart
description: Splunk OpenTelemetry Connector for Kubernetes
type: application
keywords:
- splunk
- opentelemetry
- observability
- logs
- traces
Expand Down
46 changes: 23 additions & 23 deletions helm-charts/o11y-collector/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "o11y-collector.name" -}}
{{- define "splunk-otel-collector.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Expand All @@ -12,7 +12,7 @@ Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "o11y-collector.fullname" -}}
{{- define "splunk-otel-collector.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
Expand All @@ -28,14 +28,14 @@ If release name contains chart name it will be used as a full name.
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "o11y-collector.chart" -}}
{{- define "splunk-otel-collector.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "o11y-collector.secret" -}}
{{- define "splunk-otel-collector.secret" -}}
{{- if .Values.secret.name -}}
{{- printf "%s" .Values.secret.name -}}
{{- else -}}
Expand All @@ -46,23 +46,23 @@ Create chart name and version as used by the chart label.
{{/*
Create the name of the service account to use
*/}}
{{- define "o11y-collector.serviceAccountName" -}}
{{ default (include "o11y-collector.fullname" .) .Values.serviceAccount.name }}
{{- define "splunk-otel-collector.serviceAccountName" -}}
{{ default (include "splunk-otel-collector.fullname" .) .Values.serviceAccount.name }}
{{- end -}}

{{/*
Get Signalfx ingest host
Get Splunk ingest host
*/}}
{{- define "o11y-collector.ingestHost" -}}
{{- define "splunk-otel-collector.ingestHost" -}}
{{- $_ := required "splunkRealm or ingestHost must be provided" (or .Values.ingestHost .Values.splunkRealm) }}
{{- .Values.ingestHost | default (printf "ingest.%s.signalfx.com" .Values.splunkRealm) }}
{{- end -}}

{{/*
Get Signalfx ingest URL
Get Splunk ingest URL
*/}}
{{- define "o11y-collector.ingestUrl" -}}
{{- $host := include "o11y-collector.ingestHost" . }}
{{- define "splunk-otel-collector.ingestUrl" -}}
{{- $host := include "splunk-otel-collector.ingestHost" . }}
{{- $endpoint := printf "%s://%s" .Values.ingestProtocol $host }}
{{- if or (and (eq .Values.ingestProtocol "http") (ne (toString .Values.ingestPort) "80")) (and (eq .Values.ingestProtocol "https") (ne (toString .Values.ingestPort) "443")) }}
{{- printf "%s:%s" $endpoint (toString .Values.ingestPort) }}
Expand All @@ -72,38 +72,38 @@ Get Signalfx ingest URL
{{- end -}}

{{/*
Get Signalfx API URL.
Get Splunk API URL.
*/}}
{{- define "o11y-collector.apiUrl" -}}
{{- define "splunk-otel-collector.apiUrl" -}}
{{- $_ := required "splunkRealm or apiUrl must be provided" (or .Values.apiUrl .Values.splunkRealm) }}
{{- .Values.apiUrl | default (printf "https://api.%s.signalfx.com" .Values.splunkRealm) }}
{{- end -}}

{{/*
Get splunkAccessToken.
*/}}
{{- define "o11y-collector.accessToken" -}}
{{- define "splunk-otel-collector.accessToken" -}}
{{- required "splunkAccessToken value must be provided" .Values.splunkAccessToken -}}
{{- end -}}

{{/*
Create the fluentd image name.
*/}}
{{- define "o11y-collector.image.fluentd" -}}
{{- define "splunk-otel-collector.image.fluentd" -}}
{{- printf "%s/%s:%s" .Values.image.fluentd.registry .Values.image.fluentd.name .Values.image.fluentd.tag -}}
{{- end -}}

{{/*
Create the opentelemetry collector image name.
*/}}
{{- define "o11y-collector.image.otelcol" -}}
{{- define "splunk-otel-collector.image.otelcol" -}}
{{- printf "%s/%s:%s" .Values.image.otelcol.registry .Values.image.otelcol.name .Values.image.otelcol.tag -}}
{{- end -}}

{{/*
Convert memory value from resources.limit to numeric value in MiB to be used by otel memory_limiter processor.
*/}}
{{- define "o11y-collector.convertMemToMib" -}}
{{- define "splunk-otel-collector.convertMemToMib" -}}
{{- $mem := lower . -}}
{{- if hasSuffix "e" $mem -}}
{{- trimSuffix "e" $mem | atoi | mul 1000 | mul 1000 | mul 1000 | mul 1000 -}}
Expand Down Expand Up @@ -137,20 +137,20 @@ Convert memory value from resources.limit to numeric value in MiB to be used by
{{/*
Get otel memory_limiter limit_mib value based on 80% of resources.memory.limit.
*/}}
{{- define "o11y-collector.getOtelMemLimitMib" -}}
{{- div (mul (include "o11y-collector.convertMemToMib" .resources.limits.memory) 80) 100 }}
{{- define "splunk-otel-collector.getOtelMemLimitMib" -}}
{{- div (mul (include "splunk-otel-collector.convertMemToMib" .resources.limits.memory) 80) 100 }}
{{- end -}}

{{/*
Get otel memory_limiter spike_limit_mib value based on 25% of resources.memory.limit.
*/}}
{{- define "o11y-collector.getOtelMemSpikeLimitMib" -}}
{{- div (mul (include "o11y-collector.convertMemToMib" .resources.limits.memory) 25) 100 }}
{{- define "splunk-otel-collector.getOtelMemSpikeLimitMib" -}}
{{- div (mul (include "splunk-otel-collector.convertMemToMib" .resources.limits.memory) 25) 100 }}
{{- end -}}

{{/*
Get otel memory_limiter ballast_size_mib value based on 40% of resources.memory.limit.
*/}}
{{- define "o11y-collector.getOtelMemBallastSizeMib" }}
{{- div (mul (include "o11y-collector.convertMemToMib" .resources.limits.memory) 40) 100 }}
{{- define "splunk-otel-collector.getOtelMemBallastSizeMib" }}
{{- div (mul (include "splunk-otel-collector.convertMemToMib" .resources.limits.memory) 40) 100 }}
{{- end -}}
8 changes: 4 additions & 4 deletions helm-charts/o11y-collector/templates/clusterRole.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ template "o11y-collector.fullname" . }}
name: {{ template "splunk-otel-collector.fullname" . }}
labels:
app: {{ template "o11y-collector.name" . }}
chart: {{ template "o11y-collector.chart" . }}
app: {{ template "splunk-otel-collector.name" . }}
chart: {{ template "splunk-otel-collector.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
rules:
{{- if .Values.podSecurityPolicy.create }}
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: [{{ template "o11y-collector.fullname" . }}]
resourceNames: [{{ template "splunk-otel-collector.fullname" . }}]
{{- end }}
- apiGroups:
- ""
Expand Down
10 changes: 5 additions & 5 deletions helm-charts/o11y-collector/templates/clusterRoleBinding.yaml
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ template "o11y-collector.fullname" . }}
name: {{ template "splunk-otel-collector.fullname" . }}
labels:
app: {{ template "o11y-collector.name" . }}
chart: {{ template "o11y-collector.chart" . }}
app: {{ template "splunk-otel-collector.name" . }}
chart: {{ template "splunk-otel-collector.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ template "o11y-collector.fullname" . }}
name: {{ template "splunk-otel-collector.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ template "o11y-collector.serviceAccountName" . }}
name: {{ template "splunk-otel-collector.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
Loading

0 comments on commit c311e8e

Please sign in to comment.