add k8s object receiver (signalfx#588)

* replace k8seventsreceiver with k8sobjectsreceiver, refactor variables prefixed with events, add the default configuration of objects to pull and watch for k8s objects receiver, update UPGRADING.md, add functional test for k8sobjects Co-authored-by: Dmitrii Anoshin <[email protected]>
wugutech · Jan 10, 2023 · 97af7de · 97af7de
1 parent f4cf82b
commit 97af7de
Show file tree

Hide file tree

Showing 23 changed files with 1,299 additions and 14 deletions.
diff --git a/Makefile b/Makefile
@@ -73,3 +73,15 @@ render:
 		default helm-charts/splunk-otel-collector; \
 	mv "$$dir"/splunk-otel-collector/templates/* "$$dir"; \
 	rm -rf "$$dir"/splunk-otel-collector
+
+	# cluster-receiver objects collection enabled
+	dir=rendered/manifests/cluster-receiver-objects; \
+	mkdir -p "$$dir"; \
+	helm template \
+		--namespace default \
+		--values rendered/values.yaml \
+		--output-dir "$$dir" \
+		--set logsEngine=otel,splunkObservability.logsEnabled=true,clusterReceiver.k8sObjects[0].name=pods,clusterReceiver.k8sObjects[0].mode=pull,clusterReceiver.k8sObjects[0].interval=20,clusterReceiver.k8sObjects[1].name=events,clusterReceiver.k8sObjects[1].mode=watch \
+		default helm-charts/splunk-otel-collector; \
+	mv "$$dir"/splunk-otel-collector/templates/* "$$dir"; \
+	rm -rf "$$dir"/splunk-otel-collector
diff --git a/UPGRADING.md b/UPGRADING.md
@@ -1,5 +1,66 @@
 # Upgrade guidelines
 
+## $CURRENT_VERSION to $NEXT_VERSION
+
+There is a new receiver: [Kubernetes Objects Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver) that can pull or watch any object from Kubernetes API server.
+It will replace the [Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) in the future.
+
+To migrate from Kubernetes Events Receiver to Kubernetes Object Receiver, configure `clusterReceiver` values.yaml section with:
+
+```yaml
+k8sObjects:
+  - mode: watch
+    name: events
+```
+
+There are differences in the log record formatting between the previous `k8s_events` receiver and the now adopted `k8sobjects` receiver results.
+The `k8s_events` receiver stores event messages their log body, with the following fields added as attributes:
+
+* `k8s.object.kind`
+* `k8s.object.name`
+* `k8s.object.uid`
+* `k8s.object.fieldpath`
+* `k8s.object.api_version`
+* `k8s.object.resource_version`
+* `k8s.event.reason`
+* `k8s.event.action`
+* `k8s.event.start_time`
+* `k8s.event.name`
+* `k8s.event.uid`
+* `k8s.namespace.name`
+
+Now with the `k8sobjects` receiver, the whole payload is stored in the log body and `object.message` refers to the event message.
+
+
+You can monitor more Kubernetes objects configuring by `clusterReceiver.k8sObjects` according to the instructions from the
+[Kubernetes Objects Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver) documentation.
+
+Remember to define `rbac.customRules` when needed. For example, when configuring:
+
+```yaml
+objectsEnabled: true
+k8sObjects:
+  - name: events
+    mode: watch
+    group: events.k8s.io
+    namespaces: [default]
+```
+
+You should add `events.k8s.io` API group to the `rbac.customRules`:
+
+```yaml
+rbac:
+  customRules:
+    - apiGroups:
+      - "events.k8s.io"
+      resources:
+      - events
+      verbs:
+      - get
+      - list
+      - watch
+```
+
 ## 0.58.0 to 0.59.0
 [receiver/filelogreceiver] Datatype for `force_flush_period` and `poll_interval` were changed from map to string.
 

diff --git a/ci_scripts/sck_otel_values.yaml b/ci_scripts/sck_otel_values.yaml
@@ -88,3 +88,18 @@ logsCollection:
         priority: info
     # Route journald logs to its own Splunk Index by specifying the index value below, else leave it blank. Please make sure the index exist in Splunk and is configured to receive HEC traffic. Not applicable to Splunk Observability.
     index: ""
+
+clusterReceiver:
+  enabled: true
+  # This flag enables Kubernetes objects collection using OpenTelemetry Kubernetes Object Receiver
+  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver
+  # This option requires `logsEnabled` to be set to `true` for either `splunkObservability` or `splunkPlatform`
+  # depending on where you want to send the events. Otherwise, this option will not have any effect.
+  # The receiver currently is in alpha state which means that events format might change over time.
+  # Once the receiver is stabilized, it'll be enabled by default in this helm chart
+  k8sObjects:
+    - name: pods
+    - name: namespaces
+    - name: nodes
+    - name: events
+      mode: watch
diff --git a/helm-charts/splunk-otel-collector/templates/NOTES.txt b/helm-charts/splunk-otel-collector/templates/NOTES.txt
@@ -69,3 +69,7 @@ Splunk Network Explorer is installed and configured.
 [WARNING] "clusterReceiver.k8sEventsEnabled" parameter is deprecated. Please use clusterReceiver.eventsEnabled and splunkObservability.infrastructureMonitoringEventsEnabled.
           Upgrade guidelines: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/UPGRADING.md#0532-to-0540
 {{ end }}
+{{- if not (eq (toString $clusterReceiver.eventsEnabled) "<nil>") }}
+[WARNING] "clusterReceiver.eventsEnabled" parameter is deprecated. Soon it will be replaced with "clusterReceiver.k8sObjects".
+          Upgrade guidelines: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/UPGRADING.md#$CURRENT_VERSION-to-$NEXT_VERSION
+{{ end }}
diff --git a/helm-charts/splunk-otel-collector/templates/_helpers.tpl b/helm-charts/splunk-otel-collector/templates/_helpers.tpl
@@ -395,3 +395,29 @@ compatibility with the old config group name: "otelK8sClusterReceiver".
 {{- $clusterReceiver.k8sEventsEnabled }}
 {{- end }}
 {{- end -}}
+
+
+{{/*
+Whether object collection by k8s object receiver is enabled
+*/}}
+{{- define "splunk-otel-collector.objectsEnabled" -}}
+{{- $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
+{{- gt (len $clusterReceiver.k8sObjects) 0 }}
+{{- end -}}
+
+{{/*
+Whether object collection by k8s object receiver or/and event collection by k8s event receiver is enabled
+*/}}
+{{- define "splunk-otel-collector.objectsOrEventsEnabled" -}}
+{{- $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
+{{- or $clusterReceiver.eventsEnabled (eq (include "splunk-otel-collector.objectsEnabled" .) "true") -}}
+{{- end -}}
+
+
+{{/*
+Whether clusterReceiver should be enabled
+*/}}
+{{- define "splunk-otel-collector.clusterReceiverEnabled" -}}
+{{- $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
+{{- and $clusterReceiver.enabled (or (eq (include "splunk-otel-collector.metricsEnabled" .) "true") (eq (include "splunk-otel-collector.objectsOrEventsEnabled" .) "true")) -}}
+{{- end -}}
diff --git a/helm-charts/splunk-otel-collector/templates/config/_otel-k8s-cluster-receiver-config.tpl b/helm-charts/splunk-otel-collector/templates/config/_otel-k8s-cluster-receiver-config.tpl
@@ -35,6 +35,11 @@ receivers:
     {{- if eq (include "splunk-otel-collector.distribution" .) "openshift" }}
     distribution: openshift
     {{- end }}
+  {{- if and (eq (include "splunk-otel-collector.objectsEnabled" .) "true") (eq (include "splunk-otel-collector.logsEnabled" .) "true") }}
+  k8sobjects:
+    auth_type: serviceAccount
+    objects: {{ .Values.clusterReceiver.k8sObjects | toYaml | nindent 6 }}
+  {{- end }}
   {{- if and $clusterReceiver.eventsEnabled (eq (include "splunk-otel-collector.logsEnabled" .) "true") }}
   k8s_events:
     auth_type: serviceAccount
@@ -101,6 +106,14 @@ processors:
         action: delete
   {{- end }}
 
+  {{- if and (eq (include "splunk-otel-collector.objectsEnabled" .) "true") (eq (include "splunk-otel-collector.logsEnabled" .) "true") }}
+  transform/add_sourcetype:
+    log_statements:
+      - context: log
+        statements:
+          - set(resource.attributes["com.splunk.sourcetype"], Concat(["kube:object:", attributes["event.name"]], ""))
+  {{- end }}
+
   # Resource attributes specific to the collector itself.
   resource/add_collector_k8s:
     attributes:
@@ -132,7 +145,8 @@ processors:
         value: {{ .value }}
       {{- end }}
 
-  {{- if and $clusterReceiver.eventsEnabled .Values.environment }}
+
+  {{- if and ( eq ( include "splunk-otel-collector.objectsOrEventsEnabled" . ) "true") .Values.environment }}
   resource/add_environment:
     attributes:
       - action: insert
@@ -158,7 +172,7 @@ exporters:
     timeout: 10s
   {{- end }}
 
-  {{- if and (eq (include "splunk-otel-collector.o11yLogsEnabled" .) "true") $clusterReceiver.eventsEnabled }}
+  {{- if and (eq (include "splunk-otel-collector.o11yLogsEnabled" .) "true") (eq (include "splunk-otel-collector.objectsOrEventsEnabled" .) "true") }}
   splunk_hec/o11y:
     endpoint: {{ include "splunk-otel-collector.o11yIngestUrl" . }}/v1/log
     token: "${SPLUNK_OBSERVABILITY_ACCESS_TOKEN}"
@@ -172,10 +186,12 @@ exporters:
   {{- include "splunk-otel-collector.splunkPlatformMetricsExporter" . | nindent 2 }}
   {{- end }}
 
-  {{- if and (eq (include "splunk-otel-collector.platformLogsEnabled" .) "true") $clusterReceiver.eventsEnabled }}
+  {{- if and (eq (include "splunk-otel-collector.platformLogsEnabled" .) "true") (eq (include "splunk-otel-collector.objectsOrEventsEnabled" .) "true") }}
   {{- include "splunk-otel-collector.splunkPlatformLogsExporter" . | nindent 2 }}
+  {{- if $clusterReceiver.eventsEnabled }}
     sourcetype: kube:events
   {{- end }}
+  {{- end }}
 
 service:
   telemetry:
@@ -187,6 +203,7 @@ service:
   extensions: [health_check, memory_ballast]
   {{- end }}
   pipelines:
+    {{- if or (eq (include "splunk-otel-collector.o11yMetricsEnabled" $) "true") (eq (include "splunk-otel-collector.platformMetricsEnabled" $) "true") }}
     # k8s metrics pipeline
     metrics:
       receivers: [k8s_cluster]
@@ -212,7 +229,6 @@ service:
         {{- end }}
     {{- end }}
 
-    {{- if or (eq (include "splunk-otel-collector.splunkO11yEnabled" $) "true") (eq (include "splunk-otel-collector.platformMetricsEnabled" $) "true") }}
     # Pipeline for metrics collected about the collector pod itself.
     metrics/collector:
       receivers: [prometheus/k8s_cluster_receiver]
@@ -253,6 +269,28 @@ service:
         {{- end }}
     {{- end }}
 
+    {{- if and (eq (include "splunk-otel-collector.objectsEnabled" .) "true") (eq (include "splunk-otel-collector.logsEnabled" .) "true") }}
+    logs/objects:
+      receivers:
+        - k8sobjects
+      processors:
+        - memory_limiter
+        - batch
+        - resourcedetection
+        - resource
+        - transform/add_sourcetype
+        {{- if .Values.environment }}
+        - resource/add_environment
+        {{- end }}
+      exporters:
+        {{- if (eq (include "splunk-otel-collector.o11yLogsEnabled" .) "true") }}
+        - splunk_hec/o11y
+        {{- end }}
+        {{- if (eq (include "splunk-otel-collector.platformLogsEnabled" .) "true") }}
+        - splunk_hec/platform_logs
+        {{- end }}
+    {{- end }}
+
     {{- if eq (include "splunk-otel-collector.o11yInfraMonEventsEnabled" .) "true" }}
     logs/events:
       receivers:

diff --git a/...ts/splunk-otel-collector/templates/configmap-cluster-receiver-node-discoverer-script.yaml b/...ts/splunk-otel-collector/templates/configmap-cluster-receiver-node-discoverer-script.yaml
@@ -1,5 +1,4 @@
-{{ $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
-{{ if and $clusterReceiver.enabled (eq (include "splunk-otel-collector.metricsEnabled" .) "true") (eq (include "splunk-otel-collector.distribution" .) "eks/fargate") }}
+{{ if and (eq (include "splunk-otel-collector.clusterReceiverEnabled" . ) "true" ) (eq (include "splunk-otel-collector.distribution" .) "eks/fargate") }}
 apiVersion: v1
 kind: ConfigMap
 metadata:

diff --git a/helm-charts/splunk-otel-collector/templates/configmap-cluster-receiver.yaml b/helm-charts/splunk-otel-collector/templates/configmap-cluster-receiver.yaml
@@ -1,5 +1,5 @@
 {{ $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
-{{ if and $clusterReceiver.enabled (eq (include "splunk-otel-collector.metricsEnabled" .) "true") }}
+{{ if eq (include "splunk-otel-collector.clusterReceiverEnabled" . ) "true" }}
 apiVersion: v1
 kind: ConfigMap
 metadata:

diff --git a/helm-charts/splunk-otel-collector/templates/deployment-cluster-receiver.yaml b/helm-charts/splunk-otel-collector/templates/deployment-cluster-receiver.yaml
@@ -1,5 +1,5 @@
 {{ $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
-{{- if and $clusterReceiver.enabled (eq (include "splunk-otel-collector.metricsEnabled" .) "true") }}
+{{- if eq (include "splunk-otel-collector.clusterReceiverEnabled" .) "true" }}
 apiVersion: apps/v1
 {{- /*
 eks/fargate distributions use a two-replica StatefulSet instead of a single node deployment.

diff --git a/helm-charts/splunk-otel-collector/templates/pdb-cluster-receiver.yaml b/helm-charts/splunk-otel-collector/templates/pdb-cluster-receiver.yaml
@@ -1,5 +1,4 @@
-{{ $clusterReceiver := fromYaml (include "splunk-otel-collector.clusterReceiver" .) }}
-{{- if and (and $clusterReceiver.enabled (eq (include "splunk-otel-collector.metricsEnabled" $) "true")) .Values.podDisruptionBudget }}
+{{- if and (eq (include "splunk-otel-collector.clusterReceiverEnabled" . ) "true" ) .Values.podDisruptionBudget }}
 apiVersion: policy/v1beta1
 kind: PodDisruptionBudget
 metadata:

diff --git a/helm-charts/splunk-otel-collector/values.schema.json b/helm-charts/splunk-otel-collector/values.schema.json
@@ -670,7 +670,15 @@
           "deprecated": true
         },
         "eventsEnabled": {
-          "type": "boolean"
+          "description": "[DEPRECATED] Use k8sObjects instead.",
+          "type": "boolean",
+          "deprecated": true
+        },
+        "k8sObjects": {
+          "type": "array",
+          "items": {
+            "type": "object"
+          }
         },
         "podLabels": {
           "type": "object"

diff --git a/helm-charts/splunk-otel-collector/values.yaml b/helm-charts/splunk-otel-collector/values.yaml
@@ -125,8 +125,8 @@ splunkObservability:
 
   # Option to send Kubernetes events to Splunk Observability Infrastructure Monitoring as data events:
   # https://docs.splunk.com/Observability/alerts-detectors-notifications/view-data-events.html
-  # To send Kubernetes events to Splunk Observability Log Observer, set both clusterReceiver.eventsEnabled and
-  # splunkObservability.logsEnabled to true.
+  # To send Kubernetes events to Splunk Observability Log Observer, configure clusterReceiver.k8sObjects
+  # and set splunkObservability.logsEnabled to true.
   infrastructureMonitoringEventsEnabled: false
 
   # This option just enables the shared pipeline for logs and profiling data.
@@ -431,6 +431,55 @@ clusterReceiver:
   # Once the receiver is stabilized, it'll be enabled by default in this helm chart
   eventsEnabled: false
 
+  # Kubernetes objects collection using OpenTelemetry Kubernetes Object Receiver
+  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver
+  # This option requires `logsEnabled` to be set to `true` for either `splunkObservability` or `splunkPlatform`
+  # depending on where you want to send the events. Otherwise, this option will not have any effect.
+  # The receiver currently is in alpha state which means that events format might change over time.
+  # Once the receiver is stabilized, it'll be enabled by default in this helm chart
+
+  #
+  # == Schema ==
+  # ```
+  # k8sObjects:
+  #   - <objectDefinition>
+  # ```
+  # Each `objectDefinition` has the following fields:
+  # * mode:
+  #     define in which way it collects this type of object, either "pull" or "watch".
+  #     - "poll" mode will read all objects of this type use the list API at an interval. Default mode.
+  #     - "watch" mode will setup a long connection using the watch API to just get updates.
+  # * name: [REQUIRED]
+  #     name of the object, e.g. `pods`, `namespaces`.
+  # * namespace:
+  #     only collects objects from the specified namespace, by default it's all namespaces
+  # * labelSelector:
+  #     select objects by label(s)
+  # * fieldSelector:
+  #     select objects by field(s)
+  # * interval:
+  #     the interval at which object is pulled, default 60 seconds.
+  #     Only useful for "pull" mode.
+  #
+  #
+  # == Example ==
+  # ```
+  #  k8sObjects:
+  #    - name: pods
+  #      mode: pull
+  #      label_selector: environment in (production),tier in (frontend)
+  #      field_selector: status.phase=Running
+  #      interval: 15m
+  #    - name: events
+  #      mode: watch
+  #      group: events.k8s.io
+  #      namespaces: [default]
+  # ```
+  #
+  # The configuration format in details is described here:
+  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver
+  k8sObjects: []
+
   # k8s cluster receiver extra pod labels
   podLabels: {}