Added Monitoring for Kafka Connect

Adelantos · May 31, 2018 · 22f8cf5 · 22f8cf5
1 parent 604b764
commit 22f8cf5
Show file tree

Hide file tree

Showing 6 changed files with 109 additions and 15 deletions.
diff --git a/README.md b/README.md
@@ -199,13 +199,15 @@ JMX Metrics are enabled by default for all components, Prometheus JMX Exporter i
     
     ![Zookeeper Dashboard](screenshots/zookeeper.png "Zookeeper")
     
-## Work In Progress
+## TODO
 ### External Access
 
 ### Security
 
 ### Logging
 
+### Upgrade
+
 ## Huge Thanks
 
 Huge thanks to:

diff --git a/charts/cp-kafka-connect/README.md b/charts/cp-kafka-connect/README.md
@@ -27,7 +27,7 @@ $ helm install --name my-confluent cp-helm-charts
 
 ### Install with a existing CP-Kafka release
 ```console
-$ helm install --set cp-zookeeper.url="unhinged-robin-cp-zookeeper:2181",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-kafka-connect
+$ helm install --set kafka.bootstrapServers="PLAINTEXT://lolling-chinchilla-cp-kafka-headless:9092",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-kafka-connect
 ```
 
 ### Installed Components
@@ -67,24 +67,24 @@ $ helm install --name my-kafka-connect -f my-values.yaml ./cp-kafka-connect
 
 > **Tip**: A default [values.yaml](values.yaml) is provided
 
-### REST Proxy Deployment
+### Kafka Connect Deployment
 The configuration parameters in this section control the resources requested and utilized by the cp-kafka-connect chart.
 
 | Parameter | Description | Default |
 | --------- | ----------- | ------- |
-| `replicaCount` | The number of REST Proxy Servers. | `1` |
+| `replicaCount` | The number of Kafka Connect Servers. | `1` |
 
 ### Image
 | Parameter | Description | Default |
 | --------- | ----------- | ------- |
-| `image` | Docker Image of Confluent REST Proxy. | `confluentinc/cp-kafka-connect` |
-| `imageTag` | Docker Image Tag of Confluent REST Proxy. | `4.1.0` |
-| `imagePullPolicy` | Docker Image Tag of Confluent REST Proxy. | `IfNotPresent` |
+| `image` | Docker Image of Confluent Kafka Connect. | `confluentinc/cp-kafka-connect` |
+| `imageTag` | Docker Image Tag of Confluent Kafka Connect. | `4.1.0` |
+| `imagePullPolicy` | Docker Image Tag of Confluent Kafka Connect. | `IfNotPresent` |
 
 ### Port
 | Parameter | Description | Default |
 | --------- | ----------- | ------- |
-| `servicePort` | The port on which the REST Proxy will be available and serving requests. | `8082` |
+| `servicePort` | The port on which the Kafka Connect will be available and serving requests. | `8082` |
 
 ### Kafka Connect Worker Configurations
 | Parameter | Description | Default |
@@ -100,8 +100,21 @@ The configuration parameters in this section control the resources requested and
 | --------- | ----------- | ------- |
 | `resources.requests.cpu` | The amount of CPU to request. | see [values.yaml](values.yaml) for details |
 | `resources.requests.memory` | The amount of memory to request. | see [values.yaml](values.yaml) for details |
-| `resources.requests.limit` | The upper limit CPU usage for a REST Proxy Pod. | see [values.yaml](values.yaml) for details |
-| `resources.requests.limit` | The upper limit memory usage for a REST Proxy Pod. | see [values.yaml](values.yaml) for details |
+| `resources.requests.limit` | The upper limit CPU usage for a Kafka Connect Pod. | see [values.yaml](values.yaml) for details |
+| `resources.requests.limit` | The upper limit memory usage for a Kafka Connect Pod. | see [values.yaml](values.yaml) for details |
+
+### JMX Configuration
+| Parameter | Description | Default |
+| --------- | ----------- | ------- |
+| `jmx.port` | The jmx port which JMX style metrics are exposed. | `5555` |
+
+### Prometheus JMX Exporter Configuration
+| Parameter | Description | Default |
+| --------- | ----------- | ------- |
+| `prometheus.jmx.enabled` | Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. | `true` |
+| `prometheus.jmx.image` | Docker Image for Prometheus JMX Exporter container. | `solsson/kafka-prometheus-jmx-exporter@sha256` | 
+| `prometheus.jmx.imageTag` | Docker Image Tag for Prometheus JMX Exporter container. | `a23062396cd5af1acdf76512632c20ea6be76885dfc20cd9ff40fb23846557e8` |
+| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping. | `5556` |
 
 ## Dependencies
 ### Kafka

diff --git a/charts/cp-kafka-connect/templates/_helpers.tpl b/charts/cp-kafka-connect/templates/_helpers.tpl
@@ -35,7 +35,7 @@ Create chart name and version as used by the chart label.
 Create a default fully qualified kafka headless name.
 We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
 */}}
-{{- define "cp-kafka-rest.cp-kafka-headless.fullname" -}}
+{{- define "cp-kafka-connect.cp-kafka-headless.fullname" -}}
 {{- $name := "cp-kafka-headless" -}}
 {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
 {{- end -}}
@@ -44,11 +44,11 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
 Form the Kafka URL. If Kafka is installed as part of this chart, use k8s service discovery,
 else use user-provided URL
 */}}
-{{- define "cp-schema-registry.kafka.bootstrapServers" -}}
+{{- define "cp-kafka-connect.kafka.bootstrapServers" -}}
 {{- if .Values.kafka.bootstrapServers -}}
 {{- .Values.kafka.bootstrapServers -}}
 {{- else -}}
-{{- printf "PLAINTEXT://%s:9092" (include "cp-kafka-rest.cp-kafka-headless.fullname" .) -}}
+{{- printf "PLAINTEXT://%s:9092" (include "cp-kafka-connect.cp-kafka-headless.fullname" .) -}}
 {{- end -}}
 {{- end -}}
 

diff --git a/charts/cp-kafka-connect/templates/deployment.yaml b/charts/cp-kafka-connect/templates/deployment.yaml
@@ -18,15 +18,45 @@ spec:
       labels:
         app: {{ template "cp-kafka-connect.name" . }}
         release: {{ .Release.Name }}
+      {{- if .Values.prometheus.jmx.enabled }}
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
+      {{- end }}
     spec:
       containers:
+        {{- if .Values.prometheus.jmx.enabled }}
+        - name: prometheus-jmx-exporter
+          image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
+          command:
+          - java
+          - -XX:+UnlockExperimentalVMOptions
+          - -XX:+UseCGroupMemoryLimitForHeap
+          - -XX:MaxRAMFraction=1
+          - -XshowSettings:vm
+          - -jar
+          - jmx_prometheus_httpserver.jar
+          - {{ .Values.prometheus.jmx.port | quote }}
+          - /etc/jmx-kafka-connect/jmx-kafka-connect-prometheus.yml
+          ports:
+          - containerPort: {{ .Values.prometheus.jmx.port }}
+          resources:
+{{ toYaml .Values.prometheus.jmx.resources | indent 12 }}
+          volumeMounts:
+          - name: jmx-config
+            mountPath: /etc/jmx-kafka-connect
+        {{- end }}
         - name: {{ template "cp-kafka-connect.name" . }}-server
           image: "{{ .Values.image }}:{{ .Values.imageTag }}"
           imagePullPolicy: "{{ .Values.imagePullPolicy }}"
           ports:
             - name: kafka-connect
               containerPort: {{ .Values.servicePort}}
               protocol: TCP
+            {{- if .Values.prometheus.jmx.enabled }}
+            - containerPort: {{ .Values.jmx.port }}
+              name: jmx
+            {{- end }}
           resources:
 {{ toYaml .Values.resources | indent 12 }}
           env:
@@ -37,7 +67,7 @@ spec:
             - name: CONNECT_PLUGIN_PATH
               value: {{ .Values.pluginPath }}
             - name: CONNECT_BOOTSTRAP_SERVERS
-              value: {{ template "cp-schema-registry.kafka.bootstrapServers" . }}
+              value: {{ template "cp-kafka-connect.kafka.bootstrapServers" . }}
             - name: CONNECT_GROUP_ID
               value: {{ template "cp-kafka-connect.groupId" . }}
             - name: CONNECT_CONFIG_STORAGE_TOPIC
@@ -57,4 +87,14 @@ spec:
             - name: CONNECT_INTERNAL_KEY_CONVERTER
               value: {{ .Values.connectInternalKeyConverter }}
             - name: CONNECT_INTERNAL_VALUE_CONVERTER
-              value: {{ .Values.connectInternalValueConverter }}
+              value: {{ .Values.connectInternalValueConverter }}
+            {{- if .Values.jmx.port }}
+            - name: KAFKA_JMX_PORT
+              value: "{{ .Values.jmx.port }}"
+            {{- end }}
+      volumes:
+      {{- if .Values.prometheus.jmx.enabled }}
+      - name: jmx-config
+        configMap:
+          name: {{ template "cp-kafka-connect.fullname" . }}-jmx-configmap
+      {{- end }}
diff --git a/charts/cp-kafka-connect/templates/jmx-configmap.yaml b/charts/cp-kafka-connect/templates/jmx-configmap.yaml
@@ -0,0 +1,22 @@
+{{- if and .Values.prometheus.jmx.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ template "cp-kafka-connect.fullname" . }}-jmx-configmap
+  labels:
+    app: {{ template "cp-kafka-connect.name" . }}
+    chart: {{ template "cp-kafka-connect.chart" . }}
+    release: {{ .Release.Name }}
+    heritage: {{ .Release.Service }}
+data:
+  jmx-kafka-connect-prometheus.yml: |+
+    jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
+    lowercaseOutputName: true
+    lowercaseOutputLabelNames: true
+    ssl: false
+    rules:
+    - pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
+      name: "cp_kafka_connect_connect_worker_metrics_$1"
+    - pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
+      name: "cp_kafka_connect_connect_metrics_$1_$2"
+{{- end }}
diff --git a/charts/cp-kafka-connect/values.yaml b/charts/cp-kafka-connect/values.yaml
@@ -35,6 +35,23 @@ resources: {}
   #  cpu: 100m
   #  memory: 128Mi
 
+## Monitoring
+## Kafka Connect JMX Settings
+## ref: https://kafka.apache.org/documentation/#connect_monitoring
+jmx:
+  port: 5555
+
+## Prometheus Exporter Configuration
+## ref: https://prometheus.io/docs/instrumenting/exporters/
+prometheus:
+  ## JMX Exporter Configuration
+  ## ref: https://github.com/prometheus/jmx_exporter
+  jmx:
+    enabled: true
+    image: solsson/kafka-prometheus-jmx-exporter@sha256
+    imageTag: a23062396cd5af1acdf76512632c20ea6be76885dfc20cd9ff40fb23846557e8
+    port: 5556
+
 ## You can list load balanced service endpoint, or list of all brokers (which is hard in K8s).  e.g.:
 ## bootstrapServers: "PLAINTEXT://dozing-prawn-kafka-headless:9092"
 kafka: