Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Tetragon Events with Pod Annotations Support #3527

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

cy83rc0llect0r
Copy link

Enhancement #3316

Description

This pull request implements support for including Kubernetes pod annotations in Tetragon event logs, addressing the enhancement request in issue #3316. The feature introduces an optional --enable-pod-annotations flag, allowing users to enrich process execution events with pod annotations for improved workload visibility and debugging in Kubernetes environments. The change is split into two commits for clarity:

  1. Core implementation: Adds the pod_annotations field to the Tetragon protobuf definition and integrates it into the event generation logic.
  2. Helm chart support: Provides a configuration option in the Helm chart to enable the feature.

The feature is disabled by default to maintain backward compatibility and can be activated via Helm or command-line flag.

Use Case Example: This enhancement enables tracking of Kubernetes users who execute commands in pods. For instance, a mutating admission webhook could annotate pods with the username (e.g., exec-user) and timestamp (e.g., exec-time) of an exec operation. With --enable-pod-annotations enabled, Tetragon would then include these annotations in its event logs, providing valuable audit trails for security and troubleshooting.

Changelog

Added support for including Kubernetes pod annotations in Tetragon event logs with the new `--enable-pod-annotations` flag (disabled by default). Enable it via Helm with `enablePodAnnotations=true`.

@cy83rc0llect0r cy83rc0llect0r requested a review from a team as a code owner March 20, 2025 14:34
Copy link

netlify bot commented Mar 20, 2025

Deploy Preview for tetragon ready!

Name Link
🔨 Latest commit 15b258d
🔍 Latest deploy log https://app.netlify.com/sites/tetragon/deploys/67e178d9262cd90008c95f95
😎 Deploy Preview https://deploy-preview-3527--tetragon.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@cy83rc0llect0r cy83rc0llect0r force-pushed the pr/cy83rc0llect0r/add-pod-annotations-to-event branch from 6d5b2a5 to 553e100 Compare March 20, 2025 17:06
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Please find some comments below. Also, can you rebase instead of a merge commit?

@@ -86,3 +86,6 @@ data:
cri-endpoint: "unix://{{ .Values.tetragon.cri.socketHostPath }}"
{{- end }}
enable-cgidmap: {{ .Values.tetragon.cgidmap.enabled | quote }}
{{- if .Values.tetragon.enablePodAnnotations }}
enable-pod-annotations: "true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this works because the agent flag is by default false, but if that changes then a user setting enablePodAnnotations to false in helm would not get the expected result.

How about:

enable-pod-annotations: {{- if .Values.tetragon.enablePodAnnotations }}?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved all comments:

  • Rebasing done on upstream/main to remove merge commit as requested.
  • Fixed tetragon_configmap.yaml to always set enable-pod-annotations with .Values.tetragon.podAnnotations.enabled.
  • Moved enablePodAnnotations to tetragon.podAnnotations.enabled in values.yaml for better structure.
  • Excluded local changes to podinfo_test from the PR.
  • Tested with Helm:
    • Deployed with tetragon.podAnnotations.enabled: true → annotations like test-key=test-value appear in logs.
    • Deployed with tetragon.podAnnotations.enabled: false → no annotations in logs.

Pushed to my fork (github.com/cy83rc0llect0r/tetragon). Ready for review!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is resolved.

Copy link
Author

@cy83rc0llect0r cy83rc0llect0r Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkourt
Could you please provide more details? I’ve tested various scenarios, and everything seems to be working as expected:

  • Installed with tetragon.podAnnotations.enabled=false → No annotations appear in the logs.
  • Upgraded to tetragon.podAnnotations.enabled=true → Annotations appear in the logs.
  • Upgraded to tetragon.podAnnotations.enabled=false → Annotations are no longer present in new logs.
  • Installed with tetragon.podAnnotations.enabled=true → Annotations appear in the logs.
  • Upgraded to tetragon.podAnnotations.enabled=false → Annotations are removed from new logs.
  • Upgraded to tetragon.podAnnotations.enabled=true → Annotations appear in the logs.

Copy link
Contributor

@kkourt kkourt Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will happen instead of:

flags.Bool(KeyEnablePodAnnotations, false, "Add pod annotations field to events.")

We have:

flags.Bool(KeyEnablePodAnnotations, true, "Add pod annotations field to events.")

And then use:

tetragon.podAnnotations.enabled=false

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkourt Following up on your comment:

  • Changed the code default to flags.Bool(KeyEnablePodAnnotations, true, "Add pod annotations field to events.").
  • Kept values.yaml default as tetragon.podAnnotations.enabled: false, which overrides the code default via Helm ( and also set to false again).
  • Testing:
    1. helm install tetragon ./install/kubernetes/tetragon → No annotations (Helm sets false).
    2. (after uninstall again:) helm install tetragon ./install/kubernetes/tetragon--set tetragon.podAnnotations.enabled=false → No annotations (Helm sets false).
    3. helm upgrade tetragon ./install/kubernetes/tetragon --set tetragon.podAnnotations.enabled=true → Annotations (e.g., test-key=test-value) in logs.
    4. helm upgrade tetragon ./install/kubernetes/tetragon --set tetragon.podAnnotations.enabled=false → No annotations in new logs.

Does this resolve your concern?

@@ -47,6 +47,7 @@ tetragon:
# extraEnv:
# - name: foo
# value: bar
enablePodAnnotations: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @lambdanis was advocating for having one object per configuration.

So:

podAnnotations:
    enabled: true

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved all comments:

  • Rebasing done on upstream/main to remove merge commit as requested.
  • Fixed tetragon_configmap.yaml to always set enable-pod-annotations with .Values.tetragon.podAnnotations.enabled.
  • Moved enablePodAnnotations to tetragon.podAnnotations.enabled in values.yaml for better structure.
  • Excluded local changes to podinfo_test from the PR.
  • Tested with Helm:
    • Deployed with tetragon.podAnnotations.enabled: true → annotations like test-key=test-value appear in logs.
    • Deployed with tetragon.podAnnotations.enabled: false → no annotations in logs.

Pushed to my fork (github.com/cy83rc0llect0r/tetragon). Ready for review!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for this comment, I don't think this is resolved.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkourt Following up on your comment:

Changed the code default to flags.Bool(KeyEnablePodAnnotations, true, "Add pod annotations field to events.").
Kept values.yaml default as tetragon.podAnnotations.enabled: false, which overrides the code default via Helm ( and also set to false again).
Testing:
helm install tetragon ./install/kubernetes/tetragon → No annotations (Helm sets false).
(after uninstall again:) helm install tetragon ./install/kubernetes/tetragon--set tetragon.podAnnotations.enabled=false → No annotations (Helm sets false).
helm upgrade tetragon ./install/kubernetes/tetragon --set tetragon.podAnnotations.enabled=true → Annotations (e.g., test-key=test-value) in logs.
helm upgrade tetragon ./install/kubernetes/tetragon --set tetragon.podAnnotations.enabled=false → No annotations in new logs.
Does this resolve your concern?

@cy83rc0llect0r cy83rc0llect0r force-pushed the pr/cy83rc0llect0r/add-pod-annotations-to-event branch from 6ad10e7 to 1131ac5 Compare March 21, 2025 15:48
@cy83rc0llect0r cy83rc0llect0r requested a review from kkourt March 21, 2025 15:49
This commit adds support for including Kubernetes pod annotations in Tetragon
event logs via the '--enable-pod-annotations' flag. Changes:
- Added 'pod_annotations' field to 'tetragon.Pod' in 'tetragon.proto' (ID 8).
- Regenerated protobuf files ('tetragon.pb.go').
- Added 'EnablePodAnnotations' config and flag in 'pkg/option/config.go' and 'pkg/option/flags.go'.
- Updated 'getPodInfo' in 'pkg/process/podinfo.go' to include annotations when enabled.

Enhancement cilium#3316

Signed-off-by: Amir Reza Nazarizadeh <[email protected]>
This commit updates the Helm chart to support enabling pod annotations via the
'enablePodAnnotations' value (default: false) in 'values.yaml' and
'tetragon_configmap.yaml'.

Enhancement: cilium#3316

Signed-off-by: Amir Reza Nazarizadeh <[email protected]>
@cy83rc0llect0r cy83rc0llect0r force-pushed the pr/cy83rc0llect0r/add-pod-annotations-to-event branch from 1131ac5 to 15b258d Compare March 24, 2025 15:23
@cy83rc0llect0r cy83rc0llect0r force-pushed the pr/cy83rc0llect0r/add-pod-annotations-to-event branch from 15b258d to 9ab5d2c Compare March 24, 2025 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants