Zipkin plugin: Support for Datadog tracing headers

[chrisforrette]

We've recently started integrating Kong Gateway into a set of roughly 20 existing services, all of which use Datadog for observability data. We had an easy time getting logging wired up, we've configured the Datadog plugin built by Kong, as well as the Kong integration built by Datadog, but the one area where we've struggled is tracing/APM.

The first place we started was the nginx-opentracing module as we've instrumented other Nginx services with the same thing in the past. We started simply—loading up and configuring the module—and ended up going down a rabbit hole. We took all sorts of different approaches and eventually got to attempting to compile it from scratch but we were unable to get it to work with Kong and all of its dependencies.

We are currently working on writing a custom plugin in Go, but since the plugin is detached from the actual executing query process to other services, both instrumenting and the actual reporting from tracing is a bit flimsy.

We made another attempt using the Zipkin plugin and swapping out the Datadog agent for an OpenTelemetry collector. We configured the collector with a Zipkin "collector" and a Datadog "exporter" like so:

receivers:
  zipkin:

processors:
  batch:
    timeout: 10s

exporters:
  datadog/api:

service:
  pipelines:
    traces:
      receivers: [zipkin]
      processors: [batch]
      exporters: [datadog/api]

We got traces but, while the Zipkin plugin has flexibility to use tracing headers that Datadog tracers can work with (B3 and OpenTracing), all of our services are already using Datadog-style headers so there was no interoperability and the traces fell through across service boundaries.

So I would love if datadog could be added as an option for the Zipkin plugin header_type and default_header_type configuration options!

Unfortunately I can't find any formal spec or anything around Datadog tracing headers, but they use the following headers:

• x-datadog-trace-id
• x-datadog-parent-id
• x-datadog-sampling-priority

I can see that their Go tracing library uses random uint64 types for tracing IDs: https://github.com/DataDog/dd-trace-go/blob/v1/ddtrace/tracer/tracer.go#L334

I even found prior art for adding support for OpenTracing headers to the Zipkin plugin here: Kong/kong-plugin-zipkin#95 ...and considered attempting to do this myself, but I've never written a lick of Lua and kept hitting road blocks getting a development environment set up so I could run unit tests.

I think this would go a long way to help folks using Datadog to get better observability into their services running alongside Kong! Please let me know how I can help!

[ghost]

Hi @chrisforrette, I have the same problem, but contact the support datadog they send me this project https://github.com/DataDog/kong-plugin-ddtrace.
It should only be used in development and testing/staging environments 😢.
Provavelmente este projeto será divulgado em junho deste ano.