2.21.3

Bug Fixes

• Tracing
  • configurations: Resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly

3.1.0

New Features

• Dynamic Instrumentation

  • Symbol Database is now enabled by default.

• LLM Observability

  • Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
  • Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
  • Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
  • Introduces metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.

• Tracing

  • api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
  • graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
  • graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.
  • Introduces a record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.

Bug Fixes

• ASM

  • Resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

• Code Security (IAST)

  • In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection accuracy by identifying the vulnerability at the set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.

• LLM Observability

  • botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.
  • Resolves an issue where the Gemini and Vertex AI integrations were not properly submitting spans for streamed responses in the case that the generator is not fully consumed.
  • Resolves an issue with correlating traces between APM and LLM Observability where trace IDs were always being converted to hexadecimal values which broke APM links for 64-bit trace IDs.
  • Fixes non-LLM message inputs and outputs to be rendered as json rather than Python strings.

• Profiling

  • Fixes a crash in memory allocation profiler when tracking a large number of live allocations, caused by unsigned integer wraparound which lead to invalid memory access
  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Single-Step Instrumentation

  • Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.
  • Resolves an issue in which incorrect runtime version constraints prevented injection into processes using Python 3.13.1+

• Tracing

  • asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.
  • baggage: Fixes an issue where ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.
  • configurations: This fix resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
  • configurations: Resolves an issue where accessing ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.
  • internal: Require bytecode>=0.15.1 for Python 3.12 to avoid known bug.
  • internal: Fix ddtrace internal logger initialization mutating an unlocked shared resource.
  • library: Updates the exception raised when ddtrace.patch(...) is called with an invalid argument.
  • pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.
  • runtime_metrics: Ensures only the global service name is tagged on runtime metrics, instead of every service name found in the process.
  • span: Fixes issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.
  • Captures the execution of async functions traced via DD_TRACE_METHODS, not just coroutine creation. This change increases span durations, which may affect latency-based metrics.
  • Resolves ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.

3.1.0rc1

New Features

• Dynamic Instrumentation

  • Symbol Database is now enabled by default.

• LLM Observability

  • Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
  • Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
  • Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
  • Introduces metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.

• Tracing

  • api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
  • graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
  • graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.
  • Introduces a record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.

Bug Fixes

• ASM

  • Resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

• Code Security (IAST)

  • In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection accuracy by identifying the vulnerability at the set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.

• LLM Observability

  • botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.
  • Resolves an issue where the Gemini and Vertex AI integrations were not properly submitting spans for streamed responses in the case that the generator is not fully consumed.
  • Resolves an issue with correlating traces between APM and LLM Observability where trace IDs were always being converted to hexadecimal values which broke APM links for 64-bit trace IDs.
  • Fixes non-LLM message inputs and outputs to be rendered as json rather than Python strings.

• Profiling

  • Fixes a crash in memory allocation profiler when tracking a large number of live allocations, caused by unsigned integer wraparound which lead to invalid memory access
  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Single-Step Instrumentation

  • Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.
  • Resolves an issue in which incorrect runtime version constraints prevented injection into processes using Python 3.13.1+

• Tracing

  • asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.
  • baggage: Fixes an issue where ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.
  • configurations: This fix resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
  • configurations: Resolves an issue where accessing ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.
  • internal: Require bytecode>=0.15.1 for Python 3.12 to avoid known bug.
  • internal: Fix ddtrace internal logger initialization mutating an unlocked shared resource.
  • library: Updates the exception raised when ddtrace.patch(...) is called with an invalid argument.
  • pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.
  • runtime_metrics: Ensures only the global service name is tagged on runtime metrics, instead of every service name found in the process.
  • span: Fixes issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.
  • Captures the execution of async functions traced via DD_TRACE_METHODS, not just coroutine creation. This change increases span durations, which may affect latency-based metrics.
  • Resolves ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.

2.21.2

Bug Fixes

Internal:

• Fix performance overhead of Python distribution parsing for internal telemetry.

ASM:

• This fix resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

Library Injection:

• Address partial library copy failures causing failed application startups in k8s.
• Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.

2.20.2

Bug Fixes

• ASM

  • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
  • Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• Code Security

  • SCA: Ensures that no module from IAST are loaded when IAST is disabled or unavailable.
  • Runtime Code Analysis (IAST): Avoids imports of IAST native module when IAST is not enabled.

• Tracing

  • internal: Fixes performance overhead of Python distribution parsing for internal telemetry.

2.21.1

Bug Fixes

• ASM

  • Ensure that no module from ASM are loaded when ASM is disabled or unavailable.

• Code Security

  • Runtime Code Analysis (IAST): Avoid imports of IAST native module when IAST is not enabled.
  • SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.
  • Resolve an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• Profiling

  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Tracing

  • aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.
  • span: Fix issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.

2.19.4

Bug Fixes

• ASM: This ensures that no module from ASM are loaded when ASM is disabled or unavailable.

• SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.

• Runtime Code Analysis (IAST): Avoid imports of IAST native module when IAST is not enabled.

• ASM: This fix resolves an issue where IAST modules could be loaded, even if disabled,
  which could create an ImportError exception on Windows.

3.0.0

Migration Guide: Upgrading from dd-trace-py v2 to v3

We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.

The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the versioning policy for support level definitions.

---

Summary of Significant Changes

• Removed support for Python 3.7 from the library

CI Visibility

• The new pytest plugin is now the default.
• Module, suite, and test names are now parsed from item.nodeid.
• Test names now include the class name (TestClass::test_method) for class-based tests.
• Test skipping is now done at the suite level.
• The DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable is no longer used.

LLM Observability

• Removed support for DD_LLMOBS_APP_NAME.
• The parameters argument in LLMObs.annotate() has been removed. Use metadata instead.
• Dropped support for OpenAI 0.x and Langchain v0.0.x.

Tracing

• Removed all deprecated interfaces from integrations.
• DD_TRACE_PROPAGATION_STYLE=b3 single header is no longer supported. Use b3 instead.
• Dropped support for multiple Tracer instances. The global tracer instance ddtrace.tracer must be used.
• Removed deprecated parameters from Tracer.configure(...).
• Dropped support for several deprecated tracing configurations. See “Environment Variables” section below for details.

Recommended Pre-Upgrade Steps

Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (2.21.0):

Detect Deprecation Warnings in Tests

Enable warnings as errors in pytest:

pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.py

Detect Deprecation Warnings in Applications

Set the following environment variable to raise exceptions for deprecated usage:

DD_TRACE_RAISE_DEPRECATIONWARNING=1 ddtrace-run python app.py

Alternatively, enable all deprecation warnings:

PYTHONWARNINGS=all python app.py

Environment Variables

The following environment variables have been removed or replaced in v3. Use the table below to update your configuration:

v2	Removed in v3	Replaced/Changed in v3
LLM Observability
DD_LLMOBS_APP_NAME		Use DD_LLMOBS_ML_APP instead
_DD_LLMOBS_EVALUATOR_SAMPLING_RULES		Use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead
_DD_LLMOBS_EVALUATORS		Use DD_LLMOBS_EVALUATORS instead
Tracing
DD_ANALYTICS_ENABLED	No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_HTTP_CLIENT_TAG_QUERY_STRING		Use DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING.
DD_TRACE_ANALYTICS_ENABLED	No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_TRACE_METHODS (using [] notation)		Must use : notation instead, For example: mod.submod:method1,method2;mod.submod:Class.method1. (docs)
DD_TRACE_PROPAGATION_STYLE=”b3 single header”		Use DD_TRACE_PROPAGATION_STYLE=b3 for identical behavior
DD_TRACE_SAMPLE_RATE		Use DD_TRACE_SAMPLING_RULES (docs).
DD_TRACE_SPAN_AGGREGATOR_RLOCK	No replacement (feature removed). Config variable is now a no-op.
CI Visibility
DD_PYTEST_USE_NEW_PLUGIN_BETA	No replacement (new pytest plugin is not in beta anymore).

Use the following patterns to identify the deprecated environment variables in a code base:

git grep -P -e "DD_LLMOBS_APP_NAME" \
  -e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
  -e "_DD_LLMOBS_EVALUATORS" \
  -e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
  -e "DD_TRACE_SAMPLE_RATE" \
  -e "DD_TRACE_API_VERSION=v0.3" \
  -e "DD_ANALYTICS_ENABLED" \
  -e "DD_TRACE_ANALYTICS_ENABLED" \
  -e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
  -e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
  -e "DD_TRACE_METHODS=.*\[\]"

Replace deprecated settings with the recommended alternatives where applicable.

NOTE: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.

Legacy Interfaces

The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.

v2	v3
Tracing
Tracer.configure( enabled=foo, hostname=foo, port=foo, uds_path=foo, https=foo, sampler=foo, wrap_executor=foo, priority_sampling=foo, settings=foo, dogstatsd_url=foo, writer=foo, partial_flush_enabled=foo, partial_flush_min_spans=foo, api_version=foo )	Tracer.configure()
ddtrace.opentracer.tracer.Tracer()._dd_tracer	ddtrace.tracer
ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True)	ddtrace.tracer.sampler.rules[].choose_matcher(“foo”) Callables and regex patterns are no longer allowed as arguments to choose_matcher
Span.sampled	Span.context.sampling_priority > 0
LLM Observability
LLMObs.annotate(parameters=foo)	LLMObs.annotate(metadata=foo)

Python 3.13 Support

Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.

• dd-trace-py doesn’t work on Windows with Python 3.13
• Appsec Threat Detection is not tested against Django, Flask, or FastAPI with 3.13
• Automatic Service Naming is not tested with 3.13
• The following products are not tested with 3.13:
  • Code Coverage
  • Appsec IAST
  • Data Streams Monitoring
  • CI Visibility
  • Continuous Profiling
• The following integrations are not tested with 3.13:
  • anthropic
  • consul
  • freezegun
  • gevent
  • google_generativeai
  • gunicorn
  • langchain
  • mysqlclient
  • opentracing
  • psycopg
  • psycopg2
  • pymysql
  • pytest
  • pytest-bdd
  • pytest-benchmark
  • sanic
  • selenium
  • sqlite3
  • starlette
  • tornado
  • vertexai

---

By following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!

---

New Features

• Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.

• ASM

  • Introduces use of meta struct for appsec event reports. This will improve performances. ASM will now require at least agent version 7.35.0 to work as expected.

• Code Security (IAST)

  • Introduces Code Security support to PyMySQL and MySQLDB integrations.
  • XSS detection for Django applications and Jinja2 (Flask and FastAPI applications), which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Implements the detection of the Stacktrace-Leak vulnerability for Django, Flask and FastAPI.

• LLM Observability

  • valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!

  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

  • Introduces an integration with the RAGAS evaluation framework to continuously monitor the performance of context-augmented LLM generations in production.
    The integration supports evaluating LLM inferences with the following RAGAS metrics:

    • Faithfulness: measures if the LLM response is faithful to the provided context.
    • Answer Relevancy: measures how relevant the LLM response is to the user input.
    • Context Precision: measures how effectively the context is used in the generated response.

    To learn more, see the LLM Observability evaluations guide.

Upgrade Notes

• CI Visibility
  • Official release of the new version of the pytest plugin, introducing the following features:
    • Auto Test Retries
    • Early Flake Detection
    • Improved coverage collection for ...

3.0.0rc3

Bug Fixes

• ASM

  • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.

• Code Security

  • SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.

• Tracing

  • internal: Fixes performance overhead of Python distribution parsing for internal telemetry.

2.21.0

New Features

• LLM Observability
  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

Deprecation Notes

• Tracing
  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.

Bug Fixes

• CI Visibility

  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.

• Code Security

  • Resolves an issue where the usage of callonce decorator could trigger an import loop

• LLM Observability

  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.

• Profiling

  • Fixes SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
  • cassandra, cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.