Changelogs for versions not listed here can be found at https://github.com/DataDog/dd-trace-py/releases
tracing: Fixes an issue where truncation of span attributes longer than 25000 characters would not consistently count the size of UTF-8 multibyte characters, leading to a unicode string is too large error.
CI Visibility: This fix resolves an issue where the DD_CIVISIBILITY_ITR_ENABLED was not honored properly.
tracing: Fixes a bug in distributed tracing where pickling <span class="title-ref">ddtrace.trace.Context</span> fails in coroutines. This regression was introduced in v3.7.0.
CI Visibility: This fix resolves an issue where pytest-xdist would not exit with the proper status code if ATR was enabled.
CI Visibility: This fix resolves an issue where ddtrace pytest plugin used with xdist would report test suites as failing even when all tests pass.
profiling: fixed an issue in the SynchronizedSamplePool where pool could be null when calling into ddog_ArrayQueue_ functions, leading to segfaults in the uWSGI shutdown
Code Security: IAST: Avoid excessive filtering of stacktrace locations when finding vulnerabilities. After this change, vulnerabilities that were previously discarded will now be reported. In particular, if they were found within code in site-packages or outside of the working directory.
LLM Observability: Resolves an issue where spans and evaluation metrics were not being sent via Unix sockets.
dynamic instrumentation: prevent an exception when trying to remove a probe that did not resolve to a valid source code location.
kafka: This fix resolves an issue where message headers were sent to Kafka brokers that do not support them. Message headers are turned off when the Kafka server responds with UNKNOWN_SERVER_ERROR (-1).
code origin for spans: fixes a performance issue with exit spans.
profiling: improve performance of the memory profiler for large heaps. The memory profiler previously did a linear search of tracked allocations for every free, which scaled very poorly with large heaps. Switch to a fast hash map.
unicode string is too large error.Code Security
Lib-injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.Tracing
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
Lib-Injection
LLM Observability
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (likeopen) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
pylibmc: Fixes an issue where using Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'Lib-Injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.echion which includes performance improvements.patch_all is deprecated. As an alternative to patch_all, you can use import ddtrace.auto along with DD_PATCH_MODULES if specific module patching is necessary.Dynamic Instrumentation
LLM Observability
integration tag to LLM Observability spans that are generated by an integration.agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED now defaults to None, and will be determined automatically based on the presence of a compatible Datadog agent. agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED can still be set explicitly to True or False</span>, but is now optional for agentless users.Profiling
Tracing
DD_TRACE_NATIVE_SPAN_EVENTS (disabled by default). This requires agent version 7.63.0 or later.http.referrer_hostname tag.Fixes an issue with gevent support and the typing module on CPython >= 3.12.
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
Lib-Injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.openai: Avoids creating spans for streamed Open AI chat and completion requests that use with_raw_response since the stream cannot be traced in these cases, leading to unfinished spans.Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Single-Step Instrumentation
Tracing
graphql: Fixes an issue in GraphQL patching that threw an error when locations (an Optional field) is None. This checks for locations before setting attributes.ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
LLM Observability
Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'ASM
CI Visibility
pytest-cov from being used.LLM Observability
Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
record_exception call as it was not following OTEL semantics.boto3 from the minimum versions list to avoid blocking SSI on boto3 version.Includes <span class="title-ref">dynamodb</span> in the default list of cloud services for payload tagging, i.e. <span class="title-ref">DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES</span> Note that cloud services payload tagging feature is still gated by <span class="title-ref">DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING</span> and <span class="title-ref">DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING</span>.
LLM Observability: This introduces tracing for converse calls to the Bedrock Converse API. converse calls are traced as
LLM spans with <span class="title-ref">max_tokens</span>, <span class="title-ref">temperature</span> inference parameters and input/output messages.
botocore: This introduces capturing prompts, token usage, and inference parameters for converse calls to the Bedrock API.
SCA: This add support for extended heartbeat every 24h with dependencies payload.
kafka: Adds messaging.destination.name tag to kafka produce and consume spans.
LLM Observability: Avoids submitting spans to LLM Observability when a non-default base URL is detected in request for the Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI integrations.
ASM: This introduces support for automatic instrumentation of session monitoring and blocking for Django.
ASM: This introduces new metrics span tags to keep track of truncations of WAF payloads.
runtime_metrics: Adds support for sending runtime metrics as gauge metrics (instead of distributions). To enable this feature set DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED=DD_RUNTIME_METRICS_ENABLED.
runtime_metrics: Adds support for tagging runtime metrics with the current runtime ID. To enable tagging, set DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED=True.
SCA: This fix resolves an issue where some dependencies where reported with an inaccurate name.
profiling: This fix resolves an issue where the Lock profiler would throw an AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.
tags: Updates <span class="title-ref">DD_TAGS</span> parsing to match the Datadog Agent and other SDKs. Key-value pairs can now be separated by both commas and spaces (e.g., <span class="title-ref">DD_TAGS="key1:value1, key2:value2, key3:value3"</span>). This reverts a change introduced in v2.11.0, which only allowed one separator type.
dynamic sampling: Ensures that dynamic sampling rates are always consistent with the rates received from the agent via remote configuration. Previously, setting a global sample rate and then sampling rules via remote config would result in both configurations being applied. With this fix only the most recent sampling configuration will be applied.
LLM Observability: This fix resolves an issue where enabling LLM Observability caused all botocore submodules to be patched instead
of only bedrock-runtime.
LLM Observability: Resolves issue where Pydantic model objects were encoded as unhelpful default placeholder text.
LLM Observability: Improves encoding of non-JSON serializable I/O objects by attempting to convert to string before resorting to default placeholder text.
tracing: Resolves an issue where trace information, such as span links, baggage, and trace-level tags (ex: sampling decision maker), could be lost when a new process was created while a trace was active.
internal: Fixes an issue where trimming a traceback to attach it to the span could result in the loss of the most recent frames.
pylibmc: fixes an issue where using Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'
tracing: Ensures remote sampling rules no longer overwrite existing samplers, preserving service-based sample rates set by the Agent and ensuring tracer rate limits configured by ddtrace products (e.g., ASM) are always respected.
tracing: Ensures a single DatadogSampler is initialized per Tracer instance, improving Tracer object initialization speed by approximately 7% and simplifying sampling logic.
agent: Migrates agent connection configurations to envier. This provides better documentation and validation for expected types. The following configuration are affected:
- <span class="title-ref">DD_TRACE_AGENT_TIMEOUT_SECONDS</span> Of the above impacted configurations, these had a precedence change:
DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED with DD_APM_TRACING_ENABLED. Feature set remains the same.API Gateway: Add preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
LLM Observability: Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
tracing: Introduces a record_exception method that adds an exception to a Span as a span event. Refer to [Span.record_exception](https://ddtrace.readthedocs.io/en/stable/api.html#ddtrace.trace.Span.record_exception) for more details.
LLM Observability: Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
LLM Observability: Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
Symbol Database is now enabled by default.
graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
graphql: Adds user-provided extensions to graphql error span events through the <span class="title-ref">DD_TRACE_GRAPHQL_ERROR_EXTENSIONS</span> environment variable. This is disabled by default; when set it allows users to capture their extensions.
LLM Observability: This introduces <span class="title-ref">metadata</span> argument to <span class="title-ref">LLMObs.submit_evaluation_for</span>, which enables users to
attach arbitrary structured metadata to custom evaluations.
bytecode>=0.15.1 for Python 3.12 to avoid known bug.boto3 from the minimum versions list to avoid blocking SSI on boto3 version.ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.ddtrace internal logger initialization mutating an unlocked shared resource.We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.
The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the versioning policy for support level definitions.
item.nodeid.TestClass::test_method) for class-based tests.DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable is no longer used.DD_LLMOBS_APP_NAME.parameters argument in LLMObs.annotate() has been removed. Use metadata instead.DD_TRACE_PROPAGATION_STYLE=b3 single header is no longer supported. Use b3 instead.ddtrace.tracer must be used.Tracer.configure(...).Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (2.21.0):
Enable warnings as errors in pytest:
pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.pySet the following environment variable to raise exceptions for deprecated usage:
DD_TRACE_RAISE_DEPRECATIONWARNING=1 ddtrace-run python app.pyAlternatively, enable all deprecation warnings:
PYTHONWARNINGS=all python app.pyThe following environment variables have been removed or replaced in v3. Use the table below to update your configuration:
| v2 | Removed in v3 | Replaced/Changed in v3 |
|---|---|---|
| LLM Observability | ||
DD_LLMOBS_APP_NAME | Use DD_LLMOBS_ML_APP instead | |
_DD_LLMOBS_EVALUATOR_SAMPLING_RULES | Use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead | |
_DD_LLMOBS_EVALUATORS | Use DD_LLMOBS_EVALUATORS instead | |
| Tracing | ||
DD_ANALYTICS_ENABLED | No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail. | |
DD_HTTP_CLIENT_TAG_QUERY_STRING | Use DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING. | |
DD_TRACE_ANALYTICS_ENABLED | No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail. | |
DD_TRACE_METHODS (using [] notation) | Must use : notation instead, For example: mod.submod:method1,method2;mod.submod:Class.method1. (docs) | |
DD_TRACE_PROPAGATION_STYLE=”b3 single header” | Use DD_TRACE_PROPAGATION_STYLE=b3 for identical behavior |
Use the following patterns to identify the deprecated environment variables in a code base:
git grep -P -e "DD_LLMOBS_APP_NAME" \
-e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
-e "_DD_LLMOBS_EVALUATORS" \
-e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
-e "DD_TRACE_SAMPLE_RATE" \
-e "DD_TRACE_API_VERSION=v0.3" \
-e "DD_ANALYTICS_ENABLED" \
-e "DD_TRACE_ANALYTICS_ENABLED" \
-e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
-e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
-e "DD_TRACE_METHODS=.*\[\]"Replace deprecated settings with the recommended alternatives where applicable.
NOTE: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.
The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.
| v2 | v3 |
|---|---|
| Tracing | |
Tracer.configure( enabled=foo, hostname=foo, port=foo, uds_path=foo, https=foo, sampler=foo, wrap_executor=foo, priority_sampling=foo, settings=foo, dogstatsd_url=foo, writer=foo, partial_flush_enabled=foo, partial_flush_min_spans=foo, api_version=foo ) | Tracer.configure() |
ddtrace.opentracer.tracer.Tracer()._dd_tracer | ddtrace.tracer |
ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True) | ddtrace.tracer.sampler.rules[].choose_matcher(“foo”) Callables and regex patterns are no longer allowed as arguments to choose_matcher |
Span.sampled | Span.context.sampling_priority > 0 |
| LLM Observability | |
LLMObs.annotate(parameters=foo) | LLMObs.annotate(metadata=foo) |
Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.
dd-trace-py doesn’t work on Windows with Python 3.13anthropicconsulfreezegungeventgoogle_generativeaigunicornlangchainmysqlclientopentracingpsycopgpsycopg2pymysqlpytestpytest-bddpytest-benchmarksanicseleniumsqlite3starlettetornadovertexaiBy following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!
Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.
ASM
Code Security (IAST)
LLM Observability
valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!
openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.
Introduces an integration with the RAGAS evaluation framework to continuously monitor the performance of context-augmented LLM generations in production. The integration supports evaluating LLM inferences with the following RAGAS metrics:
To learn more, see the LLM Observability evaluations guide.
ddtrace.patch(...) is called with an invalid argument.DD_TRACE_SAMPLE_RATE | Use DD_TRACE_SAMPLING_RULES (docs). |
DD_TRACE_SPAN_AGGREGATOR_RLOCK | No replacement (feature removed). Config variable is now a no-op. |
| CI Visibility |
DD_PYTEST_USE_NEW_PLUGIN_BETA | No replacement (new pytest plugin is not in beta anymore). |