LLM spans now export over APM trace; AI Guard evaluates Anthropic; Pub/Sub gets DSM

Upgrade Notes

• flask: Requests served under a non-empty WSGI SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.

<!-- -->

• flask: API endpoint discovery now reports every HTTP method the framework serves — including Werkzeug's auto-added HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].

<!-- -->

• LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.

New Features

• tracing: collect the x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.

<!-- -->

• Database Monitoring (DBM) propagation supports dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.

<!-- -->

• ai_guard: add AI Guard evaluation support to the Anthropic SDK Messages instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client (covering Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.

<!-- -->

• aiokafka: Adds kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.

<!-- -->

• ASM: This introduces the _dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.

<!-- -->

• ASM: Extends _dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).

<!-- -->

• LLM Observability: The claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.

<!-- -->

• aws_durable_execution_sdk_python: Adds distributed tracing for durable workflows across suspend/resume cycles, so a workflow that pauses and resumes in a later invocation appears as a single connected trace. See aws_durable_execution_sdk_python for details and opt-out configuration.

<!-- -->

• AAP: API endpoint discovery now covers Flask sub-applications mounted via werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.

<!-- -->

• google_cloud_pubsub: This introduces Data Streams Monitoring (DSM) context propagation for the Google Cloud Pub/Sub integration. Producer publish operations inject the DSM pathway context into message attributes, and subscriber callbacks extract it and record a consume checkpoint. To enable, set DD_DATA_STREAMS_ENABLED=true.

<!-- -->

• LLM Observability: automatically tags spans and experiments with git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.

<!-- -->

• LLM Observability: LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.

<!-- -->

• LLM Observability: experiment evaluators and summary evaluators can now return a MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.

<!-- -->

• LLM Observability: Add LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.

<!-- -->

• LLM Observability: task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.

<!-- -->

• LLM Observability: Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.

<!-- -->

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.

<!-- -->

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.

<!-- -->

• profiling: The fast stack profiler optimisation is now enabled by default.

<!-- -->

• Profiling: The heap profiler can now track allocations made through PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.

<!-- -->

• ray: Added DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.

<!-- -->

• This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence, DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.

<!-- -->

• CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).

  Example:

  from pydantic_evals.evaluators import EqualsExpected import logging

  from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter

  handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)

  while True:
  job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)

  handler.close()

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.

<!-- -->

• LLM Observability: Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).

<!-- -->

• aiokafka: Collects the Kafka cluster ID when tracing aiokafka producers and consumers and forwards it to Data Streams Monitoring, matching the behavior of the confluent_kafka integration.

<!-- -->

• aap: Adds instrumentation telemetry distributions for WAF and RASP execution durations, including waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.

<!-- -->

• AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.

<!-- -->

• azure_cosmos: This fix resolves an issue where cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.

<!-- -->

• CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.

<!-- -->

• database monitoring: Resolved a memory leak that grew with query volume when DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.

<!-- -->

• tracing: Exclude wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.

<!-- -->

• DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.

<!-- -->

• Fixed a bug in the kombu integration where programmatically setting config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.

<!-- -->

• falcon: Fixes an issue where a None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.

<!-- -->

• ASM: Fixes body inspection silently skipping requests whose Content-Type header carried a parameter (most commonly charset=utf-8).

<!-- -->

• tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.

<!-- -->

• botocore: Fixes a bug in Data Streams Monitoring where the pathway context of an SQS message was not extracted when the message body was valid JSON, silently breaking the data stream for non-SNS messages with a JSON payload. The _datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.

<!-- -->

• Fixed an issue that could have caused some instrumented code to fail to execute correctly when the original function had keyword arguments passed in as a cell variable.

<!-- -->

• celery: Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.

<!-- -->

• CI Visibility: Truncates string meta tag values to 5000 characters when encoding test-cycle payloads, preventing rejected payloads caused by tag values that exceed the backend maximum length.

<!-- -->

• LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where a span's error message showed an uncategorized unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.

<!-- -->

• LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where sequential ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.

<!-- -->

• dynamic instrumentation: Fixes debugger argument/local captures misclassifying keyword-only arguments and dropping *args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.

<!-- -->

• Django: This fix resolves an issue where async function-based middleware attributes, such as process_exception, were not preserved, causing Django to skip exception handlers under ASGI.

<!-- -->

• tracing: This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.

<!-- -->

• tracing: Fixes a race condition where extra service names could be silently dropped from Remote Configuration /v0.7/config payloads in multi-threaded applications (e.g. uWSGI).

<!-- -->

• tracing: This fix resolves a rare issue that could cause an application to hang after forking a child process.

<!-- -->

• IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted data. A concurrent or still-open request holding a tainted value could leak its taint into the current request's checks. Queries are now scoped to the calling request's slot.

<!-- -->

• langchain, botocore: This fix resolves an issue where auto-instrumented langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.

<!-- -->

• LLM Observability: Resolves an issue in the openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.

<!-- -->

• LLM Observability: Fixes an issue where reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.

<!-- -->

• LLM Observability: Fixes an issue where the span writer ignored its configured request timeout (_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.

<!-- -->

• profiling: An issue where monkey patching in gevent silently drops all lock profiling samples, caused by gevent monkey-patching threading lock primitives in place after the profiler had already patched them.

<!-- -->

• ASM: Fixes multipart body parts whose Content-Type header carried parameters being dropped.

<!-- -->

• Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first execution.

<!-- -->

• internal: This fix resolves an issue where calling awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.

<!-- -->

• internal: Fix a memory leak where reference cycles through PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.

<!-- -->

• profiling: Fixes a memory leak in native frame tracking caused by unbounded native call-site metadata growth.

<!-- -->

• pydantic_ai: Fixes APM span naming so the operation name is the generic category (pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.

<!-- -->

• runtime metrics: Fixes an issue where runtime metrics were missing container and orchestrator tags (such as pod_name) on hosts using cgroup v2.

<!-- -->

• SCA: This fix resolves an issue where unresolved runtime reachability targets could accumulate across Software Composition Analysis updates, causing resident memory usage to grow over time.

<!-- -->

• tracing: Fixes the subprocess integration silently spans for any shell command whose env-var assignment contained = in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.

<!-- -->

• dynamic instrumentation: fixes an issue where the Symbol Database uploader sends empty payloads on a recurring timer.

<!-- -->

• tracing: Applying @tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.

<!-- -->

• code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.

<!-- -->

• CI Visibility: Fixes spurious git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.

<!-- -->

• Flare: Fixes a bug where the live tracer configuration was mutated when redacting the API key before writing the flare diagnostic file.

<!-- -->

• core: This fix prevents periodic thread restarts from blocking application code from resuming in forked children.

<!-- -->

• LLM Observability: Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.

<!-- -->

• IAST: A crash that could happen at interpreter teardown has been fixed.

<!-- -->

• tracing: A rare crash happening on versions of CPython prior to 3.12 has been fixed.

<!-- -->

• LLM Observability: Resolves an issue where spans annotated with metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.

<!-- -->

• LLM Observability: Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as noise in LLM span metadata.

<!-- -->

• LLM Observability: Fixed an issue where _current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.

<!-- -->

• LLM Observability: Fixes an issue where document content on embedding and retrieval spans bypassed the span processor. LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.

<!-- -->

• LLM Observability: Fixes provider mis-attribution on openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.

<!-- -->

• Profiling: This fixes an issue where thread pool worker spans were not correctly being linked to their profiles

<!-- -->

• profiling: A bug where the Profiler could crash when Python's fault handler utility was enabled has been fixed.

<!-- -->

• profiling: fixed an issue where the stack sampler over-counted total CPU time for gevent workloads.

<!-- -->

• profiling: A rare crash that could happen after fork in fork-based applications has been fixed.

<!-- -->

• profiling: The Profiler now flushes the last profile on SIGINT and SIGTERM signals.

<!-- -->

• protobuf: This fix resolves an issue where Data Streams Monitoring schema tags were missing from protobuf spans due to incorrect patching of generated message classes.

<!-- -->

• CI Visibility: This fix resolves an issue where quarantined tests were not retried by Auto Test Retry (ATR).

<!-- -->

• tracing: Fixes a bug where running ddtrace-run caused a traceback on keyboard interrupt.

<!-- -->

• kafka: Fixes a crash in the confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.

Other Changes

• opentelemetry: Adds OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.

<!-- -->

• LLM Observability: when LLMObs is enabled in agentless mode (Datadog Agent not reachable or with DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.

<!-- -->

• profiling: Each uploaded profile now exposes the effective profiler configuration under info.profiler.settings on the upload event, so profiles can be filtered by individual settings.

<!-- -->

• ray: Cache host name and Ray runtime invariants (gcs_address, namespace, version, etc.) at first access. Per-call lookups (socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.