APM

v4.9.0

AI Guard blocks now bypass generic exception handlers; AWS Durable Execution traced

Upgrade Notes

AI Guard: ddtrace.appsec.ai_guard.AIGuardAbortError now derives from ddtrace.internal._exceptions.DDBlockException (a BaseException subclass) instead of Exception. This brings AI Guard block decisions in line with how ASM blocks are surfaced and prevents a generic except Exception: in user code from silently swallowing a block.

settings: Legacy environment variable names registered as aliases in the configuration registry now also work when set via local or fleet stable config files, not just shell environment variables. #17958

ray: Adds DD_TRACE_RAY_IGNORED_ACTORS configuration to exclude specific Ray actor methods from instrumentation. Set DD_TRACE_RAY_IGNORED_ACTORS='{"ActorA": ["method1"], "ActorB": "*"}' to leave matching methods or actors uninstrumented while continuing to trace other Ray actor methods. Matching is based on actor class name only.

Deprecation Notes

Tracing: DD_TRACE_INFERRED_SPANS_ENABLED is deprecated and will be removed in 5.0.0. Use DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED instead. The old environment variable continues to work but emits a DDTraceDeprecationWarning when set.

New Features

aws_durable_execution_sdk_python: Add tracing support for the aws-durable-execution-sdk-python library. Instruments @durable_execution workflows and DurableContext operations (step, invoke, wait, wait_for_condition, wait_for_callback, create_callback, map, parallel, run_in_child_context) to generate spans.

LLM Observability: Adds step spans to the Claude Agent SDK integration. Each inference cycle is now represented by a step container span with an llm child span for the model call and tool child spans for any tool invocations.

tracing: Adds a centralized supported-configurations.json registry of all supported DD_* and OTEL_* environment variables, following the same schema used by other Datadog tracing libraries. Accesses to unregistered environment variables now produce a debug log to help identify typos or unsupported configuration options.

AI Guard: Copies anomaly-detection attributes from the local root (service-entry) span onto every ai_guard span: ai_guard.http.useragent, ai_guard.http.client_ip, ai_guard.network.client.ip, ai_guard.usr.id and ai_guard.usr.session_id.

AI Guard: When DD_AI_GUARD_ENABLED=true is set and an ai_guard span is created during a request, the tracer now populates http.client_ip and network.client.ip on the service-entry (local root) span, mirroring the behavior used for Application Security. If AI Guard does not run during the request, no client IP tags are added. DD_TRACE_CLIENT_IP_ENABLED is ignored once AI Guard reports, and DD_TRACE_CLIENT_IP_HEADER continues to override header resolution.

ai_guard: add AI Guard evaluation support to the OpenAI SDK chat completions instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client, and evaluation is automatically skipped when a framework integration (LangChain, Strands Agents) is already evaluating the same call.

code origin for spans: The code origin for spans feature has been enabled by default.

code origin: attach code origin information to the first span generated by a function wrapped with tracer.wrap.

openfeature: This introduces a configurable initialization timeout for DataDogProvider. The timeout controls how long initialize() waits for configuration before returning, and defaults to 10 seconds. Set it via the DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS environment variable or the init_timeout constructor parameter.

CI Visibility: This introduces Jenkins custom parent ID propagation, which enables Datadog to correlate tests run from Jenkins with their Jenkins jobs and pipelines.

LLM Observability: Adds an optional cost_tags argument to LLMObs.annotate() and LLMObs.annotation_context(). Pass a list of tag keys (already set via tags or annotated previously on the same span) to have them attached to the cost and token metrics generated from LLM and embedding spans, which can help breaking down spend by team, project, org, or any custom dimension.

LLM Observability: Adds support for an optional version (string) field on each tool definition dictionary passed to LLMObs.annotate() via the tool_definitions parameter.

profiling: Add DD_PROFILING_LOCK_EXCLUDE_MODULES config to skip lock profiling for framework-internal locks. Excluded locks remain native with zero profiling overhead. Set it to a comma-separated list of module prefixes (e.g., django.db,sqlalchemy.pool,urllib3).

LLM Observability: Bedrock Agent orchestration step events (model invocations, tool/action group calls, knowledge base lookups, guardrails, rationales) are now emitted as APM child spans of the Bedrock Agent <agent_id> span when LLM Observability is enabled, with the same LLMObs payload shape as before.

Bug Fixes

tracing: Exclude wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.
ai_guard: This fix resolves a conflict between ddtrace.auto and strands when imported in the same file, which left Strands hooks silently disabled. The Strands integration now loads lazily on first attribute access so its event class identities match those the agent dispatches.

appsec: Adds telemetry metrics instrum.user_auth.missing_user_login and instrum.user_auth.missing_user_id when Django auth events cannot resolve the expected identity fields, enabling detection of misconfigured user model field mappings.

AAP: This fix resolves an issue where the usr.session_id tag was missing from the entry span of authenticated follow-up Django requests when automatic user instrumentation was enabled. They now also carry usr.session_id, matching other authenticated user-tagging paths.

azure_cosmos: This change removes the http.status_code tags from Azure CosmosDB spans and replaces them with the use of the db.response.status_code metric. For customers using ddtrace v4.8.0 and relying on the http.status_code tag of cosmosdb.query spans, this is a breaking change.

CI Visibility: Fixes an issue in the pytest plugin where a malformed log call emitted a --- Logging error --- traceback to stderr during Attempt to Fix retries, polluting pytest output and contributing to spurious test failures.

CI Visibility: Fixes an IndexError in retry bookkeeping that occurred when a test's teardown phase failed. The error produced --- Logging error --- tracebacks in stderr, which could pollute test output and cause spurious test failures during retries. #17863

CI Visibility: Fixes a regression where setting DD_TEST_MANAGEMENT_ENABLED=0 was not honored by the new pytest plugin, causing Test Management features such as quarantining, disabling tests, and Attempt to Fix to remain enabled.

CI Visibility: Fixes code coverage instrumentation on Python 3.13, 3.14, and 3.15. Resolves lost per-test line data caused by: sys.monitoring callbacks running in a snapshot context where ContextVar changes are not visible (Python 3.14+); empty modules emitting no LINE events (Python 3.13+); and ProcessPoolExecutor child coverage not being propagated to the parent context. Also fixes a stale-data bug where child process executable lines could inflate coverage denominators after stop_coverage() was called before join().

datastreams: Demotes the retry limit exceeded submitting pathway stats log from ERROR to WARNING and removes the multi-line traceback from the record. This message fires when the processor cannot reach the agent within its 1-second timeout; the dropped 10 seconds of DSM data is auto-recovered on the next flush.

LLM Observability: Fixes a concurrency bug in the Bedrock Agent integration where concurrent invoke_agent calls could orphan or cross-attribute spans due to shared class-level state. Per-invocation state is now used.

LLM Observability: This fix resolves an issue where text wrapped in Bedrock Converse guardContent content blocks was rendered as [Unsupported content type: guardContent] in traces, dropping the user's input.

Fixed an issue that could have caused some instrumented code to fail to execute correctly when the original function had keyword arguments passed in as a cell variable.

CI Visibility: Fixes an issue where tests marked as attempt-to-fix could have failures hidden when they were also quarantined or disabled.

django: Stop tagging async view and middleware spans as errored on routine ASGI cancellations (e.g. client disconnects on streaming responses), a regression introduced in 4.8.0rc4. Cancellation still propagates; the span just finishes without error.type='asyncio.exceptions.CancelledError'.

django: Fixes DD_DJANGO_DATABASE_SERVICE and DD_DJANGO_DATABASE_SERVICE_NAME, which were previously generated as DD_DJANGO-DATABASE_SERVICE and DD_DJANGO-DATABASE_SERVICE_NAME. The hyphenated names were invalid POSIX identifiers and unusable from most shells. Hyphens in integration names are now normalized to underscores when building env var names. The old hyphenated names are preserved as aliases for backward compatibility. #17952

django: API endpoint discovery now supports Django sub-applications mounted with django.urls.include(...). Endpoints are reported with their full URL path including the parent prefix — for example, a view served at /api/users/ is now reported as /api/users/ instead of losing the /api/ prefix.

django: API endpoint discovery now reports the correct HTTP methods for views decorated with @require_http_methods combined with another decorator such as @csrf_exempt; the declared methods are reported instead of a generic wildcard entry.

telemetry: tolerate malformed installed distribution metadata so a single bad dist-info entry no longer floods stderr with repeated tracebacks.

langchain, botocore: This fix resolves an issue where auto-instrumented langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.

LLM Observability: This fix resolves an issue where running an experiment with a dataset whose records had null metadata caused the summary evaluator to crash with a TypeError while preparing evaluator inputs.

LLM Observability: Changes the default model_name and model_provider of LLM and embedding spans from custom to unknown if not provided or empty. This applies to both auto-instrumented spans and manual instrumentation via LLMObs.llm() / LLMObs.embedding() and the @llm / @embedding decorators.

profiling: Fixes an issue where the lock profiler silently stopped capturing lock events when running under ddtrace-run with gevent installed.

LLM Observability: The OpenAI integration now preserves assistant message content when tool_calls are present on the same message. #17760

openfeature: This fix resolves an issue where DataDogProvider.initialize() returned before configuration was received, causing the OpenFeature SDK to mark the provider as ready to serve evaluations too early and flag evaluations to silently return default values. The provider now waits for configuration before returning.

openfeature: Fixes targeting key handling in the OpenFeature provider. None targeting key is now correctly passed to the native evaluator instead of being coerced to empty string. Flags that don't require a targeting key (static, rule-based) now evaluate successfully without one, matching the Datadog provider spec. Additionally, the Rust binding now correctly maps TargetingKeyMissing errors from libdatadog instead of returning a generic error code.

tracing: Fixes an issue where the svc.auto process tag produced garbled values such as python_-m_unittest when a process was launched with the full command as a single sys.argv[0] string (e.g. from a Docker ENTRYPOINT, a process manager, or a subprocess call with an unsplit command). The correct module or script name is now extracted in these cases. #17764

profiling: A crash that could happen in child processes after fork has been fixed.

profiling: A rare crash caused by the memory allocation profiler has been fixed.

RemoteConfig: Fixed an issue where deleted remote configurations were not applied, causing stale settings to persist.

tracing: This fix resolves a memory leak where reference cycles through a span's properties were invisible to Python's cyclic garbage collector and accumulated proportionally to traced call volume.

starlette: This fix resolves an issue where passing middleware=None caused application startup to fail when Starlette tracing was enabled.

wsgi: This fix resolves an issue where the http.url tag on inbound request spans contained the WSGI mount prefix twice (for example /admin/admin/users instead of /admin/users) when the application was served behind werkzeug.middleware.dispatcher.DispatcherMiddleware or any other in-process mount that preserves the original request URI in RAW_URI / REQUEST_URI while also setting SCRIPT_NAME.

tracing: A crash that occurred when exiting a gevent application with DD_TRACE_DEBUG=1 has been fixed.

langchain: Strips interface identifiers (e.g. chat, llm) and path prefixes (e.g. models/) when extracting the model_provider and model_name, so reported values identify the actual provider and model name rather than the LangChain interface or API resource path.

llmobs: fixes child spans created within an experiment task not inheriting the dataset_id tag. Previously only dataset_name was propagated via baggage to child spans; dataset_id is now propagated as well, making dataset, project, and experiment context (name and ID) consistent across all spans in an experiment trace.

Profiling: This fixes a bug where uploaded profiles would not have a linked span post fork

profiling: A rare crash happening on systems with small stack sizes when profiling asyncio code has been fixed.

propagation: Limits parsing of the W3C tracestate header during tracecontext extraction to 32 list-members and 512 UTF-8 bytes, consistent with the W3C Trace Context specification (https://www.w3.org/TR/trace-context/). Extra list-members and trailing whole entries that would exceed the byte budget are ignored, so unusually large headers no longer expand unbounded work during extraction. The Datadog dd= list-member is preferred: it is kept when present (including when it appears late in the header or alone exceeds the byte cap), and other vendors are dropped first. List-members longer than DD_TRACE_TRACESTATE_ITEM_MAX_CHARS (128) characters are removed first when trimming by list-member count or byte budget, so shorter vendor entries are kept when possible.

Fixed a startup deadlock when using snowflake-connector-python >= 4.4.0 with DD_TRACE_SNOWFLAKE_ENABLED=true.

tracing: This change fixes an issue in which svc_src is set to m in cases where service matches the _default_service of an active integration config. In such cases, the intended behavior is that it svc_src is equal to service. #17712

tracing: Parsing incoming baggage HTTP headers now respects DD_TRACE_BAGGAGE_MAX_ITEMS [default 64] and DD_TRACE_BAGGAGE_MAX_BYTES [default 8192], consistent with baggage injection. Previously, extraction could retain every comma-separated entry regardless of those limits. The tracer drops excess pairs and records truncation telemetry when limits apply.

Other Changes

profiling: The ECHION_ALT_VM_READ_FORCE configuration flag has been removed and support for the associated feature has been dropped.

May 22, 2026

v4.8.7

Wrapt 2.2.0 excluded; agent connection logs fixed

Bug Fixes

tracing: Exclude wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.
LLM Observability: Agent connection error logs are no longer logged when using agentless and not running a Datadog agent.

May 20, 2026

v5.104.0

Undici CONNECT span no longer unfinished; metrics delta temporality fixed

[f7f2c7c2f1] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 2 directories with 2 updates (dependabot[bot]) #8563
[4afcb7eed4] - (SEMVER-PATCH) fix(undici) unfinished CONNECT span (Pablo Erhard) #8558
[a34acbb466] - (SEMVER-PATCH) test(aerospike,couchbase): fix hook timeouts and couchbase service readiness (Roch Devost) #8550
[908e4d7558] - (SEMVER-PATCH) perf: reduce redis and ioredis bindStart overhead (Ruben Bridgewater) #8494
[85211ec3ec] - (SEMVER-PATCH) fix(metrics): fix delta temporality not being applied to OTLP counter exports (Sameeran Kunche) #8556
[32b49b2a0d] - (SEMVER-PATCH) chore(deps): bump openai (dependabot[bot]) #8539
[0bc47a6727] - (SEMVER-PATCH) perf(graphql): skip stacktrace symbolication on validation-only errors (Ruben Bridgewater) #8499
[8feb9ce389] - (SEMVER-PATCH) fix(types): import opentracing from the consumer's installed package (Ruben Bridgewater) #8525
[31353c773c] - (SEMVER-PATCH) ci: remove systematic Mocha --retries 1 on master (Roch Devost) #8552
[e769f32bb0] - (SEMVER-PATCH) fix(google-cloud-pubsub): poll for GC collection instead of fixed retry count (Roch Devost) #8553
[180549f994] - (SEMVER-PATCH) fix(mongoose): make PeerCat model registration idempotent (Roch Devost) #8551
[d7e1a534e9] - (SEMVER-PATCH) fix(aws-sdk): fix flaky SFN, SNS DSM, S3, and Bedrockruntime tests (Roch Devost) #8548
[1c8044f42a] - (SEMVER-PATCH) fix(appsec): increase beforeEach timeout in rasp-metrics integration test (Roch Devost) #8547
[e6eac5a6f0] - (SEMVER-PATCH) fix(electron): fix flaky IPC tests and Xvfb startup race (Roch Devost) #8546
[43f95f074d] - (SEMVER-PATCH) chore(ci): update dd-sts-action to v1.0.1 (Charles de Beauchesne) #8545
[d5421208ad] - (SEMVER-PATCH) ci(electron): push results to test optimization (Roch Devost) #8544
[65a9329653] - (SEMVER-PATCH) chore(deps): bump ws from 8.20.0 to 8.20.1 in /benchmark/sirun/startup/everything-fixture in the npm_and_yarn group across 1 directory (dependabot[bot]) #8542
[178fc6bd2a] - (SEMVER-PATCH) chore(ci): set ffe as owner of .github/workflows/openfeature.yml (Charles de Beauchesne) #8535
[25d47abf77] - (SEMVER-PATCH) test only Node LTS (Pablo Erhard) #8529
[a01a6b8efd] - (SEMVER-PATCH) test: prefer assert.match for string matching (Thomas Watson) #8540
[49f1bb171a] - (SEMVER-PATCH) test(electron): raise plugin-spec startApp beforeEach timeout to 10s (Ruben Bridgewater) #8536
[dcfdb77b29] - (SEMVER-PATCH) test(debugger): widen sampling spec tolerance to absorb wall-clock skew (Thomas Watson) #8534
[7302ef8258] - (SEMVER-PATCH) test: use assert.strictEqual / notStrictEqual for === and !== checks (Thomas Watson) #8538
[f74834adaa] - (SEMVER-PATCH) [test optimization] Fix early metadata encoding bug (Juan Antonio Fernández de Alba) #8528
[18d97575dc] - (SEMVER-PATCH) perf(id): memoise Identifier.toBigInt instead of re-decoding the buffer per call (Ruben Bridgewater) #8495
[54f0666bbb] - (SEMVER-PATCH) fix(rewriter): emit valid ESM imports for instrumented modules (Ruben Bridgewater) #8526
[7b5b32caca] - (SEMVER-PATCH) fix(kafkajs): skip header injection on brokers below Produce v3 (Ruben Bridgewater) #8390
[ab11acdbb6] - (SEMVER-PATCH) test: reduce agent.load overhead by reducing proxyquire calls (Ruben Bridgewater) #8099
[85baecf3cd] - (SEMVER-PATCH) perf(elasticsearch): cache per-instance tag keys and skip empty params stringify (Ruben Bridgewater) #8376
[5d72f2d380] - (SEMVER-PATCH) refactor(graphql): drop unused @graphql-tools/executor deep hook (Ruben Bridgewater) #8510
[c1eddead4b] - (SEMVER-PATCH) ci(codecov): restrict codecov/patch to PRs targeting master (Ruben Bridgewater) #8464
[9952e24b41] - (SEMVER-PATCH) chore(llmobs): fix flaky openai test (Sam Brenner) #8490
[f8614b357f] - (SEMVER-PATCH) feat(shimmer): add wrapCallback for tracer-controlled closures (Ruben Bridgewater) #8503
[3631da6a36] - (SEMVER-MINOR) feat(appsec): Always report content-length and content-type response headers when appsec is enabled (Carles Capell) #8482
[858e1cb0a2] - (SEMVER-PATCH) perf(storage): cache storage() at module load (Ruben Bridgewater) #8500
[bcab75cd29] - (SEMVER-MINOR) feat(crashtracking): capture and report uncaught exceptions (Gyuheon Oh) #8176
[28c1ff37b8] - (SEMVER-PATCH) perf(service-naming): short-circuit repeat registerExtraService calls (Ruben Bridgewater) #8496
[bdc736b843] - (SEMVER-PATCH) [test optimization] parallelize test optimization requests (Juan Antonio Fernández de Alba) #8520
[29cd9f5967] - (SEMVER-PATCH) [test optimization] use worker efd retry count in parallel cucumber (Juan Antonio Fernández de Alba) #8478
[9f243d934c] - (SEMVER-PATCH) [test optimization] merge static mocked files with recorded mocks in jest (Juan Antonio Fernández de Alba) #8477
[19dfbe4dbe] - (SEMVER-PATCH) test: targeted aerospike and electron suite timeouts (Ruben Bridgewater) #8527
[30cbcacac3] - (SEMVER-MINOR) feat(appsec): collect Datadog security-testing headers on entry spans (Christophe Papazian) #8463
[16ab502164] - (SEMVER-PATCH) [test optimization] stabilize vitest async setup instrumentation (Juan Antonio Fernández de Alba) #8519
[c497d0a4ca] - (SEMVER-PATCH) fix(mocha): correct timing for top-level it() tests (Sebastián Kay) #8480
[806f177fbf] - (SEMVER-MINOR) feat(mongodb): add opt-in query value obfuscation (Ruben Bridgewater) #8401
[f9a39f203d] - (SEMVER-PATCH) [test optimization] Stabilize Cypress intake payload waits (Juan Antonio Fernández de Alba) #8522
[f221629d3b] - (SEMVER-PATCH) chore(deps): bump axios (dependabot[bot]) #8518
[539aa4a662] - (SEMVER-PATCH) fix(grpc): handle client_streaming and bidi 3-arg overload (Crystal Luc-Magloire) #8276
[1eda439d60] - (SEMVER-PATCH) fix(openfeature): cap pending exposure events (Ruben Bridgewater) #8162
[9e41a6aae3] - (SEMVER-PATCH) deprecation(llmobs): deprecate llmobs.enable and llmobs.disable for dd-trace v7 (Sam Brenner) #8427
[934f932039] - (SEMVER-PATCH) ci(appsec): split next jobs by individual version instead of range (Roch Devost) #8472
[c4b0554b7d] - (SEMVER-PATCH) chore(deps): bump express (dependabot[bot]) #8488
[7a01a98566] - (SEMVER-PATCH) ci(apm-integrations): split next jobs by individual version instead of range (Roch Devost) #8484
[199e240fb1] - (SEMVER-PATCH) ci(all-green): rerun failed workflows on startup when all-green is rerun (Roch Devost) #8475
[4566023687] - (SEMVER-PATCH) ci: split integration-bundler into integration-esbuild and integration-webpack (Roch Devost) #8470
[ee54394360] - (SEMVER-PATCH) ci: retry yarn install on failure in datadog-ci action (Roch Devost) #8459
[d482f4b7e9] - (SEMVER-PATCH) ci: remove node_modules cache from install action (Roch Devost) #8393
[c833bf860a] - (SEMVER-PATCH) chore(benchmark): reduce sirun log iterations from 40 to 10 (Roch Devost) #8474
[94c1d4e91c] - (SEMVER-MINOR) feat(kafkajs): add kafka.messages.offsets to producer span (Piotr WOLSKI) #8468

May 19, 2026

v4.8.6

Langchain Bedrock spans now use correct model; inference profile ARN bug fixed

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

LLM Observability: Users auto instrumenting Langchain and using Bedrock inference profiles would have their spans' be associated with the inference profile's ARN instead of the underlying LLM model. We now resolve the correct underlying model if users pass base_model_id to their ChatBedrockConverse instantiation. https://github.com/DataDog/dd-trace-py/pull/18151

Full Changelog: https://github.com/DataDog/dd-trace-py/compare/v4.8.5...v4.8.6

May 18, 2026

v4.8.5

Bedrock guardContent no longer drops user input; inferred spans setting deprecated

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Deprecation Notes

Tracing: DD_TRACE_INFERRED_SPANS_ENABLED is deprecated and will be removed in 5.0.0. Use DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED instead. The old environment variable continues to work but emits a DDTraceDeprecationWarning when set.

Bug Fixes

LLM Observability: This fix resolves an issue where text wrapped in Bedrock Converse guardContent content blocks was rendered as [Unsupported content type: guardContent] in traces, dropping the user's input.

LLM Observability: The OpenAI integration now preserves assistant message content when tool_calls are present on the same message. #17760

starlette: This fix resolves an issue where passing middleware=None caused application startup to fail when Starlette tracing was enabled.

LLM Observability: Resolves an issue in the Claude Agent SDK integration where unnecessary LLM spans were being created and affecting the trace structure. The handler now only opens a new LLM span after a UserMessage that actually contained tool results, so messages without tool results no longer overwrite the in-flight LLM span.

May 13, 2026

v5.103.0

Electron support added; config validation and kafkajs fixes

[03ac52a9a1] - (SEMVER-PATCH) bump native-metrics to 3.1.2 (Roch Devost) #8473
[65ca8204f9] - (SEMVER-PATCH) fix(lint): validate config names against both index.d.ts and index.d.v5.ts (Roch Devost) #8465
[ba39be2a3f] - (SEMVER-PATCH) ci(llmobs): parallelize openai job by Node version (Roch Devost) #8471
[2b8834f73a] - (SEMVER-PATCH) ci: route high-volume CI API calls through dd-octo-sts (Roch Devost) #8391
[dc3e2ec89e] - (SEMVER-PATCH) feat(electron): add initial support (Roch Devost) #7002
[56b6048e0b] - (SEMVER-PATCH) [test optimization] stabilize web app server lifecycle (Juan Antonio Fernández de Alba) #8444
[ae0e0c8872] - (SEMVER-PATCH) [test optimization] Fix playwright v5 tests (Juan Antonio Fernández de Alba) #8458
[3325f9eed8] - (SEMVER-PATCH) ci(serverless): use tags for serverless-tools ci (Olivier Nzia) #8213
[07cf95645a] - (SEMVER-PATCH) fix(config): keep v5 IAST programmatic aliases working (Ruben Bridgewater) #8457
[991d15c649] - (SEMVER-PATCH) refactor(config): route env reads through the config singleton (Ruben Bridgewater) #8241
[423ac93b66] - (SEMVER-PATCH) chore(aiguard): Fix ai guard flaky test (Ugaitz Urien) #8454
[afdaeed5c2] - (SEMVER-PATCH) chore(deps): bump oxc-parser from 0.128.0 to 0.129.0 in the runtime-minor-and-patch-dependencies group across 1 directory (dependabot[bot]) #8446
[0e26510523] - (SEMVER-PATCH) chore(deps-dev): bump sinon from 21.1.2 to 22.0.0 (dependabot[bot]) #8447
[0284dcf408] - (SEMVER-PATCH) chore(deps-dev): bump eslint-plugin-n from 17.24.0 to 18.0.1 (dependabot[bot]) #8448
[39a17c01e2] - (SEMVER-PATCH) fix(mocha): report test_suite_end for files with no describe wrapper (Sebastián Kay) #8437
[b06d144553] - (SEMVER-PATCH) fix: pin specific @datadog/openfeature-node-server version (Ugaitz Urien) #8456
[935054053e] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 2 directories with 1 update (dependabot[bot]) #8449
[f3b4edc644] - (SEMVER-PATCH) chore(deps-dev): bump the dev-minor-and-patch-dependencies group across 1 directory with 6 updates (dependabot[bot]) #8445
[9e5e475c35] - (SEMVER-PATCH) Revert "ci: replace CodeQL workflow with default setup configuration (#8380)" (Roch Devost) #8443
[aaf5e60a0a] - (SEMVER-PATCH) ci: add dd-octo-sts policies for upcoming workflow rate-limit changes (Roch Devost) #8392
[4c42d937ee] - (SEMVER-PATCH) ci: update playwright-tools image to 1.54.1-5 (Roch Devost) #8440
[4774cc13ac] - (SEMVER-PATCH) perf(graphql): memoize the apollo signature pipeline (Ruben Bridgewater) #8308
[f8e5148693] - (SEMVER-PATCH) test(ci-visibility): wait for cypress child exit before next test (Ruben Bridgewater) #8418
[0402d979d8] - (SEMVER-PATCH) fix(kafkajs): read clusterId from existing metadata (Ruben Bridgewater) #8389
[6bc8c6cccf] - (SEMVER-PATCH) test(ci-visibility): gather payloads until child exits (Ruben Bridgewater) #8419
[e78a6c1e4f] - (SEMVER-PATCH) fix(ci): prevent Buildkite non-PR builds from being tagged as PRs (Juan Antonio Fernández de Alba) #8435
[6f1d63ccbc] - (SEMVER-PATCH) fix(pg): stop accumulating DBM comments on reused query objects (Ruben Bridgewater) #8400
[89df4e6258] - (SEMVER-PATCH) refactor: drop JSDoc @param defaults (Ruben Bridgewater) #8350
[abf97b0011] - (SEMVER-PATCH) implement ai guard telemetry metrics (Ilyas Shabi) #8093
[4ab60e2551] - (SEMVER-PATCH) refactor(config): gate experimental.appsec, plugins, ingestion shapes (Ruben Bridgewater) #8318
[626ce462f1] - (SEMVER-PATCH) [test optimization] propagate DD_CUSTOM_PARENT_ID for Jenkins (Daniel Fernández) #8433
[31899bd0b8] - (SEMVER-PATCH) refactor(config): gate experimental b3, profiling, and b3 single header (Ruben Bridgewater) #8316
[3ada9255d9] - (SEMVER-PATCH) bench: fix benchmarks running unsupported Node.js 26 (Ruben Bridgewater) #8430
[eeeccd65ce] - (SEMVER-PATCH) fix(kafka): stop mutating caller-owned message.headers (Ruben Bridgewater) #8388
[b2f62ef205] - (SEMVER-PATCH) test(setup): isolate getInstrumentation from production addHook (Ruben Bridgewater) #8396
[1b83d35bbb] - (SEMVER-PATCH) refactor(lambda): collapse runtime/ritm.js into index.js (Ruben Bridgewater) #8397
[7ffc20933c] - (SEMVER-PATCH) ci(benchmarks): fail sirun job when individual variants crash (Fayssal DEFAA) #8292
[a78d2d57d7] - (SEMVER-PATCH) perf(couchbase): cache the per-operation channel bag (Ruben Bridgewater) #8373
[fc09317297] - (SEMVER-PATCH) docs: fix legacyBaggageEnabled default value and MIGRATING.md (Ruben Bridgewater) #8317
[242bdf9f90] - (SEMVER-PATCH) fix(llmobs): avoid malformed x-datadog-tags when carrier header is unset (Grace Williams) #8366
[89ae70d027] - (SEMVER-PATCH) perf(graphql): trim per-resolver allocations (Ruben Bridgewater) #8309
[6da7add04d] - (SEMVER-PATCH) refactor(config): gate experimental.iast aliases off in v6 (Ruben Bridgewater) #8320
[973a882bd3] - (SEMVER-PATCH) test(pg): pin prepared-statement reuse under DBM full mode (Ruben Bridgewater) #8398
[c8d56550fd] - (SEMVER-PATCH) chore: support re-runs in the community PR CI script (Ruben Bridgewater) #8385
[af652f0ad2] - (SEMVER-PATCH) perf(pg): assign the injectable text directly when configurable (Ruben Bridgewater) #8372
[cd377ffa8d] - (SEMVER-PATCH) fix(fetch): rename globalThis.fetch wrapper from "value" to "fetch" (Ruben Bridgewater) #8406
[e55e65b7f8] - (SEMVER-PATCH) perf(redis): tighten per-command instrumentation (Ruben Bridgewater) #8310
[985b1d0322] - (SEMVER-PATCH) test(http): cover server-side request hook resource.name override (Ruben Bridgewater) #8399
[ff25b95aa9] - (SEMVER-PATCH) chore(ci) update one-pipeline (gh-worker-campaigns-3e9aa4[bot]) #8424
[5f6e1bb9be] - (SEMVER-PATCH) chore(benchmark): drop Node 18 from sirun matrix and add Node 26 (Ruben Bridgewater) #8423
[81e10d12de] - (SEMVER-PATCH) perf(encode): emit span fields and event attributes as compact msgpack ints (Ruben Bridgewater) #8229
[f73d984145] - (SEMVER-PATCH) ci: activate codecov coverage minimum for PRs (Ruben Bridgewater) #8209
[e8c394c014] - (SEMVER-PATCH) fix(llmobs): bound Bedrock token-headers cache (Ruben Bridgewater) #8161
[6b370862a5] - (SEMVER-MINOR) docs(appsec): drop deprecated extended-headers and rasp.bodyCollection types in v6 (Ruben Bridgewater) #8322
[2b48b7991f] - (SEMVER-PATCH) fix(benchmark): restore five crashing sirun bench groups (Ruben Bridgewater) #8307
[60cfa5c013] - (SEMVER-PATCH) [test optimization] Mitigate vitest flakiness (Juan Antonio Fernández de Alba) #8417
[841b085491] - (SEMVER-PATCH) [test optimization] Restore retries to test optimization tests (Juan Antonio Fernández de Alba) #8414
[47d6ffedc1] - (SEMVER-PATCH) [test optimization] Mitigate jest flakiness (Juan Antonio Fernández de Alba) #8415
[084086b666] - (SEMVER-PATCH) test(ci-visibility): disable real net access in git_metadata (Ruben Bridgewater) #8416
[e9eca7962d] - (SEMVER-MINOR) Send hidden tags under _dd.ci.library_configuration_error in every event of a test session (Sebastián Kay) #8274
[e276b97573] - (SEMVER-PATCH) [test optimization] bump cypress latest version (Juan Antonio Fernández de Alba) #8365
[9fad4a59bd] - (SEMVER-PATCH) [test optimization] bump vitest latest version (Juan Antonio Fernández de Alba) #8383
[1407421603] - (SEMVER-PATCH) chore: key the cache on DD_TRACE_GIT_METADATA_ENABLED (Ruben Bridgewater) #8395
[b92bdc19fa] - (SEMVER-PATCH) [test optimization] Use duration buckets for cypress EFD retries (Juan Antonio Fernández de Alba) #8290
[914484dd16] - (SEMVER-PATCH) docs(iast): drop deprecated securityControls type (Ruben Bridgewater) #8315
[0b494e8792] - (SEMVER-PATCH) chore(deps): bump the test-versions group across 1 directory with 2 updates (dependabot[bot]) #8410
[807fceb14d] - (SEMVER-PATCH) chore(deps): bump openai (dependabot[bot]) #8360
[f5ee7b20de] - (SEMVER-PATCH) fix(tracing): fix TypeError in ritm.js when requiring node:-prefixed built-in modules (Yuichi Uchida) #8304
[057685610b] - (SEMVER-PATCH) fix(stacktrace): filter dd-trace instrumentation frames for any repo directory name (Roch Devost) #8301
[a39c44ed42] - (SEMVER-PATCH) test(ws): refactor lifecycle hooks to use dynamic ports and separate afterEach (Roch Devost) #8367
[e3a718bae7] - (SEMVER-PATCH) chore(deps-dev): bump nock from 13.5.6 to 14.0.14 (dependabot[bot]) #8280
[a3bb4cd3af] - (SEMVER-PATCH) ci: replace CodeQL workflow with default setup configuration (Roch Devost) #8380
[47591c3fde] - (SEMVER-PATCH) perf(aws-sdk): trim per-response allocations (Ruben Bridgewater) #8328
[c05e122cbe] - (SEMVER-PATCH) perf(mongodb): cache the per-connection topology shape (Ruben Bridgewater) #8370
[fc437471cc] - (SEMVER-PATCH) perf(database): cache the DBM SQL injection comment per connection (Ruben Bridgewater) #8371
[8d6b8824d2] - (SEMVER-PATCH) perf(grpc): cache method metadata, drop banned for-in walks (Ruben Bridgewater) #8377
[5c2656c36b] - (SEMVER-PATCH) chore(deps): bump dc-polyfill from 0.1.10 to 0.1.11 (Brian Marks) #8369
[e05e6c6784] - (SEMVER-PATCH) perf(pubsub): trim per-message allocations in publish hot path (Ruben Bridgewater) #8374
[9fb4d152b2] - (SEMVER-PATCH) perf(mongodb): fold limit-depth and bigint sanitisation into one pass (Ruben Bridgewater) #8375
[c610358953] - (SEMVER-PATCH) fix(grpc): require a colon and a strictly numeric tail before tagging… (Ruben Bridgewater) #8378
[a46860bc55] - (SEMVER-PATCH) refactor(azure-metadata): parse WEBSITE_OWNER_NAME without regex (Ruben Bridgewater) #8348
[8e9d38b0d3] - (SEMVER-PATCH) docs(plugins): drop deprecated whitelist/blacklist plugin types from v6 surface (Ruben Bridgewater) #8321
[43de16c356] - (SEMVER-PATCH) chore(couchbase): drop SDK 2.x instrumentation hook (Ruben Bridgewater) #8362
[b893e3165d] - (SEMVER-PATCH) test: add a few database tests to cover recent reports better (Ruben Bridgewater) #7534
[26f3793cd9] - (SEMVER-PATCH) fix(aws-sdk): global crypto error (Pablo Erhard) #8368
[5802bcb274] - (SEMVER-PATCH) fix(plugin-mysql): destructure CLIENT_PORT_KEY from the constants module (Ruben Bridgewater) #8303
[5e54f8226f] - (SEMVER-PATCH) [test optimization] Bump playwright support to 1.59 (Juan Antonio Fernández de Alba) #8363
[b3c20f8cf1] - (SEMVER-PATCH) [test optimization] Bump cucumber latest version (Juan Antonio Fernández de Alba) #8364
[a5834a8f30] - (SEMVER-PATCH) [test optimization] Support Jest 30.4.1 (Juan Antonio Fernández de Alba) #8361
[aa3c020225] - (SEMVER-PATCH) fix(config): disable OTLP only when protocolVersion is explicitly set (Ruben Bridgewater) #8335
[0b803f55de] - (SEMVER-PATCH) [test-optimization] Propagate ITR skipping enabled tag to suites and tests (Andrey Marchenko) #8332
[7350d99b0f] - (SEMVER-PATCH) [test optimization] Use duration buckets for playwright EFD retries (Juan Antonio Fernández de Alba) #8289

May 12, 2026

v4.8.4

Malformed dist-info no longer floods stderr

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

telemetry: This change resolves an issue where malformed installed distribution metadata caused a single bad dist-info entry to flood stderr with repeated tracebacks.

Full Changelog: https://github.com/DataDog/dd-trace-py/compare/v4.8.3...v4.8.4

May 11, 2026

v4.8.3

CosmosDB spans: http.status_code → db.response.status_code

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

azure_cosmos: This change removes the http.status_code tags from Azure CosmosDB spans and replaces them with the use of the db.response.status_code metric. For customers using ddtrace v4.8.0 and relying on the http.status_code tag of cosmosdb.query spans, this is a breaking change.

May 8, 2026

v5.102.0

Span events handle non-strings; LLM Observability enriched

[3daff728b8] - (SEMVER-PATCH) fix: tolerate non-string span event names and whitespace-suffixed step functions JSON (Ruben Bridgewater) #8358
[130a30caf8] - (SEMVER-PATCH) chore(ci) update one-pipeline (gh-worker-campaigns-3e9aa4[bot]) #8342
[9001878b55] - (SEMVER-PATCH) refactor: dedupe redundant chars in regex character classes (Ruben Bridgewater) #8346
[6c7179306d] - (SEMVER-PATCH) refactor(appsec): convert WAF duration BigInt to Number, not via parseInt (Ruben Bridgewater) #8306
[14a0a822a3] - (SEMVER-PATCH) refactor: dedupe identical function bodies (Ruben Bridgewater) #8347
[0e7a8c7256] - (SEMVER-PATCH) refactor: activate three sonarjs test/loop rules (Ruben Bridgewater) #8349
[80076dd5af] - (SEMVER-PATCH) chore(eslint): activate n/hashbang and n/no-process-exit (Ruben Bridgewater) #8352
[46c61ffdd1] - (SEMVER-PATCH) refactor(prefer-rest-params): convert non-instrumentation wrappers (Ruben Bridgewater) #8353
[bb4f61acaa] - (SEMVER-PATCH) refactor(prefer-rest-params): convert instrumentations a-c (Ruben Bridgewater)
[a66a186421] - (SEMVER-PATCH) refactor(prefer-rest-params): convert instrumentations d-l (Ruben Bridgewater) #8355
[418179b183] - (SEMVER-PATCH) refactor(prefer-rest-params): convert instrumentations m-z (Ruben Bridgewater) #8356
[1afa83174a] - (SEMVER-PATCH) build(types): add index.d.v5.ts snapshot and v5-release swap script (Ruben Bridgewater) #8333
[66d49013dc] - (SEMVER-PATCH) update all slow benchmarks to run less iterations (Roch Devost) #8345
[16c4e16b60] - (SEMVER-PATCH) update llmobs encode unicode mixed benchmarks with less iterations (Roch Devost) #8344
[035fb236b2] - (SEMVER-PATCH) [test optimization] fix mocha EFD callback test arity (Juan Antonio Fernández de Alba) #8338
[5c7c227eb8] - (SEMVER-MINOR) feat(llmobs): enhance LLM Observability span processor span kind, embedding & retrieval input and output (Sam Brenner) #8299
[d422180bbc] - (SEMVER-PATCH) [test optimizaton] Use duration buckets for vitest EFD retries (Juan Antonio Fernández de Alba) #8288
[1764a646dc] - (SEMVER-PATCH) fix(index): Add truncate type to database plugin interfaces (Pablo Erhard) #8336
[ce8a754efa] - (SEMVER-PATCH) chore: add sonar linter (Ruben Bridgewater) #8150
[16016c6b5a] - (SEMVER-PATCH) perf(shimmer): drop assertNotClass toString call (Ruben Bridgewater) #8329
[4c45faa946] - (SEMVER-PATCH) fix(google-cloud-pubsub): honor DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED on consumer and push spans (Crystal Luc-Magloire) #8125
[771253286d] - (SEMVER-PATCH) perf(appsec): trim per-request allocations in waf and reporter (Ruben Bridgewater) #8326
[2c8447bb13] - (SEMVER-PATCH) test: decouple request specs from nock 13 internals (Ruben Bridgewater) #8305
[8d64c6e392] - (SEMVER-PATCH) [test optimization] Use duration buckets for cucumber EFD retries (Juan Antonio Fernández de Alba) #8287
[8a10090138] - (SEMVER-PATCH) perf(aws-sdk): cache instrumentation handles and dedupe per-call work (Ruben Bridgewater) #8231
[6c0da23ceb] - (SEMVER-PATCH) [test optimization] Use duration buckets for mocha EFD retries (Juan Antonio Fernández de Alba) #8286
[baeb04586b] - (SEMVER-PATCH) fix(ci): approve pnpm package-manager builds (Juan Antonio Fernández de Alba) #8331
[a200dc4d96] - (SEMVER-PATCH) chore(iast): Improve stringifyWithRanges performance (Ugaitz Urien) #8271
[19ef5f86f2] - (SEMVER-PATCH) test(jest,vitest): split integration specs into parallel CI jobs via SPEC matrix (Roch Devost) #8311
[97853d2969] - (SEMVER-PATCH) ci: add fail-fast: false to jobs with matrix (Charles de Beauchesne) #8314
[0fdc801f66] - (SEMVER-PATCH) fix(test-optimization): edge cases when computing final_status tag on playwright (Sebastián Kay) #8249
[50aa02547e] - (SEMVER-PATCH) test: reuse azure function process to lower tests run time (Roch Devost) #8102
[fe9678ab94] - (SEMVER-PATCH) fix(net): restore Socket.emit after the connect handshake (Ruben Bridgewater) #8275
[5168c02d94] - (SEMVER-PATCH) perf(encode): drop the per-span filter Array allocations in the 0.4 e… (Ruben Bridgewater) #8233
[6bf78256b2] - (SEMVER-PATCH) ci(benchmarks): bump microbenchmark splits from 5 to 6 (Fayssal DEFAA) #8283
[ff0e435df9] - (SEMVER-PATCH) chore(ci): pin all Docker images to SHA digests (Roch Devost) #8298
[c54290e695] - (SEMVER-PATCH) test(kafkajs): wait for leaders on createTopics to fix DSM flakes (Eric Firth) #8300

May 7, 2026

v4.7.2

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

CI Visibility: This fix resolves an issue where a failure response from the /search_commits endpoint caused the git metadata upload to fall back to sending the full 30-day commit history instead of aborting. This fallback could trigger cascading write load on the backend. The upload now aborts when search_commits fails, matching the behavior when the /packfile upload itself fails.

ai_guard: This fix resolves a conflict between ddtrace.auto and strands when imported in the same file, which left Strands hooks silently disabled. The Strands integration now loads lazily on first attribute access so its event class identities match those the agent dispatches.

AAP: This fix resolves an issue where Application and API Protection (AAP) was incorrectly reported as an enabled product in internal telemetry for all services by default. Previously, registering remote configuration listeners caused AAP to be reported as activated even when it was not actually enabled. This had no impact on customers as it only affected internal telemetry data. AAP is now only reported as activated when it is explicitly enabled or enabled through remote configuration.

iast: A crash has been fixed.

profiling: A crash that could happen in child processes after fork has been fixed.

tracing: This fix resolves a memory leak where reference cycles through a span's properties were invisible to Python's cyclic garbage collector and accumulated proportionally to traced call volume.

internal: A crash has been fixed.

LLM Observability: Fixes incorrect span hierarchy in LLMObs traces when using the ddtrace SDK alongside OTel-based instrumentation (e.g. Strands Agents). OTel gen_ai spans (e.g. invoke_agent) were incorrectly appearing as siblings of their SDK parent span (e.g. call_agent) rather than being nested under it.

profiling: A rare crash occurring when profiling asyncio code with many tasks or deep call stacks has been fixed.

profiling: A bug in Lock Profiling that could cause crashes when trying to access attributes of custom Lock subclasses (e.g. in Ray) has been fixed.

profiling: A race condition which could make asyncio code raise exceptions at exit has been fixed.

serverless: AWS Lambda functions now appear under their function name as the service when DD_SERVICE is not explicitly configured. Service remapping rules configured in Datadog will now apply correctly to Lambda spans.

May 6, 2026

v4.8.2

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

ai_guard
- This fix resolves a conflict between ddtrace.auto and strands when imported in the same file, which left Strands hooks silently disabled. The Strands integration now loads lazily on first attribute access so its event class identities match those the agent dispatches.
tracing
- Limits parsing of the W3C tracestate header during tracecontext extraction to 32 list-members and 512 UTF-8 bytes, consistent with the W3C Trace Context specification (https://www.w3.org/TR/trace-context/). Extra list-members and trailing whole entries that would exceed the byte budget are ignored, so unusually large headers no longer expand unbounded work during extraction. The Datadog dd= list-member is preferred: it is kept when present (including when it appears late in the header or alone exceeds the byte cap), and other vendors are dropped first. List-members longer than DD_TRACE_TRACESTATE_ITEM_MAX_CHARS (128) characters are removed first when trimming by list-member count or byte budget, so shorter vendor entries are kept when possible.
- tracing: Parsing incoming baggage HTTP headers now respects DD_TRACE_BAGGAGE_MAX_ITEMS [default 64] and DD_TRACE_BAGGAGE_MAX_BYTES [default 8192], consistent with baggage injection. Previously, extraction could retain every comma-separated entry regardless of those limits. The tracer drops excess pairs and records truncation telemetry when limits apply.

v5.101.0

[f121c4281f] - (SEMVER-PATCH) fix(scripts): harden mocha-parallel-files against crashes and silent … (Ruben Bridgewater) #8243
[fc60a84f47] - (SEMVER-PATCH) chore(deps): bump the test-versions group across 1 directory with 3 updates (dependabot[bot]) #8279
[0509edc11b] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 2 directories with 2 updates (dependabot[bot]) #8281
[6fc588e858] - (SEMVER-PATCH) Revert "ci(codeql): skip workflow for non-production file changes and merge queue (#8277)" (Ruben Bridgewater) #8291
[215ea598d4] - (SEMVER-PATCH) test(otel): scope post-finish no-op assertions to mutator keys (Ruben Bridgewater) #8285
[3616c45502] - (SEMVER-PATCH) ci: extract shared datadog-ci install action and replace coverage-upload-github-action (Roch Devost) #8278
[18c37b6d40] - (SEMVER-PATCH) refactor(config): extract git metadata loader out of Config (Ruben Bridgewater) #8238
[cbd06436a8] - (SEMVER-PATCH) fix(propagation): tighten W3C trace-context inject and extract correc… (Ruben Bridgewater) #8269
[388ae8171b] - (SEMVER-MINOR) fix(llmobs): add cost_tags to annotate and annotationContext (Xinyuan Guo) #8175
[e3a092a7d5] - (SEMVER-PATCH) perf(core): trim per-span allocations across id, span, and sampler (Ruben Bridgewater) #8235
[6ce2635162] - (SEMVER-PATCH) ci(codeql): skip workflow for non-production file changes and merge queue (Roch Devost) #8277
[b5259d74b6] - (SEMVER-PATCH) fix: guard propagation, stop mutating input, and fix log (Ruben Bridgewater) #8226
[de5649fd31] - (SEMVER-PATCH) fix(test): prevent cascading ws test failures by reliably closing wsServer in afterEach (William Conti) #8192
[57b73cfbbf] - (SEMVER-PATCH) ci: add duration and shorten time output for all green (Roch Devost) #8225
[c3f962f6f6] - (SEMVER-MINOR) feat(llmobs, anthropic): add reasoning/extended thinking support for Anthropic (Sam Brenner) #8146
[df6b6e0c71] - (SEMVER-MINOR) feat(exporter): retry through agent startup (Ruben Bridgewater) #8223
[210e9f7cc4] - (SEMVER-PATCH) update windows ci jobs to upload a node report artifact on crash (Roch Devost) #8239
[d28de77da5] - (SEMVER-PATCH) perf(plugins): trim per-message allocations in bullmq, sharedb, and d… (Ruben Bridgewater) #8232
[7afadcff5a] - (SEMVER-PATCH) ci: update all mcr references to use our mirror instead (Roch Devost) #8215
[7c559f48de] - (SEMVER-MINOR) feat(otel): forward getActiveSpan() writes onto the active Datadog span (Ruben Bridgewater) #8030
[a5733805a2] - (SEMVER-PATCH) fix(debugger): move process_tags to payload root (Thomas Watson) #8173
[6500302a1f] - (SEMVER-PATCH) perf(propagation): tighten tracestate, baggage, and tag inject paths (Ruben Bridgewater) #8234
[e8690041e0] - (SEMVER-PATCH) fix(otel): honor DD_TRACE_OTEL_ENABLED=false and OTEL_SDK_DISABLED=false (Ruben Bridgewater) #8219
[f67ac4dad3] - (SEMVER-PATCH) fix(otel): tighten OTel-bridge Span spec compliance (Ruben Bridgewater) #8242
[18eb17da71] - (SEMVER-PATCH) perf(llmobs): fast-path encodeUnicode and collapse plugin filter chains (Ruben Bridgewater) #8230
[6e32e73e40] - (SEMVER-PATCH) fix(llmobs): guard JSON.parse on streamed tool-call arguments (Ruben Bridgewater) #8227
[1234065491] - (SEMVER-PATCH) [test-optimization] Default Nx and Lage names in v6 (Juan Antonio Fernández de Alba) #8268
[aae141ef5f] - (SEMVER-PATCH) [test optimization] Use head SHA for test optimization dispatch (Juan Antonio Fernández de Alba) #8270
[4db54b181f] - (SEMVER-PATCH) chore(deps): bump axios from 1.15.0 to 1.15.2 in /integration-tests/webpack in the npm_and_yarn group across 1 directory (dependabot[bot]) #8267
[0b6d586397] - (SEMVER-PATCH) fix(scripts): resolve instrumentation ranges at both engine bounds (Ruben Bridgewater) #8251
[f6c69c6bf6] - (SEMVER-PATCH) ci: pipe GraphQL variables via stdin so commit-on-branch pushes don't… (Ruben Bridgewater) #8252
[afae167f30] - (SEMVER-PATCH) [test-optimization] Raise v6 Mocha minimum version (Juan Antonio Fernández de Alba) #8245
[d509f232a8] - (SEMVER-PATCH) [test-optimization] Raise v6 Cypress minimum version (Juan Antonio Fernández de Alba) #8247
[ea902ccf5b] - (SEMVER-PATCH) [test-optimization] Raise v6 Jest minimum version (Juan Antonio Fernández de Alba) #8246
[a329f1f575] - (SEMVER-MINOR) feat(llmobs): add tool_definitions support to Tagger (Alexandre Choura) #8082
[d34fada6ce] - (SEMVER-PATCH) chore(deps): bump the runtime-minor-and-patch-dependencies group across 2 directories with 3 updates (dependabot[bot]) #8259
[c267fbfd54] - (SEMVER-PATCH) pin node version to 24.24.1 for windows in ci (Roch Devost) #8262

v4.8.1

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Bug Fixes

django: Stop tagging async view and middleware spans as errored on routine ASGI cancellations (e.g. client disconnects on streaming responses), a regression introduced in 4.8.0rc4. Cancellation still propagates; the span just finishes without error.type='asyncio.exceptions.CancelledError'.

tracing: This fix resolves a memory leak where reference cycles through a span's properties were invisible to Python's cyclic garbage collector and accumulated proportionally to traced call volume.

May 4, 2026

v5.100.0

[d46c3485c9] - (SEMVER-PATCH) perf(propagation): rewrite tracestate parser to be linear (Ruben Bridgewater) #8256
[ed087ea187] - (SEMVER-PATCH) chore(deps-dev): bump the dev-minor-and-patch-dependencies group across 1 directory with 2 updates (dependabot[bot]) #8258
[d0eba220a0] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 2 directories with 1 update (dependabot[bot]) #8261
[33b0dc1e17] - (SEMVER-PATCH) fix(express): use the host's path-to-regexp dialect for route tagging (Ruben Bridgewater) #8224
[d064e062cf] - (SEMVER-PATCH) refactor(otel): extract bridge helpers into span-helpers.js (Ruben Bridgewater) #8220
[658450e18b] - (SEMVER-PATCH) fix(otel): return a non-recording span when the inner tracer is the noop (Ruben Bridgewater) #8218
[fe48fc0c5f] - (SEMVER-PATCH) fix(esm): expose 'tracer' as an ESM named export (Ruben Bridgewater) #8216
[90bb04871a] - (SEMVER-PATCH) fix: small correctness issues in mongodb-core and bullmq (Ruben Bridgewater) #8228
[20f9e76efe] - (SEMVER-PATCH) fix(config): align inferred service name with agent normalization (Ruben Bridgewater) #8217
[00a239a1e2] - (SEMVER-PATCH) fix(propagation): enforce baggageMax* caps on extract (Ruben Bridgewater) #8255
[15ce029c26] - (SEMVER-PATCH) ci: deprecate legacy macrobenchmarks in favor of parallel ones (Fayssal DEFAA) #8250
[9e6acd8951] - (SEMVER-MINOR) feat(test-optimization): create final_status tag on test event for playwright (Sebastián Kay) #8190
[77cc2e065d] - (SEMVER-PATCH) fix(hono): skip publishing when env.incoming is missing (Ruben Bridgewater) #8221
[c832bf7a10] - (SEMVER-PATCH) [test optimization] Update extraction of runner diagnostic dir using glob pattern matching (Sebastián Kay) #8115
[41789b4087] - (SEMVER-PATCH) refactor(config): drop more internalPropertyName aliases (Ruben Bridgewater) #8237
[c97a177432] - (SEMVER-PATCH) fix(encode): write truncated meta keys back to span.meta (Ruben Bridgewater) #8236
[e5e3e4d35b] - (SEMVER-PATCH) test: drop keep-alive on the ci-visibility web-app-server (Ruben Bridgewater) #8194
[6448470089] - (SEMVER-MINOR) [test-optimization] Honor attempt-to-fix failures (Juan Antonio Fernández de Alba) #8138
[925bd0aae7] - (SEMVER-PATCH) chore(deps): bump openai (dependabot[bot]) #8244
[b058d8b4af] - (SEMVER-PATCH) ci: remove old bundle-validate workflow (Roch Devost) #8214
[e199f82b3e] - (SEMVER-PATCH) ci: update all green script to fail when any retry failed (Roch Devost) #8222
[d109d5bf1b] - (SEMVER-PATCH) chore(codeowners): own remaining benchmark dirs and enforce in CI (Ruben Bridgewater) #8204

May 1, 2026

v5.99.1

[030413c8b3] - (SEMVER-PATCH) refactor(config): drop low-use internalPropertyName aliases (Ruben Bridgewater) #8211
[538b5746da] - (SEMVER-PATCH) ci: update all green to fail when the rate limit is reached (Roch Devost) #8212
[e1461827cb] - (SEMVER-PATCH) fix(dogstatsd): stop re-emitting stale gauges (Ruben Bridgewater) #8153
[ddfe085c82] - (SEMVER-PATCH) add github workflow to mirror images from flaky registries (Roch Devost) #8165
[1709d32f57] - (SEMVER-PATCH) bench(spans): cover tags-and-otel construction shape (Ruben Bridgewater) #8199
[a72b441efe] - (SEMVER-PATCH) fix(propagation): improve baggage implementation (Ruben Bridgewater) #8193
[f1b085a703] - (SEMVER-PATCH) bench(llmobs): add writer flush benchmark (Ruben Bridgewater) #8200
[ce42131535] - (SEMVER-PATCH) bench(aws-sdk): add inject and response-body benchmarks (Ruben Bridgewater) #8201
[35425f7a4e] - (SEMVER-PATCH) bench(dsm): add Data Streams Monitoring pathway benchmark (Ruben Bridgewater) #8202
[b47fabe6d1] - (SEMVER-PATCH) bench(propagation): add extract and inject benchmarks (Ruben Bridgewater) #8203
[6e86edfcda] - (SEMVER-PATCH) bench(encoding): cover native and legacy span-events paths (Ruben Bridgewater) #8205
[f6faa2b85f] - (SEMVER-PATCH) ci: fix flakiness workflow falling back to default for empty branch (Roch Devost) #8207
[0421807fcf] - (SEMVER-PATCH) fix(llmobs): correlate otel gen_ai spans with sdk llmobs spans (Zachary Groves) #8127
[63adfd5ae1] - (SEMVER-PATCH) bench: add plugin-graphql-long sirun benchmark (Crystal Luc-Magloire) #8089
[26bee69994] - (SEMVER-PATCH) minimize requests for all green script and retry faster (Roch Devost) #8180
[7763af7037] - (SEMVER-PATCH) ci: add days/branch/occurrences option to flakiness report (Roch Devost) #8185
[974bbce729] - (SEMVER-PATCH) test: add integration coverage harness (Ruben Bridgewater) #8071
[40c512422d] - (SEMVER-PATCH) fix(test): use ALS run() instead of enterWith() in mock agent server.emit (William Conti) #8166
[8628c2716a] - (SEMVER-PATCH) fix(appsec): bound detectedSpecificEndpoints cache (Ruben Bridgewater) #8163
[fab96f8428] - (SEMVER-PATCH) test: fix mocha retries issue (Ilyas Shabi) #8189
[476b6d1269] - (SEMVER-PATCH) test: Improve profiling tests (Ilyas Shabi) #8169

Apr 30, 2026

v4.8.0

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Upgrade Notes

claude_agent_sdk: Tool span resource names have changed from the tool name (e.g. Read, Bash) to claude_agent_sdk.tool. The specific tool name is still available in the span name (e.g. claude_agent_sdk.tool.Read). Users relying on tool resource names should update them accordingly.
ray: Adds DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED (default: False) configuration to control Ray submission tracing. Set DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED=true to trace task.submit and actor_method.submit spans. Leave it unset to trace only execution spans. See Ray integration documentation for more details.
ray: ray.job.submit spans are removed. Ray job submission outcome is now reported on the existing ray.job span through ray.job.submit_status.

Deprecation Notes

Tracing: DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED is deprecated and will be removed in 5.0.0. Use DD_TRACE_INFERRED_SPANS_ENABLED instead. The old environment variable continues to work but emits a DDTraceDeprecationWarning when set.
tracing: The pin parameter in ddtrace.contrib.dbapi.TracedConnection, ddtrace.contrib.dbapi.TracedCursor, and ddtrace.contrib.dbapi_async.TracedAsyncConnection is deprecated and will be removed in version 5.0.0. To manage configuration of DB tracing please use integration configuration and environment variables.
LLM Observability: Removes support for the RAGAS integration. As an alternative, if you have RAGAS evaluations, you can manually submit these evaluation results. See LLM Observability external evaluation documentation for more information.

New Features

AI Guard: Add DD_AI_GUARD_BLOCK environment variable. Defaults to True, which means the blocking behavior configured in the Datadog AI Guard UI (in-app) will be honored. Set to False to force monitor-only mode locally: evaluations are still performed but AIGuardAbortError is never raised, regardless of the in-app blocking setting.
AI Guard response objects now include a dict field tag_probs with the probabilities for each tag.
CI Visibility: Adds Bazel offline execution support with two modes: manifest mode (DD_TEST_OPTIMIZATION_MANIFEST_FILE), which reads settings and test data from pre-fetched cache files without network access; and payload-files mode (DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES), which writes test event, coverage, and telemetry payloads as JSON files instead of sending HTTP requests. Both modes can be used independently or together.
LLM Observability: Captures individual LLM spans for each Claude model turn within a Claude Agent SDK session. Each LLM span captures the input messages, output messages, model name, and token usage metrics (for claude_agent_sdk >= 0.1.49).
AAP: This adds Application Security support for FastAPI and Starlette applications using mounted sub-applications (via app.mount()). WAF evaluation, path parameter extraction, API endpoint discovery, and http.route reporting now correctly account for mount prefixes in sub-application routing.
google_cloud_pubsub: This adds tracing for Google Cloud Pub/Sub admin operations on topic, subscription, snapshot, and schema management methods.
google_cloud_pubsub: Adds support for Google Cloud Pub/Sub push subscriptions. When a push subscription delivers a message via HTTP, the integration now creates an inferred gcp.pubsub.receive span that captures subscription and message metadata. Use DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS to control whether the inferred span becomes a child of the producer trace or starts a new trace with the producer context attached as a span link (default: False).
LLM Observability: Add ExperimentRun.as_dataframe() to convert experiment run results into a pandas.DataFrame with a two-level MultiIndex on columns. Each top-level group (input, output, expected_output, evaluations, metadata, error, span_id, trace_id) maps to the first index level. Dict-valued fields are flattened one level deep; scalar fields use an empty string as the sub-column name. Each evaluator gets its own column containing the full evaluation result dict. Requires pandas to be installed (pip install pandas).
LLM Observability: Adds an eval_scope parameter to LLMObs.submit_evaluation() (one of "span" (default) or "trace"). Use eval_scope="trace" to associate an evaluation with an entire trace by passing the root span context.
LLM Observability: Adds LLMObs.get_spans() to retrieve span events from the Datadog platform API (GET /api/v2/llm-obs/v1/spans/events). Supports filtering by trace ID, span ID, span kind, span name, ML app, tags, and time range. Results are auto-paginated. Requires DD_API_KEY and DD_APP_KEY.
profiling: Profiles generated from fork-based servers now include a process_type tag with the value main or worker.
tracing: Support for making the default span name for @tracer.wrap include the class name has been added. For now, this is opt-in and can be enabled by setting DD_TRACE_WRAP_SPAN_NAME_INCLUDE_CLASS=true. The new naming will become the default in the next major release.
llmobs: Adds support for enabling and disabling LLMObs via Remote Configuration.
mysql: This introduces tracing support for mysql.connector.aio.connect in the MySQL integration.
profiling: Thread sub-sampling is now supported. This allows to set a maximum number of threads to capture stacks for at each sampling interval. This can be used to reduce the CPU overhead of the Stack Profiler.
llama_index: Adds APM tracing and LLM Observability support for llama-index-core>=0.11.0. Traces LLM calls, query engines, retrievers, embeddings, and agents. See the llama_index documentation for more information.
ASM: Adds a LiteLLM proxy guardrail integration for Datadog AI Guard. The ddtrace.appsec.ai_guard.integrations.litellm.DatadogAIGuardGuardrail class can be registered as a custom guardrail in the LiteLLM proxy to evaluate requests and responses against AI Guard security policies. Requires the LiteLLM proxy guardrails API v2 available since litellm>=1.46.1.
azure_cosmos: Add tracing support for Azure CosmosDB. This integration traces CRUD operations on CosmosDB databases, containers, and items.
LLM Observability: Introduces a decorator tag to LLM Observability spans that are traced by a function decorator.
CI Visibility: adds automatic log correlation and submission so that test logs appear alongside their corresponding test run in Datadog. Set DD_AGENTLESS_LOG_SUBMISSION_ENABLED=true for agentless setups, or DD_LOGS_INJECTION=true when using the Datadog Agent.
tracing: Adds support for exporting traces in OTLP HTTP/JSON format via libdatadog. Set OTEL_TRACES_EXPORTER=otlp to send spans to an OTLP endpoint instead of the Datadog Agent.

LLM Observability: Experiments accept a pydantic_evals ReportEvaluator as a summary evaluator when its evaluate return annotation is exactly ScalarResult. The scalar value is recorded as the summary evaluation. Report evaluators that declare a broader analysis return type (for example the full ReportAnalysis union) are not accepted as summary evaluators; use a class-based or function summary evaluator instead. Examples and further documentation can found in our documentation [here](https://docs.datadoghq.com/llm_observability/guide/evaluation_developer_guide).

Example:

from pydantic_evals.evaluators import ReportEvaluator
from pydantic_evals.evaluators import ReportEvaluatorContext
from pydantic_evals.reporting.analyses import ScalarResult

from ddtrace.llmobs import LLMObs

dataset = LLMObs.create_dataset(
    dataset_name="<DATASET_NAME>",
    description="<DATASET_DESCRIPTION>",
    records=[RECORD_1, RECORD_2, RECORD_3, ...]
)

class TotalCasesEvaluator(ReportEvaluator):
    def evaluate(self, ctx: ReportEvaluatorContext) -> ScalarResult:
        return ScalarResult(
            title='Total Cases',
            value=len(ctx.report.cases),
            unit='cases',
        )

def my_task(input_data, config):
    return input_data["output"]

equals_expected = EqualsExpected()
summary_evaluator = TotalCasesEvaluator()

experiment = LLMObs.experiment(
    name="<EXPERIMENT_NAME>",
    task=my_task, 
    dataset=dataset,
    evaluators=[equals_expected],
    summary_evaluators=[summary_evaluator],
    description="<EXPERIMENT_DESCRIPTION>."
)

result = experiment.run()

Bug Fixes

CI visibility: This fix resolves issues where CI provider metadata could omit pull request base branch and head commit details or report incorrect pull request values for some providers.
AAP: This fix resolves an issue where Application and API Protection (AAP) was incorrectly reported as an enabled product in internal telemetry for all services by default. Previously, registering remote configuration listeners caused AAP to be reported as activated even when it was not actually enabled. This had no impact on customers as it only affected internal telemetry data. AAP is now only reported as activated when it is explicitly enabled or enabled through remote configuration.
asgi: Fixed an issue caused network.client.ip and http.client_ip span tags being missing for FastAPI.
iast: A crash has been fixed.
lambda: Fixes a spurious Unable to create shared memory warning on every AWS Lambda cold start.
LLM Observability: Fixes an issue where an APM_TRACING remote configuration payload that did not include an llmobs section would disable LLM Observability on services where it had been enabled programmatically via LLMObs.enable(). Services that enabled LLM Observability via the DD_LLMOBS_ENABLED environment variable were unaffected. The handler now only changes LLM Observability state when the remote configuration payload explicitly carries an llmobs.enabled directive.
LLM Observability: Fixes a circular import in ddtrace.llmobs._writer when anthropic, openai, and botocore is installed.
Prevent potential crashes when the client library fails to restart a worker thread due to hitting a system resource limit.
internal: This fix resolves an issue where reading unknown attributes from ddtrace.internal.process_tags caused a KeyError instead of raising an AttributeError.
rq: Fixes compatibility with RQ 2.0. Replaces the removed Job.get_id() with the job.id property, and handles Job.get_status() now raising InvalidJobOperation for expired jobs (e.g. result_ttl=0) instead of returning None. #16682
tornado: Fixes an issue where routes inside a nested Tornado application were matched in reverse declaration order, causing a catch-all pattern to win over a more-specific route defined before it. This resulted in incorrect http.route tags on spans.
tornado: The http.route tag is now populated for routes whose regex cannot be reversed by Tornado (e.g. patterns containing non-capturing groups such as (?:a|b)). Capturing groups are still rendered as %s, consistent with Tornado's own route format, while non-capturing constructs are kept verbatim.
telemetry: This fix resolves an issue where unhandled exceptions raised by importlib.metadata during interpreter shutdown (for example, when Gunicorn workers exit uncleanly after a failed startup) caused update_imported_dependencies to surface errors through sys.excepthook. Failures while discovering dependencies for the app-dependencies-loaded telemetry payload are now logged at debug level and swallowed so they no longer propagate out of the dependency-reporting path.
profiling: Fixes noise caused by the profiler attempting to load its native module even when profiling was disabled,
profiling: A race condition which could make asyncio code raise exceptions at exit has been fixed.
remote_config: This fix resolves an issue where brief Datadog Agent connection errors could drop Remote Configuration polls, causing products such as Dynamic Instrumentation to temporarily appear disabled.
LLM Observability: Change the default model_provider and model_name to "unknown" from "custom" when a model did not match any known provider prefix in the Google GenAI, VertexAI, and Google ADK integrations.
LLM Observability: This fix resolves tracing issues for pydantic-ai >= 1.63.0 where tool spans and agent instructions were not being properly captured. This fix adds tracing to the ToolManager.execute_tool_call method for newer versions of the library to resolve this issue.
celery: remove unnecessary warning log about missing span when using Task.replace().
django: Fixes RuntimeError: coroutine ignored GeneratorExit that occurred under ASGI with async views and async middleware hooks on Python 3.13+. Async view methods and middleware hooks are now correctly detected and awaited instead of being wrapped with sync bytecode wrappers.
Code Security (IAST): Fixes a missing return in the IAST taint tracking add_aspect native function that caused redundant work when only the right operand of a string concatenation was tainted.
openai: Fixes async streaming spans never being finished when using AsyncAPIResponse (e.g. responses.create(stream=True)). The sync handle_request hook called resp.parse() without awaiting the coroutine, preventing the stream from being wrapped in TracedAsyncStream. This caused disconnected LLM Observability traces for streamed sub-agent calls via the OpenAI Agents SDK.
Fixed a race condition with internal periodic threads that could have caused a rare crash when forking.
ray: This fix resolves an issue where Ray integration spans could use an incorrect service name when the Ray job name was set after instrumentation initialization.
tracing: Fixes the svc.auto process tag attribution logic. The tag now correctly reflects the auto-detected service name derived from the script or module entrypoint, matching the service name the tracer would assign to spans.
Fixes an issue where internal background threads could cause crashes or instability in applications that fork (e.g. Gunicorn, uWSGI) or during Python shutdown. Affected applications could experience intermittent crashes or hangs on exit.
tracing: This fix resolves an issue where applications started with python -m <module> could report entrypoint.name as -m in process tags.
apm: Fixed an issue where network.client.ip and http.client_ip span tags were missing when client IP collection was enabled and request had no headers.
litellm: Fix missing LLMObs spans when routing requests through a litellm proxy. Proxy requests were incorrectly suppressed and resulted in empty or missing LLMObs spans. Proxy requests for OpenAI models are now always handled by the litellm integration.
profiling: A rare crash occurring when profiling asyncio code with many tasks or deep call stacks has been fixed.
serverless: AWS Lambda functions now appear under their function name as the service when DD_SERVICE is not explicitly configured. Service remapping rules configured in Datadog will now apply correctly to Lambda spans.
LLM Observability: Fixes an issue where deeply nested tool schemas in Anthropic and OpenAI integrations were not yet supported. The Anthropic and OpenAI integrations now check each tool's schema depth at extraction time. If a tool's schema exceeds the maximum allowed depth, the schema is truncated.
Code Security (IAST): This fix resolves a thread-safety issue in the IAST taint tracking context that could cause vulnerability detection to silently stop working under high concurrency in multi-threaded applications.
internal: A crash has been fixed.
CI Visibility: This fix resolves an issue where a failure response from the /search_commits endpoint caused the git metadata upload to fall back to sending the full 30-day commit history instead of aborting. This fallback could trigger cascading write load on the backend. The upload now aborts when search_commits fails, matching the behavior when the /packfile upload itself fails.
LLM Observability: Fixes multimodal OpenAI chat completion inputs being rendered as raw iterable objects in LLM Observability traces. Multimodal content parts (text, image, audio) are now properly materialized and formatted as readable text.
profiling: A rare crash that could occur post-fork in fork-based applications has been fixed.
profiling: A bug in Lock Profiling that could cause crashes when trying to access attributes of custom Lock subclasses (e.g. in Ray) has been fixed.
CI Visibility: This fix resolves an issue where pytest-xdist worker crashes (os._exit, SIGKILL, segfault) caused buffered test events to be lost. To enable eager flushing, set DD_TRACE_PARTIAL_FLUSH_MIN_SPANS=1.
profiling: Fixes lock profiling samples not appearing in the Thread Timeline view for events collected on macOS.
internal: Fix a potential internal thread leak in fork-heavy applications.
internal: This fix resolves an issue where a ModuleNotFoundError could be raised at startup in Python environments without the _ctypes extension module.
internal: A crash that could occur post-fork in fork-heavy applications has been fixed.
LLM Observability: Fixes incorrect span hierarchy in LLMObs traces when using the ddtrace SDK alongside OTel-based instrumentation (e.g. Strands Agents). OTel gen_ai spans (e.g. invoke_agent) were incorrectly appearing as siblings of their SDK parent span (e.g. call_agent) rather than being nested under it.
LLM Observability: Fixes model_name and model_provider reported on AWS Bedrock LLM spans as the model_id full model identifier value (e.g., "amazon.nova-lite-v1:0") and "amazon_bedrock" respectively. Bedrock spans' model_name and model_provider` now correctly match backend pricing data, which enables features including cost tracking.
LLM Observability: Fixes an issue where deferred tools (defer_loading=True) in Anthropic and OpenAI integrations caused LLMObs span payloads to include full tool descriptions and JSON schemas for every tool in a large catalog. Deferred tool definitions now have their description and schema stripped from span metadata, with only the tool name preserved.

Other Changes

remote config: Removes noisy warning log that was being emitted when an unsupported agent config payload was received.
ASM: Update default security rules to 1.18.0. Notably, this adds business logic event coverage for Stripe auto-instrumentation and expands WAF rule coverage (ZipSlip detection, file upload with double extension, broader header scanning, and expanded XXE detection).

v5.99.0

[fe44ddbe67] - (SEMVER-PATCH) ci: refresh Node patch versions on cross-window cache restores (Ruben Bridgewater) #8179
[5f582369a6] - (SEMVER-PATCH) refactor: use the same symbol to instrument pino across versions (Roch Devost) #8181
[2fa6f67207] - (SEMVER-PATCH) test: fix race condition in sandbox pack lock (Roch Devost) #8183
[b815a11c33] - (SEMVER-PATCH) fix(appsec): Fix regression on Stripe instrumentation (Carles Capell) #8186
[96c92d966e] - (SEMVER-PATCH) test: ignore Node http socketErrorListener leak (Ruben Bridgewater) #8177
[a9013ae0e9] - (SEMVER-PATCH) add dependabot ignore rule for rspack (Roch Devost) #8171
[927ac5e849] - (SEMVER-PATCH) test: do not hide mocha errors for nested suites (Ilyas Shabi) #8164
[01e27459f9] - (SEMVER-PATCH) fix(engines): set engines upper limit back to node 26 (Pablo Erhard) #8174
[4749d90ab6] - (SEMVER-PATCH) chore(ci) update one-pipeline (gh-worker-campaigns-3e9aa4[bot]) #8123
[2ffb5f95e1] - (SEMVER-PATCH) test(debugger): detect bracketed byteLength at runtime (Ruben Bridgewater) #8172
[abf7139eef] - (SEMVER-PATCH) fix: make DD_TRACING_ENABLED an alias of DD_APM_TRACING_ENABLED (Ruben Bridgewater) #8140
[74a0c98c05] - (SEMVER-PATCH) chore(deps): bump actions/setup-node (dependabot[bot]) #8168
[ef3fc9dd4f] - (SEMVER-PATCH) remove junit upload action (Roch Devost) #8148
[85e6201985] - (SEMVER-PATCH) chore(ci): run all system-tests scenario even if empty when running on master/sc… (Nicolas Catoni) #8067
[6089c4455d] - (SEMVER-PATCH) test: do not hide mocha errors (Ruben Bridgewater) #8147
[2d2ed5be7f] - (SEMVER-PATCH) test: make sure unknown config entries cause an error (Ruben Bridgewater) #8152
[9c8fd58d13] - (SEMVER-PATCH) fix: use references for special configurations (Ruben Bridgewater) #8149
[dee9732051] - (SEMVER-PATCH) [test optimization] support TypeScript 6 in browser integration tests (Juan Antonio Fernández de Alba) #8137
[bb5fa497ac] - (SEMVER-PATCH) fix: telemetry for test optimization auto instrument provider (Ruben Bridgewater) #8135
[0ad631d6eb] - (SEMVER-PATCH) chore(deps): bump oxc-parser from 0.126.0 to 0.127.0 in the runtime-minor-and-patch-dependencies group across 1 directory (dependabot[bot]) #8156
[04cfa94ad4] - (SEMVER-PATCH) ci: reduce all green initial delay to 5 minutes (Roch Devost) #8107
[960cf202b8] - (SEMVER-PATCH) test: exclude versions folder from nyc coverage (Roch Devost) #8160
[4daebfa9f8] - (SEMVER-MINOR) feat(profiling): observe async zlib and crypto events in timeline profiler (Attila Szegedi) #8042
[8fbe16e2b0] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 3 directories with 3 updates (dependabot[bot]) #8158
[581ce95744] - (SEMVER-PATCH) chore(deps-dev): bump the dev-minor-and-patch-dependencies group across 1 directory with 6 updates (dependabot[bot]) #8155
[0e63640d6c] - (SEMVER-PATCH) test,ci: remove test agent to reduce test runtime and flakiness (Ruben Bridgewater) #8090
[95010b2b4a] - (SEMVER-PATCH) ci,test: switch back to hoisted linker for bun (Roch Devost) #8142
[efaae8b89e] - (SEMVER-PATCH) ci: remove setup-bun workaround for unnecessary github api requests (Roch Devost) #8145
[a1d8a7f9d4] - (SEMVER-PATCH) chore(node): allow node 26 (Pablo Erhard) #8143
[59177dfdc7] - (SEMVER-PATCH) test: fix max listener warning in test optimization tests (Ruben Bridgewater) #8144
[60440710a3] - (SEMVER-MINOR) feat(openfeature): implement flag evaluation metrics (Oleksii Shmalko) #7993
[e68ef5869c] - (SEMVER-PATCH) fix(llmobs, anthropic): do not mutate anthropic create parameters (Sam Brenner) #8119
[9cc799de52] - (SEMVER-PATCH) test(profiler): fix Profiler telemetry flake (Attila Szegedi) #8114
[260299e4c1] - (SEMVER-PATCH) chore(scripts): flag unquoted ** globs in npm scripts (Ruben Bridgewater) #8061
[87a8e3a6de] - (SEMVER-MINOR) Update AppSec rules to 1.18.0 (simon-id) #8121
[f40600d3a5] - (SEMVER-PATCH) remove empty after blocks in cypress tests (Roch Devost) #8117
[8fc99d0283] - (SEMVER-PATCH) ci,test: warmup the nyc cache before running the tests (Roch Devost) #8130
[4cefbaff79] - (SEMVER-PATCH) test: reduce CI flakiness via span-race filters (Ruben Bridgewater) #8087
[27c35f7d22] - fix(llmobs): render BaseMessage in langgraph workflow I/O (Crystal Luc-Magloire) #8096
[df1f326b26] - (SEMVER-PATCH) Update junit upload action to v3.1.0 (Juan Antonio Fernández de Alba) #8118
[11f5fccc28] - (SEMVER-PATCH) fix: fix AppSec http memory retention (Ruben Bridgewater) #8029
[08322b5908] - (SEMVER-PATCH) split playwright and cypress tests in ci (Roch Devost) #8098
[24339c29f9] - (SEMVER-PATCH) [test optimization] Fix CODEOWNERS extraction logic and playwright reported CI metadata (Juan Antonio Fernández de Alba) #8113
[31624d78e1] - (SEMVER-PATCH) chore(deps): bump axios (dependabot[bot]) #8108
[55f04b6d03] - (SEMVER-PATCH) fix(debugger): skip malformed ddtags (Thomas Watson) #8094
[e1b4ea50a1] - (SEMVER-PATCH) chore: ignore .omc/ directory (Alberto Vara) #8092
[be7d03c5ec] - (SEMVER-PATCH) ci: split tracing-ubuntu job in ci (Roch Devost) #8104
[603307bc7f] - (SEMVER-PATCH) test,ci: fix langchain being tested against Node.js 18 (Ruben Bridgewater) #8103
[c0ae7192f5] - (SEMVER-PATCH) chore: replace getValueFromEnvSources calls with config usage (Ruben Bridgewater) #8084
[77c7719c2b] - (SEMVER-PATCH) chore: generate more narrow config types (Ruben Bridgewater) #8081
[a156a785b4] - (SEMVER-PATCH) fix: handle esm and cjs in rewriter (Ruben Bridgewater) #8000
[5435c2b9a6] - (SEMVER-PATCH) refactor(opentelemetry): simplify OTLP traces setup per PR #7531 review (Ruben Bridgewater) #8074
[fb34ecca9b] - (SEMVER-PATCH) test(otel): make tests robust against event emitter leak warnings (Ruben Bridgewater) #8075
[45e61496c9] - (SEMVER-PATCH) move instrumentation to their own workflow (Roch Devost) #8039
[0b022f346c] - (SEMVER-MINOR) feat(test-optimization): create final_status tag on test event for vitest (Sebastián Kay) #8064
[ab1f2a81ec] - (SEMVER-PATCH) chore(ci) : fix concurrency clause on master (Charles de Beauchesne) #8080
[9ef3cb5605] - (SEMVER-PATCH) ci: pin datadog-ci for junit upload (Juan Antonio Fernández de Alba) #8078
[5452091522] - (SEMVER-PATCH) [test optimization] fix cypress TS config auto-instrumentation and OTLP override (Juan Antonio Fernández de Alba) #8073
[ab1659c909] - (SEMVER-PATCH) ci: remove release status job (Roch Devost) #8070
[8cd7c2672e] - (SEMVER-PATCH) chore(deps): bump axios (dependabot[bot]) #8076
[073fbd98fd] - (SEMVER-PATCH) fix(ws): add missing ctx.span guard in bindAsyncStart and asyncStart (nikhil tallapureddy) #8002
[c9fcca945b] - (SEMVER-PATCH) test: use parallel steps while in sandbox creation for performance (Ruben Bridgewater) #8028
[f408684039] - (SEMVER-PATCH) fix(otel): correct sampling priority and origin propagation from trac… (Ruben Bridgewater) #8031
[9f10b0e7ff] - (SEMVER-PATCH) chore(project): upgrade typescript for docs (Sam Brenner) #8065
[237b091004] - (SEMVER-PATCH) fix(opentelemetry): implement setUrl on OTLP HTTP exporter base (Crystal Luc-Magloire) #8063
[4882a30f74] - (SEMVER-MINOR) feat(ai-guard): add ai_guard.event tag to the root span (Manuel Álvarez Álvarez) #7998
[f020db65c9] - (SEMVER-PATCH) test(light-my-request): run suite under withVersions (Ruben Bridgewater) #8052
[1efc8186ff] - (SEMVER-PATCH) chore(deps-dev): bump the dev-minor-and-patch-dependencies group across 2 directories with 7 updates (dependabot[bot]) #8066
[612d25e209] - (SEMVER-PATCH) chore(project): update sinon (Sam Brenner) #8062
[404f17549c] - (SEMVER-PATCH) test(openfeature): match printf-style log.debug args (Ruben Bridgewater) #8059
[2958b11023] - (SEMVER-MINOR) feat(llmobs): add support for Model Context Protocol SDK (Crystal Luc-Magloire) #7845
[1241a6187e] - (SEMVER-PATCH) ci: update flakiness report with a shorter summary (Roch Devost) #8038
[220434b97e] - (SEMVER-PATCH) test(debugger): merge caller snapshotConfig with defaults (Ruben Bridgewater) #8056
[71719a0ce1] - (SEMVER-PATCH) test(debugger): ignore transitional UNACKNOWLEDGED state (Ruben Bridgewater) #8057
[237e32cf86] - (SEMVER-PATCH) test(profiling): replace RegExp.escape with includes check (Ruben Bridgewater) #8058
[182cbdb974] - (SEMVER-PATCH) refactor(playwright): extract browser-side evaluate callbacks (Ruben Bridgewater) #8060
[734ceffeb1] - (SEMVER-PATCH) test(debugger): assert newly-exposed inputPath key (Ruben Bridgewater) #8055
[ee37f93b3a] - (SEMVER-PATCH) fix: stop reading deprecated req.connection.encrypted (Ruben Bridgewater) #8051
[3c6b53a878] - (SEMVER-PATCH) fix: support lazy startup log messages (Ruben Bridgewater) #8054
[0b7ef4161e] - (SEMVER-PATCH) chore(ci) update one-pipeline (gh-worker-campaigns-3e9aa4[bot]) #8043
[b2e23abbc3] - (SEMVER-PATCH) chore(deps): bump oxc-parser from 0.121.0 to 0.126.0 in the runtime-minor-and-patch-dependencies group across 1 directory (dependabot[bot]) #8047
[97ef341395] - (SEMVER-PATCH) chore(deps): bump the gh-actions-packages group across 5 directories with 2 updates (dependabot[bot]) #8049
[aff047fecd] - (SEMVER-PATCH) Rewriter compiler comment (Roch Devost) #8033
[89cd561765] - (SEMVER-PATCH) dont run serverless ci on release branch pushes too (Olivier Nzia) #8035
[1c136d23eb] - (SEMVER-PATCH) set network and http client ip on ai guard (Ilyas Shabi) #8034
[8539f8800f] - (SEMVER-PATCH) Skip mquery plugin flaky test (Carles Capell) #8040

Apr 28, 2026

v4.9.0rc1

Estimated end-of-life date, accurate to within three months: 10-2027 See the support level definitions for more information.

Upgrade Notes

ray: Adds DD_TRACE_RAY_IGNORED_ACTORS configuration to exclude specific Ray actor methods from instrumentation. Set DD_TRACE_RAY_IGNORED_ACTORS='{"ActorA": ["method1"], "ActorB": "*"}' to leave matching methods or actors uninstrumented while continuing to trace other Ray actor methods. Matching is based on actor class name only.

New Features

AI Guard: When DD_AI_GUARD_ENABLED=true is set and an ai_guard span is created during a request, the tracer now populates http.client_ip and network.client.ip on the service-entry (local root) span, mirroring the behavior used for Application Security. If AI Guard does not run during the request, no client IP tags are added. DD_TRACE_CLIENT_IP_ENABLED is ignored once AI Guard reports, and DD_TRACE_CLIENT_IP_HEADER continues to override header resolution.

Bug Fixes

CI Visibility: Fixes code coverage instrumentation on Python 3.13, 3.14, and 3.15. Resolves lost per-test line data caused by: sys.monitoring callbacks running in a snapshot context where ContextVar changes are not visible (Python 3.14+); empty modules emitting no LINE events (Python 3.13+); and ProcessPoolExecutor child coverage not being propagated to the parent context. Also fixes a stale-data bug where child process executable lines could inflate coverage denominators after stop_coverage() was called before join().

django: API endpoint discovery now supports Django sub-applications mounted with django.urls.include(...). Endpoints are reported with their full URL path including the parent prefix — for example, a view served at /api/users/ is now reported as /api/users/ instead of losing the /api/ prefix.

django: API endpoint discovery now reports the correct HTTP methods for views decorated with @require_http_methods combined with another decorator such as @csrf_exempt; the declared methods are reported instead of a generic wildcard entry.

LLM Observability: This fix resolves an issue where running an experiment with a dataset whose records had null metadata caused the summary evaluator to crash with a TypeError while preparing evaluator inputs.

LLM Observability: Changes the default model_name and model_provider of LLM and embedding spans from custom to unknown if not provided or empty. This applies to both auto-instrumented spans and manual instrumentation via LLMObs.llm() / LLMObs.embedding() and the @llm / @embedding decorators.

profiling: A crash that could happen in child processes after fork has been fixed.

profiling: A rare crash caused by the memory allocation profiler has been fixed.

RemoteConfig: Fixed an issue where deleted remote configurations were not applied, causing stale settings to persist.

wsgi: This fix resolves an issue where the http.url tag on inbound request spans contained the WSGI mount prefix twice (for example /admin/admin/users instead of /admin/users) when the application was served behind werkzeug.middleware.dispatcher.DispatcherMiddleware or any other in-process mount that preserves the original request URI in RAW_URI / REQUEST_URI while also setting SCRIPT_NAME.

profiling: A rare crash happening on systems with small stack sizes when profiling asyncio code has been fixed.

Apr 23, 2026

v4.8.0rc5

Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.

Upgrade Notes

claude_agent_sdk: Tool span resource names have changed from the tool name (e.g. Read, Bash) to claude_agent_sdk.tool. The specific tool name is still available in the span name (e.g. claude_agent_sdk.tool.Read). Users relying on tool resource names should update them accordingly.
ray: Adds DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED (default: False) configuration to control Ray submission tracing. Set DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED=true to trace task.submit and actor_method.submit spans. Leave it unset to trace only execution spans. See Ray integration documentation for more details.
ray: ray.job.submit spans are removed. Ray job submission outcome is now reported on the existing ray.job span through ray.job.submit_status.

Deprecation Notes

Tracing: DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED is deprecated and will be removed in 5.0.0. Use DD_TRACE_INFERRED_SPANS_ENABLED instead. The old environment variable continues to work but emits a DDTraceDeprecationWarning when set.
tracing: The pin parameter in ddtrace.contrib.dbapi.TracedConnection, ddtrace.contrib.dbapi.TracedCursor, and ddtrace.contrib.dbapi_async.TracedAsyncConnection is deprecated and will be removed in version 5.0.0. To manage configuration of DB tracing please use integration configuration and environment variables.
LLM Observability: Removes support for the RAGAS integration. As an alternative, if you have RAGAS evaluations, you can manually submit these evaluation results. See LLM Observability external evaluation documentation for more information.

New Features

AI Guard: Add DD_AI_GUARD_BLOCK environment variable. Defaults to True, which means the blocking behavior configured in the Datadog AI Guard UI (in-app) will be honored. Set to False to force monitor-only mode locally: evaluations are still performed but AIGuardAbortError is never raised, regardless of the in-app blocking setting.
AI Guard response objects now include a dict field tag_probs with the probabilities for each tag.
CI Visibility: Adds Bazel offline execution support with two modes: manifest mode (DD_TEST_OPTIMIZATION_MANIFEST_FILE), which reads settings and test data from pre-fetched cache files without network access; and payload-files mode (DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES), which writes test event, coverage, and telemetry payloads as JSON files instead of sending HTTP requests. Both modes can be used independently or together.
LLM Observability: Captures individual LLM spans for each Claude model turn within a Claude Agent SDK session. Each LLM span captures the input messages, output messages, model name, and token usage metrics (for claude_agent_sdk >= 0.1.49).
AAP: This adds Application Security support for FastAPI and Starlette applications using mounted sub-applications (via app.mount()). WAF evaluation, path parameter extraction, API endpoint discovery, and http.route reporting now correctly account for mount prefixes in sub-application routing.
google_cloud_pubsub: This adds tracing for Google Cloud Pub/Sub admin operations on topic, subscription, snapshot, and schema management methods.
google_cloud_pubsub: Adds support for Google Cloud Pub/Sub push subscriptions. When a push subscription delivers a message via HTTP, the integration now creates an inferred gcp.pubsub.receive span that captures subscription and message metadata. Use DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS to control whether the inferred span becomes a child of the producer trace or starts a new trace with the producer context attached as a span link (default: False).
LLM Observability: Add ExperimentRun.as_dataframe() to convert experiment run results into a pandas.DataFrame with a two-level MultiIndex on columns. Each top-level group (input, output, expected_output, evaluations, metadata, error, span_id, trace_id) maps to the first index level. Dict-valued fields are flattened one level deep; scalar fields use an empty string as the sub-column name. Each evaluator gets its own column containing the full evaluation result dict. Requires pandas to be installed (pip install pandas).
LLM Observability: Adds an eval_scope parameter to LLMObs.submit_evaluation() (one of "span" (default) or "trace"). Use eval_scope="trace" to associate an evaluation with an entire trace by passing the root span context.
LLM Observability: Adds LLMObs.get_spans() to retrieve span events from the Datadog platform API (GET /api/v2/llm-obs/v1/spans/events). Supports filtering by trace ID, span ID, span kind, span name, ML app, tags, and time range. Results are auto-paginated. Requires DD_API_KEY and DD_APP_KEY.
profiling: Profiles generated from fork-based servers now include a process_type tag with the value main or worker.
tracing: Support for making the default span name for @tracer.wrap include the class name has been added. For now, this is opt-in and can be enabled by setting DD_TRACE_WRAP_SPAN_NAME_INCLUDE_CLASS=true. The new naming will become the default in the next major release.
llmobs: Adds support for enabling and disabling LLMObs via Remote Configuration.
mysql: This introduces tracing support for mysql.connector.aio.connect in the MySQL integration.
profiling: Thread sub-sampling is now supported. This allows to set a maximum number of threads to capture stacks for at each sampling interval. This can be used to reduce the CPU overhead of the Stack Profiler.
llama_index: Adds APM tracing and LLM Observability support for llama-index-core>=0.11.0. Traces LLM calls, query engines, retrievers, embeddings, and agents. See the llama_index documentation for more information.
ASM: Adds a LiteLLM proxy guardrail integration for Datadog AI Guard. The ddtrace.appsec.ai_guard.integrations.litellm.DatadogAIGuardGuardrail class can be registered as a custom guardrail in the LiteLLM proxy to evaluate requests and responses against AI Guard security policies. Requires the LiteLLM proxy guardrails API v2 available since litellm>=1.46.1.
azure_cosmos: Add tracing support for Azure CosmosDB. This integration traces CRUD operations on CosmosDB databases, containers, and items.
LLM Observability: Introduces a decorator tag to LLM Observability spans that are traced by a function decorator.
CI Visibility: adds automatic log correlation and submission so that test logs appear alongside their corresponding test run in Datadog. Set DD_AGENTLESS_LOG_SUBMISSION_ENABLED=true for agentless setups, or DD_LOGS_INJECTION=true when using the Datadog Agent.
tracing: Adds support for exporting traces in OTLP HTTP/JSON format via libdatadog. Set OTEL_TRACES_EXPORTER=otlp to send spans to an OTLP endpoint instead of the Datadog Agent.

Example:

from pydantic_evals.evaluators import ReportEvaluator
from pydantic_evals.evaluators import ReportEvaluatorContext
from pydantic_evals.reporting.analyses import ScalarResult

from ddtrace.llmobs import LLMObs

dataset = LLMObs.create_dataset(
    dataset_name="<DATASET_NAME>",
    description="<DATASET_DESCRIPTION>",
    records=[RECORD_1, RECORD_2, RECORD_3, ...]
)

class TotalCasesEvaluator(ReportEvaluator):
    def evaluate(self, ctx: ReportEvaluatorContext) -> ScalarResult:
        return ScalarResult(
            title='Total Cases',
            value=len(ctx.report.cases),
            unit='cases',
        )

def my_task(input_data, config):
    return input_data["output"]

equals_expected = EqualsExpected()
summary_evaluator = TotalCasesEvaluator()

experiment = LLMObs.experiment(
    name="<EXPERIMENT_NAME>",
    task=my_task, 
    dataset=dataset,
    evaluators=[equals_expected],
    summary_evaluators=[summary_evaluator],
    description="<EXPERIMENT_DESCRIPTION>."
)

result = experiment.run()

Bug Fixes

CI visibility: This fix resolves issues where CI provider metadata could omit pull request base branch and head commit details or report incorrect pull request values for some providers.
AAP: This fix resolves an issue where Application and API Protection (AAP) was incorrectly reported as an enabled product in internal telemetry for all services by default. Previously, registering remote configuration listeners caused AAP to be reported as activated even when it was not actually enabled. This had no impact on customers as it only affected internal telemetry data. AAP is now only reported as activated when it is explicitly enabled or enabled through remote configuration.
asgi: Fixed an issue caused network.client.ip and http.client_ip span tags being missing for FastAPI.
iast: A crash has been fixed.
lambda: Fixes a spurious Unable to create shared memory warning on every AWS Lambda cold start.
LLM Observability: Fixes an issue where an APM_TRACING remote configuration payload that did not include an llmobs section would disable LLM Observability on services where it had been enabled programmatically via LLMObs.enable(). Services that enabled LLM Observability via the DD_LLMOBS_ENABLED environment variable were unaffected. The handler now only changes LLM Observability state when the remote configuration payload explicitly carries an llmobs.enabled directive.
LLM Observability: Fixes a circular import in ddtrace.llmobs._writer when anthropic, openai, and botocore is installed.
Prevent potential crashes when the client library fails to restart a worker thread due to hitting a system resource limit.
internal: This fix resolves an issue where reading unknown attributes from ddtrace.internal.process_tags caused a KeyError instead of raising an AttributeError.
rq: Fixes compatibility with RQ 2.0. Replaces the removed Job.get_id() with the job.id property, and handles Job.get_status() now raising InvalidJobOperation for expired jobs (e.g. result_ttl=0) instead of returning None. #16682
tornado: Fixes an issue where routes inside a nested Tornado application were matched in reverse declaration order, causing a catch-all pattern to win over a more-specific route defined before it. This resulted in incorrect http.route tags on spans.
tornado: The http.route tag is now populated for routes whose regex cannot be reversed by Tornado (e.g. patterns containing non-capturing groups such as (?:a|b)). Capturing groups are still rendered as %s, consistent with Tornado's own route format, while non-capturing constructs are kept verbatim.
telemetry: This fix resolves an issue where unhandled exceptions raised by importlib.metadata during interpreter shutdown (for example, when Gunicorn workers exit uncleanly after a failed startup) caused update_imported_dependencies to surface errors through sys.excepthook. Failures while discovering dependencies for the app-dependencies-loaded telemetry payload are now logged at debug level and swallowed so they no longer propagate out of the dependency-reporting path.
profiling: Fixes noise caused by the profiler attempting to load its native module even when profiling was disabled,
profiling: A race condition which could make asyncio code raise exceptions at exit has been fixed.
remote_config: This fix resolves an issue where brief Datadog Agent connection errors could drop Remote Configuration polls, causing products such as Dynamic Instrumentation to temporarily appear disabled.
LLM Observability: Change the default model_provider and model_name to "unknown" from "custom" when a model did not match any known provider prefix in the Google GenAI, VertexAI, and Google ADK integrations.
LLM Observability: This fix resolves tracing issues for pydantic-ai >= 1.63.0 where tool spans and agent instructions were not being properly captured. This fix adds tracing to the ToolManager.execute_tool_call method for newer versions of the library to resolve this issue.
celery: remove unnecessary warning log about missing span when using Task.replace().
django: Fixes RuntimeError: coroutine ignored GeneratorExit that occurred under ASGI with async views and async middleware hooks on Python 3.13+. Async view methods and middleware hooks are now correctly detected and awaited instead of being wrapped with sync bytecode wrappers.
Code Security (IAST): Fixes a missing return in the IAST taint tracking add_aspect native function that caused redundant work when only the right operand of a string concatenation was tainted.
openai: Fixes async streaming spans never being finished when using AsyncAPIResponse (e.g. responses.create(stream=True)). The sync handle_request hook called resp.parse() without awaiting the coroutine, preventing the stream from being wrapped in TracedAsyncStream. This caused disconnected LLM Observability traces for streamed sub-agent calls via the OpenAI Agents SDK.
Fixed a race condition with internal periodic threads that could have caused a rare crash when forking.
ray: This fix resolves an issue where Ray integration spans could use an incorrect service name when the Ray job name was set after instrumentation initialization.
tracing: Fixes the svc.auto process tag attribution logic. The tag now correctly reflects the auto-detected service name derived from the script or module entrypoint, matching the service name the tracer would assign to spans.
Fixes an issue where internal background threads could cause crashes or instability in applications that fork (e.g. Gunicorn, uWSGI) or during Python shutdown. Affected applications could experience intermittent crashes or hangs on exit.
tracing: This fix resolves an issue where applications started with python -m <module> could report entrypoint.name as -m in process tags.
apm: Fixed an issue where network.client.ip and http.client_ip span tags were missing when client IP collection was enabled and request had no headers.
litellm: Fix missing LLMObs spans when routing requests through a litellm proxy. Proxy requests were incorrectly suppressed and resulted in empty or missing LLMObs spans. Proxy requests for OpenAI models are now always handled by the litellm integration.
profiling: A rare crash occurring when profiling asyncio code with many tasks or deep call stacks has been fixed.
serverless: AWS Lambda functions now appear under their function name as the service when DD_SERVICE is not explicitly configured. Service remapping rules configured in Datadog will now apply correctly to Lambda spans.
LLM Observability: Fixes an issue where deeply nested tool schemas in Anthropic and OpenAI integrations were not yet supported. The Anthropic and OpenAI integrations now check each tool's schema depth at extraction time. If a tool's schema exceeds the maximum allowed depth, the schema is truncated.
Code Security (IAST): This fix resolves a thread-safety issue in the IAST taint tracking context that could cause vulnerability detection to silently stop working under high concurrency in multi-threaded applications.
internal: A crash has been fixed.
CI Visibility: This fix resolves an issue where a failure response from the /search_commits endpoint caused the git metadata upload to fall back to sending the full 30-day commit history instead of aborting. This fallback could trigger cascading write load on the backend. The upload now aborts when search_commits fails, matching the behavior when the /packfile upload itself fails.
LLM Observability: Fixes multimodal OpenAI chat completion inputs being rendered as raw iterable objects in LLM Observability traces. Multimodal content parts (text, image, audio) are now properly materialized and formatted as readable text.
profiling: A rare crash that could occur post-fork in fork-based applications has been fixed.
profiling: A bug in Lock Profiling that could cause crashes when trying to access attributes of custom Lock subclasses (e.g. in Ray) has been fixed.
CI Visibility: This fix resolves an issue where pytest-xdist worker crashes (os._exit, SIGKILL, segfault) caused buffered test events to be lost. To enable eager flushing, set DD_TRACE_PARTIAL_FLUSH_MIN_SPANS=1.
profiling: Fixes lock profiling samples not appearing in the Thread Timeline view for events collected on macOS.
internal: Fix a potential internal thread leak in fork-heavy applications.
internal: This fix resolves an issue where a ModuleNotFoundError could be raised at startup in Python environments without the _ctypes extension module.
internal: A crash that could occur post-fork in fork-heavy applications has been fixed.
LLM Observability: Fixes incorrect span hierarchy in LLMObs traces when using the ddtrace SDK alongside OTel-based instrumentation (e.g. Strands Agents). OTel gen_ai spans (e.g. invoke_agent) were incorrectly appearing as siblings of their SDK parent span (e.g. call_agent) rather than being nested under it.
LLM Observability: Fixes model_name and model_provider reported on AWS Bedrock LLM spans as the model_id full model identifier value (e.g., "amazon.nova-lite-v1:0") and "amazon_bedrock" respectively. Bedrock spans' model_name and model_provider` now correctly match backend pricing data, which enables features including cost tracking.
LLM Observability: Fixes an issue where deferred tools (defer_loading=True) in Anthropic and OpenAI integrations caused LLMObs span payloads to include full tool descriptions and JSON schemas for every tool in a large catalog. Deferred tool definitions now have their description and schema stripped from span metadata, with only the tool name preserved.

Other Changes

remote config: Removes noisy warning log that was being emitted when an unsupported agent config payload was received.
ASM: Update default security rules to 1.18.0. Notably, this adds business logic event coverage for Stripe auto-instrumentation and expands WAF rule coverage (ZipSlip detection, file upload with double extension, broader header scanning, and expanded XXE detection).

Apr 21, 2026

v5.98.0