Agent

Prelude

Release on: 2025-05-06

• Please refer to the 7.65.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Covers more gRPC status code values sent by tracer libraries
• Aggregate APM stats payloads by gRPC code

New Features

• The Agent MSI for Windows now installs the .NET APM Instrumentation libraries and automatically configures APM Instrumentation for IIS. To enable this feature, specify the following properties in the MSI command line:

  <span class="title-ref">DD_APM_INSTRUMENTATION_ENABLED=iis DD_APM_INSTRUMENTATION_LIBRARIES=dotnet:3</span>

  Requires IIS 10.0 or later. For more information see [Single Step APM Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/single-step-apm/).

• APM: Added new OpenLineage event proxy endpoint.

• A new implementation of auto multi-line detection for logs is now available. This new implementation is more flexible and powerful than the previous one. It now supports arbitrary timestamps, continuous matching, and more flexible configuration. It will be enabled automatically when logs_config.auto_multi_line_detection is set to true. You can opt out of the new implementation by setting logs_config.force_auto_multi_line_detection_v1 to true.

• APM: Container-based primary tags are now available by default.

Enhancement Notes

• Add a new host tag called orch_cluster_id that is set to the Kubernetes cluster ID generated by the cluster-agent. This is added only for the hosts that are part of a Kubernetes cluster. This tag is also used on every cluster-level resources like <span class="title-ref">kubernetes_state.*</span> metrics, kubernetes events, and orchestrator level resources.

• Add admission.datadoghq.com/apm-inject.debug: "true" annotation to inject DD_APM_INSTRUMENTATION_DEBUG=true, DD_TRACE_STARTUP_LOGS=true and DD_TRACE_DEBUG=true to an application container. This is useful for debugging Single Step Instrumentation or tracer issues in Kubernetes environments.

• Add metrics origins for Sonatype Nexus, Silverstripe CMS and Anecdote integrations.

• Add the <span class="title-ref">propagate_agent_tags setting</span>. When set to <span class="title-ref">true</span>, the tags from the Agent host are added to the check's tag for all instances.

• Agents are now built with Go 1.23.7.

• Agents are now built with Go 1.23.8.

• Allow integrations to self-declare as HA Supported.

• Improved trace context creation from Step Function execution context:

  • Now utilizes <span class="title-ref">State.RetryCount</span> and <span class="title-ref">Execution.RedriveCount</span> to generate parent IDs deterministically, preventing collisions with retry spans.
  • Supports multi-level trace merging, allowing an arbitrary number of Lambda and Step Function traces to be linked together while maintaining root context.

  This update brings feature parity with Node and Python Lambda layers in the Universal runtimes.

• The hostname subcommand for the Agent now tries to reach out to the running Agent, and falls back to computing the hostname locally if the Agent is not running. You can use the <span class="title-ref">--local</span> option to force the local computation.

• Cisco SD-WAN: improve memory usage.

• Adds a <span class="title-ref">metadata_ip_resolution_from_hostname</span> config option to use hostname DNS resolution as the preferred method to compute the host IP address

• Metrics are now sent over HTTP/2 when possible. A slight change to the connection handling to take full advantage of this means multiple requests can be in flight at any one time. This can be configured with the forwarder_max_concurrent_requests option, default is 10.

• Added configurations network_path.collector.source_excludes and network_path.collector.dest_excludes, which allow ignoring CIDR ranges in the Network Path collector.

• Added support for querying the pod list through the API server. This feature, enabled by setting <span class="title-ref">kubeletUseApiServer</span> to <span class="title-ref">true</span>, allows the Agent to retrieve pod metadata directly from the API server instead of the kubelet. This resolves issues when direct access to the kubelet <span class="title-ref">/pods</span> endpoint is restricted. The agent will continue to query the kubelet for <span class="title-ref">/metrics</span> and <span class="title-ref">/stats/summary</span>. Must add <span class="title-ref">pods</span> to Agent cluster role.

• Process checks now run in the core Agent by default on Linux. Process Agent will only run if needed for other configured features.

• Include <span class="title-ref">container_instance_arn</span> in ECS Task payloads.

• Include APIVersion and Kind to Kubernetes manifests.

• The Python runtime is now loaded when the first check is loaded, rather than when the agent starts. This will eventually avoid having to load Python when no Python check is enabled. This change can be reverted by setting python_lazy_loading: false in your configuration.

• When the API key can't be accessed in serverless, it is now logged as an error instead of debug.

• Added FIPS compliance support for Lambda Extension GovCloud customers. The Lambda Extension now uses FIPS-enabled endpoints for AWS KMS and Secrets Manager when running in GovCloud regions.

• APM: Return the correct content-type header when rejecting trace payloads.

• APM: Expose an "obfuscation_version" value via the /info endpoint. Accept a new header "Datadog-Obfuscation-Version" for incoming stats payloads; if any non-zero value is set, the trace-agent will not attempt to obfuscate these payloads as they have already been obfuscated by the tracer.

• Adds functionality to construct a DD span from <span class="title-ref">datadog.</span> attributes on an incoming span. By default, it checks for a span attribute starting with <span class="title-ref">datadog.</span>; if present, it uses this to compute the corresponding Datadog field (e.g., <span class="title-ref">datadog.service</span> is used to set <span class="title-ref">ddspan.Service</span>). This will override other sources for the same field (e.g., if both <span class="title-ref">datadog.service</span> and <span class="title-ref">service.name</span> are present, <span class="title-ref">datadog.service</span> is used). By default, if a field is missing, it will be recomputed (e.g., if there's no <span class="title-ref">datadog.env</span>, it will look for <span class="title-ref">env</span> in <span class="title-ref">deployment.environment</span>). However, if the config option <span class="title-ref">otlp_config.traces.ignore_missing_datadog_fields</span> is specified, the field will not be recomputed. This option allows users to explicitly specify blank fields if they desire (e.g., set <span class="title-ref">datadog.env = ""</span>).

  The following functionality is removed: if both <span class="title-ref">http.response.status_code</span> and <span class="title-ref">http.status_code</span> were present in the span attributes, the former was preferred. Similarly, <span class="title-ref">http.request.method</span> was preferred over <span class="title-ref">http.method</span>. In this release, the key encountered first is the one used.

Security Notes

• The OPENSSL_CONF and OPENSSL_MODULES environment variables are now set when running the <span class="title-ref">datadog-fips-agent</span> in FIPS mode. This allows applications to use the OpenSSL configuration and modules configured with the Agent installation.

Bug Fixes

• Fixes bugs in the analyze-logs subcommand, and validates the logs config for invalid syntax.
• Fixes a race condition introduced in PR #33521 around the PythonHome getting set in an init hook.
• Fix the NVIDIA Jetson check to support parsing the tegrastats output when there are multiple Graphics Processing Clusters (GPCs). In that case, the metric nvidia.jetson.gpu.freq is emitted for each GPC and tagged with gpc:0, gpc:1, and so on.
• Fixes issue where the CRI-O image status endpoint would be queried when <span class="title-ref">container_image.enabled</span> was set to <span class="title-ref">false</span>.
• CWS: Fixed a bug that prevented more than one runtime security rule from being disabled, and caused incorrect policy information to be reported in ruleset loaded events.
• Fixed a spinlock corruption in the Windows driver ddprocmon.sys that can cause system hangs and memory corruption when system-probe.exe is shutting down.
• Fixes an issue where the system-probe would spuriously log the error netlink conntracker requires NET_ADMIN capability on the Fargate CNM preview.
• Cisco SD-WAN: fix an issue that could lead to incorrect metrics tagging.
• Fix the shebang of the <span class="title-ref">gstatus</span> binary to use the embedded Python environment.
• APM: fix issue where a new span field for span events wasn't properly serialized when received in MessagePack format.
• Environment variables that the kubelet collector receives and are not explicitly defined in plaintext will now be ignored.

Other Notes

• PHP support is enabled by default in the Cluster Agent
• Metrics Origin entries for Infiniband and Celery integrations
• Update OpenSSL from 3.4.0 to 3.4.1.
• PHP support for Single Step Instrumentation is now installed by default.
• Add metric origin for the Velero integration.
• Adds the Datadog Installer service to the Agent MSI. The Datadog Installer service will start on boot but will exit unless the Remote Upgrades preview is enabled.

Datadog Cluster Agent

Prelude

Released on: 2025-05-06 Pinned to datadog-agent v7.65.0: CHANGELOG.

New Features

• [PREVIEW] Add support for mounting Datadog CSI volumes instead of hostpath volumes in the admission controller config webhook for sharing DogStatsD and APM UDS sockets with user applications. This requires the Datadog CSI driver to be installed and running on the cluster.

Enhancement Notes

• Collect terminated Kubernetes resources.