Release on: 2025-07-02
1.23.10.Fixes invalid logs compression error in DDOT, sets DDOT logs compression to gzip.
Permissions are no longer applied recursively to the Datadog installer data directory on Windows.
This fixes an issue that causes Agent updates to restrict access to the .NET APM tracer libraries that were previously installed by the DD_APM_INSTRUMENTATION_LIBRARIES option, preventing them from being loaded by IIS.
Fixes an issue in Install-Datadog.ps1 that could malform datadog.yaml and cause the Agent to fail to start. When datadog.yaml does not end with a new line the remote_updates option was incorrectly appended to the last line in the file instead of to a new line.
Released on: 2025-07-02 Pinned to datadog-agent v7.67.1: CHANGELOG.
Release on: 2025-06-18
*.datadoghq.com hosts are also valid for connections to *.datadoghq.com. (with an additional trailing dot) hosts.In the Systemd core check add the option to use regular expressions to select units to monitor.
Added a new variable <span class="title-ref">extra_dbm</span> to Aurora Autodiscovery. This variable matches the value of the <span class="title-ref">datadoghq.com/dbm</span> tag on the database instance.
Released a new <span class="title-ref">ddot-collector</span> container image that packages the [Datadog Distribution of OpenTelemetry Collector](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/).
The MacOS Agent now supports the Network Path feature by including system-probe and the <span class="title-ref">traceroute</span> module.
Windows: Added the Windows Certificate Store integration to monitor the expiration of certificates in the local machine certificate store.
Introducing a new setting <span class="title-ref">collect_ec2_instance_info</span> to collect basic EC2 instance information as host tags. This reproduces some of the behaviors of the AWS integration for users that can't enable it. The [AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services/) should still be use whenever possible as it offers a better and more in depth integration.
Feature parity between Python disk check and Go disk check. The new version of the disk check is disabled by default for now, but it will be enabled later on. It can be enabled by setting use_diskv2_check: true in your configuration.
Pretty printed/multi-line JSON messages are now aggregated into a single line when auto multiline detection is enabled. This ensures the log is treated a structured log when processed by Datadog. Aggregation can be disabled by setting logs_config.auto_multi_line.enable_json_aggregation to false.
Add a networkv2 check that is a port of the Python network check to Go. This version is disabled by default but can be enabled with use_networkv2_check in your configuration.
Adds a new diagnostic check that identifies firewall rules blocking SNMP traps and NetFlow traffic on Windows systems.
Enables support for NetPath on Windows client versions. To enable set tcp_method to syn_socket in the network_path.d configuration file.
SNMP integration now defaults to use the Core loader instead of Python.
A new core check, agentprofiling, has been introduced to automatically generate a flare with profiles when the Datadog Agent exceeds a configured memory or CPU usage threshold. When a valid config file is set, the Agent monitors its own memory and CPU usage and, upon crossing the threshold, generates a flare with profiles that is either saved locally or sent to a Zendesk ticket.
This enhancement simplifies troubleshooting memory-related issues that are difficult to reproduce or time, allowing users to passively capture critical memory data without manual intervention.
ad_allowed_env_vars, which allows users to restrict which environment variables can be resolved in Autodiscovery check configurations. When set, only the environment variables listed are resolved.ad_disable_env_var_resolution, which lets users disable environment variable resolution in Autodiscovery check configurations.1.23.9.logs.encoded_bytes_sent telemetry metric, enabling aggregation and monitoring of log compression type usage during rollout.process_config.remote_tagger config entry is removed.account_id tag.DD_SITE to prevent generating as many DNS queries as there are entries in the search section of the /etc/resolv.conf` file. If an intake full URL is explicitly set with add_url`` parameter, then, the parameter is used as-is and using FQDNs or not remains a user choice.active_session_history: true.Released on: 2025-06-18 Pinned to datadog-agent v7.67.0: CHANGELOG.
Release on: 2025-06-03
Released on: 2025-06-03 Pinned to datadog-agent v7.66.1: CHANGELOG.
This version (and previous) of the Datadog Agent is not compatible with Kubernetes 1.33+ versions due to the Feature Gate InPlacePodVerticalScaling that became enabled by default. This flag modifies the kubelet /pods output preventing the correct behaviour of the Datadog Agent. The recommendation is to upgrade to Agent v7.66.1, which is fully compatible with the latest (and previous) Kubernetes versions. More details can be found in this issue.
Release on: 2025-05-22
DDAGENTUSER_PASSWORD option when upgrading to 7.66 or later. For more information see the features release notes.disable_operation_and_resource_name_logic_v2 in DD_APM_FEATURES that replaces enable_operation_and_resource_name_logic_v2. The new operation name logic for OTLP is now opt-out instead of opt-in.Added a new WLAN check that monitors the Wi-Fi interface on the host system. This check is only available for macOS systems.
Fleet Automation now supports remote upgrades when using a custom Agent username and password on Windows.
Windows stores the password as an encrypted LSA local private data object that is only accessible to local Administrators. Windows Service Manager stores service account passwords in the same location. For more information, see the Microsoft documentation on Storing Private Data and Private Data Objects.
Uninstalling the Agent removes the encrypted password from the LSA.
To avoid providing and manually managing the account password, consider using a Group Managed Service Account (gMSA). For more information, see Installing the Agent with a gMSA account.
Adds support for persisting of non-core integrations during Agent upgrades on Windows platforms. To disable, set the INSTALL_PYTHON_THIRD_PARTY_DEPS="0" property during the installation of the MSI.
adds the ability for the Agent to tail logs via the kubelet's API.
Support multiple authentication methods for a subnet in network devices autodiscovery.
Use cdpCacheSecondaryMgmtAddr and cdpCacheAddress for CDP topology links in case cdpCachePrimaryMgmtAddr is empty or of an unsupported type.
Enable language detection via tracers metadata
| separator to the DogstatsD debug table (from agent dogstatsd-stats) when not requesting JSON output. This allows the table to render properly in markdown format.DDAGENTUSER_PASSWORD value when it is a local account. Previously, if the provided password did not match the account password, the Agent would fail to start.pod_collection_mode: node_kubelet, the Agent reported incorrect values for the kubernetes_state.container.status_report.count.waiting metric.auth to the API telemetry metrics. This tag is used to evaluate the impact of reworking the authentication system for inter-process communication.Released on: 2025-05-22 Pinned to datadog-agent v7.66.0: CHANGELOG.
valueFrom as an alternative to value in ddTraceConfigs. This enables dynamic, user-defined and label based value propagation to the tracing SDKs, like DD_SERVICE.DD_SERVICE is not consistent between containers and init containers.istio-proxy container. This fixes an issue with Kubernetes-native sidecars and the istio service mesh where a standard sidecar is moved to be the first init container by istio after it was mutated by auto-instrumentation.Release on: 2025-05-13
datadogagent service now also restarts the Datadog Installer service to ensure configuration changes take effect.Released on: 2025-05-13 Pinned to datadog-agent v7.65.2: CHANGELOG.
Release on: 2025-05-08
Released on: 2025-05-08 Pinned to datadog-agent v7.65.1: CHANGELOG.
Release on: 2025-05-06
The Agent MSI for Windows now installs the .NET APM Instrumentation libraries and automatically configures APM Instrumentation for IIS. To enable this feature, specify the following properties in the MSI command line:
<span class="title-ref">DD_APM_INSTRUMENTATION_ENABLED=iis DD_APM_INSTRUMENTATION_LIBRARIES=dotnet:3</span>
Requires IIS 10.0 or later. For more information see [Single Step APM Instrumentation](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/single-step-apm/).
APM: Added new OpenLineage event proxy endpoint.
A new implementation of auto multi-line detection for logs is now available. This new implementation is more flexible and powerful than the previous one. It now supports arbitrary timestamps, continuous matching, and more flexible configuration. It will be enabled automatically when logs_config.auto_multi_line_detection is set to true. You can opt out of the new implementation by setting logs_config.force_auto_multi_line_detection_v1 to true.
APM: Container-based primary tags are now available by default.
Add a new host tag called orch_cluster_id that is set to the Kubernetes cluster ID generated by the cluster-agent. This is added only for the hosts that are part of a Kubernetes cluster. This tag is also used on every cluster-level resources like <span class="title-ref">kubernetes_state.*</span> metrics, kubernetes events, and orchestrator level resources.
Add admission.datadoghq.com/apm-inject.debug: "true" annotation to inject DD_APM_INSTRUMENTATION_DEBUG=true, DD_TRACE_STARTUP_LOGS=true and DD_TRACE_DEBUG=true to an application container. This is useful for debugging Single Step Instrumentation or tracer issues in Kubernetes environments.
Add metrics origins for Sonatype Nexus, Silverstripe CMS and Anecdote integrations.
Add the <span class="title-ref">propagate_agent_tags setting</span>. When set to <span class="title-ref">true</span>, the tags from the Agent host are added to the check's tag for all instances.
Agents are now built with Go 1.23.7.
Agents are now built with Go 1.23.8.
Allow integrations to self-declare as HA Supported.
Improved trace context creation from Step Function execution context:
This update brings feature parity with Node and Python Lambda layers in the Universal runtimes.
The hostname subcommand for the Agent now tries to reach out to the running Agent, and falls back to computing the hostname locally if the Agent is not running. You can use the <span class="title-ref">--local</span> option to force the local computation.
Cisco SD-WAN: improve memory usage.
Adds a <span class="title-ref">metadata_ip_resolution_from_hostname</span> config option to use hostname DNS resolution as the preferred method to compute the host IP address
Metrics are now sent over HTTP/2 when possible. A slight change to the connection handling to take full advantage of this means multiple requests can be in flight at any one time. This can be configured with the forwarder_max_concurrent_requests option, default is 10.
Added configurations network_path.collector.source_excludes and network_path.collector.dest_excludes, which allow ignoring CIDR ranges in the Network Path collector.
Added support for querying the pod list through the API server. This feature, enabled by setting <span class="title-ref">kubeletUseApiServer</span> to <span class="title-ref">true</span>, allows the Agent to retrieve pod metadata directly from the API server instead of the kubelet. This resolves issues when direct access to the kubelet <span class="title-ref">/pods</span> endpoint is restricted. The agent will continue to query the kubelet for <span class="title-ref">/metrics</span> and <span class="title-ref">/stats/summary</span>. Must add <span class="title-ref">pods</span> to Agent cluster role.
Process checks now run in the core Agent by default on Linux. Process Agent will only run if needed for other configured features.
Include <span class="title-ref">container_instance_arn</span> in ECS Task payloads.
Include APIVersion and Kind to Kubernetes manifests.
The Python runtime is now loaded when the first check is loaded, rather than when the agent starts. This will eventually avoid having to load Python when no Python check is enabled. This change can be reverted by setting python_lazy_loading: false in your configuration.
When the API key can't be accessed in serverless, it is now logged as an error instead of debug.
Added FIPS compliance support for Lambda Extension GovCloud customers. The Lambda Extension now uses FIPS-enabled endpoints for AWS KMS and Secrets Manager when running in GovCloud regions.
APM: Return the correct content-type header when rejecting trace payloads.
APM: Expose an "obfuscation_version" value via the /info endpoint. Accept a new header "Datadog-Obfuscation-Version" for incoming stats payloads; if any non-zero value is set, the trace-agent will not attempt to obfuscate these payloads as they have already been obfuscated by the tracer.
Adds functionality to construct a DD span from <span class="title-ref">datadog.</span> attributes on an incoming span. By default, it checks for a span attribute starting with <span class="title-ref">datadog.</span>; if present, it uses this to compute the corresponding Datadog field (e.g., <span class="title-ref">datadog.service</span> is used to set <span class="title-ref">ddspan.Service</span>). This will override other sources for the same field (e.g., if both <span class="title-ref">datadog.service</span> and <span class="title-ref">service.name</span> are present, <span class="title-ref">datadog.service</span> is used). By default, if a field is missing, it will be recomputed (e.g., if there's no <span class="title-ref">datadog.env</span>, it will look for <span class="title-ref">env</span> in <span class="title-ref">deployment.environment</span>). However, if the config option <span class="title-ref">otlp_config.traces.ignore_missing_datadog_fields</span> is specified, the field will not be recomputed. This option allows users to explicitly specify blank fields if they desire (e.g., set <span class="title-ref">datadog.env = ""</span>).
The following functionality is removed: if both <span class="title-ref">http.response.status_code</span> and <span class="title-ref">http.status_code</span> were present in the span attributes, the former was preferred. Similarly, <span class="title-ref">http.request.method</span> was preferred over <span class="title-ref">http.method</span>. In this release, the key encountered first is the one used.
OPENSSL_CONF and OPENSSL_MODULES environment variables are now set when running the <span class="title-ref">datadog-fips-agent</span> in FIPS mode. This allows applications to use the OpenSSL configuration and modules configured with the Agent installation.tegrastats output when there are multiple Graphics Processing Clusters (GPCs). In that case, the metric nvidia.jetson.gpu.freq is emitted for each GPC and tagged with gpc:0, gpc:1, and so on.netlink conntracker requires NET_ADMIN capability on the Fargate CNM preview.Released on: 2025-05-06 Pinned to datadog-agent v7.65.0: CHANGELOG.
Release on: 2025-04-10
1.23.8.Released on: 2025-04-10 Pinned to datadog-agent v7.64.3: CHANGELOG.
Release on: 2025-04-02
Released on: 2025-04-02 Pinned to datadog-agent v7.64.2: CHANGELOG.
Release on: 2025-03-20
Released on: 2025-03-20 Pinned to datadog-agent v7.64.1: CHANGELOG
This version contains a Python package dependency (cryptography) issue that can prevent Python integrations from loading on Windows. A workaround is to set the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 at the machine level and restart the Agent. If the issue persists, the recommendation at this time is to downgrade to Agent v7.63.3 or upgrade to v7.64.1 when it becomes available.
Release on: 2025-03-19
ec2_imdsv2_transition_payload_enabled flag from false to true. If IMDSv2 hasn’t been explicitly enabled and the hostname isn’t set to the instance ID, the display name may change to the instance ID without affecting Agent behavior. For more information, see the IMDSv2 Enablement by Default documentation.DD_APM_OBFUSCATION_VALKEY_ENABLED=false. To replace all valkey command arguments with a single ?, set DD_APM_OBFUSCATION_VALKEY_REMOVE_ALL_ARGS=true (default: false).1.23.6.matched PID for the process is 0 warnings.ingestion_reason:probabilistic is set even when an OTLP span was sampled by the Error Sampler. To enable the Error Sampler for OTLP spans, you need to set DD_OTLP_CONFIG_TRACES_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE to 99 or lower, or enable DD_APM_PROBABILISTIC_SAMPLER_ENABLED and set DD_APM_PROBABILISTIC_SAMPLER_PERCENTAGE to 99 or lower.Released on: 2025-03-19 Pinned to datadog-agent v7.64.0: CHANGELOG.
Enable collection of Pod Disruption Budgets by default in the orchestrator check.
Target-based workload selection is now available for Single Step Instrumentation. This feature enables you to instrument specific workloads using pod and namespace label selectors. By applying user-defined labels, you can select workloads for instrumentation without modifying applications. For example, the following configuration injects the Python tracer with a default version for pods labeled with language=python:
instrumentation:
enabled: true
targets:
- name: "Python Services"
podSelector:
matchLabels:
language: "python"
ddTraceVersions:
python: "default"
language=java. If a pod matches both rules, the first match takes precedence:instrumentation:
enabled: true
targets:
- name: "Python Services"
podSelector:
matchLabels:
language: "python"
ddTraceVersions:
python: "default"
- name: "Java Namespaces"
namespaceSelector:
matchLabels:
language: "java"
ddTraceVersions:
python: "default"
DD_ prefix. The following example installs the Python tracer with profiling and data jobs enabled:instrumentation:
enabled: true
targets:
- name: "Python Apps"
podSelector:
matchLabels:
language: "python"
ddTraceVersions:
python: "v2"
ddTraceConfigs:
- name: "DD_PROFILING_ENABLED"
value: "true"
- name: "DD_DATA_JOBS_ENABLED"
value: "true"
kubernetes_state metrics with resource labels or annotations as tags.DD_APM_INSTRUMENTATION_VERSION=v1 has been deprecated and will default to v2.Release on: 2025-03-04
Released on: 2025-03-04 Pinned to datadog-agent v7.63.3: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Release on: 2025-02-28
Released on: 2025-02-28 Pinned to datadog-agent v7.63.2: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Release on: 2025-02-26
Released on: 2025-02-26 Pinned to datadog-agent v7.63.1: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Release on: 2025-02-19
apm_config.obfuscation.cache.max_size to set the maximum size of the cache in bytes.system-probe/dmesg.log.1.23.5.ddprocmon) service status in flare on Windowssql_obfuscation_mode parameter. The value obfuscate_and_normalize is recommended for DBM customers to enhance APM/DBM correlation.Fixes some existing metric transformer unit tests by correcting their assertions.
Datadog span.Type and span.Resource attributes are set correctly for OTel spans processed via OTel Agent and Datadog Exporter when client span type is a database span.Type.
span.Type logic update is limited to ReceiveResourceSpansV2 logic, set using <span class="title-ref">"enable_receive_resource_spans_v2"</span> in <span class="title-ref">DD_APM_FEATURES</span>
span.Resource logic update is limited to OperationAndResourceNameV2 logic, set using <span class="title-ref">"enable_operation_and_resource_name_logic_v2"</span> in <span class="title-ref">DD_APM_FEATURES</span>
Users should set a <span class="title-ref">span.type</span> attribute on their telemetry if they wish to override the default span type.
Agent flare service status search for datadog services is now case insensitive on Windows
Fixed an issue where the "source" and "service" tags were incorrectly set to "kubernetes" in logs when the Agent runs on ECS EC2.
Bypass sending blank logs configs to the integrations launcher to prevent the launcher from sending JSON parse error logs.
Respect proxy config in symdb endpoint.
Fix IsUserAnAdmin call on Windows to use correct API.
Fixed a bug that occurs when reinstalling marketplace/extra integrations for a RPM package after an Agent upgrade.
Windows installer will not abort if the LanmanServer (Server) service is not running (regression introduced in 7.47.0).
Fix the removal of non-core integrations during Agent upgrades on Windows platforms. To enable persisting non-core integration during install, set INSTALL_PYTHON_THIRD_PARTY_DEPS="1" property during the installation of the MSI.
Released on: 2025-02-19 Pinned to datadog-agent v7.63.0: CHANGELOG.
datadog.cluster_agent.cluster_checks.configs_dispatched metric emitted by the Cluster Agent telemetry. The metric values could become inaccurate after the Cluster Agent loses and then regains leader status.This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Release on: 2025-02-14
Released on: 2025-02-14 Pinned to datadog-agent v7.62.3: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Release on: 2025-02-10
Released on: 2025-02-10 Pinned to datadog-agent v7.62.2: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Released on: 2025-02-03
Released on: 2025-02-03 Pinned to datadog-agent v7.62.1: CHANGELOG.
This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.
Released on: 2025-01-29
On Windows, Agent flares now include event logs for .NET applications.
Emit new field: integration for TopologyLinkMetadata for better metrics collection.
Adds tagger tags to pod manifests.
Added the output of sestatus into the Agent flare. This information will appear in system-probe/selinux_sestatus.log.
Extended Agent telemetry histogram details, specifically:
Use HTTP zstd compression for the Agent telemetry payloads.
Agents are now built with Go 1.23.3.
Added the output of semodule -l to the Agent flare; this information appears in system-probe/selinux_semodule_list.log.
Metric payloads are compressed using <span class="title-ref">zstd</span> compression by default. This can be reverted to the previous compression kind by adding serializer_compressor_kind: zlib to the configuration.
Network Path will use recent DNS lookups to infer the destination hostname, if they are available. If a DNS lookup is not found, it will query reverse DNS the same way as before.
Adds support for the <span class="title-ref">none</span> cardinality type in the tagger.
For OpenTelemetry GraphQL request spans, the span resource name is now the GraphQL operation type and name.
All process agent endpoints now require authentication.
Extends extra configuration available for templating from Aurora Database Discovery to include %%extra_dbname%% allowing instances which are configured with non-standard DBName field to be discovered successfully
APM: The trace agent endpoint for changing the configured log level now requires authentication so it is only accessible to other Agent processes.
/var/run/syslog as the default syslog socket path on macOS.Released on: 2025-01-29 Pinned to datadog-agent v7.62.0: CHANGELOG.
Release on: 2025-01-23
1.21.11.golang.org/x/crypto to fix CVE-2024-45337.Released on: 2025-01-23 Pinned to datadog-agent v6.53.1: CHANGELOG.