chore(config): replay the agent config corpus through adp

[webern]

Human Summary

Adds tests from a corpus of actual Agent output (see #2717 to learn how). Seems pretty valuable. It has found the below issues that either need to be fixed or investigated. The design allows for bugs to exist. So these issues listed here are locked in by the tests that found them, and when they are fixed we will flip the test assertion to a green assertion.

• Accept streamed null as an empty configuration collection #2750
• Match core agent byte-size parsing for Datadog configuration #2751
• Apply core agent element coercion to string-valued configuration maps #2752
• Match core agent decoding of valid local environment inputs #2753
• Align local Datadog YAML interpretation with the core agent #2754
• Investigate scalar cast-failure compatibility with the core agent #2755
• Investigate malformed JSON and wrong-shape collection compatibility #2756
• Investigate authority of the streamed data-plane shutdown timeout #2757

AI Summary

#2717 records what the real Datadog Agent streams and what its configuration getters return. This PR uses those recordings to test what ADP actually reads. The distinction matters: a setting declared as a map can arrive as a JSON string, and the Agent's getter can accept a value that Rust rejects. Hand-written fixtures in the expected schema shape missed exactly that failure in #2700.

The goal is to make those compatibility checks part of everyday configuration development, without running the Go Agent in CI. The checked-in corpus of recordings supplies the Agent evidence; ordinary Rust unit tests exercise ADP's production code. Existing differences are covered explicitly, so we can add regression protection now and fix bugs in separate PRs.

What the replay checks

The tests live in lib/agent-data-plane-config-system/src/corpus_replay/. They look at the same recordings from several angles:

• Streamed settings. Reconstruct protobuf events and pass them through the same wire conversion and source folding used by the running process. Generated typed accessors read each supported field from DatadogConfiguration for comparison with its corresponding Agent getter, after the initial snapshot and after the final update. Settings are also deserialized in isolation so one malformed value does not hide failures in unrelated fields.
• Computed values. Compare translated byte sizes and the shutdown timeout, not only their input settings. Two implementations can agree on the string "10MB" but disagree on the byte count used at runtime.
• Local startup reads. Feed the recorded YAML and environment into ADP's own bootstrap reader, independently of the Agent stream. This catches differences in parsing and defaults before a stream arrives. These checks exclude cases whose Agent result also depends on fleet policy or CLI overrides.
• Update processing. Check deserialization, translation, and validation at each replayed operation, including rejected intermediate updates—not just the final result. A separate connected-system test starts with a usable API key, accepts an update, rejects an invalid one while retaining the last valid configuration, and then accepts a later valid update.
• Unmodeled settings. Check unsupported-key classification against the support inventory, ensure excluded and unknown keys remain unclassified, and verify these settings do not introduce deserialization or translation failures.

Comparison rules are defined by Rust value kind and Agent getter, not by setting name, so a type-level fix is checked across the recorded settings of that kind.

Most recordings have no API key. Value comparisons therefore remain independent of whether the whole configuration is runnable; otherwise missing-key validation would obscure nearly all the interesting inputs. Validation still has its own assertions, and the connected test exercises the real accept-or-reject behavior.

The production changes make these paths reusable rather than copying them into a test implementation: stream conversion moves out of the binary, startup and updates share a stage evaluator, and bootstrap loading accepts explicit inputs. The shutdown-timeout calculation is shared too. No runtime behavior change is intended.

Known differences are assertions, not exemptions

For value comparisons, matching the recorded Agent getter is the default expectation. corpus_replay/expected.rs holds hand-edited Rust entries for individual checks that currently differ. Each names the exact ADP value or error and explains whether it is a known bug or intentional behavior. Known-bug expectations also carry a Saluki tracking issue through their shared cause; missing issue references fail the replay test, and failure diagnostics link the issue. The follow-ups are tracked under #2169. An intentional rejection is a passing assertion of that specific rejection, not permission to return any error.

For example, ADP currently interprets "10MB" as 10,000,000 bytes, while the Agent's GetSizeInBytes returns 10,485,760. The test pins today's ADP result as a known bug. Fixing it makes that check fail with “This check now matches the Agent” and points to the expectation to remove. A different wrong byte count also fails. This PR records the difference; it does not fix it.

Expectations cannot exempt an entire case or automatically cover new checks. Duplicate or unused entries fail, and check identities survive generated batch renames. There is no generated ADP-results baseline or blessing command.

Failures are collected in stable order and identify the case, input location, setting, checkpoint or update, expected and actual ADP results, recorded Agent result, and expectation location. A caught production panic always fails the affected case; other cases still run.

Working with these tests

The replay runs with the normal Rust unit-test suite, without Go or Docker. For a focused run:

cargo nextest run --lib -p agent-data-plane-config-system corpus_replay

An ADP fix changes production code and the relevant expectation together—not the recorded Agent corpus. Schema updates are different: after support decisions settle, regenerate the recordings, review the upstream behavior changes, and run replay. This PR adds those steps to the configuration maintenance guidance.

This is not complete equivalence testing. Some recorded getters, provenance-dependent section reads, and other computed values still lack comparisons; lib/datadog-agent/config-recorder/docs/comparison.md and the replay code document those gaps rather than count them as matches. Live transport, authentication, and timing are outside this PR. Structured consumer decoding is tracked in #2758, effective percentile parsing in #2759, and provenance/section comparisons in #2760. Existing resolved-default findings remain in #1802 and #2484.

Change Type

• Non-functional (chore, refactoring, docs)

How did you test this PR?

• cargo nextest run --lib --bins -p agent-data-plane-config-system -p datadog-agent-config-corpus -p datadog-agent-config -p agent-data-plane-config -p agent-data-plane (655 passed).
• Mutated production behavior to simulate a fixed byte-size bug, a changed byte count, an intermediate rejected update, a missed validation rejection, and a stream-conversion panic; each produced a case-specific failure. Restored all mutations.
• make fmt, make check-docs, make check-clippy, make check-fmt, make check-unused-deps, make check-licenses, make check-deny, cargo check --workspace --tests.
• Follow-up issue-link changes: all 55 corpus_replay tests passed; make fmt, make check-docs, and make check-clippy passed. Temporary mutations verified that improved and changed byte-size results print Match core agent byte-size parsing for Datadog configuration #2751, and that a known-bug expectation without an issue fails. All mutations were restored.

References

• Related: chore(config): record the agent config stream in a corpus #2717
• Related: fix(config): decode map settings from agent stream #2700

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 216e499 · Comparison: 5b376c9 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 38.27 MiB (baseline) vs 38.21 MiB (comparison)
Size Change: -56.62 KiB (-0.14%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
core	-30.50 KiB	5072
serde_json	-26.37 KiB	793
agent_data_plane::dogstatsd_contexts::artifact	-22.53 KiB	23
anon.b0f0e7762476e62d94564a7e6a50c916.1113.llvm.18418558211630905745	+20.54 KiB	1
anon.a6926c6c21417749435f71dba1b992d8.976.llvm.1534094455157536209	-20.54 KiB	1
saluki_core::runtime::supervisor	-18.98 KiB	51
agent_data_plane::internal::remote_agent	-18.85 KiB	114
http_body_util	-18.52 KiB	76
alloc	-16.42 KiB	812
anon.87e7c38bfd928d17fa28187081c51bf0.2.llvm.240572210426274404	-16.09 KiB	1
anon.f19e40ffe4e6c23836f9daceaa6eedf5.5.llvm.13060621878149402103	+16.01 KiB	1
tracing	+15.48 KiB	4
tokio	+15.43 KiB	952
&mut serde_json	+14.81 KiB	61
serde_core	+14.65 KiB	641
anon.87e7c38bfd928d17fa28187081c51bf0.768.llvm.240572210426274404	-14.62 KiB	1
anon.f19e40ffe4e6c23836f9daceaa6eedf5.10.llvm.13060621878149402103	+14.45 KiB	1
anon.b0f0e7762476e62d94564a7e6a50c916.168.llvm.18418558211630905745	+14.40 KiB	1
anon.53bb41fcd4d7cb0fd4749e93eb822af9.155.llvm.2364525274383654594	-14.40 KiB	1
anon.87e7c38bfd928d17fa28187081c51bf0.766.llvm.240572210426274404	-12.84 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +59.9Ki  [NEW] +59.6Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h5b4fa9686eaf7472
  [NEW] +40.0Ki  [NEW] +39.9Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hfe5503d072c1731e
  [NEW] +34.0Ki  [NEW] +33.9Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h481bf233e74b84de
  [NEW] +31.8Ki  [NEW] +31.7Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h815988463bb282eb
  [NEW] +26.2Ki  [NEW] +26.0Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::connect::_{{closure}}::_{{closure}}::_{{closure}}::h35b699971391ca1f
  [NEW] +25.6Ki  [NEW] +25.4Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::hc8896c2f15248352
  [NEW] +24.6Ki  [NEW] +24.5Ki    agent_data_plane::main::_{{closure}}::h77a0d07f25d2bf0b
  [NEW] +23.8Ki  [NEW] +23.7Ki    datadog_agent_config::generated::witness::drive::h9ccc4d434aeb7d27
  [NEW] +23.3Ki  [NEW] +23.2Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h11795c74345d7b41
  +0.1% +5.24Ki  +0.3% +17.5Ki    [19736 Others]
  [DEL] -22.3Ki  [DEL] -22.2Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::hee0e8d3a30af396e
  [DEL] -23.5Ki  [DEL] -23.3Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::hc84068d640a2d683
  [DEL] -24.9Ki  [DEL] -24.8Ki    agent_data_plane::main::_{{closure}}::h3ce278d5f382d5fc
  [DEL] -25.6Ki  [DEL] -25.4Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h1f55b7becb8f9961
  [DEL] -28.4Ki  [DEL] -28.2Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::connect::_{{closure}}::_{{closure}}::_{{closure}}::h1bd6a026eeafe819
  [DEL] -28.7Ki  [DEL] -28.6Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::hb60d5262ffff75a0
  [DEL] -31.0Ki  [DEL] -30.9Ki    datadog_agent_config::generated::witness::drive::h5f0a3f05b22b2ac2
  [DEL] -31.4Ki  [DEL] -31.2Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h2f5246c806e7f392
  [DEL] -34.9Ki  [DEL] -34.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::haa79196f741be08a
  [DEL] -40.5Ki  [DEL] -40.4Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h02ed82f16b54dbc0
  [DEL] -60.0Ki  [DEL] -59.7Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h7e9578d759076d6a
  -0.1% -56.6Ki  -0.1% -44.1Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 45eed92e-8afd-43e7-b2c3-e928a02e13ec
Baseline: 216e499a · Comparison: 5b376c96 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +1.03	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.97	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.85	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.36	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.15	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 231 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 51.3 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.2 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 385 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 33.4 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

[datadog-prod-us1-6]

More details

Stream conversion, staged update retention, bootstrap inputs, computed values, and pinned Agent divergences are exercised without a concrete runtime regression identified.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit fc288a5 · @DataDog review to ask questions}