chore(agent-data-plane, config): hand back dynamic configuration tasks instead of spawning them

[tobz]

Summary

This PR updates our configuration primitives to move most of the the background tasks/asynchronicity to agent-data-plane-config-system where we can wire it up to participate in supervision.

Prior to this PR, the background work to update a configuration dynamically, as well as get dynamic updates from some arbitrary spot (like the remote agent client) into the configuration was split across two places, one of those being saluki-config. In our quest to push background talks into supervision without having to have saluki-config, this PR moves the majority of this work into agent-data-plane-config-system where it can get the proper Supervisable treatment. What remains in saluki-config is a very thin layer -- one meant simply to facilitate integrating the given updates into the resolved configuration -- which is then driven by the supervised task.

Admittedly, it's not the most satisfying or holistic split, but it moves things in the right direction prior to fully introducing supervision "scopes", and subsequent stacked PRs will be able to improve this situation.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Existing and new tests: ensuring the supervised worker keeps driving the processing of configuration updates as long as the update channel is available, etc.

References

• Closes Run the typed configuration system in the supervision tree. #2196

DADP-2

[tobz]

• chore(agent-data-plane, config): hand back dynamic configuration tasks instead of spawning them #2712 👈 (View in Graphite)
• chore(common, core): move cache and Supervisable into saluki-core #2711
• chore(context, core): move saluki-context into saluki-core #2710
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 9e63899 · Comparison: a8ad16e · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 38.48 MiB (baseline) vs 38.26 MiB (comparison)
Size Change: -231.05 KiB (-0.59%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
saluki_context	-138.59 KiB	115
core	-136.88 KiB	16551
saluki_core::data_model::event	+90.55 KiB	148
saluki_core::data_model::tags	+41.99 KiB	38
&mut serde_json	-41.35 KiB	131
saluki_core::topology::built	-31.82 KiB	35
saluki_common::cache::CacheBuilder<K,V,W,H>	-29.51 KiB	5
serde_json	+29.04 KiB	1028
saluki_core::topology::blueprint	+26.89 KiB	103
saluki_core::cache::CacheBuilder<K,V,W,H>	+26.47 KiB	5
agent_data_plane_config_system::translators::datadog_translator	-25.04 KiB	135
saluki_common::cache::expiry	-23.82 KiB	25
alloc	+22.60 KiB	3721
serde_core	-22.39 KiB	846
saluki_core::cache::expiry	+20.68 KiB	21
axum	+20.53 KiB	758
saluki_common::cache::Telemetry	-19.27 KiB	14
http_body_util	+19.14 KiB	209
saluki_core::cache::Telemetry	+19.04 KiB	14
tonic	-18.27 KiB	446

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +60.0Ki  [NEW] +59.9Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h15a965fc1b5a6f1e
  [NEW] +60.0Ki  [NEW] +59.7Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h7e9578d759076d6a
  [NEW] +40.4Ki  [NEW] +40.3Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h48199e6cde4750a2
  [NEW] +38.0Ki  [NEW] +37.8Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h252ca254b106e4c3
  [NEW] +34.9Ki  [NEW] +34.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h48f5a75cbf949eb9
  [NEW] +34.7Ki  [NEW] +34.6Ki    saluki_core::topology::blueprint::TopologyBuildState::build::_{{closure}}::h734355e2a3cc5202
  [NEW] +31.4Ki  [NEW] +31.2Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::hf3f67152355be866
  [NEW] +31.0Ki  [NEW] +30.9Ki    datadog_agent_config::generated::witness::drive::h5f0a3f05b22b2ac2
  [NEW] +28.7Ki  [NEW] +28.6Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::hb2a3129affe62bdb
  [NEW] +28.4Ki  [NEW] +28.2Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::connect::_{{closure}}::_{{closure}}::_{{closure}}::hee1adc6abddef53c
  [NEW] +27.5Ki  [NEW] +27.3Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::ha5569e4fe676e480
  [DEL] -27.5Ki  [DEL] -27.3Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h9e2eedcf1bab20f9
  [DEL] -27.6Ki  [DEL] -27.4Ki    saluki_core::topology::built::BuiltTopology::spawn_inner::_{{closure}}::hcecb1f0589b91693
  [DEL] -27.7Ki  [DEL] -27.6Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::h9053679a7c7c4282
  [DEL] -31.0Ki  [DEL] -30.9Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::hdb194efa2e919aa8
  [DEL] -35.8Ki  [DEL] -35.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h7a638dbf1937c3ba
  [DEL] -38.0Ki  [DEL] -37.8Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h154e0b8bdac85431
  [DEL] -41.1Ki  [DEL] -41.0Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::ha4dcd0a8d8fb7e5d
  [DEL] -58.7Ki  [DEL] -58.4Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hcd01371c65f5fe2d
  [DEL] -60.4Ki  [DEL] -60.2Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h4e117a76eb7fda79
  -1.2%  -298Ki  -1.2%  -230Ki    [57065 Others]
  -0.6%  -231Ki  -0.5%  -163Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 41040018-9783-420f-88e6-c062ace3e868
Baseline: 9e638990 · Comparison: a8ad16e3 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_medium	memory	⚪ +0.40	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.18	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.26	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.31	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.35	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 231 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 51.2 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.3 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 388 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 32.9 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.

[datadog-datadog-prod-us1-2]

More details

The handoff preserves initial-snapshot gating, update ordering, configuration precedence, and last-known-good behavior while moving subsequent updates under supervision.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit a241623 · @DataDog review to ask questions}

[webern]

I added Closes #2196 to your PR description because I think it does. You could change it to Progresses or Related if I'm wrong.

We have a fairly ugly collision here with #2688, so I hate approving this 😅 . Fortunately LLMs excel at cleaning of a rebase disaster.

[webern]

Just an FYI, but I'm trying to build a "crazy config stuff from the agent" test rig and I think it's about time to retire the "smoke tests" and registry. Should happen soon, the "crazy config stuff from the agent" test rig is a bit unwieldy in the repo. It's draft is at #2717. And they don't do the same thing, exactly, but anyway I think the smoke tests are useless now and I haven't forgotten about removing this stuff.