feat(agent-data-plane): add aggregator flush count and timing telemetry

[jszwedko]

Summary

When ADP handles DogStatsD, the Core Agent's aggregator stops flushing DogStatsD traffic, so its /telemetry flush series flatline or drop sharply: aggregator.flush_time, aggregator.flush_count, and aggregator.flush for events and service checks. That endpoint is the migration target for customers moving off go_expvar (DADP-179), but ADP had no flush timing or per-flush counts at all, so there was nothing to remap. This PR adds that instrumentation and maps it to the Core Agent names. Stacked on #2766 (part 1 of 2).

In ADP, events and service checks skip the aggregator and go straight to their encoders, so the encoder's flush stands in for the aggregator's event and service-check flush. Metrics still flush from dsd_agg.

flowchart LR
    subgraph dsd_agg["dsd_agg (aggregate transform)"]
        A1["aggregate_last_flush_count{data_type}"]
        A2["aggregate_last_flush_duration_nanoseconds"]
    end
    subgraph enc["dd_events_encode / dd_service_checks_encode"]
        E1["encoder_flushed_events_total"]
        E2["encoder_last_flush_events"]
        E3["encoder_last_flush_duration_nanoseconds"]
    end
    A1 --> FC["aggregator__flush_count{flush_type=series|sketches}"]
    A2 --> FT["aggregator__flush_time{flush_type=main}"]
    E1 --> F["aggregator__flush{data_type=events|service_checks}"]
    E2 --> FC2["aggregator__flush_count{flush_type=events|service_checks}"]
    E3 --> FT2["aggregator__flush_time{flush_type=event|service_check}"]

Loading

Known differences from the Core Agent:

• The encoders flush every 2s, or sooner when a payload fills, versus the Core Agent's 15s batch. So event and service-check flush_count values are smaller, and flush_time covers one encode-and-dispatch cycle.
• flush_time for metrics is only reported as flush_type="main" (the dsd_agg flush, including handing off to the encoder). The Core Agent's metric_sketch and checks_metric_sample timings include serialization, which ADP does in a separate component, so they stay unmapped.
• aggregator.flush for events and service checks only carries data_type, the same as the existing dsd_agg series. A failed encoder flush stops the encoder instead of being counted as an error.

Durations are recorded in nanoseconds to match the Core Agent's flush_time, because a remap rule can rename a metric but not rescale its value. The new encoder telemetry is also emitted by the logs encoder, which uses the same encoder wrapper, but no rule maps it.

Test plan

• Added flush_records_last_flush_counts, which runs real dsd_agg flushes and checks that the per-flush series and sketch gauges describe only the most recent flush
• Added record_flush_tracks_totals_and_last_flush for the encoder flush telemetry
• Added render_rar_telemetry_remaps_aggregator_flush_telemetry, which checks every new remapped series and its type, and that the logs encoder's series isn't remapped
• Extended rar_rules_carry_expected_help_text to pin aggregator.flush_count and aggregator.flush_time help text to the Core Agent's registered strings

🤖 Generated with Claude Code

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: a41cd43 · Comparison: 477292b · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 38.27 MiB (baseline) vs 38.24 MiB (comparison)
Size Change: -23.04 KiB (-0.06%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
tracing	-26.81 KiB	6
saluki_common::task::instrument	+23.21 KiB	7
prost	-17.31 KiB	207
anon.87e7c38bfd928d17fa28187081c51bf0.2.llvm.240572210426274404	-16.09 KiB	1
anon.b080359256cf918ed667da245626fccf.2.llvm.15973317837417986504	+16.00 KiB	1
anon.87e7c38bfd928d17fa28187081c51bf0.768.llvm.240572210426274404	-14.62 KiB	1
anon.b080359256cf918ed667da245626fccf.768.llvm.15973317837417986504	+14.62 KiB	1
anon.87e7c38bfd928d17fa28187081c51bf0.766.llvm.240572210426274404	-12.84 KiB	1
anon.b080359256cf918ed667da245626fccf.766.llvm.15973317837417986504	+12.83 KiB	1
anon.9d1057ec6821611763596b0232be6ef0.693.llvm.6678353967787269721	-11.13 KiB	1
anon.6ec044b0875770082746ae62238c0091.450.llvm.975936506186025484	+11.04 KiB	1
axum	-10.35 KiB	46
anon.6ec044b0875770082746ae62238c0091.0.llvm.975936506186025484	+8.57 KiB	1
anon.9d1057ec6821611763596b0232be6ef0.4.llvm.6678353967787269721	-8.49 KiB	1
anon.87e7c38bfd928d17fa28187081c51bf0.704.llvm.240572210426274404	-8.23 KiB	1
anon.b080359256cf918ed667da245626fccf.704.llvm.15973317837417986504	+8.23 KiB	1
anon.87e7c38bfd928d17fa28187081c51bf0.668.llvm.240572210426274404	-7.67 KiB	1
anon.b080359256cf918ed667da245626fccf.668.llvm.15973317837417986504	+7.67 KiB	1
agent_data_plane::dogstatsd_contexts::api	+7.26 KiB	3
saluki_io::deser::framing	-7.19 KiB	5

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +18.9Ki  [NEW] +18.8Ki    _<core::pin::Pin<P> as core::future::future::Future>::poll::h7886d4f245bead48
  [NEW] +16.0Ki  [NEW]     +76    anon.b080359256cf918ed667da245626fccf.2.llvm.15973317837417986504
  [NEW] +14.6Ki  [NEW]     +81    anon.b080359256cf918ed667da245626fccf.768.llvm.15973317837417986504
  [NEW] +12.8Ki  [NEW]     +82    anon.b080359256cf918ed667da245626fccf.766.llvm.15973317837417986504
  [NEW] +11.8Ki  [NEW] +11.7Ki    saluki_components::sources::otlp::logs::transform::transform_log_record::h166122ae201d1a73
  [NEW] +11.0Ki  [NEW]     +81    anon.6ec044b0875770082746ae62238c0091.450.llvm.975936506186025484
 +19e2% +9.35Ki +28e2% +9.35Ki    _<saluki_common::task::instrument::InstrumentedTask<F> as core::future::future::Future>::poll::h1a1cb0f3c2fa7b6d
  [NEW] +8.57Ki  [NEW]     +81    anon.6ec044b0875770082746ae62238c0091.0.llvm.975936506186025484
  [NEW] +8.23Ki  [NEW]     +79    anon.b080359256cf918ed667da245626fccf.704.llvm.15973317837417986504
  +0.0%    +753  -0.0%    -615    [8106 Others]
  [DEL] -8.23Ki  [DEL]     -79    anon.87e7c38bfd928d17fa28187081c51bf0.704.llvm.240572210426274404
  [DEL] -8.49Ki  [DEL]     -81    anon.9d1057ec6821611763596b0232be6ef0.4.llvm.6678353967787269721
  [DEL] -9.34Ki  [DEL] -9.20Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hae6626bbb8069929
  [DEL] -10.3Ki  [DEL] -10.2Ki    prost::encoding::message::encode::had171e864325c98f
  [DEL] -11.1Ki  [DEL]     -81    anon.9d1057ec6821611763596b0232be6ef0.693.llvm.6678353967787269721
 -86.3% -11.7Ki -87.5% -11.7Ki    _<saluki_components::sources::otlp::logs::translator::OtlpLogsTranslator as core::iter::traits::iterator::Iterator>::next::h9670cbdd41e04d9b
  [DEL] -12.8Ki  [DEL]     -82    anon.87e7c38bfd928d17fa28187081c51bf0.766.llvm.240572210426274404
  [DEL] -14.6Ki  [DEL]     -81    anon.87e7c38bfd928d17fa28187081c51bf0.768.llvm.240572210426274404
  [DEL] -15.6Ki  [DEL] -15.5Ki    prost::encoding::message::encode::he87576832411933d
  [DEL] -16.1Ki  [DEL]     -76    anon.87e7c38bfd928d17fa28187081c51bf0.2.llvm.240572210426274404
  [DEL] -16.8Ki  [DEL] -16.7Ki    _<core::pin::Pin<P> as core::future::future::Future>::poll::h1154b307d887f4d8
  -0.1% -23.0Ki  -0.1% -24.1Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 3c0b5ae5-ac36-4098-aded-0f185c1d5428
Baseline: a41cd434 · Comparison: 477292b2 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +1.03	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.41	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.09	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.32	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.82	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 230 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 51.5 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.8 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 389 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 33.3 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression. Experiments tagged (no analysis) show ⚠️ n/a: SMP ran them but produced no analysis, usually because a replicate failed and exhausted its retries. Check the SMP report for that experiment's replicate failures.