Skip to content

fix(metrics): keep historical export evidence out of current gauges - #301

Merged
lance0 merged 2 commits into
mainfrom
fix/export-metrics-historical
Oct 21, 2026
Merged

lance0 merged 2 commits into
mainfrom
fix/export-metrics-historical

Conversation

@lance0

@lance0 lance0 commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Problem

Historical evidence reported as current in metrics. After the controller loses and regains ownership, a failed or budget-cut export read keeps the previous owner's evidence for the route (#298). That includes its peer rows, counts and warning, and health and detail mark it historical: true. ExportEvidenceStore::summary() still summed those counts and warnings, and reconciliation published the result as prefixd_blackhole_export_peers{disposition} and prefixd_blackhole_unexported{reason}. Prometheus therefore reported the old owner's committed peers as current, and PrefixdBlackholeNotExported could keep firing on a historical warning.

Rotation advanced one place per pass. The export rotation cursor (#294) moved one position per pass however many reads fit in the budget. With K > 1 reads per budget, some blackholes went unread for up to N - K + 1 passes, not ceil(N/K).

Fix

  • ExportEvidenceStore::current_summary() is the same aggregation without historical routes' counts and warnings. A historical route still counts in prefixd_blackhole_export_read_errors, because its read this pass failed. Reconciliation uses it for the gauges. summary() and the health and detail JSON are unchanged. No new metric or label.
  • The cursor is read at the start of the pass and, once the pass is stored, advances to the first read the budget cut off (all candidates if none was cut off), and always by at least one so a read longer than the budget cannot pin the head. With N candidates and K reads per budget, every route is read within ceil(N/K) passes. An empty set still leaves the cursor alone, and the modulo still handles a budget larger than the set.
  • Judgement call: the cursor now advances only when the pass's evidence is stored. A pass dropped by the fence-epoch check no longer moves it, so the routes it read but never recorded get read again.
  • The docs (API reference metric paragraph, export-evidence section of the backend reference) and the CHANGELOG Fixed section are updated. The alert rule and its promtool test are unchanged: the rule still keys on the reason gauge, which now counts only current evidence.

Tests

  • historical_export_evidence_is_not_published_as_current_metrics: a committed route and an export_policy-withheld route are read at epoch 0; then ownership is lost and regained and both reads fail. Health still shows the historical rows (historical, committed 1, one unexported). The disposition gauges are all 0, the unexported gauges sum to 0, and read errors are 2. A fresh read of the withheld route makes it current again.
  • blackhole_export_rotation_advances_by_the_reads_in_each_pass: 5 candidates, 4 s reads in a 10 s budget (K = 2). After ceil(5/2) = 3 passes all five have been read, and the cursor is 6.
  • The existing K = 1 rotation test (which catches a cursor that skips past the cut-off read), the stall, fence-crossing and store tests pass unchanged.

Negative controls: I applied only the new tests to main's reconcile.rs and export_evidence.rs. Both fail:

historical_export_evidence_is_not_published_as_current_metrics
  assertion failed: historical rows are read errors, not current export
  left:  ([1.0, 1.0, 0.0], 1.0, 2.0)
  right: ([0.0, 0.0, 0.0], 0.0, 2.0)
blackhole_export_rotation_advances_by_the_reads_in_each_pass
  assertion failed: every blackhole is read within ceil(N/K) passes
  left:  [192.0.2.1/32, 192.0.2.2/32, 192.0.2.3/32, 192.0.2.4/32]
  right: [192.0.2.1/32, ..., 192.0.2.5/32]

Gate

Every just gate step passed on this content, but not as one uninterrupted run. Two just gate runs stopped at the GoBGP-only test lane on the PostgreSQL row fixture with CreateContainer(RequestTimeoutError): local Docker was failing to create any container at the time, which a plain docker create probe confirmed. Those runs had passed fmt, ruff, proto and both clippy lanes. After Docker recovered, the remaining steps ran in order:

  • cargo test --no-fail-fast for both feature sets: 1763 passed, 0 failed, 50 ignored, including the PostgreSQL container tests.
  • msrv, build, frontend, audit and deny: all passed.
  • speakers (GoBGP fixture plus rustbgpd 0.79.0): 33 passed.

git diff --check is clean.

After ownership loss and regain, a failed or budget-cut export read keeps
the previous owner's rows, marked historical in health. The export gauges
still summed those rows, so Prometheus reported the old owner's committed
peers as current and PrefixdBlackholeNotExported could keep firing on a
historical warning. Metrics now use a current-only summary: a historical
route counts as a read error, while health and detail are unchanged.

The export rotation cursor also moved one place per pass regardless of
how many reads fit in the budget. It now resumes at the first read the
budget cut off, so N blackholes at K reads per budget are all read within
ceil(N/K) passes. The cursor advances only when the pass is stored.
@lance0
lance0 marked this pull request as ready for review October 21, 2026 22:08
@lance0
lance0 merged commit 04a4aa5 into main Oct 21, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant