Skip to content

fix(alerts): clear blackhole replacement alert an hour after the last one - #303

Merged
lance0 merged 1 commit into
mainfrom
fix/replacement-alert-clears
Oct 22, 2026
Merged

lance0 merged 1 commit into
mainfrom
fix/replacement-alert-clears

Conversation

@lance0

@lance0 lance0 commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Summary

PrefixdBlackholeReplaced used prefixd_blackhole_replaced_total > 0. The counter is process-local and its {pop} series appeared only at the first replacement, so the rule could not use increase() and the critical alert latched until prefixd restarted or someone silenced it. 0.22.0 lists that as a known limitation.

  • Startup now calls init_metrics(&config.settings.pop), which exports prefixd_blackhole_replaced_total{pop="<configured POP>"} at 0. The only label value is the configured POP.
  • The rule is now increase(prefixd_blackhole_replaced_total[1h]) > 0 with severity still critical. It fires on the first replacement and clears about an hour after the last one. The annotation names the one-hour window and still calls this historical evidence, not proof of an active conflict.
  • The main binary never called init_metrics() before; only tests did. The other pre-registered series (lease renewal and state, export disposition, reconciliation status and so on) are now also exported at 0 from startup. The metrics reference already said the lease series exist at zero in every deployment.
  • Docs: the Grafana alert table and text, the metric's API reference entry and an Unreleased changelog entry. The released 0.22.0 changelog and upgrade sections are unchanged.

Verification

  • just alerts: promtool check rules finds 5 rules, and promtool test rules passes. The new cases cover these behaviours:
    • The series sits at 0, then one replacement fires it. Only the affected POP fires.
    • With no further increase it is still firing at 64m and clear at 65m and 85m.
    • A counter left at 1 by a replacement more than an hour earlier, then reset to 0 by a restart, never fires.
    • A replacement shortly before a restart stays inside the window across the reset, then clears.
    • A series that stays at 0 never fires, and the unleased-detector gauge alone does not fire this alert.
  • Negative control: the same test file run against the old > 0 expression (keeping the new annotations) fails. It still fires at the 65m and 85m clear points and in the stale-counter case at 10m, and it clears too early (20m) after a restart.
  • Unit test init_exports_the_configured_pop_replacement_counter_at_zero checks that the gathered metrics include prefixd_blackhole_replaced_total{pop="init-zero-pop"} 0 after init, before any replacement.
  • just gate passes on the head commit.

… one

PrefixdBlackholeReplaced used prefixd_blackhole_replaced_total > 0, so the
process-local counter kept the critical alert firing until prefixd
restarted. The POP series only appeared on the first replacement, which is
why the rule could not use increase().

Startup now calls init_metrics with the configured POP, exporting that
series at zero, and the rule fires on increase over one hour. The main
binary never called init_metrics before, so the other pre-registered series
are now also exported at zero from startup, as the metrics reference already
stated.
@lance0
lance0 marked this pull request as ready for review October 22, 2026 01:17
@lance0
lance0 merged commit 407c234 into main Oct 22, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant