Repository navigation
fix(alerts): clear blackhole replacement alert an hour after the last one - #303
Merged
Merged
Conversation
… one PrefixdBlackholeReplaced used prefixd_blackhole_replaced_total > 0, so the process-local counter kept the critical alert firing until prefixd restarted. The POP series only appeared on the first replacement, which is why the rule could not use increase(). Startup now calls init_metrics with the configured POP, exporting that series at zero, and the rule fires on increase over one hour. The main binary never called init_metrics before, so the other pre-registered series are now also exported at zero from startup, as the metrics reference already stated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PrefixdBlackholeReplacedusedprefixd_blackhole_replaced_total > 0. The counter is process-local and its{pop}series appeared only at the first replacement, so the rule could not useincrease()and the critical alert latched until prefixd restarted or someone silenced it. 0.22.0 lists that as a known limitation.init_metrics(&config.settings.pop), which exportsprefixd_blackhole_replaced_total{pop="<configured POP>"}at 0. The only label value is the configured POP.increase(prefixd_blackhole_replaced_total[1h]) > 0with severity still critical. It fires on the first replacement and clears about an hour after the last one. The annotation names the one-hour window and still calls this historical evidence, not proof of an active conflict.init_metrics()before; only tests did. The other pre-registered series (lease renewal and state, export disposition, reconciliation status and so on) are now also exported at 0 from startup. The metrics reference already said the lease series exist at zero in every deployment.Verification
just alerts:promtool check rulesfinds 5 rules, andpromtool test rulespasses. The new cases cover these behaviours:> 0expression (keeping the new annotations) fails. It still fires at the 65m and 85m clear points and in the stale-counter case at 10m, and it clears too early (20m) after a restart.init_exports_the_configured_pop_replacement_counter_at_zerochecks that the gathered metrics includeprefixd_blackhole_replaced_total{pop="init-zero-pop"} 0after init, before any replacement.just gatepasses on the head commit.