Conversation
The processor's NATS-connection and active-policy gauges are registered in the shared metrics registry of the single activity binary, so every subcommand serves them. The edge event exporter never sets them and reports both at zero, so once the edge exporter is scraped it raises a critical ActivityProcessorNATSDisconnected and an ActivityProcessorNoPolicies warning against a healthy system. Key changes: - Select job="activity-processor" in both rules, matching the processor dashboards and the exporter rules that already select their own job - Leave the rate-based processor rules alone: an idle zero counter cannot satisfy them Changing an expression resets the rule's alert state, so an alert that is firing when this deploys waits out its "for" again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The processor's NATS-disconnect and empty-policy alerts fire for any pod that exposes those gauges, and the edge event exporter exposes both at zero because it runs the same binary.
In staging this pages a critical alert on both edge labs while the processor is healthy.
Both rules now select only the processor's scrape job, the same selector the processor dashboards and the exporter's own alerts already use.
Test plan
Fixes #252