Skip to content

Route activity policy failures to their owners - #255

Merged
scotwells merged 1 commit into
mainfrom
codex/activity-policy-ownership
Sep 28, 2026
Merged

scotwells merged 1 commit into
mainfrom
codex/activity-policy-ownership

Conversation

@scotwells

@scotwells scotwells commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

When an activity policy fails, customers can see gaps in their resource timelines. Route those failures to the team that owns the policy so the right team can investigate and restore missing activity entries, including events already queued for retry.

Set ownership on the ActivityPolicy resource's metadata.labels, for example:

metadata:
  labels:
    meta.datumapis.com/service: network-services
    meta.datumapis.com/team: delivery

meta.datumapis.com/service becomes the alert's service label; meta.datumapis.com/team becomes its team label. Existing Alertmanager routes send service=network-services to Deliver; team=delivery records the policy's owning team. Setting the team label alone does not create a route.

Policies without an owner still alert: a missing service stays unset, a missing team falls back to sre, and missing ownership metrics do not suppress failures. Ownership changes also apply to existing retry failures.

Validated processor tests and ten alert-rule scenarios covering ownership changes, missing metadata, replicas, cluster isolation, and scrape-label collisions; CI is green.

Roll out with https://github.com/datum-cloud/infra/pull/6325 and the policy labels in datum-cloud/network-services-operator#515 and datum-cloud/dns-operator#192. Include this change in staging's federated-events branch before deployment.

Related to https://github.com/datum-cloud/infra/issues/6019.

@scotwells
scotwells merged commit df6caa7 into main Sep 28, 2026
9 checks passed
@scotwells
scotwells deleted the codex/activity-policy-ownership branch September 28, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants