Skip to content

workloadmanager: emit Kubernetes audit events for GC-triggered sandbox deletions - #515

Open
SoumaCloud wants to merge 1 commit into
volcano-sh:mainfrom
SoumaCloud:fix/gc-missing-sandbox-audit-event
Open

SoumaCloud wants to merge 1 commit into
volcano-sh:mainfrom
SoumaCloud:fix/gc-missing-sandbox-audit-event

Conversation

@SoumaCloud

Copy link
Copy Markdown

Problem

The garbage collector (gc.once()) deletes sandboxes — both idle-timed-out and
TTL-expired — without emitting any Kubernetes Event. Operators relying on
kubectl get events, Loki, Falco, or the K8s audit sink had no observable
signal to distinguish a GC sweep from an unexpected disappearance.

This was confirmed by grepping the codebase:

$ grep -n "sandboxAuditEvent\|AuditReason" pkg/workloadmanager/garbage_collection.go
(no output)

Closes #513

Solution

Introduces pkg/workloadmanager/audit.go with a sandboxAuditEvent helper
that emits a corev1.Event in the sandbox's namespace. The helper is:

  • nil-safe: returns immediately when the clientset is nil (common in unit
    tests that construct K8sClient without a full API server)
  • panic-safe: wraps the create call in a recover() so a failing event
    emission can never abort the caller's deletion path
  • non-fatal: errors are logged at V(2) and do not block the GC loop

garbage_collection.go now calls sandboxAuditEvent(AuditReasonDeleted)
immediately after each successful Kubernetes deletion (before the store record
is removed). Each Event carries:

Field Value
Reason SandboxDeleted
Type Normal
Message sandbox <ns>/<name> GC-deleted (session <id>)
Source.Component agentcube-workloadmanager
Annotation agentcube.io/owner-hash SHA-256-shortened owner ID (never the raw sub claim)

Tests

Five unit tests in audit_test.go cover:

  • SandboxCreated → EventTypeNormal
  • SandboxCreateFailed → EventTypeWarning
  • SandboxDeleted with correct annotation
  • empty ownerHash → no annotation written
  • nil clientset → no panic

All existing GC tests continue to pass (go test ./pkg/workloadmanager/...).

Signed-off-by: Soumadip Patra soum23dip@gmail.com

The garbage collector deleted sandboxes (both idle-timeout and expired)
without emitting any Kubernetes Event. Operators using `kubectl get events`,
Loki, Falco, or the K8s audit sink had no way to tell whether a sandbox
disappeared due to a GC sweep or an unexpected error.

This commit introduces pkg/workloadmanager/audit.go, which provides a
single sandboxAuditEvent helper that creates a corev1.Event in the
sandbox's namespace. The helper is nil-safe (returns early when the
clientset is nil, as in unit tests that build K8sClient without one) and
recovers from panics so it can never abort a caller's control path.

garbage_collection.go now calls sandboxAuditEvent(AuditReasonDeleted)
immediately after each successful sandbox/sandboxClaim deletion, before
the store record is removed. The Event carries:
- Reason: "SandboxDeleted"
- Type: Normal
- Message: human-readable description including namespace, name, session ID
- Annotation agentcube.io/owner-hash: SHA-256-shortened owner ID (not
  the raw sub claim) for correlating events without storing PII.

Five unit tests in audit_test.go cover the Normal/Warning type split,
per-reason annotation behaviour, empty owner-hash, and nil-clientset safety.

Fixes: volcano-sh#513

Signed-off-by: Soumadip Patra <soum23dip@gmail.com>
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign acsoto for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.11765% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.18%. Comparing base (f521ae9) to head (3b00e7c).
⚠️ Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
pkg/workloadmanager/audit.go 93.54% 1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #515      +/-   ##
==========================================
+ Coverage   60.45%   64.18%   +3.72%     
==========================================
  Files          35       36       +1     
  Lines        3616     3691      +75     
==========================================
+ Hits         2186     2369     +183     
+ Misses       1211     1089     -122     
- Partials      219      233      +14     
Flag Coverage Δ
unittests 64.18% <94.11%> (+3.72%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

workloadmanager: POST /sandbox has no per-owner rate limiting

3 participants