Skip to content

workloadmanager: add per-owner sandbox creation rate limiter - #512

Open
SoumaCloud wants to merge 2 commits into
volcano-sh:mainfrom
SoumaCloud:feature/per-owner-rate-limiter
Open

SoumaCloud wants to merge 2 commits into
volcano-sh:mainfrom
SoumaCloud:feature/per-owner-rate-limiter

Conversation

@SoumaCloud

Copy link
Copy Markdown

Problem

handleSandboxCreate already calls resolveSandboxOwnerID() on every POST /sandbox
request and attaches the result to the sandbox store entry — but no admission control
acts on that identity. A single tenant can loop sandbox creation requests and exhaust
cluster pod quota, node capacity, and network IPs for all other tenants with zero
HTTP-level protection.

Solution

Add a per-owner token bucket rate limiter checked immediately after owner identity
is resolved.

Behaviour:

  • Burst: 5 requests (absorbs short spikes without penalising normal use)
  • Sustained rate: 0.5 req/s (one new sandbox every 2 seconds per owner)
  • Response when limited: 429 Too Many Requests + Retry-After: N header so clients back off gracefully
  • Stale eviction: per-owner state removed after 10 min of inactivity (prevents unbounded memory growth)
  • Unauthenticated flows unaffected: check is skipped when ownerID == "" so integration tests, CLI tools, and admin flows continue working

Each owner gets an independent bucket — a noisy tenant cannot affect another tenant's quota.

Changes

File Change
pkg/workloadmanager/rate_limiter.go New — OwnerRateLimiter backed by golang.org/x/time/rate
pkg/workloadmanager/rate_limiter_test.go New — 7 tests covering burst, block, per-owner isolation, RetryAfter, stale eviction
pkg/workloadmanager/handlers.go Wire rate limit check after resolveSandboxOwnerID
pkg/workloadmanager/server.go Add rateLimiter field, initialize in NewServer

Tests

=== RUN   TestOwnerRateLimiter_AllowsBurst          --- PASS
=== RUN   TestOwnerRateLimiter_BlocksAfterBurst     --- PASS
=== RUN   TestOwnerRateLimiter_IndependentPerOwner  --- PASS
=== RUN   TestOwnerRateLimiter_RetryAfter           --- PASS
=== RUN   TestOwnerRateLimiter_StaleEviction        --- PASS
=== RUN   TestServerCreateSandbox_RateLimited       --- PASS
=== RUN   TestOwnerRateLimiter_NilSafe              --- PASS
ok  github.com/volcano-sh/agentcube/pkg/workloadmanager

All existing tests continue to pass.

🤖 Generated with Claude Code

POST /sandbox already resolves owner identity via resolveSandboxOwnerID
but placed no admission control on that identity. A single tenant could
loop sandbox creation requests and exhaust cluster pod quota, node
capacity, and network IPs for all other tenants with no HTTP-level
protection.

Add a per-owner token bucket rate limiter (burst=5, rate=0.5/s) checked
immediately after owner identity is resolved in handleSandboxCreate.
When the bucket is empty the handler returns 429 Too Many Requests with
a Retry-After header so clients can back off gracefully. Each owner gets
an independent bucket so a noisy tenant cannot affect others.

Stale per-owner state is evicted after 10 minutes of inactivity to
prevent unbounded memory growth in long-running deployments.

The check is skipped when ownerID is empty (unauthenticated flows,
integration tests, CLI tools) so existing admin paths are unaffected.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 78.94737% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.95%. Comparing base (f521ae9) to head (33b9a71).
⚠️ Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
pkg/workloadmanager/handlers.go 0.00% 4 Missing and 1 partial ⚠️
pkg/workloadmanager/rate_limiter.go 93.75% 2 Missing ⚠️
pkg/workloadmanager/server.go 0.00% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #512      +/-   ##
==========================================
+ Coverage   60.45%   63.95%   +3.49%     
==========================================
  Files          35       36       +1     
  Lines        3616     3695      +79     
==========================================
+ Hits         2186     2363     +177     
+ Misses       1211     1097     -114     
- Partials      219      235      +16     
Flag Coverage Δ
unittests 63.95% <78.94%> (+3.49%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… DCO sign-off

- Extract inline rate-limit block into checkOwnerRateLimit helper to keep
  handleSandboxCreate cyclomatic complexity within gocyclo limit (<=15)
- Promote golang.org/x/time from indirect to direct in go.mod (codegen check)
- Add DCO Signed-off-by to satisfy Linux Foundation contributor requirements

Signed-off-by: Soumadip Patra <tanmaytanmay1906@gmail.com>
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kevin-wangzefeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants