Repository navigation
fix(picod): propagate HTTP request context to subprocess in ExecuteHandler - #505
SoumaCloud wants to merge 4 commits into
Conversation
…ndler ExecuteHandler created a timeout context from context.Background(), making the subprocess's lifetime independent of the HTTP client. If the caller disconnected mid-execution, the subprocess kept running until the fixed timeout expired. Chain the timeout context from c.Request.Context() instead. When the HTTP client disconnects, Gin cancels c.Request.Context(), which cancels the child context, which causes exec.CommandContext to send SIGKILL to the subprocess — establishing the expected disconnect-kills-subprocess semantics. Add TestExecuteHandler_RequestCancellation_KillsSubprocess as a minimal reproducer through the same gin.Context path: it cancels the request context 100 ms into a 10-second sleep and asserts the handler returns within 5 s. Fixes volcano-sh#499 Signed-off-by: Soumadip <Soum23dip@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @SoumaCloud! It looks like this is your first PR to volcano-sh/agentcube 🎉 |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #505 +/- ##
==========================================
+ Coverage 60.45% 62.47% +2.02%
==========================================
Files 35 35
Lines 3616 3659 +43
==========================================
+ Hits 2186 2286 +100
+ Misses 1211 1147 -64
- Partials 219 226 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…linter Signed-off-by: Soumadip <Soum23dip@gmail.com>
…rocess Add TestRouterPath_DisconnectKillsSubprocess to picod_test.go. It goes through the full stack — httptest.Server, JWT auth middleware, gin router, ExecuteHandler — using the same helpers the existing e2e tests use. The test cancels the client request context 200 ms into a 10-second sleep and asserts the HTTP round-trip ends within 5 s, confirming the subprocess was killed on disconnect. Establishes disconnect semantics: cancelling the caller's context should terminate execution. exec.CommandContext propagates the cancellation as SIGKILL, so chaining from c.Request.Context() provides this automatically. Signed-off-by: Soumadip <Soum23dip@gmail.com>
|
Added What the reproducer does (Router/SDK path): The test goes through the full stack —
Without the fix ( On disconnect semantics: Disconnecting the caller should terminate execution. The agent runs commands on behalf of the caller; if the caller is gone there is no recipient for the response, and long-running commands continuing in the background exhaust host resources. |
setupTestServer returned *Server as its first value but no caller ever used it, triggering the unparam linter. Removed the unused return value and updated both call sites accordingly. Also fix 'cancelled' misspelling in comment (should be 'canceled'). Signed-off-by: Soumadip <Soum23dip@gmail.com>
|
Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages. The list of commits with invalid commit messages:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Thanks @acsoto for the detailed explanation. I've now read through the codebase to understand whether
type Store interface {
Ping(ctx context.Context) error
GetSandboxBySessionID(ctx context.Context, sessionID string) (*types.SandboxInfo, error)
StoreSandbox(ctx context.Context, sandboxStore *types.SandboxInfo) error
UpdateSandbox(ctx context.Context, sandboxStore *types.SandboxInfo) error
DeleteSandboxBySessionID(ctx context.Context, sessionID string) error
ListExpiredSandboxes(ctx context.Context, before time.Time, limit int64) ([]*types.SandboxInfo, error)
ListInactiveSandboxes(ctx context.Context, before time.Time, limit int64) ([]*types.SandboxInfo, error)
UpdateSessionLastActivity(ctx context.Context, sessionID string, at time.Time) error
Close() error
}The store handles sandbox lifecycle only — no method stores or retrieves command execution results. There is no
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
start := time.Now()
err := cmd.Run() // blocks until the subprocess exits
duration := time.Since(start).Seconds()
// ...
c.JSON(http.StatusOK, ExecuteResponse{ // response written directly to the HTTP connection
Stdout: stdout.String(),
Stderr: stderr.String(),
ExitCode: exitCode,
...
})
The E2B model works because E2B stores the result asynchronously and clients poll for it later. This codebase does the opposite: it is a synchronous request-response pattern. Happy to revise the approach if there is a design intent I'm still missing. |
Problem
ExecuteHandlercreates the subprocess context fromcontext.Background():context.Background()is never cancelled. When an HTTP client disconnects, Gin cancelsc.Request.Context()— but because the subprocess context has no parent relationship with the request context, the subprocess keeps running until the timeout expires. For a command likesleep 3600with a 60-second default timeout, the subprocess runs for a full 60 seconds after the caller has gone.Root Cause
The subprocess context must be derived from the request context, not from
context.Background(). When the parent context is cancelled (client disconnect), Go's context package propagates the cancellation to all children, which causesexec.CommandContextto send SIGKILL to the process group.Disconnect semantics
Disconnecting the caller should terminate execution. The agent runs commands on behalf of the caller; if the caller is gone there is no recipient for the response, and a long-running subprocess continuing in the background exhausts host resources. Chaining from
c.Request.Context()enforces this automatically with no extra bookkeeping.Changes
pkg/picod/execute.gocontext.WithTimeout(context.Background(), timeoutDuration)→context.WithTimeout(c.Request.Context(), timeoutDuration). One argument change.pkg/picod/execute_test.goTestExecuteHandler_RequestCancellation_KillsSubprocess: startssleep 10, cancels the request context after 100 ms, asserts the handler returns within 5 s. Without the fix the handler blocks for 30 s (the request timeout).pkg/picod/picod_test.go— Router/SDK-path reproducerTestRouterPath_DisconnectKillsSubprocess: starts a realhttptest.Serverbacked by the full Gin engine (JWT auth middleware included), makes a real HTTP request forsleep 10usingts.Client().Do(), cancels the client-side request context after 200 ms, asserts the HTTP round-trip ends within 5 s. This exercises the complete stack: network → auth middleware → router → handler → subprocess.How to verify
Both should print
--- PASS. Without the fix, both time out after 5 s with:Fixes #499