Skip to content

fix(picod): propagate HTTP request context to subprocess in ExecuteHandler - #505

Open
SoumaCloud wants to merge 4 commits into
volcano-sh:mainfrom
SoumaCloud:fix/499-picod-request-context
Open

SoumaCloud wants to merge 4 commits into
volcano-sh:mainfrom
SoumaCloud:fix/499-picod-request-context

Conversation

@SoumaCloud

@SoumaCloud SoumaCloud commented Nov 5, 2026 •

Copy link
Copy Markdown

Problem

ExecuteHandler creates the subprocess context from context.Background():

ctx, cancel := context.WithTimeout(context.Background(), timeoutDuration)
cmd := exec.CommandContext(ctx, req.Command[0], req.Command[1:]...)

context.Background() is never cancelled. When an HTTP client disconnects, Gin cancels c.Request.Context() — but because the subprocess context has no parent relationship with the request context, the subprocess keeps running until the timeout expires. For a command like sleep 3600 with a 60-second default timeout, the subprocess runs for a full 60 seconds after the caller has gone.

Root Cause

The subprocess context must be derived from the request context, not from context.Background(). When the parent context is cancelled (client disconnect), Go's context package propagates the cancellation to all children, which causes exec.CommandContext to send SIGKILL to the process group.

Disconnect semantics

Disconnecting the caller should terminate execution. The agent runs commands on behalf of the caller; if the caller is gone there is no recipient for the response, and a long-running subprocess continuing in the background exhausts host resources. Chaining from c.Request.Context() enforces this automatically with no extra bookkeeping.

Changes

pkg/picod/execute.go

  • Change context.WithTimeout(context.Background(), timeoutDuration) → context.WithTimeout(c.Request.Context(), timeoutDuration). One argument change.

pkg/picod/execute_test.go

  • Add TestExecuteHandler_RequestCancellation_KillsSubprocess: starts sleep 10, cancels the request context after 100 ms, asserts the handler returns within 5 s. Without the fix the handler blocks for 30 s (the request timeout).

pkg/picod/picod_test.go — Router/SDK-path reproducer

  • Add TestRouterPath_DisconnectKillsSubprocess: starts a real httptest.Server backed by the full Gin engine (JWT auth middleware included), makes a real HTTP request for sleep 10 using ts.Client().Do(), cancels the client-side request context after 200 ms, asserts the HTTP round-trip ends within 5 s. This exercises the complete stack: network → auth middleware → router → handler → subprocess.

How to verify

# Direct handler test
go test ./pkg/picod/... -run TestExecuteHandler_RequestCancellation_KillsSubprocess -v

# Full Router/SDK-path test
go test ./pkg/picod/... -run TestRouterPath_DisconnectKillsSubprocess -v

Both should print --- PASS. Without the fix, both time out after 5 s with:

ExecuteHandler did not return after request context cancellation; subprocess was not killed (pre-fix behavior)

Fixes #499

…ndler

ExecuteHandler created a timeout context from context.Background(), making
the subprocess's lifetime independent of the HTTP client. If the caller
disconnected mid-execution, the subprocess kept running until the fixed
timeout expired.

Chain the timeout context from c.Request.Context() instead. When the HTTP
client disconnects, Gin cancels c.Request.Context(), which cancels the
child context, which causes exec.CommandContext to send SIGKILL to the
subprocess — establishing the expected disconnect-kills-subprocess semantics.

Add TestExecuteHandler_RequestCancellation_KillsSubprocess as a minimal
reproducer through the same gin.Context path: it cancels the request context
100 ms into a 10-second sleep and asserts the handler returns within 5 s.

Fixes volcano-sh#499

Signed-off-by: Soumadip <Soum23dip@gmail.com>
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign acsoto for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@volcano-sh-bot

Copy link
Copy Markdown
Contributor

Welcome @SoumaCloud! It looks like this is your first PR to volcano-sh/agentcube 🎉

@codecov-commenter

codecov-commenter commented Nov 5, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.47%. Comparing base (f521ae9) to head (aaf6140).
⚠️ Report is 25 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #505      +/-   ##
==========================================
+ Coverage   60.45%   62.47%   +2.02%     
==========================================
  Files          35       35              
  Lines        3616     3659      +43     
==========================================
+ Hits         2186     2286     +100     
+ Misses       1211     1147      -64     
- Partials      219      226       +7     
Flag Coverage Δ
unittests 62.47% <100.00%> (+2.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…linter

Signed-off-by: Soumadip <Soum23dip@gmail.com>
…rocess

Add TestRouterPath_DisconnectKillsSubprocess to picod_test.go. It goes through
the full stack — httptest.Server, JWT auth middleware, gin router, ExecuteHandler —
using the same helpers the existing e2e tests use. The test cancels the client
request context 200 ms into a 10-second sleep and asserts the HTTP round-trip
ends within 5 s, confirming the subprocess was killed on disconnect.

Establishes disconnect semantics: cancelling the caller's context should
terminate execution. exec.CommandContext propagates the cancellation as SIGKILL,
so chaining from c.Request.Context() provides this automatically.

Signed-off-by: Soumadip <Soum23dip@gmail.com>
@SoumaCloud

Copy link
Copy Markdown
Author

Added TestRouterPath_DisconnectKillsSubprocess in pkg/picod/picod_test.go (latest commit f0a990e).

What the reproducer does (Router/SDK path):

The test goes through the full stack — httptest.NewServer(server.engine) → JWT auth middleware → gin router → ExecuteHandler — using the same setupTestServer / createToken helpers the existing end-to-end tests use:

  1. Starts a real HTTP server with the gin engine (auth middleware wired in).
  2. POSTs {"command":["sleep","10"],"timeout":"60s"} with a valid JWT.
  3. Cancels the client-side request context 200 ms after the subprocess starts (simulates disconnect).
  4. Asserts the HTTP round-trip ends within 5 s.

Without the fix (context.Background() parent), sleep 10 would run to completion and the round-trip would block for the full 60-second timeout. With the fix (c.Request.Context() parent), gin cancels the request context on disconnect, which propagates through exec.CommandContext as SIGKILL — the test passes in ~200 ms.

On disconnect semantics:

Disconnecting the caller should terminate execution. The agent runs commands on behalf of the caller; if the caller is gone there is no recipient for the response, and long-running commands continuing in the background exhaust host resources. exec.CommandContext with c.Request.Context() as the parent provides this behaviour atomically with no extra goroutine or signal bookkeeping.

setupTestServer returned *Server as its first value but no caller ever used it,
triggering the unparam linter. Removed the unused return value and updated both
call sites accordingly.

Also fix 'cancelled' misspelling in comment (should be 'canceled').

Signed-off-by: Soumadip <Soum23dip@gmail.com>
@volcano-sh-bot

Copy link
Copy Markdown
Contributor

Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

  • bedc5d7 fix(picod): propagate HTTP request context to subprocess in ExecuteHandler
Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@SoumaCloud

Copy link
Copy Markdown
Author

Thanks @acsoto for the detailed explanation. I've now read through the codebase to understand whether ExecuteHandler is actually decoupled in the E2B sense, and the evidence shows it is not:

pkg/store/interface.go — the full Store interface:

type Store interface {
    Ping(ctx context.Context) error
    GetSandboxBySessionID(ctx context.Context, sessionID string) (*types.SandboxInfo, error)
    StoreSandbox(ctx context.Context, sandboxStore *types.SandboxInfo) error
    UpdateSandbox(ctx context.Context, sandboxStore *types.SandboxInfo) error
    DeleteSandboxBySessionID(ctx context.Context, sessionID string) error
    ListExpiredSandboxes(ctx context.Context, before time.Time, limit int64) ([]*types.SandboxInfo, error)
    ListInactiveSandboxes(ctx context.Context, before time.Time, limit int64) ([]*types.SandboxInfo, error)
    UpdateSessionLastActivity(ctx context.Context, sessionID string, at time.Time) error
    Close() error
}

The store handles sandbox lifecycle only — no method stores or retrieves command execution results. There is no StoreResult, no GetResult, no task-ID-based polling.

pkg/picod/execute.go lines 128–161 — the handler is fully synchronous:

var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr

start := time.Now()
err := cmd.Run()          // blocks until the subprocess exits
duration := time.Since(start).Seconds()

// ...

c.JSON(http.StatusOK, ExecuteResponse{   // response written directly to the HTTP connection
    Stdout:    stdout.String(),
    Stderr:    stderr.String(),
    ExitCode:  exitCode,
    ...
})

cmd.Run() blocks. The response is written directly to c.JSON — there is no task ID returned, no background goroutine, no result queue, and no polling endpoint. If the client disconnects before cmd.Run() returns, the response is never delivered but the subprocess keeps running until the timeout — there is no benefit to the decoupling.

The E2B model works because E2B stores the result asynchronously and clients poll for it later. This codebase does the opposite: it is a synchronous request-response pattern. context.Background() here does not add E2B-style decoupling; it just orphans subprocesses on disconnect.

Happy to revise the approach if there is a design intent I'm still missing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PicoD subprocess execution does not respect incoming HTTP request cancellation

3 participants