Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions evals/terminal-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,12 @@ The task container must provide Debian/Ubuntu Linux with glibc, root access, Bas
Linux x86_64 and aarch64 have separate pinned Node archive checksums.
The installer fails explicitly on unsupported platforms and refuses to reuse an existing installation.

[pins.toml](src/diffex_terminal_bench/pins.toml) records Node `22.23.2`, Diffex `0.2.7`, the public npm registry, archive checksums, npm integrity, and a 600-second installation deadline.
[pins.toml](src/diffex_terminal_bench/pins.toml) records Node `22.23.2`, Diffex `0.3.0`, the public npm registry, archive checksums, npm integrity, and a 600-second installation deadline.
The deadline includes apt prerequisites, Node download, npm installation, and all three CLI checks.
GNU `timeout` terminates the installer process group at the deadline and kills it after a ten-second grace period.
Harbor's command transport allows another five seconds to collect the result.

Setup installs Node under `/opt/diffex-terminal-bench/node` and uses `npm install --ignore-scripts --save-exact --registry=https://registry.npmjs.org/ @diffexai/diffex@0.2.7` with a private prefix.
Setup installs Node under `/opt/diffex-terminal-bench/node` and uses `npm install --ignore-scripts --save-exact --registry=https://registry.npmjs.org/ @diffexai/diffex@0.3.0` with a private prefix.
An isolated home and npm cache prevent host or task npm settings from changing installation.
Node archive SHA-256, registry metadata, installed package version, and npm lockfile integrity must match the checked-in pins.
The `/usr/local/bin/diffex` wrapper always invokes the pinned Node binary.
Expand Down Expand Up @@ -132,7 +132,7 @@ Run `npm run check` from the repository root after changing code.
### Recorded validation

The following historical smoke records used Diffex 0.2.2.
They do not qualify the updated Diffex 0.2.6 / Terminal-Bench 2.1 configuration.
They do not qualify the updated Diffex 0.3.0 / Terminal-Bench 2.1 configuration.

On 2026-09-14, both pinned Ubuntu images passed through Harbor 0.23.0 on Docker Desktop with an arm64 host.

Expand Down Expand Up @@ -250,11 +250,11 @@ Startup network updates are disabled with the supported offline setting so the p

The `context-only` project policy retains ancestor `AGENTS.md`/`CLAUDE.md` context and disables project trust, discovered extensions, skills, prompt templates, and themes.
It does not pass `--no-context-files` or replace the system prompt.
The pinned npm `0.2.6` release includes `dist/AGENTS.md`, which matches the adapter's checked-in release instructions.
The pinned npm `0.3.0` release includes `dist/AGENTS.md`, which matches the adapter's checked-in release instructions.
The adapter retains that packaged context without adding a global duplicate.
The effective system prompt must contain the release instructions, and their SHA-256 is recorded.

The checked-in `benchmark.toml` and `execution.yaml` enable subagents, adding `spawn_agent`, `agent_action`, and `wait_for_agents` to the explicit tool allowlist.
The checked-in `benchmark.toml` and `execution.yaml` enable subagents, adding `spawn_agent`, `agent_action`, and `agent_status` to the explicit tool allowlist.
Diffex decides whether to call those tools; enabling them does not require a child on every task.
Children share their trial's sandbox, and the pinned CLI owns child scheduling and its default concurrency limit.
The staged runner's `-n/--concurrency` controls simultaneous trials and sandboxes separately.
Expand Down Expand Up @@ -304,7 +304,7 @@ Tool errors alone do not fail execution: the assistant can inspect and recover f
`DiffexExecutionError` subclasses Harbor's `NonZeroAgentExitCodeError`, so infrastructure failures retain exception information rather than becoming ordinary zero-reward results.
The adapter never sets or infers verifier reward.
A completed assistant that produces an incorrect solution returns normally and can earn zero from the verifier.
The pinned `0.2.6` release returns nonzero for terminal assistant errors and aborts in JSON mode as well as text mode.
The pinned `0.3.0` release returns nonzero for terminal assistant errors and aborts in JSON mode as well as text mode.
The adapter also parses native events to classify provider failures and reject incomplete output independently of process exit status.

The launcher writes stdout and stderr directly to files and retains native sessions during execution.
Expand Down
4 changes: 2 additions & 2 deletions evals/terminal-bench/RELEASE_GATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ Each invocation extracts a fresh copy of the verified archive; it does not reuse
The preparation command writes `results/release-gate/dataset.json` without creating sandboxes or calling a model.

Harbor 0.23.0 and Daytona SDK 0.210.0 are locked in the benchmark's private Python environment.
Node 22.23.2, Diffex 0.2.6, npm integrity, registry, and Node archive checksums come from [pins.toml](src/diffex_terminal_bench/pins.toml).
Node 22.23.2, Diffex 0.3.0, npm integrity, registry, and Node archive checksums come from [pins.toml](src/diffex_terminal_bench/pins.toml).
The default is `openai/gpt-5.6-luna`, `max` thinking, tools `read,bash,edit,write,update_plan`, enabled subagents, disabled evolution, and `context-only` project resources.
Enabled subagents add `spawn_agent`, `agent_action`, and `wait_for_agents` to the model's tools; Diffex chooses whether to use them.
Enabled subagents add `spawn_agent`, `agent_action`, and `agent_status` to the model's tools; Diffex chooses whether to use them.
Children execute inside the parent trial's sandbox and their sessions and usage remain in the trial evidence.
Change and commit the configuration before starting a new qualification sequence.
Every subsequent stage must match the exact recorded identity, including the adapter commit and dependency lock.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"mistral": ("MISTRAL_API_KEY",),
}
BUILTIN_TOOLS = ("read", "bash", "edit", "write", "update_plan", "grep", "find", "ls")
SUBAGENT_TOOLS = ("spawn_agent", "agent_action", "wait_for_agents")
SUBAGENT_TOOLS = ("spawn_agent", "agent_action", "agent_status")


class ExecutionOptions(InstalledAgentOptions):
Expand Down
4 changes: 2 additions & 2 deletions evals/terminal-bench/src/diffex_terminal_bench/pins.toml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
harbor_version = "0.23.0"
node_version = "22.23.2"
diffex_version = "0.2.7"
diffex_version = "0.3.0"
npm_registry = "https://registry.npmjs.org/"
diffex_integrity = "sha512-+LXbkWtgLZ/AMxN3uKnUFpCeG1iw1mH4DMKhBx1SjKYmmvZw7NWFCiLdlMFSFrHTt6N5IE9YV8FIjefzxx36eg=="
diffex_integrity = "sha512-MeD6G7az/b/SSj4yXrtN3bgRB4tPFwR26pVToPuvBDOTr8VU0ZtKNz0f/GLt5d4X4dx5d4/8FQzWn16LpYhSyw=="
# Must match the instructions shipped by diffex_version, not the current checkout.
release_agents_sha256 = "6f194cd7f04d80e08cd3c8b22096de33ba5dbb2ba42176567c466579f8ec26db"
setup_timeout_sec = 600
Expand Down
5 changes: 3 additions & 2 deletions evals/terminal-bench/tests/test_execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ async def upload(source, target):
agent = DiffexAgent(logs_dir=self.logs_dir, model_name="openai/gpt-5", subagents="enabled",
extra_env={"OPENAI_API_KEY": "test-only"})
await agent.run("Task", environment, AgentContext())
self.assertTrue({"spawn_agent", "agent_action", "wait_for_agents"}.issubset(configs[0]["tools"]))
self.assertEqual(configs[0]["tools"], ["read", "bash", "edit", "write", "update_plan",
"spawn_agent", "agent_action", "agent_status"])
self.assertTrue(any("chown -R 1000" in call.kwargs["command"] for call in environment.exec.call_args_list))

async def test_checked_in_eval_policies_expose_subagent_tools(self):
Expand All @@ -134,7 +135,7 @@ async def upload(source, target):
await agent.run("Task", environment, AgentContext())
self.assertEqual(requested[0]["subagents"], "enabled")
self.assertEqual(requested[0]["tools"], ["read", "bash", "edit", "write", "update_plan",
"spawn_agent", "agent_action", "wait_for_agents"])
"spawn_agent", "agent_action", "agent_status"])

async def test_missing_credentials_and_ambiguous_models_fail_before_exec(self):
with patch.dict(os.environ, {}, clear=True):
Expand Down
75 changes: 75 additions & 0 deletions packages/coding-agent/docs/background-processes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Background process lifetime

Local shell commands normally belong to their Diffex session.
Abort, disposal, print-mode exit, and normal subagent completion terminate surviving background jobs, including jobs started with `nohup`.

An environment supervisor can explicitly take responsibility for successful background jobs on Linux and macOS:

```sh
diffex --print \
--background-process-lifetime=environment \
--background-process-ownership-dir=/run/trial/diffex-shell-ownership \
"Start the task service"
```

The default lifetime is `session`.
Environment lifetime requires an explicit ownership directory and applies to all built-in local bash executions, direct session bash commands, and child sessions in that invocation.
Relative CLI directory paths are resolved against the initial working directory, including after session replacement.
Custom or remote `BashOperations` implementations manage their own process lifecycle.

A shell process group becomes environment-owned only when its launching shell exits with code `0` before cancellation or timeout.
Handoff happens at shell exit, before the next provider request or final assistant response.
Failed commands and active commands remain session-owned.
Once handed off, background processes survive later provider errors, interruption, child failure, session replacement, and disposal.
The failed or interrupted invocation still retains its failure outcome.

The environment supervisor must terminate transferred processes when the environment is destroyed.
Services should redirect their output to files: environment mode bounds inherited stdout/stderr draining to 100 milliseconds after shell exit so a continuously logging service cannot hold the CLI open.

## SDK

```ts
const { session } = await createAgentSession({
backgroundProcessPolicy: {
lifetime: "environment",
ownershipDirectory: "/run/trial/diffex-shell-ownership",
},
});
```

The same option is available on `AgentSessionConfig` and `createAgentSessionFromServices`.
SDK ownership directories must be absolute.
Runtime factories should capture this invocation-level policy and pass it to each replacement session.
Child sessions inherit the parent policy, including children restored within the invocation.

## Supervisor handoff records

Use a separate ownership directory for each environment execution.
Diffex writes one versioned JSON file per shell execution, named `<executionId>.json`.
The execution ID also appears in the inherited `DIFFEX_SHELL_EXECUTION_ID` environment variable.
Records contain `version: 1`, `executionId`, `processGroupId`, `supervisorPid`, `owner`, `state`, and a snapshot of live `members`.
Each member contains its PID, process group ID, and process start identity.
Linux identities combine the boot ID and `/proc/<pid>/stat` start ticks; macOS identities use `ps lstart` in UTC and the C locale.

The initial record is `owner: "session", state: "active"`.
Successful handoff replaces it with `owner: "environment", state: "background"`.
Confirmed termination of an untransferred group records `owner: "session", state: "terminated"`.
Member snapshots are evidence at publication time, not a continuously updated process inventory.

Each replacement is flushed to a temporary file and atomically renamed before the process group is removed from session and global shutdown tracking.
Only the completed `.json` file establishes ownership; ignore `.tmp` files.
The atomic rename is the handoff commit point, so a supervisor can recover ownership after a CLI crash or SIGKILL without depending on session-history flushing or a successful final response.
Publication failure leaves ownership with Diffex and fails the shell operation.
Publication and in-memory release contain no asynchronous boundary, so ordinary cancellation cannot interleave with the handoff.

A supervisor must validate current process identities before signaling or preserving a recorded group and follow inherited execution markers for newly created descendants.
Never signal a numeric PID or group ID from a stale record without checking identity.
Missing, invalid, or ambiguous records are not proof of an environment handoff.
Records are a cooperation protocol within the trusted trial environment, not a security boundary against task code that can modify that environment.

## Terminal-Bench integration

This option provides the Diffex ownership boundary described by process-lifetime issue 1.
The adapter must enable it for every task once its pinned Diffex release supports these flags and its pre-verifier cleanup consumes the ownership records.
The separate adapter cleanup change in issue 2 is still required: marker-based cleanup that kills all trial processes will also kill transferred services.
Final environment deletion remains responsible for removing task services after verification.
37 changes: 37 additions & 0 deletions packages/coding-agent/src/cli/args.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import type { TuiMode } from "../core/settings-manager.ts";
export type Mode = "text" | "json" | "rpc";

export interface Args {
backgroundProcessLifetime?: "session" | "environment";
backgroundProcessOwnershipDir?: string;
provider?: string;
model?: string;
apiKey?: string;
Expand Down Expand Up @@ -82,6 +84,25 @@ export function parseArgs(args: string[]): Args {
if (mode === "text" || mode === "json" || mode === "rpc") {
result.mode = mode;
}
} else if (arg === "--background-process-lifetime" || arg.startsWith("--background-process-lifetime=")) {
const value = arg.includes("=") ? arg.slice(arg.indexOf("=") + 1) : args[i + 1];
if (!arg.includes("=") && value !== undefined && !value.startsWith("-")) i++;
if (value === "session" || value === "environment") result.backgroundProcessLifetime = value;
else
result.diagnostics.push({
type: "error",
message: "--background-process-lifetime requires session or environment",
});
} else if (
arg === "--background-process-ownership-dir" ||
arg.startsWith("--background-process-ownership-dir=")
) {
const value = arg.includes("=") ? arg.slice(arg.indexOf("=") + 1) : args[i + 1];
if (value && !value.startsWith("-")) {
result.backgroundProcessOwnershipDir = value;
if (!arg.includes("=")) i++;
} else
result.diagnostics.push({ type: "error", message: "--background-process-ownership-dir requires a path" });
} else if (arg === "--continue" || arg === "-c") {
result.continue = true;
} else if (arg === "--resume" || arg === "-r") {
Expand Down Expand Up @@ -222,6 +243,18 @@ export function parseArgs(args: string[]): Args {
}
}

if (result.backgroundProcessLifetime === "environment" && !result.backgroundProcessOwnershipDir) {
result.diagnostics.push({
type: "error",
message: "Environment lifetime requires --background-process-ownership-dir",
});
}
if (result.backgroundProcessOwnershipDir && result.backgroundProcessLifetime !== "environment") {
result.diagnostics.push({
type: "error",
message: "--background-process-ownership-dir requires environment lifetime",
});
}
return result;
}

Expand Down Expand Up @@ -259,6 +292,10 @@ ${chalk.bold("Options:")}
--append-system-prompt <text> Append text or file contents to the system prompt (can be used multiple times)
--mode <mode> Output mode: text (default), json, or rpc
--print, -p Non-interactive mode: process prompt and exit
--background-process-lifetime <session|environment>
Background shell ownership (default: session)
--background-process-ownership-dir <path>
Supervisor handoff records (required for environment lifetime)
--continue, -c Continue previous session
--resume, -r Select a session to resume
--session <path|id> Use specific session file or partial UUID
Expand Down
2 changes: 2 additions & 0 deletions packages/coding-agent/src/core/agent-session-services.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ export interface CreateAgentSessionServicesOptions {
* have been resolved against those services.
*/
export interface CreateAgentSessionFromServicesOptions {
backgroundProcessPolicy?: CreateAgentSessionOptions["backgroundProcessPolicy"];
services: AgentSessionServices;
sessionManager: SessionManager;
sessionStartEvent?: SessionStartEvent;
Expand Down Expand Up @@ -240,6 +241,7 @@ export async function createInternalAgentSessionFromServices(
registerSubagentRuntimeDiagnosticSink(options.sessionManager, options.services.diagnostics);
return createInternalAgentSession(
{
backgroundProcessPolicy: options.backgroundProcessPolicy,
cwd: options.services.cwd,
agentDir: options.services.agentDir,
modelRuntime: options.services.modelRuntime,
Expand Down
11 changes: 9 additions & 2 deletions packages/coding-agent/src/core/agent-session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ import type {
HarnessWorkspaceIdentity,
} from "@diffexai/diffex-harness-state";
import { getThemeByName, theme } from "../modes/interactive/theme/theme.ts";
import { OwnedShellProcesses } from "../utils/owned-shell-processes.ts";
import { type BackgroundProcessPolicy, OwnedShellProcesses } from "../utils/owned-shell-processes.ts";
import { canonicalizePath, resolvePath } from "../utils/paths.ts";
import { sleep } from "../utils/sleep.ts";
import { normalizeToolResultImages } from "../utils/tool-result-images.ts";
Expand Down Expand Up @@ -303,6 +303,8 @@ function withoutDeletedHeaders(headers: ProviderHeaders | undefined): Record<str
}

export interface AgentSessionConfig {
/** Lifetime of background groups launched by successful local shell commands. */
backgroundProcessPolicy?: BackgroundProcessPolicy;
agent: Agent;
sessionManager: SessionManager;
settingsManager: SettingsManager;
Expand Down Expand Up @@ -520,7 +522,7 @@ export class AgentSession {
private readonly _bashAbortControllers = new Set<AbortController>();
private _pendingBashMessages: BashExecutionMessage[] = [];
private readonly _ownedOperations = new Set<Promise<unknown>>();
private readonly _shellProcesses = new OwnedShellProcesses();
private readonly _shellProcesses: OwnedShellProcesses;
private readonly _bashOperations = new Set<Promise<BashResult>>();

// Extension system
Expand Down Expand Up @@ -583,6 +585,7 @@ export class AgentSession {
private _disposePromise: Promise<void> | undefined;

constructor(config: AgentSessionConfig) {
this._shellProcesses = new OwnedShellProcesses(config.backgroundProcessPolicy);
this.agent = config.agent;
this.sessionManager = config.sessionManager;
this.settingsManager = config.settingsManager;
Expand Down Expand Up @@ -3089,6 +3092,10 @@ export class AgentSession {
await this._shellProcesses.terminateAll();
}

get backgroundProcessPolicy(): BackgroundProcessPolicy {
return this._shellProcesses.policy;
}

get ownedShellProcessCount(): number {
return this._shellProcesses.count;
}
Expand Down
4 changes: 4 additions & 0 deletions packages/coding-agent/src/core/sdk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { clampThinkingLevel, type Message, type Model, streamSimple } from "@dif
import type { HarnessStateStore, HarnessVersionBundle } from "@diffexai/diffex-harness-state";
import { EvolutionMemoryWriter, FileEvolutionPersistence } from "#evolution";
import { getAgentDir, VERSION } from "../config.ts";
import type { BackgroundProcessPolicy } from "../utils/owned-shell-processes.ts";
import { resolvePath } from "../utils/paths.ts";
import { AgentSession } from "./agent-session.ts";
import type { AgentSessionLifecycleObserver } from "./agent-session-lifecycle.ts";
Expand Down Expand Up @@ -81,6 +82,8 @@ export interface SubagentOptions extends SubagentSupervisionOptions {
}

export interface CreateAgentSessionOptions {
/** Local shell background lifetime. Defaults to session-owned cleanup. */
backgroundProcessPolicy?: BackgroundProcessPolicy;
/** Working directory for project-local discovery. Default: process.cwd() */
cwd?: string;
/** Global config directory. Default: ~/.diffex/agent */
Expand Down Expand Up @@ -665,6 +668,7 @@ export async function createInternalAgentSession(
}

const session = new AgentSession({
backgroundProcessPolicy: options.backgroundProcessPolicy,
agent,
sessionManager,
settingsManager,
Expand Down
Loading
Loading