Repository navigation
Find and fix Responses disconnects across official and external routes #114
Description
Problem
Codex Desktop sessions routed through CodexHub frequently reconnect and can end with:
stream disconnected before completion: error sending request for url (http://127.0.0.1:9099/v1/responses)
The symptom affects both official OpenAI models and configured external GLM-5.2 routes, across the v0.1.4 release and the latest dev portable. A healthy local /health endpoint therefore does not prove an end-to-end Responses stream is healthy.
Current evidence
Official OpenAI route
A replay of the real 2026-07-13 14:15-15:05 Asia/Shanghai telemetry window produced six upstream transport failures for Codex App requests:
- GPT-5.6 Terra: two pre-response
RemoteDisconnectedretries for one >=1 MB request. - GPT-5.4 Mini: one pre-response
RemoteDisconnectedretry. - GPT-5.6 Sol: one pre-response
RemoteDisconnectedretry and one write timeout. - GPT-5.6 Terra: one post-open
IncompleteRead(0 bytes read)after 3,249 SSE lines / 325,905 bytes had already been relayed; Gateway recorded status 502. - Two exact downstream cancellations were separately excluded by the analyzer.
The corresponding Codex UI showed reconnect attempts and IncompleteRead(0 bytes read).
External GLM route
The reporter supplied the matching event and access logs. The configured route identifies itself as company/glm-5.2; do not infer Ollama Cloud or another upstream from the model name alone.
The analyzer found 23 real upstream_retry events across two Codex App requests:
- One 347,480-byte request emitted 8 retries with
ConnectionResetError [WinError 10054]. - One 350,493-byte request emitted 15 retries with
ConnectionAbortedError [WinError 10053]. - Both advertised
max_attempts=30. - Neither request emitted a terminal
request_completeorrequest_errorevent in the supplied event log. - The access log nevertheless recorded HTTP 200 after the long-running requests, while
/healthcontinued returning 200. - Telemetry classified the same retries as
failure_phase=stream_bodyandfailure_side=upstream_open, an inconsistency that must be resolved before the fields drive retry policy.
Codex exhausted five reconnect attempts and reported the localhost /v1/responses stream-disconnect error.
These observations prove a cross-route user symptom and a missing/ambiguous Gateway terminal state, not a single-model failure. They do not yet prove one shared upstream-network root cause.
App versus CLI boundary
The reporter adds a durable observation from the faulty period beginning the prior evening: equivalent Codex CLI connections remained stable, while Codex Desktop App sessions disconnected. CLI-only clean runs are therefore negative controls, not a reproduction or clearance of the P0 defect. The primary causal boundary now includes the Desktop App/app-server/rollout and stream-consumer lifecycle in addition to Gateway and upstream transport.
Ranked root-cause hypotheses to test
- Codex Desktop App/app-server/rollout or its Responses stream consumer closes, cancels, or resumes the localhost request differently from Codex CLI.
- The shared effective system-proxy/network path terminates both official and external upstream connections.
- Gateway's Python transport differs materially from direct Codex traffic in proxy discovery, connection reuse, TCP/TLS setup, request writes, or SSE reads.
- A Windows-local component triggers WinError 10053 for the Gateway or Desktop process or their long-lived/reused connections.
- Individual upstream services reset connections independently, with payload size or stream duration as a contributing variable.
- Retry budgeting and post-open terminalization do not cause the first disconnect, but amplify it into a long reconnect storm and obscure the initiating phase.
Each hypothesis must be tested by changing one variable at a time. Do not infer the network path or root cause from provider identity or exception text alone.
Investigation priority
The initiating disconnect is P0. The Worker must first identify the causal boundary and show why the connection breaks.
Retry limits, terminal SSE behavior, and telemetry consistency are secondary containment. They may be implemented alongside a proven root-cause fix, but they cannot by themselves satisfy or close this Issue. A PR that only bounds retries or improves terminalization is a partial slice and must retain Refs #114.
Outcome
Identify, reproduce, and remove or productively mitigate the mechanism that causes Responses connections to break across official and external routes. Then ensure any residual fault is bounded and protocol-valid instead of becoming a reconnect storm.
Scope
- Build deterministic fault-injection replays for official passthrough and external Responses routes.
- Establish the initiating failure boundary before proposing retry or terminalization changes; a containment-only change is explicitly partial.
- Cover upstream-open
RemoteDisconnected, upstream-open/write timeout, established-connectionWinError 10053, and post-openIncompleteReadafter partial SSE. - Use a sealed/isolated Codex Desktop App environment as the primary reproducer. Compare App direct official, App through Gateway official, and App through Gateway external under one App build, host, network window, model/workload, and Responses configuration. Run Codex CLI only as a contemporaneous negative control. Do not use Computer Use or mutate the reporter's shared Desktop state.
- Record the effective system-proxy/explicit-proxy/direct route as evidence; do not change the user's global proxy, credentials, or shared Desktop configuration.
- Preserve
supports_websockets=falsefor mixed-provider mode. - Change only the proven Gateway transport/retry/SSE boundary. Protocol translation or provider policy changes require a discussion gate.
- Relate findings to closed diagnostic Issues Add official transport recurrence analyzer for post-open failures #12 and Diagnose generic Gateway upstream transport instability across providers #16 without reopening their already-completed analyzer scope.
Acceptance criteria
- One fast deterministic harness reproduces the exact localhost stream-disconnect class for both official and external Responses paths.
- A controlled same-workload A/B changes one variable at a time and identifies the initiating failure boundary: network/proxy path, Gateway transport implementation, Windows-local abort, or upstream service.
- The chosen root-cause fix or mitigation changes that A/B result and removes the disconnect, not merely the retry count.
- If the initiating cause is environmental, the product response (for example explicit per-process proxying, transport replacement, detection, or actionable diagnosis) is demonstrated; closure with only
root cause unknownis not accepted. - Pre-response transient retries are bounded by a documented request budget and cannot produce an unbounded/high-count retry storm.
- Once response headers/SSE content are exposed, Gateway never replays model content and never silently converts the failure into an unclassified socket close.
- Post-open failure behavior is verified against Codex Desktop App: it receives the chosen protocol-valid terminal outcome or one precise non-retryable error, without five opaque localhost reconnects. A contemporaneous CLI negative control remains stable under the same route/model/network window.
- Official and external paths emit the same sanitized phase/side/retry-budget/terminalization fields where their semantics match.
- A same-workload A/B distinguishes direct official traffic, official through Gateway, and GLM through Gateway under the same effective network route.
- The App-versus-CLI delta is captured with correlated App/app-server/rollout identity, client stream-consumer/continuation state, Gateway request phase, connection identity, and terminal outcome, showing which side closes first.
- No credential, global system-proxy, TUN, or shared Codex Desktop mutation is required.
- Regression tests cover open failure, partial-stream read failure, downstream cancellation, retry-budget exhaustion, and a healthy long stream.
- Relevant focused and full Python/Rust/frontend gates pass.
Verification
- Sealed/isolated Codex Desktop App profile/state/config as the primary surface, Responses HTTP, WebSocket disabled; App-bundled Codex CLI under isolated
CODEX_HOMEonly as the same-window negative control. - Existing analyzer command over a sanitized fixture and the retained aggregate evidence.
- Focused routing/telemetry tests plus full Python and repository-required gates.
- Sanitized A/B report identifying route, provider/model provenance, request phase, lines/bytes exposed, retry budget, and terminal outcome.
Expected hotset
src-python/codex_proxy.pyupstream open/relay/retry and terminalization seams.src-python/proxy_telemetry.pyonly if additional sanitized phase fields are required.tests/test_routing.py, transport analyzer fixtures/tests, and one isolated CLI smoke harness.- Avoid protocol translation, provider catalog, and unrelated UI code unless a discussion gate is accepted.
Priority and relationship
P0 in the Third-party model agentic reliability milestone. This blocks reliable official and GLM Worker operation and is independent of #106 Task rollout/control-plane reliability.
Activity
NOirBRight commentedon Jul 13, 2026
WORKER_DISPATCH
NOirBRight commentedon Jul 13, 2026
Plan for #114:
Collision evidence: the branch starts at the assigned base and both worker-side and upstream-only path sets are empty.
[-]Prevent Responses transport faults from causing Codex reconnect storms[/-][+]Find and fix Responses disconnects across official and external routes[/+]NOirBRight commentedon Jul 13, 2026
ORCHESTRATOR_DECISION — ROOT CAUSE FIRST
The initiating disconnect is the P0 outcome. Retry count, reconnect storm suppression, terminal SSE behavior, and telemetry consistency are secondary containment only.
Required ordering:
A containment-only PR must remain partial with Refs #114 and cannot close the Issue.
NOirBRight commentedon Jul 13, 2026
Direction update for #114:
NOirBRight commentedon Jul 13, 2026
Orchestrator decision: choose A — wait for and capture a reproducible faulty window.
The clean-window A/B is valuable negative evidence, but it does not explain the first disconnect. It currently rules out only deterministic necessity: CodexHub itself, a ~347480-byte synthetic request, and auto/explicit routing did not reproduce the failure.
Therefore:
A later containment slice may be considered as an explicitly non-closing
Refs #114PR, but only after root-cause evidence or a separate authorization.NOirBRight commentedon Jul 13, 2026
ORCHESTRATOR_DIRECTION_CORRECTION — FIRST-CLOSING-SIDE EVIDENCE
The pending Direct Official long-stream rate control is cancelled. Reporter evidence already establishes that Direct Official is stable in sustained normal use while inserting CodexHub materially raises disconnects for both official and external models. #114 will not spend further maintainer time re-proving that rate difference.
The next diagnostic gate is one known-fault CodexHub-routed capture that orders the first failure across four correlated boundaries: Desktop downstream socket/consumer, Gateway upstream transport, Gateway relay/terminalization, and upstream service. Evidence must identify
first_closing_side,first_failure_phase, effective per-process proxy mode, new/reused connection state, retry attempt/budget, bytes/SSE lines exposed, upstream terminal observation, downstream terminal forwarding, and monotonic event ordering.Only after the first close is localized may a one-variable probe be chosen: connection reuse, process-local explicit proxy, relay/terminal parsing, or Desktop cancellation behavior. A non-reproducing run stops and leaves the capture armed for the next naturally faulty window; it does not trigger repeated Direct controls. No production transport/retry/SSE change or root-cause claim is authorized before this evidence exists.
NOirBRight commentedon Jul 13, 2026
TEST_INVALIDATED — STALE CATALOG, NO NETWORK RESULT
The armed Gateway Official-auto session was stopped before a workload because its isolated model list lacked all 5.6 official bindings. The #114 branch was still based on pre-PR-#107
dev; this is a test-environment catalog mismatch, not disconnect evidence.Read-only reconciliation shows #114 HEAD
dc692206, currentorigin/dev014d6f60(merged PR #107), merge-base29161909, and zero changed-path overlap. The existing Worker will merge currentdevnon-destructively, verify a fresh connected isolated catalog contains the current 5.6 bindings, and only then arm a replacement Gateway localization session. No request from the invalid session will be collected as root-cause evidence.Merge origin/dev into issue #114 branch
NOirBRight commentedon Jul 13, 2026
CATALOG GATE RECOVERED
The invalid stale-catalog session was cleaned pre-request. Current
origin/dev@014d6f60was merged normally into the #114 branch and pushed asd4329dee; the worktree is clean. Independent fresh connected model-list checks confirmed gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna are present with Max support. Focused 514 tests and full Python 1074 passed / 1 skipped.A new unlaunched Gateway Official-auto first-close session is armed. No production behavior change or root-cause claim has been made.
NOirBRight commentedon Jul 13, 2026
CAPTURE RESULT — NO INITIATING CLOSE REPRODUCED
The isolated Gateway Official-auto capture completed and was cleaned up without touching shared Desktop, CodexHub, proxy, or credential state. Sanitized evidence is preserved at commit
ba124ca3.Ordered evidence for the prolonged request:
request_complete, with no duplicate or silent terminal;The maintainer observed renderer output only after the >120-second cutoff. The premature manual
completedmarker is retained as an operator-label mismatch and is not used as terminal evidence; the real terminal is established only by the ordered Gateway trace.Classification remains
first_closing_side=unknown,first_failure_phase=unknown, reasonno_boundary_failure_observed. This run proves a healthy long-lived Gateway stream and validates the capture path, but it did not reproduce the reported disconnect and therefore supports neither a root-cause claim nor a production transport change.#114 remains open. Do not repeat Direct/External rate controls. Resume the existing first-close capture only for the next natural Desktop disconnect window.
24 remaining items
NOirBRight commentedon Jul 20, 2026
V6 sequencing decision: #114 is natively blocked by #20. After the E2E foundation and serial #18 → #19 → #20 transport chain land, run strict fault injection on one exact Debug SHA across (1) Direct Official control, (2) Gateway Luna, and (3) Gateway Volc. Hold client/build/workload constant; classify the first failing boundary, connection disposition, terminal/completion count, fallback/reconnect behavior, and sanitized timing. Do not infer causality from a later 499 and do not use host shared credentials or sessions.
NOirBRight commentedon Jul 25, 2026
{ "contract": { "acceptance": [ "The deterministic matrix identifies the first failing boundary for pre-header reset/timeout, request-write failure, post-open incomplete read, downstream cancellation, and healthy completion on Official and Ollama Cloud Native Responses paths.", "No main-generation request spends more than 180 seconds before its first upstream response boundary, all delays and attempts share that budget, and Official uses no more than two open attempts.", "A non-Official main-generation replay remains subject to #17: safe pre-write or explicit guaranteed idempotency only; capacity hints, retry headers, or zero downstream bytes cannot make an ambiguous post-write request safe.", "Pre-exposure budget exhaustion emits one sanitized HTTP 504 no-retry failure and one completion. Post-exposure failure emits one protocol-valid terminal through the shared commit seam, with no content replay, fallback, duplicate terminal, or later success.", "The real A/B records Direct Official, Gateway Official Luna, and Gateway Ollama Cloud Native Responses in the same bounded network window, with Desktop primary and CLI negative-control outcomes and no global proxy/TUN mutation.", "Evidence orders effective proxy mode, connection disposition, request phase, attempts/budget, first header/SSE, bytes/events exposed, first closing side, App/app-server identity, terminal forwarding, and exactly-once completion without sensitive content.", "A proven CodexHub defect changes the A/B result after a focused red/green fix. If no disconnect reproduces, deterministic fault coverage and the bounded current-candidate non-reproduction are explicitly retained rather than represented as a universal root-cause proof.", "Protocol translation, Provider selection, catalogs, managed-client materialization, tools, transparent byte identity, cancellation, and the #223/#225 terminal/event contracts remain regression-green.", "No intermediate full 12-case E2E is run; #227 consumes the qualified normal-path evidence on the final frozen 0.1.7 candidate." ], "change_claims": { "paths": [ "src-python/codex_proxy.py", "src-python/proxy_telemetry.py", "tests/test_routing.py", "tests/test_proxy_event_logging.py", "tests/test_proxy_shutdown.py" ], "resources": [ "gateway-pre-response-total-budget", "official-responses-transport", "ollama-cloud-native-responses-transport", "debug-first-close-recorder", "isolated-codex-desktop-profile", "isolated-codex-cli-control", "process-local-proxy-observation" ] }, "dependencies": { "dispatch_after": [ 17, 226 ], "merge_after": [ 17, 226 ] }, "design": [ "Run only after #17 and #226, which transitively include the downstream-write, reader-lifecycle, event-boundary, and selected Ollama Cloud path prerequisites. Reconcile the old external Chat evidence out of the Responses conclusion; the third-party Responses leg is Ollama Cloud Native Responses.", "Use the existing Debug recorder and one exact candidate to run deterministic localhost faults for Official Responses and Ollama Cloud Native Responses: reset and timeout before headers, reset during request write, incomplete read after exposed SSE, downstream cancellation, and healthy long stream. The same fault must produce the same first-boundary and terminal accounting independent of transport chunking.", "Enforce one total pre-response wall-clock budget of 180 seconds for a main-generation request, including all open attempts and retry delays, with at most two Official open attempts. Clamp each wait to the remaining shared budget; #17's retry-safety decision continues to govern whether a non-Official attempt is replayable.", "Before downstream exposure, budget exhaustion returns one sanitized protocol-correct gateway_pre_response_budget_exhausted failure with HTTP 504 and an explicit no-retry signal. After headers or SSE exposure, never replay model generation; use the #223/#225 request-scoped terminal seam for one protocol-valid failure and one completion.", "On the same exact Debug candidate, run a bounded isolated causal comparison: Direct Official control, Gateway Official Luna, and Gateway Ollama Cloud Native Responses, with Codex Desktop as the primary surface and the bundled Codex CLI as a contemporaneous negative control. Hold build, profile, model-class workload, machine, and network window constant.", "Record effective process-local proxy mode, new/reused/unobserved connection disposition, attempt and shared budget, first header/SSE timing, bytes and events exposed, first closing side, terminal forwarding, completion count, and App/app-server identity. Never change the global proxy or TUN and never retain prompts, responses, credentials, Task IDs, or host paths.", "If the comparison proves a CodexHub-owned transport defect within the declared hotset, observe a focused red fixture, apply only that fix, and repeat the affected local gates and bounded comparison before freeze. A new UI, Provider policy, transport replacement, or system-proxy setting requires a separately approved child contract.", "A non-reproducing current-candidate comparison can clear the 0.1.7 gate only when all deterministic faults pass, both Gateway routes complete the bounded workload without opaque reconnect, and the exact limits of the non-reproduction are recorded. Do not duplicate #227's full 12-case matrix." ], "difficulty": "heavy", "done_when": [ "Focused transport-fault and total-budget regressions are observed failing before any production fix and passing afterward.", "python -m pytest -q tests/test_routing.py tests/test_proxy_event_logging.py tests/test_proxy_shutdown.py", "python -m pytest -q --ignore=tests/test_real_client_e2e.py --junitxml=.pytest-results/junit-core.xml --durations=0", "git diff --check", "python scripts/report_quality_gates.py (report-only)", "One bounded same-candidate Direct Official, Gateway Official Luna, and Gateway Ollama Cloud comparison completes with Desktop primary and CLI negative-control evidence.", "The durable disposition identifies a proven/fixed CodexHub boundary or a bounded current-candidate non-reproduction with exact limits; it does not claim the older external Chat evidence as Native Responses.", "If repository files change, exact-local-SHA Standards and Spec reviews report no blocker before one push/PR update and one exact-SHA GitHub Actions run.", "No intermediate 12-case real-client qualification is run; #141 may begin only after this disposition is complete." ], "priority": "P1", "risk": "strict", "sha256": "350d6dd4c7964fba7da8e7d9ea6d8bd59f4fd7685b7796db01f85e7ffc61e8c9", "unresolved_decisions": [] }, "dispatch": null }NOirBRight commentedon Jul 26, 2026
Pre-dispatch current-dev gap audit
This is read-only preparation on
dev@1758b830130ac3bf4a0d55f08b69fce342f4a969. It does not dispatch #114, change production code, run a live A/B, or claim the post-#17/#226 tree is already qualified.Confirmed current gaps against the V2 contract:
DEFAULT_OFFICIAL_UPSTREAM_OPEN_ATTEMPTSis3, while this Issue requires no more than two Official open attempts.official_upstream_open_attempts()accepts any positive environment override and therefore does not enforce the required upper bound._open_upstream_response()passes the configured per-attempt timeout unchanged to every open. The current default upstream timeout is 300 seconds.gateway_pre_response_budget_exhaustedclassification or explicit pre-exposure HTTP 504/no-retry terminal currently exists.Confirmed reusable foundations already present:
Therefore the later #114 implementation should remain narrow after #17/#226 merge:
The current validated contract hash, including the path-gated Python-core command, is
350d6dd4c7964fba7da8e7d9ea6d8bd59f4fd7685b7796db01f85e7ffc61e8c9.NOirBRight commentedon Jul 27, 2026
Deterministic containment candidate frozen
Partial PR #242 is frozen at
b6ab722a3fda6c52973409fd6272e8aadf26d1ea(treecf79fc8e47ae9b47b9d484a46641e48c8254b049). It implements only the contract's deterministic pre-response containment:Evidence:
The required same-candidate Desktop/CLI A/B was not run: the dedicated isolated profile contains zero existing Tasks, and the companion #141 contract forbids creating one during diagnosis. This comment and PR make no initiating-root-cause claim. #114 remains open and continues to block #227; PR #242 is
Refs #114, not a closing PR. Exact-SHA Actions run 30291838819 is in progress.NOirBRight commentedon Jul 27, 2026
Partial containment merged; root-cause gate remains open
PR #242 passed exact-head Actions run 30291838819 with all eight required jobs successful and was squash-merged to
devasfad95c869bf3726461d9f9a764020863a5ba0317.The frozen PR tree and remote
devtree are bothcf79fc8e47ae9b47b9d484a46641e48c8254b049.This merge completes only the deterministic 180-second/open-attempt containment. The required isolated live A/B and initiating-boundary disposition are still absent, so #114 remains open and continues to block #227.
NOirBRight commentedon Jul 28, 2026
0.1.7 takeover — bounded current-candidate A/B
Exact candidate:
fad95c869bf3726461d9f9a764020863a5ba0317(treecf79fc8e47ae9b47b9d484a46641e48c8254b049).One dedicated profile, one persistent Task, Codex Desktop build
26.721.4979.0, and app-bundled CLI0.146.0-alpha.3.1were used. No shared Desktop state, global proxy/TUN mutation, direct database/session editing, or Task creation occurred during evidence.Transport result:
--ephemeralcontrols: the same three legs all completed.request_complete=200, with zero retry, request-error, or opaque reconnect marker.A separate whole-GUI shell differential then cold-started the same dedicated GUI profile and supported Task deep link once connected and once Direct. Both legs kept exactly one GUI root and one app-server child alive for the bounded observation window, produced no WER/Application Error record, and reproduced no whole-root restart (
0/1connected,0/1Direct). The same Task remained natively recoverable afterward.Fail-closed limit: the actual GUI composer did not submit the workload. The transport workload was driven through the Desktop-identity app-server API, while the whole-GUI run covered only cold-start/navigation shell and recovery. Therefore this is a bounded transport non-reproduction, not the contract's complete Desktop-primary causal comparison and not a universal root-cause claim.
Sanitized local evidence schemas:
codexhub.issue-114-141-bounded-ab.v1andcodexhub.issue-141-gui-shell-ab.v1. They retain no Task IDs, prompt/response content, credentials, or host paths.#114 remains open and continues to block the release candidate.
NOirBRight commentedon Jul 28, 2026
Final 0.1.7 disposition — bounded current-candidate non-reproduction
This supersedes the prior GUI-composer limitation. The exact candidate remains
fad95c869bf3726461d9f9a764020863a5ba0317(treecf79fc8e47ae9b47b9d484a46641e48c8254b049), Desktop build26.721.4979.0, bundled CLI0.146.0-alpha.3.1.The isolated packaged Desktop GUI now submitted the same bounded workload on the one persistent Task for all required primary legs:
request_complete=200, zero retry and zero request-error; terminal SSE observed.request_complete=200, zero retry and zero request-error; the native Task turn completed.The supported Task count stayed exactly one and its completed turn count advanced
4 → 6 → 7. Connected and Direct each retained one GUI root and one app-server child, reproduced no whole-root restart (0/1each), and produced no matching WER/Application Error event. The earlier bundled-CLI controls for the same three route classes also completed.One separate Desktop background-resume sequence occurred during connected startup: an initial Official request completed
200, then a follow-up was cancelled by the client before any downstream output and recorded one499atfailure_side=downstream_write. The GUI root remained alive, no retry/request-error/opaque reconnect followed, and both foreground composer workloads then completed normally. Ordered evidence therefore classifies this as a client-side background cancellation, not an earlier Gateway invalid, duplicate, or unterminated sequence and not a GUI-root crash.PR #242 already supplied the deterministic RED/green fault coverage and merged containment: one 180-second request budget, at most two Official opens, #17 retry-safety preservation, one pre-exposure non-retryable 504, and shared post-exposure terminalization. Exact local and Actions gates are retained in the earlier comments.
Disposition: the V2 contract's non-reproduction path is satisfied—deterministic faults pass and both Gateway routes complete the bounded actual-Desktop workload without opaque reconnect. This is deliberately a current-candidate non-reproduction with stated limits, not a universal explanation of historic network failures. No additional Gateway retry/transport change is justified.
Sanitized evidence schema:
codexhub.issue-114-141-gui-composer-ab.v1; it retains no Task/window/request IDs, retry identity, prompt/response content, credentials, or host paths.