perf(chat): reduce stream allocations and bound native Chat completion

[lidge-jun]

Summary

Carry of #4389 by @olddonkey onto current dev. Stacked on #4473; this PR targets that branch, so its own diff is just this change.

Repeated byte measurements allocated full UTF-8 copies of requests and growing stream buffers, which amplified allocation pressure during concurrent turns. Native Chat could also wait indefinitely after response headers, return partial success after cancellation, or collect CRLF responses as empty output.

• Count request and response sizes without measurement arrays; accumulate compaction and Chat text sizes incrementally while preserving Unicode coercion, admission limits, and ownership.
• Scan SSE input incrementally and encode only delivered output after admission. Release reservations on failed enqueue and stop rewrites after cancellation or disposal.
• Apply the configured native Chat stall timeout to meaningful progress, pausing it under downstream backpressure. Return typed stall errors and cancellation status instead of partial success; preserve accepted terminals and support LF, CRLF and multiline data.
• Update architecture notes and server configuration docs in English and seven translations, and add focused regressions for allocation volume, Unicode, size boundaries, cancellation, terminal precedence and cleanup.

For 1,024 fragments of 1 KiB, compaction measurement arrays fall from a cumulative 1,025 MiB to zero, and native SSE and rewrite encoding volume now equals delivered output. Those are synthetic allocation measurements, not measured production RSS savings, and no live provider load test was run here.

Conflicts resolved

The source branch was 266 commits behind dev, so three resolutions were judged against current behavior rather than the branch's original base.

• src/chat/outbound.ts — collectChatCompletion. dev added service-tier capture to the same loop this change rewrites from replaceRetained to appendRetained with a retainedBytes ledger. The rewrite is taken wholesale and dev's three service-tier deltas are re-applied onto it: the serviceTier declaration, the parsed.service_tier capture beside parsed.usage, and the service_tier spread on the returned completion. Verified by diffing the result against both sides — the only delta against the source branch is dev's own. dev's refusal-ledger change from fix(chat): stop refusal bookkeeping from failing refusal-free turns #4468 is in the streaming function and merged without conflict.
• src/adapters/openai-chat.ts. Positional only: dev inserted the parallel_tool_calls block directly above, so git anchored the hunk onto a duplicate of the debug block. dev's code is kept and the one real change, new TextEncoder().encode(bodyJson).length to Buffer.byteLength(bodyJson, "utf8"), is applied to the live site. The other two byte-counting sites in the file merged cleanly.
• Twenty structure and docs-site documents. Two shapes, both mechanical: dev added an intro paragraph at the same anchor (both sides kept, dev first), or dev added a table row above a row this change rewrites (row kept, rewrite applied).

The 600-line structure budget

structure/transports/responses.md is exactly at its 600-line budget on dev, so this change could not add its two new owning sections there at all — structure:check failed at 630 lines. structure/AGENTS.md says an over-budget doc is split along a topic boundary with its own manifest entry, and that a grace.oversizeDocs entry is only for a split already planned, so this takes the split. Request-copy and stream-buffer accounting are one topic and now live in structure/transports/byte-accounting.md; responses.md is byte-identical to dev again. Both documents sit in structure/transports/, so the seventeen cross-references keep their relative prefix and change only the file name.

structure/gui-and-management-api.md hit 602 lines for the same reason, and its cross-reference is dropped instead: the dashboard and management API own neither the request-decompression nor the SSE-rewrite path, which makes it the least load-bearing of the seventeen.

Maintainers should know this affects other lanes: responses.md having zero headroom on dev means any branch adding a line to it fails structure:check.

Attribution, per AGENTS.md and CREDITS.md — the landing commit must carry this trailer, and the squash message must not drop it:

Co-authored-by: Olddonkey <22208754+olddonkey@users.noreply.github.com>

This is the middle link of lane C in devlog/_plan/260913_contributor_carry_train/, stacked on #4473 with #4457 above it. Retarget to dev once #4473 lands.

Verification

Source head 176cbbd2ec1f492c74174eecc54435af2d7cb8db carried onto #4473's head 6645ccb06.

• bun test on the eleven files this change and its parent touch — 536 pass, 0 fail: tests/adapters/bridge-nonstreaming-terminal, tests/adapters/translator-budget, tests/lib/debug, tests/responses/chat-completions-endpoint, tests/responses/openai-responses-passthrough, tests/responses/sse-payload-rewrite, tests/usage/request-decompress, tests/responses/chat-conversation-affinity, tests/lab/core-lab-boundary, tests/ci-workflows/structure-ssot, tests/adapters/openai/openai-chat-system-order.
• bun run typecheck — clean.
• bun run structure:check — passes after the split above. bun run structure:index regenerated INDEX.md.
• bun run privacy:scan — passes.
• The local full suite was not run; per the lane policy hosted CI on the lane tip is the suite proof. This branch carries [skip ci] on its head commit, so its own ci check does not run and the lane tip run covers it.

Because this is a performance change, the question worth reviewing is whether any allocation removal changes observable behavior on a truncated or folded stream. It does not. The behavior that does change is the lifecycle and correctness policy this PR advertises, and the two are worth separating.

The allocation removals are behavior-preserving. The shared block scanner delimits on the same four blank-line shapes as dev's nextSseBlock regex (\n\n, \r\n\r\n, \r\n\n, \n\r\n), and CR-only still never delimits on either side; byte-split events, several events per chunk, Unicode and surrogate pairs split across append all produce identical blocks and leftovers. The incremental byte subtraction is exact, because SSE delimiters are CR/LF only, so .length equals their UTF-8 size; remaining bytes matched Buffer.byteLength(tail) after every consume. Admission overlap is unchanged, so a buffer already at maxTurnBytes still cannot consume — fail-closed as before. An unterminated final event is still not forwarded: the HTTP rewrite flushes the raw tail exactly as dev did, and native Chat still fails it closed as upstream_sse_unterminated. Incremental compaction counting matches dev's full recount, including a surrogate pair split across two deltas; a 14,641-sequence sweep over empty, ASCII, CJK, emoji, lone high/low surrogates and NFD input found zero divergences. snapshot || doneText || deltas precedence and terminal ownership are untouched.

Buffer.byteLength and TextEncoder differ only on non-string input, where the former throws and the latter coerces. Every changed call site is already guarded by JSON.stringify, typeof === "string", or TextDecoder.decode, so that hazard is not reachable.

Three real behavior changes, all intentional. Each follows from a bullet above rather than from removing an allocation:

1. After a client abort, already-buffered complete events are no longer emitted. dev read the next block before checking the abort, so a second event in the same chunk could still reach a client that had gone away.
2. data: with an empty value is forwarded instead of failing. dev ran JSON.parse("") on it and killed the stream with upstream_sse_invalid.
3. A cancelled non-streaming native request returns 499 instead of a 200 assembled from whatever frames had arrived — the "partial success after cancellation" this PR set out to fix.

One leniency is given up. data: lines within a single event are now joined per the EventSource contract, which is what makes CRLF and a JSON payload split across data: lines work. A colonless data line in the same event as a JSON line used to be ignored and the JSON parsed; joined, it no longer parses. This is spec-conformant and uncommon on Chat wires, and it is deliberate.

Scope check on the collector. collectChatCompletion changed framing too, but it is not a user-visible surface: both production call sites feed it opencodex's own generated SSE — nativeChatSse output in chat-native.ts and responsesSseToChatCompletionsSse output in chat-completions.ts — never third-party SSE. Verified against a clean origin/dev checkout: a malformed upstream block returns the identical 502 upstream returned malformed SSE JSON on both, because the relay sits in front of the collector and dev's relay already joined data: lines.

Audit method: three independent read-only reviewers, one per area, each comparing this branch against origin/dev and refs/carry/4389 with executable probes rather than reading alone. Two further collectChatCompletion test files outside the source PR's focused set were found and run: tests/responses/sse-unspaced-data-fields.test.ts and tests/responses/chat-json-sse-fallback.test.ts, plus tests/responses/compaction-progress.test.ts — 58 pass, 0 fail.

Checklist

• Scope stays focused and avoids unrelated cleanup. The structure split is not unrelated cleanup: it is the only way this change can document the areas it modifies.
• Docs or release notes were updated when needed.
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. No authentication, credential, workflow, dependency or release surface is touched, and privacy:scan passes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

• ^dev$
• ^preview$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: fc23073a-ec52-48ca-9b06-82355ee7b1b8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-13T07:15:43.379885Z	7e23433	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

✅ Deterministic PR hygiene checks passed.

[lidge-jun]

리뷰 · 우선순위 69 / 80

설명
이 PR은 기여자 @olddonkey의 #4389를 현재 dev 쪽으로 옮긴 캐리입니다. 베이스가 dev가 아니라 #4473(codex/260913-carry-ocg-deepseek-system-order)이고, 본문에도 lane C의 가운데 고리라고 적혀 있습니다. 그래서 지금 tip(f7d9dbad0, #4484 Devin 통합이 막 올라온 상태)에 바로 합칠 수는 없고, #4473이 먼저 들어간 뒤에 dev로 리타겟해야 합니다.

하는 일은 크게 두 갈래입니다. 첫째, 요청·응답·스트림 버퍼 크기를 잴 때마다 UTF-8 전체 복사본을 새로 만들지 않게 바꿉니다. Buffer.byteLength(..., "utf8")로 바꾸고, compaction·Responses 텍스트는 조각마다 누적 바이트를 더하는 방식으로 바꿉니다. 둘째, 네이티브 Chat이 헤더만 받고 끝없이 기다리거나, 취소 뒤에 부분 성공을 돌려주거나, CRLF/여러 줄 data를 빈 결과로 모으던 문제를 고칩니다. stallTimeoutSec가 Responses뿐 아니라 네이티브 Chat의 “의미 있는 진행”에도 적용되고, 느린 클라이언트 읽기 동안에는 타이머를 멈추며, 스톨은 upstream_stall_timeout(스트리밍 오류 / 비스트리밍 502)으로 끝냅니다.

현재 tip과의 관계는 이렇습니다. tip에는 이미 #4468 거절(refusal) 장부 수정과 #4471 단위 마감 문서가 들어 있고, 방금 #4484로 devin-cli가 devin으로 합쳐졌습니다. 이 PR은 그 위에 올라가는 성능·수명주기 스택이라서, tip 방향(오디오·쿼터·Devin·서브에이전트)과 충돌하지 않고 오히려 동시 턴 부담을 줄이는 쪽입니다. 본문이 말한 것처럼 collectChatCompletion은 replaceRetained → appendRetained + retainedBytes로 바뀌면서 tip의 service_tier 캡처를 다시 얹었고, #4468 거절 장부는 스트리밍 쪽에 있어서 충돌 없이 합쳐졌다고 합니다.

문서 쪽도 중요합니다. tip의 structure/transports/responses.md가 600줄 한도에 딱 맞아서, 새 소유 문단을 거기 넣으면 structure:check가 깨집니다. 그래서 요청·스트림 바이트 계산을 structure/transports/byte-accounting.md로 갈라내고, gui-and-management-api.md 교차 참조는 부담이 가장 적은 쪽으로 빼 버렸습니다. 이건 정리 작업이 아니라 한도 때문에 필요한 분리입니다. 다만 tip에서 responses.md에 한 줄만 더해도 다른 레인 전부가 깨질 수 있다는 경고는 메인테이너가 기억해 둘 만합니다.

검증은 이 변경과 부모가 만지는 열한 파일 위주 bun test 536통과, typecheck·structure:check·privacy:scan 통과라고 적혀 있습니다. 헤드에 [skip ci]가 있어서 이 PR 자체 CI는 돌지 않고, 레인 tip CI가 전수 증명 역할을 합니다. 합성 할당 측정(1,024×1KiB에서 compaction 측정 배열이 누적 1,025MiB → 0)은 광고된 대로 프로덕션 RSS 절감이 아니고, 라이브 프로바이더 부하 테스트도 없다고 스스로 밝혀 두었습니다.

경로/심볼 - 베이스가 아직 열린 #4473이라 tip에 단독 머지하면 스택이 꼬입니다. #4473 머지 후 dev 리타겟이 필수입니다.
경로/심볼 - 헤드 [skip ci]라서 이 PR 단독으로는 전수 CI 증명이 없습니다. 레인 tip 체크가 초록인지 확인한 뒤 합쳐야 합니다.
structure/transports/responses.md - tip에서 줄 수 여유가 0입니다. 이 PR의 byte-accounting 분리는 맞고, 이후 다른 레인이 같은 파일에 한 줄만 추가해도 structure:check가 실패합니다.
src/chat/outbound.ts / collectChatCompletion - tip의 service_tier와 #4468 refusal ledger를 다시 얹었다고 본문에 적혀 있으니, 리타겟·리베이스 때 그 세 델타가 빠지지 않았는지 diff로 한 번 더 보면 좋습니다.
테스트 범위 - 프레임 분할·취소 우선순위·예약 해제 회귀는 있으나, 프로덕션 RSS/동시 턴 부하는 측정되지 않았습니다. 머지 판단은 동작 보존 + 할당 회귀 테스트로 충분하고, 벤치 숫자는 기대하지 않는 편이 맞습니다.

메인테이너의 판단이 필요한 지점

• #4473을 먼저 머지할지, 아니면 이 PR을 잠시 대기시킬지(스택 순서 유지 vs tip 병목).
• squash 메시지에 Co-authored-by: Olddonkey <22208754+olddonkey@users.noreply.github.com> 트레일러를 꼭 남길지(AGENTS/CREDITS 규칙).
• responses.md 600줄 한도를 앞으로 어떻게 운영할지(강제 분리 vs 한도 상향 vs grace).
• 레인 tip CI만으로 전수 증명을 받아들일지, 리타겟 후 이 PR에서 [skip ci]를 빼고 한 번 더 돌릴지.

너의 추천
#4473이 dev에 들어간 뒤 이 PR을 dev로 리타겟하고, 레인 tip(또는 리타겟 후) CI가 초록이면 머지하세요. squash 때 Olddonkey co-author 트레일러를 유지하고, byte-accounting.md 분리는 그대로 가져가세요. tip에 단독으로 먼저 넣지 마세요.

이 댓글은 grok-bot이 작성했습니다

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e2343326a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve TextEncoder sizing on older Bun runtimes

When the proxy runs under Bun 1.2.14, Buffer.byteLength does not match TextEncoder for unpaired UTF-16 surrogates: for example, "\ud800x\udc00" measures as 5 bytes here instead of the 7 bytes actually emitted as UTF-8, causing the newly added retained UTF-8 sizing test to fail. User-controlled JSON can produce these strings through escaped surrogates, so retainedUtf8Bytes can undercount admission limits and truncateRetainedUtf8 can return values exceeding its byte cap; use an allocation-free code-unit counter or a compatible fallback instead of delegating this contract to Bun's Buffer.byteLength.

AGENTS.md reference: src/AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

[lidge-jun]

Landed on dev as part of lane C. Closing manually: this branch's head is a verified ancestor of origin/dev after tip #4487 merged as 55bb9f3, but GitHub did not close it automatically because the chain was retargeted at merge time.

Tip-only CI record, owner-authorized for this batch: Cross-platform CI run 34744712476 concluded success on 9b30902, the exact merged head, and the lane is cumulative so that run executed this branch's content as a strict subset. This pull request's own ci check never ran; its head commit carries [skip ci] by design.