Tolerate a Chutes stream chunk that omits id

[lloydmak99]

Split out of #952 so the status fix and this parser fix can be reviewed separately.

The bug

Reproduced against production. Streaming zai-org/GLM-5.1-FP8, varying only the input context size:

input ctx	x-serving-provider	result
32K / 64K	near	OK — TTFT 19.6s / 48.7s
96K	—	HTTP 400 — clean rejection
128K / 200K	absent	HTTP 200, 2 events, in-band error, then [DONE]

data: {"error":{"message":"Failed to perform completion: Chutes stream chunk parse:
       missing field `id` at line 1 column 184","type":"server_error"}}
data: [DONE]

Above a context threshold the request falls back to Chutes, whose decrypted frames omit id. ChatCompletionChunk requires it, so one missing scalar kills an otherwise healthy stream at chutes/e2ee_stream.rs:88.

That the 96K path already returns a clean 400 is what makes this a defect rather than a limit — the system can reject properly; this path instead falls back to a provider it cannot parse.

Why the fix is contained to the Chutes boundary

ChatCompletionChunk is shared by every provider. Making id optional there would weaken validation fleet-wide and let a genuinely malformed chunk from any backend parse silently. So the frame is deserialized to a Value, a missing id is filled, and the typed chunk is built from that. models.rs is untouched.

Unknown fields already survive via the flatten map — covered by a new test, since DeepSeek emits an extra prompt_text field at 400K context. Provider chunk shapes genuinely diverge: DeepSeek at 300K returned "id":"c678a193…", at 400K "id":"chatcmpl-6ce9a499…" plus "prompt_text":null.

raw_bytes are asserted byte-identical to the input — those bytes are signature-relevant and must pass through verbatim.

The part worth a careful look

extract_inference_id_from_chunk previously hashed whatever it was given, so an empty id produced the same inference id for every affected stream — worse than having none, because it looks valid and would corrupt attestation lookups. It now returns Option<Uuid>, and absent stays absent, matching how Inference-Id is already omitted when a stream fails before its first chunk.

This inverts an existing unit test — test_extract_inference_id_from_chunk_empty_id asserted that an empty id should still yield a non-nil UUID. That assertion was encoding the bug. Flagging it rather than letting it pass as a routine test edit.

Tests added

• a frame with no id parses, yields a usable chunk, and raw_bytes are byte-identical
• a frame with an unknown extra field still parses and preserves it (guards the DeepSeek shape)
• a genuinely malformed frame still errors — the parser must not become one that accepts anything
• absent/empty id yields no inference id rather than a hash of the empty string

Verification

• cargo clippy -p inference_providers -p services -p api --all-targets -- -D warnings — clean
• cargo test -p inference_providers --lib chutes — 112 passed, 0 failed
• cargo test -p api --lib extract_inference_id — 4 passed, 0 failed

E2E tests were not executed locally — they require PostgreSQL and a dstack/TEE socket; both tests panic at setup on Failed to derive signing keys from dstack. CI is the judge there.

Interaction with #952

Independent changes, but both touch extract_inference_id_from_chunk and its call sites. Whichever merges second needs a trivial merge in that function. #952 makes a pre-first-event error return a real HTTP status; this PR makes the stream not fail in the first place.

Suggested acceptance test post-merge

Against staging, varying only context size on zai-org/GLM-5.1-FP8: 128K/200K should stream normally instead of returning 200 + in-band error; 96K should still return 400; healthy control unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fm9f3maaY2wd6yHH8FEjR5

[claude]

Review — Tolerate a Chutes stream chunk that omits id

The diagnosis and the containment argument are right: keeping id required on the shared ChatCompletionChunk and normalizing only at the Chutes boundary is the correct call, and the deliberate flag on the inverted test_extract_inference_id_from_chunk_empty_id assertion is appreciated. stream_chat_id = Some(...).filter(|id| !id.is_empty()) is also a correct companion fix — it stops a get_provider_tier_for_chat_id("") lookup from matching the "" key the pool writes.

One blocker below: the empty-id hazard the PR itself identifies is fixed on the header path but not on the billing path, and this PR is what makes that path reachable.

⚠️ Critical — empty id collapses all affected streams onto one inference_id, silently dropping usage

extract_inference_id_from_chunk now returns None for an empty id, but the usage path computes its own inference id from the same string and was not changed:

• crates/services/src/completions/mod.rs:517 — self.last_chat_id = Some(chat_chunk.id.clone()) stores Some("") for every Chutes chunk that omitted id.
• crates/services/src/completions/mod.rs:279 — hash_inference_id_to_uuid(&chat_id) hashes "" into a constant UUID (Uuid::new_v5(NAMESPACE_DNS, b"")).
• crates/database/src/repositories/organization_usage.rs:107 — ON CONFLICT (organization_id, inference_id) WHERE inference_id IS NOT NULL DO NOTHING, backed by the unique index from V0045.

Failure scenario: org X issues two 128K-context streams against zai-org/GLM-5.1-FP8 that fall back to Chutes. The first records usage under H(""). The second hits the conflict, and at organization_usage.rs:180-183 the transaction is rolled back — no organization_usage_log row, no organization_balance update. Every subsequent such stream for that org is dropped too, indefinitely: unbilled tokens and spend limits not enforced, with only a debug! line.

Before this PR those streams died at the parse boundary and never reached usage recording. The fix converts a loud stream failure into silent revenue loss, which is strictly worse — and it is the exact hazard the PR description calls out ("worse than having none, because it looks valid").

Note the naive fix does not work: gating last_chat_id on non-empty routes into the (Some(usage), None) arm at mod.rs:251, which logs and returns — also no billing. The id needs to become optional independently of whether usage is recorded, e.g.:

// mod.rs:517 — absence stays absent, matching the route fix
if !chat_chunk.id.is_empty() {
    self.last_chat_id = Some(chat_chunk.id.clone());
}

plus letting record_usage_and_metrics proceed on (Some(usage), None) with inference_id: None / provider_request_id: None. RecordUsageServiceRequest.inference_id is already Option<Uuid> and the partial unique index excludes NULLs, so the row inserts and the balance updates while simply forgoing the idempotency key — the same "absent stays absent" principle this PR applies to the Inference-Id header.

(Related but benign: inference_provider_pool/mod.rs:3316 still calls store_chat_id_mapping(""). Harmless today — the route no longer looks up "", Chutes' supports_chat_signatures() is false at chutes/mod.rs:1711 so the signature fetch short-circuits, and pin_chat_connection is the default no-op for Chutes. Worth skipping the write anyway rather than relying on all three staying true.)

Minor — "id": null still kills the stream

object.entry("id").or_insert_with(...) at e2ee_stream.rs:88-92 fires only when the key is absent. An explicit "id": null still fails from_value with invalid type: null, expected a string — the same fatal path, unfixed. Given the PR documents DeepSeek emitting "prompt_text": null, an explicit-null id is plausible from the same family of shapes:

if matches!(object.get("id"), None | Some(serde_json::Value::Null)) {
    object.insert("id".to_string(), serde_json::Value::String(String::new()));
}

Minor — the raw_bytes byte-identity test is self-fulfilling

inner_event builds raw_bytes as format!("data: {content}\n\n"), so inner_event_accepts_missing_id_and_preserves_raw_bytes passes because the test frame is already in exactly that canonical form. A bare-JSON frame, or one with different data: spacing, would not round-trip byte-identically. The behavior is pre-existing and unchanged here, so not a blocker — but the description's "raw_bytes are asserted byte-identical to the input ... those bytes are signature-relevant" claims more than the test establishes.

Good

• Confining the normalization to the Chutes boundary rather than weakening models.rs — correct, and the inner_event_rejects_missing_choices test is the right guard against the parser becoming permissive.
• The unknown-field test pins the flatten-map behavior against the real DeepSeek shape.
• Explicitly flagging the inverted test assertion instead of letting it slip through as a routine edit.

⚠️ Issues found — the billing-path fix should land with this change, since this PR is what makes that path reachable.

[ironloopai]

Review · Status

🟩 Completed

IronLoop completed the review and posted it to GitHub.

Result

Open submitted review →

Run details

• Run: 0050226e-1a1d-4965-a06b-f4a281690d85
• Base: main at c8b50f2
• Head: fix/chutes-missing-id at f9a419f
• Created: 2026-08-27 06:29 UTC
• Updated: 2026-08-27 06:55 UTC

_{Automatic trigger · attempt 1 of 3 · completed in 25m 59s}

[github-actions]

✅ OpenCodeReview: Review failed: 0 finding(s); 2 of 2 selected item(s) failed.

[ironloopai]

Review · Summary

Two issues remain in the new id-less Chutes stream path.

Findings: 🔴 High 1 · 🟠 Medium 1

_{Code-specific findings are attached to the diff.}

Validation

• ✅ Chutes E2EE stream unit tests — 12 focused parser and stream tests passed.
• ✅ Inference-ID extraction unit tests — 4 focused API tests passed.

Review details

• Run: 0050226e-1a1d-4965-a06b-f4a281690d85
• Attempts: 1

[ironloopai]

🔴 High · Keep id-less chunks synchronized after sanitization

The placeholder is inserted only in the local value used to build chunk; raw_bytes remains id-less. The downstream Chutes rewrite sanitizes that raw JSON but cannot deserialize it back into ChatCompletionChunk, so it retains the original chunk. The normal attested streaming path reserializes that stale chunk for usage shaping, which can re-expose fields the Chutes allowlist removed (such as prompt_token_ids and prompt_sha256) and lose the canonical model rewrite. Normalize the value used by downstream typed rebuilds too, or replace the parsed chunk after sanitization.

[ironloopai]

🟠 Medium · Do not use an empty provider id as a signature key

For a default attestation-supported stream, usage shaping enables gateway signing. If the first accepted Chutes chunk has no id, its parsed id is "", and the stream tail stores gateway signatures under that key. The signature table upserts on (chat_id, signing_algo), so each affected completion overwrites the prior completion's signatures and cannot retain a per-completion signature. Skip storage when the provider id is absent, or mint and return a unique gateway identifier.

[PierreLeGuen]

Containing the tolerance to the Chutes boundary is the right call, but raw_bytes still omit id, so downstream re-derives (rewrite_sse_event_model, think extraction) silently no-op and unsanitized Chutes internals reach clients; the empty id also flows on as a real chat id, collapsing the usage idempotency key to…

Blocking findings:

• crates/inference_providers/src/attested/chutes/e2ee_stream.rs:92 — The missing id is filled only in the local chunk_json used to build the typed chunk; the emitted raw_bytes (data. Impact: For exactly the id-less streams this PR unblocks, rewrite_sse_event_modelnever replacesev.chunk`, so the parsed… Fix: Do not leave the frame id-less downstream.

Checks: cargo +1.92.0 test -p inference_providers --lib chutes — 112 passed, 0 failed, 1 ignored (matches the PR's stated result); cargo +1.92.0 test -p api --lib extract_inference_id — 4 passed, 0 failed, including the deliberately inverted test_extract_inference_id_from_chunk_empty_id; cargo +1.92.0 clippy -p inference_providers -p api --all-targets -- -D warnings — clean

[lloydmak99]

Both findings addressed in 1926b78.

🔴 High — raw and typed chunks now derive from the same value

You were right that the divergence was the bug. The placeholder was inserted only into the value used to build the typed chunk, so raw_bytes stayed id-less, rewrite_sse_event_model's typed rebuild failed, and the original chunk survived sanitization — re-exposing prompt_token_ids / prompt_sha256 and losing the canonical model rewrite on exactly the streams this PR unblocks.

Both representations are now built from the value carrying the id, so every downstream round-trip through raw_bytes succeeds. Regression test asserts the sanitized result drops the stripped fields and carries the canonical model — it fails without the fix.

🟠 Medium — minted a stable per-stream id instead of an empty string

Took the "mint a unique gateway identifier" option, since it fixes both findings at once. chutes-gateway-<uuid>, derived per response session:

• stable across every chunk of a stream, so chat_id grouping, sticky routing and signature storage all behave
• unique per completion, so (chat_id, signing_algo) rows cannot collide
• visibly synthetic, so it can't be mistaken for a provider id in logs or support cases

One thing my own verification caught before this landed

The first version re-serialized every frame. serde_json here has no preserve_order, so Value::Object is a BTreeMap and a round-trip sorts keys alphabetically — that would have changed the wire bytes of every Chutes stream in production, not just the id-less ones.

Re-serialization is now conditional: frames that already carry an id are emitted byte-for-byte as before. Guarded by a byte-identity test written to fail against the unconditional version.

Verification

• cargo clippy -p inference_providers --all-targets -- -D warnings — clean
• cargo test -p inference_providers --lib chutes — 116 passed, 0 failed
• sanitization regression test confirmed failing before the fix

E2E not run locally — needs PostgreSQL and a dstack/TEE socket; both tests panic at setup on Failed to derive signing keys from dstack.

[PierreLeGuen]

Normalizing a missing or empty id at the Chutes E2EE boundary is correct and coherent. One non-blocking gap: a JSON-null id is not normalized.

Optional follow-ups:

• crates/inference_providers/src/attested/chutes/e2ee_stream.rs:94 — The new normalization treats a JSON-null id as present. Fix: Fold null into the replaced set so absent, empty, and null normalize identically.

Checks: cargo +1.92.0 clippy -p inference_providers -p services -p api --all-targets -- -D warnings — clean, no warnings (a parallel attempt in a second environment aborted on a read-only…; cargo +1.92.0 test -p inference_providers --lib — 424 passed, 0 failed, 1 ignored; cargo +1.92.0 test -p inference_providers --lib chutes — 116 passed, 0 failed (includes the new e2ee_stream tests)

[lloydmak99]

@PierreLeGuen follow-up addressed — a JSON-null id now normalizes with absent and empty.

You were right that it was a gap rather than a nicety: Some(_) => false caught Value::Null, so no synthetic id was inserted, and from_value::<ChatCompletionChunk> then failed on id: String from null — reproducing the same Chutes stream chunk parse error this PR exists to prevent, via a different frame shape.

Re-serialization stays conditional, so frames carrying a real id are still emitted byte-for-byte identical — the byte-identity test is unchanged and green.

Verified: clippy -p inference_providers --all-targets -- -D warnings clean; cargo test -p inference_providers --lib chutes 117 passed, 0 failed (up from 116; the new test covers absent, empty and null in one loop and asserts raw_bytes and the typed chunk carry the same id).