feat: context-length tier routing (262k fleet vs 1M tier, best-effort Chutes overflow)

[PierreLeGuen]

What

Lets a NEAR-served model run two capacity tiers behind one canonical id — built for GLM-5.2 after cvm-compose-files#128 (2xTP4 fleet at 262k ctx) + cvm-compose-files#129 (TP8 host on glm-5-2-long.completions.near.ai at 1M ctx). Requests whose estimated context requirement doesn't fit the base fleet route to the long tier; a saturated long tier (SGLang --max-queued-requests → 503) falls back to Chutes via the existing retry chain, best-effort.

The tier is data, not code: one PATCH on the existing model row —

{"long_context": {"inference_url": "https://glm-5-2-long.completions.near.ai",
                  "max_context_tokens": 1048576,
                  "base_max_context_tokens": 262144}}

expand_inference_endpoints turns the row into two inference-url entries under the same canonical id, flowing through the normal load_inference_url_models lifecycle (attestation discovery, fingerprint pinning, fail-closed, teardown on removal). Invalid blocks are dropped loudly (identity expansion). No catalog change: /v1/models is untouched, contextLength stays 1048576 (Phala requirement), clients send z-ai/glm-5.2 for every size, same pricing row — the tiering is invisible to users.

Routing (builds on #838)

• Best-fit capacity ordering within a tier: the sort key gains a trailing capacity rank, so short requests stay on the base fleet (the single 1M host is their 5xx failover, not a round-robin peer) and requests that don't fit sort the base fleet last (→ long tier first, Chutes behind). Inside the all-overflow group the rank inverts to biggest-first (closest-to-fitting is the best bet for over-estimates).
• Context requirement refinement (refine_context_requirement): gated on ≥2 distinct declared capacities — every single-capacity model routes byte-identically to main (the service-side estimator and both hint call sites are untouched from feat: adaptive load balancing (TTFT EMA + prefix-hash cache routing + context-length routing) #838). For multi-tier models: ceil(countable_bytes/4 × 1.2) + media/template overhead + max_tokens reserve; near a tier boundary ([0.7, 1.3]×cap) it asks the base fleet for an exact count via the attested POST /v1/tokenize passthrough (new defaulted count_tokens trait method, implemented on the nearai provider over its fingerprint-pinned client; best-effort, 5s budget, concurrency-capped at 4, skipped for encrypted payloads).
• Context-length 400 fall-through: a 400 whose body says the input exceeds the backend's window escapes the 4xx fast-return (same mechanism as Staging: z-ai/glm-5.1 intermittently 400s for tool calls when CHUTES_MODELS fallback is configured #797's model-not-found) only when a sibling with a strictly larger declared window exists — an under-estimated glm-5.2 request self-heals via the 1M tier, while GLM-5.1-style models (Chutes fallback with no declared window) keep today's fast-return and the engine's actionable error.
• CHUTES_MODELS slug takes an optional @<max_context_tokens> suffix (e.g. z-ai/glm-5.2=zai-org/GLM-5.2-TEE@1048576 — verified: that chute serves the full 1M window).
• Admin PATCH runtime registration now builds from the merged post-upsert rows, so a partial PATCH can't tear down the long tier, and providerConfig-only PATCHes take effect immediately.
• METRIC_PROVIDER_REQUESTS gains provider_ctx:{capacity|unbounded} and context_tier:{long|default} tags for long-tier-share dashboards.

Review

Adversarially reviewed (4 dimensions, per-finding verification); all confirmed findings fixed: estimator/hint changes reverted to keep single-capacity models byte-identical, all-overflow ordering inverted, 400 fall-through gated on a larger sibling, invalid long_context blocks rejected, tokenize amplification bounded (semaphore + narrowed band), admin PATCH source-of-truth fixed. Privacy: tokenize text travels only over the pinned attested transport; all new logs carry numbers/ids only.

Testing

• cargo test --lib --bins (1,067 tests), cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings — clean.
• New unit tests: expansion contract (identity/valid/invalid blocks), best-fit + all-overflow ordering, refinement gating (single vs multi capacity), context-400 fall-through WITH bigger sibling + fast-return WITHOUT, engine-phrasing matcher, Chutes @max_ctx parsing, pinned-capacity recording, metric tags.
• Staging plan (after review): PATCH the staging row with a lowered base_max_context_tokens to exercise tier selection cheaply, one real ~250k-token request end-to-end, dead-SNI drill for the Chutes leg, /v1/tokenize shape verification (fail-open to the heuristic if it differs).

Rollout ordering (hard invariant)

Deploys inert (no long_context in any row ⇒ single provider ⇒ behavior identical). Routing + catalog PATCH + cvm-compose-files#129 must ALL be live before any host flips to 2xTP4, else >262k inputs (p99=318k) 400 on the TP4 fleet. Sequencing details in cvm-compose-files#129.

[gemini-code-assist]

Code Review

This pull request implements context-length tier routing, allowing models to declare multiple capacity tiers (e.g., base and long-context) via provider_config. The system estimates request context requirements using a byte-based heuristic, which can be refined with an exact token count via /v1/tokenize near tier boundaries, and routes requests to the best-fit provider. Feedback on these changes suggests lowering the tokenize request timeout from 5 seconds to 2 seconds to avoid latency issues on slow backends, and replacing try_from().unwrap_or() with an explicit saturating cast when converting the required token count to u32.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

Review — context-length tier routing

Reviewed the full diff focusing on correctness, production-safety (rolling updates / multi-tier state), and Rust concerns. No critical/blocking issues found. This is a notably careful change — the design fails toward existing behavior at every branch and the test coverage matches the surface area. Notes below are non-blocking.

Verified safe

• Backward-compatible by construction. expand_inference_endpoints is the identity for any row without a valid long_context block, and refine_context_requirement early-returns unless a model has ≥2 distinct declared capacities. Single-capacity / undeclared models route byte-identically. The rollout is inert until a row carries long_context — good for rolling updates.
• No infinite fall-through. A context-length 400 only escapes the 4xx fast-return when a sibling with a strictly larger declared window exists. The 1M long tier has no larger sibling (Chutes' window is None = "unknown ≠ larger"), so it fast-returns instead of re-uploading to Chutes. Failure counter is correctly left untouched.
• Concurrency/locks. Tokenize semaphore permit is held across the .await (dropped at end of the if let block); all provider_load_state locks use unwrap_or_else(|e| e.into_inner()) poison recovery. needs_creation (mod.rs:4084) and reused (mod.rs:4007) both seed max_context_tokens, so admin PATCHes that only change context numbers take effect without provider recreation.
• Privacy. count_tokens sends prompt text only over the fingerprint-pinned attested client; response body (may echo token ids) and the text are never logged. All new log lines carry numbers/ids only, per CLAUDE.md.

Minor observations (non-blocking)

• Tokenize band ignores output_reserve. near_boundary is checked against pre_factor (input estimate, pre-safety-factor, excluding max_tokens reserve), while the final required includes the reserve. A request with large max_tokens whose input sits just below band_low × cap routes on the heuristic (never triggering the exact count) even though the reserve pushes it across the tier boundary. Routing stays safe (overflow → long tier, which fits), just less precise than an exact count would be. Fine as-is; worth a one-line comment noting the band is deliberately input-only.
• is_context_length_exceeded_error catches max_tokens exceeds the model's context length. On a multi-tier model a genuinely absurd max_tokens will fall through base→long before the long tier fast-returns — one wasted full-payload re-upload. Bounded (single sibling) and already acknowledged in the PR body's tradeoff; noting for completeness.
• Non-streaming latency. The non-streaming path now runs refine_context_requirement, which for boundary-sized requests on multi-tier models can add one ~5s /v1/tokenize round-trip on the critical path (concurrency-capped at 4, try_acquire → skip under contention). Intentional and scoped to multi-tier models only, but it's new latency on a path that previously had none.

Nice touch gating the all-overflow ordering inversion (biggest-window-first) and building admin re-registration from the merged post-upsert rows.

✅ Approved

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01aca41e23

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[PierreLeGuen]

Review disposition (all threads resolved, fixes in d4e5a63):

• Tokenize band now includes the max_tokens reserve (codex P2 + claude-review note): boundary decision uses input estimate + reserve vs each capacity.
• Context-400 fall-through no longer clobbers a retryable error (codex P2): [long 503 → base 400] terminates with the retryable 503 so the round retries the capable tier; regression test added.
• /v1/tokenize timeout 5s → 2s (gemini + claude-review's non-streaming-latency note): fail-open fast on the critical path.
• Saturating cast for the u64→u32 requirement (gemini).

Deferred by design → #861: E2EE long-context single-tier limitation, absurd-max_tokens pre-flight, SGLang tokenize-shape verification on staging, long-context pricing.

[lloydmak99]

Approve. Reviewed the full diff and traced the high-complexity paths against the source.

Verified

• expand_inference_endpoints: identity by default; invalid long_context blocks dropped loudly without touching the base entry (tiers can't collapse into a round-robin onto the single long host). Used on both the DB-load (flat_map) and admin-PATCH paths, built from merged post-upsert rows so a partial PATCH can't tear down the tier.
• Sort capacity_rank: traced all four cases (short→base first; overflow→long/Chutes/base; all-overflow→biggest-first via u32::MAX-capacity; no-hint→unchanged). Matches the tests; single-capacity models order byte-identically.
• Context-400 fall-through: gated on a strictly-larger declared sibling, non-retryable, no failure-counter bump, and keep_prior_retryable correctly stops a base-tier 400 from clobbering the long-tier retryable 503.
• count_tokens/refinement: best-effort, 2s timeout, semaphore(4), skipped for encrypted payloads, text only over the pinned attested transport, numbers-only logging; gated on ≥2 distinct capacities so all other models are untouched.
• Both provider paths (create + reuse) set max_context_tokens; both expanded URLs accumulate under one id.

The four bot P2s (timeout, saturating cast, output-reserve in the band check, retryable-clobber) match the areas I independently flagged and are all fixed in d4e5a63. CI green.

Rollout note (already documented): routing + catalog PATCH + cvm-compose-files#129 must all be live before any host flips to 2xTP4.

[PierreLeGuen]

End-to-end validation (full dev environment, per review request):

1. New e2e suite (crates/api/tests/e2e_all/glm52_tier_routing.rs, commit f40cd94 — permanent CI coverage): 9 tests driving the full app (real DB, real pool, real nearai providers registered via the real admin PATCH) against two wiremock backends with 1k/10k declared windows. All green: small→base with zero long-tier traffic, oversize→long, exact /v1/tokenize overriding the byte heuristic in both directions at the band boundary, engine context-400 fall-through, saturated-long surfacing as retryable 429 (never the base fleet's context-400 — regression for the review fix), streaming tier selection over SSE, catalog transparency, and the full base→long→pinned-attested-fallback chain. cargo test --test e2e_all: 628 passed / 1 pre-existing env-dependent failure that reproduces identically without this change.

2. Real-engine stack (#862): the same behavior verified with a REAL model (Qwen2.5-0.5B on CPU via llama.cpp) and REAL tokenizer counts — demo 8/8, including the boundary case where the true count (801 tokens) kept a request on base that the byte heuristic (~1,201) would have mis-routed, and a curl-visible saturation drill returning 429.

Rollout invariant unchanged: this + the catalog PATCH + cvm-compose-files#129 must be live before any host flips to 2xTP4.