fix: auto-cap max_tokens to the model window per prompt

[tjluyao]

What

The vLLM executor passes a request's max_tokens straight through to SamplingParams. vLLM rejects a request when prompt_tokens + max_tokens > max_model_len, so a workflow that hard-codes a large max_tokens fails every invocation regardless of prompt size.

Concretely: a dsm_clause_entailment_audit op requesting max_tokens: 65536 against QuantTrio/Qwen3.6-27B-AWQ (native window 40960) can never succeed — even though the op only emits a short verdict. The surfaced error names neither the parameter nor the op:

mp(vllm) failed: This model's maximum context length is 8192 tokens.
However, you requested 0 output tokens and your prompt contains at least
8193 input tokens...

(The "0 output tokens" is vLLM clamping the output budget to zero once the prompt alone overflows; the real 65536 only becomes visible once the window is large enough to admit the prompt.) This was misdiagnosed as an auth/config problem more than once.

Fix

Auto-cap max_tokens per prompt to window − prompt_tokens − margin, bounded by a floor and never above what was requested. LLM.generate accepts either one SamplingParams or a per-prompt list, so the list is built only when a clamp actually bites — the common case keeps sharing one object.

Deliberately conservative:

• Window unknown (engine shape unrecognised) → no clamp. A missing window must never silently shrink a caller's budget.
• Prompt at/over the window → floor, leaving vLLM to raise the real "input too long" error rather than hiding it behind a 1-token generation.
• Multimodal (TextPrompt) entries → keep the requested budget. Raw text undercounts image tokens, so clamping on text alone could permit more than fits; preserving current behaviour is the safe choice.

The window and max_tokens are read through typed attribute access on the live engine (llm_engine.model_config.max_model_len), guarded by try/except, so a vLLM version skew degrades to "no clamp" rather than crashing.

When a clamp bites it is surfaced to the caller on the result item's diagnostics field (diagnostics.auto_cap = {max_tokens, requested}), so a shorter-than-requested completion is self-explaining instead of a silent truncation. The grouped/table output path aggregates it per member. diagnostics is engine-owned and kept distinct from the user-owned metadata.

Tests

tests/worker/test_vllm_auto_cap_max_tokens.py covers the pure clamp (fit, window-unknown, already-fits, floor, never-above-requested) and the wiring on a bare instance with fakes (per-prompt list, mixed batch, multimodal passthrough, tokenizer-failure fallback) plus the grouped-remap aggregation. The pure clamp is builtins-only; the wiring needs no GPU deps. The suite passes under uv run pytest.

Note

Pairs with the deploy-side change that stopped Lumilake capping max_model_len at 8192 fleet-wide (deploy_infra#51). That fix lets the prompt fit the 40960 window; this fix stops an oversized max_tokens from overflowing it from the other side.

[kaiitunnz]

LGTM.