diff --git a/README.md b/README.md index 825e8b10..6e35c3ed 100644 --- a/README.md +++ b/README.md @@ -167,10 +167,12 @@ Python screenshot worker, Valen-Preview-0923 has a Python reference worker for text and one-image state, CLM has a stub-encoder contract recipe, and OmniJev-4B v1.1 runs on a native Rust/CUDA worker. D3 (Decision 3.0) has a Python reference worker that runs the checkpoint's own -pinned runtime for text, image and video state. pplx-decider-v1.1-27b has a -Python reference worker for the pinned Perplexity checkpoint. Public231 -concurrency-1 latency for heads `1ba07d1`, `f968e576`, `ffae182`, `4b99b73`, and `fcf5e2b` is recorded in -the recipe: +pinned runtime for text, image and video state, and Vela 2.0 0.3B has one +that serves all five SystemOne question kinds (`choice`/`noul`/`score` plus +the `span`/`set` extensions) on the checkpoint's own pinned runtime. +pplx-decider-v1.1-27b has a Python reference worker for the pinned Perplexity +checkpoint. Public231 concurrency-1 latency for heads `1ba07d1`, `f968e576`, +`ffae182`, `4b99b73`, and `fcf5e2b` is recorded in the recipe: | Model | Status | | --- | --- | @@ -188,6 +190,7 @@ the recipe: | LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions | | OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) | | D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) | +| Vela 2.0 0.3B | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; all five kinds (`choice`/`noul`/`score`/`span`/`set`) for text state; CPU FP32 validated (author-run, evidence in the integration PR) | | Clef-flash (9B) and Clef (27B) | [Python reference worker](recipe/clef/reference.md) for text `choice`/`noul`/`score`; flash is the bring-up variant. Image, video and native CUDA are not served. L20X-labeled lab Public231 C1 primary is tip `594528a9`: p50 164.969 ms, p95 216.755 ms, `/health` `fla+causal_conv1d`, versus `9030065` run1 p50 247.518 ms, p95 413.025 ms (Δp50 −82.549 ms, Δp95 −196.270 ms) | | pplx-decider-v1.1-27b | [Python reference worker](recipe/pplx_decider/README.md) on the pinned checkpoint; `choice`/`noul`/`score` text plus up to four images. Lab L20X-labeled GPUs are the H200 target. Kept Public231 primary is `fcf5e2b`: 231/231, p50 87.782 ms, p95 291.183 ms, grouped-value FlashInfer with q and k packed before the L2 view (p50 −1.30%, p95 −1.56% versus `4b99b73` 88.942 / 295.797). `4b99b73` was p50 −4.20%, p95 −0.22% versus `ffae182` 92.844 / 296.458. Uncached all-sequence FlashInfer `3c72f02` is not the primary (p50 88.808 ms, p95 300.751 ms). Short-sequence FlashInfer `a50b9aa` missed its gate (p50 95.446 ms, p95 300.871 ms). cuDNN-first SDPA `0dcaa0ac` missed its gate (p50 138.501 ms, p95 332.859 ms). Fused gate-up `8b4ee326` missed its gate (p50 102.437 ms, p95 304.970 ms). Always-on dynamic compile `e36b32ed` missed the both-percentiles gate (p50 97.751 ms, p95 263.124 ms; p95 improved). The 512-token split `7e3b1be` missed its gate (p50 100.037 ms, p95 257.084 ms). Isolated-median compile `34ba5de` missed its gate (p50 97.084 ms, p95 256.679 ms; the 136-token probe was eager). The FlashInfer state-workspace ring `17a7b120` missed its gate (p50 89.165 ms, p95 295.723 ms). Always-on FLA residual RMSNorm `3afb472` missed its gate (p50 93.935 ms, p95 274.902 ms). Length-gated FLA residual RMSNorm `705ae089` missed its gate (p50 94.604 ms, p95 274.835 ms). `use_cp=auto` `a6f4d9a` missed its gate (p50 112.294 ms, p95 299.419 ms). Grouped-value `3ecdf5a` crashed in warmup (0/231, no p50 or p95). FLA fused gated RMSNorm `801ef7b` missed its gate (p50 89.172 ms, p95 274.790 ms). FLA `swiglu_fwd` `2bbbe4e` missed its gate (owner closeout p50 97.613 ms, p95 292.795 ms) and is not installed | diff --git a/docs/supported-models.md b/docs/supported-models.md index 1eb8b4e1..c5dbc4c2 100644 --- a/docs/supported-models.md +++ b/docs/supported-models.md @@ -28,6 +28,7 @@ Models that are being added are also tracked in issues labeled [new model](https | CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code | | OmniJev-4B v1.1 | [Native Rust/CUDA worker](../recipe/omnijev/native.md) on the shared Qwen3.5 vision and language path, eager | Contract, preparation, pixels, heads and finishing checked against the reference; no CPU inference | Compared with the reference on compute capability 8.9 for [140 answers](../recipe/omnijev/native.md#comparison-with-the-reference) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, the export from `export.py` | | D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) on the checkpoint's pinned runtime — text, image and video state over `choice`/`noul`/`score`, with the deliberate transport differences documented in the recipe; the native worker is in progress ([#136](https://github.com/ThinkFlowLab/system1-omni/issues/136)) | Unverified | Author-run on NVIDIA A100 (sm_80): serving smoke and reference parity for text and media (exact decisions, max delta 0.0); evidence in the integration and media PRs. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB); [media budgets and the 64 MiB total request cap](../recipe/d3/reference.md) | +| Vela 2.0 0.3B | [Python reference worker](../recipe/vela/reference.md) running the checkpoint's pinned runtime | **CPU FP32 validated** (author-run, x86 WSL2, 24 cores): startup with all-kind warmup, the four example requests plus five synthetic rejections, and exact-match reference parity against the pinned upstream server on all nine (four 200s byte-equal incl. key order, agreed 422/413); byte-identical frontend forwarding; evidence in the integration PR. Owner: @MrDongsls. | Unverified (bf16-autocast is an untested deployment option) | Not supported | Python 3.12, the [pinned requirements](../recipe/vela/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) (1.24 GB FP32 weights; ~4.4 GB with ONNX/assets) | | Clef-flash (9B) | [Python reference worker](../recipe/clef/reference.md); text `choice`/`noul`/`score`; non-thinking prompt; `thinking.enabled: false` admitted and omitted from the prompt; 16384-token refusal | Contract tests only; no inference | Operator-reported Public231 C1 on the L20X-labeled lab host. Baseline `9030065` run1 p50 247.518 ms, p95 413.025 ms, 231/231. Tip `594528a9` remeasure, warmup 5, concurrency 1, stock Public231 with `thinking.enabled` false, metric `successful_latency`: 231/231, p50 164.969 ms, p95 216.755 ms, `/health` `fla+causal_conv1d`. Versus run1: Δp50 −82.549 ms (−33.35%), Δp95 −196.270 ms (−47.52%). Not re-run here | Not supported | Python 3.12, [card-tested torch 2.11 and transformers 5.10.2](../recipe/clef/requirements.txt), pinned `Cloudflare/clef-flash` snapshot (~19.1 GB) | | Clef (27B) | Same worker and text contract as Clef-flash, selected with `--variant clef` | Contract tests only; no inference | Not run | Not supported | Same stack; pinned `Cloudflare/clef` snapshot (~55.0 GB). Weights and joint-head config differ from flash | | pplx-decider-v1.1-27b | [Python reference worker](../recipe/pplx_decider/README.md) on the checkpoint's own runtime. Text and up to four images; `choice`/`noul`/`score`. Native CUDA executor is not this worker. Lab L20X-labeled GPUs are the H200 target. The causal-conv1d build keeps sm_90 and sm_89. Capability 8.9 is ready on FLA chunk. Capability 9.0 uses all-sequence FlashInfer. The kept primary rotates 48 `cu_seqlens` buffers per device and length, passes `use_cp` false, skips the q/k repeat, and packs q and k before the L2-norm view. `801ef7b` fused the 48 linear gated RMSNorms and missed the median (p50 89.172 ms, p95 274.790 ms), so that kernel stays off. `2bbbe4e` replaced `silu(gate) * up` with FLA `swiglu_fwd` and the owner closeout rejected it (p50 97.613 ms, p95 292.795 ms), so that kernel stays off. Gate and up GEMMs stay separate. Residual RMSNorm and the gated output norm stay eager. An unknown `PPLX_GDN_PREFILL` fails closed on every device, including capability 8.9. `3ecdf5a` crashed in warmup (0/231). Always-on fusion `3afb472` and the 512-token gate `705ae089` missed the median. The 48-slot FlashInfer state ring `17a7b120` missed the gate. `PPLX_GDN_PREFILL=fla` reproduces the prior `f968e576` chunk. `PPLX_GDN_PREFILL=short` is the rejected ≤256 dispatch. SDPA stays on the checkpoint order. The fused gate-up GEMM is refused. The forward stays eager. `PPLX_COMPILE=dynamic` and `PPLX_COMPILE=dynamic-tail` are rejected opt-ins | Contract checks only; no inference | Reported load on 1× lab L20X (sm_89, ~140 GiB) at head `1ba07d1`: BF16 resident ~50333 MiB, no OOM. Historical primary: 231/231, p50 98.777 ms, p95 341.202 ms. Prior primary is the `f968e576` remeasure on the lab L20X-labeled GPU, `compute_capability=9.0`, `fla-chunk`, `flashinfer` null: 231/231, p50 94.077 ms, p95 300.200 ms (p50 −4.76%, p95 −12.02%). Prior primary `ffae182`: 231/231, p50 92.844 ms, p95 296.458 ms (p50 −1.31%, p95 −1.25%). Prior primary `4b99b73`: 231/231, p50 88.942 ms, p95 295.797 ms (p50 −4.20%, p95 −0.22%), 48-slot `cu_seqlens`, quality `choice_acc=0.9065` (139), `noul_acc=0.8784` (74), `score_mae=0.2035` (18). Kept primary is `fcf5e2b`: 231/231, p50 87.782 ms, p95 291.183 ms (p50 −1.30%, p95 −1.56%), grouped-value FlashInfer with q and k packed, same quality digits. Uncached all-sequence FlashInfer `3c72f02`: p50 88.808 ms, p95 300.751 ms, gate failed. Short-sequence FlashInfer `a50b9aa`: p50 95.446 ms, p95 300.871 ms, gate failed. cuDNN-first SDPA `0dcaa0ac`: p50 138.501 ms, p95 332.859 ms, gate failed. Fused gate-up `8b4ee326`: p50 102.437 ms, p95 304.970 ms, gate failed. Always-on dynamic compile `e36b32ed`: p50 97.751 ms, p95 263.124 ms, gate failed (p95 improved). 512-token compile `7e3b1be`: p50 100.037 ms, p95 257.084 ms, gate failed. Isolated-median compile `34ba5de`: p50 97.084 ms, p95 256.679 ms, gate failed (median probe was eager). FlashInfer state-workspace ring `17a7b120`: p50 89.165 ms, p95 295.723 ms, gate failed. Always-on FLA residual RMSNorm `3afb472`: p50 93.935 ms, p95 274.902 ms, gate failed. Length-gated FLA residual RMSNorm `705ae089`: p50 94.604 ms, p95 274.835 ms, gate failed. `use_cp=auto` `a6f4d9a`: p50 112.294 ms, p95 299.419 ms, gate failed. Grouped-value `3ecdf5a` crashed in warmup (0/231, no p50 or p95). The fused linear gated-norm candidate has no sample. Does not enter the H800 board | Not supported | Python 3.12, pinned checkpoint revision `14c9d90137962d300f3de95633f7b6535e80c5ce`, checkpoint `requirements.txt`, `causal-conv1d` 1.7.0, `flash-linear-attention` 0.5.2, optional `flashinfer-python` 0.6.16.post3. Shard sizes in the release manifest sum with the readout to about 48.6 GiB before activations | @@ -43,6 +44,9 @@ StartLux serves Instant only; AutoThink is not implemented, and the weights are D3's reference worker serves the same three kinds for text, image and video state (inline base64 media under the pinned runtime's per-item budgets); its media serving evidence is author-run on the A100 above. +Vela 2.0 0.3B's reference worker serves all five kinds (`choice`, `noul`, +`score`, plus the `span`/`set` extensions with their Noul views) for text +state on CPU FP32; the hybrid members (0.8B/4B/9B) are separate slices. Clef-flash and Clef serve the same three kinds for text only. The worker refuses image and video inputs. The flash CUDA cell records two operator-reported Public231 concurrency-1 results on the L20X-labeled lab @@ -70,8 +74,8 @@ compile tips missed the gate and are not the primary. Compile improved p95 and raised p50, including after the median route proved eager. The FlashInfer state-workspace ring `17a7b120` missed the gate (p50 89.165 ms, p95 295.723 ms) and is not the primary. Always-on FLA residual -RMSNorm `3afb472` missed the gate (p50 93.935 ms, p95 274.902 ms) and -is not the primary. Length-gated FLA residual RMSNorm `705ae089` +RMSNorm `3afb472` missed the gate (p50 93.935 ms, p95 274.902 ms) and is +not the primary. Length-gated FLA residual RMSNorm `705ae089` missed the gate (p50 94.604 ms, p95 274.835 ms) and is not the primary. `use_cp=auto` `a6f4d9a` missed the gate (p50 112.294 ms, p95 299.419 ms) and is not the primary. Grouped-value `3ecdf5a` crashed in warmup (0/231) and is not a latency result. FLA fused gated RMSNorm `801ef7b` missed the gate (p50 89.172 ms, p95 274.790 ms) and is not the primary. FLA `swiglu_fwd` `2bbbe4e` missed the gate (owner closeout p50 97.613 ms, p95 292.795 ms) and is not installed. CLM has a row per path: the merged `clm-serve` recipe exercises those answer shapes with stub diff --git a/mkdocs.yml b/mkdocs.yml index 065901a4..7e3be0ad 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -84,6 +84,7 @@ nav: - CLM stub-encoder contract: recipe/clm/README.md - LFM2.5-350M choice worker: recipe/lfm2/README.md - D3 reference worker: recipe/d3/reference.md + - Vela 2.0 0.3B reference worker: recipe/vela/reference.md - Clef reference worker: recipe/clef/reference.md - pplx-decider reference worker: recipe/pplx_decider/README.md - Contributing: CONTRIBUTING.md diff --git a/recipe/README.md b/recipe/README.md index c7bdf791..041c091d 100644 --- a/recipe/README.md +++ b/recipe/README.md @@ -48,6 +48,10 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting verify its runtime manifest, and serve `choice`/`noul`/`score` decisions over text, image and video state through the Python worker and the Rust frontend (text serving and parity validated on A100; media budgets and caps in the recipe). +- [Vela 2.0 0.3B reference worker](vela/reference.md): download the pinned + vllm-sr/Vela-2.0-0.3B package, pass its full-hash package gate, and serve all + five SystemOne question kinds (`choice`/`noul`/`score` plus the `span`/`set` + extensions) through the Python worker and the Rust frontend on CPU FP32. - [Clef reference worker](clef/reference.md): load pinned Clef-flash (9B) or Clef (27B) and serve text `choice`/`noul`/`score`. Image, video and native CUDA are not served. Operator-reported flash Public231 concurrency-1 diff --git a/recipe/vela/compare_reference.py b/recipe/vela/compare_reference.py new file mode 100644 index 00000000..fde4cd15 --- /dev/null +++ b/recipe/vela/compare_reference.py @@ -0,0 +1,295 @@ +#!/usr/bin/env python3 +"""Compare the Vela 2.0 0.3B worker pipeline against the pinned upstream server. + +Runs each fixture request through two paths that share one loaded pinned +engine, so the comparison isolates this repository's admission and +orchestration from the checkpoint's own code: + +- reference: the pinned package's own ``vela2_serve.py`` app, driven + through FastAPI's test client exactly as a client would (routing, body + parsing, whole-request 422/413 mapping included); +- worker: the production ``frontend.vela.decide`` pipeline (protocol + envelope admission, pinned ``system_one_request`` execution, response + finishing). + +Both paths execute the same verified runtime, and the worker delegates +response construction to the same pinned code — so the criterion is the +charter's reference parity: exact match, including JSON key order, with +rejections agreeing on status and body (``{"detail": [{"loc", "msg", +"type"}]}``). ``--export`` additionally freezes the oracle fixture — +request, SystemOne response and the predict()-level answers — for the +future native encoder slices, and writes it only after a passing, +non-rejected comparison. Requires the pinned package, the reference +dependencies, and fastapi+uvicorn+httpx; see recipe/vela/reference.md. + +The default fixture set is the recipe's four example requests plus +five synthetic rejections: an invalid span question (whole-request +422), a schema that alone exceeds the 8,192-token window (413), and +the three envelope 422s (missing field, extra field, wrong model +name) that freeze the worker's preamble admission against the pinned +runtime's. CPU FP32 is the oracle setting; GPU bf16-autocast is a +deployment option, not this tool's default. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +MODEL_NAME = "vllm-sr/Vela-2.0-0.3B" + + +def parse_args(argv=None): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--model", required=True, help="pinned Vela 2.0 0.3B package directory") + parser.add_argument("--device", default="cpu", help="torch device (default: the CPU FP32 oracle)") + parser.add_argument( + "--fixture", + action="append", + default=None, + help="request body file (repeatable; default: the recipe examples plus the " + "synthetic invalid-span and over-limit rejections)", + ) + parser.add_argument( + "--export", + default=None, + help="write the frozen oracle fixture (request, SystemOne response, predict-level " + "answers per passing 200 case); only passing, non-rejected comparisons are written", + ) + return parser.parse_args(argv) + + +def over_limit_body(): + """A schema that alone cannot fit the 8,192-token window. + + 255 choice options with ~300-character descriptions: far past + ``max_length`` before any state is rendered, which is exactly the + upstream 413 condition (questions + schema alone; state text would be + truncated, never 413'd). + """ + + criteria = { + f"option_{index}": ( + f"a deliberately long description repeating filler text to overflow the " + f"window for option {index}: " + "filler " * 40 + ) + for index in range(255) + } + return { + "model": MODEL_NAME, + "state": "short", + "questions": { + "big": {"type": "choice", "instructions": "Pick one.", "criteria": criteria} + }, + } + + +def invalid_span_body(): + """A span question whose criteria is a list, not a label mapping.""" + + return { + "model": MODEL_NAME, + "state": "short", + "questions": { + "bad": { + "type": "span", + "instructions": "Which spans?", + "criteria": ["PERSON", "EMAIL_ADDRESS"], + } + }, + } + + +def missing_model_body(): + """An envelope missing the model field (the preamble's 'missing' 422).""" + + return {"state": "short", "questions": {"q": {"type": "noul", "instructions": "any?"}}} + + +def extra_field_body(): + """An envelope with a field the preamble does not permit ('extra_forbidden').""" + + return { + "model": MODEL_NAME, + "state": "short", + "questions": {"q": {"type": "noul", "instructions": "any?"}}, + "images": [], + } + + +def wrong_model_body(): + """Another member's name on this runtime ('value_error.model' 422).""" + + return { + "model": "vllm-sr/Vela-2.0-0.8B", + "state": "short", + "questions": {"q": {"type": "noul", "instructions": "any?"}}, + } + + +def default_cases(): + return [ + ("example-request", HERE / "example-request.json"), + ("example-request-set-threshold", HERE / "example-request-set-threshold.json"), + ("example-request-cjk", HERE / "example-request-cjk.json"), + ("example-request-score", HERE / "example-request-score.json"), + ("invalid-span-422", invalid_span_body()), + ("over-limit-413", over_limit_body()), + ("missing-model-422", missing_model_body()), + ("extra-field-422", extra_field_body()), + ("wrong-model-422", wrong_model_body()), + ] + + +def load_cases(paths): + if not paths: + cases = default_cases() + else: + cases = [(Path(path).name, path) for path in paths] + return [ + (name, body if isinstance(body, dict) else json.loads(Path(body).read_text(encoding="utf-8"))) + for name, body in cases + ] + + +def reference_response(client, body): + """One request through the pinned upstream server app. + + Returns the (status, parsed-body) pair a client observes. + """ + + response = client.post("/v1/systemone", json=body) + return response.status_code, response.json() + + +def worker_response(raw, executor, decide): + """One request through the production worker pipeline. + + Returns (200, response) or the (status, body) rejection a client + would observe. + """ + + try: + return 200, decide(raw, executor) + except Exception as exc: # noqa: BLE001 - reported through the comparison + status = getattr(exc, "status", 500) + body = getattr(exc, "body", None) + return status, body if body is not None else {"detail": str(exc)} + + +def same_result(worker, reference) -> bool: + """Order-preserving equality of the two observed results. + + ``dict == dict`` ignores insertion order, but key order is part of + the parity criterion; text dumped from each side keeps that order in + the comparison. + """ + + return json.dumps(worker, ensure_ascii=False) == json.dumps(reference, ensure_ascii=False) + + +def compare_case(name, worker, reference): + """Exact-match comparison of the two observed results. + + Returns (report lines, passed). Key order is part of the match: + insertion order is the pinned runtime's answer/question order. + """ + + lines = [f"case: {name}", "reference path: pinned vela2_serve app (FastAPI test client)"] + problems = [] + if same_result(worker, reference): + status = worker[0] + lines.append(f"both paths: HTTP {status}") + if status == 200: + answers = worker[1].get("answers", {}) + usage = worker[1].get("usage", {}) + lines.append(f"answers: {len(answers)} key(s) in pinned order") + lines.append(f"input_tokens: {usage.get('input_tokens')} == reference") + else: + lines.append(f"rejection body: {json.dumps(worker[1], ensure_ascii=False)[:160]}") + lines.append("exact match (status, body, key order): PASS") + return lines, True + problems.append("results differ") + if worker[0] != reference[0]: + problems.append(f"status {worker[0]} != {reference[0]}") + lines.append(f"worker: HTTP {worker[0]}") + lines.append(f"reference: HTTP {reference[0]}") + lines.append("exact match (status, body, key order): FAIL: " + "; ".join(problems)) + return lines, False + + +def export_allowed(ok, worker): + """The oracle freezes served 200 responses, never a rejection.""" + + return ok and worker[0] == 200 + + +def export_oracle(name, body, response, predict, path): + """Freeze one passing case for the future native encoder slices.""" + + oracle = { + "case": name, + "request": body, + "system_one": response, + "predict": predict, + } + Path(path).write_text(json.dumps(oracle, indent=1, ensure_ascii=False) + "\n", encoding="utf-8") + return oracle + + +def main(argv=None): + args = parse_args(argv) + sys.path.insert(0, str(HERE.parents[1] / "src")) + from fastapi.testclient import TestClient + + from frontend.vela import decide + from models.vela.engine import Vela2Executor + + executor = Vela2Executor(args.model, device=args.device) + # Both runtime imports stay inside a no-bytecode window: the package + # gate refuses existing caches, and vela2_serve is a fresh import that + # would otherwise write the __pycache__ the next worker start rejects. + previous_dont_write_bytecode = sys.dont_write_bytecode + sys.dont_write_bytecode = True + try: + import vela2_inference + import vela2_serve + finally: + sys.dont_write_bytecode = previous_dont_write_bytecode + + app = vela2_serve.create_app(executor.engine, model_name=executor.model_name, api_key=None) + client = TestClient(app) + + exported = 0 + failed = 0 + try: + for name, body in load_cases(args.fixture): + raw = json.dumps(body).encode("utf-8") + worker = worker_response(raw, executor, decide) + reference = reference_response(client, body) + lines, ok = compare_case(name, worker, reference) + print("\n".join(lines)) + if not ok: + failed += 1 + elif args.export and export_allowed(ok, worker): + req, _plan = vela2_inference.system_one_to_predict(body["state"], body["questions"]) + target = Path(args.export) + if target.is_dir(): + target = target / f"oracle-{name}.json" + export_oracle(name, body, worker[1], executor.engine.predict(req), target) + exported += 1 + print() + finally: + executor.close() + if args.export: + print(f"oracle cases exported: {exported}" if exported else + "oracle export skipped: no passing 200-response comparison to freeze") + print("compare_reference: ok" if not failed else f"compare_reference: {failed} FAILED") + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/recipe/vela/example-request-cjk.json b/recipe/vela/example-request-cjk.json new file mode 100644 index 00000000..bd3ab688 --- /dev/null +++ b/recipe/vela/example-request-cjk.json @@ -0,0 +1,32 @@ +{ + "model": "vllm-sr/Vela-2.0-0.3B", + "state": { + "request": "客户张伟（zhang.wei@example.cn）于2026年10月1日在上海市浦东新区世纪大道100号办理了业务，联系电话13800138000。", + "source": "客户的业务申请需要在三个工作日内审核完成。", + "answer": "客户的业务已经即时完成了。" + }, + "questions": { + "pii": { + "type": "span", + "instructions": "Which spans are personal information?", + "criteria": { + "PERSON": "a person's name", + "EMAIL_ADDRESS": "an e-mail address", + "PHONE_NUMBER": "a telephone number", + "STREET_ADDRESS": "a street address or postal address" + }, + "over": "request" + }, + "unsupported": { + "type": "span", + "instructions": "Which spans of the answer are not supported by the context?", + "criteria": { + "unsupported": "a claim not supported by the context" + } + }, + "billing": { + "type": "noul", + "instructions": "这个请求是否与账单有关？" + } + } +} diff --git a/recipe/vela/example-request-score.json b/recipe/vela/example-request-score.json new file mode 100644 index 00000000..fa37f20c --- /dev/null +++ b/recipe/vela/example-request-score.json @@ -0,0 +1,16 @@ +{ + "model": "vllm-sr/Vela-2.0-0.3B", + "state": "Hello, I would like to check the status of my order. It has been five days and I have heard nothing back.", + "questions": { + "tone": { + "type": "score", + "instructions": "How formal is the message?", + "criteria": ["casual", "neutral", "formal"] + }, + "urgency": { + "type": "score", + "instructions": "How urgently does the sender need a reply?", + "criteria": ["no rush", "soon", "today", "immediately"] + } + } +} diff --git a/recipe/vela/example-request-set-threshold.json b/recipe/vela/example-request-set-threshold.json new file mode 100644 index 00000000..901927fc --- /dev/null +++ b/recipe/vela/example-request-set-threshold.json @@ -0,0 +1,16 @@ +{ + "model": "vllm-sr/Vela-2.0-0.3B", + "state": "My card was charged twice and the parcel never arrived.", + "questions": { + "issues": { + "type": "set", + "instructions": "Which issues does the customer report?", + "criteria": { + "billing": "payments, charges, refunds or invoices", + "shipping": "delivery of an order or a parcel", + "login": "signing in, passwords or account access" + }, + "threshold": 0.15 + } + } +} diff --git a/recipe/vela/example-request.json b/recipe/vela/example-request.json new file mode 100644 index 00000000..8e0557b1 --- /dev/null +++ b/recipe/vela/example-request.json @@ -0,0 +1,51 @@ +{ + "model": "vllm-sr/Vela-2.0-0.3B", + "state": { + "request": "Hi, I'm Tom Baker (tom.baker@example.com). What is the maximum daily dose of paracetamol for an adult?", + "source": "For adults, the maximum dose of paracetamol is 4 grams in 24 hours, taken as 500 mg to 1 g every 4 to 6 hours.", + "answer": "Adults can take up to 6 grams of paracetamol in 24 hours, in doses of 500 mg to 1 g every 4 to 6 hours." + }, + "questions": { + "pii": { + "type": "span", + "instructions": "Which spans are personal information?", + "criteria": { + "AGE": "a person's age", + "CREDIT_CARD": "a payment card number", + "DATE_TIME": "a date, time or date of birth", + "DOMAIN_NAME": "an internet domain name or website address", + "EMAIL_ADDRESS": "an e-mail address", + "GPE": "a country, city, state or other geopolitical place", + "IBAN_CODE": "an international bank account number (IBAN)", + "IP_ADDRESS": "an IPv4 or IPv6 address", + "NRP": "a nationality, religious or political group", + "ORGANIZATION": "the name of a company, institution or other organisation", + "PERSON": "a person's name", + "PHONE_NUMBER": "a telephone number", + "STREET_ADDRESS": "a street address or postal address", + "TITLE": "a personal or professional title such as Dr. or Mrs.", + "US_DRIVER_LICENSE": "a US driver's licence number", + "US_SSN": "a US social security number", + "ZIP_CODE": "a postal or ZIP code" + }, + "over": "request" + }, + "halu": { + "type": "span", + "instructions": "Which spans of the answer are not supported by the context?", + "criteria": { + "unsupported": "a claim not supported by the context" + } + }, + "domain": { + "type": "choice", + "instructions": "Which subject area is this request about?", + "over": "request", + "criteria": { + "health": "medicine, clinical practice, nutrition, ageing or sexual health", + "math": "arithmetic, algebra, geometry, statistics or other mathematics", + "other": "a subject that fits none of the listed areas" + } + } + } +} diff --git a/recipe/vela/reference.md b/recipe/vela/reference.md new file mode 100644 index 00000000..98d30f9c --- /dev/null +++ b/recipe/vela/reference.md @@ -0,0 +1,271 @@ +# Vela 2.0 0.3B reference worker + +This recipe serves Vela 2.0 0.3B through the Python reference worker and +the existing Rust frontend. The model-side implementation is under +[`src/models/vela/`](../../src/models/vela/), and the worker is +[`src/frontend/vela.py`](../../src/frontend/vela.py). + +Run the commands from the repository root with Python 3.12. The default +and evidence-backed setting is CPU FP32 — the reference oracle; a CUDA +device works for serving but its bf16-autocast backbone is a deployment +option, not the parity baseline. The recipe records the procedure; +measured results and verbatim run output live in the pull request +description, not here. + +## Scope + +The slice is: + +- package: [`vllm-sr/Vela-2.0-0.3B`](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) + at the pinned revision below; +- the Python reference worker behind the Rust frontend, executing the + checkpoint's own pinned runtime (`vela2_inference.py` and siblings) + after a full-hash package gate; +- text/JSON state and all five typed kinds — `choice`, `noul`, `score`, + `span`, `set` — in one request, with the span/set response extensions + (`spans`/`sets`/`thresholds` and their Noul views) served + forward-untouched; +- FP32 parameters, one bidirectional forward per assembled sequence, + calibrated readouts; zero generated tokens. + +Out of scope for this slice: the hybrid members (0.8B/4B/9B — their +reference workers are a separate slice), the ONNX backend, any native +Rust/CUDA executor (the frozen RFC's ModernBERT-profile route), and +`noul_calibration` (a hybrid-only feature that does not exist on 0.3B). + +## Architecture and ownership + +~~~text +Rust frontend + -> Python worker + -> admit: JSON transport + upstream envelope preamble (protocol) + -> verify: config identity pin, full SHA-256 of the 7-file table, + exact weights size, then import it (engine) + -> execute: the checkpoint's own vela2_inference system_one_request + -> finish: response-order/key-set checks + formula mirrors (postprocess) +~~~ + +The ownership boundary follows +[`docs/architecture.md`](../../docs/architecture.md) and the +[model contract](../../src/models/vela/README.md): + +- `protocol.py` owns the envelope preamble with upstream's exact 422 + triples; deeper validation (state parts, question semantics, the 413) + is the pinned runtime's, verbatim; +- `preprocess.py` maps the runtime's `SystemOneError` to 422/413; +- `engine.py` owns the pinned runtime and the package gate; +- `postprocess.py` owns the pure formulas and response finishing; +- `frontend.vela` owns HTTP lifecycle, the single-flight 503 lock, the + `VELA2_API_KEY` 401 gate and warmup-before-readiness; +- The Rust frontend forwards `/v1/systemone` and `/health` unchanged. + +## Pinned artifacts + +| Artifact | Value | +| --- | --- | +| Package | [vllm-sr/Vela-2.0-0.3B](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) | +| Pinned revision | `d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409` | +| Package gate | config structural identity + a 7-file SHA-256 table pinned per revision in `engine.py`, full hash before import, no unlisted importables, `__pycache__` or directory symlinks in the package root, exact 1,236,473,252-byte weights check | +| License | Apache-2.0; tokenizer material under Gemma Terms of Use | + +The gate's file table is checked in (not read from the package, which +ships no manifest), so a later snapshot — self-consistent or not — +cannot pass as the pinned revision. Re-check before changing the pin: + +~~~sh +curl -sS "https://huggingface.co/api/models/vllm-sr/Vela-2.0-0.3B/revision/d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409" | jq -r '.sha' +sha256sum weights/vela-2.0-0.3b/{vela2_inference.py,modeling_vela2.py,configuration_vela2.py,vela2_serve.py,config.json,calibration.json,tokenizer.json} +~~~ + +## Install and prepare the artifacts + +~~~sh +python3.12 -m venv .venv +.venv/bin/python -m pip install --upgrade pip +.venv/bin/python -m pip install -r recipe/vela/requirements.txt pytest +~~~ + +~~~sh +export VELA_PKG="$PWD/weights/vela-2.0-0.3b" +export VELA_REV=d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409 +mkdir -p weights +.venv/bin/hf download vllm-sr/Vela-2.0-0.3B --revision "$VELA_REV" --local-dir "$VELA_PKG" +~~~ + +The full package is about 4.4 GB (the 1.24 GB FP32 safetensors, a 34 MB +tokenizer, the ONNX exports and doc assets); serving needs roughly +2 GB of RAM for the FP32 model plus working memory for the batch rows. +The worker refuses to start when the config is not the pinned member's, +when any of the 7 gated files fails its hash, when the package root +holds an importable entry the table does not name, an existing +`__pycache__` or a directory symlink, or when the weights file is not +the pinned size — that +gate runs before any package code is imported. + +## Start the worker + +~~~sh +PYTHONPATH=src .venv/bin/python -m frontend.vela \ + --model "$VELA_PKG" --device cpu --port 8000 +~~~ + +The worker verifies the package, loads the FP32 model, runs a real +warmup request covering every question kind, and only then binds the +port: + +~~~sh +curl -sS -i http://127.0.0.1:8000/health # {"status": "ok"} (upstream verbatim) +curl -sS -i http://127.0.0.1:8000/ready # {"ready": true} +curl -sS -i http://127.0.0.1:8000/v1/models +~~~ + +Optional: set `VELA2_API_KEY` to require `Authorization: Bearer +<$VELA2_API_KEY>` on `/v1/systemone` and `/v1/models` (401 with the +upstream message otherwise), and `VELA2_MODEL_NAME` (or `--name`) to +serve under a different name. One request is served at a time; a +concurrent one gets 503. + +## Example requests + +Four complete runnable requests ship with the recipe: + +- [`example-request.json`](example-request.json) — the model card's PII + example: a span question over the 17 trained PII labels (the PII + length-rule threshold is applied and echoed), a hallucination span + question over the answer part, and a domain `choice` question; +- [`example-request-set-threshold.json`](example-request-set-threshold.json) + — a `set` question with a per-question `threshold` override (the + applied value 0.15 is echoed in `thresholds`; the pinned predict + level marks overrides with rule `given` on span answers); +- [`example-request-cjk.json`](example-request-cjk.json) — CJK state + exercising Unicode code-point offsets (one unit per CJK character); +- [`example-request-score.json`](example-request-score.json) — two + `score` questions (three and four ordered levels) covering the fifth + answer kind in the default parity run. + +~~~sh +curl -sS http://127.0.0.1:8000/v1/systemone \ + -H 'Content-Type: application/json' \ + --data-binary @recipe/vela/example-request.json +~~~ + +Expected response shape (a serving smoke test, not an accuracy result; +the exact numbers come from the checkpoint and are frozen by the parity +tool below): + +~~~json +{ + "model": "vllm-sr/Vela-2.0-0.3B", + "answers": { + "pii": {"type": "noul", "noul": 0.0}, + "halu": {"type": "noul", "noul": 0.0}, + "domain": {"type": "choice", "choice": "…", "confidence": 0.0, "probabilities": {"…": 0.0}} + }, + "usage": {"input_tokens": 0, "output_tokens": 0}, + "spans": {"pii": [{"label": "…", "start": 0, "end": 0, "text": "…", "probability": 0.0}]}, + "thresholds": {"pii": 0.0, "halu": 0.55} +} +~~~ + +## Start and verify the Rust frontend + +~~~sh +cargo build --release --locked +OMNI_JEV_BIND=127.0.0.1:8080 \ +OMNI_JEV_BACKEND_URL=http://127.0.0.1:8000 \ + target/release/omni-jev +~~~ + +Send the same request through port 8080 and compare byte-for-byte: + +~~~sh +curl -fsS http://127.0.0.1:8000/v1/systemone \ + -H 'Content-Type: application/json' \ + --data-binary @recipe/vela/example-request.json > /tmp/vela-worker.json +curl -fsS http://127.0.0.1:8080/v1/systemone \ + -H 'Content-Type: application/json' \ + --data-binary @recipe/vela/example-request.json > /tmp/vela-frontend.json +cmp /tmp/vela-worker.json /tmp/vela-frontend.json +~~~ + +Worker responses are forwarded unchanged. Frontend-generated 502/504 +responses are frontend errors, not model responses. + +## Error behavior + +The request, response and error contract — the two error layers, the +five answer shapes, usage counting and the state-truncation order — is +the [model contract](../../src/models/vela/README.md). In short: +transport errors follow the shared worker discipline; every invalid +field rejects the whole request with the upstream 422 +`{"detail": [{"loc", "msg", "type"}]}`; 413 `too_long` fires only when +questions plus schema alone exceed 8,192 tokens; and over-limit state +text is **truncated as the reference truncates it** (shrink part first, +floor 64 then 1, then protected parts; windowed re-coverage of cut +labelled parts) — a deliberate, loudly documented divergence from the +repository's no-silent-truncation convention, with the guardrail risk +recorded in the model contract. + +## Repository-side validation + +~~~sh +PYTHONPATH=src .venv/bin/python -m pytest tests/vela -q +PYTHONPATH=src .venv/bin/python -m compileall -q src/models/vela src/frontend/vela.py tests/vela +.venv/bin/python recipe/vela/test_compare_reference.py +cargo fmt --all --check +cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings +cargo test --workspace --locked --features omni-laya/serve +cargo build --workspace --release --locked --features omni-laya/serve +~~~ + +Run these from the repository root. The recorded outputs of a run belong +in the pull request description, not in this recipe. + +## Reference parity (author-side, opt-in) + +[`compare_reference.py`](compare_reference.py) drives the fixture set — +the four example requests and five synthetic rejections (an +invalid-span 422, an over-limit 413, and the missing-field / +extra-field / wrong-model envelope 422s) — through two paths that +share one loaded engine: the pinned +`vela2_serve.py` app via FastAPI's test client, and the production +`frontend.vela.decide` pipeline. The criterion is the charter's +reference parity: **exact match** including JSON key order for 200s, +and agreement on status and body for rejections (needs `pip install +fastapi uvicorn httpx` on top of the requirements file). +`--export tests/vela/fixtures/oracle-.json` freezes each passing +200 case — request, SystemOne response and predict()-level answers — as +the frozen comparison baseline for the frozen RFC's native encoder +slices, and writes it only after a passing comparison. + +~~~sh +PYTHONPATH=src .venv/bin/python recipe/vela/compare_reference.py \ + --model "$VELA_PKG" --device cpu \ + --export tests/vela/fixtures/ +~~~ + +Run it by hand and record the verbatim output, host, package revision +and dependency versions in the pull request description. CPU FP32 is +the oracle setting; keep flash-linear-attention out of the fixture path +(it does not exist on this encoder member anyway) and do not substitute +GPU bf16-autocast numbers for the baseline. + +## Troubleshooting and limits + +- The package directory must be the pinned revision's download; the + worker refuses foreign configs, hash mismatches, unlisted importables, + `__pycache__` bytecode caches, directory symlinks and wrong weights + sizes at startup, before importing anything. +- The 8 MiB body cap is slice-scoped (upstream sets no total-body + limit); a valid request bounded by the 8,192-token window is far + below it. +- `/health` and `/ready` are upstream's minimal bodies on this member, + not the richer shape of some other workers in this repository. +- Long states are truncated (see Error behavior); if a consumer must + never miss tail text, check `usage.input_tokens` against the expected + budget or enforce limits upstream of the worker. +- GPU bf16-autocast serving (`--device cuda`) is untested in this + slice's evidence; treat it as a deployment option to validate + separately. +- GPU evidence for the supported-models table is recorded in the + integration pull request, not in this recipe. diff --git a/recipe/vela/requirements.txt b/recipe/vela/requirements.txt new file mode 100644 index 00000000..9f2cb6cf --- /dev/null +++ b/recipe/vela/requirements.txt @@ -0,0 +1,15 @@ +# Reference-worker dependencies for Vela 2.0 0.3B (torch backend, CPU FP32 +# oracle setting). The upstream package ships no requirements file; its +# vela2_inference.py header names numpy + tokenizers, and the torch backend +# adds torch + transformers + safetensors. Versions are the ones the worker +# was validated with (Python 3.12); torch CPU wheels are enough — the GPU +# bf16-autocast deployment setting is not part of this slice's evidence. +torch==2.11.0 +transformers==5.19.0 +safetensors==0.8.0 +tokenizers==0.23.2 +numpy==2.5.3 +# Downloads of the pinned revision (hf download). +huggingface_hub==1.33.0 +# The reference comparison additionally needs the upstream server stack: +# pip install fastapi uvicorn httpx diff --git a/recipe/vela/test_compare_reference.py b/recipe/vela/test_compare_reference.py new file mode 100644 index 00000000..d97aaa45 --- /dev/null +++ b/recipe/vela/test_compare_reference.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +"""Regression tests for recipe/vela/compare_reference.py. + +Checks the pure comparison, case-loading and export helpers without the +pinned package, weights, torch or fastapi: exact-match verdicts +including key order, rejection agreement, the default fixture set's +shape (the invalid-span 422 and over-limit 413 synthetic cases), the +oracle export gate, and the oracle export structure. + + python recipe/vela/test_compare_reference.py +""" + +from __future__ import annotations + +import json +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +import compare_reference as tool + + +def response(answers=None, input_tokens=17): + value = { + "model": tool.MODEL_NAME, + "answers": answers if answers is not None else {"q": {"type": "noul", "noul": 0.75}}, + "usage": {"input_tokens": input_tokens, "output_tokens": 0}, + } + if answers is None: + value["thresholds"] = {"q": 0.3} + return value + + +def main() -> None: + results: list[tuple[str, bool, str]] = [] + + def check(name: str, condition: bool, detail: str = "") -> None: + results.append((name, condition, detail)) + + # 1. Matching 200 responses pass; every divergence fails. + reference = response() + lines, ok = tool.compare_case("fixture", (200, response()), (200, reference)) + report = "\n".join(lines) + check( + "exact 200 match passes", + ok and "exact match (status, body, key order): PASS" in report, + report, + ) + + lines, ok = tool.compare_case("fixture", (200, response(input_tokens=18)), (200, reference)) + check("token divergence fails", not ok, "\n".join(lines)) + + reordered = {"q": {"type": "noul", "noul": 0.75}} + other_first = {"other": {"type": "noul", "noul": 0.5}, "q": {"type": "noul", "noul": 0.75}} + lines, ok = tool.compare_case("fixture", (200, response(other_first)), (200, response(reordered))) + check("answer-set divergence fails", not ok, "\n".join(lines)) + + drifted = {"q": {"type": "noul", "noul": 0.7500000001}} + lines, ok = tool.compare_case("fixture", (200, response(drifted)), (200, reference)) + check("numeric drift fails (exact-match criterion)", not ok, "\n".join(lines)) + + same_answers = {"q": {"type": "noul", "noul": 0.75}, "other": {"type": "noul", "noul": 0.5}} + reordered_answers = {"other": {"type": "noul", "noul": 0.5}, "q": {"type": "noul", "noul": 0.75}} + lines, ok = tool.compare_case( + "fixture", (200, response(same_answers)), (200, response(reordered_answers)) + ) + check("key-order divergence fails (order is part of the match)", not ok, "\n".join(lines)) + + # 2. Rejections agree only on both status and body. + rejection = {"detail": [{"loc": ["body", "questions", "bad", "criteria"], + "msg": "span criteria must be an object with 1 to 255 entries", + "type": "value_error"}]} + lines, ok = tool.compare_case("fixture", (422, rejection), (422, rejection)) + check("matching 422 rejections pass", ok, "\n".join(lines)) + + lines, ok = tool.compare_case("fixture", (413, rejection), (422, rejection)) + check("status mismatch fails", not ok, "\n".join(lines)) + + other = {"detail": [{"loc": ["body", "questions"], "msg": "different", "type": "value_error"}]} + lines, ok = tool.compare_case("fixture", (422, other), (422, rejection)) + check("body mismatch fails", not ok, "\n".join(lines)) + + lines, ok = tool.compare_case("fixture", (200, response()), (422, rejection)) + check("worker answering a rejected request fails", not ok, "\n".join(lines)) + + # 3. The default case set carries the charter's fixture list shape. + cases = tool.default_cases() + names = [name for name, _ in cases] + check( + "default cases are the four examples plus five rejections", + names == ["example-request", "example-request-set-threshold", "example-request-cjk", + "example-request-score", + "invalid-span-422", "over-limit-413", + "missing-model-422", "extra-field-422", "wrong-model-422"], + str(names), + ) + score = json.loads( + (tool.HERE / "example-request-score.json").read_text(encoding="utf-8") + ) + check( + "the score example carries score questions with level criteria", + score["questions"] + and all(question["type"] == "score" for question in score["questions"].values()) + and all(isinstance(question["criteria"], list) for question in score["questions"].values()), + json.dumps(score)[:160], + ) + missing = dict(cases[6][1]) + check( + "the missing-model case is an envelope without the model field", + set(missing) == {"state", "questions"}, + json.dumps(missing)[:120], + ) + extra = dict(cases[7][1]) + check( + "the extra-field case carries a field the preamble forbids", + "images" in extra and set(extra) == {"model", "state", "questions", "images"}, + json.dumps(extra)[:120], + ) + wrong = dict(cases[8][1]) + check( + "the wrong-model case names the 0.8B member", + wrong["model"] == "vllm-sr/Vela-2.0-0.8B", + wrong["model"], + ) + invalid = dict(cases[4][1]) + check( + "the invalid-span case is a span question with list criteria", + invalid["questions"]["bad"]["type"] == "span" + and isinstance(invalid["questions"]["bad"]["criteria"], list), + json.dumps(invalid)[:200], + ) + over = dict(cases[5][1]) + check( + "the over-limit case is questions+schema overflow by construction", + len(over["questions"]["big"]["criteria"]) == 255, + json.dumps(over)[:120], + ) + + # 4. The oracle export freezes the full native-comparison surface. + body = {"model": tool.MODEL_NAME, "state": "s", + "questions": {"q": {"type": "noul", "instructions": "x"}}} + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "oracle.json" + oracle = tool.export_oracle( + "case", body, response(), {"answers": [{"id": "q", "type": "choice"}]}, path + ) + reread = json.loads(path.read_text(encoding="utf-8")) + check( + "oracle export round-trips", + oracle == reread + and oracle["request"] == body + and oracle["system_one"]["answers"]["q"]["noul"] == 0.75 + and oracle["predict"]["answers"][0]["id"] == "q", + json.dumps(oracle)[:200], + ) + + check("export allowed for a passing 200 case", + tool.export_allowed(True, (200, response()))) + check("export refused for a failed comparison", + not tool.export_allowed(False, (200, response()))) + check("export refused for an agreed rejection", + not tool.export_allowed(True, (422, rejection))) + + failed = 0 + for name, passed, detail in results: + print(f" {'ok ' if passed else 'FAIL'} {name}") + if not passed: + failed += 1 + print(" " + detail.strip().replace("\n", "\n ")[:400]) + print( + "compare_reference tests: ok" + if not failed + else f"compare_reference tests: {failed} FAILED" + ) + sys.exit(1 if failed else 0) + + +if __name__ == "__main__": + main() diff --git a/src/frontend/vela.py b/src/frontend/vela.py new file mode 100644 index 00000000..c863a018 --- /dev/null +++ b/src/frontend/vela.py @@ -0,0 +1,230 @@ +"""Python reference worker for Vela 2.0 0.3B. + +The Rust frontend remains the public transport layer. This worker owns +only HTTP lifecycle and request orchestration; wire admission lives in +models.vela.protocol, the pinned-runtime error mapping in +models.vela.preprocess, the pinned runtime and its package gate in +models.vela.engine, and response finishing in models.vela.postprocess. + +The error surface has two layers: transport errors follow the shared +Python-worker discipline, and every request rejection the pinned runtime +makes — an invalid field, an invalid question, a questions-overflow — is +the upstream whole-request 422 ``{"detail": [{"loc", "msg", "type"}]}`` +(or 413 ``too_long``), passed through verbatim. There are no per-question +errors. Over-limit state text is truncated exactly as the pinned +reference truncates it (see the model contract); only questions plus +schema that cannot fit the 8,192-token window are refused. +""" + +from __future__ import annotations + +import argparse +import hmac +import json +import logging +import os +import socket +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any + +from models.vela.engine import Vela2Executor +from models.vela.postprocess import finish_response +from models.vela.preprocess import map_runtime_error, to_request_error +from models.vela.protocol import MODEL_NAME, MAX_BODY, RequestError, parse_body, admit_request + +LOG = logging.getLogger(__name__) + +# Upstream vela2_serve.py /v1/models entry, kept verbatim so clients see +# the reference server's description and release date. +MODELS_ENTRY = { + "name": None, # the served model name, filled per request + "release_date": "2026-09-29", + "description": ( + "Vela 2.0 0.3B: SystemOne Choice / Noul / Score " + "with span and set extensions, 8,192-token context" + ), +} +UNAUTHORIZED_BODY = {"detail": "Missing or invalid API key"} + + +def decide(raw: bytes, executor: Vela2Executor) -> dict[str, Any]: + """Admit, execute, and finish one complete request.""" + + body = admit_request(parse_body(raw), executor.served) + questions = body["questions"] + try: + response = executor.system_one(body) + except executor.SystemOneError as exc: + raise to_request_error(exc) from exc + except ValueError as exc: + # The pinned runtime is the validation authority for what the wire + # layer cannot foresee (state rendering, question semantics, the + # window budget); its rejections are client errors, not failures. + raise map_runtime_error(exc) from exc + return finish_response(questions, response) + + +class WorkerServer(ThreadingHTTPServer): + daemon_threads = False + + def __init__(self, address, executor: Vela2Executor, api_key: str | None = None): + self.executor = executor + self.api_key = api_key + self.inference_lock = threading.Lock() + self._executor_closed = False + super().__init__(address, Handler) + + def server_close(self): + super().server_close() + if not self._executor_closed: + self.executor.close() + self._executor_closed = True + + +class Handler(BaseHTTPRequestHandler): + def setup(self): + super().setup() + self.connection.settimeout(15) + + def log_message(self, format, *args): + # Do not log paths, prompts, or arbitrary request headers. + pass + + def send_json(self, status: int, value: dict): + raw = json.dumps(value, ensure_ascii=False, allow_nan=False).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + try: + self.wfile.write(raw) + except (BrokenPipeError, ConnectionResetError): + pass + + def _send_request_error(self, exc): + if exc.body is not None: + self.send_json(exc.status, exc.body) + else: + self.send_json(exc.status, {"detail": str(exc)}) + + def _unauthorized(self) -> bool: + """Upstream vela2_serve.py: no key configured means no gate; with + VELA2_API_KEY set, exactly ``Authorization: Bearer `` passes.""" + + key = self.server.api_key + if not key: + return False + got = self.headers.get("authorization", "") + try: + authorized = hmac.compare_digest(got, f"Bearer {key}") + except TypeError: + # Header bytes outside ASCII (a raw 0xff, say) decode to + # non-ASCII text, which compare_digest refuses to compare; a + # value that cannot be compared cannot equal the key either. + authorized = False + if authorized: + return False + self.send_json(401, UNAUTHORIZED_BODY) + return True + + def do_GET(self): + path = self.path + if path == "/health": + # Upstream verbatim; the worker binds the port only after + # loading and warmup, so answering at all means ready. + self.send_json(200, {"status": "ok"}) + elif path == "/ready": + self.send_json(200, {"ready": True}) + elif path == "/v1/models": + if self._unauthorized(): + return + entry = dict(MODELS_ENTRY) + entry["name"] = self.server.executor.model_name + self.send_json(200, {"models": [entry]}) + else: + self.send_json(404, {"detail": "unknown route"}) + + def do_POST(self): + if self.path != "/v1/systemone": + self.send_json(404, {"detail": "unknown route"}) + return + if self.headers.get("Transfer-Encoding"): + self.send_json(411, {"detail": "Content-Length is required; chunked requests are unsupported"}) + return + lengths = self.headers.get_all("Content-Length", []) + if len(lengths) != 1: + self.send_json(411, {"detail": "one Content-Length is required"}) + return + try: + length = int(lengths[0]) + except ValueError: + self.send_json(400, {"detail": "invalid Content-Length"}) + return + if length < 0 or length > MAX_BODY: + self.send_json(413, {"detail": "request exceeds the 8 MiB body limit"}) + return + if self.headers.get_content_type() != "application/json": + self.send_json(415, {"detail": "Content-Type must be application/json"}) + return + if self._unauthorized(): + return + if not self.server.inference_lock.acquire(blocking=False): + self.send_json(503, {"detail": "worker busy"}) + return + try: + raw = self.rfile.read(length) + if len(raw) != length: + self.send_json(400, {"detail": "incomplete body"}) + return + self.send_json(200, decide(raw, self.server.executor)) + except RequestError as exc: + self._send_request_error(exc) + except (TimeoutError, socket.timeout): + self.send_json(408, {"detail": "request body timed out"}) + except Exception as exc: + LOG.error("vela inference failed: %s", type(exc).__name__) + self.send_json(500, {"detail": "inference failed"}) + finally: + self.server.inference_lock.release() + + +def parse_args(argv=None): + parser = argparse.ArgumentParser(description="Python reference worker for Vela 2.0 0.3B") + parser.add_argument("--model", required=True, help="pinned Vela 2.0 0.3B package directory") + parser.add_argument("--device", default="cpu", help="torch device (reference contract: cpu)") + parser.add_argument("--batch-size", type=int, default=None) + parser.add_argument("--name", default=os.environ.get("VELA2_MODEL_NAME", MODEL_NAME), + help="model name accepted in the request's model field") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=8000) + return parser.parse_args(argv) + + +def main(argv=None): + args = parse_args(argv) + logging.basicConfig(level=logging.INFO) + executor = Vela2Executor( + args.model, + device=args.device, + model_name=args.name, + batch_size=args.batch_size, + ) + server = None + try: + executor.warmup() + server = WorkerServer((args.host, args.port), executor, + api_key=os.environ.get("VELA2_API_KEY") or None) + LOG.info("%s worker ready on %s:%s", executor.model_name, args.host, args.port) + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + if server is not None: + server.server_close() + else: + executor.close() + + +if __name__ == "__main__": + main() diff --git a/src/models/vela/README.md b/src/models/vela/README.md new file mode 100644 index 00000000..8c6c65f6 --- /dev/null +++ b/src/models/vela/README.md @@ -0,0 +1,199 @@ +# Vela 2.0 0.3B reference contract + +This directory contains the System1-Omni integration slice for +[Vela 2.0 0.3B](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) +(`vela2-unified`). It is a reference-serving boundary only: a Python +worker that runs the checkpoint's own pinned runtime for text state and +all five typed question kinds (`choice`, `noul`, `score`, `span`, `set`) +on `/v1/systemone`. Native execution is not part of this slice; per the +frozen RFC this worker is the Phase-1 entry that freezes the +text/token_ids input, the position-level final-hidden readout contract, +the error/length boundaries and the parity baseline for later +ModernBERT-profile work. + +The shared Vela 2.0 wire module is deliberately shaped for the family: +the envelope admission (`protocol`), the pinned-runtime error mapping +(`preprocess`) and response finishing (`postprocess`) are member-neutral; +this slice ships the 0.3B pinned executor (`engine`). The hybrid members +(0.8B/4B/9B) reuse the same wire layer with their own executors. + +## Pinned artifacts + +| Artifact | Value | +| --- | --- | +| Package | [vllm-sr/Vela-2.0-0.3B](https://huggingface.co/vllm-sr/Vela-2.0-0.3B) | +| Pinned revision | `d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409` | +| Identity | `vela2-unified` / `Vela2Model`, 309,114,369 parameters, all FP32 (1,236,473,252-byte safetensors) | +| Runtime | the checkpoint's own flat files: `vela2_inference.py`, `modeling_vela2.py`, `configuration_vela2.py`, `vela2_serve.py` | +| Calibration | the package's own `calibration.json` (never cross-copied between members) | +| License | Apache-2.0; the inherited tokenizer material carries Gemma Terms of Use | +| Runtime dependencies | [recipe/vela/requirements.txt](../../recipe/vela/requirements.txt) (torch, transformers, safetensors, tokenizers, numpy) | + +The package ships no manifest, so the executor's startup gate checks the +config's structural identity (model type, architecture, `max_length` +8,192, the 8 marker ids), then hashes every small file the runtime reads +or executes — the four runtime modules, `config.json`, +`calibration.json` and `tokenizer.json` — against the table checked in +for the pinned revision, before anything is imported, and refuses +importable entries the table does not name plus any existing +`__pycache__` bytecode cache or directory symlink (`model.safetensors` is +checked by exact size; the runtime's strict safetensors load rejects +truncated or structurally mismatched weights, but not same-size +bit-level corruption). +A later self-consistent snapshot rewrites its own files and fails the +checked-in pin. + +## Current serving scope + +- The request envelope is upstream's three-field form, exactly: + `{"model", "state", "questions"}`. Any other key is a whole-request + 422 `extra_forbidden`; a missing key is a 422 `missing`; `model` must + be non-blank and equal the served name (`vllm-sr/Vela-2.0-0.3B`, or + `VELA2_MODEL_NAME`/`--name`), otherwise 422 `value_error.model`. +- `state` is a plain string or a JSON object/array. An object is mapped + onto typed parts (`user`/`context`/`answer`) by the pinned runtime's + key roles (request/user/prompt → user; source/context/document/tools/ + previous_answer → context; answer/response → answer; unknown keys → + context); non-string values render as canonical JSON. A string holding + a JSON object is read as that object. +- `questions` is a non-empty object of question ids to questions. All + five kinds are served forward-untouched; per-question limits (choice + 2–255 options, score 2–10 distinct levels, span/set 1–255 criteria) + and `over` resolution are the pinned runtime's validation, verbatim. +- `over` names a state key, a part name, or a list of them; default is + every part (span: the answer part if present, else user, else the + first part). `threshold` is allowed only on span/set and must be a + number in [0, 1]. Set-label Noul views (`".