Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,21 @@ jobs:
python -m pip install pytest
PYTHONPATH=src python -m pytest tests/clef -q

kev-export:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: CPU-only export dependencies
run: |
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -r recipe/kev/requirements.txt
- name: Export contract tests (no weights or GPU)
run: python -m unittest discover -s tests/kev -p 'test_*.py' -v

cuda-contract:
# The CUDA backend contract and its checker. No CUDA toolkit and no GPU: both
# read files as text.
Expand Down
9 changes: 9 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[workspace]
members = ["src/frontend", "src/runtime", "src/models/clm", "src/models/cua_s1/native", "src/models/qwen3_5/native", "src/models/open_jev/native", "src/models/jev_vl/native", "src/models/omnijev/native", "src/models/laya", "src/backends/cuda", "src/models/decider/native", "src/models/jemm/native", "src/models/startlux/native", "src/models/clef/native"]
members = ["src/frontend", "src/runtime", "src/models/clm", "src/models/cua_s1/native", "src/models/qwen3_5/native", "src/models/open_jev/native", "src/models/jev_vl/native", "src/models/omnijev/native", "src/models/laya", "src/backends/cuda", "src/models/decider/native", "src/models/jemm/native", "src/models/startlux/native", "src/models/clef/native", "src/models/kev/native"]
resolver = "3"
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ checkpoint. Public231 concurrency-1 latency for heads `1ba07d1`, `f968e576`,
| CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) |
| Kev-4B | [Pinned CPU export and Rust pointer head](recipe/kev/README.md); no worker or full-model parity yet ([#27](https://github.com/InferMatrixLab/system1-omni/issues/27)) |
| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |
| Vela 2.0 0.3B | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; all five kinds (`choice`/`noul`/`score`/`span`/`set`) for text state; CPU FP32 validated (author-run, evidence in the integration PR) |
| Vela 2.0 0.8B hybrid | [Python reference worker](recipe/vela/reference.md) on the checkpoint's pinned runtime; same surface plus `span_heads`; CPU FP32 parity recorded in the integration PR |
Expand Down
4 changes: 4 additions & 0 deletions docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Models that are being added are also tracked in issues labeled [new model](https
| Clef (27B) | Same `omni-clef` server and text contract as Clef-flash, selected with `--variant clef` | Contract tests only; no inference | Not run | Not supported | Same stack; pinned `Cloudflare/clef` snapshot (~55.0 GB). Weights and joint-head config differ from flash |
| pplx-decider-v1.1-27b | [Python reference worker](../recipe/pplx_decider/README.md) on the checkpoint's own runtime. Text and up to four images; `choice`/`noul`/`score`. Native CUDA executor is not this worker. Lab L20X-labeled GPUs are the H200 target. The causal-conv1d build keeps sm_90 and sm_89. Capability 8.9 is ready on FLA chunk. Capability 9.0 uses all-sequence FlashInfer. The kept primary rotates 48 `cu_seqlens` buffers per device and length, passes `use_cp` false, skips the q/k repeat, and packs q and k before the L2-norm view. `801ef7b` fused the 48 linear gated RMSNorms and missed the median (p50 89.172 ms, p95 274.790 ms), so that kernel stays off. `2bbbe4e` replaced `silu(gate) * up` with FLA `swiglu_fwd` and the owner closeout rejected it (p50 97.613 ms, p95 292.795 ms), so that kernel stays off. Gate and up GEMMs stay separate. Residual RMSNorm and the gated output norm stay eager. An unknown `PPLX_GDN_PREFILL` fails closed on every device, including capability 8.9. `3ecdf5a` crashed in warmup (0/231). Always-on fusion `3afb472` and the 512-token gate `705ae089` missed the median. The 48-slot FlashInfer state ring `17a7b120` missed the gate. `PPLX_GDN_PREFILL=fla` reproduces the prior `f968e576` chunk. `PPLX_GDN_PREFILL=short` is the rejected ≤256 dispatch. SDPA stays on the checkpoint order. The fused gate-up GEMM is refused. The forward stays eager. `PPLX_COMPILE=dynamic` and `PPLX_COMPILE=dynamic-tail` are rejected opt-ins | Contract checks only; no inference | Reported load on 1× lab L20X (sm_89, ~140 GiB) at head `1ba07d1`: BF16 resident ~50333 MiB, no OOM. Historical primary: 231/231, p50 98.777 ms, p95 341.202 ms. Prior primary is the `f968e576` remeasure on the lab L20X-labeled GPU, `compute_capability=9.0`, `fla-chunk`, `flashinfer` null: 231/231, p50 94.077 ms, p95 300.200 ms (p50 −4.76%, p95 −12.02%). Prior primary `ffae182`: 231/231, p50 92.844 ms, p95 296.458 ms (p50 −1.31%, p95 −1.25%). Prior primary `4b99b73`: 231/231, p50 88.942 ms, p95 295.797 ms (p50 −4.20%, p95 −0.22%), 48-slot `cu_seqlens`, quality `choice_acc=0.9065` (139), `noul_acc=0.8784` (74), `score_mae=0.2035` (18). Kept primary is `fcf5e2b`: 231/231, p50 87.782 ms, p95 291.183 ms (p50 −1.30%, p95 −1.56%), grouped-value FlashInfer with q and k packed, same quality digits. Uncached all-sequence FlashInfer `3c72f02`: p50 88.808 ms, p95 300.751 ms, gate failed. Short-sequence FlashInfer `a50b9aa`: p50 95.446 ms, p95 300.871 ms, gate failed. cuDNN-first SDPA `0dcaa0ac`: p50 138.501 ms, p95 332.859 ms, gate failed. Fused gate-up `8b4ee326`: p50 102.437 ms, p95 304.970 ms, gate failed. Always-on dynamic compile `e36b32ed`: p50 97.751 ms, p95 263.124 ms, gate failed (p95 improved). 512-token compile `7e3b1be`: p50 100.037 ms, p95 257.084 ms, gate failed. Isolated-median compile `34ba5de`: p50 97.084 ms, p95 256.679 ms, gate failed (median probe was eager). FlashInfer state-workspace ring `17a7b120`: p50 89.165 ms, p95 295.723 ms, gate failed. Always-on FLA residual RMSNorm `3afb472`: p50 93.935 ms, p95 274.902 ms, gate failed. Length-gated FLA residual RMSNorm `705ae089`: p50 94.604 ms, p95 274.835 ms, gate failed. `use_cp=auto` `a6f4d9a`: p50 112.294 ms, p95 299.419 ms, gate failed. Grouped-value `3ecdf5a` crashed in warmup (0/231, no p50 or p95). The fused linear gated-norm candidate has no sample. Does not enter the H800 board | Not supported | Python 3.12, pinned checkpoint revision `14c9d90137962d300f3de95633f7b6535e80c5ce`, checkpoint `requirements.txt`, `causal-conv1d` 1.7.0, `flash-linear-attention` 0.5.2, optional `flashinfer-python` 0.6.16.post3. Shard sizes in the release manifest sum with the readout to about 48.6 GiB before activations |

Kev-4B has [CPU checkpoint preparation and a Rust pointer head](../recipe/kev/README.md)
only. No Kev worker, backbone parity, GPU serving or full-model accuracy is
established by that preparation slice; native integration remains [#27](https://github.com/InferMatrixLab/system1-omni/issues/27).

- **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
- **Author-run:** recorded by the recipe author on the stated hardware and not independently reproduced; the row names its owner.
- **Unverified:** the worker accepts this device, but no recipe or merged pull request covers it.
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ nav:
- JEV-27B-VL experimental worker: recipe/jev_vl/README.md
- JEV-27B-VL validation status: recipe/jev_vl/validation.md
- OmniJev preparation: recipe/omnijev/README.md
- Kev-4B preparation: recipe/kev/README.md
- OmniJev native worker: recipe/omnijev/native.md
- CLM stub-encoder contract: recipe/clm/README.md
- LFM2.5-350M choice worker: recipe/lfm2/README.md
Expand Down
2 changes: 2 additions & 0 deletions recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
- [OmniJev-4B v1.1 preparation](omnijev/README.md): export the checkpoint and
regenerate the CPU fixtures; [native worker](omnijev/native.md): serve it on CUDA
and compare it with the reference.
- [Kev-4B preparation](kev/README.md): export all LoRA projections and validate
the Rust FP32 pointer head on CPU; worker integration remains pending.
- [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
CPU with a stub encoder, and what the response comparison has to allow for.
- [pplx-decider reference worker](pplx_decider/README.md): pin
Expand Down
77 changes: 77 additions & 0 deletions recipe/kev/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Kev-4B: CPU export and pointer-head validation

This is a preparation recipe for [#27](https://github.com/InferMatrixLab/system1-omni/issues/27),
not a runnable Kev server. The [model contract](../../src/models/kev/README.md)
records the pinned artifacts, numerical boundaries and missing worker pieces.
All commands below run from the repository root. No GPU is needed.

## Environment and downloads

Use Python 3.12 and the stable Rust toolchain used by this repository. The CPU
export reads about 9.5 GB of inputs and writes about 8.5 GB; reserve at least 25 GB
of free disk and 24 GB of host RAM. It processes one source shard at a time.

```sh
python3.12 -m venv .venv-kev
.venv-kev/bin/pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
.venv-kev/bin/pip install -r recipe/kev/requirements.txt huggingface-hub==1.33.0

.venv-kev/bin/hf download Qwen/Qwen3.5-4B-Base \
--revision 1001bb4d826a52d1f399e183466143f4da7b741b \
--include 'config.json' 'tokenizer.json' 'tokenizer_config.json' \
'model.safetensors.index.json' '*.safetensors' --local-dir weights/kev-base
.venv-kev/bin/hf download jaredpalmer/kev-4b \
--revision 139fdd94f1b6a6ad80cc15e08fcb99cac885a101 \
--include 'adapter_config.json' 'adapter_model.safetensors' 'head.pt' \
--local-dir weights/kev-adapter

.venv-kev/bin/python recipe/kev/export.py \
--base weights/kev-base --checkpoint weights/kev-adapter --out weights/kev-native
```

Every required input must match its pinned SHA-256. The output directory must not
already exist. `kev_export.json` lists merged targets, output hashes, FP32 head
dimensions and calibration, plus the base tokenizer's five delimiter IDs.
Do not replace these artifacts with another Kev size or a bare Qwen checkpoint.

## CPU checks

```sh
.venv-kev/bin/python -m unittest discover -s tests/kev -p 'test_*.py' -v
cargo test --locked -p omni-kev-native -- --list
cargo test --locked -p omni-kev-native
KEV_HEAD="$PWD/weights/kev-native/head.safetensors" \
cargo test --release --locked -p omni-kev-native --test head \
-- --ignored --nocapture
```

The final test is opt-in because it needs the released head, but it is still
CPU-only. It compares the Rust implementation against the committed small oracle
for 1/2/7/31/255 options. Default CI executes the ordinary head tests without
downloading weights. Full backbone inference is not part of these checks.

To regenerate the oracle using the official class rather than this implementation:

```sh
git clone https://github.com/jaredpalmer/kev.git /tmp/kev-reference
git -C /tmp/kev-reference checkout --detach 2074cc44256200bb37540103fa4572901faa10f3
.venv-kev/bin/python tests/kev/generate_head_fixture.py \
--reference-model /tmp/kev-reference/kev/model.py \
--head weights/kev-adapter/head.pt --out /tmp/kev-head-reference.json
diff -u tests/kev/data/head.json /tmp/kev-head-reference.json
```

The generator verifies the complete source and head hashes, extracts only the
upstream `PointerHead` class, and uses CPU PyTorch. It does not need Transformers,
PEFT or the backbone. The fixture's Torch version includes the wheel's local
suffix, so a different wheel can change provenance even when numbers agree.

## Next integration slice

There is no worker binary, frontend launch command, health endpoint or example
HTTP request yet. The next slice adds exact prompt/row preparation and typed
answers, verified export loading, shared Qwen execution and serial admission.
It must qualify the BF16 merge against the full reference, then test real warmup
and first inference directly and through the frontend. Record hardware, full
dependency pins and numerical gates before those runs. No latency or accuracy
claim follows from export success or isolated head agreement.
Loading
Loading