diff --git a/.agents/skills/new-model-integration/SKILL.md b/.agents/skills/new-model-integration/SKILL.md new file mode 100644 index 00000000..6a9e839a --- /dev/null +++ b/.agents/skills/new-model-integration/SKILL.md @@ -0,0 +1,100 @@ +--- +name: new-model-integration +description: Bring a new open model into System1-Omni on its own GitHub issue and draft pull request, then a recipe and operator checklist, a Public231 C1 measurement, and continuous optimization on that same branch. Use when adding model support. Keep serving-chain work on separate issues from the new-model RFC. +--- + +# Integrate a new open model + +Follow this order for a model that is not already supported. Implementation +placement, contracts, tests, and docs stay in +[add-new-model](../add-new-model/SKILL.md). This skill is the sequence around +that work: issue, draft PR, recipe, Public231 C1, then measured optimization. +Complete the [contributor-intent gate](../self-review/SKILL.md#human-contributor-intent-gate) +before editing and before opening the draft. + +Do not merge, and do not mark the PR ready, without a separate explicit owner +yes. A green check or a C1 number is not that yes. + +## 1. Open an independent issue + +Open a new GitHub issue for this model. Do not add the model to another +model's issue or to a serving-chain issue. + +Use the `new model` label. Title it `[New Model]:` for an implementation +request, or `[RFC]:` when the issue is still a design proposal, matching +issues such as #136, #114, and #121. Say which checkpoint, modalities, and +question types are in scope. Link the rolling tracker +[#83](https://github.com/InferMatrixLab/system1-omni/issues/83) and the +roadmap [#125](https://github.com/InferMatrixLab/system1-omni/issues/125); +do not use either tracker as a substitute for the model's own issue. + +Serving-chain work stays on its own issues. The roadmap separates model +milestones from shared serving features, and it says reference serving, native +execution, and optimized execution are separate milestones. Shared +`/v1/systemone` contract work is #61, observation reuse is #85, and +admission, batching, and scheduling stay with #150 and the feature items on +#125. Link those issues when the model depends on them. Do not implement that +shared work inside the new-model RFC or its PR. A cross-request batching or +scheduler change needs its own follow-up issue. + +## 2. Open a draft pull request + +Implement on a branch whose PR targets the model's issue and stays **draft**. +[CONTRIBUTING](../../../CONTRIBUTING.md#self-review-before-requesting-review) +keeps draft PRs in draft until self-review is complete. Use +[precheck-pr](../precheck-pr/SKILL.md) before asking to mark it ready, and +wait for the owner's separate yes before ready or merge. + +Keep the diff to this model's processors, executor, recipe, tests, and the +docs `add-new-model` lists. Reuse a shared backend only when the model +actually needs it. A new model is not a reason to redesign the frontend, +scheduler, or batching policy. + +## 3. Recipe and operator checklist + +Land the operator-facing recipe in the same draft, under `recipe//`, +before calling the integration runnable. `add-new-model` lists the required +pages (model README, `recipe/README.md`, root README supported-model status, +`mkdocs.yml`, and `docs/architecture.md` when a shared boundary changes). + +The recipe must let an operator do all of the following without private +host notes: + +- pin the checkpoint, tokenizer, adapters, and dependency versions +- build and launch the worker and, when used, the frontend +- check health and send one real example request +- run the model's validation commands and read the stated limits +- see which hardware was actually tested, named as that hardware + +GPU claims name the measured `device` and `compute capability`. When the +recipe is documenting the production target, the preferred label is +**H200 / SM90**. If the host reported a different nvidia-smi string, the +recipe records that string and the measured capability, and does not rewrite +them. Do not describe untested hardware as validated. Do not put hostnames, +SSH targets, usernames, or reservation-tool names in the recipe. + +## 4. Public231 C1 + +When the draft serves the model, measure it with +[public231-c1](../public231-c1/SKILL.md). Every paste records the draft SHA, +`device`, `compute capability`, the serving pass rate, and pooled p50/p95. +The documented production target is H200 / SM90. The protocol does not treat +a different measured identity as INCONCLUSIVE; paired runs still have to share +the recorded identity. A recipe that does not yet serve stays explicit about +that: do not invent a C1 result. CPU contract tests are not a latency baseline. + +## 5. Continuous optimization on the same branch + +Further latency work stays on this model's draft branch. Run it under +[optimize-remeasure-gate](../optimize-remeasure-gate/SKILL.md): paste the C1 +p50/p95, `device`, and `compute capability` first, and do not stack the next +tip commit until that tip is remeasured with the same protocol on the same +named hardware. Incomplete evidence is INCONCLUSIVE. Quality and contract +checks stay green. Each tip states its RSI phase (bring-up/scheduling, +parallelism/comms, kernel, or launch/post) and which alternatives were +considered versus the primary; without that it is not merge-clean. Feed each +delta back to the agent before the next edit. + +Optimized execution remains a separate milestone from reference serving and +from native bring-up. Promote a number into the supported-model matrix only +when the recipe and the measured SHA agree with it. diff --git a/.agents/skills/optimize-remeasure-gate/SKILL.md b/.agents/skills/optimize-remeasure-gate/SKILL.md new file mode 100644 index 00000000..6dd861f8 --- /dev/null +++ b/.agents/skills/optimize-remeasure-gate/SKILL.md @@ -0,0 +1,135 @@ +--- +name: optimize-remeasure-gate +description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land until that tip is remeasured with the same protocol on the same recorded device and compute capability. Every paste includes those two fields. The documented production target is H200 / SM90. Each tip names its RSI phase and the primary change versus alternatives considered, or it is not merge-clean. Incomplete evidence is INCONCLUSIVE. The pull request stays draft until a separate owner yes. +--- + +# Optimize only against a remeasured tip + +Use this after a Public231 C1 baseline measured with +[public231-c1](../public231-c1/SKILL.md). It is the loop for later latency +work on that deployment, including a Cloud Agent or other coding agent that +would otherwise keep committing. Read +[the benchmark self-review rules](../../../benchmarks/README.md#self-review-and-ab-evidence) +before changing inference or serving. + +The gate controls the next commit. It does not authorize a merge. + +## Paste the baseline before editing + +Put the measured primary numbers in the task before the first optimization +edit, and again before every later tip. Every paste in this loop includes +`device` and `compute capability`: the baseline paste, the tip remeasure, +and the delta fed back. A paste that omits either field is incomplete. + +- baseline full git SHA (`runtime_revision`) +- pooled p50 and p95 in milliseconds +- serving pass rate (`successful_requests / planned_requests`) +- clock basis `post_to_body_json_and_schema_validation` +- `device` (the nvidia-smi name string; runner metadata `gpu`) +- `compute capability` + +The documented production target is H200 / SM90. A paste of that machine +reads `device: H200` and `compute capability: SM90`. If the host prints a +different name, paste that string and the capability that was measured. +Those are the primary figures. Per-round values stay attached so a pooled +number can be checked. A recalled or rounded figure, a single round, or a +percentile averaged across rounds is not a baseline. + +## One tip, then remeasure + +A tip is one optimization commit on the draft branch. Do not add another +commit on top of it until that tip has its own Public231 C1 remeasure. The +gate requires the same measurement protocol and the same named hardware as +the baseline. It does not require the SKU to be H200, L20X, SM90, or sm_89. + +Same measurement protocol means warmup 5, concurrency 1, three measured +rounds, the same manifest, and the same clock +(`post_to_body_json_and_schema_validation`). Same named hardware means the +same recorded `device` string and the same recorded `compute capability`. +The remeasure SHA must be the tip SHA. A measurement of an older commit does +not clear the tip. + +Keep controls matched to the baseline (manifest, precision, request order, +cache policy, thinking treatment, server shape). State any necessary +difference. Do not reset the comparison by changing the clock, the quantile +method, or the named hardware. + +## INCONCLUSIVE stops the loop + +Mark the tip **INCONCLUSIVE** and do not stack the next commit when any of +these are true: + +- the baseline p50/p95 were not recorded before the edit +- fewer than three measured rounds finished +- `evidence_complete` is not true, or warmup failed +- any measured request failed, was incomplete, or was unattempted +- `device` or `compute capability` is missing on the baseline or the tip +- the tip's `device` string or `compute capability` differs from the + baseline, so the two sides are not the same named hardware +- the tip does not use the same measurement protocol as the baseline +- the clock basis is missing or is not + `post_to_body_json_and_schema_validation` +- the measured SHA is not the tip SHA +- pooled p50/p95 were averaged from round percentiles instead of recomputed + from the pooled samples + +INCONCLUSIVE is not a win and not a regression. Do not drop failures, relax a +tolerance, or add undeclared rounds to produce a preferred delta. A recorded +`device` other than H200, or a `compute capability` other than SM90, is not +by itself INCONCLUSIVE. + +## Quality and contract stay green + +A latency delta is acceptable only when the tip's quality and contract checks +are green: + +- the [validation for the files the tip touched](../../../CONTRIBUTING.md#choose-validation-by-changed-area) +- the model request/response contract tests that cover the changed path +- `python benchmarks/bench.py compare` against the baseline outputs when the + change can affect decisions or probabilities, with tolerances declared + before the run + +A faster tip that fails a contract test, flips decisions beyond the declared +tolerance, or lowers the serving pass rate is not an accepted tip. Do not +stack past it. + +## Feed the delta back + +Before the agent proposes the next change, give it this record: + +- baseline SHA, pooled p50, pooled p95, pass rate +- tip SHA, pooled p50, pooled p95, pass rate +- p50 and p95 delta (tip minus baseline), in milliseconds +- clock basis, `device`, and `compute capability` for both sides +- RSI phase, alternatives considered, and which change is primary +- verdict: improved, regressed, or INCONCLUSIVE +- quality/contract result + +The agent uses that record as the next baseline only when the verdict is +improved or regressed and the evidence is complete. An INCONCLUSIVE tip stays +the head until it is fixed and remeasured, or until the tip is reverted. + +## RSI phase, or the tip is not merge-clean + +Every tip states which RSI phase it belongs to, one of: + +- bring-up/scheduling +- parallelism/comms +- kernel +- launch/post + +It also states which alternatives were considered and which of them is the +primary change in the tip. A tip that omits the phase, or omits alternatives +versus the primary, is not merge-clean. Put that statement in the tip record +and in the delta fed back to the agent. A conclusive latency result does not +fill in a missing phase. + +## Draft, and a separate yes to merge + +Leave the pull request in draft through the loop. Do not mark it ready for +review and do not merge it because a remeasure improved p50 or p95. Merge +and ready-for-review each need a separate explicit yes from the owner. +Passing CI is not that yes. A tip that is not merge-clean on the RSI phase +stays unmerged even when the owner has not yet been asked. Room posts of a +delta follow the metrics-only rule in [public231-c1](../public231-c1/SKILL.md) +and include `device` and `compute capability`. diff --git a/.agents/skills/public231-c1/SKILL.md b/.agents/skills/public231-c1/SKILL.md new file mode 100644 index 00000000..75aa1f8e --- /dev/null +++ b/.agents/skills/public231-c1/SKILL.md @@ -0,0 +1,169 @@ +--- +name: public231-c1 +description: Measure Public231 C1 request latency for a model or pull request. Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock. Every paste records measured device and compute capability. The documented production target is H200 / SM90; a different measured identity stays a valid run and pairs only with that same recorded identity. Includes how to treat a thinking field and what a room post may contain. +--- + +# Public231 C1 + +Measure one deployment of the frozen 231-task Public231 text workload at +concurrency 1. Read [the benchmark protocol](../../../benchmarks/README.md) and +[the latency leaderboard](../../../benchmarks/latency-leaderboard.md) first. +`benchmarks/smoke.jsonl` is a contract smoke file, not this campaign. + +This skill records a measurement. It does not authorize a merge, a hardware +relabel, or a room post that contains operator metadata. + +## Measured hardware + +The production GPU, when a document is naming that intended target, is +**H200 / SM90**. That is the preferred label in examples. It is not a gate. +A run is not INCONCLUSIVE because `device` is not H200, not L20X, or not +another SKU, and not because `compute capability` is not SM90 or not sm_89. + +Every paste of a run records both measured fields: + +- `device`: the device-name string the host reported. `nvidia-smi` is the + source. The runner metadata field for that string is `gpu`. +- `compute capability`: the capability from the worker or device log. + +Preferred production paste: + +```text +device: H200 +compute capability: SM90 +``` + +If `nvidia-smi` prints a different name, paste that string as `device` and +paste the capability that was actually reported. Do not rewrite either field +to H200 / SM90, and do not rewrite them to L20X / sm_89. Archived notes in +[the GDN note](../../../benchmarks/gdn/README.md), +[the prefill-batching note](../../../benchmarks/prefill_batching/README.md), and +[the Open-Jev 27B validation](../../../recipe/open_jev/validation.md) disagree +about a driver string versus a corrected label for those campaigns. This +protocol keeps the string this host printed. If either field cannot be read, +leave it missing. Do not invent it. + +Paired runs, including a baseline and a later tip, compare only when both +sides use this same measurement protocol and the same named hardware: the +same recorded `device` string and the same recorded `compute capability`. +A different recorded identity is a separate group. It does not by itself make +either run inconclusive. + +## Freeze the run + +Use one frozen Public231 text manifest of 231 tasks. Record +`manifest_sha256` from the run's `config.json`. The leaderboard's canonical +dataset hash and the issue #130 JSONL hash are different representations of +related inputs; do not treat them as the same file checksum, and do not mix +this runner's clock with the historical H800 body-receipt clock. + +The runner replays request fields unchanged aside from the served model alias +(`--protocol systemone`, the default). Keep precision, request order, cache +policy, and the same server process across the three measured rounds. Exclude +process startup, readiness, and warmup from the latency samples. A feasibility +run, if you take one, stays out of the comparison. + +Set runner metadata as +[the protocol requires](../../../benchmarks/README.md#run-against-a-ready-gpu-server): +`gpu`, `gpu_ids`, `driver`, `cuda`, `precision`, `model_revision`, +`runtime_revision`, `cache_policy`, `cuda_evidence`, and `reservation`. +`runtime_revision` is the full git SHA that was actually measured. Fill +`reservation` under the execution host's own rules. Do not copy that field, +or any hostname, SSH target, username, filesystem path, or reservation-tool +name, into a room post. + +Pass the model wire contract the server implements. Decider's scalar Score is +two decimals, so that deployment needs `--score-decimals 2` on feasibility and +measured runs or warmup will reject valid responses. The default is 4. + +## Thinking field + +C1 is a no-thinking measurement unless the campaign text freezes thinking on. +Document which of these two treatments the manifest uses, and use only one +inside a campaign: + +1. **Accept `enabled: false`.** Leave a `thinking` object in the request and + set `enabled` to `false`. Keep the key when the frozen manifest or the + server branches on the object being present. +2. **Strip the field.** Delete `thinking` from the no-thinking twin so the + body matches workers whose no-thinking path is the absence of the field. + On the JEV-VL single-question contract the field is optional; the + implemented string values are `default` and `off`, and `auto` / `on` are + rejected. + +Do not send thinking-on content and call the run C1. Do not flip Cua-S1's chat +template with `enable_thinking=False`; that changes the prompt suffix and is a +different contract (`src/models/cua_s1/README.md`). The `jev-vl` bench adapter +rebuilds a one-choice body and drops other fields, so it is not the Public231 +text campaign. + +## Measure + +`bench.py` defaults `--warmup` to 3. This protocol passes **5** on every +measured round. The historical H800 board warmed up once for all three +traversals and used a different clock; do not copy that shape. Invoke `run` +once per fresh output directory. Three traversals are still 231 tasks, not 693. + +```sh +python benchmarks/bench.py validate + +python benchmarks/bench.py run \ + --endpoint --model \ + --metadata --phase measured \ + --concurrency 1 --warmup 5 \ + --output +``` + +Repeat the measured command for rounds 2 and 3 on the same server, each with +its own new directory. Then recompute each round with no extra requests: + +```sh +python benchmarks/bench.py summary +``` + +## What to report + +Pool only rounds that share deployment, manifest, and timing settings. +Concatenate `latency_ms` from `responses.jsonl` where `state` is `completed` +and `error` is null. Exclude warmups and unsuccessful records. For sorted +samples `x` of length `n`, p50 is the statistical median and p95 is the +zero-based nearest rank `x[ceil(0.95 * n) - 1]`. Recompute both from the +pooled samples. Do not average the three round percentiles. + +The request clock is +`latency_basis=post_to_body_json_and_schema_validation`: POST through complete +body receipt, JSON parsing, and wire schema validation. Hashing and +persistence are outside that clock. Historical leaderboard rows that stop at +body receipt and use linear-interpolation p95 are a different group. Do not +publish a speedup across those clocks. + +Publish: + +- full `runtime_revision` SHA and `manifest_sha256` +- `device` and `compute capability` (both, on every paste) +- thinking treatment (`enabled: false` or stripped) +- per-round and pooled p50/p95 in milliseconds +- serving pass rate: `successful_requests / planned_requests`, plus failed, + incomplete, and unattempted counts +- `evidence_complete` from each `summary.json` + +Labelled choice/noul accuracy and score MAE in +`quality_on_successful_requests` are conditional on successes. Report them +separately. They are not the serving pass rate, and a fast success subset is +not a ranked result. + +A round is not a C1 baseline when warmup failed, `evidence_complete` is false, +any measured request failed or was incomplete, or `device` or +`compute capability` was not recorded. Call that run **INCONCLUSIVE** and do +not repair hashes or counts to make it look complete. Failing to match H200, +SM90, L20X, or sm_89 is not a reason to mark the run INCONCLUSIVE. + +## Room posts + +Post metrics only: model or PR, SHA, `device`, `compute capability`, warmup 5, +concurrency 1, three rounds, clock basis, thinking treatment, pass rate, and +p50/p95, plus INCONCLUSIVE when that is the result. A production-target +example is `device: H200` and `compute capability: SM90`; if the host string +differs, post the recorded pair instead. +Omit hostnames, SSH targets, usernames, filesystem paths, and reservation-tool +names. Local output directories stay local.