From 2d47e52158d7cb1c501364c8dcb18ef8a0296f18 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 25 Oct 2026 00:39:40 +0000 Subject: [PATCH 1/3] skills: add Public231 C1, remeasure, and new-model recipes Document the lab L20X C1 measurement, the one-tip remeasure gate for agent optimization loops, and the issue-then-draft sequence for a new open model. Skill docs only. Co-authored-by: Hongsheng Liu --- .agents/skills/new-model-integration/SKILL.md | 91 +++++++++++ .../skills/optimize-remeasure-gate/SKILL.md | 100 ++++++++++++ .agents/skills/public231-c1/SKILL.md | 143 ++++++++++++++++++ 3 files changed, 334 insertions(+) create mode 100644 .agents/skills/new-model-integration/SKILL.md create mode 100644 .agents/skills/optimize-remeasure-gate/SKILL.md create mode 100644 .agents/skills/public231-c1/SKILL.md diff --git a/.agents/skills/new-model-integration/SKILL.md b/.agents/skills/new-model-integration/SKILL.md new file mode 100644 index 00000000..fdcbcd09 --- /dev/null +++ b/.agents/skills/new-model-integration/SKILL.md @@ -0,0 +1,91 @@ +--- +name: new-model-integration +description: Bring a new open model into System1-Omni on its own GitHub issue and draft pull request, then a recipe and operator checklist, a Public231 C1 measurement, and continuous optimization on that same branch. Use when adding model support. Keep serving-chain work on separate issues from the new-model RFC. +--- + +# Integrate a new open model + +Follow this order for a model that is not already supported. Implementation +placement, contracts, tests, and docs stay in +[add-new-model](../add-new-model/SKILL.md). This skill is the sequence around +that work: issue, draft PR, recipe, Public231 C1, then measured optimization. +Complete the [contributor-intent gate](../self-review/SKILL.md#human-contributor-intent-gate) +before editing and before opening the draft. + +Do not merge, and do not mark the PR ready, without a separate explicit owner +yes. A green check or a C1 number is not that yes. + +## 1. Open an independent issue + +Open a new GitHub issue for this model. Do not add the model to another +model's issue or to a serving-chain issue. + +Use the `new model` label. Title it `[New Model]:` for an implementation +request, or `[RFC]:` when the issue is still a design proposal, matching +issues such as #136, #114, and #121. Say which checkpoint, modalities, and +question types are in scope. Link the rolling tracker +[#83](https://github.com/InferMatrixLab/system1-omni/issues/83) and the +roadmap [#125](https://github.com/InferMatrixLab/system1-omni/issues/125); +do not use either tracker as a substitute for the model's own issue. + +Serving-chain work stays on its own issues. The roadmap separates model +milestones from shared serving features, and it says reference serving, native +execution, and optimized execution are separate milestones. Shared +`/v1/systemone` contract work is #61, observation reuse is #85, and +admission, batching, and scheduling stay with #150 and the feature items on +#125. Link those issues when the model depends on them. Do not implement that +shared work inside the new-model RFC or its PR. A cross-request batching or +scheduler change needs its own follow-up issue. + +## 2. Open a draft pull request + +Implement on a branch whose PR targets the model's issue and stays **draft**. +[CONTRIBUTING](../../../CONTRIBUTING.md#self-review-before-requesting-review) +keeps draft PRs in draft until self-review is complete. Use +[precheck-pr](../precheck-pr/SKILL.md) before asking to mark it ready, and +wait for the owner's separate yes before ready or merge. + +Keep the diff to this model's processors, executor, recipe, tests, and the +docs `add-new-model` lists. Reuse a shared backend only when the model +actually needs it. A new model is not a reason to redesign the frontend, +scheduler, or batching policy. + +## 3. Recipe and operator checklist + +Land the operator-facing recipe in the same draft, under `recipe//`, +before calling the integration runnable. `add-new-model` lists the required +pages (model README, `recipe/README.md`, root README supported-model status, +`mkdocs.yml`, and `docs/architecture.md` when a shared boundary changes). + +The recipe must let an operator do all of the following without private +host notes: + +- pin the checkpoint, tokenizer, adapters, and dependency versions +- build and launch the worker and, when used, the frontend +- check health and send one real example request +- run the model's validation commands and read the stated limits +- see which hardware was actually tested, named as that hardware + +GPU claims name the measured device and compute capability. Do not describe +untested hardware as validated. Do not put hostnames, SSH targets, usernames, +or reservation-tool names in the recipe. + +## 4. Public231 C1 + +When the draft serves the model, measure it with +[public231-c1](../public231-c1/SKILL.md) on lab L20X / sm_89. Record the +draft SHA, serving pass rate, and pooled p50/p95. A recipe that does not yet +serve stays explicit about that: do not invent a C1 result. CPU contract tests +are not a latency baseline. + +## 5. Continuous optimization on the same branch + +Further latency work stays on this model's draft branch. Run it under +[optimize-remeasure-gate](../optimize-remeasure-gate/SKILL.md): paste the C1 +p50/p95 first, and do not stack the next tip commit until that tip has its own +L20X remeasure. Incomplete evidence is INCONCLUSIVE. Quality and contract +checks stay green. Feed each delta back to the agent before the next edit. + +Optimized execution remains a separate milestone from reference serving and +from native bring-up. Promote a number into the supported-model matrix only +when the recipe and the measured SHA agree with it. diff --git a/.agents/skills/optimize-remeasure-gate/SKILL.md b/.agents/skills/optimize-remeasure-gate/SKILL.md new file mode 100644 index 00000000..dc3e704a --- /dev/null +++ b/.agents/skills/optimize-remeasure-gate/SKILL.md @@ -0,0 +1,100 @@ +--- +name: optimize-remeasure-gate +description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land on the previous tip until that tip has an L20X remeasure, evidence is complete or explicitly INCONCLUSIVE, quality and contract checks are green, and the pull request stays draft until a separate owner yes. +--- + +# Optimize only against a remeasured tip + +Use this after a Public231 C1 baseline measured with +[public231-c1](../public231-c1/SKILL.md). It is the loop for later latency +work on that deployment, including a Cloud Agent or other coding agent that +would otherwise keep committing. Read +[the benchmark self-review rules](../../../benchmarks/README.md#self-review-and-ab-evidence) +before changing inference or serving. + +The gate controls the next commit. It does not authorize a merge. + +## Paste the baseline before editing + +Put the measured primary numbers in the task before the first optimization +edit, and again before every later tip: + +- baseline full git SHA (`runtime_revision`) +- pooled p50 and p95 in milliseconds +- serving pass rate (`successful_requests / planned_requests`) +- clock basis `post_to_body_json_and_schema_validation` +- hardware label L20X / sm_89 + +Those are the primary figures. Per-round values stay attached so a pooled +number can be checked. A recalled or rounded figure, a single round, or a +percentile averaged across rounds is not a baseline. + +## One tip, then remeasure + +A tip is one optimization commit on the draft branch. Do not add another +commit on top of it until that tip has its own L20X Public231 C1 remeasure: +warmup 5, concurrency 1, three measured rounds, same manifest and clock as +the baseline. The remeasure SHA must be the tip SHA. A measurement of an +older commit does not clear the tip. + +Keep controls matched to the baseline (manifest, precision, request order, +cache policy, thinking treatment, server shape). State any necessary +difference. Do not reset the comparison by changing the clock, the quantile +method, or the hardware label. + +## INCONCLUSIVE stops the loop + +Mark the tip **INCONCLUSIVE** and do not stack the next commit when any of +these are true: + +- the baseline p50/p95 were not recorded before the edit +- fewer than three measured rounds finished +- `evidence_complete` is not true, or warmup failed +- any measured request failed, was incomplete, or was unattempted +- the device is not confirmed L20X / sm_89 +- the clock basis is missing or is not + `post_to_body_json_and_schema_validation` +- the measured SHA is not the tip SHA +- pooled p50/p95 were averaged from round percentiles instead of recomputed + from the pooled samples + +INCONCLUSIVE is not a win and not a regression. Do not drop failures, relax a +tolerance, or add undeclared rounds to produce a preferred delta. + +## Quality and contract stay green + +A latency delta is acceptable only when the tip's quality and contract checks +are green: + +- the [validation for the files the tip touched](../../../CONTRIBUTING.md#choose-validation-by-changed-area) +- the model request/response contract tests that cover the changed path +- `python benchmarks/bench.py compare` against the baseline outputs when the + change can affect decisions or probabilities, with tolerances declared + before the run + +A faster tip that fails a contract test, flips decisions beyond the declared +tolerance, or lowers the serving pass rate is not an accepted tip. Do not +stack past it. + +## Feed the delta back + +Before the agent proposes the next change, give it this record: + +- baseline SHA, pooled p50, pooled p95, pass rate +- tip SHA, pooled p50, pooled p95, pass rate +- p50 and p95 delta (tip minus baseline), in milliseconds +- clock basis and L20X / sm_89 +- verdict: improved, regressed, or INCONCLUSIVE +- quality/contract result + +The agent uses that record as the next baseline only when the verdict is +improved or regressed and the evidence is complete. An INCONCLUSIVE tip stays +the head until it is fixed and remeasured, or until the tip is reverted. + +## Draft, and a separate yes to merge + +Leave the pull request in draft through the loop. Do not mark it ready for +review and do not merge it because a remeasure improved p50 or p95. Merge +and ready-for-review each need a separate explicit yes from the owner. +Passing CI is not that yes. Room posts of a delta follow the metrics-only +rule in [public231-c1](../public231-c1/SKILL.md). diff --git a/.agents/skills/public231-c1/SKILL.md b/.agents/skills/public231-c1/SKILL.md new file mode 100644 index 00000000..21733002 --- /dev/null +++ b/.agents/skills/public231-c1/SKILL.md @@ -0,0 +1,143 @@ +--- +name: public231-c1 +description: Measure Public231 C1 request latency for a model or pull request on lab L20X (sm_89). Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock, including how to treat a thinking field and what a room post may contain. +--- + +# Public231 C1 on lab L20X + +Measure one deployment of the frozen 231-task Public231 text workload at +concurrency 1. Read [the benchmark protocol](../../../benchmarks/README.md) and +[the latency leaderboard](../../../benchmarks/latency-leaderboard.md) first. +`benchmarks/smoke.jsonl` is a contract smoke file, not this campaign. + +This skill records a measurement. It does not authorize a merge, a hardware +relabel, or a room post that contains operator metadata. + +## Hardware label + +Report the device as **L20X / sm_89**. Do not call it H200. + +Older notes correct a different archived part whose driver string was +`NVIDIA L20X` while the compute capability was 9.0. Those corrections live in +[the GDN note](../../../benchmarks/gdn/README.md), +[the prefill-batching note](../../../benchmarks/prefill_batching/README.md), and +[the Open-Jev 27B validation](../../../recipe/open_jev/validation.md). They do +not apply to this protocol. If the worker or device log shows a compute +capability other than 8.9, stop and mark the hardware identity unverified. +Do not substitute H200. + +## Freeze the run + +Use one frozen Public231 text manifest of 231 tasks. Record +`manifest_sha256` from the run's `config.json`. The leaderboard's canonical +dataset hash and the issue #130 JSONL hash are different representations of +related inputs; do not treat them as the same file checksum, and do not mix +this runner's clock with the historical H800 body-receipt clock. + +The runner replays request fields unchanged aside from the served model alias +(`--protocol systemone`, the default). Keep precision, request order, cache +policy, and the same server process across the three measured rounds. Exclude +process startup, readiness, and warmup from the latency samples. A feasibility +run, if you take one, stays out of the comparison. + +Set runner metadata as +[the protocol requires](../../../benchmarks/README.md#run-against-a-ready-gpu-server): +`gpu`, `gpu_ids`, `driver`, `cuda`, `precision`, `model_revision`, +`runtime_revision`, `cache_policy`, `cuda_evidence`, and `reservation`. +`runtime_revision` is the full git SHA that was actually measured. Fill +`reservation` under the execution host's own rules. Do not copy that field, +or any hostname, SSH target, username, filesystem path, or reservation-tool +name, into a room post. + +Pass the model wire contract the server implements. Decider's scalar Score is +two decimals, so that deployment needs `--score-decimals 2` on feasibility and +measured runs or warmup will reject valid responses. The default is 4. + +## Thinking field + +C1 is a no-thinking measurement unless the campaign text freezes thinking on. +Document which of these two treatments the manifest uses, and use only one +inside a campaign: + +1. **Accept `enabled: false`.** Leave a `thinking` object in the request and + set `enabled` to `false`. Keep the key when the frozen manifest or the + server branches on the object being present. +2. **Strip the field.** Delete `thinking` from the no-thinking twin so the + body matches workers whose no-thinking path is the absence of the field. + On the JEV-VL single-question contract the field is optional; the + implemented string values are `default` and `off`, and `auto` / `on` are + rejected. + +Do not send thinking-on content and call the run C1. Do not flip Cua-S1's chat +template with `enable_thinking=False`; that changes the prompt suffix and is a +different contract (`src/models/cua_s1/README.md`). The `jev-vl` bench adapter +rebuilds a one-choice body and drops other fields, so it is not the Public231 +text campaign. + +## Measure + +`bench.py` defaults `--warmup` to 3. This protocol passes **5** on every +measured round. The historical H800 board warmed up once for all three +traversals and used a different clock; do not copy that shape. Invoke `run` +once per fresh output directory. Three traversals are still 231 tasks, not 693. + +```sh +python benchmarks/bench.py validate + +python benchmarks/bench.py run \ + --endpoint --model \ + --metadata --phase measured \ + --concurrency 1 --warmup 5 \ + --output +``` + +Repeat the measured command for rounds 2 and 3 on the same server, each with +its own new directory. Then recompute each round with no extra requests: + +```sh +python benchmarks/bench.py summary +``` + +## What to report + +Pool only rounds that share deployment, manifest, and timing settings. +Concatenate `latency_ms` from `responses.jsonl` where `state` is `completed` +and `error` is null. Exclude warmups and unsuccessful records. For sorted +samples `x` of length `n`, p50 is the statistical median and p95 is the +zero-based nearest rank `x[ceil(0.95 * n) - 1]`. Recompute both from the +pooled samples. Do not average the three round percentiles. + +The request clock is +`latency_basis=post_to_body_json_and_schema_validation`: POST through complete +body receipt, JSON parsing, and wire schema validation. Hashing and +persistence are outside that clock. Historical leaderboard rows that stop at +body receipt and use linear-interpolation p95 are a different group. Do not +publish a speedup across those clocks. + +Publish: + +- full `runtime_revision` SHA and `manifest_sha256` +- hardware label L20X / sm_89 +- thinking treatment (`enabled: false` or stripped) +- per-round and pooled p50/p95 in milliseconds +- serving pass rate: `successful_requests / planned_requests`, plus failed, + incomplete, and unattempted counts +- `evidence_complete` from each `summary.json` + +Labelled choice/noul accuracy and score MAE in +`quality_on_successful_requests` are conditional on successes. Report them +separately. They are not the serving pass rate, and a fast success subset is +not a ranked result. + +A round is not a C1 baseline when warmup failed, `evidence_complete` is false, +any measured request failed or was incomplete, or the hardware label is not +L20X / sm_89. Call that run **INCONCLUSIVE** and do not repair hashes or +counts to make it look complete. + +## Room posts + +Post metrics only: model or PR, SHA, L20X / sm_89, warmup 5, concurrency 1, +three rounds, clock basis, thinking treatment, pass rate, and p50/p95, plus +INCONCLUSIVE when that is the result. Omit hostnames, SSH targets, usernames, +filesystem paths, and reservation-tool names. Local output directories stay +local. From 0ae6d39ef0cf9c1aa4dc193b6e3b6c1019270531 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 25 Oct 2026 07:05:21 +0000 Subject: [PATCH 2/3] skills: gate C1 on protocol and named hardware Stop treating a non-L20X device as inconclusive. Record the measured device name and compute capability, and require that pair plus the same measurement protocol on both sides of a comparison. Co-authored-by: Hongsheng Liu --- .agents/skills/new-model-integration/SKILL.md | 12 +++-- .../skills/optimize-remeasure-gate/SKILL.md | 27 ++++++---- .agents/skills/public231-c1/SKILL.md | 50 +++++++++++-------- 3 files changed, 55 insertions(+), 34 deletions(-) diff --git a/.agents/skills/new-model-integration/SKILL.md b/.agents/skills/new-model-integration/SKILL.md index fdcbcd09..d97d8e56 100644 --- a/.agents/skills/new-model-integration/SKILL.md +++ b/.agents/skills/new-model-integration/SKILL.md @@ -73,8 +73,9 @@ or reservation-tool names in the recipe. ## 4. Public231 C1 When the draft serves the model, measure it with -[public231-c1](../public231-c1/SKILL.md) on lab L20X / sm_89. Record the -draft SHA, serving pass rate, and pooled p50/p95. A recipe that does not yet +[public231-c1](../public231-c1/SKILL.md). Record the draft SHA, the measured +device name and compute capability, the serving pass rate, and pooled p50/p95. +The protocol does not require a particular SKU. A recipe that does not yet serve stays explicit about that: do not invent a C1 result. CPU contract tests are not a latency baseline. @@ -82,9 +83,10 @@ are not a latency baseline. Further latency work stays on this model's draft branch. Run it under [optimize-remeasure-gate](../optimize-remeasure-gate/SKILL.md): paste the C1 -p50/p95 first, and do not stack the next tip commit until that tip has its own -L20X remeasure. Incomplete evidence is INCONCLUSIVE. Quality and contract -checks stay green. Feed each delta back to the agent before the next edit. +p50/p95 first, and do not stack the next tip commit until that tip is +remeasured with the same protocol on the same named hardware. Incomplete +evidence is INCONCLUSIVE. Quality and contract checks stay green. Feed each +delta back to the agent before the next edit. Optimized execution remains a separate milestone from reference serving and from native bring-up. Promote a number into the supported-model matrix only diff --git a/.agents/skills/optimize-remeasure-gate/SKILL.md b/.agents/skills/optimize-remeasure-gate/SKILL.md index dc3e704a..08aa7ad4 100644 --- a/.agents/skills/optimize-remeasure-gate/SKILL.md +++ b/.agents/skills/optimize-remeasure-gate/SKILL.md @@ -1,6 +1,6 @@ --- name: optimize-remeasure-gate -description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land on the previous tip until that tip has an L20X remeasure, evidence is complete or explicitly INCONCLUSIVE, quality and contract checks are green, and the pull request stays draft until a separate owner yes. +description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land on the previous tip until that tip is remeasured with the same protocol on the same named hardware, evidence is complete or explicitly INCONCLUSIVE, quality and contract checks are green, and the pull request stays draft until a separate owner yes. --- # Optimize only against a remeasured tip @@ -23,7 +23,7 @@ edit, and again before every later tip: - pooled p50 and p95 in milliseconds - serving pass rate (`successful_requests / planned_requests`) - clock basis `post_to_body_json_and_schema_validation` -- hardware label L20X / sm_89 +- measured device name and compute capability Those are the primary figures. Per-round values stay attached so a pooled number can be checked. A recalled or rounded figure, a single round, or a @@ -32,15 +32,20 @@ percentile averaged across rounds is not a baseline. ## One tip, then remeasure A tip is one optimization commit on the draft branch. Do not add another -commit on top of it until that tip has its own L20X Public231 C1 remeasure: -warmup 5, concurrency 1, three measured rounds, same manifest and clock as -the baseline. The remeasure SHA must be the tip SHA. A measurement of an -older commit does not clear the tip. +commit on top of it until that tip has its own Public231 C1 remeasure. The +gate requires the same measurement protocol and the same named hardware as +the baseline, not a particular SKU. + +Same measurement protocol means warmup 5, concurrency 1, three measured +rounds, the same manifest, and the same clock +(`post_to_body_json_and_schema_validation`). Same named hardware means the +same measured device name and the same compute capability. The remeasure SHA +must be the tip SHA. A measurement of an older commit does not clear the tip. Keep controls matched to the baseline (manifest, precision, request order, cache policy, thinking treatment, server shape). State any necessary difference. Do not reset the comparison by changing the clock, the quantile -method, or the hardware label. +method, or the named hardware. ## INCONCLUSIVE stops the loop @@ -51,7 +56,11 @@ these are true: - fewer than three measured rounds finished - `evidence_complete` is not true, or warmup failed - any measured request failed, was incomplete, or was unattempted -- the device is not confirmed L20X / sm_89 +- the measured device name or compute capability is missing on the baseline + or the tip +- the tip's device name or compute capability differs from the baseline, so + the two sides are not the same named hardware +- the tip does not use the same measurement protocol as the baseline - the clock basis is missing or is not `post_to_body_json_and_schema_validation` - the measured SHA is not the tip SHA @@ -83,7 +92,7 @@ Before the agent proposes the next change, give it this record: - baseline SHA, pooled p50, pooled p95, pass rate - tip SHA, pooled p50, pooled p95, pass rate - p50 and p95 delta (tip minus baseline), in milliseconds -- clock basis and L20X / sm_89 +- clock basis, measured device name, and compute capability - verdict: improved, regressed, or INCONCLUSIVE - quality/contract result diff --git a/.agents/skills/public231-c1/SKILL.md b/.agents/skills/public231-c1/SKILL.md index 21733002..fe261e2e 100644 --- a/.agents/skills/public231-c1/SKILL.md +++ b/.agents/skills/public231-c1/SKILL.md @@ -1,9 +1,9 @@ --- name: public231-c1 -description: Measure Public231 C1 request latency for a model or pull request on lab L20X (sm_89). Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock, including how to treat a thinking field and what a room post may contain. +description: Measure Public231 C1 request latency for a model or pull request. Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock, with the measured device name and compute capability recorded. Includes how to treat a thinking field and what a room post may contain. --- -# Public231 C1 on lab L20X +# Public231 C1 Measure one deployment of the frozen 231-task Public231 text workload at concurrency 1. Read [the benchmark protocol](../../../benchmarks/README.md) and @@ -13,18 +13,27 @@ concurrency 1. Read [the benchmark protocol](../../../benchmarks/README.md) and This skill records a measurement. It does not authorize a merge, a hardware relabel, or a room post that contains operator metadata. -## Hardware label +## Measured hardware -Report the device as **L20X / sm_89**. Do not call it H200. +Record the device name and compute capability the run actually measured, in +the run evidence (`gpu` in the runner metadata, plus the compute capability +from the worker or device log). This protocol does not require a particular +SKU. A run on another device is still a C1 run when the rest of the protocol +is intact. -Older notes correct a different archived part whose driver string was -`NVIDIA L20X` while the compute capability was 9.0. Those corrections live in -[the GDN note](../../../benchmarks/gdn/README.md), +A comparison between paired runs, including a baseline and a later tip, is +valid only when both sides use this same measurement protocol and the same +named hardware: the same device name and the same compute capability. +Different hardware is a separate group. It does not by itself make either +run inconclusive. + +Do not replace the measured name or compute capability with a different SKU. +Archived notes in [the GDN note](../../../benchmarks/gdn/README.md), [the prefill-batching note](../../../benchmarks/prefill_batching/README.md), and -[the Open-Jev 27B validation](../../../recipe/open_jev/validation.md). They do -not apply to this protocol. If the worker or device log shows a compute -capability other than 8.9, stop and mark the hardware identity unverified. -Do not substitute H200. +[the Open-Jev 27B validation](../../../recipe/open_jev/validation.md) disagree +about a driver string versus a corrected label for those campaigns. Copy the +pair this run measured. If the device name or compute capability cannot be +read, leave it missing. Do not invent it. ## Freeze the run @@ -117,7 +126,7 @@ publish a speedup across those clocks. Publish: - full `runtime_revision` SHA and `manifest_sha256` -- hardware label L20X / sm_89 +- measured device name and compute capability - thinking treatment (`enabled: false` or stripped) - per-round and pooled p50/p95 in milliseconds - serving pass rate: `successful_requests / planned_requests`, plus failed, @@ -130,14 +139,15 @@ separately. They are not the serving pass rate, and a fast success subset is not a ranked result. A round is not a C1 baseline when warmup failed, `evidence_complete` is false, -any measured request failed or was incomplete, or the hardware label is not -L20X / sm_89. Call that run **INCONCLUSIVE** and do not repair hashes or -counts to make it look complete. +any measured request failed or was incomplete, or the device name or compute +capability was not recorded. Call that run **INCONCLUSIVE** and do not repair +hashes or counts to make it look complete. The device being a particular SKU +is not a reason to mark the run INCONCLUSIVE. ## Room posts -Post metrics only: model or PR, SHA, L20X / sm_89, warmup 5, concurrency 1, -three rounds, clock basis, thinking treatment, pass rate, and p50/p95, plus -INCONCLUSIVE when that is the result. Omit hostnames, SSH targets, usernames, -filesystem paths, and reservation-tool names. Local output directories stay -local. +Post metrics only: model or PR, SHA, measured device name and compute +capability, warmup 5, concurrency 1, three rounds, clock basis, thinking +treatment, pass rate, and p50/p95, plus INCONCLUSIVE when that is the result. +Omit hostnames, SSH targets, usernames, filesystem paths, and reservation-tool +names. Local output directories stay local. From c1dd64536bdd919e59c7b26c9b87f193becbd7dd Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 25 Oct 2026 07:07:02 +0000 Subject: [PATCH 3/3] skills: name production GPU H200/SM90 and require RSI phase Document H200 / SM90 as the production label without gating on it. Every paste records the measured device string and compute capability, paired runs must share that identity, and a tip is not merge-clean without its RSI phase and the primary versus alternatives. Co-authored-by: Hongsheng Liu --- .agents/skills/new-model-integration/SKILL.md | 29 +++++---- .../skills/optimize-remeasure-gate/SKILL.md | 54 ++++++++++++---- .agents/skills/public231-c1/SKILL.md | 64 ++++++++++++------- 3 files changed, 98 insertions(+), 49 deletions(-) diff --git a/.agents/skills/new-model-integration/SKILL.md b/.agents/skills/new-model-integration/SKILL.md index d97d8e56..6a9e839a 100644 --- a/.agents/skills/new-model-integration/SKILL.md +++ b/.agents/skills/new-model-integration/SKILL.md @@ -66,26 +66,33 @@ host notes: - run the model's validation commands and read the stated limits - see which hardware was actually tested, named as that hardware -GPU claims name the measured device and compute capability. Do not describe -untested hardware as validated. Do not put hostnames, SSH targets, usernames, -or reservation-tool names in the recipe. +GPU claims name the measured `device` and `compute capability`. When the +recipe is documenting the production target, the preferred label is +**H200 / SM90**. If the host reported a different nvidia-smi string, the +recipe records that string and the measured capability, and does not rewrite +them. Do not describe untested hardware as validated. Do not put hostnames, +SSH targets, usernames, or reservation-tool names in the recipe. ## 4. Public231 C1 When the draft serves the model, measure it with -[public231-c1](../public231-c1/SKILL.md). Record the draft SHA, the measured -device name and compute capability, the serving pass rate, and pooled p50/p95. -The protocol does not require a particular SKU. A recipe that does not yet -serve stays explicit about that: do not invent a C1 result. CPU contract tests -are not a latency baseline. +[public231-c1](../public231-c1/SKILL.md). Every paste records the draft SHA, +`device`, `compute capability`, the serving pass rate, and pooled p50/p95. +The documented production target is H200 / SM90. The protocol does not treat +a different measured identity as INCONCLUSIVE; paired runs still have to share +the recorded identity. A recipe that does not yet serve stays explicit about +that: do not invent a C1 result. CPU contract tests are not a latency baseline. ## 5. Continuous optimization on the same branch Further latency work stays on this model's draft branch. Run it under [optimize-remeasure-gate](../optimize-remeasure-gate/SKILL.md): paste the C1 -p50/p95 first, and do not stack the next tip commit until that tip is -remeasured with the same protocol on the same named hardware. Incomplete -evidence is INCONCLUSIVE. Quality and contract checks stay green. Feed each +p50/p95, `device`, and `compute capability` first, and do not stack the next +tip commit until that tip is remeasured with the same protocol on the same +named hardware. Incomplete evidence is INCONCLUSIVE. Quality and contract +checks stay green. Each tip states its RSI phase (bring-up/scheduling, +parallelism/comms, kernel, or launch/post) and which alternatives were +considered versus the primary; without that it is not merge-clean. Feed each delta back to the agent before the next edit. Optimized execution remains a separate milestone from reference serving and diff --git a/.agents/skills/optimize-remeasure-gate/SKILL.md b/.agents/skills/optimize-remeasure-gate/SKILL.md index 08aa7ad4..6dd861f8 100644 --- a/.agents/skills/optimize-remeasure-gate/SKILL.md +++ b/.agents/skills/optimize-remeasure-gate/SKILL.md @@ -1,6 +1,6 @@ --- name: optimize-remeasure-gate -description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land on the previous tip until that tip is remeasured with the same protocol on the same named hardware, evidence is complete or explicitly INCONCLUSIVE, quality and contract checks are green, and the pull request stays draft until a separate owner yes. +description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land until that tip is remeasured with the same protocol on the same recorded device and compute capability. Every paste includes those two fields. The documented production target is H200 / SM90. Each tip names its RSI phase and the primary change versus alternatives considered, or it is not merge-clean. Incomplete evidence is INCONCLUSIVE. The pull request stays draft until a separate owner yes. --- # Optimize only against a remeasured tip @@ -17,14 +17,20 @@ The gate controls the next commit. It does not authorize a merge. ## Paste the baseline before editing Put the measured primary numbers in the task before the first optimization -edit, and again before every later tip: +edit, and again before every later tip. Every paste in this loop includes +`device` and `compute capability`: the baseline paste, the tip remeasure, +and the delta fed back. A paste that omits either field is incomplete. - baseline full git SHA (`runtime_revision`) - pooled p50 and p95 in milliseconds - serving pass rate (`successful_requests / planned_requests`) - clock basis `post_to_body_json_and_schema_validation` -- measured device name and compute capability +- `device` (the nvidia-smi name string; runner metadata `gpu`) +- `compute capability` +The documented production target is H200 / SM90. A paste of that machine +reads `device: H200` and `compute capability: SM90`. If the host prints a +different name, paste that string and the capability that was measured. Those are the primary figures. Per-round values stay attached so a pooled number can be checked. A recalled or rounded figure, a single round, or a percentile averaged across rounds is not a baseline. @@ -34,13 +40,14 @@ percentile averaged across rounds is not a baseline. A tip is one optimization commit on the draft branch. Do not add another commit on top of it until that tip has its own Public231 C1 remeasure. The gate requires the same measurement protocol and the same named hardware as -the baseline, not a particular SKU. +the baseline. It does not require the SKU to be H200, L20X, SM90, or sm_89. Same measurement protocol means warmup 5, concurrency 1, three measured rounds, the same manifest, and the same clock (`post_to_body_json_and_schema_validation`). Same named hardware means the -same measured device name and the same compute capability. The remeasure SHA -must be the tip SHA. A measurement of an older commit does not clear the tip. +same recorded `device` string and the same recorded `compute capability`. +The remeasure SHA must be the tip SHA. A measurement of an older commit does +not clear the tip. Keep controls matched to the baseline (manifest, precision, request order, cache policy, thinking treatment, server shape). State any necessary @@ -56,10 +63,9 @@ these are true: - fewer than three measured rounds finished - `evidence_complete` is not true, or warmup failed - any measured request failed, was incomplete, or was unattempted -- the measured device name or compute capability is missing on the baseline - or the tip -- the tip's device name or compute capability differs from the baseline, so - the two sides are not the same named hardware +- `device` or `compute capability` is missing on the baseline or the tip +- the tip's `device` string or `compute capability` differs from the + baseline, so the two sides are not the same named hardware - the tip does not use the same measurement protocol as the baseline - the clock basis is missing or is not `post_to_body_json_and_schema_validation` @@ -68,7 +74,9 @@ these are true: from the pooled samples INCONCLUSIVE is not a win and not a regression. Do not drop failures, relax a -tolerance, or add undeclared rounds to produce a preferred delta. +tolerance, or add undeclared rounds to produce a preferred delta. A recorded +`device` other than H200, or a `compute capability` other than SM90, is not +by itself INCONCLUSIVE. ## Quality and contract stay green @@ -92,7 +100,8 @@ Before the agent proposes the next change, give it this record: - baseline SHA, pooled p50, pooled p95, pass rate - tip SHA, pooled p50, pooled p95, pass rate - p50 and p95 delta (tip minus baseline), in milliseconds -- clock basis, measured device name, and compute capability +- clock basis, `device`, and `compute capability` for both sides +- RSI phase, alternatives considered, and which change is primary - verdict: improved, regressed, or INCONCLUSIVE - quality/contract result @@ -100,10 +109,27 @@ The agent uses that record as the next baseline only when the verdict is improved or regressed and the evidence is complete. An INCONCLUSIVE tip stays the head until it is fixed and remeasured, or until the tip is reverted. +## RSI phase, or the tip is not merge-clean + +Every tip states which RSI phase it belongs to, one of: + +- bring-up/scheduling +- parallelism/comms +- kernel +- launch/post + +It also states which alternatives were considered and which of them is the +primary change in the tip. A tip that omits the phase, or omits alternatives +versus the primary, is not merge-clean. Put that statement in the tip record +and in the delta fed back to the agent. A conclusive latency result does not +fill in a missing phase. + ## Draft, and a separate yes to merge Leave the pull request in draft through the loop. Do not mark it ready for review and do not merge it because a remeasure improved p50 or p95. Merge and ready-for-review each need a separate explicit yes from the owner. -Passing CI is not that yes. Room posts of a delta follow the metrics-only -rule in [public231-c1](../public231-c1/SKILL.md). +Passing CI is not that yes. A tip that is not merge-clean on the RSI phase +stays unmerged even when the owner has not yet been asked. Room posts of a +delta follow the metrics-only rule in [public231-c1](../public231-c1/SKILL.md) +and include `device` and `compute capability`. diff --git a/.agents/skills/public231-c1/SKILL.md b/.agents/skills/public231-c1/SKILL.md index fe261e2e..75aa1f8e 100644 --- a/.agents/skills/public231-c1/SKILL.md +++ b/.agents/skills/public231-c1/SKILL.md @@ -1,6 +1,6 @@ --- name: public231-c1 -description: Measure Public231 C1 request latency for a model or pull request. Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock, with the measured device name and compute capability recorded. Includes how to treat a thinking field and what a room post may contain. +description: Measure Public231 C1 request latency for a model or pull request. Use when a campaign needs warmup 5, concurrency 1, three measured rounds, SHA, pass rate, and p50/p95 on the bench.py clock. Every paste records measured device and compute capability. The documented production target is H200 / SM90; a different measured identity stays a valid run and pairs only with that same recorded identity. Includes how to treat a thinking field and what a room post may contain. --- # Public231 C1 @@ -15,25 +15,39 @@ relabel, or a room post that contains operator metadata. ## Measured hardware -Record the device name and compute capability the run actually measured, in -the run evidence (`gpu` in the runner metadata, plus the compute capability -from the worker or device log). This protocol does not require a particular -SKU. A run on another device is still a C1 run when the rest of the protocol -is intact. +The production GPU, when a document is naming that intended target, is +**H200 / SM90**. That is the preferred label in examples. It is not a gate. +A run is not INCONCLUSIVE because `device` is not H200, not L20X, or not +another SKU, and not because `compute capability` is not SM90 or not sm_89. -A comparison between paired runs, including a baseline and a later tip, is -valid only when both sides use this same measurement protocol and the same -named hardware: the same device name and the same compute capability. -Different hardware is a separate group. It does not by itself make either -run inconclusive. +Every paste of a run records both measured fields: -Do not replace the measured name or compute capability with a different SKU. -Archived notes in [the GDN note](../../../benchmarks/gdn/README.md), +- `device`: the device-name string the host reported. `nvidia-smi` is the + source. The runner metadata field for that string is `gpu`. +- `compute capability`: the capability from the worker or device log. + +Preferred production paste: + +```text +device: H200 +compute capability: SM90 +``` + +If `nvidia-smi` prints a different name, paste that string as `device` and +paste the capability that was actually reported. Do not rewrite either field +to H200 / SM90, and do not rewrite them to L20X / sm_89. Archived notes in +[the GDN note](../../../benchmarks/gdn/README.md), [the prefill-batching note](../../../benchmarks/prefill_batching/README.md), and [the Open-Jev 27B validation](../../../recipe/open_jev/validation.md) disagree -about a driver string versus a corrected label for those campaigns. Copy the -pair this run measured. If the device name or compute capability cannot be -read, leave it missing. Do not invent it. +about a driver string versus a corrected label for those campaigns. This +protocol keeps the string this host printed. If either field cannot be read, +leave it missing. Do not invent it. + +Paired runs, including a baseline and a later tip, compare only when both +sides use this same measurement protocol and the same named hardware: the +same recorded `device` string and the same recorded `compute capability`. +A different recorded identity is a separate group. It does not by itself make +either run inconclusive. ## Freeze the run @@ -126,7 +140,7 @@ publish a speedup across those clocks. Publish: - full `runtime_revision` SHA and `manifest_sha256` -- measured device name and compute capability +- `device` and `compute capability` (both, on every paste) - thinking treatment (`enabled: false` or stripped) - per-round and pooled p50/p95 in milliseconds - serving pass rate: `successful_requests / planned_requests`, plus failed, @@ -139,15 +153,17 @@ separately. They are not the serving pass rate, and a fast success subset is not a ranked result. A round is not a C1 baseline when warmup failed, `evidence_complete` is false, -any measured request failed or was incomplete, or the device name or compute -capability was not recorded. Call that run **INCONCLUSIVE** and do not repair -hashes or counts to make it look complete. The device being a particular SKU -is not a reason to mark the run INCONCLUSIVE. +any measured request failed or was incomplete, or `device` or +`compute capability` was not recorded. Call that run **INCONCLUSIVE** and do +not repair hashes or counts to make it look complete. Failing to match H200, +SM90, L20X, or sm_89 is not a reason to mark the run INCONCLUSIVE. ## Room posts -Post metrics only: model or PR, SHA, measured device name and compute -capability, warmup 5, concurrency 1, three rounds, clock basis, thinking -treatment, pass rate, and p50/p95, plus INCONCLUSIVE when that is the result. +Post metrics only: model or PR, SHA, `device`, `compute capability`, warmup 5, +concurrency 1, three rounds, clock basis, thinking treatment, pass rate, and +p50/p95, plus INCONCLUSIVE when that is the result. A production-target +example is `device: H200` and `compute capability: SM90`; if the host string +differs, post the recorded pair instead. Omit hostnames, SSH targets, usernames, filesystem paths, and reservation-tool names. Local output directories stay local.