skills: add Public231 C1, remeasure gate, and new-model integration

.agents/skills/new-model-integration/SKILL.md

@@ -0,0 +1,100 @@
+---
+name: new-model-integration
+description: Bring a new open model into System1-Omni on its own GitHub issue and draft pull request, then a recipe and operator checklist, a Public231 C1 measurement, and continuous optimization on that same branch. Use when adding model support. Keep serving-chain work on separate issues from the new-model RFC.
+---
+
+# Integrate a new open model
+
+Follow this order for a model that is not already supported. Implementation
+placement, contracts, tests, and docs stay in
+[add-new-model](../add-new-model/SKILL.md). This skill is the sequence around
+that work: issue, draft PR, recipe, Public231 C1, then measured optimization.
+Complete the [contributor-intent gate](../self-review/SKILL.md#human-contributor-intent-gate)
+before editing and before opening the draft.
+
+Do not merge, and do not mark the PR ready, without a separate explicit owner
+yes. A green check or a C1 number is not that yes.
+
+## 1. Open an independent issue
+
+Open a new GitHub issue for this model. Do not add the model to another
+model's issue or to a serving-chain issue.
+
+Use the `new model` label. Title it `[New Model]:` for an implementation
+request, or `[RFC]:` when the issue is still a design proposal, matching
+issues such as #136, #114, and #121. Say which checkpoint, modalities, and
+question types are in scope. Link the rolling tracker
+[#83](https://github.com/InferMatrixLab/system1-omni/issues/83) and the
+roadmap [#125](https://github.com/InferMatrixLab/system1-omni/issues/125);
+do not use either tracker as a substitute for the model's own issue.
+
+Serving-chain work stays on its own issues. The roadmap separates model
+milestones from shared serving features, and it says reference serving, native
+execution, and optimized execution are separate milestones. Shared
+`/v1/systemone` contract work is #61, observation reuse is #85, and
+admission, batching, and scheduling stay with #150 and the feature items on
+#125. Link those issues when the model depends on them. Do not implement that
+shared work inside the new-model RFC or its PR. A cross-request batching or
+scheduler change needs its own follow-up issue.
+
+## 2. Open a draft pull request
+
+Implement on a branch whose PR targets the model's issue and stays **draft**.
+[CONTRIBUTING](../../../CONTRIBUTING.md#self-review-before-requesting-review)
+keeps draft PRs in draft until self-review is complete. Use
+[precheck-pr](../precheck-pr/SKILL.md) before asking to mark it ready, and
+wait for the owner's separate yes before ready or merge.
+
+Keep the diff to this model's processors, executor, recipe, tests, and the
+docs `add-new-model` lists. Reuse a shared backend only when the model
+actually needs it. A new model is not a reason to redesign the frontend,
+scheduler, or batching policy.
+
+## 3. Recipe and operator checklist
+
+Land the operator-facing recipe in the same draft, under `recipe/<model>/`,
+before calling the integration runnable. `add-new-model` lists the required
+pages (model README, `recipe/README.md`, root README supported-model status,
+`mkdocs.yml`, and `docs/architecture.md` when a shared boundary changes).
+
+The recipe must let an operator do all of the following without private
+host notes:
+
+- pin the checkpoint, tokenizer, adapters, and dependency versions
+- build and launch the worker and, when used, the frontend
+- check health and send one real example request
+- run the model's validation commands and read the stated limits
+- see which hardware was actually tested, named as that hardware
+
+GPU claims name the measured `device` and `compute capability`. When the
+recipe is documenting the production target, the preferred label is
+**H200 / SM90**. If the host reported a different nvidia-smi string, the
+recipe records that string and the measured capability, and does not rewrite
+them. Do not describe untested hardware as validated. Do not put hostnames,
+SSH targets, usernames, or reservation-tool names in the recipe.
+
+## 4. Public231 C1
+
+When the draft serves the model, measure it with
+[public231-c1](../public231-c1/SKILL.md). Every paste records the draft SHA,
+`device`, `compute capability`, the serving pass rate, and pooled p50/p95.
+The documented production target is H200 / SM90. The protocol does not treat
+a different measured identity as INCONCLUSIVE; paired runs still have to share
+the recorded identity. A recipe that does not yet serve stays explicit about
+that: do not invent a C1 result. CPU contract tests are not a latency baseline.
+
+## 5. Continuous optimization on the same branch
+
+Further latency work stays on this model's draft branch. Run it under
+[optimize-remeasure-gate](../optimize-remeasure-gate/SKILL.md): paste the C1
+p50/p95, `device`, and `compute capability` first, and do not stack the next
+tip commit until that tip is remeasured with the same protocol on the same
+named hardware. Incomplete evidence is INCONCLUSIVE. Quality and contract
+checks stay green. Each tip states its RSI phase (bring-up/scheduling,
+parallelism/comms, kernel, or launch/post) and which alternatives were
+considered versus the primary; without that it is not merge-clean. Feed each
+delta back to the agent before the next edit.
+
+Optimized execution remains a separate milestone from reference serving and
+from native bring-up. Promote a number into the supported-model matrix only
+when the recipe and the measured SHA agree with it.

.agents/skills/optimize-remeasure-gate/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: optimize-remeasure-gate
+description: Gate continuous optimization after a measured Public231 C1 baseline, including Cloud Agent and coding-agent loops. Use when a new tip must not land until that tip is remeasured with the same protocol on the same recorded device and compute capability. Every paste includes those two fields. The documented production target is H200 / SM90. Each tip names its RSI phase and the primary change versus alternatives considered, or it is not merge-clean. Incomplete evidence is INCONCLUSIVE. The pull request stays draft until a separate owner yes.
+---
+
+# Optimize only against a remeasured tip
+
+Use this after a Public231 C1 baseline measured with
+[public231-c1](../public231-c1/SKILL.md). It is the loop for later latency
+work on that deployment, including a Cloud Agent or other coding agent that
+would otherwise keep committing. Read
+[the benchmark self-review rules](../../../benchmarks/README.md#self-review-and-ab-evidence)
+before changing inference or serving.
+
+The gate controls the next commit. It does not authorize a merge.
+
+## Paste the baseline before editing
+
+Put the measured primary numbers in the task before the first optimization
+edit, and again before every later tip. Every paste in this loop includes
+`device` and `compute capability`: the baseline paste, the tip remeasure,
+and the delta fed back. A paste that omits either field is incomplete.
+
+- baseline full git SHA (`runtime_revision`)
+- pooled p50 and p95 in milliseconds
+- serving pass rate (`successful_requests / planned_requests`)
+- clock basis `post_to_body_json_and_schema_validation`
+- `device` (the nvidia-smi name string; runner metadata `gpu`)
+- `compute capability`
+
+The documented production target is H200 / SM90. A paste of that machine
+reads `device: H200` and `compute capability: SM90`. If the host prints a
+different name, paste that string and the capability that was measured.
+Those are the primary figures. Per-round values stay attached so a pooled
+number can be checked. A recalled or rounded figure, a single round, or a
+percentile averaged across rounds is not a baseline.
+
+## One tip, then remeasure
+
+A tip is one optimization commit on the draft branch. Do not add another
+commit on top of it until that tip has its own Public231 C1 remeasure. The
+gate requires the same measurement protocol and the same named hardware as
+the baseline. It does not require the SKU to be H200, L20X, SM90, or sm_89.
+
+Same measurement protocol means warmup 5, concurrency 1, three measured
+rounds, the same manifest, and the same clock
+(`post_to_body_json_and_schema_validation`). Same named hardware means the
+same recorded `device` string and the same recorded `compute capability`.
+The remeasure SHA must be the tip SHA. A measurement of an older commit does
+not clear the tip.
+
+Keep controls matched to the baseline (manifest, precision, request order,
+cache policy, thinking treatment, server shape). State any necessary
+difference. Do not reset the comparison by changing the clock, the quantile
+method, or the named hardware.
+
+## INCONCLUSIVE stops the loop
+
+Mark the tip **INCONCLUSIVE** and do not stack the next commit when any of
+these are true:
+
+- the baseline p50/p95 were not recorded before the edit
+- fewer than three measured rounds finished
+- `evidence_complete` is not true, or warmup failed
+- any measured request failed, was incomplete, or was unattempted
+- `device` or `compute capability` is missing on the baseline or the tip
+- the tip's `device` string or `compute capability` differs from the
+  baseline, so the two sides are not the same named hardware
+- the tip does not use the same measurement protocol as the baseline
+- the clock basis is missing or is not
+  `post_to_body_json_and_schema_validation`
+- the measured SHA is not the tip SHA
+- pooled p50/p95 were averaged from round percentiles instead of recomputed
+  from the pooled samples
+
+INCONCLUSIVE is not a win and not a regression. Do not drop failures, relax a
+tolerance, or add undeclared rounds to produce a preferred delta. A recorded
+`device` other than H200, or a `compute capability` other than SM90, is not
+by itself INCONCLUSIVE.
+
+## Quality and contract stay green
+
+A latency delta is acceptable only when the tip's quality and contract checks
+are green:
+
+- the [validation for the files the tip touched](../../../CONTRIBUTING.md#choose-validation-by-changed-area)
+- the model request/response contract tests that cover the changed path
+- `python benchmarks/bench.py compare` against the baseline outputs when the
+  change can affect decisions or probabilities, with tolerances declared
+  before the run
+
+A faster tip that fails a contract test, flips decisions beyond the declared
+tolerance, or lowers the serving pass rate is not an accepted tip. Do not
+stack past it.
+
+## Feed the delta back
+
+Before the agent proposes the next change, give it this record:
+
+- baseline SHA, pooled p50, pooled p95, pass rate
+- tip SHA, pooled p50, pooled p95, pass rate
+- p50 and p95 delta (tip minus baseline), in milliseconds
+- clock basis, `device`, and `compute capability` for both sides
+- RSI phase, alternatives considered, and which change is primary
+- verdict: improved, regressed, or INCONCLUSIVE
+- quality/contract result
+
+The agent uses that record as the next baseline only when the verdict is
+improved or regressed and the evidence is complete. An INCONCLUSIVE tip stays
+the head until it is fixed and remeasured, or until the tip is reverted.
+
+## RSI phase, or the tip is not merge-clean
+
+Every tip states which RSI phase it belongs to, one of:
+
+- bring-up/scheduling
+- parallelism/comms
+- kernel
+- launch/post
+
+It also states which alternatives were considered and which of them is the
+primary change in the tip. A tip that omits the phase, or omits alternatives
+versus the primary, is not merge-clean. Put that statement in the tip record
+and in the delta fed back to the agent. A conclusive latency result does not
+fill in a missing phase.
+
+## Draft, and a separate yes to merge
+
+Leave the pull request in draft through the loop. Do not mark it ready for
+review and do not merge it because a remeasure improved p50 or p95. Merge
+and ready-for-review each need a separate explicit yes from the owner.
+Passing CI is not that yes. A tip that is not merge-clean on the RSI phase
+stays unmerged even when the owner has not yet been asked. Room posts of a
+delta follow the metrics-only rule in [public231-c1](../public231-c1/SKILL.md)
+and include `device` and `compute capability`.