[Decision] published skills/** teaching errors are found by hand (21 of 109 lane cards in two weeks): run the skill evals or a schema check over skill examples in CI (a new gate), or keep maintainer-scored evals #22059

Activity

1. objectstack-fleet

   added

   tooling

   needs-user-decision

   on Oct 7, 2026
2. objectstack-fleet commented on Oct 7, 2026

   

   objectstack-fleetbot

   on Oct 7, 2026

   ContributorAuthor

   More actions

   Box intake (triage): domain:skills · priority:p3 added. The card's shape is complete: 维护者速读, the option table with costs, the four axes, Prior rulings read:, the confidence gap and 裁后执行 are all present. The quoted rule text was checked on main: SKILL.md:255 carries 「新增门禁默认否」, and skills/README.md:67 says the evals/ fixtures are "used by maintainers to score the skill".

   Triage seat (objectstack-wide, seat post #6015) · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-07T06:00Z. ⛔ Not a claim.

   • Why p3: today's path works. Teaching errors are caught by hand and close in a median of 9 hours, as the card measures, and no release waits on this ruling.
   • For the director: the rule's default answer for an unnamed new gate is 「不加」 (C). B rests on reading it as a blind-spot fix of an existing gate, and the card puts that reading to the maintainer itself. Triage adds no recommendation.
   • Nothing else changes. The card waits in the box for the director, beside decision: open v18 now and ship it in stages — release the last 17.x from main first without waiting for #21908's deny (A), skip the last 17.x (B), or keep #22009's order (C)? #22050 and [Decision] the >5000-line SIZE limb counts a pure deletion the maintainer already ruled (ruling 208), and the Tier H wait for an authorized APPROVED has no clock — PR #22002 idles green #22051.
3. objectstack-fleet

   added

   domain:skills

   priority:p3

   on Oct 7, 2026
4. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Ruling: batch #287 item 2 · letter B (with the probe as its precondition) · maintainer 「批 #287 同意」 2026-10-08T02:21Z

   Director seat, summon #35, session_01VYToj6PQehTEKNrjGM9akg (GitHub os-zhuang; written as objectstack-fleet[bot] via the relay). Batch #287 was presented in chat twice, the second time in the detailed form the maintainer asked for, with this card's options A / B / C / D and the question whether B counts as a new gate under SKILL.md:255 (新增门禁默认否); the maintainer answered 「批 #287 同意」, which takes B as presented: the seat's framing of B as a blind-spot fix of the existing check-corpus-claim-drift gate, and, where that reading is contested, the maintainer's own word on this batch is the naming the rule asks for. Freshness gate: the body and the one comment (6031996518, triage's intake) were re-read before this record; nothing newer exists. Thread-read: 6031996518.

   The ruling

   B, in two steps, the second conditional on the first.

   1. The probe. Extract the example code blocks from the published skills/objectstack-* texts (TypeScript, JSON and YAML fences alike; a TypeScript example needs a transform into a shape the schemas accept, and whether that transform covers the catalog's fences is what the probe measures), validate them against packages/spec's schemas, and replay the validator over the fix diffs of the 21 teaching-error cards named in the card's premise 1. Report the hit rate.
   2. The gate step, only if the hit rate is at least one half. The validation joins check-corpus-claim-drift's own CI step (no new job, no model call, no inference cost), with its self-test and several of the 21 cases as fixed fixtures. A structural teaching error (a wrong key, a wrong nesting, a retired spelling) then fails the pull request loudly; a semantic one (a rule stated backwards) still needs a reader, as the card says plainly. If the hit rate is under one half, the ruling falls to C: no gate, the finding process as today, and the probe's readings stay on this card.

   ⛔ Not taken: A (a model scoring the evals/ fixtures in CI: a non-deterministic gate that adds inference cost and a new source of false reds), D (an offline report nobody reads, the fate evals/ already has).

   Why B on the first axis: the spec is the declaration and a shipped example must parse under it; the check reuses the schemas and the corpus gate that exist, where A would add a part the project then has to keep calibrated.

   State

   • needs-user-decision → pm:queue in this act; tooling · domain:skills · priority:p3 unchanged, no assignee: the domain:skills seat claims the probe from its queue; the gate step follows on this card only on a passing probe. Body: Ruled: line added in the same act.

   ---

   Generated by Claude Code
5. objectstack-fleet

   added

   pm:queue

   pm:dispatched

   and removed

   needs-user-decision

   pm:queue

   on Oct 8, 2026
6. objectstack-fleet

   assigned

   huangyiirene

   on Oct 8, 2026
7. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Claim: PM loop round 2
   Session: session_01CXydFDyiQwNbGFkmwrcRQq
   Account: huangyiirene (the seat's linked user as GET /user answers it; always the card's assignee)
   Branch: claude/issue-22059-skill-example-schema-probe
   Worktree: objectstack-issue-22059
   Domain: domain:skills
   Seat: domain:skills#1
   File surface: step ① of the ruling (the probe, this dispatch): scripts/check-corpus-claim-drift.mjs — a report-only probe mode beside its existing batteries, or a sibling module under scripts/ that it imports — with --self-test cases; the probe READS the published skills/objectstack-*/**/*.md texts and the fix diffs of the 21 teaching-error cards (REST reads) and reports the hit rate on this card. ⛔ In step ① no edit to skills/**, packages/spec/**, .github/workflows/**, package.json, scripts/corpus-claim-drift-baseline.json or scripts/pm/dispatch-gates.mjs (FROZEN). Step ② (the gate step: the validation joins check-corpus-claim-drift's existing CI step with fixtures) follows on this card and branch only on a hit rate of at least one half and only after this seat reads the probe report. Not a governed surface: a code PR, green + ACCEPT → queue landing. Stop on breach; explain in the report
   Container & model: M, mode:subagent, model: opus — dispatch-gates --tier --repo objectstack-ai/objectstack scripts/check-corpus-claim-drift.mjs prints no path-derived mandate ("the surface hits none of the 3 declared glob(s) … floor sonnet · default opus · ceiling fable"); reason: the default judgment tier for a measurement-first probe whose extraction and transform choices decide a gate ruling, judged from the card content
   Clause-②: no
   Responsibility: n/a — not a defect card (a ruled decision: letter B, ruling 6050857613, maintainer 「批 #287 同意」 at 2026-10-08T02:21Z)
   Thread-read: 6050857613
   Serial constraints cleared: no open PR touches scripts/check-corpus-claim-drift.mjs, its baseline or lint.yml (nine open PRs' file lists read at 2026-10-08T02:26Z); same-day churn none on the file (last touch cc1b8ac94d at 2026-09-13T08:08Z); file-disjoint from #22123 (same batch, skills/objectstack-upgrade/SKILL.md); scripts/pm/dispatch-gates.mjs FROZEN and not in this surface; verify lock free, queue empty (arrival depth 1 at this read); the lane's open P0/P1: none ⇒ p3 is a candidate (NORTH-STAR 「优先级」 3 read). Ruling-ref: 6050857613 (retrieved in this act, with its Governing text: items). Readings taken at 2026-10-08T02:28Z.

   ---

   Generated by Claude Code
8. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   os-dev-report

   {
     "issue": 22059,
     "status": "done",
     "branch": "claude/issue-22059-skill-example-schema-probe",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/22171",
     "session": "session_01CXydFDyiQwNbGFkmwrcRQq",
     "premise_still_valid": false,
     "summary": "Step one of ruling B is delivered as a report-only probe: `node scripts/check-corpus-claim-drift.mjs --probe-examples` (module `scripts/skill-example-probe.mjs`, exit 0 on any reading, wired into no check script, workflow or baseline), with four new self-test batteries (65 cases). HIT RATE 2/19 = 10.5% over the cards filed before #22059 that have a landed published-catalog repair (2/20 = 10.0% over all 20 the listing matched at filing; 2/22 = 9.1% over today's 22). The 2 hits are #20173 (page:header actions; ComponentPropsMap) and #21392 (select with no options; FieldSchema refinement), both in os:check blocks the TS type gate passes. Misses over the 22: prose 10, comment-or-string 4, neither 2, transform 2, not-in-corpus 1 (#18964), no-fix 1 (#22123). Ceiling with a perfect transform: 6/19 = 31.6%, because 9 of the 19 repairs touched no example fence. Transform coverage: 283 population fences (262 typescript, 16 ts, 1 tsx, 2 json, 2 jsonc, 0 yaml). Of the 279 TS fences, the transform produced a value for 203 (72.8%) and 115 (41.2%) were read and mapped whole. Control on today's catalog: 110 green, 3 red. Two of the reds are real teaching errors (`skills/objectstack-ui/rules/actions.md:77` and `:176`, both throw under the built `defineAction`; the TS gate `check:skill-examples` is green over :77). The third is the upgrade skill's deliberate protocol-16 example. Premise corrections: the listing gives 20 cards at filing, not 21 (#22120 and #22123 were created later; #18964 is a seat process finding). Four gates already read examples opt-in (check:skill-examples, check:yaml-examples, check:doc-formula-expressions, check:doc-security-posture). By the ruling's own terms the readings sit under one half; the step-two or C decision is the seat's.",
     "tests": "Head e6d00e1421. (1) `pnpm check:corpus-claim-drift`: exit 0, self-test OK with the Probe batteries at 14/18/21/12 cases, gate 'OK, no new claim sites' over 252 files. (2) `node scripts/check-corpus-claim-drift.mjs --probe-examples --json F`: exit 0. Apart from its head line, the output is byte-identical on every run from before the first commit to e6d00e1421. (3) Ablations via scripts/ablation-replace.mjs, each landed (anchor 1 to 0, blob moved) and each restored (blob == HEAD, git diff HEAD empty): opaque suppression off gives self-test exit 1 (2 cases); the HIT rule off gives exit 1 (1 case); the probe registration call off gives exit 1 (all 4 Probe batteries 'DID NOT RUN'). (4) `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 30 commands; all 30 were run on e6d00e1421 and all exited 0, `pnpm check:pm-dispatch-gates` included ('1976 cases pass', EXIT=0). --ran reconciliation: 30 derived, 30 run, 0 UNRUN. Roster families under scripts/ also run: check-published-list-mirrors (both), check:dts-closure, check:engine-double-contract, check:i18n-stale-fill and check-dts-references --self-test exit 0. NOT MEASURED: check:published-readme-exports, reason: PREREQUISITE NOT MET, exit 3, 45 packages unbuilt. check:console-injection, reason: no console dist, skipped. Test Core CI shard steps and the type-check lanes are CI's own; no package TypeScript is touched. An earlier battery run on 4a648ffec1 failed 1 of 1976 cases (compound self-test function names unledgered). The names were fixed in e6d00e1421 rather than ledgered. (5) eslint narrowed to the 2 touched files: `eslint --no-inline-config --format json` gives 2 files, 0 errors, 0 warnings. The population is eslint.config.mjs `files: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}']`. Invariance: no parserOptions.project or typed rules, so the diff cannot move a verdict on an untouched file. (6) Control: `pnpm --filter @objectstack/spec run check:skill-examples` exits 0 ('262 prose examples type-check across 3 surface(s)') with actions.md:77 in it. The spec and the client-react closure were built first through os-verify-lock. (7) pnpm check:nul-bytes exits 0; the control-byte self-scan of both files is empty.",
     "mcp_calls": "0 — none",
     "api_writes": "3 relay dispatches — POST /repos/objectstack-ai/objectstack/dispatches ×3, executed as objectstack-fleet[bot]: (a) pr_create, i.e. POST /pulls (draft #22171; read back 13547/13547 bytes identical); (b) label-write, i.e. POST /issues/22171/labels (skip-changeset) plus POST /issues/22171/assignees (huangyiirene); read back matching; size/xl came from another actor; (c) this os-dev-report comment, i.e. POST /issues/22059/comments. git push is not REST. Reads (REST GET, not writes): issue 22059 and its comments (2), the domain:skills listing (3 pages), 22 card timelines, PR 22171 read-back (1).",
     "open_questions": [],
     "out_of_scope_findings": [
       "class: c · reach: copying the published example verbatim makes the built `defineAction(...)` throw at module load. Measured at e6d00e1421; os validate and the metadata save parse with the same ActionSchema. · evidence: skills/objectstack-ui/rules/actions.md:77 (os:check block 'Modal-typed action'): `type: 'modal'` plus `body` is refused by the ActionSchema refinement on `body` ('`body` only runs for `type: 'script'` …'). actions.md:176 ('opensInNewTab + newTabUrl'): `type: 'url'` with no `target` is refused: 'Action 'target' is required when type is 'url', 'flow', 'modal', 'api', or 'form'.' The TS gate check:skill-examples is green over :77. One family (one file, one schema): file as one card. · Seam: spec:ActionSchema refinements (body vs type, target vs type) → runtime:defineAction / metadata save · dedupe words: actions.md modal body · defineAction target required url · opensInNewTab newTabUrl target · Modal-typed action body script",
       "carrier: step two of #22059 if it lands (an expect-red convention), else 承接者:无 · noted, not filed — skills/objectstack-upgrade/references/examples-upgrade.md:7 reads red (connectors[].fieldMappings, label, type) because it is the deliberate protocol-16 FROM shape the upgrade skill migrates; a probe gap, not a teaching error (PR Acceptance notes)"
     ]
   }

   ---

   Generated by Claude Code
9. objectstack-fleet

   removed

   pm:dispatched

   on Oct 8, 2026
10. objectstack-fleet

    unassigned

    huangyiirene

    on Oct 8, 2026
11. objectstack-fleet commented on Oct 8, 2026

    

    objectstack-fleetbot

    on Oct 8, 2026

    ContributorAuthor

    More actions

    Close-out — ruling B step ① delivered, readings under one half ⇒ C · skills seat 1, session_01CXydFDyiQwNbGFkmwrcRQq, 2026-10-08T04:02Z

    ACCEPT of the probe readings (dev report 6051826214; PR #22171, head e6d00e1421), reviewed against the PR body's replay table and the tree, not the report's narrative:

    • Hit rate 2 / 19 = 10.5% over the cards filed before this one that have a landed published-catalog repair; 2 / 20 over the listing at filing; 2 / 22 over today's. Ceiling with a perfect transform 6 / 19 = 31.6%, because 9 of the 19 repairs touched no example fence and 4 more changed only comments or strings inside one. Spot-check by this seat: replay row skills(objectstack-ui): the dashboards rule tells authors to override a compareTo series with series.dashArray / series.opacity — a dashboard widget's chartConfig.series is a tombstone the schema refuses by name #21211 (class "prose", 0 / 0 fences) against its landing PR docs(skills): the dashboards rule stops teaching a per-series override on the compareTo comparison series #21283: one file, +2 / −2, no fence opener changed. Holds.
    • Positive controls: the two HITs are the cards' own known errors (skills(objectstack-ui): pages.md and its eval teach full Action objects in page:header.properties.actions; the contract is action ids (ruling B on #11592) and os validate flags the taught shape #20173 page:header actions as objects; finding(skills): the objectstack-upgrade skill's "v17-canonical shapes, compiled" block authors a select with no options, which FieldSchema refuses once PR #21390 lands #21392 select with no options), both in os:check blocks the TypeScript gate passes. Catalog control: 110 green, 3 red; the two actions.md reds re-read by this seat on origin/main ec8f37c890 (sources and refinements) are real teaching errors and are filed as finding(skills): objectstack-ui rules/actions.md teaches two action shapes the ActionSchema refinements refuse — a modal carrying a body (:77) and a url action with no target (:176); copying either makes defineAction throw #22176; the third is the upgrade skill's deliberate protocol-16 shape, a probe gap, not an error.
    • Not re-run by this seat: the probe itself (it needs a built packages/spec/dist; the review stayed read-only). The ruling's bar is one half; no reading, denominator or ceiling comes near it, so the decision does not turn on the measurement's last digit.

    Decision, from the ruling's own text: "If the hit rate is under one half, the ruling falls to C: no gate, the finding process as today, and the probe's readings stay on this card." So: no step ②, no gate, no change to check-corpus-claim-drift's CI step. The readings stay here (the report above and PR #22171's body). PR #22171 is closed without merging (its note says why; the branch stays at e6d00e1421 as the probe's source of record). The card closes completed: the decision is made and executed.

    Premise corrections, acknowledged in public as the dev made them: the premise's listing matched 20 cards at filing, not 21 (#22120 and #22123 were created after this card; #18964 is a seat process finding, not in the published corpus). Four existing gates already read example blocks opt-in (check:skill-examples, check:yaml-examples, check:doc-formula-expressions, check:doc-security-posture), against the body's "none reads what an example teaches": the correct sentence is that none reads a Zod refinement, which is where both HITs and both new reds sit. No follow-up card: nothing further turns on either correction once C is taken.

    Carriers for the dev's acceptance notes: the expect-red convention for the upgrade skill's FROM shapes, the 86 ownership misses and the Plugin hold-out were gaps a gate would have had to close; with no gate they have no carrier and are recorded here only.

    State: pm:dispatched and the assignee come off in this act; closed completed. Reads and writes of the delivery: mcp_calls 0; api_writes 3, all relay dispatches as objectstack-fleet[bot].

    ---

    Generated by Claude Code
12. objectstack-fleet

    closed this as completedon Oct 8, 2026