docs: Prism v3 scoring (source-tree submissions, G1–G8 battery, Zone B) - #3
docs: Prism v3 scoring (source-tree submissions, G1–G8 battery, Zone B)#3echobt wants to merge 1 commit into
Conversation
…e B, baselines)
Recipe 1.3.0: two-phase pod flow, METRICS_JSON v2 (flat battery.metrics
org.* map + g2/g4 contamination mirrors), source-tree ZIP intake with
kernels/, attribution planner, Zone B self-report POST route
(/v1/submissions/{id}/zone-b), CAP_EXCEEDED terminal path. Leaf score
stays v2 pure bpb while PRISM_SCORING_MODE=shadow (default).
📝 WalkthroughWalkthroughThe documentation updates recipe 1.3.0 and defines v3 shadow evaluation, composite scoring, reporting APIs, Zone B submissions, and full source-tree ZIP submission rules. ChangesRecipe 1.3.0 and v3 documentation
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/getting-started.md`:
- Around line 20-23: The submission contract in the getting-started
documentation uses absolute “two scripts only” wording despite allowing the
recipe 1.3.0 source-tree alternative. Replace that wording with “the default
submission contains two seam files” or explicitly state the source-tree
exception in the same contract section, while preserving the existing
source-tree details.
In `@docs/scoring.md`:
- Around line 51-53: Update the fenced code block containing the POST route
example to specify the text language, changing the bare opening fence to a
text-labeled fence while preserving the route content.
- Around line 34-48: Update the Zone B examples and generated zone-b intake
contract so train() self-report fields use the metrics.miner.* namespace instead
of bare train_loss, train_steps, and train_seconds keys. Verify all baseline and
top-model examples follow this namespaced contract, and ensure no bare names
remain emitted.
In `@docs/submit.md`:
- Around line 34-36: In the documentation text around the harness submission
guidance, replace “Send tree ZIPs” with “Send source-tree ZIPs,” preserving the
surrounding JSON field and validation wording.
In `@README.md`:
- Around line 44-47: Apply the /challenge/prism gateway prefix to every
specified route reference: update the Zone B route in README.md:44-47 and
docs/scoring.md:51-53, and prefix the metrics, anchors, and preregistration
routes in docs/scoring.md:72-73. No other documentation changes are needed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3139625d-a249-4660-bb66-28e5e184f84b
📒 Files selected for processing (5)
README.mddocs/api.mddocs/getting-started.mddocs/scoring.mddocs/submit.md
| No offline weights, no network at pod runtime beyond the pinned dataset pull. | ||
| Since recipe **1.3.0** you may alternatively ship a full **source tree** (the two seam | ||
| files plus `prism.toml`, `count_params.py`, `kernels/`, `vendor.lock`) — see | ||
| [Submit](submit.md#source-tree-zip-recipe--130). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Qualify the two-script contract.
The document says “two scripts only” and then permits a full source tree. Replace the absolute wording with “the default submission contains two seam files” or state the exception in the same contract section.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/getting-started.md` around lines 20 - 23, The submission contract in the
getting-started documentation uses absolute “two scripts only” wording despite
allowing the recipe 1.3.0 source-tree alternative. Replace that wording with
“the default submission contains two seam files” or explicitly state the
source-tree exception in the same contract section, while preserving the
existing source-tree details.
| **What the harness reports (METRICS_JSON v2).** Every v1 key (`bpb`, `tokens_seen`, | ||
| `wall_clock_seconds`, `gpu_type`, `n_params`, `telemetry`, …) plus the v3 blocks: | ||
| `flow`, `eval_tier` (`"private"` | `"public_dev"`), `gate`, `probe_curve` (G6), | ||
| `train_metrics` (your Zone B dict, sanitized, never scored), and `battery`. The | ||
| battery's canonical surface is `battery.metrics` — a **flat** map of | ||
| `org.<group>.<name>` keys to a bare float or `{value, clusters}` (`clusters` are | ||
| per-template means, the units of randomization for the clustered bootstrap). A metric | ||
| that was never measured is **absent, never fabricated**. `battery.mirrors` carries the | ||
| contamination-gap pairs for G2/G4: the same metric scored on the public dev-seed asset | ||
| family vs a private mirror family — in the `public_dev` tier no private assets exist, | ||
| so each pair is degenerate (gap 0, honestly labelled). | ||
|
|
||
| Your `train()` return dict lands in **Zone B** (`miner.*` keys): displayed but labelled | ||
| participant-reported, validated at ingest, **never scored**. Never emit `org.*` keys — | ||
| that quarantines the report as anti-cheat evidence. You can also post additional |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 8 \
'train_metrics|miner\.[A-Za-z0-9_.]+|finish_evaluation|return[[:space:]]*\{' \
. --glob '*.py' --glob '*.md'Repository: BaseIntelligence/prism
Length of output: 16621
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- repo files mentioning train_metrics/battery/miner keys ---\n'
rg -n -C 5 'train_metrics|battery\.metrics|battery\.mirrors|miner\.[A-Za-z0-9_.]+|train_metrics|train_loss|metrics' README.md docs examples top-model --glob '!*.lock' --glob '!*.pyc' || true
printf '\n--- docs/scoring.md relevant sections ---\n'
sed -n '1,140p' docs/scoring.md
printf '\n--- baseline/top-model docs around Zone B if present ---\n'
sed -n '38,70p' examples/baseline/README.md 2>/dev/null || true
sed -n '1,40p' top-model/README.md 2>/dev/null || trueRepository: BaseIntelligence/prism
Length of output: 251
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- repo files mentioning train_metrics/battery/miner keys ---'
rg -n -C 5 'train_metrics|battery\.metrics|battery\.mirrors|miner\.[A-Za-z0-9_.]+|train_metrics|train_loss|metrics' README.md docs examples top-model --glob '!*.lock' --glob '!*.pyc' || true
echo
echo '--- docs/scoring.md relevant sections ---'
sed -n '1,140p' docs/scoring.md
echo
echo '--- baseline/top-model docs around Zone B if present ---'
sed -n '38,70p' examples/baseline/README.md 2>/dev/null || true
sed -n '1,40p' top-model/README.md 2>/dev/null || trueRepository: BaseIntelligence/prism
Length of output: 14418
Align the Zone B return contract with the harness.
train() must return keys namespaced under metrics.miner.* for the Zone B self-report path, but the baseline and top-model examples return bare keys (train_loss, train_steps, train_seconds). Either confirm that the harness renames/synthesizes those keys, or update the examples and any generated zone-b intake contract to stop emitting bare names.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/scoring.md` around lines 34 - 48, Update the Zone B examples and
generated zone-b intake contract so train() self-report fields use the
metrics.miner.* namespace instead of bare train_loss, train_steps, and
train_seconds keys. Verify all baseline and top-model examples follow this
namespaced contract, and ensure no bare names remain emitted.
| ``` | ||
| POST /v1/submissions/{id}/zone-b | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a language to the fenced code block.
Use text for the route example. The current bare fence fails Markdownlint rule MD040 at Line 51.
Proposed fix
-```
+```text
POST /v1/submissions/{id}/zone-b
-```
+```📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ``` | |
| POST /v1/submissions/{id}/zone-b | |
| ``` |
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 51-51: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/scoring.md` around lines 51 - 53, Update the fenced code block
containing the POST route example to specify the text language, changing the
bare opening fence to a text-labeled fence while preserving the route content.
Source: Linters/SAST tools
| (size budgets, banned-pattern scan, canonical hash) and re-audited in-pod by the | ||
| harness cheatguard. Send tree ZIPs via the JSON `zip_base64` field (the raw-zip path | ||
| rejects them with a pointer to `zip_base64` so the full tree is validated and |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the established source-tree ZIPs term.
Replace “Send tree ZIPs” with “Send source-tree ZIPs.” This removes the ambiguous wording and addresses the grammar warning at Line 35.
🧰 Tools
🪛 LanguageTool
[grammar] ~35-~35: Ensure spelling is correct
Context: ... in-pod by the harness cheatguard. Send tree ZIPs via the JSON zip_base64 field (t...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/submit.md` around lines 34 - 36, In the documentation text around the
harness submission guidance, replace “Send tree ZIPs” with “Send source-tree
ZIPs,” preserving the surrounding JSON field and validation wording.
Source: Linters/SAST tools
| [Scoring & competition](docs/scoring.md). Your `train()` return dict is a labelled, | ||
| never-scored **Zone B** self-report; additional reports can be posted to | ||
| `POST /v1/submissions/{id}/zone-b`. You may now also submit **full source trees** | ||
| (with custom `kernels/`) instead of only two scripts — see [Submit](docs/submit.md). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Apply the gateway prefix consistently to all new route references.
The new documentation omits /challenge/prism from several miner routes. This conflicts with docs/api.md and can cause 404 responses.
README.md#L44-L47: change the Zone B route to/challenge/prism/v1/submissions/{id}/zone-b.docs/scoring.md#L51-L53: change the Zone B route to/challenge/prism/v1/submissions/{id}/zone-b.docs/scoring.md#L72-L73: prefix the metrics, anchors, and preregistration routes with/challenge/prism.
📍 Affects 2 files
README.md#L44-L47(this comment)docs/scoring.md#L51-L53docs/scoring.md#L72-L73
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 44 - 47, Apply the /challenge/prism gateway prefix to
every specified route reference: update the Zone B route in README.md:44-47 and
docs/scoring.md:51-53, and prefix the metrics, anchors, and preregistration
routes in docs/scoring.md:72-73. No other documentation changes are needed.
1 participant
Summary
prism.tomlmanifest,count_params.py, customkernels/,vendor.lock) instead of only two scripts, with intake validation and in-pod cheatguard re-audit.battery.metrics(org.*keys) plusbattery.mirrorscontamination-gap pairs.train()metrics are labelled, validated, never scored; new out-of-band envelope +POST /v1/submissions/{id}/zone-broute (hash-chained, capped, verdictedok/flagged/quarantined,422on malformed).transformer_pp) and Hybrid Delta (hybrid_delta) ship as the v3 anchor reference family.eval_tier(privatevspublic_dev) documented; inpublic_devthe mirror pairs are degenerate (gap 0, honestly labelled).PRISM_SCORING_MODE=shadow(default); composite flip is a governance decision after anchors are pre-registered.Status
v3 is in final verification on the
prism-betterbranch of the monorepo. This PR is docs-only: it syncs the public miner docs with the current mirror (docs/external-miner/prism.md) and recipe contract. No miner-facing behavior changes are being shipped here.Test plan
zone-b,metrics?zone=a|b,attribution,anchors,preregistration)examples/baseline/guidance still matches the stated telemetry-hook contractSummary by CodeRabbit