Skip to content

docs: Prism v3 scoring (source-tree submissions, G1–G8 battery, Zone B) #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
echobt wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Prism v3 scoring (source-tree submissions, G1–G8 battery, Zone B) #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,17 @@ miners — HTTP submit only.
| Production gateway | `https://chain.joinbase.ai` |
| Staging gateway | `http://staging.api.joinbase.ai` |
| Submit path | `/challenge/prism/v1/submissions` |
| Recipe | v1.2.0 — telemetry hooks required |
| Recipe | v1.3.0 — telemetry hooks required; source-tree ZIPs + v3 battery (shadow) |

**v3 (shadow-by-default):** recipe 1.3.0 runs your submission through a two-phase pod
flow (train → operator-staged private eval assets → eval) and measures it on the
**G1–G8 battery** (intrinsic fit, downstream, recall, reasoning, long-context, sample
efficiency, inference efficiency, stability) alongside the usual bpb. The leaf score is
still pure bpb until governance flips composite scoring on — see
[Scoring & competition](docs/scoring.md). Your `train()` return dict is a labelled,
never-scored **Zone B** self-report; additional reports can be posted to
`POST /v1/submissions/{id}/zone-b`. You may now also submit **full source trees**
(with custom `kernels/`) instead of only two scripts — see [Submit](docs/submit.md).
Comment on lines +44 to +47

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Apply the gateway prefix consistently to all new route references.

The new documentation omits /challenge/prism from several miner routes. This conflicts with docs/api.md and can cause 404 responses.

  • README.md#L44-L47: change the Zone B route to /challenge/prism/v1/submissions/{id}/zone-b.
  • docs/scoring.md#L51-L53: change the Zone B route to /challenge/prism/v1/submissions/{id}/zone-b.
  • docs/scoring.md#L72-L73: prefix the metrics, anchors, and preregistration routes with /challenge/prism.
📍 Affects 2 files
  • README.md#L44-L47 (this comment)
  • docs/scoring.md#L51-L53
  • docs/scoring.md#L72-L73
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 44 - 47, Apply the /challenge/prism gateway prefix to
every specified route reference: update the Zone B route in README.md:44-47 and
docs/scoring.md:51-53, and prefix the metrics, anchors, and preregistration
routes in docs/scoring.md:72-73. No other documentation changes are needed.


This repository holds **miner documentation and examples only**. Control-plane source
lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base).
Expand Down
5 changes: 5 additions & 0 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ Replace `{GATEWAY}` with `https://chain.joinbase.ai` (prod) or
| `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic records |
| `GET /challenge/prism/v1/submissions/{id}/events` | Stage timeline |
| `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row |
| `GET /challenge/prism/v1/submissions/{id}/metrics?zone=a\|b` | Zone A battery rows / Zone B self-report chain (v3) |
| `POST /challenge/prism/v1/submissions/{id}/zone-b` | Miner Zone B self-report intake — validated, chained, stored (v3) |
| `POST /challenge/prism/v1/submissions/{id}/attribution` | 2×2 arch/kernel attribution run plans (v3) |
| `GET /challenge/prism/v1/anchors` | v3 anchor-set registry + status |
| `GET /challenge/prism/v1/preregistration` | v3 anchor pre-registration hash-commits |
| `GET /challenge/prism/v1/submissions?miner=<hex>` | Your submissions |
| `GET /challenge/prism/v1/architectures` | Published archs + per-arch best bpb |
| `GET /challenge/prism/v1/recipe` | Versioned recipe descriptor + pin |
Expand Down
8 changes: 5 additions & 3 deletions docs/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Getting started

## The contract (recipe v1.2.0)
## The contract (recipe v1.3.0)

You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them,
downloads the pinned dataset, verifies its SHA-256, times the run, and reports
Expand All @@ -17,8 +17,10 @@ def train(model, ctx):
budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train)."""
```

No third source file, no offline weights, no network at pod runtime beyond the pinned
dataset pull.
No offline weights, no network at pod runtime beyond the pinned dataset pull.
Since recipe **1.3.0** you may alternatively ship a full **source tree** (the two seam
files plus `prism.toml`, `count_params.py`, `kernels/`, `vendor.lock`) — see
[Submit](submit.md#source-tree-zip-recipe--130).
Comment on lines +20 to +23

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Qualify the two-script contract.

The document says “two scripts only” and then permits a full source tree. Replace the absolute wording with “the default submission contains two seam files” or state the exception in the same contract section.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/getting-started.md` around lines 20 - 23, The submission contract in the
getting-started documentation uses absolute “two scripts only” wording despite
allowing the recipe 1.3.0 source-tree alternative. Replace that wording with
“the default submission contains two seam files” or explicitly state the
source-tree exception in the same contract section, while preserving the
existing source-tree details.


## Telemetry hooks (required since recipe 1.1.0)

Expand Down
73 changes: 73 additions & 0 deletions docs/scoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,79 @@
lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the
submission is coherent and not cheating; their quality notes never move the score.

## v3 composite (shadow-by-default)

Since recipe 1.3.0 every run is *also* measured on the organizer-run **G1–G8 battery**
(Zone A — computed by the harness, never by your code):

| Group | Axis | Weight |
|-------|------|--------|
| G1 | intrinsic fit (frozen-val + multi-domain/fresh-crawl bpb) | 0.25 |
| G2 | commonsense/reading 0-shot core | 0.15 |
| G3 | retrieval/associative recall (gated ≥ 0.25) | 0.10 |
| G4 | reasoning at small scale | 0.15 |
| G5 | long-context | 0.15 |
| G6 | sample efficiency (train probe curve) | 0.075 |
| G7 | inference efficiency | 0.075 |
| G8 | training stability + µP (gated ≥ 0.5) | 0.05 |

The battery runs in a **two-phase pod flow**: training completes first, then the
operator stages private eval assets (held-out + fresh-crawl data) and a fresh eval
process measures the model. While `PRISM_SCORING_MODE=shadow` (default) your leaf score
stays **pure bpb, bit-identical to v2**. After the reference baselines
(**Transformer++**, **hybrid delta**) are measured and the anchor set is
pre-registered, governance may flip to `composite`: anchor-normalized group scores,
gates, a weighted geometric mean, and bootstrap lower-confidence-bound ranking
(`lattice = round(SCORE_MAX × max(0, C − 1.645·SE))`, `scoring_version 3`).

**What the harness reports (METRICS_JSON v2).** Every v1 key (`bpb`, `tokens_seen`,
`wall_clock_seconds`, `gpu_type`, `n_params`, `telemetry`, …) plus the v3 blocks:
`flow`, `eval_tier` (`"private"` | `"public_dev"`), `gate`, `probe_curve` (G6),
`train_metrics` (your Zone B dict, sanitized, never scored), and `battery`. The
battery's canonical surface is `battery.metrics` — a **flat** map of
`org.<group>.<name>` keys to a bare float or `{value, clusters}` (`clusters` are
per-template means, the units of randomization for the clustered bootstrap). A metric
that was never measured is **absent, never fabricated**. `battery.mirrors` carries the
contamination-gap pairs for G2/G4: the same metric scored on the public dev-seed asset
family vs a private mirror family — in the `public_dev` tier no private assets exist,
so each pair is degenerate (gap 0, honestly labelled).

Your `train()` return dict lands in **Zone B** (`miner.*` keys): displayed but labelled
participant-reported, validated at ingest, **never scored**. Never emit `org.*` keys —
that quarantines the report as anti-cheat evidence. You can also post additional
Comment on lines +34 to +48

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 8 \
  'train_metrics|miner\.[A-Za-z0-9_.]+|finish_evaluation|return[[:space:]]*\{' \
  . --glob '*.py' --glob '*.md'

Repository: BaseIntelligence/prism

Length of output: 16621

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- repo files mentioning train_metrics/battery/miner keys ---\n'
rg -n -C 5 'train_metrics|battery\.metrics|battery\.mirrors|miner\.[A-Za-z0-9_.]+|train_metrics|train_loss|metrics' README.md docs examples top-model --glob '!*.lock' --glob '!*.pyc' || true

printf '\n--- docs/scoring.md relevant sections ---\n'
sed -n '1,140p' docs/scoring.md

printf '\n--- baseline/top-model docs around Zone B if present ---\n'
sed -n '38,70p' examples/baseline/README.md 2>/dev/null || true
sed -n '1,40p' top-model/README.md 2>/dev/null || true

Repository: BaseIntelligence/prism

Length of output: 251

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- repo files mentioning train_metrics/battery/miner keys ---'
rg -n -C 5 'train_metrics|battery\.metrics|battery\.mirrors|miner\.[A-Za-z0-9_.]+|train_metrics|train_loss|metrics' README.md docs examples top-model --glob '!*.lock' --glob '!*.pyc' || true

echo
echo '--- docs/scoring.md relevant sections ---'
sed -n '1,140p' docs/scoring.md

echo
echo '--- baseline/top-model docs around Zone B if present ---'
sed -n '38,70p' examples/baseline/README.md 2>/dev/null || true
sed -n '1,40p' top-model/README.md 2>/dev/null || true

Repository: BaseIntelligence/prism

Length of output: 14418

Align the Zone B return contract with the harness.

train() must return keys namespaced under metrics.miner.* for the Zone B self-report path, but the baseline and top-model examples return bare keys (train_loss, train_steps, train_seconds). Either confirm that the harness renames/synthesizes those keys, or update the examples and any generated zone-b intake contract to stop emitting bare names.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/scoring.md` around lines 34 - 48, Update the Zone B examples and
generated zone-b intake contract so train() self-report fields use the
metrics.miner.* namespace instead of bare train_loss, train_steps, and
train_seconds keys. Verify all baseline and top-model examples follow this
namespaced contract, and ensure no bare names remain emitted.

self-reports out-of-band:

```
POST /v1/submissions/{id}/zone-b
```
Comment on lines +51 to +53

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a language to the fenced code block.

Use text for the route example. The current bare fence fails Markdownlint rule MD040 at Line 51.

Proposed fix 
-```
+```text
 POST /v1/submissions/{id}/zone-b
-```
+```
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```
POST /v1/submissions/{id}/zone-b
```
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)

[warning] 51-51: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/scoring.md` around lines 51 - 53, Update the fenced code block
containing the POST route example to specify the text language, changing the
bare opening fence to a text-labeled fence while preserving the route content.

Source: Linters/SAST tools


```json
{
"schema_version": "<recipe version>",
"prev_hash": "<previous report_hash — optional>",
"metrics": {
"miner.<group>.<name>": {"kind": "scalar | series | histogram"}
}
}
```

Reports chain per submission (`prev_hash` → the previous `report_hash`; omit it for
master-chained ingest) and are capped at 64 scalars / 16 series / 10 000 points / 1 MB.
Each report is validated against organizer ground truth (token/step/wall-clock
counters, MFU ceiling, terminal-loss band) and the cross-miner cohort, and lands a
stored verdict — `ok` / `flagged` / `quarantined`. Verdicts are evidence, never an
auto-zero. Malformed or over-cap envelopes reject `422` and store nothing.

Per-run rows: `GET /v1/submissions/{id}/metrics?zone=a|b`; anchor registry:
`GET /v1/anchors` and `GET /v1/preregistration`.

Kernel-carrying source trees can be decomposed with the 2×2 **attribution** planner
(`POST /v1/submissions/{id}/attribution`): your architecture on reference kernels vs
the reference architecture on your kernels, isolating arch vs kernel contributions.

Note: a model over the **350M parameter cap** is now a terminal `Score(0)`
(`CAP_EXCEEDED`), not a retryable failure.

## Anti-copy (architecture-only)

- A **pre-LLM copy gate** compares your `architecture.py` against earlier submissions
Expand Down
17 changes: 17 additions & 0 deletions docs/submit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,23 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
--data-binary @submission.zip
```

## Source-tree ZIP (recipe ≥ 1.3.0)

Instead of only two scripts you may submit a full **source tree**: the two seam files
plus optional `prism.toml` (entry point), `count_params.py`, a `kernels/` directory of
custom ops (pure Python + torch, per `KERNEL_INTERFACE.md` — no prebuilt binaries, no
`ctypes`, no I/O or threads), and a `vendor.lock`. Trees are validated at intake
(size budgets, banned-pattern scan, canonical hash) and re-audited in-pod by the
harness cheatguard. Send tree ZIPs via the JSON `zip_base64` field (the raw-zip path
rejects them with a pointer to `zip_base64` so the full tree is validated and
Comment on lines +34 to +36

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use the established source-tree ZIPs term.

Replace “Send tree ZIPs” with “Send source-tree ZIPs.” This removes the ambiguous wording and addresses the grammar warning at Line 35.

🧰 Tools
🪛 LanguageTool

[grammar] ~35-~35: Ensure spelling is correct
Context: ... in-pod by the harness cheatguard. Send tree ZIPs via the JSON zip_base64 field (t...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/submit.md` around lines 34 - 36, In the documentation text around the
harness submission guidance, replace “Send tree ZIPs” with “Send source-tree
ZIPs,” preserving the surrounding JSON field and validation wording.

Source: Linters/SAST tools

retained):

```bash
curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
-H 'content-type: application/json' \
-d '{"miner_hotkey":"<hex>","zip_base64":"<base64 of tree.zip>"}'
```

## JSON (local / scripting)

```bash
Expand Down
Loading