From a79a8d32a8f96401f89423d11485ce6049e9a2ad Mon Sep 17 00:00:00 2001 From: echobt <154886644+echobt@users.noreply.github.com> Date: Thu, 6 Aug 2026 13:46:25 +0000 Subject: [PATCH] docs: summarize Prism v3 (source-tree submissions, G1-G8 battery, Zone B, baselines) Recipe 1.3.0: two-phase pod flow, METRICS_JSON v2 (flat battery.metrics org.* map + g2/g4 contamination mirrors), source-tree ZIP intake with kernels/, attribution planner, Zone B self-report POST route (/v1/submissions/{id}/zone-b), CAP_EXCEEDED terminal path. Leaf score stays v2 pure bpb while PRISM_SCORING_MODE=shadow (default). --- README.md | 12 ++++++- docs/api.md | 5 +++ docs/getting-started.md | 8 +++-- docs/scoring.md | 73 +++++++++++++++++++++++++++++++++++++++++ docs/submit.md | 17 ++++++++++ 5 files changed, 111 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index c8b0c4f..f8ee475 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,17 @@ miners — HTTP submit only. | Production gateway | `https://chain.joinbase.ai` | | Staging gateway | `http://staging.api.joinbase.ai` | | Submit path | `/challenge/prism/v1/submissions` | -| Recipe | v1.2.0 — telemetry hooks required | +| Recipe | v1.3.0 — telemetry hooks required; source-tree ZIPs + v3 battery (shadow) | + +**v3 (shadow-by-default):** recipe 1.3.0 runs your submission through a two-phase pod +flow (train → operator-staged private eval assets → eval) and measures it on the +**G1–G8 battery** (intrinsic fit, downstream, recall, reasoning, long-context, sample +efficiency, inference efficiency, stability) alongside the usual bpb. The leaf score is +still pure bpb until governance flips composite scoring on — see +[Scoring & competition](docs/scoring.md). Your `train()` return dict is a labelled, +never-scored **Zone B** self-report; additional reports can be posted to +`POST /v1/submissions/{id}/zone-b`. You may now also submit **full source trees** +(with custom `kernels/`) instead of only two scripts — see [Submit](docs/submit.md). This repository holds **miner documentation and examples only**. Control-plane source lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). diff --git a/docs/api.md b/docs/api.md index 16c62e9..5aada15 100644 --- a/docs/api.md +++ b/docs/api.md @@ -13,6 +13,11 @@ Replace `{GATEWAY}` with `https://chain.joinbase.ai` (prod) or | `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic records | | `GET /challenge/prism/v1/submissions/{id}/events` | Stage timeline | | `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row | +| `GET /challenge/prism/v1/submissions/{id}/metrics?zone=a\|b` | Zone A battery rows / Zone B self-report chain (v3) | +| `POST /challenge/prism/v1/submissions/{id}/zone-b` | Miner Zone B self-report intake — validated, chained, stored (v3) | +| `POST /challenge/prism/v1/submissions/{id}/attribution` | 2×2 arch/kernel attribution run plans (v3) | +| `GET /challenge/prism/v1/anchors` | v3 anchor-set registry + status | +| `GET /challenge/prism/v1/preregistration` | v3 anchor pre-registration hash-commits | | `GET /challenge/prism/v1/submissions?miner=` | Your submissions | | `GET /challenge/prism/v1/architectures` | Published archs + per-arch best bpb | | `GET /challenge/prism/v1/recipe` | Versioned recipe descriptor + pin | diff --git a/docs/getting-started.md b/docs/getting-started.md index d53ca74..050f1c0 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,6 @@ # Getting started -## The contract (recipe v1.2.0) +## The contract (recipe v1.3.0) You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them, downloads the pinned dataset, verifies its SHA-256, times the run, and reports @@ -17,8 +17,10 @@ def train(model, ctx): budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train).""" ``` -No third source file, no offline weights, no network at pod runtime beyond the pinned -dataset pull. +No offline weights, no network at pod runtime beyond the pinned dataset pull. +Since recipe **1.3.0** you may alternatively ship a full **source tree** (the two seam +files plus `prism.toml`, `count_params.py`, `kernels/`, `vendor.lock`) — see +[Submit](submit.md#source-tree-zip-recipe--130). ## Telemetry hooks (required since recipe 1.1.0) diff --git a/docs/scoring.md b/docs/scoring.md index aaaebec..96aabb4 100644 --- a/docs/scoring.md +++ b/docs/scoring.md @@ -6,6 +6,79 @@ lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the submission is coherent and not cheating; their quality notes never move the score. +## v3 composite (shadow-by-default) + +Since recipe 1.3.0 every run is *also* measured on the organizer-run **G1–G8 battery** +(Zone A — computed by the harness, never by your code): + +| Group | Axis | Weight | +|-------|------|--------| +| G1 | intrinsic fit (frozen-val + multi-domain/fresh-crawl bpb) | 0.25 | +| G2 | commonsense/reading 0-shot core | 0.15 | +| G3 | retrieval/associative recall (gated ≥ 0.25) | 0.10 | +| G4 | reasoning at small scale | 0.15 | +| G5 | long-context | 0.15 | +| G6 | sample efficiency (train probe curve) | 0.075 | +| G7 | inference efficiency | 0.075 | +| G8 | training stability + µP (gated ≥ 0.5) | 0.05 | + +The battery runs in a **two-phase pod flow**: training completes first, then the +operator stages private eval assets (held-out + fresh-crawl data) and a fresh eval +process measures the model. While `PRISM_SCORING_MODE=shadow` (default) your leaf score +stays **pure bpb, bit-identical to v2**. After the reference baselines +(**Transformer++**, **hybrid delta**) are measured and the anchor set is +pre-registered, governance may flip to `composite`: anchor-normalized group scores, +gates, a weighted geometric mean, and bootstrap lower-confidence-bound ranking +(`lattice = round(SCORE_MAX × max(0, C − 1.645·SE))`, `scoring_version 3`). + +**What the harness reports (METRICS_JSON v2).** Every v1 key (`bpb`, `tokens_seen`, +`wall_clock_seconds`, `gpu_type`, `n_params`, `telemetry`, …) plus the v3 blocks: +`flow`, `eval_tier` (`"private"` | `"public_dev"`), `gate`, `probe_curve` (G6), +`train_metrics` (your Zone B dict, sanitized, never scored), and `battery`. The +battery's canonical surface is `battery.metrics` — a **flat** map of +`org..` keys to a bare float or `{value, clusters}` (`clusters` are +per-template means, the units of randomization for the clustered bootstrap). A metric +that was never measured is **absent, never fabricated**. `battery.mirrors` carries the +contamination-gap pairs for G2/G4: the same metric scored on the public dev-seed asset +family vs a private mirror family — in the `public_dev` tier no private assets exist, +so each pair is degenerate (gap 0, honestly labelled). + +Your `train()` return dict lands in **Zone B** (`miner.*` keys): displayed but labelled +participant-reported, validated at ingest, **never scored**. Never emit `org.*` keys — +that quarantines the report as anti-cheat evidence. You can also post additional +self-reports out-of-band: + +``` +POST /v1/submissions/{id}/zone-b +``` + +```json +{ + "schema_version": "", + "prev_hash": "", + "metrics": { + "miner..": {"kind": "scalar | series | histogram"} + } +} +``` + +Reports chain per submission (`prev_hash` → the previous `report_hash`; omit it for +master-chained ingest) and are capped at 64 scalars / 16 series / 10 000 points / 1 MB. +Each report is validated against organizer ground truth (token/step/wall-clock +counters, MFU ceiling, terminal-loss band) and the cross-miner cohort, and lands a +stored verdict — `ok` / `flagged` / `quarantined`. Verdicts are evidence, never an +auto-zero. Malformed or over-cap envelopes reject `422` and store nothing. + +Per-run rows: `GET /v1/submissions/{id}/metrics?zone=a|b`; anchor registry: +`GET /v1/anchors` and `GET /v1/preregistration`. + +Kernel-carrying source trees can be decomposed with the 2×2 **attribution** planner +(`POST /v1/submissions/{id}/attribution`): your architecture on reference kernels vs +the reference architecture on your kernels, isolating arch vs kernel contributions. + +Note: a model over the **350M parameter cap** is now a terminal `Score(0)` +(`CAP_EXCEEDED`), not a retryable failure. + ## Anti-copy (architecture-only) - A **pre-LLM copy gate** compares your `architecture.py` against earlier submissions diff --git a/docs/submit.md b/docs/submit.md index 0ea6583..117d895 100644 --- a/docs/submit.md +++ b/docs/submit.md @@ -25,6 +25,23 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ --data-binary @submission.zip ``` +## Source-tree ZIP (recipe ≥ 1.3.0) + +Instead of only two scripts you may submit a full **source tree**: the two seam files +plus optional `prism.toml` (entry point), `count_params.py`, a `kernels/` directory of +custom ops (pure Python + torch, per `KERNEL_INTERFACE.md` — no prebuilt binaries, no +`ctypes`, no I/O or threads), and a `vendor.lock`. Trees are validated at intake +(size budgets, banned-pattern scan, canonical hash) and re-audited in-pod by the +harness cheatguard. Send tree ZIPs via the JSON `zip_base64` field (the raw-zip path +rejects them with a pointer to `zip_base64` so the full tree is validated and +retained): + +```bash +curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ + -H 'content-type: application/json' \ + -d '{"miner_hotkey":"","zip_base64":""}' +``` + ## JSON (local / scripting) ```bash