Skip to content

docs: sync miner guide to recipe 1.4.0 #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
echobt wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: sync miner guide to recipe 1.4.0 #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,30 +21,33 @@

## What it is

PRISM is a research challenge: you try **new architectures** and the challenge re-executes
them fairly. You submit **two Python scripts** — `architecture.py` (`build_model(ctx)`) and
`training.py` (`train(model, ctx)`) — and the operator runs them on a GPU pod against a
pinned FineWeb-Edu shard. Score is pure **bits-per-byte** (bpb, lower is better) measured
by the operator harness. There is **no** miner Docker image, no CVM, no on-chain write from
miners — HTTP submit only.
PRISM is a research challenge: you try **new architectures** (and optionally
training recipes / tokenizers) and the challenge re-executes them fairly. You
submit a ZIP — either the classic two scripts (`architecture.py` +
`training.py`) or a **source-tree** with helpers, `kernels/`, and optional
`tokenizer/` — and the operator runs them on a GPU pod against a pinned
FineWeb-Edu shard. Live leaf score is pure **bits-per-byte** (bpb, lower is
better); v3 also measures a G1–G8 battery in shadow mode. There is **no** miner
Docker image, no CVM, no on-chain write from miners — HTTP submit only.

| | |
|---|---|
| Challenge id | `prism` |
| Production gateway | `https://chain.joinbase.ai` |
| Staging gateway | `http://staging.api.joinbase.ai` |
| Submit path | `/challenge/prism/v1/submissions` |
| Recipe | v1.2.0 — telemetry hooks required |
| Recipe | **v1.4.0** — miner-chosen tokenizer; G5 = RULER + BABILong + natural docs (**pretrain-only**) |

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use exact names for the natural G5 packs.

“Natural docs” does not identify the two separate scored packs.

  • README.md#L39-L39: Name natural MCQ and HELMET RAG explicitly.
  • docs/README.md#L12-L14: Use the same exact G5 names in the recipe summary.
📍 Affects 2 files
  • README.md#L39-L39 (this comment)
  • docs/README.md#L12-L14
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` at line 39, Update the recipe summaries to replace the vague
“natural docs” label with the exact natural G5 pack names, “natural MCQ” and
“HELMET RAG”; apply this change in README.md lines 39-39 and docs/README.md
lines 12-14 so both summaries use identical naming.


This repository holds **miner documentation and examples only**. Control-plane source
lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base).
This repository holds **miner documentation and examples only**. Control-plane
source lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base).

## Start here

1. Read [Getting started](docs/getting-started.md).
2. Copy [`examples/baseline/`](examples/baseline/) — it shows the required telemetry
hooks (`prism_telemetry.report` + `finish_evaluation`).
3. Zip `architecture.py` + `training.py` and submit — see [Submit](docs/submit.md).
1. Read [Getting started](docs/getting-started.md) — tokenizer + source-tree
contracts matter from recipe **1.3.0 / 1.4.0**.
2. Copy [`examples/baseline/`](examples/baseline/) — required telemetry hooks
(`prism_telemetry.report` + `finish_evaluation`) and `ctx["tokenizer"]`.
3. Zip and submit — see [Submit](docs/submit.md).
Comment on lines +48 to +50

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Mark finish_evaluation() as optional.

prism_telemetry.report(...) is required. finish_evaluation() is an optional early-stop signal. Grouping both under “required telemetry hooks” gives miners an incorrect contract.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 48 - 50, Update the README instructions around the
baseline example so only prism_telemetry.report is described as required;
identify finish_evaluation as an optional early-stop signal while retaining the
ctx["tokenizer"] requirement.

4. Poll events until `terminated`, then check your bpb — see [API](docs/api.md).

```bash
Expand All @@ -68,6 +71,8 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
2. **Copying someone's `architecture.py`** — the pre-GPU copy gate rejects byte/AST
copies of *earlier* architectures with zero score, no appeal. Starting from the
published baseline is fine.
3. **Submitting again while gated** — one accepted architecture submission per hotkey;
a second one returns `409 submission_gated`. Training-only entries on published
architectures are separate slots (one per `(hotkey, arch_id)`).
3. **Hub downloads / hardcoded GPT-2** — the pod has **no network**. Use
`ctx["tokenizer"]` (and size embeddings from `ctx["vocab_size"]`). GPT-2 is
only the harness **fallback** when you declare nothing — not a challenge rule.
A second architecture submit while gated returns `409 submission_gated`;
training-only entries on published archs are separate slots.
15 changes: 8 additions & 7 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

| Page | What it covers |
|------|----------------|
| [Getting started](getting-started.md) | Contract, hooks, dataset, budgets |
| [Submit](submit.md) | ZIP/JSON submit, gating, retries, training-only entries |
| [Scoring & competition](scoring.md) | bpb lattice, anti-copy, architecture competition, top-model |
| [API](api.md) | Routes, statuses, telemetry |
| [Getting started](getting-started.md) | Contract, tokenizer, source-tree, hooks, dataset, budgets |
| [Submit](submit.md) | ZIP/JSON/source-tree submit, gating, retries, training-only |
| [Scoring & competition](scoring.md) | bpb lattice, v3 G1–G8 (G5 pretrain-only), anti-copy, top-model |
| [API](api.md) | Routes, statuses, telemetry, Zone B / attribution |
| [Troubleshooting](troubleshooting.md) | Common failures |
| [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks |
| [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks + `ctx["tokenizer"]` |

Normative sources (BASE monorepo): `docs/PRISM.md`, `docs/PRISM_RECIPE.md`,
`docs/external-miner/prism.md`.
**Recipe:** `1.4.0` (miner-chosen tokenizer; G5 = RULER + BABILong + natural
docs, pretrain-only). Normative freeze lives in the BASE monorepo:
`docs/PRISM.md`, `docs/PRISM_RECIPE.md`, `docs/external-miner/prism.md`.
12 changes: 9 additions & 3 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,22 @@ Replace `{GATEWAY}` with `https://chain.joinbase.ai` (prod) or

| Route | What it tells you |
|-------|-------------------|
| `POST /challenge/prism/v1/submissions` | Submit zip / JSON / training-only |
| `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic records |
| `POST /challenge/prism/v1/submissions` | Submit zip / JSON / `zip_base64` source-tree / training-only |
| `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic (+ composite block on v3) |
| `GET /challenge/prism/v1/submissions/{id}/events` | Stage timeline |
| `GET /challenge/prism/v1/submissions/{id}/metrics?zone=a\|b` | Zone A battery / Zone B self-report chain (v3) |
| `POST /challenge/prism/v1/submissions/{id}/zone-b` | Miner Zone B self-report intake (v3) |
| `POST /challenge/prism/v1/submissions/{id}/attribution` | 2×2 arch/kernel attribution plans (v3, `kernels/` trees) |
| `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row |
| `GET /challenge/prism/v1/submissions?miner=<hex>` | Your submissions |
| `GET /challenge/prism/v1/architectures` | Published archs + per-arch best bpb |
| `GET /challenge/prism/v1/anchors` | v3 anchor-set registry + status |
| `GET /challenge/prism/v1/preregistration` | v3 anchor pre-registration hash-commits |
| `GET /challenge/prism/v1/recipe` | Versioned recipe descriptor + pin |
| `GET /challenge/prism/v1/recipe/baseline` | Official baseline scripts |
| `GET /challenge/prism/v1/status` | Backend / epoch / queues / recipe pin |
| `GET /v1/site/arenas/prism/submissions/{id}/telemetry` | Loss curve / gradients / layer stats |
| `GET /health` | Liveness (direct challenge port; gateway may proxy) |

## Poll example

Expand All @@ -42,7 +48,7 @@ Terminal states to know:
| `rejected` | Pre-LLM copy gate: byte/AST copy of an *earlier* architecture (`Score(0)`, no GPU time, no LLM review) |
| `failed` | Infra retries exhausted (`auto_retry` events) or harness/internal failure |
| `terminated` with `score.kind = "no_score"` | `ChallengeInternal` — operator-side, never a miner zero |
| `terminated` with score 0 | Cheat / suspicious / copied verdict (see the `scoring` event detail) |
| `terminated` with score 0 | Cheat / suspicious / copied / `CAP_EXCEEDED` / missing telemetry (see the `scoring` event detail) |

Submit errors: `403 hotkey_not_in_metagraph`, `404 unknown_arch`,
`409 submission_gated`, `503 metagraph_unavailable`.
Expand Down
91 changes: 80 additions & 11 deletions docs/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,91 @@
# Getting started

## The contract (recipe v1.2.0)
## The contract (recipe v1.4.0)

You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them,
downloads the pinned dataset, verifies its SHA-256, times the run, and reports
`METRICS_JSON` (bpb, tokens, steps, wall clock, gpu, params).
You ship either:

1. **Two scripts** — `architecture.py` (`build_model(ctx)`) + `training.py`
(`train(model, ctx)`), or
2. A **source-tree ZIP** (recipe ≥ 1.3.0) — those seams plus optional helpers,
`kernels/`, `tokenizer/`, `prism.toml`, `count_params.py`, `vendor.lock`.

The operator harness imports your seams, downloads the pinned dataset, verifies
its SHA-256, times the run, and reports `METRICS_JSON` (bpb, `bits_per_byte`,
tokenizer spec, tokens, steps, wall clock, gpu, params — plus the v3 battery
when enabled).

```python
# architecture.py
def build_model(ctx):
"""Return a model given the recipe context (devices, dims, seed)."""
"""Return a model. Size embeddings from ctx["vocab_size"]."""

# optional — must live beside build_model (not in training.py)
def build_tokenizer(ctx):
"""Return your tokenizer (offline). See Tokenizer below."""

# training.py
def train(model, ctx):
"""Train the model; must respect ctx.budget():
budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train)."""
"""Train; respect ctx.budget():
budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train).
Use ctx["tokenizer"] — never from_pretrained("<hub id>") on the pod."""
```

Models must stay **≤ 350M parameters** after `build_model`. Since 1.3.0 a
breach is a **terminal Score(0)** (`CAP_EXCEEDED`), not a retryable failure.

## Tokenizer (yours — recipe ≥ 1.4.0)

**GPT-2 is no longer the challenge rule.** The harness resolves one tokenizer
per run and injects it as `ctx["tokenizer"]`, with vocab at `ctx["vocab_size"]`.
Declaration order (first match wins, always offline):

| Order | How you declare | Notes |
|-------|-----------------|-------|
| 1 | `tokenizer/` in a source-tree ZIP | Staged under `submission/tokenizer/` on the pod; ≤ **12** files, ≤ **8 MiB** total |
| 2 | `build_tokenizer(ctx)` in `architecture.py` | Must sit beside `build_model` — a hook in `training.py` is rejected |
| 3 | *(declare nothing)* | Pinned `gpt2` **fallback** (pre-1.4 behavior) — a default, not a rule |

```python
# architecture.py
def build_tokenizer(ctx):
"""Anything offline: train a BPE on ctx["dataset_path"], wrap a vendored
implementation, or hand-roll a byte-level tokenizer. Must satisfy:

tok(text, add_special_tokens=False)["input_ids"] -> list[int]
tok.decode(ids) -> str # roundtrips plain ASCII
len(tok) or tok.vocab_size -> int # 256 .. 262144
tok.eos_token_id -> int | None
"""
Comment on lines +48 to +58

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
rg -n -C 3 'def build_tokenizer|ctx\["tokenizer"\]|return_tensors|padding|input_ids|attention_mask|\.to\(device\)' \
  docs/getting-started.md examples/baseline/training.py

Repository: BaseIntelligence/prism

Length of output: 4343

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '--- docs/getting-started.md relevant section ---\n'
sed -n '1,90p' docs/getting-started.md
printf '\n--- examples/baseline/training.py relevant section ---\n'
sed -n '48,105p' examples/baseline/training.py

Repository: BaseIntelligence/prism

Length of output: 251

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- docs/getting-started.md relevant section ---'
sed -n '1,90p' docs/getting-started.md
printf '%s\n' ''
printf '%s\n' '--- examples/baseline/training.py relevant section ---'
sed -n '48,105p' examples/baseline/training.py

Repository: BaseIntelligence/prism

Length of output: 6191

Document the full baseline tokenizer contract.

docs/getting-started.md only specifies single-text tokenization, decode, vocab size, and eos_token_id. The baseline passes a list of texts and calls tok(batch_txt, return_tensors="pt", truncation=True, max_length=block, padding=True).to(device) before reading enc.input_ids. Either document and enforce the batch / BatchEncoding / padding contract, or change the baseline to use the documented minimum.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/getting-started.md` around lines 48 - 58, The tokenizer contract in
build_tokenizer must cover the baseline’s batch call and BatchEncoding behavior,
including batched text input, truncation, padding, return_tensors="pt",
.to(device), and enc.input_ids. Document and enforce these requirements, or
update the baseline to use only the currently documented single-text interface;
keep the chosen contract consistent throughout the baseline and getting-started
guide.

```

Your pod has **no network** (`unshare --net`), so `from_pretrained("<hub id>")`
inside your code fails closed. The harness validates the tokenizer and
fingerprints it; eval re-resolves it and refuses to score a mismatch — so
`build_tokenizer` must be deterministic.

**Fairness.** Different vocabs change tokenization, not the unit —
`bits_per_byte` (bits over UTF-8 bytes) is the tokenizer-neutral anchor. The
legacy `bpb` key is bits per *token* and only comparable at equal tokenizers.

## Source-tree submissions (recipe ≥ 1.3.0)

Optional layout (flat or one shared top-level folder):

```text
prism.toml # optional: entry = "train.py"
architecture.py # seam: build_model (+ optional build_tokenizer)
training.py # seam: train (or train.py)
count_params.py # optional
kernels/ # optional custom ops (pure Python + torch)
tokenizer/ # optional HF-style tokenizer files
vendor.lock # optional vendored *.py lock
```

No third source file, no offline weights, no network at pod runtime beyond the pinned
dataset pull.
Caps (intake): ≤ **128** files, ≤ **4 MiB**/file, ≤ **16 MiB** total
uncompressed (≤ 8 MiB compressed). The validated tree is staged on the pod under
`submission/` so sibling imports (`import kernels`) and `tokenizer/` resolve.
Trees with `kernels/` are eligible for 2×2 **attribution**
(`POST /v1/submissions/{id}/attribution`). See [Submit](submit.md).

## Telemetry hooks (required since recipe 1.1.0)

Expand Down Expand Up @@ -69,8 +136,10 @@ eval as `ChallengeInternal` — never a miner score.
| Train wall clock | 6.0 h per submission |
| Pod lifetime | 7.0 h (train + bootstrap margin) |
| Hard step cap | 20 000 |
| Source size | 128 KiB per script |
| Model parameters | ≤ **350 000 000** after `build_model` |
| Two-script source size | 128 KiB per seam script |
| Source-tree | ≤ 128 files, ≤ 4 MiB/file, ≤ 16 MiB total |
| `tokenizer/` (in tree) | ≤ 12 files, ≤ 8 MiB total |
| Model parameters | ≤ **350 000 000** after `build_model` (`CAP_EXCEEDED` → Score(0)) |

## Recipe pin

Expand Down
55 changes: 53 additions & 2 deletions docs/scoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
# Scoring & competition

## Pure bpb
## Pure bpb (live leaf)

`final_score = score_from_bpb(measured_bpb)` on the integer lattice `[0, SCORE_MAX]` —
lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the
submission is coherent and not cheating; their quality notes never move the score.

**Fairness across tokenizers.** `bits_per_byte` (bits over UTF-8 bytes of the scored
region) is the tokenizer-neutral anchor reported in `METRICS_JSON`. The legacy `bpb`
key is bits per *token* and is only comparable across submissions that share a
tokenizer.
Comment on lines +3 to +12

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Files:\n'
git ls-files | rg '(^|/)(README\.md|docs/(scoring|README|submit|troubling|.*\.md)$)' || true

printf '\nRelevant docs excerpts:\n'
sed -n '1,80p' README.md 2>/dev/null || true
printf '\n--- docs/scoring.md ---\n'
sed -n '1,60p' docs/scoring.md 2>/dev/null || true

printf '\nSearch for exact identifiers:\n'
rg -n "score_from_bpb|measured_bpb|bits_per_byte|bits-per-byte|bits per|METRICS_JSON|bpb" README.md docs 2>/dev/null || true

Repository: BaseIntelligence/prism

Length of output: 8656

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'docs/getting-started.md excerpts:\n'
sed -n '1,90p' docs/getting-started.md 2>/dev/null || true

printf '\nFull occurrences of relevant terms with filenames:\n'
rg -n -C 2 "measured_bpb|score_from_bpb|bits_per_byte|METRICS_JSON|bits-per-byte|bits per (token|byte)|bpb" README.md docs 2>/dev/null || true

printf '\nExact source-line probe for current claimed occurrences:\n'
python3 - <<'PY'
from pathlib import Path
files = ["README.md", "docs/scoring.md", "docs/getting-started.md"]
terms = ["measured_bpb", "score_from_bpb", "bits_per_byte", "METRICS_JSON", "bits-per-byte", "bits per token"]
for path in files:
    try:
        text = Path(path).read_text(encoding="utf-8")
    except FileNotFoundError as e:
        print(f"{path}: MISSING\n")
        continue
    lines = text.splitlines()
    print(f"\n--- {path} ({len(lines)} lines) ---")
    hits, term_hits = [], {t: [] for t in terms}
    for i, line in enumerate(lines, 1):
        if any(t in line for t in terms):
            hits.append(i)
        for t in terms:
            if t in line:
                term_hits[t].append(i)
    print("term hits:", {t: hits_ for t, hits_ in term_hits.items() if hits_})
    for i in (1,29,30,39,43,50,13,63,67,68) if path=="docs/getting-started.md" else (1,5,6,9,11):
        if 1 <= i <= len(lines):
            print(f"{i}: {lines[i-1]}")
PY

Repository: BaseIntelligence/prism

Length of output: 11122

Use one canonical name for the live-score input.

docs/scoring.md introduces measured_bpb without saying it equals the tokenizer-neutral bits_per_byte value, and README.md says live score is “bits-per-byte (bpb)” without distinguishing the legacy per-token bpb key. Define whether measured_bpb is bits_per_byte and replace the README equivalence with the canonical metric name.

📍 Affects 2 files
  • docs/scoring.md#L3-L12 (this comment)
  • README.md#L29-L30
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/scoring.md` around lines 3 - 12, In docs/scoring.md, explicitly define
measured_bpb as the tokenizer-neutral bits_per_byte value used for live scoring,
while distinguishing it from the legacy per-token bpb key. In README.md lines
29-30, replace the “bits-per-byte (bpb)” wording with the canonical
measured_bpb/bits_per_byte name and preserve the distinction from legacy bpb.


## Anti-copy (architecture-only)

- A **pre-LLM copy gate** compares your `architecture.py` against earlier submissions
Expand All @@ -27,10 +32,54 @@ Per epoch, your emission is the **max** of:
trainer** this epoch.

Max, never summed — architecture owners are rewarded when *anyone* trains well on
their architecture. `Score(0)` rows (cheat / copy-gate) never set an arch's best.
their architecture. `Score(0)` rows (cheat / copy-gate / `CAP_EXCEEDED`) never set an
arch's best.

Published architectures and their best bpb so far: `GET /v1/architectures`.

## v3 scoring (shadow-by-default)

Recipe ≥ 1.3.0 harnesses run a **two-phase** pod flow: your code trains
(`phase=train`), checkpoints, then a fresh eval subprocess runs frozen-val bpb plus
the **G1–G8 battery** (intrinsic fit, commonsense/reading, retrieval/recall,
reasoning, long-context, sample efficiency, inference efficiency, training
stability/µP). Battery metrics are organizer-measured (**Zone A**, `org.*`) — your
code never emits them.

While scoring mode is `shadow` (default), the **leaf score stays pure bpb**,
bit-identical to v2. After reference baselines are measured and anchors
pre-registered, governance may flip to `composite`. Inspect anchors at
`GET /v1/anchors` and `GET /v1/preregistration`; per-run rows at
`GET /v1/submissions/{id}/metrics?zone=a|b`.

### G5 long-context (recipe ≥ 1.4.0 — pretrain-only)

G5 scores a **base LM**, not an instruction-tuned chat model: completion-style /
few-shot base prompts, short exact-match or multiple-choice logprob — **no** chat
templates, free-form summarization, or LLM-as-judge on the ranked path. Length
targets are counted in tokens of **your** tokenizer (`ctx["tokenizer"]`).

Scored keys (group weight 0.15 total):

| Key | Weight |
|-----|--------|
| `org.g5.ruler_acc` | 0.35 |
| `org.g5.babilong_acc` | 0.25 |
| `org.g5.natural_mcq_acc` | 0.15 |
| `org.g5.helmet_rag_acc` | 0.15 |
| `org.g5.lstar` | 0.10 |

**L\*** is the highest length where pooled RULER+BABILong accuracy stays ≥ 90% of the
shortest-grid accuracy and ≥ 0.25 (else 0). Natural MCQ / HELMET RAG packs are
mirrored like G2/G4.

### Zone B (self-report, never scored)

Your `train()` return dict (`train_metrics` in `METRICS_JSON` v2) is **Zone B**:
participant-reported, displayed-but-labelled, validated at ingest, and **never
scored**. Do not emit `org.*` keys. Optional out-of-band reports:
`POST /v1/submissions/{id}/zone-b`.

## Top-model publish

Whenever a new **global-best bpb** lands, the master publishes the winning
Expand All @@ -56,6 +105,8 @@ uses the model as-is at that point, before any cap fires.

Leaves per epoch feed the BASE gateway seal (`/v1/weights/latest`); prism's emission
share is owner-controlled via the trust root. Miners never write on-chain weights.
Scores land in the leaf set emitted at the first chain-epoch boundary **after** your
run finalizes.

## Next

Expand Down
45 changes: 42 additions & 3 deletions docs/submit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Preferred path: a **ZIP** through the production or staging gateway.

## ZIP (preferred)
## ZIP (two-script, preferred for simple entries)

| Header / body | Value |
|---------------|--------|
Expand All @@ -25,6 +25,37 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
--data-binary @submission.zip
```

## Source-tree ZIP (recipe ≥ 1.3.0)

Full trees (helpers, `kernels/`, `tokenizer/`, …) should go through the JSON
intake with `zip_base64` so the tree is validated and retained. Raw
`application/zip` accepts the classic two-script layout; a multi-file tree on
that path is rejected with a pointer to `zip_base64`.

```bash
# pack the tree (paths relative to project root)
cd my-submission
zip -r ../tree.zip . -x '*.pyc' -x '__pycache__/*' -x '.git/*'

python3 - <<'PY'
import base64, json, pathlib
raw = pathlib.Path("tree.zip").read_bytes()
print(json.dumps({
"miner_hotkey": "<64 lowercase hex>",
"zip_base64": base64.b64encode(raw).decode(),
"label": "optional",
}))
PY

curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
-H 'content-type: application/json' \
-d @submission.json
Comment on lines +35 to +52

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Fix the source-tree packaging example before publishing it.

After cd my-submission, the ZIP is written to ../tree.zip, but the Python snippet reads tree.zip from my-submission. The snippet also prints JSON to stdout, while the next command reads submission.json, which the snippet never creates.

Proposed fix 
-python3 - <<'PY'
+python3 - <<'PY' > submission.json
 import base64, json, pathlib
-raw = pathlib.Path("tree.zip").read_bytes()
+raw = pathlib.Path("../tree.zip").read_bytes()
 print(json.dumps({
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```bash
# pack the tree (paths relative to project root)
cd my-submission
zip -r ../tree.zip . -x '*.pyc' -x '__pycache__/*' -x '.git/*'
python3 - <<'PY'
import base64, json, pathlib
raw = pathlib.Path("tree.zip").read_bytes()
print(json.dumps({
"miner_hotkey": "<64 lowercase hex>",
"zip_base64": base64.b64encode(raw).decode(),
"label": "optional",
}))
PY
curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \
-H 'content-type: application/json' \
-d @submission.json
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/submit.md` around lines 35 - 52, Update the packaging example so the
Python snippet reads the ZIP from the path actually created by the preceding zip
command, and write its JSON output to submission.json for the subsequent curl
command instead of stdout. Keep the existing archive contents, payload fields,
and submission endpoint unchanged.

```

Caps: ≤ 128 files, ≤ 4 MiB/file, ≤ 16 MiB total uncompressed; `tokenizer/` ≤ 12
files / ≤ 8 MiB. The validated tree is staged on the pod under `submission/`.
Comment on lines +55 to +56

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Keep all intake cap summaries complete.

The normative cap table includes an 8 MiB compressed ZIP limit, but these intake summaries omit it.

  • docs/submit.md#L55-L56: Add ≤ 8 MiB compressed to the source-tree caps.
  • docs/troubleshooting.md#L5-L5: Mirror the compressed limit and tokenizer-directory sub-cap in the 400-error checklist.
📍 Affects 2 files
  • docs/submit.md#L55-L56 (this comment)
  • docs/troubleshooting.md#L5-L5
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/submit.md` around lines 55 - 56, Add the normative ≤ 8 MiB compressed
ZIP limit to the source-tree cap summary in docs/submit.md:55-56. Also update
the 400-error checklist in docs/troubleshooting.md:5 to include both the
compressed limit and the tokenizer/ sub-cap, preserving the existing intake
limits.

See [Getting started](getting-started.md#source-tree-submissions-recipe--130).

## JSON (local / scripting)

```bash
Expand Down Expand Up @@ -62,6 +93,7 @@ swapped), the watcher reopens your slot automatically.

Training-only entries are **separate slots**: one accepted entry per `(hotkey, arch_id)`
— you may train on many published architectures, one script per arch.
Training-only intake accepts the **two-script** layout only (not a full source tree).

@coderabbitai coderabbitai Bot Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Correct the training-only layout description.

Training-only intake sends training.py with arch_id; the architecture is pulled from the registry. “Two-script layout only” conflicts with the table and the next paragraph. Describe this as the legacy training seam, not as a two-script payload.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/submit.md` at line 96, Update the training-only intake description near
the two-script layout statement to identify it as the legacy training seam:
training.py is submitted with arch_id, while the architecture is retrieved from
the registry. Remove the claim that the payload is limited to a two-script
layout and keep the surrounding table and follow-up paragraph consistent.


```bash
# JSON
Expand All @@ -85,8 +117,8 @@ from the registry (miner-sent architecture is rejected on these rows). Unknown

- Infra failures (pod provisioning, review/similarity/LLM infra) **auto-retry up to 3
times**. Retry budget exhausted → `failed`, slot `blocked`.
- Cheat / rejected verdicts are **terminal** — no auto-retry. Manual retry for
infra-class failures: `POST /v1/submissions/{id}/retry`.
- Cheat / rejected / `CAP_EXCEEDED` verdicts are **terminal** — no auto-retry. Manual
retry for infra-class failures: `POST /v1/submissions/{id}/retry`.

## Gateways

Expand All @@ -97,6 +129,13 @@ from the registry (miner-sent architecture is rejected on these rows). Unknown

Always use the `/challenge/prism/...` prefix on those hosts.

Inspect recipe pins before coding:

```bash
curl -sS "$GATEWAY/challenge/prism/v1/recipe"
curl -sS "$GATEWAY/challenge/prism/v1/recipe/baseline"
```

## Next

→ [Scoring & competition](scoring.md)
Expand Down
Loading
Loading