diff --git a/README.md b/README.md index c8b0c4f..c8a4e37 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,14 @@ ## What it is -PRISM is a research challenge: you try **new architectures** and the challenge re-executes -them fairly. You submit **two Python scripts** — `architecture.py` (`build_model(ctx)`) and -`training.py` (`train(model, ctx)`) — and the operator runs them on a GPU pod against a -pinned FineWeb-Edu shard. Score is pure **bits-per-byte** (bpb, lower is better) measured -by the operator harness. There is **no** miner Docker image, no CVM, no on-chain write from -miners — HTTP submit only. +PRISM is a research challenge: you try **new architectures** (and optionally +training recipes / tokenizers) and the challenge re-executes them fairly. You +submit a ZIP — either the classic two scripts (`architecture.py` + +`training.py`) or a **source-tree** with helpers, `kernels/`, and optional +`tokenizer/` — and the operator runs them on a GPU pod against a pinned +FineWeb-Edu shard. Live leaf score is pure **bits-per-byte** (bpb, lower is +better); v3 also measures a G1–G8 battery in shadow mode. There is **no** miner +Docker image, no CVM, no on-chain write from miners — HTTP submit only. | | | |---|---| @@ -34,17 +36,18 @@ miners — HTTP submit only. | Production gateway | `https://chain.joinbase.ai` | | Staging gateway | `http://staging.api.joinbase.ai` | | Submit path | `/challenge/prism/v1/submissions` | -| Recipe | v1.2.0 — telemetry hooks required | +| Recipe | **v1.4.0** — miner-chosen tokenizer; G5 = RULER + BABILong + natural docs (**pretrain-only**) | -This repository holds **miner documentation and examples only**. Control-plane source -lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). +This repository holds **miner documentation and examples only**. Control-plane +source lives in [BaseIntelligence/base](https://github.com/BaseIntelligence/base). ## Start here -1. Read [Getting started](docs/getting-started.md). -2. Copy [`examples/baseline/`](examples/baseline/) — it shows the required telemetry - hooks (`prism_telemetry.report` + `finish_evaluation`). -3. Zip `architecture.py` + `training.py` and submit — see [Submit](docs/submit.md). +1. Read [Getting started](docs/getting-started.md) — tokenizer + source-tree + contracts matter from recipe **1.3.0 / 1.4.0**. +2. Copy [`examples/baseline/`](examples/baseline/) — required telemetry hooks + (`prism_telemetry.report` + `finish_evaluation`) and `ctx["tokenizer"]`. +3. Zip and submit — see [Submit](docs/submit.md). 4. Poll events until `terminated`, then check your bpb — see [API](docs/api.md). ```bash @@ -68,6 +71,8 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ 2. **Copying someone's `architecture.py`** — the pre-GPU copy gate rejects byte/AST copies of *earlier* architectures with zero score, no appeal. Starting from the published baseline is fine. -3. **Submitting again while gated** — one accepted architecture submission per hotkey; - a second one returns `409 submission_gated`. Training-only entries on published - architectures are separate slots (one per `(hotkey, arch_id)`). +3. **Hub downloads / hardcoded GPT-2** — the pod has **no network**. Use + `ctx["tokenizer"]` (and size embeddings from `ctx["vocab_size"]`). GPT-2 is + only the harness **fallback** when you declare nothing — not a challenge rule. + A second architecture submit while gated returns `409 submission_gated`; + training-only entries on published archs are separate slots. diff --git a/docs/README.md b/docs/README.md index 2dfa34a..7939579 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,12 +2,13 @@ | Page | What it covers | |------|----------------| -| [Getting started](getting-started.md) | Contract, hooks, dataset, budgets | -| [Submit](submit.md) | ZIP/JSON submit, gating, retries, training-only entries | -| [Scoring & competition](scoring.md) | bpb lattice, anti-copy, architecture competition, top-model | -| [API](api.md) | Routes, statuses, telemetry | +| [Getting started](getting-started.md) | Contract, tokenizer, source-tree, hooks, dataset, budgets | +| [Submit](submit.md) | ZIP/JSON/source-tree submit, gating, retries, training-only | +| [Scoring & competition](scoring.md) | bpb lattice, v3 G1–G8 (G5 pretrain-only), anti-copy, top-model | +| [API](api.md) | Routes, statuses, telemetry, Zone B / attribution | | [Troubleshooting](troubleshooting.md) | Common failures | -| [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks | +| [`examples/baseline/`](../examples/baseline/) | Reference recipe with hooks + `ctx["tokenizer"]` | -Normative sources (BASE monorepo): `docs/PRISM.md`, `docs/PRISM_RECIPE.md`, -`docs/external-miner/prism.md`. +**Recipe:** `1.4.0` (miner-chosen tokenizer; G5 = RULER + BABILong + natural +docs, pretrain-only). Normative freeze lives in the BASE monorepo: +`docs/PRISM.md`, `docs/PRISM_RECIPE.md`, `docs/external-miner/prism.md`. diff --git a/docs/api.md b/docs/api.md index 16c62e9..bb09a1f 100644 --- a/docs/api.md +++ b/docs/api.md @@ -9,16 +9,22 @@ Replace `{GATEWAY}` with `https://chain.joinbase.ai` (prod) or | Route | What it tells you | |-------|-------------------| -| `POST /challenge/prism/v1/submissions` | Submit zip / JSON / training-only | -| `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic records | +| `POST /challenge/prism/v1/submissions` | Submit zip / JSON / `zip_base64` source-tree / training-only | +| `GET /challenge/prism/v1/submissions/{id}` | Detail + bpb + review/similarity/agentic (+ composite block on v3) | | `GET /challenge/prism/v1/submissions/{id}/events` | Stage timeline | +| `GET /challenge/prism/v1/submissions/{id}/metrics?zone=a\|b` | Zone A battery / Zone B self-report chain (v3) | +| `POST /challenge/prism/v1/submissions/{id}/zone-b` | Miner Zone B self-report intake (v3) | +| `POST /challenge/prism/v1/submissions/{id}/attribution` | 2×2 arch/kernel attribution plans (v3, `kernels/` trees) | | `POST /challenge/prism/v1/submissions/{id}/retry` | Requeue an infra-failed row | | `GET /challenge/prism/v1/submissions?miner=` | Your submissions | | `GET /challenge/prism/v1/architectures` | Published archs + per-arch best bpb | +| `GET /challenge/prism/v1/anchors` | v3 anchor-set registry + status | +| `GET /challenge/prism/v1/preregistration` | v3 anchor pre-registration hash-commits | | `GET /challenge/prism/v1/recipe` | Versioned recipe descriptor + pin | | `GET /challenge/prism/v1/recipe/baseline` | Official baseline scripts | | `GET /challenge/prism/v1/status` | Backend / epoch / queues / recipe pin | | `GET /v1/site/arenas/prism/submissions/{id}/telemetry` | Loss curve / gradients / layer stats | +| `GET /health` | Liveness (direct challenge port; gateway may proxy) | ## Poll example @@ -42,7 +48,7 @@ Terminal states to know: | `rejected` | Pre-LLM copy gate: byte/AST copy of an *earlier* architecture (`Score(0)`, no GPU time, no LLM review) | | `failed` | Infra retries exhausted (`auto_retry` events) or harness/internal failure | | `terminated` with `score.kind = "no_score"` | `ChallengeInternal` — operator-side, never a miner zero | -| `terminated` with score 0 | Cheat / suspicious / copied verdict (see the `scoring` event detail) | +| `terminated` with score 0 | Cheat / suspicious / copied / `CAP_EXCEEDED` / missing telemetry (see the `scoring` event detail) | Submit errors: `403 hotkey_not_in_metagraph`, `404 unknown_arch`, `409 submission_gated`, `503 metagraph_unavailable`. diff --git a/docs/getting-started.md b/docs/getting-started.md index d53ca74..d19386f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,24 +1,91 @@ # Getting started -## The contract (recipe v1.2.0) +## The contract (recipe v1.4.0) -You ship **two scripts only**. The operator harness (`prism_harness.py`) imports them, -downloads the pinned dataset, verifies its SHA-256, times the run, and reports -`METRICS_JSON` (bpb, tokens, steps, wall clock, gpu, params). +You ship either: + +1. **Two scripts** — `architecture.py` (`build_model(ctx)`) + `training.py` + (`train(model, ctx)`), or +2. A **source-tree ZIP** (recipe ≥ 1.3.0) — those seams plus optional helpers, + `kernels/`, `tokenizer/`, `prism.toml`, `count_params.py`, `vendor.lock`. + +The operator harness imports your seams, downloads the pinned dataset, verifies +its SHA-256, times the run, and reports `METRICS_JSON` (bpb, `bits_per_byte`, +tokenizer spec, tokens, steps, wall clock, gpu, params — plus the v3 battery +when enabled). ```python # architecture.py def build_model(ctx): - """Return a model given the recipe context (devices, dims, seed).""" + """Return a model. Size embeddings from ctx["vocab_size"].""" + +# optional — must live beside build_model (not in training.py) +def build_tokenizer(ctx): + """Return your tokenizer (offline). See Tokenizer below.""" # training.py def train(model, ctx): - """Train the model; must respect ctx.budget(): - budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train).""" + """Train; respect ctx.budget(): + budget.max_steps <= 20000 and budget.max_seconds <= 21600 (6h train). + Use ctx["tokenizer"] — never from_pretrained("") on the pod.""" +``` + +Models must stay **≤ 350M parameters** after `build_model`. Since 1.3.0 a +breach is a **terminal Score(0)** (`CAP_EXCEEDED`), not a retryable failure. + +## Tokenizer (yours — recipe ≥ 1.4.0) + +**GPT-2 is no longer the challenge rule.** The harness resolves one tokenizer +per run and injects it as `ctx["tokenizer"]`, with vocab at `ctx["vocab_size"]`. +Declaration order (first match wins, always offline): + +| Order | How you declare | Notes | +|-------|-----------------|-------| +| 1 | `tokenizer/` in a source-tree ZIP | Staged under `submission/tokenizer/` on the pod; ≤ **12** files, ≤ **8 MiB** total | +| 2 | `build_tokenizer(ctx)` in `architecture.py` | Must sit beside `build_model` — a hook in `training.py` is rejected | +| 3 | *(declare nothing)* | Pinned `gpt2` **fallback** (pre-1.4 behavior) — a default, not a rule | + +```python +# architecture.py +def build_tokenizer(ctx): + """Anything offline: train a BPE on ctx["dataset_path"], wrap a vendored + implementation, or hand-roll a byte-level tokenizer. Must satisfy: + + tok(text, add_special_tokens=False)["input_ids"] -> list[int] + tok.decode(ids) -> str # roundtrips plain ASCII + len(tok) or tok.vocab_size -> int # 256 .. 262144 + tok.eos_token_id -> int | None + """ +``` + +Your pod has **no network** (`unshare --net`), so `from_pretrained("")` +inside your code fails closed. The harness validates the tokenizer and +fingerprints it; eval re-resolves it and refuses to score a mismatch — so +`build_tokenizer` must be deterministic. + +**Fairness.** Different vocabs change tokenization, not the unit — +`bits_per_byte` (bits over UTF-8 bytes) is the tokenizer-neutral anchor. The +legacy `bpb` key is bits per *token* and only comparable at equal tokenizers. + +## Source-tree submissions (recipe ≥ 1.3.0) + +Optional layout (flat or one shared top-level folder): + +```text +prism.toml # optional: entry = "train.py" +architecture.py # seam: build_model (+ optional build_tokenizer) +training.py # seam: train (or train.py) +count_params.py # optional +kernels/ # optional custom ops (pure Python + torch) +tokenizer/ # optional HF-style tokenizer files +vendor.lock # optional vendored *.py lock ``` -No third source file, no offline weights, no network at pod runtime beyond the pinned -dataset pull. +Caps (intake): ≤ **128** files, ≤ **4 MiB**/file, ≤ **16 MiB** total +uncompressed (≤ 8 MiB compressed). The validated tree is staged on the pod under +`submission/` so sibling imports (`import kernels`) and `tokenizer/` resolve. +Trees with `kernels/` are eligible for 2×2 **attribution** +(`POST /v1/submissions/{id}/attribution`). See [Submit](submit.md). ## Telemetry hooks (required since recipe 1.1.0) @@ -69,8 +136,10 @@ eval as `ChallengeInternal` — never a miner score. | Train wall clock | 6.0 h per submission | | Pod lifetime | 7.0 h (train + bootstrap margin) | | Hard step cap | 20 000 | -| Source size | 128 KiB per script | -| Model parameters | ≤ **350 000 000** after `build_model` | +| Two-script source size | 128 KiB per seam script | +| Source-tree | ≤ 128 files, ≤ 4 MiB/file, ≤ 16 MiB total | +| `tokenizer/` (in tree) | ≤ 12 files, ≤ 8 MiB total | +| Model parameters | ≤ **350 000 000** after `build_model` (`CAP_EXCEEDED` → Score(0)) | ## Recipe pin diff --git a/docs/scoring.md b/docs/scoring.md index aaaebec..5799e62 100644 --- a/docs/scoring.md +++ b/docs/scoring.md @@ -1,11 +1,16 @@ # Scoring & competition -## Pure bpb +## Pure bpb (live leaf) `final_score = score_from_bpb(measured_bpb)` on the integer lattice `[0, SCORE_MAX]` — lower bpb, higher score. The LLM reviews are **gates, not graders**: they verify the submission is coherent and not cheating; their quality notes never move the score. +**Fairness across tokenizers.** `bits_per_byte` (bits over UTF-8 bytes of the scored +region) is the tokenizer-neutral anchor reported in `METRICS_JSON`. The legacy `bpb` +key is bits per *token* and is only comparable across submissions that share a +tokenizer. + ## Anti-copy (architecture-only) - A **pre-LLM copy gate** compares your `architecture.py` against earlier submissions @@ -27,10 +32,54 @@ Per epoch, your emission is the **max** of: trainer** this epoch. Max, never summed — architecture owners are rewarded when *anyone* trains well on -their architecture. `Score(0)` rows (cheat / copy-gate) never set an arch's best. +their architecture. `Score(0)` rows (cheat / copy-gate / `CAP_EXCEEDED`) never set an +arch's best. Published architectures and their best bpb so far: `GET /v1/architectures`. +## v3 scoring (shadow-by-default) + +Recipe ≥ 1.3.0 harnesses run a **two-phase** pod flow: your code trains +(`phase=train`), checkpoints, then a fresh eval subprocess runs frozen-val bpb plus +the **G1–G8 battery** (intrinsic fit, commonsense/reading, retrieval/recall, +reasoning, long-context, sample efficiency, inference efficiency, training +stability/µP). Battery metrics are organizer-measured (**Zone A**, `org.*`) — your +code never emits them. + +While scoring mode is `shadow` (default), the **leaf score stays pure bpb**, +bit-identical to v2. After reference baselines are measured and anchors +pre-registered, governance may flip to `composite`. Inspect anchors at +`GET /v1/anchors` and `GET /v1/preregistration`; per-run rows at +`GET /v1/submissions/{id}/metrics?zone=a|b`. + +### G5 long-context (recipe ≥ 1.4.0 — pretrain-only) + +G5 scores a **base LM**, not an instruction-tuned chat model: completion-style / +few-shot base prompts, short exact-match or multiple-choice logprob — **no** chat +templates, free-form summarization, or LLM-as-judge on the ranked path. Length +targets are counted in tokens of **your** tokenizer (`ctx["tokenizer"]`). + +Scored keys (group weight 0.15 total): + +| Key | Weight | +|-----|--------| +| `org.g5.ruler_acc` | 0.35 | +| `org.g5.babilong_acc` | 0.25 | +| `org.g5.natural_mcq_acc` | 0.15 | +| `org.g5.helmet_rag_acc` | 0.15 | +| `org.g5.lstar` | 0.10 | + +**L\*** is the highest length where pooled RULER+BABILong accuracy stays ≥ 90% of the +shortest-grid accuracy and ≥ 0.25 (else 0). Natural MCQ / HELMET RAG packs are +mirrored like G2/G4. + +### Zone B (self-report, never scored) + +Your `train()` return dict (`train_metrics` in `METRICS_JSON` v2) is **Zone B**: +participant-reported, displayed-but-labelled, validated at ingest, and **never +scored**. Do not emit `org.*` keys. Optional out-of-band reports: +`POST /v1/submissions/{id}/zone-b`. + ## Top-model publish Whenever a new **global-best bpb** lands, the master publishes the winning @@ -56,6 +105,8 @@ uses the model as-is at that point, before any cap fires. Leaves per epoch feed the BASE gateway seal (`/v1/weights/latest`); prism's emission share is owner-controlled via the trust root. Miners never write on-chain weights. +Scores land in the leaf set emitted at the first chain-epoch boundary **after** your +run finalizes. ## Next diff --git a/docs/submit.md b/docs/submit.md index 0ea6583..f829ffc 100644 --- a/docs/submit.md +++ b/docs/submit.md @@ -2,7 +2,7 @@ Preferred path: a **ZIP** through the production or staging gateway. -## ZIP (preferred) +## ZIP (two-script, preferred for simple entries) | Header / body | Value | |---------------|--------| @@ -25,6 +25,37 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ --data-binary @submission.zip ``` +## Source-tree ZIP (recipe ≥ 1.3.0) + +Full trees (helpers, `kernels/`, `tokenizer/`, …) should go through the JSON +intake with `zip_base64` so the tree is validated and retained. Raw +`application/zip` accepts the classic two-script layout; a multi-file tree on +that path is rejected with a pointer to `zip_base64`. + +```bash +# pack the tree (paths relative to project root) +cd my-submission +zip -r ../tree.zip . -x '*.pyc' -x '__pycache__/*' -x '.git/*' + +python3 - <<'PY' +import base64, json, pathlib +raw = pathlib.Path("tree.zip").read_bytes() +print(json.dumps({ + "miner_hotkey": "<64 lowercase hex>", + "zip_base64": base64.b64encode(raw).decode(), + "label": "optional", +})) +PY + +curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ + -H 'content-type: application/json' \ + -d @submission.json +``` + +Caps: ≤ 128 files, ≤ 4 MiB/file, ≤ 16 MiB total uncompressed; `tokenizer/` ≤ 12 +files / ≤ 8 MiB. The validated tree is staged on the pod under `submission/`. +See [Getting started](getting-started.md#source-tree-submissions-recipe--130). + ## JSON (local / scripting) ```bash @@ -62,6 +93,7 @@ swapped), the watcher reopens your slot automatically. Training-only entries are **separate slots**: one accepted entry per `(hotkey, arch_id)` — you may train on many published architectures, one script per arch. +Training-only intake accepts the **two-script** layout only (not a full source tree). ```bash # JSON @@ -85,8 +117,8 @@ from the registry (miner-sent architecture is rejected on these rows). Unknown - Infra failures (pod provisioning, review/similarity/LLM infra) **auto-retry up to 3 times**. Retry budget exhausted → `failed`, slot `blocked`. -- Cheat / rejected verdicts are **terminal** — no auto-retry. Manual retry for - infra-class failures: `POST /v1/submissions/{id}/retry`. +- Cheat / rejected / `CAP_EXCEEDED` verdicts are **terminal** — no auto-retry. Manual + retry for infra-class failures: `POST /v1/submissions/{id}/retry`. ## Gateways @@ -97,6 +129,13 @@ from the registry (miner-sent architecture is rejected on these rows). Unknown Always use the `/challenge/prism/...` prefix on those hosts. +Inspect recipe pins before coding: + +```bash +curl -sS "$GATEWAY/challenge/prism/v1/recipe" +curl -sS "$GATEWAY/challenge/prism/v1/recipe/baseline" +``` + ## Next → [Scoring & competition](scoring.md) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0e8c5ed..fdbf932 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2,7 +2,11 @@ | Symptom | What to check | |---------|----------------| -| `400` on submit | Contract shape: `def build_model(` in `architecture.py`, `def train(` in `training.py`, ≤ 128 KiB per script | +| `400` on submit | Contract shape: `def build_model(` in `architecture.py`, `def train(` in `training.py`, size limits; source-tree caps (128 files / 4 MiB/file / 16 MiB); banned patterns (`ctypes`, prebuilt binaries, …) | +| Source-tree rejected on raw ZIP | Use JSON `zip_base64` so the full tree is validated and retained | +| Tokenizer / hub errors on pod | Pod has **no network** — use `ctx["tokenizer"]`; declare via `tokenizer/` or `build_tokenizer` in `architecture.py` (not `training.py`) | +| `build_tokenizer` ignored / rejected | Hook must live in `architecture.py` beside `build_model` | +| `CAP_EXCEEDED` / Score 0 | Model > 350M params after `build_model` — terminal, not retried | | `403 hotkey_not_in_metagraph` | Hotkey not registered on the subnet; check the hex (64 lowercase, no `0x`) | | `404 unknown_arch` | Training-only `arch_id` not in the registry — `GET /v1/architectures` | | `409 submission_gated` | You already have an accepted submission (1-max per hotkey, per `(hotkey, arch_id)` for training-only) | diff --git a/examples/baseline/README.md b/examples/baseline/README.md index b9b85e5..5921549 100644 --- a/examples/baseline/README.md +++ b/examples/baseline/README.md @@ -1,7 +1,13 @@ # PRISM baseline recipe (example) The official baseline submission — a tiny GPT-style causal transformer (~12M params) -plus an AdamW training loop that demonstrates the **required telemetry hooks**. +plus an AdamW training loop that demonstrates the **required telemetry hooks** and +the **`ctx["tokenizer"]`** contract (recipe ≥ 1.4.0). + +This baseline does **not** ship `build_tokenizer` or a `tokenizer/` directory — +the harness falls back to the pinned **gpt2** tokenizer. That is a baseline +*choice*, not a challenge rule. Competitive entries may train or vendor their +own tokenizer (see [getting started](../../docs/getting-started.md#tokenizer-yours--recipe--140)). Starting from this baseline is always allowed (the anti-copy gate exempts it). To be competitive, ship your **own** architecture — see @@ -11,8 +17,8 @@ competitive, ship your **own** architecture — see | File | Role | |------|------| -| `architecture.py` | `build_model(ctx)` → TinyGPT (tied embeddings, causal mask, block=512) | -| `training.py` | `train(model, ctx)` → AdamW loop with `prism_telemetry.report(...)` + `finish_evaluation()` | +| `architecture.py` | `build_model(ctx)` → TinyGPT sized from `ctx["vocab_size"]` | +| `training.py` | `train(model, ctx)` → AdamW loop with `ctx["tokenizer"]`, `prism_telemetry.report(...)` + `finish_evaluation()` | ## Hook pattern (required since recipe 1.1.0) @@ -41,6 +47,15 @@ Inside the operator harness the real module captures your series into in-memory model as-is. It raises a `BaseException` through `train()`, so it cannot be swallowed by your own `except Exception` blocks. +## Tokenizer pattern (recipe ≥ 1.4.0) + +On the pod, always use the harness-injected tokenizer: + +```python +tok = ctx["tokenizer"] # never from_pretrained("") — pod has no network +vocab = int(ctx["vocab_size"]) # size embeddings / head from this +``` + ## Submit ```bash @@ -52,3 +67,5 @@ curl -sS -X POST "$GATEWAY/challenge/prism/v1/submissions" \ ``` The same sources are always available live at `GET /v1/recipe/baseline`. +For a multi-file source tree (`kernels/`, `tokenizer/`, …), see +[Submit — source-tree ZIP](../../docs/submit.md#source-tree-zip-recipe--130). diff --git a/examples/baseline/architecture.py b/examples/baseline/architecture.py index 3792bd5..1df16a0 100644 --- a/examples/baseline/architecture.py +++ b/examples/baseline/architecture.py @@ -1,7 +1,12 @@ -"""PRISM baseline architecture (recipe v1). +"""PRISM baseline architecture (recipe v1.4.0). Tiny GPT-style causal transformer (~12M params) — the reference submission against which LLM similarity review calibrates "copied". + +This baseline does **not** declare `build_tokenizer`; the harness falls back +to the pinned gpt2 tokenizer. That is a baseline *choice*, not a challenge +rule — ship `tokenizer/` files and/or `build_tokenizer(ctx)` beside +`build_model` to use your own. """ import torch @@ -41,6 +46,11 @@ def forward(self, ids): def build_model(ctx): - """Recipe contract entrypoint. ctx carries device/seed/caps (unused here).""" + """Recipe contract entrypoint. ctx carries device/seed/caps. + + `ctx["vocab_size"]` is the vocab of the tokenizer the harness resolved for + this submission (recipe >= 1.4.0) — size embeddings from it rather than + assuming the pinned fallback's 50257. + """ torch.manual_seed(int(ctx.get("seed", 0))) - return TinyGPT() + return TinyGPT(vocab=int(ctx.get("vocab_size", 50257))) diff --git a/examples/baseline/training.py b/examples/baseline/training.py index 708ad7a..b8b8331 100644 --- a/examples/baseline/training.py +++ b/examples/baseline/training.py @@ -1,4 +1,4 @@ -"""PRISM baseline training (recipe v1). +"""PRISM baseline training (recipe v1.4.0). Reads `ctx["train_rows"]` texts from the pinned shard, runs AdamW at a modest LR for a small fixed step budget (baseline is deliberately mediocre: it is @@ -9,13 +9,19 @@ `report(loss=..., step=..., grad_norm=..., layer_stats=...)` periodically and `finish_evaluation()` to end the eval early — the harness scores the in-memory model either way. + +Tokenizer contract (recipe >= 1.4.0): use `ctx["tokenizer"]`. It is the +tokenizer *this submission* declared (`tokenizer/` files or a +`build_tokenizer(ctx)` hook) or the pinned gpt2 fallback when it declares +none, as here. Never `from_pretrained` a hub id yourself on the pod: the +training subprocess has no network, and the eval phase re-resolves the same +tokenizer from the submission. """ import time import pyarrow.parquet as pq import torch -from transformers import GPT2TokenizerFast try: import prism_telemetry @@ -39,16 +45,31 @@ def _texts(path, n): return xs[:n] +def _tokenizer(ctx): + """Prefer harness-injected `ctx["tokenizer"]`. + + Local/dev fallback loads gpt2 only when the harness is absent — that is + *this baseline's* tokenizer choice, not a challenge rule. On the pod the + harness always injects `ctx["tokenizer"]` (and has already warmed the + fallback cache when you declare none). + """ + tok = ctx.get("tokenizer") + if tok is None: + from transformers import GPT2TokenizerFast + + tok = GPT2TokenizerFast.from_pretrained("gpt2") + if getattr(tok, "pad_token", None) is None and getattr(tok, "eos_token", None) is not None: + tok.pad_token = tok.eos_token + return tok + + def train(model, ctx): """Recipe contract entrypoint: returns a metrics dict (val is harness-side).""" device = ctx["device"] torch.manual_seed(int(ctx["seed"])) guard = ctx.get("guard", lambda: None) - tok = GPT2TokenizerFast.from_pretrained("gpt2") - if tok.pad_token is None: - tok.pad_token = tok.eos_token - + tok = _tokenizer(ctx) texts = _texts(ctx["dataset_path"], int(ctx.get("train_rows", 2048))) block = model.block if hasattr(model, "block") else 512 @@ -70,8 +91,11 @@ def train(model, ctx): ids = enc.input_ids out = model(ids[:, :-1]) logits = out.logits if hasattr(out, "logits") else out + pad_id = getattr(tok, "pad_token_id", None) + if pad_id is None: + pad_id = -100 loss = torch.nn.functional.cross_entropy( - logits.reshape(-1, logits.shape[-1]), ids[:, 1:].reshape(-1), ignore_index=tok.pad_token_id + logits.reshape(-1, logits.shape[-1]), ids[:, 1:].reshape(-1), ignore_index=pad_id ) opt.zero_grad(set_to_none=True) loss.backward()