Simplify memorization experiments: two runs and direct trajectory analysis

[charlesmartin14]

Requested change

Make the nanoGPT memorization experiment easy to run and analyze. Default to the staged verbatim comparison, not an 80-run campaign.

User commands

• Tonight: caffeinate -dimsu python study.py run --optimizer adamw
• Analyze: python study.py analyze
• Matched Muon later: caffeinate -dimsu python study.py run --optimizer muon --latest
• Both in a fresh timestamped study: caffeinate -dimsu python study.py run

The launcher creates readable timestamped directories under /tmp, streams and saves logs, performs real smoke checks, preserves failures and interruptions, and only resumes when explicitly requested. It does not change shell options or exit the user's shell. run_full.sh now delegates to this two-run workflow; the extended conditions remain opt-in in the unchanged low-level runner. The original extended README is preserved as PROTOCOL.md.

Analysis

Read saved JSON/CSV only, including partial or running studies. Export per-canary and dose/prefix trajectories, first observed exact matches, observed peaks and latest values, reconstructed presentation counts, same-step zero-dose contrasts, and per-matrix spectral plots. Keep raw/clipped alpha distinct. Compare optimizers only at common steps with matching seed/data/source/inventory/hardware. Label incomplete data and missing onset honestly; never use alpha as the definition of memorization. No model loading, retraining, or new WeightWatcher analysis in the reporting command.

Validation

Locally executed the combined tests: 43 passed in 4.91s (14 existing contract tests plus 29 new workflow/analysis cases). Real subprocess-return handling and plot-file generation are tested. New dedicated GitHub Actions workflow runs these tests and a dry run.

run.py, configs/suite.json, GPT, AdamW, Muon, WeightWatcher settings, and existing checkpoint fingerprints are unchanged. The source-hash tests verify this. No full training campaign, numerical WeightWatcher integration, or Mac/CUDA execution was performed during this update. This is not a fix for the previously reported nonfinite-gradient error; its cause remains undiagnosed.

The user explicitly requested creation and merge of this PR, followed by ordinary run instructions.

[charlesmartin14]

Pre-merge validation: all 43 local memorization component tests passed; study.py/analyze.py compile, run_full.sh parses, and the dry run schedules exactly two smoke checks plus two full verbatim runs (or one of each with --optimizer adamw). Uploaded study.py, analyze.py, and test_workflow.py blob hashes match the locally tested files.

GitHub's new Memorization workflow tests and Baseline source syntax workflows both passed. The older one-head-nanogpt job again reports 159 passed / 2 failed: test_single_matrix_wrapper_preserves_loader_metadata and test_notebook_displays_the_actual_results_variable in test_angular_rg_notebook_smoke.py. These are the same existing angular-notebook source assertions documented on PR #129. This PR changes neither their tests nor their notebooks; no existing checks were disabled. The core-baselines job was still running at this review.

The user explicitly requested PR creation and merge. This update changes orchestration, reporting, documentation, and dedicated workflow tests only. It does not change the training source/configuration or repair the reported nonfinite gradient. No new target-hardware training results are claimed.