Joshua Shay Kricheli*, Alexander Lawrence Reid*, Soumajyoti Sarkar, Venkata Gandikota, Paulo Shakarian
* equal contribution
- What is this?
- Installation
- Quick start
- Repository structure
- Full pipeline
- Paper item to command
- Known behaviours
- Testing
- Citing
- Acknowledgments
The code and data pipeline behind the NeurIPS 2026 paper Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation.
Scaling laws are fitted on small models and used to predict large ones. Whether such a fit extrapolates depends on the tokens-per-parameter (TPP) ratios D/N that the training runs cover:
| Collinear (CO) design | Non-collinear (NC) design |
|---|---|
| Grows N and D together along a few TPP rays | Spreads the same number of runs over TPP |
| The law's parameters are badly conditioned | The parameters are well conditioned |
| Extrapolates poorly | Wins 97.3% of the extrapolation comparisons |
We test this on about 1,900 trained models over five corpora and four scaling laws, and give a criterion, computable before any training, that tells which designs are well conditioned.
All the data the analysis needs is public on HuggingFace and needs no account:
leibnitz-lab/colinear_scaling_models- training metrics, checkpoints and pre-computed outputs of every trained model;leibnitz-lab/public_checkpoint_reanalysis- run tables of the ten published studies re-analysed in the camera-ready version.
uv is the only tool needed. One locked environment (Python 3.10) runs the analysis, the training and the tests.
git clone https://github.com/lab-v2/tpp-coverage.git && cd tpp-coverage
uv syncReproduce the paper's tables and figures from the public data, no GPU:
uv run python src/tpp_analysis/download_and_extract.py --results-only
uv run python src/tpp_analysis/run_analysis.py all --data-dir extracted_losses/ --n-workers "$(nproc)"
uv run python src/tpp_analysis/generate_appendix_tables.py full_coverage_results/
uv run bash src/tpp_analysis/visualizations/run_appendix_figures.shAdd --clear-cache to run_analysis.py to refit every law from the loss tables; the refit
reproduces the released results bit for bit (about 20 minutes on 50 cores).
%%{init: {"theme": "base", "flowchart": {"curve": "basis", "nodeSpacing": 30, "rankSpacing": 50, "padding": 14}, "themeVariables": {"fontFamily": "Inter, -apple-system, Segoe UI, Helvetica, sans-serif", "fontSize": "15px", "lineColor": "#94A3B8", "clusterBkg": "transparent", "clusterBorder": "#94A3B8", "titleColor": "#94A3B8", "edgeLabelBackground": "transparent"}}}%%
flowchart TB
subgraph TRAIN["TRAIN · GPU"]
direction LR
A("Download<br/>5 corpora") --> B("Generate the<br/>N × D grid") --> C("Train<br/>1,900 models")
end
subgraph ANALYSE["ANALYSE · CPU"]
direction LR
D("Extract<br/>losses") --> E("Fit 4<br/>scaling laws") --> F("Tables<br/>and figures")
end
H[("Public data<br/>on HuggingFace")]
G("Camera-ready<br/>analyses")
TRAIN ==> ANALYSE
H -.-> ANALYSE
H -.-> G
classDef train fill:#F76900,stroke:#FFB27A,stroke-width:1px,color:#FFFFFF
classDef analyse fill:#1F2A5C,stroke:#7C8DD6,stroke-width:1px,color:#FFFFFF
classDef data fill:#334155,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
classDef extra fill:#475569,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
class A,B,C train
class D,E,F analyse
class H data
class G extra
Every folder below has one job; a linked folder has its own README.
| Folder | What it holds |
|---|---|
src/ |
The importable Python packages |
src/tpp_analysis/ |
The paper's analysis: corpora, job generation, loss extraction, law fits, tables and figures |
src/training_models/ |
Llama-style models and the distributed trainer |
src/tpp_large_scale/ |
The analysis package of large_scale/ |
camera_ready/ |
The two analyses added for the camera-ready version |
camera_ready/grid_classification/ |
The paper's criterion applied to 24 published scaling-law designs |
camera_ready/public_checkpoints/ |
The reanalysis of ten public training-loss releases |
large_scale/ |
The paper's analysis rerun on an independent grid of 461 C4 runs up to 354M parameters |
cluster/ |
A larger Wikipedia stratum (to 380.6M parameters) trained on the paper's protocol |
cluster/queue/ |
The SLURM task queue that trained it |
cluster/wiki300m/ |
Its task generator and the queue of trained cells |
cluster/audit/ |
Its re-analyses and their result tables |
docs/ |
Step-by-step guides: downloading, training, analysis, figures, n-gram KL, and the known behaviours |
slides/ |
The five-minute NeurIPS 2026 talk (Beamer source and PDF) |
website/ |
The project page at lab-v2.github.io/tpp-coverage |
tests/ |
The pytest suite, one folder per component |
.github/ |
CI on the project's own runners: tests, ruff check ., the privacy scan and the merge rule |
images/ |
The logos shown in this README |
The experiment folders (camera_ready/*, large_scale/, cluster/) share one layout: a
README saying what the folder holds and how to run it, the scripts, and the committed outputs
in a results folder (the grid classification keeps its results.csv beside its scripts,
where its pre-registered protocol names it). Their tests sit in tests/<folder>/.
| Step | Command | Guide |
|---|---|---|
| 1. Download and tokenize the corpora | src/tpp_analysis/dataset_downloaders/download_{wiki,cosmopedia,redpj,peso,c4}.py |
TRAINING |
| 2. Write the jobs of the trained grid | src/tpp_analysis/generate_jobs/generate_jobs.py --dataset <name> |
TRAINING |
| 3. Train (4 GPUs per run) | src/training_models/training/dist_trainer_main.py under torchrun |
TRAINING |
| 4. Extract per-epoch losses | src/tpp_analysis/get_results/get_results.py from_local <metrics dir> |
DOWNLOADING |
| 5. Fit the laws, write the tables | run_analysis.py, run_special_analysis.py, run_prelim_bigtpp.py, generate_appendix_tables.py |
ANALYSIS |
| 6. Figures | run_appendix_figures.sh, notebooks in visualizations/ |
VISUALIZATIONS |
Steps 1-3 need GPUs (about 3,600 GPU-hours in total). Step 4 can start from the released
metrics: download_and_extract.py without --results-only fetches them (~0.5 GB).
Paths are under src/tpp_analysis/ unless noted.
| Paper item | Command |
|---|---|
| Main win-rate table (97.3%) and R2/RMSE summary | run_analysis.py full_coverage |
| Appendix metrics, fitted parameters, win rate by holdout split | generate_appendix_tables.py full_coverage_results/ |
| BF16 replication (98.7%) | run_special_analysis.py, then generate_appendix_tables.py special_results/ --special |
| BigTPP preliminary (63.3%) | run_prelim_bigtpp.py |
| Regime-A subset table | dispatch_seeds.sh 0 21, subset_analysis/aggregate_seed_runs.py, subset_analysis/per_law_winrate_ci.py --mode first |
| Figure 1 and appendix three-panel figures | run_analysis.py seed_variance, then visualizations/run_appendix_figures.sh |
| Architecture table, n-layers figure | utility/audit_architectures.py, then utility/plot_nlayers_distribution.py --mode runs |
| n-gram KL table | ngram_analysis/run_all_ngrams.sh (needs the tokenized corpora) |
| Published-design classification (24 designs) | camera_ready/grid_classification/classify.py |
| Public-checkpoint reanalysis | camera_ready/public_checkpoints/ |
The theorems and the severity, eigenvalue and R_min tables are analytic; no script computes them.
Each of these changes a reproduced number when it is missed. What each one changes, what it leaves alone and what to do: Known behaviours.
- Padding was scored. The released losses include padded positions, so they sit below the
loss on real tokens; the win rates and conditioning comparisons are unaffected. The trainer
now masks padding;
--score-paddingreproduces a released run. - BF16 is a full cast.
--use-bf16callsmodel.bfloat16(), so weights, gradients and AdamW states are all BF16; only the logits are cast to FP32 for the loss. - Four GPUs per run. The GPU count sets the global batch and the step count the learning-rate schedule follows, so another count trains a different model.
- One training seed. Every run of the paper uses seed 42; the analysis seeds are fitting restarts and subset draws, so the intervals measure fitting variance.
- Two releases are incomplete. C4 has 99% of its runs (321 of 324, three NC holdout runs missing) and the BF16 replication 93% (302 of 324); a retrain matches the paper only without the missing runs.
- Sizes are targets.
--model-sizetrains the shape its search finds: up to 8.8% larger for the paper's targets and 13% for a 150M target. The analysis uses the realized count. - Run names set the role. A run whose name lacks
NxD_GRID,COLINEARandHOLDOUTis analysed as NC training. - Fits are cached by name.
run_analysis.pyre-renders stored fits; pass--clear-cacheafter changing the loss tables. - Scale pairs can sit at a start draw. In 183 of 900 Kaplan and 510 of 900 Droppo-Elibol fits the optimizer left (N_c, D_c) at a start draw, so do not reuse those pairs as estimates.
- Bitwise on one driver. A CUDA driver update moved the training loss by O(1e-3); compare
a retrained run with
--atol 1e-2.
uv run pytest -n autoThe suite checks law recovery on synthetic data, every entry point's --help, that the
released tables are reproduced, the camera-ready analyses, the README's links, images,
diagram and citation, and that no private host or path is in the tree.
CI runs the suite, ruff check . and the privacy scan on every pull request and every push
to main. A pull request may merge only when its author is a lab-v2 member, or when a
member approved its latest commit (the pr-authorization check).
If you use this work, please cite our paper:
@article{kricheli2026tpp,
title = {Tokens-per-Parameter Coverage Is Critical for Robust {LLM} Scaling Law Extrapolation},
author = {Kricheli, Joshua Shay and Reid, Alexander Lawrence and Sarkar, Soumajyoti and Gandikota, Venkata and Shakarian, Paulo},
journal = {arXiv preprint arXiv:2605.08541},
year = {2026},
eprint = {2605.08541},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2605.08541},
note = {Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)}
}This work was done at the Leibniz Lab, Department of Electrical Engineering and Computer
Science, Syracuse University. This research was funded by ARO grant W911NF-24-1-0007.
Separately, this work used resources at NCSA through allocation CIS230318 from the Advanced
Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program
(Boerner et al., 2023), which is supported by U.S.
National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
Three AMD EPYC CPU chips, donated by AMD, were used for the experiments. This research was
supported in part through computational resources provided by Syracuse University (NSF award
ACI-1341006). The code is released under the GNU
General Public License v2 (LICENSE).
