Skip to content

About

Code for the NeurIPS 2026 paper 'Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation'

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Tokens-per-Parameter Coverage Is Critical for
Robust LLM Scaling Law Extrapolation

arXiv Project page NeurIPS 2026 Data on HuggingFace License: GPL-2.0 Python 3.10 uv Ruff

Joshua Shay Kricheli*, Alexander Lawrence Reid*, Soumajyoti Sarkar, Venkata Gandikota, Paulo Shakarian
* equal contribution


What is this?

The code and data pipeline behind the NeurIPS 2026 paper Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation.

Scaling laws are fitted on small models and used to predict large ones. Whether such a fit extrapolates depends on the tokens-per-parameter (TPP) ratios D/N that the training runs cover:

Collinear (CO) design Non-collinear (NC) design
Grows N and D together along a few TPP rays Spreads the same number of runs over TPP
The law's parameters are badly conditioned The parameters are well conditioned
Extrapolates poorly Wins 97.3% of the extrapolation comparisons

We test this on about 1,900 trained models over five corpora and four scaling laws, and give a criterion, computable before any training, that tells which designs are well conditioned.

All the data the analysis needs is public on HuggingFace and needs no account:

Installation

uv is the only tool needed. One locked environment (Python 3.10) runs the analysis, the training and the tests.

git clone https://github.com/lab-v2/tpp-coverage.git && cd tpp-coverage
uv sync

Quick start

Reproduce the paper's tables and figures from the public data, no GPU:

uv run python src/tpp_analysis/download_and_extract.py --results-only
uv run python src/tpp_analysis/run_analysis.py all --data-dir extracted_losses/ --n-workers "$(nproc)"
uv run python src/tpp_analysis/generate_appendix_tables.py full_coverage_results/
uv run bash src/tpp_analysis/visualizations/run_appendix_figures.sh

Add --clear-cache to run_analysis.py to refit every law from the loss tables; the refit reproduces the released results bit for bit (about 20 minutes on 50 cores).

Repository structure

%%{init: {"theme": "base", "flowchart": {"curve": "basis", "nodeSpacing": 30, "rankSpacing": 50, "padding": 14}, "themeVariables": {"fontFamily": "Inter, -apple-system, Segoe UI, Helvetica, sans-serif", "fontSize": "15px", "lineColor": "#94A3B8", "clusterBkg": "transparent", "clusterBorder": "#94A3B8", "titleColor": "#94A3B8", "edgeLabelBackground": "transparent"}}}%%
flowchart TB
    subgraph TRAIN["TRAIN  ·  GPU"]
        direction LR
        A("Download<br/>5 corpora") --> B("Generate the<br/>N × D grid") --> C("Train<br/>1,900 models")
    end
    subgraph ANALYSE["ANALYSE &nbsp;·&nbsp; CPU"]
        direction LR
        D("Extract<br/>losses") --> E("Fit 4<br/>scaling laws") --> F("Tables<br/>and figures")
    end
    H[("Public data<br/>on HuggingFace")]
    G("Camera-ready<br/>analyses")
    TRAIN ==> ANALYSE
    H -.-> ANALYSE
    H -.-> G
    classDef train fill:#F76900,stroke:#FFB27A,stroke-width:1px,color:#FFFFFF
    classDef analyse fill:#1F2A5C,stroke:#7C8DD6,stroke-width:1px,color:#FFFFFF
    classDef data fill:#334155,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
    classDef extra fill:#475569,stroke:#94A3B8,stroke-width:1px,color:#FFFFFF
    class A,B,C train
    class D,E,F analyse
    class H data
    class G extra
Loading

Layout

Every folder below has one job; a linked folder has its own README.

Folder What it holds
src/ The importable Python packages
src/tpp_analysis/ The paper's analysis: corpora, job generation, loss extraction, law fits, tables and figures
src/training_models/ Llama-style models and the distributed trainer
src/tpp_large_scale/ The analysis package of large_scale/
camera_ready/ The two analyses added for the camera-ready version
camera_ready/grid_classification/ The paper's criterion applied to 24 published scaling-law designs
camera_ready/public_checkpoints/ The reanalysis of ten public training-loss releases
large_scale/ The paper's analysis rerun on an independent grid of 461 C4 runs up to 354M parameters
cluster/ A larger Wikipedia stratum (to 380.6M parameters) trained on the paper's protocol
cluster/queue/ The SLURM task queue that trained it
cluster/wiki300m/ Its task generator and the queue of trained cells
cluster/audit/ Its re-analyses and their result tables
docs/ Step-by-step guides: downloading, training, analysis, figures, n-gram KL, and the known behaviours
slides/ The five-minute NeurIPS 2026 talk (Beamer source and PDF)
website/ The project page at lab-v2.github.io/tpp-coverage
tests/ The pytest suite, one folder per component
.github/ CI on the project's own runners: tests, ruff check ., the privacy scan and the merge rule
images/ The logos shown in this README

The experiment folders (camera_ready/*, large_scale/, cluster/) share one layout: a README saying what the folder holds and how to run it, the scripts, and the committed outputs in a results folder (the grid classification keeps its results.csv beside its scripts, where its pre-registered protocol names it). Their tests sit in tests/<folder>/.

Full pipeline

Step Command Guide
1. Download and tokenize the corpora src/tpp_analysis/dataset_downloaders/download_{wiki,cosmopedia,redpj,peso,c4}.py TRAINING
2. Write the jobs of the trained grid src/tpp_analysis/generate_jobs/generate_jobs.py --dataset <name> TRAINING
3. Train (4 GPUs per run) src/training_models/training/dist_trainer_main.py under torchrun TRAINING
4. Extract per-epoch losses src/tpp_analysis/get_results/get_results.py from_local <metrics dir> DOWNLOADING
5. Fit the laws, write the tables run_analysis.py, run_special_analysis.py, run_prelim_bigtpp.py, generate_appendix_tables.py ANALYSIS
6. Figures run_appendix_figures.sh, notebooks in visualizations/ VISUALIZATIONS

Steps 1-3 need GPUs (about 3,600 GPU-hours in total). Step 4 can start from the released metrics: download_and_extract.py without --results-only fetches them (~0.5 GB).

Paper item to command

Paths are under src/tpp_analysis/ unless noted.

Paper item Command
Main win-rate table (97.3%) and R2/RMSE summary run_analysis.py full_coverage
Appendix metrics, fitted parameters, win rate by holdout split generate_appendix_tables.py full_coverage_results/
BF16 replication (98.7%) run_special_analysis.py, then generate_appendix_tables.py special_results/ --special
BigTPP preliminary (63.3%) run_prelim_bigtpp.py
Regime-A subset table dispatch_seeds.sh 0 21, subset_analysis/aggregate_seed_runs.py, subset_analysis/per_law_winrate_ci.py --mode first
Figure 1 and appendix three-panel figures run_analysis.py seed_variance, then visualizations/run_appendix_figures.sh
Architecture table, n-layers figure utility/audit_architectures.py, then utility/plot_nlayers_distribution.py --mode runs
n-gram KL table ngram_analysis/run_all_ngrams.sh (needs the tokenized corpora)
Published-design classification (24 designs) camera_ready/grid_classification/classify.py
Public-checkpoint reanalysis camera_ready/public_checkpoints/

The theorems and the severity, eigenvalue and R_min tables are analytic; no script computes them.

Known behaviours

Each of these changes a reproduced number when it is missed. What each one changes, what it leaves alone and what to do: Known behaviours.

  • Padding was scored. The released losses include padded positions, so they sit below the loss on real tokens; the win rates and conditioning comparisons are unaffected. The trainer now masks padding; --score-padding reproduces a released run.
  • BF16 is a full cast. --use-bf16 calls model.bfloat16(), so weights, gradients and AdamW states are all BF16; only the logits are cast to FP32 for the loss.
  • Four GPUs per run. The GPU count sets the global batch and the step count the learning-rate schedule follows, so another count trains a different model.
  • One training seed. Every run of the paper uses seed 42; the analysis seeds are fitting restarts and subset draws, so the intervals measure fitting variance.
  • Two releases are incomplete. C4 has 99% of its runs (321 of 324, three NC holdout runs missing) and the BF16 replication 93% (302 of 324); a retrain matches the paper only without the missing runs.
  • Sizes are targets. --model-size trains the shape its search finds: up to 8.8% larger for the paper's targets and 13% for a 150M target. The analysis uses the realized count.
  • Run names set the role. A run whose name lacks NxD_GRID, COLINEAR and HOLDOUT is analysed as NC training.
  • Fits are cached by name. run_analysis.py re-renders stored fits; pass --clear-cache after changing the loss tables.
  • Scale pairs can sit at a start draw. In 183 of 900 Kaplan and 510 of 900 Droppo-Elibol fits the optimizer left (N_c, D_c) at a start draw, so do not reuse those pairs as estimates.
  • Bitwise on one driver. A CUDA driver update moved the training loss by O(1e-3); compare a retrained run with --atol 1e-2.

Testing

uv run pytest -n auto

The suite checks law recovery on synthetic data, every entry point's --help, that the released tables are reproduced, the camera-ready analyses, the README's links, images, diagram and citation, and that no private host or path is in the tree.

CI runs the suite, ruff check . and the privacy scan on every pull request and every push to main. A pull request may merge only when its author is a lab-v2 member, or when a member approved its latest commit (the pr-authorization check).

Citing

If you use this work, please cite our paper:

@article{kricheli2026tpp,
  title         = {Tokens-per-Parameter Coverage Is Critical for Robust {LLM} Scaling Law Extrapolation},
  author        = {Kricheli, Joshua Shay and Reid, Alexander Lawrence and Sarkar, Soumajyoti and Gandikota, Venkata and Shakarian, Paulo},
  journal       = {arXiv preprint arXiv:2605.08541},
  year          = {2026},
  eprint        = {2605.08541},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2605.08541},
  note          = {Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)}
}

Acknowledgments

This work was done at the Leibniz Lab, Department of Electrical Engineering and Computer Science, Syracuse University. This research was funded by ARO grant W911NF-24-1-0007. Separately, this work used resources at NCSA through allocation CIS230318 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program (Boerner et al., 2023), which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. Three AMD EPYC CPU chips, donated by AMD, were used for the experiments. This research was supported in part through computational resources provided by Syracuse University (NSF award ACI-1341006). The code is released under the GNU General Public License v2 (LICENSE).

Syracuse University     Syracuse University College of Engineering and Computer Science     Leibniz Lab

About

Code for the NeurIPS 2026 paper 'Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation'

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages