This project implements language models in dependency-free C++, eliminating the need for PyTorch or Python to train a transformer locally. The core implementation is a decoder-only GPT architecture featuring custom tensors, embeddings, multi-head causal self-attention, layer normalization, cross-entropy loss, and an analytical backward pass with the AdamW optimizer - all contained within main.cpp ,llm.mm and the include/ directory also a token level [BPE] tokenizer.h implementation inside include. With no autograd engine or external frameworks, every gradient is explicitly derived and written out. The model achieves a validation loss of 1.6371 nats after 76 minutes of CPU training on 31.4 million characters, demonstrating that character-level language modeling at this scale is highly tractable on commodity hardware without external dependencies. On a GPU (CUDA/bfloat16), a validation loss of 2.3918 is reached in under 83 minutes, achieving a peak throughput of 19.6k tokens per second.
| S.No. | time | val_bpb / Metric | scale | Date | Contributors |
|---|---|---|---|---|---|
| 1 | 168 hours | 29.41 PPL (~0.93 BPB) | 124M (32x TPU v3) | Feb 2019 | OpenAI (GPT-2 Small) |
| 2 | 45 min | 3.28 Val Loss (~0.748 BPB) | 124M (8x H100) | May 2024 | Andrej Karpathy (llm.c) |
| 3 | 2.98 min | 3.28 Val Loss (~0.748 BPB) | 124M (8x H100) | Feb 2025 | Keller Jordan et al. (Modded-NanoGPT) |
| 4 | 72 hours | 22.7 PPL (~0.85 BPB) | 125M (32x A100) | Feb 2024 | Meta (MobileLLM-125M) |
| 5 | ~24 hours | ~1.02 BPB | 135M (64x H100) | Jul 2024 | Hugging Face (SmolLM-135M) |
| 6 | 61.3 min | 0.7176 | 10.82M (T4) | Mar 2026 | @Eamon2009 |
| 7 | 6.1 min | 0.9250 | 1.99M (T4) | Feb 2026 | @Eamon2009 |
| 8 | 39.4 min | 1.3145 | 0.82M (CPU) | July 2026 | @Eamon2009 |
| 9 | 76.2 min | 1.6371 | 0.82M (CPU) | Jan 2026 | @Eamon2009 |
More broadly, the primary contribution of this work lies in its absolute transparency. Every gradient in the backward pass is explicitly written and readable, and every tensor operation is a standard C++ function. By exposing exactly what frameworks like PyTorch compute under the hood, this implementation provides a clear educational pathway. We believe that this fundamental understanding is the true foundation of genuine expertise in deep learning. The point of this repo is the C++ core. The PyTorch exist to make the model usable, but if you're here to train a GPT without a framework doing the work for you, include/backward.h is where to start seeing optimization without torch.
The custom C++ backend is transparent but slow: a CPU executes scalar matrix multiplication at roughly 1-10 GFLOP/s. An NVIDIA RTX 4090 delivers ∼80 TFLOP/s-an 8,000–80,000× speedup for the same computation.The LibTorch port replaces the custom backend with PyTorch’s C++ API, gaining cuBLAS-accelerated matrix operations. The transformer architecture remains unchanged only the compute layer is modified. A single line migrates the model to GPU:
model->to(torch::kCUDA)
which transfers all parameters to GPU memory; all subsequent torch::matmul calls dispatch to cuBLAS automatically.
technical notes: docs
The fastest way to see the whole pipeline - tokenize, train, checkpoint, generate - using the bundled character-level corpus:
get the data set first :
cd data # you set the file size for data set
python data_set.py # also get any dataset from hugging face datasets # run this
g++ -std=c++17 -O3 -march=native -fopenmp -I. -Iinclude -o llm.exe main.cpp
./llm.exe data/input.txtshould see something like this
[DATA] Total tokens : 3521179
[DATA] Train tokens : 3169061
[DATA] Val tokens : 352118
██╗ ██╗ ███╗ ███╗ ██████╗ ██████╗ ██████╗
██║ ██║ ████╗ ████║ ██╔════╝ ██╔══██╗██╔══██╗
██║ ██║ ██╔████╔██║ █████╗██║ ██████╔╝██████╔╝
██║ ██║ ██║╚██╔╝██║ ╚════╝██║ ██╔═══╝ ██╔═══╝
███████╗███████╗██║ ╚═╝ ██║ █████╗╚██████╗ ██║ ██║
╚══════╝╚══════╝╚═╝ ╚═╝ ╚════╝ ╚═════╝ ╚═╝ ╚═╝
+------------------------------------------+------------------------------------------+
| LLM Architecture |
+------------------------------------------+------------------------------------------+
| Max Context Length : 64 | Vocab Size (BPE) : 2056 |
| Number of Layers : 4 | Attention Heads : 2 |
| Embedding Channels : 128 | Total Parameters : 1328392 |
| Repetition Penalty : 10 | Repetition Window : 10 |
+------------------------------------------+------------------------------------------+
+-------------------------------------------------------------------------------------+
| Host Hardware Specs |
+-------------------------------------------------------------------------------------+
| Host CPU Device : AMD Ryzen 5 PRO 3500U w/ Radeon... |
| Host RAM (Total) : 8045 MB |
+-------------------------------------------------------------------------------------+
step 1/5000(0.02%) | train loss 7.650238 | val loss 7.652169 | lr 1.00e-06 | 4016.84 ms | 509 tok/s | ram 189.6 MB
step 2/5000(0.04%) | train loss 7.648808 | val loss 7.652169 | lr 2.00e-06 | 4053.90 ms | 505 tok/s | ram 190.7 MB
step 3/5000(0.06%) | train loss 7.658056 | val loss 7.652169 | lr 3.00e-06 | 4381.06 ms | 467 tok/s | ram 190.7 MB
step 4/5000(0.08%) | train loss 7.648185 | val loss 7.652169 | lr 4.00e-06 | 4514.13 ms | 453 tok/s | ram 189.8 MB
step 5/5000(0.10%) | train loss 7.646149 | val loss 7.652169 | lr 5.00e-06 | 4429.38 ms | 462 tok/s | ram 190.3 MB
step 6/5000(0.12%) | train loss 7.644379 | val loss 7.652169 | lr 6.00e-06 | 4443.55 ms | 460 tok/s | ram 190.8 MB
This trains from scratch on data/input.txt and writes the best checkpoint to best_model.bin. Once you have a checkpoint, generate or chat with it:
./llm.exe data/input.txt --generate
./llm.exe data/input.txt --chat --chat-tokens 300debugging tip: drop -O2 for -g when compiling if you want to step through include/backward.h or include/gpt.h in a debugger — the manual backward pass is much easier to follow one breakpoint at a time.
llm.exe [data_path] [--generate] [--chat] [--chat-tokens N].
├── .ci/ # CI/CD pipelines and Docker configurations
├── .github/ # GitHub Actions workflows and issue templates
├── assets/ # Project images, banners, and hardware diagrams
├── benches/
│ └── bench.cpp # C++ benchmarking script for performance testing
├── config/
│ └── config.h # Global configuration parameters
├── data/
│ ├── dataset.py # Data loading and preprocessing pipeline
│ ├── data_set.py # Alternative dataset handling logic
│ ├── export.py # Script to export models or tensors
│ └── input.txt # Raw text data used for training/testing
├── docs/ # Additional documentation and generated reports
├── engine/ # Core backend implementation
│ ├── llm.pt # Primary PyTorch model checkpoint
│ ├── mini-quadtrix.pt # Minimal PyTorch model for testing
│ └── llm.cpp/ # Low-level C++/CUDA/Metal engine
│ ├── CMakeLists.txt # Engine-specific build configuration
│ ├── llm.cu # CUDA implementation for Nvidia GPUs
│ ├── make # Engine Makefile compilation script
│ ├── train.mm # Objective-C++ Metal script for Apple Silicon training
│ ├── config/
│ │ └── config.h # Engine-specific configuration header
│ └── include/ # Neural network mathematical headers
│ ├── attention.h # Self-attention module definitions
│ ├── cuda_kernels.cuh # Custom CUDA kernel definitions
│ ├── layer.cuh # Layer abstractions for GPU
│ ├── tensor.cuh # Core tensor math operations
│ └── ... # (Other low-level neural net headers)
├── include/ # High-level C++ API headers
│ ├── attention.h # High-level attention interfaces
│ ├── gpt.h # GPT model architecture definitions
│ ├── llm-cpp.hpp # Main library interface for external use
│ ├── tokenizer.h # Text tokenization logic
│ └── torch_bridge.h # Interoperability layer for PyTorch tensors
├── scripts/
│ └── build.sh # Automation script for building the project
├── train_test/ # Experimental and testing scripts
│ ├── model.py # Python model architecture definitions
│ ├── test.c # C-based functional testing
│ └── train2.mm # Experimental Metal training iterations
├── .clang-format # Code style rules for C/C++ files
├── .clang-tidy # Linter configuration for C/C++ static analysis
├── benchmark.cpp # Entry point for running system benchmarks
├── CMakeLists.txt # Root CMake build configuration
├── llm.mm # Apple Silicon (Metal) main inference entry point
├── main.cpp # Main application C++ entry point
├── README.md # Main project documentation
├── requirements.txt # Python dependencies for the project
└── shards.cpp # C++ implementation for handling data shards
| Argument | Description |
|---|---|
data_path |
Plain-text corpus used to build the tokenizer and train/validation split |
--generate |
Load weights and continuously generate text |
--chat |
Load weights and start interactive terminal chat |
--chat-tokens N |
Max generated tokens per chat response |
| Env var | Default | Description |
|---|---|---|
GPT_DATA_PATH |
data/input.txt |
Override the default training corpus |
GPT_MODEL_PATH |
best_model.bin |
Override the checkpoint path |
No third-party runtime dependency - it builds from main.cpp, config/config.h, and include/*.h alone.
- Byte Pair Encoding (BPE) Tokenizer - Built entirely from scratch. Compiles a custom vocabulary directly from the training corpus by running iterative token-pair merges until it hits a targeted vocabulary threshold.
- Train/validation split via
DataLoader - Token + positional embeddings
- Multi-head causal self-attention with explicit QKV projections
- Pre-layer-norm residual transformer blocks
- Feed-forward MLP with ReLU
- Cross-entropy loss
- Fully analytical backward pass - every gradient (attention, layer norm, MLP, embeddings) is derived mathematically and coded explicitly in
include/backward.h, not autograd - AdamW optimizer (first/second moment estimates, weight decay)
- Checkpoint save/load
- Autoregressive generation and terminal chat mode
Hyperparameters live in engine/llm.cpp/config/config.h and require a rebuild to take effect:
// note: The c++ version only runs on cpu not on GPU
static const unsigned int SEED = 1337;
static const double TRAIN_SPLIT = 0.9;
static const int BATCH_SIZE = 32;
static const int BLOCK_SIZE = 64;
static const int MAX_ITERS = 5000;
static const int EVAL_INTERVAL = 500;
static const float LEARNING_RATE = 5e-4f;
static const int EVAL_ITERS = 25;
static const int N_EMBD = 128;
static const int N_HEAD = 2;
static const int N_LAYER = 4;
static const float DROPOUT = 0.05f;
static const int BPE_VOCAB_SIZE = 2048; | Metric | Character-Level | Small Scale | Large Scale |
|---|---|---|---|
| Parameters | 0.83M | 2.00M | 19.17M |
| Layers | 4 | 4 | 4 |
| Embedding dim | 128 | 200 | 200 |
| Attention heads | 4 | 4 | 4 |
| Context length | 64 | 200 | 200 |
| Vocab | 105 char | 110 char | ~50K BPE |
| Corpus | TinyStories | TinyStories | Children's Stories |
| Iterations | 3,000 | 5,000 | 5,000 |
| Train loss | 1.5632 | 0.9045 | — |
| Val loss | 1.6371 | 0.9301 | — |
| Gen. gap | 0.0739 | 0.0256 | — |
See run.md and the leaderboard in the full docs for more configurations.
| Project | Focus | Language | Autograd |
|---|---|---|---|
| nanoGPT / minGPT | Minimal, educational GPT training | Python | PyTorch |
| llama2.c | Inference-only | C | None |
| llm.cpp | Training and inference, manual backward pass | C++ | C++ |
I'd like the C++ core (main.cpp, include/, config/) to stay dependency-free and to stay the part of this repo that explin transformer internals directly. The PyTorch engine, include, and ci are welcome to grow more features, integrations, and CI polish. If you build a port to another language or framework, I'm happy to link to it from a notable-forks section; just open an issue or PR.
- Vaswani et al., "Attention Is All You Need", 2017
- Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2 technical work), 2019
- Brown et al., "Language Models are Few-Shot Learners" (GPT-3 paper), 2020
- Meta AI, "The Llama 3 Herd of Models" (Llama 3 paper), 2024
- Andrej Karpathy, nanoGPT repository as an educational reference point
- HuggingFace Datasets for FineWeb and other pretraining/fine-tuning datasets
- Karpathy, A. (2024). Let's reproduce GPT-2 (124M)
- Note: We express our thanks to Andrej Karpathy for his instructional content. Concepts regarding the multi-head attention structure, learning rate schedule, and binary token shard loading were implemented using his walkthrough.
If you find llm.cpp helpful in your research cite as:
@misc{llm.cpp,
author = {Eamon Sippy},
title = {llm.cpp: LLM training in C++ \& Python},
year = {2026},
publisher = {GitHub},
journal = {GitHub repository},
url = {https://github.com/LMGNU/llm.cpp}
}
GPL-3.0