A 250M language model that preserves layer-wise corrections, revises its representation for 16 steps with a shared editor, and reads from the entire 17-state trajectory.
Deltaxis combines three mechanisms on top of a decoder-only Transformer:
- each backbone sublayer keeps its change as a separate Delta source;
- two shared refiner blocks revise the representation for 16 sequential steps;
- a token-level readout mixes the backbone output and all 16 intermediate states.
If a conventional language model submits only its final draft, Deltaxis retains the first draft and every revision that follows.
input tokens
│
▼
14-layer Full-Delta backbone ───────────────► h0 (first draft)
│
▼
2 shared refiner blocks × 16 steps ────────► h1 ... h16
│
▼
17-state trajectory readout
│
▼
next-token distribution
This repository contains everything needed to build the model, train a compatible tokenizer, prepare a corpus, train the Body, train the trajectory readout, generate text, and run a terminal chat. It does not include a dataset, tokenizer file, checkpoint, or pretrained weights.
A conventional Transformer accumulates every layer's work into one residual state. Deltaxis also keeps the individual changes produced by attention and MLP sublayers.
Later layers can route over those changes directly:
layer 1 attention Delta ─┐
layer 1 MLP Delta ─┤
layer 2 attention Delta ─┼─► Full-Delta Router ─► current layer
layer 2 MLP Delta ─┤
embedding anchor ─┘
Router queries start at zero, so every available Delta begins with equal weight. Training can then teach each layer which earlier corrections are useful.
The backbone output is the initial state h0. A pair of distinct refiner blocks is reused 16 times:
h1 = RefinerPair(h0, step=1)
h2 = RefinerPair(h1, step=2)
...
h16 = RefinerPair(h15, step=16)
The model does not store 16 separate copies of the refiner. Reusing the same two blocks keeps the deployable parameter count near 250M while adding substantially more sequential computation to each forward pass.
The shared refiner is told which revision it is performing. This allows the same weights to learn different behavior near the beginning and end of the trajectory.
The final prediction does not have to trust h16 alone. The readout scores:
h0, h1, h2, ... h16
for every token position and mixes their next-token probabilities. A simple position can use a shallow state while a position that benefits from more processing can use a deeper one.
Training is split into two phases. The Body learns first. It is then frozen while the trajectory readout is trained.
See ARCHITECTURE.md for the equations and tensor-level path.
| Component | Setting |
|---|---|
| Vocabulary | 49,152 |
| Hidden size | 1,024 |
| Feed-forward size | 4,032 |
| Backbone | 14 Full-Delta blocks |
| Refiner | 2 shared blocks |
| Refinement steps | 16 |
| Readout candidates | 17 (h0 through h16) |
| Attention | Full causal MHA, 8 heads × 128 |
| Activation | ReLU² |
| Normalization | RMSNorm and parameter-free QK RMSNorm |
| Position encoding | RoPE |
| Deploy parameters | 250,399,877 |
| Training parameters with TOP head | 300,731,525 |
- Python 3.11 or newer;
- a CUDA-capable NVIDIA GPU and a matching PyTorch build;
- UTF-8 training text or JSONL;
- a separate validation split;
- a 49,152-piece SentencePiece tokenizer;
- enough storage for token shards and checkpoints.
The included defaults target a 24 GB GPU. Deltaxis backpropagates through all 16 revisions, so its compute cost is higher than that of a conventional 250M Transformer.
Plain text is enough for general language modelling. For dialogue, include a consistent conversation format in the training corpus, for example:
{"text":"User: How high is Mount Fuji?\nAssistant: Mount Fuji is 3,776 metres high."}Data preparation appends <|eod|> to every document automatically. This repository trains a causal language model; a model trained only on ordinary prose will not automatically become an instruction-following assistant.
git clone https://github.com/AwakeningOS/Deltaxis-250M.git
cd Deltaxis-250M
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"Run the small CPU suite and verify the parameter contract:
python scripts/count_parameters.py
python -m pytestAn existing SentencePiece model can be used if it satisfies this contract:
vocabulary size: 49,152
<unk>: 0
<|pad|>: 1
<|bos|>: 2
<|eos|>: 3
<|eod|>: 4
byte fallback pieces: 5 through 260
Otherwise, train one from your own corpus. Plain UTF-8 input uses one document per non-empty line; JSONL uses the text field by default.
python scripts/train_tokenizer.py \
--input corpus/train.jsonl \
--output-prefix tokenizer/deltaxis-49152This writes:
tokenizer/deltaxis-49152.model
tokenizer/deltaxis-49152.vocab
The tokenizer is part of the data and checkpoint contract. Do not switch tokenizers in the middle of a run.
Input may be either:
- UTF-8 text, with one document per non-empty line; or
- JSONL, using a string field named
textby default.
Keep training and validation documents in separate files.
python scripts/prepare_data.py \
--tokenizer-model tokenizer/deltaxis-49152.model \
--train corpus/train.jsonl \
--validation corpus/validation.jsonl \
--output-dir data/my-corpus \
--eod-id 4Multiple files can be supplied after --train and --validation. Use --text-field for a different JSONL field.
The command writes uint16 token shards and manifest.json. The manifest records the tokenizer SHA-256, vocabulary size, EOD ID, split sizes, and shard paths. Training stops if these values do not match.
Reference command for an RTX 3090-class 24 GB GPU:
python scripts/train.py \
--phase body \
--data-dir data/my-corpus \
--tokenizer-model tokenizer/deltaxis-49152.model \
--run-dir runs/body \
--device cuda:0 \
--max-steps 1000The reference batch is:
sequence 2,048 × micro batch 1 × accumulation 256
= 524,288 tokens per optimizer step
Checkpoints, train.jsonl, validation.jsonl, and the run contract are written under runs/body. Starting the same command with the same run directory resumes from its newest checkpoint.
To stop at 100M seen tokens:
python scripts/train.py \
--phase body \
--data-dir data/my-corpus \
--tokenizer-model tokenizer/deltaxis-49152.model \
--run-dir runs/body \
--device cuda:0 \
--stop-at-tokens 100000000 \
--max-steps 1000000If you reduce the total batch, explicitly choose how the learning rate is scaled:
python scripts/train.py \
--phase body \
--data-dir data/my-corpus \
--tokenizer-model tokenizer/deltaxis-49152.model \
--run-dir runs/body-small-batch \
--device cuda:0 \
--gradient-accumulation-steps 32 \
--batch-lr-mode sqrt \
--max-steps 1000Use a new run directory if you change the batch, learning rate, tokenizer, data manifest, or runtime settings.
After Body training, freeze the Body and train only the trajectory readout:
python scripts/train.py \
--phase readout \
--body-checkpoint runs/body/step_00001000.pt \
--data-dir data/my-corpus \
--tokenizer-model tokenizer/deltaxis-49152.model \
--run-dir runs/readout \
--device cuda:0 \
--max-steps 1000Finalize the readout checkpoint before inference:
python scripts/finalize_readout.py \
--checkpoint runs/readout/step_00001000.pt \
--output checkpoints/deltaxis-250m.ptThe product path refuses an unfinalized checkpoint, preventing an untrained readout from being used silently.
python scripts/generate.py \
--checkpoint checkpoints/deltaxis-250m.pt \
--tokenizer-model tokenizer/deltaxis-49152.model \
--prompt "Artificial intelligence is" \
--device cuda:0Generation uses a KV cache by default. The prompt is processed once; subsequent steps send only the newly generated token through the model. Because Deltaxis has 14 Backbone attention stages and 16 revisions with two Refiner blocks, the runtime keeps 46 separate KV-cache slots. The shared Refiner weights do not share cache contents: each revision receives a different hidden state.
The cache is fixed to the model context length and is discarded after the
generation request. Generation stops when that context is full. Use
--no-kv-cache only when checking compatibility against full-sequence
recomputation.
On the development RTX 3090, eager BF16 generation from a 512-token prompt for 16 token predictions took 3.43 seconds with full-sequence recomputation and 1.07 seconds with the cache, a 3.21× end-to-end speedup. After prefill, single-token decode improved from about 214 ms to 56.8 ms per token. The 46-stage cache occupied 0.719 GiB at the 4,096-token capacity. These figures are hardware-, software-, and sequence-dependent.
If the checkpoint was trained on a compatible dialogue format:
python scripts/chat.py \
--checkpoint checkpoints/deltaxis-250m.pt \
--tokenizer-model tokenizer/deltaxis-49152.model \
--device cuda:0 \
--user-prefix User \
--assistant-prefix AssistantUse /reset to clear the conversation and /quit to exit.
Each assistant reply prefills the current transcript once and then uses single-token cached decoding. The next user turn starts a fresh prefill from the updated transcript, keeping the terminal implementation simple and exact.
The trainer enables practical defaults for a 24 GB GPU:
- whole-model
torch.compile; - a compile recompile limit of 16;
- vocabulary chunks of 16,384;
- PyTorch SDPA;
- activation checkpointing for the first 14 refinement visits, with the final two visits kept directly to avoid their backward recomputation;
- trajectory telemetry only at the end of an accumulation;
- detailed optimizer update metrics every 100 steps;
- a complete parameter finite scan every 100 steps;
- approximately hourly checkpoints.
The first step of a new process may spend several minutes compiling. Long runs amortize that cold start. On the development RTX 3090, whole-model compilation improved steady throughput from about 1,822 tok/s for the best eager setting to about 3,226 tok/s. A separate checkpoint-boundary comparison selected a two-visit direct suffix, improving its matched baseline from about 3,069 to 3,199 tok/s while remaining below 16 GiB of PyTorch reserved memory. These numbers are hardware-, software-, data-, and shape-dependent; benchmark your own system.
Disable compilation when diagnosing compatibility problems:
python scripts/train.py ... --torch-compile-mode noneIf memory is insufficient, reduce sequence length or gradient accumulation first. Increasing the micro batch can help some models, but Deltaxis keeps activations for a 16-step trajectory and can run out of memory quickly.
This runs a Body forward/backward and optimizer step, readout training step, and finalized product inference:
python scripts/gpu_smoke.py --device cuda:0 --allow-gpuIt allocates the full model on the GPU. The ordinary test suite uses tiny CPU configurations.
The repository includes a fixed growth evaluation suite:
- 3,072 synthetic multiple-choice diagnostics across 12 ability families;
- a 768-item core subset for quick checks;
- 42 fixed Japanese prompts for reading model outputs directly;
- deterministic generation, answer keys, independent solving, and validation.
Validate the published questions without a model:
python evaluation/scripts/validate_suite.pyEvaluate an early Body checkpoint on the quick subset:
python evaluation/scripts/evaluate_deltaxis.py \
--checkpoint runs/body/step_00001000.pt \
--tokenizer-model tokenizer/deltaxis-49152.model \
--phase body \
--subset core \
--output evaluations/body-core.jsonl \
--device cuda:0Use --subset full for a formal checkpoint. The official development project
compares three different model architectures at 100-million-token intervals.
Keep the evaluation directory out of every training corpus.
- Pretrained weights are not published.
- Datasets and tokenizer files are not included.
- The repository can be cloned and trained from random initialization on your own data.
- Generation and terminal chat use stage-separated KV caches by default.
- Useful dialogue requires sufficient pretraining plus dialogue or instruction data.