Skip to content

About

A Full-Delta Transformer with Step-Conditioned Persistent Trajectory Refinement

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Deltaxis-250M

A 250M language model that preserves layer-wise corrections, revises its representation for 16 steps with a shared editor, and reads from the entire 17-state trajectory.

日本語

Deltaxis combines three mechanisms on top of a decoder-only Transformer:

  1. each backbone sublayer keeps its change as a separate Delta source;
  2. two shared refiner blocks revise the representation for 16 sequential steps;
  3. a token-level readout mixes the backbone output and all 16 intermediate states.

If a conventional language model submits only its final draft, Deltaxis retains the first draft and every revision that follows.

input tokens
    │
    ▼
14-layer Full-Delta backbone ───────────────► h0 (first draft)
    │
    ▼
2 shared refiner blocks × 16 steps ────────► h1 ... h16
                                               │
                                               ▼
                                  17-state trajectory readout
                                               │
                                               ▼
                                    next-token distribution

This repository contains everything needed to build the model, train a compatible tokenizer, prepare a corpus, train the Body, train the trajectory readout, generate text, and run a terminal chat. It does not include a dataset, tokenizer file, checkpoint, or pretrained weights.

How the architecture works

Full-Delta backbone

A conventional Transformer accumulates every layer's work into one residual state. Deltaxis also keeps the individual changes produced by attention and MLP sublayers.

Later layers can route over those changes directly:

layer 1 attention Delta ─┐
layer 1 MLP Delta       ─┤
layer 2 attention Delta ─┼─► Full-Delta Router ─► current layer
layer 2 MLP Delta       ─┤
embedding anchor        ─┘

Router queries start at zero, so every available Delta begins with equal weight. Training can then teach each layer which earlier corrections are useful.

Sixteen revisions with shared weights

The backbone output is the initial state h0. A pair of distinct refiner blocks is reused 16 times:

h1  = RefinerPair(h0,  step=1)
h2  = RefinerPair(h1,  step=2)
...
h16 = RefinerPair(h15, step=16)

The model does not store 16 separate copies of the refiner. Reusing the same two blocks keeps the deployable parameter count near 250M while adding substantially more sequential computation to each forward pass.

Step Conditioner

The shared refiner is told which revision it is performing. This allows the same weights to learn different behavior near the beginning and end of the trajectory.

Seventeen-state trajectory readout

The final prediction does not have to trust h16 alone. The readout scores:

h0, h1, h2, ... h16

for every token position and mixes their next-token probabilities. A simple position can use a shallow state while a position that benefits from more processing can use a deeper one.

Training is split into two phases. The Body learns first. It is then frozen while the trajectory readout is trained.

See ARCHITECTURE.md for the equations and tensor-level path.

Model configuration

Component Setting
Vocabulary 49,152
Hidden size 1,024
Feed-forward size 4,032
Backbone 14 Full-Delta blocks
Refiner 2 shared blocks
Refinement steps 16
Readout candidates 17 (h0 through h16)
Attention Full causal MHA, 8 heads × 128
Activation ReLU²
Normalization RMSNorm and parameter-free QK RMSNorm
Position encoding RoPE
Deploy parameters 250,399,877
Training parameters with TOP head 300,731,525

What you need to train it

  • Python 3.11 or newer;
  • a CUDA-capable NVIDIA GPU and a matching PyTorch build;
  • UTF-8 training text or JSONL;
  • a separate validation split;
  • a 49,152-piece SentencePiece tokenizer;
  • enough storage for token shards and checkpoints.

The included defaults target a 24 GB GPU. Deltaxis backpropagates through all 16 revisions, so its compute cost is higher than that of a conventional 250M Transformer.

Plain text is enough for general language modelling. For dialogue, include a consistent conversation format in the training corpus, for example:

{"text":"User: How high is Mount Fuji?\nAssistant: Mount Fuji is 3,776 metres high."}

Data preparation appends <|eod|> to every document automatically. This repository trains a causal language model; a model trained only on ordinary prose will not automatically become an instruction-following assistant.

1. Install

git clone https://github.com/AwakeningOS/Deltaxis-250M.git
cd Deltaxis-250M

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Run the small CPU suite and verify the parameter contract:

python scripts/count_parameters.py
python -m pytest

2. Create or supply a tokenizer

An existing SentencePiece model can be used if it satisfies this contract:

vocabulary size: 49,152
<unk>: 0
<|pad|>: 1
<|bos|>: 2
<|eos|>: 3
<|eod|>: 4
byte fallback pieces: 5 through 260

Otherwise, train one from your own corpus. Plain UTF-8 input uses one document per non-empty line; JSONL uses the text field by default.

python scripts/train_tokenizer.py \
  --input corpus/train.jsonl \
  --output-prefix tokenizer/deltaxis-49152

This writes:

tokenizer/deltaxis-49152.model
tokenizer/deltaxis-49152.vocab

The tokenizer is part of the data and checkpoint contract. Do not switch tokenizers in the middle of a run.

3. Prepare the dataset

Input may be either:

  • UTF-8 text, with one document per non-empty line; or
  • JSONL, using a string field named text by default.

Keep training and validation documents in separate files.

python scripts/prepare_data.py \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --train corpus/train.jsonl \
  --validation corpus/validation.jsonl \
  --output-dir data/my-corpus \
  --eod-id 4

Multiple files can be supplied after --train and --validation. Use --text-field for a different JSONL field.

The command writes uint16 token shards and manifest.json. The manifest records the tokenizer SHA-256, vocabulary size, EOD ID, split sizes, and shard paths. Training stops if these values do not match.

4. Train the Body

Reference command for an RTX 3090-class 24 GB GPU:

python scripts/train.py \
  --phase body \
  --data-dir data/my-corpus \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --run-dir runs/body \
  --device cuda:0 \
  --max-steps 1000

The reference batch is:

sequence 2,048 × micro batch 1 × accumulation 256
= 524,288 tokens per optimizer step

Checkpoints, train.jsonl, validation.jsonl, and the run contract are written under runs/body. Starting the same command with the same run directory resumes from its newest checkpoint.

To stop at 100M seen tokens:

python scripts/train.py \
  --phase body \
  --data-dir data/my-corpus \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --run-dir runs/body \
  --device cuda:0 \
  --stop-at-tokens 100000000 \
  --max-steps 1000000

If you reduce the total batch, explicitly choose how the learning rate is scaled:

python scripts/train.py \
  --phase body \
  --data-dir data/my-corpus \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --run-dir runs/body-small-batch \
  --device cuda:0 \
  --gradient-accumulation-steps 32 \
  --batch-lr-mode sqrt \
  --max-steps 1000

Use a new run directory if you change the batch, learning rate, tokenizer, data manifest, or runtime settings.

5. Train the 17-state readout

After Body training, freeze the Body and train only the trajectory readout:

python scripts/train.py \
  --phase readout \
  --body-checkpoint runs/body/step_00001000.pt \
  --data-dir data/my-corpus \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --run-dir runs/readout \
  --device cuda:0 \
  --max-steps 1000

Finalize the readout checkpoint before inference:

python scripts/finalize_readout.py \
  --checkpoint runs/readout/step_00001000.pt \
  --output checkpoints/deltaxis-250m.pt

The product path refuses an unfinalized checkpoint, preventing an untrained readout from being used silently.

6. Generate text

python scripts/generate.py \
  --checkpoint checkpoints/deltaxis-250m.pt \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --prompt "Artificial intelligence is" \
  --device cuda:0

Generation uses a KV cache by default. The prompt is processed once; subsequent steps send only the newly generated token through the model. Because Deltaxis has 14 Backbone attention stages and 16 revisions with two Refiner blocks, the runtime keeps 46 separate KV-cache slots. The shared Refiner weights do not share cache contents: each revision receives a different hidden state.

The cache is fixed to the model context length and is discarded after the generation request. Generation stops when that context is full. Use --no-kv-cache only when checking compatibility against full-sequence recomputation.

On the development RTX 3090, eager BF16 generation from a 512-token prompt for 16 token predictions took 3.43 seconds with full-sequence recomputation and 1.07 seconds with the cache, a 3.21× end-to-end speedup. After prefill, single-token decode improved from about 214 ms to 56.8 ms per token. The 46-stage cache occupied 0.719 GiB at the 4,096-token capacity. These figures are hardware-, software-, and sequence-dependent.

7. Chat in the terminal

If the checkpoint was trained on a compatible dialogue format:

python scripts/chat.py \
  --checkpoint checkpoints/deltaxis-250m.pt \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --device cuda:0 \
  --user-prefix User \
  --assistant-prefix Assistant

Use /reset to clear the conversation and /quit to exit.

Each assistant reply prefills the current transcript once and then uses single-token cached decoding. The next user turn starts a fresh prefill from the updated transcript, keeping the terminal implementation simple and exact.

Training-efficiency defaults

The trainer enables practical defaults for a 24 GB GPU:

  • whole-model torch.compile;
  • a compile recompile limit of 16;
  • vocabulary chunks of 16,384;
  • PyTorch SDPA;
  • activation checkpointing for the first 14 refinement visits, with the final two visits kept directly to avoid their backward recomputation;
  • trajectory telemetry only at the end of an accumulation;
  • detailed optimizer update metrics every 100 steps;
  • a complete parameter finite scan every 100 steps;
  • approximately hourly checkpoints.

The first step of a new process may spend several minutes compiling. Long runs amortize that cold start. On the development RTX 3090, whole-model compilation improved steady throughput from about 1,822 tok/s for the best eager setting to about 3,226 tok/s. A separate checkpoint-boundary comparison selected a two-visit direct suffix, improving its matched baseline from about 3,069 to 3,199 tok/s while remaining below 16 GiB of PyTorch reserved memory. These numbers are hardware-, software-, data-, and shape-dependent; benchmark your own system.

Disable compilation when diagnosing compatibility problems:

python scripts/train.py ... --torch-compile-mode none

If memory is insufficient, reduce sequence length or gradient accumulation first. Increasing the micro batch can help some models, but Deltaxis keeps activations for a 16-step trajectory and can run out of memory quickly.

Full-model GPU smoke test

This runs a Body forward/backward and optimizer step, readout training step, and finalized product inference:

python scripts/gpu_smoke.py --device cuda:0 --allow-gpu

It allocates the full model on the GPU. The ordinary test suite uses tiny CPU configurations.

Measure growth during pretraining

The repository includes a fixed growth evaluation suite:

  • 3,072 synthetic multiple-choice diagnostics across 12 ability families;
  • a 768-item core subset for quick checks;
  • 42 fixed Japanese prompts for reading model outputs directly;
  • deterministic generation, answer keys, independent solving, and validation.

Validate the published questions without a model:

python evaluation/scripts/validate_suite.py

Evaluate an early Body checkpoint on the quick subset:

python evaluation/scripts/evaluate_deltaxis.py \
  --checkpoint runs/body/step_00001000.pt \
  --tokenizer-model tokenizer/deltaxis-49152.model \
  --phase body \
  --subset core \
  --output evaluations/body-core.jsonl \
  --device cuda:0

Use --subset full for a formal checkpoint. The official development project compares three different model architectures at 100-million-token intervals. Keep the evaluation directory out of every training corpus.

Current scope

  • Pretrained weights are not published.
  • Datasets and tokenizer files are not included.
  • The repository can be cloned and trained from random initialization on your own data.
  • Generation and terminal chat use stage-separated KV caches by default.
  • Useful dialogue requires sufficient pretraining plus dialogue or instruction data.

License

MIT

About

A Full-Delta Transformer with Step-Conditioned Persistent Trajectory Refinement

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages