Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/cli-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -448,8 +448,8 @@ explicit flags take precedence.
| `--no-mtp` | Disable multi-token speculation: load no draft model, keep no verification rows, and decode one token per step. | Speculation on |
| `--mtp-only` | Omit MTP-disabled prefill, decode, maintenance, and mixed graphs during preparation. Retain width-one speculative rounds and vision graphs. Requires positive draft depth; rejects `--no-mtp`. The core cannot load with native `--mtp-draft-tokens 0`. | Capture both execution modes |
| `--draft-body-quantization {mxfp8}` | Prepare eligible DFlash2 body context, attention latent/query/output, and MLP projections as MXFP8, independently of target conversion and proposal-head precision. Kimi K3.1 Pro requires `--speculative-method dflash2` and positive learned draft depth; DSpark and other recipes reject it. Normalization, convolution, candidate-selector, and absorbed key/value weights retain their precision. Omission preserves checkpoint precision or the existing `online_mxfp8_scope all` behavior. | No additional override |
| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. This does not quantize the draft body. | Checkpoint precision |
| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Requires positive draft depth. | Checkpoint precision |
| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. Inkling 2 Small NVFP4 supports both. This does not quantize the draft body. | Checkpoint precision |
| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Inkling 2 Small NVFP4 supports FP8 for its recurrent correction head. Requires positive draft depth. | Checkpoint precision |
| `--vocabulary-parallel-sampling` | Prepare vocabulary-sharded DSpark proposals and block rejection sampling. DeepSeek V4.1 on four GPUs only; three-GPU preparation rejects it. Requires positive draft depth and recipe sampling defaults without top-k, top-p, or min-p filtering, and rejects `--markov-head-quantization nvfp4`. | Full-vocabulary sampling |
| `--mtp-draft-tokens N` | Most draft tokens verified in one round; see `MODEL --help` for its limit. Verification capacities cover counts from 0 through `N` within the speculative token capacity. Each complete speculative program records its actual proposal depth independently of target width and aggregate rows. Prefix selection uses measured complete-program cost and completed acceptance or confidence. Zero retains draft state with width-one execution; `--no-mtp` excludes draft weights and state. | Recipe declaration |
| `--speculative-token-capacity N` | Maximum physical target-token rows in wider speculative graphs; width one covers every request bucket. Applies to every speculative recipe. | Recipe declaration: 32 for Muse Spark and 128 for the other recipes that declare one. Gemma 4 declares none and derives it from concurrency and draft depth (6 at its defaults) |
Expand Down
12 changes: 12 additions & 0 deletions docs/model-preparation.md
Original file line number Diff line number Diff line change
Expand Up @@ -1325,6 +1325,18 @@ reasoning, tool calls, and structured responses. Sampling defaults are
temperature 1.0 and top-p 1.0; the prompt encoder defaults to reasoning effort
0.9 when the request does not specify one.

Each drafting round projects eight block rows through the vocabulary head and
then, for each of up to seven draft positions, corrects those logits with the
recurrent head: its output map folded through the vocabulary head, one
vocabulary row by 256 values. Both read BF16 by default, 463 MB and 25.7 MB
per rank. `--draft-head-quantization nvfp4` or `mxfp8` prepares a quantized
copy of the target's vocabulary shard that only proposals read, and
`--markov-head-quantization fp8` stores the folded correction head as E4M3
values with one FP32 scale per vocabulary row. Both are computed at load from
checkpoint values; target verification keeps the BF16 head, so a quantized
proposal head changes which tokens are proposed and how often they are
accepted, never the output distribution. The default omits both flags.

Export with default tuning and Torch compilation:

```sh
Expand Down
6 changes: 3 additions & 3 deletions ignition/components/parallel_draft.py
Original file line number Diff line number Diff line change
Expand Up @@ -442,11 +442,11 @@ def propose(self, advance_rng=True, allowed=None, *, probabilities):
hidden = features["hidden"].view(b, block, -1)
context = features["recurrent_context"]
state = self.recurrent.begin(context)
# The draft states and target context do not depend on the draws: project them once.
fixed = self.recurrent.project_context(hidden[:, o.logit_offset:], context)
scores = torch.empty_like(draft_logits)[:, o.logit_offset:o.logit_offset + w - 1]
for index in range(w - 1):
state, logits = self.recurrent.step(state, markov_token,
hidden[:, index + o.logit_offset].contiguous(), context,
base[:, index])
state, logits = self.recurrent.step(state, markov_token, fixed[:, index], base[:, index])
scores[:, index].copy_(logits)
self.full_proposal.run_step(index, scores, drafts, pending=markov_token,
seeds=seed, offsets=base_offset, probabilities=probabilities, valid=valid,
Expand Down
51 changes: 40 additions & 11 deletions ignition/components/recurrent_proposal.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

from .embedding import PreparedEmbedding
from .linear import UnquantizedProjection
from .markov_sampling import PreparedRowScaledFP8MarkovProjection, RowScaledFP8MarkovWeight


def recurrent_gate(raw, state):
Expand All @@ -27,33 +28,61 @@ def corrected_logits(base, correction, softcap):


class PreparedRecurrentProposalHead:
"""Keep recurrence state inside one block while retaining its actual sampled q."""
"""Keep recurrence state inside one block while retaining its actual sampled q.

def __init__(self, weights, tokens, *, session, name):
The recurrence input has a part that depends on the previous draw (state and predecessor
embedding) and a part that does not (draft hidden state and target context). `project_context`
computes the second part for every position of the block in one projection; `step` adds the
first. `weights.head` maps the recurrent output to vocabulary corrections as BF16 rows, or as
row-scaled E4M3 rows whose FP32 product takes the BF16 bias before rounding to BF16.
"""

def __init__(self, weights, tokens, *, positions, session, name):
rows, device = tokens.numel(), tokens.device
rank = weights.embedding.shape[1]
hidden = weights.context_weight.shape[1] // 2
self.rank, self.softcap = rank, weights.softcap
if type(positions) is not int or positions < 1:
raise ValueError("a recurrent proposal head corrects at least one position per row")
self.rank, self.softcap, self.positions = rank, weights.softcap, positions
self.context_bias = weights.bias
embedding = torch.empty((rows, rank), dtype=torch.bfloat16, device=device)
dynamic = torch.empty((rows, 2 * rank), dtype=torch.bfloat16, device=device)
context = torch.empty((rows, 2 * hidden), dtype=torch.bfloat16, device=device)
context = torch.empty((rows * positions, 2 * hidden), dtype=torch.bfloat16, device=device)
fixed = torch.empty((rows * positions, 3 * rank), dtype=torch.float32, device=device)
raw = torch.empty((rows, 3 * rank), dtype=torch.float32, device=device)
correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device)
self.embed = PreparedEmbedding(weights.embedding, tokens, embedding,
session=session, name=name + ".embedding").functional(session=session, dynamic_inputs=True)
self.context = UnquantizedProjection(context, weights.context_weight, raw,
self.context = UnquantizedProjection(context, weights.context_weight, fixed,
session=session, name=name + ".context").functional(session=session)
self.dynamic = UnquantizedProjection(dynamic, weights.state_weight, raw,
session=session, name=name + ".dynamic").functional(session=session)
self.output = UnquantizedProjection(embedding, weights.head, correction,
bias=weights.head_bias, session=session, name=name + ".output").functional(session=session)
self.output_bias = None
if isinstance(weights.head, RowScaledFP8MarkovWeight):
correction = torch.empty((rows, weights.head.out_features), dtype=torch.float32, device=device)
self.output = PreparedRowScaledFP8MarkovProjection(embedding, weights.head, correction,
session=session, name=name + ".output").functional(session=session)
self.output_bias = weights.head_bias
else:
correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device)
self.output = UnquantizedProjection(embedding, weights.head, correction,
bias=weights.head_bias, session=session, name=name + ".output").functional(session=session)

def begin(self, hidden):
return torch.zeros((hidden.shape[0], self.rank), dtype=torch.bfloat16, device=hidden.device)

def step(self, state, previous, hidden, context, base):
fixed = self.context(torch.cat((hidden, context), -1)) + self.context_bias
def project_context(self, hidden, context):
"""Return FP32 [rows, positions, 3 * rank] recurrence inputs from [rows, positions, hidden] draft states."""
rows, positions, width = hidden.shape
if positions != self.positions:
raise ValueError("recurrent context projection covers every prepared block position")
source = torch.cat((hidden, context[:, None].expand(rows, positions, width)), -1)
fixed = self.context(source.view(rows * positions, 2 * width)) + self.context_bias
return fixed.view(rows, positions, 3 * self.rank)

def step(self, state, previous, fixed, base):
dynamic = self.dynamic(torch.cat((state, self.embed(token_ids=previous)), -1))
state, latent = recurrent_gate(dynamic + fixed, state)
return state, corrected_logits(base, self.output(latent), self.softcap)
correction = self.output(latent)
if self.output_bias is not None:
correction = (correction + self.output_bias.float()).bfloat16()
return state, corrected_logits(base, correction, self.softcap)
21 changes: 15 additions & 6 deletions ignition/components/ring_commit.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,15 @@
def _commit(Staged, Backing, Offsets, Rows, Starts, Lengths, ROW_STRIDE: tl.constexpr,
STAGED_STRIDE: tl.constexpr, REGION_STRIDE: tl.constexpr,
ROWS_STRIDE: tl.constexpr, STARTS_STRIDE: tl.constexpr, LENGTHS_STRIDE: tl.constexpr,
WINDOW: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr):
WINDOW: tl.constexpr, CANDIDATES: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr):
item, request, region = tl.program_id(0), tl.program_id(1), tl.program_id(2)
length = tl.load(Lengths + request * LENGTHS_STRIDE)
start, row = tl.load(Starts + request * STARTS_STRIDE), tl.load(Rows + request * ROWS_STRIDE)
token = tl.maximum(0, length - WINDOW) + item
byte = tl.arange(0, BLOCK)
live = (token < length) & (row > 0) & (byte < RECORD)
value = tl.load(Staged + region * REGION_STRIDE + request * STAGED_STRIDE
+ (WINDOW + token) * RECORD + byte, live, other=0)
+ (CANDIDATES + token) * RECORD + byte, live, other=0)
offset = tl.load(Offsets + region)
tl.store(Backing + row * ROW_STRIDE + offset + ((start + token) % WINDOW) * RECORD + byte,
value, live)
Expand All @@ -35,9 +35,15 @@ class PreparedRingCommit:
Admission validates distinct positive destination rows and bounds accepted
counts by token_capacity. Row zero and unaccepted ring records are preserved.
The complete backing tensor exposes mutations to forward compilation.

Each request's staging holds its candidate records from record index
`candidate_offset`, which defaults to `window_tokens` for staging that
places the candidates after one window of history. Position `p` commits to
ring slot `p mod window_tokens`.
"""

def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None):
def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None,
candidate_offset=None):
if (backing.device.type != "cuda" or backing.dtype != torch.uint8 or backing.ndim != 2
or backing.stride(1) != 1 or backing.stride(0) < backing.shape[1]
or rows.ndim != 1 or rows.numel() == 0 or starts.shape != rows.shape
Expand All @@ -46,6 +52,9 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok
raise ValueError("ring commits require byte rows and matching int64 row/position vectors")
if any(type(value) is not int or value <= 0 for value in (token_capacity, window_tokens, record_bytes)):
raise ValueError("ring and staging capacities must be positive integers")
candidate_offset = window_tokens if candidate_offset is None else candidate_offset
if type(candidate_offset) is not int or candidate_offset < 0:
raise ValueError("staged candidates start at a nonnegative record index")
offsets = tuple(offsets)
size = window_tokens * record_bytes
if (not offsets or any(type(offset) is not int or offset < 0 or offset + size > backing.shape[1]
Expand All @@ -57,12 +66,12 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok
record_numeric_tensor(self.offsets)
from ignition.preparation.scratch import bind_scratch

shape = (len(offsets), rows.numel(), (window_tokens + token_capacity) * record_bytes)
shape = (len(offsets), rows.numel(), (candidate_offset + token_capacity) * record_bytes)
self.staging_shape, self.rows_shape = shape, tuple(rows.shape)
self.staging = bind_scratch({"staging": (shape, torch.uint8)}, device=backing.device,
workspace=None if staging is None else {"staging": staging})["staging"]
self.staging.zero_()
self.window_tokens, self.record_bytes = window_tokens, record_bytes
self.window_tokens, self.record_bytes, self.candidate_offset = window_tokens, record_bytes, candidate_offset
self.grid = (min(token_capacity, window_tokens), rows.numel(), len(offsets))

def functional(self):
Expand All @@ -76,4 +85,4 @@ def __call__(self, counts, *, staging, rows, starts):
staging, self.backing, self.offsets, rows, starts, counts,
self.backing.stride(0), staging.stride(1), staging.stride(0),
rows.stride(0), starts.stride(0), counts.stride(0),
self.window_tokens, self.record_bytes, triton.next_power_of_2(self.record_bytes))
self.window_tokens, self.candidate_offset, self.record_bytes, triton.next_power_of_2(self.record_bytes))
24 changes: 18 additions & 6 deletions ignition/components/window_staging.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ class Query:
window: int
history_dtype: str
first_position: int
lookback: int


CONTRACT = native_contract(component_id="flare.window_stage", query_type=Query, tactics=lambda q: (0,))
Expand Down Expand Up @@ -50,10 +51,12 @@ def compile_staging(payload, ordinal):
tuple(cute.sym_int(64) for _ in range(rank - 1)) + (1,), assumed_align=4)
for rank, dtype in _operands(q.history_dtype))
with torch.cuda.device(ordinal):
# The kernel takes a lookback only when it differs from the history length.
lookback = {} if q.lookback == q.window else {"lookback": q.lookback}
return compile_kernel(WindowStageKernel(heads=q.heads, dim=q.dim, capacity=q.capacity,
window=q.window, first_position=q.first_position), *operands, cutlass.Int32(1), current_cuda_stream(),
compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2,
(q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position)))
window=q.window, first_position=q.first_position, **lookback), *operands, cutlass.Int32(1),
current_cuda_stream(), compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2,
(q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position, q.lookback)))


@dataclass(frozen=True)
Expand Down Expand Up @@ -113,10 +116,16 @@ def _overlap(left, right):


class PreparedWindowStaging:
"""Gather paired history and candidates with BF16 selection and FP8 saturation."""
"""Gather paired history and candidates with BF16 selection and FP8 saturation.

The history's second dimension is its ring: position `p` occupies slot `p`
modulo that length. Each request's output lists positions in order from
`max(start - lookback, first_position)`; `lookback` defaults to the ring
length and may be shorter when the ring retains more than attention reads.
"""

def __init__(self, history_key, history_value, key, value, output_key, output_value, metadata, *,
counts=None, first_position=0, session, name):
counts=None, first_position=0, lookback=None, session, name):
from b12x.attention.paged_kv_write._preparation import _check_layout

m = metadata
Expand All @@ -133,6 +142,9 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va
or output_key.shape[0] * output_key.shape[1] % rows or output_key.numel() == 0
or type(first_position) is not int):
raise ValueError("window staging requires matching positive BF16 candidates, BF16/FP8 history and FP8 output geometry")
lookback = history_key.shape[1] if lookback is None else lookback
if type(lookback) is not int or not 0 < lookback <= history_key.shape[1]:
raise ValueError("window staging looks back over a positive count of at most the history's positions")
capacity = output_key.shape[0] * output_key.shape[1] // rows
self.page_shapes = tuple(output_key.shape), tuple(output_value.shape)
outputs = tuple(t.view(rows, capacity, *key.shape[1:]) for t in (output_key, output_value))
Expand All @@ -152,7 +164,7 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va
if _overlap(*outputs):
raise ValueError("window staging outputs must not overlap")
q = Query(key.shape[1], key.shape[2], capacity, history_key.shape[1],
str(history_key.dtype).removeprefix("torch."), first_position)
str(history_key.dtype).removeprefix("torch."), first_position, lookback)
payload = CONTRACT.encode_query(q)
arguments = self.arguments
device_index = key.device.index
Expand Down
Loading