diff --git a/docs/cli-design.md b/docs/cli-design.md index cde639a96..23b3e74cf 100644 --- a/docs/cli-design.md +++ b/docs/cli-design.md @@ -448,8 +448,8 @@ explicit flags take precedence. | `--no-mtp` | Disable multi-token speculation: load no draft model, keep no verification rows, and decode one token per step. | Speculation on | | `--mtp-only` | Omit MTP-disabled prefill, decode, maintenance, and mixed graphs during preparation. Retain width-one speculative rounds and vision graphs. Requires positive draft depth; rejects `--no-mtp`. The core cannot load with native `--mtp-draft-tokens 0`. | Capture both execution modes | | `--draft-body-quantization {mxfp8}` | Prepare eligible DFlash2 body context, attention latent/query/output, and MLP projections as MXFP8, independently of target conversion and proposal-head precision. Kimi K3.1 Pro requires `--speculative-method dflash2` and positive learned draft depth; DSpark and other recipes reject it. Normalization, convolution, candidate-selector, and absorbed key/value weights retain their precision. Omission preserves checkpoint precision or the existing `online_mxfp8_scope all` behavior. | No additional override | -| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. This does not quantize the draft body. | Checkpoint precision | -| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Requires positive draft depth. | Checkpoint precision | +| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. Inkling 2 Small NVFP4 supports both. This does not quantize the draft body. | Checkpoint precision | +| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Inkling 2 Small NVFP4 supports FP8 for its recurrent correction head. Requires positive draft depth. | Checkpoint precision | | `--vocabulary-parallel-sampling` | Prepare vocabulary-sharded DSpark proposals and block rejection sampling. DeepSeek V4.1 on four GPUs only; three-GPU preparation rejects it. Requires positive draft depth and recipe sampling defaults without top-k, top-p, or min-p filtering, and rejects `--markov-head-quantization nvfp4`. | Full-vocabulary sampling | | `--mtp-draft-tokens N` | Most draft tokens verified in one round; see `MODEL --help` for its limit. Verification capacities cover counts from 0 through `N` within the speculative token capacity. Each complete speculative program records its actual proposal depth independently of target width and aggregate rows. Prefix selection uses measured complete-program cost and completed acceptance or confidence. Zero retains draft state with width-one execution; `--no-mtp` excludes draft weights and state. | Recipe declaration | | `--speculative-token-capacity N` | Maximum physical target-token rows in wider speculative graphs; width one covers every request bucket. Applies to every speculative recipe. | Recipe declaration: 32 for Muse Spark and 128 for the other recipes that declare one. Gemma 4 declares none and derives it from concurrency and draft depth (6 at its defaults) | diff --git a/docs/model-preparation.md b/docs/model-preparation.md index 1771ce7a2..1457b42d1 100644 --- a/docs/model-preparation.md +++ b/docs/model-preparation.md @@ -1325,6 +1325,18 @@ reasoning, tool calls, and structured responses. Sampling defaults are temperature 1.0 and top-p 1.0; the prompt encoder defaults to reasoning effort 0.9 when the request does not specify one. +Each drafting round projects eight block rows through the vocabulary head and +then, for each of up to seven draft positions, corrects those logits with the +recurrent head: its output map folded through the vocabulary head, one +vocabulary row by 256 values. Both read BF16 by default, 463 MB and 25.7 MB +per rank. `--draft-head-quantization nvfp4` or `mxfp8` prepares a quantized +copy of the target's vocabulary shard that only proposals read, and +`--markov-head-quantization fp8` stores the folded correction head as E4M3 +values with one FP32 scale per vocabulary row. Both are computed at load from +checkpoint values; target verification keeps the BF16 head, so a quantized +proposal head changes which tokens are proposed and how often they are +accepted, never the output distribution. The default omits both flags. + Export with default tuning and Torch compilation: ```sh diff --git a/ignition/components/parallel_draft.py b/ignition/components/parallel_draft.py index fcf439d39..34652bbdc 100644 --- a/ignition/components/parallel_draft.py +++ b/ignition/components/parallel_draft.py @@ -442,11 +442,11 @@ def propose(self, advance_rng=True, allowed=None, *, probabilities): hidden = features["hidden"].view(b, block, -1) context = features["recurrent_context"] state = self.recurrent.begin(context) + # The draft states and target context do not depend on the draws: project them once. + fixed = self.recurrent.project_context(hidden[:, o.logit_offset:], context) scores = torch.empty_like(draft_logits)[:, o.logit_offset:o.logit_offset + w - 1] for index in range(w - 1): - state, logits = self.recurrent.step(state, markov_token, - hidden[:, index + o.logit_offset].contiguous(), context, - base[:, index]) + state, logits = self.recurrent.step(state, markov_token, fixed[:, index], base[:, index]) scores[:, index].copy_(logits) self.full_proposal.run_step(index, scores, drafts, pending=markov_token, seeds=seed, offsets=base_offset, probabilities=probabilities, valid=valid, diff --git a/ignition/components/recurrent_proposal.py b/ignition/components/recurrent_proposal.py index d76b02730..8c21b4465 100644 --- a/ignition/components/recurrent_proposal.py +++ b/ignition/components/recurrent_proposal.py @@ -7,6 +7,7 @@ from .embedding import PreparedEmbedding from .linear import UnquantizedProjection +from .markov_sampling import PreparedRowScaledFP8MarkovProjection, RowScaledFP8MarkovWeight def recurrent_gate(raw, state): @@ -27,33 +28,61 @@ def corrected_logits(base, correction, softcap): class PreparedRecurrentProposalHead: - """Keep recurrence state inside one block while retaining its actual sampled q.""" + """Keep recurrence state inside one block while retaining its actual sampled q. - def __init__(self, weights, tokens, *, session, name): + The recurrence input has a part that depends on the previous draw (state and predecessor + embedding) and a part that does not (draft hidden state and target context). `project_context` + computes the second part for every position of the block in one projection; `step` adds the + first. `weights.head` maps the recurrent output to vocabulary corrections as BF16 rows, or as + row-scaled E4M3 rows whose FP32 product takes the BF16 bias before rounding to BF16. + """ + + def __init__(self, weights, tokens, *, positions, session, name): rows, device = tokens.numel(), tokens.device rank = weights.embedding.shape[1] hidden = weights.context_weight.shape[1] // 2 - self.rank, self.softcap = rank, weights.softcap + if type(positions) is not int or positions < 1: + raise ValueError("a recurrent proposal head corrects at least one position per row") + self.rank, self.softcap, self.positions = rank, weights.softcap, positions self.context_bias = weights.bias embedding = torch.empty((rows, rank), dtype=torch.bfloat16, device=device) dynamic = torch.empty((rows, 2 * rank), dtype=torch.bfloat16, device=device) - context = torch.empty((rows, 2 * hidden), dtype=torch.bfloat16, device=device) + context = torch.empty((rows * positions, 2 * hidden), dtype=torch.bfloat16, device=device) + fixed = torch.empty((rows * positions, 3 * rank), dtype=torch.float32, device=device) raw = torch.empty((rows, 3 * rank), dtype=torch.float32, device=device) - correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device) self.embed = PreparedEmbedding(weights.embedding, tokens, embedding, session=session, name=name + ".embedding").functional(session=session, dynamic_inputs=True) - self.context = UnquantizedProjection(context, weights.context_weight, raw, + self.context = UnquantizedProjection(context, weights.context_weight, fixed, session=session, name=name + ".context").functional(session=session) self.dynamic = UnquantizedProjection(dynamic, weights.state_weight, raw, session=session, name=name + ".dynamic").functional(session=session) - self.output = UnquantizedProjection(embedding, weights.head, correction, - bias=weights.head_bias, session=session, name=name + ".output").functional(session=session) + self.output_bias = None + if isinstance(weights.head, RowScaledFP8MarkovWeight): + correction = torch.empty((rows, weights.head.out_features), dtype=torch.float32, device=device) + self.output = PreparedRowScaledFP8MarkovProjection(embedding, weights.head, correction, + session=session, name=name + ".output").functional(session=session) + self.output_bias = weights.head_bias + else: + correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device) + self.output = UnquantizedProjection(embedding, weights.head, correction, + bias=weights.head_bias, session=session, name=name + ".output").functional(session=session) def begin(self, hidden): return torch.zeros((hidden.shape[0], self.rank), dtype=torch.bfloat16, device=hidden.device) - def step(self, state, previous, hidden, context, base): - fixed = self.context(torch.cat((hidden, context), -1)) + self.context_bias + def project_context(self, hidden, context): + """Return FP32 [rows, positions, 3 * rank] recurrence inputs from [rows, positions, hidden] draft states.""" + rows, positions, width = hidden.shape + if positions != self.positions: + raise ValueError("recurrent context projection covers every prepared block position") + source = torch.cat((hidden, context[:, None].expand(rows, positions, width)), -1) + fixed = self.context(source.view(rows * positions, 2 * width)) + self.context_bias + return fixed.view(rows, positions, 3 * self.rank) + + def step(self, state, previous, fixed, base): dynamic = self.dynamic(torch.cat((state, self.embed(token_ids=previous)), -1)) state, latent = recurrent_gate(dynamic + fixed, state) - return state, corrected_logits(base, self.output(latent), self.softcap) + correction = self.output(latent) + if self.output_bias is not None: + correction = (correction + self.output_bias.float()).bfloat16() + return state, corrected_logits(base, correction, self.softcap) diff --git a/ignition/components/ring_commit.py b/ignition/components/ring_commit.py index ec5b69486..1eb37ca41 100644 --- a/ignition/components/ring_commit.py +++ b/ignition/components/ring_commit.py @@ -15,7 +15,7 @@ def _commit(Staged, Backing, Offsets, Rows, Starts, Lengths, ROW_STRIDE: tl.constexpr, STAGED_STRIDE: tl.constexpr, REGION_STRIDE: tl.constexpr, ROWS_STRIDE: tl.constexpr, STARTS_STRIDE: tl.constexpr, LENGTHS_STRIDE: tl.constexpr, - WINDOW: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr): + WINDOW: tl.constexpr, CANDIDATES: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr): item, request, region = tl.program_id(0), tl.program_id(1), tl.program_id(2) length = tl.load(Lengths + request * LENGTHS_STRIDE) start, row = tl.load(Starts + request * STARTS_STRIDE), tl.load(Rows + request * ROWS_STRIDE) @@ -23,7 +23,7 @@ def _commit(Staged, Backing, Offsets, Rows, Starts, Lengths, ROW_STRIDE: tl.cons byte = tl.arange(0, BLOCK) live = (token < length) & (row > 0) & (byte < RECORD) value = tl.load(Staged + region * REGION_STRIDE + request * STAGED_STRIDE - + (WINDOW + token) * RECORD + byte, live, other=0) + + (CANDIDATES + token) * RECORD + byte, live, other=0) offset = tl.load(Offsets + region) tl.store(Backing + row * ROW_STRIDE + offset + ((start + token) % WINDOW) * RECORD + byte, value, live) @@ -35,9 +35,15 @@ class PreparedRingCommit: Admission validates distinct positive destination rows and bounds accepted counts by token_capacity. Row zero and unaccepted ring records are preserved. The complete backing tensor exposes mutations to forward compilation. + + Each request's staging holds its candidate records from record index + `candidate_offset`, which defaults to `window_tokens` for staging that + places the candidates after one window of history. Position `p` commits to + ring slot `p mod window_tokens`. """ - def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None): + def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None, + candidate_offset=None): if (backing.device.type != "cuda" or backing.dtype != torch.uint8 or backing.ndim != 2 or backing.stride(1) != 1 or backing.stride(0) < backing.shape[1] or rows.ndim != 1 or rows.numel() == 0 or starts.shape != rows.shape @@ -46,6 +52,9 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok raise ValueError("ring commits require byte rows and matching int64 row/position vectors") if any(type(value) is not int or value <= 0 for value in (token_capacity, window_tokens, record_bytes)): raise ValueError("ring and staging capacities must be positive integers") + candidate_offset = window_tokens if candidate_offset is None else candidate_offset + if type(candidate_offset) is not int or candidate_offset < 0: + raise ValueError("staged candidates start at a nonnegative record index") offsets = tuple(offsets) size = window_tokens * record_bytes if (not offsets or any(type(offset) is not int or offset < 0 or offset + size > backing.shape[1] @@ -57,12 +66,12 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok record_numeric_tensor(self.offsets) from ignition.preparation.scratch import bind_scratch - shape = (len(offsets), rows.numel(), (window_tokens + token_capacity) * record_bytes) + shape = (len(offsets), rows.numel(), (candidate_offset + token_capacity) * record_bytes) self.staging_shape, self.rows_shape = shape, tuple(rows.shape) self.staging = bind_scratch({"staging": (shape, torch.uint8)}, device=backing.device, workspace=None if staging is None else {"staging": staging})["staging"] self.staging.zero_() - self.window_tokens, self.record_bytes = window_tokens, record_bytes + self.window_tokens, self.record_bytes, self.candidate_offset = window_tokens, record_bytes, candidate_offset self.grid = (min(token_capacity, window_tokens), rows.numel(), len(offsets)) def functional(self): @@ -76,4 +85,4 @@ def __call__(self, counts, *, staging, rows, starts): staging, self.backing, self.offsets, rows, starts, counts, self.backing.stride(0), staging.stride(1), staging.stride(0), rows.stride(0), starts.stride(0), counts.stride(0), - self.window_tokens, self.record_bytes, triton.next_power_of_2(self.record_bytes)) + self.window_tokens, self.candidate_offset, self.record_bytes, triton.next_power_of_2(self.record_bytes)) diff --git a/ignition/components/window_staging.py b/ignition/components/window_staging.py index 652240018..7ae6a3504 100644 --- a/ignition/components/window_staging.py +++ b/ignition/components/window_staging.py @@ -22,6 +22,7 @@ class Query: window: int history_dtype: str first_position: int + lookback: int CONTRACT = native_contract(component_id="flare.window_stage", query_type=Query, tactics=lambda q: (0,)) @@ -50,10 +51,12 @@ def compile_staging(payload, ordinal): tuple(cute.sym_int(64) for _ in range(rank - 1)) + (1,), assumed_align=4) for rank, dtype in _operands(q.history_dtype)) with torch.cuda.device(ordinal): + # The kernel takes a lookback only when it differs from the history length. + lookback = {} if q.lookback == q.window else {"lookback": q.lookback} return compile_kernel(WindowStageKernel(heads=q.heads, dim=q.dim, capacity=q.capacity, - window=q.window, first_position=q.first_position), *operands, cutlass.Int32(1), current_cuda_stream(), - compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2, - (q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position))) + window=q.window, first_position=q.first_position, **lookback), *operands, cutlass.Int32(1), + current_cuda_stream(), compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2, + (q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position, q.lookback))) @dataclass(frozen=True) @@ -113,10 +116,16 @@ def _overlap(left, right): class PreparedWindowStaging: - """Gather paired history and candidates with BF16 selection and FP8 saturation.""" + """Gather paired history and candidates with BF16 selection and FP8 saturation. + + The history's second dimension is its ring: position `p` occupies slot `p` + modulo that length. Each request's output lists positions in order from + `max(start - lookback, first_position)`; `lookback` defaults to the ring + length and may be shorter when the ring retains more than attention reads. + """ def __init__(self, history_key, history_value, key, value, output_key, output_value, metadata, *, - counts=None, first_position=0, session, name): + counts=None, first_position=0, lookback=None, session, name): from b12x.attention.paged_kv_write._preparation import _check_layout m = metadata @@ -133,6 +142,9 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va or output_key.shape[0] * output_key.shape[1] % rows or output_key.numel() == 0 or type(first_position) is not int): raise ValueError("window staging requires matching positive BF16 candidates, BF16/FP8 history and FP8 output geometry") + lookback = history_key.shape[1] if lookback is None else lookback + if type(lookback) is not int or not 0 < lookback <= history_key.shape[1]: + raise ValueError("window staging looks back over a positive count of at most the history's positions") capacity = output_key.shape[0] * output_key.shape[1] // rows self.page_shapes = tuple(output_key.shape), tuple(output_value.shape) outputs = tuple(t.view(rows, capacity, *key.shape[1:]) for t in (output_key, output_value)) @@ -152,7 +164,7 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va if _overlap(*outputs): raise ValueError("window staging outputs must not overlap") q = Query(key.shape[1], key.shape[2], capacity, history_key.shape[1], - str(history_key.dtype).removeprefix("torch."), first_position) + str(history_key.dtype).removeprefix("torch."), first_position, lookback) payload = CONTRACT.encode_query(q) arguments = self.arguments device_index = key.device.index diff --git a/ignition/recipes/inkling2/attention.py b/ignition/recipes/inkling2/attention.py index e0c885333..819ea633b 100644 --- a/ignition/recipes/inkling2/attention.py +++ b/ignition/recipes/inkling2/attention.py @@ -23,8 +23,10 @@ from ignition.components.head_gate import PreparedHeadGate from ignition.components.linear import UnquantizedProjection from ignition.components.paged_attention import PreparedPagedAttention, PreparedPagedKVWriter +from ignition.components.ring_commit import PreparedRingCommit from ignition.components.rms_norm import PreparedRMSNorm from ignition.components.state import gather_rows, scatter_rows +from ignition.components.window_staging import PreparedWindowStaging from ignition.preparation.persistent import persistent_empty from .arithmetic import l2_clip, log_scaling @@ -125,7 +127,9 @@ class LayerCache: Local layers also expose each continuation row's K and V rings as bytes (`key_rings`, `value_rings`, `[rows, ring bytes]`), which prefill reads and - rewrites one request row at a time. + rewrites one request row at a time, and the continuation rows with the byte + offsets of the K and V rings (`rows`, `ring_offsets`), where verification + commits accepted positions. """ key: torch.Tensor @@ -135,13 +139,17 @@ class LayerCache: ring_row_pages: int = 0 key_rings: torch.Tensor | None = None value_rings: torch.Tensor | None = None + rows: torch.Tensor | None = None + ring_offsets: tuple[int, int] | None = None @classmethod def of(cls, pool, layer): if pool.text.local(layer): return cls(pool.ring_pages(layer, "key"), pool.ring_pages(layer, "value"), pool.history(layer, "key_convolution"), pool.history(layer, "value_convolution"), - pool.ring_row_pages, pool.ring_rows(layer, "key"), pool.ring_rows(layer, "value")) + pool.ring_row_pages, pool.ring_rows(layer, "key"), pool.ring_rows(layer, "value"), + pool.reserved_backing, + tuple(pool.layout.region(f"layer{layer}.{kind}").offset for kind in ("key", "value"))) return cls(pool.cache(layer, "key"), pool.cache(layer, "value"), pool.history(layer, "key_convolution"), pool.history(layer, "value_convolution")) @@ -230,6 +238,16 @@ def example(role, shape, dtype=torch.bfloat16): cache_value = example("inkling.attention.staged_value", (count, PAGE_TOKENS, self.kv_heads, self.dim), KV_DTYPE) pages = torch.arange(count, dtype=torch.int32, device=device).view(requests, self.staged_pages) self.staged_table = pages + if mode == "verify": + # One kernel stages each request's preceding window and its rows from the K and V rings; + # one kernel commits the accepted rows' records to both rings. + shape = (-1, self.ring, self.kv_heads, self.dim) + self.staging = PreparedWindowStaging(cache.key_rings.view(KV_DTYPE).view(shape), + cache.value_rings.view(KV_DTYPE).view(shape), key, value, cache_key, cache_value, metadata, + lookback=self.window - 1, session=session, name=f"{name}.staging").functional(session=session) + self.ring_commit = PreparedRingCommit(cache.rows, cache.ring_offsets, metadata.state_rows, + metadata.starts, token_capacity=metadata.max_query_tokens, window_tokens=self.ring, + record_bytes=self.kv_heads * self.dim * KV_DTYPE.itemsize, candidate_offset=0).functional() else: cache_key, cache_value = cache.key, cache.value pages = (example("inkling.attention.ring_table", (requests, self.ring // PAGE_TOKENS), torch.int32) @@ -336,10 +354,8 @@ def _decode(self, v, m): def _verify(self, v, m): if self.local: - cache_key, lengths = stage_batch(self.cache.key_rings, v["key"], m, - window=self.window, pages=self.staged_pages) - cache_value, _ = stage_batch(self.cache.value_rings, v["value"], m, - window=self.window, pages=self.staged_pages) + cache_key, cache_value = self.staging(v["key"], v["value"], metadata=m) + lengths = staged_lengths(m, self.window) pages = self.staged_table kwargs = dict(k_cache=cache_key, v_cache=cache_value) else: @@ -355,38 +371,26 @@ def commit(self, values, counts, metadata): self.key_convolution.commit(values["key_previous"], values["key_raw"], metadata, counts=counts) self.value_convolution.commit(values["value_previous"], values["value_raw"], metadata, counts=counts) if self.local: - retain_batch(self.cache.key_rings, values["key"], counts, metadata) - retain_batch(self.cache.value_rings, values["value"], counts, metadata) + records = accepted_records(values["key"], values["value"], metadata) + self.ring_commit(counts, staging=records, rows=metadata.state_rows, starts=metadata.starts) -def stage_batch(rings, values, m, *, window, pages): - rows, capacity = m.starts.numel(), pages * PAGE_TOKENS - ring = torch.empty((rows, rings.shape[1]), dtype=torch.uint8, device=rings.device) - gather_rows(rings, m.state_rows, ring) - ring = ring.view(KV_DTYPE).view(rows, -1, *values.shape[1:]) +def staged_lengths(m, window): + """Each request's staged key count: its preceding `window - 1` positions, bounded by its start, and its rows.""" first = (m.starts - window + 1).clamp_min(0) - positions = first[:, None] + torch.arange(capacity, device=values.device) - previous = ring[torch.arange(rows, device=values.device)[:, None], positions.remainder(ring.shape[1])].to(values.dtype) - index = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) - staged = torch.where((positions < m.starts[:, None])[:, :, None, None], previous, values[index]) - lengths = torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() - staged = torch.where((positions < (m.starts + m.lengths)[:, None])[:, :, None, None], staged, 0) - return staged.clamp(-448, 448).to(KV_DTYPE).view(rows * pages, PAGE_TOKENS, *values.shape[1:]), lengths - - -def retain_batch(rings, values, counts, m): - rows = m.starts.numel() - previous = torch.empty((rows, rings.shape[1]), dtype=torch.uint8, device=rings.device) - gather_rows(rings, m.state_rows, previous) - ring = previous.view(KV_DTYPE).view(rows, -1, *values.shape[1:]) - capacity = ring.shape[1] - first = m.starts + counts - capacity - positions = first[:, None] + (torch.arange(capacity, device=values.device) - first[:, None]).remainder(capacity) - indices = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) - current = values[indices].clamp(-448, 448) - keep = ((positions < m.starts[:, None]) | (positions < 0))[:, :, None, None] - retained = torch.where(keep, ring.to(values.dtype), current).to(KV_DTYPE) - scatter_rows(retained.view(torch.uint8).view(rows, -1), m.state_rows, rings, counts > 0) + return torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() + + +def accepted_records(key, value, m): + """FP8 K and V records of each request's rows, as bytes `[2, requests, width * record bytes]`. + + Record `t` of a request is its row `t`; columns past a request's rows hold other + requests' records, which a commit bounded by the request's accepted count never reads. + """ + width = m.max_query_tokens + index = (m.query_start_loc[:-1, None] + torch.arange(width, device=key.device)).clamp(0, key.shape[0] - 1) + records = torch.stack((key[index], value[index])).clamp(-448, 448).to(KV_DTYPE) + return records.view(torch.uint8).view(2, index.shape[0], -1) def _page_slots(m): diff --git a/ignition/recipes/inkling2/bootstrap.py b/ignition/recipes/inkling2/bootstrap.py index 8210ca87e..cb7f6ca7d 100644 --- a/ignition/recipes/inkling2/bootstrap.py +++ b/ignition/recipes/inkling2/bootstrap.py @@ -75,6 +75,9 @@ class PreparationConfig: speculative_token_capacity: int | None = 64 speculative_graph_budget: int = 64 mtp_only: bool = False + # Proposal-only quantized copies derived from checkpoint values at load; None keeps BF16. + draft_head_quantization: str | None = None + markov_head_quantization: str | None = None benchmark_selection: object | None = None def __post_init__(self): @@ -107,6 +110,13 @@ def __post_init__(self): raise ValueError("--mtp-only requires enabled learned speculation with positive draft depth") if self.drafting and (not self.speculative_widths or not set(self.speculative_widths) <= set(range(1, 9))): raise ValueError("Inkling 2 verification widths must be in [1, 8]") + if self.draft_head_quantization not in (None, "nvfp4", "mxfp8"): + raise ValueError("Inkling 2 draft head quantization must be omitted, nvfp4 or mxfp8") + if self.markov_head_quantization not in (None, "fp8"): + raise ValueError("Inkling 2 recurrent correction head quantization must be omitted or fp8") + if (self.draft_head_quantization is not None or self.markov_head_quantization is not None) and ( + not self.drafting or self.draft_tokens < 1): + raise ValueError("proposal head quantization requires learned speculation with positive draft depth") if self.snapshot_budget_bytes <= 0 or self.headroom_bytes < 0: raise ValueError("snapshot budget must be positive and headroom nonnegative") object.__setattr__(self, "image_concurrency", image_capacity(self.max_active, self.image_concurrency) if self.vision else 0) @@ -217,7 +227,9 @@ def prepare_rank(snapshot, config, *, rank, group, progress=None, core_storage): raise ValueError("a decoder-layer prefix must include a global-attention layer, whose K/V use pages") with phase(progress, "Load weights"): weights = ModelWeights.load(manifest, device, shard=shard, layers=config.decoder_layers, - progress=progress, drafting=config.drafting, vision=config.vision) + progress=progress, drafting=config.drafting, vision=config.vision, + draft_head_quantization=config.draft_head_quantization, + markov_head_quantization=config.markov_head_quantization) weights_bytes = torch.cuda.memory_allocated(device) if torch.cuda.get_device_properties(device).is_integrated: from ignition.preparation.host_memory import release_page_cache @@ -406,7 +418,9 @@ def add(name, forward, kind, rows, tokens, fields, enabled=None, packed=0): prefill_capacities=prefills, free_device_bytes=free, total_device_bytes=total, allocated_tensor_bytes=torch.cuda.memory_allocated(device), reserved_tensor_bytes=torch.cuda.memory_reserved(device), headroom_bytes=config.headroom_bytes, - tuning=tuning, compiled_decoder=config.compile_decoder) + tuning=tuning, compiled_decoder=config.compile_decoder, + draft_head_quantization=config.draft_head_quantization, + markov_head_quantization=config.markov_head_quantization) # Each decoder layer's weights form one stable payload; vocabulary shards, norms and the output query share misc. rebuild_weights = declare_weights(weights, components=("embedding", "embedding_norm", "final_norm", "output_query", "unembedding", "draft", "vision")) diff --git a/ignition/recipes/inkling2/dspark.py b/ignition/recipes/inkling2/dspark.py index 93944531e..83095c9e6 100644 --- a/ignition/recipes/inkling2/dspark.py +++ b/ignition/recipes/inkling2/dspark.py @@ -9,7 +9,7 @@ from ignition.components.collectives import PreparedAllReduce from ignition.components.embedding import PreparedEmbedding -from ignition.components.linear import UnquantizedProjection +from ignition.components.linear import QuantizedProjection, UnquantizedProjection from ignition.components.mlp import MLPScratch, PreparedMLP from ignition.components.paged_attention import PreparedPagedAttention, PreparedPagedKVWriter from ignition.components.recurrent_proposal import PreparedRecurrentProposalHead @@ -18,7 +18,8 @@ from ignition.components.state import gather_rows, save_hidden from ignition.preparation.persistent import persistent_copy, persistent_zeros -from .dspark_state import context_slots, mix_features, rotary, stage_paged_tail, stage_window +from .dspark_state import (block_inputs, context_slots, mix_features, rotary_table, rotate, stage, tail_plan, + window_plan) def inverse_frequencies(draft, device): @@ -66,11 +67,12 @@ def __call__(self, auxiliary, counts, metadata): m = metadata full_slots = context_slots(m, counts) local_slots = context_slots(m, counts, window=self.window) + table = rotary_table(m.positions, self.inverse) for index, (projection, key_norm, context_norm, writer, local) in enumerate(self.stages): context = mix_features(auxiliary, self.alpha[index], context_norm, self.eps) key, value = projection(context).chunk(2, -1) key = key_norm(key.contiguous().view(-1, self.key_shape[-1])).view(self.key_shape) - writer(rotary(key, m.positions, self.inverse), value.contiguous().view(self.key_shape), + writer(rotate(key, table), value.contiguous().view(self.key_shape), local_slots if local else full_slots) indices = (m.query_start_loc[:-1].long() + counts.long() - 1).clamp(0, auxiliary.shape[0] - 1) context = mix_features(auxiliary[indices], self.alpha[-1], self.recurrent_norm, self.eps) @@ -82,6 +84,8 @@ def scratch_tensors(self): @dataclass(frozen=True) class DraftScratch: + # One row per request: the block's first position is the only one that embeds a token. + anchor: torch.Tensor hidden: torch.Tensor accumulator: torch.Tensor qkv: torch.Tensor @@ -99,7 +103,9 @@ def allocate(cls, config, shard, tokens, block, device): start, end = shard.padded_partition(config.vocabulary, alignment=64) def empty(*shape, dtype=torch.bfloat16): return torch.empty(shape, dtype=dtype, device=device) - return cls(empty(tokens, hidden), empty(tokens, hidden, dtype=torch.float32), + if tokens % block: + raise ValueError("Inkling2 DSpark scratch holds whole blocks") + return cls(empty(tokens // block, hidden), empty(tokens, hidden), empty(tokens, hidden, dtype=torch.float32), empty(tokens, (q + 2 * kv) * d.head_dim), empty(tokens, q, d.head_dim), empty(tokens, kv, d.head_dim), empty(tokens, q, d.head_dim), empty(tokens, hidden), MLPScratch.allocate(tokens, hidden, d.text["intermediate_size"] // shard.world_size, device), @@ -108,7 +114,11 @@ def empty(*shape, dtype=torch.bfloat16): def rows(self, tokens, samples): if samples != tokens: raise ValueError("Inkling2 DSpark projects every block position") - return type(self)(*(self.mlp.rows(tokens) if field == "mlp" else getattr(self, field)[:tokens] + block = self.hidden.shape[0] // self.anchor.shape[0] + if tokens % block: + raise ValueError("Inkling2 DSpark scratch holds whole blocks") + counts = {"mlp": None, "anchor": tokens // block} + return type(self)(*(self.mlp.rows(tokens) if field == "mlp" else getattr(self, field)[:counts.get(field, tokens)] for field in self.__dataclass_fields__)) @@ -118,7 +128,6 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru d = config.draft self.query_shape, self.key_shape = tuple(scratch.query.shape), tuple(scratch.key.shape) self.window, self.local = d.window, index in d.local_layers - self.inverse = inverse_frequencies(d, scratch.hidden.device) self.qkv = UnquantizedProjection(scratch.hidden, weights.qkv, scratch.qkv, session=session, name=name + ".qkv").functional(session=session) self.query_norm = norm(scratch.query.flatten(0, 1), weights.query_norm, scratch.query.flatten(0, 1), @@ -134,6 +143,7 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru shape = (rows * self.pages_per_row, 128, *self.key_shape[1:]) key, value = (torch.empty(shape, dtype=torch.bfloat16, device=scratch.hidden.device) for _ in range(2)) tail = {} + self.staging = (window_plan, d.window, self.pages_per_row) else: self.key_cache, self.value_cache = pool.key(index, draft=True), pool.value(index, draft=True) self.pages = None @@ -142,6 +152,7 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru shape = (rows * self.tail_pages, key.shape[1], *self.key_shape[1:]) tail = {role: torch.empty(shape, dtype=key.dtype, device=key.device) for role in ("tail_key", "tail_value")} + self.staging = (tail_plan, key.shape[1], self.tail_pages) self.attention = PreparedPagedAttention(scratch.query, key, value, self.pages if self.local else metadata.page_table, metadata.lengths, metadata.query_start_loc, scratch.attended, mode="verify", width=d.block_size, window=-1, causal=False, @@ -155,26 +166,23 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru self.reduce = PreparedAllReduce(scratch.partial, scratch.hidden, runtime=runtime, coordinator=coordinator, session=session, name=name + ".reduce").functional(session=session) - def __call__(self, hidden, metadata): + def __call__(self, hidden, metadata, plan, table): + """Attend with `plan`, this layer's staging plan, and `table`, the rows' rotary cosines and sines.""" m = metadata q_width = self.query_shape[1] * self.query_shape[2] kv_width = self.key_shape[1] * self.key_shape[2] q, k, v = self.qkv(hidden).split((q_width, kv_width, kv_width), -1) q = self.query_norm(q.contiguous().view(-1, self.query_shape[-1])).view(self.query_shape) k = self.key_norm(k.contiguous().view(-1, self.key_shape[-1])).view(self.key_shape) - q, k = rotary(q, m.positions, self.inverse), rotary(k, m.positions, self.inverse) + q, k = rotate(q, table), rotate(k, table) v = v.contiguous().view(self.key_shape) if self.local: - key = stage_window(self.key_cache, k, m, self.pages_per_row) - value = stage_window(self.value_cache, v, m, self.pages_per_row) - lengths = torch.where(m.lengths > 0, m.starts.clamp(max=self.window).int() + m.lengths, 0) + key, value = stage(self.key_cache, k, plan), stage(self.value_cache, v, plan) tail = {} else: key, value = self.key_cache, self.value_cache - tail = dict(tail_key=stage_paged_tail(key, k, m, self.tail_pages), - tail_value=stage_paged_tail(value, v, m, self.tail_pages)) - lengths = torch.where(m.lengths > 0, m.starts.int() + m.lengths, 0) - attended = self.attention(q, k_cache=key, v_cache=value, cache_seqlens=lengths, + tail = dict(tail_key=stage(key, k, plan), tail_value=stage(value, v, plan)) + attended = self.attention(q, k_cache=key, v_cache=value, cache_seqlens=plan.lengths, cu_seqlens_q=m.query_start_loc, **tail, **({} if self.local else {"page_table": m.page_table})) return self.reduce(self.output(attended.flatten(1))) @@ -195,13 +203,13 @@ def __init__(self, index, weights, owner, scratch, metadata, pool, *, session, n self.reduce = PreparedAllReduce(s.output, scratch.hidden, runtime=runtime, coordinator=coordinator, session=session, name=name + ".mlp.reduce").functional(session=session) - def __call__(self, hidden, residual, metadata): + def __call__(self, hidden, residual, metadata, plan, table): if residual is None: residual = hidden else: hidden = hidden.float() + residual.float() residual = hidden.bfloat16() - attended = self.attention(self.input_norm(hidden), metadata) + attended = self.attention(self.input_norm(hidden), metadata, plan, table) summed = attended.float() + residual.float() residual = summed.bfloat16() return self.reduce(self.mlp(self.post_norm(summed))), residual @@ -221,36 +229,50 @@ def __init__(self, weights, scratch, metadata, pool, *, runtime, coordinator, se first, _ = weights.shard.padded_partition(weights.config.vocabulary, alignment=64) self.vocabulary_start = persistent_copy(torch.tensor([first], dtype=torch.int64, device=device)) self.embedding_weight = weights.embedding - ids = metadata.token_ids.clamp(0, weights.embedding.shape[0] - 1) - self.embedding = PreparedEmbedding(weights.embedding, ids, scratch.hidden, session=session, + self.inverse = inverse_frequencies(d, device) + # Positions after the first take the learned mask, so only each request's first row is + # embedded, reduced across the vocabulary shards and normalized. + ids = metadata.token_ids[::self.block].clamp(0, weights.embedding.shape[0] - 1) + self.embedding = PreparedEmbedding(weights.embedding, ids, scratch.anchor, session=session, name=name + ".embed").functional(session=session, dynamic_inputs=True) - self.embed_reduce = PreparedAllReduce(scratch.hidden, scratch.hidden, runtime=runtime, + self.embed_reduce = PreparedAllReduce(scratch.anchor, scratch.anchor, runtime=runtime, coordinator=coordinator, session=session, name=name + ".embed.reduce").functional(session=session) - self.embed_norm = norm(scratch.hidden, weights.embedding_norm, scratch.hidden, + self.embed_norm = norm(scratch.anchor, weights.embedding_norm, scratch.anchor, eps=d.text["rms_norm_eps"], session=session, name=name + ".embed.norm") self.layers = [DraftLayer(i, layer, weights, scratch, metadata, pool, runtime=runtime, coordinator=coordinator, session=session, name=f"{name}.{i}", workspace=workspace) for i, layer in enumerate(weights.draft.layers)] + # Layers with equal history geometry share one staging plan per forward. + self.staging = tuple(dict.fromkeys(layer.attention.staging for layer in self.layers)) + self.layer_plans = tuple(self.staging.index(layer.attention.staging) for layer in self.layers) self.norm = norm(scratch.accumulator, weights.draft.norm, scratch.hidden, eps=d.text["rms_norm_eps"], session=session, name=name + ".norm") - self.head = UnquantizedProjection(scratch.hidden, weights.head, scratch.local_logits, - session=session, name=name + ".head").functional(session=session) + if weights.draft.draft_head is None: + self.head = UnquantizedProjection(scratch.hidden, weights.head, scratch.local_logits, + session=session, name=name + ".head").functional(session=session) + else: + # A quantized copy of the target's vocabulary shard serves proposals; verification keeps the target's. + self.head = QuantizedProjection(scratch.hidden, weights.draft.draft_head, scratch.local_logits, + session=session, name=name + ".head", activation_mode="a16", + workspace=workspace).functional(session=session) self.recurrent = PreparedRecurrentProposalHead(weights.draft.recurrent, - torch.zeros(metadata.lengths.numel(), dtype=torch.int64, device=device), session=session, name=name + ".recurrent") + torch.zeros(metadata.lengths.numel(), dtype=torch.int64, device=device), + positions=self.block - 1, session=session, name=name + ".recurrent") def release_outputs(self): return self def __call__(self, metadata): - ids = metadata.token_ids - self.vocabulary_start + ids = metadata.token_ids[::self.block] - self.vocabulary_start owned = (ids >= 0) & (ids < self.embedding_weight.shape[0]) local = self.embedding(token_ids=ids.clamp(0, self.embedding_weight.shape[0] - 1)) - hidden = self.embed_norm(self.embed_reduce(torch.where(owned[:, None], local, 0))) - positions = torch.arange(hidden.shape[0], device=hidden.device) % self.block - hidden = torch.where((positions == 0)[:, None], hidden, self.mask) + anchor = self.embed_norm(self.embed_reduce(torch.where(owned[:, None], local, 0))) + hidden = block_inputs(anchor, self.mask, self.block) + table = rotary_table(metadata.positions, self.inverse) + plans = [plan(metadata, *geometry) for plan, *geometry in self.staging] residual = None - for layer in self.layers: - hidden, residual = layer(hidden, residual, metadata) + for layer, plan in zip(self.layers, self.layer_plans): + hidden, residual = layer(hidden, residual, metadata, plans[plan], table) hidden = self.norm(hidden.float() + residual.float())[metadata.sample_indices] logits = self.head(hidden) context = torch.empty((metadata.state_rows.numel(), self.recurrent_context.shape[1]), diff --git a/ignition/recipes/inkling2/dspark_state.py b/ignition/recipes/inkling2/dspark_state.py index 71c9399f3..a6c9886a9 100644 --- a/ignition/recipes/inkling2/dspark_state.py +++ b/ignition/recipes/inkling2/dspark_state.py @@ -3,6 +3,8 @@ """Inkling2 draft feature mixing, rotary positions, and committed context selection.""" +from dataclasses import dataclass + import torch @@ -14,46 +16,78 @@ def mix_features(features, alpha, weight, eps): return (mixed * scale * weight.float()).bfloat16() -def rotary(values, positions, inverse): +def block_inputs(anchor, mask, block): + """Expand one normalized first-position embedding per request into its block of learned masks.""" + return torch.cat((anchor[:, None], mask.expand(anchor.shape[0], block - 1, -1)), 1).flatten(0, 1) + + +def rotary_table(positions, inverse): + """Return FP32 cosines and sines of each row's position, shared by every rotation of those rows.""" angle = positions.clamp_min(0).float()[:, None] * inverse[None] - cosine, sine = angle.cos()[:, None], angle.sin()[:, None] + return angle.cos()[:, None], angle.sin()[:, None] + + +def rotate(values, table): + cosine, sine = table left, right = values.float().chunk(2, -1) return torch.cat((left * cosine - right * sine, right * cosine + left * sine), -1).to(values.dtype) -def stage_window(history, values, metadata, pages): - """Stage the same committed suffix before every query in a noncausal block.""" +def rotary(values, positions, inverse): + return rotate(values, rotary_table(positions, inverse)) + + +@dataclass(frozen=True) +class StagingPlan: + """Where each slot of a request's private attention pages comes from. + + The plan depends on request metadata only, so one plan serves the key and value histories of + every draft layer with the same history geometry. A slot holds committed history + (`history[rows, slots]`), a row of the block being drafted (`values[indices]`), or zero. + `lengths` is each request's visible key count. + """ + + rows: torch.Tensor + slots: torch.Tensor + indices: torch.Tensor + committed: torch.Tensor + valid: torch.Tensor + lengths: torch.Tensor + page_tokens: int + + +def window_plan(metadata, window, pages, *, page_tokens=128): + """Plan the same committed suffix of a `window`-position ring before every query of a noncausal block.""" m = metadata - rows, window = m.starts.numel(), history.shape[1] - capacity = pages * 128 - offsets = torch.arange(capacity, device=values.device)[None] + offsets = torch.arange(pages * page_tokens, device=m.starts.device)[None] prefix = m.starts.clamp(max=window)[:, None] positions = m.starts[:, None] - prefix + offsets - previous = history[m.state_rows[:, None], positions.remainder(window)] - indices = (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, values.shape[0] - 1) - current = values[indices] - valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) - return torch.where(valid[:, :, None, None], - torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view( - rows * pages, 128, *values.shape[1:]) + return StagingPlan(m.state_rows[:, None], positions.remainder(window), + (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, m.positions.numel() - 1), + offsets < prefix, (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]), + torch.where(m.lengths > 0, m.starts.clamp(max=window).int() + m.lengths, 0), page_tokens) -def stage_paged_tail(history, values, metadata, pages): - """Copy the committed partial page and proposal rows into private attention pages.""" +def tail_plan(metadata, page_tokens, pages): + """Plan the committed partial page followed by the block's rows in private tail pages.""" m = metadata - rows, page_tokens = m.starts.numel(), history.shape[1] - offsets = torch.arange(pages * page_tokens, device=values.device)[None] + rows = m.starts.numel() + offsets = torch.arange(pages * page_tokens, device=m.starts.device)[None] prefix = m.starts.remainder(page_tokens)[:, None] columns = (m.starts // page_tokens).clamp(max=m.page_table.shape[1] - 1) - requests = torch.arange(rows, device=values.device) + requests = torch.arange(rows, device=m.starts.device) source_pages = m.page_table[requests, columns].long().clamp_min(0) - previous = history[source_pages[:, None], offsets.remainder(page_tokens)] - indices = (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, values.shape[0] - 1) - current = values[indices] - valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) - return torch.where(valid[:, :, None, None], - torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view( - rows * pages, page_tokens, *values.shape[1:]) + return StagingPlan(source_pages[:, None], offsets.remainder(page_tokens), + (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, m.positions.numel() - 1), + offsets < prefix, (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]), + torch.where(m.lengths > 0, m.starts.int() + m.lengths, 0), page_tokens) + + +def stage(history, values, plan): + """Copy the planned history and block rows into private attention pages without changing either source.""" + staged = torch.where(plan.valid[:, :, None, None], + torch.where(plan.committed[:, :, None, None], history[plan.rows, plan.slots], values[plan.indices]), 0) + return staged.view(-1, plan.page_tokens, *values.shape[1:]) def accepted_rows(metadata, counts, *, window=None): diff --git a/ignition/recipes/inkling2/dspark_weights.py b/ignition/recipes/inkling2/dspark_weights.py index 37ace67cb..8d7652ac3 100644 --- a/ignition/recipes/inkling2/dspark_weights.py +++ b/ignition/recipes/inkling2/dspark_weights.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 Flare community contributors # SPDX-License-Identifier: Apache-2.0 -"""Checkpoint-preserving tensor partitions for Inkling2's dense DSpark decoder.""" +"""Tensor partitions for Inkling2's dense DSpark decoder and its optional quantized proposal heads.""" from dataclasses import dataclass from functools import cached_property @@ -10,6 +10,8 @@ import torch +from ignition.components.markov_sampling import fp8_markov_head +from ignition.components.vocabulary import mxfp8_draft_head, nvfp4_draft_head from ignition.components.weights import DirectWeightLoader, MLPWeights from ignition.preparation.checkpoint import IndexedCheckpoint, _read_header from ignition.preparation.persistent import persistent_copy, persistent_empty @@ -74,25 +76,36 @@ class DraftLayerWeights: mlp: MLPWeights +DRAFT_HEAD_FORMATS = {"nvfp4": nvfp4_draft_head, "mxfp8": mxfp8_draft_head} + + +def _transient(shape, dtype, device): + return torch.empty(shape, dtype=dtype, device=device) + + @dataclass(frozen=True) class RecurrentWeights: embedding: torch.Tensor state_weight: torch.Tensor context_weight: torch.Tensor bias: torch.Tensor - head: torch.Tensor + # Vocabulary rows by recurrent rank: BF16, or row-scaled E4M3 when the correction head is quantized. + head: object head_bias: torch.Tensor context_norm: torch.Tensor softcap: float = 30. -def fold_recurrent_head(head, hidden_out, hidden_bias, *, chunk=4096): - """Fold the BF16 output map through the vocabulary head with FP64 products.""" +def fold_recurrent_head(head, hidden_out, hidden_bias, *, chunk=4096, allocate=persistent_empty): + """Fold the BF16 output map through the vocabulary head with FP64 products. + + `allocate` places the folded matrix; a caller that quantizes it passes a transient allocator. + """ if (head.ndim != 2 or hidden_out.ndim != 2 or hidden_out.shape[0] != head.shape[1] or hidden_bias.shape != (head.shape[1],) or any(t.dtype != torch.bfloat16 or t.device != head.device for t in (head, hidden_out, hidden_bias))): raise ValueError("recurrent output folding requires compatible BF16 output and vocabulary matrices") - weight = persistent_empty((head.shape[0], hidden_out.shape[1]), torch.bfloat16, head.device) + weight = allocate((head.shape[0], hidden_out.shape[1]), torch.bfloat16, head.device) bias = persistent_empty((head.shape[0],), torch.bfloat16, head.device) projection, offset = hidden_out.double(), hidden_bias.double() for start in range(0, head.shape[0], chunk): @@ -110,9 +123,22 @@ class DraftWeights: norm: torch.Tensor layers: tuple[DraftLayerWeights, ...] recurrent: RecurrentWeights + # Quantized copy of the target vocabulary shard that only proposals read; None projects with the target's. + draft_head: object | None = None @classmethod - def load(cls, snapshot, config, device, shard, *, target_head, progress=None): + def load(cls, snapshot, config, device, shard, *, target_head, progress=None, + draft_head_quantization=None, markov_head_quantization=None): + """Load one rank's drafter and derive its proposal heads from checkpoint values. + + `draft_head_quantization` (`nvfp4` or `mxfp8`) adds a quantized copy of `target_head` for the block's + vocabulary logits. `markov_head_quantization` (`fp8`) stores the recurrent correction head, the + output map folded through `target_head`, as row-scaled E4M3. The target head keeps BF16. + """ + if draft_head_quantization not in (None, *DRAFT_HEAD_FORMATS): + raise ValueError("Inkling2 draft head quantization must be omitted, nvfp4 or mxfp8") + if markov_head_quantization not in (None, "fp8"): + raise ValueError("Inkling2 recurrent correction head quantization must be omitted or fp8") snapshot = Path(snapshot) sidecar, tensors = _read_header(snapshot / "mtp.safetensors", "mtp.safetensors") manifest = DraftCheckpoint(snapshot, tensors, (sidecar,)) @@ -160,8 +186,16 @@ def vector(name, width): raise ValueError("Inkling2 predecessor embedding must be a BF16 vocabulary by recurrent-rank matrix") hidden_out = loader.load_tensor("markov_head.hidden_out.weight", temporary=True) hidden_bias = vector("markov_head.hidden_out.bias", hidden) - head, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias) + if markov_head_quantization is None: + head, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias) + else: + folded, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias, allocate=_transient) + head = fp8_markov_head(folded) + del folded recurrent = RecurrentWeights(embedding, persistent_copy(state[:, :2 * rank].contiguous()), persistent_copy(state[:, 2 * rank:].contiguous()), persistent_copy(vector("markov_head.rnn_proj.bias", 3 * rank).float()), head, head_bias, vector("markov_head.dflare_norm.weight", hidden)) - return cls(alpha, vector("mask_embed", hidden), vector("norm.weight", hidden), tuple(layers), recurrent) + draft_head = (None if draft_head_quantization is None + else DRAFT_HEAD_FORMATS[draft_head_quantization](target_head)) + return cls(alpha, vector("mask_embed", hidden), vector("norm.weight", hidden), tuple(layers), recurrent, + draft_head) diff --git a/ignition/recipes/inkling2/recipe.py b/ignition/recipes/inkling2/recipe.py index 192907937..29e3b9c01 100644 --- a/ignition/recipes/inkling2/recipe.py +++ b/ignition/recipes/inkling2/recipe.py @@ -22,12 +22,11 @@ def prepare(snapshot, plan, *, compile_forwards, kernel_cache, progress, retain_ if plan.prompt_logprobs: raise ValueError("Inkling 2 preparation does not capture prompt scoring graphs") speculation = plan.speculation - if speculation is not None and (speculation.draft_head_quantization is not None - or speculation.markov_head_quantization is not None): - raise ValueError("Inkling 2 retains the bundled draft checkpoint precision") drafting = {} if speculation is None else dict(speculative_widths=speculation.widths, speculative_fixed_width=speculation.fixed_width, speculative_token_capacity=plan.speculative_token_capacity, - speculative_graph_budget=speculation.graph_budget or 64) + speculative_graph_budget=speculation.graph_budget or 64, + draft_head_quantization=speculation.draft_head_quantization, + markov_head_quantization=speculation.markov_head_quantization) config = PreparationConfig(collective_tuning=plan.collective_tuning, collective_ladder=plan.collective_ladder, prompt_logprobs=plan.prompt_logprobs, compile_workers=plan.compile_workers, native_dense_activations=plan.native_dense_activations, @@ -54,6 +53,8 @@ def usage(prepared): RECIPE = Recipe(name="inkling2", served_model_name="Inkling-2-Small-NVFP4", prepare=prepare, usage=usage, sampling_defaults=SAMPLING_DEFAULTS, + # Proposal-only copies: the block's vocabulary head, and the recurrent correction head as row-scaled E4M3. + draft_head_quantizations=("nvfp4", "mxfp8"), markov_head_quantizations=("fp8",), checkpoints=(Checkpoint(alias="inkling-2-small-nvfp4", repository="/data/models/Inkling-2-Small-NVFP4", revision="local", architecture="Inkling2ForConditionalGeneration", stored_bytes=344_366_116_864, device_bytes=344_366_116_864, expansion_bytes=5_673_717_760, draft_bytes=2_684_203_392),), diff --git a/ignition/recipes/inkling2/weights.py b/ignition/recipes/inkling2/weights.py index 7a851c2d1..72731dba7 100644 --- a/ignition/recipes/inkling2/weights.py +++ b/ignition/recipes/inkling2/weights.py @@ -72,7 +72,8 @@ def embed_norm(self): return self.embedding_norm @classmethod - def load(cls, manifest, device, *, shard, progress=None, drafting=True, vision=True, layers=None): + def load(cls, manifest, device, *, shard, progress=None, drafting=True, vision=True, layers=None, + draft_head_quantization=None, markov_head_quantization=None): before = torch.cuda.memory_allocated(device) with InklingWeightLoader(manifest, device, shard=shard, progress=progress, vision=vision) as loader: c = loader.config @@ -105,7 +106,8 @@ def vocabulary_rows(name): draft = None if drafting: from .dspark_weights import DraftWeights - draft = DraftWeights.load(manifest.snapshot, c, device, shard, target_head=head, progress=progress) + draft = DraftWeights.load(manifest.snapshot, c, device, shard, target_head=head, progress=progress, + draft_head_quantization=draft_head_quantization, markov_head_quantization=markov_head_quantization) return cls(c, shard, loaded, embedding, embed_norm, norm, query, head, torch.cuda.memory_allocated(device) - before, draft, tower) diff --git a/tests/cli/test_inkling2.py b/tests/cli/test_inkling2.py index c614160f0..8d3472320 100644 --- a/tests/cli/test_inkling2.py +++ b/tests/cli/test_inkling2.py @@ -55,3 +55,63 @@ def capture_config(**kwargs): assert config.speculative_widths == tuple(range(1, 9)) assert config.speculative_token_capacity == 64 assert config.speculative_graph_budget == 64 + + +def test_inkling_declares_quantized_proposal_heads_with_checkpoint_precision_as_default(): + assert RECIPE.draft_head_quantizations == ("nvfp4", "mxfp8") + assert RECIPE.markov_head_quantizations == ("fp8",) + assert RECIPE.speculation.draft_head_quantization is None + assert RECIPE.speculation.markov_head_quantization is None + model = ResolvedModel(RECIPE, RECIPE.checkpoints[0]) + args = _parser().parse_args(["export", model.checkpoint.alias, "--draft-head-quantization", "nvfp4", + "--markov-head-quantization", "fp8"]) + speculation = preparation.speculation(args, model) + assert (speculation.draft_head_quantization, speculation.markov_head_quantization) == ("nvfp4", "fp8") + assert speculation.draft_tokens == 7 + + +@pytest.mark.parametrize("draft_head,markov_head", [(None, None), ("nvfp4", "fp8"), ("mxfp8", None), (None, "fp8")]) +def test_inkling_adapter_passes_proposal_head_formats_to_preparation(monkeypatch, tmp_path, draft_head, markov_head): + from ignition.recipes.inkling2 import bootstrap + + devices = tuple(gpu(i, 95) for i in range(4)) + speculation = replace(RECIPE.speculation, draft_head_quantization=draft_head, + markov_head_quantization=markov_head) + plan = derive(model=RECIPE, gpus=devices, selected=(0, 1, 2, 3), concurrency=8, speculation=speculation) + captured = [] + configuration = bootstrap.PreparationConfig + + class ConfigurationCaptured(Exception): + pass + + def capture_config(**kwargs): + captured.append(configuration(**kwargs)) + raise ConfigurationCaptured + + monkeypatch.setattr(bootstrap, "PreparationConfig", capture_config) + with pytest.raises(ConfigurationCaptured): + RECIPE.prepare(tmp_path, plan, compile_forwards=False, kernel_cache=None, progress=None) + config = captured.pop() + assert (config.draft_head_quantization, config.markov_head_quantization) == (draft_head, markov_head) + + +def test_inkling_plan_rejects_an_nvfp4_recurrent_correction_head(): + from ignition.errors import UsageError + + devices = tuple(gpu(i, 95) for i in range(4)) + speculation = replace(RECIPE.speculation, markov_head_quantization="nvfp4") + with pytest.raises(UsageError, match="--markov-head-quantization nvfp4"): + derive(model=RECIPE, gpus=devices, selected=(0, 1, 2, 3), concurrency=8, speculation=speculation) + + +@pytest.mark.parametrize("arguments,message", [ + (dict(draft_head_quantization="fp8"), "draft head quantization"), + (dict(markov_head_quantization="nvfp4"), "correction head quantization"), + (dict(draft_head_quantization="nvfp4", drafting=False), "positive draft depth"), + (dict(markov_head_quantization="fp8", speculative_widths=(1,)), "positive draft depth"), +]) +def test_inkling_preparation_config_validates_proposal_head_formats(arguments, message): + from ignition.recipes.inkling2.bootstrap import PreparationConfig + + with pytest.raises(ValueError, match=message): + PreparationConfig(**arguments) diff --git a/tests/components/test_inkling2_dspark.py b/tests/components/test_inkling2_dspark.py index bd5a0d430..c2f727db5 100644 --- a/tests/components/test_inkling2_dspark.py +++ b/tests/components/test_inkling2_dspark.py @@ -10,10 +10,11 @@ from ignition.components.paged_batch import Metadata from ignition.components.parallel import TensorParallelShard -from ignition.components.recurrent_proposal import corrected_logits, recurrent_gate -from ignition.recipes.inkling2.dspark_state import context_slots, mix_features, rotary, stage_paged_tail, stage_window -from ignition.recipes.inkling2.dspark_weights import fold_recurrent_head, mtp_quantization -from ignition.recipes.inkling2.dspark import PreparedContext, PreparedDraft, SpeculativePreparation +from ignition.components.recurrent_proposal import PreparedRecurrentProposalHead, corrected_logits, recurrent_gate +from ignition.recipes.inkling2.dspark_state import (block_inputs, context_slots, mix_features, rotary, rotary_table, + rotate, stage, tail_plan, window_plan) +from ignition.recipes.inkling2.dspark_weights import DraftWeights, fold_recurrent_head, mtp_quantization +from ignition.recipes.inkling2.dspark import DraftScratch, PreparedContext, PreparedDraft, SpeculativePreparation @pytest.mark.parametrize("draft_tokens", [0, 1, 7]) @@ -31,6 +32,7 @@ def test_speculative_preparation_retains_context_at_zero_draft_depth(draft_token assert preparation.forwards.context is PreparedContext assert preparation.forwards.draft is PreparedDraft assert preparation.draft_scratch.hidden.shape == (8, 128) + assert preparation.draft_scratch.anchor.shape == (1, 128) assert not preparation.offset.count_nonzero() assert not preparation.generation.count_nonzero() @@ -87,7 +89,9 @@ def test_noncausal_staging_retains_one_prefix_for_all_eight_positions(start): m = Metadata.pack(torch.tensor([start]), torch.tensor([8]), torch.tensor([1]), torch.tensor([[0] * 16], dtype=torch.int32), torch.zeros(1, block, dtype=torch.int64), torch.tensor([1], dtype=torch.int32), token_capacity=8, sample_count=8) - staged = stage_window(history, values, m, 9).flatten(0, 1) + plan = window_plan(m, window, 9) + staged = stage(history, values, plan).flatten(0, 1) + assert plan.lengths.tolist() == [min(start, window) + block] prefix = min(start, window) expected = torch.arange(start - prefix, start + block).bfloat16() assert torch.equal(staged[:prefix + block, 0, 0], expected) @@ -115,7 +119,9 @@ def test_paged_tail_preserves_committed_prefix_with_one_target_page(): m = Metadata.pack(torch.tensor([127]), torch.tensor([8]), torch.tensor([1]), torch.tensor([[2]], dtype=torch.int32), torch.zeros(1, 8, dtype=torch.int64), torch.tensor([1], dtype=torch.int32), token_capacity=8, sample_count=8) - staged = stage_paged_tail(history, proposal, m, 2).flatten(0, 1) + plan = tail_plan(m, 128, 2) + staged = stage(history, proposal, plan).flatten(0, 1) + assert plan.lengths.tolist() == [135] assert torch.equal(staged[:127], history[2, :127]) assert torch.equal(staged[127:135], proposal) assert not staged[135:].count_nonzero() @@ -147,3 +153,145 @@ def test_mtp_mxfp8_metadata_requires_activation_quantization_and_complete_covera mtp_quantization(changed, names) with pytest.raises(ValueError, match="cover exactly"): mtp_quantization(config, names + ["undeclared.weight"]) + + +def test_one_staging_plan_serves_key_and_value_histories_of_every_layer(): + generator = torch.Generator().manual_seed(9) + window, block, pages = 1024, 8, 9 + m = Metadata.pack(torch.tensor([1027, 5, 300]), torch.tensor([8, 8, 0]), torch.tensor([2, 1, 0]), + torch.tensor([[0] * 16, [1] * 16, [-1] * 16], dtype=torch.int32), torch.zeros(3, block, dtype=torch.int64), + torch.tensor([3], dtype=torch.int32), token_capacity=24, sample_count=24) + plan = window_plan(m, window, pages) + offsets = torch.arange(pages * 128)[None] + prefix = m.starts.clamp(max=window)[:, None] + for _ in range(2): + history = torch.randn(3, window, 2, 4, generator=generator).bfloat16() + values = torch.randn(24, 2, 4, generator=generator).bfloat16() + previous = history[m.state_rows[:, None], (m.starts[:, None] - prefix + offsets).remainder(window)] + current = values[(m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, 23)] + valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) + expected = torch.where(valid[:, :, None, None], + torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view(3 * pages, 128, 2, 4) + assert torch.equal(stage(history, values, plan), expected) + assert plan.lengths.tolist() == [1032, 13, 0] + assert not stage(history, values, plan)[2 * pages:].count_nonzero() + + +def test_shared_rotary_table_matches_per_call_rotation(): + generator = torch.Generator().manual_seed(12) + positions = torch.tensor([0, 7, 1025, -1, 1048575]) + inverse = 1 / (1e6 ** (torch.arange(0, 128, 2).float() / 128)) + table = rotary_table(positions, inverse) + for heads in (2, 12): + values = torch.randn(5, heads, 128, generator=generator).bfloat16() + assert torch.equal(rotate(values, table), rotary(values, positions, inverse)) + + +def test_block_inputs_embed_only_each_request_first_position(): + generator = torch.Generator().manual_seed(15) + anchor = torch.randn(3, 32, generator=generator).bfloat16() + mask = torch.randn(32, generator=generator).bfloat16() + positions = torch.arange(24) % 8 + expected = torch.where((positions == 0)[:, None], anchor.repeat_interleave(8, 0), mask) + assert torch.equal(block_inputs(anchor, mask, 8), expected) + + +def test_draft_scratch_keeps_one_anchor_row_per_block(): + config = SimpleNamespace(text={"hidden_size": 128}, vocabulary=256, + draft=SimpleNamespace(heads=4, kv_heads=4, head_dim=128, text={"intermediate_size": 256})) + scratch = DraftScratch.allocate(config, TensorParallelShard(0, 4), 32, 8, torch.device("cpu")) + assert scratch.anchor.shape == (4, 128) and scratch.hidden.shape == (32, 128) + rows = scratch.rows(16, 16) + assert rows.anchor.shape == (2, 128) and rows.hidden.shape == (16, 128) + assert rows.anchor.data_ptr() == scratch.anchor.data_ptr() + with pytest.raises(ValueError, match="whole blocks"): + scratch.rows(12, 12) + with pytest.raises(ValueError, match="whole blocks"): + DraftScratch.allocate(config, TensorParallelShard(0, 4), 12, 8, torch.device("cpu")) + + +def _recurrent_head(weight, bias, head, head_bias, positions, *, row_scaled): + """Bind FP64 reference projections to the recurrent head's arithmetic.""" + rank = head.shape[1] + operation = object.__new__(PreparedRecurrentProposalHead) + operation.rank, operation.softcap, operation.positions = rank, 30., positions + operation.context_bias = bias + operation.context = lambda source: (source.double() @ weight[:, 2 * rank:].double().T).float() + operation.dynamic = lambda source: (source.double() @ weight[:, :2 * rank].double().T).float() + operation.embed = lambda token_ids: torch.zeros(token_ids.numel(), rank, dtype=torch.bfloat16) + if row_scaled: + operation.output = lambda latent: (latent.double() @ head.double().T).float() + operation.output_bias = head_bias + else: + operation.output = lambda latent: (latent.double() @ head.double().T + head_bias.double()).bfloat16() + operation.output_bias = None + return operation + + +def test_block_context_projection_matches_one_projection_per_position(): + generator = torch.Generator().manual_seed(18) + rows, positions, hidden, rank, vocabulary = 3, 7, 16, 8, 24 + weight = (torch.randn(3 * rank, 2 * rank + 2 * hidden, generator=generator) / 8).bfloat16() + bias = torch.randn(3 * rank, generator=generator) / 8 + head = (torch.randn(vocabulary, rank, generator=generator) / 4).bfloat16() + head_bias = (torch.randn(vocabulary, generator=generator) / 4).bfloat16() + states = torch.randn(rows, positions, hidden, generator=generator).bfloat16() + context = torch.randn(rows, hidden, generator=generator).bfloat16() + operation = _recurrent_head(weight, bias, head, head_bias, positions, row_scaled=False) + fixed = operation.project_context(states, context) + assert fixed.shape == (rows, positions, 3 * rank) and fixed.dtype == torch.float32 + for index in range(positions): + source = torch.cat((states[:, index], context), -1) + assert torch.equal(fixed[:, index], operation.context(source) + bias) + with pytest.raises(ValueError, match="every prepared block position"): + operation.project_context(states[:, :3], context) + + +def test_row_scaled_correction_adds_the_bf16_bias_in_fp32_before_bf16_rounding(): + generator = torch.Generator().manual_seed(21) + rows, hidden, rank, vocabulary = 2, 16, 8, 24 + weight = (torch.randn(3 * rank, 2 * rank + 2 * hidden, generator=generator) / 8).bfloat16() + bias = torch.randn(3 * rank, generator=generator) / 8 + head = (torch.randn(vocabulary, rank, generator=generator) / 4).bfloat16() + head_bias = (torch.randn(vocabulary, generator=generator) / 4).bfloat16() + base = torch.randn(rows, vocabulary, generator=generator).bfloat16() + fixed = torch.randn(rows, 3 * rank, generator=generator) + previous = torch.zeros(rows, dtype=torch.int64) + outputs = [] + for row_scaled in (False, True): + operation = _recurrent_head(weight, bias, head, head_bias, 1, row_scaled=row_scaled) + state, logits = operation.step(operation.begin(base), previous, fixed, base) + outputs.append((state, logits)) + state, latent = recurrent_gate(fixed, torch.zeros(rows, rank, dtype=torch.bfloat16)) + correction = ((latent.double() @ head.double().T).float() + head_bias.float()).bfloat16() + assert torch.equal(outputs[1][0], state) + assert torch.equal(outputs[1][1], corrected_logits(base, correction, 30.)) + # With identical weights the two bias placements differ only by one FP32 addition before BF16 rounding. + assert torch.equal(outputs[0][0], outputs[1][0]) + torch.testing.assert_close(outputs[0][1].float(), outputs[1][1].float(), rtol=2**-7, atol=2**-9) + + +def test_folded_correction_head_uses_the_supplied_allocator(): + generator = torch.Generator().manual_seed(24) + head = torch.randn(64, 32, generator=generator).bfloat16() + weight = torch.randn(32, 8, generator=generator).bfloat16() + bias = torch.randn(32, generator=generator).bfloat16() + calls = [] + + def transient(shape, dtype, device): + calls.append(shape) + return torch.empty(shape, dtype=dtype, device=device) + + folded, offset = fold_recurrent_head(head, weight, bias, allocate=transient) + assert calls == [(64, 8)] + persistent, persistent_offset = fold_recurrent_head(head, weight, bias) + assert torch.equal(folded, persistent) and torch.equal(offset, persistent_offset) + + +@pytest.mark.parametrize("arguments,message", [ + (dict(draft_head_quantization="fp8"), "draft head quantization"), + (dict(markov_head_quantization="nvfp4"), "correction head quantization"), +]) +def test_draft_weights_reject_unsupported_head_formats_before_reading(tmp_path, arguments, message): + with pytest.raises(ValueError, match=message): + DraftWeights.load(tmp_path, None, torch.device("cpu"), TensorParallelShard(0, 4), target_head=None, **arguments) diff --git a/tests/components/test_inkling2_dspark_cuda.py b/tests/components/test_inkling2_dspark_cuda.py index 7da8173e2..4a837f851 100644 --- a/tests/components/test_inkling2_dspark_cuda.py +++ b/tests/components/test_inkling2_dspark_cuda.py @@ -20,7 +20,10 @@ from ignition.components.parallel import TensorParallelShard from ignition.recipes.inkling2.dspark import DraftAttention, DraftScratch, PreparedContext, PreparedDraft from ignition.components.weights import MLPWeights -from ignition.recipes.inkling2.dspark_state import mix_features, rotary +from ignition.components.markov_sampling import fp8_markov_head +from ignition.components.vocabulary import nvfp4_draft_head +from ignition.recipes.inkling2.dspark import inverse_frequencies +from ignition.recipes.inkling2.dspark_state import mix_features, rotary, rotary_table from ignition.recipes.inkling2.dspark_weights import DraftWeights from ignition.components.weights import DirectWeightLoader from ignition.preparation.checkpoint import inspect_indexed_checkpoint @@ -31,9 +34,10 @@ reason="requires an explicitly assigned CUDA device") -@pytest.mark.parametrize("batch,depth", [(1, 7), (4, 7), (4, 1)]) +@pytest.mark.parametrize("batch,depth,row_scaled", [(1, 7, False), (4, 7, False), (4, 1, False), + (1, 7, True), (4, 7, True)]) @torch.no_grad() -def test_sequential_recurrent_draws_record_actual_q_and_reset_state(batch, depth): +def test_sequential_recurrent_draws_record_actual_q_and_reset_state(batch, depth, row_scaled): device = torch.device("cuda:0") generator = torch.Generator().manual_seed(422) hidden, rank, vocabulary, local = 128, 256, 513, 576 @@ -43,6 +47,10 @@ def values(*shape, denominator=64): recurrent = SimpleNamespace(embedding=values(vocabulary, rank), state_weight=weight[:, :2 * rank].contiguous(), context_weight=weight[:, 2 * rank:].contiguous(), bias=values(3 * rank).float(), head=values(local, rank), head_bias=values(local), softcap=30.) + correction_head = recurrent.head + if row_scaled: + recurrent.head = fp8_markov_head(correction_head) + correction_head = recurrent.head.values.float() * recurrent.head.scale[:, None] features, context = values(batch, depth, hidden, denominator=8), values(batch, hidden, denominator=8) backing = values(batch, 8, local, denominator=2) base = backing[:, 1:depth + 1] @@ -65,7 +73,7 @@ def values(*shape, denominator=64): with RecordedPreparationSession(device=device, checkpoint="synthetic-recurrent-head", workload_schema="conditional-proposals", numerical_policy="bf16-checkpoint", autotune_effort="off", compile_workers=2) as session: - head = PreparedRecurrentProposalHead(recurrent, pending, session=session, name="head") + head = PreparedRecurrentProposalHead(recurrent, pending, positions=depth, session=session, name="head") sampler = PreparedVocabularySoftmax(base, q, valid, requests, slots, generations, live, tokens, start=0, runtime=None, coordinator=None, session=session, name="q", pending=pending, seeds=seeds, offsets=offsets, tokens=tokens) @@ -75,9 +83,10 @@ def values(*shape, denominator=64): def forward(): state = head.begin(context) + fixed = head.project_context(features, context) previous = pending for index in range(depth): - state, logits = head.step(state, previous, features[:, index].contiguous(), context, base[:, index]) + state, logits = head.step(state, previous, fixed[:, index], base[:, index]) observed_states[:, index].copy_(state) scores[:, index].copy_(logits) sampler.run_step(index, scores, tokens, pending=pending, seeds=seeds, offsets=offsets, @@ -112,9 +121,20 @@ def forward(): torch.testing.assert_close(raw.double(), concatenated, rtol=2e-7, atol=2e-7) state, latent = recurrent_gate(raw, state) assert torch.equal(observed_states[:active, index], state[:active]) - correction = (latent.double() @ recurrent.head.double().T + recurrent.head_bias.double()).bfloat16() - expected_scores = corrected_logits(base[:, index], correction, 30.) - assert torch.equal(scores[:active, index], expected_scores[:active]) + if row_scaled: + # The E4M3 product is FP32; the BF16 bias joins it before the BF16 rounding. + correction = ((latent.double() @ correction_head.double().T).float() + + recurrent.head_bias.float()).bfloat16() + expected_scores = corrected_logits(base[:, index], correction, 30.) + # FP32 accumulation order can move a correction by one BF16 step. + torch.testing.assert_close(scores[:active, index].float(), expected_scores[:active].float(), + rtol=2**-7, atol=2**-9) + expected_scores = scores[:, index].clone() + else: + correction = (latent.double() @ correction_head.double().T + + recurrent.head_bias.double()).bfloat16() + expected_scores = corrected_logits(base[:, index], correction, 30.) + assert torch.equal(scores[:active, index], expected_scores[:active]) expected_q = expected_scores[:, :vocabulary].float().softmax(-1) torch.testing.assert_close(q[:active, index, :vocabulary], expected_q[:active], rtol=3e-6, atol=1e-9) assert not q[:, index, vocabulary:].count_nonzero() @@ -241,14 +261,20 @@ def values(*shape): operation = DraftAttention(layer, weights, config, scratch, m, pool, shard=TensorParallelShard(0, 1), runtime=None, coordinator=None, session=session, name="attention", workspace=arena) session.tune(show_progress=False) - actual = operation(source, m) + inverse = inverse_frequencies(d, device) + + def attend(): + plan = operation.staging[0](m, *operation.staging[1:]) + return operation(source, m, plan, rotary_table(m.positions, inverse)) + + actual = attend() qkv = (source[:8].double() @ weights.qkv.double().T).bfloat16() q, k, v = qkv.split((heads * dimension, kv_heads * dimension, kv_heads * dimension), -1) def normalize(x, count): x = x.view(8, count, dimension).float() return (x * torch.rsqrt(x.square().mean(-1, keepdim=True) + 1e-6)).bfloat16() - q = rotary(normalize(q, heads), m.positions[:8], operation.inverse) - k = rotary(normalize(k, kv_heads), m.positions[:8], operation.inverse) + q = rotary(normalize(q, heads), m.positions[:8], inverse) + k = rotary(normalize(k, kv_heads), m.positions[:8], inverse) v = v.reshape(8, kv_heads, dimension) if layer == 0: positions = torch.arange(max(0, start - d.window), start, device=device) % d.window @@ -276,7 +302,7 @@ def normalize(x, count): replayed = torch.empty_like(actual) graph = torch.cuda.CUDAGraph() with session.capture(), torch.cuda.graph(graph): - replayed.copy_(operation(source, m)) + replayed.copy_(attend()) allocated = torch.cuda.memory_allocated() graph.replay() torch.cuda.synchronize() @@ -300,8 +326,9 @@ def normalize(x, count): graph.reset() +@pytest.mark.parametrize("draft_head", [None, "nvfp4"]) @torch.no_grad() -def test_five_layer_block_uses_learned_masks_and_replays_without_allocating(): +def test_five_layer_block_uses_learned_masks_and_replays_without_allocating(draft_head): device, hidden, heads, kv_heads, dimension, vocabulary, rank = torch.device("cuda:0"), 128, 12, 2, 128, 576, 256 generator = torch.Generator().manual_seed(341) def values(*shape): @@ -319,7 +346,10 @@ def ones(width): embedding_norm=ones(hidden), head=values(vocabulary, hidden), draft=SimpleNamespace(layers=layers, mask_embedding=values(hidden), norm=ones(hidden), recurrent=SimpleNamespace(embedding=values(vocabulary, rank), context_weight=values(3 * rank, 2 * hidden), state_weight=values(3 * rank, 2 * rank), - bias=values(3 * rank).float(), head=values(vocabulary, rank), head_bias=values(vocabulary), softcap=30.))) + bias=values(3 * rank).float(), head=values(vocabulary, rank), head_bias=values(vocabulary), softcap=30.), + draft_head=None)) + if draft_head is not None: + weights.draft.draft_head = nvfp4_draft_head(weights.head) regions = {f"draft{index}.{kind}": values(3, 1024, kv_heads, dimension) for index in (0, 1, 3, 4) for kind in ("key", "value")} regions["draft.rnn_context"] = values(3, hidden) @@ -347,6 +377,12 @@ def forward(): return features features = forward() assert features["hidden"].shape == (8, hidden) + if draft_head is not None: + # The proposal head reads its NVFP4 copy; four-bit weights keep the logits' direction. + reference = features["hidden"].double() @ weights.head.double().T + similarity = torch.nn.functional.cosine_similarity(output.double(), reference, dim=-1) + assert similarity.min() > .95 + assert not torch.equal(output, reference.bfloat16()) assert torch.equal(features["recurrent_context"], regions["draft.rnn_context"][1:2]) torch.cuda.synchronize() expected = output.clone() diff --git a/tests/components/test_ring_commit_cuda.py b/tests/components/test_ring_commit_cuda.py index aa89b793b..3cb4b4851 100644 --- a/tests/components/test_ring_commit_cuda.py +++ b/tests/components/test_ring_commit_cuda.py @@ -181,3 +181,13 @@ def forward(): worker.close() if graph is not None: graph.reset() + + +@pytest.mark.parametrize("candidate_offset", [-1, 1.5]) +def test_reject_candidates_before_the_staging(candidate_offset): + device = torch.device("cuda", 0) + backing = torch.zeros((3, 4096), dtype=torch.uint8, device=device) + rows, starts = (torch.ones(1, dtype=torch.int64, device=device) for _ in range(2)) + with pytest.raises(ValueError, match="nonnegative record index"): + PreparedRingCommit(backing, (0,), rows, starts, token_capacity=4, window_tokens=8, record_bytes=16, + candidate_offset=candidate_offset) diff --git a/tests/components/test_window_staging_cuda.py b/tests/components/test_window_staging_cuda.py index 4ace3f129..f9f73c692 100644 --- a/tests/components/test_window_staging_cuda.py +++ b/tests/components/test_window_staging_cuda.py @@ -104,3 +104,10 @@ def test_reject_unsupported_staging_views(failure): values = values[0].float(), values[1] with pytest.raises(ValueError): PreparedWindowStaging(*history, *values, *outputs, m, session=None, name='invalid') + + +@pytest.mark.parametrize('lookback', [0, 2049, 1023.]) +def test_reject_a_lookback_outside_the_history(lookback): + history, values, outputs, m, _ = _fixture(8, 128, 1) + with pytest.raises(ValueError, match='looks back'): + PreparedWindowStaging(*history, *values, *outputs, m, lookback=lookback, session=None, name='invalid') diff --git a/tests/recipes/test_inkling2_target.py b/tests/recipes/test_inkling2_target.py index fb0d9fee9..33a2fb0f1 100644 --- a/tests/recipes/test_inkling2_target.py +++ b/tests/recipes/test_inkling2_target.py @@ -162,48 +162,6 @@ def forward(): assert (((partial[:n].double() - exact).abs() - gamma * accumulated).max() <= 0).item() -@pytest.mark.skipif(os.environ.get('FLARE_RUN_CUDA_TESTS') != '1', reason='requires an assigned SM120 GPU') -def test_accepted_window_prefix_commits_replay_without_candidate_mutation(): - from ignition.recipes.inkling2.attention import retain_batch, stage_batch - - device = torch.device('cuda', 0) - m = _metadata([8, 8, 8], [0, 1150, 3499], device) - m.state_rows.copy_(torch.tensor([3, 1, 2], device=device)) - initial = torch.randn(5, 1152, 2, 128, device=device).to(torch.float8_e4m3fn) - rings = initial.clone().view(torch.uint8).view(5, -1) - values = torch.randn(24, 2, 128, dtype=torch.bfloat16, device=device) - staged, lengths = stage_batch(rings, values, m, window=1024, pages=9) - torch.testing.assert_close(rings, initial.view(torch.uint8).view(5, -1), rtol=0, atol=0) - assert lengths.tolist() == [8, 1031, 1031] - for lane, start in enumerate(m.starts.tolist()): - first = max(start - 1023, 0) - expected = torch.cat((initial[m.state_rows[lane], first % 1152:].bfloat16(), initial[m.state_rows[lane], :first % 1152].bfloat16())) - previous = expected[:start - first] - retained = torch.cat((previous, values[lane * 8:(lane + 1) * 8])).to(torch.float8_e4m3fn) - actual = staged.view(3, 1152, 2, 128)[lane] - torch.testing.assert_close(actual[:retained.shape[0]].view(torch.uint8), retained.view(torch.uint8), rtol=0, atol=0) - assert not actual[retained.shape[0]:].view(torch.uint8).any() - counts = torch.zeros(3, dtype=torch.int32, device=device) - retain_batch(rings, values, counts, m) - graph = torch.cuda.CUDAGraph() - with torch.cuda.graph(graph): - retain_batch(rings, values, counts, m) - for accepted in range(9): - rings.copy_(initial.view(torch.uint8).view(5, -1)) - selected = [accepted, (accepted + 1) % 9, 8 - accepted] - counts.copy_(torch.tensor(selected, dtype=torch.int32, device=device)) - before = torch.cuda.memory_stats()['allocation.all.allocated'] - graph.replay() - torch.cuda.synchronize() - assert torch.cuda.memory_stats()['allocation.all.allocated'] == before - expected = initial.clone() - for lane, count in enumerate(selected): - for index in range(count): - position = (int(m.starts[lane]) + index) % 1152 - expected[m.state_rows[lane], position] = values[lane * 8 + index].to(torch.float8_e4m3fn) - torch.testing.assert_close(rings, expected.view(torch.uint8).view(5, -1), rtol=0, atol=0) - - @pytest.mark.skipif(os.environ.get('FLARE_RUN_CUDA_TESTS') != '1', reason='requires an assigned SM120 GPU') @pytest.mark.parametrize('tokens,streams,sources,add', [(1, streams, sources, add) for streams in (1, 3) for sources in range(1, 10) for add in (False, True)] diff --git a/tests/recipes/test_inkling2_window.py b/tests/recipes/test_inkling2_window.py new file mode 100644 index 000000000..ccf1dfac4 --- /dev/null +++ b/tests/recipes/test_inkling2_window.py @@ -0,0 +1,231 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 Flare community contributors +# SPDX-License-Identifier: Apache-2.0 + +"""Inkling 2 verification stages sliding-window rings and commits accepted rows byte for byte. + +The references gather each request's whole ring, select in BF16 and rewrite the ring, +which states the staged pages and committed ring bytes independently of the kernels. +""" + +import os + +import pytest +import torch + +from ignition.components.paged_batch import Metadata +from ignition.recipes.inkling2.attention import accepted_records, staged_lengths + +KV, PAGE, WINDOW, RING, HEADS, DIM, WIDTH = torch.float8_e4m3fn, 128, 1024, 1152, 2, 128, 8 +PAGES = -(-(WINDOW - 1 + WIDTH) // PAGE) +RECORD = HEADS * DIM +cuda = pytest.mark.skipif(os.environ.get("FLARE_RUN_CUDA_TESTS") != "1", reason="requires an assigned SM120 GPU") + + +def reference_stage(rings, values, m): + """FP8 pages `[requests * PAGES, PAGE, heads, dim]` of each request's window and rows, and key counts.""" + rows, capacity = m.starts.numel(), PAGES * PAGE + ring = rings[m.state_rows].view(KV).view(rows, RING, *values.shape[1:]) + first = (m.starts - WINDOW + 1).clamp_min(0) + positions = first[:, None] + torch.arange(capacity, device=values.device) + previous = ring[torch.arange(rows, device=values.device)[:, None], positions.remainder(RING)].to(values.dtype) + index = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) + staged = torch.where((positions < m.starts[:, None])[:, :, None, None], previous, values[index]) + lengths = torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() + staged = torch.where((positions < (m.starts + m.lengths)[:, None])[:, :, None, None], staged, 0) + return staged.clamp(-448, 448).to(KV).view(rows * PAGES, PAGE, *values.shape[1:]), lengths + + +def reference_retain(rings, values, counts, m): + """Rewrite the ring of every request that accepted rows, in place.""" + rows = m.starts.numel() + ring = rings[m.state_rows].view(KV).view(rows, RING, *values.shape[1:]) + first = m.starts + counts - RING + positions = first[:, None] + (torch.arange(RING, device=values.device) - first[:, None]).remainder(RING) + indices = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) + current = values[indices].clamp(-448, 448) + keep = ((positions < m.starts[:, None]) | (positions < 0))[:, :, None, None] + retained = torch.where(keep, ring.to(values.dtype), current).to(KV).view(torch.uint8).view(rows, -1) + selected = counts > 0 + rings[m.state_rows[selected]] = retained[selected] + + +def metadata(lengths, starts, rows, device="cpu"): + m = Metadata.allocate(len(lengths) * WIDTH, len(lengths), 1, device, max_query_tokens=WIDTH) + update(m, lengths, starts, rows) + return m + + +def update(m, lengths, starts, rows): + device, count = m.starts.device, m.starts.numel() + m.set(torch.as_tensor(starts, device=device), torch.as_tensor(lengths, dtype=torch.int32, device=device), + torch.as_tensor(rows, device=device), torch.zeros(count, 1, dtype=torch.int32, device=device), + torch.zeros(count, WIDTH, dtype=torch.int64, device=device), torch.tensor([count], device=device)) + + +def candidates(tokens, generator, device="cpu"): + values = torch.randn(tokens, HEADS, DIM, generator=generator) * 4 + values.view(-1)[:6].copy_(torch.tensor([449., -449., 0., -0., 448., 1.0625])) + return values.bfloat16().to(device) + + +def test_staged_lengths_count_the_preceding_window_and_the_request_rows(): + m = metadata([8, 3, 0, 8, 1], [0, 5, 77, 1023, 5000], [1, 2, 0, 3, 4]) + assert staged_lengths(m, WINDOW).tolist() == [8, 8, 0, 1031, 1024] + values = candidates(m.positions.numel(), torch.Generator().manual_seed(1)) + rings = torch.zeros(5, RING * RECORD, dtype=torch.uint8) + assert torch.equal(staged_lengths(m, WINDOW), reference_stage(rings, values, m)[1]) + + +@pytest.mark.parametrize("starts", [[0, 1, 1144], [1151, 1152, 1153], [1016, 1023, 1024], [3499, 2297, 1048568]]) +def test_accepted_records_commit_to_the_bytes_of_a_rewritten_ring(starts): + generator = torch.Generator().manual_seed(sum(starts)) + lengths, rows = [8, 5, 8], [3, 1, 2] + m = metadata(lengths, starts, rows) + values = (candidates(m.positions.numel(), generator), candidates(m.positions.numel(), generator)) + initial = torch.randn(2, 5, RING, HEADS, DIM, generator=generator).to(KV).view(torch.uint8).view(2, 5, -1) + records = accepted_records(*values, m) + assert records.shape == (2, 3, WIDTH * RECORD) and records.dtype == torch.uint8 + for accepted in range(9): + counts = torch.tensor([min(accepted, 8), min((accepted + 1) % 9, 5), 8 - accepted], dtype=torch.int32) + for region in range(2): + expected = initial[region].clone() + reference_retain(expected, values[region], counts, m) + actual = initial[region].clone().view(5, RING, RECORD) + # The commit stores record `t` of a request at slot `(start + t) mod ring` for `t` below its count. + for request, count in enumerate(counts.tolist()): + for token in range(count): + actual[rows[request], (starts[request] + token) % RING] = records[region, request].view( + WIDTH, RECORD)[token] + assert torch.equal(actual.view(5, -1), expected) + + +def _layer(batch, device, generator): + """Continuation rows with K and V rings at distinct offsets, and the layer's prepared operations.""" + from ignition.components.ring_commit import PreparedRingCommit + from ignition.components.window_staging import PreparedWindowStaging + + offsets = (256, 256 + RING * RECORD + 768) + row_bytes = offsets[1] + RING * RECORD + 256 + backing = torch.randint(0, 256, (batch + 2, row_bytes), dtype=torch.uint8, generator=generator).to(device) + for offset in offsets: + ring = torch.randn(batch + 2, RING, HEADS, DIM, generator=generator).to(KV).view(torch.uint8) + backing[:, offset:offset + RING * RECORD].copy_(ring.view(batch + 2, -1).to(device)) + rings = tuple(backing[:, offset:offset + RING * RECORD] for offset in offsets) + m = metadata([WIDTH] * batch, [0] * batch, list(range(1, batch + 1)), device) + tokens = batch * WIDTH + key, value = (torch.zeros(tokens, HEADS, DIM, dtype=torch.bfloat16, device=device) for _ in range(2)) + staged = tuple(torch.zeros(batch * PAGES, PAGE, HEADS, DIM, dtype=KV, device=device) for _ in range(2)) + + def build(session): + shape = (-1, RING, HEADS, DIM) + staging = PreparedWindowStaging(rings[0].view(KV).view(shape), rings[1].view(KV).view(shape), key, value, + *staged, m, lookback=WINDOW - 1, session=session, name="staging").functional(session=session) + commit = PreparedRingCommit(backing, offsets, m.state_rows, m.starts, token_capacity=WIDTH, + window_tokens=RING, record_bytes=RECORD, candidate_offset=0).functional() + return staging, commit + + return backing, rings, m, key, value, build + + +def _cases(batch, generator): + """Request starts, row counts and state rows at sequence start, window and ring boundaries.""" + boundaries = [0, 1, 7, 1015, 1016, 1023, 1024, 1025, 1143, 1144, 1151, 1152, 1153, 2297, 3499, 1048568] + for shift in range(4): + starts = [boundaries[(shift * 5 + 3 * lane) % len(boundaries)] for lane in range(batch)] + lengths = [int(torch.randint(0 if shift else 1, WIDTH + 1, (1,), generator=generator)) for _ in range(batch)] + if shift == 0: + lengths = [WIDTH] * batch + rows = torch.randperm(batch, generator=generator).add(1).tolist() + yield starts, lengths, rows + + +@cuda +@pytest.mark.parametrize("batch", [1, 8, 32]) +@torch.no_grad() +def test_verification_staging_and_commit_match_the_whole_ring_references(batch): + from ignition.preparation.tuning import RecordedPreparationSession + + device, generator = torch.device("cuda", 0), torch.Generator().manual_seed(600 + batch) + backing, rings, m, key, value, build = _layer(batch, device, generator) + counts = torch.zeros(batch, dtype=torch.int32, device=device) + observed = tuple(torch.zeros(batch * PAGES, PAGE, HEADS, DIM, dtype=KV, device=device) for _ in range(2)) + with RecordedPreparationSession(device=device, checkpoint="inkling2-window-verification", + workload_schema="window-verification", numerical_policy="exact", compile_workers=1) as session: + staging, commit = build(session) + session.tune(show_progress=False) + + def forward(): + staged = staging(key, value, metadata=m) + for output, pages in zip(observed, staged): + output.copy_(pages) + commit(counts, staging=accepted_records(key, value, m), rows=m.state_rows, starts=m.starts) + + forward() + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + forward() + for starts, lengths, rows in _cases(batch, generator): + for accepted in ("none", "all", "depth", "random"): + update(m, lengths, starts, rows) + key.copy_(candidates(batch * WIDTH, generator, device)) + value.copy_(candidates(batch * WIDTH, generator, device)) + limit = torch.tensor(lengths, dtype=torch.int32) + selected = {"none": torch.zeros_like(limit), "all": limit, + "depth": torch.minimum(limit, torch.arange(batch, dtype=torch.int32) % (WIDTH + 1)), + "random": (torch.rand(batch, generator=generator) * (limit + 1)).int()}[accepted] + counts.copy_(selected.to(device)) + before, expected = backing.clone(), backing.clone() + references = [] + for offset, values in zip((256, 256 + RING * RECORD + 768), (key, value)): + region = expected[:, offset:offset + RING * RECORD] + references.append(reference_stage(region, values, m)) + reference_retain(region, values, counts, m) + allocated = torch.cuda.memory_stats()["allocation.all.allocated"] + graph.replay() + torch.cuda.synchronize() + assert torch.cuda.memory_stats()["allocation.all.allocated"] == allocated + live = (m.lengths > 0).repeat_interleave(PAGES) + for output, (pages, counted) in zip(observed, references): + assert torch.equal(output.view(torch.uint8)[live], pages.view(torch.uint8)[live]) + # A request without rows stages no keys; attention reads none of its pages. + assert not output.view(torch.uint8)[~live].any() + assert torch.equal(staged_lengths(m, WINDOW), counted) + assert torch.equal(backing, expected) + if accepted == "none": + assert torch.equal(backing, before) + graph.reset() + + +@cuda +@torch.no_grad() +def test_successive_rounds_keep_rings_identical_across_ring_wraps(): + from ignition.preparation.tuning import RecordedPreparationSession + + batch = 4 + device, generator = torch.device("cuda", 0), torch.Generator().manual_seed(77) + backing, rings, m, key, value, build = _layer(batch, device, generator) + counts = torch.zeros(batch, dtype=torch.int32, device=device) + with RecordedPreparationSession(device=device, checkpoint="inkling2-window-rounds", + workload_schema="window-verification", numerical_policy="exact", compile_workers=1) as session: + staging, commit = build(session) + session.tune(show_progress=False) + expected = backing.clone() + starts, rows = [0, 1000, 1140, 2300], [4, 2, 1, 3] + for _ in range(500): + lengths = torch.randint(1, WIDTH + 1, (batch,), generator=generator).tolist() + update(m, lengths, starts, rows) + key.copy_(candidates(batch * WIDTH, generator, device)) + value.copy_(candidates(batch * WIDTH, generator, device)) + accepted = [int(torch.randint(0, length + 1, (1,), generator=generator)) for length in lengths] + counts.copy_(torch.tensor(accepted, dtype=torch.int32)) + staged = staging(key, value, metadata=m) + for offset, values, pages in zip((256, 256 + RING * RECORD + 768), (key, value), staged): + region = expected[:, offset:offset + RING * RECORD] + assert torch.equal(pages.view(torch.uint8), reference_stage(region, values, m)[0].view(torch.uint8)) + reference_retain(region, values, counts, m) + commit(counts, staging=accepted_records(key, value, m), rows=m.state_rows, starts=m.starts) + assert torch.equal(backing, expected) + starts = [start + count for start, count in zip(starts, accepted)] + # Every request has crossed its window; three have wrapped their rings. + assert starts[0] > WINDOW and min(starts[1:]) > RING