From ebf16a680fffd1e3c353d68768188e1b1ba4890c Mon Sep 17 00:00:00 2001 From: Jason Cook Date: Sun, 15 Nov 2026 06:04:22 -0500 Subject: [PATCH] Cut the fixed cost of Inkling 2 drafting rounds and add proposal-head formats On DGX Spark TP4 a drafting round of Inkling 2 Small cost about 39 ms plus 4.5 ms per verified draft against a 28 ms decode step, so 11 ms of each round bought no tokens and the balanced policy rarely chose to draft. Verification staged each sliding-window layer's history with PyTorch gathers over whole rings and committed accepted rows by rewriting whole rings, the recurrent head projected its draw-independent context once per draft position, and proposal heads read BF16 only. These costs do not depend on the draws or on the number of drafts, so they can leave the per-round path without changing what verification computes. Verification attention stages each request's preceding window and its candidate rows with one PreparedWindowStaging kernel per layer and commits accepted rows with one PreparedRingCommit kernel. PreparedWindowStaging gains lookback, the count of preceding positions staged from a ring that retains more than attention reads, and PreparedRingCommit gains candidate_offset, the record index at which staged candidates start; both default to their earlier meaning. The recurrent proposal head projects the block's states and target context once per block, proposals read one embedding row per request, and staging plans are shared between layers. --draft-head-quantization nvfp4 or mxfp8 and --markov-head-quantization fp8 prepare proposal-only copies computed from checkpoint values at load; target verification keeps the BF16 head, so they change which tokens are proposed, never the output distribution. Both flags default to omitted. A drafting round at one request costs about 34.7 ms plus 4.7 ms per verified draft: the fixed cost over a decode step falls from about 11 ms to about 6 ms and the per-draft cost is unchanged. The preparation graph pool falls from 2,840 MiB to 2,092 MiB per rank. Evidence: one exported core per tree served by flare run at TP4 on GB10, one request, ordinary prompts, a fresh launch per verification row budget. Rounds verifying 2 to 8 target rows took 39.3, 45.0, 49.9, 54.4, 58.9, 62.1 and 67.3 ms against 43.7, 48.5, 53.2, 57.4, 61.6, 67.1 and 70.5 ms. Drafting rounds took 215 to 248 ms at 16 requests (237 to 272 ms) and 279 ms at 32 (342 to 461 ms). The measured core used --draft-head-quantization nvfp4; acceptance at draft depths 1 to 4 was 81, 63, 48 and 35% on ordinary prompts against 79, 61, 48 and 36% with the BF16 head, so the NVFP4 head neither raised nor lowered acceptance. The measured core also contained the prefill changes of the preceding commit. On a GB10 with the checkpoint, test_inkling2_target.py (87), test_inkling2_window.py (9) and test_inkling2_dspark_cuda.py (15) pass, covering verification attention against the reference and staged and committed rings against PyTorch. Limitations: verification staging requires WindowStageKernel's lookback from FlashInfer (local-inference-lab/flashinfer#112), which the pinned revision lacks. --markov-head-quantization fp8 and the MXFP8 draft head were not measured in serving. Aggregate decode throughput is within a few percent of the unmodified tree. lookback and candidate_offset extend two existing component contracts with defaulted parameters; no graph kind is added. Design-Signed-Off-By: flare-02; design-review=ed22 --- docs/cli-design.md | 4 +- docs/model-preparation.md | 12 + ignition/components/parallel_draft.py | 6 +- ignition/components/recurrent_proposal.py | 51 +++- ignition/components/ring_commit.py | 21 +- ignition/components/window_staging.py | 24 +- ignition/recipes/inkling2/attention.py | 74 +++--- ignition/recipes/inkling2/bootstrap.py | 18 +- ignition/recipes/inkling2/dspark.py | 82 ++++--- ignition/recipes/inkling2/dspark_state.py | 86 +++++-- ignition/recipes/inkling2/dspark_weights.py | 50 +++- ignition/recipes/inkling2/recipe.py | 9 +- ignition/recipes/inkling2/weights.py | 6 +- tests/cli/test_inkling2.py | 60 +++++ tests/components/test_inkling2_dspark.py | 160 +++++++++++- tests/components/test_inkling2_dspark_cuda.py | 64 +++-- tests/components/test_ring_commit_cuda.py | 10 + tests/components/test_window_staging_cuda.py | 7 + tests/recipes/test_inkling2_target.py | 42 ---- tests/recipes/test_inkling2_window.py | 231 ++++++++++++++++++ 20 files changed, 820 insertions(+), 197 deletions(-) create mode 100644 tests/recipes/test_inkling2_window.py diff --git a/docs/cli-design.md b/docs/cli-design.md index cde639a96..23b3e74cf 100644 --- a/docs/cli-design.md +++ b/docs/cli-design.md @@ -448,8 +448,8 @@ explicit flags take precedence. | `--no-mtp` | Disable multi-token speculation: load no draft model, keep no verification rows, and decode one token per step. | Speculation on | | `--mtp-only` | Omit MTP-disabled prefill, decode, maintenance, and mixed graphs during preparation. Retain width-one speculative rounds and vision graphs. Requires positive draft depth; rejects `--no-mtp`. The core cannot load with native `--mtp-draft-tokens 0`. | Capture both execution modes | | `--draft-body-quantization {mxfp8}` | Prepare eligible DFlash2 body context, attention latent/query/output, and MLP projections as MXFP8, independently of target conversion and proposal-head precision. Kimi K3.1 Pro requires `--speculative-method dflash2` and positive learned draft depth; DSpark and other recipes reject it. Normalization, convolution, candidate-selector, and absorbed key/value weights retain their precision. Omission preserves checkpoint precision or the existing `online_mxfp8_scope all` behavior. | No additional override | -| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. This does not quantize the draft body. | Checkpoint precision | -| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Requires positive draft depth. | Checkpoint precision | +| `--draft-head-quantization {nvfp4,mxfp8}` | Prepare a separate proposal vocabulary head while retaining target-head precision. NVFP4 supports DeepSeek V4/V4.1; MXFP8 supports DeepSeek V4.1. Qwen3.8 dense supports both when the source checkpoint head is BF16, including `qwen3.8-27b-qad`; the packed NVIDIA head cannot be converted through this option. Flash Next supports both with `--speculative-method dflash2` and a BF16 target vocabulary head. Kimi K3.1 Pro supports both with `--speculative-method dflash2` only. Inkling 2 Small NVFP4 supports both. This does not quantize the draft body. | Checkpoint precision | +| `--markov-head-quantization {nvfp4,fp8}` | Prepare the draft Markov transition projection as NVFP4 or row-scaled E4M3 FP8. DeepSeek V4 supports NVFP4; DeepSeek V4.1 supports both. Inkling 2 Small NVFP4 supports FP8 for its recurrent correction head. Requires positive draft depth. | Checkpoint precision | | `--vocabulary-parallel-sampling` | Prepare vocabulary-sharded DSpark proposals and block rejection sampling. DeepSeek V4.1 on four GPUs only; three-GPU preparation rejects it. Requires positive draft depth and recipe sampling defaults without top-k, top-p, or min-p filtering, and rejects `--markov-head-quantization nvfp4`. | Full-vocabulary sampling | | `--mtp-draft-tokens N` | Most draft tokens verified in one round; see `MODEL --help` for its limit. Verification capacities cover counts from 0 through `N` within the speculative token capacity. Each complete speculative program records its actual proposal depth independently of target width and aggregate rows. Prefix selection uses measured complete-program cost and completed acceptance or confidence. Zero retains draft state with width-one execution; `--no-mtp` excludes draft weights and state. | Recipe declaration | | `--speculative-token-capacity N` | Maximum physical target-token rows in wider speculative graphs; width one covers every request bucket. Applies to every speculative recipe. | Recipe declaration: 32 for Muse Spark and 128 for the other recipes that declare one. Gemma 4 declares none and derives it from concurrency and draft depth (6 at its defaults) | diff --git a/docs/model-preparation.md b/docs/model-preparation.md index 1771ce7a2..1457b42d1 100644 --- a/docs/model-preparation.md +++ b/docs/model-preparation.md @@ -1325,6 +1325,18 @@ reasoning, tool calls, and structured responses. Sampling defaults are temperature 1.0 and top-p 1.0; the prompt encoder defaults to reasoning effort 0.9 when the request does not specify one. +Each drafting round projects eight block rows through the vocabulary head and +then, for each of up to seven draft positions, corrects those logits with the +recurrent head: its output map folded through the vocabulary head, one +vocabulary row by 256 values. Both read BF16 by default, 463 MB and 25.7 MB +per rank. `--draft-head-quantization nvfp4` or `mxfp8` prepares a quantized +copy of the target's vocabulary shard that only proposals read, and +`--markov-head-quantization fp8` stores the folded correction head as E4M3 +values with one FP32 scale per vocabulary row. Both are computed at load from +checkpoint values; target verification keeps the BF16 head, so a quantized +proposal head changes which tokens are proposed and how often they are +accepted, never the output distribution. The default omits both flags. + Export with default tuning and Torch compilation: ```sh diff --git a/ignition/components/parallel_draft.py b/ignition/components/parallel_draft.py index fcf439d39..34652bbdc 100644 --- a/ignition/components/parallel_draft.py +++ b/ignition/components/parallel_draft.py @@ -442,11 +442,11 @@ def propose(self, advance_rng=True, allowed=None, *, probabilities): hidden = features["hidden"].view(b, block, -1) context = features["recurrent_context"] state = self.recurrent.begin(context) + # The draft states and target context do not depend on the draws: project them once. + fixed = self.recurrent.project_context(hidden[:, o.logit_offset:], context) scores = torch.empty_like(draft_logits)[:, o.logit_offset:o.logit_offset + w - 1] for index in range(w - 1): - state, logits = self.recurrent.step(state, markov_token, - hidden[:, index + o.logit_offset].contiguous(), context, - base[:, index]) + state, logits = self.recurrent.step(state, markov_token, fixed[:, index], base[:, index]) scores[:, index].copy_(logits) self.full_proposal.run_step(index, scores, drafts, pending=markov_token, seeds=seed, offsets=base_offset, probabilities=probabilities, valid=valid, diff --git a/ignition/components/recurrent_proposal.py b/ignition/components/recurrent_proposal.py index d76b02730..8c21b4465 100644 --- a/ignition/components/recurrent_proposal.py +++ b/ignition/components/recurrent_proposal.py @@ -7,6 +7,7 @@ from .embedding import PreparedEmbedding from .linear import UnquantizedProjection +from .markov_sampling import PreparedRowScaledFP8MarkovProjection, RowScaledFP8MarkovWeight def recurrent_gate(raw, state): @@ -27,33 +28,61 @@ def corrected_logits(base, correction, softcap): class PreparedRecurrentProposalHead: - """Keep recurrence state inside one block while retaining its actual sampled q.""" + """Keep recurrence state inside one block while retaining its actual sampled q. - def __init__(self, weights, tokens, *, session, name): + The recurrence input has a part that depends on the previous draw (state and predecessor + embedding) and a part that does not (draft hidden state and target context). `project_context` + computes the second part for every position of the block in one projection; `step` adds the + first. `weights.head` maps the recurrent output to vocabulary corrections as BF16 rows, or as + row-scaled E4M3 rows whose FP32 product takes the BF16 bias before rounding to BF16. + """ + + def __init__(self, weights, tokens, *, positions, session, name): rows, device = tokens.numel(), tokens.device rank = weights.embedding.shape[1] hidden = weights.context_weight.shape[1] // 2 - self.rank, self.softcap = rank, weights.softcap + if type(positions) is not int or positions < 1: + raise ValueError("a recurrent proposal head corrects at least one position per row") + self.rank, self.softcap, self.positions = rank, weights.softcap, positions self.context_bias = weights.bias embedding = torch.empty((rows, rank), dtype=torch.bfloat16, device=device) dynamic = torch.empty((rows, 2 * rank), dtype=torch.bfloat16, device=device) - context = torch.empty((rows, 2 * hidden), dtype=torch.bfloat16, device=device) + context = torch.empty((rows * positions, 2 * hidden), dtype=torch.bfloat16, device=device) + fixed = torch.empty((rows * positions, 3 * rank), dtype=torch.float32, device=device) raw = torch.empty((rows, 3 * rank), dtype=torch.float32, device=device) - correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device) self.embed = PreparedEmbedding(weights.embedding, tokens, embedding, session=session, name=name + ".embedding").functional(session=session, dynamic_inputs=True) - self.context = UnquantizedProjection(context, weights.context_weight, raw, + self.context = UnquantizedProjection(context, weights.context_weight, fixed, session=session, name=name + ".context").functional(session=session) self.dynamic = UnquantizedProjection(dynamic, weights.state_weight, raw, session=session, name=name + ".dynamic").functional(session=session) - self.output = UnquantizedProjection(embedding, weights.head, correction, - bias=weights.head_bias, session=session, name=name + ".output").functional(session=session) + self.output_bias = None + if isinstance(weights.head, RowScaledFP8MarkovWeight): + correction = torch.empty((rows, weights.head.out_features), dtype=torch.float32, device=device) + self.output = PreparedRowScaledFP8MarkovProjection(embedding, weights.head, correction, + session=session, name=name + ".output").functional(session=session) + self.output_bias = weights.head_bias + else: + correction = torch.empty((rows, weights.head.shape[0]), dtype=torch.bfloat16, device=device) + self.output = UnquantizedProjection(embedding, weights.head, correction, + bias=weights.head_bias, session=session, name=name + ".output").functional(session=session) def begin(self, hidden): return torch.zeros((hidden.shape[0], self.rank), dtype=torch.bfloat16, device=hidden.device) - def step(self, state, previous, hidden, context, base): - fixed = self.context(torch.cat((hidden, context), -1)) + self.context_bias + def project_context(self, hidden, context): + """Return FP32 [rows, positions, 3 * rank] recurrence inputs from [rows, positions, hidden] draft states.""" + rows, positions, width = hidden.shape + if positions != self.positions: + raise ValueError("recurrent context projection covers every prepared block position") + source = torch.cat((hidden, context[:, None].expand(rows, positions, width)), -1) + fixed = self.context(source.view(rows * positions, 2 * width)) + self.context_bias + return fixed.view(rows, positions, 3 * self.rank) + + def step(self, state, previous, fixed, base): dynamic = self.dynamic(torch.cat((state, self.embed(token_ids=previous)), -1)) state, latent = recurrent_gate(dynamic + fixed, state) - return state, corrected_logits(base, self.output(latent), self.softcap) + correction = self.output(latent) + if self.output_bias is not None: + correction = (correction + self.output_bias.float()).bfloat16() + return state, corrected_logits(base, correction, self.softcap) diff --git a/ignition/components/ring_commit.py b/ignition/components/ring_commit.py index ec5b69486..1eb37ca41 100644 --- a/ignition/components/ring_commit.py +++ b/ignition/components/ring_commit.py @@ -15,7 +15,7 @@ def _commit(Staged, Backing, Offsets, Rows, Starts, Lengths, ROW_STRIDE: tl.constexpr, STAGED_STRIDE: tl.constexpr, REGION_STRIDE: tl.constexpr, ROWS_STRIDE: tl.constexpr, STARTS_STRIDE: tl.constexpr, LENGTHS_STRIDE: tl.constexpr, - WINDOW: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr): + WINDOW: tl.constexpr, CANDIDATES: tl.constexpr, RECORD: tl.constexpr, BLOCK: tl.constexpr): item, request, region = tl.program_id(0), tl.program_id(1), tl.program_id(2) length = tl.load(Lengths + request * LENGTHS_STRIDE) start, row = tl.load(Starts + request * STARTS_STRIDE), tl.load(Rows + request * ROWS_STRIDE) @@ -23,7 +23,7 @@ def _commit(Staged, Backing, Offsets, Rows, Starts, Lengths, ROW_STRIDE: tl.cons byte = tl.arange(0, BLOCK) live = (token < length) & (row > 0) & (byte < RECORD) value = tl.load(Staged + region * REGION_STRIDE + request * STAGED_STRIDE - + (WINDOW + token) * RECORD + byte, live, other=0) + + (CANDIDATES + token) * RECORD + byte, live, other=0) offset = tl.load(Offsets + region) tl.store(Backing + row * ROW_STRIDE + offset + ((start + token) % WINDOW) * RECORD + byte, value, live) @@ -35,9 +35,15 @@ class PreparedRingCommit: Admission validates distinct positive destination rows and bounds accepted counts by token_capacity. Row zero and unaccepted ring records are preserved. The complete backing tensor exposes mutations to forward compilation. + + Each request's staging holds its candidate records from record index + `candidate_offset`, which defaults to `window_tokens` for staging that + places the candidates after one window of history. Position `p` commits to + ring slot `p mod window_tokens`. """ - def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None): + def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tokens, record_bytes, staging=None, + candidate_offset=None): if (backing.device.type != "cuda" or backing.dtype != torch.uint8 or backing.ndim != 2 or backing.stride(1) != 1 or backing.stride(0) < backing.shape[1] or rows.ndim != 1 or rows.numel() == 0 or starts.shape != rows.shape @@ -46,6 +52,9 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok raise ValueError("ring commits require byte rows and matching int64 row/position vectors") if any(type(value) is not int or value <= 0 for value in (token_capacity, window_tokens, record_bytes)): raise ValueError("ring and staging capacities must be positive integers") + candidate_offset = window_tokens if candidate_offset is None else candidate_offset + if type(candidate_offset) is not int or candidate_offset < 0: + raise ValueError("staged candidates start at a nonnegative record index") offsets = tuple(offsets) size = window_tokens * record_bytes if (not offsets or any(type(offset) is not int or offset < 0 or offset + size > backing.shape[1] @@ -57,12 +66,12 @@ def __init__(self, backing, offsets, rows, starts, *, token_capacity, window_tok record_numeric_tensor(self.offsets) from ignition.preparation.scratch import bind_scratch - shape = (len(offsets), rows.numel(), (window_tokens + token_capacity) * record_bytes) + shape = (len(offsets), rows.numel(), (candidate_offset + token_capacity) * record_bytes) self.staging_shape, self.rows_shape = shape, tuple(rows.shape) self.staging = bind_scratch({"staging": (shape, torch.uint8)}, device=backing.device, workspace=None if staging is None else {"staging": staging})["staging"] self.staging.zero_() - self.window_tokens, self.record_bytes = window_tokens, record_bytes + self.window_tokens, self.record_bytes, self.candidate_offset = window_tokens, record_bytes, candidate_offset self.grid = (min(token_capacity, window_tokens), rows.numel(), len(offsets)) def functional(self): @@ -76,4 +85,4 @@ def __call__(self, counts, *, staging, rows, starts): staging, self.backing, self.offsets, rows, starts, counts, self.backing.stride(0), staging.stride(1), staging.stride(0), rows.stride(0), starts.stride(0), counts.stride(0), - self.window_tokens, self.record_bytes, triton.next_power_of_2(self.record_bytes)) + self.window_tokens, self.candidate_offset, self.record_bytes, triton.next_power_of_2(self.record_bytes)) diff --git a/ignition/components/window_staging.py b/ignition/components/window_staging.py index 652240018..7ae6a3504 100644 --- a/ignition/components/window_staging.py +++ b/ignition/components/window_staging.py @@ -22,6 +22,7 @@ class Query: window: int history_dtype: str first_position: int + lookback: int CONTRACT = native_contract(component_id="flare.window_stage", query_type=Query, tactics=lambda q: (0,)) @@ -50,10 +51,12 @@ def compile_staging(payload, ordinal): tuple(cute.sym_int(64) for _ in range(rank - 1)) + (1,), assumed_align=4) for rank, dtype in _operands(q.history_dtype)) with torch.cuda.device(ordinal): + # The kernel takes a lookback only when it differs from the history length. + lookback = {} if q.lookback == q.window else {"lookback": q.lookback} return compile_kernel(WindowStageKernel(heads=q.heads, dim=q.dim, capacity=q.capacity, - window=q.window, first_position=q.first_position), *operands, cutlass.Int32(1), current_cuda_stream(), - compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2, - (q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position))) + window=q.window, first_position=q.first_position, **lookback), *operands, cutlass.Int32(1), + current_cuda_stream(), compile_spec=KernelCompileSpec.from_key("attention.window_stage", 2, + (q.heads, q.dim, q.capacity, q.window, q.history_dtype, q.first_position, q.lookback))) @dataclass(frozen=True) @@ -113,10 +116,16 @@ def _overlap(left, right): class PreparedWindowStaging: - """Gather paired history and candidates with BF16 selection and FP8 saturation.""" + """Gather paired history and candidates with BF16 selection and FP8 saturation. + + The history's second dimension is its ring: position `p` occupies slot `p` + modulo that length. Each request's output lists positions in order from + `max(start - lookback, first_position)`; `lookback` defaults to the ring + length and may be shorter when the ring retains more than attention reads. + """ def __init__(self, history_key, history_value, key, value, output_key, output_value, metadata, *, - counts=None, first_position=0, session, name): + counts=None, first_position=0, lookback=None, session, name): from b12x.attention.paged_kv_write._preparation import _check_layout m = metadata @@ -133,6 +142,9 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va or output_key.shape[0] * output_key.shape[1] % rows or output_key.numel() == 0 or type(first_position) is not int): raise ValueError("window staging requires matching positive BF16 candidates, BF16/FP8 history and FP8 output geometry") + lookback = history_key.shape[1] if lookback is None else lookback + if type(lookback) is not int or not 0 < lookback <= history_key.shape[1]: + raise ValueError("window staging looks back over a positive count of at most the history's positions") capacity = output_key.shape[0] * output_key.shape[1] // rows self.page_shapes = tuple(output_key.shape), tuple(output_value.shape) outputs = tuple(t.view(rows, capacity, *key.shape[1:]) for t in (output_key, output_value)) @@ -152,7 +164,7 @@ def __init__(self, history_key, history_value, key, value, output_key, output_va if _overlap(*outputs): raise ValueError("window staging outputs must not overlap") q = Query(key.shape[1], key.shape[2], capacity, history_key.shape[1], - str(history_key.dtype).removeprefix("torch."), first_position) + str(history_key.dtype).removeprefix("torch."), first_position, lookback) payload = CONTRACT.encode_query(q) arguments = self.arguments device_index = key.device.index diff --git a/ignition/recipes/inkling2/attention.py b/ignition/recipes/inkling2/attention.py index e0c885333..819ea633b 100644 --- a/ignition/recipes/inkling2/attention.py +++ b/ignition/recipes/inkling2/attention.py @@ -23,8 +23,10 @@ from ignition.components.head_gate import PreparedHeadGate from ignition.components.linear import UnquantizedProjection from ignition.components.paged_attention import PreparedPagedAttention, PreparedPagedKVWriter +from ignition.components.ring_commit import PreparedRingCommit from ignition.components.rms_norm import PreparedRMSNorm from ignition.components.state import gather_rows, scatter_rows +from ignition.components.window_staging import PreparedWindowStaging from ignition.preparation.persistent import persistent_empty from .arithmetic import l2_clip, log_scaling @@ -125,7 +127,9 @@ class LayerCache: Local layers also expose each continuation row's K and V rings as bytes (`key_rings`, `value_rings`, `[rows, ring bytes]`), which prefill reads and - rewrites one request row at a time. + rewrites one request row at a time, and the continuation rows with the byte + offsets of the K and V rings (`rows`, `ring_offsets`), where verification + commits accepted positions. """ key: torch.Tensor @@ -135,13 +139,17 @@ class LayerCache: ring_row_pages: int = 0 key_rings: torch.Tensor | None = None value_rings: torch.Tensor | None = None + rows: torch.Tensor | None = None + ring_offsets: tuple[int, int] | None = None @classmethod def of(cls, pool, layer): if pool.text.local(layer): return cls(pool.ring_pages(layer, "key"), pool.ring_pages(layer, "value"), pool.history(layer, "key_convolution"), pool.history(layer, "value_convolution"), - pool.ring_row_pages, pool.ring_rows(layer, "key"), pool.ring_rows(layer, "value")) + pool.ring_row_pages, pool.ring_rows(layer, "key"), pool.ring_rows(layer, "value"), + pool.reserved_backing, + tuple(pool.layout.region(f"layer{layer}.{kind}").offset for kind in ("key", "value"))) return cls(pool.cache(layer, "key"), pool.cache(layer, "value"), pool.history(layer, "key_convolution"), pool.history(layer, "value_convolution")) @@ -230,6 +238,16 @@ def example(role, shape, dtype=torch.bfloat16): cache_value = example("inkling.attention.staged_value", (count, PAGE_TOKENS, self.kv_heads, self.dim), KV_DTYPE) pages = torch.arange(count, dtype=torch.int32, device=device).view(requests, self.staged_pages) self.staged_table = pages + if mode == "verify": + # One kernel stages each request's preceding window and its rows from the K and V rings; + # one kernel commits the accepted rows' records to both rings. + shape = (-1, self.ring, self.kv_heads, self.dim) + self.staging = PreparedWindowStaging(cache.key_rings.view(KV_DTYPE).view(shape), + cache.value_rings.view(KV_DTYPE).view(shape), key, value, cache_key, cache_value, metadata, + lookback=self.window - 1, session=session, name=f"{name}.staging").functional(session=session) + self.ring_commit = PreparedRingCommit(cache.rows, cache.ring_offsets, metadata.state_rows, + metadata.starts, token_capacity=metadata.max_query_tokens, window_tokens=self.ring, + record_bytes=self.kv_heads * self.dim * KV_DTYPE.itemsize, candidate_offset=0).functional() else: cache_key, cache_value = cache.key, cache.value pages = (example("inkling.attention.ring_table", (requests, self.ring // PAGE_TOKENS), torch.int32) @@ -336,10 +354,8 @@ def _decode(self, v, m): def _verify(self, v, m): if self.local: - cache_key, lengths = stage_batch(self.cache.key_rings, v["key"], m, - window=self.window, pages=self.staged_pages) - cache_value, _ = stage_batch(self.cache.value_rings, v["value"], m, - window=self.window, pages=self.staged_pages) + cache_key, cache_value = self.staging(v["key"], v["value"], metadata=m) + lengths = staged_lengths(m, self.window) pages = self.staged_table kwargs = dict(k_cache=cache_key, v_cache=cache_value) else: @@ -355,38 +371,26 @@ def commit(self, values, counts, metadata): self.key_convolution.commit(values["key_previous"], values["key_raw"], metadata, counts=counts) self.value_convolution.commit(values["value_previous"], values["value_raw"], metadata, counts=counts) if self.local: - retain_batch(self.cache.key_rings, values["key"], counts, metadata) - retain_batch(self.cache.value_rings, values["value"], counts, metadata) + records = accepted_records(values["key"], values["value"], metadata) + self.ring_commit(counts, staging=records, rows=metadata.state_rows, starts=metadata.starts) -def stage_batch(rings, values, m, *, window, pages): - rows, capacity = m.starts.numel(), pages * PAGE_TOKENS - ring = torch.empty((rows, rings.shape[1]), dtype=torch.uint8, device=rings.device) - gather_rows(rings, m.state_rows, ring) - ring = ring.view(KV_DTYPE).view(rows, -1, *values.shape[1:]) +def staged_lengths(m, window): + """Each request's staged key count: its preceding `window - 1` positions, bounded by its start, and its rows.""" first = (m.starts - window + 1).clamp_min(0) - positions = first[:, None] + torch.arange(capacity, device=values.device) - previous = ring[torch.arange(rows, device=values.device)[:, None], positions.remainder(ring.shape[1])].to(values.dtype) - index = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) - staged = torch.where((positions < m.starts[:, None])[:, :, None, None], previous, values[index]) - lengths = torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() - staged = torch.where((positions < (m.starts + m.lengths)[:, None])[:, :, None, None], staged, 0) - return staged.clamp(-448, 448).to(KV_DTYPE).view(rows * pages, PAGE_TOKENS, *values.shape[1:]), lengths - - -def retain_batch(rings, values, counts, m): - rows = m.starts.numel() - previous = torch.empty((rows, rings.shape[1]), dtype=torch.uint8, device=rings.device) - gather_rows(rings, m.state_rows, previous) - ring = previous.view(KV_DTYPE).view(rows, -1, *values.shape[1:]) - capacity = ring.shape[1] - first = m.starts + counts - capacity - positions = first[:, None] + (torch.arange(capacity, device=values.device) - first[:, None]).remainder(capacity) - indices = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) - current = values[indices].clamp(-448, 448) - keep = ((positions < m.starts[:, None]) | (positions < 0))[:, :, None, None] - retained = torch.where(keep, ring.to(values.dtype), current).to(KV_DTYPE) - scatter_rows(retained.view(torch.uint8).view(rows, -1), m.state_rows, rings, counts > 0) + return torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() + + +def accepted_records(key, value, m): + """FP8 K and V records of each request's rows, as bytes `[2, requests, width * record bytes]`. + + Record `t` of a request is its row `t`; columns past a request's rows hold other + requests' records, which a commit bounded by the request's accepted count never reads. + """ + width = m.max_query_tokens + index = (m.query_start_loc[:-1, None] + torch.arange(width, device=key.device)).clamp(0, key.shape[0] - 1) + records = torch.stack((key[index], value[index])).clamp(-448, 448).to(KV_DTYPE) + return records.view(torch.uint8).view(2, index.shape[0], -1) def _page_slots(m): diff --git a/ignition/recipes/inkling2/bootstrap.py b/ignition/recipes/inkling2/bootstrap.py index 8210ca87e..cb7f6ca7d 100644 --- a/ignition/recipes/inkling2/bootstrap.py +++ b/ignition/recipes/inkling2/bootstrap.py @@ -75,6 +75,9 @@ class PreparationConfig: speculative_token_capacity: int | None = 64 speculative_graph_budget: int = 64 mtp_only: bool = False + # Proposal-only quantized copies derived from checkpoint values at load; None keeps BF16. + draft_head_quantization: str | None = None + markov_head_quantization: str | None = None benchmark_selection: object | None = None def __post_init__(self): @@ -107,6 +110,13 @@ def __post_init__(self): raise ValueError("--mtp-only requires enabled learned speculation with positive draft depth") if self.drafting and (not self.speculative_widths or not set(self.speculative_widths) <= set(range(1, 9))): raise ValueError("Inkling 2 verification widths must be in [1, 8]") + if self.draft_head_quantization not in (None, "nvfp4", "mxfp8"): + raise ValueError("Inkling 2 draft head quantization must be omitted, nvfp4 or mxfp8") + if self.markov_head_quantization not in (None, "fp8"): + raise ValueError("Inkling 2 recurrent correction head quantization must be omitted or fp8") + if (self.draft_head_quantization is not None or self.markov_head_quantization is not None) and ( + not self.drafting or self.draft_tokens < 1): + raise ValueError("proposal head quantization requires learned speculation with positive draft depth") if self.snapshot_budget_bytes <= 0 or self.headroom_bytes < 0: raise ValueError("snapshot budget must be positive and headroom nonnegative") object.__setattr__(self, "image_concurrency", image_capacity(self.max_active, self.image_concurrency) if self.vision else 0) @@ -217,7 +227,9 @@ def prepare_rank(snapshot, config, *, rank, group, progress=None, core_storage): raise ValueError("a decoder-layer prefix must include a global-attention layer, whose K/V use pages") with phase(progress, "Load weights"): weights = ModelWeights.load(manifest, device, shard=shard, layers=config.decoder_layers, - progress=progress, drafting=config.drafting, vision=config.vision) + progress=progress, drafting=config.drafting, vision=config.vision, + draft_head_quantization=config.draft_head_quantization, + markov_head_quantization=config.markov_head_quantization) weights_bytes = torch.cuda.memory_allocated(device) if torch.cuda.get_device_properties(device).is_integrated: from ignition.preparation.host_memory import release_page_cache @@ -406,7 +418,9 @@ def add(name, forward, kind, rows, tokens, fields, enabled=None, packed=0): prefill_capacities=prefills, free_device_bytes=free, total_device_bytes=total, allocated_tensor_bytes=torch.cuda.memory_allocated(device), reserved_tensor_bytes=torch.cuda.memory_reserved(device), headroom_bytes=config.headroom_bytes, - tuning=tuning, compiled_decoder=config.compile_decoder) + tuning=tuning, compiled_decoder=config.compile_decoder, + draft_head_quantization=config.draft_head_quantization, + markov_head_quantization=config.markov_head_quantization) # Each decoder layer's weights form one stable payload; vocabulary shards, norms and the output query share misc. rebuild_weights = declare_weights(weights, components=("embedding", "embedding_norm", "final_norm", "output_query", "unembedding", "draft", "vision")) diff --git a/ignition/recipes/inkling2/dspark.py b/ignition/recipes/inkling2/dspark.py index 93944531e..83095c9e6 100644 --- a/ignition/recipes/inkling2/dspark.py +++ b/ignition/recipes/inkling2/dspark.py @@ -9,7 +9,7 @@ from ignition.components.collectives import PreparedAllReduce from ignition.components.embedding import PreparedEmbedding -from ignition.components.linear import UnquantizedProjection +from ignition.components.linear import QuantizedProjection, UnquantizedProjection from ignition.components.mlp import MLPScratch, PreparedMLP from ignition.components.paged_attention import PreparedPagedAttention, PreparedPagedKVWriter from ignition.components.recurrent_proposal import PreparedRecurrentProposalHead @@ -18,7 +18,8 @@ from ignition.components.state import gather_rows, save_hidden from ignition.preparation.persistent import persistent_copy, persistent_zeros -from .dspark_state import context_slots, mix_features, rotary, stage_paged_tail, stage_window +from .dspark_state import (block_inputs, context_slots, mix_features, rotary_table, rotate, stage, tail_plan, + window_plan) def inverse_frequencies(draft, device): @@ -66,11 +67,12 @@ def __call__(self, auxiliary, counts, metadata): m = metadata full_slots = context_slots(m, counts) local_slots = context_slots(m, counts, window=self.window) + table = rotary_table(m.positions, self.inverse) for index, (projection, key_norm, context_norm, writer, local) in enumerate(self.stages): context = mix_features(auxiliary, self.alpha[index], context_norm, self.eps) key, value = projection(context).chunk(2, -1) key = key_norm(key.contiguous().view(-1, self.key_shape[-1])).view(self.key_shape) - writer(rotary(key, m.positions, self.inverse), value.contiguous().view(self.key_shape), + writer(rotate(key, table), value.contiguous().view(self.key_shape), local_slots if local else full_slots) indices = (m.query_start_loc[:-1].long() + counts.long() - 1).clamp(0, auxiliary.shape[0] - 1) context = mix_features(auxiliary[indices], self.alpha[-1], self.recurrent_norm, self.eps) @@ -82,6 +84,8 @@ def scratch_tensors(self): @dataclass(frozen=True) class DraftScratch: + # One row per request: the block's first position is the only one that embeds a token. + anchor: torch.Tensor hidden: torch.Tensor accumulator: torch.Tensor qkv: torch.Tensor @@ -99,7 +103,9 @@ def allocate(cls, config, shard, tokens, block, device): start, end = shard.padded_partition(config.vocabulary, alignment=64) def empty(*shape, dtype=torch.bfloat16): return torch.empty(shape, dtype=dtype, device=device) - return cls(empty(tokens, hidden), empty(tokens, hidden, dtype=torch.float32), + if tokens % block: + raise ValueError("Inkling2 DSpark scratch holds whole blocks") + return cls(empty(tokens // block, hidden), empty(tokens, hidden), empty(tokens, hidden, dtype=torch.float32), empty(tokens, (q + 2 * kv) * d.head_dim), empty(tokens, q, d.head_dim), empty(tokens, kv, d.head_dim), empty(tokens, q, d.head_dim), empty(tokens, hidden), MLPScratch.allocate(tokens, hidden, d.text["intermediate_size"] // shard.world_size, device), @@ -108,7 +114,11 @@ def empty(*shape, dtype=torch.bfloat16): def rows(self, tokens, samples): if samples != tokens: raise ValueError("Inkling2 DSpark projects every block position") - return type(self)(*(self.mlp.rows(tokens) if field == "mlp" else getattr(self, field)[:tokens] + block = self.hidden.shape[0] // self.anchor.shape[0] + if tokens % block: + raise ValueError("Inkling2 DSpark scratch holds whole blocks") + counts = {"mlp": None, "anchor": tokens // block} + return type(self)(*(self.mlp.rows(tokens) if field == "mlp" else getattr(self, field)[:counts.get(field, tokens)] for field in self.__dataclass_fields__)) @@ -118,7 +128,6 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru d = config.draft self.query_shape, self.key_shape = tuple(scratch.query.shape), tuple(scratch.key.shape) self.window, self.local = d.window, index in d.local_layers - self.inverse = inverse_frequencies(d, scratch.hidden.device) self.qkv = UnquantizedProjection(scratch.hidden, weights.qkv, scratch.qkv, session=session, name=name + ".qkv").functional(session=session) self.query_norm = norm(scratch.query.flatten(0, 1), weights.query_norm, scratch.query.flatten(0, 1), @@ -134,6 +143,7 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru shape = (rows * self.pages_per_row, 128, *self.key_shape[1:]) key, value = (torch.empty(shape, dtype=torch.bfloat16, device=scratch.hidden.device) for _ in range(2)) tail = {} + self.staging = (window_plan, d.window, self.pages_per_row) else: self.key_cache, self.value_cache = pool.key(index, draft=True), pool.value(index, draft=True) self.pages = None @@ -142,6 +152,7 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru shape = (rows * self.tail_pages, key.shape[1], *self.key_shape[1:]) tail = {role: torch.empty(shape, dtype=key.dtype, device=key.device) for role in ("tail_key", "tail_value")} + self.staging = (tail_plan, key.shape[1], self.tail_pages) self.attention = PreparedPagedAttention(scratch.query, key, value, self.pages if self.local else metadata.page_table, metadata.lengths, metadata.query_start_loc, scratch.attended, mode="verify", width=d.block_size, window=-1, causal=False, @@ -155,26 +166,23 @@ def __init__(self, index, weights, config, scratch, metadata, pool, *, shard, ru self.reduce = PreparedAllReduce(scratch.partial, scratch.hidden, runtime=runtime, coordinator=coordinator, session=session, name=name + ".reduce").functional(session=session) - def __call__(self, hidden, metadata): + def __call__(self, hidden, metadata, plan, table): + """Attend with `plan`, this layer's staging plan, and `table`, the rows' rotary cosines and sines.""" m = metadata q_width = self.query_shape[1] * self.query_shape[2] kv_width = self.key_shape[1] * self.key_shape[2] q, k, v = self.qkv(hidden).split((q_width, kv_width, kv_width), -1) q = self.query_norm(q.contiguous().view(-1, self.query_shape[-1])).view(self.query_shape) k = self.key_norm(k.contiguous().view(-1, self.key_shape[-1])).view(self.key_shape) - q, k = rotary(q, m.positions, self.inverse), rotary(k, m.positions, self.inverse) + q, k = rotate(q, table), rotate(k, table) v = v.contiguous().view(self.key_shape) if self.local: - key = stage_window(self.key_cache, k, m, self.pages_per_row) - value = stage_window(self.value_cache, v, m, self.pages_per_row) - lengths = torch.where(m.lengths > 0, m.starts.clamp(max=self.window).int() + m.lengths, 0) + key, value = stage(self.key_cache, k, plan), stage(self.value_cache, v, plan) tail = {} else: key, value = self.key_cache, self.value_cache - tail = dict(tail_key=stage_paged_tail(key, k, m, self.tail_pages), - tail_value=stage_paged_tail(value, v, m, self.tail_pages)) - lengths = torch.where(m.lengths > 0, m.starts.int() + m.lengths, 0) - attended = self.attention(q, k_cache=key, v_cache=value, cache_seqlens=lengths, + tail = dict(tail_key=stage(key, k, plan), tail_value=stage(value, v, plan)) + attended = self.attention(q, k_cache=key, v_cache=value, cache_seqlens=plan.lengths, cu_seqlens_q=m.query_start_loc, **tail, **({} if self.local else {"page_table": m.page_table})) return self.reduce(self.output(attended.flatten(1))) @@ -195,13 +203,13 @@ def __init__(self, index, weights, owner, scratch, metadata, pool, *, session, n self.reduce = PreparedAllReduce(s.output, scratch.hidden, runtime=runtime, coordinator=coordinator, session=session, name=name + ".mlp.reduce").functional(session=session) - def __call__(self, hidden, residual, metadata): + def __call__(self, hidden, residual, metadata, plan, table): if residual is None: residual = hidden else: hidden = hidden.float() + residual.float() residual = hidden.bfloat16() - attended = self.attention(self.input_norm(hidden), metadata) + attended = self.attention(self.input_norm(hidden), metadata, plan, table) summed = attended.float() + residual.float() residual = summed.bfloat16() return self.reduce(self.mlp(self.post_norm(summed))), residual @@ -221,36 +229,50 @@ def __init__(self, weights, scratch, metadata, pool, *, runtime, coordinator, se first, _ = weights.shard.padded_partition(weights.config.vocabulary, alignment=64) self.vocabulary_start = persistent_copy(torch.tensor([first], dtype=torch.int64, device=device)) self.embedding_weight = weights.embedding - ids = metadata.token_ids.clamp(0, weights.embedding.shape[0] - 1) - self.embedding = PreparedEmbedding(weights.embedding, ids, scratch.hidden, session=session, + self.inverse = inverse_frequencies(d, device) + # Positions after the first take the learned mask, so only each request's first row is + # embedded, reduced across the vocabulary shards and normalized. + ids = metadata.token_ids[::self.block].clamp(0, weights.embedding.shape[0] - 1) + self.embedding = PreparedEmbedding(weights.embedding, ids, scratch.anchor, session=session, name=name + ".embed").functional(session=session, dynamic_inputs=True) - self.embed_reduce = PreparedAllReduce(scratch.hidden, scratch.hidden, runtime=runtime, + self.embed_reduce = PreparedAllReduce(scratch.anchor, scratch.anchor, runtime=runtime, coordinator=coordinator, session=session, name=name + ".embed.reduce").functional(session=session) - self.embed_norm = norm(scratch.hidden, weights.embedding_norm, scratch.hidden, + self.embed_norm = norm(scratch.anchor, weights.embedding_norm, scratch.anchor, eps=d.text["rms_norm_eps"], session=session, name=name + ".embed.norm") self.layers = [DraftLayer(i, layer, weights, scratch, metadata, pool, runtime=runtime, coordinator=coordinator, session=session, name=f"{name}.{i}", workspace=workspace) for i, layer in enumerate(weights.draft.layers)] + # Layers with equal history geometry share one staging plan per forward. + self.staging = tuple(dict.fromkeys(layer.attention.staging for layer in self.layers)) + self.layer_plans = tuple(self.staging.index(layer.attention.staging) for layer in self.layers) self.norm = norm(scratch.accumulator, weights.draft.norm, scratch.hidden, eps=d.text["rms_norm_eps"], session=session, name=name + ".norm") - self.head = UnquantizedProjection(scratch.hidden, weights.head, scratch.local_logits, - session=session, name=name + ".head").functional(session=session) + if weights.draft.draft_head is None: + self.head = UnquantizedProjection(scratch.hidden, weights.head, scratch.local_logits, + session=session, name=name + ".head").functional(session=session) + else: + # A quantized copy of the target's vocabulary shard serves proposals; verification keeps the target's. + self.head = QuantizedProjection(scratch.hidden, weights.draft.draft_head, scratch.local_logits, + session=session, name=name + ".head", activation_mode="a16", + workspace=workspace).functional(session=session) self.recurrent = PreparedRecurrentProposalHead(weights.draft.recurrent, - torch.zeros(metadata.lengths.numel(), dtype=torch.int64, device=device), session=session, name=name + ".recurrent") + torch.zeros(metadata.lengths.numel(), dtype=torch.int64, device=device), + positions=self.block - 1, session=session, name=name + ".recurrent") def release_outputs(self): return self def __call__(self, metadata): - ids = metadata.token_ids - self.vocabulary_start + ids = metadata.token_ids[::self.block] - self.vocabulary_start owned = (ids >= 0) & (ids < self.embedding_weight.shape[0]) local = self.embedding(token_ids=ids.clamp(0, self.embedding_weight.shape[0] - 1)) - hidden = self.embed_norm(self.embed_reduce(torch.where(owned[:, None], local, 0))) - positions = torch.arange(hidden.shape[0], device=hidden.device) % self.block - hidden = torch.where((positions == 0)[:, None], hidden, self.mask) + anchor = self.embed_norm(self.embed_reduce(torch.where(owned[:, None], local, 0))) + hidden = block_inputs(anchor, self.mask, self.block) + table = rotary_table(metadata.positions, self.inverse) + plans = [plan(metadata, *geometry) for plan, *geometry in self.staging] residual = None - for layer in self.layers: - hidden, residual = layer(hidden, residual, metadata) + for layer, plan in zip(self.layers, self.layer_plans): + hidden, residual = layer(hidden, residual, metadata, plans[plan], table) hidden = self.norm(hidden.float() + residual.float())[metadata.sample_indices] logits = self.head(hidden) context = torch.empty((metadata.state_rows.numel(), self.recurrent_context.shape[1]), diff --git a/ignition/recipes/inkling2/dspark_state.py b/ignition/recipes/inkling2/dspark_state.py index 71c9399f3..a6c9886a9 100644 --- a/ignition/recipes/inkling2/dspark_state.py +++ b/ignition/recipes/inkling2/dspark_state.py @@ -3,6 +3,8 @@ """Inkling2 draft feature mixing, rotary positions, and committed context selection.""" +from dataclasses import dataclass + import torch @@ -14,46 +16,78 @@ def mix_features(features, alpha, weight, eps): return (mixed * scale * weight.float()).bfloat16() -def rotary(values, positions, inverse): +def block_inputs(anchor, mask, block): + """Expand one normalized first-position embedding per request into its block of learned masks.""" + return torch.cat((anchor[:, None], mask.expand(anchor.shape[0], block - 1, -1)), 1).flatten(0, 1) + + +def rotary_table(positions, inverse): + """Return FP32 cosines and sines of each row's position, shared by every rotation of those rows.""" angle = positions.clamp_min(0).float()[:, None] * inverse[None] - cosine, sine = angle.cos()[:, None], angle.sin()[:, None] + return angle.cos()[:, None], angle.sin()[:, None] + + +def rotate(values, table): + cosine, sine = table left, right = values.float().chunk(2, -1) return torch.cat((left * cosine - right * sine, right * cosine + left * sine), -1).to(values.dtype) -def stage_window(history, values, metadata, pages): - """Stage the same committed suffix before every query in a noncausal block.""" +def rotary(values, positions, inverse): + return rotate(values, rotary_table(positions, inverse)) + + +@dataclass(frozen=True) +class StagingPlan: + """Where each slot of a request's private attention pages comes from. + + The plan depends on request metadata only, so one plan serves the key and value histories of + every draft layer with the same history geometry. A slot holds committed history + (`history[rows, slots]`), a row of the block being drafted (`values[indices]`), or zero. + `lengths` is each request's visible key count. + """ + + rows: torch.Tensor + slots: torch.Tensor + indices: torch.Tensor + committed: torch.Tensor + valid: torch.Tensor + lengths: torch.Tensor + page_tokens: int + + +def window_plan(metadata, window, pages, *, page_tokens=128): + """Plan the same committed suffix of a `window`-position ring before every query of a noncausal block.""" m = metadata - rows, window = m.starts.numel(), history.shape[1] - capacity = pages * 128 - offsets = torch.arange(capacity, device=values.device)[None] + offsets = torch.arange(pages * page_tokens, device=m.starts.device)[None] prefix = m.starts.clamp(max=window)[:, None] positions = m.starts[:, None] - prefix + offsets - previous = history[m.state_rows[:, None], positions.remainder(window)] - indices = (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, values.shape[0] - 1) - current = values[indices] - valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) - return torch.where(valid[:, :, None, None], - torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view( - rows * pages, 128, *values.shape[1:]) + return StagingPlan(m.state_rows[:, None], positions.remainder(window), + (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, m.positions.numel() - 1), + offsets < prefix, (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]), + torch.where(m.lengths > 0, m.starts.clamp(max=window).int() + m.lengths, 0), page_tokens) -def stage_paged_tail(history, values, metadata, pages): - """Copy the committed partial page and proposal rows into private attention pages.""" +def tail_plan(metadata, page_tokens, pages): + """Plan the committed partial page followed by the block's rows in private tail pages.""" m = metadata - rows, page_tokens = m.starts.numel(), history.shape[1] - offsets = torch.arange(pages * page_tokens, device=values.device)[None] + rows = m.starts.numel() + offsets = torch.arange(pages * page_tokens, device=m.starts.device)[None] prefix = m.starts.remainder(page_tokens)[:, None] columns = (m.starts // page_tokens).clamp(max=m.page_table.shape[1] - 1) - requests = torch.arange(rows, device=values.device) + requests = torch.arange(rows, device=m.starts.device) source_pages = m.page_table[requests, columns].long().clamp_min(0) - previous = history[source_pages[:, None], offsets.remainder(page_tokens)] - indices = (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, values.shape[0] - 1) - current = values[indices] - valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) - return torch.where(valid[:, :, None, None], - torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view( - rows * pages, page_tokens, *values.shape[1:]) + return StagingPlan(source_pages[:, None], offsets.remainder(page_tokens), + (m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, m.positions.numel() - 1), + offsets < prefix, (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]), + torch.where(m.lengths > 0, m.starts.int() + m.lengths, 0), page_tokens) + + +def stage(history, values, plan): + """Copy the planned history and block rows into private attention pages without changing either source.""" + staged = torch.where(plan.valid[:, :, None, None], + torch.where(plan.committed[:, :, None, None], history[plan.rows, plan.slots], values[plan.indices]), 0) + return staged.view(-1, plan.page_tokens, *values.shape[1:]) def accepted_rows(metadata, counts, *, window=None): diff --git a/ignition/recipes/inkling2/dspark_weights.py b/ignition/recipes/inkling2/dspark_weights.py index 37ace67cb..8d7652ac3 100644 --- a/ignition/recipes/inkling2/dspark_weights.py +++ b/ignition/recipes/inkling2/dspark_weights.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 Flare community contributors # SPDX-License-Identifier: Apache-2.0 -"""Checkpoint-preserving tensor partitions for Inkling2's dense DSpark decoder.""" +"""Tensor partitions for Inkling2's dense DSpark decoder and its optional quantized proposal heads.""" from dataclasses import dataclass from functools import cached_property @@ -10,6 +10,8 @@ import torch +from ignition.components.markov_sampling import fp8_markov_head +from ignition.components.vocabulary import mxfp8_draft_head, nvfp4_draft_head from ignition.components.weights import DirectWeightLoader, MLPWeights from ignition.preparation.checkpoint import IndexedCheckpoint, _read_header from ignition.preparation.persistent import persistent_copy, persistent_empty @@ -74,25 +76,36 @@ class DraftLayerWeights: mlp: MLPWeights +DRAFT_HEAD_FORMATS = {"nvfp4": nvfp4_draft_head, "mxfp8": mxfp8_draft_head} + + +def _transient(shape, dtype, device): + return torch.empty(shape, dtype=dtype, device=device) + + @dataclass(frozen=True) class RecurrentWeights: embedding: torch.Tensor state_weight: torch.Tensor context_weight: torch.Tensor bias: torch.Tensor - head: torch.Tensor + # Vocabulary rows by recurrent rank: BF16, or row-scaled E4M3 when the correction head is quantized. + head: object head_bias: torch.Tensor context_norm: torch.Tensor softcap: float = 30. -def fold_recurrent_head(head, hidden_out, hidden_bias, *, chunk=4096): - """Fold the BF16 output map through the vocabulary head with FP64 products.""" +def fold_recurrent_head(head, hidden_out, hidden_bias, *, chunk=4096, allocate=persistent_empty): + """Fold the BF16 output map through the vocabulary head with FP64 products. + + `allocate` places the folded matrix; a caller that quantizes it passes a transient allocator. + """ if (head.ndim != 2 or hidden_out.ndim != 2 or hidden_out.shape[0] != head.shape[1] or hidden_bias.shape != (head.shape[1],) or any(t.dtype != torch.bfloat16 or t.device != head.device for t in (head, hidden_out, hidden_bias))): raise ValueError("recurrent output folding requires compatible BF16 output and vocabulary matrices") - weight = persistent_empty((head.shape[0], hidden_out.shape[1]), torch.bfloat16, head.device) + weight = allocate((head.shape[0], hidden_out.shape[1]), torch.bfloat16, head.device) bias = persistent_empty((head.shape[0],), torch.bfloat16, head.device) projection, offset = hidden_out.double(), hidden_bias.double() for start in range(0, head.shape[0], chunk): @@ -110,9 +123,22 @@ class DraftWeights: norm: torch.Tensor layers: tuple[DraftLayerWeights, ...] recurrent: RecurrentWeights + # Quantized copy of the target vocabulary shard that only proposals read; None projects with the target's. + draft_head: object | None = None @classmethod - def load(cls, snapshot, config, device, shard, *, target_head, progress=None): + def load(cls, snapshot, config, device, shard, *, target_head, progress=None, + draft_head_quantization=None, markov_head_quantization=None): + """Load one rank's drafter and derive its proposal heads from checkpoint values. + + `draft_head_quantization` (`nvfp4` or `mxfp8`) adds a quantized copy of `target_head` for the block's + vocabulary logits. `markov_head_quantization` (`fp8`) stores the recurrent correction head, the + output map folded through `target_head`, as row-scaled E4M3. The target head keeps BF16. + """ + if draft_head_quantization not in (None, *DRAFT_HEAD_FORMATS): + raise ValueError("Inkling2 draft head quantization must be omitted, nvfp4 or mxfp8") + if markov_head_quantization not in (None, "fp8"): + raise ValueError("Inkling2 recurrent correction head quantization must be omitted or fp8") snapshot = Path(snapshot) sidecar, tensors = _read_header(snapshot / "mtp.safetensors", "mtp.safetensors") manifest = DraftCheckpoint(snapshot, tensors, (sidecar,)) @@ -160,8 +186,16 @@ def vector(name, width): raise ValueError("Inkling2 predecessor embedding must be a BF16 vocabulary by recurrent-rank matrix") hidden_out = loader.load_tensor("markov_head.hidden_out.weight", temporary=True) hidden_bias = vector("markov_head.hidden_out.bias", hidden) - head, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias) + if markov_head_quantization is None: + head, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias) + else: + folded, head_bias = fold_recurrent_head(target_head, hidden_out, hidden_bias, allocate=_transient) + head = fp8_markov_head(folded) + del folded recurrent = RecurrentWeights(embedding, persistent_copy(state[:, :2 * rank].contiguous()), persistent_copy(state[:, 2 * rank:].contiguous()), persistent_copy(vector("markov_head.rnn_proj.bias", 3 * rank).float()), head, head_bias, vector("markov_head.dflare_norm.weight", hidden)) - return cls(alpha, vector("mask_embed", hidden), vector("norm.weight", hidden), tuple(layers), recurrent) + draft_head = (None if draft_head_quantization is None + else DRAFT_HEAD_FORMATS[draft_head_quantization](target_head)) + return cls(alpha, vector("mask_embed", hidden), vector("norm.weight", hidden), tuple(layers), recurrent, + draft_head) diff --git a/ignition/recipes/inkling2/recipe.py b/ignition/recipes/inkling2/recipe.py index 192907937..29e3b9c01 100644 --- a/ignition/recipes/inkling2/recipe.py +++ b/ignition/recipes/inkling2/recipe.py @@ -22,12 +22,11 @@ def prepare(snapshot, plan, *, compile_forwards, kernel_cache, progress, retain_ if plan.prompt_logprobs: raise ValueError("Inkling 2 preparation does not capture prompt scoring graphs") speculation = plan.speculation - if speculation is not None and (speculation.draft_head_quantization is not None - or speculation.markov_head_quantization is not None): - raise ValueError("Inkling 2 retains the bundled draft checkpoint precision") drafting = {} if speculation is None else dict(speculative_widths=speculation.widths, speculative_fixed_width=speculation.fixed_width, speculative_token_capacity=plan.speculative_token_capacity, - speculative_graph_budget=speculation.graph_budget or 64) + speculative_graph_budget=speculation.graph_budget or 64, + draft_head_quantization=speculation.draft_head_quantization, + markov_head_quantization=speculation.markov_head_quantization) config = PreparationConfig(collective_tuning=plan.collective_tuning, collective_ladder=plan.collective_ladder, prompt_logprobs=plan.prompt_logprobs, compile_workers=plan.compile_workers, native_dense_activations=plan.native_dense_activations, @@ -54,6 +53,8 @@ def usage(prepared): RECIPE = Recipe(name="inkling2", served_model_name="Inkling-2-Small-NVFP4", prepare=prepare, usage=usage, sampling_defaults=SAMPLING_DEFAULTS, + # Proposal-only copies: the block's vocabulary head, and the recurrent correction head as row-scaled E4M3. + draft_head_quantizations=("nvfp4", "mxfp8"), markov_head_quantizations=("fp8",), checkpoints=(Checkpoint(alias="inkling-2-small-nvfp4", repository="/data/models/Inkling-2-Small-NVFP4", revision="local", architecture="Inkling2ForConditionalGeneration", stored_bytes=344_366_116_864, device_bytes=344_366_116_864, expansion_bytes=5_673_717_760, draft_bytes=2_684_203_392),), diff --git a/ignition/recipes/inkling2/weights.py b/ignition/recipes/inkling2/weights.py index 7a851c2d1..72731dba7 100644 --- a/ignition/recipes/inkling2/weights.py +++ b/ignition/recipes/inkling2/weights.py @@ -72,7 +72,8 @@ def embed_norm(self): return self.embedding_norm @classmethod - def load(cls, manifest, device, *, shard, progress=None, drafting=True, vision=True, layers=None): + def load(cls, manifest, device, *, shard, progress=None, drafting=True, vision=True, layers=None, + draft_head_quantization=None, markov_head_quantization=None): before = torch.cuda.memory_allocated(device) with InklingWeightLoader(manifest, device, shard=shard, progress=progress, vision=vision) as loader: c = loader.config @@ -105,7 +106,8 @@ def vocabulary_rows(name): draft = None if drafting: from .dspark_weights import DraftWeights - draft = DraftWeights.load(manifest.snapshot, c, device, shard, target_head=head, progress=progress) + draft = DraftWeights.load(manifest.snapshot, c, device, shard, target_head=head, progress=progress, + draft_head_quantization=draft_head_quantization, markov_head_quantization=markov_head_quantization) return cls(c, shard, loaded, embedding, embed_norm, norm, query, head, torch.cuda.memory_allocated(device) - before, draft, tower) diff --git a/tests/cli/test_inkling2.py b/tests/cli/test_inkling2.py index c614160f0..8d3472320 100644 --- a/tests/cli/test_inkling2.py +++ b/tests/cli/test_inkling2.py @@ -55,3 +55,63 @@ def capture_config(**kwargs): assert config.speculative_widths == tuple(range(1, 9)) assert config.speculative_token_capacity == 64 assert config.speculative_graph_budget == 64 + + +def test_inkling_declares_quantized_proposal_heads_with_checkpoint_precision_as_default(): + assert RECIPE.draft_head_quantizations == ("nvfp4", "mxfp8") + assert RECIPE.markov_head_quantizations == ("fp8",) + assert RECIPE.speculation.draft_head_quantization is None + assert RECIPE.speculation.markov_head_quantization is None + model = ResolvedModel(RECIPE, RECIPE.checkpoints[0]) + args = _parser().parse_args(["export", model.checkpoint.alias, "--draft-head-quantization", "nvfp4", + "--markov-head-quantization", "fp8"]) + speculation = preparation.speculation(args, model) + assert (speculation.draft_head_quantization, speculation.markov_head_quantization) == ("nvfp4", "fp8") + assert speculation.draft_tokens == 7 + + +@pytest.mark.parametrize("draft_head,markov_head", [(None, None), ("nvfp4", "fp8"), ("mxfp8", None), (None, "fp8")]) +def test_inkling_adapter_passes_proposal_head_formats_to_preparation(monkeypatch, tmp_path, draft_head, markov_head): + from ignition.recipes.inkling2 import bootstrap + + devices = tuple(gpu(i, 95) for i in range(4)) + speculation = replace(RECIPE.speculation, draft_head_quantization=draft_head, + markov_head_quantization=markov_head) + plan = derive(model=RECIPE, gpus=devices, selected=(0, 1, 2, 3), concurrency=8, speculation=speculation) + captured = [] + configuration = bootstrap.PreparationConfig + + class ConfigurationCaptured(Exception): + pass + + def capture_config(**kwargs): + captured.append(configuration(**kwargs)) + raise ConfigurationCaptured + + monkeypatch.setattr(bootstrap, "PreparationConfig", capture_config) + with pytest.raises(ConfigurationCaptured): + RECIPE.prepare(tmp_path, plan, compile_forwards=False, kernel_cache=None, progress=None) + config = captured.pop() + assert (config.draft_head_quantization, config.markov_head_quantization) == (draft_head, markov_head) + + +def test_inkling_plan_rejects_an_nvfp4_recurrent_correction_head(): + from ignition.errors import UsageError + + devices = tuple(gpu(i, 95) for i in range(4)) + speculation = replace(RECIPE.speculation, markov_head_quantization="nvfp4") + with pytest.raises(UsageError, match="--markov-head-quantization nvfp4"): + derive(model=RECIPE, gpus=devices, selected=(0, 1, 2, 3), concurrency=8, speculation=speculation) + + +@pytest.mark.parametrize("arguments,message", [ + (dict(draft_head_quantization="fp8"), "draft head quantization"), + (dict(markov_head_quantization="nvfp4"), "correction head quantization"), + (dict(draft_head_quantization="nvfp4", drafting=False), "positive draft depth"), + (dict(markov_head_quantization="fp8", speculative_widths=(1,)), "positive draft depth"), +]) +def test_inkling_preparation_config_validates_proposal_head_formats(arguments, message): + from ignition.recipes.inkling2.bootstrap import PreparationConfig + + with pytest.raises(ValueError, match=message): + PreparationConfig(**arguments) diff --git a/tests/components/test_inkling2_dspark.py b/tests/components/test_inkling2_dspark.py index bd5a0d430..c2f727db5 100644 --- a/tests/components/test_inkling2_dspark.py +++ b/tests/components/test_inkling2_dspark.py @@ -10,10 +10,11 @@ from ignition.components.paged_batch import Metadata from ignition.components.parallel import TensorParallelShard -from ignition.components.recurrent_proposal import corrected_logits, recurrent_gate -from ignition.recipes.inkling2.dspark_state import context_slots, mix_features, rotary, stage_paged_tail, stage_window -from ignition.recipes.inkling2.dspark_weights import fold_recurrent_head, mtp_quantization -from ignition.recipes.inkling2.dspark import PreparedContext, PreparedDraft, SpeculativePreparation +from ignition.components.recurrent_proposal import PreparedRecurrentProposalHead, corrected_logits, recurrent_gate +from ignition.recipes.inkling2.dspark_state import (block_inputs, context_slots, mix_features, rotary, rotary_table, + rotate, stage, tail_plan, window_plan) +from ignition.recipes.inkling2.dspark_weights import DraftWeights, fold_recurrent_head, mtp_quantization +from ignition.recipes.inkling2.dspark import DraftScratch, PreparedContext, PreparedDraft, SpeculativePreparation @pytest.mark.parametrize("draft_tokens", [0, 1, 7]) @@ -31,6 +32,7 @@ def test_speculative_preparation_retains_context_at_zero_draft_depth(draft_token assert preparation.forwards.context is PreparedContext assert preparation.forwards.draft is PreparedDraft assert preparation.draft_scratch.hidden.shape == (8, 128) + assert preparation.draft_scratch.anchor.shape == (1, 128) assert not preparation.offset.count_nonzero() assert not preparation.generation.count_nonzero() @@ -87,7 +89,9 @@ def test_noncausal_staging_retains_one_prefix_for_all_eight_positions(start): m = Metadata.pack(torch.tensor([start]), torch.tensor([8]), torch.tensor([1]), torch.tensor([[0] * 16], dtype=torch.int32), torch.zeros(1, block, dtype=torch.int64), torch.tensor([1], dtype=torch.int32), token_capacity=8, sample_count=8) - staged = stage_window(history, values, m, 9).flatten(0, 1) + plan = window_plan(m, window, 9) + staged = stage(history, values, plan).flatten(0, 1) + assert plan.lengths.tolist() == [min(start, window) + block] prefix = min(start, window) expected = torch.arange(start - prefix, start + block).bfloat16() assert torch.equal(staged[:prefix + block, 0, 0], expected) @@ -115,7 +119,9 @@ def test_paged_tail_preserves_committed_prefix_with_one_target_page(): m = Metadata.pack(torch.tensor([127]), torch.tensor([8]), torch.tensor([1]), torch.tensor([[2]], dtype=torch.int32), torch.zeros(1, 8, dtype=torch.int64), torch.tensor([1], dtype=torch.int32), token_capacity=8, sample_count=8) - staged = stage_paged_tail(history, proposal, m, 2).flatten(0, 1) + plan = tail_plan(m, 128, 2) + staged = stage(history, proposal, plan).flatten(0, 1) + assert plan.lengths.tolist() == [135] assert torch.equal(staged[:127], history[2, :127]) assert torch.equal(staged[127:135], proposal) assert not staged[135:].count_nonzero() @@ -147,3 +153,145 @@ def test_mtp_mxfp8_metadata_requires_activation_quantization_and_complete_covera mtp_quantization(changed, names) with pytest.raises(ValueError, match="cover exactly"): mtp_quantization(config, names + ["undeclared.weight"]) + + +def test_one_staging_plan_serves_key_and_value_histories_of_every_layer(): + generator = torch.Generator().manual_seed(9) + window, block, pages = 1024, 8, 9 + m = Metadata.pack(torch.tensor([1027, 5, 300]), torch.tensor([8, 8, 0]), torch.tensor([2, 1, 0]), + torch.tensor([[0] * 16, [1] * 16, [-1] * 16], dtype=torch.int32), torch.zeros(3, block, dtype=torch.int64), + torch.tensor([3], dtype=torch.int32), token_capacity=24, sample_count=24) + plan = window_plan(m, window, pages) + offsets = torch.arange(pages * 128)[None] + prefix = m.starts.clamp(max=window)[:, None] + for _ in range(2): + history = torch.randn(3, window, 2, 4, generator=generator).bfloat16() + values = torch.randn(24, 2, 4, generator=generator).bfloat16() + previous = history[m.state_rows[:, None], (m.starts[:, None] - prefix + offsets).remainder(window)] + current = values[(m.query_start_loc[:-1, None] + offsets - prefix).clamp(0, 23)] + valid = (m.lengths[:, None] > 0) & (offsets < prefix + m.lengths[:, None]) + expected = torch.where(valid[:, :, None, None], + torch.where((offsets < prefix)[:, :, None, None], previous, current), 0).view(3 * pages, 128, 2, 4) + assert torch.equal(stage(history, values, plan), expected) + assert plan.lengths.tolist() == [1032, 13, 0] + assert not stage(history, values, plan)[2 * pages:].count_nonzero() + + +def test_shared_rotary_table_matches_per_call_rotation(): + generator = torch.Generator().manual_seed(12) + positions = torch.tensor([0, 7, 1025, -1, 1048575]) + inverse = 1 / (1e6 ** (torch.arange(0, 128, 2).float() / 128)) + table = rotary_table(positions, inverse) + for heads in (2, 12): + values = torch.randn(5, heads, 128, generator=generator).bfloat16() + assert torch.equal(rotate(values, table), rotary(values, positions, inverse)) + + +def test_block_inputs_embed_only_each_request_first_position(): + generator = torch.Generator().manual_seed(15) + anchor = torch.randn(3, 32, generator=generator).bfloat16() + mask = torch.randn(32, generator=generator).bfloat16() + positions = torch.arange(24) % 8 + expected = torch.where((positions == 0)[:, None], anchor.repeat_interleave(8, 0), mask) + assert torch.equal(block_inputs(anchor, mask, 8), expected) + + +def test_draft_scratch_keeps_one_anchor_row_per_block(): + config = SimpleNamespace(text={"hidden_size": 128}, vocabulary=256, + draft=SimpleNamespace(heads=4, kv_heads=4, head_dim=128, text={"intermediate_size": 256})) + scratch = DraftScratch.allocate(config, TensorParallelShard(0, 4), 32, 8, torch.device("cpu")) + assert scratch.anchor.shape == (4, 128) and scratch.hidden.shape == (32, 128) + rows = scratch.rows(16, 16) + assert rows.anchor.shape == (2, 128) and rows.hidden.shape == (16, 128) + assert rows.anchor.data_ptr() == scratch.anchor.data_ptr() + with pytest.raises(ValueError, match="whole blocks"): + scratch.rows(12, 12) + with pytest.raises(ValueError, match="whole blocks"): + DraftScratch.allocate(config, TensorParallelShard(0, 4), 12, 8, torch.device("cpu")) + + +def _recurrent_head(weight, bias, head, head_bias, positions, *, row_scaled): + """Bind FP64 reference projections to the recurrent head's arithmetic.""" + rank = head.shape[1] + operation = object.__new__(PreparedRecurrentProposalHead) + operation.rank, operation.softcap, operation.positions = rank, 30., positions + operation.context_bias = bias + operation.context = lambda source: (source.double() @ weight[:, 2 * rank:].double().T).float() + operation.dynamic = lambda source: (source.double() @ weight[:, :2 * rank].double().T).float() + operation.embed = lambda token_ids: torch.zeros(token_ids.numel(), rank, dtype=torch.bfloat16) + if row_scaled: + operation.output = lambda latent: (latent.double() @ head.double().T).float() + operation.output_bias = head_bias + else: + operation.output = lambda latent: (latent.double() @ head.double().T + head_bias.double()).bfloat16() + operation.output_bias = None + return operation + + +def test_block_context_projection_matches_one_projection_per_position(): + generator = torch.Generator().manual_seed(18) + rows, positions, hidden, rank, vocabulary = 3, 7, 16, 8, 24 + weight = (torch.randn(3 * rank, 2 * rank + 2 * hidden, generator=generator) / 8).bfloat16() + bias = torch.randn(3 * rank, generator=generator) / 8 + head = (torch.randn(vocabulary, rank, generator=generator) / 4).bfloat16() + head_bias = (torch.randn(vocabulary, generator=generator) / 4).bfloat16() + states = torch.randn(rows, positions, hidden, generator=generator).bfloat16() + context = torch.randn(rows, hidden, generator=generator).bfloat16() + operation = _recurrent_head(weight, bias, head, head_bias, positions, row_scaled=False) + fixed = operation.project_context(states, context) + assert fixed.shape == (rows, positions, 3 * rank) and fixed.dtype == torch.float32 + for index in range(positions): + source = torch.cat((states[:, index], context), -1) + assert torch.equal(fixed[:, index], operation.context(source) + bias) + with pytest.raises(ValueError, match="every prepared block position"): + operation.project_context(states[:, :3], context) + + +def test_row_scaled_correction_adds_the_bf16_bias_in_fp32_before_bf16_rounding(): + generator = torch.Generator().manual_seed(21) + rows, hidden, rank, vocabulary = 2, 16, 8, 24 + weight = (torch.randn(3 * rank, 2 * rank + 2 * hidden, generator=generator) / 8).bfloat16() + bias = torch.randn(3 * rank, generator=generator) / 8 + head = (torch.randn(vocabulary, rank, generator=generator) / 4).bfloat16() + head_bias = (torch.randn(vocabulary, generator=generator) / 4).bfloat16() + base = torch.randn(rows, vocabulary, generator=generator).bfloat16() + fixed = torch.randn(rows, 3 * rank, generator=generator) + previous = torch.zeros(rows, dtype=torch.int64) + outputs = [] + for row_scaled in (False, True): + operation = _recurrent_head(weight, bias, head, head_bias, 1, row_scaled=row_scaled) + state, logits = operation.step(operation.begin(base), previous, fixed, base) + outputs.append((state, logits)) + state, latent = recurrent_gate(fixed, torch.zeros(rows, rank, dtype=torch.bfloat16)) + correction = ((latent.double() @ head.double().T).float() + head_bias.float()).bfloat16() + assert torch.equal(outputs[1][0], state) + assert torch.equal(outputs[1][1], corrected_logits(base, correction, 30.)) + # With identical weights the two bias placements differ only by one FP32 addition before BF16 rounding. + assert torch.equal(outputs[0][0], outputs[1][0]) + torch.testing.assert_close(outputs[0][1].float(), outputs[1][1].float(), rtol=2**-7, atol=2**-9) + + +def test_folded_correction_head_uses_the_supplied_allocator(): + generator = torch.Generator().manual_seed(24) + head = torch.randn(64, 32, generator=generator).bfloat16() + weight = torch.randn(32, 8, generator=generator).bfloat16() + bias = torch.randn(32, generator=generator).bfloat16() + calls = [] + + def transient(shape, dtype, device): + calls.append(shape) + return torch.empty(shape, dtype=dtype, device=device) + + folded, offset = fold_recurrent_head(head, weight, bias, allocate=transient) + assert calls == [(64, 8)] + persistent, persistent_offset = fold_recurrent_head(head, weight, bias) + assert torch.equal(folded, persistent) and torch.equal(offset, persistent_offset) + + +@pytest.mark.parametrize("arguments,message", [ + (dict(draft_head_quantization="fp8"), "draft head quantization"), + (dict(markov_head_quantization="nvfp4"), "correction head quantization"), +]) +def test_draft_weights_reject_unsupported_head_formats_before_reading(tmp_path, arguments, message): + with pytest.raises(ValueError, match=message): + DraftWeights.load(tmp_path, None, torch.device("cpu"), TensorParallelShard(0, 4), target_head=None, **arguments) diff --git a/tests/components/test_inkling2_dspark_cuda.py b/tests/components/test_inkling2_dspark_cuda.py index 7da8173e2..4a837f851 100644 --- a/tests/components/test_inkling2_dspark_cuda.py +++ b/tests/components/test_inkling2_dspark_cuda.py @@ -20,7 +20,10 @@ from ignition.components.parallel import TensorParallelShard from ignition.recipes.inkling2.dspark import DraftAttention, DraftScratch, PreparedContext, PreparedDraft from ignition.components.weights import MLPWeights -from ignition.recipes.inkling2.dspark_state import mix_features, rotary +from ignition.components.markov_sampling import fp8_markov_head +from ignition.components.vocabulary import nvfp4_draft_head +from ignition.recipes.inkling2.dspark import inverse_frequencies +from ignition.recipes.inkling2.dspark_state import mix_features, rotary, rotary_table from ignition.recipes.inkling2.dspark_weights import DraftWeights from ignition.components.weights import DirectWeightLoader from ignition.preparation.checkpoint import inspect_indexed_checkpoint @@ -31,9 +34,10 @@ reason="requires an explicitly assigned CUDA device") -@pytest.mark.parametrize("batch,depth", [(1, 7), (4, 7), (4, 1)]) +@pytest.mark.parametrize("batch,depth,row_scaled", [(1, 7, False), (4, 7, False), (4, 1, False), + (1, 7, True), (4, 7, True)]) @torch.no_grad() -def test_sequential_recurrent_draws_record_actual_q_and_reset_state(batch, depth): +def test_sequential_recurrent_draws_record_actual_q_and_reset_state(batch, depth, row_scaled): device = torch.device("cuda:0") generator = torch.Generator().manual_seed(422) hidden, rank, vocabulary, local = 128, 256, 513, 576 @@ -43,6 +47,10 @@ def values(*shape, denominator=64): recurrent = SimpleNamespace(embedding=values(vocabulary, rank), state_weight=weight[:, :2 * rank].contiguous(), context_weight=weight[:, 2 * rank:].contiguous(), bias=values(3 * rank).float(), head=values(local, rank), head_bias=values(local), softcap=30.) + correction_head = recurrent.head + if row_scaled: + recurrent.head = fp8_markov_head(correction_head) + correction_head = recurrent.head.values.float() * recurrent.head.scale[:, None] features, context = values(batch, depth, hidden, denominator=8), values(batch, hidden, denominator=8) backing = values(batch, 8, local, denominator=2) base = backing[:, 1:depth + 1] @@ -65,7 +73,7 @@ def values(*shape, denominator=64): with RecordedPreparationSession(device=device, checkpoint="synthetic-recurrent-head", workload_schema="conditional-proposals", numerical_policy="bf16-checkpoint", autotune_effort="off", compile_workers=2) as session: - head = PreparedRecurrentProposalHead(recurrent, pending, session=session, name="head") + head = PreparedRecurrentProposalHead(recurrent, pending, positions=depth, session=session, name="head") sampler = PreparedVocabularySoftmax(base, q, valid, requests, slots, generations, live, tokens, start=0, runtime=None, coordinator=None, session=session, name="q", pending=pending, seeds=seeds, offsets=offsets, tokens=tokens) @@ -75,9 +83,10 @@ def values(*shape, denominator=64): def forward(): state = head.begin(context) + fixed = head.project_context(features, context) previous = pending for index in range(depth): - state, logits = head.step(state, previous, features[:, index].contiguous(), context, base[:, index]) + state, logits = head.step(state, previous, fixed[:, index], base[:, index]) observed_states[:, index].copy_(state) scores[:, index].copy_(logits) sampler.run_step(index, scores, tokens, pending=pending, seeds=seeds, offsets=offsets, @@ -112,9 +121,20 @@ def forward(): torch.testing.assert_close(raw.double(), concatenated, rtol=2e-7, atol=2e-7) state, latent = recurrent_gate(raw, state) assert torch.equal(observed_states[:active, index], state[:active]) - correction = (latent.double() @ recurrent.head.double().T + recurrent.head_bias.double()).bfloat16() - expected_scores = corrected_logits(base[:, index], correction, 30.) - assert torch.equal(scores[:active, index], expected_scores[:active]) + if row_scaled: + # The E4M3 product is FP32; the BF16 bias joins it before the BF16 rounding. + correction = ((latent.double() @ correction_head.double().T).float() + + recurrent.head_bias.float()).bfloat16() + expected_scores = corrected_logits(base[:, index], correction, 30.) + # FP32 accumulation order can move a correction by one BF16 step. + torch.testing.assert_close(scores[:active, index].float(), expected_scores[:active].float(), + rtol=2**-7, atol=2**-9) + expected_scores = scores[:, index].clone() + else: + correction = (latent.double() @ correction_head.double().T + + recurrent.head_bias.double()).bfloat16() + expected_scores = corrected_logits(base[:, index], correction, 30.) + assert torch.equal(scores[:active, index], expected_scores[:active]) expected_q = expected_scores[:, :vocabulary].float().softmax(-1) torch.testing.assert_close(q[:active, index, :vocabulary], expected_q[:active], rtol=3e-6, atol=1e-9) assert not q[:, index, vocabulary:].count_nonzero() @@ -241,14 +261,20 @@ def values(*shape): operation = DraftAttention(layer, weights, config, scratch, m, pool, shard=TensorParallelShard(0, 1), runtime=None, coordinator=None, session=session, name="attention", workspace=arena) session.tune(show_progress=False) - actual = operation(source, m) + inverse = inverse_frequencies(d, device) + + def attend(): + plan = operation.staging[0](m, *operation.staging[1:]) + return operation(source, m, plan, rotary_table(m.positions, inverse)) + + actual = attend() qkv = (source[:8].double() @ weights.qkv.double().T).bfloat16() q, k, v = qkv.split((heads * dimension, kv_heads * dimension, kv_heads * dimension), -1) def normalize(x, count): x = x.view(8, count, dimension).float() return (x * torch.rsqrt(x.square().mean(-1, keepdim=True) + 1e-6)).bfloat16() - q = rotary(normalize(q, heads), m.positions[:8], operation.inverse) - k = rotary(normalize(k, kv_heads), m.positions[:8], operation.inverse) + q = rotary(normalize(q, heads), m.positions[:8], inverse) + k = rotary(normalize(k, kv_heads), m.positions[:8], inverse) v = v.reshape(8, kv_heads, dimension) if layer == 0: positions = torch.arange(max(0, start - d.window), start, device=device) % d.window @@ -276,7 +302,7 @@ def normalize(x, count): replayed = torch.empty_like(actual) graph = torch.cuda.CUDAGraph() with session.capture(), torch.cuda.graph(graph): - replayed.copy_(operation(source, m)) + replayed.copy_(attend()) allocated = torch.cuda.memory_allocated() graph.replay() torch.cuda.synchronize() @@ -300,8 +326,9 @@ def normalize(x, count): graph.reset() +@pytest.mark.parametrize("draft_head", [None, "nvfp4"]) @torch.no_grad() -def test_five_layer_block_uses_learned_masks_and_replays_without_allocating(): +def test_five_layer_block_uses_learned_masks_and_replays_without_allocating(draft_head): device, hidden, heads, kv_heads, dimension, vocabulary, rank = torch.device("cuda:0"), 128, 12, 2, 128, 576, 256 generator = torch.Generator().manual_seed(341) def values(*shape): @@ -319,7 +346,10 @@ def ones(width): embedding_norm=ones(hidden), head=values(vocabulary, hidden), draft=SimpleNamespace(layers=layers, mask_embedding=values(hidden), norm=ones(hidden), recurrent=SimpleNamespace(embedding=values(vocabulary, rank), context_weight=values(3 * rank, 2 * hidden), state_weight=values(3 * rank, 2 * rank), - bias=values(3 * rank).float(), head=values(vocabulary, rank), head_bias=values(vocabulary), softcap=30.))) + bias=values(3 * rank).float(), head=values(vocabulary, rank), head_bias=values(vocabulary), softcap=30.), + draft_head=None)) + if draft_head is not None: + weights.draft.draft_head = nvfp4_draft_head(weights.head) regions = {f"draft{index}.{kind}": values(3, 1024, kv_heads, dimension) for index in (0, 1, 3, 4) for kind in ("key", "value")} regions["draft.rnn_context"] = values(3, hidden) @@ -347,6 +377,12 @@ def forward(): return features features = forward() assert features["hidden"].shape == (8, hidden) + if draft_head is not None: + # The proposal head reads its NVFP4 copy; four-bit weights keep the logits' direction. + reference = features["hidden"].double() @ weights.head.double().T + similarity = torch.nn.functional.cosine_similarity(output.double(), reference, dim=-1) + assert similarity.min() > .95 + assert not torch.equal(output, reference.bfloat16()) assert torch.equal(features["recurrent_context"], regions["draft.rnn_context"][1:2]) torch.cuda.synchronize() expected = output.clone() diff --git a/tests/components/test_ring_commit_cuda.py b/tests/components/test_ring_commit_cuda.py index aa89b793b..3cb4b4851 100644 --- a/tests/components/test_ring_commit_cuda.py +++ b/tests/components/test_ring_commit_cuda.py @@ -181,3 +181,13 @@ def forward(): worker.close() if graph is not None: graph.reset() + + +@pytest.mark.parametrize("candidate_offset", [-1, 1.5]) +def test_reject_candidates_before_the_staging(candidate_offset): + device = torch.device("cuda", 0) + backing = torch.zeros((3, 4096), dtype=torch.uint8, device=device) + rows, starts = (torch.ones(1, dtype=torch.int64, device=device) for _ in range(2)) + with pytest.raises(ValueError, match="nonnegative record index"): + PreparedRingCommit(backing, (0,), rows, starts, token_capacity=4, window_tokens=8, record_bytes=16, + candidate_offset=candidate_offset) diff --git a/tests/components/test_window_staging_cuda.py b/tests/components/test_window_staging_cuda.py index 4ace3f129..f9f73c692 100644 --- a/tests/components/test_window_staging_cuda.py +++ b/tests/components/test_window_staging_cuda.py @@ -104,3 +104,10 @@ def test_reject_unsupported_staging_views(failure): values = values[0].float(), values[1] with pytest.raises(ValueError): PreparedWindowStaging(*history, *values, *outputs, m, session=None, name='invalid') + + +@pytest.mark.parametrize('lookback', [0, 2049, 1023.]) +def test_reject_a_lookback_outside_the_history(lookback): + history, values, outputs, m, _ = _fixture(8, 128, 1) + with pytest.raises(ValueError, match='looks back'): + PreparedWindowStaging(*history, *values, *outputs, m, lookback=lookback, session=None, name='invalid') diff --git a/tests/recipes/test_inkling2_target.py b/tests/recipes/test_inkling2_target.py index fb0d9fee9..33a2fb0f1 100644 --- a/tests/recipes/test_inkling2_target.py +++ b/tests/recipes/test_inkling2_target.py @@ -162,48 +162,6 @@ def forward(): assert (((partial[:n].double() - exact).abs() - gamma * accumulated).max() <= 0).item() -@pytest.mark.skipif(os.environ.get('FLARE_RUN_CUDA_TESTS') != '1', reason='requires an assigned SM120 GPU') -def test_accepted_window_prefix_commits_replay_without_candidate_mutation(): - from ignition.recipes.inkling2.attention import retain_batch, stage_batch - - device = torch.device('cuda', 0) - m = _metadata([8, 8, 8], [0, 1150, 3499], device) - m.state_rows.copy_(torch.tensor([3, 1, 2], device=device)) - initial = torch.randn(5, 1152, 2, 128, device=device).to(torch.float8_e4m3fn) - rings = initial.clone().view(torch.uint8).view(5, -1) - values = torch.randn(24, 2, 128, dtype=torch.bfloat16, device=device) - staged, lengths = stage_batch(rings, values, m, window=1024, pages=9) - torch.testing.assert_close(rings, initial.view(torch.uint8).view(5, -1), rtol=0, atol=0) - assert lengths.tolist() == [8, 1031, 1031] - for lane, start in enumerate(m.starts.tolist()): - first = max(start - 1023, 0) - expected = torch.cat((initial[m.state_rows[lane], first % 1152:].bfloat16(), initial[m.state_rows[lane], :first % 1152].bfloat16())) - previous = expected[:start - first] - retained = torch.cat((previous, values[lane * 8:(lane + 1) * 8])).to(torch.float8_e4m3fn) - actual = staged.view(3, 1152, 2, 128)[lane] - torch.testing.assert_close(actual[:retained.shape[0]].view(torch.uint8), retained.view(torch.uint8), rtol=0, atol=0) - assert not actual[retained.shape[0]:].view(torch.uint8).any() - counts = torch.zeros(3, dtype=torch.int32, device=device) - retain_batch(rings, values, counts, m) - graph = torch.cuda.CUDAGraph() - with torch.cuda.graph(graph): - retain_batch(rings, values, counts, m) - for accepted in range(9): - rings.copy_(initial.view(torch.uint8).view(5, -1)) - selected = [accepted, (accepted + 1) % 9, 8 - accepted] - counts.copy_(torch.tensor(selected, dtype=torch.int32, device=device)) - before = torch.cuda.memory_stats()['allocation.all.allocated'] - graph.replay() - torch.cuda.synchronize() - assert torch.cuda.memory_stats()['allocation.all.allocated'] == before - expected = initial.clone() - for lane, count in enumerate(selected): - for index in range(count): - position = (int(m.starts[lane]) + index) % 1152 - expected[m.state_rows[lane], position] = values[lane * 8 + index].to(torch.float8_e4m3fn) - torch.testing.assert_close(rings, expected.view(torch.uint8).view(5, -1), rtol=0, atol=0) - - @pytest.mark.skipif(os.environ.get('FLARE_RUN_CUDA_TESTS') != '1', reason='requires an assigned SM120 GPU') @pytest.mark.parametrize('tokens,streams,sources,add', [(1, streams, sources, add) for streams in (1, 3) for sources in range(1, 10) for add in (False, True)] diff --git a/tests/recipes/test_inkling2_window.py b/tests/recipes/test_inkling2_window.py new file mode 100644 index 000000000..ccf1dfac4 --- /dev/null +++ b/tests/recipes/test_inkling2_window.py @@ -0,0 +1,231 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 Flare community contributors +# SPDX-License-Identifier: Apache-2.0 + +"""Inkling 2 verification stages sliding-window rings and commits accepted rows byte for byte. + +The references gather each request's whole ring, select in BF16 and rewrite the ring, +which states the staged pages and committed ring bytes independently of the kernels. +""" + +import os + +import pytest +import torch + +from ignition.components.paged_batch import Metadata +from ignition.recipes.inkling2.attention import accepted_records, staged_lengths + +KV, PAGE, WINDOW, RING, HEADS, DIM, WIDTH = torch.float8_e4m3fn, 128, 1024, 1152, 2, 128, 8 +PAGES = -(-(WINDOW - 1 + WIDTH) // PAGE) +RECORD = HEADS * DIM +cuda = pytest.mark.skipif(os.environ.get("FLARE_RUN_CUDA_TESTS") != "1", reason="requires an assigned SM120 GPU") + + +def reference_stage(rings, values, m): + """FP8 pages `[requests * PAGES, PAGE, heads, dim]` of each request's window and rows, and key counts.""" + rows, capacity = m.starts.numel(), PAGES * PAGE + ring = rings[m.state_rows].view(KV).view(rows, RING, *values.shape[1:]) + first = (m.starts - WINDOW + 1).clamp_min(0) + positions = first[:, None] + torch.arange(capacity, device=values.device) + previous = ring[torch.arange(rows, device=values.device)[:, None], positions.remainder(RING)].to(values.dtype) + index = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) + staged = torch.where((positions < m.starts[:, None])[:, :, None, None], previous, values[index]) + lengths = torch.where(m.lengths > 0, m.starts - first + m.lengths, 0).int() + staged = torch.where((positions < (m.starts + m.lengths)[:, None])[:, :, None, None], staged, 0) + return staged.clamp(-448, 448).to(KV).view(rows * PAGES, PAGE, *values.shape[1:]), lengths + + +def reference_retain(rings, values, counts, m): + """Rewrite the ring of every request that accepted rows, in place.""" + rows = m.starts.numel() + ring = rings[m.state_rows].view(KV).view(rows, RING, *values.shape[1:]) + first = m.starts + counts - RING + positions = first[:, None] + (torch.arange(RING, device=values.device) - first[:, None]).remainder(RING) + indices = (m.query_start_loc[:-1, None] + positions - m.starts[:, None]).clamp(0, values.shape[0] - 1) + current = values[indices].clamp(-448, 448) + keep = ((positions < m.starts[:, None]) | (positions < 0))[:, :, None, None] + retained = torch.where(keep, ring.to(values.dtype), current).to(KV).view(torch.uint8).view(rows, -1) + selected = counts > 0 + rings[m.state_rows[selected]] = retained[selected] + + +def metadata(lengths, starts, rows, device="cpu"): + m = Metadata.allocate(len(lengths) * WIDTH, len(lengths), 1, device, max_query_tokens=WIDTH) + update(m, lengths, starts, rows) + return m + + +def update(m, lengths, starts, rows): + device, count = m.starts.device, m.starts.numel() + m.set(torch.as_tensor(starts, device=device), torch.as_tensor(lengths, dtype=torch.int32, device=device), + torch.as_tensor(rows, device=device), torch.zeros(count, 1, dtype=torch.int32, device=device), + torch.zeros(count, WIDTH, dtype=torch.int64, device=device), torch.tensor([count], device=device)) + + +def candidates(tokens, generator, device="cpu"): + values = torch.randn(tokens, HEADS, DIM, generator=generator) * 4 + values.view(-1)[:6].copy_(torch.tensor([449., -449., 0., -0., 448., 1.0625])) + return values.bfloat16().to(device) + + +def test_staged_lengths_count_the_preceding_window_and_the_request_rows(): + m = metadata([8, 3, 0, 8, 1], [0, 5, 77, 1023, 5000], [1, 2, 0, 3, 4]) + assert staged_lengths(m, WINDOW).tolist() == [8, 8, 0, 1031, 1024] + values = candidates(m.positions.numel(), torch.Generator().manual_seed(1)) + rings = torch.zeros(5, RING * RECORD, dtype=torch.uint8) + assert torch.equal(staged_lengths(m, WINDOW), reference_stage(rings, values, m)[1]) + + +@pytest.mark.parametrize("starts", [[0, 1, 1144], [1151, 1152, 1153], [1016, 1023, 1024], [3499, 2297, 1048568]]) +def test_accepted_records_commit_to_the_bytes_of_a_rewritten_ring(starts): + generator = torch.Generator().manual_seed(sum(starts)) + lengths, rows = [8, 5, 8], [3, 1, 2] + m = metadata(lengths, starts, rows) + values = (candidates(m.positions.numel(), generator), candidates(m.positions.numel(), generator)) + initial = torch.randn(2, 5, RING, HEADS, DIM, generator=generator).to(KV).view(torch.uint8).view(2, 5, -1) + records = accepted_records(*values, m) + assert records.shape == (2, 3, WIDTH * RECORD) and records.dtype == torch.uint8 + for accepted in range(9): + counts = torch.tensor([min(accepted, 8), min((accepted + 1) % 9, 5), 8 - accepted], dtype=torch.int32) + for region in range(2): + expected = initial[region].clone() + reference_retain(expected, values[region], counts, m) + actual = initial[region].clone().view(5, RING, RECORD) + # The commit stores record `t` of a request at slot `(start + t) mod ring` for `t` below its count. + for request, count in enumerate(counts.tolist()): + for token in range(count): + actual[rows[request], (starts[request] + token) % RING] = records[region, request].view( + WIDTH, RECORD)[token] + assert torch.equal(actual.view(5, -1), expected) + + +def _layer(batch, device, generator): + """Continuation rows with K and V rings at distinct offsets, and the layer's prepared operations.""" + from ignition.components.ring_commit import PreparedRingCommit + from ignition.components.window_staging import PreparedWindowStaging + + offsets = (256, 256 + RING * RECORD + 768) + row_bytes = offsets[1] + RING * RECORD + 256 + backing = torch.randint(0, 256, (batch + 2, row_bytes), dtype=torch.uint8, generator=generator).to(device) + for offset in offsets: + ring = torch.randn(batch + 2, RING, HEADS, DIM, generator=generator).to(KV).view(torch.uint8) + backing[:, offset:offset + RING * RECORD].copy_(ring.view(batch + 2, -1).to(device)) + rings = tuple(backing[:, offset:offset + RING * RECORD] for offset in offsets) + m = metadata([WIDTH] * batch, [0] * batch, list(range(1, batch + 1)), device) + tokens = batch * WIDTH + key, value = (torch.zeros(tokens, HEADS, DIM, dtype=torch.bfloat16, device=device) for _ in range(2)) + staged = tuple(torch.zeros(batch * PAGES, PAGE, HEADS, DIM, dtype=KV, device=device) for _ in range(2)) + + def build(session): + shape = (-1, RING, HEADS, DIM) + staging = PreparedWindowStaging(rings[0].view(KV).view(shape), rings[1].view(KV).view(shape), key, value, + *staged, m, lookback=WINDOW - 1, session=session, name="staging").functional(session=session) + commit = PreparedRingCommit(backing, offsets, m.state_rows, m.starts, token_capacity=WIDTH, + window_tokens=RING, record_bytes=RECORD, candidate_offset=0).functional() + return staging, commit + + return backing, rings, m, key, value, build + + +def _cases(batch, generator): + """Request starts, row counts and state rows at sequence start, window and ring boundaries.""" + boundaries = [0, 1, 7, 1015, 1016, 1023, 1024, 1025, 1143, 1144, 1151, 1152, 1153, 2297, 3499, 1048568] + for shift in range(4): + starts = [boundaries[(shift * 5 + 3 * lane) % len(boundaries)] for lane in range(batch)] + lengths = [int(torch.randint(0 if shift else 1, WIDTH + 1, (1,), generator=generator)) for _ in range(batch)] + if shift == 0: + lengths = [WIDTH] * batch + rows = torch.randperm(batch, generator=generator).add(1).tolist() + yield starts, lengths, rows + + +@cuda +@pytest.mark.parametrize("batch", [1, 8, 32]) +@torch.no_grad() +def test_verification_staging_and_commit_match_the_whole_ring_references(batch): + from ignition.preparation.tuning import RecordedPreparationSession + + device, generator = torch.device("cuda", 0), torch.Generator().manual_seed(600 + batch) + backing, rings, m, key, value, build = _layer(batch, device, generator) + counts = torch.zeros(batch, dtype=torch.int32, device=device) + observed = tuple(torch.zeros(batch * PAGES, PAGE, HEADS, DIM, dtype=KV, device=device) for _ in range(2)) + with RecordedPreparationSession(device=device, checkpoint="inkling2-window-verification", + workload_schema="window-verification", numerical_policy="exact", compile_workers=1) as session: + staging, commit = build(session) + session.tune(show_progress=False) + + def forward(): + staged = staging(key, value, metadata=m) + for output, pages in zip(observed, staged): + output.copy_(pages) + commit(counts, staging=accepted_records(key, value, m), rows=m.state_rows, starts=m.starts) + + forward() + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + forward() + for starts, lengths, rows in _cases(batch, generator): + for accepted in ("none", "all", "depth", "random"): + update(m, lengths, starts, rows) + key.copy_(candidates(batch * WIDTH, generator, device)) + value.copy_(candidates(batch * WIDTH, generator, device)) + limit = torch.tensor(lengths, dtype=torch.int32) + selected = {"none": torch.zeros_like(limit), "all": limit, + "depth": torch.minimum(limit, torch.arange(batch, dtype=torch.int32) % (WIDTH + 1)), + "random": (torch.rand(batch, generator=generator) * (limit + 1)).int()}[accepted] + counts.copy_(selected.to(device)) + before, expected = backing.clone(), backing.clone() + references = [] + for offset, values in zip((256, 256 + RING * RECORD + 768), (key, value)): + region = expected[:, offset:offset + RING * RECORD] + references.append(reference_stage(region, values, m)) + reference_retain(region, values, counts, m) + allocated = torch.cuda.memory_stats()["allocation.all.allocated"] + graph.replay() + torch.cuda.synchronize() + assert torch.cuda.memory_stats()["allocation.all.allocated"] == allocated + live = (m.lengths > 0).repeat_interleave(PAGES) + for output, (pages, counted) in zip(observed, references): + assert torch.equal(output.view(torch.uint8)[live], pages.view(torch.uint8)[live]) + # A request without rows stages no keys; attention reads none of its pages. + assert not output.view(torch.uint8)[~live].any() + assert torch.equal(staged_lengths(m, WINDOW), counted) + assert torch.equal(backing, expected) + if accepted == "none": + assert torch.equal(backing, before) + graph.reset() + + +@cuda +@torch.no_grad() +def test_successive_rounds_keep_rings_identical_across_ring_wraps(): + from ignition.preparation.tuning import RecordedPreparationSession + + batch = 4 + device, generator = torch.device("cuda", 0), torch.Generator().manual_seed(77) + backing, rings, m, key, value, build = _layer(batch, device, generator) + counts = torch.zeros(batch, dtype=torch.int32, device=device) + with RecordedPreparationSession(device=device, checkpoint="inkling2-window-rounds", + workload_schema="window-verification", numerical_policy="exact", compile_workers=1) as session: + staging, commit = build(session) + session.tune(show_progress=False) + expected = backing.clone() + starts, rows = [0, 1000, 1140, 2300], [4, 2, 1, 3] + for _ in range(500): + lengths = torch.randint(1, WIDTH + 1, (batch,), generator=generator).tolist() + update(m, lengths, starts, rows) + key.copy_(candidates(batch * WIDTH, generator, device)) + value.copy_(candidates(batch * WIDTH, generator, device)) + accepted = [int(torch.randint(0, length + 1, (1,), generator=generator)) for length in lengths] + counts.copy_(torch.tensor(accepted, dtype=torch.int32)) + staged = staging(key, value, metadata=m) + for offset, values, pages in zip((256, 256 + RING * RECORD + 768), (key, value), staged): + region = expected[:, offset:offset + RING * RECORD] + assert torch.equal(pages.view(torch.uint8), reference_stage(region, values, m)[0].view(torch.uint8)) + reference_retain(region, values, counts, m) + commit(counts, staging=accepted_records(key, value, m), rows=m.state_rows, starts=m.starts) + assert torch.equal(backing, expected) + starts = [start + count for start, count in zip(starts, accepted)] + # Every request has crossed its window; three have wrapped their rings. + assert starts[0] > WINDOW and min(starts[1:]) > RING