Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/TODO.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
host memory admission. Its nine-worker assertion requires 296 GiB available
under the 32 GiB worker allowance and 8 GiB headroom; smaller hosts admit fewer.

[ ] Extend DeepSeek V4.1 row-sharded-attention validation beyond the retained
SM121 TP4 native profile: fresh default-dependency whole-model preparation,
other supported GPU/TP geometries, scored-prefill modes, and remaining
shard-tail capacities. Use operator-specific bounds and complete state checks.
See docs/model-preparation.md for the qualified execution scope.

[ ] Ensure Rust bridge updates relink the WinUI executable during incremental
builds. MSBuild can retain flare-winui.exe after the bridge library changes;
verify loose and single-file outputs without requiring /t:Rebuild.
Expand Down Expand Up @@ -479,3 +485,12 @@
[ ] Publish Debian packages with releases and update the APT repository from
them. release.yml builds no .deb, and no tracked tool builds or signs an
APT index. See debian.md and release.md.

[ ] Qualify row-sharded prefill at three ranks. The DeepSeek V4.1 recipe's
4,096-row chunk has no per-rank block at three ranks, though a 6,144-row
chunk would. Close when an export and native serve with --row-shard-prefill
complete at three ranks.

[ ] Qualify row-sharded prefill on SM103, where its projections and experts run
a different kernel path from SM121. Close when an export and native serve
with --row-shard-prefill complete on SM103.
2 changes: 2 additions & 0 deletions docs/cli-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -461,6 +461,8 @@ explicit flags take precedence.
| `--autotune-effort off\|default\|audit` | Kernel search effort. Off uses heuristics without races; default races the candidates that pass each kernel family's performance exclusions; audit lifts those exclusions and races every legal candidate, to check that the default search prunes no winners. Overrides `[preparation] autotune_effort`. | Default |
| `--joint-tuning` | Jointly select DeepSeek V4.1 QKV projection, query normalization, and query expansion plans by timing complete prepared layers across every prefill, decode, packed verification, and draft shape. Requires MXFP8 query normalization and autotuning enabled. Command line only. | off |
| `--shard-hyperconnections` | Partition Qwen3.8 Flash target and MTP hyperconnection bottlenecks across TP ranks and all-reduce BF16 gate logits before sigmoid. Requires TP2 or TP4. Command line only; changing it requires re-export. | off (replicated) |
| `--row-shard-prefill` | Replace each DeepSeek V4.1 prefill tensor-parallel all-reduce with a reduce-scatter, run the hyperconnection on the rank's contiguous row shard, and gather the normalized branch input back to every rank. The collective pair preserves the pipelined two-shot reduction's arithmetic; whole-decoder numerical equivalence remains unqualified. Preparation requires the runtime's ABI 11 or later row-shard surfaces; the proxy pinned in this tree records ABI 14. Requires TP2 or more; three ranks and SM103 are refused, and a layer whose row count or pipe capacity does not fit keeps the all-reduce path. Command line only; changing it requires re-export. | off |
| `--row-shard-attention` | Also run DeepSeek V4.1 prefill attention on the rank's row shard: every rank loads all 64 attention heads and all 8 output groups for the layers a prepared prefill runs on the shard, which the plan charges as resident weights (17 layers, 2.12 GB per rank at TP4 by default; 37, 4.62 GB with `--prompt-logprobs`; 35, 4.37 GB for scored `--mtp-only`), gathers the KV latent and the compressed-latent projections so its caches hold every row, and needs no attention reduction. Changes reduction rounding relative to the all-reduce path; see the [DeepSeek V4.1 preparation guide](model-preparation.md#deepseek-v41-flash). Requires `--row-shard-prefill`; a layer whose gathers do not fit keeps the partitioned path. Command line only; changing it requires re-export. | off |
| `--collective-ladder PATH` | JSON file of final joint collective geometries and runtime profiles. Required when the resolved collective tuning policy is `runtime`; rejected with `static`. See [offline ladders](../ignition/preparation/ladders/README.md). | None |
| `--collective-tuning static\|runtime` | Static autotunes collectives during preparation; runtime uses the required ladder file for final compiled geometry and captures runtime submode/active-CTA selection for eligible BF16 PCIe reductions. Overrides `[preparation] collective_tuning`, then the recipe declaration. | Static |
| `--lock-gpu-clocks` | Lock each tuning GPU at its maximum supported graphics frequency during b12x and Inductor autotuning, then reset driver clock defaults. Frozen warmup, capture, and serving remain unlocked. Requires root or noninteractive sudo access to `nvidia-smi`. Command line only; inactive with `--autotune-effort off`. | off |
Expand Down
50 changes: 50 additions & 0 deletions docs/model-preparation.md
Original file line number Diff line number Diff line change
Expand Up @@ -599,6 +599,56 @@ seams determine the measured contexts. The option defaults to off and requires
MXFP8 query normalization on the selected GPUs and `--autotune-effort default`
or `audit`. It adds preparation time without changing the serving graph catalog.

`ignition export deepseek-v4.1-flash --row-shard-prefill --row-shard-attention`
computes prefill attention on the row shard the reduced prefill already leaves on
each rank. At TP4, a layer that a prepared prefill runs on the shard loads all 64
attention heads and all 8 output groups of its query and output projections
beside the rank's partitioned 16 heads and 2 groups. Decode, the 128-row suffix
layers, and layers without sharded prefill use the partitioned projections.
Packed prefill stops at the decoder boundary, so 17 of the 40 backbone layers
hold replicated projections and attention sinks, totaling 2.12 GB per rank: 1.59
GB outside that rank's partition and 0.53 GB for an additional copy of its
partition. Each replicated layer retains 124,780,800 bytes: 112 MiB of MXFP8
values, two 3.5 MiB scale representations, and 256 bytes of FP32 sinks. Scored
prefill replays no suffix and reaches every layer below the last, so
`--prompt-logprobs` raises the totals to 37 layers and 4.62 GB: 3.46 GB outside
the partition and 1.15 GB duplicating it. A draft-only plan builds no prefill
graph without the draft, and those graphs are what keep a draft target layer on
full rows, so the scored plan under `--mtp-only` replicates 35 layers and 4.37
GB. The plan's per-rank weight estimate charges the same layers, so the page
budget it sizes leaves room for them instead of spending it on KV pages. The
weight estimate determines planned page, context, and snapshot capacities;
changing those capacities requires re-export. Existing cores retain their
captured storage and capacities. Each rank gathers the 512-wide KV latent plus
the two compressed-latent projection inputs before the full-row window,
index, and compressed-latent cache writes. This preserves full-row cache
coverage; numerical comparisons use the operator-specific bounds described below. The
attention output of a rank's own query rows is complete once
the grouped output projection has summed all eight groups, so no attention
reduction runs at all. That sum happens in one kernel here, where the
partitioned path sums four rank partials through an FP32 all-reduce, so prefill
logits are expected to agree with the all-reduce path to BF16 rounding rather
than exactly; comparisons require operator-specific rounding bounds. The option requires `--row-shard-prefill`, is
recorded in the plan and the core's rebuild metadata, and changes prepared weight
layouts, so changing it requires a fresh export and `--reuse-weights` requires
matching layouts; `--prompt-logprobs` and `--mtp-only` select the replicated
layers too and carry the same requirements. A prefill capacity whose shard rows
or gathers do not fit keeps the partitioned attention path for that graph and
prints one preparation line naming the capacity. The replicated weights load for
the layers a prefill kind reaches, not for the capacities that shard, so a
capacity that keeps the partitioned path leaves them resident; disabling the
option and re-exporting is the only way to recover that memory. Row-sharded
attention has compiled collective and exported-core native validation on four
SM121 GB10 hosts with the retained FlashInfer `7b35510b`/ABI-14 cores. Checks
cover partial prefill, exact checkpoint/Restore and inactive-state preservation,
replay, scratch reuse, and retained-memory/allocation stability. GPU component
and compiler-boundary checks also pass with the pinned FlashInfer `4fd836dc`;
a whole-model core prepared with that revision has not been exported for these
checks. The grouped output projection satisfies the tested operator rounding
budget; floating cache differences and baseline replay variation remain visible.
These results do not qualify other hardware/TP configurations, scored-prefill
modes, or every captured shape; follow-up checks are tracked in docs/TODO.txt.

`--speculative-token-capacity` bounds physical target rows;
`--speculative-graph-budget` independently controls the number of packed rounds.
For each power-of-two request bucket `B`, the catalog retains capacities
Expand Down
4 changes: 4 additions & 0 deletions helios/proto/runtime_core.proto
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ message RebuildManifest {
string collective_ladder_json = 26;
// Preparation-only routed W4A16 decode/verification with calibrated A4 prefill.
bool force_moe_decode_a16 = 27;
// Preparation-only row-sharded prefill collectives; requires proxy ABI 11 or later.
bool row_shard_prefill = 28;
// Preparation-only row-sharded prefill attention with replicated attention weights.
bool row_shard_attention = 29;
// Preparation-only online conversion group; omitted means all eligible projections.
string online_mxfp8_scope = 30;
// Preparation context group size; zero/absent retains full-TP context sharding when enabled.
Expand Down
8 changes: 8 additions & 0 deletions ignition/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,14 @@ def _preparation_arguments(running, *, model_default="defaults.model"):
advanced.add_argument("--joint-tuning", action="store_true",
help="jointly tune DeepSeek V4.1 attention projection plans across the prepared"
" prefill, decode, and speculative shapes (default: off)")
advanced.add_argument("--row-shard-prefill", action="store_true",
help="reduce DeepSeek V4.1 prefill into one contiguous row shard per rank, run the"
" hyperconnection on that shard, and gather the branch input every row reads"
" (default: tensor-parallel all-reduce)")
advanced.add_argument("--row-shard-attention", action="store_true",
help="also run DeepSeek V4.1 prefill attention on that row shard, replicating every"
" head and output group on every rank and gathering the KV latent instead of"
" the attention output; requires --row-shard-prefill (default: off)")
advanced.add_argument("--lock-gpu-clocks", action="store_true",
help="lock each tuning GPU to its maximum supported graphics clock during"
" autotuning; requires root or noninteractive sudo for nvidia-smi")
Expand Down
1 change: 1 addition & 0 deletions ignition/cli/bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ def run(args):
lock_gpu_clocks=args.lock_gpu_clocks, native_dense_activations=args.native_dense_activations,
force_moe_decode_a16=args.force_moe_decode_a16,
shard_hyperconnections=args.shard_hyperconnections, joint_tuning=args.joint_tuning,
row_shard_prefill=args.row_shard_prefill, row_shard_attention=args.row_shard_attention,
image_concurrency=args.image_concurrency, mixed_prefill=args.mixed_prefill,
prompt_logprobs=args.prompt_logprobs, expert_scale_compression=args.expert_scale_compression,
online_mxfp8=args.online_mxfp8, online_mxfp8_scope=args.online_mxfp8_scope,
Expand Down
6 changes: 4 additions & 2 deletions ignition/cli/preparation.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ def derive(configuration: Configuration, model: ResolvedModel, host: Host, *, gp
compile_workers: int | None = None, tuning_gpus: str | None = None, autotune_effort: str | None = None, collective_tuning: str | None = None, collective_ladder=None,
image_concurrency: int | None = None, mla_compute_mode: str | None = None,
lock_gpu_clocks: bool = False, native_dense_activations: bool = False,
joint_tuning: bool = False, shard_hyperconnections: bool = False,
joint_tuning: bool = False, shard_hyperconnections: bool = False, row_shard_prefill: bool = False,
row_shard_attention: bool = False,
force_moe_decode_a16: bool | None = None) -> planning.Plan:
"""Select GPUs and estimate memory for one checkpoint from the configuration and command-line intent."""
from ignition.preparation.collective_ladder import resolve as resolve_ladder
Expand All @@ -66,7 +67,8 @@ def derive(configuration: Configuration, model: ResolvedModel, host: Host, *, gp
stream_overlap=stream_overlap, roce_ring=roce_ring, roce_chain=roce_chain,
mixed_prefill=mixed_prefill, prompt_logprobs=prompt_logprobs, expert_scale_compression=expert_scale_compression,
image_concurrency=image_capacity_setting(configuration, model, image_concurrency),
online_mxfp8=online_mxfp8, online_mxfp8_scope=online_mxfp8_scope, language_model_only=language_model_only, mla_compute_mode=mla_compute_mode)
online_mxfp8=online_mxfp8, online_mxfp8_scope=online_mxfp8_scope, language_model_only=language_model_only, mla_compute_mode=mla_compute_mode,
row_shard_prefill=row_shard_prefill, row_shard_attention=row_shard_attention)
result = replace(result, compile_workers=workers, hub_cache=configuration.directory("hub_cache"),
autotune_effort=tuning_effort(configuration, autotune_effort),
collective_tuning=mode, collective_ladder=ladder,
Expand Down
7 changes: 5 additions & 2 deletions ignition/cli/serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,8 @@ def _export_hosts(args):
expert_scale_compression=args.expert_scale_compression,
online_mxfp8=args.online_mxfp8, online_mxfp8_scope=args.online_mxfp8_scope,
language_model_only=args.language_model_only,
mla_compute_mode=args.mla_compute_mode)
mla_compute_mode=args.mla_compute_mode,
row_shard_prefill=args.row_shard_prefill, row_shard_attention=args.row_shard_attention)
collective_mode = collective_tuning_mode(configuration, model, args.collective_tuning)
plan = replace(plan, compile_workers=compiler_workers(configuration, args.compile_workers), hub_cache=hub_cache,
autotune_effort=tuning_effort(configuration, args.autotune_effort),
Expand Down Expand Up @@ -277,6 +278,7 @@ def __call__(self, message):
lock_gpu_clocks=args.lock_gpu_clocks, native_dense_activations=args.native_dense_activations,
force_moe_decode_a16=configuration.get("preparation.force_moe_decode_a16", args.force_moe_decode_a16).value,
shard_hyperconnections=args.shard_hyperconnections, joint_tuning=args.joint_tuning,
row_shard_prefill=args.row_shard_prefill, row_shard_attention=args.row_shard_attention,
image_concurrency=args.image_concurrency,
speculation=speculation(args, model, configuration), prefill_chunk=args.prefill_chunk_size,
context_parallel=context_parallel(args, model), context_parallel_size=getattr(args, "dcp_size", None),
Expand Down Expand Up @@ -467,7 +469,8 @@ def debug_serve(args) -> int:
cache_states=args.prefix_cache_states, mixed_prefill=args.mixed_prefill, prompt_logprobs=args.prompt_logprobs,
expert_scale_compression=args.expert_scale_compression, online_mxfp8=args.online_mxfp8,
online_mxfp8_scope=args.online_mxfp8_scope,
language_model_only=args.language_model_only, mla_compute_mode=args.mla_compute_mode)
language_model_only=args.language_model_only, mla_compute_mode=args.mla_compute_mode,
row_shard_prefill=args.row_shard_prefill, row_shard_attention=args.row_shard_attention)
collective_mode = collective_tuning_mode(configuration, model, args.collective_tuning)
plan = replace(plan, compile_workers=compiler_workers(configuration, args.compile_workers), hub_cache=hub_cache,
autotune_effort=tuning_effort(configuration, args.autotune_effort),
Expand Down
Loading