Repository navigation
lfm2: a cheaper DSpark hold (one concurrent encoder) - #1394
BlivionIaG wants to merge 14 commits into
Conversation
ashhart
left a comment
ashhart
left a comment
There was a problem hiding this comment.
Thanks @BlivionIaG. This builds on #1391, so it inherits that review's blockers (runtime lossy BF16 conversion, the 4-bit buffer mismatch, dropped async failure flags, the contract gaps). Its concurrent encoder and shared stage flush also need device race, resume and replacement-equality checks.
|
Thanks @ashhart. Device checks on an M4 Pro at the stack (details in the body, head |
|
An M3 receipt for this head, since the body says no run covers M1, M2 or M3. M3 Pro (36 GB), macOS 26.7.1, Zig 0.17.0, head
|
Thanks for the run, now we have proof it works |
|
Thanks @BlivionIaG, and thanks to @jschmied for the M3 Pro receipt. The concurrent hold encoder puts a barrier after This PR's own changes look ready. It is blocked only by what it inherits from #1391 and #1392: the guards, the |
|
An M1 receipt for this head, for the load-time memory the body lists as not measured. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, head 2e5966f: zig build native -Dcpu=apple_m1: built. |
|
An M1 receipt for this head: the load-time memory the body lists as not measured, and a slow-prompt finding on M1 that main has too. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, Load-time memory (head
Cold prompts are slow on M1, on this head and on main
|
|
Thanks @Arminova for the M1 Pro receipt. The load peak (13.63 GiB without a drafter) is the number the body was missing. The slow cold prompts show on main too, so this stack didn't cause them. The long prompt that ignored SIGTERM looks like the same whole-prompt pass, which has no cancel point on main either. We'll look at it with #1377. @BlivionIaG, nothing has been pushed to #1390-#1394 since our last comments, so where things stand is unchanged. This PR's own changes are fine. What still blocks the stack:
Once that's in, we'll run these on the stack head: drafted against plain and concurrent against solo at 1 to 16 streams, the long-context cells, and alternated speed rounds against main. |
2e5966f to
017919f
Compare
|
@ashhart Following up on your last checklist: the cumulative fixes are now pushed here, at
Receipts, reproducer and limitations. The final allocation-only follow-up removes duplicate probability and held-grammar buffers without changing arithmetic or decoding policy. Its shared-lane/native checks pass (341 tests, three skipped), and the separate 64-cell M5 before/after comparison completed with matching token hashes and throughput changes between -0.45% and +0.68%. That comparison is additional evidence, not a repeat of the full hardware suite on the final commit. All nine CI checks are green on |
… holds follow the grants A drafter that gives each held chain draft's chance (Drafter.VTable.chances) may set Facts.chances. Drafted.facts then sets Model.draft_probabilities and Model.granted_depth, so the window allocator prices each stream's verify rows by those chances, as it does a head's node chances. With Facts.confidence_floor it instead verifies the drafts at least that likely (a fixed window when 0). With granted_depth a stream's next hold is as deep as its last grant plus a slack: one more draft when the grant took every held draft, none when it stopped short of them (Stream.grant_short), and after a grant of none a probe of draft_slack drafts at the depth rule's probe interval. Facts.slack sets one slack for all three. A shared round's granted depths are cut so the drafting streams' blocks fit one hold pass of batch_rows. When the shared-round guard starts a drafted stretch, its first round grants every stream one draft, so the stretch measures drafted rounds rather than plain ones left by grants of none. All of it is behind Facts.chances and Model.granted_depth, off by default; windows.allocate records grant_short only for granted depths. Tests: chances price the chain's rows and drafted replies equal one-token decode alone and at 1, 4 and 8 shared streams, with the scheduled verify ragged across streams and verifying fewer rows than a fixed window; the facts reach the lane config; granted depths, their slacks and the one-pass cut; holds as wide as the grants.
…py, and holds kept only while they pay The depth rule asks one head draft for a stream with a copy ahead, and a shared round's granted holds were priced only by their verify rows. A drafter whose hold runs as a pass of its own (Facts.copy_first) pays that pass for the one draft, and the copy round then drops it; Model.copy_first now asks it for none. A drafter may also give its hold pass's timed cost by blocks and depth (Facts.hold_costs, priced by hold_price.zig). With granted depths a shared round then keeps granted holds, most expected drafts a row first, only while each raises the next round's expected tokens a ms over its verify rows and the pass; a probe after a grant of none keeps its hold. A stream's last grant records the drafts it expected to keep (Stream.grant_gain). Both are off by default (copy_first false, hold_costs empty). Tests: with copy_first no hold is asked for a copy round and replies equal one-token decode at 1 and 3 streams; a hold's cost between and past its timed passes; granted holds kept while they pay, none for a dear pass, and a probe's hold kept.
Bound the reused ordering buffer by batch_rows, and cover reuse with an allocator that refuses further allocations. Interpolate between the measured depths surrounding the requested depth, including nearest-end extrapolation. Query a stateful copy proposer once for a granted-depth budget decision.
The lane core can put an external drafter in front of a target. The drafter reads the target's hidden rows after some layers (its taps). LFM2-MoE had no taps, so it could draft only with the core's own heads. prepare_features names the layers. From then on, each forward copies each row's output of those layers into the scratch (lfm2_tap, a plain copy of bf16 values). features() gives a stream's rows of the latest verify or one-token round. prefill_features hands each prompt chunk's rows to the drafter before the chunk's end is kept for reuse. The prompt's last row is kept beside the stream's caches for its first draft. A drafting stream resumes only a kept state that also holds its drafter's context (Snap.draft). Without a drafter nothing changes: no taps are kept and every forward runs as before. lfm2_tap registers with the kernel contract (metal_lfm2_tap.zig): the third of five taps' columns at 1, 10, 32 and 2,048 rows must hold the layer's bits and the other columns what the call found. Gate 5 checks its launch against every Apple chip we serve and this GPU. test-lfm2-dspark checks the same copy through the family's own kernels.
…F16 documentation
98f5cc7 to
f9de08d
Compare
|
@ashhart The intermediate branches are now rebuilt too, rather than leaving fixes only at the cumulative tip. All are on current main
No M5 was accessed for this restack. New M4 Pro / 48 GiB checks of the final integration:
Reproduction, raw hashes/results, actual load peaks and limitations. CI is running on the new heads; these local results do not stand in for pending CI or your review. The previously measured 1–2% slowdown at four/eight streams is still open against your performance requirement. Restacking does not fix or waive it, and the task owner’s accepted tradeoff is not presented as your approval. No new performance comparison or 65k/130k long-context run was made in this M4 replay; earlier receipts keep their original source scope. Could you re-review the propagated correctness/base fixes, with that remaining performance item kept explicit? |
Runs shared DSpark holds in one concurrent encoder, shares staged absorbs, measures shared verification increments, and reuses final lane probability/grammar buffers.
Rebuilt on main
d33dcfc767ee093393c6bfa00b3949bd7b4d8e88. The rejected #1379/#1382 guard commits are excluded. No M5 was accessed for this restack. Fresh CI is running; maintainer re-review remains required.Scope and review fixes
Includes the clean #1390–#1393 chain and #1381. All Zig/build source files are byte-identical to the previously tested integration
f73bb5590(public parents: prior #1394 head98f5cc72band maind33dcfc76); the rebase changes history and receipt placement, not arithmetic or policy. Current main’s unrelated contracts are retained.Validation
Code
f9de08d58; final receipt head87d1fbbd0. 73/73 scoped build steps, 355 tests passed and three skipped; native/reference/contract tools compile. Lean has zero new findings. The new local M4 replay passes 11 kernels / all 17 controls red, 9/9 FP32-reference picks, zero-allocation steady rounds, cache/resume/fresh equality, cancellation/replacement at 6/16 slots (zero wrong/errors), 29 sampled cases and eight 8-bit equality cells. M4 scope and full results. It uses context 4096; no new long-context or performance qualification is claimed. The default build passes 239/239 steps. Prior same-source M5 integration report has 72 throughput cells, six retention batches and matching tokens. Earlier long-context/cancel/cache receipts retain their original source identifiers.Remaining performance requirement
The existing M5 integration loses about 1–2% at four/eight streams while gaining about 7–9% at 16. This restack does not claim to fix or waive that tradeoff, and Ash has not approved an exception to the no-slowdown requirement. The accepted task-owner tradeoff is not presented as maintainer approval. No stream-count/context guard is introduced.
Part of #1374. The chain’s intermediate branches are now updated rather than leaving fixes only in the cumulative tip.