Repository navigation
lfm3: the backends share the rope frequencies and the layer schedule - #1839
Merged
Merged
Conversation
Rotary.inverseFrequencies() builds RoPE's inverse frequencies as the CUDA backend does: theta to the 2i/128 in float64, rounded to float32, one over it in float32, zero past the rotated pairs. A test pins both of LFM3's settings (theta 1e6 partial 0.5, theta 1e4 partial 1) to torch 2.14's float32 rows bit for bit; a float32 pow is off by 1 to 3 ulps in 16 of 32 and 28 of 64 entries. count() gives the layers of a kind in the admitted schedule, LayerType.window() the keys a layer's attention reads back (the sliding window, or 0 for all of them), and ring() the keys a sliding ring holds when a pass writes its rows before the first of them reads its window. A test checks 13 conv, 6 full and 5 sliding layers and both sizes.
Owner
|
Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LFM3's backends each kept their own copy of three host-side facts from the family config. This moves them into
families/lfm3/config.zig, so every backend reads the same numbers:Rotary.inverseFrequencies(): RoPE's inverse frequencies, computed as theta^(2i/128) in float64, rounded to float32, inverted in float32, and zero past the rotated pairs. A test pins both of LFM3's settings to torch 2.14's float32 rows, bit for bit: theta 1e6 with partial 0.5 for the full layers, and theta 1e4 with partial 1 for the sliding layers. A float32 pow, which the Intel GPU copy used, is 1–3 ulp off in 16 of 32 and 28 of 64 entries.count(kind): the number of layers of a kind in the admitted schedule (13 conv, 6 full, 5 sliding).LayerType.window(): the keys a layer's attention reads back (256 for a sliding layer, 0 meaning all).ring(rows): the keys a sliding ring holds when a pass writes its rows before the first of them reads its window.There are no users on main yet. Two of our open PRs are the first:
Checks on this head: lean has no new findings;
zig build test, the contract, golden, security and fuzz steps pass;git diff --checkis clean.