You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Design Go structures to hold the model parameters (e.g. a struct for each Transformer block containing its weight tensors). Implement a loader that reads the embedded weight file and populates these structures. This will involve unpacking the 2-bit ternary weights (-1 → 00, 0 → 01, +1 → 10; 11 unused): the model uses ternary weight values {-1, 0, +1}, which are packed as four 2-bit values per byte ar5iv.org.
Decode each byte by extracting 2-bit segments and mapping them to -1/0/+1 (using the quantization scheme from training that maps weights to ternary values ar5iv.org
).
Verify that the shapes of the loaded weight matrices match the expected dimensions of the model layers.
Loader must also read the per-matrix abs-mean float32 scale that accompanies every packed tensor and store it alongside the weights for later de-quantization.
Activity
Address all review comments for issue #173
Address all review comments for issue #173
Address all review comments for issue #173
feat(bitnet): implement model structures and weight loading for issue #…
feat(bitnet): implement model structures and weight loading for issue #…
fix(bitnet): address PR review feedback and align ternary weights tes…
Address all review comments for issue #173
feat(bitnet): implement core model structures and weight loading (#198)