You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Develop a core linear algebra routine for BitNet’s binary-weight linear layer (BitLinear). This function takes an input vector (or matrix) of 8-bit activations and multiplies it by a weight matrix with 1.58-bit (ternary) weights to produce an 8-bit output ar5iv.org.
To do this efficiently, use bitwise operations: unpack the weight matrix bits (as prepared in step 3) on the fly, interpret 2-bit codes as -1, 0, or +1, and compute the dot-products. Exploit binary arithmetic where possible (e.g., treat -1 as +1 with a sign bit to use XNOR or bit-count tricks if inputs were binary, though here inputs are 8-bit). Accumulate results in a higher precision type (e.g. int32) then quantize/clamp the output back to int8 range. No bias terms should be added in these projections ar5iv.org.
Make sure the BitLinear function is general and can handle weight matrices of various shapes (it will be used for Q, K, V projections, feed-forward layers, etc.). Optimize inner loops with bit-level operations and verify correctness on small test matrices.
Activity
174 implement tokenizer llama3 bpe (#199)
200 improve pr messages (#201)
feat(bitnet): implement token embedding layer (#202)
176 set model constants architecture hyperparameters (#203)
feat(tensor): add BitLinear layer and tests (#178)
feat(tensor): implement model weights and ternary weights benchmarks
feat(tensor): add BitLinear layer and tests (#178) (#206)
fix(benchmarks): update benchmark scripts and gitignore