You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement the SubLayerNorm normalization used in BitNet ar5iv.org.
This is a variant of layer normalization applied at the sub-layer level to stabilize training, and in inference it will normalize the vectors before certain operations. SubLN likely operates similarly to LayerNorm or RMSNorm (as indicated by an epsilon in config), normalizing each token’s hidden state. Implement it by computing the mean (or RMS) and variance as needed across the feature dimension and then scaling. Include a learnable scale parameter (gamma) for each normalized vector dimension, but no bias offset ar5iv.org.
Use the epsilon value from the config (e.g. 1e-5) for numerical stability. This SubLN will be used before the attention and feed-forward sublayers (since BitNet uses a pre-normalization approach like LLaMA). Test the normalization on dummy data (compare with standard LayerNorm outputs for sanity).
Activity
feat(bitnet): implement Sub-Layer Normalization (SubLN) for #179
feat(scripts): add task and PR detection scripts
chore(scripts): update PR description template
chore(scripts): add task management and preview scripts
feat(bitnet): implement Sub-Layer Normalization (SubLN) for #179 (#207)