Skip to content

Implement Sub-Layer Normalization (SubLN) #179

Closed

Description

@thejhh

Implement the SubLayerNorm normalization used in BitNet
ar5iv.org.

This is a variant of layer normalization applied at the sub-layer level to stabilize training, and in inference it will normalize the vectors before certain operations. SubLN likely operates similarly to LayerNorm or RMSNorm (as indicated by an epsilon in config), normalizing each token’s hidden state. Implement it by computing the mean (or RMS) and variance as needed across the feature dimension and then scaling. Include a learnable scale parameter (gamma) for each normalized vector dimension, but no bias offset
ar5iv.org.

Use the epsilon value from the config (e.g. 1e-5) for numerical stability. This SubLN will be used before the attention and feed-forward sublayers (since BitNet uses a pre-normalization approach like LLaMA). Test the normalization on dummy data (compare with standard LayerNorm outputs for sanity).

Metadata

Metadata

Assignees

Labels

bitnetBitNet implementationtask

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions