Set Model Constants (BitNet-b1.58-2B-4T Hyperparameters) #176

Define constants for the model architecture based on BitNet’s configuration. For example, set the hidden dimension, number of layers, number of attention heads, etc., according to the BitNet b1.58‑2B spec.

From the config: e.g.,

• hidden_size=2560

• intermediate_size=6912

• num_hidden_layers=30

• num_attention_heads=20

• num_key_value_heads=5

• max_position_embeddings=4096

• rms_norm_eps = 1e-5 – required by every RMSNorm layer.
  huggingface.co

• rope_theta = 500 000.0 – RoPE base value for long-context support.
  huggingface.co

• Vocabulary & special tokens:

  • vocab_size = 128 256
  • bos_token_id = 128 000
  • eos_token_id = 128 001`
  • Essential for tokenizer and embedding setup:
    huggingface.co

• tie_word_embeddings=true – makes the LM head share weights with the input embedding matrix.
  huggingface.co

• (Optionally) initializer_range = 0.02 – useful when the code needs to create scratch tensors in tests.

huggingface.co
huggingface.co
.)

Also note activation functions and normalization type from the config (e.g. hidden_act="relu2" indicating squared ReLU). Ensure you account for any architecture specifics like shared key/value heads (in BitNet, 20 heads with 5 unique K/V sets) and the absence of bias terms in all layers
ar5iv.org.