Implement Rotary Positional Encoding #177

Add Rotary Position Embeddings (RoPE) to the model’s attention mechanism
ar5iv.org.

Implement functions to apply RoPE to the query and key tensors of each attention head. Use the context length (4096) and rotation base (rope_theta) from the config to parameterize RoPE (BitNet uses a larger theta, ~5e5, to support 4096 tokens without degradation
huggingface.co
).

This step will involve intertwining sine-cosine rotations into the query/key vectors based on token position indices. Test the RoPE implementation on known inputs (or compare with an existing RoPE reference) to ensure it’s injecting position information correctly.