Attention Output Projection #184

Implement the final projection of the concatenated attention contexts back to the model’s hidden size. Use another BitLinear layer to map the combined head output (from step 13) to the hidden dimension (this is often just a linear layer that mixes the heads). This BitLinear uses the trained projection matrix (ternary weights) and, as always, no bias
ar5iv.org.

Each token’s context vector (size = hidden_size) is transformed by this linear layer, producing the output of the attention sublayer for that token. This output remains in 8-bit quantized form. Verify that the output shape matches the input hidden state shape. At this point, implement the residual connection for attention: add the attention output for each token to the original sublayer input (the input prior to QKV projection) to form the updated hidden state.