Skip to content

Apply Attention Weights to Values #183

Closed

Description

@thejhh

Use the attention weights to compute the attention output for each head. For each attention head, multiply the weight matrix (from step 12) by the head’s Value matrix $V$ (shape [seq_len, head_dim]). This yields the context matrix for that head (of shape [seq_len, head_dim]), where each token now contains a weighted sum of value vectors from other tokens. Implement this multiplication in an efficient manner, again using higher precision for accumulation if $V$ is 8-bit. The result should then be quantized/clamped back to 8-bit range. After computing the context for all heads, concatenate the context vectors of all heads together for each token (resulting in a [seq_len, hidden_size] matrix, since hidden_size = num_heads * head_dim).

Metadata

Metadata

Assignees

Labels

bitnetBitNet implementationtask

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions