Feed-Forward Network (FFN) Sublayer #185

Implement the two-layer feed-forward network for the transformer block. This consists of: (a) a BitLinear layer that expands the hidden dimension to some larger intermediate size (e.g. 6912) using 1-bit weights; (b) apply the ReLU² activation function to the intermediate outputs
ar5iv.org
; (c) a second BitLinear layer that projects the intermediate results back down to the hidden size. Both linear layers use the quantized weights and have no biases. After the first linear, ensure the 8-bit activation output is properly quantized before applying ReLU² (the activation itself will output non-negative values, which you then may quantize again). After the second linear, quantize/clamp the output to 8-bit as well. This FFN transforms each token’s representation through a non-linear layer and is applied independently per token (position). Test the FFN implementation with a small input to ensure dimensions and activation are correct.