You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Write the activation function for ReLU² (squared ReLU), which BitNet uses in its feed-forward networks ar5iv.org.
This function should take an input x and compute y = max(0, x)². Ensure that after applying ReLU², the result can be quantized back to 8-bit if needed (since BitNet uses 8-bit activations) ar5iv.org.
This may involve applying a scaling factor or clamping to the 8-bit range based on the dynamic range of the data. Integrate this activation function into your code so it can be used in the feed-forward layers of each transformer block.
Activity
feat(bitnet): implement squared ReLU activation (ReLU²) and tests (#180)
feat(bitnet): implement squared ReLU activation (ReLU²) (#208)