Skip to content

Implement Squared ReLU Activation #180

Closed

Description

@thejhh

Write the activation function for ReLU² (squared ReLU), which BitNet uses in its feed-forward networks
ar5iv.org.

This function should take an input x and compute y = max(0, x)². Ensure that after applying ReLU², the result can be quantized back to 8-bit if needed (since BitNet uses 8-bit activations)
ar5iv.org.

This may involve applying a scaling factor or clamping to the 8-bit range based on the dynamic range of the data. Integrate this activation function into your code so it can be used in the feed-forward layers of each transformer block.

Metadata

Metadata

Assignees

Labels

bitnetBitNet implementationtask

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions