Pinned Loading

1. interpreting-transformers-with-boolean-symmetries interpreting-transformers-with-boolean-symmetries Public

   We investigate symmetries in single layer transformers used to represent boolean functions and construct bounds in the number of attention heads required to represent them.

2. rkhs-and-deep-learning rkhs-and-deep-learning Public

   An intuitive derivation of smoothing splines from variational calculus, demonstrating their relationship to reproducing kernel Hilbert spaces (RKHS) and regularized neural networks.

   Python

3. interpreting-transformers-as-a-markov-process interpreting-transformers-as-a-markov-process Public

   We construct a fully interpretable single-attention-head transformer, which we train to predict Markov chains. The transformer represents a typical architecture, but we are able to solve it analyti…

   Python

4. generalization-in-resnets-and-SLT generalization-in-resnets-and-SLT Public

   Study of how neuron redundancy may bias ResNets toward collective class transport over sample-wise transport, via the lens of singular learning theory.