Collaborative exploration of activation functions in neural networks, covering the range of functions available in torch.nn (and beyond), from foundational ones (ReLU, Sigmoid, Tanh) to more specialised and modern variants (GELU, SiLU, Mish and others).
The work is published as two websites: The Nonlinearity Index (https://nonlinearity-index.vercel.app) and Neural Atlas (https://neural-atlas-ten.vercel.app).
Both sites are static and hosted on Vercel.
| Site | What it is | Source |
|---|---|---|
| The Nonlinearity Index | A catalogue of activation functions: formula, behaviour, and a plot of each curve and its derivative, organised by family. Includes a use-case guide to which functions get chosen for which tasks and why. 374 functions, most of them taken from the Kunc & Kléma (2024) survey and checked against its formulas. | NJRProjects/NonlinearityIndex/ |
| Neural Atlas | A field guide to Machine Learning, Deep Learning and Reinforcement Learning: about 195 topics with formulas and code, three interactive labs, and a Python playground that runs 26 example programs in the browser. | NJRProjects/ML/main.html |
This repository is a joint deep-dive into why activation functions matter, not just what they do. Goals include:
- Understanding the mathematical behaviour of each function (shape, range, derivative)
- Comparing gradient-flow characteristics (for example, vanishing and exploding gradients)
- Identifying practical use cases and trade-offs (for example, why GELU is preferred in Transformers and why ReLU dominates CNNs)
- Hands-on implementation, both from scratch (plain Python/NumPy) and with
nnbuilt-ins - Visualising function curves and their derivatives side by side
Exploration spans the ~400 activation functions available in torch.nn and related libraries, prioritised by relevance and common usage rather than exhaustive coverage of every variant. The Nonlinearity Index tracks progress toward that number.
ActivationFWorks/
├── NJRProjects/ individual work (NJR)
│ ├── Sigmoid_Activation/ first function in the series: notebook on sigmoid and its gradient
│ ├── ML/ Neural Atlas site (main.html) and an ML revision notebook
│ └── NonlinearityIndex/ The Nonlinearity Index site
├── RSProjects/ individual work (RS)
└── Collab_projects/ joint synthesis
Individual folders hold each person's exploration; Collab_projects/ is for the comparisons and write-ups we put together jointly.
Plain HTML and CSS, generated by a small Python script that uses only the standard library. Every plot is pre-rendered as an inline SVG, so the pages work without JavaScript.
To add a function, append an entry to catalog.py (the fields are documented at the top of the file), optionally reference it from use_cases.py, then rebuild:
cd NJRProjects/NonlinearityIndex
python3 build.py # validates the data and regenerates public/
vercel deploy --prod # publishes public/ to nonlinearity-index.vercel.appSee NJRProjects/NonlinearityIndex/README.md for details.
A single self-contained file, NJRProjects/ML/main.html. Topics live in the TRACKS array in its script, and the runnable Python examples are the <script type="text/x-python"> blocks near the end of the file. The playground runs them with Pyodide (Python compiled to WebAssembly), so numpy, pandas, scikit-learn and matplotlib work in the browser; PyTorch does not.
To preview locally, serve the folder over HTTP rather than opening the file directly, because the playground's background worker needs it:
cd NJRProjects/ML
python3 -m http.server 8000 # then open http://localhost:8000/main.html
vercel deploy --prod # publishes to neural-atlas-ten.vercel.appDeploying either site needs access to the Vercel team that owns the projects.
Work here follows the same style as the rest of the collaboration: concepts explained first, hands-on implementation second. Findings, comparisons and notebooks are split between individual exploration and joint synthesis as understanding develops.
This repository is expected to evolve as functions are explored and comparisons are added. Treat it as a living reference rather than a fixed structure. Corrections to formulas, sources or use-case claims on either site are welcome.