Visualization of mean field and neural tangent kernel regime

Fast and Easy Infinite Neural Networks in Python

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch

Example code for the NNGeometry PyTorch library

Code for experiments in my blog post on the Neural Tangent Kernel: https://eigentales.com/NTK

Simple implementation of muP, based on Spectral Condition for Feature Learning. The implementation is SGD only, dont use it for Adam

Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training