Note:
N.B. The originalmainbranch of this repository has been renamed tomain_old. This branch contains the historical development of the code base, including the sparse neural network experiments that are reported in [1]. All important functionality has been separated into a number of independent repositories. Moreover, a substantial amount of documentation has been added to the C++ libraries. Similar documentation for the Python libraries is on the way.
The Nerva framework consists of a collection of C++ and Python libraries for neural networks. Originally the library was developed for experimenting with truly sparse neural networks in C++. In the meantime, several Python implementations have been added. The following repositories are available:
| Repository | Description |
|---|---|
| nerva-rowwise | A C++ implementation with data in row-wise layout |
| nerva-colwise | A C++ implementation with data in column-wise layout |
| nerva-jax | A Python implementation using JAX data structures |
| nerva-numpy | A Python implementation using NumPy data structures |
| nerva-tensorflow | A Python implementation using TensorFlow data structures |
| nerva-torch | A Python implementation using PyTorch data structures |
| nerva-sympy | A symbolic Python implementation used for validation |
The Nerva libraries have the following features:
- They support common layers, loss functions and activation functions.
- They support mini-batches, and all equations (including backpropagation!) are given in matrix form.
- Precise mathematical specifications of the equations are available in this specification document.
The nerva-colwise and nerva-rowwise libraries have the following additional features:
- They support truly sparse layers. The weight matrices of these layers are stored using a sparse matrix representation (CSR).
- They include Python bindings.
- Only multilayer perceptrons are supported.
- Only the CPU is supported.
The following papers about Nerva are available:
[1] Nerva: a Truly Sparse Implementation of Neural Networks, https://arxiv.org/abs/2407.17437. It introduces the library, and describes a number of static sparse training experiments.
[2] Batch Matrix-form Equations and Implementation of Multilayer Perceptrons, https://arxiv.org/abs/TODO. It describes the implementation of the Nerva libraries in great detail.
If you have questions, or if you would like to contribute to the Nerva libraries, you can email Wieger Wesselink ([email protected]).