HyperMixer: An MLP-based Green AI Alternative to Transformers

This is a very quick (and maybe wrong) PyTorch reimplementation of the HyperMixer.

Usage

import torch
from hyper_mixer import HyperMixerBlock

hyper_mixer_block = HyperMixerBlock(dim=32)
input = torch.rand(3, 256, 32)
output = hyper_mixer_block(input)
print(output.shape)

Reference

@article{Liu2021,
    title={{HyperMixer: An MLP-based Green AI Alternative to Transformers}},
    author={Mai, Florian and Pannatier, Arnaud and Fehr, Fabio and Chen, Haolin and Marelli, Francois 
    and Fleuret, Francois and Henderson, James},
    journal={arXiv preprint arXiv:2203.03691},
    year={2022}
}