An example implementation of the bucketMul algorithm - you can read about it here.
With it you can smoothly adjust—in real time—the number of calculations performed during the inference of an LLM model.
At 50% effort, it performs as fast as regular matrix multiplications on Apple Silicon chips; at 25% effort, it is twice as fast while still retaining most of the quality.
You also have the option to skip loading the least important weights.
You can quickly get started by downloading the precompiled binaries available at: Effort Engine v0.0.1
To bypass macOS Gatekeeper, hold option while clicking to open the downloaded application for the first time.
On the first run, you will be prompted to download the converted weights necessary for operation. Subsequently, a matrix multiplication benchmark will execute to demonstrate the capabilities of the engine.
The sources are in Swift & Metal.
Download and open effort.xcodeproj. It should work straight away.
- More Information: Visit our project page.
- See it in Action: Watch a demo on Asciinema.
Ton of things to fix, looking for collabolators! :)