CUDA Templates for Linear Algebra Subroutines

SGLang is a fast serving framework for large language models and vision language models.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

A high-throughput and memory-efficient inference and serving engine for LLMs

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Universal LLM Deployment Engine with ML Compilation

An Open Source Machine Learning Framework for Everyone

Infrastructure to enable deployment of ML models to low-power resource-constrained embedded targets (including microcontrollers and digital signal processors).

TinyNeuralNetwork is an efficient and easy-to-use deep learning model compression framework.