SpargeAttention: A training-free sparse attention that can accelerate any model inference.

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

FlashMLA: Efficient MLA decoding kernels

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, Flash-Attention, Paged-Attention, Parallelism, etc. 🎉🎉

📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.

Private Cloud Compute (PCC)

Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.

Development repository for the Triton language and compiler

This repository contains integer operators on GPUs for PyTorch.

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Fast and memory-efficient exact attention

✨ Light and Fast AI Assistant. Support: Web | iOS | MacOS | Android | Linux | Windows

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, …

The collection of pre-trained, state-of-the-art AI models for ailia SDK

System for AI Education Resource.

Try to reproduce mediapipe with OpenCV_lite and MNN.

Convert Caffe models to ONNX.

Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information

Step-by-step GEMM optimization tutorial on OpenCL GPU platforms

MLX: An array framework for Apple silicon

LLM inference in C/C++

The programming language Ficus

A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.

MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba. Full multimodal LLM Android App:[MNN-LLM-Android](./apps/Android/MnnLlmChat/READ…

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and…

official pypi project for libfacedetection

The interactive graphing library for Python ✨

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production