Production-tested AI infrastructure tools for efficient AGI development and community-driven innovation

Repository for MLCommons Chakra schema and tools

Summary of some awesome work for optimizing LLM inference

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).

My learning notes/codes for ML SYS.

LaTeX Proposal Template for the University of Chinese Academy of Sciences

The code based on vLLM for the paper “ Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention”.

Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

Super-Efficient RLHF Training of LLMs with Parameter Reallocation

Curated collection of papers in machine learning systems

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

UPMEM LLM Framework allows profiling PyTorch layers and functions and simulate those layers/functions with a given hardware profile.

Fast Multimodal LLM on Mobile Devices

LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale

📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

AIOS: AI Agent Operating System

A Overview of Efficiently Serving Foundation Models across Edge Devices

GLake: optimizing GPU memory management and IO transmission.

SGLang is a fast serving framework for large language models and vision language models.

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

[TMLR 2024] Efficient Large Language Models: A Survey

A list of tutorials, paper, talks, and open-source projects for emerging compiler and architecture

awesome llm plaza: daily tracking all sorts of awesome topics of llm, e.g. llm for coding, robotics, reasoning, multimod etc.

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, Flash-Attention, Paged-Attention, Parallelism, etc. 🎉🎉

An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"

A list of papers, docs, codes about efficient AIGC. This repo is aimed to provide the info for efficient AIGC research, including language and vision, we are continuously improving the project. Wel…

NeuPIMs Simulator