Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"

DeepEP: an efficient expert-parallel communication library

A generalized framework for subspace tuning methods in parameter efficient fine-tuning.

Overseas Summer Research Guidance 海外暑研申请指南

DeMo: Decoupled Momentum Optimization

official implementation of paper SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

[MLSys 2024] Does Compressing Activations Help Model Parallel Training?

Code for experiments with activations and gradients compression for model-parallel training.

Graphic notes on Gilbert Strang's "Linear Algebra for Everyone"

Minimalistic 4D-parallelism distributed training framework for education purpose

Simple Implementation of the CVPR 2024 Paper "JointSQ: Joint Sparsification-Quantization for Distributed Learning"

Making large AI models cheaper, faster and more accessible

LDAdam - Adaptive Optimization from Low-Dimensional Gradient Statistics

Pytorch distributed backend extension with compression support

GRACE - GRAdient ComprEssion for distributed deep learning

Practical low-rank gradient compression for distributed optimization: https://arxiv.org/abs/1905.13727

A PyTorch native library for large model training

nanoGPT style version of Llama 3.1

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

《开源大模型食用指南》针对中国宝宝量身打造的基于Linux环境快速微调（全参数/Lora）、部署国内外开源大模型（LLM）/多模态大模型（MLLM）教程

Machine Learning and Computer Vision Engineer - Technical Interview Questions

QLoRA: Efficient Finetuning of Quantized LLMs

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)

Retrieval-Augmented Theorem Provers for Lean

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

[NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch