Inference Code for Paper "Harder Tasks Need More Experts: Dynamic Routing in MoE Models"

Training Sparse Autoencoders on Language Models

欢迎来到 LLM-Dojo，这里是一个开源大模型学习场所，使用简洁且易阅读的代码构建模型训练框架(支持各种主流模型如Qwen、Llama、GLM等等)、RLHF框架(DPO/CPO/KTO/PPO)等各种功能。👩‍🎓👨‍🎓

awesome SAE papers

Bringing BERT into modernity via both architecture changes and scaling

LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment

An Efficient LLM Fine-Tuning Factory Optimized for MoE PEFT

Blazingly fast LLM inference.

[NAACL'24] Self-data filtering of LLM instruction-tuning data using a novel perplexity-based difficulty score, without using any other models

[NAACL'24] Self-data filtering of LLM instruction-tuning data using a novel perplexity-based difficulty score, without using any other models

X-LoRA: Mixture of LoRA Experts

Adapt an LLM model to a Mixture-of-Experts model using Parameter Efficient finetuning (LoRA), injecting the LoRAs in the FFN.

Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

RapidIn: Scalable Influence Estimation for Large Language Models (LLMs). The implementation for paper "Token-wise Influential Training Data Retrieval for Large Language Models" (Accepted on ACL 2024).

This repository contains various advanced techniques for Retrieval-Augmented Generation (RAG) systems.

Tools for working with Gauss-Newton Hessian in PyTorch

A collection of large question answering datasets

An Open Large Reasoning Model for Real-World Solutions

Data for "Datamodels: Predicting Predictions with Training Data"

Influence Functions with (Eigenvalue-corrected) Kronecker-Factored Approximate Curvature

An accessibility tool to assist in FFXIV gameplay and compensate for human imperfections.

[ICML 2024] Selecting High-Quality Data for Training Language Models

AI Logging for Interpretability and Explainability🔬

Official repository for MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models [NeurIPS 2024]

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]

DSIR large-scale data selection framework for language model training