Awesome-Align-LLM-Human

A collection of papers and resources about aligning large language models (LLMs) with human.

We hope this repository can help researchers and practitioners to get a better understanding of this emerging field. If this repository is helpful for you, plase help us by citing this paper:

@article{aligning_llm_human,
    title={Aligning Large Language Models with Human: A Survey},
    author={Yufei Wang and Wanjun Zhong and Liangyou Li and Fei Mi and Xingshan Zeng and Wenyong Huang and Lifeng Shang and Xin Jiang and Qun Liu},
    journal={arXiv preprint arXiv:2307.12966},
    year={2023}
}

Table of Contents

• Awesome-Aligning-LLM-Human
  • Alignment Data
    • Instructions From Human
    • Instructions From Strong LLMs
    • Instructions Data Management
  • Alignment Training
    • Online Human Alignment
    • Offline Human Alignment
    • Parameter-Efficient Training
  • Alignment Evaluation
    • Evaluation Benchmarks
    • Evaluation Paradigms

Alignment Data

Instructions From Human

NLP Benchmarks

• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts [paper]
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks [paper]
• The FLAN collection: Designing data and methods for effective instruction tuning [paper]
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor [paper]
• The OIG Dataset [Blog]

Hand-crafted Instructions

• Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM [Blog]
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment [Paper]
• Chinese open instruction generalist: A preliminary release [Paper]
• ShareGPT [Blog]

Instructions From Strong LLMs

Self-Instruct

Improving Input Quality

• Self-Instruct: Aligning Language Models with Self-Generated Instructions [Paper]
• LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions [Paper]
• Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data [Paper]
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias [Paper]
• WizardLM: Empowering Large Language Models to Follow Complex Instructions [Paper]
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct [Paper]
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor [paper]
• Textbooks Are All You Need [Paper]

Improving Output Quality

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [Paper]
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 [Paper]
• Lion: Adversarial Distillation of Closed-Source Large Language Model [Paper]
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision [Paper]
• ExpertPrompting: Instructing Large Language Models to be Distinguished Experts [Paper]
• Phoenix: Democratizing ChatGPT across Languages [Paper]

Multi-Turn Instructions

• Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality [Blog]
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations [Paper]
• CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society [Paper]
• Selfee: Iterative self-revising llm empowered by self-feedback generation [Blog]

Multilingual Instructions

• Phoenix: Democratizing ChatGPT across Languages [Paper]
• BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models [Paper]
• Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation [Paper]

Instructions Data Management

Instruction Implications

• How far can camels go? exploring the state of instruction tuning on open resources [Paper]
• Flacuna: Unleashing the problem solving power of vicuna using flan fine-tuning [Paper]
• Scaling data-constrained language models [Paper]
• Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation [Paper]

Instruction Quantity

• Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning [Paper]
• LIMA: Less Is More for Alignment [Paper]
• Instruction Mining: High-Quality Instruction Data Selection for Large Language Models [Paper]
• AlpaGasus: Training A Better Alpaca with Fewer Data [Paper]

Alignment Training

Online Human Alignment

• Training language models to follow instructions with human feedback [Paper]
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment [Paper]

Offline Human Alignment

Rank-based Training

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model [Paper]
• Preference Ranking Optimization for Human Alignment [Paper]
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears [Paper]
• Calibrating Sequence likelihood Improves Conditional Language Generation [Paper]

Language-based Training

• OpenChat: Less is More for Open-source Models [Github]
• Languages are rewards: Hindsight finetuning using human feedback [Paper]
• Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits [Paper]
• Training Socially Aligned Language Models in Simulated Human Society [Paper]
• Selfee: Iterative self-revising llm empowered by self-feedback generation [Blog]

Parameter-Efficient Training

• LoRA: Low-Rank Adaptation of Large Language Models [Paper]
• QLoRA: Efficient Finetuning of Quantized LLMs [Paper]
• Prefix-Tuning: Optimizing Continuous Prompts for Generation [Paper]
• The Power of Scale for Parameter-Efficient Prompt Tuning [Paper]
• Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning [Paper]
• Parameter-Efficient Fine-Tuning Design Spaces [Paper]

Alignment Evaluation

Evaluation Benchmarks

Closed-set Benchmarks

General Knowledge

• Measuring Massive Multitask Language Understanding [Paper]
• CMMLU: Measuring massive multitask language understanding in Chinese [Paper]
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models [Paper]
• KoLA: Carefully Benchmarking World Knowledge of Large Language Models [Paper]
• M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models [Paper]
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models [Paper]

Reasoning

• Training Verifiers to Solve Math Word Problems [Paper]
• Measuring Massive Multitask Language Understanding [Paper]
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge [Paper]
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies [Paper]
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [Paper]
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them [Paper]

Coding

• Program Synthesis with Large Language Models [Paper]
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation [Paper]
• Evaluating Large Language Models Trained on Code [Paper]
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation [Paper]

Open-set Benchmarks

• Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality [Blog]
• Self-Instruct: Aligning Language Models with Self-Generated Instructions [Paper]
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment [Paper]
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [Paper]
• Judging LLM-as-a-judge with MT-Bench and Chatbot Arena [Paper]
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback [Paper]

Evaluation Paradigms

Human-based Evaluation

• Self-Instruct: Aligning Language Models with Self-Generated Instructions [Paper]
• LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions [Paper]
• Training language models to follow instructions with human feedback [Paper]
• Judging LLM-as-a-judge with MT-Bench and Chatbot Arena [Paper]

LLMs-based Evaluation

Reference-free Evaluation

• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment [Paper]
• GPTScore: Evaluate as You Desire [Paper]
• Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study [Paper]
• Can Large Language Models Be an Alternative to Human Evaluations? [Paper]
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation [Paper]
• AlignScore: Evaluating Factual Consistency with A Unified Alignment Function [Paper]

LLMs bias in Evaluation

• Large Language Models are not Fair Evaluators [Paper]
• Style Over Substance: Evaluation Biases for Large Language Models [Paper]
• Judging LLM-as-a-judge with MT-Bench and Chatbot Arena [Paper]

LLMs for Evaluation

• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization [Paper]