New README (Lightning-AI#1099)

Co-authored-by: awaelchli <[email protected]> Co-authored-by: Carlos Mocholí <[email protected]>
yuzc19 · Mar 15, 2024 · 17b5c98 · 17b5c98
1 parent a96a984
commit 17b5c98
Show file tree

Hide file tree

Showing 7 changed files with 329 additions and 110 deletions.
diff --git a/README.md b/README.md
diff --git a/images/1.webp b/images/1.webp
diff --git a/images/2.webp b/images/2.webp
diff --git a/images/3.webp b/images/3.webp
diff --git a/images/4.webp b/images/4.webp
diff --git a/tutorials/download_model_weights.md b/tutorials/download_model_weights.md
@@ -2,6 +2,32 @@
 
 LitGPT supports a variety of LLM architectures with publicly available weights. You can download model weights and access a list of supported models using the LitGPT `download.py` script.
 
+
+| Model                                        | Model size                               | Reference                                                                                                                    |
+|----------------------------------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
+| Code Llama by Meta AI                        | 7B, 13B, 34B, 70B                        | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950)                                                                      |
+| Dolly by Databricks                          | 3B, 7B, 12B                              | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) |
+| Falcon by TII UAE                            | 7B, 40B, 180B                            | [TII 2023](https://falconllm.tii.ae)                                                                                         |
+| FreeWilly2 (Stable Beluga 2) by Stability AI | 70B                                      | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models)                             |
+| Function Calling Llama 2 by Trelis           | 7B                                       | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2)                                   |
+| Gemma by Google                              | 2B, 7B                                   | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf)                         |
+| Llama 2 by Meta AI                           | 7B, 13B, 70B                             | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288)                                                                      |
+| LongChat by LMSYS                            | 7B, 13B                                  | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/)                                                            |
+| Mistral and Mixtral by Mistral AI            | 7B                                       | [Mistral website](https://mistral.ai/)                                                                                       |
+| Nous-Hermes by NousResearch                  | 7B, 13B, 70B                             | [Org page](https://huggingface.co/NousResearch)                                                                              |
+| OpenLLaMA by OpenLM Research                 | 3B, 7B, 13B                              | [Geng & Liu 2023](https://github.com/openlm-research/open_llama)                                                             |
+| Phi by Microsoft Research                    | 1.3B, 2.7B                               | [Li et al. 2023](https://arxiv.org/abs/2309.05463)                                                                           |
+| Platypus by Lee at el.                       | 7B, 13B, 70B                             | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317)                                                               |
+| Pythia by EleutherAI                         | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373)                                                                     |
+| RedPajama-INCITE by Together                 | 3B, 7B                                   | [Together 2023](https://together.ai/blog/redpajama-models-v1)                                                                |
+| StableCode by Stability AI                   | 3B                                       | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding)                                           |
+| StableLM by Stability AI                     | 3B, 7B                                   | [Stability AI 2023](https://github.com/Stability-AI/StableLM)                                                                |
+| StableLM Zephyr by Stability AI              | 3B                                       | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding)                                           |
+| TinyLlama by Zhang et al.                    | 1.1B                                     | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama)                                                                   |
+| Vicuna by LMSYS                              | 7B, 13B, 33B                             | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/)                                                                  |
+
+
+
 &nbsp;
 ## General Instructions
 

diff --git a/tutorials/finetune.md b/tutorials/finetune.md
@@ -0,0 +1,47 @@
+# Finetuning
+
+We provide a simple training scripts (`litgpt/finetune/*.py`) that instruction-tunes a pretrained model on the [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset.
+For example, you can either use
+
+LoRA ([Hu et al. 2021](https://arxiv.org/abs/2106.09685)):
+
+```bash
+litgpt finetune lora
+```
+
+or Adapter ([Zhang et al. 2023](https://arxiv.org/abs/2303.16199)):
+
+```bash
+litgpt finetune adapter
+```
+
+or Adapter v2 ([Gao et al. 2023](https://arxiv.org/abs/2304.15010)):
+
+```bash
+litgpt finetune adapter_v2
+```
+
+
+The finetuning requires at least one GPU with ~12 GB memory (RTX 3060).
+
+It is expected that you have downloaded the pretrained weights as described above.
+More details about each finetuning method and how you can apply it to your own data can be found in our technical how-to guides.
+
+
+### Finetuning how-to guides
+
+These technical tutorials illustrate how to run the finetuning code.
+
+- [Full-parameter finetuning](funetuning.md)
+- [Finetune with Adapters](finetune_adapter.md)
+- [Finetune with LoRA or QLoRA](finetune_lora.md)
+
+&nbsp;
+
+### Understanding finetuning -- conceptual tutorials
+
+Looking for conceptual tutorials and explanations? We have some additional articles below:
+
+- [Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-Adapters](https://lightning.ai/pages/community/article/understanding-llama-adapters/)
+
+- [Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)](https://lightning.ai/pages/community/tutorial/lora-llm/)