Independent implementation of LLaMA that is fully open source under the Apache 2.0 license.
This implementation builds on nanoGPT. Weights are distributed by Meta under a research-only license.
We believe that AI should be fully open source and part of the collective knowledge.
The original LLaMA code is GPL licensed which means any project using it must also be released under GPL.
This "taints" any other code and prevents integration with the rest of the ecosystem.
Lit-LLaMA solves that for good.
Lit-LLaMA is:
- Simple: Single-file implementation without boilerplate.
- Correct: Numerically equivalent to the original model.
- Optimized: Runs on consumer hardware or at scale.
- Open-source: No strings attached.
Join our Discord to build high-performance, truly open-source models for the common benefit of the community.
Clone the repo
git clone https://github.com/Lightning-AI/lit-llama
cd lit-llamainstall dependencies
pip install -r requirements.txtYou are all set! 🎉
To generate text predictions, download the model weights following the instructions on the official LLaMA repository.
Once downloaded, you should have a folder like this:
checkpoints/llama
├── 7B
│ ├── checklist.chk
│ ├── consolidated.00.pth
│ └── params.json
├── 13B
│ ...
├── tokenizer_checklist.chk
└── tokenizer.model
Convert the weights to the Lit-LLaMA format:
python scripts/convert_checkpoint.py \
--output_dir checkpoints/lit-llama \
--ckpt_dir checkpoints/llama \
--tokenizer_path checkpoints/llama/tokenizer.model \
--model_size 7BRun inference:
python generate.py --prompt "Hello, my name is"This will run the 7B model and require ~26 GB of GPU memory (A100 GPU).
For GPUs with less memory, enable quantization (--quantize llm.int8) or use bfloat16 (--dtype bfloat16). Quantization will take longer to load but require ~8GB of memory. bfloat16 is closer to the "full deal" and runs on ~10GB of GPU memory.
This can run on any consumer GPU.
python generate.py --quantize llm.int8 --prompt "Hello, my name is"See python generate.py --help for more options.
You can also use GPTQ-style int4 quantization, but this needs conversions of the weights first:
python quantize.py --checkpoint_path state_dict.pth --tokenizer_path tokenizer.model --output_path llama-7b-gptq.4bit.pt --dtype bfloat16 --quantize gptq.int4With the generated quantized checkpoint generation works as usual with --quantize gptq.int4, bringing GPU usage to about ~5GB. As only the weights of the Linear layers are quantized, it is useful to use --dtype bfloat16 even with the quantization enabled.
We provide a simple training scripts in finetune_lora.py and finetune_adapter.py that instruction-tunes a pretrained model on the Alpaca dataset using the techniques of LoRA and Adapter.
-
Download the data and generate a instruction tuning dataset:
python scripts/prepare_alpaca.py
-
Run the finetuning script
python finetune_lora.py
or
python finetune_adapter.py
It is expected that you have downloaded the pretrained weights as described above. The finetuning requires a GPU with ~24 GB memory (GTX 3090). Note: For some GPU models you might need to install PyTorch nightly (see issue #101).
We're in a quest towards fully open source AI.
Join us and start contributing, especially on the following areas:
Look at train.py for a starting point towards pre-training / fine-tuning using Lightning Fabric.
Don't forget to join our Discord!
- @karpathy for nanoGPT
- @FacebookResearch for the original LLaMA implementation
- @TimDettmers for bitsandbytes
- @Microsoft for LoRA
- @IST-DASLab for GPTQ
Lit-LLaMA is released under the Apache 2.0 license.