Skip to content

Commit

Permalink
add readme for lora_ft
Browse files Browse the repository at this point in the history
  • Loading branch information
Eric-mingjie committed Oct 23, 2023
1 parent 46a18e9 commit 4938954
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions lora_ft/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Here we provide the script for the lora fine-tuning experiments in the paper. Th
This codebase is based on [run_clm.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling). Here we adapt this code with LoRA fine-tuning on the C4 training dataset. Some custom changes we make in the code include:
- [loc 1](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L374): set up LLaMA-7B for LoRA fine-tuning;
- [loc 2](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L521): set up training arguments for Trainer.
- [loc 3](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L364): we are using the tokenizer from vicuna, which are the same as the original LLaMA tokenizer but also fix the issues of some special tokens.
- [loc 4](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L319): location to load the c4 training dataset.
- [loc 3](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L364): load the tokenizer from vicuna, which are the same as the original LLaMA tokenizer but also fix the issues of some special tokens.
- [loc 4](https://github.com/locuslab/wanda/blob/main/lora_ft/finetune_lm.py#L319): load the c4 training dataset.

To train a LoRA adapter, run the command:
```sh
Expand Down

0 comments on commit 4938954

Please sign in to comment.