Skip to content

Commit

Permalink
Updated GitHub markdown alerts (Lightning-AI#841)
Browse files Browse the repository at this point in the history
  • Loading branch information
Andrei-Aksionov authored Dec 20, 2023
1 parent 608e8ed commit 9a2065e
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 8 deletions.
7 changes: 1 addition & 6 deletions tutorials/finetune_lora.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,17 +167,12 @@ python scripts/merge_lora.py \
--out_dir "out/lora_merged/stablelm-base-alpha-3b/"
```

> [!Note]
> [!IMPORTANT]
> If you changed the LoRA hyperparameters (`lora_r`, `lora_key`, etc.) in the
> `finetune/lora.py` script, it is important to update the hyperparameter configuration
> in the `scripts/merge_lora.py` script accordingly. Otherwise, you will encounter size
> mismatch errors upon merging.
> [!Note]
> If the model was trained with quantized weights, it is possible to merge LoRA
> and quantized pretrained weights by providing `--quantize` argument,
> the same that was used for the finetuning stage.
After merging, we can use the `base.py` file for inference using the new checkpoint file. Note that if your new checkpoint directory is different from the original checkpoint directory, we also have to copy over the tokenizer and config files:

```bash
Expand Down
4 changes: 2 additions & 2 deletions tutorials/quantize.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,14 @@ Memory used: 14.50 GB

To reduce the memory requirements further, Lit-GPT supports several quantization techniques, which are shown below.

> [!NOTE]
> [!TIP]
> Most quantization examples below also use the `--precision bf16-true` setting explained above. If your GPU does not support `bfloat16`, you can change it to `--precision 16-true`.
## `bnb.nf4`

Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2305.14314v1) to learn more about how it works.

> [!NOTE]
> [!IMPORTANT]
> `bitsandbytes` only supports `CUDA` devices and the `Linux` operating system.
> Windows users should use [WSL2](https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl).
Expand Down

0 comments on commit 9a2065e

Please sign in to comment.