pku_llam2_finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "notebookRunGroups": {
     "groupValue": ""
    }
   },
   "source": [
    "# llama2微调流程的步骤\n",
    "\n",
    "* **安装环境**：安装大模型llama2、大模型微工具llama-recipes，并测试。\n",
    "* **准备微调数据**：收集并准备课程要求的、用于微调llama2模型的数据集。\n",
    "* **数据预处理**：对数据执行必要的预处理步骤，如清洗、分词、编码等。\n",
    "* **上传数据集**：将预处理后的数据上传到北大超算平台指定的存储系统。\n",
    "* **配置微调参数**：设置llama2模型微调所需的参数，包括学习率、批大小、训练周期等。\n",
    "* **提交微调任务**：在北大超算平台上，根据设置的超算环境和微调参数，提交llama2模型的微调任务。\n",
    "* **监测任务状态**：在北大超算平台上监测微调任务的执行状态，直至任务完成。\n",
    "* **结果分析**：当任务完成后，分析微调结果，如损失函数的变化、模型的准确率等指标。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **安装环境**：安装大模型llama2、大模型微工具llama-recipes，并测试。\n",
    "https://github.com/denofiend/acm/blob/master/pku_llama2_install_and_test.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **准备微调数据**：收集并准备课程要求的、用于微调llama2模型的数据集。\n",
    "### 注意事项：\n",
    "1. 北大超算连不上huggingface.co，所以要提前下载数据集\n",
    "2. 准备数据集的格式有json、csv等等，json、csv、excel数据格式都可以，但是按照课程要求的微调数据格式准备具体数据\n",
    "3. 建议使用未名二的GPU80G规格，因为模型和数据集预计使用52G存储，GPU40不够\n",
    "4. GPU80G规格只有两个节点，需要排队。\n",
    "5. 需要准备训练数据集 和  验证数据集。训练数据集可以用GPT、代码生成，验证数据集可以是大家讨论出来的。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### json数据生成的方法\n",
    "\n",
    "可以通过下面的promot询问大模型，生成python代码来生成：\n",
    "\n",
    "***写出python代码生成数据集，数据集的格式是json，需要对中国32个省市地区的每个问题模版都生成问题。要求如下：1. 请变换问题的问法，但是答案不变，要求问题的省市地区和答案中的地区参数保持一致，请生成10条不同的问题模版，2. 并将北京市替换为中国其他32个省市地区，省市地区去掉市、省、自治区，并将api_args里面的北京市修改成对应值，3. json文件名要和api对应起来，4. 一条json数据一行。5. 将2023 修改为[2023,2030]之间随机值, 5 年为 1-10随机值，并修改2023/01/01\",\"5\"对应的值，2是年份的差值，并将api_args修改成对应值，格式如下：\n",
    "请预测 2023 年未来 5 年风电发电量,api: power_generation_prediction api_args:[\"全国月度发电量\",\"风电\",\"年\",\"2023/01/01\",\"5\",\"True\",\"True\"]***\n",
    "\n",
    "### 20个APi，每个APi都建议生成如下的python代码，来生成训练数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "\n",
    "# 定义省市自治区列表\n",
    "regions = [\n",
    "    \"北京\", \"天津\", \"河北\", \"山西\", \"内蒙古\", \"辽宁\", \"吉林\", \"黑龙江\",\n",
    "    \"上海\", \"江苏\", \"浙江\", \"安徽\", \"福建\", \"江西\", \"山东\", \"河南\",\n",
    "    \"湖北\", \"湖南\", \"广东\", \"广西\", \"海南\", \"重庆\", \"四川\", \"贵州\",\n",
    "    \"云南\", \"西藏\", \"陕西\", \"甘肃\", \"青海\", \"宁夏\", \"新疆\", \"全国\"\n",
    "]\n",
    "\n",
    "# 生成随机数字函数（1到10）\n",
    "def get_random_number():\n",
    "    return random.randint(1, 10)\n",
    "\n",
    "# 生成20个不同的问题模板，这里可以问题GPT，或者同组讨论\n",
    "question_templates = [\n",
    "    \"请预测未来 {number} 年{region}铁路里程的变化趋势。\",\n",
    "    \"未来 {number} 年{region}铁路扩展情况如何？\",\n",
    "    \"如何评价{region}接下来 {number} 年的铁路发展潜力？\",\n",
    "    \"未来 {number} 年内{region}铁路发展的预期目标是？\",\n",
    "    \"对{region}接下来 {number} 年铁路里程增长的预测。\",\n",
    "    # ... 添加更多问题模板以满足20个模板的要求\n",
    "]\n",
    "\n",
    "# 答案模板生成函数\n",
    "def make_answer(region, number):\n",
    "    return f'api: railway_prediction api_args:[\"各省铁路年度里程 1\",\"{region}\", \"年\", \"{number}\", \"True\", \"True\"]'\n",
    "\n",
    "# 生成数据集，对每个省市地区都生成问题\n",
    "dataset = []\n",
    "with open('railway_prediction.json', 'w', encoding='utf-8') as json_file:\n",
    "    for region in regions:\n",
    "        for _ in range(20):  # 每个地区生成20个问题\n",
    "            number = get_random_number()\n",
    "            question_template = random.choice(question_templates)\n",
    "            question = question_template.format(region=region, number=number)\n",
    "            answer = make_answer(region, number)\n",
    "            json_file.write(json.dumps({\"query\": question, \"answer\": answer}, ensure_ascii=False) + '\\n')\n",
    "\n",
    "print(\"数据集已生成并保存为dataset.json文件\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### path/to/datasets/train/train.json 数据举例\n",
    "\n",
    "\n",
    "> {\"query\": \"未来6年内，预计全国的公路里程增长率将是多少?\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 6, True, False]\"}   \n",
    "> {\"query\": \"未来9年内，预计全国的公路里程增长率将是多少?\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 9, True, False]\"}   \n",
    "> {\"query\": \"全国在接下来的8年里公路发展的预测增长率是多少?\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 8, True, False]\"}   \n",
    "> {\"query\": \"全国在接下来的5年里公路发展的预测增长率是多少?\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 5, True, False]\"}   \n",
    "> {\"query\": \"请预测全国未来3年的公路里程可能增长的百分比。\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 3, True, False]\"}   \n",
    "> {\"query\": \"请预测全国未来1年的公路里程可能增长的百分比。\", \"answer\": \"api: highway_prediction api_args:[\\\"各省公路年度里程\\\", \\\"全国\\\", \\\"2024\\\", 1, True, False]\"}   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## **数据预处理**：对数据执行必要的预处理步骤，如清洗、分词、编码等。\n",
    "### 1. 使用custom_datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 微调数据集 转换代码请参考下面，放到pku_datasets.py文件中，需要修改 path/to/datasets/train/train.json\n",
    "\n",
    "# Copyright (c) Meta Platforms, Inc. and affiliates.\n",
    "# This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.\n",
    "\n",
    "# For dataset details visit: https://huggingface.co/datasets/samsum\n",
    "\n",
    "import copy\n",
    "import datasets\n",
    "\n",
    "# Copyright (c) Meta Platforms, Inc. and affiliates.\n",
    "# This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.\n",
    "\n",
    "from dataclasses import dataclass\n",
    "import logging\n",
    "\n",
    "def get_custom_dataset(dataset_config, tokenizer, split: str):\n",
    "    if \"train\" == split:\n",
    "        dataset = datasets.load_dataset(\"json\", data_files=\"path/to/datasets/train/train.json\")['train']\n",
    "    else:\n",
    "        dataset = datasets.load_dataset(\"json\", data_files=\"path/to/datasets/validation/validation.json\")['train']\n",
    "    print(dataset[0])\n",
    "\n",
    "    prompt = (\n",
    "            f\"query:\\n{{query}}\\nanswer:\\n{{answer}}\"\n",
    "    )\n",
    "\n",
    "    def apply_prompt_template(sample):\n",
    "        prompt = tokenizer.encode(tokenizer.bos_token + sample[\"query\"], add_special_tokens=False)\n",
    "        summary = tokenizer.encode(sample[\"answer\"] +  tokenizer.eos_token, add_special_tokens=False)\n",
    "\n",
    "        sample = {\n",
    "            \"input_ids\": prompt + summary,\n",
    "            \"attention_mask\" : [1] * (len(prompt) + len(summary)),\n",
    "            \"labels\": [-100] * len(prompt) + summary,\n",
    "            }\n",
    "        return sample\n",
    "\n",
    "\n",
    "    dataset = dataset.map(apply_prompt_template, remove_columns=list(dataset.features))\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 下载samsum等数据集的方法\n",
    "因为超算不能连接huggingface下载数据集，所以要自己写download_dataset.py，放到缓存中，这样就可以微调了。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "\n",
    "  \n",
    "# 加载samsum数据集  \n",
    "#dataset = datasets.load_dataset(\"https://hf-mirror.com/datasets/samsum\")\n",
    "# 将 https://hf-mirror.com/datasets/samsum/blob/main/samsum.py 放到  ~/.cache/huggingface/datasets/samsum/samsum.py\n",
    "dataset = datasets.load_dataset(\"path/to/.cache/huggingface/datasets/samsum/samsum.py\")\n",
    "\n",
    "\n",
    "print(dataset)\n",
    "  \n",
    "# 显示数据集的一些信息  \n",
    "print(dataset)  \n",
    "  \n",
    "# 如果你想查看一些样本，你可以这样做：  \n",
    "for sample in dataset[\"train\"][:5]:  \n",
    "    print(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 直接修改sansum是的代码\n",
    "路径是：/lustre/home/2301xxxxxx/llama-recipes/src/llama_recipes/datasets/samsum_dataset.py\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright (c) Meta Platforms, Inc. and affiliates.\n",
    "# This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.\n",
    "\n",
    "# For dataset details visit: https://huggingface.co/datasets/samsum\n",
    "\n",
    "import copy\n",
    "import datasets\n",
    "\n",
    "\n",
    "def get_preprocessed_samsum(dataset_config, tokenizer, split):\n",
    "    if \"train\" == split:\n",
    "        dataset = datasets.load_dataset(\"json\", data_files=\"path/to/datasets/train/train.json\")['train']\n",
    "    else:\n",
    "        dataset = datasets.load_dataset(\"json\", data_files=\"path/to/datasets/validation/validation.json\")['train']\n",
    "\n",
    "    print(dataset[0])\n",
    "\n",
    "    prompt = (\n",
    "        f\"query:\\n{{query}}\\n---\\nanswer:\\n\"\n",
    "    )\n",
    "\n",
    "    def apply_prompt_template(sample):\n",
    "        return {\n",
    "            \"query\": prompt.format(query=sample[\"query\"]),\n",
    "            \"answer\": sample[\"answer\"],\n",
    "        }\n",
    "\n",
    "    dataset = dataset.map(apply_prompt_template, remove_columns=list(dataset.features))\n",
    "\n",
    "    def tokenize_add_label(sample):\n",
    "        prompt = tokenizer.encode(tokenizer.bos_token + sample[\"query\"], add_special_tokens=False)\n",
    "        summary = tokenizer.encode(sample[\"answer\"] + tokenizer.eos_token, add_special_tokens=False)\n",
    "\n",
    "        sample = {\n",
    "            \"input_ids\": prompt + summary,\n",
    "            \"attention_mask\" : [1] * (len(prompt) + len(summary)),\n",
    "            \"labels\": [-100] * len(prompt) + summary,\n",
    "            }\n",
    "\n",
    "        return sample\n",
    "\n",
    "    dataset = dataset.map(tokenize_add_label, remove_columns=list(dataset.features))\n",
    "\n",
    "    return dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 3. 请手动将llama-recipes的代码中的 attn_implementation 注释，代码如下\n",
    "\n",
    "130     else:\n",
    "131         model = LlamaForCausalLM.from_pretrained(\n",
    "132             train_config.model_name,\n",
    "133             load_in_8bit=True if train_config.quantization else None,\n",
    "134             device_map=\"auto\" if train_config.quantization else None,\n",
    "135             use_cache=use_cache,\n",
    "136             #attn_implementation=\"sdpa\" if train_config.use_fast_kernels else None,\n",
    "137         )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **上传数据集**：将预处理后的数据上传到北大超算平台指定的存储系统。\n",
    "\n",
    "1. 将本地20个API生成的数据合并到 train.json 中。\n",
    "2. 将 train.json 文件放到北大超算的 path/to/datasets/train/train.json "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **配置微调参数**：设置llama2模型微调所需的参数，包括学习率、批大小、训练周期等。\n",
    "1. 进入 llama-recipes/src/llama_recipes/configs 目录\n",
    "2. 修改 peft.py 文件，参照下面"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class lora_config:\n",
    "     r: int=8\n",
    "     #lora_alpha: int=32\n",
    "     lora_alpha: int=16\n",
    "     target_modules: List[str] = field(default_factory=lambda: [\"q_proj\", \"v_proj\"])\n",
    "     bias= \"none\"\n",
    "     task_type: str= \"CAUSAL_LM\"\n",
    "     #lora_dropout: float=0.05\n",
    "     lora_dropout: float=0.1\n",
    "     inference_mode: bool = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **提交微调任务**：在北大超算平台上，根据设置的超算环境和微调参数，提交llama2模型的微调任务。\n",
    "\n",
    "使用命令行提交方式，详细看下面的脚本和提交命令, 参考文档地址：https://hpc.pku.edu.cn/ug/guide/slurm/sbatch\n",
    "\n",
    "或者使用sbach --help自己研究一下，\n",
    "\n",
    "### 使用custom_dataset，提交到集群的job.sh，注意使用custom_dataset需要先训练samsum_dataset数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/bin/bash\n",
    "#SBATCH -o job.%j.out\n",
    "#SBATCH --partition=GPU80G\n",
    "#SBATCH --qos=low\n",
    "#SBATCH -J myFirstGPUJob\n",
    "#SBATCH --nodes=1             \n",
    "#SBATCH --ntasks-per-node=6\n",
    "#SBATCH --gres=gpu:1             \n",
    "\n",
    "nvidia-smi\n",
    "\n",
    "\n",
    "lr=2e-4  # 学习率\n",
    "lora_rank=64 # LoRA低秩矩阵的维数\n",
    "lora_alpha=128 #  LoRA低秩矩阵的缩放系数，为一个常数超参，调整alpha与调整学习率类似\n",
    "target_modules=\"q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj\" # 可训练的 LORA 模块，q_proj、k_proj和v_proj是多头注意力机制中的三个线性变换，用于将输入的token映射到一个高维向量空间中，以便于模型对输入进行处理；o_proj则是多头注意力机制的输出层，它将模型的输出映射到一个概率分布上，以便于模型预测下一个token；gate_proj、down_proj和up_proj则是在LoRA微调方法中使用的一些层\n",
    "modules_to_save=\"embed_tokens,lm_head\" # 需要保存的模块，embed_tokens层将输入的token映射到一个高维向量空间中，以便于模型对输入进行处理。lm_head层则是预测下一个token的输出层，它将模型的输出映射到一个概率分布上，以便于模型预测下一个token\n",
    "lora_dropout=0.05 # LoRA 层的丢弃（dropout）率，取值范围为[0, 1)\n",
    "\n",
    "gradient_accumulation_steps=1\n",
    "output_dir=~/llama/models_hsf/0316PEFT/model\n",
    "\n",
    "\n",
    "python3 -m llama_recipes.finetuning --use_peft --peft_method lora --quantization --lora_config.r ${lora_rank} --lora_config.lora_alpha ${lora_alpha} --lora_config.lora_dropout ${lora_dropout} --lora_config.target_modules ${target_modules} --lr ${lr} --gradient_accumulation_steps ${gradient_accumulation_steps} --model_name ~/llama/models_hf/7B --output_dir ${output_dir} --dataset \"custom_dataset\" --custom_dataset.file \"/lustre/home/2301221855/finetuning/pku_dataset.py\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用samsum_dataset，提交到集群的job.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/bin/bash\n",
    "#SBATCH -o job.%j.out\n",
    "#SBATCH --partition=GPU80G\n",
    "#SBATCH --qos=low\n",
    "#SBATCH -J myFirstGPUJob\n",
    "#SBATCH --nodes=2             \n",
    "#SBATCH --ntasks-per-node=6\n",
    "#SBATCH --gres=gpu:1             \n",
    "\n",
    "nvidia-smi\n",
    "\n",
    "# 开始微调\n",
    "HF_ENDPOINT=https://hf-mirror.com python3 -m llama_recipes.finetuning --use_peft --peft_method lora --quantization --model_name ~/llama/models_hf/7B --output_dir ~/llama/models_hsf/0303PEFT/model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提交命令\n",
    "sbatch --export=HF_ENDPOINT=https://hf-mirror.com job.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **监测任务状态**：在北大超算平台上监测微调任务的执行状态，直至任务完成。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看到job.$j.out文件中有下面的输出，就说明成功了，否则解决碰到的问题\n",
    "\n",
    "***\n",
    "we are about to save the PEFT modules   \n",
    "PEFT modules are saved in llama/models_hsf/0303PEFT/model directory   \n",
    "best eval loss on epoch 3 is 0.014066883362829685   \n",
    "Epoch 3: train_perplexity=1.0157, train_epoch_loss=0.0156, epoch time 2195.944413188845s  \n",
    "Key: avg_train_prep, Value: 1.0356412331263225   \n",
    "Key: avg_train_loss, Value: 0.034785668986539044   \n",
    "Key: avg_eval_prep, Value: 1.0268224477767944   \n",
    "Key: avg_eval_loss, Value: 0.02639332755158345   \n",
    "Key: avg_epoch_time, Value: 2196.3332363131144   \n",
    "Key: avg_checkpoint_time, Value: 0.17571138218045235   \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 查看排队任务\n",
    "squeue -o \"%.18i %.9P %.12j %.12u %.12T %.12M %.16l %.6D %R\" -u $USER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 取消任务\n",
    "scancel 2113606"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **结果分析**：当任务完成后，分析微调结果，如损失函数的变化、模型的准确率等指标。\n",
    "新建jupyter交互任务，将下面代码复制进去后执行，会看到最后的成功结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 重新训练了一轮，1万测试数据，1000条验证数据，老师的测试数据没有在训练数据中，测试微调后的模型编号 0316，老师的8条测试数据通过7条，有一定的泛化能力\n",
    "# \n",
    "# 训练参数如下：\n",
    "# lr=2e-4             # 学习率\n",
    "# lora_rank=64        # LoRA低秩矩阵的维数\n",
    "# lora_alpha=128      #  LoRA低秩矩阵的缩放系数，为一个常数超参，调整alpha与调整学习率类似\n",
    "# lora_dropout=0.05   # LoRA 层的丢弃（dropout）率，取值范围为[0, 1)\n",
    "# gradient_accumulation_steps=1\n",
    "# output_dir=~/llama/models_hsf/0316PEFT/model\n",
    "# target_modules=\"q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj\" \n",
    "#  可训练的 LORA 模块 target_modeules参数解释：\n",
    "#   1. q_proj、k_proj和v_proj是多头注意力机制中的三个线性变换，用于将输入的token映射到一个高维向量空间中，以便于模型对输入进行处理；\n",
    "#   2. o_proj则是多头注意力机制的输出层，它将模型的输出映射到一个概率分布上，以便于模型预测下一个token；\n",
    "#   3. gate_proj通常指的是一个门控（gating）机制，它通过学习一个介于0到1之间的权重来控制信息的流动。这类似于LSTM（长短期记忆网络）或GRU（门控循环单元）中的门控概念，允许模型动态地决定在每个时间步保留或忽略多少信息。在注意力机制中，门控投影也可以用来调节不同成分的重要性。\n",
    "#   4. down_projdown_proj通常表示一个降维操作，它通过一个线性层或卷积层将输入数据的维度减少。这种投影通常用于减少计算量或提取更抽象的特征。例如，在构建一个编码器-解码器结构时，编码器可能会使用降维投影来生成一个压缩表示。\n",
    "#   5. up_proj通常表示一个升维操作，它通过一个线性层或转置卷积层（有时称为反卷积层）将输入数据的维度增加。这种操作在生成型任务中很常见，如在自动编码器的解码器部分或图像分割中用于从低维特征映射回到高维像素空间。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "import torch\n",
    "from transformers import LlamaForCausalLM, LlamaTokenizer\n",
    "from peft import PeftModel, PeftConfig\n",
    "\n",
    "model_id=\"/lustre/home/2301221855/llama/models_hf/7B\"\n",
    "tokenizer = LlamaTokenizer.from_pretrained(model_id)\n",
    "model =LlamaForCausalLM.from_pretrained(model_id, load_in_8bit=True, device_map='auto', torch_dtype=torch.float16)\n",
    "\n",
    "model = PeftModel.from_pretrained(model, \"/lustre/home/2301221855/llama/models_hsf/0316PEFT/model\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_prompt = \"\"\"\n",
    "\n",
    "query:\n",
    "请帮我分析 SmallNewEngland 地区考虑可再生比例，机组备用，电力机组最大容量，氢能 耦合且绿氢比例为 60%的情况\n",
    "answer:\n",
    "\"\"\"\n",
    "\n",
    "model_input = tokenizer(eval_prompt, return_tensors=\"pt\").to(\"cuda\")\n",
    "\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    print(tokenizer.decode(model.generate(**model_input, max_new_tokens=200)[0], skip_special_tokens=True))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Loading checkpoint shards: 100% 2/2 [00:05<00:00,  2.64s/it]\n",
    "\n",
    "\n",
    "query:\n",
    "请帮我分析 SmallNewEngland 地区考虑可再生比例，机组备用，电力机组最大容量，氢能 耦合且绿氢比例为 60%的情况\n",
    "answer:\n",
    " api:dolphyn_ride\n",
    "api_args:['SmallNewEngland',1,1,1,1,1,0.6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## FAQ\n",
    "1. 如碰到 attn_implementation 报错，请手动将llama-recipes的代码中的 attn_implementation 注释，代码如下\n",
    "\n",
    "130     else:\n",
    "131         model = LlamaForCausalLM.from_pretrained(\n",
    "132             train_config.model_name,\n",
    "133             load_in_8bit=True if train_config.quantization else None,\n",
    "134             device_map=\"auto\" if train_config.quantization else None,\n",
    "135             use_cache=use_cache,\n",
    "136             #attn_implementation=\"sdpa\" if train_config.use_fast_kernels else None,\n",
    "137         )\n",
    "\n",
    "2. export的环境变量：\n",
    "HF_ENDPOINT=https://hf-mirror.com \n",
    "PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:2\n",
    "\n",
    "3. jupyter not found错误，在命令行执行：\n",
    "conda install numpy pandas matplotlib ipykernel \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}