notebooks/zh-CN/rag_zephyr_langchain.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kih21u1tyr-I"
      },
      "source": [
        "# 用 Hugging Face Zephyr 和 LangChain 针对 Github issues 构建简单的 RAG\n",
        "\n",
        "_作者: [Maria Khalusova](https://github.com/MKhalusova)_\n",
        "\n",
        "本 notebook 展示了如何使用 [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) 模型和 LangChain 快速构建一个针对项目 GitHub issues 的简单 RAG。\n",
        "\n",
        "\n",
        "\n",
        "**什么是 RAG**\n",
        "\n",
        "RAG 是一个很流行的方法，用来解决强大的 LLM 不知道具体内容的问题，因为具体内容不在其训练数据中，或者当它看到它之前时产生幻觉。这样的具体内容可能是专有的、敏感的，或者，就像这个例子中一样，是最近的和更新的。\n",
        "\n",
        "如果你的数据集是静态的和不需要定期更新的，那么你可能会考虑微调一个大模型。但在大多数情况下，微调模型花费巨大并且重复去微调的话(比如，处理数据漂移的时候)，可能会导致“模型偏移”。这种情况模型行为的变换就不是设计的那样了。\n",
        "\n",
        "**RAG (检索增强生成)** 并不需要模型微调。相反， RAG 通过提供检索到的额外的相关内容喂给 LLM 以此来获得更好的回答。\n",
        "\n",
        "这里是一个简单说明：\n",
        "\n",
        "![RAG diagram](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/rag-diagram.png)\n",
        "\n",
        "* 额外的数据通过独立的嵌入模型会被转化为嵌入向量，这些向量会储存在向量数据库里。嵌入模型通常都比较小，因此在常规偏差上更新嵌入向量相比于微调模型会更快，便宜，和简单。\n",
        "\n",
        "* 与此同时，由于不需要微调，给了你极大的自由度去切换选择你自己的更强的 LLM，或者对于更快速的推理去切换更小的蒸馏模型。\n",
        "\n",
        "让我们用开源的 LLM ，嵌入模型，和 LangChain 快速构建一个针对项目 GitHub issues 的简单 RAG。\n",
        "\n",
        "\n",
        "首先安装相关依赖："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lC9frDOlyi38"
      },
      "outputs": [],
      "source": [
        "!pip install -q torch transformers accelerate bitsandbytes transformers sentence-transformers faiss-gpu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "-aYENQwZ-p_c"
      },
      "outputs": [],
      "source": [
        "# If running in Google Colab, you may need to run this cell to make sure you're using UTF-8 locale to install LangChain\n",
        "import locale\n",
        "locale.getpreferredencoding = lambda: \"UTF-8\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W5HhMZ2c-NfU"
      },
      "outputs": [],
      "source": [
        "!pip install -q langchain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8po01vMWzXL"
      },
      "source": [
        "## 准备数据\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cCmQywC04x6"
      },
      "source": [
        "在这个例子中，我们会从[PEFT 库的仓库](https://github.com/huggingface/peft)加载所有的 issues（包括现在开放的和已经关闭的）。\n",
        "\n",
        "首先，你需要获取一个 [GitHub 个人权限 token](https://github.com/settings/tokens?type=beta) 来访问 GitHub API。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8MoD7NbsNjlM"
      },
      "outputs": [],
      "source": [
        "from getpass import getpass\n",
        "ACCESS_TOKEN = getpass(\"YOUR_GITHUB_PERSONAL_TOKEN\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fccecm3a10N6"
      },
      "source": [
        "下一步，我们将会加载  [huggingface/peft](https://github.com/huggingface/peft) 仓库中所有的 issues:\n",
        "- 默认情况下， PR 也被认定为 issues，这里我们要设置 `include_prs=False` 来排除 PR。\n",
        "- 设置 `state = \"all\"` 意味着我们会把开放和已经关闭的 issues 都加载了。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "8EKMit4WNDY8"
      },
      "outputs": [],
      "source": [
        "from langchain.document_loaders import GitHubIssuesLoader\n",
        "\n",
        "loader = GitHubIssuesLoader(\n",
        "    repo=\"huggingface/peft\",\n",
        "    access_token=ACCESS_TOKEN,\n",
        "    include_prs=False,\n",
        "    state=\"all\"\n",
        ")\n",
        "\n",
        "docs = loader.load()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CChTrY-k2qO5"
      },
      "source": [
        "个人仓库的 issues 内容可能会长于一个嵌入模型可以最为输入处理的长度。如果我们想要嵌入所有可用的内容，我们需要把文档分割成适当大小的块。\n",
        "\n",
        "最普通直接的切块方法就是定义一个固定的块大小，以及判断块之间是否加入重叠。保存一些块之间的重叠允许我们去保存一些语义上下文。\n",
        "\n",
        "其他方法通常更复杂，会考虑到文档的结构和上下文。例如，人们可能希望根据句子或段落来分割文档，然而，固定大小的分块在大多数常见情况下都表现得很好，所以我们将在这里采用这种方法。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OmsXOf59Pmm-"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import CharacterTextSplitter\n",
        "\n",
        "splitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=30)\n",
        "\n",
        "chunked_docs = splitter.split_documents(docs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAt_zPVlXOn7"
      },
      "source": [
        "## 创建嵌入和检索器"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-mvat6JQl4yp"
      },
      "source": [
        "现在所有的文档都设置成立合适的大小，我们可以用他们的嵌入创建一个数据集了。\n",
        "\n",
        "为了创建文档块嵌入，我们将会使用 `HuggingFaceEmbeddings` 和 [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) 嵌入模型。在 Hub 上有许多其他的嵌入模型可用，你也可以查看 [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) 关注表现最好的模型。\n",
        "\n",
        "为了创建向量数据库，我们将会使用 `FAISS` 库。这个库提供高效的相似度搜索和稠密向量的聚类，正是我们需要的。FAISS 目前是大规模数据集上 NN 搜索最常用的库之一。\n",
        "\n",
        "我们通过 LangChain 的 API 来获取嵌入模型和 FAISS 向量数据库。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ixmCdRzBQ5gu"
      },
      "outputs": [],
      "source": [
        "from langchain.vectorstores import FAISS\n",
        "from langchain.embeddings import HuggingFaceEmbeddings\n",
        "\n",
        "db = FAISS.from_documents(chunked_docs,\n",
        "                          HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iCgEPi0nnN6"
      },
      "source": [
        "我们需要一种方式，来返回给定无结构的查询所需要的文档。针对这个，我们会使用 `as_retriever` 方法，使用 `db` 作为支柱：\n",
        "- `search_type=\"similarity\"` 意味着我们会执行查询和文档之间的相似度搜索\n",
        "- `search_kwargs={'k': 4}` 指示我们指定返回的最高的 4 个结果\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "mBTreCQ9noHK"
      },
      "outputs": [],
      "source": [
        "retriever = db.as_retriever(\n",
        "    search_type=\"similarity\",\n",
        "    search_kwargs={'k': 4}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WgEhlISJpTgj"
      },
      "source": [
        "向量数据库和检索器现在设置好了，下一步我们需要设置好链中的下一块 - 模型。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tzQxx0HkXVFU"
      },
      "source": [
        "## 加载量化模型"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jy1cC65p_GD"
      },
      "source": [
        "针对本例，我们选择 [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), 一个小而强大的模型。\n",
        "\n",
        "随着每周都会出好多模型，你可能会想要替换这个模型到最新的最好的模型。最好的方式是查看 [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)。\n",
        "\n",
        "为了推理更快，我们将加载模型的量化版本："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L-ggaa763VRo"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
        "\n",
        "model_name = 'HuggingFaceH4/zephyr-7b-beta'\n",
        "\n",
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVNRJALyXYHG"
      },
      "source": [
        "## 设置 LLM 链"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RUUNneJ1smhl"
      },
      "source": [
        "最后，我们有了所有的需要设置的 LLM 链的部分。\n",
        "\n",
        "首先，使用加载的模型和他的tokenizer创建一个文本生成的流水线(pipeline)\n",
        "\n",
        "下一步，创建一个提示模板-这个应该遵循模型的格式，所以如果你替换了模型检查点，确保使用合适的格式。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "cR0k1cRWz8Pm"
      },
      "outputs": [],
      "source": [
        "from langchain.llms import HuggingFacePipeline\n",
        "from langchain.prompts import PromptTemplate\n",
        "from transformers import pipeline\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "text_generation_pipeline = pipeline(\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    task=\"text-generation\",\n",
        "    temperature=0.2,\n",
        "    do_sample=True,\n",
        "    repetition_penalty=1.1,\n",
        "    return_full_text=True,\n",
        "    max_new_tokens=400,\n",
        ")\n",
        "\n",
        "llm = HuggingFacePipeline(pipeline=text_generation_pipeline)\n",
        "\n",
        "prompt_template = \"\"\"\n",
        "<|system|>\n",
        "Answer the question based on your knowledge. Use the following context to help:\n",
        "\n",
        "{context}\n",
        "\n",
        "</s>\n",
        "<|user|>\n",
        "{question}\n",
        "</s>\n",
        "<|assistant|>\n",
        "\n",
        " \"\"\"\n",
        "\n",
        "prompt = PromptTemplate(\n",
        "    input_variables=[\"context\", \"question\"],\n",
        "    template=prompt_template,\n",
        ")\n",
        "\n",
        "llm_chain = prompt | llm | StrOutputParser()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l19UKq5HXfSp"
      },
      "source": [
        "注意：你也可以使用 `tokenizer.apply_chat_template` 转换列表消息为合适聊天格式的字符串（字典也行  `{'role': 'user', 'content': '(...)'}`）\n",
        "\n",
        "最后，我们需要将 LLM 链与检索器(retriever)结合起来创建一个 RAG 链。我们将原始问题以及检索到的文档上下文传递到最后生成步骤："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "_rI3YNp9Xl4s"
      },
      "outputs": [],
      "source": [
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "retriever = db.as_retriever()\n",
        "\n",
        "rag_chain = (\n",
        " {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
        "    | llm_chain\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UsCOhfDDXpaS"
      },
      "source": [
        "## 比较结果\n",
        "\n",
        "让我们看看对于特定领域库的问题不同的 RAG 的生成的回答。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "W7F07fQLXusU"
      },
      "outputs": [],
      "source": [
        "question = \"How do you combine multiple adapters?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KC0rJYU1x1ir"
      },
      "source": [
        "首先，让我们看看仅仅通过模型自身不加检索内容能得到什么答案:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "id": "GYh-HG1l0De5",
        "outputId": "277d8e89-ce9b-4e04-c11b-639ad2645759"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\" To combine multiple adapters, you need to ensure that they are compatible with each other and the devices you want to connect. Here's how you can do it:\\n\\n1. Identify the adapters you need: Determine which adapters you require to connect the devices you want to use together. For example, if you want to connect a USB-C device to an HDMI monitor, you may need a USB-C to HDMI adapter and a USB-C to USB-A adapter (if your computer only has USB-A ports).\\n\\n2. Connect the first adapter: Plug in the first adapter into the device you want to connect. For instance, if you're connecting a USB-C laptop to an HDMI monitor, plug the USB-C to HDMI adapter into the laptop's USB-C port.\\n\\n3. Connect the second adapter: Next, connect the second adapter to the first one. In this case, connect the USB-C to USB-A adapter to the USB-C port of the USB-C to HDMI adapter.\\n\\n4. Connect the final device: Finally, connect the device you want to use to the second adapter. For example, connect the HDMI cable from the monitor to the HDMI port on the USB-C to HDMI adapter.\\n\\n5. Test the connection: Turn on both devices and check whether everything is working correctly. If necessary, adjust the settings on your devices to ensure optimal performance.\\n\\nBy combining multiple adapters, you can connect a variety of devices together, even if they don't have the same type of connector. Just be sure to choose adapters that are compatible with all the devices you want to connect and test the connection thoroughly before relying on it for critical tasks.\""
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "llm_chain.invoke({\"context\":\"\", \"question\": question})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i-TIWr3wx9w8"
      },
      "source": [
        "可以看到，模型将这个问题解释为关于物理电脑适配器的问题，而在 PEFT 的背景下，“适配器”指的是 LoRA 适配器。\n",
        "让我们看看添加 GitHub issues 的上下文是否有助于模型给出更相关的答案：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "id": "FZpNA3o10H10",
        "outputId": "31f9aed3-3dd7-4ff8-d1a8-866794fefe80"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\" Based on the provided context, it seems that combining multiple adapters is still an open question in the community. Here are some possibilities:\\n\\n  1. Save the output from the base model and pass it to each adapter separately, as described in the first context snippet. This allows you to run multiple adapters simultaneously and reuse the output from the base model. However, this approach requires loading and running each adapter separately.\\n\\n  2. Export everything into a single PyTorch model, as suggested in the second context snippet. This would involve saving all the adapters and their weights into a single model, potentially making it larger and more complex. The advantage of this approach is that it would allow you to run all the adapters simultaneously without having to load and run them separately.\\n\\n  3. Merge multiple Lora adapters, as mentioned in the third context snippet. This involves adding multiple distinct, independent behaviors to a base model by merging multiple Lora adapters. It's not clear from the context how this would be done, but it suggests that there might be a recommended way of doing it.\\n\\n  4. Combine adapters through a specific architecture, as proposed in the fourth context snippet. This involves merging multiple adapters into a single architecture, potentially creating a more complex model with multiple behaviors. Again, it's not clear from the context how this would be done.\\n\\n   Overall, combining multiple adapters is still an active area of research, and there doesn't seem to be a widely accepted solution yet. If you're interested in exploring this further, it might be worth reaching out to the Hugging Face community or checking out their documentation for more information.\""
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rag_chain.invoke(question)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hZQedZKSyrwO"
      },
      "source": [
        "我们可以看到，加入检索的信息后，同一个模型能够对于特定库的问题给出更准确、更相关的答案。\n",
        "\n",
        "值得注意的是，将多个适配器结合用于推理的功能已经被添加到库中，人们可以在文档中找到这些信息，因此在下一个迭代的RAG中，包含文档嵌入可能是有价值的。"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}