02-a-indexing-vectorized-content.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📚 Prerequisites\n",
    "\n",
    "Before running this notebook, ensure you have configured Azure AI services, set the appropriate configuration parameters, and set up a Conda environment to ensure reproducibility. You can find the setup instructions and how to create a Conda environment in the [REQUIREMENTS.md](REQUIREMENTS.md) file.\n",
    "\n",
    "## 📋 Table of Contents\n",
    "\n",
    "This notebook guides you through the following sections:\n",
    "\n",
    "> **💡 Note:** Please refer to the notebook `01-creation-indexes.ipynb` for detailed information and steps on how to create Azure AI Search Indexes.\n",
    "\n",
    "1. [**Indexing Vectorized Content from multiple formats and sources**](#index-documents)\n",
    "    - Chunk, vectorize, and index local PDF files and website addresses.\n",
    "    - Download, chunk, vectorize, and index all `.docx` files from a SharePoint site.\n",
    "    \n",
    "2. [**Indexing Vectorized Content from complex layout documents laveraging OCR Capabilities**](#index-images)\n",
    "    - Leverage complex OCR, image recognition using Azure Document Intelligence. Chunk, vectorize, and index extracted metadata from Dcouments\n",
    "\n",
    "3. [**Indexing Vectorized Content from Audio**](#index-audio) (TODO)\n",
    "    - Process WAV audio data using Azure AI Speech transalations capabilities, chunk, vectorize, and index audio files stored in Blob Storage and indexed in Azure AI Search.\n",
    "\n",
    "Before you start, please take a look at the README. It contains detailed instructions, diagrams, and information about the class structure and automation used in this project, specifically in the `AzureAIIndexer` backend. [README.md](README.md) file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory changed to C:\\Users\\pablosal\\Desktop\\gbbai-azure-ai-search-indexing\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# Define the target directory\n",
    "target_directory = r\"C:\\Users\\pablosal\\Desktop\\gbbai-azure-ai-search-indexing\"  # change your directory here\n",
    "\n",
    "# Check if the directory exists\n",
    "if os.path.exists(target_directory):\n",
    "    # Change the current working directory\n",
    "    os.chdir(target_directory)\n",
    "    print(f\"Directory changed to {os.getcwd()}\")\n",
    "else:\n",
    "    print(f\"Directory {target_directory} does not exist.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Azure AI Search Indexes \n",
    "\n",
    "Please refer to the notebook [01-creation-indexes.ipynb](01-creation-indexes.ipynb) for detailed information and steps on how to create Azure AI Search Indexes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📚 Indexing Vectorized Content from Multiple Sources and Various Formats\n",
    "\n",
    "In this section, we will explore how to index vectorized content from various sources and in different formats. This includes local PDF files, website addresses, `.docx` files from a SharePoint site, and more. We will chunk, vectorize, and index these different types of content, leveraging the power of Azure AI Search Indexes. This process allows us to create a comprehensive, searchable index that can handle a wide range of queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-13 10:41:34,678 - micro - MainProcess - INFO     Loading OpenAIEmbeddings object with model, deployment foundational-canadaeast-ada, and chunk size 1000 (ai_search_indexing.py:load_embedding_model:162)\n",
      "c:\\Users\\pablosal\\AppData\\Local\\anaconda3\\envs\\vector-indexing-azureaisearch\\lib\\site-packages\\langchain_core\\_api\\deprecation.py:189: LangChainDeprecationWarning: The class `AzureOpenAIEmbeddings` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use langchain_openai.AzureOpenAIEmbeddings instead.\n",
      "  warn_deprecated(\n",
      "c:\\Users\\pablosal\\AppData\\Local\\anaconda3\\envs\\vector-indexing-azureaisearch\\lib\\site-packages\\langchain_core\\_api\\deprecation.py:189: LangChainDeprecationWarning: The class `OpenAIEmbeddings` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use langchain_openai.OpenAIEmbeddings instead.\n",
      "  warn_deprecated(\n",
      "2024-03-13 10:41:42,766 - micro - MainProcess - INFO     AzureOpenAIEmbeddings object has been created successfully. You can now access the embeddings\n",
      "                using the '.embeddings' attribute. (ai_search_indexing.py:load_embedding_model:175)\n",
      "vector_search_configuration is not a known attribute of class <class 'azure.search.documents.indexes.models._index.SearchField'> and will be ignored\n",
      "2024-03-13 10:41:46,660 - micro - MainProcess - INFO     The Azure AI search index 'test-index-002' has been loaded correctly. (ai_search_indexing.py:load_azureai_index:226)\n",
      "2024-03-13 10:41:46,681 - micro - MainProcess - INFO     Successfully loaded environment variables: TENANT_ID, CLIENT_ID, CLIENT_SECRET (sharepoint_data_extractor.py:load_environment_variables_from_env_file:87)\n",
      "2024-03-13 10:41:47,342 - micro - MainProcess - INFO     New access token retrieved. (sharepoint_data_extractor.py:msgraph_auth:121)\n"
     ]
    }
   ],
   "source": [
    "# Import the AzureAIndexer class from the ai_search_indexing module\n",
    "from src.indexers.ai_search_indexing import AzureAIndexer\n",
    "\n",
    "DEPLOYMENT_NAME = os.getenv(\"AZURE_AOAI_EMBEDDING_DEPLOYMENT_ID\")\n",
    "INDEX_NAME = os.getenv(\"AZURE_SEARCH_INDEX_NAME_DOCUMENTS\")\n",
    "\n",
    "# Create an instance of the AzureAIndexer class\n",
    "azure_search_indexer_client = AzureAIndexer(\n",
    "    index_name=INDEX_NAME, embedding_azure_deployment_name=DEPLOYMENT_NAME\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing PDFs, DOCX, and Images from Blob Storage \n",
    "\n",
    "The `load_files_and_split_into_chunks` function is a powerful tool for indexing and processing documents. It is designed to streamline the initial steps of loading files, splitting them into manageable chunks, and preparing your documents for further processing and conversion. \n",
    "\n",
    "Here are its key features:\n",
    "\n",
    "- **Multi-Format Support**: The function can process documents in different formats (PDFs, Word documents, images, etc.) from various sources (blob storage, URLs, local paths). You can pass a list of file paths, each possibly in a different format.\n",
    "\n",
    "- **Automated File Loading**: The function efficiently loads files into memory, eliminating the need for manual file handling. It manages the reading and processing of each file.\n",
    "\n",
    "- **Advanced Text Splitting**: After loading, the function splits the text into manageable chunks, crucial for processing large documents. You can customize the chunk size and overlap according to your needs.\n",
    "\n",
    "- **Versatile Splitting Options**: You can choose from various splitters - 'by_title', 'by_character_recursive', 'by_character_brute_force' - to fit your specific text processing requirements.\n",
    "\n",
    "- **Encoding Capabilities**: The function can optionally use an encoder during splitting. This feature is particularly useful for certain text analysis tasks. You can specify the model used for encoding (default is \"gpt-4\").\n",
    "\n",
    "- **OCR Capabilities**: If the 'ocr' parameter is set to True, the function will use Optical Character Recognition (OCR) via Azure Document Intelligence to extract text from images or scanned documents.\n",
    "\n",
    "- **Verbose Logging**: You can enable detailed logging for in-depth progress tracking and easier debugging.\n",
    "\n",
    "- **High Customizability**: The function's behavior can be tailored to your needs with additional keyword arguments. This includes options like retaining separators in chunks, using separators as regex patterns, and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:09:17,518 - micro - MainProcess - INFO     Reading .pdf file from local path C:\\Users\\pablosal\\Desktop\\gbbai-azure-ai-search-indexing\\utils\\data\\autogen.pdf. (from_blob.py:load_document:67)\n",
      "2024-03-05 16:09:17,519 - micro - MainProcess - INFO     Loading file with Loader PyPDFLoader (from_blob.py:load_document:79)\n",
      "2024-03-05 16:09:20,866 - micro - MainProcess - INFO     Reading .pdf file from https://arxiv.org/pdf/2308.08155.pdf. (from_blob.py:load_document:75)\n",
      "2024-03-05 16:09:20,867 - micro - MainProcess - INFO     Loading file with Loader PyPDFLoader (from_blob.py:load_document:79)\n",
      "2024-03-05 16:09:24,994 - micro - MainProcess - INFO     Successfully downloaded blob file autogen.pdf (blob_data_extractors.py:extract_content:93)\n",
      "2024-03-05 16:09:25,034 - micro - MainProcess - INFO     Reading .pdf file from temporary location C:\\Users\\pablosal\\AppData\\Local\\Temp\\tmpt7zsi48i originally sourced from https://testeastusdev001.blob.core.windows.net/testretrieval/autogen.pdf. (from_blob.py:load_document:63)\n",
      "2024-03-05 16:09:25,036 - micro - MainProcess - INFO     Loading file with Loader PyPDFLoader (from_blob.py:load_document:79)\n",
      "2024-03-05 16:09:26,909 - micro - MainProcess - INFO     Deleted temporary file: C:\\Users\\pablosal\\AppData\\Local\\Temp\\tmpt7zsi48i (from_blob.py:load_document_from_bytes:132)\n",
      "2024-03-05 16:09:26,910 - micro - MainProcess - INFO     Reading .docx file from local path C:\\Users\\pablosal\\Desktop\\gbbai-azure-ai-search-indexing\\utils\\data\\test.docx. (from_blob.py:load_document:67)\n",
      "2024-03-05 16:09:26,911 - micro - MainProcess - INFO     Loading file with Loader Docx2txtLoader (from_blob.py:load_document:79)\n",
      "2024-03-05 16:09:27,167 - micro - MainProcess - INFO     Successfully downloaded blob file test.docx (blob_data_extractors.py:extract_content:93)\n",
      "2024-03-05 16:09:27,206 - micro - MainProcess - INFO     Reading .docx file from temporary location C:\\Users\\pablosal\\AppData\\Local\\Temp\\tmpu5mabavh originally sourced from https://testeastusdev001.blob.core.windows.net/testretrieval/test.docx. (from_blob.py:load_document:63)\n",
      "2024-03-05 16:09:27,208 - micro - MainProcess - INFO     Loading file with Loader Docx2txtLoader (from_blob.py:load_document:79)\n",
      "2024-03-05 16:09:27,392 - micro - MainProcess - INFO     Deleted temporary file: C:\\Users\\pablosal\\AppData\\Local\\Temp\\tmpu5mabavh (from_blob.py:load_document_from_bytes:132)\n",
      "2024-03-05 16:09:27,394 - micro - MainProcess - INFO     Creating a splitter of type: by_character_recursive (by_character.py:get_splitter:63)\n",
      "2024-03-05 16:09:27,394 - micro - MainProcess - INFO     Obtained splitter of type: RecursiveCharacterTextSplitter (by_character.py:split_documents_in_chunks_from_documents:175)\n",
      "2024-03-05 16:09:27,479 - micro - MainProcess - INFO     Number of chunks obtained: 1408 (by_character.py:split_documents_in_chunks_from_documents:178)\n"
     ]
    }
   ],
   "source": [
    "# Define file paths and URLs\n",
    "local_pdf_path = \"utils/data/autogen.pdf\"\n",
    "remote_pdf_url = \"https://arxiv.org/pdf/2308.08155.pdf\"\n",
    "blob_pdf_url = (\n",
    "    \"https://testeastusdev001.blob.core.windows.net/testretrieval/autogen.pdf\"\n",
    ")\n",
    "local_word_path = \"utils/data/test.docx\"\n",
    "remote_word_url = (\n",
    "    \"https://testeastusdev001.blob.core.windows.net/testretrieval/test.docx\"\n",
    ")\n",
    "\n",
    "# Combine all paths and URLs into a list. This is optional if you want to process multiple files at once.\n",
    "# It will also work by passing a string for simple file processing.\n",
    "file_sources = [\n",
    "    local_pdf_path,\n",
    "    remote_pdf_url,\n",
    "    blob_pdf_url,\n",
    "    local_word_path,\n",
    "    remote_word_url,\n",
    "]\n",
    "\n",
    "# Define parameters for the load_files_and_split_into_chunks function\n",
    "splitter_params = {\n",
    "    \"splitter_type\": \"by_character_recursive\",\n",
    "    \"use_encoder\": False,\n",
    "    \"chunk_size\": 512,\n",
    "    \"chunk_overlap\": 128,\n",
    "    \"verbose\": False,\n",
    "    \"keep_separator\": True,\n",
    "    \"is_separator_regex\": False,\n",
    "    \"model_name\": \"gpt-4\",\n",
    "}\n",
    "\n",
    "# Load files and split them into chunks\n",
    "document_chunks_to_index = azure_search_indexer_client.load_files_and_split_into_chunks(\n",
    "    file_paths=file_sources, **splitter_params\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:09:50,886 - micro - MainProcess - INFO     Embedding and indexing initiated for 1408 text chunks. (ai_search_indexing.py:index_text_embeddings:498)\n",
      "2024-03-05 16:12:09,347 - micro - MainProcess - INFO     Embedding and indexing completed for 1408 text chunks. (ai_search_indexing.py:index_text_embeddings:502)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Index the document chunks using the Azure Search Indexer client\n",
    "azure_search_indexer_client.index_text_embeddings(document_chunks_to_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing Pdfs and Docs from Sharepoint\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_names = [\"testdocx.docx\", \"autogen.pdf\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:12:09,410 - micro - MainProcess - INFO     Getting the Site ID... (sharepoint_data_extractor.py:get_site_id:191)\n",
      "2024-03-05 16:12:10,026 - micro - MainProcess - ERROR    HTTP Error: 403 Client Error: Forbidden for url: https://graph.microsoft.com/v1.0/sites/mngenvmcap747548.sharepoint.com:/sites/Contoso:/ (sharepoint_data_extractor.py:_make_ms_graph_request:173)\n",
      "2024-03-05 16:12:10,028 - micro - MainProcess - ERROR    Error retrieving Site ID: 403 Client Error: Forbidden for url: https://graph.microsoft.com/v1.0/sites/mngenvmcap747548.sharepoint.com:/sites/Contoso:/ (sharepoint_data_extractor.py:get_site_id:198)\n",
      "2024-03-05 16:12:10,032 - micro - MainProcess - ERROR    Failed to retrieve site_id (sharepoint_data_extractor.py:get_site_and_drive_ids:469)\n",
      "2024-03-05 16:12:10,033 - micro - MainProcess - ERROR    Error loading file testdocx.docx: Site ID or Drive ID is missing. (from_sharepoint.py:load_documents:243)\n",
      "2024-03-05 16:12:10,035 - micro - MainProcess - INFO     Getting the Site ID... (sharepoint_data_extractor.py:get_site_id:191)\n",
      "2024-03-05 16:12:10,597 - micro - MainProcess - ERROR    HTTP Error: 403 Client Error: Forbidden for url: https://graph.microsoft.com/v1.0/sites/mngenvmcap747548.sharepoint.com:/sites/Contoso:/ (sharepoint_data_extractor.py:_make_ms_graph_request:173)\n",
      "2024-03-05 16:12:10,607 - micro - MainProcess - ERROR    Error retrieving Site ID: 403 Client Error: Forbidden for url: https://graph.microsoft.com/v1.0/sites/mngenvmcap747548.sharepoint.com:/sites/Contoso:/ (sharepoint_data_extractor.py:get_site_id:198)\n",
      "2024-03-05 16:12:10,613 - micro - MainProcess - ERROR    Failed to retrieve site_id (sharepoint_data_extractor.py:get_site_and_drive_ids:469)\n",
      "2024-03-05 16:12:10,615 - micro - MainProcess - ERROR    Error loading file autogen.pdf: Site ID or Drive ID is missing. (from_sharepoint.py:load_documents:243)\n",
      "2024-03-05 16:12:10,616 - micro - MainProcess - ERROR    No documents were loaded. (from_sharepoint.py:load_documents:246)\n",
      "2024-03-05 16:12:10,617 - micro - MainProcess - INFO     Creating a splitter of type: by_character_recursive (by_character.py:get_splitter:63)\n",
      "2024-03-05 16:12:10,618 - micro - MainProcess - INFO     Obtained splitter of type: RecursiveCharacterTextSplitter (by_character.py:split_documents_in_chunks_from_documents:175)\n",
      "2024-03-05 16:12:10,619 - micro - MainProcess - INFO     Number of chunks obtained: 0 (by_character.py:split_documents_in_chunks_from_documents:178)\n"
     ]
    }
   ],
   "source": [
    "# Define parameters for the load_files_and_split_into_chunks function\n",
    "splitter_params = {\n",
    "    \"splitter_type\": \"by_character_recursive\",\n",
    "    \"use_encoder\": False,\n",
    "    \"chunk_size\": 512,\n",
    "    \"chunk_overlap\": 128,\n",
    "    \"verbose\": False,\n",
    "    \"keep_separator\": True,\n",
    "    \"is_separator_regex\": False,\n",
    "    \"model_name\": \"gpt-4\",\n",
    "}\n",
    "\n",
    "document_chunks_to_index = (\n",
    "    azure_search_indexer_client.load_files_and_split_into_chunks_from_sharepoint(\n",
    "        site_domain=os.environ[\"SITE_DOMAIN\"],\n",
    "        site_name=os.environ[\"SITE_NAME\"],\n",
    "        file_names=file_names,\n",
    "        **splitter_params,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:12:10,658 - micro - MainProcess - INFO     Embedding and indexing initiated for 0 text chunks. (ai_search_indexing.py:index_text_embeddings:498)\n",
      "2024-03-05 16:12:10,661 - micro - MainProcess - INFO     Embedding and indexing completed for 0 text chunks. (ai_search_indexing.py:index_text_embeddings:502)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Index the document chunks using the Azure Search Indexer client\n",
    "azure_search_indexer_client.index_text_embeddings(document_chunks_to_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📚 Indexing Vectorized Content from Complex Layout Documents Leveraging OCR Capabilities\n",
    "\n",
    "In this section, we will be using Azure's Document Intelligence in the backend to extract elements from complex layout documents. This process involves extracting the title and other metadata from the documents, which allows us to chunk the document by sections. \n",
    "\n",
    "We use a simple algorithm that chunks the document based on sections. This approach ensures that each chunk is semantically coherent and can be indexed separately. However, there's a risk that some sections might be longer than the context window of our model. To mitigate this, we can add an additional layer to the algorithm that further chunks sections based on a cutoff count number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_blob = \"https://testeastusdev001.blob.core.windows.net/customskillspdf/instruction-manual-fieldvue-dvc6200-hw2-digital-valve-controller-en-123052.pdf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:12:10,726 - micro - MainProcess - INFO     Blob URL detected. Extracting content. (ocr_document_intelligence.py:analyze_document:147)\n",
      "2024-03-05 16:12:12,043 - micro - MainProcess - INFO     Successfully downloaded blob file instruction-manual-fieldvue-dvc6200-hw2-digital-valve-controller-en-123052.pdf (blob_data_extractors.py:extract_content:93)\n",
      "2024-03-05 16:12:45,280 - micro - MainProcess - INFO     Successfully extracted content from https://testeastusdev001.blob.core.windows.net/customskillspdf/instruction-manual-fieldvue-dvc6200-hw2-digital-valve-controller-en-123052.pdf (ocr_data_extractors.py:extract_content:82)\n",
      "2024-03-05 16:12:45,285 - micro - MainProcess - INFO     Section headings: ['## Scope of Manual', '## Conventions Used in this Manual', '## Description', '## Specifications', '## Related Documents', '### Table 1-2. Specifications', '### Communication Protocol', '#### Input Signal', '### Output Signal', '#### Table 1-2. Specifications (continued)', '##### Connections'] (by_title.py:split_text_by_headings:34)\n",
      "2024-03-05 16:12:45,286 - micro - MainProcess - INFO     Number of chunks: 13 (by_title.py:split_text_by_headings:35)\n",
      "2024-03-05 16:12:45,289 - micro - MainProcess - INFO     Processed chunk 1 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,298 - micro - MainProcess - INFO     Processed chunk 2 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,302 - micro - MainProcess - INFO     Processed chunk 3 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,313 - micro - MainProcess - INFO     Processed chunk 4 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,320 - micro - MainProcess - INFO     Processed chunk 5 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,323 - micro - MainProcess - INFO     Processed chunk 6 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,327 - micro - MainProcess - INFO     Processed chunk 7 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,331 - micro - MainProcess - INFO     Processed chunk 8 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,334 - micro - MainProcess - INFO     Processed chunk 9 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,340 - micro - MainProcess - INFO     Processed chunk 10 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,343 - micro - MainProcess - INFO     Processed chunk 11 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,348 - micro - MainProcess - INFO     Processed chunk 12 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,352 - micro - MainProcess - INFO     Processed chunk 13 of 13 (by_title.py:combine_chunks:58)\n",
      "2024-03-05 16:12:45,354 - micro - MainProcess - INFO     Processed document 1 of 1 (by_title.py:split_documents_in_chunks_from_documents:104)\n"
     ]
    }
   ],
   "source": [
    "# Define parameters for the load_files_and_split_into_chunks function\n",
    "splitter_params = {\n",
    "    \"splitter_type\": \"by_title\",\n",
    "    \"ocr\": True,\n",
    "    \"ocr_output_format\": \"markdown\",\n",
    "    \"pages\": \"3-7\",\n",
    "}\n",
    "\n",
    "document_chunks_to_index = azure_search_indexer_client.load_files_and_split_into_chunks(\n",
    "    file_paths=document_blob,\n",
    "    **splitter_params,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:12:45,385 - micro - MainProcess - INFO     Embedding and indexing initiated for 7 text chunks. (ai_search_indexing.py:index_text_embeddings:498)\n",
      "2024-03-05 16:12:46,434 - micro - MainProcess - INFO     Embedding and indexing completed for 7 text chunks. (ai_search_indexing.py:index_text_embeddings:502)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Index the document chunks using the Azure Search Indexer client\n",
    "azure_search_indexer_client.index_text_embeddings(document_chunks_to_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here, we are extracting the text from complex documents using OCR and passing it to a batched recursive iterator, sectioned by title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:12:46,459 - micro - MainProcess - INFO     Blob URL detected. Extracting content. (ocr_document_intelligence.py:analyze_document:147)\n",
      "2024-03-05 16:12:46,666 - micro - MainProcess - INFO     Successfully downloaded blob file instruction-manual-fieldvue-dvc6200-hw2-digital-valve-controller-en-123052.pdf (blob_data_extractors.py:extract_content:93)\n",
      "2024-03-05 16:13:18,284 - micro - MainProcess - INFO     Successfully extracted content from https://testeastusdev001.blob.core.windows.net/customskillspdf/instruction-manual-fieldvue-dvc6200-hw2-digital-valve-controller-en-123052.pdf (ocr_data_extractors.py:extract_content:82)\n",
      "2024-03-05 16:13:18,286 - micro - MainProcess - INFO     Creating a splitter of type: by_character_recursive (by_character.py:get_splitter:63)\n",
      "2024-03-05 16:13:18,287 - micro - MainProcess - INFO     Obtained splitter of type: RecursiveCharacterTextSplitter (by_character.py:split_documents_in_chunks_from_documents:175)\n",
      "2024-03-05 16:13:18,290 - micro - MainProcess - INFO     Number of chunks obtained: 38 (by_character.py:split_documents_in_chunks_from_documents:178)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chunk Number: 1, Character Count: 504, Token Count: 107\n",
      "Chunk Number: 2, Character Count: 339, Token Count: 63\n",
      "Chunk Number: 3, Character Count: 362, Token Count: 65\n",
      "Chunk Number: 4, Character Count: 511, Token Count: 96\n",
      "Chunk Number: 5, Character Count: 472, Token Count: 95\n",
      "Chunk Number: 6, Character Count: 510, Token Count: 99\n",
      "Chunk Number: 7, Character Count: 183, Token Count: 36\n",
      "Chunk Number: 8, Character Count: 285, Token Count: 83\n",
      "Chunk Number: 9, Character Count: 409, Token Count: 91\n",
      "Chunk Number: 10, Character Count: 304, Token Count: 55\n",
      "Chunk Number: 11, Character Count: 380, Token Count: 70\n",
      "Chunk Number: 12, Character Count: 494, Token Count: 159\n",
      "Chunk Number: 13, Character Count: 487, Token Count: 142\n",
      "Chunk Number: 14, Character Count: 242, Token Count: 54\n",
      "Chunk Number: 15, Character Count: 499, Token Count: 140\n",
      "Chunk Number: 16, Character Count: 188, Token Count: 56\n",
      "Chunk Number: 17, Character Count: 454, Token Count: 88\n",
      "Chunk Number: 18, Character Count: 456, Token Count: 113\n",
      "Chunk Number: 19, Character Count: 479, Token Count: 123\n",
      "Chunk Number: 20, Character Count: 503, Token Count: 129\n",
      "Chunk Number: 21, Character Count: 376, Token Count: 88\n",
      "Chunk Number: 22, Character Count: 443, Token Count: 104\n",
      "Chunk Number: 23, Character Count: 265, Token Count: 80\n",
      "Chunk Number: 24, Character Count: 505, Token Count: 125\n",
      "Chunk Number: 25, Character Count: 370, Token Count: 99\n",
      "Chunk Number: 26, Character Count: 371, Token Count: 92\n",
      "Chunk Number: 27, Character Count: 415, Token Count: 113\n",
      "Chunk Number: 28, Character Count: 410, Token Count: 146\n",
      "Chunk Number: 29, Character Count: 465, Token Count: 155\n",
      "Chunk Number: 30, Character Count: 391, Token Count: 109\n",
      "Chunk Number: 31, Character Count: 343, Token Count: 69\n",
      "Chunk Number: 32, Character Count: 429, Token Count: 112\n",
      "Chunk Number: 33, Character Count: 477, Token Count: 137\n",
      "Chunk Number: 34, Character Count: 450, Token Count: 113\n",
      "Chunk Number: 35, Character Count: 467, Token Count: 104\n",
      "Chunk Number: 36, Character Count: 460, Token Count: 108\n",
      "Chunk Number: 37, Character Count: 373, Token Count: 121\n",
      "Chunk Number: 38, Character Count: 295, Token Count: 87\n"
     ]
    }
   ],
   "source": [
    "# Define parameters for the load_files_and_split_into_chunks function\n",
    "splitter_params = {\n",
    "    \"splitter_type\": \"by_character_recursive\",\n",
    "    \"ocr\": True,\n",
    "    \"ocr_output_format\": \"text\",\n",
    "    \"pages\": \"3-7\",\n",
    "    \"use_encoder\": False,\n",
    "    \"chunk_size\": 512,\n",
    "    \"chunk_overlap\": 128,\n",
    "    \"verbose\": False,\n",
    "    \"keep_separator\": True,\n",
    "    \"is_separator_regex\": False,\n",
    "    \"verbose\": True,\n",
    "}\n",
    "\n",
    "document_chunks_to_index = azure_search_indexer_client.load_files_and_split_into_chunks(\n",
    "    file_paths=document_blob,\n",
    "    **splitter_params,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-05 16:13:18,334 - micro - MainProcess - INFO     Embedding and indexing initiated for 38 text chunks. (ai_search_indexing.py:index_text_embeddings:498)\n",
      "2024-03-05 16:13:22,068 - micro - MainProcess - INFO     Embedding and indexing completed for 38 text chunks. (ai_search_indexing.py:index_text_embeddings:502)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Index the document chunks using the Azure Search Indexer client\n",
    "azure_search_indexer_client.index_text_embeddings(document_chunks_to_index)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "build-your-own-copilot",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}