06_sanity_check.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ad895f4-b3e0-4d9e-86d5-9cbfcb85fa75",
   "metadata": {},
   "source": [
    "# Sanity Check: Local Inference\n",
    "\n",
    "You need to generate predictions so you can test the model.  Axolotl uploaded the trained model to\n",
    "\n",
    "[parlance-labs/hc-mistral-alpaca](https://huggingface.co/parlance-labs/hc-mistral-alpaca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "858e5904-d02f-43da-9e5d-304ac888c89b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b05cd8c38364cc0a4614dcbe5159f04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "adapter_config.json:   0%|          | 0.00/654 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4edd69315eac45c482e1d1d4f863b575",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors.index.json:   0%|          | 0.00/25.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87dfbe78acd947f8a3c1a83364c3f69f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd0204f4db2e498aa3d8bfbfed6bac36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00002.safetensors:   0%|          | 0.00/9.94G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37b7ab8f5acb42ea99d42d34fe5d9bf0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00002.safetensors:   0%|          | 0.00/4.54G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc2a1c019f2541f1a72fa3addf7cd424",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d624e5ed9ca4e88b5ff7718abdf6cec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48a931ae0cd44cf8b0127adca1d7954e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/1.02k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b59a115d75b4731b2ef5e7f70bcf052",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.model:   0%|          | 0.00/493k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a22310b9eba472fa992f732982f4def",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/437 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47dc13b3af944f91a615dbd3917a3d39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "adapter_model.safetensors:   0%|          | 0.00/336M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from peft import AutoPeftModelForCausalLM\n",
    "from transformers import AutoTokenizer\n",
    "model_id='parlance-labs/hc-mistral-alpaca' # this will be different for you based upon hub_model_id\n",
    "model = AutoPeftModelForCausalLM.from_pretrained(model_id).cuda()\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "tokenizer.pad_token = tokenizer.eos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e487f417-b4f4-4fe2-a90a-2bb5ef7a98a6",
   "metadata": {},
   "source": [
    "### Prompt Template\n",
    "\n",
    "Next, we have to construct a prompt template that is as close as possible to the prompt template we saw earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ddef1949-74c5-4806-963c-a9f831707109",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prompt(nlq, cols):\n",
    "    return f\"\"\"Honeycomb is an observability platform that allows you to write queries to inspect trace data. You are an assistant that takes a natural language query (NLQ) and a list of valid columns and produce a Honeycomb query.\n",
    "\n",
    "### Instruction:\n",
    "\n",
    "NLQ: \"{nlq}\"\n",
    "\n",
    "Columns: {cols}\n",
    "\n",
    "### Response:\n",
    "\"\"\"\n",
    "\n",
    "def prompt_tok(nlq, cols, return_ids=False):\n",
    "    _p = prompt(nlq, cols)\n",
    "    input_ids = tokenizer(_p, return_tensors=\"pt\", truncation=True).input_ids.cuda()\n",
    "    out_ids = model.generate(input_ids=input_ids, max_new_tokens=5000, \n",
    "                          do_sample=False)\n",
    "    ids = out_ids.detach().cpu().numpy()\n",
    "    if return_ids: return out_ids\n",
    "    return tokenizer.batch_decode(ids, \n",
    "                                  skip_special_tokens=True)[0][len(_p):]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cba7d50e-a089-4339-a9f0-528dbe287249",
   "metadata": {},
   "source": [
    "## Sanity Check Examples\n",
    "\n",
    "Next, we sanity check a few examples to make sure that:\n",
    "\n",
    "- Our prompt template is constructed correctly (indeed I made some mistakes at first!)\n",
    "- The model is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3e8afc19-a87f-4e7f-8e36-16c35c93864c",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlq = \"Exception count by exception and caller\"\n",
    "cols = ['error', 'exception.message', 'exception.type', 'exception.stacktrace', 'SampleRate', 'name', 'db.user', 'type', 'duration_ms', 'db.name', 'service.name', 'http.method', 'db.system', 'status_code', 'db.operation', 'library.name', 'process.pid', 'net.transport', 'messaging.system', 'rpc.system', 'http.target', 'db.statement', 'library.version', 'status_message', 'parent_name', 'aws.region', 'process.command', 'rpc.method', 'span.kind', 'serializer.name', 'net.peer.name', 'rpc.service', 'http.scheme', 'process.runtime.name', 'serializer.format', 'serializer.renderer', 'net.peer.port', 'process.runtime.version', 'http.status_code', 'telemetry.sdk.language', 'trace.parent_id', 'process.runtime.description', 'span.num_events', 'messaging.destination', 'net.peer.ip', 'trace.trace_id', 'telemetry.instrumentation_library', 'trace.span_id', 'span.num_links', 'meta.signal_type', 'http.route']\n",
    "\n",
    "nlq2 = \"ssv-customer-identification\"\n",
    "cols2 = ['ssv_debit_card_fast_processing_sli', 'ssv_debit_card_fast_processing_sli_exclude_warmup', 'fincrime_ems_success_sli', 'event_timestamp', 'fincrime_ems_sync_sla_upheld_rate', 'fincrime_ems_sync_sla_missed_rate_sli', 'fincrime_ems_success', 'interbank_bacs_debit_payments_successfully_processed_on_processing_date', 'interbank_bacs_credit_payments_successfully_processed_on_processing_date', 'date_iso', 'CustomerIdentificationExists', 'interbank_saga_processes_success_rate', 'fincrime_ems_sync_sla_missed_rate', 'fincrime_ems_success_rate', 'interbank_inbound_fps_payment_latency', 'sagaStatusExists', 'timestamp.is_nine_to_five', 'timestamp.event_hour', 'did_skitty_onboarding_fail', 'timestamp.is_weekend', 'onfido.api.url', 'is_weekend', 'isPending', 'was_skitty_onboarding_successful', 'trace.parent_id', 'duration_ms', 'http.status_code', 'name', 'service.name', 'exception.message', 'parent_name', 'db.statement', 'error', 'http.route']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0f3050d2-758f-45e1-add1-43d9e73bb257",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exception count by exception and caller \n",
      " \"{'breakdowns': ['exception.message', 'exception.type'], 'calculations': [{'op': 'COUNT'}], 'filters': [{'column': 'exception.message', 'op': 'exists'}, {'column': 'exception.type', 'op': 'exists'}], 'orders': [{'op': 'COUNT', 'order': 'descending'}], 'time_range': 7200}\"\n"
     ]
    }
   ],
   "source": [
    "out = prompt_tok(nlq, cols)\n",
    "print(nlq, '\\n', out.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8600b0f-802c-43a5-b709-f0b5b7a15d81",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exception count by exception and caller \n",
      " \"{'breakdowns': ['ssv_debit_card_fast_processing_sli', 'ssv_debit_card_fast_processing_sli_exclude_warmup'], 'calculations': [{'op': 'COUNT'}], 'filters': [{'column': 'ssv_debit_card_fast_processing_sli', 'op': 'exists'}, {'column': 'ssv_debit_card_fast_processing_sli_exclude_warmup', 'op': 'exists'}], 'orders': [{'op': 'COUNT', 'order': 'descending'}], 'time_range': 7200}\"\n"
     ]
    }
   ],
   "source": [
    "out = prompt_tok(nlq2, cols2)\n",
    "print(nlq, '\\n', out.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e70980d-7abd-4987-92ab-452c315a14f0",
   "metadata": {},
   "source": [
    "## Homework - Token Check\n",
    "\n",
    "Optional Homework!  Check that there aren't any token issues https://hamel.dev/notes/llm/finetuning/05_tokenizer_gotchas.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcee5d46-eb8a-4ffb-b805-5ed240c1c86e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}