intro.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🎓 𝗗𝗦𝗣: 𝗗EMONSTRATE–𝗦EARCH–𝗣REDICT\n",
    "\n",
    "This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
    "\n",
    "Below, we'll build **five** small 𝗗𝗦𝗣 programs that implement strategies for this task. The programs range from extremely simple ones (programs 1 and 2) to moderate ones (programs 3 and 4). Program 5 considers a fairly advanced pipeline."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation\n",
    "\n",
    "If you haven't installed **DSP** already, let's do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: dsp-ml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (0.1.1)\n",
      "Requirement already satisfied: spacy in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (3.5.0)\n",
      "Requirement already satisfied: backoff in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2.2.1)\n",
      "Requirement already satisfied: jupyter in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.0.0)\n",
      "Requirement already satisfied: pandas in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.5.3)\n",
      "Requirement already satisfied: joblib in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (1.2.0)\n",
      "Requirement already satisfied: openai in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (0.26.1)\n",
      "Requirement already satisfied: regex in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from dsp-ml) (2022.10.31)\n",
      "Requirement already satisfied: ipywidgets in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (8.0.4)\n",
      "Requirement already satisfied: notebook in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.5.2)\n",
      "Requirement already satisfied: ipykernel in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.20.2)\n",
      "Requirement already satisfied: jupyter-console in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (6.4.4)\n",
      "Requirement already satisfied: qtconsole in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (5.4.0)\n",
      "Requirement already satisfied: nbconvert in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter->dsp-ml) (7.2.8)\n",
      "Requirement already satisfied: requests>=2.20 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (2.28.2)\n",
      "Requirement already satisfied: tqdm in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (4.64.1)\n",
      "Requirement already satisfied: aiohttp in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from openai->dsp-ml) (3.8.3)\n",
      "Requirement already satisfied: pytz>=2020.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2022.7.1)\n",
      "Requirement already satisfied: numpy>=1.20.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (1.24.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pandas->dsp-ml) (2.8.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.4.5)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.11)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.10.4)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.8)\n",
      "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (8.1.7)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.9)\n",
      "Requirement already satisfied: setuptools in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (65.6.3)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (6.3.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (2.0.7)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.0.4)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.3.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (23.0)\n",
      "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.7.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.0.8)\n",
      "Requirement already satisfied: jinja2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (3.1.2)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (1.1.1)\n",
      "Requirement already satisfied: pathy>=0.10.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from spacy->dsp-ml) (0.10.1)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy->dsp-ml) (4.4.0)\n",
      "Requirement already satisfied: six>=1.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas->dsp-ml) (1.16.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2.1.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from requests>=2.20->openai->dsp-ml) (2022.12.7)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.7.9)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from thinc<8.2.0,>=8.1.0->spacy->dsp-ml) (0.0.4)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from typer<0.8.0,>=0.3.0->spacy->dsp-ml) (8.1.3)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.3)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (4.0.2)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (22.2.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (1.8.2)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from aiohttp->openai->dsp-ml) (6.0.4)\n",
      "Requirement already satisfied: jupyter-client>=6.1.12 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (7.4.9)\n",
      "Requirement already satisfied: pyzmq>=17 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (25.0.0)\n",
      "Requirement already satisfied: tornado>=6.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (6.2)\n",
      "Requirement already satisfied: nest-asyncio in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.5.6)\n",
      "Requirement already satisfied: debugpy>=1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (1.6.5)\n",
      "Requirement already satisfied: traitlets>=5.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.8.1)\n",
      "Requirement already satisfied: ipython>=7.23.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (8.8.0)\n",
      "Requirement already satisfied: comm>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.2)\n",
      "Requirement already satisfied: psutil in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (5.9.4)\n",
      "Requirement already satisfied: matplotlib-inline>=0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipykernel->jupyter->dsp-ml) (0.1.6)\n",
      "Requirement already satisfied: jupyterlab-widgets~=3.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (3.0.5)\n",
      "Requirement already satisfied: widgetsnbextension~=4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipywidgets->jupyter->dsp-ml) (4.0.5)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jinja2->spacy->dsp-ml) (2.1.2)\n",
      "Requirement already satisfied: pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (2.14.0)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-console->jupyter->dsp-ml) (3.0.36)\n",
      "Requirement already satisfied: bleach in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.0.1)\n",
      "Requirement already satisfied: nbclient>=0.5.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.2)\n",
      "Requirement already satisfied: mistune<3,>=2.0.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (2.0.4)\n",
      "Requirement already satisfied: defusedxml in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.7.1)\n",
      "Requirement already satisfied: nbformat>=5.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.7.3)\n",
      "Requirement already satisfied: importlib-metadata>=3.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (6.0.0)\n",
      "Requirement already satisfied: jupyter-core>=4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (5.1.3)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.5.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (0.2.2)\n",
      "Requirement already satisfied: beautifulsoup4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (4.11.1)\n",
      "Requirement already satisfied: tinycss2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbconvert->jupyter->dsp-ml) (1.2.1)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (1.8.0)\n",
      "Requirement already satisfied: prometheus-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.15.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.17.1)\n",
      "Requirement already satisfied: nbclassic>=0.4.7 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.4.8)\n",
      "Requirement already satisfied: argon2-cffi in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (21.3.0)\n",
      "Requirement already satisfied: ipython-genutils in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from notebook->jupyter->dsp-ml) (0.2.0)\n",
      "Requirement already satisfied: qtpy>=2.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from qtconsole->jupyter->dsp-ml) (2.3.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from importlib-metadata>=3.6->nbconvert->jupyter->dsp-ml) (3.11.0)\n",
      "Requirement already satisfied: backcall in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.0)\n",
      "Requirement already satisfied: pexpect>4.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (4.8.0)\n",
      "Requirement already satisfied: pickleshare in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.7.5)\n",
      "Requirement already satisfied: stack-data in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.6.2)\n",
      "Requirement already satisfied: decorator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.18.2)\n",
      "Requirement already satisfied: entrypoints in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-client>=6.1.12->ipykernel->jupyter->dsp-ml) (0.4)\n",
      "Requirement already satisfied: platformdirs>=2.5 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-core>=4.7->nbconvert->jupyter->dsp-ml) (2.6.2)\n",
      "Requirement already satisfied: jupyter-server>=1.8 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.1.0)\n",
      "Requirement already satisfied: notebook-shim>=0.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.2.2)\n",
      "Requirement already satisfied: jsonschema>=2.6 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (4.17.3)\n",
      "Requirement already satisfied: fastjsonschema in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.16.2)\n",
      "Requirement already satisfied: wcwidth in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->dsp-ml) (0.2.6)\n",
      "Requirement already satisfied: ptyprocess in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from terminado>=0.8.3->notebook->jupyter->dsp-ml) (0.7.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi->notebook->jupyter->dsp-ml) (21.2.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from beautifulsoup4->nbconvert->jupyter->dsp-ml) (2.3.2.post1)\n",
      "Requirement already satisfied: webencodings in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from bleach->nbconvert->jupyter->dsp-ml) (0.5.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.8.3)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (5.10.2)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (0.19.3)\n",
      "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.3.10)\n",
      "Requirement already satisfied: websocket-client in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.4.2)\n",
      "Requirement already satisfied: jupyter-events>=0.4.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.6.3)\n",
      "Requirement already satisfied: anyio<4,>=3.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (3.6.2)\n",
      "Requirement already satisfied: jupyter-server-terminals in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.4.4)\n",
      "Requirement already satisfied: cffi>=1.0.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (1.15.1)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (2.2.1)\n",
      "Requirement already satisfied: pure-eval in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (0.2.2)\n",
      "Requirement already satisfied: executing>=1.2.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from stack-data->ipython>=7.23.1->ipykernel->jupyter->dsp-ml) (1.2.0)\n",
      "Requirement already satisfied: sniffio>=1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (1.3.0)\n",
      "Requirement already satisfied: pycparser in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter->dsp-ml) (2.21)\n",
      "Requirement already satisfied: rfc3339-validator in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.4)\n",
      "Requirement already satisfied: python-json-logger>=2.0.4 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (2.0.4)\n",
      "Requirement already satisfied: pyyaml>=5.3 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (6.0)\n",
      "Requirement already satisfied: rfc3986-validator>=0.1.1 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->dsp-ml) (0.1.1)\n",
      "Requirement already satisfied: fqdn in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.5.1)\n",
      "Requirement already satisfied: uri-template in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.0)\n",
      "Requirement already satisfied: webcolors>=1.11 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.12)\n",
      "Requirement already satisfied: isoduration in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (20.11.0)\n",
      "Requirement already satisfied: jsonpointer>1.13 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (2.3)\n",
      "Requirement already satisfied: arrow>=0.15.0 in /future/u/okhattab/home/anaconda3/envs/py38/lib/python3.8/site-packages (from isoduration->jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter->dsp-ml) (1.2.3)\n",
      "env: DSP_NOTEBOOK_CACHEDIR=cache\n"
     ]
    }
   ],
   "source": [
    "!pip install dsp-ml\n",
    "\n",
    "try:\n",
    "    import google.colab  # When on google Colab, let's clone the notebook so we download the cache.\n",
    "    !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
    "    %set_env DSP_NOTEBOOK_CACHEDIR dsp/cache\n",
    "\n",
    "except:\n",
    "    %set_env DSP_NOTEBOOK_CACHEDIR cache  # Set up the cache for the examples below."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Up\n",
    "\n",
    "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
    "\n",
    "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
    "\n",
    "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
    "\n",
    "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import os\n",
    "import dsp\n",
    "\n",
    "openai_key = os.getenv('OPENAI_API_KEY')  # or replace with your API key (optional)\n",
    "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
    "\n",
    "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
    "rm = dsp.ColBERTv2(url=colbert_server)\n",
    "\n",
    "dsp.settings.configure(lm=lm, rm=rm)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task Examples\n",
    "\n",
    "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
    "\n",
    "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
    "\n",
    "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
    "         ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
    "         ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
    "         ('In what year was the star of To Hell and Back born?', ['1925']),\n",
    "         ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
    "         ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
    "\n",
    "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
    "       ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
    "       ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
    "       ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
    "       ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
    "       ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
    "       ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
    "       ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
    "       ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
    "       (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
    "       (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
    "       ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
    "       ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
    "\n",
    "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program 1: Vanilla GPT-3.5\n",
    "\n",
    "Let's begin by implementing the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
    "\n",
    "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
    "\n",
    "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
    "Answer = dsp.dsp.Type(prefix=\"Final Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
    "\n",
    "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's define the actual program, `vanilla_LM_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vanilla_LM_QA(question: str) -> str:\n",
    "    demos = dsp.sample(train, k=7)\n",
    "    example = dsp.Example(question=question, demos=demos)\n",
    "\n",
    "    example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
    "    return completions.answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's invoke the program on a sample question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
      "E. L. Doctorow\n"
     ]
    }
   ],
   "source": [
    "print(dev[0].question)\n",
    "print(vanilla_LM_QA(dev[0].question))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the last call to the LM to learn more about the behavior of the program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "Final Answer: ${a short factoid answer, often between 1 and 5 words}\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Final Answer: U.S. National Book Award\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Final Answer: Foxcatcher\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Final Answer: Kevin Greutert\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Final Answer: Butch Vig\n",
      "\n",
      "Question: What city was the victim of Joseph Druces working in?\n",
      "Final Answer: Boston, Massachusetts\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Final Answer: 1925\n",
      "\n",
      "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
      "Final Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lm.inspect_history(n=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the program has organized the examples in a simple prompt and generated an answer for the `dev[0]` question we asked.\n",
    "\n",
    "Let's try on the remaining development examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 13/13 [00:00<00:00, 384.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answered 4 / 13 correctly.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_364bd th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_364bd td {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_364bd\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_364bd_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_364bd_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
       "      <th id=\"T_364bd_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
       "      <th id=\"T_364bd_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_364bd_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_364bd_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
       "      <td id=\"T_364bd_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_364bd_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_364bd_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
       "      <td id=\"T_364bd_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
       "      <td id=\"T_364bd_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
       "      <td id=\"T_364bd_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_364bd_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_364bd_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
       "      <td id=\"T_364bd_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
       "      <td id=\"T_364bd_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_364bd_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_364bd_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
       "      <td id=\"T_364bd_row3_col2\" class=\"data row3 col2\" >1966</td>\n",
       "      <td id=\"T_364bd_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_364bd_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
       "      <td id=\"T_364bd_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
       "      <td id=\"T_364bd_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
       "      <td id=\"T_364bd_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_364bd_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_364bd_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
       "      <td id=\"T_364bd_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
       "      <td id=\"T_364bd_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_364bd_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
       "      <td id=\"T_364bd_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
       "      <td id=\"T_364bd_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
       "      <td id=\"T_364bd_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_364bd_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_364bd_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
       "      <td id=\"T_364bd_row7_col2\" class=\"data row7 col2\" >July 3, 1961</td>\n",
       "      <td id=\"T_364bd_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_364bd_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_364bd_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
       "      <td id=\"T_364bd_row8_col2\" class=\"data row8 col2\" >Susquehanna River</td>\n",
       "      <td id=\"T_364bd_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_364bd_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
       "      <td id=\"T_364bd_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
       "      <td id=\"T_364bd_row9_col2\" class=\"data row9 col2\" >Mars exploration</td>\n",
       "      <td id=\"T_364bd_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_364bd_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
       "      <td id=\"T_364bd_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
       "      <td id=\"T_364bd_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
       "      <td id=\"T_364bd_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_364bd_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_364bd_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
       "      <td id=\"T_364bd_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
       "      <td id=\"T_364bd_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_364bd_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_364bd_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_364bd_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
       "      <td id=\"T_364bd_row12_col2\" class=\"data row12 col2\" >1457</td>\n",
       "      <td id=\"T_364bd_row12_col3\" class=\"data row12 col3\" >❌</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2d505339a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from dsp.evaluation.utils import evaluate\n",
    "\n",
    "evaluate(vanilla_LM_QA, dev)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program 2: Retrieve-then-Read w/ GPT-3.5\n",
    "\n",
    "The program above relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
    "\n",
    "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
    "\n",
    "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
    "\n",
    "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "Context = dsp.Type(\n",
    "    prefix=\"Context:\\n\",\n",
    "    desc=\"${one or more sources, some of which may contain relevant content or clues}\",\n",
    "    format=dsp.passages2text\n",
    ")\n",
    "\n",
    "qa_template_with_passages = dsp.Template(\n",
    "    instructions=qa_template.instructions,\n",
    "    context=Context(), question=Question(), answer=Answer()\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieve_then_read_QA(question: str) -> str:\n",
    "    demos = dsp.sample(train, k=7)\n",
    "    passages = dsp.retrieve(question, k=1)\n",
    "    \n",
    "    example = dsp.Example(question=question, context=passages, demos=demos)\n",
    "    example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
    "\n",
    "    return completions.answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Final Answer: U.S. National Book Award\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Final Answer: Foxcatcher\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Final Answer: Kevin Greutert\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Final Answer: Butch Vig\n",
      "\n",
      "Question: What city was the victim of Joseph Druces working in?\n",
      "Final Answer: Boston, Massachusetts\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Final Answer: 1925\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Final Answer: ${a short factoid answer, often between 1 and 5 words}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
      "\n",
      "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
      "\n",
      "Final Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('E. L. Doctorow', None)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_then_read_QA(dev[0].question), lm.inspect_history(n=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's evaluate this on the dev examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 13/13 [00:00<00:00, 208.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answered 5 / 13 correctly.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_5ecf5 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_5ecf5 td {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_5ecf5\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_5ecf5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_5ecf5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
       "      <th id=\"T_5ecf5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
       "      <th id=\"T_5ecf5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_5ecf5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_5ecf5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
       "      <td id=\"T_5ecf5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_5ecf5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_5ecf5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
       "      <td id=\"T_5ecf5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
       "      <td id=\"T_5ecf5_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
       "      <td id=\"T_5ecf5_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_5ecf5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_5ecf5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
       "      <td id=\"T_5ecf5_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida</td>\n",
       "      <td id=\"T_5ecf5_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_5ecf5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_5ecf5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
       "      <td id=\"T_5ecf5_row3_col2\" class=\"data row3 col2\" >1856</td>\n",
       "      <td id=\"T_5ecf5_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_5ecf5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
       "      <td id=\"T_5ecf5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
       "      <td id=\"T_5ecf5_row4_col2\" class=\"data row4 col2\" >Studs Terkel</td>\n",
       "      <td id=\"T_5ecf5_row4_col3\" class=\"data row4 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_5ecf5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_5ecf5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
       "      <td id=\"T_5ecf5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
       "      <td id=\"T_5ecf5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_5ecf5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
       "      <td id=\"T_5ecf5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
       "      <td id=\"T_5ecf5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
       "      <td id=\"T_5ecf5_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_5ecf5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_5ecf5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
       "      <td id=\"T_5ecf5_row7_col2\" class=\"data row7 col2\" >5 August 1862</td>\n",
       "      <td id=\"T_5ecf5_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_5ecf5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_5ecf5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
       "      <td id=\"T_5ecf5_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
       "      <td id=\"T_5ecf5_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_5ecf5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
       "      <td id=\"T_5ecf5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
       "      <td id=\"T_5ecf5_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
       "      <td id=\"T_5ecf5_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_5ecf5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
       "      <td id=\"T_5ecf5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
       "      <td id=\"T_5ecf5_row10_col2\" class=\"data row10 col2\" >Los Angeles, California</td>\n",
       "      <td id=\"T_5ecf5_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_5ecf5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_5ecf5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
       "      <td id=\"T_5ecf5_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_5ecf5_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ecf5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_5ecf5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_5ecf5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
       "      <td id=\"T_5ecf5_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
       "      <td id=\"T_5ecf5_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2d5051aee0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(retrieve_then_read_QA, dev)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program 3: Retrieve-then-Read w/ Self-Consistency\n",
    "\n",
    "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
    "\n",
    "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
    "\n",
    "In particular:\n",
    "\n",
    "- We will include 5 passages (instead of a single passage) into the prompt.\n",
    "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
    "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "Rationale = dsp.Type(\n",
    "    prefix=\"Rationale: Let's think step by step.\",\n",
    "    desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
    ")\n",
    "\n",
    "qa_template_with_CoT = dsp.Template(\n",
    "    instructions=qa_template.instructions,\n",
    "    context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet above updates our template to include a `rationale` field.\n",
    "\n",
    "Below, we define the new program `retrieve_then_read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
    "\n",
    "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dsp.transformation\n",
    "def QA_predict(example: dsp.Example, sc=True):\n",
    "    if sc:\n",
    "        example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
    "        completions = dsp.majority(completions)\n",
    "    else:\n",
    "        example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
    "    \n",
    "    return example.copy(answer=completions.answer)\n",
    "\n",
    "def retrieve_then_read_QA_v2(question: str) -> str:\n",
    "    demos = dsp.sample(train, k=7)\n",
    "    passages = dsp.retrieve(question, k=5)\n",
    "    example = dsp.Example(question=question, context=passages, demos=demos)\n",
    "    \n",
    "    return QA_predict(example).answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Final Answer: U.S. National Book Award\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Final Answer: Foxcatcher\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Final Answer: Kevin Greutert\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Final Answer: Butch Vig\n",
      "\n",
      "Question: What city was the victim of Joseph Druces working in?\n",
      "Final Answer: Boston, Massachusetts\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Final Answer: 1925\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
      "\n",
      "Final Answer: ${a short factoid answer, often between 1 and 5 words}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
      "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
      "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
      "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
      "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
      "\n",
      "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
      "\n",
      "Rationale: Let's think step by step.\u001b[32m The song was co-written by the singer, Jeremy McKinnon. A quick search of his name reveals that he was born in Ocala, Florida.\n",
      "\n",
      "Final Answer: Ocala, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('Ocala, Florida', None)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_then_read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's run this on the other dev examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 13/13 [00:00<00:00, 16.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answered 6 / 13 correctly.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_3695c th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_3695c td {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_3695c\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_3695c_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_3695c_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
       "      <th id=\"T_3695c_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
       "      <th id=\"T_3695c_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_3695c_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_3695c_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
       "      <td id=\"T_3695c_row0_col2\" class=\"data row0 col2\" >Julia Peterkin</td>\n",
       "      <td id=\"T_3695c_row0_col3\" class=\"data row0 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_3695c_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
       "      <td id=\"T_3695c_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
       "      <td id=\"T_3695c_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
       "      <td id=\"T_3695c_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_3695c_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_3695c_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
       "      <td id=\"T_3695c_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
       "      <td id=\"T_3695c_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_3695c_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_3695c_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
       "      <td id=\"T_3695c_row3_col2\" class=\"data row3 col2\" >1824</td>\n",
       "      <td id=\"T_3695c_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_3695c_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
       "      <td id=\"T_3695c_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
       "      <td id=\"T_3695c_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
       "      <td id=\"T_3695c_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_3695c_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_3695c_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
       "      <td id=\"T_3695c_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
       "      <td id=\"T_3695c_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_3695c_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
       "      <td id=\"T_3695c_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
       "      <td id=\"T_3695c_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
       "      <td id=\"T_3695c_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_3695c_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_3695c_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
       "      <td id=\"T_3695c_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
       "      <td id=\"T_3695c_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_3695c_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_3695c_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
       "      <td id=\"T_3695c_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
       "      <td id=\"T_3695c_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_3695c_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
       "      <td id=\"T_3695c_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
       "      <td id=\"T_3695c_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
       "      <td id=\"T_3695c_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_3695c_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
       "      <td id=\"T_3695c_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
       "      <td id=\"T_3695c_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
       "      <td id=\"T_3695c_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_3695c_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_3695c_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
       "      <td id=\"T_3695c_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_3695c_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3695c_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_3695c_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_3695c_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
       "      <td id=\"T_3695c_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
       "      <td id=\"T_3695c_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2c97dcf2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(retrieve_then_read_QA_v2, dev)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program 4: Multi-Hop Retrieval\n",
    "\n",
    "From these simple programs, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
    "\n",
    "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
    "\n",
    "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "SearchRationale = dsp.Type(\n",
    "    prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
    "    desc=\"${missing information}\"\n",
    ")\n",
    "\n",
    "SearchQuery = dsp.Type(\n",
    "    prefix=\"Search Query:\",\n",
    "    desc=\"${a simple question for seeking the missing or required information}\"\n",
    ")\n",
    "\n",
    "rewrite_template = dsp.Template(\n",
    "    instructions=\"Write a search query that will help answer a complex question.\",\n",
    "    question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in `hop_template`, we will use the retrieved information from earlier hops to generate additional search queries for missing information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "CondenseRationale = dsp.Type(\n",
    "    prefix=\"Rationale: Let's think step by step. We now know that\",\n",
    "    desc=\"${information from the context that may provide useful clues for the question}\"\n",
    ")\n",
    "\n",
    "hop_template = dsp.Template(\n",
    "    instructions=rewrite_template.instructions,\n",
    "    context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
    "\n",
    "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
    "\n",
    "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsp.utils import deduplicate\n",
    "\n",
    "@dsp.transformation\n",
    "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
    "    example.context = []\n",
    "    \n",
    "    for hop in range(max_hops):\n",
    "        # Generate a query based\n",
    "        template = rewrite_template if hop == 0 else hop_template\n",
    "        example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
    "\n",
    "        # Retrieve k results based on the query generated\n",
    "        passages = dsp.retrieve(completions.query, k=k)\n",
    "\n",
    "        # Update the context by concatenating old and new passages\n",
    "        example.context = deduplicate(example.context + passages)\n",
    "\n",
    "    return example\n",
    "\n",
    "\n",
    "def multihop_QA_v1(question: str) -> str:\n",
    "    demos = dsp.sample(train, k=7)\n",
    "    x = dsp.Example(question=question, demos=demos)\n",
    "    \n",
    "    x = multihop_search_v1(x)\n",
    "    x = QA_predict(x, sc=False)\n",
    "\n",
    "    return x.answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run this on one example and inspect the last 3 calls to the LM (i.e., search query 1, search query 2, and QA)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Write a search query that will help answer a complex question.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out ${missing information}\n",
      "Search Query: ${a simple question for seeking the missing or required information}\n",
      "\n",
      "---\n",
      "\n",
      "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
      "Search Query: Who sings Right Back At It Again\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Write a search query that will help answer a complex question.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Rationale: Let's think step by step. We now know that ${information from the context that may provide useful clues for the question}\n",
      "\n",
      "Search Query: ${a simple question for seeking the missing or required information}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
      "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
      "\n",
      "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
      "\n",
      "Rationale: Let's think step by step. We now know that\u001b[32m the song was co-written by the singer, Jeremy McKinnon.\n",
      "\n",
      "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Final Answer: U.S. National Book Award\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Final Answer: Foxcatcher\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Final Answer: Kevin Greutert\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Final Answer: Butch Vig\n",
      "\n",
      "Question: What city was the victim of Joseph Druces working in?\n",
      "Final Answer: Boston, Massachusetts\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Final Answer: 1925\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
      "\n",
      "Final Answer: ${a short factoid answer, often between 1 and 5 words}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
      "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
      "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
      "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
      "\n",
      "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
      "\n",
      "Rationale: Let's think step by step.\u001b[32m The song Right Back At It Again was co-written by the singer Jeremy McKinnon. A quick search shows that he was born in Gainesville, Florida.\n",
      "\n",
      "Final Answer: Gainesville, Florida\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('Gainesville, Florida', None)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's run this on the remaining dev examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 13/13 [00:00<00:00, 83.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answered 7 / 13 correctly.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_83eb7 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_83eb7 td {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_83eb7\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_83eb7_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_83eb7_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
       "      <th id=\"T_83eb7_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
       "      <th id=\"T_83eb7_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_83eb7_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_83eb7_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
       "      <td id=\"T_83eb7_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_83eb7_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_83eb7_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
       "      <td id=\"T_83eb7_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
       "      <td id=\"T_83eb7_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
       "      <td id=\"T_83eb7_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_83eb7_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_83eb7_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
       "      <td id=\"T_83eb7_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
       "      <td id=\"T_83eb7_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_83eb7_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_83eb7_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
       "      <td id=\"T_83eb7_row3_col2\" class=\"data row3 col2\" >1947</td>\n",
       "      <td id=\"T_83eb7_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_83eb7_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
       "      <td id=\"T_83eb7_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
       "      <td id=\"T_83eb7_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
       "      <td id=\"T_83eb7_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_83eb7_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_83eb7_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
       "      <td id=\"T_83eb7_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
       "      <td id=\"T_83eb7_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_83eb7_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
       "      <td id=\"T_83eb7_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
       "      <td id=\"T_83eb7_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
       "      <td id=\"T_83eb7_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_83eb7_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_83eb7_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
       "      <td id=\"T_83eb7_row7_col2\" class=\"data row7 col2\" >1965</td>\n",
       "      <td id=\"T_83eb7_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_83eb7_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_83eb7_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
       "      <td id=\"T_83eb7_row8_col2\" class=\"data row8 col2\" >River Foyle</td>\n",
       "      <td id=\"T_83eb7_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_83eb7_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
       "      <td id=\"T_83eb7_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
       "      <td id=\"T_83eb7_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
       "      <td id=\"T_83eb7_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_83eb7_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
       "      <td id=\"T_83eb7_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
       "      <td id=\"T_83eb7_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
       "      <td id=\"T_83eb7_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_83eb7_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_83eb7_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
       "      <td id=\"T_83eb7_row11_col2\" class=\"data row11 col2\" >Carrington</td>\n",
       "      <td id=\"T_83eb7_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_83eb7_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_83eb7_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_83eb7_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
       "      <td id=\"T_83eb7_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
       "      <td id=\"T_83eb7_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2d50533ee0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(multihop_QA_v1, dev)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Program 5: Multi-Hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
    "\n",
    "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
    "\n",
    "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
    "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
    "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We address these problems automatically in the program below.\n",
    "\n",
    "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dsp.transformation\n",
    "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
    "    # Prepare unaugmented demonstrations for the example.\n",
    "    x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
    "    \n",
    "    # Search. And skip examples where search fails.\n",
    "    # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
    "    x = multihop_search_v1(x)\n",
    "    if not dsp.passage_match(x.context, d.answer): return None\n",
    "    \n",
    "    # Predict. And skip examples where predict fails.\n",
    "    x = QA_predict(x, sc=False)\n",
    "    if not dsp.answer_match(x.answer, d.answer): return None\n",
    "    \n",
    "    return d.copy(**x)\n",
    "\n",
    "@dsp.transformation\n",
    "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
    "    demos = dsp.sample(train, k=7)\n",
    "    x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
    "    return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now implement `multihop_search_v2` as part of `multihop_QA_v2`. In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021)  in a few lines of code. In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dsp.transformation\n",
    "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
    "    example.context = []\n",
    "\n",
    "    for hop in range(max_hops):\n",
    "        # Generate queries\n",
    "        template = rewrite_template if hop == 0 else hop_template\n",
    "        example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
    "        \n",
    "        # Collect the queries and search with result fusion\n",
    "        queries = [c.query for c in completions] + [example.question]\n",
    "        example.context = dsp.retrieveEnsemble(queries, k=k)\n",
    "\n",
    "        # Arrange the passages for the next hop\n",
    "        if hop > 0:\n",
    "            example.context = [completions[0].rationale] + example.context\n",
    "    \n",
    "    return example\n",
    "\n",
    "def multihop_QA_v2(question: str) -> str:\n",
    "    x = dsp.Example(question=question)\n",
    "    x = multihop_demonstrate(x)\n",
    "    x = multihop_search_v2(x)\n",
    "    x = QA_predict(x)\n",
    "    return x.answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Write a search query that will help answer a complex question.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out ${missing information}\n",
      "Search Query: ${a simple question for seeking the missing or required information}\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav\n",
      "Search Query: \"Gary Zukav\" first book\n",
      "\n",
      "---\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
      "Search Query: \"Heir to the Du Pont family fortune\"\n",
      "\n",
      "---\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
      "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
      "\n",
      "---\n",
      "\n",
      "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
      "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
      "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Write a search query that will help answer a complex question.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Rationale: Let's think step by step. We now know that ${information from the context that may provide useful clues for the question}\n",
      "\n",
      "Search Query: ${a simple question for seeking the missing or required information}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
      "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "\n",
      "Rationale: Let's think step by step. We now know that Gary Zukav's first book is \"The Dancing Wu Li Masters\" and it was written with the help of Jack Sarfatti and other physicists.\n",
      "\n",
      "Search Query: What award did \"The Dancing Wu Li Masters\" receive?\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
      "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "\n",
      "Rationale: Let's think step by step. We now know that John du Pont was an American philanthropist and heir to the Du Pont family fortune. He was also a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach.\n",
      "\n",
      "Search Query: What wrestling team did John du Pont sponsor?\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
      "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "\n",
      "Rationale: Let's think step by step. We now know that Peter Outerbridge played William Easton in the 2009 movie.\n",
      "\n",
      "Search Query: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
      "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
      "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
      "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
      "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
      "\n",
      "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
      "\n",
      "Rationale: Let's think step by step. We now know that\u001b[32m the winner of the 1971 San Francisco mayoral election was Joseph Alioto and he was a member of the Democratic Party.\n",
      "\n",
      "Search Query: When was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer questions with short factoid answers.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Final Answer: Butch Vig\n",
      "\n",
      "Question: What city was the victim of Joseph Druces working in?\n",
      "Final Answer: Boston, Massachusetts\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Final Answer: 1925\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context:\n",
      "${one or more sources, some of which may contain relevant content or clues}\n",
      "\n",
      "Question: ${the question to be answered}\n",
      "\n",
      "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
      "\n",
      "Final Answer: ${a short factoid answer, often between 1 and 5 words}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
      "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
      "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
      "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "\n",
      "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it received a U.S. National Book Award.\n",
      "\n",
      "Final Answer: U.S. National Book Award\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
      "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
      "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
      "[4] «John du Pont | stamps while in prison, he was not allowed to bring them there. After his mother's death, du Pont developed the Liseter Hall Farm in Newtown Square as a high-quality wrestling facility for amateur wrestlers. He called the private group \"Team Foxcatcher,\" after his father's noted racing stable. Du Pont established an Olympic swimming and wrestling training center and sponsored competitive events at the estate. He also allowed some people, such as Olympic champion wrestlers Mark Schultz and later his older brother Dave Schultz and his wife, to live in houses on the grounds for years. Dave Schultz also coached the»\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "\n",
      "Rationale: Let's think step by step. The heir to the Du Pont family fortune was John du Pont. He was convicted of murdering wrestling coach Dave Schultz in 1996. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. He sponsored competitive events at the estate.\n",
      "\n",
      "Final Answer: Team Foxcatcher\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
      "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
      "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "\n",
      "Rationale: Let's think step by step. We know that Peter Outerbridge was in the movie \"Saw VI\" and that he played the character William Easton. We also know that the movie was released in 2009. From this, we can deduce that the director of the movie was Kevin Greutert.\n",
      "\n",
      "Final Answer: Kevin Greutert\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «the winner of the 1971 San Francisco mayoral election was Joseph Alioto and he was a member of the Democratic Party.»\n",
      "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
      "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
      "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
      "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
      "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
      "\n",
      "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
      "\n",
      "Rationale: Let's think step by step.\u001b[32m We know that Joseph Alioto was the winner of the 1971 San Francisco mayoral election and that he was a member of the Democratic Party. We also know that the Democratic Party was founded in 1828. From this, we can deduce that the party was founded in 1828.\n",
      "\n",
      "Final Answer:\n",
      "\n",
      "1828\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('1828', None)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 13/13 [00:01<00:00,  8.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answered 12 / 13 correctly.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_b6174 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_b6174 td {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_b6174\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_b6174_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_b6174_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
       "      <th id=\"T_b6174_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
       "      <th id=\"T_b6174_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_b6174_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_b6174_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
       "      <td id=\"T_b6174_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_b6174_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_b6174_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
       "      <td id=\"T_b6174_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
       "      <td id=\"T_b6174_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
       "      <td id=\"T_b6174_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_b6174_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_b6174_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
       "      <td id=\"T_b6174_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
       "      <td id=\"T_b6174_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_b6174_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_b6174_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
       "      <td id=\"T_b6174_row3_col2\" class=\"data row3 col2\" >1828</td>\n",
       "      <td id=\"T_b6174_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_b6174_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
       "      <td id=\"T_b6174_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
       "      <td id=\"T_b6174_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
       "      <td id=\"T_b6174_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_b6174_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_b6174_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
       "      <td id=\"T_b6174_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
       "      <td id=\"T_b6174_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_b6174_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
       "      <td id=\"T_b6174_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
       "      <td id=\"T_b6174_row6_col2\" class=\"data row6 col2\" >Cheshire</td>\n",
       "      <td id=\"T_b6174_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_b6174_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_b6174_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
       "      <td id=\"T_b6174_row7_col2\" class=\"data row7 col2\" >1958</td>\n",
       "      <td id=\"T_b6174_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_b6174_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_b6174_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
       "      <td id=\"T_b6174_row8_col2\" class=\"data row8 col2\" >Castletown River</td>\n",
       "      <td id=\"T_b6174_row8_col3\" class=\"data row8 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_b6174_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
       "      <td id=\"T_b6174_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
       "      <td id=\"T_b6174_row9_col2\" class=\"data row9 col2\" >Aeronautics</td>\n",
       "      <td id=\"T_b6174_row9_col3\" class=\"data row9 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_b6174_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
       "      <td id=\"T_b6174_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
       "      <td id=\"T_b6174_row10_col2\" class=\"data row10 col2\" >Surrey</td>\n",
       "      <td id=\"T_b6174_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_b6174_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_b6174_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
       "      <td id=\"T_b6174_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_b6174_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b6174_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_b6174_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_b6174_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
       "      <td id=\"T_b6174_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
       "      <td id=\"T_b6174_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f2c9404ee80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate(multihop_QA_v2, dev)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Programs...\n",
    "\n",
    "While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
    "\n",
    "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
    "\n",
    "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
    "\n",
    "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
    "\n",
    "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
    "\n",
    "```\n",
    "SearchQueryWithStopping = dsp.Type(\n",
    "    prefix=\"Search Query:\",\n",
    "    desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Program 4 with max_hops=4 and automatic stopping."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py38",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "ff8889dd26469456638524e92174013821322a2ab8e175954d0360930982a899"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}