scrape_nhs_conditions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape NHS Conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will scrape NHS Conditions (using the [scrape_nhs_conditions](https://pypi.org/project/scrape-nhs-conditions/) python package), pulling the text content from all of the pages. \n",
    "\n",
    "To run this you'll need to first set up an API key from the NHS Developer portal, and add it to the '.secrets' file (see the '.secrets_example' file for an example of how this should look)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrape_nhs_conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrape_nhs_conditions.data_ingestion.simple_nhs_conditions_scrape as snhscs\n",
    "import scrape_nhs_conditions.processing.process_html as ph\n",
    "\n",
    "output_file = \"nhs_conditions_raw.jsonl\"\n",
    "docs_directory = \"docs\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrape the Raw HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below will scrape the raw HTML from the pages, saving it into the output file described below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "snhscs.run_nhs_conditions_scraper(output_file=output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process the raw HTML into text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ph.process_nhs_conditions_json(json_file=output_file, docs_directory=docs_directory)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}