python/using-an-api.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zshenv\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n",
      "zshenv\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "!pip install -Uq requests\n",
    "import requests\n",
    "!mkdir -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using an API\n",
    "\n",
    "Learning outcomes\n",
    "\n",
    "- difference between API & webscraping\n",
    "- what JSON is (and why it's like a Python `dict`)\n",
    "- how to properly handle files in Python\n",
    "- what a REST API is\n",
    "- how to use the `requests` library\n",
    "\n",
    "## API versus web-scraping\n",
    "\n",
    "**Both are ways to sample data from the internet**\n",
    "\n",
    "API\n",
    "- structured\n",
    "- provided as a service (you are talking to a server via a REST API)\n",
    "- limited data / rate limits / paid / require auth (sometimes)\n",
    "- most will give back JSON (maybe XML or CSV)\n",
    "\n",
    "Web scraping\n",
    "- less structure\n",
    "- parsing HTML meant for your browser\n",
    "\n",
    "Neither is better than the other\n",
    "\n",
    "- API developer can limit what data is accessible through the API\n",
    "- API developer can not maintain the API\n",
    "- website page can change HTML structure\n",
    "- website page can have dynamic (Javascript) content that requires execution (usually done by the browser) before the correct HTML is available\n",
    "\n",
    "Much of the work in using an API is figuring out how to properly construct URL's for `GET` requests\n",
    "- requires looking at their documentation (& ideally a Python example!)\n",
    "\n",
    "## Where to find APIs\n",
    "\n",
    "- [ProgrammableWeb](https://www.programmableweb.com/apis/directory) - a collection of available API's\n",
    "- For the *Developer* or *For Developers* documentation on your favourite website,\n",
    "- [public-apis/public-apis](https://github.com/public-apis/public-apis),\n",
    "- [Wikipedia API documentation](https://en.wikipedia.org/api/rest_v1/#/Page%20content).\n",
    "\n",
    "## Using API's\n",
    "\n",
    "Most API's require authentication\n",
    "\n",
    "- so they API developer knows who you are\n",
    "- can charge you\n",
    "- can limit access\n",
    "- commonly via key or OAuth (both of which may be free)\n",
    "\n",
    "All the API's we use here are unauthenticated - this is to avoid the time of you all signing up\n",
    "\n",
    "If your app requries authentication, it's usually done by passing in your credentials into the request (as a header)\n",
    "\n",
    "```python\n",
    "response = requests.get(url, auth=auth)\n",
    "```\n",
    "\n",
    "## JSON strings\n",
    "\n",
    "JSON (JavaScript Object Notation) is a:\n",
    "- lightweight data-interchange format (text)\n",
    "- easy for humans to read and write \n",
    "- easy for machines to parse and generate\n",
    "- based on key, value pairs\n",
    "\n",
    "You can think of the Python `dict` as JSON like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan-turing'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'name': 'alan-turing'}\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But true JSON is just a string (only text).  We can use `json.dumps` from the standard library to turn the `dict` into a JSON string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"name\": \"alan-turing\"}'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = json.dumps(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use `json.loads` to turn this string back into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan-turing'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.loads(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, reading & writing to files\n",
    "\n",
    "### Reading from a file\n",
    "\n",
    "We open files using the Python `open` builtin function, followed by a `read`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'# Python\\n\\nWhy Python?\\n\\n- fast to write - suitable for research,\\n- allows quick iteration over ideas,'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./readme.md', 'r').read()[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note the `./path` - the `.` refers to the current working directory.  It's not straightforward to know where this is - is it where the notebook is, or where the notebook server is running?\n",
    "\n",
    "If we wanted to read the file as separate lines, we could use `readlines()` (note we would still need to manually strip off the `\\n` characters later)\n",
    "\n",
    "(Note `/n` is used as a newline indicator in text files - you never see it because your editor interprets it as a line break :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['# Python\\n',\n",
       " '\\n',\n",
       " 'Why Python?\\n',\n",
       " '\\n',\n",
       " '- fast to write - suitable for research,\\n',\n",
       " '- allows quick iteration over ideas,\\n',\n",
       " '- can be put into production.\\n',\n",
       " '- massive ecosystem, mature language.\\n']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./readme.md', 'r').readlines()[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `open`\n",
    "\n",
    "`open(path, mode)`\n",
    "\n",
    "Common values for the mode:\n",
    "- `r` read\n",
    "- `rb` read binary\n",
    "- `w+` write (`+` to create file if it doesn't exist)\n",
    "- `a` append\n",
    "\n",
    "Note there are options for both reading & writing - we actually use `open` for both reading & writing.\n",
    "\n",
    "We open a file using the Python builtin `open`, which is then followed by either a read or write stage\n",
    "\n",
    "- open the file\n",
    "- read the file OR write to the file\n",
    "\n",
    "Notice that the file is read in as a single string, with the newline character `\\n` separating lines\n",
    "\n",
    "- this is how all text files are structured\n",
    "- your editor does the line splitting for you\n",
    "\n",
    "### Writing to a file (without context management)\n",
    "\n",
    "We can write to a new file using the same `open` builtin\n",
    "\n",
    "- open the file\n",
    "- write to the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./data/output.data', 'w').write('We make this file to show how not to do it\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can do the same file write by explicitly assiging the file object to a variable (note the `a` to append)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('We make this file slightly differently to show how not to do it\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The issue with the code above is that we aren't closing the file - we can fix this by intentionally closing the file.  \n",
    "\n",
    "One way to do this is to use `.close()` when we are done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('This time we close the file manually\\n')\n",
    "fi.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This requires us to remember to close (also an additional line).\n",
    "\n",
    "### Reading files with context management\n",
    "\n",
    "The Pythonic way of handling opening & closing of files is context management:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./readme.md', 'r') as fi:\n",
    "    data = fi.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to a file with context management\n",
    "\n",
    "Now that we understand context management, we can save our `data` dict as JSON, using `json.dump` to write the dict to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'name': 'alan turing'}\n",
    "with open('./data/output.json', 'w') as fi:\n",
    "    json.dump(data, fi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it worked by loading the file, here using `json.load` to load from the file object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'alan turing'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./data/output.json', 'r') as fi:\n",
    "    data = json.load(fi)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST API's\n",
    "\n",
    "[REST - Wiki](https://en.wikipedia.org/wiki/Representational_state_transfer)\n",
    "\n",
    "REST is a set of constraints that allow **stateless communication of text data on the internet**\n",
    "\n",
    "- REST = REpresentational State Transfer\n",
    "- API = Application Programming Interface\n",
    "\n",
    "REST\n",
    "- communication of resources (located at URLs / URIs)\n",
    "- requests for a resource are responded to with a text payload (HTML, JSON etc)\n",
    "- these requests are made using HTTP (determines how messages are formatted, what actions (methods) can be taken)\n",
    "- common HTTP methods are `GET` and `POST`\n",
    "\n",
    "HTTP methods\n",
    "- GET - retrieve information about the REST API resource\n",
    "- POST - create a REST API resource\n",
    "- PUT - update a REST API resource\n",
    "- DELETE - delete a REST API resource or related component\n",
    "\n",
    "RESTful APIs enable you to develop any kind of web application having all possible CRUD (create, retrieve, update, delete) operations\n",
    "\n",
    "- can do anything we would want to do with a database\n",
    "\n",
    "*Further reading*\n",
    "- [Web Architecture 101](https://engineering.videoblocks.com/web-architecture-101-a3224e126947) for more detail on how the web works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - sunrise API\n",
    "\n",
    "Docs - https://sunrise-sunset.org/api\n",
    "\n",
    "First we need to form the url\n",
    "- use `?` to separate the API server name from the parameters for our request\n",
    "- use `&` to separate the parameters from each other\n",
    "- use `+` instead of space in the parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'results': {'date': '2024-01-13',\n",
       "  'sunrise': '7:27:13 AM',\n",
       "  'sunset': '5:08:14 PM',\n",
       "  'first_light': '5:52:52 AM',\n",
       "  'last_light': '6:42:35 PM',\n",
       "  'dawn': '6:57:52 AM',\n",
       "  'dusk': '5:37:35 PM',\n",
       "  'solar_noon': '12:17:43 PM',\n",
       "  'golden_hour': '4:27:22 PM',\n",
       "  'day_length': '9:41:00',\n",
       "  'timezone': 'America/New_York',\n",
       "  'utc_offset': -300},\n",
       " 'status': 'OK'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = requests.get(\"https://api.sunrisesunset.io/json?lat=38.907192&lng=-77.036873\")\n",
    "data = res.json()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This response is JSON - `requests.json()` turns it into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common to have a top level heirarchy to dig through to get the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['results', 'status'])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the interesting stuff is in `results`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'date': '2024-01-13',\n",
       " 'sunrise': '7:27:13 AM',\n",
       " 'sunset': '5:08:14 PM',\n",
       " 'first_light': '5:52:52 AM',\n",
       " 'last_light': '6:42:35 PM',\n",
       " 'dawn': '6:57:52 AM',\n",
       " 'dusk': '5:37:35 PM',\n",
       " 'solar_noon': '12:17:43 PM',\n",
       " 'golden_hour': '4:27:22 PM',\n",
       " 'day_length': '9:41:00',\n",
       " 'timezone': 'America/New_York',\n",
       " 'utc_offset': -300}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - Chronicling America API\n",
    "\n",
    "Docs - https://chroniclingamerica.loc.gov/about/api/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://chroniclingamerica.loc.gov/search/pages/results/?proxtext=germany&format=json'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the `requests` HTTP library to perform a `GET` request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP response\n",
    "\n",
    "What we recieved above is an *HTTP response**\n",
    "\n",
    "[HTTP Response - Wiki](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Response_message)\n",
    "\n",
    "The response message consists of the following:\n",
    "\n",
    "- a status line which includes the status code and reason message (e.g., HTTP/1.1 200 OK, which indicates that the client's request succeeded)\n",
    "- response header fields (e.g., Content-Type: text/html)\n",
    "- an optional message body\n",
    "\n",
    "## What can we do with this HTTP response in `requests`?\n",
    "\n",
    "The Python builtin `dir` gives us all the attributes & methods of a Python object.\n",
    "\n",
    "This also includes all the `__` dunder (literally double-under) methods) - which we filter out using a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['_content',\n",
       " '_content_consumed',\n",
       " '_next',\n",
       " 'apparent_encoding',\n",
       " 'close',\n",
       " 'connection',\n",
       " 'content',\n",
       " 'cookies',\n",
       " 'elapsed',\n",
       " 'encoding',\n",
       " 'headers',\n",
       " 'history',\n",
       " 'is_permanent_redirect',\n",
       " 'is_redirect',\n",
       " 'iter_content',\n",
       " 'iter_lines',\n",
       " 'json',\n",
       " 'links',\n",
       " 'next',\n",
       " 'ok',\n",
       " 'raise_for_status',\n",
       " 'raw',\n",
       " 'reason',\n",
       " 'request',\n",
       " 'status_code',\n",
       " 'text',\n",
       " 'url']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[o for o in dir(response) if '__' not in o]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the HTTP status code (used to communicate things like everything OK (200), stop making requests etc - see [List of HTTP status codes - Wiki](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTTP response headers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Date': 'Sat, 13 Jan 2024 22:38:11 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Last-Modified': 'Sat, 13 Jan 2024 22:37:33 GMT', 'X-Robots-Tag': 'noindex, nofollow', 'Expires': 'Sun, 14 Jan 2024 22:37:33 GMT', 'Cache-Control': 's-maxage=86400, public, max-age=86400', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'X-requested-with', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Varnish': '6539974', 'Via': '1.1 varnish (Varnish/5.2)', 'CF-Cache-Status': 'HIT', 'Age': '45', 'Server': 'cloudflare', 'CF-RAY': '845120c71d031c50-AKL'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the response body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"totalItems\": 2594186, \"endIndex\": 20, \"startIndex\": 1, \"itemsPerPage\": 20, \"items\": [{\"sequence\": 11, \"county\": [\"Wayne\"], \"edition\": \"Home ed.\", \"frequency\": \"Daily\", \"id\": \"/lccn/sn88063294/1944-09-12/ed-1/seq-11/\", \"subject\": [\"Detroit (Mich.)--Newspapers.\", \"Michigan--Detroit.--fast--(OCoLC)fst01205010\", \"Michigan--Wayne County.--fast--(OCoLC)fst01206628\", \"Wayne County (Mich.)--Newspapers.\"], \"city\": [\"Detroit\"], \"date\": \"19440912\", \"title\": \"Detroit evening times.\", \"end_year\": 1958, \"note\": [\"Archived issues are available in digital format from the Library of Congress Chronicling America online collection.\", \"Description based on: 22nd yr., no. 27 (Nov. 1, 1921).\", \"Issue called: Michigan centennial ed., May 2, 1937.\", \"Most Sunday issues for Aug. 6, 1922-Jan. 12, 1958 called: Detroit Sunday times.\", \"Some issues include supplement called: Goodfellows Special/Old Newsboys ed., published in Dec. of each year <Dce. 16, 1926-Dec. 1948>.\", \"The word \\\\\"evening\\\\\" appears within orna'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.text[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "data = json.loads(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the keys of the dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['totalItems', 'endIndex', 'startIndex', 'itemsPerPage', 'items'])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the values using the square bracket indexing with a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2594186"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['totalItems']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While JSON is a simple text format, it can become complex due to\n",
    "\n",
    "- nesting (JSON inside JSON)\n",
    "- lists of JSON\n",
    "\n",
    "An example is our `items`, which has been parsed as a Python `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data['items'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`items` is a list of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['sequence', 'county', 'edition', 'frequency', 'id', 'subject', 'city', 'date', 'title', 'end_year', 'note', 'state', 'section_label', 'type', 'place_of_publication', 'start_year', 'edition_label', 'publisher', 'language', 'alt_title', 'lccn', 'country', 'ocr_eng', 'batch', 'title_normal', 'url', 'place', 'page'])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item = data['items'][0]\n",
    "item.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate over both the keys and values as a pair using `items()`.\n",
    "\n",
    "Below we use a check  a quick check that the value isn't too long before printing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "county: ['Wayne']\n",
      "edition: Home ed.\n",
      "frequency: Daily\n",
      "id: /lccn/sn88063294/1944-09-12/ed-1/seq-11/\n",
      "subject: ['Detroit (Mich.)--Newspapers.', 'Michigan--Detroit.--fast--(OCoLC)fst01205010', 'Michigan--Wayne County.--fast--(OCoLC)fst01206628', 'Wayne County (Mich.)--Newspapers.']\n",
      "city: ['Detroit']\n",
      "date: 19440912\n",
      "title: Detroit evening times.\n",
      "note: ['Archived issues are available in digital format from the Library of Congress Chronicling America online collection.', 'Description based on: 22nd yr., no. 27 (Nov. 1, 1921).', 'Issue called: Michigan centennial ed., May 2, 1937.', 'Most Sunday issues for Aug. 6, 1922-Jan. 12, 1958 called: Detroit Sunday times.', 'Some issues include supplement called: Goodfellows Special/Old Newsboys ed., published in Dec. of each year <Dce. 16, 1926-Dec. 1948>.', 'The word \"evening\" appears within ornament.']\n",
      "state: ['Michigan']\n",
      "section_label: \n",
      "type: page\n",
      "place_of_publication: Detroit, Mich\n",
      "edition_label: REDLINE\n",
      "publisher: [s.n.]\n",
      "language: ['English']\n",
      "alt_title: ['Detroit Sunday times', 'Detroit times']\n",
      "lccn: sn88063294\n",
      "country: Michigan\n",
      "ocr_eng: Where the Battle of Germany Will Be Fought\n",
      "|DETROir.*KfTIJvIES| -Sea\n",
      "batch: mimtptc_holly_ver03\n",
      "title_normal: detroit evening times.\n",
      "url: https://chroniclingamerica.loc.gov/lccn/sn88063294/1944-09-12/ed-1/seq-11.json\n",
      "place: ['Michigan--Wayne--Detroit']\n",
      "page: \n"
     ]
    }
   ],
   "source": [
    "from collections.abc import Iterable\n",
    "\n",
    "for k, v in item.items():\n",
    "    if isinstance(v, Iterable) and len(v) < 100:\n",
    "        print(f'{k}: {v}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish this exercise by only taking articles that appear between two years, and save those to disk.\n",
    "\n",
    "Normally you would apply this kind of filtering in the API request - we are going to filter in memory.\n",
    "\n",
    "First we need a bit of data cleaning of our date, which is an integer representation of time (but is a `str`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'19440912'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here use `strptime` to convert the integer into a proper datetime:\n",
    "- ([Python's strftime directives](http://strftime.org/) is very useful!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(1944, 9, 12, 0, 0)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime as dt\n",
    "dt.strptime(item['date'], \"%Y%m%d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's put this data cleaning & filtering into a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "items = data['items']\n",
    "\n",
    "start = 1900\n",
    "extract = []\n",
    "for item in items:\n",
    "    item['date'] = dt.strptime(item['date'], \"%Y%m%d\")\n",
    "    \n",
    "    if item['date'].year > start:\n",
    "        extract.append(item)\n",
    "        \n",
    "len(extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a list of dictionaries, which plays very nice with `pandas`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zshenv\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>county</th>\n",
       "      <th>edition</th>\n",
       "      <th>frequency</th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>city</th>\n",
       "      <th>date</th>\n",
       "      <th>title</th>\n",
       "      <th>end_year</th>\n",
       "      <th>note</th>\n",
       "      <th>state</th>\n",
       "      <th>section_label</th>\n",
       "      <th>type</th>\n",
       "      <th>place_of_publication</th>\n",
       "      <th>start_year</th>\n",
       "      <th>edition_label</th>\n",
       "      <th>publisher</th>\n",
       "      <th>language</th>\n",
       "      <th>alt_title</th>\n",
       "      <th>lccn</th>\n",
       "      <th>country</th>\n",
       "      <th>ocr_eng</th>\n",
       "      <th>batch</th>\n",
       "      <th>title_normal</th>\n",
       "      <th>url</th>\n",
       "      <th>place</th>\n",
       "      <th>page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "      <td>[Wayne]</td>\n",
       "      <td>Home ed.</td>\n",
       "      <td>Daily</td>\n",
       "      <td>/lccn/sn88063294/1944-09-12/ed-1/seq-11/</td>\n",
       "      <td>[Detroit (Mich.)--Newspapers., Michigan--Detro...</td>\n",
       "      <td>[Detroit]</td>\n",
       "      <td>1944-09-12</td>\n",
       "      <td>Detroit evening times.</td>\n",
       "      <td>1958</td>\n",
       "      <td>[Archived issues are available in digital form...</td>\n",
       "      <td>[Michigan]</td>\n",
       "      <td></td>\n",
       "      <td>page</td>\n",
       "      <td>Detroit, Mich</td>\n",
       "      <td>1921</td>\n",
       "      <td>REDLINE</td>\n",
       "      <td>[s.n.]</td>\n",
       "      <td>[English]</td>\n",
       "      <td>[Detroit Sunday times, Detroit times]</td>\n",
       "      <td>sn88063294</td>\n",
       "      <td>Michigan</td>\n",
       "      <td>Where the Battle of Germany Will Be Fought\\n|D...</td>\n",
       "      <td>mimtptc_holly_ver03</td>\n",
       "      <td>detroit evening times.</td>\n",
       "      <td>https://chroniclingamerica.loc.gov/lccn/sn8806...</td>\n",
       "      <td>[Michigan--Wayne--Detroit]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16</td>\n",
       "      <td>[San Francisco]</td>\n",
       "      <td>None</td>\n",
       "      <td>Daily</td>\n",
       "      <td>/lccn/sn85066387/1908-11-08/ed-1/seq-16/</td>\n",
       "      <td>[California--San Francisco Bay Area.--fast--(O...</td>\n",
       "      <td>[San Francisco]</td>\n",
       "      <td>1908-11-08</td>\n",
       "      <td>The San Francisco call. [volume]</td>\n",
       "      <td>1913</td>\n",
       "      <td>[\"San Francisco\" appears above, and later acro...</td>\n",
       "      <td>[California]</td>\n",
       "      <td></td>\n",
       "      <td>page</td>\n",
       "      <td>San Francisco [Calif.]</td>\n",
       "      <td>1895</td>\n",
       "      <td></td>\n",
       "      <td>Charles M. Shortridge</td>\n",
       "      <td>[English]</td>\n",
       "      <td>[Call, Call-chronicle-examiner, Sunday call]</td>\n",
       "      <td>sn85066387</td>\n",
       "      <td>California</td>\n",
       "      <td>JACK— he w.45 11111^^ Germany,\\nI^ll/ riFSIIOV...</td>\n",
       "      <td>curiv_llano_ver01</td>\n",
       "      <td>san francisco call.</td>\n",
       "      <td>https://chroniclingamerica.loc.gov/lccn/sn8506...</td>\n",
       "      <td>[California--San Francisco--San Francisco]</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sequence           county   edition frequency                                        id                                            subject             city       date                             title  end_year                                               note         state section_label  type    place_of_publication  start_year edition_label              publisher   language                                     alt_title        lccn     country                                            ocr_eng                batch            title_normal                                                url                                       place page\n",
       "0        11          [Wayne]  Home ed.     Daily  /lccn/sn88063294/1944-09-12/ed-1/seq-11/  [Detroit (Mich.)--Newspapers., Michigan--Detro...        [Detroit] 1944-09-12            Detroit evening times.      1958  [Archived issues are available in digital form...    [Michigan]                page           Detroit, Mich        1921       REDLINE                 [s.n.]  [English]         [Detroit Sunday times, Detroit times]  sn88063294    Michigan  Where the Battle of Germany Will Be Fought\\n|D...  mimtptc_holly_ver03  detroit evening times.  https://chroniclingamerica.loc.gov/lccn/sn8806...                  [Michigan--Wayne--Detroit]     \n",
       "1        16  [San Francisco]      None     Daily  /lccn/sn85066387/1908-11-08/ed-1/seq-16/  [California--San Francisco Bay Area.--fast--(O...  [San Francisco] 1908-11-08  The San Francisco call. [volume]      1913  [\"San Francisco\" appears above, and later acro...  [California]                page  San Francisco [Calif.]        1895                Charles M. Shortridge  [English]  [Call, Call-chronicle-examiner, Sunday call]  sn85066387  California  JACK— he w.45 11111^^ Germany,\\nI^ll/ riFSIIOV...    curiv_llano_ver01     san francisco call.  https://chroniclingamerica.loc.gov/lccn/sn8506...  [California--San Francisco--San Francisco]   16"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!pip install -q pandas\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(extract)\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - downloading images\n",
    "\n",
    "We can also `requests` to download things other than text - such as images.\n",
    "\n",
    "Below we do a requests and see we get back a binary string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'�PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x02 \\x00\\x00\\x00�\\x08\\x06\\x00\\x00\\x00�#W\\x1b\\x00\\x004�IDATx\\x01��\\x03�%?\\x1e��m۶}����(�Y:��n��z��m۶���w�$=�ח�\\u05fb�^����N�g���|��_'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png'\n",
    "res = requests.get(url)\n",
    "res.text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use context management to dump the contents of the binary string into a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./data/google-logo.png', 'wb') as fi:\n",
    "    fi.write(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the Google logo locally (you may need to re-run this cell)\n",
    "\n",
    "![](./data/google-logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exericse (group) - earthquake API \n",
    "\n",
    "Let's as a group write a program to get data from the USGS Earthquake Catalog - [documentation](https://earthquake.usgs.gov/fdsnws/event/1/#methods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = \"2014-01-01\"\n",
    "end = \"2014-01-02\"\n",
    "url = f\"https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime={start}&endtime={end}\"\n",
    "\n",
    "import os\n",
    "os.makedirs('./data/earthquakes', exist_ok=True)\n",
    "os.makedirs(f'./data/earthquakes/{start}', exist_ok=True)\n",
    "\n",
    "response = requests.get(url)\n",
    "data = response.json()\n",
    "features = data['features']\n",
    "for feature in features:\n",
    "    properties = feature['properties']\n",
    "  \n",
    "    url = properties['url'].split('/')[-1]\n",
    "    path = f'./data/earthquakes/{start}/{url}.json'\n",
    "    with open(path, 'w') as fi:\n",
    "        json.dump(properties, fi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise (individual) - Wikipedia API\n",
    "\n",
    "Now for an open-ended exercise for you! Your task is to:\n",
    "- create a database of countries\n",
    "- in a folder called `countries` (you will need to make the folder - you can do this in bash or Python)\n",
    "- each country in it's own folder\n",
    "- start with germany & new zealand\n",
    "\n",
    "V1 of your program should:\n",
    "- save the url you use to request the data\n",
    "- save the title\n",
    "- save the `line` parameter of each section (`data['parse']['sections']`)\n",
    "- save all in a single JSON\n",
    "\n",
    "V2 of your program should also:\n",
    "- save all '.png' & '.jpg' images as images, with the url as the image name\n",
    "- save all external links as CSV\n",
    "\n",
    "Much of the work will be understanding how the Wikipedia API works - useful resources are below:\n",
    "\n",
    "- [Main API page](https://www.mediawiki.org/wiki/API:Main_page),\n",
    "- [What the actions are](https://www.mediawiki.org/w/api.php),\n",
    "- [Python examples](https://github.com/wikimedia/mediawiki-api-demos/tree/master/python),\n",
    "- [Wikipedia API documentation](https://en.wikipedia.org/api/rest_v1/#/Page%20content).\n",
    "\n",
    "Please also feel free to work on another API - happy to assist you with this as well :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Etymology',\n",
       "  'number': '1',\n",
       "  'index': '1',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 13212,\n",
       "  'anchor': 'Etymology',\n",
       "  'linkAnchor': 'Etymology'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'History',\n",
       "  'number': '2',\n",
       "  'index': '2',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 15785,\n",
       "  'anchor': 'History',\n",
       "  'linkAnchor': 'History'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Germanic tribes, Roman frontier and the Frankish Empire',\n",
       "  'number': '2.1',\n",
       "  'index': '3',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 19765,\n",
       "  'anchor': 'Germanic_tribes,_Roman_frontier_and_the_Frankish_Empire',\n",
       "  'linkAnchor': 'Germanic_tribes,_Roman_frontier_and_the_Frankish_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'East Francia and the Holy Roman Empire',\n",
       "  'number': '2.2',\n",
       "  'index': '4',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 24541,\n",
       "  'anchor': 'East_Francia_and_the_Holy_Roman_Empire',\n",
       "  'linkAnchor': 'East_Francia_and_the_Holy_Roman_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'German Confederation and Empire',\n",
       "  'number': '2.3',\n",
       "  'index': '5',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 32576,\n",
       "  'anchor': 'German_Confederation_and_Empire',\n",
       "  'linkAnchor': 'German_Confederation_and_Empire'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Weimar Republic and Nazi Germany',\n",
       "  'number': '2.4',\n",
       "  'index': '6',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 41187,\n",
       "  'anchor': 'Weimar_Republic_and_Nazi_Germany',\n",
       "  'linkAnchor': 'Weimar_Republic_and_Nazi_Germany'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'East and West Germany',\n",
       "  'number': '2.5',\n",
       "  'index': '7',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 54471,\n",
       "  'anchor': 'East_and_West_Germany',\n",
       "  'linkAnchor': 'East_and_West_Germany'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Reunified Germany and the European Union',\n",
       "  'number': '2.6',\n",
       "  'index': '8',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 62641,\n",
       "  'anchor': 'Reunified_Germany_and_the_European_Union',\n",
       "  'linkAnchor': 'Reunified_Germany_and_the_European_Union'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Geography',\n",
       "  'number': '3',\n",
       "  'index': '9',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 69140,\n",
       "  'anchor': 'Geography',\n",
       "  'linkAnchor': 'Geography'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Climate',\n",
       "  'number': '3.1',\n",
       "  'index': '10',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 71474,\n",
       "  'anchor': 'Climate',\n",
       "  'linkAnchor': 'Climate'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Biodiversity',\n",
       "  'number': '3.2',\n",
       "  'index': '11',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 74045,\n",
       "  'anchor': 'Biodiversity',\n",
       "  'linkAnchor': 'Biodiversity'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Politics',\n",
       "  'number': '4',\n",
       "  'index': '12',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 81174,\n",
       "  'anchor': 'Politics',\n",
       "  'linkAnchor': 'Politics'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Constituent states',\n",
       "  'number': '4.1',\n",
       "  'index': '13',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 86601,\n",
       "  'anchor': 'Constituent_states',\n",
       "  'linkAnchor': 'Constituent_states'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Law',\n",
       "  'number': '4.2',\n",
       "  'index': '14',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 93907,\n",
       "  'anchor': 'Law',\n",
       "  'linkAnchor': 'Law'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Foreign relations',\n",
       "  'number': '4.3',\n",
       "  'index': '15',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 98686,\n",
       "  'anchor': 'Foreign_relations',\n",
       "  'linkAnchor': 'Foreign_relations'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Military',\n",
       "  'number': '4.4',\n",
       "  'index': '16',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 104871,\n",
       "  'anchor': 'Military',\n",
       "  'linkAnchor': 'Military'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Economy',\n",
       "  'number': '5',\n",
       "  'index': '17',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 114628,\n",
       "  'anchor': 'Economy',\n",
       "  'linkAnchor': 'Economy'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Infrastructure',\n",
       "  'number': '5.1',\n",
       "  'index': '18',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 129434,\n",
       "  'anchor': 'Infrastructure',\n",
       "  'linkAnchor': 'Infrastructure'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Tourism',\n",
       "  'number': '5.2',\n",
       "  'index': '19',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 138205,\n",
       "  'anchor': 'Tourism',\n",
       "  'linkAnchor': 'Tourism'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Demographics',\n",
       "  'number': '6',\n",
       "  'index': '20',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 140209,\n",
       "  'anchor': 'Demographics',\n",
       "  'linkAnchor': 'Demographics'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Religion',\n",
       "  'number': '6.1',\n",
       "  'index': '21',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 146957,\n",
       "  'anchor': 'Religion',\n",
       "  'linkAnchor': 'Religion'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Languages',\n",
       "  'number': '6.2',\n",
       "  'index': '22',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 153436,\n",
       "  'anchor': 'Languages',\n",
       "  'linkAnchor': 'Languages'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Education',\n",
       "  'number': '6.3',\n",
       "  'index': '23',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 156351,\n",
       "  'anchor': 'Education',\n",
       "  'linkAnchor': 'Education'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Health',\n",
       "  'number': '6.4',\n",
       "  'index': '24',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 161787,\n",
       "  'anchor': 'Health',\n",
       "  'linkAnchor': 'Health'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Culture',\n",
       "  'number': '7',\n",
       "  'index': '25',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 165539,\n",
       "  'anchor': 'Culture',\n",
       "  'linkAnchor': 'Culture'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Music',\n",
       "  'number': '7.1',\n",
       "  'index': '26',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 169311,\n",
       "  'anchor': 'Music',\n",
       "  'linkAnchor': 'Music'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Art, design and architecture',\n",
       "  'number': '7.2',\n",
       "  'index': '27',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 172729,\n",
       "  'anchor': 'Art,_design_and_architecture',\n",
       "  'linkAnchor': 'Art,_design_and_architecture'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Literature and philosophy',\n",
       "  'number': '7.3',\n",
       "  'index': '28',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 178420,\n",
       "  'anchor': 'Literature_and_philosophy',\n",
       "  'linkAnchor': 'Literature_and_philosophy'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Media',\n",
       "  'number': '7.4',\n",
       "  'index': '29',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 182937,\n",
       "  'anchor': 'Media',\n",
       "  'linkAnchor': 'Media'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Cuisine',\n",
       "  'number': '7.5',\n",
       "  'index': '30',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 188446,\n",
       "  'anchor': 'Cuisine',\n",
       "  'linkAnchor': 'Cuisine'},\n",
       " {'toclevel': 2,\n",
       "  'level': '3',\n",
       "  'line': 'Sports',\n",
       "  'number': '7.6',\n",
       "  'index': '31',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 193793,\n",
       "  'anchor': 'Sports',\n",
       "  'linkAnchor': 'Sports'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'See also',\n",
       "  'number': '8',\n",
       "  'index': '32',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 199241,\n",
       "  'anchor': 'See_also',\n",
       "  'linkAnchor': 'See_also'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'Notes',\n",
       "  'number': '9',\n",
       "  'index': '33',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 199301,\n",
       "  'anchor': 'Notes',\n",
       "  'linkAnchor': 'Notes'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'References',\n",
       "  'number': '10',\n",
       "  'index': '34',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 199332,\n",
       "  'anchor': 'References',\n",
       "  'linkAnchor': 'References'},\n",
       " {'toclevel': 1,\n",
       "  'level': '2',\n",
       "  'line': 'External links',\n",
       "  'number': '11',\n",
       "  'index': '35',\n",
       "  'fromtitle': 'Germany',\n",
       "  'byteoffset': 199916,\n",
       "  'anchor': 'External_links',\n",
       "  'linkAnchor': 'External_links'}]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "url = f\"https://en.wikipedia.org/w/api.php?action=parse&page={term}&format=json\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "\n",
    "data['parse']['sections']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "term = 'germany'\n",
    "os.makedirs(f'./data/countries/{term}', exist_ok=True)\n",
    "os.makedirs(f'./data/countries/{term}/images', exist_ok=True)\n",
    "\n",
    "url = f\"https://en.wikipedia.org/w/api.php?action=parse&page={term}&format=json\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "\n",
    "data = data['parse']\n",
    "out = {\n",
    "    'url': url,\n",
    "    'title': data['title']\n",
    "}\n",
    "\n",
    "path = os.path.join('data', 'countries', term, 'data.json')\n",
    "with open(path, 'w') as fi:\n",
    "    json.dump(out, fi)\n",
    "    \n",
    "for img in data['images']:\n",
    "    url = f'https://en.wikipedia.org/w/api.php?action=query&format=json&list=allimages&aifrom={img}&ailimit=1'\n",
    "    res = requests.get(url)\n",
    "    res = res.json()\n",
    "    with open(f'./data/countries/{term}/images/{img}', 'wb') as fi:\n",
    "        url = res['query']['allimages'][0]['url']\n",
    "        res = requests.get(url)\n",
    "        fi.write(res.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "term = 'germany'\n",
    "os.makedirs(f'./data/countries/{term}', exist_ok=True)\n",
    "os.makedirs(f'./data/countries/{term}/images', exist_ok=True)\n",
    "\n",
    "url = f\"https://en.wikipedia.org/w/api.php?action=parse&page={term}&format=json\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "\n",
    "data = data['parse']\n",
    "out = {\n",
    "    'url': url,\n",
    "    'title': data['title']\n",
    "}\n",
    "\n",
    "path = os.path.join('data', 'countries', term, 'data.json')\n",
    "with open(path, 'w') as fi:\n",
    "    json.dump(out, fi)\n",
    "    \n",
    "for img in data['images']:\n",
    "    url = f'https://en.wikipedia.org/w/api.php?action=query&format=json&list=allimages&aifrom={img}&ailimit=1'\n",
    "    res = requests.get(url)\n",
    "    res = res.json()\n",
    "    with open(f'./data/countries/{term}/images/{img}', 'wb') as fi:\n",
    "        url = res['query']['allimages'][0]['url']\n",
    "        res = requests.get(url)\n",
    "        fi.write(res.content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}