Web Screping Material

itversity-material/01-python-and-sql/01_getting_started_with_postgresql/06_exercise_loading_data_into_postgres_database.ipynb

@@ -84,7 +84,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.12"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

itversity-material/01-python-and-sql/35_processing_rest_payload/02_overview_of_rest_apis.ipynb

@@ -0,0 +1,67 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a94e5edb",
+   "metadata": {},
+   "source": [
+    "## Overview of REST APIs\n",
+    "\n",
+    "Let us get an overview of REST APIs. REST based APIs are extensively used for building large scale applications.\n",
+    "* REST stands for **Representational State Transfer**.\n",
+    "* It is extensively used to build modern applications as it simplifies the process of building complex and large scale applications.\n",
+    "* Here are the use cases where REST APIs are extensively used.\n",
+    "  * Mobile applications\n",
+    "  * Web applications\n",
+    "  * Data integration between multiple applications\n",
+    "* For mobile and web applications, we typically develop frontend and backend separately and integrate them over REST.\n",
+    "* Here are the commonly used REST based requests.\n",
+    "  * GET - Get the data from the backend\n",
+    "  * POST - Insert or update the data in the backend\n",
+    "  * PUT - Update the data in the backend\n",
+    "  * DELETE - Delete the data in the backend\n",
+    "* As part of data engineering projects, we typically use REST to get the data from external applications.\n",
+    "* If required, we can take care of authentication and authorization by leveraging **Headers** while placing the requests.\n",
+    "* Let us see an example using GitHub REST APIs. As a developer you might want to do the following:\n",
+    "  * Get your repositories or public repositories. You can place **GET** request from external applications.\n",
+    "  * Create a new repository under your GitHub Account. You can place **POST** request from external application.\n",
+    "  * Delete an existing repository under your GitHub Account. You can place **DELETE** request form external application.\n",
+    "* Here are different ways you can validate REST APIs without getting into programming.\n",
+    "  * Using browser (it might not work for complex requests)\n",
+    "  * Using `curl` command.\n",
+    "  * Using a client based utility called as **postman**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "40c1f1f1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!curl https://api.github.com"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

itversity-material/01-python-and-sql/35_processing_rest_payload/04_overview_of_postman.ipynb

@@ -0,0 +1,62 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7d76eed4",
+   "metadata": {},
+   "source": [
+    "## Overview of Postman\n",
+    "\n",
+    "**Postman** is the most popular way of exploring REST APIs with out getting into programming nuances. Let us get an overview of **postman**.\n",
+    "* You can use postman via browser, however we recommend to download and install the client utility.\n",
+    "* Here are the some of the key features related to Postman.\n",
+    "  * It is a freemium product, and provide lot of features at no cost.\n",
+    "  * Grouping related APIs to collections.\n",
+    "  * Passing values dynamically using variables. Variables can be global or collection level.\n",
+    "  * Ability to see the direct curl command as well as examples using standard programming languages.\n",
+    "  * Ability to automate the execution.\n",
+    "* Here is an example of making `GET` call via `requests` module using Python as programming language."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6f9228be",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = \"https://api.github.com/users/dgadiraju/repos\"\n",
+    "\n",
+    "payload={}\n",
+    "headers = {}\n",
+    "\n",
+    "response = requests.request(\"GET\", url, headers=headers, data=payload)\n",
+    "\n",
+    "print(response.text)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}