section-2-data-science-and-ml-tools/introduction-to-numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Quick Introduction to Numerical Data Manipulation with Python and NumPy\n",
    "\n",
    "\n",
    "## What is NumPy?\n",
    "\n",
    "[NumPy](https://docs.scipy.org/doc/numpy/index.html) stands for numerical Python. It's the backbone of all kinds of scientific and numerical computing in Python.\n",
    "\n",
    "And since machine learning is all about turning data into numbers and then figuring out the patterns, NumPy often comes into play.\n",
    "\n",
    "<img src=\"../images/numpy-6-step-ml-framework-tools-numpy-highlight.png\" alt=\"a 6 step machine learning framework along will tools you can use for each step\" width=\"700\"/>\n",
    "\n",
    "## Why NumPy?\n",
    "\n",
    "You can do numerical calculations using pure Python. In the beginning, you might think Python is fast but once your data gets large, you'll start to notice slow downs.\n",
    "\n",
    "One of the main reasons you use NumPy is because it's fast. Behind the scenes, the code has been optimized to run using C. Which is another programming language, which can do things much faster than Python.\n",
    "\n",
    "The benefit of this being behind the scenes is you don't need to know any C to take advantage of it. You can write your numerical computations in Python using NumPy and get the added speed benefits.\n",
    "\n",
    "If your curious as to what causes this speed benefit, it's a process called vectorization. [Vectorization](https://en.wikipedia.org/wiki/Vectorization) aims to do calculations by avoiding loops as loops can create potential bottlenecks.\n",
    "\n",
    "NumPy achieves vectorization through a process called [broadcasting](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html#module-numpy.doc.broadcasting).\n",
    "\n",
    "\n",
    "## What does this notebook cover?\n",
    "\n",
    "The NumPy library is very capable. However, learning everything off by heart isn't necessary. Instead, this notebook focuses on the main concepts of NumPy and the `ndarray` datatype.\n",
    "\n",
    "You can think of the `ndarray` datatype as a very flexible array of numbers.\n",
    "\n",
    "More specifically, we'll look at:\n",
    "* NumPy datatypes & attributes\n",
    "* Creating arrays\n",
    "* Viewing arrays & matrices (indexing)\n",
    "* Manipulating & comparing arrays\n",
    "* Sorting arrays\n",
    "* Use cases (examples of turning things into numbers)\n",
    "\n",
    "After going through it, you'll have the base knolwedge of NumPy you need to keep moving forward.\n",
    "\n",
    "## Where can I get help?\n",
    "If you get stuck or think of something you'd like to do which this notebook doesn't cover, don't fear!\n",
    "\n",
    "The recommended steps you take are:\n",
    "1. **Try it** - Since NumPy is very friendly, your first step should be to use what you know and try figure out the answer to your own question (getting it wrong is part of the process). If in doubt, run your code.\n",
    "2. **Search for it** - If trying it on your own doesn't work, since someone else has probably tried to do something similar, try searching for your problem. You'll likely end up in 1 of 2 places:\n",
    "    * [NumPy documentation](https://docs.scipy.org/doc/numpy/index.html) - the ground truth for everything NumPy, this resource covers all of the NumPy functionality.\n",
    "    * [Stack Overflow](https://stackoverflow.com/) - this is the developers Q&A hub, it's full of questions and answers of different problems across a wide range of software development topics and chances are, there's one related to your problem.\n",
    "    \n",
    "An example of searching for a NumPy function might be:\n",
    "\n",
    "> \"how to find unique elements in a numpy array\"\n",
    "\n",
    "Searching this on Google leads to the NumPy documentation for the `np.uniquie()` function: https://docs.scipy.org/doc/numpy/reference/generated/numpy.unique.html\n",
    "\n",
    "The next steps here are to read through the documentation, check the examples and see if they line up to the problem you're trying to solve. If they do, **rewrite the code** to suit your needs, run it, and see what the outcomes are.\n",
    "\n",
    "3. **Ask for help** - If you've been through the above 2 steps and you're still stuck, you might want to ask your question on [Stack Overflow](https://www.stackoverflow.com). Be as specific as possible and provide details on what you've tried.\n",
    "\n",
    "Remember, you don't have to learn all of the functions off by heart to begin with. \n",
    "\n",
    "What's most important is continually asking yourself, \"what am I trying to do with the data?\".\n",
    "\n",
    "Start by answering that question and then practicing finding the code which does it.\n",
    "\n",
    "Let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Importing NumPy\n",
    "\n",
    "To get started using NumPy, the first step is to import it. \n",
    "\n",
    "The most common way (and method you should use) is to import NumPy as the abbreviation `np`.\n",
    "\n",
    "If you see the letters `np` used anywhere in machine learning or data science, it's probably referring to the NumPy library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. DataTypes and attributes\n",
    "\n",
    "**NOTE:** Important to remember the main type in NumPy is `ndarray`, even seemingly different kinds of arrays are still `ndarray`'s. This means an operation you do on one array, will work on another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1-dimensonal array, also referred to as a vector\n",
    "a1 = np.array([1, 2, 3])\n",
    "\n",
    "# 2-dimensional array, also referred to as matrix\n",
    "a2 = np.array([[1, 2.0, 3.3],\n",
    "               [4, 5, 6.5]])\n",
    "\n",
    "# 3-dimensional array, also referred to as a matrix\n",
    "a3 = np.array([[[1, 2, 3],\n",
    "                [4, 5, 6],\n",
    "                [7, 8, 9]],\n",
    "                [[10, 11, 12],\n",
    "                 [13, 14, 15],\n",
    "                 [16, 17, 18]]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3,), 1, dtype('int64'), 3, numpy.ndarray)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1.shape, a1.ndim, a1.dtype, a1.size, type(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2, 3), 2, dtype('float64'), 6, numpy.ndarray)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2.shape, a2.ndim, a2.dtype, a2.size, type(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2, 3, 3), 3, dtype('int64'), 18, numpy.ndarray)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3.shape, a3.ndim, a3.dtype, a3.size, type(a3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1. , 2. , 3.3],\n",
       "       [4. , 5. , 6.5]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 1,  2,  3],\n",
       "        [ 4,  5,  6],\n",
       "        [ 7,  8,  9]],\n",
       "\n",
       "       [[10, 11, 12],\n",
       "        [13, 14, 15],\n",
       "        [16, 17, 18]]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Anatomy of an array\n",
    "\n",
    "<img src=\"../images/numpy-anatomy-of-an-array-updated.png\" alt=\"anatomy of a numpy array\"/>\n",
    "\n",
    "Key terms:\n",
    "* **Array** - A list of numbers, can be multi-dimensional.\n",
    "* **Scalar** - A single number (e.g. `7`).\n",
    "* **Vector** - A list of numbers with 1-dimesion (e.g. `np.array([1, 2, 3])`).\n",
    "* **Matrix** - A (usually) multi-deminsional list of numbers (e.g. `np.array([[1, 2, 3], [4, 5, 6]])`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pandas DataFrame out of NumPy arrays\n",
    "\n",
    "This is to examplify how NumPy is the backbone of many other libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  b  c\n",
       "0  5  5  8\n",
       "1  3  6  0\n",
       "2  5  1  4\n",
       "3  8  9  5\n",
       "4  1  4  9"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(np.random.randint(10, size=(5, 3)), \n",
    "                                    columns=['a', 'b', 'c'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1. , 2. , 3.3],\n",
       "       [4. , 5. , 6.5]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2\n",
       "0  1.0  2.0  3.3\n",
       "1  4.0  5.0  6.5"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame(a2)\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Creating arrays\n",
    "\n",
    "* `np.array()`\n",
    "* `np.ones()`\n",
    "* `np.zeros()`\n",
    "* `np.random.rand(5, 3)`\n",
    "* `np.random.randint(10, size=5)`\n",
    "* `np.random.seed()` - pseudo random numbers\n",
    "* Searching the documentation example (finding `np.unique()` and using it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a simple array\n",
    "simple_array = np.array([1, 2, 3])\n",
    "simple_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1, 2, 3]), dtype('int64'))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_array = np.array((1, 2, 3))\n",
    "simple_array, simple_array.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.],\n",
       "       [1., 1.]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an array of ones\n",
    "ones = np.ones((10, 2))\n",
    "ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The default datatype is 'float64'\n",
    "ones.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1],\n",
       "       [1, 1]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can change the datatype with .astype()\n",
    "ones.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an array of zeros\n",
    "zeros = np.zeros((5, 3, 3))\n",
    "zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zeros.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 4, 6, 8])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an array within a range of values\n",
    "range_array = np.arange(0, 10, 2)\n",
    "range_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 2, 4],\n",
       "       [0, 9, 0],\n",
       "       [6, 7, 5],\n",
       "       [0, 1, 9],\n",
       "       [5, 3, 1]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random array\n",
    "random_array = np.random.randint(10, size=(5, 3))\n",
    "random_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.20153841, 0.37582038, 0.60296171],\n",
       "       [0.78922978, 0.61651979, 0.39181855],\n",
       "       [0.21325988, 0.53159683, 0.97521884],\n",
       "       [0.12675152, 0.34538155, 0.60466876],\n",
       "       [0.66714451, 0.92307148, 0.85628976]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random array of floats (between 0 & 1)\n",
    "np.random.random((5, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.41006771, 0.25834425, 0.75566325],\n",
       "       [0.48997703, 0.23325182, 0.47949583],\n",
       "       [0.00629117, 0.98519241, 0.25080785],\n",
       "       [0.17224824, 0.47306183, 0.30073575],\n",
       "       [0.54365477, 0.05714738, 0.43913006]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.random((5, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.66962815, 0.70759503, 0.81030571],\n",
       "       [0.45215695, 0.00173544, 0.71588354],\n",
       "       [0.30440026, 0.97664503, 0.07787752],\n",
       "       [0.49839944, 0.2900812 , 0.83955077],\n",
       "       [0.06752272, 0.73464427, 0.0827841 ]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random 5x3 array of floats (between 0 & 1), similar to above\n",
    "np.random.rand(5, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.98809371, 0.50318032, 0.73969971],\n",
       "       [0.20350304, 0.15134646, 0.46323102],\n",
       "       [0.86984225, 0.13964899, 0.01276344],\n",
       "       [0.59930525, 0.83124397, 0.55221531],\n",
       "       [0.35222831, 0.12840147, 0.89773117]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.rand(5, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy uses pseudo-random numbers, which means, the numbers look random but aren't really, they're predetermined.\n",
    "\n",
    "For consistency, you might want to keep the random numbers you generate similar throughout experiments.\n",
    "\n",
    "To do this, you can use [`np.random.seed()`](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.seed.html).\n",
    "\n",
    "What this does is it tells NumPy, \"Hey, I want you to create random numbers but keep them aligned with the seed.\"\n",
    "\n",
    "Let's see it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5, 0, 3],\n",
       "       [3, 7, 9],\n",
       "       [3, 5, 2],\n",
       "       [4, 7, 6],\n",
       "       [8, 8, 1]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set random seed to 0\n",
    "np.random.seed(0)\n",
    "\n",
    "# Make 'random' numbers\n",
    "np.random.randint(10, size=(5, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `np.random.seed()` set, every time you run the cell above, the same random numbers will be generated.\n",
    "\n",
    "What if `np.random.seed()` wasn't set?\n",
    "\n",
    "Every time you run the cell below, a new set of numbers will appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6, 7, 7],\n",
       "       [8, 1, 5],\n",
       "       [9, 8, 9],\n",
       "       [4, 3, 0],\n",
       "       [3, 5, 0]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make more random numbers\n",
    "np.random.randint(10, size=(5, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see it in action again, we'll stay consistent and set the random seed to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5, 0, 3],\n",
       "       [3, 7, 9],\n",
       "       [3, 5, 2],\n",
       "       [4, 7, 6],\n",
       "       [8, 8, 1]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set random seed to same number as above\n",
    "np.random.seed(0)\n",
    "\n",
    "# The same random numbers come out\n",
    "np.random.randint(10, size=(5, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because `np.random.seed()` is set to 0, the random numbers are the same as the cell with `np.random.seed()` set to 0 as well.\n",
    "\n",
    "Setting `np.random.seed()` is not 100% necessary but it's helpful to keep numbers the same throughout your experiments.\n",
    "\n",
    "For example, say you wanted to split your data randomly into training and test sets.\n",
    "\n",
    "Every time you randomly split, you might get different rows in each set.\n",
    "\n",
    "If you shared your work with someone else, they'd get different rows in each set too.\n",
    "\n",
    "Setting `np.random.seed()` ensures there's still randomness, it just makes the randomness repeatable. Hence the 'pseudo-random' numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2\n",
       "0  5  0  3\n",
       "1  3  7  9\n",
       "2  3  5  2\n",
       "3  4  7  6\n",
       "4  8  8  1"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "df = pd.DataFrame(np.random.randint(10, size=(5, 3)))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What unique values are in the array a3?\n",
    "\n",
    "Now you've seen a few different ways to create arrays, as an exercise, try find out what NumPy function you could use to find the unique values are within the `a3` array.\n",
    "\n",
    "You might want to search some like, \"how to find the unqiue values in a numpy array\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Viewing arrays and matrices (indexing)\n",
    "\n",
    "Remember, because arrays and matrices are both `ndarray`'s, they can be viewed in similar ways.\n",
    "\n",
    "Let's check out our 3 arrays again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1. , 2. , 3.3],\n",
       "       [4. , 5. , 6.5]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 1,  2,  3],\n",
       "        [ 4,  5,  6],\n",
       "        [ 7,  8,  9]],\n",
       "\n",
       "       [[10, 11, 12],\n",
       "        [13, 14, 15],\n",
       "        [16, 17, 18]]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Array shapes are always listed in the format `(row, column, n, n, n...)` where `n` is optional extra dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1. , 2. , 3.3])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6],\n",
       "       [7, 8, 9]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a3[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4. , 5. , 6.5])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get 2nd row (index 1) of a2\n",
    "a2[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 1,  2],\n",
       "        [ 4,  5]],\n",
       "\n",
       "       [[10, 11],\n",
       "        [13, 14]]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the first 2 values of the first 2 rows of both arrays\n",
    "a3[:2, :2, :2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This takes a bit of practice, especially when the dimensions get higher. Usually, it takes me a little trial and error of trying to get certain values, viewing the output in the notebook and trying again.\n",
    "\n",
    "NumPy arrays get printed from outside to inside. This means the number at the end of the shape comes first, and the number at the start of the shape comes last."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[[6, 7, 7, 8, 1],\n",
       "         [5, 9, 8, 9, 4],\n",
       "         [3, 0, 3, 5, 0],\n",
       "         [2, 3, 8, 1, 3]],\n",
       "\n",
       "        [[3, 3, 7, 0, 1],\n",
       "         [9, 9, 0, 4, 7],\n",
       "         [3, 2, 7, 2, 0],\n",
       "         [0, 4, 5, 5, 6]],\n",
       "\n",
       "        [[8, 4, 1, 4, 9],\n",
       "         [8, 1, 1, 7, 9],\n",
       "         [9, 3, 6, 7, 2],\n",
       "         [0, 3, 5, 9, 4]]],\n",
       "\n",
       "\n",
       "       [[[4, 6, 4, 4, 3],\n",
       "         [4, 4, 8, 4, 3],\n",
       "         [7, 5, 5, 0, 1],\n",
       "         [5, 9, 3, 0, 5]],\n",
       "\n",
       "        [[0, 1, 2, 4, 2],\n",
       "         [0, 3, 2, 0, 7],\n",
       "         [5, 9, 0, 2, 7],\n",
       "         [2, 9, 2, 3, 3]],\n",
       "\n",
       "        [[2, 3, 4, 1, 2],\n",
       "         [9, 1, 4, 6, 8],\n",
       "         [2, 3, 0, 0, 6],\n",
       "         [0, 6, 3, 3, 8]]]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a4 = np.random.randint(10, size=(2, 3, 4, 5))\n",
    "a4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 4, 5)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a4.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[[6, 7, 7, 8],\n",
       "         [5, 9, 8, 9],\n",
       "         [3, 0, 3, 5],\n",
       "         [2, 3, 8, 1]],\n",
       "\n",
       "        [[3, 3, 7, 0],\n",
       "         [9, 9, 0, 4],\n",
       "         [3, 2, 7, 2],\n",
       "         [0, 4, 5, 5]],\n",
       "\n",
       "        [[8, 4, 1, 4],\n",
       "         [8, 1, 1, 7],\n",
       "         [9, 3, 6, 7],\n",
       "         [0, 3, 5, 9]]],\n",
       "\n",
       "\n",
       "       [[[4, 6, 4, 4],\n",
       "         [4, 4, 8, 4],\n",
       "         [7, 5, 5, 0],\n",
       "         [5, 9, 3, 0]],\n",
       "\n",
       "        [[0, 1, 2, 4],\n",
       "         [0, 3, 2, 0],\n",
       "         [5, 9, 0, 2],\n",
       "         [2, 9, 2, 3]],\n",
       "\n",
       "        [[2, 3, 4, 1],\n",
       "         [9, 1, 4, 6],\n",
       "         [2, 3, 0, 0],\n",
       "         [0, 6, 3, 3]]]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get only the first 4 numbers of each single vector\n",
    "a4[:, :, :, :4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`a4`'s shape is (2, 3, 4, 5), this means it gets displayed like so:\n",
    "* Inner most array = size 5\n",
    "* Next array = size 4\n",
    "* Next array = size 3\n",
    "* Outer most array = size 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Manipulating and comparying arrays\n",
    "* Arithmetic\n",
    "    * `+`, `-`, `*`, `/`, `//`, `**`, `%`\n",
    "    * `np.exp()`\n",
    "    * `np.log()`\n",
    "    * [Dot product](https://www.mathsisfun.com/algebra/matrix-multiplying.html) - `np.dot()`\n",
    "    * Broadcasting\n",
    "* Aggregation\n",
    "    * `np.sum()` - faster than `.sum()`, make demo, np is really fast\n",
    "    * `np.mean()`\n",
    "    * `np.std()`\n",
    "    * `np.var()`\n",
    "    * `np.min()`\n",
    "    * `np.max()`\n",
    "    * `np.argmin()` - find index of minimum value\n",
    "    * `np.argmax()` - find index of maximum value\n",
    "    * These work on all `ndarray`'s\n",
    "        * `a4.min(axis=0)` -- you can use axis as well\n",
    "* Reshaping\n",
    "    * `np.reshape()`\n",
    "* Transposing\n",
    "    * `a3.T` \n",
    "* Comparison operators\n",
    "    * `>`\n",
    "    * `<`\n",
    "    * `<=`\n",
    "    * `>=`\n",
    "    * `x != 3`\n",
    "    * `x == 3`\n",
    "    * `np.sum(x > 3)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arithmetic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1.])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ones = np.ones(3)\n",
    "ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2., 3., 4.])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add two arrays\n",
    "a1 + ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 2.])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subtract two arrays\n",
    "a1 - ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 2., 3.])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply two arrays\n",
    "a1 * ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. ,  4. ,  9.9],\n",
       "       [ 4. , 10. , 19.5]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply two arrays\n",
    "a1 * a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3,), (2, 3))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1.shape, a2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "operands could not be broadcast together with shapes (2,3) (2,3,3) ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-37d8620f1b2c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0ma2\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0ma3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: operands could not be broadcast together with shapes (2,3) (2,3,3) "
     ]
    }
   ],
   "source": [
    "a2 * a3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting\n",
    "\n",
    "- What is broadcasting?\n",
    "    - Broadcasting is a feature of NumPy which performs an operation across multiple dimensions of data without replicating the data. This saves time and space. For example, if you have a 3x3 array (A) and want to add a 1x3 array (B), NumPy will add the row of (B) to every row of (A).\n",
    "\n",
    "- Rules of Broadcasting\n",
    "    1. If the two arrays differ in their number of dimensions, the shape of the one with fewer dimensions is padded with ones on its leading (left) side.\n",
    "    2. If the shape of the two arrays does not match in any dimension, the array with shape equal to 1 in that dimension is stretched to match the other shape.\n",
    "    3. If in any dimension the sizes disagree and neither is equal to 1, an error is raised.\n",
    "    \n",
    "    \n",
    "**The broadcasting rule:**\n",
    "In order to broadcast, the size of the trailing axes for both arrays in an operation must be either the same size or one of them must be one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 + a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2 + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Raises an error because there's a shape mismatch\n",
    "a2 + a3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide two arrays\n",
    "a1 / ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide using floor division\n",
    "a2 // a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take an array to a power\n",
    "a1 ** 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also use np.square()\n",
    "np.square(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modulus divide (what's the remainder)\n",
    "a1 % 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also find the log or exponential of an array using `np.log()` and `np.exp()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the log of an array\n",
    "np.log(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the exponential of an array\n",
    "np.exp(a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregation\n",
    "\n",
    "Aggregation - bringing things together, doing a similar thing on a number of things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use NumPy's `np.sum()` on NumPy arrays and Python's `sum()` on Python lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "massive_array = np.random.random(100000)\n",
    "massive_array.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit sum(massive_array) # Python sum()\n",
    "%timeit np.sum(massive_array) # NumPy np.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random \n",
    "massive_list = [random.randint(0, 10) for i in range(100000)]\n",
    "len(massive_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "massive_list[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit sum(massive_list)\n",
    "%timeit np.sum(massive_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the mean\n",
    "np.mean(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the max\n",
    "np.max(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the min\n",
    "np.min(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the standard deviation\n",
    "np.std(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the variance\n",
    "np.var(a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The standard deviation is the square root of the variance\n",
    "np.sqrt(np.var(a2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What's mean?**\n",
    "\n",
    "Mean is the same as average. You can find the average of a set of numbers by adding them up and dividing them by how many there are.\n",
    "\n",
    "**What's standard deviation?**\n",
    "\n",
    "[Standard deviation](https://www.mathsisfun.com/data/standard-deviation.html) is a measure of how spread out numbers are.\n",
    "\n",
    "**What's variance?**\n",
    "\n",
    "The [variance](https://www.mathsisfun.com/data/standard-deviation.html) is the averaged squared differences of the mean.\n",
    "\n",
    "To work it out, you:\n",
    "1. Work out the mean\n",
    "2. For each number, subtract the mean and square the result\n",
    "3. Find the average of the squared differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo of variance\n",
    "high_var_array = np.array([1, 100, 200, 300, 4000, 5000])\n",
    "low_var_array = np.array([2, 4, 6, 8, 10])\n",
    "\n",
    "np.var(high_var_array), np.var(low_var_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.std(high_var_array), np.std(low_var_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The standard deviation is the square root of the variance\n",
    "np.sqrt(np.var(high_var_array))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.hist(high_var_array)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(low_var_array)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2 + a3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.reshape(2, 3, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.reshape(2, 3, 1) + a3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.T.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.random(size=(5,3,3))\n",
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix.T.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dot\n",
    "\n",
    "* TODO - create graphic for dot versus element-wise also known as Hadamard product\n",
    "* TODO - why would someone use dot versus element-wise?\n",
    "* A dot product models real world problems well, it's a method of finding patterns between data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "mat1 = np.random.randint(10, size=(3, 3))\n",
    "mat2 = np.random.randint(10, size=(3, 2))\n",
    "\n",
    "mat1.shape, mat2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(mat1, mat2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "mat3 = np.random.randint(10, size=(4,3))\n",
    "mat4 = np.random.randint(10, size=(4,3))\n",
    "mat3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(mat3, mat4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat3.T.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dot product\n",
    "np.dot(mat3.T, mat4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Element-wise multiplication, also known as Hadamard product\n",
    "mat3 * mat4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dot product practical example, nut butter sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "sales_amounts = np.random.randint(20, size=(5, 3))\n",
    "sales_amounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weekly_sales = pd.DataFrame(sales_amounts,\n",
    "                            index=[\"Mon\", \"Tues\", \"Wed\", \"Thurs\", \"Fri\"],\n",
    "                            columns=[\"Almond butter\", \"Peanut butter\", \"Cashew butter\"])\n",
    "weekly_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prices = np.array([10, 8, 12])\n",
    "prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "butter_prices = pd.DataFrame(prices.reshape(1, 3),\n",
    "                             index=[\"Price\"],\n",
    "                             columns=[\"Almond butter\", \"Peanut butter\", \"Cashew butter\"])\n",
    "butter_prices.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weekly_sales.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the total amount of sales for a whole day\n",
    "total_sales = prices.dot(sales_amounts)\n",
    "total_sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shapes aren't aligned, we need the middle two numbers to be the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_amounts.T.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To make the middle numbers the same, we can transpose\n",
    "total_sales = prices.dot(sales_amounts.T)\n",
    "total_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "butter_prices.shape, weekly_sales.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daily_sales = butter_prices.dot(weekly_sales.T)\n",
    "daily_sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Need to transpose again\n",
    "weekly_sales[\"Total\"] = daily_sales.T\n",
    "weekly_sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison operators "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 > a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 >= a2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 > 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 == a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 == a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Sorting arrays\n",
    "\n",
    "* `np.sort()`\n",
    "* `np.argsort()`\n",
    "* `np.argmax()`\n",
    "* `np.argmin()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sort(random_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argsort(random_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the indices that would sort an array\n",
    "np.argsort(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# No axis\n",
    "np.argmin(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Down the vertical\n",
    "np.argmax(random_array, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Across the horizontal\n",
    "np.argmin(random_array, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Use case\n",
    "\n",
    "Turning an image of a panda into numbers.\n",
    "\n",
    "<img src=\"../images/numpy-panda.png\" alt=\"photo of a panda waving\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "from matplotlib.image import imread\n",
    "\n",
    "panda = imread('../images/numpy-panda.png')\n",
    "print(type(panda))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2330, 3500, 3)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panda.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        ...,\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765]],\n",
       "\n",
       "       [[0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        ...,\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765]],\n",
       "\n",
       "       [[0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        [0.05490196, 0.10588235, 0.06666667],\n",
       "        ...,\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765],\n",
       "        [0.16470589, 0.12941177, 0.09411765]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0.13333334, 0.07450981, 0.05490196],\n",
       "        [0.12156863, 0.0627451 , 0.04313726],\n",
       "        [0.10980392, 0.05098039, 0.03137255],\n",
       "        ...,\n",
       "        [0.02745098, 0.02745098, 0.03529412],\n",
       "        [0.02745098, 0.02745098, 0.03529412],\n",
       "        [0.02745098, 0.02745098, 0.03529412]],\n",
       "\n",
       "       [[0.13333334, 0.07450981, 0.05490196],\n",
       "        [0.12156863, 0.0627451 , 0.04313726],\n",
       "        [0.12156863, 0.0627451 , 0.04313726],\n",
       "        ...,\n",
       "        [0.02352941, 0.02352941, 0.03137255],\n",
       "        [0.02352941, 0.02352941, 0.03137255],\n",
       "        [0.02352941, 0.02352941, 0.03137255]],\n",
       "\n",
       "       [[0.13333334, 0.07450981, 0.05490196],\n",
       "        [0.12156863, 0.0627451 , 0.04313726],\n",
       "        [0.12156863, 0.0627451 , 0.04313726],\n",
       "        ...,\n",
       "        [0.02352941, 0.02352941, 0.03137255],\n",
       "        [0.02352941, 0.02352941, 0.03137255],\n",
       "        [0.02352941, 0.02352941, 0.03137255]]], dtype=float32)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../images/numpy-car-photo.png\" alt=\"photo of a car\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(431, 575, 4)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car = imread(\"../images/numpy-car-photo.png\")\n",
    "car.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(431, 575, 3)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "car[:,:,:3].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../images/numpy-dog-photo.png\" alt=\"photo a dog\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(432, 575, 4)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dog = imread(\"../images/numpy-dog-photo.png\")\n",
    "dog.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.70980394, 0.80784315, 0.88235295, 1.        ],\n",
       "        [0.72156864, 0.8117647 , 0.8862745 , 1.        ],\n",
       "        [0.7411765 , 0.8156863 , 0.8862745 , 1.        ],\n",
       "        ...,\n",
       "        [0.49803922, 0.6862745 , 0.8392157 , 1.        ],\n",
       "        [0.49411765, 0.68235296, 0.8392157 , 1.        ],\n",
       "        [0.49411765, 0.68235296, 0.8352941 , 1.        ]],\n",
       "\n",
       "       [[0.69411767, 0.8039216 , 0.8862745 , 1.        ],\n",
       "        [0.7019608 , 0.8039216 , 0.88235295, 1.        ],\n",
       "        [0.7058824 , 0.80784315, 0.88235295, 1.        ],\n",
       "        ...,\n",
       "        [0.5019608 , 0.6862745 , 0.84705883, 1.        ],\n",
       "        [0.49411765, 0.68235296, 0.84313726, 1.        ],\n",
       "        [0.49411765, 0.68235296, 0.8392157 , 1.        ]],\n",
       "\n",
       "       [[0.6901961 , 0.8       , 0.88235295, 1.        ],\n",
       "        [0.69803923, 0.8039216 , 0.88235295, 1.        ],\n",
       "        [0.7058824 , 0.80784315, 0.88235295, 1.        ],\n",
       "        ...,\n",
       "        [0.5019608 , 0.6862745 , 0.84705883, 1.        ],\n",
       "        [0.49803922, 0.6862745 , 0.84313726, 1.        ],\n",
       "        [0.49803922, 0.6862745 , 0.84313726, 1.        ]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0.9098039 , 0.81960785, 0.654902  , 1.        ],\n",
       "        [0.8352941 , 0.7490196 , 0.6509804 , 1.        ],\n",
       "        [0.72156864, 0.6313726 , 0.5372549 , 1.        ],\n",
       "        ...,\n",
       "        [0.01568628, 0.07058824, 0.02352941, 1.        ],\n",
       "        [0.03921569, 0.09411765, 0.03529412, 1.        ],\n",
       "        [0.03921569, 0.09019608, 0.05490196, 1.        ]],\n",
       "\n",
       "       [[0.9137255 , 0.83137256, 0.6784314 , 1.        ],\n",
       "        [0.8117647 , 0.7294118 , 0.627451  , 1.        ],\n",
       "        [0.65882355, 0.5686275 , 0.47843137, 1.        ],\n",
       "        ...,\n",
       "        [0.00392157, 0.05490196, 0.03529412, 1.        ],\n",
       "        [0.03137255, 0.09019608, 0.05490196, 1.        ],\n",
       "        [0.04705882, 0.10588235, 0.06666667, 1.        ]],\n",
       "\n",
       "       [[0.9137255 , 0.83137256, 0.68235296, 1.        ],\n",
       "        [0.76862746, 0.68235296, 0.5882353 , 1.        ],\n",
       "        [0.59607846, 0.5058824 , 0.44313726, 1.        ],\n",
       "        ...,\n",
       "        [0.03921569, 0.10196079, 0.07058824, 1.        ],\n",
       "        [0.02745098, 0.08235294, 0.05882353, 1.        ],\n",
       "        [0.05098039, 0.11372549, 0.07058824, 1.        ]]], dtype=float32)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dog"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}