content/Introduction_to_Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Pandas\n",
    "\n",
    "*Written by Luke Chang & Jin Cheong*\n",
    "\n",
    "Analyzing data requires being facile with manipulating and transforming datasets to be able to test specific hypotheses. Data come in all different types of flavors and there are many different tools in the Python ecosystem to work with pretty much any type of data you might encounter. For example, you might be interested in working with functional neuroimaging data that is four dimensional. Three dimensional matrices contain brain activations in space, and these data can change over time in the 4th dimension. This type of data is well suited for [numpy](https://numpy.org/) and specialized brain imaging packages such as [nilearn](https://nilearn.github.io/). The majority of data, however, is typically in some version of a two-dimensional observations by features format as might be seen in an excel spreadsheet, a SQL table, or in a comma delimited format (i.e., csv). \n",
    "\n",
    "In Python, the [Pandas](https://pandas.pydata.org/) library is a powerful tool to work with this type of data. This is a very large library with a tremendous amount of functionality. In this tutorial, we will cover the basics of how to load and manipulate data and will focus on common to data munging tasks.\n",
    "\n",
    "For those interested in diving deeper into Pandas, there are many online resources. There is the [Pandas online documention](https://pandas.pydata.org/), [stackoverflow](https://stackoverflow.com/questions/tagged/pandas), and [medium blogposts](https://medium.com/search?q=pandas). I highly recommend  Jake Vanderplas's terrific [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/).   In addition, here is a brief [video](https://neurohackademy.org/course/complex-data-structures/) by Tal Yarkoni providing a useful introduction to pandas. \n",
    "\n",
    "After the tutorial you will have the chance to apply the methods to a new set of data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas Objects\n",
    "\n",
    "Pandas has several objects that are commonly used (i.e., Series, DataFrame, Index). At it's core, Pandas Objects are enhanced numpy arrays where columns and rows can have special names and there are lots of methods to operate on the data. See Jake Vanderplas's [tutorial](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.01-Introducing-Pandas-Objects.ipynb) for a more in depth overview."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series\n",
    "A pandas `Series` is a one-dimensional array of indexed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.906085Z",
     "start_time": "2020-03-31T04:06:34.654377Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    3\n",
       "3    4\n",
       "4    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.Series([1, 2, 3, 4, 5])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The indices can be integers like in the example above. Alternatively, the indices can be labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.912944Z",
     "start_time": "2020-03-31T04:06:34.908326Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1\n",
       "b    2\n",
       "c    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series([1,2,3], index=['a', 'b', 'c'])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, `Series` can be easily created from dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.919658Z",
     "start_time": "2020-03-31T04:06:34.914814Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    5\n",
       "B    3\n",
       "C    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series({'A':5, 'B':3, 'C':1})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame\n",
    "If a `Series` is a one-dimensional indexed array, the `DataFrame` is a two-dimensional indexed array. It can be thought of as a collection of Series objects, where each Series represents a column, or as an enhanced 2D numpy array. \n",
    "\n",
    "In a `DataFrame`, the index refers to labels for each row, while columns describe each column.\n",
    "\n",
    "First, let's create a `DataFrame` using random numbers generated from numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.931582Z",
     "start_time": "2020-03-31T04:06:34.921814Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.828627</td>\n",
       "      <td>0.860628</td>\n",
       "      <td>0.239354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.701561</td>\n",
       "      <td>0.990040</td>\n",
       "      <td>0.955123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.526356</td>\n",
       "      <td>0.573977</td>\n",
       "      <td>0.854763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.360077</td>\n",
       "      <td>0.396195</td>\n",
       "      <td>0.303766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.188879</td>\n",
       "      <td>0.446423</td>\n",
       "      <td>0.063622</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2\n",
       "0  0.828627  0.860628  0.239354\n",
       "1  0.701561  0.990040  0.955123\n",
       "2  0.526356  0.573977  0.854763\n",
       "3  0.360077  0.396195  0.303766\n",
       "4  0.188879  0.446423  0.063622"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "data = pd.DataFrame(np.random.random((5, 3)))\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also initialize with column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.941794Z",
     "start_time": "2020-03-31T04:06:34.933174Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.493188</td>\n",
       "      <td>0.513151</td>\n",
       "      <td>0.500130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.706082</td>\n",
       "      <td>0.967668</td>\n",
       "      <td>0.762955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.853237</td>\n",
       "      <td>0.361205</td>\n",
       "      <td>0.297108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.021289</td>\n",
       "      <td>0.920400</td>\n",
       "      <td>0.609904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.917379</td>\n",
       "      <td>0.207424</td>\n",
       "      <td>0.451640</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C\n",
       "0  0.493188  0.513151  0.500130\n",
       "1  0.706082  0.967668  0.762955\n",
       "2  0.853237  0.361205  0.297108\n",
       "3  0.021289  0.920400  0.609904\n",
       "4  0.917379  0.207424  0.451640"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame(np.random.random((5, 3)), columns=['A', 'B', 'C'])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-30T03:26:39.232314Z",
     "start_time": "2020-03-30T03:26:39.227566Z"
    }
   },
   "source": [
    "Alternatively, we could create a `DataFrame` from multiple `Series` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.951676Z",
     "start_time": "2020-03-31T04:06:34.943440Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Numbers</th>\n",
       "      <th>Letters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Numbers Letters\n",
       "0        1       a\n",
       "1        2       b\n",
       "2        3       c\n",
       "3        4       d"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = pd.Series([1, 2, 3, 4])\n",
    "b = pd.Series(['a', 'b', 'c', 'd'])\n",
    "data = pd.DataFrame({'Numbers':a, 'Letters':b})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or a python dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:34.984981Z",
     "start_time": "2020-03-31T04:06:34.955336Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Capital</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>California</td>\n",
       "      <td>Sacremento</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Colorado</td>\n",
       "      <td>Denver</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New Hampshire</td>\n",
       "      <td>Concord</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           State     Capital\n",
       "0     California  Sacremento\n",
       "1       Colorado      Denver\n",
       "2  New Hampshire     Concord"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({'State':['California', 'Colorado', 'New Hampshire'], \n",
    "                     'Capital':['Sacremento', 'Denver', 'Concord']})\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "Loading data is fairly straightfoward in Pandas. Type `pd.read` then press tab to see a list of functions that can load specific file formats such as: csv, excel, spss, and sql. \n",
    "\n",
    "In this example, we will use `pd.read_csv` to load a .csv file into a dataframe. \n",
    "Note that read_csv() has many options that can be used to make sure you load the data correctly. You can explore the docstrings for a function to get more information about the inputs and general useage guidelines by running `pd.read_csv?` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.042712Z",
     "start_time": "2020-03-31T04:06:34.987943Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mSignature:\u001b[0m\n",
      "\u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'FilePath | ReadCsvBuffer[bytes] | ReadCsvBuffer[str]'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0msep\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdelimiter\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mheader\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"int | Sequence[int] | None | Literal['infer']\"\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'infer'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mnames\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Sequence[Hashable] | None | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mindex_col\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'IndexLabel | Literal[False] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0musecols\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'UsecolsArgType'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'DtypeArg | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mengine\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'CSVEngine | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mconverters\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Mapping[Hashable, Callable] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mtrue_values\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'list | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfalse_values\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'list | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mskipinitialspace\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mskiprows\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'list[int] | int | Callable[[Hashable], bool] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mskipfooter\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mnrows\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mna_values\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Hashable | Iterable[Hashable] | Mapping[Hashable, Iterable[Hashable]] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mkeep_default_na\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mna_filter\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mverbose\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mskip_blank_lines\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mparse_dates\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | Sequence[Hashable] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0minfer_datetime_format\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mkeep_date_col\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdate_parser\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Callable | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdate_format\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | dict[Hashable, str] | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdayfirst\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mcache_dates\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mchunksize\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mcompression\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'CompressionOptions'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'infer'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mthousands\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdecimal\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'.'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mlineterminator\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mquotechar\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'\"'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mquoting\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdoublequote\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mescapechar\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mcomment\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mencoding\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mencoding_errors\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'strict'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdialect\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str | csv.Dialect | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mon_bad_lines\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'error'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdelim_whitespace\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mlow_memory\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mmemory_map\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfloat_precision\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"Literal['high', 'legacy'] | None\"\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mstorage_options\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'StorageOptions | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdtype_backend\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'DtypeBackend | lib.NoDefault'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mno_default\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'DataFrame | TextFileReader'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m\n",
      "Read a comma-separated values (csv) file into DataFrame.\n",
      "\n",
      "Also supports optionally iterating or breaking of the file\n",
      "into chunks.\n",
      "\n",
      "Additional help can be found in the online docs for\n",
      "`IO Tools <https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html>`_.\n",
      "\n",
      "Parameters\n",
      "----------\n",
      "filepath_or_buffer : str, path object or file-like object\n",
      "    Any valid string path is acceptable. The string could be a URL. Valid\n",
      "    URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is\n",
      "    expected. A local file could be: file://localhost/path/to/table.csv.\n",
      "\n",
      "    If you want to pass in a path object, pandas accepts any ``os.PathLike``.\n",
      "\n",
      "    By file-like object, we refer to objects with a ``read()`` method, such as\n",
      "    a file handle (e.g. via builtin ``open`` function) or ``StringIO``.\n",
      "sep : str, default ','\n",
      "    Character or regex pattern to treat as the delimiter. If ``sep=None``, the\n",
      "    C engine cannot automatically detect\n",
      "    the separator, but the Python parsing engine can, meaning the latter will\n",
      "    be used and automatically detect the separator from only the first valid\n",
      "    row of the file by Python's builtin sniffer tool, ``csv.Sniffer``.\n",
      "    In addition, separators longer than 1 character and different from\n",
      "    ``'\\s+'`` will be interpreted as regular expressions and will also force\n",
      "    the use of the Python parsing engine. Note that regex delimiters are prone\n",
      "    to ignoring quoted data. Regex example: ``'\\r\\t'``.\n",
      "delimiter : str, optional\n",
      "    Alias for ``sep``.\n",
      "header : int, Sequence of int, 'infer' or None, default 'infer'\n",
      "    Row number(s) containing column labels and marking the start of the\n",
      "    data (zero-indexed). Default behavior is to infer the column names: if no ``names``\n",
      "    are passed the behavior is identical to ``header=0`` and column\n",
      "    names are inferred from the first line of the file, if column\n",
      "    names are passed explicitly to ``names`` then the behavior is identical to\n",
      "    ``header=None``. Explicitly pass ``header=0`` to be able to\n",
      "    replace existing names. The header can be a list of integers that\n",
      "    specify row locations for a :class:`~pandas.MultiIndex` on the columns\n",
      "    e.g. ``[0, 1, 3]``. Intervening rows that are not specified will be\n",
      "    skipped (e.g. 2 in this example is skipped). Note that this\n",
      "    parameter ignores commented lines and empty lines if\n",
      "    ``skip_blank_lines=True``, so ``header=0`` denotes the first line of\n",
      "    data rather than the first line of the file.\n",
      "names : Sequence of Hashable, optional\n",
      "    Sequence of column labels to apply. If the file contains a header row,\n",
      "    then you should explicitly pass ``header=0`` to override the column names.\n",
      "    Duplicates in this list are not allowed.\n",
      "index_col : Hashable, Sequence of Hashable or False, optional\n",
      "  Column(s) to use as row label(s), denoted either by column labels or column\n",
      "  indices.  If a sequence of labels or indices is given, :class:`~pandas.MultiIndex`\n",
      "  will be formed for the row labels.\n",
      "\n",
      "  Note: ``index_col=False`` can be used to force pandas to *not* use the first\n",
      "  column as the index, e.g., when you have a malformed file with delimiters at\n",
      "  the end of each line.\n",
      "usecols : Sequence of Hashable or Callable, optional\n",
      "    Subset of columns to select, denoted either by column labels or column indices.\n",
      "    If list-like, all elements must either\n",
      "    be positional (i.e. integer indices into the document columns) or strings\n",
      "    that correspond to column names provided either by the user in ``names`` or\n",
      "    inferred from the document header row(s). If ``names`` are given, the document\n",
      "    header row(s) are not taken into account. For example, a valid list-like\n",
      "    ``usecols`` parameter would be ``[0, 1, 2]`` or ``['foo', 'bar', 'baz']``.\n",
      "    Element order is ignored, so ``usecols=[0, 1]`` is the same as ``[1, 0]``.\n",
      "    To instantiate a :class:`~pandas.DataFrame` from ``data`` with element order\n",
      "    preserved use ``pd.read_csv(data, usecols=['foo', 'bar'])[['foo', 'bar']]``\n",
      "    for columns in ``['foo', 'bar']`` order or\n",
      "    ``pd.read_csv(data, usecols=['foo', 'bar'])[['bar', 'foo']]``\n",
      "    for ``['bar', 'foo']`` order.\n",
      "\n",
      "    If callable, the callable function will be evaluated against the column\n",
      "    names, returning names where the callable function evaluates to ``True``. An\n",
      "    example of a valid callable argument would be ``lambda x: x.upper() in\n",
      "    ['AAA', 'BBB', 'DDD']``. Using this parameter results in much faster\n",
      "    parsing time and lower memory usage.\n",
      "dtype : dtype or dict of {Hashable : dtype}, optional\n",
      "    Data type(s) to apply to either the whole dataset or individual columns.\n",
      "    E.g., ``{'a': np.float64, 'b': np.int32, 'c': 'Int64'}``\n",
      "    Use ``str`` or ``object`` together with suitable ``na_values`` settings\n",
      "    to preserve and not interpret ``dtype``.\n",
      "    If ``converters`` are specified, they will be applied INSTEAD\n",
      "    of ``dtype`` conversion.\n",
      "\n",
      "    .. versionadded:: 1.5.0\n",
      "\n",
      "        Support for ``defaultdict`` was added. Specify a ``defaultdict`` as input where\n",
      "        the default determines the ``dtype`` of the columns which are not explicitly\n",
      "        listed.\n",
      "engine : {'c', 'python', 'pyarrow'}, optional\n",
      "    Parser engine to use. The C and pyarrow engines are faster, while the python engine\n",
      "    is currently more feature-complete. Multithreading is currently only supported by\n",
      "    the pyarrow engine.\n",
      "\n",
      "    .. versionadded:: 1.4.0\n",
      "\n",
      "        The 'pyarrow' engine was added as an *experimental* engine, and some features\n",
      "        are unsupported, or may not work correctly, with this engine.\n",
      "converters : dict of {Hashable : Callable}, optional\n",
      "    Functions for converting values in specified columns. Keys can either\n",
      "    be column labels or column indices.\n",
      "true_values : list, optional\n",
      "    Values to consider as ``True`` in addition to case-insensitive variants of 'True'.\n",
      "false_values : list, optional\n",
      "    Values to consider as ``False`` in addition to case-insensitive variants of 'False'.\n",
      "skipinitialspace : bool, default False\n",
      "    Skip spaces after delimiter.\n",
      "skiprows : int, list of int or Callable, optional\n",
      "    Line numbers to skip (0-indexed) or number of lines to skip (``int``)\n",
      "    at the start of the file.\n",
      "\n",
      "    If callable, the callable function will be evaluated against the row\n",
      "    indices, returning ``True`` if the row should be skipped and ``False`` otherwise.\n",
      "    An example of a valid callable argument would be ``lambda x: x in [0, 2]``.\n",
      "skipfooter : int, default 0\n",
      "    Number of lines at bottom of file to skip (Unsupported with ``engine='c'``).\n",
      "nrows : int, optional\n",
      "    Number of rows of file to read. Useful for reading pieces of large files.\n",
      "na_values : Hashable, Iterable of Hashable or dict of {Hashable : Iterable}, optional\n",
      "    Additional strings to recognize as ``NA``/``NaN``. If ``dict`` passed, specific\n",
      "    per-column ``NA`` values.  By default the following values are interpreted as\n",
      "    ``NaN``: \" \", \"#N/A\", \"#N/A N/A\", \"#NA\", \"-1.#IND\", \"-1.#QNAN\", \"-NaN\", \"-nan\",\n",
      "    \"1.#IND\", \"1.#QNAN\", \"<NA>\", \"N/A\", \"NA\", \"NULL\", \"NaN\", \"None\",\n",
      "    \"n/a\", \"nan\", \"null \".\n",
      "\n",
      "keep_default_na : bool, default True\n",
      "    Whether or not to include the default ``NaN`` values when parsing the data.\n",
      "    Depending on whether ``na_values`` is passed in, the behavior is as follows:\n",
      "\n",
      "    * If ``keep_default_na`` is ``True``, and ``na_values`` are specified, ``na_values``\n",
      "      is appended to the default ``NaN`` values used for parsing.\n",
      "    * If ``keep_default_na`` is ``True``, and ``na_values`` are not specified, only\n",
      "      the default ``NaN`` values are used for parsing.\n",
      "    * If ``keep_default_na`` is ``False``, and ``na_values`` are specified, only\n",
      "      the ``NaN`` values specified ``na_values`` are used for parsing.\n",
      "    * If ``keep_default_na`` is ``False``, and ``na_values`` are not specified, no\n",
      "      strings will be parsed as ``NaN``.\n",
      "\n",
      "    Note that if ``na_filter`` is passed in as ``False``, the ``keep_default_na`` and\n",
      "    ``na_values`` parameters will be ignored.\n",
      "na_filter : bool, default True\n",
      "    Detect missing value markers (empty strings and the value of ``na_values``). In\n",
      "    data without any ``NA`` values, passing ``na_filter=False`` can improve the\n",
      "    performance of reading a large file.\n",
      "verbose : bool, default False\n",
      "    Indicate number of ``NA`` values placed in non-numeric columns.\n",
      "\n",
      "    .. deprecated:: 2.2.0\n",
      "skip_blank_lines : bool, default True\n",
      "    If ``True``, skip over blank lines rather than interpreting as ``NaN`` values.\n",
      "parse_dates : bool, list of Hashable, list of lists or dict of {Hashable : list}, default False\n",
      "    The behavior is as follows:\n",
      "\n",
      "    * ``bool``. If ``True`` -> try parsing the index. Note: Automatically set to\n",
      "      ``True`` if ``date_format`` or ``date_parser`` arguments have been passed.\n",
      "    * ``list`` of ``int`` or names. e.g. If ``[1, 2, 3]`` -> try parsing columns 1, 2, 3\n",
      "      each as a separate date column.\n",
      "    * ``list`` of ``list``. e.g.  If ``[[1, 3]]`` -> combine columns 1 and 3 and parse\n",
      "      as a single date column. Values are joined with a space before parsing.\n",
      "    * ``dict``, e.g. ``{'foo' : [1, 3]}`` -> parse columns 1, 3 as date and call\n",
      "      result 'foo'. Values are joined with a space before parsing.\n",
      "\n",
      "    If a column or index cannot be represented as an array of ``datetime``,\n",
      "    say because of an unparsable value or a mixture of timezones, the column\n",
      "    or index will be returned unaltered as an ``object`` data type. For\n",
      "    non-standard ``datetime`` parsing, use :func:`~pandas.to_datetime` after\n",
      "    :func:`~pandas.read_csv`.\n",
      "\n",
      "    Note: A fast-path exists for iso8601-formatted dates.\n",
      "infer_datetime_format : bool, default False\n",
      "    If ``True`` and ``parse_dates`` is enabled, pandas will attempt to infer the\n",
      "    format of the ``datetime`` strings in the columns, and if it can be inferred,\n",
      "    switch to a faster method of parsing them. In some cases this can increase\n",
      "    the parsing speed by 5-10x.\n",
      "\n",
      "    .. deprecated:: 2.0.0\n",
      "        A strict version of this argument is now the default, passing it has no effect.\n",
      "\n",
      "keep_date_col : bool, default False\n",
      "    If ``True`` and ``parse_dates`` specifies combining multiple columns then\n",
      "    keep the original columns.\n",
      "date_parser : Callable, optional\n",
      "    Function to use for converting a sequence of string columns to an array of\n",
      "    ``datetime`` instances. The default uses ``dateutil.parser.parser`` to do the\n",
      "    conversion. pandas will try to call ``date_parser`` in three different ways,\n",
      "    advancing to the next if an exception occurs: 1) Pass one or more arrays\n",
      "    (as defined by ``parse_dates``) as arguments; 2) concatenate (row-wise) the\n",
      "    string values from the columns defined by ``parse_dates`` into a single array\n",
      "    and pass that; and 3) call ``date_parser`` once for each row using one or\n",
      "    more strings (corresponding to the columns defined by ``parse_dates``) as\n",
      "    arguments.\n",
      "\n",
      "    .. deprecated:: 2.0.0\n",
      "       Use ``date_format`` instead, or read in as ``object`` and then apply\n",
      "       :func:`~pandas.to_datetime` as-needed.\n",
      "date_format : str or dict of column -> format, optional\n",
      "    Format to use for parsing dates when used in conjunction with ``parse_dates``.\n",
      "    The strftime to parse time, e.g. :const:`\"%d/%m/%Y\"`. See\n",
      "    `strftime documentation\n",
      "    <https://docs.python.org/3/library/datetime.html\n",
      "    #strftime-and-strptime-behavior>`_ for more information on choices, though\n",
      "    note that :const:`\"%f\"` will parse all the way up to nanoseconds.\n",
      "    You can also pass:\n",
      "\n",
      "    - \"ISO8601\", to parse any `ISO8601 <https://en.wikipedia.org/wiki/ISO_8601>`_\n",
      "        time string (not necessarily in exactly the same format);\n",
      "    - \"mixed\", to infer the format for each element individually. This is risky,\n",
      "        and you should probably use it along with `dayfirst`.\n",
      "\n",
      "    .. versionadded:: 2.0.0\n",
      "dayfirst : bool, default False\n",
      "    DD/MM format dates, international and European format.\n",
      "cache_dates : bool, default True\n",
      "    If ``True``, use a cache of unique, converted dates to apply the ``datetime``\n",
      "    conversion. May produce significant speed-up when parsing duplicate\n",
      "    date strings, especially ones with timezone offsets.\n",
      "\n",
      "iterator : bool, default False\n",
      "    Return ``TextFileReader`` object for iteration or getting chunks with\n",
      "    ``get_chunk()``.\n",
      "chunksize : int, optional\n",
      "    Number of lines to read from the file per chunk. Passing a value will cause the\n",
      "    function to return a ``TextFileReader`` object for iteration.\n",
      "    See the `IO Tools docs\n",
      "    <https://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking>`_\n",
      "    for more information on ``iterator`` and ``chunksize``.\n",
      "\n",
      "compression : str or dict, default 'infer'\n",
      "    For on-the-fly decompression of on-disk data. If 'infer' and 'filepath_or_buffer' is\n",
      "    path-like, then detect compression from the following extensions: '.gz',\n",
      "    '.bz2', '.zip', '.xz', '.zst', '.tar', '.tar.gz', '.tar.xz' or '.tar.bz2'\n",
      "    (otherwise no compression).\n",
      "    If using 'zip' or 'tar', the ZIP file must contain only one data file to be read in.\n",
      "    Set to ``None`` for no decompression.\n",
      "    Can also be a dict with key ``'method'`` set\n",
      "    to one of {``'zip'``, ``'gzip'``, ``'bz2'``, ``'zstd'``, ``'xz'``, ``'tar'``} and\n",
      "    other key-value pairs are forwarded to\n",
      "    ``zipfile.ZipFile``, ``gzip.GzipFile``,\n",
      "    ``bz2.BZ2File``, ``zstandard.ZstdDecompressor``, ``lzma.LZMAFile`` or\n",
      "    ``tarfile.TarFile``, respectively.\n",
      "    As an example, the following could be passed for Zstandard decompression using a\n",
      "    custom compression dictionary:\n",
      "    ``compression={'method': 'zstd', 'dict_data': my_compression_dict}``.\n",
      "\n",
      "    .. versionadded:: 1.5.0\n",
      "        Added support for `.tar` files.\n",
      "\n",
      "    .. versionchanged:: 1.4.0 Zstandard support.\n",
      "\n",
      "thousands : str (length 1), optional\n",
      "    Character acting as the thousands separator in numerical values.\n",
      "decimal : str (length 1), default '.'\n",
      "    Character to recognize as decimal point (e.g., use ',' for European data).\n",
      "lineterminator : str (length 1), optional\n",
      "    Character used to denote a line break. Only valid with C parser.\n",
      "quotechar : str (length 1), optional\n",
      "    Character used to denote the start and end of a quoted item. Quoted\n",
      "    items can include the ``delimiter`` and it will be ignored.\n",
      "quoting : {0 or csv.QUOTE_MINIMAL, 1 or csv.QUOTE_ALL, 2 or csv.QUOTE_NONNUMERIC, 3 or csv.QUOTE_NONE}, default csv.QUOTE_MINIMAL\n",
      "    Control field quoting behavior per ``csv.QUOTE_*`` constants. Default is\n",
      "    ``csv.QUOTE_MINIMAL`` (i.e., 0) which implies that only fields containing special\n",
      "    characters are quoted (e.g., characters defined in ``quotechar``, ``delimiter``,\n",
      "    or ``lineterminator``.\n",
      "doublequote : bool, default True\n",
      "   When ``quotechar`` is specified and ``quoting`` is not ``QUOTE_NONE``, indicate\n",
      "   whether or not to interpret two consecutive ``quotechar`` elements INSIDE a\n",
      "   field as a single ``quotechar`` element.\n",
      "escapechar : str (length 1), optional\n",
      "    Character used to escape other characters.\n",
      "comment : str (length 1), optional\n",
      "    Character indicating that the remainder of line should not be parsed.\n",
      "    If found at the beginning\n",
      "    of a line, the line will be ignored altogether. This parameter must be a\n",
      "    single character. Like empty lines (as long as ``skip_blank_lines=True``),\n",
      "    fully commented lines are ignored by the parameter ``header`` but not by\n",
      "    ``skiprows``. For example, if ``comment='#'``, parsing\n",
      "    ``#empty\\na,b,c\\n1,2,3`` with ``header=0`` will result in ``'a,b,c'`` being\n",
      "    treated as the header.\n",
      "encoding : str, optional, default 'utf-8'\n",
      "    Encoding to use for UTF when reading/writing (ex. ``'utf-8'``). `List of Python\n",
      "    standard encodings\n",
      "    <https://docs.python.org/3/library/codecs.html#standard-encodings>`_ .\n",
      "\n",
      "encoding_errors : str, optional, default 'strict'\n",
      "    How encoding errors are treated. `List of possible values\n",
      "    <https://docs.python.org/3/library/codecs.html#error-handlers>`_ .\n",
      "\n",
      "    .. versionadded:: 1.3.0\n",
      "\n",
      "dialect : str or csv.Dialect, optional\n",
      "    If provided, this parameter will override values (default or not) for the\n",
      "    following parameters: ``delimiter``, ``doublequote``, ``escapechar``,\n",
      "    ``skipinitialspace``, ``quotechar``, and ``quoting``. If it is necessary to\n",
      "    override values, a ``ParserWarning`` will be issued. See ``csv.Dialect``\n",
      "    documentation for more details.\n",
      "on_bad_lines : {'error', 'warn', 'skip'} or Callable, default 'error'\n",
      "    Specifies what to do upon encountering a bad line (a line with too many fields).\n",
      "    Allowed values are :\n",
      "\n",
      "    - ``'error'``, raise an Exception when a bad line is encountered.\n",
      "    - ``'warn'``, raise a warning when a bad line is encountered and skip that line.\n",
      "    - ``'skip'``, skip bad lines without raising or warning when they are encountered.\n",
      "\n",
      "    .. versionadded:: 1.3.0\n",
      "\n",
      "    .. versionadded:: 1.4.0\n",
      "\n",
      "        - Callable, function with signature\n",
      "          ``(bad_line: list[str]) -> list[str] | None`` that will process a single\n",
      "          bad line. ``bad_line`` is a list of strings split by the ``sep``.\n",
      "          If the function returns ``None``, the bad line will be ignored.\n",
      "          If the function returns a new ``list`` of strings with more elements than\n",
      "          expected, a ``ParserWarning`` will be emitted while dropping extra elements.\n",
      "          Only supported when ``engine='python'``\n",
      "\n",
      "    .. versionchanged:: 2.2.0\n",
      "\n",
      "        - Callable, function with signature\n",
      "          as described in `pyarrow documentation\n",
      "          <https://arrow.apache.org/docs/python/generated/pyarrow.csv.ParseOptions.html\n",
      "          #pyarrow.csv.ParseOptions.invalid_row_handler>`_ when ``engine='pyarrow'``\n",
      "\n",
      "delim_whitespace : bool, default False\n",
      "    Specifies whether or not whitespace (e.g. ``' '`` or ``'\\t'``) will be\n",
      "    used as the ``sep`` delimiter. Equivalent to setting ``sep='\\s+'``. If this option\n",
      "    is set to ``True``, nothing should be passed in for the ``delimiter``\n",
      "    parameter.\n",
      "\n",
      "    .. deprecated:: 2.2.0\n",
      "        Use ``sep=\"\\s+\"`` instead.\n",
      "low_memory : bool, default True\n",
      "    Internally process the file in chunks, resulting in lower memory use\n",
      "    while parsing, but possibly mixed type inference.  To ensure no mixed\n",
      "    types either set ``False``, or specify the type with the ``dtype`` parameter.\n",
      "    Note that the entire file is read into a single :class:`~pandas.DataFrame`\n",
      "    regardless, use the ``chunksize`` or ``iterator`` parameter to return the data in\n",
      "    chunks. (Only valid with C parser).\n",
      "memory_map : bool, default False\n",
      "    If a filepath is provided for ``filepath_or_buffer``, map the file object\n",
      "    directly onto memory and access the data directly from there. Using this\n",
      "    option can improve performance because there is no longer any I/O overhead.\n",
      "float_precision : {'high', 'legacy', 'round_trip'}, optional\n",
      "    Specifies which converter the C engine should use for floating-point\n",
      "    values. The options are ``None`` or ``'high'`` for the ordinary converter,\n",
      "    ``'legacy'`` for the original lower precision pandas converter, and\n",
      "    ``'round_trip'`` for the round-trip converter.\n",
      "\n",
      "storage_options : dict, optional\n",
      "    Extra options that make sense for a particular storage connection, e.g.\n",
      "    host, port, username, password, etc. For HTTP(S) URLs the key-value pairs\n",
      "    are forwarded to ``urllib.request.Request`` as header options. For other\n",
      "    URLs (e.g. starting with \"s3://\", and \"gcs://\") the key-value pairs are\n",
      "    forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more\n",
      "    details, and for more examples on storage options refer `here\n",
      "    <https://pandas.pydata.org/docs/user_guide/io.html?\n",
      "    highlight=storage_options#reading-writing-remote-files>`_.\n",
      "\n",
      "dtype_backend : {'numpy_nullable', 'pyarrow'}, default 'numpy_nullable'\n",
      "    Back-end data type applied to the resultant :class:`DataFrame`\n",
      "    (still experimental). Behaviour is as follows:\n",
      "\n",
      "    * ``\"numpy_nullable\"``: returns nullable-dtype-backed :class:`DataFrame`\n",
      "      (default).\n",
      "    * ``\"pyarrow\"``: returns pyarrow-backed nullable :class:`ArrowDtype`\n",
      "      DataFrame.\n",
      "\n",
      "    .. versionadded:: 2.0\n",
      "\n",
      "Returns\n",
      "-------\n",
      "DataFrame or TextFileReader\n",
      "    A comma-separated values (csv) file is returned as two-dimensional\n",
      "    data structure with labeled axes.\n",
      "\n",
      "See Also\n",
      "--------\n",
      "DataFrame.to_csv : Write DataFrame to a comma-separated values (csv) file.\n",
      "read_table : Read general delimited file into DataFrame.\n",
      "read_fwf : Read a table of fixed-width formatted lines into DataFrame.\n",
      "\n",
      "Examples\n",
      "--------\n",
      ">>> pd.read_csv('data.csv')  # doctest: +SKIP\n",
      "\u001b[0;31mFile:\u001b[0m      /opt/anaconda3/envs/py312/lib/python3.12/site-packages/pandas/io/parsers/readers.py\n",
      "\u001b[0;31mType:\u001b[0m      function"
     ]
    }
   ],
   "source": [
    "pd.read_csv?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load a csv file we will need to specify either the relative or absolute path to the file.\n",
    "\n",
    "The command `pwd` will print the path of the current working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.048733Z",
     "start_time": "2020-03-31T04:06:35.044825Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/lukechang/Github/dartbrains/content'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T03:32:48.507978Z",
     "start_time": "2019-04-12T03:32:48.503985Z"
    }
   },
   "source": [
    "We will now load the Pandas has many ways to read data different data formats into a dataframe.  Here we will use the `pd.read_csv` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.055740Z",
     "start_time": "2020-03-31T04:06:35.050405Z"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('https://raw.githubusercontent.com/ljchang/dartbrains/master/data/salary/salary.csv', sep = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ways to check the dataframe\n",
    "There are many ways to examine your dataframe. One easy way is to just call the dataframe variable itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.070274Z",
     "start_time": "2020-03-31T04:06:35.057267Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>53662</td>\n",
       "      <td>1</td>\n",
       "      <td>neuro</td>\n",
       "      <td>1.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>57185</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>9.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>52254</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>61885</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>23.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>49542</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>3.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>77 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender departm  years   age  publications\n",
       "0    86285       0     bio   26.0  64.0            72\n",
       "1    77125       0     bio   28.0  58.0            43\n",
       "2    71922       0     bio   10.0  38.0            23\n",
       "3    70499       0     bio   16.0  46.0            64\n",
       "4    66624       0     bio   11.0  41.0            23\n",
       "..     ...     ...     ...    ...   ...           ...\n",
       "72   53662       1   neuro    1.0  31.0             3\n",
       "73   57185       1    stat    9.0  39.0             7\n",
       "74   52254       1    stat    2.0  32.0             9\n",
       "75   61885       1    math   23.0  60.0             9\n",
       "76   49542       1    math    3.0  33.0             5\n",
       "\n",
       "[77 rows x 6 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-01-16T18:35:36.660858",
     "start_time": "2017-01-16T18:35:36.656319"
    }
   },
   "source": [
    "However, often the dataframes can be large and we may be only interested in seeing the first few rows.  `df.head()` is useful for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.080036Z",
     "start_time": "2020-03-31T04:06:35.071946Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender departm  years   age  publications\n",
       "0   86285       0     bio   26.0  64.0            72\n",
       "1   77125       0     bio   28.0  58.0            43\n",
       "2   71922       0     bio   10.0  38.0            23\n",
       "3   70499       0     bio   16.0  46.0            64\n",
       "4   66624       0     bio   11.0  41.0            23"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the top row, you have column names, that can be called like a dictionary (a dataframe can be essentially thought of as a dictionary with column names as the keys). The left most column (0,1,2,3,4...) is called the index of the dataframe. The default index is sequential integers, but it can be set to anything as long as each row is unique (e.g., subject IDs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.085359Z",
     "start_time": "2020-03-31T04:06:35.081513Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Indexes\n",
      "RangeIndex(start=0, stop=77, step=1)\n",
      "Columns\n",
      "Index(['salary', 'gender', 'departm', 'years', 'age', 'publications'], dtype='object')\n",
      "Columns are like keys of a dictionary\n",
      "Index(['salary', 'gender', 'departm', 'years', 'age', 'publications'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(\"Indexes\")\n",
    "print(df.index)\n",
    "print(\"Columns\")\n",
    "print(df.columns)\n",
    "print(\"Columns are like keys of a dictionary\")\n",
    "print(df.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the values of a column by calling it directly. Single bracket returns a `Series` and double bracket returns a `dataframe`.\n",
    "\n",
    "Let's return the first 10 rows of salary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.091462Z",
     "start_time": "2020-03-31T04:06:35.087009Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    86285\n",
       "1    77125\n",
       "2    71922\n",
       "3    70499\n",
       "4    66624\n",
       "5    64451\n",
       "6    64366\n",
       "7    59344\n",
       "8    58560\n",
       "9    58294\n",
       "Name: salary, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['salary'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-30T05:43:56.143768Z",
     "start_time": "2020-03-30T05:43:56.139066Z"
    }
   },
   "source": [
    "`shape` is another useful method for getting the dimensions of the matrix.  \n",
    "\n",
    "We will print the number of rows and columns in this data set using fstring formatting. First, you need to specify a string starting with 'f', like this `f'anything'`. It is easy to insert variables with curly brackets like this `f'rows: {rows}'`. \n",
    "\n",
    "[Here](https://realpython.com/python-f-strings/) is more info about formatting text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.095691Z",
     "start_time": "2020-03-31T04:06:35.093060Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 77 rows and 6 columns in this data set\n"
     ]
    }
   ],
   "source": [
    "rows, cols = df.shape\n",
    "print(f'There are {rows} rows and {cols} columns in this data set') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-01-16T18:36:43.976308",
     "start_time": "2017-01-16T18:36:43.972047"
    }
   },
   "source": [
    "### Describing the data\n",
    "We can use the `.describe()` method to get a quick summary of the continuous values of the data frame. We will `.transpose()` the output to make it slightly easier to read. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.120997Z",
     "start_time": "2020-03-31T04:06:35.097367Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>salary</th>\n",
       "      <td>77.0</td>\n",
       "      <td>67748.519481</td>\n",
       "      <td>15100.581435</td>\n",
       "      <td>44687.0</td>\n",
       "      <td>57185.0</td>\n",
       "      <td>62607.0</td>\n",
       "      <td>75382.0</td>\n",
       "      <td>112800.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>77.0</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.387783</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>years</th>\n",
       "      <td>76.0</td>\n",
       "      <td>14.973684</td>\n",
       "      <td>8.617770</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>76.0</td>\n",
       "      <td>45.486842</td>\n",
       "      <td>9.005914</td>\n",
       "      <td>31.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>publications</th>\n",
       "      <td>77.0</td>\n",
       "      <td>21.831169</td>\n",
       "      <td>15.240530</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>72.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              count          mean           std      min      25%      50%  \\\n",
       "salary         77.0  67748.519481  15100.581435  44687.0  57185.0  62607.0   \n",
       "gender         77.0      0.142857      0.387783      0.0      0.0      0.0   \n",
       "years          76.0     14.973684      8.617770      1.0      8.0     14.0   \n",
       "age            76.0     45.486842      9.005914     31.0     38.0     44.0   \n",
       "publications   77.0     21.831169     15.240530      3.0      9.0     19.0   \n",
       "\n",
       "                  75%       max  \n",
       "salary        75382.0  112800.0  \n",
       "gender            0.0       2.0  \n",
       "years            23.0      34.0  \n",
       "age              53.0      65.0  \n",
       "publications     33.0      72.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get quick summary of a pandas series, or specific column of a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.128873Z",
     "start_time": "2020-03-31T04:06:35.122936Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count      77\n",
       "unique      7\n",
       "top       bio\n",
       "freq       16\n",
       "Name: departm, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.departm.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, you will want to know how many data points are associated with a specific variable for categorical data. The `value_counts` method can be used for this goal.\n",
    "\n",
    "For example, how many males and females are in this dataset?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.134924Z",
     "start_time": "2020-03-31T04:06:35.130347Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender\n",
       "0    67\n",
       "1     9\n",
       "2     1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['gender'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that there are more than 2 genders specified in our data. \n",
    "\n",
    "This is likely an error in the data collection process. It's always up to the data analyst to decide what to do in these cases. Because we don't know what the true value should have been, let's just remove the row from the dataframe by finding all rows that are not '2'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.142133Z",
     "start_time": "2020-03-31T04:06:35.136515Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender\n",
       "0    67\n",
       "1     9\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.loc[df['gender']!=2]\n",
    "\n",
    "df['gender'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-01-16T18:36:01.082619",
     "start_time": "2017-01-16T18:36:01.078188"
    }
   },
   "source": [
    "### Dealing with missing values\n",
    "Data are always messy and often have lots of missing values. There are many different ways, in which missing data might present `NaN`, `None`, or `NA`, Sometimes researchers code missing values with specific numeric codes such as 999999. It is important to find these as they can screw up your analyses if they are hiding in your data. \n",
    "\n",
    "If the missing values are using a standard pandas or numpy value such as `NaN`, `None`, or `NA`, we can identify where the missing values are as booleans using the `isnull()` method. \n",
    "\n",
    "The `isnull()` method will return a dataframe with True/False values on whether a datapoint is null or not a number (nan)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.155870Z",
     "start_time": "2020-03-31T04:06:35.143747Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>76 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender  departm  years    age  publications\n",
       "0    False   False    False  False  False         False\n",
       "1    False   False    False  False  False         False\n",
       "2    False   False    False  False  False         False\n",
       "3    False   False    False  False  False         False\n",
       "4    False   False    False  False  False         False\n",
       "..     ...     ...      ...    ...    ...           ...\n",
       "72   False   False    False  False  False         False\n",
       "73   False   False    False  False  False         False\n",
       "74   False   False    False  False  False         False\n",
       "75   False   False    False  False  False         False\n",
       "76   False   False    False  False  False         False\n",
       "\n",
       "[76 rows x 6 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we wanted to count the number of missing values for each column in the dataset.\n",
    "\n",
    "One thing that is  nice about Python is that you can chain commands, which means that the output of one method can be the input into the next method.  This allows us to write intuitive and concise code.  Notice how we take the `sum()` of all of the null cases.\n",
    "We can chain the `.null()` and `.sum()` methods to see how many null values are added up in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.162180Z",
     "start_time": "2020-03-31T04:06:35.157460Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "salary          0\n",
       "gender          0\n",
       "departm         0\n",
       "years           1\n",
       "age             1\n",
       "publications    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the boolean indexing once again to see the datapoints that have missing values. We chained the method `.any()` which will check if there are any True values for a given axis.  Axis=0 indicates rows, while Axis=1 indicates columns.  So here we are creating a boolean index for row where *any* column has a missing value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.176895Z",
     "start_time": "2020-03-31T04:06:35.167557Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>64762</td>\n",
       "      <td>0</td>\n",
       "      <td>chem</td>\n",
       "      <td>25.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>104828</td>\n",
       "      <td>0</td>\n",
       "      <td>geol</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.0</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender departm  years   age  publications\n",
       "18   64762       0    chem   25.0   NaN            29\n",
       "24  104828       0    geol    NaN  50.0            44"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T04:32:55.078779Z",
     "start_time": "2019-04-12T04:32:55.074997Z"
    }
   },
   "source": [
    "You may look at where the values are not null. Note that indexes 18, and 24 are missing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.195014Z",
     "start_time": "2020-03-31T04:06:35.180054Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>53662</td>\n",
       "      <td>1</td>\n",
       "      <td>neuro</td>\n",
       "      <td>1.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>57185</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>9.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>52254</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>61885</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>23.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>49542</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>3.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender departm  years   age  publications\n",
       "0    86285       0     bio   26.0  64.0            72\n",
       "1    77125       0     bio   28.0  58.0            43\n",
       "2    71922       0     bio   10.0  38.0            23\n",
       "3    70499       0     bio   16.0  46.0            64\n",
       "4    66624       0     bio   11.0  41.0            23\n",
       "..     ...     ...     ...    ...   ...           ...\n",
       "72   53662       1   neuro    1.0  31.0             3\n",
       "73   57185       1    stat    9.0  39.0             7\n",
       "74   52254       1    stat    2.0  32.0             9\n",
       "75   61885       1    math   23.0  60.0             9\n",
       "76   49542       1    math    3.0  33.0             5\n",
       "\n",
       "[74 rows x 6 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[~df.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-01-16T18:36:16.705384",
     "start_time": "2017-01-16T18:36:16.701256"
    }
   },
   "source": [
    "There are different techniques for dealing with missing data.  An easy one is to simply remove rows that have any missing values using the `dropna()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.201423Z",
     "start_time": "2020-03-31T04:06:35.196779Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/1379821321.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df.dropna(inplace=True)\n"
     ]
    }
   ],
   "source": [
    "df.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check to make sure the missing rows are removed.  Let's also check the new dimensions of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.208038Z",
     "start_time": "2020-03-31T04:06:35.202811Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 74 rows and 6 columns in this data set\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "salary          0\n",
       "gender          0\n",
       "departm         0\n",
       "years           0\n",
       "age             0\n",
       "publications    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows, cols = df.shape\n",
    "print(f'There are {rows} rows and {cols} columns in this data set') \n",
    "\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create New Columns\n",
    "You can create new columns to fit your needs. \n",
    "For instance you can set initialize a new column with zeros. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.212343Z",
     "start_time": "2020-03-31T04:06:35.209529Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/2945664157.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['pubperyear'] = 0\n"
     ]
    }
   ],
   "source": [
    "df['pubperyear'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can create a new column pubperyear, which is the ratio of the number of papers published per year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.293475Z",
     "start_time": "2020-03-31T04:06:35.213833Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/1818352125.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['pubperyear'] = df['publications']/df['years']\n"
     ]
    }
   ],
   "source": [
    "df['pubperyear'] = df['publications']/df['years']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing and slicing Data\n",
    "\n",
    "Indexing in Pandas can be tricky. There are many ways to index in pandas, for this tutorial we will focus on four: loc, iloc, boolean, and indexing numpy values. For a more in depth overview see Jake Vanderplas's tutorial](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.02-Data-Indexing-and-Selection.ipynb), where he also covers more advanced topics, such as [hierarchical indexing](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.05-Hierarchical-Indexing.ipynb). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing with Keys\n",
    "First, we will cover indexing with keys using the `.loc` method. This method references the explicit index with a key name. It works for both index names and also column names. Note that often the keys for rows are integers by default.\n",
    "\n",
    "In this example, we will return rows 10-20 on the salary column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.299033Z",
     "start_time": "2020-03-31T04:06:35.295104Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10    56092\n",
       "11    54452\n",
       "12    54269\n",
       "13    55125\n",
       "14    97630\n",
       "15    82444\n",
       "16    76291\n",
       "17    75382\n",
       "19    62607\n",
       "20    60373\n",
       "Name: salary, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[10:20, 'salary']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can return multiple columns using a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.307877Z",
     "start_time": "2020-03-31T04:06:35.300639Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>64451</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>64366</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>59344</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>58560</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>58294</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>56092</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  publications\n",
       "0    86285            72\n",
       "1    77125            43\n",
       "2    71922            23\n",
       "3    70499            64\n",
       "4    66624            23\n",
       "5    64451            44\n",
       "6    64366            22\n",
       "7    59344            11\n",
       "8    58560             8\n",
       "9    58294            12\n",
       "10   56092             4"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:10, ['salary', 'publications']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing with Integers\n",
    "Next we wil try `.iloc`.  This method references the implicit python index using integer indexing (starting from 0, exclusive of last number).  You can think of this like row by column indexing using integers.\n",
    "\n",
    "For example, let's grab the first 3 rows and columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.316278Z",
     "start_time": "2020-03-31T04:06:35.309562Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender departm\n",
       "0   86285       0     bio\n",
       "1   77125       0     bio\n",
       "2   71922       0     bio"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0:3, 0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a new data frame with just Males and another for just Females. Notice, how we added the `.reset_index(drop=True)` method?   This is because assigning a new dataframe based on indexing another dataframe will retain the *original* index.  We need to explicitly tell pandas to reset the index if we want it to start from zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.322758Z",
     "start_time": "2020-03-31T04:06:35.317890Z"
    }
   },
   "outputs": [],
   "source": [
    "male_df = df[df.gender == 0].reset_index(drop=True)\n",
    "female_df = df[df.gender == 1].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing with booleans\n",
    "\n",
    "Boolean or logical indexing is useful if you need to sort the data based on some True or False value.  \n",
    "\n",
    "For instance, who are the people with salaries greater than 90K but lower than 100K ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.334142Z",
     "start_time": "2020-03-31T04:06:35.324474Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pubperyear</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>97630</td>\n",
       "      <td>0</td>\n",
       "      <td>chem</td>\n",
       "      <td>34.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>43</td>\n",
       "      <td>1.264706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>92951</td>\n",
       "      <td>0</td>\n",
       "      <td>neuro</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>20</td>\n",
       "      <td>1.818182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>96936</td>\n",
       "      <td>0</td>\n",
       "      <td>physics</td>\n",
       "      <td>15.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>17</td>\n",
       "      <td>1.133333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender  departm  years   age  publications  pubperyear\n",
       "14   97630       0     chem   34.0  64.0            43    1.264706\n",
       "30   92951       0    neuro   11.0  41.0            20    1.818182\n",
       "54   96936       0  physics   15.0  50.0            17    1.133333"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[ (df.salary > 90000) & (df.salary < 100000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-30T15:29:35.327223Z",
     "start_time": "2020-03-30T15:29:35.323356Z"
    }
   },
   "source": [
    "This also works with the `.loc` method, which is what you need to do if you want to return specific columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.343918Z",
     "start_time": "2020-03-31T04:06:35.335868Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>97630</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>92951</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>96936</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender\n",
       "14   97630       0\n",
       "30   92951       0\n",
       "54   96936       0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[ (df.salary > 90000) & (df.salary < 100000), ['salary', 'gender']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can also return a numpy matrix from a pandas data frame by accessing the `.values` property. This returns a numpy array that can be indexed using numpy integer indexing and slicing.\n",
    "\n",
    "As an example, let's grab the last 10 rows and the first 3 columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.349196Z",
     "start_time": "2020-03-31T04:06:35.345467Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[53638, 0, 'math'],\n",
       "       [59139, 1, 'bio'],\n",
       "       [52968, 1, 'bio'],\n",
       "       [55949, 1, 'chem'],\n",
       "       [58893, 1, 'neuro'],\n",
       "       [53662, 1, 'neuro'],\n",
       "       [57185, 1, 'stat'],\n",
       "       [52254, 1, 'stat'],\n",
       "       [61885, 1, 'math'],\n",
       "       [49542, 1, 'math']], dtype=object)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.values[-10:, :3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Renaming\n",
    "Part of cleaning up the data is renaming with more sensible names. This is easy to do with Pandas.\n",
    "\n",
    "### Renaming Columns\n",
    "We can rename columns with the `.rename` method by passing in a dictionary using the `{'Old Name':'New Name'}`. We either need to assigne the result to a new variable or add `inplace=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.356020Z",
     "start_time": "2020-03-31T04:06:35.350659Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/144029972.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df.rename({'departm':'department','pubperyear':'pub_per_year'}, axis=1, inplace=True)\n"
     ]
    }
   ],
   "source": [
    "df.rename({'departm':'department','pubperyear':'pub_per_year'}, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Renaming Rows\n",
    "Often we may want to change the coding scheme for a variable. For example, it is hard to remember what zeros and ones mean in the gender variable. We can make this easier by changing these with a dictionary `{0:'male', 1:'female'}` with the `replace` method. We can do this `inplace=True` or we can assign it to a new variable. As an example, we will assign this to a new variable to also retain the original lablels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.368702Z",
     "start_time": "2020-03-31T04:06:35.357544Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/1691057888.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['gender_name'] = df['gender'].replace({0:'male', 1:'female'})\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>department</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pub_per_year</th>\n",
       "      <th>gender_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "      <td>2.769231</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "      <td>1.535714</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.300000</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.090909</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender department  years   age  publications  pub_per_year  \\\n",
       "0   86285       0        bio   26.0  64.0            72      2.769231   \n",
       "1   77125       0        bio   28.0  58.0            43      1.535714   \n",
       "2   71922       0        bio   10.0  38.0            23      2.300000   \n",
       "3   70499       0        bio   16.0  46.0            64      4.000000   \n",
       "4   66624       0        bio   11.0  41.0            23      2.090909   \n",
       "\n",
       "  gender_name  \n",
       "0        male  \n",
       "1        male  \n",
       "2        male  \n",
       "3        male  \n",
       "4        male  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['gender_name'] = df['gender'].replace({0:'male', 1:'female'})\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations\n",
    "One of the really fun things about pandas once you get the hang of it is how easy it is to perform operations on the data. It is trivial to compute simple summaries of the data. We can also leverage the object-oriented nature of a pandas object, we can chain together multiple commands.\n",
    "\n",
    "For example, let's grab the mean of a few columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.375340Z",
     "start_time": "2020-03-31T04:06:35.370201Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "years           14.972973\n",
       "age             45.567568\n",
       "publications    21.662162\n",
       "dtype: float64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:,['years', 'age', 'publications']].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also turn these values into a plot with the `plot` method, which we will cover in more detail in future tutorials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.654717Z",
     "start_time": "2020-03-31T04:06:35.376621Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAHjCAYAAACD5X0uAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAh50lEQVR4nO3dfZjVdZ3/8dcBZYQcEDUHWJEgJ+8Q3TTvskQTWnWz1d0ur7yvdbe8S9TN23VFbYF0Q3aXlnQrRUtNN1ctTWFT0Y1YkTSV1FZTYdWRFGK4W9Dh/P4o59csZo3C5zDnPB7Xda6r8/l+h3lrx4vnfM53vqdSrVarAQAopFetBwAAGov4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABS1Sa0H+L/Wrl2bl156Kc3NzalUKrUeBwD4A1Sr1SxbtixDhgxJr15vv7ex0cXHSy+9lKFDh9Z6DADgHVi4cGG23Xbbtz1no4uP5ubmJL8evn///jWeBgD4Q7S3t2fo0KGdf4+/nY0uPt58q6V///7iAwB6mD/kkgkXnAIARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBRm9R6AGD9ed95d9Z6hLrx/KTDaj0C1C07HwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAot5VfEycODGVSiXjxo3rXKtWqxk/fnyGDBmSvn37ZvTo0Zk/f/67nRMAqBPvOD7mzp2bq6++OqNGjeqyfvnll2fy5MmZOnVq5s6dm0GDBmXMmDFZtmzZux4WAOj53lF8LF++PMccc0z+9V//NQMHDuxcr1armTJlSi688MIceeSRGTlyZKZPn56VK1fmhhtuWG9DAwA91zuKj1NPPTWHHXZYDj744C7rzz33XNra2jJ27NjOtaamphxwwAGZPXv2W/5Zq1evTnt7e5cHAFC/NunuF9x00035yU9+krlz565zrK2tLUnS0tLSZb2lpSUvvPDCW/55EydOzCWXXNLdMQCAHqpbOx8LFy7MGWeckW9961vZbLPNfud5lUqly/NqtbrO2pvOP//8LF26tPOxcOHC7owEAPQw3dr5mDdvXhYtWpQ99tijc62joyMPPPBApk6dmqeffjrJr3dABg8e3HnOokWL1tkNeVNTU1OampreyewAQA/UrZ2Pj33sY3n88cfz6KOPdj723HPPHHPMMXn00UczYsSIDBo0KDNnzuz8mjVr1mTWrFnZb7/91vvwAEDP062dj+bm5owcObLL2nve855stdVWnevjxo3LhAkT0tramtbW1kyYMCH9+vXL0Ucfvf6mBgB6rG5fcPr7nHPOOVm1alVOOeWULFmyJHvvvXdmzJiR5ubm9f2tAIAeqFKtVqu1HuK3tbe3Z8CAAVm6dGn69+9f63GgR3nfeXfWeoS68fykw2o9AvQo3fn722e7AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgqG7Fx7Rp0zJq1Kj0798//fv3z7777psf/OAHncer1WrGjx+fIUOGpG/fvhk9enTmz5+/3ocGAHqubsXHtttum0mTJuXhhx/Oww8/nIMOOiif/OQnOwPj8ssvz+TJkzN16tTMnTs3gwYNypgxY7Js2bINMjwA0PN0Kz4+8YlP5NBDD80HPvCBfOADH8jf//3fZ/PNN8+cOXNSrVYzZcqUXHjhhTnyyCMzcuTITJ8+PStXrswNN9ywoeYHAHqYd3zNR0dHR2666aasWLEi++67b5577rm0tbVl7Nixnec0NTXlgAMOyOzZs3/nn7N69eq0t7d3eQAA9avb8fH4449n8803T1NTUz7/+c/n3//937Pzzjunra0tSdLS0tLl/JaWls5jb2XixIkZMGBA52Po0KHdHQkA6EG6HR877LBDHn300cyZMycnn3xyTjjhhPzsZz/rPF6pVLqcX61W11n7beeff36WLl3a+Vi4cGF3RwIAepBNuvsFffr0yfbbb58k2XPPPTN37tz84z/+Y84999wkSVtbWwYPHtx5/qJFi9bZDfltTU1NaWpq6u4YAEAP9a7v81GtVrN69eoMHz48gwYNysyZMzuPrVmzJrNmzcp+++33br8NAFAnurXzccEFF+SQQw7J0KFDs2zZstx00025//77c/fdd6dSqWTcuHGZMGFCWltb09ramgkTJqRfv345+uijN9T8AEAP0634eOWVV3Lcccfl5ZdfzoABAzJq1KjcfffdGTNmTJLknHPOyapVq3LKKadkyZIl2XvvvTNjxow0NzdvkOEBgJ6nUq1Wq7Ue4re1t7dnwIABWbp0afr371/rcaBHed95d9Z6hLrx/KTDaj0C9Cjd+fvbZ7sAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoKhNaj0AAPXrfefdWesR6sbzkw6r9QjrjZ0PAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEV1Kz4mTpyYD33oQ2lubs4222yTP/uzP8vTTz/d5ZxqtZrx48dnyJAh6du3b0aPHp358+ev16EBgJ6rW/Exa9asnHrqqZkzZ05mzpyZN954I2PHjs2KFSs6z7n88sszefLkTJ06NXPnzs2gQYMyZsyYLFu2bL0PDwD0PN26vfrdd9/d5fk111yTbbbZJvPmzctHP/rRVKvVTJkyJRdeeGGOPPLIJMn06dPT0tKSG264IZ/73OfW3+QAQI/0rq75WLp0aZJkyy23TJI899xzaWtry9ixYzvPaWpqygEHHJDZs2e/5Z+xevXqtLe3d3kAAPXrHcdHtVrNWWedlf333z8jR45MkrS1tSVJWlpaupzb0tLSeez/mjhxYgYMGND5GDp06DsdCQDoAd5xfJx22ml57LHHcuONN65zrFKpdHlerVbXWXvT+eefn6VLl3Y+Fi5c+E5HAgB6gG5d8/Gm008/PXfccUceeOCBbLvttp3rgwYNSvLrHZDBgwd3ri9atGid3ZA3NTU1pamp6Z2MAQD0QN3a+ahWqznttNNy66235t57783w4cO7HB8+fHgGDRqUmTNndq6tWbMms2bNyn777bd+JgYAerRu7XyceuqpueGGG3L77benubm58zqOAQMGpG/fvqlUKhk3blwmTJiQ1tbWtLa2ZsKECenXr1+OPvroDfIPAAD0LN2Kj2nTpiVJRo8e3WX9mmuuyYknnpgkOeecc7Jq1aqccsopWbJkSfbee+/MmDEjzc3N62VgAKBn61Z8VKvV33tOpVLJ+PHjM378+Hc6EwBQx3y2CwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFLVJrQfoyd533p21HqEuPD/psFqPAEBBdj4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUFS34+OBBx7IJz7xiQwZMiSVSiW33XZbl+PVajXjx4/PkCFD0rdv34wePTrz589fX/MCAD1ct+NjxYoV2W233TJ16tS3PH755Zdn8uTJmTp1aubOnZtBgwZlzJgxWbZs2bseFgDo+Tbp7hcccsghOeSQQ97yWLVazZQpU3LhhRfmyCOPTJJMnz49LS0tueGGG/K5z33u3U0LAPR46/Waj+eeey5tbW0ZO3Zs51pTU1MOOOCAzJ49+y2/ZvXq1Wlvb+/yAADq13qNj7a2tiRJS0tLl/WWlpbOY//XxIkTM2DAgM7H0KFD1+dIAMBGZoP8tkulUunyvFqtrrP2pvPPPz9Lly7tfCxcuHBDjAQAbCS6fc3H2xk0aFCSX++ADB48uHN90aJF6+yGvKmpqSlNTU3rcwwAYCO2Xnc+hg8fnkGDBmXmzJmda2vWrMmsWbOy3377rc9vBQD0UN3e+Vi+fHmeeeaZzufPPfdcHn300Wy55ZbZbrvtMm7cuEyYMCGtra1pbW3NhAkT0q9fvxx99NHrdXAAoGfqdnw8/PDDOfDAAzufn3XWWUmSE044Iddee23OOeecrFq1KqecckqWLFmSvffeOzNmzEhzc/P6mxoA6LG6HR+jR49OtVr9nccrlUrGjx+f8ePHv5u5AIA65bNdAICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQlPgAAIoSHwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUJT4AACKEh8AQFHiAwAoSnwAAEWJDwCgKPEBABQlPgCAosQHAFCU+AAAihIfAEBR4gMAKEp8AABFiQ8AoCjxAQAUJT4AgKLEBwBQ1AaLj3/5l3/J8OHDs9lmm2WPPfbIgw8+uKG+FQDQg2yQ+PjOd76TcePG5cILL8wjjzySj3zkIznkkEOyYMGCDfHtAIAeZIPEx+TJk/OXf/mXOemkk7LTTjtlypQpGTp0aKZNm7Yhvh0A0INssr7/wDVr1mTevHk577zzuqyPHTs2s2fPXuf81atXZ/Xq1Z3Ply5dmiRpb29f36Otd2tXr6z1CHWhJ/x/3VN4Ta4/Xpfrh9fk+rOxvybfnK9arf7ec9d7fLz66qvp6OhIS0tLl/WWlpa0tbWtc/7EiRNzySWXrLM+dOjQ9T0aG6kBU2o9AazL65KNTU95TS5btiwDBgx423PWe3y8qVKpdHlerVbXWUuS888/P2eddVbn87Vr12bx4sXZaqut3vJ8/nDt7e0ZOnRoFi5cmP79+9d6HPCaZKPkdbl+VKvVLFu2LEOGDPm95673+Nh6663Tu3fvdXY5Fi1atM5uSJI0NTWlqampy9oWW2yxvsdqaP379/cfFBsVr0k2Rl6X797v2/F403q/4LRPnz7ZY489MnPmzC7rM2fOzH777be+vx0A0MNskLddzjrrrBx33HHZc889s+++++bqq6/OggUL8vnPf35DfDsAoAfZIPFx1FFH5bXXXsull16al19+OSNHjsxdd92VYcOGbYhvx+/Q1NSUiy++eJ23taBWvCbZGHldllep/iG/EwMAsJ74bBcAoCjxAQAUJT4AgKLEBwBQlPgAAIoSH0ARzzzzTO65556sWrUqyR/24VNAfRIfdWbhwoX5n//5n87nDz30UMaNG5err766hlPRyF577bUcfPDB+cAHPpBDDz00L7/8cpLkpJNOytlnn13j6YBaEB915uijj859992XJGlra8uYMWPy0EMP5YILLsill15a4+loRGeeeWY22WSTLFiwIP369etcP+qoo3L33XfXcDIamR/Uakt81Jknnngie+21V5Lk5ptvzsiRIzN79uzccMMNufbaa2s7HA1pxowZ+fKXv5xtt922y3pra2teeOGFGk1Fo/ODWm2Jjzrz+uuvd94i+D/+4z9y+OGHJ0l23HHHzu1uKGnFihVddjze9Oqrr7qdNTXjB7XaEh91ZpdddsnXvva1PPjgg5k5c2b+5E/+JEny0ksvZauttqrxdDSij370o7nuuus6n1cqlaxduzZXXHFFDjzwwBpORiPzg1ptbZAPlqN2vvzlL+eII47IFVdckRNOOCG77bZbkuSOO+7orHwo6Yorrsjo0aPz8MMPZ82aNTnnnHMyf/78LF68OD/60Y9qPR4N6s0f1A477LDMnDkzl112WRI/qJXig+XqSLVazYIFCzJw4MB0dHRk4MCBnceef/759OvXL9tss00NJ6RRtbW1Zdq0aZk3b17Wrl2bD37wgzn11FMzePDgWo9Gg7r//vtzxBFHpL29PSeccEK++c1vJkkuuOCCPPXUU7n11ltrPGF9Ex91ZO3atdlss80yf/78tLa21nocgI1aR0dH2tvb/aBWA952qSO9evVKa2trXnvtNfHBRuOxxx57y/VKpZLNNtss2223nQtPqYnevXt3CY8ked/73lebYRqMnY86c+edd2bSpEmZNm1aRo4cWetxIL169UqlUkny/+9q+ubzJNl0001z1FFH5aqrrspmm21WkxlpPK+88kr+5m/+Jj/84Q+zaNGide6429HRUaPJGoP4qDMDBw7MypUr88Ybb6RPnz7p27dvl+OLFy+u0WQ0qttvvz3nnntuvvjFL2avvfZKtVrN3Llz85WvfCUXX3xx3njjjZx33nk56qij8g//8A+1HpcGccghh2TBggU57bTTMnjw4C5BnCSf/OQnazRZYxAfdWb69Olve/yEE04oNAn82l577ZXLLrssH//4x7us33PPPbnooovy0EMP5bbbbsvZZ5+dZ599tkZT0miam5vz4IMPZvfdd6/1KA3JNR91RlywsXn88cczbNiwddaHDRuWxx9/PEmy++67u7cCRQ0dOtSHG9aQm4zVsVWrVqW9vb3LA0rbcccdM2nSpKxZs6Zz7fXXX8+kSZOy4447JklefPHFtLS01GpEGtCUKVNy3nnn5fnnn6/1KA3J2y51ZsWKFTn33HNz880357XXXlvnuIuoKG327Nk5/PDD06tXr4waNSqVSiWPPfZYOjo68v3vfz/77LNPrr/++rS1teWLX/xircelQfz29XH9+vXLpptu2uW46+M2LPFRZ0499dTcd999ufTSS3P88cfnq1/9al588cVcddVVmTRpUo455phaj0gDWr58eb71rW/l5z//earVanbcccccffTRaW5urvVoNCjXx9WW+Kgz2223Xa677rqMHj06/fv3z09+8pNsv/32uf7663PjjTfmrrvuqvWINKif/exnWbBgQZe3X5J0fqYG0DhccFpnFi9enOHDhydJ+vfv37l1uP/+++fkk0+u5Wg0qF/84hc54ogj8vjjj6dSqaRarXb5tUZvBVIrHR0due222/Lkk0+mUqlk5513zuGHH57evXvXerS654LTOjNixIjOC6h23nnn3HzzzUmS733ve9liiy1qNxgN64wzzsjw4cPzyiuvpF+/fnniiScya9as7Lnnnrn//vtrPR4N6plnnslOO+2U448/Prfeemv+7d/+Lccee2x22WUXv/JdgLdd6syVV16Z3r175wtf+ELuu+++HHbYYeno6Mgbb7yRyZMn54wzzqj1iDSYrbfeOvfee29GjRqVAQMG5KGHHsoOO+yQe++9N2effXYeeeSRWo9IAzr00ENTrVbz7W9/O1tuuWWS5LXXXsuxxx6bXr165c4776zxhPVNfNS5BQsW5OGHH8773//+7LbbbrUehwY0cODAzJs3LyNGjMj73//+fP3rX8+BBx6YZ599NrvuumtWrlxZ6xFpQO95z3syZ86c7Lrrrl3Wf/rTn+bDH/5wli9fXqPJGoNrPurY//7v/2a77bbLdtttV+tRaGAjR47MY489lhEjRmTvvffO5Zdfnj59+uTqq6/OiBEjaj0eDaqpqSnLli1bZ3358uXp06dPDSZqLK75qDMdHR257LLL8kd/9EfZfPPN84tf/CJJctFFF+Ub3/hGjaejEf3t3/5t1q5dmyT50pe+lBdeeCEf+chHctddd+Wf/umfajwdjepP//RP89d//df5r//6r1Sr1VSr1cyZMyef//zn/QZWAd52qTOXXnpppk+fnksvvTR/9Vd/lSeeeCIjRozIzTffnCuvvDI//vGPaz0iZPHixRk4cOA6H+YFpfzqV7/KCSeckO9973udNxh74403cvjhh+faa6/NgAEDajxhfRMfdWb77bfPVVddlY997GNpbm7OT3/604wYMSJPPfVU9t133yxZsqTWIwJsNP77v/87Tz31VKrVanbeeedsv/32tR6pIbjmo868+OKLb/kfz9q1a/P666/XYCKAjVdra2taW1trPUbDER91ZpdddsmDDz64zqeI3nLLLfnjP/7jGk0FUHtnnXVWLrvssrznPe/JWWed9bbnTp48udBUjUl81JmLL744xx13XF588cWsXbs2t956a55++ulcd911+f73v1/r8QBq5pFHHuncAXZ/mdpyzUcduueeezJhwoTMmzcva9euzQc/+MH83d/9XcaOHVvr0QDAr9rWmxNPPDF9+/bNrFmzsnz58qxcuTL/+Z//KTwAfstnP/vZt7zPx4oVK/LZz362BhM1FvFRZ5YtW5axY8emtbU1EyZMyEsvvVTrkQA2OtOnT8+qVavWWV+1alWuu+66GkzUWMRHnfnud7+bF198MaeddlpuueWWDBs2LIccckhuueUWv+0CNLz29vYsXbo01Wo1y5YtS3t7e+djyZIlueuuu7LNNtvUesy655qPOvfII4/km9/8Zr7+9a9n8803z7HHHptTTjnFr5YBDalXr15ve3O7SqWSSy65JBdeeGHBqRqP33apYy+//HJmzJiRGTNmpHfv3jn00EMzf/787Lzzzrn88stz5pln1npEgKLuu+++VKvVHHTQQfnud7/b+Ym2SdKnT58MGzYsQ4YMqeGEjcHOR515/fXXc8cdd+Saa67JjBkzMmrUqJx00kk55phj0tzcnCS56aabcvLJJ7vbKdCwXnjhhQwdOjS9ern6oBbsfNSZwYMHZ+3atfn0pz+dhx56KLvvvvs653z84x/PFltsUXw2gI3FmzdiXLlyZRYsWJA1a9Z0OT5q1KhajNUw7HzUmeuvvz6f+tSnstlmm9V6FICN1i9/+ct85jOfyQ9+8IO3PN7R0VF4osZiv6nOHHfcccID4PcYN25clixZkjlz5qRv3765++67M3369LS2tuaOO+6o9Xh1z9suADSce++9N7fffns+9KEPpVevXhk2bFjGjBmT/v37Z+LEiTnssMNqPWJds/MBQMNZsWJF5/08ttxyy/zyl79Mkuy66675yU9+UsvRGoL4AKDh7LDDDnn66aeTJLvvvnuuuuqqvPjii/na176WwYMH13i6+ueCUwAazre//e28/vrrOfHEE/PII4/k4x//eF577bX06dMn1157bY466qhaj1jXxAcADW/lypV56qmnst1222Xrrbeu9Th1T3wAAEW55gOAhvMXf/EXmTRp0jrrV1xxRT71qU/VYKLGYucDgIbz3ve+N/fee2923XXXLuuPP/54Dj744Lzyyis1mqwx2PkAoOEsX748ffr0WWd90003TXt7ew0maiziA4CGM3LkyHznO99ZZ/2mm27KzjvvXIOJGos7nALQcC666KL8+Z//eZ599tkcdNBBSZIf/vCHufHGG3PLLbfUeLr655oPABrSnXfemQkTJuTRRx9N3759M2rUqFx88cU54IADaj1a3RMfAEBRrvkAAIpyzQcADWHLLbfMz3/+82y99dYZOHBgKpXK7zx38eLFBSdrPOIDgIZw5ZVXprm5OUkyZcqU2g7T4FzzAQAUZecDgIbQnZuH9e/ffwNOgp0PABpCr1693vY6jySpVqupVCrp6OgoNFVjsvMBQEO47777aj0Cv2HnAwAoys4HAA1pyZIl+cY3vpEnn3wylUolO+20Uz7zmc9kyy23rPVodc/OBwANZ9asWTn88MMzYMCA7LnnnkmSefPm5Ve/+lXuuOMOt1jfwMQHAA1n5MiR2W+//TJt2rT07t07SdLR0ZFTTjklP/rRj/LEE0/UeML6Jj4AaDh9+/bNo48+mh122KHL+tNPP53dd989q1atqtFkjcFnuwDQcD74wQ/mySefXGf9ySefzO67715+oAbjglMAGsJjjz3W+b+/8IUv5IwzzsgzzzyTffbZJ0kyZ86cfPWrX82kSZNqNWLD8LYLAA3hzZuM/b6/9txkbMOz8wFAQ3juuedqPQK/YecDACjKzgcADee666572+PHH398oUkak50PABrOwIEDuzx//fXXs3LlyvTp0yf9+vXL4sWLazRZY/CrtgA0nCVLlnR5LF++PE8//XT233//3HjjjbUer+7Z+QCA33j44Ydz7LHH5qmnnqr1KHXNzgcA/Ebv3r3z0ksv1XqMuueCUwAazh133NHlebVazcsvv5ypU6fmwx/+cI2mahzedgGg4fTq1XXjv1Kp5L3vfW8OOuigfOUrX8ngwYNrNFljEB8ANLS1a9cmWTdI2HD8mwagIX3jG9/IyJEj07dv3/Tt2zcjR47M17/+9VqP1RBc8wFAw7noooty5ZVX5vTTT8++++6bJPnxj3+cM888M88//3y+9KUv1XjC+uZtFwAaztZbb51//ud/zqc//eku6zfeeGNOP/30vPrqqzWarDF42wWAhtPR0ZE999xznfU99tgjb7zxRg0maiziA4CGc+yxx2batGnrrF999dU55phjajBRY/G2CwAN5/TTT891112XoUOHZp999kmSzJkzJwsXLszxxx+fTTfdtPPcyZMn12rMuiU+AGg4Bx544B90XqVSyb333ruBp2k84gMAKMo1HwBAUeIDAChKfAAARYkPAKAo8QEAFCU+AICixAcAUNT/A7PNJFQzI50hAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "df.loc[:,['years', 'age', 'publications']].mean().plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps we want to see if there are any correlations in our dataset. We can do this with the `.corr` method. More recent versions of Pandas might produce an error if there are any columns containing string data. To avoid this issue set `numeric_only=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.665207Z",
     "start_time": "2020-03-31T04:06:35.656326Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pub_per_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>salary</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.300071</td>\n",
       "      <td>0.303275</td>\n",
       "      <td>0.275534</td>\n",
       "      <td>0.427426</td>\n",
       "      <td>-0.016988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender</th>\n",
       "      <td>-0.300071</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.275468</td>\n",
       "      <td>-0.277098</td>\n",
       "      <td>-0.249410</td>\n",
       "      <td>0.024210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>years</th>\n",
       "      <td>0.303275</td>\n",
       "      <td>-0.275468</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.958181</td>\n",
       "      <td>0.323965</td>\n",
       "      <td>-0.541125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.275534</td>\n",
       "      <td>-0.277098</td>\n",
       "      <td>0.958181</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.328285</td>\n",
       "      <td>-0.498825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>publications</th>\n",
       "      <td>0.427426</td>\n",
       "      <td>-0.249410</td>\n",
       "      <td>0.323965</td>\n",
       "      <td>0.328285</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.399865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pub_per_year</th>\n",
       "      <td>-0.016988</td>\n",
       "      <td>0.024210</td>\n",
       "      <td>-0.541125</td>\n",
       "      <td>-0.498825</td>\n",
       "      <td>0.399865</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                salary    gender     years       age  publications  \\\n",
       "salary        1.000000 -0.300071  0.303275  0.275534      0.427426   \n",
       "gender       -0.300071  1.000000 -0.275468 -0.277098     -0.249410   \n",
       "years         0.303275 -0.275468  1.000000  0.958181      0.323965   \n",
       "age           0.275534 -0.277098  0.958181  1.000000      0.328285   \n",
       "publications  0.427426 -0.249410  0.323965  0.328285      1.000000   \n",
       "pub_per_year -0.016988  0.024210 -0.541125 -0.498825      0.399865   \n",
       "\n",
       "              pub_per_year  \n",
       "salary           -0.016988  \n",
       "gender            0.024210  \n",
       "years            -0.541125  \n",
       "age              -0.498825  \n",
       "publications      0.399865  \n",
       "pub_per_year      1.000000  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.corr(numeric_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merging Data\n",
    "Another common data manipulation goal is to merge datasets. There are multiple ways to do this in pandas, we will cover concatenation, append, and merge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concatenation\n",
    "Concatenation describes the process of *stacking* dataframes together. Older versions of pandas also had an `.append()` method, which has been deprecated since pandas 1.4. The main thing to consider is to make sure that the shapes of the two dataframes are the same as well as the index labels. For example, if we wanted to vertically stack two dataframe, they need to have the same column names. \n",
    "\n",
    "Remember that we previously created two separate dataframes for males and females?  Let's put them back together using the `pd.concat` method. Note how the index of this output retains the old index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.671004Z",
     "start_time": "2020-03-31T04:06:35.666780Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "combined_data = pd.concat([female_df, male_df], axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-30T15:56:06.841587Z",
     "start_time": "2020-03-30T15:56:06.838466Z"
    }
   },
   "source": [
    "We can reset the index using the `reset_index` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.689990Z",
     "start_time": "2020-03-31T04:06:35.672497Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>departm</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pubperyear</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "      <td>2.769231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "      <td>1.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.090909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>53662</td>\n",
       "      <td>1</td>\n",
       "      <td>neuro</td>\n",
       "      <td>1.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>57185</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>9.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.777778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>52254</td>\n",
       "      <td>1</td>\n",
       "      <td>stat</td>\n",
       "      <td>2.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>9</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>61885</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>23.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9</td>\n",
       "      <td>0.391304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>49542</td>\n",
       "      <td>1</td>\n",
       "      <td>math</td>\n",
       "      <td>3.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1.666667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary  gender departm  years   age  publications  pubperyear\n",
       "0    86285       0     bio   26.0  64.0            72    2.769231\n",
       "1    77125       0     bio   28.0  58.0            43    1.535714\n",
       "2    71922       0     bio   10.0  38.0            23    2.300000\n",
       "3    70499       0     bio   16.0  46.0            64    4.000000\n",
       "4    66624       0     bio   11.0  41.0            23    2.090909\n",
       "..     ...     ...     ...    ...   ...           ...         ...\n",
       "69   53662       1   neuro    1.0  31.0             3    3.000000\n",
       "70   57185       1    stat    9.0  39.0             7    0.777778\n",
       "71   52254       1    stat    2.0  32.0             9    4.500000\n",
       "72   61885       1    math   23.0  60.0             9    0.391304\n",
       "73   49542       1    math    3.0  33.0             5    1.666667\n",
       "\n",
       "[74 rows x 7 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([male_df, female_df], axis = 0).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also concatenate columns in addition to rows. Make sure that the number of rows are the same in each dataframe. For this example, we will just create two new data frames with a subset of the columns and then combine them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.701410Z",
     "start_time": "2020-03-31T04:06:35.691598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender   age  publications\n",
       "0   86285       0  64.0            72\n",
       "1   77125       0  58.0            43\n",
       "2   71922       0  38.0            23\n",
       "3   70499       0  46.0            64\n",
       "4   66624       0  41.0            23"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df[['salary', 'gender']]\n",
    "df2 = df[['age', 'publications']]\n",
    "df3 = pd.concat([df1, df2], axis=1)\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merge\n",
    "The most powerful method of merging data is using the `pd.merge` method. This allows you to merge datasets of different shapes and sizes on specific variables that match. This is very common when you need to merge multiple sql tables together for example. \n",
    "\n",
    "In this example, we are creating two separate data frames that have different states and columns and will merge on the `State` column.\n",
    "\n",
    "First, we will only retain rows where there is a match across dataframes, using `how=inner`. This is equivalent to an 'and' join in sql."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.726003Z",
     "start_time": "2020-03-31T04:06:35.714878Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Capital</th>\n",
       "      <th>Population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>California</td>\n",
       "      <td>Sacremento</td>\n",
       "      <td>39512223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>New Hampshire</td>\n",
       "      <td>Concord</td>\n",
       "      <td>1359711</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           State     Capital Population\n",
       "0     California  Sacremento   39512223\n",
       "1  New Hampshire     Concord    1359711"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = pd.DataFrame({'State':['California', 'Colorado', 'New Hampshire'], \n",
    "                    'Capital':['Sacremento', 'Denver', 'Concord']})\n",
    "df2 = pd.DataFrame({'State':['California', 'New Hampshire', 'New York'], \n",
    "                    'Population':['39512223', '1359711', '19453561']})\n",
    "\n",
    "df3 = pd.merge(left=df1, right=df2, on='State', how='inner')\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how there are only two rows in the merged dataframe.\n",
    "\n",
    "We can also be more inclusive and match on `State` column, but retain all rows. This is equivalent to an 'or' join."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.736745Z",
     "start_time": "2020-03-31T04:06:35.727441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Capital</th>\n",
       "      <th>Population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>California</td>\n",
       "      <td>Sacremento</td>\n",
       "      <td>39512223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Colorado</td>\n",
       "      <td>Denver</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New Hampshire</td>\n",
       "      <td>Concord</td>\n",
       "      <td>1359711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>New York</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19453561</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           State     Capital Population\n",
       "0     California  Sacremento   39512223\n",
       "1       Colorado      Denver        NaN\n",
       "2  New Hampshire     Concord    1359711\n",
       "3       New York         NaN   19453561"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = pd.merge(left=df1, right=df2, on='State', how='outer')\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very handy way to merge data when you have lots of files with missing data.  See Jake Vanderplas's [tutorial](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.07-Merge-and-Join.ipynb) for a more in depth overview."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping\n",
    "We've seen above that it is very easy to summarize data over columns using the builtin functions such as `pd.mean()`. Sometimes we are interested in summarizing data over different groups of rows. For example, what is the mean of participants in Condition A compared to Condition B?\n",
    "\n",
    "This is suprisingly easy to compute in pandas using the `groupby` operator, where we aggregate data using a specific operation over different labels.\n",
    "\n",
    "One useful way to conceptualize this is using the **Split, Apply, Combine** operation (similar to map-reduce).\n",
    "\n",
    "![](../images/pandas/split-apply-combine.png)\n",
    "\n",
    "This figure is taken from Jake Vanderplas's tutorial and highlights how input data can be *split* on some key and then an operation such as sum can be *applied* separately to each split. Finally, the results of the applied function for each key can be *combined* into a new data frame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groupby\n",
    "In this example, we will use the `groupby` operator to split the data based on gender labels and separately calculate the mean for each group. Note that newer versions of pandas might throw an error if you try to perform a numeric computation such as `.mean()` on a dataframe containing columns of string data. Use the flag `numeric_only=True` to avoid this issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.748994Z",
     "start_time": "2020-03-31T04:06:35.738332Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pub_per_year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>55719.666667</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>38.888889</td>\n",
       "      <td>11.555556</td>\n",
       "      <td>2.043170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>69108.492308</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.846154</td>\n",
       "      <td>46.492308</td>\n",
       "      <td>23.061538</td>\n",
       "      <td>1.924709</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   salary  gender      years        age  publications  \\\n",
       "gender_name                                                             \n",
       "female       55719.666667     1.0   8.666667  38.888889     11.555556   \n",
       "male         69108.492308     0.0  15.846154  46.492308     23.061538   \n",
       "\n",
       "             pub_per_year  \n",
       "gender_name                \n",
       "female           2.043170  \n",
       "male             1.924709  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('gender_name').mean(numeric_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other default aggregation methods include `.count()`, `.mean()`, `.median()`, `.min()`, `.max()`, `.std()`, `.var()`, and `.sum()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform\n",
    "While the split, apply, combine operation that we just demonstrated is extremely usefuly to quickly summarize data based on a grouping key, the resulting data frame is compressed to one row per grouping label.\n",
    "\n",
    "Sometimes, we would like to perform an operation over groups, but retain the original data shape. One common example is standardizing data within a subject or grouping variable. Normally, you might think to loop over subject ids and separately z-score or center a variable and then recombine the subject data using a vertical concatenation operation.\n",
    "\n",
    "The `transform` method in pandas can make this much easier and faster!\n",
    "\n",
    "Suppose we want to compute the standardized salary separately for each department. We can standardize using a z-score which requires subtracting the departmental mean from each professor's salary in that department, and then dividing it by the departmental standard deviation. \n",
    "\n",
    "We can do this by using the `groupby(key)` method chained with the `.transform(function)` method. It will group the dataframe by the key column, perform the \"function\" transformation of the data and return data in same format. We can then assign the results to a new column in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/2225281835.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['salary_dept_z'] = (df['salary'] - df['salary'].groupby(df['department']).transform('mean'))/df['salary'].groupby(df['department']).transform('std')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>department</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pub_per_year</th>\n",
       "      <th>gender_name</th>\n",
       "      <th>salary_dept_z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "      <td>2.769231</td>\n",
       "      <td>male</td>\n",
       "      <td>2.468065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "      <td>1.535714</td>\n",
       "      <td>male</td>\n",
       "      <td>1.493198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.300000</td>\n",
       "      <td>male</td>\n",
       "      <td>0.939461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>male</td>\n",
       "      <td>0.788016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.090909</td>\n",
       "      <td>male</td>\n",
       "      <td>0.375613</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender department  years   age  publications  pub_per_year  \\\n",
       "0   86285       0        bio   26.0  64.0            72      2.769231   \n",
       "1   77125       0        bio   28.0  58.0            43      1.535714   \n",
       "2   71922       0        bio   10.0  38.0            23      2.300000   \n",
       "3   70499       0        bio   16.0  46.0            64      4.000000   \n",
       "4   66624       0        bio   11.0  41.0            23      2.090909   \n",
       "\n",
       "  gender_name  salary_dept_z  \n",
       "0        male       2.468065  \n",
       "1        male       1.493198  \n",
       "2        male       0.939461  \n",
       "3        male       0.788016  \n",
       "4        male       0.375613  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['salary_dept_z'] = (df['salary'] - df['salary'].groupby(df['department']).transform('mean'))/df['salary'].groupby(df['department']).transform('std')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T02:37:08.134136Z",
     "start_time": "2020-03-31T02:37:08.121669Z"
    }
   },
   "source": [
    "This worked well, but is also pretty verbose. We can simplify the syntax a little bit more using a `lambda` function, where we can define the zscore function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.829918Z",
     "start_time": "2020-03-31T04:06:35.811444Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/5b/m183lc3x27n9krrzz85z2x1c0000gn/T/ipykernel_28379/94412960.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['salary_dept_z'] = df['salary'].groupby(df['department']).transform(calc_zscore)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>gender</th>\n",
       "      <th>department</th>\n",
       "      <th>years</th>\n",
       "      <th>age</th>\n",
       "      <th>publications</th>\n",
       "      <th>pub_per_year</th>\n",
       "      <th>gender_name</th>\n",
       "      <th>salary_dept_z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>86285</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72</td>\n",
       "      <td>2.769231</td>\n",
       "      <td>male</td>\n",
       "      <td>2.468065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>77125</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>28.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>43</td>\n",
       "      <td>1.535714</td>\n",
       "      <td>male</td>\n",
       "      <td>1.493198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71922</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>10.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.300000</td>\n",
       "      <td>male</td>\n",
       "      <td>0.939461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70499</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>16.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>64</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>male</td>\n",
       "      <td>0.788016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66624</td>\n",
       "      <td>0</td>\n",
       "      <td>bio</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>23</td>\n",
       "      <td>2.090909</td>\n",
       "      <td>male</td>\n",
       "      <td>0.375613</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   salary  gender department  years   age  publications  pub_per_year  \\\n",
       "0   86285       0        bio   26.0  64.0            72      2.769231   \n",
       "1   77125       0        bio   28.0  58.0            43      1.535714   \n",
       "2   71922       0        bio   10.0  38.0            23      2.300000   \n",
       "3   70499       0        bio   16.0  46.0            64      4.000000   \n",
       "4   66624       0        bio   11.0  41.0            23      2.090909   \n",
       "\n",
       "  gender_name  salary_dept_z  \n",
       "0        male       2.468065  \n",
       "1        male       1.493198  \n",
       "2        male       0.939461  \n",
       "3        male       0.788016  \n",
       "4        male       0.375613  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_zscore = lambda x: (x - x.mean()) / x.std()\n",
    "\n",
    "df['salary_dept_z'] = df['salary'].groupby(df['department']).transform(calc_zscore)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more in depth overview of data aggregation and grouping, check out Jake Vanderplas's [tutorial](http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/03.08-Aggregation-and-Grouping.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshaping Data\n",
    "The last topic we will cover in this tutorial is reshaping data. Data is often in the form of observations by features, in which there is a single row for each independent observation of data and a separate column for each feature of the data. This is commonly referred to as as the *wide* format. However, when running regression or plotting in libraries such as seaborn, we often want our data in the *long* format, in which each grouping variable is specified in a separate column.\n",
    "\n",
    "In this section we cover how to go from wide to long using the `melt` operation and from long to wide using the `pivot` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Melt\n",
    "To `melt` a dataframe into the long format, we need to specify which variables are the `id_vars`, which ones should be combined into a `value_var`, and finally, what we should label the column name for the `value_var`, and also for the `var_name`. We will call the values 'Ratings' and variables 'Condition'.\n",
    "\n",
    "First, we need to create a dataset to play with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.839588Z",
     "start_time": "2020-03-31T04:06:35.831470Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.288893</td>\n",
       "      <td>0.189514</td>\n",
       "      <td>0.662512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.494610</td>\n",
       "      <td>0.012093</td>\n",
       "      <td>0.181321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.631140</td>\n",
       "      <td>0.999889</td>\n",
       "      <td>0.113135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.584941</td>\n",
       "      <td>0.987392</td>\n",
       "      <td>0.176449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.967856</td>\n",
       "      <td>0.157259</td>\n",
       "      <td>0.465654</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID         A         B         C\n",
       "0  1.0  0.288893  0.189514  0.662512\n",
       "1  2.0  0.494610  0.012093  0.181321\n",
       "2  3.0  0.631140  0.999889  0.113135\n",
       "3  4.0  0.584941  0.987392  0.176449\n",
       "4  5.0  0.967856  0.157259  0.465654"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame(np.vstack([np.arange(1,6), np.random.random((3, 5))]).T, columns=['ID','A', 'B', 'C'])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's melt the dataframe into the long format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.852174Z",
     "start_time": "2020-03-31T04:06:35.841058Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Condition</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.288893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.494610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.631140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.584941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>A</td>\n",
       "      <td>0.967856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.189514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.012093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3.0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.999889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4.0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.987392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5.0</td>\n",
       "      <td>B</td>\n",
       "      <td>0.157259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.662512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2.0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.181321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>3.0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.113135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4.0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.176449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5.0</td>\n",
       "      <td>C</td>\n",
       "      <td>0.465654</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ID Condition    Rating\n",
       "0   1.0         A  0.288893\n",
       "1   2.0         A  0.494610\n",
       "2   3.0         A  0.631140\n",
       "3   4.0         A  0.584941\n",
       "4   5.0         A  0.967856\n",
       "5   1.0         B  0.189514\n",
       "6   2.0         B  0.012093\n",
       "7   3.0         B  0.999889\n",
       "8   4.0         B  0.987392\n",
       "9   5.0         B  0.157259\n",
       "10  1.0         C  0.662512\n",
       "11  2.0         C  0.181321\n",
       "12  3.0         C  0.113135\n",
       "13  4.0         C  0.176449\n",
       "14  5.0         C  0.465654"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_long = pd.melt(data,id_vars='ID', value_vars=['A', 'B', 'C'], var_name='Condition', value_name='Rating')\n",
    "df_long\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the id variable is repeated for each condition?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pivot\n",
    "We can also go back to the wide data format from a long dataframe using `pivot`. We just need to specify the variable containing the labels which will become the `columns` and the `values` column that will be broken into separate columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:06:35.865063Z",
     "start_time": "2020-03-31T04:06:35.854444Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Condition</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.288893</td>\n",
       "      <td>0.189514</td>\n",
       "      <td>0.662512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2.0</th>\n",
       "      <td>0.494610</td>\n",
       "      <td>0.012093</td>\n",
       "      <td>0.181321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.0</th>\n",
       "      <td>0.631140</td>\n",
       "      <td>0.999889</td>\n",
       "      <td>0.113135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4.0</th>\n",
       "      <td>0.584941</td>\n",
       "      <td>0.987392</td>\n",
       "      <td>0.176449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5.0</th>\n",
       "      <td>0.967856</td>\n",
       "      <td>0.157259</td>\n",
       "      <td>0.465654</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Condition         A         B         C\n",
       "ID                                     \n",
       "1.0        0.288893  0.189514  0.662512\n",
       "2.0        0.494610  0.012093  0.181321\n",
       "3.0        0.631140  0.999889  0.113135\n",
       "4.0        0.584941  0.987392  0.176449\n",
       "5.0        0.967856  0.157259  0.465654"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_wide = df_long.pivot(index='ID', columns='Condition', values='Rating')\n",
    "df_wide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Exercises ( Homework)\n",
    "The following exercises uses the dataset \"salary_exercise.csv\" adapted from material available [here](http://data.princeton.edu/wws509/datasets/#salary)\n",
    "\n",
    "These are the salary data used in Weisberg's book, consisting of observations on six variables for 52 tenure-track professors in a small college. The variables are:\n",
    "\n",
    " - sx = Sex, coded 1 for female and 0 for male\n",
    " - rk = Rank, coded\n",
    " - 1 for assistant professor,\n",
    " - 2 for associate professor, and\n",
    " - 3 for full professor\n",
    " - yr = Number of years in current rank\n",
    " - dg = Highest degree, coded 1 if doctorate, 0 if masters\n",
    " - yd = Number of years since highest degree was earned\n",
    " - sl = Academic year salary, in dollars.\n",
    "\n",
    "Reference: S. Weisberg (1985). Applied Linear Regression, Second Edition. New York: John Wiley and Sons. Page 194.\n",
    "\n",
    "### Exercise 1\n",
    "\n",
    "Read the salary_exercise.csv into a dataframe, and change the column names to a more readable format such as sex, rank, yearsinrank, degree, yearssinceHD, and salary.   \n",
    "\n",
    "Clean the data by excluding rows with any missing value. \n",
    "\n",
    "What are the overall mean, standard deviation, min, and maximum of professors' salary? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:04:56.411197Z",
     "start_time": "2020-03-31T04:04:55.140Z"
    }
   },
   "outputs": [],
   "source": [
    "salary_file_url = 'https://raw.githubusercontent.com/ljchang/dartbrains/master/data/salary/salary_exercise.csv'\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "Create two separate dataframes based on the type of degree. Now calculate the mean salary of the 5 oldest professors of each degree type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-31T04:02:23.981203Z",
     "start_time": "2020-03-31T04:02:23.966211Z"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "What is the correlation between the standardized salary *across* all ranks and the standardized salary *within* ranks?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "py312",
   "language": "python",
   "name": "py312"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "44px",
    "width": "252px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "231.837px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}