Outliers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dealing with Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Sometimes outliers can mess up an analysis; you usually don't want a handful of data points to skew the overall results. Let's revisit our example of income data, with Donald Trump thrown in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEJCAYAAAB/pOvWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEU9JREFUeJzt3X+s3XV9x/Hny1YY6lCwXcMKrJ3rNIXIlI51agyOP6i6\nrLgpqVPpTEOzwIxbZibsj7ksqYPEbI5ssHSolM3IOiWj2cSNVBkuCuwyQShY6USkFWj9yeYytPDe\nH+fDdrifW3p7z+09ve3zkZycz/l8P5/P+XzS5r7u9/s953NTVUiSNOx5456AJOnIYzhIkjqGgySp\nYzhIkjqGgySpYzhIkjqGgySpYzhIkjqGgySps3DcE5ipRYsW1bJly8Y9DUmaV+66665vVdXig7Wb\nt+GwbNkyJiYmxj0NSZpXkjw8nXZeVpIkdQwHSVLHcJAkdQwHSVLHcJAkdQ4aDkk+mmRvkvuG6k5O\nckuSB9vzSUPHLk+yK8nOJOcP1Z+d5N527KokafXHJ/nbVn9HkmWzu0RJ0qGazpnDdcCaSXWXAdur\nagWwvb0myUpgHXBG63N1kgWtzzXAxcCK9nhmzA3Ad6vqZ4A/Ba6c6WIkSbPjoOFQVbcB35lUvRbY\n0spbgAuG6m+oqier6iFgF3BOklOAE6vq9hr8XdLrJ/V5ZqxPAuc9c1YhSRqPmd5zWFJVj7byY8CS\nVl4KPDLUbnerW9rKk+uf1aeq9gPfB146w3lJkmbByN+QrqpKUrMxmYNJshHYCHD66afPeJxll/3j\nlPVfv+LNMx5Tko4mMz1zeLxdKqI97231e4DThtqd2ur2tPLk+mf1SbIQeDHw7anetKo2V9Wqqlq1\nePFBtwaRJM3QTMNhG7C+ldcDNw3Vr2ufQFrO4Mbzne0S1BNJVrf7CRdN6vPMWG8FPtvuS0iSxuSg\nl5WSfAI4F1iUZDfwAeAKYGuSDcDDwIUAVbUjyVbgfmA/cGlVPdWGuoTBJ59OAG5uD4CPAH+dZBeD\nG9/rZmVlkqQZO2g4VNXbD3DovAO03wRsmqJ+Ajhzivr/Ad52sHlIkuaO35CWJHUMB0lSx3CQJHUM\nB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lS\nx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQ\nJHUMB0lSx3CQJHUMB0lSZ6RwSHJ5kvuT3JfkE0l+LMnJSW5J8mB7PmlS+11JdiY5f6j+7CT3tmNX\nJcko85IkjWbG4ZBkGbAROLuqzgQWAOuAy4DtVbUC2N5ek2RlO34GsAa4OsmCNtw1wMXAivZYM9N5\nSZJGN8qZwxPAj4ATkiwEXgB8E1gLbGlttgAXtPJa4IaqerKqHgJ2AeckOQU4sapur6oCrh/qI0ka\ngxmHQ1V9B/gQ8A3gUeD7VfXPwJKqerQ1ewxY0spLgUeGhtjd6pa28uR6SdKYjHJZ6WXA7wDLgZ8E\nXpjkncNt2plAjTTDZ7/nxiQTSSb27ds3W8NKkiYZ5bLSKuALVbWvqn4E3Ai8Bni8XSqiPe9t7fcA\npw31P7XV7WnlyfWdqtpcVauqatXixYtHmLok6bmMEg47gdVJXtA+XXQe8ACwDVjf2qwHbmrlbcC6\nJMcnWc7gxvOd7RLUE0lWt3EuGuojSRqDhTPtWFV3J7kemACeBr4EbAZeBGxNsgF4GLiwtd+RZCtw\nP7AfuLSqnmrDXQJcB5wA3NwekqQxmXE4AFTVlcCVk6qfZHAWMVX7TcCmKeongDNHmYskafb4DWlJ\nUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdw\nkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1\nDAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUmekcEjykiSfTPKVJA8k+cUkJye5JcmD7fmk\nofaXJ9mVZGeS84fqz05ybzt2VZKMMi9J0mhGPXP4M+AzVfUK4CzgAeAyYHtVrQC2t9ckWQmsA84A\n1gBXJ1nQxrkGuBhY0R5rRpyXJGkEMw6HJC8GXg98BKCqflhV3wPWAltasy3ABa28Frihqp6sqoeA\nXcA5SU4BTqyq26uqgOuH+kiSxmCUM4flwD7gY0m+lOTaJC8EllTVo63NY8CSVl4KPDLUf3erW9rK\nk+slSWMySjgsBF4NXFNVrwJ+QLuE9Ix2JlAjvMezJNmYZCLJxL59+2ZrWEnSJKOEw25gd1Xd0V5/\nkkFYPN4uFdGe97bje4DThvqf2ur2tPLk+k5Vba6qVVW1avHixSNMXZL0XGYcDlX1GPBIkpe3qvOA\n+4FtwPpWtx64qZW3AeuSHJ9kOYMbz3e2S1BPJFndPqV00VAfSdIYLByx/3uAjyc5Dvga8G4GgbM1\nyQbgYeBCgKrakWQrgwDZD1xaVU+1cS4BrgNOAG5uD0nSmIwUDlV1N7BqikPnHaD9JmDTFPUTwJmj\nzEWSNHv8hrQkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4\nSJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6\nhoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6I4dDkgVJvpTkH9rrk5PckuTB9nzS\nUNvLk+xKsjPJ+UP1Zye5tx27KklGnZckaeZm48zhvcADQ68vA7ZX1Qpge3tNkpXAOuAMYA1wdZIF\nrc81wMXAivZYMwvzkiTN0EjhkORU4M3AtUPVa4EtrbwFuGCo/oaqerKqHgJ2AeckOQU4sapur6oC\nrh/qI0kag1HPHD4M/B7w9FDdkqp6tJUfA5a08lLgkaF2u1vd0laeXC9JGpMZh0OSXwb2VtVdB2rT\nzgRqpu8xxXtuTDKRZGLfvn2zNawkaZJRzhxeC/xKkq8DNwC/lORvgMfbpSLa897Wfg9w2lD/U1vd\nnlaeXN+pqs1VtaqqVi1evHiEqUuSnsuMw6GqLq+qU6tqGYMbzZ+tqncC24D1rdl64KZW3gasS3J8\nkuUMbjzf2S5BPZFkdfuU0kVDfSRJY7DwMIx5BbA1yQbgYeBCgKrakWQrcD+wH7i0qp5qfS4BrgNO\nAG5uD0nSmMxKOFTVrcCtrfxt4LwDtNsEbJqifgI4czbmIkkand+QliR1DAdJUsdwkCR1DAdJUsdw\nkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1\nDAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJ\nUsdwkCR1DAdJUsdwkCR1ZhwOSU5L8rkk9yfZkeS9rf7kJLckebA9nzTU5/Iku5LsTHL+UP3ZSe5t\nx65KktGWJUkaxShnDvuB362qlcBq4NIkK4HLgO1VtQLY3l7Tjq0DzgDWAFcnWdDGuga4GFjRHmtG\nmJckaUQzDoeqerSq/r2V/xN4AFgKrAW2tGZbgAtaeS1wQ1U9WVUPAbuAc5KcApxYVbdXVQHXD/WR\nJI3BrNxzSLIMeBVwB7Ckqh5thx4DlrTyUuCRoW67W93SVp5cP9X7bEwykWRi3759szF1SdIURg6H\nJC8CPgX8dlU9MXysnQnUqO8xNN7mqlpVVasWL148W8NKkiYZKRySPJ9BMHy8qm5s1Y+3S0W0572t\nfg9w2lD3U1vdnlaeXC9JGpNRPq0U4CPAA1X1J0OHtgHrW3k9cNNQ/bokxydZzuDG853tEtQTSVa3\nMS8a6iNJGoOFI/R9LfAu4N4kd7e63weuALYm2QA8DFwIUFU7kmwF7mfwSadLq+qp1u8S4DrgBODm\n9pAkjcmMw6Gq/hU40PcRzjtAn03ApinqJ4AzZzoXSdLs8hvSkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO\n4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ\n6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgO\nkqSO4SBJ6hwx4ZBkTZKdSXYluWzc85GkY9kREQ5JFgB/AbwRWAm8PcnK8c5Kko5dR0Q4AOcAu6rq\na1X1Q+AGYO2Y5yRJx6wjJRyWAo8Mvd7d6iRJY7Bw3BM4FEk2Ahvby/9KsnOGQy0CvtWNf+VMZzYv\nTLnmo5xrPja45kPzU9NpdKSEwx7gtKHXp7a6Z6mqzcDmUd8syURVrRp1nPnENR8bXPOxYS7WfKRc\nVvo3YEWS5UmOA9YB28Y8J0k6Zh0RZw5VtT/JbwH/BCwAPlpVO8Y8LUk6Zh0R4QBQVZ8GPj1Hbzfy\npal5yDUfG1zzseGwrzlVdbjfQ5I0zxwp9xwkSUeQozocDrYlRwauase/nOTV45jnbJrGmt/R1npv\nki8kOWsc85xN0916JcnPJ9mf5K1zOb/DYTprTnJukruT7EjyL3M9x9k0jf/Xi5J8Jsk9bb3vHsc8\nZ1OSjybZm+S+Axw/vD+/quqofDC4sf0fwE8DxwH3ACsntXkTcDMQYDVwx7jnPQdrfg1wUiu/8VhY\n81C7zzK4r/XWcc97Dv6dXwLcD5zeXv/EuOd9mNf7h8CVrbwY+A5w3LjnPuK6Xw+8GrjvAMcP68+v\no/nMYTpbcqwFrq+B24GXJDllric6iw665qr6QlV9t728ncF3Suaz6W698h7gU8DeuZzcYTKdNf86\ncGNVfQOgqubzuqez3seAH08S4EUMwmH/3E5zdlXVbQzWcSCH9efX0RwO09mS42jbtuNQ17OBwW8e\n89lB15xkKfAW4Jo5nNfhNJ1/558FTkpya5K7klw0Z7ObfdNZ718x2LTzm8C9wHur6um5md7YHNaf\nX0fMR1k1t5K8gUE4vG7cc5kDHwbeX1VPD36xPCYsBM4GzgNOAL6Y5Paq+up4p3XYXA58GXgD8DLg\nliSfr6onxjut+etoDofpbMkxrW075pFprSfJK4FrgTdW1bfnaG6Hy3TWvAq4oQXDIuBNSfZX1d/P\nzRRn3XTWvBv4dlX9APhBktuAs4D5GA7TWe9rgQ/W4GL8riQPAa8A7pybKY7FYf35dTRfVprOlhzb\ngIvaXf/VwPer6tG5nugsOuiak5wO3Ai86yj5LfKga66q5VW1rKqWAZ8ELpnHwQDT+799E/C6JAuT\nvAD4BeCBOZ7nbJnOer/C4CyJJEuAlwNfm9NZzr3D+vPrqD1zqANsyZHkN9vxv2TwyZU3AbuA/wbm\n9cffprnmPwBeClzdfpPeX/N407JprvmoMp01V9UDST7D4FLL08C1VTXlRyKPdNP8N/4g8LEkX2bw\nS+/7q2pe79Sa5BPAucCiJLuBDwDPh7n5+eU3pCVJnaP5spIkaYYMB0lSx3CQJHUMB0lSx3CQpHng\nYBvxTWr7U0m2tw35bk1yyNvkGA6SND9cB6yZZtsPMdh36ZXAHwF/fKhvZjhI0jww1UZ8SV7Wtiq/\nK8nnk7yiHVrJYBdigM8x9WaUz8lwkKT5azPwnqo6G3gfcHWrvwf41VZ+C4Mda196KAMftd+QlqSj\nWZIXMfj7LH83tKHk8e35fcCfJ/kN4DYGey49dSjjGw6SND89D/heVf3c5ANV9U3amUMLkV+rqu8d\n6uCSpHmmbUf+UJK3wf/92dCzWnlRkmd+vl8OfPRQxzccJGkeaBvxfRF4eZLdSTYA7wA2JLkH2MH/\n33g+F9iZ5KvAEmDTIb+fG+9JkibzzEGS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEmd\n/wW8L+YMtJCLMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x279597c5358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "\n",
    "incomes = np.random.normal(27000, 15000, 10000)\n",
    "incomes = np.append(incomes, [1000000000])\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.hist(incomes, 50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That's not very helpful to look at. One billionaire ended up squeezing everybody else into a single line in my histogram. Plus it skewed my mean income significantly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "126713.54327205669"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "incomes.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It's important to dig into what is causing your outliers, and understand where they are coming from. You also need to think about whether removing them is a valid thing to do, given the spirit of what it is you're trying to analyze. If I know I want to understand more about the incomes of \"typical Americans\", filtering out billionaires seems like a legitimate thing to do.\n",
    "\n",
    "Here's something a little more robust than filtering out billionaires - it filters out anything beyond two standard deviations of the median value in the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEcVJREFUeJzt3WuMnFd9x/Hvrw4EGiiJG9cysVObyqJypHLpKiUFqoAL\ncQHhVKoiI9EaSOUXTbn0IrDLi76yFEqFoGpTsLjULYHEDdBYQEmDaVRVrWIcCJDYMTG51Lbs2NxL\nK6Wy+ffFnJCJ8XpnvTve3ePvRxrNec7zPDvn7Gh/c/bMM2dSVUiS+vUzc90ASdJ4GfSS1DmDXpI6\nZ9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzl0w1w0AuPTSS2vlypVz3QxJWlDuueeeb1fVkqmO\nmxdBv3LlSvbs2TPXzZCkBSXJo6Mc59SNJHXOoJekzhn0ktQ5g16SOmfQS1LnRgr6JBcnuS3JA0n2\nJbkqyeIkdyZ5sN1fMnT8liQHkuxPcs34mi9JmsqoI/oPAF+oql8GXgDsAzYDu6pqNbCrbZNkDbAB\nuAJYB9yUZNFsN1ySNJopgz7Jc4DfAD4CUFX/V1XfB9YD29th24FrW3k9cEtVPV5VDwMHgCtnu+GS\npNGMMqJfBRwHPpbkq0k+nOQiYGlVHWnHHAWWtvJlwMGh8w+1OknSHBjlk7EXAC8G3lpVdyf5AG2a\n5glVVUmm9S3jSTYBmwAuv/zy6ZwqzWsrN3/utPWP3Pjac9wSaWCUoD8EHKqqu9v2bQyC/rEky6rq\nSJJlwLG2/zCwYuj85a3uKapqG7ANYGJiYlovEtI4GNDq1ZRTN1V1FDiY5Pmtai2wF9gJbGx1G4Hb\nW3knsCHJhUlWAauB3bPaaknSyEZd1OytwM1Jng48BLyZwYvEjiTXA48C1wFU1f1JdjB4MTgB3FBV\nJ2e95ZKkkYwU9FV1LzBxml1rJzl+K7B1Bu2SJM0SPxkrSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16S\nOmfQS1LnDHpJ6tyon4yVzluTrYEjLRSO6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnvLxSmmN+\nhaHGzRG9JHXOEb3OO34ASucbR/SS1DmDXpI6Z9BLUucMeknqnG/Gqku+4So9yRG9JHXOoJekzhn0\nktS5kYI+ySNJvpHk3iR7Wt3iJHcmebDdXzJ0/JYkB5LsT3LNuBovSZradEb0r6iqF1bVRNveDOyq\nqtXArrZNkjXABuAKYB1wU5JFs9hmSdI0zOSqm/XA1a28HbgLeFerv6WqHgceTnIAuBL4zxk8lrTg\neSWQ5sqoI/oCvpjkniSbWt3SqjrSykeBpa18GXBw6NxDrU6SNAdGHdG/rKoOJ/kF4M4kDwzvrKpK\nUtN54PaCsQng8ssvn86pkqRpGGlEX1WH2/0x4DMMpmIeS7IMoN0fa4cfBlYMnb681Z36M7dV1URV\nTSxZsuTseyBJOqMpgz7JRUme/UQZeDVwH7AT2NgO2wjc3so7gQ1JLkyyClgN7J7thkuSRjPK1M1S\n4DNJnjj+E1X1hSRfBnYkuR54FLgOoKruT7ID2AucAG6oqpNjab0kaUpTBn1VPQS84DT13wHWTnLO\nVmDrjFsnSZoxPxkrSZ1z9UotaF6bLk3NEb0kdc6gl6TOGfSS1Dnn6KV5arL3Hx658bXnuCVa6BzR\nS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM55eaUWBJc6kM6eI3pJ6pxBL0mdM+glqXMGvSR1zqCX\npM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdW7kRc2SLAL2AIer6nVJFgO3AiuBR4Dr\nqup77dgtwPXASeBtVXXHLLdbOm/5XbKarumM6N8O7Bva3gzsqqrVwK62TZI1wAbgCmAdcFN7kZAk\nzYGRgj7JcuC1wIeHqtcD21t5O3DtUP0tVfV4VT0MHACunJ3mSpKma9QR/fuBdwI/HqpbWlVHWvko\nsLSVLwMODh13qNVJkubAlEGf5HXAsaq6Z7JjqqqAms4DJ9mUZE+SPcePH5/OqZKkaRhlRP9S4PVJ\nHgFuAV6Z5OPAY0mWAbT7Y+34w8CKofOXt7qnqKptVTVRVRNLliyZQRckSWcy5VU3VbUF2AKQ5Grg\nT6vqjUneC2wEbmz3t7dTdgKfSPI+4LnAamD37DddPfIrA6XZN5PvjL0R2JHkeuBR4DqAqro/yQ5g\nL3ACuKGqTs64pZKkszKtoK+qu4C7Wvk7wNpJjtsKbJ1h2yRJs8BPxkpS5wx6SeqcQS9JnTPoJalz\nBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUudmsgSCpHnEb57SZBzRS1LnDHpJ6pxBL0mdM+glqXMGvSR1\nzqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnXOtG82JydZlkTT7HNFLUucMeknqnEEvSZ0z6CWp\nc1MGfZJnJNmd5GtJ9iW5sdUvTnJnkgfb/SVD52xJciDJ/iTXjLMDkqQzG2VE/zjwyqp6AfArwCuS\nvBzYDOyqqtXArrZNkjXABuAKYB1wU5JF42i8JGlqUwZ9DfyobT4NWAR8D1gPbG/124FrW3k9cEtV\nPV5VDwMHgCtntdWSpJGNNEefZFGSe4FjwF1VdR+wtKqOtEOOAktb+TLg4NDph1rdqT9zU5I9SfYc\nP378rDsgSTqzkYK+qk5W1QuB5cDLk7zilP0F1HQeuKq2VdVEVU0sWbJkOqdKkqZhWlfdVNX3gc8B\nE8BjSZYBtPtj7bDDwIqh05a3OknSHBjlqpslSS5u5WcCrwLuBXYCG9thG4HbW3knsCHJhUlWAauB\n3bPdcEnSaEZZ62YZsD3JzzB4Yfh4Vd2Z5CvAjiTXA48C1wFU1f1JdgB7gRPADVV1cjzN13znmjbS\n3Jsy6Kvq68CLTlP/HWDtJOdsBbbOuHWSpBlz9Uqpc2f6r+qRG197DluiueISCJLUOYNekjpn0EtS\n5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI65xIImhUuXibNX47oJalzBr0kdc6gl6TO\nOUcvnccme2/F5Yv74ohekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6N+Un\nY5OsAP4eWAoUsK2qPpBkMXArsBJ4BLiuqr7XztkCXA+cBN5WVXeMpfU651ylUlp4RhnRnwD+pKrW\nAC8BbkiyBtgM7Kqq1cCutk3btwG4AlgH3JRk0TgaL0ma2pRBX1VHquorrfzfwD7gMmA9sL0dth24\ntpXXA7dU1eNV9TBwALhythsuSRrNtObok6wEXgTcDSytqiNt11EGUzsweBE4OHTaoVYnSZoDIwd9\nkmcBnwLeUVU/HN5XVcVg/n5kSTYl2ZNkz/Hjx6dzqiRpGkYK+iRPYxDyN1fVp1v1Y0mWtf3LgGOt\n/jCwYuj05a3uKapqW1VNVNXEkiVLzrb9kqQpTBn0SQJ8BNhXVe8b2rUT2NjKG4Hbh+o3JLkwySpg\nNbB79posSZqOUb545KXA7wLfSHJvq/sz4EZgR5LrgUeB6wCq6v4kO4C9DK7YuaGqTs56yyVJI5ky\n6Kvq34FMsnvtJOdsBbbOoF2SpFniVwlK+il+xWBfXAJBkjpn0EtS5wx6SeqcQS9JnTPoJalzBr0k\ndc7LKyWNzMsuFyZH9JLUOUf0Oi2/SUrqhyN6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6\nZ9BLUucMeknqnEEvSZ0z6CWpcwa9JHXORc0kzZjLF89vBv15zlUqpf45dSNJnTPoJalzUwZ9ko8m\nOZbkvqG6xUnuTPJgu79kaN+WJAeS7E9yzbgaLkkazSgj+r8D1p1StxnYVVWrgV1tmyRrgA3AFe2c\nm5IsmrXWSpKmbcqgr6p/A757SvV6YHsrbweuHaq/paoer6qHgQPAlbPUVknSWTjbOfqlVXWklY8C\nS1v5MuDg0HGHWt1PSbIpyZ4ke44fP36WzZAkTWXGl1dWVSWpszhvG7ANYGJiYtrnS5r/vL5+fjjb\nEf1jSZYBtPtjrf4wsGLouOWtTpI0R8426HcCG1t5I3D7UP2GJBcmWQWsBnbPrImSpJmYcuomySeB\nq4FLkxwC/hy4EdiR5HrgUeA6gKq6P8kOYC9wArihqk6Oqe2SpBFMGfRV9YZJdq2d5PitwNaZNEqS\nNHtc6+Y84Zo20vnLoJd0znk1zrnlWjeS1DmDXpI659RNZ5yLl3QqR/SS1DmDXpI6Z9BLUucMeknq\nnEEvSZ3zqpsFyCtrJE2HI3pJ6pxBL0mdc+pG0rzhGjjj4YhekjrniH4e801XSbPBEb0kdc6gl6TO\nOXUjad7zTdqZMejnAefiJY2TQS9pwXKkPxqDXlJ3fAF4KoP+HHKKRtJc8KobSeqcI/oxcOQuaT4Z\n24g+ybok+5McSLJ5XI8jSTqzsYzokywC/gZ4FXAI+HKSnVW1dxyPJ0mjOJv/tnt4A3dcUzdXAgeq\n6iGAJLcA64EFGfROxUjnr+n+/c/HF4ZxBf1lwMGh7UPAr43psQxiSfPeXF7yOWdvxibZBGxqmz9K\nsn+u2jJNlwLfnutGjIl9W3h67Rcs0L7lPVMe8pR+jXD8mfziKAeNK+gPAyuGtpe3up+oqm3AtjE9\n/tgk2VNVE3PdjnGwbwtPr/2Cfvs2F/0a11U3XwZWJ1mV5OnABmDnmB5LknQGYxnRV9WJJH8I3AEs\nAj5aVfeP47EkSWc2tjn6qvo88Plx/fw5tOCmm6bBvi08vfYL+u3bOe9XqupcP6Yk6RxyrRtJ6tx5\nH/RJ3pvkgSRfT/KZJBcP7dvSlnDYn+SaofpfTfKNtu+vkqTVX5jk1lZ/d5KVQ+dsTPJgu208l32c\nykJYriLJiiT/mmRvkvuTvL3VL05yZ/u93pnkkqFzZu35O0d9XJTkq0k+21Pfklyc5Lb2d7YvyVU9\n9K21c2+S+5J8Mskz5m2/quq8vgGvBi5o5fcA72nlNcDXgAuBVcC3gEVt327gJUCAfwZ+q9X/AfDB\nVt4A3NrKi4GH2v0lrXzJXPe9tW1R69vzgKe3Pq+Z63adpp3LgBe38rOBb7bn6C+Aza1+8ziev3PY\nxz8GPgF8tm130TdgO/D7rfx04OKF3jdgJfAw8My2vQN403zt15z/Ac+nG/DbwM2tvAXYMrTvDuAq\nBoHzwFD9G4APDR/Tyhcw+FBEho9p+z4EvGGu+9vachVwx9D2U/o9X2/A7QzWUtoPLGt1y4D9s/38\nnaP+LAd2Aa/kyaBf8H0DntMCMafUL+i+MRi0fbPdXwB8lsGgcV7267yfujnFWxi8osLpl3G4rN0O\nnab+KedU1QngB8DPn+FnzQfzuW2n1f6FfRFwN7C0qo60XUeBpa08m8/fufB+4J3Aj4fqeujbKuA4\n8LE2LfXhJBexwPtWVd8F/hL4L+AI8IOq+hfmab/Oi6BP8sU2j3bqbf3QMe8GTgA3z11LNZUkzwI+\nBbyjqn44vK8GQ58FdxlZktcBx6rqnsmOWah9YzASfTHwt1X1IuB/GExp/MRC7FuSXwL+iMEL2XOB\ni5K8cfiY+dSv8+KLR6rqN8+0P8mbgNcBa9uTA5Mv43C4lU+tHz7nUJILGPzb+p1Wf/Up59w1/Z6M\nxZTLVcwXSZ7GIORvrqpPt+rHkiyrqiNJlgHHWv1sPn/j9lLg9UleAzwD+LkkH6ePvh0CDlXV3W37\nNgZBv9D7NgH8R1UdB0jyaeDXmaf9Oi9G9GeSZB2Df5lfX1X/O7RrJ7ChvfO9ClgN7G7/lv0wyUva\nu+O/x2C++Ilznrii5neAL7UXjjuAVye5pL0L/+pWNx8siOUq2u/6I8C+qnrf0K7h3/lGnvpczNbz\nN1ZVtaWqllfVSga//y9V1Rs76dtR4GCS57eqtQyWK1/ofdsPvCTJz7b2rAX2zdt+jfMNi4VwAw4w\nmAe7t90+OLTv3QzeHd9Peye81U8A97V9f82THzx7BvCP7WfuBp43dM5bWv0B4M1z3e9TfgevYfDG\n0reAd891eyZp48sY/Bv89aHn6jUM5ix3AQ8CXwQWj+P5O4f9vJon34ztom/AC4E97bn7JwZXni34\nvgHvYvCidR/wDwyuqJmX/fKTsZLUufN+6kaSemfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z\n9JLUuf8HxDzT6rZ6sbQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2795b8394e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def reject_outliers(data):\n",
    "    u = np.median(data)\n",
    "    s = np.std(data)\n",
    "    filtered = [e for e in data if (u - 2 * s < e < u + 2 * s)]\n",
    "    return filtered\n",
    "\n",
    "filtered = reject_outliers(incomes)\n",
    "\n",
    "plt.hist(filtered, 50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That looks better. And, our mean is more, well, meangingful now as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26726.214626383888"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(filtered)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Instead of a single outlier, add several randomly-generated outliers to the data. Experiment with different values of the multiple of the standard deviation to identify outliers, and see what effect it has on the final results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}