A01. Statistics - Basic Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A01. Statistics - Basic Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Data Science from Scratch (pdf)](http://math.ecnu.edu.cn/~lfzhou/seminar/[Joel_Grus]_Data_Science_from_Scratch_First_Princ.pdf#page=96)<br/>\n",
    "[How to Read Mathematical Formulae (video)](https://www.youtube.com/watch?v=-mu3TYZ_udM)<br/>\n",
    "[Latex Editor](https://www.latex4technics.com/)<br/>\n",
    "[List of Mathematical Symbols](https://www.rapidtables.com/math/symbols/Basic_Math_Symbols.html)<br/>\n",
    "[List of Latex Symbols](https://oeis.org/wiki/List_of_LaTeX_mathematical_symbols)<br/>\n",
    "[Standard Deviation](https://simple.wikipedia.org/wiki/Standard_deviation)<br/>\n",
    "[Variance and Standard Deviation](https://www.sciencebuddies.org/science-fair-projects/science-fair/variance-and-standard-deviation)<br/>\n",
    "[Bessel's Correction](https://en.wikipedia.org/wiki/Bessel%27s_correction)<br/>\n",
    "[Pearson Correlation Coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)<br/>\n",
    "[Simpson's Paradox](http://www.statisticshowto.com/what-is-simpsons-paradox/)<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pandas_profiling as pdpf\n",
    "import missingno\n",
    "from scipy import stats\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Text & Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics is the practice or science of collecting and analysing numerical data in large quantities, especially for the purpose of inferring proportions in a whole from those in a representative sample. Essentially it refers to gaining an understanding of something from data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Some random data\n",
    "np_data = np.random.randint(low=0,high=20,size=20)\n",
    "np_data2 = np.random.randint(low=0,high=20,size=20)\n",
    "pd_data = pd.DataFrame(np_data.tolist(), columns = ['data'])\n",
    "pd_data2 = pd.DataFrame(np_data2.tolist(), columns = ['data2'])\n",
    "pd_data2 = pd.concat([pd_data, pd_data2], axis=1)\n",
    "\n",
    "# Basic Statistics for Numpy\n",
    "np_data.min()                            # Minimum value\n",
    "np_data.max()                            # Maximum value\n",
    "np_range = np_data.max() - np_data.min() # Range\n",
    "np.median(np_data)                       # Median value\n",
    "stats.mode(np_data)                      # Mode value\n",
    "np_data.sum()                            # Sum\n",
    "np_data.cumsum()                         # Cumulative sum\n",
    "np.percentile(np_data,10)                # Percentiles\n",
    "\n",
    "# Basic Statistics for Pandas\n",
    "pd_data['data'].min()                                     # Minimum value\n",
    "pd_data['data'].max()                                     # Maximum value\n",
    "pd_range = pd_data['data'].max() - pd_data['data'].min()  # Range\n",
    "pd_data['data'].median()                                  # Median value\n",
    "pd_data['data'].mode()                                    # Mode value\n",
    "pd_data['data'].sum()                                     # Sum \n",
    "pd_data['data'].cumsum()                                  # Cumulative sum\n",
    "pd_data['data'].quantile(0.1)                             # Percentiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Stats Exploratoin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Automated stats using Pandas\n",
    "df = pd.DataFrame.from_csv(\"https://raw.githubusercontent.com/nickhould/craft-beers-dataset/master/data/processed/beers.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.info()                 # Basic Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.dtypes                 # Data Types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.describe()             # Basic stats for a dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pdpf.ProfileReport(df)    # Creating a profile report from a pandas dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.hist(bins=50, figsize=(20,15))    # Plotting the data\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "missingno.matrix(df)                  # Visualising missing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statistical Functions & Formulae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Moments**\n",
    "\n",
    "In statistics, moments are quantities that are related to the shape of a set of numbers. “Shape of a set of numbers,” means “what a histogram based on the numbers looks like” — how spread out it is, how symmetric it is, and more. They are as follows:\n",
    "\n",
    "1. Mean\n",
    "2. Variance\n",
    "3. Skewness\n",
    "4. Kurtosis\n",
    "\n",
    "There is more information [here](http://www.statisticshowto.com/what-is-a-moment/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Arithmetic Mean (1st Moment)**<br/>\n",
    "\n",
    "The mean is the average of a set of numerical values, as calculated by adding them together and dividing by the number of terms in the set.\n",
    "\n",
    "<span style=\"color:#888888\">${\\displaystyle A={\\frac {1}{n}}\\sum _{i=1}^{n}a_{i}={\\frac {a_{1}+a_{2}+\\cdots +a_{n}}{n}}}$</span>\n",
    "\n",
    "Where:  \n",
    "<span style=\"color:#888888\">$A$ = Mean  \n",
    "$i$ = First observation in the data  \n",
    "$n$ = Number of observations in the dataset   \n",
    "$\\sum$ = Sum   \n",
    "$a$ = Individual value in the dataset  \n",
    "</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.mean(np_data)\n",
    "pd_data['data'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Variance (2nd Moment)**  \n",
    "Variance measures how far a data set is spread out. The technical definition is “The average of the squared differences from the mean,” but all it really does is to give you a very general idea of the spread of your data. It is expressed in square units whereas **Standard Deviation** is expressed in the same units at the mean. Note that there are slightly different ways of calculation variance depending upon whether you are working with a sample or a full population. See [Bessel's Correction](https://en.wikipedia.org/wiki/Bessel%27s_correction) for more details. The example below uses Bessel's correction.  \n",
    "\n",
    "<span style=\"color:#888888\">${\\displaystyle s^{2} = {\\frac{1}{N-1} \\sum_{i=1}^N (x_i - \\overline{x})^2}}$</span>\n",
    "\n",
    "Where:  \n",
    "<span style=\"color:#888888\">$s^{2}$ = Variance  \n",
    "$N$ = The population size  \n",
    "$x_i$ = ?  \n",
    "$\\overline{x}$ = The mean (estimated)\n",
    "</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.var(np_data, ddof=1)           # Not N-1 by default!\n",
    "pd_data['data'].var(ddof=1)       # N-1 by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Standard Deviation**<br/>\n",
    "The standard deviation is a measurement statisticians use for the amount of variability (or spread) among the numbers in a data set. As the term implies, a standard deviation is a standard (or typical) amount of deviation (or distance) from the average (or mean, as statisticians like to call it). So the standard deviation, in very rough terms, is the average distance from the mean. Note that there are slightly different ways of calculating standard deviation depending upon whether you are working with a sample or a full population. See [Bessel's Correction](https://en.wikipedia.org/wiki/Bessel%27s_correction) for more details.\n",
    "\n",
    "The standard deviation is also used to describe where most of the data should fall, in a relative sense, compared to the average. For example, **if your data have the form of a bell-shaped curve (also known as a normal distribution)**:\n",
    "\n",
    "* 68% of data falls within the one standard deviation of the mean.  \n",
    "* 95% fall within two standard deviations.  \n",
    "* 99.7% fall within three standard deviations \n",
    "\n",
    "A standard deviation of 3 means that a majority of observations (about 68%, assuming a normal distribution) are either 3 more or 3 less than the average (one standard deviation).<br/>\n",
    "Most observations (95%) will have a value either 6 more, or 6 less than the average (two standard deviations).<br/>\n",
    "Almost all observations (99.7%) will have a value either 9 more, or 9 less than the average (three standard deviations).<br/>\n",
    "\n",
    "This result is called the empirical rule, or the 68–95–99.7% rule. A low standard deviation means that most of the numbers are very close to the average. A high standard deviation means that the numbers are spread out. \n",
    "\n",
    "<span style=\"color:#888888\">${\\displaystyle s = \\sqrt{\\frac{1}{N-1} \\sum_{i=1}^N (x_i - \\overline{x})^2}}$</span>\n",
    "\n",
    "Where:\n",
    "\n",
    "<span style=\"color:#888888\">$s$ = Standard Deviation  \n",
    "$N$ = The population size  \n",
    "$x_i$ = = $x$ variable indexed with ${i}$  \n",
    "$\\overline{x}$ = The mean (estimated)\n",
    "</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.std(np_data, ddof=1)     # Not N-1 by default!\n",
    "pd_data['data'].std()       # N-1 by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Pearson Correlation**<br/>\n",
    "\n",
    "The Pearson Correlation Coefficient to measure the strength and direction of the linear relationship between two numerical variables X and Y. As with the examples above this is calculated for a sample dataset. The formula for a full population is slightly different. See [https://en.wikipedia.org/wiki/Pearson_correlation_coefficient](the Pearson Correlation Coefficient Wikipedia page) for more details. Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations.\n",
    "\n",
    "Values returned will range between -1 and 1, with positive numbers indicating a positive correlation and negative numbers indicating a negative correlation.\n",
    "\n",
    "Note that you should always take account of [Simpson's Paradox](http://www.statisticshowto.com/what-is-simpsons-paradox/) when correlation. This is a phenomenon in probability and statistics, in which a trend appears in several different groups of data but disappears or reverses when these groups are combined. It is sometimes given the descriptive title **reversal paradox** or **amalgamation paradox**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:#888888\">${\\displaystyle r =\\frac{\\sum _{i=1}^n (x_i - \\bar{x})(y_i - \\bar{y})}{\\sqrt{\\sum ^n _{i=1}(x_i - \\bar{x})^2} \\sqrt{\\sum ^n _{i=1}(y_i - \\bar{y})^2}}}$</span>\n",
    "\n",
    "Where:  \n",
    "<span style=\"color:#888888\">$n$ = Sample size,  \n",
    "$x_i$ = $x$ variable indexed with ${i}$   \n",
    "$y_i$ = $y$ variable indexed with ${i}$  \n",
    "$\\overline{x}$ = The mean of $x$ (estimated) \n",
    "$\\overline{y}$ = The mean of $y$ (estimated) \n",
    "</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.corrcoef(np_data,np_data2)              # Correlation\n",
    "pd_data2.corr()                            # Correlation\n",
    "chart = sns.pairplot(pd_data2,kind=\"reg\")  # Seaborn pairplot"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}