Week 10/02 - Assembly Methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assembly methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine different ML methods with different settings and apply one method to achieve consensus.\n",
    "\n",
    "Diversity is a very good option.\n",
    "\n",
    "Esamble's methods have stood out for winning many ML competitions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bagging (Bootstrap Aggregation)\n",
    "\n",
    "What if instead of relying on the opinion of a single \"*expert*\" we consult the opinion of several experts in parallel and try to reach a consensus?\n",
    "\n",
    "The final answer is the combination (for example by voting) between the individual answers\n",
    "\n",
    "#### Algorithms\n",
    "\n",
    "1. Random Forest\n",
    "2. Voting Classifiers / Regressors\n",
    "3. Can be applied on any family of ML models\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/850/1*_pfQ7Xf-BAwfQXtaBbNTEg.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boosting \n",
    "\n",
    "We will ask an expert for his opinion on a problem, we measure his possible error, and then using that calculated error we ask another expert for his judgment on the same problem\n",
    "\n",
    "\n",
    "Boosting is a sequential method, it seeks to gradually strengthen a learning model using always the residual error of the previous stages, the final result is also achieved by consensus among all models.\n",
    "\n",
    "\n",
    "#### Algorithms\n",
    "\n",
    "1. AdaBoost\n",
    "2. Gradient Tree Boosting\n",
    "3. XGBoost\n",
    "\n",
    "\n",
    "<img src=\"https://www.cs.us.es/~fsancho/images/2018-12/boosting.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "from sklearn import datasets\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>125</td>\n",
       "      <td>212</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>168</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>140</td>\n",
       "      <td>203</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>155</td>\n",
       "      <td>1</td>\n",
       "      <td>3.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>145</td>\n",
       "      <td>174</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>125</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>61</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>148</td>\n",
       "      <td>203</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>161</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>138</td>\n",
       "      <td>294</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>106</td>\n",
       "      <td>0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  slope  \\\n",
       "0   52    1   0       125   212    0        1      168      0      1.0      2   \n",
       "1   53    1   0       140   203    1        0      155      1      3.1      0   \n",
       "2   70    1   0       145   174    0        1      125      1      2.6      0   \n",
       "3   61    1   0       148   203    0        1      161      0      0.0      2   \n",
       "4   62    0   0       138   294    1        1      106      0      1.9      1   \n",
       "\n",
       "   ca  thal  target  \n",
       "0   2     3       0  \n",
       "1   0     3       0  \n",
       "2   0     3       0  \n",
       "3   1     3       0  \n",
       "4   3     2       0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('../Datasets/Week9/heart.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1025.000000\n",
       "mean        0.513171\n",
       "std         0.500070\n",
       "min         0.000000\n",
       "25%         0.000000\n",
       "50%         1.000000\n",
       "75%         1.000000\n",
       "max         1.000000\n",
       "Name: target, dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['target'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df.drop(['target'], 1)\n",
    "y = df['target']\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.35)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.7158774373259053\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier().fit(x_train, y_train)\n",
    "knn_pred = knn.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, knn_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.7576601671309192\n"
     ]
    }
   ],
   "source": [
    "bag = BaggingClassifier(base_estimator=KNeighborsClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, bag_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.9888579387186629\n"
     ]
    }
   ],
   "source": [
    "dtc = DecisionTreeClassifier().fit(x_train,y_train)\n",
    "dtc_pred = dtc.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, dtc_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.9888579387186629\n"
     ]
    }
   ],
   "source": [
    "bag = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, bag_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.924791086350975\n"
     ]
    }
   ],
   "source": [
    "booster = GradientBoostingClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, boost_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy: 0.9025069637883009\n"
     ]
    }
   ],
   "source": [
    "booster = AdaBoostClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy:', accuracy_score(y_test, boost_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset wines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113.0</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
       "0    14.23        1.71  2.43               15.6      127.0           2.80   \n",
       "1    13.20        1.78  2.14               11.2      100.0           2.65   \n",
       "2    13.16        2.36  2.67               18.6      101.0           2.80   \n",
       "3    14.37        1.95  2.50               16.8      113.0           3.85   \n",
       "4    13.24        2.59  2.87               21.0      118.0           2.80   \n",
       "\n",
       "   flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity   hue  \\\n",
       "0        3.06                  0.28             2.29             5.64  1.04   \n",
       "1        2.76                  0.26             1.28             4.38  1.05   \n",
       "2        3.24                  0.30             2.81             5.68  1.03   \n",
       "3        3.49                  0.24             2.18             7.80  0.86   \n",
       "4        2.69                  0.39             1.82             4.32  1.04   \n",
       "\n",
       "   od280/od315_of_diluted_wines  proline  \n",
       "0                          3.92   1065.0  \n",
       "1                          3.40   1050.0  \n",
       "2                          3.17   1185.0  \n",
       "3                          3.45   1480.0  \n",
       "4                          2.93    735.0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wines = datasets.load_wine()\n",
    "df = pd.DataFrame(wines.data, columns=wines.feature_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df \n",
    "y = wines.target\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy KNN: 0.6111111111111112\n",
      "Acuraccy Bagging KNN: 0.5833333333333334\n",
      "Acuraccy DT: 0.9166666666666666\n",
      "Acuraccy Bagging DT: 0.9722222222222222\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier().fit(x_train, y_train)\n",
    "knn_pred = knn.predict(x_test)\n",
    "print('Acuraccy KNN:', accuracy_score(y_test, knn_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=KNeighborsClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging KNN:', accuracy_score(y_test, bag_pred))\n",
    "\n",
    "dtc = DecisionTreeClassifier().fit(x_train,y_train)\n",
    "dtc_pred = dtc.predict(x_test)\n",
    "print('Acuraccy DT:', accuracy_score(y_test, dtc_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging DT:', accuracy_score(y_test, bag_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy GBC: 0.9444444444444444\n",
      "Acuraccy Ada: 0.9444444444444444\n"
     ]
    }
   ],
   "source": [
    "booster = GradientBoostingClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy GBC:', accuracy_score(y_test, boost_pred))\n",
    "\n",
    "booster = AdaBoostClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy Ada:', accuracy_score(y_test, boost_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Cancer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38          122.80     1001.0          0.11840   \n",
       "1        20.57         17.77          132.90     1326.0          0.08474   \n",
       "2        19.69         21.25          130.00     1203.0          0.10960   \n",
       "3        11.42         20.38           77.58      386.1          0.14250   \n",
       "4        20.29         14.34          135.10     1297.0          0.10030   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760          0.3001              0.14710         0.2419   \n",
       "1           0.07864          0.0869              0.07017         0.1812   \n",
       "2           0.15990          0.1974              0.12790         0.2069   \n",
       "3           0.28390          0.2414              0.10520         0.2597   \n",
       "4           0.13280          0.1980              0.10430         0.1809   \n",
       "\n",
       "   mean fractal dimension  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0                 0.07871  ...         25.38          17.33           184.60   \n",
       "1                 0.05667  ...         24.99          23.41           158.80   \n",
       "2                 0.05999  ...         23.57          25.53           152.50   \n",
       "3                 0.09744  ...         14.91          26.50            98.87   \n",
       "4                 0.05883  ...         22.54          16.67           152.20   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer = datasets.load_breast_cancer()\n",
    "df = pd.DataFrame(cancer.data, columns=cancer.feature_names)\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df \n",
    "y = cancer.target\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy KNN: 0.956140350877193\n",
      "Acuraccy Bagging KNN: 0.9649122807017544\n",
      "Acuraccy DT: 0.9122807017543859\n",
      "Acuraccy Bagging DT: 0.9912280701754386\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier().fit(x_train, y_train)\n",
    "knn_pred = knn.predict(x_test)\n",
    "print('Acuraccy KNN:', accuracy_score(y_test, knn_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=KNeighborsClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging KNN:', accuracy_score(y_test, bag_pred))\n",
    "\n",
    "dtc = DecisionTreeClassifier().fit(x_train,y_train)\n",
    "dtc_pred = dtc.predict(x_test)\n",
    "print('Acuraccy DT:', accuracy_score(y_test, dtc_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging DT:', accuracy_score(y_test, bag_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy GBC: 0.9736842105263158\n",
      "Acuraccy Ada: 0.9824561403508771\n"
     ]
    }
   ],
   "source": [
    "booster = GradientBoostingClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy GBC:', accuracy_score(y_test, boost_pred))\n",
    "\n",
    "booster = AdaBoostClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy Ada:', accuracy_score(y_test, boost_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
       "0                5.1               3.5                1.4               0.2\n",
       "1                4.9               3.0                1.4               0.2\n",
       "2                4.7               3.2                1.3               0.2\n",
       "3                4.6               3.1                1.5               0.2\n",
       "4                5.0               3.6                1.4               0.2"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris = datasets.load_iris()\n",
    "df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df \n",
    "y = iris.target\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy KNN: 1.0\n",
      "Acuraccy Bagging KNN: 1.0\n",
      "Acuraccy DT: 1.0\n",
      "Acuraccy Bagging DT: 1.0\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier().fit(x_train, y_train)\n",
    "knn_pred = knn.predict(x_test)\n",
    "print('Acuraccy KNN:', accuracy_score(y_test, knn_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=KNeighborsClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging KNN:', accuracy_score(y_test, bag_pred))\n",
    "\n",
    "dtc = DecisionTreeClassifier().fit(x_train,y_train)\n",
    "dtc_pred = dtc.predict(x_test)\n",
    "print('Acuraccy DT:', accuracy_score(y_test, dtc_pred))\n",
    "\n",
    "bag = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50).fit(x_train, y_train)\n",
    "bag_pred = bag.predict(x_test)\n",
    "print('Acuraccy Bagging DT:', accuracy_score(y_test, bag_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acuraccy GBC: 1.0\n",
      "Acuraccy Ada: 0.9666666666666667\n"
     ]
    }
   ],
   "source": [
    "booster = GradientBoostingClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy GBC:', accuracy_score(y_test, boost_pred))\n",
    "\n",
    "booster = AdaBoostClassifier(n_estimators=50).fit(x_train, y_train)\n",
    "boost_pred = booster.predict(x_test)\n",
    "print('Acuraccy Ada:', accuracy_score(y_test, boost_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}