04 - Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering\n",
    "\n",
    "In contrast to *supervised* machine learning, *unsupervised* learning is used when there is no \"ground truth\" from which to train and validate label predictions. The most common form of unsupervised learning is *clustering*, which is simllar conceptually to *classification*, except that the the training data does not include known values for the class label to be predicted. Clustering works by separating the training cases based on similarities that can be determined from their feature values. Think of it this way; the numeric features of a given entity can be though of as vector coordinates that define the entity's position in n-dimensional space. What a clustering model seeks to do is to identify groups, or *clusters*, of entities that are close to one another while being separated from other clusters.\n",
    "\n",
    "For example, let's take a look at the Palmer Islands penguin dataset, which contains measurements of penguins.\n",
    "\n",
    "Let's start by examining a dataset that contains observations of multiple classes. We'll use a dataset that contains observations of three different species of penguin.\n",
    "\n",
    "> **Citation**: The penguins dataset used in the this exercise is a subset of data collected and made available by [Dr. Kristen\n",
    "Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php)\n",
    "and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a\n",
    "member of the [Long Term Ecological Research\n",
    "Network](https://lternet.edu/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# load the training dataset (dropping rows with nulls)\n",
    "penguins = pd.read_csv('data/penguins.csv').dropna()\n",
    "\n",
    "# Display a random sample of 10 observations (just the features)\n",
    "penguin_features = penguins[penguins.columns[0:4]]\n",
    "penguin_features.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the dataset contains four data points (or *features*) for each instance (*observation*) of an penguin. So you could interpret these as coordinates that describe each instance's location in four-dimensional space.\n",
    "\n",
    "Now, of course four dimensional space is difficult to visualise in a three-dimensional world, or on a two-dimensional plot; so we'll take advantage of a mathematical technique called *Principal Component Analysis* (PCA) to analyze the relationships between the features and summarize each observation as coordinates for two principal components - in other words, we'll translate the four dimensional feature values into two-dimensional coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Normalize the numeric features so they're on the same scale\n",
    "scaled_features = MinMaxScaler().fit_transform(penguin_features[penguins.columns[0:4]])\n",
    "\n",
    "# Get two principal components\n",
    "pca = PCA(n_components=2).fit(scaled_features)\n",
    "penguins_2d = pca.transform(scaled_features)\n",
    "penguins_2d[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the data points translated to two dimensions, we can visualize them in a plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "plt.scatter(penguins_2d[:,0],penguins_2d[:,1])\n",
    "plt.xlabel('Dimension 1')\n",
    "plt.ylabel('Dimension 2')\n",
    "plt.title('Penguin Data')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you can see at least two, arguably three, reasonably distinct groups of data points; but here lies one of the fundamental problems with clustering - without known class labels, how do you know how many clusters to separate your data into?\n",
    "\n",
    "One way we can try to find out is to use a data sample to create a series of clustering models with an incrementing number of clusters, and measure how tightly the data points are grouped within each cluster. A metric often used to measure this tightness is the *within cluster sum of squares* (WCSS), with lower values meaning that the data points are closer. You can then plot the WCSS for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing the libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.cluster import KMeans\n",
    "%matplotlib inline\n",
    "\n",
    "# Create 10 models with 1 to 10 clusters\n",
    "wcss = []\n",
    "for i in range(1, 11):\n",
    "    kmeans = KMeans(n_clusters = i)\n",
    "    # Fit the Iris data points\n",
    "    kmeans.fit(penguin_features.values)\n",
    "    # Get the WCSS (inertia) value\n",
    "    wcss.append(kmeans.inertia_)\n",
    "    \n",
    "#Plot the WCSS values onto a line graph\n",
    "plt.plot(range(1, 11), wcss)\n",
    "plt.title('WCSS by Clusters')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('WCSS')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot shows a large reduction in WCSS (so greater *tightness*) as the number of clusters increases from one to two, and a further noticable reduction from two to three clusters. After that, the reduction is less pronounced, resulting in an \"elbow\" in the chart at around three clusters. This is a good indication that there are two to three reasonably well separated clusters of data points.\n",
    "\n",
    "## K-Means Clustering\n",
    "\n",
    "The algorithm we used to create our test clusters is *K-Means*. This is a commonly used clustering algorithm that separates a dataset into *K* clusters of equal variance. The number of clusters, *K*, is user defined. The basic algorithm has the following steps:\n",
    "\n",
    "1. A set of K centroids are randomly chosen.\n",
    "2. Clusters are formed by assigning the data points to their closest centroid.\n",
    "3. The means of each cluster is computed and the centroid is moved to the mean.\n",
    "4. Steps 2 and 3 are repeated until a stopping criteria is met. Typically, the algorithm terminates when each new iteration results in negligable movement of centroids and the clusters become static.\n",
    "5. When the clusters stop changing, the algorithm has *converged*, defining the locations of the clusters - note that the random starting point for the centroids means that re-running the algorithm could result in slightly different clusters, so training usually involves multiple iterations, reinitializing the centroids each time, and the model with the best WCSS is selected.\n",
    "\n",
    "Let's try using K-Means on our penguin data with a K value of 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# Create a model based on 3 centroids\n",
    "model = KMeans(n_clusters=3, init='k-means++', n_init=100, max_iter=1000)\n",
    "# Fit to the iris data and predict the cluster assignments for each data point\n",
    "km_clusters = model.fit_predict(penguin_features.values)\n",
    "# View the cluster assignments\n",
    "km_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see those cluster assignments with the two-dimensional data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_clusters(samples, clusters):\n",
    "    col_dic = {0:'blue',1:'green',2:'orange'}\n",
    "    mrk_dic = {0:'*',1:'x',2:'+'}\n",
    "    colors = [col_dic[x] for x in clusters]\n",
    "    markers = [mrk_dic[x] for x in clusters]\n",
    "    for sample in range(len(clusters)):\n",
    "        plt.scatter(samples[sample][0], samples[sample][1], color = colors[sample], marker=markers[sample], s=100)\n",
    "    plt.xlabel('Dimension 1')\n",
    "    plt.ylabel('Dimension 2')\n",
    "    plt.title('Assignments')\n",
    "    plt.show()\n",
    "\n",
    "plot_clusters(penguins_2d, km_clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what's the practical use of clustering? In some cases, you may have data that you need to group into distict clusters without knowing how many clusters there are or what they indicate. For example a marketing organization might want to separate customers into distinct segments, and then investigate how those segments exhibit different purchasing behaviors.\n",
    "\n",
    "Sometimes, clustering is used as an initial step towards creating a classification model. You start by identifying distinct groups of data points, and then assign class labels to those clusters. You can then use this labelled data to train a classification model.\n",
    "\n",
    "In the case of the penguin data, the different species of penguin are already known, so we can use the class labels identifying the species to plot the class assignments and compare them to the clusters identified by our unsupervised algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguin_species = penguins[penguins.columns[4]]\n",
    "plot_clusters(penguins_2d, penguin_species.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There may be some differences in the cluster assignments and class labels, but the K-Means model should have done a reasonable job of clustering the penguin observations so that birds of the same species are generally in the same cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical Clustering\n",
    "\n",
    "Hierarchical clustering methods make fewer distributional assumptions when compared to K-means methods. However, K-means methods are generally more scalable, sometimes very much so.\n",
    "\n",
    "Hierarchical clustering creates clusters by either a *divisive* method or *agglomerative* method. The divisive method is a \"top down\" approach starting with the entire dataset and then finding partitions in a stepwise manner. Agglomerative clustering is a \"bottom up** approach. In this lab you will work with agglomerative clustering which roughly works as follows:\n",
    "\n",
    "1. The linkage distances between each of the data points is computed.\n",
    "2. Points are clustered pairwise with their nearest neighbor.\n",
    "3. Linkage distances between the clusters are computed.\n",
    "4. Clusters are combined pairwise into larger clusters.\n",
    "5. Steps 3 and 4 are repeated until all data points are in a single cluster.\n",
    "\n",
    "The linkage function can be computed in a number of ways:\n",
    "- Ward linkage measures the increase in variance for the clusters being linked,\n",
    "- Average linkage uses the mean pairwise distance between the members of the two clusters,\n",
    "- Complete or Maximal linkage uses the maximum distance between the members of the two clusters.\n",
    "\n",
    "Several different distance metrics are used to compute linkage functions:\n",
    "- Euclidian or l2 distance is the most widely used. This metric is only choice for the Ward linkage method.\n",
    "- Manhattan or l1 distance is robust to outliers and has other interesting properties.\n",
    "- Cosine similarity, is the dot product between the location vectors divided by the magnitudes of the vectors. Notice that this metric is a measure of similarity, whereas the other two metrics are measures of difference. Similarity can be quite useful when working with data such as images or text documents.\n",
    "\n",
    "### Agglomerative Clustering\n",
    "\n",
    "Let's see an example of clustering the penguin data using an agglomerative clustering algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "agg_model = AgglomerativeClustering(n_clusters=3)\n",
    "agg_clusters = agg_model.fit_predict(penguin_features.values)\n",
    "agg_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what do the agglomerative cluster assignments look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "def plot_clusters(samples, clusters):\n",
    "    col_dic = {0:'blue',1:'green',2:'orange'}\n",
    "    mrk_dic = {0:'*',1:'x',2:'+'}\n",
    "    colors = [col_dic[x] for x in clusters]\n",
    "    markers = [mrk_dic[x] for x in clusters]\n",
    "    for sample in range(len(clusters)):\n",
    "        plt.scatter(samples[sample][0], samples[sample][1], color = colors[sample], marker=markers[sample], s=100)\n",
    "    plt.xlabel('Dimension 1')\n",
    "    plt.ylabel('Dimension 2')\n",
    "    plt.title('Assignments')\n",
    "    plt.show()\n",
    "\n",
    "plot_clusters(penguins_2d, agg_clusters)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, you've explored clustering; an unsupervised form of machine learning.\n",
    "\n",
    "To learn more about clustering with scikit-learn, see the [scikit-learn documentation](https://scikit-learn.org/stable/modules/clustering.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}