feature-engineering.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install category_encoders -Uq\n",
    "!pip install spacy -q\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "import category_encoders as ce\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import skewnorm\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import spacy\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "from create_dataset import ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "\n",
    "## References\n",
    "\n",
    "Gabby Shklovsky - Random Forests Best Practices for the Business World - PyData NYC 2017 [youtube](https://www.youtube.com/watch?v=E7VLE-U07x0) - [slides](https://www.youtube.com/redirect?q=https%3A%2F%2Fwww.slideshare.net%2FPyData%2Frandom-forests-best-practices-for-the-business-world&redir_token=HgV_RBYb_uD_jYV6nYygn8RpyKR8MTU2OTkwODE2N0AxNTY5ODIxNzY3&v=E7VLE-U07x0&event=video_description)\n",
    "\n",
    "Art of Feature Engineering for Data Science - Nabeel Sarwar - [youtube](https://youtu.be/leTyvBPhYzw)\n",
    "\n",
    "Feature Engineering with H2O - Dmitry Larko, Senior Data Scientist, H2O.ai - [youtube](https://youtu.be/irkV4sYExX4)\n",
    "\n",
    "[Building Machine Learning Powered Applications: Going from Idea to Product](https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X)\n",
    "\n",
    "[Datacamp categorical data tutorial](https://www.datacamp.com/community/tutorials/categorical-data)\n",
    "\n",
    "[Smarter Ways to Encode Categorical Data for Machine Learning](https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159)\n",
    "\n",
    "[Vincent Warmerdam: Winning with Simple, even Linear, Models | PyData London 2018](https://www.youtube.com/watch?v=68ABAU_V8qI)\n",
    "\n",
    "[Why giving your algorithm ALL THE FEATURES does not always work - Thomas Huijskens](https://www.youtube.com/watch?v=JsArBz46_3s)\n",
    "\n",
    "[Feature Engineering - Elite Data Science](https://elitedatascience.com/feature-engineering)\n",
    "\n",
    "[Feature Engineering for ML](https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering\n",
    "\n",
    "Adding columns or transform existing columns\n",
    "- this includes the target (target transformation)\n",
    "\n",
    "## How can I generate features?\n",
    "\n",
    "Domain knowledge\n",
    "- use business insight -> features\n",
    "\n",
    "Exploratory data analysis\n",
    "\n",
    "Build features from features\n",
    "- ratios\n",
    "- polynomials\n",
    "- log / exponentials / sigmoids\n",
    "- Fourier / Laplace transforms\n",
    "\n",
    "## Data leakage\n",
    "\n",
    "Feature available during training that isn't available during testing / production time\n",
    "\n",
    "Duplicates in the train & test set\n",
    "\n",
    "Training / learning statistics from the test set\n",
    "- common mistake to refit normalization / standardization\n",
    "\n",
    "Feature engineering can often cause data leakage\n",
    "- mean target encoding on the entire dataset, before splitting test & train\n",
    "\n",
    "## Nominal vs ordinal\n",
    "\n",
    "Nominal (no order) versus ordinal (order)\n",
    "\n",
    "Some methods place order onto a nominal series during the encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform from continuous to categorical\n",
    "\n",
    "On the target \n",
    "- moving from a regression to classification problem \n",
    "- key thing to ask - how do we use our prediction in our business problem\n",
    "\n",
    "On features \n",
    "- removes noise (& maybe signal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.cut(ds.loc[:, 'contract-length'].fillna(0), bins=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with categorical variables\n",
    "\n",
    "Machine always think in numbers\n",
    "- all categorical features need to be transformed into numbers at somepoint\n",
    "- most ML algorithms get upset if you don't do it - some will do it on the fly\n",
    "\n",
    "## One hot encoding\n",
    "\n",
    "- curse of dimensionality makes dimensionality increase exponential\n",
    "- lose the explicit relationship of the feature (model now just sees a lot of columns, and has to learn their relationship)\n",
    "\n",
    "There are a few ways to do one-hot encoding in the Python\n",
    "- `sklearn.preprocessing.OneHotEncoder()`\n",
    "- `pd.get_dummies()`\n",
    "\n",
    "Recommend `sklearn.preprocessing.OneHotEncoder()`\n",
    "- creates a stateful transformer that can be reused (i.e. on the test data)\n",
    "- using `pd.get_dummies`, you are hoping that the columns will be encoded in the same way at test time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "#  note that I use a missing value token here\n",
    "cat = ds.loc[:, 'customers-category'].to_frame().fillna('missing')\n",
    "\n",
    "#  will by default return a sparse matrix - you can turn this off\n",
    "enc = OneHotEncoder(sparse=False)\n",
    "enc.fit_transform(cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label encoding\n",
    "\n",
    "- `0, 1, 2, 3`\n",
    "- is an ordinal encoding - even if feature is not ordinal, you are imposing this structure on the data\n",
    "\n",
    "Decision trees sort the data based on the feature being split on\n",
    "- this makes the decision boundary meaningful\n",
    "- inherently ordinal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = LabelEncoder()\n",
    "enc.fit_transform(cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean encode\n",
    "\n",
    "- put the training data average for the target for that class\n",
    "- could also use other statistics like median, quantiles or variance\n",
    "\n",
    "The code below will mean encode a categorical feature using the average of another column (often the target):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_encode(data, col, on):\n",
    "    group = data.groupby(col).mean()\n",
    "    data.loc[:, col+'-original'] = data.loc[:, col]\n",
    "    mapper = {k: v for k, v in zip(group.index, group.loc[:, on].values)}\n",
    "\n",
    "    data.loc[:, col] = data.loc[:, col].replace(mapper)\n",
    "    data.loc[:, col].fillna(value=np.mean(data.loc[:, col]), inplace=True)\n",
    "    return data\n",
    "\n",
    "\n",
    "def test_mean_encoding():\n",
    "    store1 = pd.DataFrame(\n",
    "        {'store': ['A'] * 3,\n",
    "         'Sales': [100, 200, 300],\n",
    "         'noise': [0, 0, 0]}\n",
    "    )\n",
    "\n",
    "    store2 = pd.DataFrame(\n",
    "        {'store': ['B'] * 3,\n",
    "         'Sales': [10, 20, 30],\n",
    "         'noise': [0, 0, 0]}\n",
    "    )\n",
    "\n",
    "    data = pd.concat([store1, store2], axis=0)\n",
    "    data = mean_encode(data, col='store', on='Sales')\n",
    "    np.testing.assert_array_equal(\n",
    "        data.loc[:, 'store'], np.array([200, 200, 200, 20, 20, 20])\n",
    "    )\n",
    "    \n",
    "test_mean_encoding()\n",
    "data = mean_encode(ds.copy(), 'customers-category', 'contract-length')\n",
    "data.drop(['contract-length', 'location'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean & frequency encoding\n",
    "\n",
    "See *Feature Engineering with H2O - Dmitry Larko, Senior Data Scientist, H2O.ai - [youtube](https://youtu.be/irkV4sYExX4)*\n",
    "\n",
    "If we have a category with only a few samples, the mean we encode will be higher variance\n",
    "- we can tell the model this by encoding the weighted average of the overall mean & the mean of this category\n",
    "- we trust the mean encoding less if we have less samples\n",
    "- requires a hyperparameter $\\lambda$\n",
    "- function that turns a category into a weight\n",
    "\n",
    "$$ \\lambda(category) * \\mu(category) + (1 - \\lambda(category)) * mean(data) $$\n",
    "\n",
    "A simple choice of $\\lambda$ is a frequentist probability\n",
    "\n",
    "$$\\lambda(category) = \\frac{freq(category)}{freq(all categories)}$$\n",
    "\n",
    "### Practical\n",
    "\n",
    "Implement mean & frequency encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary encoding\n",
    "\n",
    "Encoding the string using it's binary (0110100 etc) representation\n",
    "\n",
    "In Python we can do this using the `bin()` builtin:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bin(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `category_encoders` library can do this for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ['A', 'BB', 'B', 'A', 'C', 'D', 'A', 'E']\n",
    "\n",
    "enc = ce.BinaryEncoder()\n",
    "\n",
    "print('{} classes'.format(len(set(data))))\n",
    "\n",
    "pd.concat([enc.fit_transform(data), pd.DataFrame(data, columns=['data'])], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency encoding\n",
    "\n",
    "Encode the categorical feature based on their relative frequency\n",
    "- probability of seeing this category\n",
    "- tell the model about rare categories\n",
    "- but you can't distinguish categories with the same frequency (unlikely in very large datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ['A', 'B', 'A', 'C', 'D', 'A', 'E']\n",
    "counter = Counter(data)\n",
    "freq_enc = [counter[x] / len(data) for x in data]\n",
    "\n",
    "pd.DataFrame(\n",
    "    {'data': data,\n",
    "     'freq_enc': freq_enc}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interaction features\n",
    "\n",
    "Also called **feature crossing**\n",
    "\n",
    "Multiplying features together\n",
    "\n",
    "Define interaction effects as new features directly\n",
    "- even for trees - because trees are locally greedy\n",
    "\n",
    "Grouping sparse classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    'late': [1, 1, 0],\n",
    "    'tired': [1, 0, 1]\n",
    "}\n",
    "\n",
    "d = pd.DataFrame(data)\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add columns based on the combination of these features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d.loc[:, 'late-and-tired'] = ((d.loc[:, 'late'] == 1) * (d.loc[:, 'tired'] == 1)).astype(int)\n",
    "\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This kind of if/and relationship is something that a decision tree is good at learning:\n",
    "- but you can always help your model out\n",
    "- encode domain knowledge where you can!\n",
    "\n",
    "This kind of feature engineering will be uncomfortable for those who are familiar with linear models, where introducing this kind of co-linearity is incorrect :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indirect features\n",
    "\n",
    "*Gabby Shklovsky - Random Forests Best Practices for the Business World - PyData NYC 2017 [youtube](https://www.youtube.com/watch?v=E7VLE-U07x0) - [slides](https://www.youtube.com/redirect?q=https%3A%2F%2Fwww.slideshare.net%2FPyData%2Frandom-forests-best-practices-for-the-business-world&redir_token=HgV_RBYb_uD_jYV6nYygn8RpyKR8MTU2OTkwODE2N0AxNTY5ODIxNzY3&v=E7VLE-U07x0&event=video_description)*\n",
    "\n",
    "Non-predictive features don't hurt so much with trees\n",
    "- **use multiple metrics that are proxies** for same concept as predictors\n",
    "- two or three things that measure it indirectly\n",
    "- absolute and relative differences\n",
    "- percentage & absolute changes - sometimes % is predictive, sometimes $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factor out linear relationships\n",
    "\n",
    "*Gabby Shklovsky - Random Forests Best Practices for the Business World - PyData NYC 2017 [youtube](https://www.youtube.com/watch?v=E7VLE-U07x0) - [slides](https://www.youtube.com/redirect?q=https%3A%2F%2Fwww.slideshare.net%2FPyData%2Frandom-forests-best-practices-for-the-business-world&redir_token=HgV_RBYb_uD_jYV6nYygn8RpyKR8MTU2OTkwODE2N0AxNTY5ODIxNzY3&v=E7VLE-U07x0&event=video_description)*\n",
    "\n",
    "- factor out linear relationships between predictor and response\n",
    "- a strong linear relationship will overpower subtle non linearities (will dominate the tree splits)\n",
    "- trees model a linear relationship in a non-linear way -> unlikely to pick up non linear effects\n",
    "\n",
    "### Differencing\n",
    "\n",
    "- difference = linear transformation\n",
    "- **predict the difference** between last year and now (not the actual sales)\n",
    "- last spend is still predictive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame(\n",
    "    {'sales': [10, 50, 100], 'year': [2012, 2013, 2014]}\n",
    ")\n",
    "\n",
    "data.loc[:, 'last-year-sales-feature'] = data.loc[:, 'sales'].shift()\n",
    "data.loc[:, 'model-target'] = data.loc[:, 'sales'].diff()\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality reduction\n",
    "\n",
    "### PCA & t-SNE\n",
    "\n",
    "Both forms of dimensionality reduction, that can create new columns\n",
    "- these can sit alongside or replace the high dimensional data\n",
    "\n",
    "See [visualization.ipynb]\n",
    "\n",
    "### Clustering\n",
    "\n",
    "Can be used to create a new feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame([np.random.uniform(0, x, 1000) for x in range(1, 10)]).T\n",
    "\n",
    "mdl = KMeans(3)\n",
    "mdl.fit(data)\n",
    "data.loc[:, 'cluster'] = mdl.labels_\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datetime\n",
    "\n",
    "### Month, year, day columns\n",
    "\n",
    "These can then be either be label / one-hot encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame(\n",
    "    {'date': pd.date_range('01-01-2018', '02-01-2018', freq='1h')}\n",
    ")\n",
    "\n",
    "data.loc[:, 'month'] = data.loc[:, 'date'].dt.month\n",
    "data.loc[:, 'minute'] = data.loc[:, 'date'].dt.minute\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cyclical datetime features\n",
    "\n",
    "Encoding cyclical continuous features - [blog post](https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hours_in_day = 24\n",
    "\n",
    "def transform_hourly(self, x):\n",
    "    h = x.index.hour\n",
    "\n",
    "    sin = np.sin(2 * np.pi * h / self.hours_in_day)\n",
    "    cos = np.cos(2 * np.pi * h / self.hours_in_day)\n",
    "\n",
    "    out = pd.DataFrame(index=x.index)\n",
    "    out.loc[:, 'sin_h'] = sin\n",
    "    out.loc[:, 'cos_h'] = cos\n",
    "    return out\n",
    "\n",
    "\n",
    "def transform_hh(self, x):\n",
    "    hh = x.index.hour + (x.index.minute / 60)\n",
    "\n",
    "    sin = np.sin(2 * np.pi * hh / self.hours_in_day)\n",
    "    cos = np.cos(2 * np.pi * hh / self.hours_in_day)\n",
    "\n",
    "    out = pd.DataFrame(index=x.index)\n",
    "    out.loc[:, 'sin_hh'] = sin\n",
    "    out.loc[:, 'cos_hh'] = cos\n",
    "    return out\n",
    "\n",
    "\n",
    "def transform(x, max_value):\n",
    "    sin = np.sin(2 * np.pi * x / max_value)\n",
    "    cos = np.cos(2 * np.pi * x / max_value)\n",
    "\n",
    "    out = pd.DataFrame()\n",
    "    out.loc[:, 'sin'] = sin\n",
    "    out.loc[:, 'cos'] = cos\n",
    "\n",
    "    return out\n",
    "\n",
    "rng = pd.DataFrame(index=pd.date_range('01-01-2020', '01-02-2020', freq='1h'))\n",
    "df = transform(rng.index.hour, 24)\n",
    "df.plot('sin', 'cos', kind='scatter')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time series decomposition\n",
    "\n",
    "Seasonality, trend can become features\n",
    "\n",
    "More important than other areas to ask the question\n",
    "- what will I have available at test time?\n",
    "- example of weather versus weather forecasts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP\n",
    "\n",
    "Bag of words, tfidf\n",
    "\n",
    "Tokenization\n",
    "- harder than splitting on whitespace!\n",
    "- ngrams\n",
    "- NLTK, SpaCy\n",
    "\n",
    "Word / doc embeddings\n",
    "- word to vec\n",
    "\n",
    "### Part of speech tagging with SpaCy\n",
    "\n",
    "A simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc.\n",
    "\n",
    "https://spacy.io/usage/linguistic-features\n",
    "\n",
    "https://spacy.io/api/token\n",
    "\n",
    "**Need to run**:\n",
    "```bash\n",
    "$ python -m spacy download en_core_web_sm\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_ents(doc, verbose=False):\n",
    "    if verbose:\n",
    "        print(doc)\n",
    "        print('---')\n",
    "        \n",
    "    doc = nlp(doc)\n",
    "    ents = []\n",
    "    for token in doc:\n",
    "        \n",
    "        if verbose:\n",
    "            print({\n",
    "                'text': token.text,\n",
    "                'coarse POS': token.pos_,\n",
    "                'fine POS': token.tag_,\n",
    "                'syntatic dependency relation': token.dep_,\n",
    "                'stop': token.is_stop\n",
    "            })\n",
    "        if token.pos_ == 'PROPN':\n",
    "            ents.append(token)\n",
    "            \n",
    "    return doc, ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm') \n",
    "doc, ents = find_ents(\"Apple is looking at buying U.K. startup for $1 billion to prevent coronavirus\")\n",
    "ents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Possible for a single word to have a different part of speech tag in different sentences based on different contexts\n",
    "\n",
    "- I love you vs. Lets make love"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc, ents = find_ents(\"I care for you\", verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc, ents = find_ents(\"He needs a lot of care\", verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Augmentation\n",
    "\n",
    "Common in computer vision\n",
    "- rotation etc\n",
    "\n",
    "SMOTE\n",
    "- synthetic data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}