feature_penalties/feature_penalties.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature weights and penalties tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some applications it would be helpful to give model a hint to reduce the use of specific features. Some features may be less interpretable, which can be an important consideration when the model needs to be explained to non-technical audiences.\n",
    "\n",
    "Another application is in time series, where a continuous time feature is useful to model changes over time, but vulnerable to overfitting because everything in a time series could be explained with enough splits on time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes such hints may be harmful because the whole concept of machine learning is to use statistical analysis to assign optimal feature weights. Anyway catboost gives such a possibility through feature weights and penalties."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature weights are designed to change some features usage by specifying weight to each of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do we need feature penalties for? Some object features can be more computationally expensive than another. For example, we want to make a diagnosis. It's much cheaper to take blood sample than X-ray, and in most cases blood sample is enough to diagnose. So it's would save us some resources to not X-ray if diagnosis is clear after taking blood sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Cost efficient gradient boosting](https://papers.nips.cc/paper/6753-cost-efficient-gradient-boosting.pdf) is supposed to build models with less expensive-to-calculate features usage and less applying time. Feature computation time is specified through *per-object-feature-penalties* and *first-feature-use-penalties* (if feature is computed once for all objects) parameters. You can scale all feature penalties at once through *penalties-coefficient* param."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Algorithm uses given parameters by modifying split candidate score in the following way:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Score' = Score * \\prod_{f\\in S}W_f - \\sum_{f\\in S}P_f*U(f) - \\sum_{f\\in S}\\sum_{x \\in L}EP_f*U(f, x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$W_f - feature\\: weight$\n",
    "\n",
    "$P_f - feature\\: penalty$\n",
    "\n",
    "$EP_f - elementwise\\: feature\\: penalty$\n",
    "\n",
    "$S - current\\:split$\n",
    "\n",
    "$L - current\\: leaf$\n",
    "\n",
    "$U(f) = \n",
    "\\begin{cases}\n",
    "    0,& \\text{if } f \\text{ was used in model already}\\\\\n",
    "    1,& \\text{otherwise}\n",
    "\\end{cases}$\n",
    "\n",
    "$U(f, x) = \n",
    "\\begin{cases}\n",
    "    0,& \\text{if } f \\text{ was used already for object } x\\\\\n",
    "    1,& \\text{otherwise}\n",
    "\\end{cases}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a synthetic dataset to demonstrate feature penalties influence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "import catboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 33\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CEGB claims to produce trees with deep branches but low expected depth and feature cost, so let's try to build it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a synthetic dataset for regression. Each next feature is supposed to be more costly than previous. Part of the data label depends on the first feature, part of the rest data label depends on the second feature and so on. So a perfect model would use only *n_features - 1* splits with each feature except the last one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_dataset(n_objects, n_features):\n",
    "    assert(n_objects % n_features == 0)\n",
    "    \n",
    "    X = np.random.rand(n_objects, n_features)\n",
    "    y = np.zeros(n_objects, dtype=int)\n",
    "    \n",
    "    piece_len = n_objects // n_features\n",
    "    for lvl in range(n_features):\n",
    "        X[:, lvl] = X[:, lvl] / 2\n",
    "        X[piece_len*lvl:piece_len*(lvl+1), lvl] = X[piece_len*lvl:piece_len*(lvl+1), lvl] + 0.5\n",
    "        y[piece_len*lvl:piece_len*(lvl+1)] = lvl\n",
    "    \n",
    "    X, y = sklearn.utils.shuffle(X, y, random_state=seed)\n",
    "    return catboost.Pool(X, y, cat_features=[], feature_names=list(map(str, range(n_features))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_objects=2**13\n",
    "n_features=4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_pool = generate_dataset(n_objects, n_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterations_count=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "common_params = {\n",
    "    'random_seed': seed,\n",
    "    'grow_policy': 'Lossguide',\n",
    "    'objective': 'RMSE',\n",
    "    'iterations': iterations_count,\n",
    "    'depth': n_features-1,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_wo_penalties = catboost.CatBoostRegressor(\n",
    "    **common_params,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.5\n",
      "0:\tlearn: 0.5723978\ttotal: 66ms\tremaining: 594ms\n",
      "1:\tlearn: 0.2942938\ttotal: 89.7ms\tremaining: 359ms\n",
      "2:\tlearn: 0.1588871\ttotal: 113ms\tremaining: 265ms\n",
      "3:\tlearn: 0.0942765\ttotal: 136ms\tremaining: 204ms\n",
      "4:\tlearn: 0.0639308\ttotal: 158ms\tremaining: 158ms\n",
      "5:\tlearn: 0.0498856\ttotal: 180ms\tremaining: 120ms\n",
      "6:\tlearn: 0.0456871\ttotal: 202ms\tremaining: 86.4ms\n",
      "7:\tlearn: 0.0434958\ttotal: 224ms\tremaining: 56.1ms\n",
      "8:\tlearn: 0.0427214\ttotal: 244ms\tremaining: 27.1ms\n",
      "9:\tlearn: 0.0261691\ttotal: 265ms\tremaining: 0us\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7fc04a135110>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_wo_penalties.fit(small_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"788pt\" height=\"305pt\"\n",
       " viewBox=\"0.00 0.00 787.50 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 783.5,-301 783.5,4 -4,4\"/>\n",
       "<!-- node_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>node_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"455\" cy=\"-279\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"455\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">3, value&gt;0.500058</text>\n",
       "</g>\n",
       "<!-- node_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>node_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"348\" cy=\"-192\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"348\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">2, value&gt;0.508266</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_1 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M433.347,-261.3943C417.2485,-248.3048 395.1013,-230.2973 377.4022,-215.9064\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"379.5878,-213.1726 369.6208,-209.5796 375.1717,-218.6039 379.5878,-213.1726\"/>\n",
       "<text text-anchor=\"middle\" x=\"419\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>node_8</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"564\" cy=\"-192\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"564\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.920095</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_8 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_8</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M477.0578,-261.3943C493.4572,-248.3048 516.0183,-230.2973 534.0483,-215.9064\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"536.3427,-218.5533 541.975,-209.5796 531.9759,-213.0823 536.3427,-218.5533\"/>\n",
       "<text text-anchor=\"middle\" x=\"529.5\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- node_2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>node_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"138\" cy=\"-105\" rx=\"92.8835\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"138\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">1, value&gt;0.50625</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;node_2 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;node_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M308.0063,-175.4312C273.5954,-161.1752 223.8274,-140.5571 187.0243,-125.3101\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"188.067,-121.9536 177.4888,-121.3596 185.3878,-128.4206 188.067,-121.9536\"/>\n",
       "<text text-anchor=\"middle\" x=\"267\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>node_5</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"348\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"348\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">2, value&gt;0.324005</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;node_5 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;node_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M348,-173.9735C348,-162.1918 348,-146.5607 348,-133.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"351.5001,-133.0033 348,-123.0034 344.5001,-133.0034 351.5001,-133.0033\"/>\n",
       "<text text-anchor=\"middle\" x=\"360.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_0 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>leaf_0</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"104,-36 0,-36 0,0 104,0 104,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"52\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.726</text>\n",
       "</g>\n",
       "<!-- node_2&#45;&gt;leaf_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>node_2&#45;&gt;leaf_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M120.1807,-86.9735C107.7272,-74.3752 90.9229,-57.3755 77.1135,-43.4055\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"79.3157,-40.6547 69.7964,-36.0034 74.3374,-45.5757 79.3157,-40.6547\"/>\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_1 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>leaf_1</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"226,-36 122,-36 122,0 226,0 226,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.250</text>\n",
       "</g>\n",
       "<!-- node_2&#45;&gt;leaf_1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>node_2&#45;&gt;leaf_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M145.4592,-86.9735C150.431,-74.9585 157.0593,-58.9401 162.6767,-45.3646\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"165.9608,-46.5818 166.5503,-36.0034 159.4927,-43.9053 165.9608,-46.5818\"/>\n",
       "<text text-anchor=\"middle\" x=\"171.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_2 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>leaf_2</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"361.5,-36 262.5,-36 262.5,0 361.5,0 361.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"312\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.000</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;leaf_2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;leaf_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M340.5408,-86.9735C335.569,-74.9585 328.9407,-58.9401 323.3233,-45.3646\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"326.5073,-43.9053 319.4497,-36.0034 320.0392,-46.5818 326.5073,-43.9053\"/>\n",
       "<text text-anchor=\"middle\" x=\"344\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_3 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>leaf_3</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"478.5,-36 379.5,-36 379.5,0 478.5,0 478.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"429\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.250</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;leaf_3 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;leaf_3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M364.7833,-86.9735C376.5127,-74.3752 392.3401,-57.3755 405.3466,-43.4055\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"407.9857,-45.7073 412.2382,-36.0034 402.8624,-40.9374 407.9857,-45.7073\"/>\n",
       "<text text-anchor=\"middle\" x=\"406.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- node_9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>node_9</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"564\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"564\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.714181</text>\n",
       "</g>\n",
       "<!-- node_8&#45;&gt;node_9 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>node_8&#45;&gt;node_9</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M564,-173.9735C564,-162.1918 564,-146.5607 564,-133.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"567.5001,-133.0033 564,-123.0034 560.5001,-133.0034 567.5001,-133.0033\"/>\n",
       "<text text-anchor=\"middle\" x=\"575\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_4 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>leaf_4</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"779.5,-123 680.5,-123 680.5,-87 779.5,-87 779.5,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"730\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.000</text>\n",
       "</g>\n",
       "<!-- node_8&#45;&gt;leaf_4 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>node_8&#45;&gt;leaf_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M596.7963,-174.8116C622.4333,-161.3753 658.3064,-142.5743 686.3669,-127.868\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"688.3249,-130.7934 695.5575,-123.0512 685.0754,-124.5933 688.3249,-130.7934\"/>\n",
       "<text text-anchor=\"middle\" x=\"669.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_5 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>leaf_5</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"604.5,-36 505.5,-36 505.5,0 604.5,0 604.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"555\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.749</text>\n",
       "</g>\n",
       "<!-- node_9&#45;&gt;leaf_5 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>node_9&#45;&gt;leaf_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M562.1352,-86.9735C560.9164,-75.1918 559.2994,-59.5607 557.9129,-46.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"561.3729,-45.5901 556.8624,-36.0034 554.4101,-46.3105 561.3729,-45.5901\"/>\n",
       "<text text-anchor=\"middle\" x=\"572\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_6 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>leaf_6</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"721.5,-36 622.5,-36 622.5,0 721.5,0 721.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"672\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.000</text>\n",
       "</g>\n",
       "<!-- node_9&#45;&gt;leaf_6 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>node_9&#45;&gt;leaf_6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M585.8554,-87.3943C601.8905,-74.4771 623.8713,-56.7703 641.6156,-42.4763\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"643.8417,-45.1774 649.4336,-36.1785 639.4504,-39.7261 643.8417,-45.1774\"/>\n",
       "<text text-anchor=\"middle\" x=\"637.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fc07adaf690>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_wo_penalties.plot_tree(tree_idx=0, pool=small_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the model without feature penalties uses some extra splits except three required ones, including splits with \"costly\" features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_penalties(n_features, base=1.3):\n",
    "    return list([base**lvl for lvl in range(n_features)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_with_penalties = catboost.CatBoostRegressor(\n",
    "    **common_params,\n",
    "    penalties_coefficient=0.001,\n",
    "    per_object_feature_penalties=get_penalties(n_features)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.5\n",
      "0:\tlearn: 0.5818967\ttotal: 26.4ms\tremaining: 237ms\n",
      "1:\tlearn: 0.3140865\ttotal: 44.3ms\tremaining: 177ms\n",
      "2:\tlearn: 0.1791524\ttotal: 64.4ms\tremaining: 150ms\n",
      "3:\tlearn: 0.1166760\ttotal: 86.3ms\tremaining: 129ms\n",
      "4:\tlearn: 0.0765730\ttotal: 108ms\tremaining: 108ms\n",
      "5:\tlearn: 0.0616796\ttotal: 127ms\tremaining: 84.8ms\n",
      "6:\tlearn: 0.0547723\ttotal: 147ms\tremaining: 63ms\n",
      "7:\tlearn: 0.0481709\ttotal: 169ms\tremaining: 42.2ms\n",
      "8:\tlearn: 0.0460998\ttotal: 190ms\tremaining: 21.1ms\n",
      "9:\tlearn: 0.0428211\ttotal: 214ms\tremaining: 0us\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7fc04a163e10>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_with_penalties.fit(small_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that loss of model with penalties is just twice the original loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"807pt\" height=\"305pt\"\n",
       " viewBox=\"0.00 0.00 806.50 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 802.5,-301 802.5,4 -4,4\"/>\n",
       "<!-- node_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>node_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"363.5\" cy=\"-279\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"363.5\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.497255</text>\n",
       "</g>\n",
       "<!-- node_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>node_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"259.5\" cy=\"-192\" rx=\"92.8835\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"259.5\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">1, value&gt;0.50625</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_1 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M342.454,-261.3943C326.9479,-248.4228 305.668,-230.6213 288.5438,-216.2962\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"290.4305,-213.3114 280.5147,-209.5796 285.9391,-218.6805 290.4305,-213.3114\"/>\n",
       "<text text-anchor=\"middle\" x=\"329.5\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_5 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>node_5</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"469.5\" cy=\"-192\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"469.5\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.920095</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_5 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M384.9507,-261.3943C400.755,-248.4228 422.4441,-230.6213 439.8977,-216.2962\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"442.5719,-218.6293 448.0812,-209.5796 438.1309,-213.2184 442.5719,-218.6293\"/>\n",
       "<text text-anchor=\"middle\" x=\"436\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- node_2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>node_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"110.5\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"110.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">2, value&gt;0.500036</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;node_2 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;node_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M230.0624,-174.8116C206.8733,-161.2716 174.354,-142.2839 149.0828,-127.5282\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"150.5316,-124.3212 140.1311,-122.3014 147.002,-130.3662 150.5316,-124.3212\"/>\n",
       "<text text-anchor=\"middle\" x=\"205.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>leaf_0</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"331.5,-123 227.5,-123 227.5,-87 331.5,-87 331.5,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.250</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;leaf_0 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;leaf_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M263.644,-173.9735C266.3793,-162.0751 270.0171,-146.2508 273.1182,-132.7606\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"276.5318,-133.5333 275.3613,-123.0034 269.7098,-131.965 276.5318,-133.5333\"/>\n",
       "<text text-anchor=\"middle\" x=\"284\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>leaf_1</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"99,-36 0,-36 0,0 99,0 99,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"49.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.733</text>\n",
       "</g>\n",
       "<!-- node_2&#45;&gt;leaf_1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>node_2&#45;&gt;leaf_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M97.8608,-86.9735C89.2728,-74.7252 77.7679,-58.3165 68.1341,-44.5766\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"70.7298,-42.1819 62.123,-36.0034 64.9983,-46.2006 70.7298,-42.1819\"/>\n",
       "<text text-anchor=\"middle\" x=\"95.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>leaf_2</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"216,-36 117,-36 117,0 216,0 216,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"166.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.250</text>\n",
       "</g>\n",
       "<!-- node_2&#45;&gt;leaf_2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>node_2&#45;&gt;leaf_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M122.1032,-86.9735C129.9872,-74.7252 140.5491,-58.3165 149.3933,-44.5766\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"152.4422,-46.3064 154.9116,-36.0034 146.5561,-42.5176 152.4422,-46.3064\"/>\n",
       "<text text-anchor=\"middle\" x=\"155\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- node_6 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>node_6</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"453.5\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"453.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.714181</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;node_6 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;node_6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M466.1848,-173.9735C464.018,-162.1918 461.1433,-146.5607 458.6785,-133.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"462.0621,-132.2053 456.811,-123.0034 455.1775,-133.4715 462.0621,-132.2053\"/>\n",
       "<text text-anchor=\"middle\" x=\"473.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_9 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>node_9</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"669.5\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"669.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.131665</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;node_9 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;node_9</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M508.062,-175.2255C540.434,-161.1437 586.8303,-140.9613 621.5841,-125.8434\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"623.2825,-128.9215 631.0564,-121.723 620.4902,-122.5025 623.2825,-128.9215\"/>\n",
       "<text text-anchor=\"middle\" x=\"594\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_3 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>leaf_3</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"393.5,-36 289.5,-36 289.5,0 393.5,0 393.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"341.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.714</text>\n",
       "</g>\n",
       "<!-- node_6&#45;&gt;leaf_3 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>node_6&#45;&gt;leaf_3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M430.8351,-87.3943C414.2061,-74.4771 391.4112,-56.7703 373.0098,-42.4763\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"374.9466,-39.5489 364.9022,-36.1785 370.6524,-45.0771 374.9466,-39.5489\"/>\n",
       "<text text-anchor=\"middle\" x=\"415.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_4 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>leaf_4</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"515.5,-36 411.5,-36 411.5,0 515.5,0 515.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"463.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.747</text>\n",
       "</g>\n",
       "<!-- node_6&#45;&gt;leaf_4 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>node_6&#45;&gt;leaf_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M455.572,-86.9735C456.9262,-75.1918 458.7229,-59.5607 460.2634,-46.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"463.7658,-46.3376 461.4306,-36.0034 456.8115,-45.5383 463.7658,-46.3376\"/>\n",
       "<text text-anchor=\"middle\" x=\"472\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_5 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>leaf_5</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"676,-36 577,-36 577,0 676,0 676,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"626.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.000</text>\n",
       "</g>\n",
       "<!-- node_9&#45;&gt;leaf_5 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>node_9&#45;&gt;leaf_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M660.5904,-86.9735C654.5942,-74.8418 646.5808,-58.6287 639.8299,-44.9698\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"642.9668,-43.4173 635.3982,-36.0034 636.6915,-46.519 642.9668,-43.4173\"/>\n",
       "<text text-anchor=\"middle\" x=\"662.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_6 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>leaf_6</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"798.5,-36 694.5,-36 694.5,0 798.5,0 798.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"746.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.743</text>\n",
       "</g>\n",
       "<!-- node_9&#45;&gt;leaf_6 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>node_9&#45;&gt;leaf_6</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M685.4544,-86.9735C696.5014,-74.4919 711.3722,-57.6899 723.6705,-43.7944\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"726.5593,-45.8114 730.566,-36.0034 721.3174,-41.172 726.5593,-45.8114\"/>\n",
       "<text text-anchor=\"middle\" x=\"725\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fc04a131810>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_with_penalties.plot_tree(tree_idx=0, pool=small_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that model with penalties doesn't have unnecessary splits with \"costly\" features 1 and 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Too low feature penalties may have no effect on model while too big feature penalties may significantly decrease model quality. Be careful to use it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to achieve similar results using only feature weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_weights(n_features, base=1.1):\n",
    "    return list([(1/base)**lvl for lvl in range(n_features)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_with_weights = catboost.CatBoostRegressor(\n",
    "    **common_params,\n",
    "    feature_weights=get_weights(n_features)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.5\n",
      "0:\tlearn: 0.6531579\ttotal: 18.3ms\tremaining: 165ms\n",
      "1:\tlearn: 0.3379231\ttotal: 36.1ms\tremaining: 144ms\n",
      "2:\tlearn: 0.1898801\ttotal: 53.7ms\tremaining: 125ms\n",
      "3:\tlearn: 0.1257772\ttotal: 73.9ms\tremaining: 111ms\n",
      "4:\tlearn: 0.0868733\ttotal: 95.1ms\tremaining: 95.1ms\n",
      "5:\tlearn: 0.0705268\ttotal: 118ms\tremaining: 78.4ms\n",
      "6:\tlearn: 0.0644072\ttotal: 140ms\tremaining: 60ms\n",
      "7:\tlearn: 0.0601975\ttotal: 163ms\tremaining: 40.8ms\n",
      "8:\tlearn: 0.0548494\ttotal: 184ms\tremaining: 20.4ms\n",
      "9:\tlearn: 0.0371757\ttotal: 208ms\tremaining: 0us\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7fc04a167e50>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_with_weights.fit(small_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"721pt\" height=\"305pt\"\n",
       " viewBox=\"0.00 0.00 720.50 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 716.5,-301 716.5,4 -4,4\"/>\n",
       "<!-- node_0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>node_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"253.5\" cy=\"-279\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"253.5\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.497255</text>\n",
       "</g>\n",
       "<!-- node_1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>node_1</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"149.5\" cy=\"-192\" rx=\"92.8835\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"149.5\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">1, value&gt;0.50625</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_1 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M232.454,-261.3943C216.9479,-248.4228 195.668,-230.6213 178.5438,-216.2962\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"180.4305,-213.3114 170.5147,-209.5796 175.9391,-218.6805 180.4305,-213.3114\"/>\n",
       "<text text-anchor=\"middle\" x=\"219.5\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>node_4</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"359.5\" cy=\"-192\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"359.5\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.920095</text>\n",
       "</g>\n",
       "<!-- node_0&#45;&gt;node_4 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>node_0&#45;&gt;node_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M274.9507,-261.3943C290.755,-248.4228 312.4441,-230.6213 329.8977,-216.2962\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"332.5719,-218.6293 338.0812,-209.5796 328.1309,-213.2184 332.5719,-218.6293\"/>\n",
       "<text text-anchor=\"middle\" x=\"326\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>leaf_0</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"99,-123 0,-123 0,-87 99,-87 99,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"49.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.494</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;leaf_0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;leaf_0</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M129.2635,-174.3943C114.5499,-161.5934 94.4299,-144.089 78.0787,-129.8635\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"80.2365,-127.1016 70.3948,-123.1785 75.6419,-132.3827 80.2365,-127.1016\"/>\n",
       "<text text-anchor=\"middle\" x=\"116.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_1 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>leaf_1</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"221.5,-123 117.5,-123 117.5,-87 221.5,-87 221.5,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"169.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.250</text>\n",
       "</g>\n",
       "<!-- node_1&#45;&gt;leaf_1 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>node_1&#45;&gt;leaf_1</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M153.644,-173.9735C156.3793,-162.0751 160.0171,-146.2508 163.1182,-132.7606\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"166.5318,-133.5333 165.3613,-123.0034 159.7098,-131.965 166.5318,-133.5333\"/>\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- node_5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>node_5</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"348.5\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"348.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.714181</text>\n",
       "</g>\n",
       "<!-- node_4&#45;&gt;node_5 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>node_4&#45;&gt;node_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M357.2208,-173.9735C355.7311,-162.1918 353.7548,-146.5607 352.0602,-133.1581\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"355.5031,-132.4853 350.7763,-123.0034 348.5584,-133.3634 355.5031,-132.4853\"/>\n",
       "<text text-anchor=\"middle\" x=\"366.5\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- node_8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>node_8</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"564.5\" cy=\"-105\" rx=\"98.5829\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"564.5\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">0, value&gt;0.131665</text>\n",
       "</g>\n",
       "<!-- node_4&#45;&gt;node_8 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>node_4&#45;&gt;node_8</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M398.5415,-175.4312C431.9198,-161.2657 480.0998,-140.8186 515.9571,-125.6011\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"517.4173,-128.7836 525.2553,-121.6551 514.6826,-122.3399 517.4173,-128.7836\"/>\n",
       "<text text-anchor=\"middle\" x=\"487\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_2 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>leaf_2</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"302.5,-36 198.5,-36 198.5,0 302.5,0 302.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.714</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;leaf_2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;leaf_2</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M328.1943,-86.9735C313.8717,-74.2586 294.4989,-57.0603 278.6815,-43.0183\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"280.5816,-40.0249 270.7797,-36.0034 275.9343,-45.2597 280.5816,-40.0249\"/>\n",
       "<text text-anchor=\"middle\" x=\"316.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_3 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>leaf_3</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"424.5,-36 320.5,-36 320.5,0 424.5,0 424.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"372.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.747</text>\n",
       "</g>\n",
       "<!-- node_5&#45;&gt;leaf_3 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>node_5&#45;&gt;leaf_3</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M353.4728,-86.9735C356.7551,-75.0751 361.1205,-59.2508 364.8419,-45.7606\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"368.2482,-46.5741 367.5336,-36.0034 361.5002,-44.7125 368.2482,-46.5741\"/>\n",
       "<text text-anchor=\"middle\" x=\"375\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "<!-- leaf_4 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>leaf_4</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"590,-36 491,-36 491,0 590,0 590,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"540.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = 0.000</text>\n",
       "</g>\n",
       "<!-- node_8&#45;&gt;leaf_4 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>node_8&#45;&gt;leaf_4</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M559.5272,-86.9735C556.2449,-75.0751 551.8795,-59.2508 548.1581,-45.7606\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"551.4998,-44.7125 545.4664,-36.0034 544.7518,-46.5741 551.4998,-44.7125\"/>\n",
       "<text text-anchor=\"middle\" x=\"565.5\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">No</text>\n",
       "</g>\n",
       "<!-- leaf_5 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>leaf_5</title>\n",
       "<polygon fill=\"none\" stroke=\"#ff0000\" points=\"712.5,-36 608.5,-36 608.5,0 712.5,0 712.5,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"660.5\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">val = &#45;0.743</text>\n",
       "</g>\n",
       "<!-- node_8&#45;&gt;leaf_5 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>node_8&#45;&gt;leaf_5</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M584.3913,-86.9735C598.4216,-74.2586 617.399,-57.0603 632.8936,-43.0183\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"635.5747,-45.3121 640.6342,-36.0034 630.874,-40.1251 635.5747,-45.3121\"/>\n",
       "<text text-anchor=\"middle\" x=\"630\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">Yes</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fc04a0f2250>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_with_weights.plot_tree(tree_idx=0, pool=small_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that tree also reduces \"costly\" features usage. Metric result is also comparable with model without feature weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see that feature weights and penalties usage reduce overall feature strength of \"costly\" features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_fs_col_name(feature_idx):\n",
    "    return 'Feature strength {}'.format(feature_idx)\n",
    "\n",
    "fs_df = pd.DataFrame(\n",
    "    data=[\n",
    "        cb_wo_penalties.get_feature_importance(\n",
    "            type=catboost.EFstrType.PredictionValuesChange,\n",
    "            data=small_pool\n",
    "        ),\n",
    "        cb_with_penalties.get_feature_importance(\n",
    "            type=catboost.EFstrType.PredictionValuesChange,\n",
    "            data=small_pool\n",
    "        ),\n",
    "        cb_with_weights.get_feature_importance(\n",
    "            type=catboost.EFstrType.PredictionValuesChange,\n",
    "            data=small_pool\n",
    "        ),\n",
    "    ],\n",
    "    columns=list(map(get_fs_col_name, range(n_features))),\n",
    "    index=['Simple regressor', 'Regressor with feature penalties', 'Regressor with feature weights']\n",
    ")\n",
    "for idx in range(1, n_features):\n",
    "    fs_df['Relative feature strength {}'.format(idx)] = fs_df[get_fs_col_name(idx)] / fs_df[get_fs_col_name(0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature strength 0</th>\n",
       "      <th>Feature strength 1</th>\n",
       "      <th>Feature strength 2</th>\n",
       "      <th>Feature strength 3</th>\n",
       "      <th>Relative feature strenght 1</th>\n",
       "      <th>Relative feature strenght 2</th>\n",
       "      <th>Relative feature strenght 3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Simple regressor</td>\n",
       "      <td>13.090980</td>\n",
       "      <td>13.028802</td>\n",
       "      <td>25.600115</td>\n",
       "      <td>48.280102</td>\n",
       "      <td>0.995250</td>\n",
       "      <td>1.955554</td>\n",
       "      <td>3.688043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Regressor with feature penalties</td>\n",
       "      <td>60.241950</td>\n",
       "      <td>30.182175</td>\n",
       "      <td>9.568341</td>\n",
       "      <td>0.007533</td>\n",
       "      <td>0.501016</td>\n",
       "      <td>0.158832</td>\n",
       "      <td>0.000125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Regressor with feature weights</td>\n",
       "      <td>57.798581</td>\n",
       "      <td>23.163002</td>\n",
       "      <td>0.013350</td>\n",
       "      <td>19.025068</td>\n",
       "      <td>0.400754</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>0.329162</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  Feature strength 0  Feature strength 1  \\\n",
       "Simple regressor                           13.090980           13.028802   \n",
       "Regressor with feature penalties           60.241950           30.182175   \n",
       "Regressor with feature weights             57.798581           23.163002   \n",
       "\n",
       "                                  Feature strength 2  Feature strength 3  \\\n",
       "Simple regressor                           25.600115           48.280102   \n",
       "Regressor with feature penalties            9.568341            0.007533   \n",
       "Regressor with feature weights              0.013350           19.025068   \n",
       "\n",
       "                                  Relative feature strenght 1  \\\n",
       "Simple regressor                                     0.995250   \n",
       "Regressor with feature penalties                     0.501016   \n",
       "Regressor with feature weights                       0.400754   \n",
       "\n",
       "                                  Relative feature strenght 2  \\\n",
       "Simple regressor                                     1.955554   \n",
       "Regressor with feature penalties                     0.158832   \n",
       "Regressor with feature weights                       0.000231   \n",
       "\n",
       "                                  Relative feature strenght 3  \n",
       "Simple regressor                                     3.688043  \n",
       "Regressor with feature penalties                     0.000125  \n",
       "Regressor with feature weights                       0.329162  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fs_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}