house-prices-using-pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.018846,
     "end_time": "2023-03-24T19:55:59.282803",
     "exception": false,
     "start_time": "2023-03-24T19:55:59.263957",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction  <a id='introduction'></a>\n",
    "\n",
    "This is a  starter code for those who want to work with sklearn pipelines. The reader may quickly get a better score by applying a grid search, EDA, and feature engineering concepts. Please note that the score may change due to the indeterministic nature of the model.\n",
    "\n",
    "Thank you for reading.\n",
    "\n",
    "\n",
    "# Table of Contents\n",
    "* [Introduction](#introduction)\n",
    "* [Helper Functions](#functions)\n",
    "* [Preprocessing](#preprocessing) \n",
    "* [Validation](#validation) \n",
    "* [Cross-validation using full training set](#cross-validation)    \n",
    "* [Prediction](#prediction) \n",
    "* [Saving and submission](#saving)  \n",
    "* [References](#references)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2023-03-24T19:55:59.332529Z",
     "iopub.status.busy": "2023-03-24T19:55:59.327382Z",
     "iopub.status.idle": "2023-03-24T19:56:02.958703Z",
     "shell.execute_reply": "2023-03-24T19:56:02.958074Z",
     "shell.execute_reply.started": "2022-09-27T12:14:38.405784Z"
    },
    "papermill": {
     "duration": 3.657537,
     "end_time": "2023-03-24T19:56:02.958857",
     "exception": false,
     "start_time": "2023-03-24T19:55:59.301320",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/input/house-prices-advanced-regression-techniques/sample_submission.csv\n",
      "/kaggle/input/house-prices-advanced-regression-techniques/data_description.txt\n",
      "/kaggle/input/house-prices-advanced-regression-techniques/train.csv\n",
      "/kaggle/input/house-prices-advanced-regression-techniques/test.csv\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the read-only \"../input/\" directory\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under\n",
    "# the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as\n",
    "# output when you create a version using \"Save & Run All\" \n",
    "# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the\n",
    "# current session\n",
    "\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, make_scorer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler,OneHotEncoder, OrdinalEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline, FeatureUnion, make_pipeline\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from xgboost import XGBRegressor\n",
    "pd.set_option('display.max_columns', None)\n",
    "import category_encoders as ce\n",
    "from sklearn.feature_selection import SelectKBest, f_classif, f_regression\n",
    "from pandas_profiling import ProfileReport\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016203,
     "end_time": "2023-03-24T19:56:02.993822",
     "exception": false,
     "start_time": "2023-03-24T19:56:02.977619",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Helper functions   <a id='functions'></a>   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.015632,
     "end_time": "2023-03-24T19:56:03.025963",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.010331",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Tip:</b> We will use some helper functions throughout the notebook. Collecting them in one place is a good idea, making the code more organized.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.177790Z",
     "iopub.status.busy": "2023-03-24T19:56:03.176937Z",
     "iopub.status.idle": "2023-03-24T19:56:03.180118Z",
     "shell.execute_reply": "2023-03-24T19:56:03.179581Z",
     "shell.execute_reply.started": "2022-09-27T12:14:41.893312Z"
    },
    "papermill": {
     "duration": 0.138253,
     "end_time": "2023-03-24T19:56:03.180254",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.042001",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "def save_file (predictions):\n",
    "    \"\"\"Save submission file.\"\"\"\n",
    "    # Save test predictions to file\n",
    "    output = pd.DataFrame({'Id': sample_submission_file.Id,\n",
    "                       'SalePrice': predictions})\n",
    "    output.to_csv('submission.csv', index=False)\n",
    "    print (\"Submission file is saved\")\n",
    "\n",
    "def calculate_root_mean_squared_log_error(y_true, y_pred):\n",
    "    \"\"\"Calculate root mean squared error of log(y_true) and log(y_pred)\"\"\"\n",
    "    if len(y_pred)!=len(y_true): return 'error_mismatch'\n",
    "    y_pred_new = [math.log(x+1) for x in y_pred]\n",
    "    y_true_new = [math.log(x+1) for x in y_true]\n",
    "    return mean_squared_error(y_true_new, y_pred_new, squared=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.225987Z",
     "iopub.status.busy": "2023-03-24T19:56:03.225206Z",
     "iopub.status.idle": "2023-03-24T19:56:03.339485Z",
     "shell.execute_reply": "2023-03-24T19:56:03.338796Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.017017Z"
    },
    "papermill": {
     "duration": 0.142487,
     "end_time": "2023-03-24T19:56:03.339611",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.197124",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data is OK\n"
     ]
    }
   ],
   "source": [
    "# Read the data\n",
    "train_data = pd.read_csv('/kaggle/input/house-prices-advanced-regression-techniques/train.csv',\n",
    "                         index_col='Id')\n",
    "X_test = pd.read_csv('/kaggle/input/house-prices-advanced-regression-techniques/test.csv',\n",
    "                     index_col='Id')\n",
    "X = train_data.copy()\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X.SalePrice              \n",
    "X.drop(['SalePrice', 'Utilities'], axis=1, inplace=True)\n",
    "X_test.drop(['Utilities'], axis=1, inplace=True)\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,\n",
    "                                                                random_state=0)\n",
    "\n",
    "sample_submission_file = pd.read_csv(\"/kaggle/input/house-prices-advanced-regression-techniques/sample_submission.csv\")\n",
    "\n",
    "print('Data is OK')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016691,
     "end_time": "2023-03-24T19:56:03.372854",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.356163",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Preprocessing  <a id='preprocessing'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.420548Z",
     "iopub.status.busy": "2023-03-24T19:56:03.419845Z",
     "iopub.status.idle": "2023-03-24T19:56:03.426827Z",
     "shell.execute_reply": "2023-03-24T19:56:03.426245Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.145044Z"
    },
    "papermill": {
     "duration": 0.037634,
     "end_time": "2023-03-24T19:56:03.426961",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.389327",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing values in each column:\n",
      "LotFrontage      212\n",
      "Alley           1097\n",
      "MasVnrType         6\n",
      "MasVnrArea         6\n",
      "BsmtQual          28\n",
      "BsmtCond          28\n",
      "BsmtExposure      28\n",
      "BsmtFinType1      28\n",
      "BsmtFinType2      29\n",
      "Electrical         1\n",
      "FireplaceQu      551\n",
      "GarageType        58\n",
      "GarageYrBlt       58\n",
      "GarageFinish      58\n",
      "GarageQual        58\n",
      "GarageCond        58\n",
      "PoolQC          1164\n",
      "Fence            954\n",
      "MiscFeature     1119\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Select object columns\n",
    "categorical_cols = [cname for cname in X_train.columns if X_train[cname].dtype == \"object\"]\n",
    "\n",
    "# Select numeric columns\n",
    "numerical_cols = [cname for cname in X_train.columns if X_train[cname].dtype in ['int64',\n",
    "                                                                                 'float64']]\n",
    "\n",
    "# Number of missing values in each column of training data\n",
    "missing_val_count_by_column_train = (X_train.isnull().sum())\n",
    "print(\"Number of missing values in each column:\")\n",
    "print(missing_val_count_by_column_train[missing_val_count_by_column_train > 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.469131Z",
     "iopub.status.busy": "2023-03-24T19:56:03.467600Z",
     "iopub.status.idle": "2023-03-24T19:56:03.473606Z",
     "shell.execute_reply": "2023-03-24T19:56:03.472927Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.167706Z"
    },
    "papermill": {
     "duration": 0.02996,
     "end_time": "2023-03-24T19:56:03.473735",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.443775",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing values in numerical columns:\n",
      "LotFrontage    212\n",
      "MasVnrArea       6\n",
      "GarageYrBlt     58\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Number of missing values in numerical columns\n",
    "missing_val_count_by_column_numeric = (X_train[numerical_cols].isnull().sum())\n",
    "print(\"Number of missing values in numerical columns:\")\n",
    "print(missing_val_count_by_column_numeric[missing_val_count_by_column_numeric > 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.515375Z",
     "iopub.status.busy": "2023-03-24T19:56:03.514665Z",
     "iopub.status.idle": "2023-03-24T19:56:03.517724Z",
     "shell.execute_reply": "2023-03-24T19:56:03.517155Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.182795Z"
    },
    "papermill": {
     "duration": 0.02715,
     "end_time": "2023-03-24T19:56:03.517854",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.490704",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Imputation lists\n",
    "\n",
    "# imputation to null values of these numerical columns need to be 'constant'\n",
    "constant_num_cols = ['GarageYrBlt', 'MasVnrArea']\n",
    "\n",
    "# imputation to null values of these numerical columns need to be 'mean'\n",
    "mean_num_cols = list(set(numerical_cols).difference(set(constant_num_cols)))\n",
    "\n",
    "# imputation to null values of these categorical columns need to be 'constant'\n",
    "constant_categorical_cols = ['Alley', 'MasVnrType', 'BsmtQual', 'BsmtCond', 'BsmtExposure',\n",
    "                             'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu','GarageType',\n",
    "                             'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence',\n",
    "                             'MiscFeature']\n",
    "\n",
    "# imputation to null values of these categorical columns need to be 'most_frequent'\n",
    "mf_categorical_cols = list(set(categorical_cols).difference(set(constant_categorical_cols)))\n",
    "\n",
    "my_cols = constant_num_cols + mean_num_cols + constant_categorical_cols + mf_categorical_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.563075Z",
     "iopub.status.busy": "2023-03-24T19:56:03.562384Z",
     "iopub.status.idle": "2023-03-24T19:56:03.565803Z",
     "shell.execute_reply": "2023-03-24T19:56:03.565082Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.192813Z"
    },
    "papermill": {
     "duration": 0.03106,
     "end_time": "2023-03-24T19:56:03.565936",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.534876",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define transformers\n",
    "# Preprocessing for numerical data\n",
    "\n",
    "numerical_transformer_m = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='mean')),\n",
    "    ('scaler', StandardScaler())])\n",
    "\n",
    "numerical_transformer_c = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value=0)),\n",
    "    ('scaler', StandardScaler())])\n",
    "\n",
    "\n",
    "# Preprocessing for categorical data for most frequent\n",
    "categorical_transformer_mf = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown = 'ignore', sparse = False))\n",
    "])\n",
    "\n",
    "# Preprocessing for categorical data for constant\n",
    "categorical_transformer_c = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value='NA')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown = 'ignore', sparse = False))\n",
    "])\n",
    "\n",
    "\n",
    "# Bundle preprocessing for numerical and categorical data\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num_mean', numerical_transformer_m, mean_num_cols),\n",
    "        ('num_constant', numerical_transformer_c, constant_num_cols),\n",
    "        ('cat_mf', categorical_transformer_mf, mf_categorical_cols),\n",
    "        ('cat_c', categorical_transformer_c, constant_categorical_cols)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.607873Z",
     "iopub.status.busy": "2023-03-24T19:56:03.606866Z",
     "iopub.status.idle": "2023-03-24T19:56:03.609921Z",
     "shell.execute_reply": "2023-03-24T19:56:03.609230Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.206741Z"
    },
    "papermill": {
     "duration": 0.027195,
     "end_time": "2023-03-24T19:56:03.610062",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.582867",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define Model\n",
    "model = XGBRegressor(learning_rate = 0.01,\n",
    "                            n_estimators=2500,\n",
    "                            max_depth=4,\n",
    "                            min_child_weight=1,\n",
    "                            gamma=0,\n",
    "                            subsample=0.7,\n",
    "                            colsample_bytree=0.6,\n",
    "                            reg_alpha = 0.1,\n",
    "                            reg_lambda = 1.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.01717,
     "end_time": "2023-03-24T19:56:03.644655",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.627485",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Validation with early_stopping_rounds  <a id='validation'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016917,
     "end_time": "2023-03-24T19:56:03.678646",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.661729",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<div class=\"alert alert-block alert-danger\">  \n",
    "<p>If we want to use early_stopping_rounds with our pipeline, we cannot use the validation set (X_valid) directly. This is because sklearn pipelines do not process the eval_set used with early_stopping_rounds. As a result, we need to process our validation set before using early_stopping_rounds.</p> \n",
    "\n",
    "<p>There is a great danger here. If we forget to process our validation set and if processed data has the same number of columns as the unprocessed data, we may not see an error. Validation with unprocessed data may mislead us.</p>\n",
    "\n",
    "<p>To process the eval_set, we need to fit_transform X_valid by using our preprocessor, a pipeline consisting of transformers that do not have an estimator.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.724043Z",
     "iopub.status.busy": "2023-03-24T19:56:03.723287Z",
     "iopub.status.idle": "2023-03-24T19:56:03.820292Z",
     "shell.execute_reply": "2023-03-24T19:56:03.819552Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.221827Z"
    },
    "papermill": {
     "duration": 0.124153,
     "end_time": "2023-03-24T19:56:03.820420",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.696267",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Preprocessing of validation data\n",
    "X_valid_eval = preprocessor.fit(X_train, y_train).transform (X_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.860973Z",
     "iopub.status.busy": "2023-03-24T19:56:03.860017Z",
     "iopub.status.idle": "2023-03-24T19:56:03.863565Z",
     "shell.execute_reply": "2023-03-24T19:56:03.864107Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.337226Z"
    },
    "papermill": {
     "duration": 0.026766,
     "end_time": "2023-03-24T19:56:03.864283",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.837517",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 296 features after transformation\n"
     ]
    }
   ],
   "source": [
    "# Display the number of remaining columns after transformation \n",
    "print(\"We have\", X_valid_eval.shape[1], \"features after transformation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.905239Z",
     "iopub.status.busy": "2023-03-24T19:56:03.904395Z",
     "iopub.status.idle": "2023-03-24T19:56:03.906900Z",
     "shell.execute_reply": "2023-03-24T19:56:03.907370Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.344714Z"
    },
    "papermill": {
     "duration": 0.025712,
     "end_time": "2023-03-24T19:56:03.907547",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.881835",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define XGBRegressor fitting parameters for the pipeline\n",
    "fit_params = {\"model__early_stopping_rounds\": 50,\n",
    "              \"model__eval_set\": [(X_valid_eval, y_valid)],\n",
    "              \"model__verbose\": False,\n",
    "              \"model__eval_metric\" : \"rmsle\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:03.955803Z",
     "iopub.status.busy": "2023-03-24T19:56:03.955046Z",
     "iopub.status.idle": "2023-03-24T19:56:09.026927Z",
     "shell.execute_reply": "2023-03-24T19:56:09.027759Z",
     "shell.execute_reply.started": "2022-09-27T12:14:42.358053Z"
    },
    "papermill": {
     "duration": 5.102909,
     "end_time": "2023-03-24T19:56:09.027967",
     "exception": false,
     "start_time": "2023-03-24T19:56:03.925058",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.1240876772999255\n"
     ]
    }
   ],
   "source": [
    "# Create and Evaluate the Pipeline\n",
    "# Bundle preprocessing and modeling code in a pipeline\n",
    "my_pipeline = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                              ('model', model)\n",
    "                             ])\n",
    "\n",
    "# Preprocessing of training data, fit model \n",
    "my_pipeline.fit(X_train, y_train, **fit_params)\n",
    "\n",
    "# Get predictions\n",
    "preds = my_pipeline.predict(X_valid)\n",
    "\n",
    "# Evaluate the model\n",
    "score = calculate_root_mean_squared_log_error(y_valid,preds)\n",
    "\n",
    "print(\"Score: {}\".format(score))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.022503,
     "end_time": "2023-03-24T19:56:09.073376",
     "exception": false,
     "start_time": "2023-03-24T19:56:09.050873",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Cross-validation using full training set <a id='cross-validation'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:09.133636Z",
     "iopub.status.busy": "2023-03-24T19:56:09.132784Z",
     "iopub.status.idle": "2023-03-24T19:56:09.135009Z",
     "shell.execute_reply": "2023-03-24T19:56:09.135792Z",
     "shell.execute_reply.started": "2022-09-27T12:14:50.162233Z"
    },
    "papermill": {
     "duration": 0.038696,
     "end_time": "2023-03-24T19:56:09.135979",
     "exception": false,
     "start_time": "2023-03-24T19:56:09.097283",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Preprocessing of training data, fit model \n",
    "X_cv = X[my_cols].copy()\n",
    "X_sub = X_test[my_cols].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:09.194726Z",
     "iopub.status.busy": "2023-03-24T19:56:09.193344Z",
     "iopub.status.idle": "2023-03-24T19:56:58.073224Z",
     "shell.execute_reply": "2023-03-24T19:56:58.073887Z",
     "shell.execute_reply.started": "2022-09-27T12:14:50.178072Z"
    },
    "papermill": {
     "duration": 48.915273,
     "end_time": "2023-03-24T19:56:58.074083",
     "exception": false,
     "start_time": "2023-03-24T19:56:09.158810",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score:\n",
      " [0.10659199 0.1361177  0.1259693  0.10805361 0.12018209]\n",
      "Score mean: 0.11938293745885334\n",
      "Score std: 0.011099849692667089\n"
     ]
    }
   ],
   "source": [
    "# Multiply by -1 since sklearn calculates *negative* MAE\n",
    "scores = -1 * cross_val_score(my_pipeline, X_cv, y,\n",
    "                              cv=5,\n",
    "                              scoring=make_scorer(calculate_root_mean_squared_log_error,\n",
    "                                                  greater_is_better=False))\n",
    "\n",
    "print(\"Score:\\n\", scores)\n",
    "print(\"Score mean: {}\".format(scores.mean()))\n",
    "print(\"Score std: {}\".format(scores.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.022962,
     "end_time": "2023-03-24T19:56:58.121107",
     "exception": false,
     "start_time": "2023-03-24T19:56:58.098145",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Prediction   <a id='prediction'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:56:58.176733Z",
     "iopub.status.busy": "2023-03-24T19:56:58.175858Z",
     "iopub.status.idle": "2023-03-24T19:57:10.899141Z",
     "shell.execute_reply": "2023-03-24T19:57:10.899797Z",
     "shell.execute_reply.started": "2022-09-27T12:15:40.059942Z"
    },
    "papermill": {
     "duration": 12.754816,
     "end_time": "2023-03-24T19:57:10.900027",
     "exception": false,
     "start_time": "2023-03-24T19:56:58.145211",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fit model\n",
    "my_pipeline.fit(X_cv, y)\n",
    "\n",
    "# Get predictions\n",
    "preds = my_pipeline.predict(X_sub)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.022838,
     "end_time": "2023-03-24T19:57:10.947074",
     "exception": false,
     "start_time": "2023-03-24T19:57:10.924236",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Saving and submission   <a id='saving'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-24T19:57:11.002303Z",
     "iopub.status.busy": "2023-03-24T19:57:11.001272Z",
     "iopub.status.idle": "2023-03-24T19:57:11.287475Z",
     "shell.execute_reply": "2023-03-24T19:57:11.287959Z",
     "shell.execute_reply.started": "2022-09-27T12:15:52.547121Z"
    },
    "papermill": {
     "duration": 0.317994,
     "end_time": "2023-03-24T19:57:11.288128",
     "exception": false,
     "start_time": "2023-03-24T19:57:10.970134",
     "status": "completed"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission file is saved\n"
     ]
    }
   ],
   "source": [
    "# Use predefined utility function\n",
    "save_file(preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.019246,
     "end_time": "2023-03-24T19:57:11.326587",
     "exception": false,
     "start_time": "2023-03-24T19:57:11.307341",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# References   <a id='references'></a>\n",
    "* [10-simple-hacks-to-speed-up-your-data-analysis - Parul Pandey](https://www.kaggle.com/parulpandey/10-simple-hacks-to-speed-up-your-data-analysis)\n",
    "* [Dataset Transformations - Scikit-learn](https://scikit-learn.org/stable/data_transforms.html)\n",
    "* [Intermediate Machine Learning Course - Pipelines](https://www.kaggle.com/alexisbcook/pipelines)\n",
    "* [Kaggle Learn](https://www.kaggle.com/learn/overview)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "papermill": {
   "duration": 77.173108,
   "end_time": "2023-03-24T19:57:11.452792",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-03-24T19:55:54.279684",
   "version": "2.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}