src/tests/test_deepchecks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deepchecks Testing using the Tabular Sub-Package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we assess our data using the Deepchecks library, specifically utilizing the Tabular Sub-Package for data inspection.\n",
    "\n",
    "We first import the needed libraries. If errors related to the deepchecks libray occur, we suggest to install the deepchecks library with pip in your virtual environment. Here is the command you need to run: pip install deepchecks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install deepchecks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: deepchecks in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (0.17.5)\n",
      "Requirement already satisfied: pandas<2.2.0,>=1.1.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (2.1.1)\n",
      "Requirement already satisfied: numpy>=1.19 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (1.26.1)\n",
      "Requirement already satisfied: scikit-learn>=0.23.2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (1.3.1)\n",
      "Requirement already satisfied: jsonpickle>=2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (3.0.2)\n",
      "Requirement already satisfied: PyNomaly>=0.3.3 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (0.3.3)\n",
      "Requirement already satisfied: typing-extensions>=4.0.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (4.8.0)\n",
      "Requirement already satisfied: tqdm>=4.62.3 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (4.66.1)\n",
      "Requirement already satisfied: category-encoders>=2.3.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (2.6.3)\n",
      "Requirement already satisfied: scipy<=1.10.1,>=1.4.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (1.10.1)\n",
      "Requirement already satisfied: plotly>=5.13.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (5.17.0)\n",
      "Requirement already satisfied: matplotlib>=3.3.4 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (3.8.0)\n",
      "Requirement already satisfied: pyzmq<24.0.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (23.2.1)\n",
      "Requirement already satisfied: beautifulsoup4>=4.11.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (4.12.2)\n",
      "Requirement already satisfied: requests>=2.22.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (2.31.0)\n",
      "Requirement already satisfied: statsmodels>=0.13.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (0.14.0)\n",
      "Requirement already satisfied: ipython<8,>=7.15.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (7.34.0)\n",
      "Requirement already satisfied: ipykernel>=5.3.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (6.26.0)\n",
      "Requirement already satisfied: ipywidgets<8,>=7.6.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from deepchecks) (7.8.1)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from beautifulsoup4>=4.11.1->deepchecks) (2.5)\n",
      "Requirement already satisfied: patsy>=0.5.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from category-encoders>=2.3.0->deepchecks) (0.5.3)\n",
      "Requirement already satisfied: comm>=0.1.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (0.2.0)\n",
      "Requirement already satisfied: debugpy>=1.6.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (1.8.0)\n",
      "Requirement already satisfied: jupyter-client>=6.1.12 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (7.4.9)\n",
      "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (5.5.0)\n",
      "Requirement already satisfied: matplotlib-inline>=0.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (0.1.6)\n",
      "Requirement already satisfied: nest-asyncio in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (1.5.8)\n",
      "Requirement already satisfied: packaging in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (23.2)\n",
      "Requirement already satisfied: psutil in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (5.9.6)\n",
      "Requirement already satisfied: tornado>=6.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (6.3.3)\n",
      "Requirement already satisfied: traitlets>=5.4.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipykernel>=5.3.0->deepchecks) (5.13.0)\n",
      "Requirement already satisfied: setuptools>=18.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (65.5.0)\n",
      "Requirement already satisfied: jedi>=0.16 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (0.19.1)\n",
      "Requirement already satisfied: decorator in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (5.1.1)\n",
      "Requirement already satisfied: pickleshare in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (3.0.39)\n",
      "Requirement already satisfied: pygments in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (2.16.1)\n",
      "Requirement already satisfied: backcall in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (0.2.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipython<8,>=7.15.0->deepchecks) (0.4.6)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipywidgets<8,>=7.6.5->deepchecks) (0.2.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.6.6 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipywidgets<8,>=7.6.5->deepchecks) (3.6.6)\n",
      "Requirement already satisfied: jupyterlab-widgets<3,>=1.0.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from ipywidgets<8,>=7.6.5->deepchecks) (1.1.7)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (1.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (4.44.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (1.4.5)\n",
      "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (10.1.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (3.1.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from matplotlib>=3.3.4->deepchecks) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from pandas<2.2.0,>=1.1.5->deepchecks) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from pandas<2.2.0,>=1.1.5->deepchecks) (2023.3)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from plotly>=5.13.1->deepchecks) (8.2.3)\n",
      "Requirement already satisfied: python-utils in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from PyNomaly>=0.3.3->deepchecks) (3.8.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from requests>=2.22.0->deepchecks) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from requests>=2.22.0->deepchecks) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from requests>=2.22.0->deepchecks) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from requests>=2.22.0->deepchecks) (2023.7.22)\n",
      "Requirement already satisfied: joblib>=1.1.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from scikit-learn>=0.23.2->deepchecks) (1.3.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from scikit-learn>=0.23.2->deepchecks) (3.2.0)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jedi>=0.16->ipython<8,>=7.15.0->deepchecks) (0.8.3)\n",
      "Requirement already satisfied: entrypoints in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jupyter-client>=6.1.12->ipykernel>=5.3.0->deepchecks) (0.4)\n",
      "Requirement already satisfied: platformdirs>=2.5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel>=5.3.0->deepchecks) (3.11.0)\n",
      "Requirement already satisfied: pywin32>=300 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel>=5.3.0->deepchecks) (306)\n",
      "Requirement already satisfied: six in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from patsy>=0.5.1->category-encoders>=2.3.0->deepchecks) (1.16.0)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8,>=7.15.0->deepchecks) (0.2.9)\n",
      "Requirement already satisfied: notebook>=4.4.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (6.5.6)\n",
      "Requirement already satisfied: jinja2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (3.1.2)\n",
      "Requirement already satisfied: argon2-cffi in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (23.1.0)\n",
      "Requirement already satisfied: nbformat in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (5.9.2)\n",
      "Requirement already satisfied: nbconvert>=5 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (6.5.4)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.8.2)\n",
      "Requirement already satisfied: terminado>=0.8.3 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.17.1)\n",
      "Requirement already satisfied: prometheus-client in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.18.0)\n",
      "Requirement already satisfied: nbclassic>=0.4.7 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.0.0)\n",
      "Requirement already satisfied: jupyter-server>=1.8 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.24.0)\n",
      "Requirement already satisfied: notebook-shim>=0.2.3 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.2.3)\n",
      "Requirement already satisfied: lxml in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (4.9.3)\n",
      "Requirement already satisfied: bleach in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (6.1.0)\n",
      "Requirement already satisfied: defusedxml in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.7.1)\n",
      "Requirement already satisfied: jupyterlab-pygments in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.2.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (2.1.3)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.8.4)\n",
      "Requirement already satisfied: nbclient>=0.5.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.9.0)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.5.0)\n",
      "Requirement already satisfied: tinycss2 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.2.1)\n",
      "Requirement already satisfied: fastjsonschema in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (2.18.1)\n",
      "Requirement already satisfied: jsonschema>=2.6 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (4.19.2)\n",
      "Requirement already satisfied: pywinpty>=1.1.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (2.0.12)\n",
      "Requirement already satisfied: argon2-cffi-bindings in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (21.2.0)\n",
      "Requirement already satisfied: attrs>=22.2.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (23.1.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (2023.7.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.30.2)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.12.0)\n",
      "Requirement already satisfied: anyio<4,>=3.1.0 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (3.7.1)\n",
      "Requirement already satisfied: websocket-client in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.6.4)\n",
      "Requirement already satisfied: cffi>=1.0.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.16.0)\n",
      "Requirement already satisfied: webencodings in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (0.5.1)\n",
      "Requirement already satisfied: sniffio>=1.1 in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (1.3.0)\n",
      "Requirement already satisfied: pycparser in c:\\users\\ivanr\\desktop\\university\\master\\secondyear\\firstsemester\\se4ai\\project\\musicexpress\\env\\lib\\site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.6->ipywidgets<8,>=7.6.5->deepchecks) (2.21)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.2.1 -> 23.3.1\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "from deepchecks.tabular import Dataset\n",
    "from deepchecks.tabular.suites import data_integrity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we define variables for later use in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_directory = os.getcwd()\n",
    "current_directory = os.path.dirname(current_directory)\n",
    "curr_dir_path = os.path.dirname(current_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Raw Data\n",
    "train_data_directory = curr_dir_path + '\\\\data\\\\interim\\\\keep grinding..csv'\n",
    "test_data_directory = curr_dir_path + '\\\\data\\\\interim\\\\Spotify\\'s Most Played All-Time [Updated Weekly]  Most Streamed  Top Played  500Mil+.csv'\n",
    "\n",
    "TRAIN_REPORT_PATH = curr_dir_path + '\\\\reports\\\\deepchecks\\\\QA_suite_training_report.html'\n",
    "TEST_REPORT_PATH = curr_dir_path + '\\\\reports\\\\deepchecks\\\\QA_suite_test_report.html'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Processed Data\n",
    "processed_train_data_directory = curr_dir_path + '\\\\data\\\\processed\\\\trainSet.csv'\n",
    "processed_test_data_directory = curr_dir_path + '\\\\data\\\\processed\\\\testSet.csv'\n",
    "\n",
    "PROCESSED_TRAIN_REPORT_PATH = curr_dir_path + '\\\\reports\\\\deepchecks\\\\QA_suite_processed_training_report.html'\n",
    "PROCESSED_TEST_REPORT_PATH = curr_dir_path + '\\\\reports\\\\deepchecks\\\\QA_suite_processed_test_report.html'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Raw Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick dataset overview before proceeding to the test suite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Artist</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Liveness</th>\n",
       "      <th>Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Serotonin</td>\n",
       "      <td>Call Me Karizma</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.159</td>\n",
       "      <td>-9.299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Joy</td>\n",
       "      <td>Thornhill</td>\n",
       "      <td>0.910</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-3.417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Like</td>\n",
       "      <td>Alissic</td>\n",
       "      <td>0.423</td>\n",
       "      <td>0.143</td>\n",
       "      <td>-8.773</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Name           Artist  Energy  Liveness  Loudness\n",
       "0  Serotonin  Call Me Karizma   0.418     0.159    -9.299\n",
       "1        Joy        Thornhill   0.910     0.252    -3.417\n",
       "2       Like          Alissic   0.423     0.143    -8.773"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv(train_data_directory)\n",
    "df_train.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Artist</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Liveness</th>\n",
       "      <th>Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blinding Lights</td>\n",
       "      <td>The Weeknd</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.0897</td>\n",
       "      <td>-5.934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shape of You</td>\n",
       "      <td>Ed Sheeran</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.0931</td>\n",
       "      <td>-3.183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dance Monkey</td>\n",
       "      <td>Tones And I</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.1490</td>\n",
       "      <td>-6.400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Name       Artist  Energy  Liveness  Loudness\n",
       "0  Blinding Lights   The Weeknd   0.730    0.0897    -5.934\n",
       "1     Shape of You   Ed Sheeran   0.652    0.0931    -3.183\n",
       "2     Dance Monkey  Tones And I   0.588    0.1490    -6.400"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.read_csv(test_data_directory)\n",
    "df_test.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the 'Name' and 'Artist' columns as Categorial Features as requested by the Deepchecks library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['Name'] = df_train['Name'].astype('category')\n",
    "df_train['Artist'] = df_train['Artist'].astype('category')\n",
    "df_test['Name'] = df_test['Name'].astype('category')\n",
    "df_test['Artist'] = df_test['Artist'].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the Dataset method, we convert the pandas dataframes into the required data structure for the test suite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4><b>Dataset Description</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Column</th>\n",
       "      <th style=\"min-width: 15px;\">DType</th>\n",
       "      <th style=\"min-width: 15px;\">Kind</th>\n",
       "      <th style=\"min-width: 15px;\">Additional Info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Name</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Artist</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Energy</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Liveness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loudness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><h4><b>Dataset Content</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Name</th>\n",
       "      <th style=\"min-width: 15px;\">Artist</th>\n",
       "      <th style=\"min-width: 15px;\">Energy</th>\n",
       "      <th style=\"min-width: 15px;\">Liveness</th>\n",
       "      <th style=\"min-width: 15px;\">Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Serotonin</td>\n",
       "      <td>Call Me Karizma</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.159</td>\n",
       "      <td>-9.299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Joy</td>\n",
       "      <td>Thornhill</td>\n",
       "      <td>0.910</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-3.417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Like</td>\n",
       "      <td>Alissic</td>\n",
       "      <td>0.423</td>\n",
       "      <td>0.143</td>\n",
       "      <td>-8.773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Can You Afford to Be An Individual?</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.859</td>\n",
       "      <td>0.127</td>\n",
       "      <td>-6.103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>This Feels Like the End</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.899</td>\n",
       "      <td>0.241</td>\n",
       "      <td>-7.082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>605</th>\n",
       "      <td>Tomorrow Is Closed</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.835</td>\n",
       "      <td>0.155</td>\n",
       "      <td>-4.695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>606</th>\n",
       "      <td>Do You Love Me Yet?</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.801</td>\n",
       "      <td>0.108</td>\n",
       "      <td>-6.183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>607</th>\n",
       "      <td>Welcome to the DCC</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.129</td>\n",
       "      <td>-4.759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>608</th>\n",
       "      <td>CODE MISTAKE</td>\n",
       "      <td>CORPSE</td>\n",
       "      <td>0.967</td>\n",
       "      <td>0.203</td>\n",
       "      <td>-3.607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>609</th>\n",
       "      <td>Kingdom Of Cards</td>\n",
       "      <td>Bad Omens</td>\n",
       "      <td>0.873</td>\n",
       "      <td>0.136</td>\n",
       "      <td>-5.802</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_train = Dataset(df_train, cat_features=['Name','Artist'])\n",
    "ds_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4><b>Dataset Description</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Column</th>\n",
       "      <th style=\"min-width: 15px;\">DType</th>\n",
       "      <th style=\"min-width: 15px;\">Kind</th>\n",
       "      <th style=\"min-width: 15px;\">Additional Info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Name</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Artist</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Energy</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Liveness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loudness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><h4><b>Dataset Content</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Name</th>\n",
       "      <th style=\"min-width: 15px;\">Artist</th>\n",
       "      <th style=\"min-width: 15px;\">Energy</th>\n",
       "      <th style=\"min-width: 15px;\">Liveness</th>\n",
       "      <th style=\"min-width: 15px;\">Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blinding Lights</td>\n",
       "      <td>The Weeknd</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.0897</td>\n",
       "      <td>-5.934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shape of You</td>\n",
       "      <td>Ed Sheeran</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.0931</td>\n",
       "      <td>-3.183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dance Monkey</td>\n",
       "      <td>Tones And I</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.1490</td>\n",
       "      <td>-6.400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Someone You Loved</td>\n",
       "      <td>Lewis Capaldi</td>\n",
       "      <td>0.405</td>\n",
       "      <td>0.1050</td>\n",
       "      <td>-5.679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sunflower - Spider-Man: Into the Spider-Verse</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>0.522</td>\n",
       "      <td>0.0685</td>\n",
       "      <td>-4.368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>495</th>\n",
       "      <td>Hey There Delilah</td>\n",
       "      <td>Plain White T's</td>\n",
       "      <td>0.291</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>-10.572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>idontwannabeyouanymore</td>\n",
       "      <td>Billie Eilish</td>\n",
       "      <td>0.412</td>\n",
       "      <td>0.1160</td>\n",
       "      <td>-8.461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>Zombie</td>\n",
       "      <td>The Cranberries</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.3660</td>\n",
       "      <td>-8.957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>No Lie</td>\n",
       "      <td>Sean Paul</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.2060</td>\n",
       "      <td>-2.862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>Candy Paint</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>0.667</td>\n",
       "      <td>0.0753</td>\n",
       "      <td>-5.941</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_test = Dataset(df_test, cat_features=['Name','Artist'])\n",
    "ds_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This test suite involves eleven tests, although we retain only the most relevant tests, including:\n",
    "\n",
    "- **Mixed Data Type**: scrutinizes data type variations within a dataset, pinpointing columns or fields with mixed data types, ensuring data consistency.\n",
    "\n",
    "- **Mixed Nulls**: examines the presence of null or missing values in the dataset, with a specific focus on columns where nulls are intertwined with non-null values.\n",
    "\n",
    "- **Data Duplicates**: assesses the dataset for duplicate records, identifying instances where identical data points occur more than once.\n",
    "\n",
    "These tests offer a comprehensive analysis of data quality, assisting us in identifying potential issues and enhancing data reliability for song recommendations.\n",
    "\n",
    "Yet, we show all the included tests before removing the ones we do not need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data Integrity Suite: [\n",
       "\t0: IsSingleValue\n",
       "\t\tConditions:\n",
       "\t\t\t0: Does not contain only a single value\n",
       "\t1: SpecialCharacters\n",
       "\t\tConditions:\n",
       "\t\t\t0: Ratio of samples containing solely special character is less or equal to 0.1%\n",
       "\t2: MixedNulls\n",
       "\t\tConditions:\n",
       "\t\t\t0: Number of different null types is less or equal to 1\n",
       "\t3: MixedDataTypes\n",
       "\t\tConditions:\n",
       "\t\t\t0: Rare data types in column are either more than 10% or less than 1% of the data\n",
       "\t4: StringMismatch\n",
       "\t\tConditions:\n",
       "\t\t\t0: No string variants\n",
       "\t5: DataDuplicates\n",
       "\t\tConditions:\n",
       "\t\t\t0: Duplicate data ratio is less or equal to 5%\n",
       "\t6: StringLengthOutOfBounds\n",
       "\t\tConditions:\n",
       "\t\t\t0: Ratio of string length outliers is less or equal to 0%\n",
       "\t7: ConflictingLabels\n",
       "\t\tConditions:\n",
       "\t\t\t0: Ambiguous sample ratio is less or equal to 0%\n",
       "\t8: OutlierSampleDetection\n",
       "\t9: FeatureLabelCorrelation(ppscore_params={}, random_state=42)\n",
       "\t\tConditions:\n",
       "\t\t\t0: Features' Predictive Power Score is less than 0.8\n",
       "\t10: FeatureFeatureCorrelation\n",
       "\t\tConditions:\n",
       "\t\t\t0: Not more than 0 pairs are correlated above 0.9\n",
       "\t11: IdentifierLabelCorrelation(ppscore_params={})\n",
       "\t\tConditions:\n",
       "\t\t\t0: Identifier columns PPS is less or equal to 0\n",
       "]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "suite = data_integrity()\n",
    "suite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we remove all the tests that we do not need, leaving only the ones listed before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data Integrity Suite: [\n",
       "\t2: MixedNulls\n",
       "\t\tConditions:\n",
       "\t\t\t0: Number of different null types is less or equal to 1\n",
       "\t3: MixedDataTypes\n",
       "\t\tConditions:\n",
       "\t\t\t0: Rare data types in column are either more than 10% or less than 1% of the data\n",
       "\t5: DataDuplicates\n",
       "\t\tConditions:\n",
       "\t\t\t0: Duplicate data ratio is less or equal to 5%\n",
       "]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "suite.remove(0) #removed check if does not contain only a single value\n",
    "suite.remove(1) #removed check Ratio of samples containing solely special character is less or equal to 0.1%\n",
    "suite.remove(4) #removed check if there is No string variants\n",
    "suite.remove(6) #removed check if Ratio of string length outliers is less or equal to 0% \n",
    "suite.remove(7) #removed check if  Ambiguous sample ratio is less or equal to 0%\n",
    "suite.remove(8) #removed check OutlierSampleDetection\n",
    "suite.remove(9) #removed check if Features' Predictive Power Score is less than 0.8\n",
    "suite.remove(10) #removed check if Not more than 0 pairs are correlated above 0.9\n",
    "suite.remove(11) #removed check if Identifier columns PPS is less or equal to 0\n",
    "suite "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now execute the suite on the raw train dataset, and the Deepchecks UI presents a summary of the results for each test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "        progress {\n",
       "            -webkit-appearance: none;\n",
       "            border: none;\n",
       "            border-radius: 3px;\n",
       "            width: 300px;\n",
       "            height: 20px;\n",
       "            vertical-align: middle;\n",
       "            margin-right: 10px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-bar {\n",
       "            border-radius: 3px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-value {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "        progress::-moz-progress-bar {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "    </style>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77287ddda21d4217a00388ff26b8103c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(HTML(value='\\n<h1 id=\"summary_CQYRQRKO20BOHAP4NTQSZ1LO1\">Data Integrity Sui…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "suite_training = suite.run(ds_train)\n",
    "suite_training.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the report as an HTML file in the dedicated folder. If it already exists, we delete the previous one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c:\\\\Users\\\\ivanr\\\\Desktop\\\\University\\\\Master\\\\SecondYear\\\\FirstSemester\\\\SE4AI\\\\project\\\\MusicExpress\\\\reports\\\\deepchecks\\\\QA_suite_training_report.html'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if os.path.exists(TRAIN_REPORT_PATH):\n",
    "    os.remove(TRAIN_REPORT_PATH)\n",
    "\n",
    "suite_training.save_as_html(TRAIN_REPORT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same pipeline is executed on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "        progress {\n",
       "            -webkit-appearance: none;\n",
       "            border: none;\n",
       "            border-radius: 3px;\n",
       "            width: 300px;\n",
       "            height: 20px;\n",
       "            vertical-align: middle;\n",
       "            margin-right: 10px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-bar {\n",
       "            border-radius: 3px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-value {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "        progress::-moz-progress-bar {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "    </style>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae6154ae9f5f41be98d8151ca309916b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(HTML(value='\\n<h1 id=\"summary_VWLZ6A6OBTGN34H4U72M51DKP\">Data Integrity Sui…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "suite_test = suite.run(ds_test)\n",
    "suite_test.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c:\\\\Users\\\\ivanr\\\\Desktop\\\\University\\\\Master\\\\SecondYear\\\\FirstSemester\\\\SE4AI\\\\project\\\\MusicExpress\\\\reports\\\\deepchecks\\\\QA_suite_test_report.html'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if os.path.exists(TEST_REPORT_PATH):\n",
    "    os.remove(TEST_REPORT_PATH)\n",
    "\n",
    "suite_test.save_as_html(TEST_REPORT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processed Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat the previously mentioned steps on the processed data, avoiding redundant descriptions since they have already been presented earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Artist</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Liveness</th>\n",
       "      <th>Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Serotonin</td>\n",
       "      <td>Call Me Karizma</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.159</td>\n",
       "      <td>0.671575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Joy</td>\n",
       "      <td>Thornhill</td>\n",
       "      <td>0.910</td>\n",
       "      <td>0.252</td>\n",
       "      <td>0.897120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Like</td>\n",
       "      <td>Alissic</td>\n",
       "      <td>0.423</td>\n",
       "      <td>0.143</td>\n",
       "      <td>0.691744</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Name           Artist  Energy  Liveness  Loudness\n",
       "0  Serotonin  Call Me Karizma   0.418     0.159  0.671575\n",
       "1        Joy        Thornhill   0.910     0.252  0.897120\n",
       "2       Like          Alissic   0.423     0.143  0.691744"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed_train = pd.read_csv(processed_train_data_directory)\n",
    "df_processed_train.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Artist</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Liveness</th>\n",
       "      <th>Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blinding Lights</td>\n",
       "      <td>The Weeknd</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.0897</td>\n",
       "      <td>0.756423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shape of You</td>\n",
       "      <td>Ed Sheeran</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.0931</td>\n",
       "      <td>0.927975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dance Monkey</td>\n",
       "      <td>Tones And I</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.1490</td>\n",
       "      <td>0.727363</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Name       Artist  Energy  Liveness  Loudness\n",
       "0  Blinding Lights   The Weeknd   0.730    0.0897  0.756423\n",
       "1     Shape of You   Ed Sheeran   0.652    0.0931  0.927975\n",
       "2     Dance Monkey  Tones And I   0.588    0.1490  0.727363"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed_test = pd.read_csv(processed_test_data_directory)\n",
    "df_processed_test.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_processed_train['Name'] = df_processed_train['Name'].astype('category')\n",
    "df_processed_train['Artist'] = df_processed_train['Artist'].astype('category')\n",
    "df_processed_test['Name'] = df_processed_test['Name'].astype('category')\n",
    "df_processed_test['Artist'] = df_processed_test['Artist'].astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4><b>Dataset Description</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Column</th>\n",
       "      <th style=\"min-width: 15px;\">DType</th>\n",
       "      <th style=\"min-width: 15px;\">Kind</th>\n",
       "      <th style=\"min-width: 15px;\">Additional Info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Name</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Artist</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Energy</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Liveness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loudness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><h4><b>Dataset Content</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Name</th>\n",
       "      <th style=\"min-width: 15px;\">Artist</th>\n",
       "      <th style=\"min-width: 15px;\">Energy</th>\n",
       "      <th style=\"min-width: 15px;\">Liveness</th>\n",
       "      <th style=\"min-width: 15px;\">Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Serotonin</td>\n",
       "      <td>Call Me Karizma</td>\n",
       "      <td>0.418</td>\n",
       "      <td>0.159</td>\n",
       "      <td>0.671575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Joy</td>\n",
       "      <td>Thornhill</td>\n",
       "      <td>0.910</td>\n",
       "      <td>0.252</td>\n",
       "      <td>0.897120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Like</td>\n",
       "      <td>Alissic</td>\n",
       "      <td>0.423</td>\n",
       "      <td>0.143</td>\n",
       "      <td>0.691744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Can You Afford to Be An Individual?</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.859</td>\n",
       "      <td>0.127</td>\n",
       "      <td>0.794126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>This Feels Like the End</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.899</td>\n",
       "      <td>0.241</td>\n",
       "      <td>0.756586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>605</th>\n",
       "      <td>Tomorrow Is Closed</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.835</td>\n",
       "      <td>0.155</td>\n",
       "      <td>0.848115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>606</th>\n",
       "      <td>Do You Love Me Yet?</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.801</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.791058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>607</th>\n",
       "      <td>Welcome to the DCC</td>\n",
       "      <td>Nothing But Thieves</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.129</td>\n",
       "      <td>0.845661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>608</th>\n",
       "      <td>CODE MISTAKE</td>\n",
       "      <td>CORPSE</td>\n",
       "      <td>0.967</td>\n",
       "      <td>0.203</td>\n",
       "      <td>0.889835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>609</th>\n",
       "      <td>Kingdom Of Cards</td>\n",
       "      <td>Bad Omens</td>\n",
       "      <td>0.873</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.805667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_processed_train = Dataset(df_processed_train, cat_features=['Name','Artist'])\n",
    "ds_processed_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4><b>Dataset Description</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Column</th>\n",
       "      <th style=\"min-width: 15px;\">DType</th>\n",
       "      <th style=\"min-width: 15px;\">Kind</th>\n",
       "      <th style=\"min-width: 15px;\">Additional Info</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Name</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Artist</td>\n",
       "      <td>categorical</td>\n",
       "      <td>Categorical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Energy</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Liveness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Loudness</td>\n",
       "      <td>floating</td>\n",
       "      <td>Numerical Feature</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><h4><b>Dataset Content</b></h4><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th style=\"min-width: 15px;\"></th>\n",
       "      <th style=\"min-width: 15px;\">Name</th>\n",
       "      <th style=\"min-width: 15px;\">Artist</th>\n",
       "      <th style=\"min-width: 15px;\">Energy</th>\n",
       "      <th style=\"min-width: 15px;\">Liveness</th>\n",
       "      <th style=\"min-width: 15px;\">Loudness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blinding Lights</td>\n",
       "      <td>The Weeknd</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.0897</td>\n",
       "      <td>0.756423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shape of You</td>\n",
       "      <td>Ed Sheeran</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.0931</td>\n",
       "      <td>0.927975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dance Monkey</td>\n",
       "      <td>Tones And I</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.1490</td>\n",
       "      <td>0.727363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Someone You Loved</td>\n",
       "      <td>Lewis Capaldi</td>\n",
       "      <td>0.405</td>\n",
       "      <td>0.1050</td>\n",
       "      <td>0.772325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sunflower - Spider-Man: Into the Spider-Verse</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>0.522</td>\n",
       "      <td>0.0685</td>\n",
       "      <td>0.854078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>495</th>\n",
       "      <td>Hey There Delilah</td>\n",
       "      <td>Plain White T's</td>\n",
       "      <td>0.291</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>0.467199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>idontwannabeyouanymore</td>\n",
       "      <td>Billie Eilish</td>\n",
       "      <td>0.412</td>\n",
       "      <td>0.1160</td>\n",
       "      <td>0.598840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>Zombie</td>\n",
       "      <td>The Cranberries</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.3660</td>\n",
       "      <td>0.567910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>No Lie</td>\n",
       "      <td>Sean Paul</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.2060</td>\n",
       "      <td>0.947992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>Candy Paint</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>0.667</td>\n",
       "      <td>0.0753</td>\n",
       "      <td>0.755987</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_processed_test = Dataset(df_processed_test, cat_features=['Name','Artist'])\n",
    "ds_processed_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "        progress {\n",
       "            -webkit-appearance: none;\n",
       "            border: none;\n",
       "            border-radius: 3px;\n",
       "            width: 300px;\n",
       "            height: 20px;\n",
       "            vertical-align: middle;\n",
       "            margin-right: 10px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-bar {\n",
       "            border-radius: 3px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-value {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "        progress::-moz-progress-bar {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "    </style>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d77c6c96b5cc4266adcf1510ec615626",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(HTML(value='\\n<h1 id=\"summary_KSOBGBV0CU3VGGPP56S0W6EJM\">Data Integrity Sui…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "suite_processed_training = suite.run(ds_processed_train)\n",
    "suite_processed_training.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c:\\\\Users\\\\ivanr\\\\Desktop\\\\University\\\\Master\\\\SecondYear\\\\FirstSemester\\\\SE4AI\\\\project\\\\MusicExpress\\\\reports\\\\deepchecks\\\\QA_suite_processed_training_report.html'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if os.path.exists(PROCESSED_TRAIN_REPORT_PATH):\n",
    "    os.remove(PROCESSED_TRAIN_REPORT_PATH)\n",
    "\n",
    "suite_processed_training.save_as_html(PROCESSED_TRAIN_REPORT_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <style>\n",
       "        progress {\n",
       "            -webkit-appearance: none;\n",
       "            border: none;\n",
       "            border-radius: 3px;\n",
       "            width: 300px;\n",
       "            height: 20px;\n",
       "            vertical-align: middle;\n",
       "            margin-right: 10px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-bar {\n",
       "            border-radius: 3px;\n",
       "            background-color: aliceblue;\n",
       "        }\n",
       "        progress::-webkit-progress-value {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "        progress::-moz-progress-bar {\n",
       "            background-color: #9d60fb;\n",
       "        }\n",
       "    </style>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n",
      "c:\\Users\\ivanr\\Desktop\\University\\Master\\SecondYear\\FirstSemester\\SE4AI\\project\\MusicExpress\\env\\Lib\\site-packages\\deepchecks\\tabular\\checks\\data_integrity\\mixed_nulls.py:108: FutureWarning:\n",
      "\n",
      "is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9fdbbf477654af1b991d64dfce18e88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(HTML(value='\\n<h1 id=\"summary_YSYWPQCISFUPCC5ADW37C69N0\">Data Integrity Sui…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "suite_processed_test = suite.run(ds_processed_test)\n",
    "suite_processed_test.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c:\\\\Users\\\\ivanr\\\\Desktop\\\\University\\\\Master\\\\SecondYear\\\\FirstSemester\\\\SE4AI\\\\project\\\\MusicExpress\\\\reports\\\\deepchecks\\\\QA_suite_processed_test_report.html'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if os.path.exists(PROCESSED_TEST_REPORT_PATH):\n",
    "    os.remove(PROCESSED_TEST_REPORT_PATH)\n",
    "\n",
    "suite_processed_test.save_as_html(PROCESSED_TEST_REPORT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the notebook, the data does not present issues with respect to the chosen tests."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}