04. Content-based Course Recommender System Using User Profile and Course Genres.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMML321ENSkillsNetwork32585014-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Content-based Course Recommender System Using User Profile and Course Genres**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most common type of content-based recommendation system is to recommend items to users based on their profiles. The user's profile revolves around that user's preferences and tastes. It is shaped based on user ratings, including the number of times a user has clicked on different items or liked those items.\n",
    "\n",
    "The recommendation process is based on the similarity between those items. The similarity or closeness of items is measured based on the similarity in the content of those items. When we say content, we're talking about things like the item's category, tag, genre, and so on. Essentially the features about an item.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For online course recommender systems, we already know how to extract features from courses (such as genres or BoW features). Next, based on the course genres and users' ratings, we want to further build user profiles (if unknown).\n",
    "\n",
    "A user profile can be seen as the user feature vector that mathematically represents a user's learning interests.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the user profile feature vectors and course genre feature vectors constructed, we can use several computational methods, such as a simple dot product, to compute or predict an interest score for each course and recommend those courses with high interest scores.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/module\\_3/images/user_profile_score.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Generate a user profile based on course genres and rating\n",
    "*   Generate course recommendations based on a user's profile and course genres\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare and setup the lab environment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's install and import the required packages.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-learn==1.0.2 in c:\\python310\\lib\\site-packages (1.0.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\python310\\lib\\site-packages (from scikit-learn==1.0.2) (3.1.0)\n",
      "Requirement already satisfied: scipy>=1.1.0 in c:\\python310\\lib\\site-packages (from scikit-learn==1.0.2) (1.9.3)\n",
      "Requirement already satisfied: numpy>=1.14.6 in c:\\python310\\lib\\site-packages (from scikit-learn==1.0.2) (1.23.4)\n",
      "Requirement already satisfied: joblib>=0.11 in c:\\python310\\lib\\site-packages (from scikit-learn==1.0.2) (1.2.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install scikit-learn==1.0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# also set a random state\n",
    "rs = 123"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets generate user profiles using course genres and ratings\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we have a very simple course genre dataset that contains only three genres: `Python`, `Database`, and `MachineLearning`.\n",
    "\n",
    "We also have two courses: `Machine Learning with Python` and `SQL Learning with Python` and their genres as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Python</th>\n",
       "      <th>Database</th>\n",
       "      <th>MachineLearning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Machine Learning with Python</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SQL with Python</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Title  Python  Database  MachineLearning\n",
       "0  Machine Learning with Python       1         0                1\n",
       "1               SQL with Python       1         1                0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "course_genres = ['Python', 'Database', 'MachineLearning']\n",
    "courses = [['Machine Learning with Python', 1, 0, 1], [\"SQL with Python\", 1, 1, 0]]\n",
    "courses_df = pd.DataFrame(courses, columns = ['Title'] + course_genres)\n",
    "courses_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the dataset:\n",
    "\n",
    "*   Course `Machine Learning with Python` has `Python` and `MachineLearning` genres\n",
    "*   Course `SQL with Python` has `Python` and `Database` genres\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's create another simple user rating dataframe containing the ratings from two users.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>User</th>\n",
       "      <th>Title</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>user0</td>\n",
       "      <td>Machine Learning with Python</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>user1</td>\n",
       "      <td>SQL with Python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    User                         Title  Rating\n",
       "0  user0  Machine Learning with Python       3\n",
       "1  user1               SQL with Python       2"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users = [['user0', 'Machine Learning with Python', 3], ['user1', 'SQL with Python', 2]]\n",
    "users_df = pd.DataFrame(users, columns = ['User', 'Title', 'Rating'])\n",
    "users_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose user 0 rated `Machine Learning with Python` as 3 (completed with a certificate)  and user 1 rated `SQL with Python` as 2 (just audited or not completed).\n",
    "\n",
    "Based on their course ratings and course genres. Can we generate a profile vector for each user?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Intuitively, since user 0 has completed the course `Machine Learning with Python`, they should be interested in those genres associated with the course, i.e.,Machine Learning and Python.\n",
    "\n",
    "On the other hand, user 0 has not taken the `SQL with Python` so it is likely they are not interested in the database genre.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To quantify such user interests, we could multiply user 0's rating vector with a course genre matrix and get the weighted genre's vector for the courses:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User 0 rated course 0 as 3 and course 1 as 0/NA (unknown or not interested)\n",
    "u0 = np.array([[3, 0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The course genre's matrix\n",
    "C = courses_df[['Python', 'Database', 'MachineLearning']].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before multiple them, let's first print their shapes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User profile vector shape (1, 2) and course genre matrix shape (2, 3)\n"
     ]
    }
   ],
   "source": [
    "print(f\"User profile vector shape {u0.shape} and course genre matrix shape {C.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we multiple a $1 x 2$ vector with a $2 x 3$ matrix, we will get a 1 x 3 vector representing the user profile vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$u\\_0C = \\begin{bmatrix} 3 & 0 \\end{bmatrix} \\begin{bmatrix} 1 & 0 & 1 \\\\\\ 1 & 1 & 0 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 0, 3]], dtype=int64)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u0_weights = np.matmul(u0, C)\n",
    "u0_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Python', 'Database', 'MachineLearning']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "course_genres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the result. This `u0_weights` is also called the weighted genre vector and represents the interests of the user for each genre based on the courses they have rated. As we can see from the results, user0 seems interested in `Python` and `MachineLearning` with a rating of 3.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can calculate the weighted genre matrix for user 1:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$u\\_1C = \\begin{bmatrix} 0 & 2 \\end{bmatrix} \\begin{bmatrix} 1 & 0 & 1 \\\\\\ 1 & 1 & 0 \\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User 1 rated course 0 as 0 (unknown or not interested) and course 1 as 2\n",
    "u1 = np.array([[0, 2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2, 2, 0]], dtype=int64)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u1_weights = np.matmul(u1, C)\n",
    "u1_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the `u1_weights` vector, user0 seems very interested in `Python` and `Database` with a value 2.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine the two weighted genre vectors and create a user profile dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.concatenate((u0_weights.reshape(1, 3), u1_weights.reshape(1, 3)), axis=0)\n",
    "profiles_df = pd.DataFrame(weights, columns=['Python', 'Database', 'MachineLearning'])\n",
    "profiles_df.insert(0, 'user', ['user0', 'user1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>Python</th>\n",
       "      <th>Database</th>\n",
       "      <th>MachineLearning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>user0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>user1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    user  Python  Database  MachineLearning\n",
       "0  user0       3         0                3\n",
       "1  user1       2         2                0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profiles_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this `profiles_df` clearly shows the user profiles or course interests.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate recommendation scores for some new courses\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the user profiles generated, we can see that `user0` is very interested in Python and machine learning, and `user1` is very interested in Python and database.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, suppose we published some new courses titled as `Python 101`, `Database 101`, and `Machine Learning with R`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Python</th>\n",
       "      <th>Database</th>\n",
       "      <th>MachineLearning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Python 101</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Database 101</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Machine Learning with R</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Title  Python  Database  MachineLearning\n",
       "0               Python 101       1         0                0\n",
       "1             Database 101       0         1                0\n",
       "2  Machine Learning with R       0         0                1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_courses = [['Python 101', 1, 0, 0], [\"Database 101\", 0, 1, 0], [\"Machine Learning with R\", 0, 0, 1]]\n",
    "new_courses_df = pd.DataFrame(new_courses, columns = ['Title', 'Python', 'Database', 'MachineLearning'])\n",
    "new_courses_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, how can we calculate a recommendation score for each new course with respect to `user0` and `user1`, using user profile vectors and genre vectors?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One simple but effective way is to apply the dot product to the user profile vector and course genre vector (as they always have the same shape). Since we have two users and three courses, we need to perform a matrix multiplication:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>Python</th>\n",
       "      <th>Database</th>\n",
       "      <th>MachineLearning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>user0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>user1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    user  Python  Database  MachineLearning\n",
       "0  user0       3         0                3\n",
       "1  user1       2         2                0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profiles_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert the course genre dataframe into a 2-D numpy array:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0],\n",
       "       [0, 1, 0],\n",
       "       [0, 0, 1]], dtype=int64)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the title column\n",
    "new_courses_df = new_courses_df.loc[:, new_courses_df.columns != 'Title']\n",
    "course_matrix = new_courses_df.values\n",
    "course_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# course matrix shape\n",
    "course_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the above output, the course matrix is a `3 x 3` matrix and each row vector is a course genre vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can convert the user profile dataframe into another 2-d numpy array:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 0, 3],\n",
       "       [2, 2, 0]], dtype=int64)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the user column\n",
    "profiles_df = profiles_df.loc[:, profiles_df.columns != 'user']\n",
    "profile_matrix = profiles_df.values\n",
    "profile_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The profile matrix is a 2 x 3 matrix and each row is a user profile vector:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we multiply the course matrix and the user profile matrix, we can get the 2 x 3 course recommendation matrix with each element `(i, j)` representing a recommendation score of course `i` to user `j`. Intuitively, if a user `j` is interested in some topics(genres) and if a course `i` also has the same topics(genres), it means the user profile vector and course genre vector share many common dimensions and a dot product is likely to have a large value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 2],\n",
       "       [0, 2],\n",
       "       [3, 0]], dtype=int64)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = np.matmul(course_matrix, profile_matrix.T)\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add the course titles and user ids back to make the results more clear:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_df = pd.DataFrame(scores, columns=['User0', 'User1'])\n",
    "scores_df.index = ['Python 101', 'Database 101', 'Machine Learning with R']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>User0</th>\n",
       "      <th>User1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Python 101</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Database 101</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Machine Learning with R</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         User0  User1\n",
       "Python 101                   3      2\n",
       "Database 101                 0      2\n",
       "Machine Learning with R      3      0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# recommendation score dataframe\n",
    "scores_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the score results, we can see that:\n",
    "\n",
    "*   For user0, the recommended courses are `Python 101` and `Machine Learning with R` because user0 is very interested in Python and machine learning\n",
    "*   For user1, the recommended courses are `Python 101` and `Database 101` because user1 seems very interested in topics like Python and database\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK: Generate course recommendations based on user profile and course genre vectors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By now you have learned how to calculate recommendation scores using a user profile vector and a course genre vector.  Now, let's work on some real-world datasets to generate real personalized courses recommendations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will load a user's profile dataframe and a course genre dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "course_genre_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/datasets/course_genre.csv\"\n",
    "course_genres_df = pd.read_csv(course_genre_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>COURSE_ID</th>\n",
       "      <th>TITLE</th>\n",
       "      <th>Database</th>\n",
       "      <th>Python</th>\n",
       "      <th>CloudComputing</th>\n",
       "      <th>DataAnalysis</th>\n",
       "      <th>Containers</th>\n",
       "      <th>MachineLearning</th>\n",
       "      <th>ComputerVision</th>\n",
       "      <th>DataScience</th>\n",
       "      <th>BigData</th>\n",
       "      <th>Chatbot</th>\n",
       "      <th>R</th>\n",
       "      <th>BackendDev</th>\n",
       "      <th>FrontendDev</th>\n",
       "      <th>Blockchain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ML0201EN</td>\n",
       "      <td>robots are coming  build iot apps with watson ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ML0122EN</td>\n",
       "      <td>accelerating deep learning with gpu</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GPXX0ZG0EN</td>\n",
       "      <td>consuming restful services using the reactive ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RP0105EN</td>\n",
       "      <td>analyzing big data in r using apache spark</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GPXX0Z2PEN</td>\n",
       "      <td>containerizing  packaging  and running a sprin...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    COURSE_ID                                              TITLE  Database  \\\n",
       "0    ML0201EN  robots are coming  build iot apps with watson ...         0   \n",
       "1    ML0122EN                accelerating deep learning with gpu         0   \n",
       "2  GPXX0ZG0EN  consuming restful services using the reactive ...         0   \n",
       "3    RP0105EN         analyzing big data in r using apache spark         1   \n",
       "4  GPXX0Z2PEN  containerizing  packaging  and running a sprin...         0   \n",
       "\n",
       "   Python  CloudComputing  DataAnalysis  Containers  MachineLearning  \\\n",
       "0       0               0             0           0                0   \n",
       "1       1               0             0           0                1   \n",
       "2       0               0             0           0                0   \n",
       "3       0               0             1           0                0   \n",
       "4       0               0             0           1                0   \n",
       "\n",
       "   ComputerVision  DataScience  BigData  Chatbot  R  BackendDev  FrontendDev  \\\n",
       "0               0            0        0        0  0           1            1   \n",
       "1               0            1        0        0  0           0            0   \n",
       "2               0            0        0        0  0           1            1   \n",
       "3               0            0        1        0  1           0            0   \n",
       "4               0            0        0        0  0           1            0   \n",
       "\n",
       "   Blockchain  \n",
       "0           0  \n",
       "1           0  \n",
       "2           0  \n",
       "3           0  \n",
       "4           0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "course_genres_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_genre_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/datasets/user_profile.csv\"\n",
    "profile_df = pd.read_csv(profile_genre_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>Database</th>\n",
       "      <th>Python</th>\n",
       "      <th>CloudComputing</th>\n",
       "      <th>DataAnalysis</th>\n",
       "      <th>Containers</th>\n",
       "      <th>MachineLearning</th>\n",
       "      <th>ComputerVision</th>\n",
       "      <th>DataScience</th>\n",
       "      <th>BigData</th>\n",
       "      <th>Chatbot</th>\n",
       "      <th>R</th>\n",
       "      <th>BackendDev</th>\n",
       "      <th>FrontendDev</th>\n",
       "      <th>Blockchain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>52.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>40.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>24.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  Database  Python  CloudComputing  DataAnalysis  Containers  \\\n",
       "0     2      52.0    14.0             6.0          43.0         3.0   \n",
       "1     4      40.0     2.0             4.0          28.0         0.0   \n",
       "2     5      24.0     8.0            18.0          24.0         0.0   \n",
       "3     7       2.0     0.0             0.0           2.0         0.0   \n",
       "4     8       6.0     0.0             0.0           4.0         0.0   \n",
       "\n",
       "   MachineLearning  ComputerVision  DataScience  BigData  Chatbot     R  \\\n",
       "0             33.0             0.0         29.0     41.0      2.0  18.0   \n",
       "1             14.0             0.0         20.0     24.0      0.0   6.0   \n",
       "2             30.0             0.0         22.0     14.0      2.0  14.0   \n",
       "3              0.0             0.0          0.0      2.0      0.0   0.0   \n",
       "4              0.0             0.0          0.0      6.0      0.0   2.0   \n",
       "\n",
       "   BackendDev  FrontendDev  Blockchain  \n",
       "0        34.0          9.0         6.0  \n",
       "1         6.0          0.0         2.0  \n",
       "2        26.0          4.0         6.0  \n",
       "3         0.0          0.0         0.0  \n",
       "4         0.0          0.0         0.0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The profile dataframe contains the course interests for each user, for example, user 8 is very interested in R, data analysis, database, and big data:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>Database</th>\n",
       "      <th>Python</th>\n",
       "      <th>CloudComputing</th>\n",
       "      <th>DataAnalysis</th>\n",
       "      <th>Containers</th>\n",
       "      <th>MachineLearning</th>\n",
       "      <th>ComputerVision</th>\n",
       "      <th>DataScience</th>\n",
       "      <th>BigData</th>\n",
       "      <th>Chatbot</th>\n",
       "      <th>R</th>\n",
       "      <th>BackendDev</th>\n",
       "      <th>FrontendDev</th>\n",
       "      <th>Blockchain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  Database  Python  CloudComputing  DataAnalysis  Containers  \\\n",
       "4     8       6.0     0.0             0.0           4.0         0.0   \n",
       "\n",
       "   MachineLearning  ComputerVision  DataScience  BigData  Chatbot    R  \\\n",
       "4              0.0             0.0          0.0      6.0      0.0  2.0   \n",
       "\n",
       "   BackendDev  FrontendDev  Blockchain  \n",
       "4         0.0          0.0         0.0  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile_df[profile_df['user'] == 8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's load a test dataset, containing test users to whom we want to make course recommendations:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_users_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/datasets/rs_content_test.csv\"\n",
    "test_users_df = pd.read_csv(test_users_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at how many test users we have in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total numbers of test users 1000\n"
     ]
    }
   ],
   "source": [
    "test_users = test_users_df.groupby(['user']).max().reset_index(drop=False)\n",
    "test_user_ids = test_users['user'].to_list()\n",
    "print(f\"Total numbers of test users {len(test_user_ids)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then for each test user in the test dataset, you need to first find out which courses are unknown/unselected to them. For example, suppose we have a user `1078030` with profile:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>Database</th>\n",
       "      <th>Python</th>\n",
       "      <th>CloudComputing</th>\n",
       "      <th>DataAnalysis</th>\n",
       "      <th>Containers</th>\n",
       "      <th>MachineLearning</th>\n",
       "      <th>ComputerVision</th>\n",
       "      <th>DataScience</th>\n",
       "      <th>BigData</th>\n",
       "      <th>Chatbot</th>\n",
       "      <th>R</th>\n",
       "      <th>BackendDev</th>\n",
       "      <th>FrontendDev</th>\n",
       "      <th>Blockchain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18204</th>\n",
       "      <td>1078030</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          user  Database  Python  CloudComputing  DataAnalysis  Containers  \\\n",
       "18204  1078030       0.0    12.0             0.0           9.0         0.0   \n",
       "\n",
       "       MachineLearning  ComputerVision  DataScience  BigData  Chatbot    R  \\\n",
       "18204             12.0             0.0          6.0      0.0      0.0  0.0   \n",
       "\n",
       "       BackendDev  FrontendDev  Blockchain  \n",
       "18204         0.0          0.0         0.0  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_user_profile = profile_df[profile_df['user'] == 1078030]\n",
    "test_user_profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0., 12.,  0.,  9.,  0., 12.,  0.,  6.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now let's get the test user vector by excluding the `user` column\n",
    "test_user_vector = test_user_profile.iloc[0, 1:].values\n",
    "test_user_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can first find their enrolled courses in `test_users_df`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "enrolled_courses = test_users_df[test_users_df['user'] == 1078030]['item'].to_list()\n",
    "enrolled_courses = set(enrolled_courses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'DA0101EN',\n",
       " 'DV0101EN',\n",
       " 'ML0101ENv3',\n",
       " 'ML0115EN',\n",
       " 'ML0120ENv2',\n",
       " 'ML0122ENv1',\n",
       " 'PY0101EN',\n",
       " 'ST0101EN'}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enrolled_courses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then print the entire course list:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AI0111EN',\n",
       " 'BC0101EN',\n",
       " 'BC0201EN',\n",
       " 'BC0202EN',\n",
       " 'BD0101EN',\n",
       " 'BD0111EN',\n",
       " 'BD0115EN',\n",
       " 'BD0121EN',\n",
       " 'BD0123EN',\n",
       " 'BD0131EN',\n",
       " 'BD0133EN',\n",
       " 'BD0135EN',\n",
       " 'BD0137EN',\n",
       " 'BD0141EN',\n",
       " 'BD0143EN',\n",
       " 'BD0145EN',\n",
       " 'BD0151EN',\n",
       " 'BD0153EN',\n",
       " 'BD0211EN',\n",
       " 'BD0212EN',\n",
       " 'BD0221EN',\n",
       " 'BD0223EN',\n",
       " 'BENTEST4',\n",
       " 'CB0101EN',\n",
       " 'CB0103EN',\n",
       " 'CB0105ENv1',\n",
       " 'CB0201EN',\n",
       " 'CC0101EN',\n",
       " 'CC0103EN',\n",
       " 'CC0120EN',\n",
       " 'CC0121EN',\n",
       " 'CC0150EN',\n",
       " 'CC0201EN',\n",
       " 'CC0210EN',\n",
       " 'CC0250EN',\n",
       " 'CC0271EN',\n",
       " 'CL0101EN',\n",
       " 'CNSC02EN',\n",
       " 'CO0101EN',\n",
       " 'CO0193EN',\n",
       " 'CO0201EN',\n",
       " 'CO0301EN',\n",
       " 'CO0302EN',\n",
       " 'CO0401EN',\n",
       " 'COM001EN',\n",
       " 'CP0101EN',\n",
       " 'DA0101EN',\n",
       " 'DA0151EN',\n",
       " 'DA0201EN',\n",
       " 'DAI101EN',\n",
       " 'DB0101EN',\n",
       " 'DB0111EN',\n",
       " 'DB0113EN',\n",
       " 'DB0115EN',\n",
       " 'DB0151EN',\n",
       " 'DE0205EN',\n",
       " 'DJ0101EN',\n",
       " 'DP0101EN',\n",
       " 'DS0101EN',\n",
       " 'DS0103EN',\n",
       " 'DS0105EN',\n",
       " 'DS0107',\n",
       " 'DS0110EN',\n",
       " 'DS0132EN',\n",
       " 'DS0201EN',\n",
       " 'DS0301EN',\n",
       " 'DS0321EN',\n",
       " 'DV0101EN',\n",
       " 'DV0151EN',\n",
       " 'DW0101EN',\n",
       " 'DX0106EN',\n",
       " 'DX0107EN',\n",
       " 'DX0108EN',\n",
       " 'EE0101EN',\n",
       " 'GPXX01AVEN',\n",
       " 'GPXX01DCEN',\n",
       " 'GPXX01RYEN',\n",
       " 'GPXX03HFEN',\n",
       " 'GPXX0435EN',\n",
       " 'GPXX048OEN',\n",
       " 'GPXX04HEEN',\n",
       " 'GPXX04MXEN',\n",
       " 'GPXX04P5EN',\n",
       " 'GPXX04TNEN',\n",
       " 'GPXX04V3EN',\n",
       " 'GPXX04XJEN',\n",
       " 'GPXX05LMEN',\n",
       " 'GPXX05P1EN',\n",
       " 'GPXX05RDEN',\n",
       " 'GPXX06KEEN',\n",
       " 'GPXX06RFEN',\n",
       " 'GPXX06ZLEN',\n",
       " 'GPXX0725EN',\n",
       " 'GPXX0742EN',\n",
       " 'GPXX07REN',\n",
       " 'GPXX07UGEN',\n",
       " 'GPXX07YGEN',\n",
       " 'GPXX08WYEN',\n",
       " 'GPXX097UEN',\n",
       " 'GPXX0A1YEN',\n",
       " 'GPXX0ADEN',\n",
       " 'GPXX0BSAEN',\n",
       " 'GPXX0BUBEN',\n",
       " 'GPXX0D14EN',\n",
       " 'GPXX0E3QEN',\n",
       " 'GPXX0FFCEN',\n",
       " 'GPXX0FTCEN',\n",
       " 'GPXX0G31EN',\n",
       " 'GPXX0G3KEN',\n",
       " 'GPXX0G81EN',\n",
       " 'GPXX0HAAEN',\n",
       " 'GPXX0HC7EN',\n",
       " 'GPXX0HZ2EN',\n",
       " 'GPXX0I4FEN',\n",
       " 'GPXX0IBEN',\n",
       " 'GPXX0IHMEN',\n",
       " 'GPXX0JGFEN',\n",
       " 'GPXX0JLHEN',\n",
       " 'GPXX0JZ4EN',\n",
       " 'GPXX0KHHEN',\n",
       " 'GPXX0KV4EN',\n",
       " 'GPXX0KY1EN',\n",
       " 'GPXX0LLEEN',\n",
       " 'GPXX0M6UEN',\n",
       " 'GPXX0M7ZEN',\n",
       " 'GPXX0MIIEN',\n",
       " 'GPXX0MP0EN',\n",
       " 'GPXX0NHZEN',\n",
       " 'GPXX0PG8EN',\n",
       " 'GPXX0PICEN',\n",
       " 'GPXX0Q8AEN',\n",
       " 'GPXX0QJFEN',\n",
       " 'GPXX0QQ3EN',\n",
       " 'GPXX0QR3EN',\n",
       " 'GPXX0QS6EN',\n",
       " 'GPXX0QTEEN',\n",
       " 'GPXX0QU9EN',\n",
       " 'GPXX0RL8EN',\n",
       " 'GPXX0RQLEN',\n",
       " 'GPXX0SDXEN',\n",
       " 'GPXX0T0FEN',\n",
       " 'GPXX0T3CEN',\n",
       " 'GPXX0TY1EN',\n",
       " 'GPXX0UMSEN',\n",
       " 'GPXX0UN5EN',\n",
       " 'GPXX0W7KEN',\n",
       " 'GPXX0WRDEN',\n",
       " 'GPXX0WTIEN',\n",
       " 'GPXX0XENEN',\n",
       " 'GPXX0XFQEN',\n",
       " 'GPXX0XV3EN',\n",
       " 'GPXX0YBFEN',\n",
       " 'GPXX0YMEEN',\n",
       " 'GPXX0YXHEN',\n",
       " 'GPXX0Z2PEN',\n",
       " 'GPXX0ZG0EN',\n",
       " 'GPXX0ZMZEN',\n",
       " 'GPXX0ZYVEN',\n",
       " 'HCC104EN',\n",
       " 'HCC105EN',\n",
       " 'IT0101EN',\n",
       " 'LB0101ENv1',\n",
       " 'LB0103ENv1',\n",
       " 'LB0105ENv1',\n",
       " 'LB0107ENv1',\n",
       " 'LB0109ENv1',\n",
       " 'LB0111EN',\n",
       " 'ML0101EN',\n",
       " 'ML0101ENv3',\n",
       " 'ML0103EN',\n",
       " 'ML0109EN',\n",
       " 'ML0111EN',\n",
       " 'ML0115EN',\n",
       " 'ML0120EN',\n",
       " 'ML0120ENv2',\n",
       " 'ML0120ENv3',\n",
       " 'ML0122EN',\n",
       " 'ML0122ENv1',\n",
       " 'ML0122ENv3',\n",
       " 'ML0151EN',\n",
       " 'ML0201EN',\n",
       " 'OS0101EN',\n",
       " 'PA0101EN',\n",
       " 'PA0103EN',\n",
       " 'PA0107EN',\n",
       " 'PA0109EN',\n",
       " 'PHPM002EN',\n",
       " 'PY0101EN',\n",
       " 'QC0101EN',\n",
       " 'RAVSCTEST1',\n",
       " 'RP0101EN',\n",
       " 'RP0103',\n",
       " 'RP0103EN',\n",
       " 'RP0105EN',\n",
       " 'RP0151EN',\n",
       " 'SC0101EN',\n",
       " 'SC0103EN',\n",
       " 'SC0105EN',\n",
       " 'SECM03EN',\n",
       " 'SN0111EN',\n",
       " 'ST0101EN',\n",
       " 'ST0201EN',\n",
       " 'ST0301EN',\n",
       " 'SW0101EN',\n",
       " 'SW0201EN',\n",
       " 'TA0105',\n",
       " 'TA0105EN',\n",
       " 'TA0106EN',\n",
       " 'TMP0101EN',\n",
       " 'TMP0105EN',\n",
       " 'TMP0106',\n",
       " 'TMP107',\n",
       " 'WA0101EN',\n",
       " 'WA0103EN',\n",
       " 'excourse01',\n",
       " 'excourse02',\n",
       " 'excourse03',\n",
       " 'excourse04',\n",
       " 'excourse05',\n",
       " 'excourse06',\n",
       " 'excourse07',\n",
       " 'excourse08',\n",
       " 'excourse09',\n",
       " 'excourse10',\n",
       " 'excourse11',\n",
       " 'excourse12',\n",
       " 'excourse13',\n",
       " 'excourse14',\n",
       " 'excourse15',\n",
       " 'excourse16',\n",
       " 'excourse17',\n",
       " 'excourse18',\n",
       " 'excourse19',\n",
       " 'excourse20',\n",
       " 'excourse21',\n",
       " 'excourse22',\n",
       " 'excourse23',\n",
       " 'excourse24',\n",
       " 'excourse25',\n",
       " 'excourse26',\n",
       " 'excourse27',\n",
       " 'excourse28',\n",
       " 'excourse29',\n",
       " 'excourse30',\n",
       " 'excourse31',\n",
       " 'excourse32',\n",
       " 'excourse33',\n",
       " 'excourse34',\n",
       " 'excourse35',\n",
       " 'excourse36',\n",
       " 'excourse37',\n",
       " 'excourse38',\n",
       " 'excourse39',\n",
       " 'excourse40',\n",
       " 'excourse41',\n",
       " 'excourse42',\n",
       " 'excourse43',\n",
       " 'excourse44',\n",
       " 'excourse45',\n",
       " 'excourse46',\n",
       " 'excourse47',\n",
       " 'excourse48',\n",
       " 'excourse49',\n",
       " 'excourse50',\n",
       " 'excourse51',\n",
       " 'excourse52',\n",
       " 'excourse53',\n",
       " 'excourse54',\n",
       " 'excourse55',\n",
       " 'excourse56',\n",
       " 'excourse57',\n",
       " 'excourse58',\n",
       " 'excourse59',\n",
       " 'excourse60',\n",
       " 'excourse61',\n",
       " 'excourse62',\n",
       " 'excourse63',\n",
       " 'excourse64',\n",
       " 'excourse65',\n",
       " 'excourse66',\n",
       " 'excourse67',\n",
       " 'excourse68',\n",
       " 'excourse69',\n",
       " 'excourse70',\n",
       " 'excourse71',\n",
       " 'excourse72',\n",
       " 'excourse73',\n",
       " 'excourse74',\n",
       " 'excourse75',\n",
       " 'excourse76',\n",
       " 'excourse77',\n",
       " 'excourse78',\n",
       " 'excourse79',\n",
       " 'excourse80',\n",
       " 'excourse81',\n",
       " 'excourse82',\n",
       " 'excourse83',\n",
       " 'excourse84',\n",
       " 'excourse85',\n",
       " 'excourse86',\n",
       " 'excourse87',\n",
       " 'excourse88',\n",
       " 'excourse89',\n",
       " 'excourse90',\n",
       " 'excourse91',\n",
       " 'excourse92',\n",
       " 'excourse93'}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_courses = set(course_genres_df['COURSE_ID'].values)\n",
    "all_courses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can use all courses to subtract the enrolled courses to get a set of all unknown courses for user `1078030`, and we want to find potential interested courses hidden in the unknown course list.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AI0111EN',\n",
       " 'BC0101EN',\n",
       " 'BC0201EN',\n",
       " 'BC0202EN',\n",
       " 'BD0101EN',\n",
       " 'BD0111EN',\n",
       " 'BD0115EN',\n",
       " 'BD0121EN',\n",
       " 'BD0123EN',\n",
       " 'BD0131EN',\n",
       " 'BD0133EN',\n",
       " 'BD0135EN',\n",
       " 'BD0137EN',\n",
       " 'BD0141EN',\n",
       " 'BD0143EN',\n",
       " 'BD0145EN',\n",
       " 'BD0151EN',\n",
       " 'BD0153EN',\n",
       " 'BD0211EN',\n",
       " 'BD0212EN',\n",
       " 'BD0221EN',\n",
       " 'BD0223EN',\n",
       " 'BENTEST4',\n",
       " 'CB0101EN',\n",
       " 'CB0103EN',\n",
       " 'CB0105ENv1',\n",
       " 'CB0201EN',\n",
       " 'CC0101EN',\n",
       " 'CC0103EN',\n",
       " 'CC0120EN',\n",
       " 'CC0121EN',\n",
       " 'CC0150EN',\n",
       " 'CC0201EN',\n",
       " 'CC0210EN',\n",
       " 'CC0250EN',\n",
       " 'CC0271EN',\n",
       " 'CL0101EN',\n",
       " 'CNSC02EN',\n",
       " 'CO0101EN',\n",
       " 'CO0193EN',\n",
       " 'CO0201EN',\n",
       " 'CO0301EN',\n",
       " 'CO0302EN',\n",
       " 'CO0401EN',\n",
       " 'COM001EN',\n",
       " 'CP0101EN',\n",
       " 'DA0151EN',\n",
       " 'DA0201EN',\n",
       " 'DAI101EN',\n",
       " 'DB0101EN',\n",
       " 'DB0111EN',\n",
       " 'DB0113EN',\n",
       " 'DB0115EN',\n",
       " 'DB0151EN',\n",
       " 'DE0205EN',\n",
       " 'DJ0101EN',\n",
       " 'DP0101EN',\n",
       " 'DS0101EN',\n",
       " 'DS0103EN',\n",
       " 'DS0105EN',\n",
       " 'DS0107',\n",
       " 'DS0110EN',\n",
       " 'DS0132EN',\n",
       " 'DS0201EN',\n",
       " 'DS0301EN',\n",
       " 'DS0321EN',\n",
       " 'DV0151EN',\n",
       " 'DW0101EN',\n",
       " 'DX0106EN',\n",
       " 'DX0107EN',\n",
       " 'DX0108EN',\n",
       " 'EE0101EN',\n",
       " 'GPXX01AVEN',\n",
       " 'GPXX01DCEN',\n",
       " 'GPXX01RYEN',\n",
       " 'GPXX03HFEN',\n",
       " 'GPXX0435EN',\n",
       " 'GPXX048OEN',\n",
       " 'GPXX04HEEN',\n",
       " 'GPXX04MXEN',\n",
       " 'GPXX04P5EN',\n",
       " 'GPXX04TNEN',\n",
       " 'GPXX04V3EN',\n",
       " 'GPXX04XJEN',\n",
       " 'GPXX05LMEN',\n",
       " 'GPXX05P1EN',\n",
       " 'GPXX05RDEN',\n",
       " 'GPXX06KEEN',\n",
       " 'GPXX06RFEN',\n",
       " 'GPXX06ZLEN',\n",
       " 'GPXX0725EN',\n",
       " 'GPXX0742EN',\n",
       " 'GPXX07REN',\n",
       " 'GPXX07UGEN',\n",
       " 'GPXX07YGEN',\n",
       " 'GPXX08WYEN',\n",
       " 'GPXX097UEN',\n",
       " 'GPXX0A1YEN',\n",
       " 'GPXX0ADEN',\n",
       " 'GPXX0BSAEN',\n",
       " 'GPXX0BUBEN',\n",
       " 'GPXX0D14EN',\n",
       " 'GPXX0E3QEN',\n",
       " 'GPXX0FFCEN',\n",
       " 'GPXX0FTCEN',\n",
       " 'GPXX0G31EN',\n",
       " 'GPXX0G3KEN',\n",
       " 'GPXX0G81EN',\n",
       " 'GPXX0HAAEN',\n",
       " 'GPXX0HC7EN',\n",
       " 'GPXX0HZ2EN',\n",
       " 'GPXX0I4FEN',\n",
       " 'GPXX0IBEN',\n",
       " 'GPXX0IHMEN',\n",
       " 'GPXX0JGFEN',\n",
       " 'GPXX0JLHEN',\n",
       " 'GPXX0JZ4EN',\n",
       " 'GPXX0KHHEN',\n",
       " 'GPXX0KV4EN',\n",
       " 'GPXX0KY1EN',\n",
       " 'GPXX0LLEEN',\n",
       " 'GPXX0M6UEN',\n",
       " 'GPXX0M7ZEN',\n",
       " 'GPXX0MIIEN',\n",
       " 'GPXX0MP0EN',\n",
       " 'GPXX0NHZEN',\n",
       " 'GPXX0PG8EN',\n",
       " 'GPXX0PICEN',\n",
       " 'GPXX0Q8AEN',\n",
       " 'GPXX0QJFEN',\n",
       " 'GPXX0QQ3EN',\n",
       " 'GPXX0QR3EN',\n",
       " 'GPXX0QS6EN',\n",
       " 'GPXX0QTEEN',\n",
       " 'GPXX0QU9EN',\n",
       " 'GPXX0RL8EN',\n",
       " 'GPXX0RQLEN',\n",
       " 'GPXX0SDXEN',\n",
       " 'GPXX0T0FEN',\n",
       " 'GPXX0T3CEN',\n",
       " 'GPXX0TY1EN',\n",
       " 'GPXX0UMSEN',\n",
       " 'GPXX0UN5EN',\n",
       " 'GPXX0W7KEN',\n",
       " 'GPXX0WRDEN',\n",
       " 'GPXX0WTIEN',\n",
       " 'GPXX0XENEN',\n",
       " 'GPXX0XFQEN',\n",
       " 'GPXX0XV3EN',\n",
       " 'GPXX0YBFEN',\n",
       " 'GPXX0YMEEN',\n",
       " 'GPXX0YXHEN',\n",
       " 'GPXX0Z2PEN',\n",
       " 'GPXX0ZG0EN',\n",
       " 'GPXX0ZMZEN',\n",
       " 'GPXX0ZYVEN',\n",
       " 'HCC104EN',\n",
       " 'HCC105EN',\n",
       " 'IT0101EN',\n",
       " 'LB0101ENv1',\n",
       " 'LB0103ENv1',\n",
       " 'LB0105ENv1',\n",
       " 'LB0107ENv1',\n",
       " 'LB0109ENv1',\n",
       " 'LB0111EN',\n",
       " 'ML0101EN',\n",
       " 'ML0103EN',\n",
       " 'ML0109EN',\n",
       " 'ML0111EN',\n",
       " 'ML0120EN',\n",
       " 'ML0120ENv3',\n",
       " 'ML0122EN',\n",
       " 'ML0122ENv3',\n",
       " 'ML0151EN',\n",
       " 'ML0201EN',\n",
       " 'OS0101EN',\n",
       " 'PA0101EN',\n",
       " 'PA0103EN',\n",
       " 'PA0107EN',\n",
       " 'PA0109EN',\n",
       " 'PHPM002EN',\n",
       " 'QC0101EN',\n",
       " 'RAVSCTEST1',\n",
       " 'RP0101EN',\n",
       " 'RP0103',\n",
       " 'RP0103EN',\n",
       " 'RP0105EN',\n",
       " 'RP0151EN',\n",
       " 'SC0101EN',\n",
       " 'SC0103EN',\n",
       " 'SC0105EN',\n",
       " 'SECM03EN',\n",
       " 'SN0111EN',\n",
       " 'ST0201EN',\n",
       " 'ST0301EN',\n",
       " 'SW0101EN',\n",
       " 'SW0201EN',\n",
       " 'TA0105',\n",
       " 'TA0105EN',\n",
       " 'TA0106EN',\n",
       " 'TMP0101EN',\n",
       " 'TMP0105EN',\n",
       " 'TMP0106',\n",
       " 'TMP107',\n",
       " 'WA0101EN',\n",
       " 'WA0103EN',\n",
       " 'excourse01',\n",
       " 'excourse02',\n",
       " 'excourse03',\n",
       " 'excourse04',\n",
       " 'excourse05',\n",
       " 'excourse06',\n",
       " 'excourse07',\n",
       " 'excourse08',\n",
       " 'excourse09',\n",
       " 'excourse10',\n",
       " 'excourse11',\n",
       " 'excourse12',\n",
       " 'excourse13',\n",
       " 'excourse14',\n",
       " 'excourse15',\n",
       " 'excourse16',\n",
       " 'excourse17',\n",
       " 'excourse18',\n",
       " 'excourse19',\n",
       " 'excourse20',\n",
       " 'excourse21',\n",
       " 'excourse22',\n",
       " 'excourse23',\n",
       " 'excourse24',\n",
       " 'excourse25',\n",
       " 'excourse26',\n",
       " 'excourse27',\n",
       " 'excourse28',\n",
       " 'excourse29',\n",
       " 'excourse30',\n",
       " 'excourse31',\n",
       " 'excourse32',\n",
       " 'excourse33',\n",
       " 'excourse34',\n",
       " 'excourse35',\n",
       " 'excourse36',\n",
       " 'excourse37',\n",
       " 'excourse38',\n",
       " 'excourse39',\n",
       " 'excourse40',\n",
       " 'excourse41',\n",
       " 'excourse42',\n",
       " 'excourse43',\n",
       " 'excourse44',\n",
       " 'excourse45',\n",
       " 'excourse46',\n",
       " 'excourse47',\n",
       " 'excourse48',\n",
       " 'excourse49',\n",
       " 'excourse50',\n",
       " 'excourse51',\n",
       " 'excourse52',\n",
       " 'excourse53',\n",
       " 'excourse54',\n",
       " 'excourse55',\n",
       " 'excourse56',\n",
       " 'excourse57',\n",
       " 'excourse58',\n",
       " 'excourse59',\n",
       " 'excourse60',\n",
       " 'excourse61',\n",
       " 'excourse62',\n",
       " 'excourse63',\n",
       " 'excourse64',\n",
       " 'excourse65',\n",
       " 'excourse66',\n",
       " 'excourse67',\n",
       " 'excourse68',\n",
       " 'excourse69',\n",
       " 'excourse70',\n",
       " 'excourse71',\n",
       " 'excourse72',\n",
       " 'excourse73',\n",
       " 'excourse74',\n",
       " 'excourse75',\n",
       " 'excourse76',\n",
       " 'excourse77',\n",
       " 'excourse78',\n",
       " 'excourse79',\n",
       " 'excourse80',\n",
       " 'excourse81',\n",
       " 'excourse82',\n",
       " 'excourse83',\n",
       " 'excourse84',\n",
       " 'excourse85',\n",
       " 'excourse86',\n",
       " 'excourse87',\n",
       " 'excourse88',\n",
       " 'excourse89',\n",
       " 'excourse90',\n",
       " 'excourse91',\n",
       " 'excourse92',\n",
       " 'excourse93'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unknown_courses = all_courses.difference(enrolled_courses)\n",
    "unknown_courses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the genre vectors for those unknown courses as well:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 1, 1, 0],\n",
       "       [0, 1, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 1, 1, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 1, 0],\n",
       "       [0, 0, 0, ..., 1, 1, 0],\n",
       "       [0, 0, 0, ..., 1, 1, 0]], dtype=int64)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unknown_course_genres = course_genres_df[course_genres_df['COURSE_ID'].isin(unknown_courses)]\n",
    "# Now let's get the course matrix by excluding `COURSE_ID` and `TITLE` columns:\n",
    "course_matrix = unknown_course_genres.iloc[:, 2:].values\n",
    "course_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the user profile vector for user `1078030`  and all the unseen course genres vectors above, you can use the dot product to calculate the recommendation score for each unknown course. e.g., the recommendation score for course `accelerating deep learning with gpu` is:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30.0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score = np.dot(course_matrix[1], test_user_vector)\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, we will need to choose a recommendation score threshold. If the score of any course is above the threshold, we may recommend that course to the user.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The workflow can be summarized in the following flowchart:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/module\\_3/images/recommend_courses.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's calculate the recommendation scores of all courses for all the 1000 test users.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We first reload all datasets again, and create an empty dictionary to store the results\n",
    "test_users_df = pd.read_csv(test_users_url)\n",
    "profile_df = pd.read_csv(profile_genre_url)\n",
    "course_genres_df = pd.read_csv(course_genre_url)\n",
    "res_dict = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only want to recommend courses with very high scores so we may set a score threshold to filter out those courses with low scores.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only keep the score larger than the recommendation threshold\n",
    "# The threshold can be fine-tuned to adjust the size of generated recommendations\n",
    "score_threshold = 10.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We defined a function called `generate_recommendation_scores()` to compute the recommendation scores of all the unknown courses for all test users.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*TODO: Complete the generate_recommendation_scores() function blow to generate recommendation score for all users. You may also implement the task with different solutions.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_recommendation_scores():\n",
    "    users = []\n",
    "    courses = []\n",
    "    scores = []\n",
    "    for user_id in test_user_ids:\n",
    "        test_user_profile = profile_df[profile_df['user'] == user_id]\n",
    "        # get user vector for the current user id\n",
    "        test_user_vector = profile_df[profile_df['user'] == user_id].iloc[0, 1:].values\n",
    "        \n",
    "        # get the unknown course ids for the current user id\n",
    "        enrolled_courses = test_users_df[test_users_df['user'] == user_id]['item'].to_list()\n",
    "        unknown_courses = all_courses.difference(enrolled_courses)\n",
    "        unknown_course_df = course_genres_df[course_genres_df['COURSE_ID'].isin(unknown_courses)]\n",
    "        unknown_course_ids = unknown_course_df['COURSE_ID'].values\n",
    "        \n",
    "        # user np.dot() to get the recommendation scores for each course\n",
    "        recommendation_scores = np.dot(unknown_course_df.iloc[:, 2:].values, test_user_vector)\n",
    "\n",
    "        # Append the results into the users, courses, and scores list\n",
    "        for i in range(0, len(unknown_course_ids)):\n",
    "            score = recommendation_scores[i]\n",
    "            # Only keep the courses with high recommendation score\n",
    "            if score >= score_threshold:\n",
    "                users.append(user_id)\n",
    "                courses.append(unknown_course_ids[i])\n",
    "                scores.append(recommendation_scores[i])\n",
    "                \n",
    "    return users, courses, scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0., 12.,  0.,  9.,  0., 12.,  0.,  6.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile_df[profile_df['user'] == 1078030].iloc[0, 1:].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: Instead of using some absolute score threshold, you may also try sorting the scores for each user and return the top-ranked courses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you have completed the function `generate_recommendation_scores()` above, you can test it and generate recommendation scores and save the courses recommendations into a dataframe with three columns: `USER`, `COURSE_ID`, `SCORE`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return users, courses, and scores lists for the dataframe\n",
    "users, courses, scores = generate_recommendation_scores()\n",
    "res_dict['USER'] = users\n",
    "res_dict['COURSE_ID'] = courses\n",
    "res_dict['SCORE'] = scores\n",
    "res_df = pd.DataFrame(res_dict, columns=['USER', 'COURSE_ID', 'SCORE'])\n",
    "# Save the dataframe \n",
    "res_df.to_csv(\"profile_rs_results.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>USER</th>\n",
       "      <th>COURSE_ID</th>\n",
       "      <th>SCORE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>37465</td>\n",
       "      <td>RP0105EN</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37465</td>\n",
       "      <td>GPXX06RFEN</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37465</td>\n",
       "      <td>CC0271EN</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37465</td>\n",
       "      <td>BD0145EN</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37465</td>\n",
       "      <td>DE0205EN</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53406</th>\n",
       "      <td>2087663</td>\n",
       "      <td>excourse88</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53407</th>\n",
       "      <td>2087663</td>\n",
       "      <td>excourse89</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53408</th>\n",
       "      <td>2087663</td>\n",
       "      <td>excourse90</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53409</th>\n",
       "      <td>2087663</td>\n",
       "      <td>excourse92</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53410</th>\n",
       "      <td>2087663</td>\n",
       "      <td>excourse93</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>53411 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          USER   COURSE_ID  SCORE\n",
       "0        37465    RP0105EN   27.0\n",
       "1        37465  GPXX06RFEN   12.0\n",
       "2        37465    CC0271EN   15.0\n",
       "3        37465    BD0145EN   24.0\n",
       "4        37465    DE0205EN   15.0\n",
       "...        ...         ...    ...\n",
       "53406  2087663  excourse88   15.0\n",
       "53407  2087663  excourse89   15.0\n",
       "53408  2087663  excourse90   15.0\n",
       "53409  2087663  excourse92   15.0\n",
       "53410  2087663  excourse93   15.0\n",
       "\n",
       "[53411 rows x 3 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your recommendation results may look like the following screenshot:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the course recommendation list generated for each test user, you also need to perform some analytic tasks to answer the following two questions:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   On average, how many new courses have been recommended per test user?\n",
    "*   What are the most frequently recommended courses? Return the top-10 commonly recommended courses across all test users.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, suppose we have only 3 test users, each user receives the following course recommendations:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   User1: \\['course1', 'course2']\n",
    "*   User2: \\['course3', 'course4']\n",
    "*   User3: \\['course3', 'course4', 'course5']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, the average recommended courses per user is: $(2 + 2 + 3) / 3 = 2.33$. The top-2 recommended courses are: `course3`: 2 times, and `course4`: 2 times.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the answers may depend on your score threshold. A lower score threshold yields more recommended courses but with smaller confidence so that some test users may receive very long course recommendation lists and feel overwhelmed.\n",
    "\n",
    "Ideally, we should limit the maximum course recommendations to be less than 20 courses per user. As such, the average course recommendations per user should also be less than 20 or so. This makes sure we only recommend relevant courses with high confidence (score).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you first learned how to generate a user profile vector based on the user's course ratings and course genre vectors. Then, with the user profile generated, you applied a simple dot product between the user profile vector and the course genre vector to generate a course recommendation score.\n",
    "\n",
    "The idea is if a user is interested in certain topics (genres) and if a course also has similar topics (genres), which means the user vector and course genre vector share common dimensions, and a dot product is able to capture such similarity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Yan Luo](https://www.linkedin.com/in/yan-luo-96288783/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMML321ENSkillsNetwork32585014-2022-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description          |\n",
    "| ----------------- | ------- | ---------- | --------------------------- |\n",
    "| 2021-10-25        | 1.0     | Yan        | Created the initial version |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2021 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}