Furniture Object Detector & Recommender using Detectron2 & VGG16

This repository contains all the code to complete iDECOR, a Furniture Recommender that help users who have recently moved to explore IKEA products at ease.

After uploading a room scene image, IDECOR returns users with similar-styled furnitures from IKEA product catalog.

LET'S SEE HOW IT WORKS!

Business Values of this Project

Amazon is a giant in ecommerce which has its 35% revenue generated by its recommendation engine. This proportion has potential to increase up to 60% based off its competitors.

This project is to drive conversions-to-sales of IKEA's and other furniture ecommerce stores, by providing better onsite recommendations.

Project Overview

• Developed a customized solution for furniture shoppers, by wrapping furniture detection and recommendation system into a seamless process on Streamlit.

• Leveraged 2 Deep Learning technologies:

  • Detectron2 (developed by Facebook AI): Detect every furniture item in user's uploaded image
  • VGG16 model: Detect design features and generate vector representations for respective furniture items

• Returned users with 5 most-similar furniture items, from product catalogue web-scraped from IKEA Hong Kong using Selenium

• Made iDECOR available across 6 furniture products, and will expand to more brands/ categories with API support in the future.

What is in this repo?

• app are some functions to deploy iDECOR with Streamlit.
• ikea contains programs to scrape IKEA product images and stores scraped data.
• img contains illustrations for this README.md
• model-evaluation are some plots made upon model inference.
• model contains the program and notebook that perform model training of Detectron2 and VGG16 models.
• open-images contains the codes to download and annotate images downloaded from ‘Open Images Dataset V6’
  • save annotation files here to run the program: https://drive.google.com/file/d/11dKuACP-K8PWYrQX-2br3Xj83ryQKbt4/view?usp=sharing
• iDECOR.pdf is a powerpoint which illustrates the complete framework of this project.

Table of Contents

• System Architecture
• Data Collection
• Data Preprocessing
• Modelling
  • Object Detection: Detectron2 & Faster R-CNN
  • Similarity Detection: VGG16 vs InceptionV3
• Evaluation
• Deployment
  • Streamlit

System Archirecture

The flowchart shows 5 main stages during product development with the tools and technologies used.

Data Collection

3 datasets are acquired for this project via API, direct download and web scraping respectively.

1. Annotated Furniture Images

   • Used in Object Detection Model training
   • Data Source: Open Images Dataset V6
   • Obtained around 10,000 images across 6 categories*
   • Images annotated with image-level labels, object bounding boxes

2. Style-labelled Furniture Images

   • Used in Similarity Detection Model training
   • Data Source: Bonn Furniture Style Dataset
   • Obtained around 13,000 images across 6 categories* and 15 styles
   • Style labelled by interior design professionals

3. IKEA Product Catalogue Dataset

   • Used as Recommendation Source
   • Data Source: IKEA Hong Kong
   • Web-scraped around 1,400 products in any category

*6 Categories: Bed, Cabinetry, Chair, Couch, Lamp, Table

Data Preprocessing

We executed some common pre-processing steps, and handle special cases at a later point to improve our results. Here are some major techniques we used in data cleaning:

• Balancing Categories in dataset
• Detectron2 Formatting with Annotations
• Mutli-label Binarizers: Encoding multiple labels per image instance
• Image Resizing
• Dropping Duplicates
• Image Augmentation

We also took advantage of the high computational power of Google Colab and Google Cloud Platform (GCP) to help proceeding to model training.

Model Engineering

Object Detection

Detectron2 was selected to perform Object Detection in this product, due to its modular design with numerous pre-trained models built in, that we could be adopt transfer learning more flexibly.

Faster R-CNN is a Region Proposal Network (RPN) introduced in object detection network to hypothesize object locations and classify object categories in images.

It performed shorter inference time and fewer training memory when compared to Mask R-CNN, and higher accuracy compared to Fast R-CNN.

Similarity Detection

We applied transfer learning of VGG16 and InceptionV3 as Similarity Detection models, using Tensorflow and Keras, to generate vectors that encode the design feature of the furniture detected.

Different number of blocks and dense layers are freezed to experiment on the best accuracy.

Recommender

After obtaining desgin feature vectors from last model, it will then query the feature index libraries using cosine similarity as the similarity search metrics.

Top 5 furnitures with the closest distances are returned as the recommended furnitures.

Evaluation

Deployment

To make iDECOR an interactive experience online, we created a data application using Streamlit framework. It takes few lines of code to get started, and is very intuitive to use, allowing us to prototype a beautiful machine learning application very efficiently.

---

Prerequisites

• CPU
• GPU x 1

Installation

• Detectron2 Library
  • Built on PyTorch
  • NVIDIA GPU required