notebooks/multimodal_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multimodal model\n",
    "This notebook lets you instantiate a model and run a forward pass from a multimodal sample. For this, you must have done the following:\n",
    "- downloaded and preprocessed the S3DIS dataset (if you haven't, instantiating the dataset will launch it for you though). You may edit this code to load any other multimodal dataset you have on your machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select you GPU\n",
    "I_GPU = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to use autoreload\n",
    "# %load_ext autoreload\n",
    "# %autoreload 2\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import torch\n",
    "from time import time\n",
    "from omegaconf import OmegaConf\n",
    "start = time()\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "torch.cuda.set_device(I_GPU)\n",
    "DIR = os.path.dirname(os.getcwd())\n",
    "ROOT = os.path.join(DIR, \"..\")\n",
    "sys.path.insert(0, ROOT)\n",
    "sys.path.insert(0, DIR)\n",
    "\n",
    "from torch_points3d.utils.config import hydra_read\n",
    "from torch_points3d.core.multimodal.data import MMBatch\n",
    "from torch_points3d.datasets.segmentation.multimodal.s3dis import S3DISFusedDataset\n",
    "from torch_points3d.models.model_factory import instantiate_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset and model configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset and model configurations are parsed in the following cell using Hydra. For the **multimodal semantic segmentation** task, dataset configs live in `conf/data/segmentation/multimodal` and model configs live in `conf/models/segmentation/multimodal`. You can create a new model there and run a forward pass on it in this notebook to debug it.\n",
    "\n",
    "For now, supported multimodal datasets are [S3DIS](http://buildingparser.stanford.edu/dataset.html), [ScanNet](http://www.scan-net.org/) and [KITTI-360](http://www.cvlibs.net/datasets/kitti-360), while supported multimodal architectures are based on [MinkowskiNet](https://arxiv.org/abs/1904.08755)-like backbones. It would be relatively, using Torch-Points3D, to extend the same models to other backbones such as [PointNet](https://arxiv.org/abs/1706.02413), [KP-Conv](https://arxiv.org/abs/1904.08889), etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set your dataset root directory, where the data was/will be downloaded\n",
    "DATA_ROOT = '/path/to/your/dataset/root/directory'\n",
    "\n",
    "config_file = 'segmentation/multimodal/s3disfused-sparse'  # dataset config, S3DIS here \n",
    "models_config = 'segmentation/multimodal/sparseconv3d'     # family of models based on sparseconv3d backbone\n",
    "model_name = 'Res16UNet34-L4-early-ade20k-interpolate'     # name of the specific model we want to use\n",
    "\n",
    "overrides = [\n",
    "    'task=segmentation',\n",
    "    f'data={config_file}',\n",
    "    f'models={models_config}',\n",
    "    f'model_name={model_name}',\n",
    "    f'data.dataroot={DATA_ROOT}',\n",
    "]\n",
    "\n",
    "cfg = hydra_read(overrides)\n",
    "# print(OmegaConf.to_yaml(cfg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset creation\n",
    "\n",
    "The dataset will now be created. If you have not downloaded or preprocessed the dataset before, it will be performed here (but this will take some time though). Otherwise, it will should load normally within a few seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset instantiation\n",
    "start = time()\n",
    "dataset = S3DISFusedDataset(cfg.data)\n",
    "# print(dataset)\n",
    "print(f\"Time = {time() - start:0.1f} sec.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model creation\n",
    "\n",
    "The following cell will instantiate teh model, based on the config. In Torch-Points3d, instantiating a model often requires information about the dataset (*e.g.* the number of classes). For this reason, `instantiate_model` requires the `dataset` to be passed as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model instantiation\n",
    "print(f\"Model: {cfg.model_name}\")\n",
    "model = instantiate_model(cfg, dataset)\n",
    "model = model.train().cuda()\n",
    "n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "print(f\"Model parameters : {n_params / 10**6:0.1f} M\")\n",
    "# print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward pass on a multimodal batch\n",
    "\n",
    "We can now create a batch of `batch_size` multimodal samples from `dataset` and run a forward and backward pass on `model`. This can help us debug the model before launching a full training experiemnt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 4\n",
    "\n",
    "# Create a batch of multimodal samples\n",
    "print(f\"\\nBatch creation\")\n",
    "batch = MMBatch.from_mm_data_list([dataset.train_dataset[i] for i in range(batch_size)])\n",
    "# print(batch)\n",
    "\n",
    "# Set some model attributes based on the input batch. Moves batch to \n",
    "# device\n",
    "print(f\"\\nForward pass\")\n",
    "model.set_input(batch, model.device)\n",
    "\n",
    "# Forward pass. Output will be stored in model attributes\n",
    "batch = model(batch)\n",
    "\n",
    "# Loss belongs to the model attributes and is automatically computed \n",
    "# when running forward pass\n",
    "print(f\"\\nLoss\")\n",
    "model.loss_seg\n",
    "\n",
    "# Backward pass\n",
    "print(f\"\\nBackward pass\")\n",
    "model.backward()\n",
    "\n",
    "print(f\"\\nOK\")\n",
    "\n",
    "del batch"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tp3d",
   "language": "python",
   "name": "tp3d"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}