Reduce, Reuse, Recycle: Modular Multi-Object Navigation

This is an implementation of our paper Reduce, Reuse, Recycle: Modular Multi-Object Navigation. webpage

Architecture Overview

Installing dependencies:

This code is tested on python 3.8.13, pytorch v1.11.0 and CUDA V11.2. Install pytorch from https://pytorch.org/ according to your machine configuration.

conda create -n mon python=3.8 cmake=3.14.0
conda activate mon

This code uses forked versions of habitat-sim and habitat-lab.

Installing habitat-sim:

For headless machines with GPU

git clone [email protected]:sonia-raychaudhuri/habitat-sim.git
cd habitat-sim
python -m pip install -r requirements.txt
python setup.py build_ext --parallel 4 install --headless --bullet 

For machines with attached display

git clone [email protected]:sonia-raychaudhuri/habitat-sim.git
cd habitat-sim
python -m pip install -r requirements.txt
python setup.py build_ext --parallel 4 install --bullet 

Installing habitat-lab:

git clone [email protected]:sonia-raychaudhuri/habitat-lab.git
cd habitat-lab
pip install -e .

Setup

Clone the repository and install the requirements:

git clone [email protected]:3dlg-hcvc/multi-obj-nav.git
cd multi-obj-nav
python -m pip install -r requirements.txt

Downloading data and checkpoints

Download HM3D scenes here and place the data in: multi-obj-nav/data/scene_datasets/hm3d.

Download objects:

wget -O multion_cyl_objects.zip "https://aspis.cmpt.sfu.ca/projects/multion-challenge/2022/challenge/dataset/multion_cyl_objects"
wget -O multion_real_objects.zip "https://aspis.cmpt.sfu.ca/projects/multion-challenge/2022/challenge/dataset/multion_real_objects"

Extract them under multi-obj-nav/data.

Download the dataset.

# Replace {n} with 1, 3, 5 for 1ON, 3ON & 5ON respectively; Replace {obj_type} with CYL or REAL for Cylinder and Real/Natural objects respectively; Replace {split} with minival, val or train for different data splits.

wget -O {n}_ON_{obj_type}_{split}.zip "https://aspis.cmpt.sfu.ca/projects/multion-challenge/2022/challenge/dataset/{n}_ON_{obj_type}_{split}"

Extract them and place them inside multi-obj-nav/data in the following format:

multi-obj-nav/
  data/
    scene_datasets/
      hm3d/
          ...
    multion_cyl_objects/
        ...
    multion_real_objects/
        ...
    5_ON_CYL/
        train/
            content/
                ...
            train.json.gz
        minival/
            content/
                ...
            minival.json.gz
        val/
            content/
                ...
            val.json.gz
    5_ON_REAL/
        train/
            content/
                ...
            train.json.gz
        minival/
            content/
                ...
            minival.json.gz
        val/
            content/
                ...
            val.json.gz

Usage

Pre-trained models

Download the pretrained PointNav model, trained on HM3D here and update the path here and here.

Download the following checkpoints for Object Detection and place under multi-obj-nav/data/object_detection_models:

wget "https://aspis.cmpt.sfu.ca/projects/multion/rrr/pretrained_models/obj_det_real.ckpt"
wget "https://aspis.cmpt.sfu.ca/projects/multion/rrr/pretrained_models/obj_det_cylinder.ckpt"
wget "https://aspis.cmpt.sfu.ca/projects/multion/rrr/pretrained_models/knn_colors.pkl"

Evaluation

Evaluation will run on the 3_ON val set by default.

# For evaluating with OraSem agent on 3ON cylinders dataset
python run.py  --exp-config baselines/config/pointnav/hier_w_proj_ora_sem_map.yaml --run-type eval

# For evaluating with OraSem agent on 3ON real/natural objects dataset
python run.py  --exp-config baselines/config/pointnav/hier_w_proj_ora_sem_map_real.yaml --run-type eval

# For evaluating with PredSem agent on 3ON cylinders dataset
python run.py  --exp-config baselines/config/pointnav/hier_w_proj_pred_sem_map.yaml --run-type eval

# For evaluating with PredSem agent on 3ON real/natural objects dataset
python run.py  --exp-config baselines/config/pointnav/hier_w_proj_pred_sem_map_real.yaml --run-type eval

Citation

Sonia Raychaudhuri, Tommaso Campari, Unnat Jain, Manolis Savva, Angel X. Chang, 2023. Reduce, Reuse, Recycle: Modular Multi-Object Navigation. PDF

Bibtex

  @misc{raychaudhuri2023reduce,
    title={Reduce, Reuse, Recycle: Modular Multi-Object Navigation}, 
    author={Sonia Raychaudhuri and Tommaso Campari and Unnat Jain and Manolis Savva and Angel X. Chang},
    year={2023},
    eprint={2304.03696},
    archivePrefix={arXiv},
}

Acknowledgements

The members at SFU were supported by Canada CIFAR AI Chair grant, Canada Research Chair grant, NSERC Discovery Grant and a research grant by Facebook AI Research. Experiments at SFU were enabled by support from WestGrid and Compute Canada. TC was supported by the PNRR project Future AI Research (FAIRPE00000013), under the NRRP MUR program funded by the NextGenerationEU. We also thank Angelica, Jiayi, Shawn, Bita, Yongsen, Arjun, Justin, Matthew, and Shivansh for comments on early drafts of this paper. This repository is built upon Habitat Lab and multiON.