xfeatSLAM

Introduction

xfeatSLAM is a Visual SLAM system that integrates the lightweight and efficient XFeat architecture into the ORB-SLAM3 pipeline. This project is an experimental combination of ORB-SLAM3 with the XFeat model to create a SLAM system utilizing deep learning-based image descriptors.

Typically, models used to obtain deep learning-based local features provide accurate descriptions but are highly resource-intensive. The lightweight nature of XFeat makes it particularly well-suited for environments with limited processing power, such as mobile robots and embedded systems, where real-time performance is critical.

First, the entire XFeat model is implemented in C++ using PyTorch C++ API. I have separated the code for this implementation into this repo. This implementation is then integrated into the ORB-SLAM3 pipeline, following an approach similar to that used in GCNv2 SLAM.

Dependencies

C++17 Compiler

xfeatSLAM uses libtorch and requires a compiler that supports at least C++17. It has been tested with GCC and G++ compilers v11.4.0.

PyTorch

xfeatSLAM uses PyTorch C++ API (libtorch) for running the C++ Implementation of XFeat model.

Please avoid using the pre-built version of libtorch as it may cause linking issues (CXX11 ABI issue)

Pangolin

Pangolin is used for visualization and UI.

OpenCV

OpenCV is used for image and feature processing.

The version of OpenCV used for testing is v4.5.4.

Eigen3

Eigen3 is requried for matrix operations.

Project was tested using Eigen3 v3.4.0.

DBoW2, Sophus and g2o (included in thirdparty folder)

The project use modified versions of the DBoW2 library to perform place recognition, Sophus for Lie groups, and g2o library to perform non-linear optimizations. All three modified libraries (which are BSD) are included in the thirdparty folder.

Getting Started

To download xfeatSLAM:

git clone https://github.com/udaysankar01/xfeatSLAM
cd xfeatSLAM

To install all the necessary packages:

chmod +x project_setup.sh
./project_setup.sh

To build the project:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j8

RGB-D Example

To run deep RGB-D SLAM:

./examples/RGB-D/rgbd_tum Vocabulary/ORBvoc.txt \
                          examples/RGB-D/TUMX.yaml \
                          PATH_TO_SEQUENCE_FOLDER \
                          ASSOCIATIONS_FILE

To use ORB feature descriptor instead of XFeat, set the environment variable USE_ORB as 1.

USE_ORB=1 ./examples/RGB-D/rgbd_tum Vocabulary/ORBvoc.txt \
                                    examples/RGB-D/TUMX.yaml \
                                    PATH_TO_SEQUENCE_FOLDER \
                                    ASSOCIATIONS_FILE

Monocular Example

To run deep Monocular SLAM:

./examples/Monocular/mono_tum Vocabulary/ORBvoc.txt \
                              examples/Monocular/TUMX.yaml \
                              PATH_TO_SEQUENCE_FOLDER

Support for additional sensors will be added soon.

Bibtex Citation

@misc{potje2024xfeatacceleratedfeatureslightweight,
      title={XFeat: Accelerated Features for Lightweight Image Matching},
      author={Guilherme Potje and Felipe Cadar and Andre Araujo and Renato Martins and Erickson R. Nascimento},
      year={2024},
      eprint={2404.19174},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2404.19174},
}

@article{ORBSLAM3_TRO,
  title={{ORB-SLAM3}: An Accurate Open-Source Library for Visual, Visual-Inertial
           and Multi-Map {SLAM}},
  author={Campos, Carlos AND Elvira, Richard AND G\´omez, Juan J. AND Montiel,
          Jos\'e M. M. AND Tard\'os, Juan D.},
  journal={IEEE Transactions on Robotics},
  volume={37},
  number={6},
  pages={1874-1890},
  year={2021}
 }

@unknown{unknown,
author = {Tang, Jiexiong and Ericson, Ludvig and Folkesson, John and Jensfelt, Patric},
year = {2019},
month = {02},
pages = {},
title = {GCNv2: Efficient Correspondence Prediction for Real-Time SLAM}
}