SFND 2D Feature Tracking

The idea of the camera course is to build a collision detection system - that's the overall goal for the Final Project. As a preparation for this, you will now build the feature tracking part and test various detector / descriptor combinations to see which ones perform best. This mid-term project consists of four parts:

• First, you will focus on loading images, setting up data structures and putting everything into a ring buffer to optimize memory load.
• Then, you will integrate several keypoint detectors such as HARRIS, FAST, BRISK and SIFT and compare them with regard to number of keypoints and speed.
• In the next part, you will then focus on descriptor extraction and matching using brute force and also the FLANN approach we discussed in the previous lesson.
• In the last part, once the code framework is complete, you will test the various algorithms in different combinations and compare them with regard to some performance measures.

See the classroom instruction and code comments for more details on each of these parts. Once you are finished with this project, the keypoint matching part will be set up and you can proceed to the next lesson, where the focus is on integrating Lidar points and on object detection using deep-learning.

Dependencies for Running Locally

1. cmake >= 2.8

• All OSes: click here for installation instructions

2. make >= 4.1 (Linux, Mac), 3.81 (Windows)

• Linux: make is installed by default on most Linux distros
• Mac: install Xcode command line tools to get make
• Windows: Click here for installation instructions

3. OpenCV >= 4.1

• All OSes: refer to the official instructions
• This must be compiled from source using the -D OPENCV_ENABLE_NONFREE=ON cmake flag for testing the SIFT and SURF detectors. If using homebrew: $> brew install --build-from-source opencv will install required dependencies and compile opencv with the opencv_contrib module by default (no need to set -DOPENCV_ENABLE_NONFREE=ON manually).
• The OpenCV 4.1.0 source code can be found here

4. gcc/g++ >= 5.4

• Linux: gcc / g++ is installed by default on most Linux distros
• Mac: same deal as make - install Xcode command line tools
• Windows: recommend using either MinGW-w64 or Microsoft's VCPKG, a C++ package manager. VCPKG maintains its own binary distributions of OpenCV and many other packages. To see what packages are available, type vcpkg search at the command prompt. For example, once you've VCPKG installed, you can install OpenCV 4.1 with the command:

c:\vcpkg> vcpkg install opencv4[nonfree,contrib]:x64-windows

Then, add C:\vcpkg\installed\x64-windows\bin and C:\vcpkg\installed\x64-windows\debug\bin to your user's PATH variable. Also, set the CMake Toolchain File to c:\vcpkg\scripts\buildsystems\vcpkg.cmake.

Basic Build Instructions

1. Clone this repo.
2. Make a build directory in the top level directory: mkdir build && cd build
3. Compile: cmake .. && make
4. Run it: ./2D_feature_tracking.

Performance Evaluation

Top 4 Detector-Descriptor pairs

Detector	Descritor	Average Number of Keypoints Matched	Average Time Computation (ms)
FAST	BRIEF	35	1.30048
FAST	ORB	34	2.26665
SHITOMASI	ORB	100	9.54437
SHITOMASI	BRIEF	104	9.8693

List of all results

Keypoints Leading Vehicle count

Detector	Average Number of Keypoints
SHITOMASI	110
HARRIS	24
FAST	41
BRISK	276
ORB	116
AKAZE	167
SIFT	138

Keypoint Matched Leading Vehicle count and Time to compute keypoint detection and descriptor extraction

Detector	Descritor	Average Number of Keypoints Matched	Average Time Computation (ms)
AKAZE	AKAZE	125	82.4137
SHITOMASI	BRISK	76	38.8758
HARRIS	BRISK	14	39.9434
FAST	BRISK	27	28.3885
BRISK	BRISK	157	95.7682
ORB	BRISK	75	50.1811
SIFT	BRISK	59	84.9078
SHITOMASI	BRIEF	104	9.8693
HARRIS	BRIEF	19	14.5849
FAST	BRIEF	35	1.30048
BRISK	BRIEF	189	64.5819
ORB	BRIEF	60	21.0474
SIFT	BRIEF	78	72.5434
SHITOMASI	ORB	100	9.54437
HARRIS	ORB	17	14.6905
FAST	ORB	34	2.26665
BRISK	ORB	167	66.1544
ORB	ORB	84	23.576
SIFT	ORB	-	-
SHITOMASI	FREAK	85	37.2443
HARRIS	FREAK	16	38.1
FAST	FREAK	29	28.9637
BRISK	FREAK	169	90.299
ORB	FREAK	46	45.1091
SIFT	FREAK	66	97.0222
SHITOMASI	SIFT	103	23.8803
HARRIS	SIFT	18	25.389
FAST	SIFT	32	17.0679
BRISK	SIFT	182	84.5988
ORB	SIFT	84	43.5799
SIFT	SIFT	88	130.201