Pytorch Face Landmark Detection

Implementation of face landmark detection with PyTorch. The models were trained using coordinate-based and heatmap-based regression methods. A video demo and image detection results were displayed here.

• Support 68-point and 39-point landmark inference.
• Support different backbone networks.
• Support ONNX inference.
• Support heatmap-based inference.

Inference

Test on a sample folder and save the landmark detection results. Support different backbones.

python3 test_batch_mtcnn_multiple.py --backbone MobileNet or PFLD or MobileFaceNet

Optimize with ONNX and test on a camera with MTCNN as a face detector.

python3 test_camera_mtcnn_onnx.py

Optimize with ONNX and test on a camera with a lightweight face detector. It can achieve real-time speed on CPU.

python3 test_camera_light_onnx.py

Benchmark Results on 300W

• Inter-ocular Normalization (ION)

Algorithms	Common	Challenge	Full Set	Param # (M)
ResNet18 (224×224)	3.73	7.14	4.39	11.76
Res2Net50_v1b (224×224)	3.43	6.77	4.07	26.00
Res2Net50_v1b_SE (224×224)	3.37	6.67	4.01	27.05
Res2Net50_v1b_ExternalData (224×224)	3.30	5.92	3.81	26.00
HRNet_w18_small_v2 (224×224)	3.57	6.85	4.20	13.83

• Inter-ocular Normalization (ION) with Lightweight Models

Algorithms	Common	Challenge	Full Set	Param # (M)	CPU Inference (s)
MobileNetV2 (224×224)	3.70	7.27	4.39	3.74	1.2
MobileNetV2_SE (224×224)	3.63	7.01	4.28	4.15	/
MobileNetV2_SE_RE (224×224)	3.63	6.66	4.21	4.15	/
MobileNetV2_ExternalData (224×224)	3.48	6.0	3.96	3.74	1.2
MobileNetV2 (56×56)	4.50	8.50	5.27	3.74	0.01 (onnx)
MobileNetV2_SE_ExternalData (56×56)	4.10	6.89	4.64	4.10	0.01 (onnx)
PFLD_ExternalData (112×112)	3.59	6.22	4.09	0.73	0.01 (onnx)
MobileFaceNet_ExternalData (112×112)	3.47	6.08	3.97	1.01	/
MobileFaceNet_SE_ExternalData (112×112)	3.45	5.96	3.94	1.03	/

Note: SE (squeeze-and-excitation module), RE (random erasing module).

• Heatmap Inference (still under test)

Algorithms	Common	Challenge	Full Set	Param # (M)
Hourglass2	3.06	5.54	3.55	8.73

Visualization Results

• Face alignment on 300W dataset

• Semi-frontal face alignment on Menpo dataset

• Profile face alignment on Menpo dataset

TODO

The following features will be added soon.

• Still to come:
  • Support for the 39-point detection
  • Support for the 106 point detection
  • Support for heatmap-based inferences

Public Datasets:

• 68-point: 300W, Menpo, 300-VW, 300W-LP, 300W-Style
• 98-point: WFLW
• 106-point: JD

References:

• https://github.com/rwightman/pytorch-image-models
• https://github.com/Res2Net/Res2Net-PretrainedModels
• https://github.com/HRNet/HRNet-Image-Classification
• https://github.com/lzx1413/pytorch_face_landmark
• https://github.com/polarisZhao/PFLD-pytorch