Benchmark.md

Benchmark

Environment

This benchmark is tested on CentOS7 with GPU A10 and CPU Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz, and its environment is as follows:

Package	Version
CUDA	11.3
cuDNN	8.3.2.44
TensorRT	8.4.1.5
Python	3.8.13
PyTorch	1.12.1

Performance

The following is the result of comparison between pytorch eager and poros, which measured by average latency time (ms) of model infering 1000 times.

1. ResNet50

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	6.17	1.70
2	6.02	2.41
4	6.33	3.23
8	8.55	4.75
16	16.22	7.82
32	32.09	14.00

Model source: https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py

2. VGG16

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	3.20	2.71
2	4.97	3.78
4	8.20	6.09
8	14.64	10.20
16	27.47	19.17
32	53.09	36.47

Model source: https://github.com/pytorch/vision/blob/main/torchvision/models/vgg.py

3. MobileNetV2

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	3.85	0.65
2	3.75	0.86
4	3.90	1.19
8	4.18	2.08
16	8.43	3.83
32	16.57	7.14

Model source: https://github.com/tonylins/pytorch-mobilenet-v2/blob/master/MobileNetV2.py

4. InceptionV3

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	10.05	2.51
2	10.13	3.22
4	10.08	3.70
8	10.15	4.95
16	12.51	7.11
32	21.43	11.22

Model source: https://github.com/pytorch/vision/blob/main/torchvision/models/inception.py

5. Efficientnet_b0

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	8.28	1.28
2	8.50	1.57
4	8.49	2.29
8	8.83	3.65
16	10.65	6.62
32	20.51	12.51

Model source: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/efficientnet.py

6. Bert-base-uncased

Input shape: bx128

Batch size	PyTorch (ms)	Poros (ms)
1	6.40	2.02
2	7.14	2.59
4	11.58	4.39
8	21.64	8.41
16	44.20	16.90
32	92.69	32.21

Model source: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py

7. Vision Transformer (ViT)

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	6.38	3.07
2	10.35	4.57
4	19.06	8.37
8	36.71	16.34
16	73.84	29.92
32	147.70	58.11

Model source: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/vision_transformer.py

8. YOLOv5s

Input shape: bx3x640x640

Batch size	PyTorch (ms)	Poros (ms)
1	6.17	2.22
2	5.93	3.96
4	10.02	6.84
8	20.02	12.86
16	38.17	24.80
32	77.19	49.16

Model source: https://github.com/ultralytics/yolov5/blob/master/models/yolo.py

9. Swin Transformer

Input shape: bx3x224x224

Batch size	PyTorch (ms)	Poros (ms)
1	14.11	7.68
2	22.73	11.99
4	42.21	21.74
8	83.07	42.18
16	162.34	78.34
32	317.43	149.72

Model source: https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py