[Model] Add Paddle3D smoke model (PaddlePaddle#1766)

* add smoke model * add 3d vis * update code * update doc * mv paddle3d from detection to perception * update result for velocity * update code for CI * add set input data for TRT backend * add serving support for smoke model * update code * update code * update code --------- Co-authored-by: DefTruth <[email protected]>
Petal99 · Apr 14, 2023 · a509dd8 · a509dd8
1 parent 81fbd54
commit a509dd8
Show file tree

Hide file tree

Showing 53 changed files with 2,610 additions and 26 deletions.
diff --git a/cmake/opencv.cmake b/cmake/opencv.cmake
@@ -199,6 +199,6 @@ else()
     endif()
     find_package(OpenCV REQUIRED PATHS ${OpenCV_DIR} NO_DEFAULT_PATH)
     include_directories(${OpenCV_INCLUDE_DIRS})
-    list(APPEND DEPEND_LIBS opencv_core opencv_video opencv_highgui opencv_imgproc opencv_imgcodecs)
+    list(APPEND DEPEND_LIBS opencv_core opencv_video opencv_highgui opencv_imgproc opencv_imgcodecs opencv_calib3d opencv_features2d opencv_flann)
   endif()
 endif()
diff --git a/docs/api/vision_results/perception_result.md b/docs/api/vision_results/perception_result.md
@@ -0,0 +1,48 @@
+English | [简体中文](detection_result_CN.md)
+# PerceptionResult target detection result
+
+The PerceptionResult code is defined in `fastdeploy/vision/common/result.h`, which is used to indicate the detected 3D target: two-dimensional target frame, target frame length, width and height, target category and target confidence, target orientation angle and observation angle etc.
+
+## C++ definition
+
+```c++
+fastdeploy::vision::PerceptionResult
+```
+
+```c++
+struct PerceptionResult {
+   std::vector<float> scores;
+   std::vector<int32_t> label_ids;
+   std::vector<std::array<float, 7>> boxes;
+   std::vector<std::array<float, 3>> center;
+   std::vector<float>observation_angle;
+   std::vector<float>yaw_angle;
+   std::vector<std::array<float, 3>>velocity;
+   void Clear();
+   std::string Str();
+};
+```
+
+- **scores**: Member variable, indicating the confidence of all detected targets, `scores.size()` indicates the number of detected boxes
+- **label_ids**: Member variable, representing all detected target categories, the number of elements is consistent with `scores.size()`
+- **boxes**: Member variable, representing the coordinates of all detected target boxes, the number of elements is consistent with `scores.size()`, and each box represents xmin, ymin, xmax, ymax in turn with 7 float values , h, w, l, the coordinates of the upper left and lower right corners and the length, width and height of the 3D box
+- **center**: member variable, indicating the center point coordinates of all detected target frames, the number of elements is consistent with `scores.size()`, and each frame uses 3 float values to represent the center point coordinates of the frame in turn
+- **observation_angle**: Member variable, indicating the observation angle of the detected frame, and the number of elements is consistent with `scores.size()`
+- **yaw_angle**: Member variable, indicating the orientation angle of the detected frame, the number of elements is consistent with `scores.size()`
+- **velocity**: Member variable, indicating the velocity of the detected frame, the number of elements is consistent with `scores.size()`
+- **Clear()**: member function, used to clear the results stored in the structure
+- **Str()**: member function, output the information in the structure as a string (for Debug)
+
+## Python definition
+
+```python
+fastdeploy.vision.PerceptionResult
+```
+
+- **scores**(list of float): Member variable, indicating the confidence of all detected targets, `scores.size()` indicates the number of detected frames
+- **label_ids**(list of int): Member variable, representing all detected target categories, the number of elements is consistent with `scores.size()`
+- **boxes**(list of list(float)): Member variable, indicating the coordinates of all detected target boxes, the number of elements is the same as `scores.size()`, and each box is in order of 7 float values Indicates xmin, ymin, xmax, ymax, h, w, l, that is, the coordinates of the upper left and lower right corners and the length, width and height of the 3D box
+- **center**(list of list(float)): Member variable, which represents the coordinates of the center points of all detected target frames, the number of elements is the same as `scores.size()`, and each frame is represented by 3 floats The values in turn represent the coordinates of the center point of the box
+- **observation_angle**: member variable, indicating the orientation angle of the detected frame, and the number of elements is consistent with `scores.size()`
+- **yaw_angle**: Member variable, indicating the orientation angle of the detected frame, the number of elements is consistent with `scores.size()`
+- **velocity**: Member variable, indicating the velocity of the detected frame, the number of elements is consistent with `scores.size()`
diff --git a/docs/api/vision_results/perception_result_CN.md b/docs/api/vision_results/perception_result_CN.md
@@ -0,0 +1,48 @@
+简体中文 ｜ [English](perception_result.md)
+# PerceptionResult 目标检测结果
+
+PerceptionResult`fastdeploy/vision/common/result.h`中，用于表明检测出来的3D目标的：二维目标框、目标框长宽高、目标类别和目标置信度、目标朝向角和观测角等。
+
+## C++ 定义
+
+```c++
+fastdeploy::vision::PerceptionResult
+```  
+
+```c++
+struct PerceptionResult {
+  std::vector<float> scores;
+  std::vector<int32_t> label_ids;
+  std::vector<std::array<float, 7>> boxes;
+  std::vector<std::array<float, 3>> center;
+  std::vector<float>observation_angle;
+  std::vector<float>yaw_angle;
+  std::vector<std::array<float, 3>>velocity;
+  void Clear();
+  std::string Str();
+};
+```
+
+- **scores**: 成员变量，表示检测出来的所有目标置信度，`scores.size()`表示检测出来框的个数
+- **label_ids**: 成员变量，表示检测出来的所有目标类别，其元素个数与`scores.size()`一致
+- **boxes**: 成员变量，表示检测出来的所有目标框坐标，其元素个数与`scores.size()`一致，每个框以7个float数值依次表示xmin, ymin, xmax, ymax，h, w, l， 即左上角和右下角坐标以及3D框的长宽高
+- **center**: 成员变量，表示检测出来的所有目标框中心点坐标，其元素个数与`scores.size()`一致，每个框以3个float数值依次表示框中心点坐标
+- **observation_angle**: 成员变量，表示检测出来的框的观测角，其元素个数与`scores.size()`一致
+- **yaw_angle**: 成员变量，表示检测出来的框的朝向角，其元素个数与`scores.size()`一致
+- **velocity**: 成员变量，表示检测出来的框的速度，其元素个数与`scores.size()`一致
+- **Clear()**: 成员函数，用于清除结构体中存储的结果
+- **Str()**: 成员函数，将结构体中的信息以字符串形式输出（用于Debug）
+
+## Python 定义
+
+```python
+fastdeploy.vision.PerceptionResult  
+```
+
+- **scores**(list of float): 成员变量，表示检测出来的所有目标置信度，`scores.size()`表示检测出来框的个数
+- **label_ids**(list of int): 成员变量，表示检测出来的所有目标类别，其元素个数与`scores.size()`一致
+- **boxes**(list of list(float)): 成员变量，表示检测出来的所有目标框坐标，其元素个数与`scores.size()`一致，每个框以7个float数值依次表示xmin, ymin, xmax, ymax，h, w, l， 即左上角和右下角坐标以及3D框的长宽高
+- **center**(list of list(float)): 成员变量，表示检测出来的所有目标框中心点坐标，其元素个数与`scores.size()`一致，每个框以3个float数值依次表示框中心点坐标
+- **observation_angle**: 成员变量，表示检测出来的框的朝向角，其元素个数与`scores.size()`一致
+- **yaw_angle**: 成员变量，表示检测出来的框的朝向角，其元素个数与`scores.size()`一致
+- **velocity**: 成员变量，表示检测出来的框的速度，其元素个数与`scores.size()`一致
diff --git a/examples/vision/perception/paddle3d/smoke/README.md b/examples/vision/perception/paddle3d/smoke/README.md
@@ -0,0 +1,11 @@
+English | [简体中文](README_CN.md)
+
+# Smoke Ready-to-deploy Model
+
+The Smoke deployment model implements the Smoke model from Paddle3D. For more detailed information about the model, please refer to [Smoke Introduction](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
+
+
+## Detailed Deployment Documents
+
+- [Python Deployment](python)
+- [C++ Deployment](cpp)
diff --git a/examples/vision/perception/paddle3d/smoke/README_CN.md b/examples/vision/perception/paddle3d/smoke/README_CN.md
@@ -0,0 +1,12 @@
+[English](README.md) | 简体中文
+
+# Smoke 准备部署模型
+
+Smoke 部署模型实现来自 Paddle3D 的 Smoke 模型，模型相关的更多详细信息可以参考[Smoke 介绍](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
+
+
+
+## 详细部署文档
+
+- [Python部署](python)
+- [C++部署](cpp)
diff --git a/examples/vision/perception/paddle3d/smoke/cpp/CMakeLists.txt b/examples/vision/perception/paddle3d/smoke/cpp/CMakeLists.txt
@@ -0,0 +1,14 @@
+PROJECT(infer_demo C CXX)
+CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
+
+# 指定下载解压后的fastdeploy库路径
+option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
+
+include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
+
+# 添加FastDeploy依赖头文件
+include_directories(${FASTDEPLOY_INCS})
+
+add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
+# 添加FastDeploy库依赖
+target_link_libraries(infer_demo ${FASTDEPLOY_LIBS})
diff --git a/examples/vision/perception/paddle3d/smoke/cpp/README.md b/examples/vision/perception/paddle3d/smoke/cpp/README.md
@@ -0,0 +1,80 @@
+English | [简体中文](README_CN.md)
+# Smoke C++ Deployment Example
+
+This directory provides an example of `infer.cc` to quickly complete the deployment of Smoke on CPU/GPU.
+
+Before deployment, the following two steps need to be confirmed
+
+- 1. The hardware and software environment meets the requirements, refer to [FastDeploy environment requirements](../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
+- 2. According to the development environment, download the precompiled deployment library and samples code, refer to [FastDeploy prebuilt library](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
+
+Taking CPU inference on Linux as an example, execute the following command in this directory to complete the compilation test. To support this model, you need to ensure FastDeploy version 1.0.6 or higher (x.x.x>=1.0.6)
+
+```bash
+mkdir build
+cd build
+# Download the FastDeploy precompiled library, users can choose the appropriate version to use in the `FastDeploy precompiled library` mentioned above
+wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
+tar xvf fastdeploy-linux-x64-x.x.x.tgz
+cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
+make -j
+
+wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
+tar -xf smoke.tar.gz
+wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
+
+# CPU
+./infer_demo smoke smoke_test.png 0
+# GPU
+./infer_demo smoke smoke_test.png 1
+
+```
+
+The visual result after running is shown in the figure below
+
+<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
+
+The above commands are only applicable to Linux or MacOS. For the usage of SDK under Windows, please refer to:
+- [How to use FastDeploy C++ SDK in Windows](../../../../../docs/en/faq/use_sdk_on_windows.md)
+
+## Smoke C++ interface
+
+### Class Smoke
+
+```c++
+fastdeploy::vision::detection::Smoke(
+        const string& model_file,
+        const string& params_file,
+        const string& config_file,
+        const RuntimeOption& runtime_option = RuntimeOption(),
+        const ModelFormat& model_format = ModelFormat::PADDLE)
+```
+
+Smoke model loading and initialization.
+
+**parameter**
+
+> * **model_file**(str): model file path
+> * **params_file**(str): parameter file path
+> * **config_file**(str): configuration file path
+> * **runtime_option**(RuntimeOption): Backend reasoning configuration, the default is None, that is, the default configuration is used
+> * **model_format**(ModelFormat): model format, the default is Paddle format
+
+#### Predict function
+
+> ```c++
+> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
+> ```
+>
+> Model prediction interface, the input image directly outputs the detection result.
+>
+> **parameters**
+>
+> > * **im**: input image, note that it must be in HWC, BGR format
+> > * **result**: Detection result, including the detection frame, the confidence of each frame, PerceptionResult description reference [visual model prediction results](../../../../../docs/api /vision_results/)
+
+
+- [Model Introduction](../../)
+- [Python deployment](../python)
+- [Vision Model Prediction Results](../../../../../docs/api/vision_results/)
+- [How to switch model inference backend engine](../../../../../docs/en/faq/how_to_change_backend.md)
diff --git a/examples/vision/perception/paddle3d/smoke/cpp/README_CN.md b/examples/vision/perception/paddle3d/smoke/cpp/README_CN.md
@@ -0,0 +1,80 @@
+[English](README.md) | 简体中文
+# Smoke C++部署示例
+
+本目录下提供 `infer.cc` 快速完成 Smoke 在 CPU/GPU 上部署的示例。
+
+在部署前，需确认以下两个步骤
+
+- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)  
+- 2. 根据开发环境，下载预编译部署库和samples代码，参考[FastDeploy预编译库](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
+
+以Linux上 CPU 推理为例，在本目录执行如下命令即可完成编译测试，支持此模型需保证 FastDeploy 版本1.0.6以上(x.x.x>=1.0.6)
+
+```bash
+mkdir build
+cd build
+# 下载FastDeploy预编译库，用户可在上文提到的`FastDeploy预编译库`中自行选择合适的版本使用
+wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
+tar xvf fastdeploy-linux-x64-x.x.x.tgz
+cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
+make -j
+
+wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
+tar -xf smoke.tar.gz
+wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
+
+# CPU推理
+./infer_demo smoke smoke_test.png 0
+# GPU推理
+./infer_demo smoke smoke_test.png 1
+
+```
+
+运行完成可视化结果如下图所示
+
+<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
+
+以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:  
+- [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md)
+
+## Smoke C++ 接口
+
+### Smoke 类
+
+```c++
+fastdeploy::vision::detection::Smoke(
+        const string& model_file,
+        const string& params_file,
+        const string& config_file,
+        const RuntimeOption& runtime_option = RuntimeOption(),
+        const ModelFormat& model_format = ModelFormat::PADDLE)
+```
+
+Smoke模型加载和初始化。
+
+**参数**
+
+> * **model_file**(str): 模型文件路径
+> * **params_file**(str): 参数文件路径
+> * **config_file**(str): 配置文件路径
+> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
+> * **model_format**(ModelFormat): 模型格式，默认为Paddle格式
+
+#### Predict函数
+
+> ```c++
+> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
+> ```
+>
+> 模型预测接口，输入图像直接输出检测结果。
+>
+> **参数**
+>
+> > * **im**: 输入图像，注意需为HWC，BGR格式
+> > * **result**: 检测结果，包括检测框，各个框的置信度, PerceptionResult 说明参考[视觉模型预测结果](../../../../../docs/api/vision_results/)
+
+
+- [模型介绍](../../)
+- [Python部署](../python)
+- [视觉模型预测结果](../../../../../docs/api/vision_results/)
+- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)
diff --git a/examples/vision/perception/paddle3d/smoke/cpp/infer.cc b/examples/vision/perception/paddle3d/smoke/cpp/infer.cc
@@ -0,0 +1,83 @@
+// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "fastdeploy/vision.h"
+#ifdef WIN32
+const char sep = '\\';
+#else
+const char sep = '/';
+#endif
+
+void InitAndInfer(const std::string& model_dir, const std::string& image_file,
+                  const fastdeploy::RuntimeOption& option) {
+  auto model_file = model_dir + sep + "smoke.pdmodel";
+  auto params_file = model_dir + sep + "smoke.pdiparams";
+  auto config_file = model_dir + sep + "infer_cfg.yml";
+  fastdeploy::vision::EnableFlyCV();
+  auto model = fastdeploy::vision::perception::Smoke(
+      model_file, params_file, config_file, option,
+      fastdeploy::ModelFormat::PADDLE);
+  assert(model.Initialized());
+
+  auto im = cv::imread(image_file);
+
+  fastdeploy::vision::PerceptionResult res;
+  if (!model.Predict(im, &res)) {
+    std::cerr << "Failed to predict." << std::endl;
+    return;
+  }
+  std::cout << res.Str() << std::endl;
+
+  auto vis_im = fastdeploy::vision::VisPerception(im, res, config_file);
+  cv::imwrite("vis_result.jpg", vis_im);
+  std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
+}
+
+int main(int argc, char* argv[]) {
+  if (argc < 4) {
+    std::cout << "Usage: infer_demo path/to/paddle_model"
+                 "path/to/image "
+                 "run_option, "
+                 "e.g ./infer_demo ./smoke ./00000.png 0"
+              << std::endl;
+    std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
+                 "with gpu; 2: run with paddle-trt"
+              << std::endl;
+    return -1;
+  }
+
+  fastdeploy::RuntimeOption option;
+  if (std::atoi(argv[3]) == 0) {
+    option.UseCpu();
+  } else if (std::atoi(argv[3]) == 1) {
+    option.UseGpu();
+  } else if (std::atoi(argv[3]) == 2) {
+    option.UseGpu();
+    option.UseTrtBackend();
+    option.EnablePaddleToTrt();
+    option.SetTrtInputShape("images", {1, 3, 384, 1280});
+    option.SetTrtInputShape("down_ratios", {1, 2});
+    option.SetTrtInputShape("trans_cam_to_img", {1, 3, 3});
+    option.SetTrtInputData("trans_cam_to_img",
+                           {721.53771973, 0., 609.55932617, 0., 721.53771973,
+                            172.85400391, 0, 0, 1});
+    option.EnablePaddleTrtCollectShape();
+  }
+  option.UsePaddleBackend();
+
+  std::string model_dir = argv[1];
+  std::string test_image = argv[2];
+  InitAndInfer(model_dir, test_image, option);
+  return 0;
+}