[Model] add style transfer model (PaddlePaddle#922)

* add style transfer model * add examples for generation model * add unit test * add speed comparison * add speed comparison * add variable for constant * add preprocessor and postprocessor * add preprocessor and postprocessor * fix * fix according to review Co-authored-by: DefTruth <[email protected]>
zyl75391 · Jan 3, 2023 · 87bcb5d · 87bcb5d
1 parent f72846c
commit 87bcb5d
Show file tree

Hide file tree

Showing 23 changed files with 966 additions and 1 deletion.
diff --git a/examples/vision/generation/anemigan/README.md b/examples/vision/generation/anemigan/README.md
@@ -0,0 +1,36 @@
+# 图像生成模型
+
+FastDeploy目前支持PaddleHub预训练模型库中如下风格迁移模型的部署
+
+| 模型 | 说明 | 模型格式 |
+| :--- | :--- | :------- |
+|[animegan_v1_hayao_60](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v1_hayao_60&en_category=GANs)|可将输入的图像转换成宫崎骏动漫风格，模型权重转换自AnimeGAN V1官方开源项目|paddle|
+|[animegan_v2_paprika_97](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_paprika_97&en_category=GANs)|可将输入的图像转换成今敏红辣椒动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_hayao_64](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_hayao_64&en_category=GANs)|可将输入的图像转换成宫崎骏动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_shinkai_53](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_shinkai_53&en_category=GANs)|可将输入的图像转换成新海诚动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_shinkai_33](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_shinkai_33&en_category=GANs)|可将输入的图像转换成新海诚动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_paprika_54](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_paprika_54&en_category=GANs)|可将输入的图像转换成今敏红辣椒动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_hayao_99](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_hayao_99&en_category=GANs)|可将输入的图像转换成宫崎骏动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_paprika_74](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_paprika_74&en_category=GANs)|可将输入的图像转换成今敏红辣椒动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+|[animegan_v2_paprika_98](https://www.paddlepaddle.org.cn/hubdetail?name=animegan_v2_paprika_98&en_category=GANs)|可将输入的图像转换成今敏红辣椒动漫风格，模型权重转换自AnimeGAN V2官方开源项目|paddle|
+
+## FastDeploy paddle backend部署和hub速度对比(ips, 越高越好)
+| Device | FastDeploy | Hub |
+| :--- | :--- | :------- |
+|  CPU   |  0.075     | 0.069|
+|  GPU   |  8.33      | 8.26 |
+
+
+
+## 下载预训练模型
+使用fastdeploy.download_model即可以下载模型, 例如下载animegan_v1_hayao_60
+```python
+import fastdeploy as fd
+fd.download_model(name='animegan_v1_hayao_60', path='./', format='paddle')
+```
+将会在当前目录获得animegan_v1_hayao_60的预训练模型。
+
+## 详细部署文档
+
+- [Python部署](python)
+- [C++部署](cpp)
diff --git a/examples/vision/generation/anemigan/cpp/CMakeLists.txt b/examples/vision/generation/anemigan/cpp/CMakeLists.txt
@@ -0,0 +1,13 @@
+PROJECT(infer_demo C CXX)
+CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
+
+# 指定下载解压后的fastdeploy库路径
+option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
+include(${FASTDEPLOY_INSTALL_DIR}/utils/gflags.cmake)
+include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
+
+# 添加FastDeploy依赖头文件
+include_directories(${FASTDEPLOY_INCS})
+
+add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
+target_link_libraries(infer_demo ${FASTDEPLOY_LIBS} ${GFLAGS_LIBRARIES})
diff --git a/examples/vision/generation/anemigan/cpp/README.md b/examples/vision/generation/anemigan/cpp/README.md
@@ -0,0 +1,84 @@
+# AnimeGAN C++部署示例
+
+本目录下提供`infer.cc`快速完成AnimeGAN在CPU/GPU部署的示例。
+
+在部署前，需确认以下两个步骤
+
+- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)  
+- 2. 根据开发环境，下载预编译部署库和samples代码，参考[FastDeploy预编译库](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
+
+以Linux上AnimeGAN推理为例，在本目录执行如下命令即可完成编译测试，支持此模型需保证FastDeploy版本1.0.2以上(x.x.x>=1.0.2)
+
+```bash
+mkdir build
+cd build
+# 下载FastDeploy预编译库，用户可在上文提到的`FastDeploy预编译库`中自行选择合适的版本使用
+wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
+tar xvf fastdeploy-linux-x64-x.x.x.tgz
+cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
+make -j
+
+# 下载准备好的模型文件和测试图片
+wget https://bj.bcebos.com/paddlehub/fastdeploy/style_transfer_testimg.jpg
+wget https://bj.bcebos.com/paddlehub/fastdeploy/animegan_v1_hayao_60_v1.0.0.tgz
+tar xvfz animegan_v1_hayao_60_v1.0.0.tgz
+
+# CPU推理
+./infer_demo --model animegan_v1_hayao_60 --image style_transfer_testimg.jpg  --device cpu
+# GPU推理
+./infer_demo --model animegan_v1_hayao_60 --image style_transfer_testimg.jpg  --device gpu
+```
+
+以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:  
+- [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md)
+
+## AnimeGAN C++接口
+
+### AnimeGAN类
+
+```c++
+fastdeploy::vision::generation::AnimeGAN(
+        const string& model_file,
+        const string& params_file = "",
+        const RuntimeOption& runtime_option = RuntimeOption(),
+        const ModelFormat& model_format = ModelFormat::PADDLE)
+```
+
+AnimeGAN模型加载和初始化，其中model_file为导出的Paddle模型结构文件，params_file为模型参数文件。
+
+**参数**
+
+> * **model_file**(str): 模型文件路径
+> * **params_file**(str): 参数文件路径
+> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
+> * **model_format**(ModelFormat): 模型格式，默认为Paddle格式
+
+#### Predict函数
+
+> ```c++
+> bool AnimeGAN::Predict(cv::Mat& image, cv::Mat* result)
+> ```
+>
+> 模型预测入口，输入图像输出风格迁移后的结果。
+>
+> **参数**
+>
+> > * **image**: 输入数据，注意需为HWC，BGR格式
+> > * **result**: 风格转换后的图像，BGR格式
+
+#### BatchPredict函数
+
+> ```c++
+> bool AnimeGAN::BatchPredict(const std::vector<cv::Mat>& images, std::vector<cv::Mat>* results);
+> ```
+>
+> 模型预测入口，输入一组图像并输出风格迁移后的结果。
+>
+> **参数**
+>
+> > * **images**: 输入数据，一组图像数据，注意需为HWC，BGR格式
+> > * **results**: 风格转换后的一组图像，BGR格式
+
+- [模型介绍](../../)
+- [Python部署](../python)
+- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)
diff --git a/examples/vision/generation/anemigan/cpp/infer.cc b/examples/vision/generation/anemigan/cpp/infer.cc
@@ -0,0 +1,69 @@
+// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "fastdeploy/vision.h"
+#include "gflags/gflags.h"
+
+DEFINE_string(model, "", "Directory of the inference model.");
+DEFINE_string(image, "", "Path of the image file.");
+DEFINE_string(device, "cpu",
+              "Type of inference device, support 'cpu' or 'gpu'.");
+
+void PrintUsage() {
+  std::cout << "Usage: infer_demo --model model_path --image img_path --device [cpu|gpu]"
+            << std::endl;
+  std::cout << "Default value of device: cpu" << std::endl;
+}
+
+bool CreateRuntimeOption(fastdeploy::RuntimeOption* option) {
+  if (FLAGS_device == "gpu") {
+    option->UseGpu();
+    } 
+  else if (FLAGS_device == "cpu") {
+    option->SetPaddleMKLDNN(false);
+    return true;
+  } else {
+    std::cerr << "Only support device CPU/GPU now, "  << FLAGS_device << " is not supported." << std::endl;
+    return false;
+  }
+
+  return true;
+}
+
+int main(int argc, char* argv[]) {
+  google::ParseCommandLineFlags(&argc, &argv, true);
+  auto option = fastdeploy::RuntimeOption();
+  if (!CreateRuntimeOption(&option)) {
+    PrintUsage();
+    return -1;
+  }
+
+  auto model = fastdeploy::vision::generation::AnimeGAN(FLAGS_model+"/model.pdmodel", FLAGS_model+"/model.pdiparams", option);
+  if (!model.Initialized()) {
+    std::cerr << "Failed to initialize." << std::endl;
+    return -1;
+  }
+
+  auto im = cv::imread(FLAGS_image);
+  cv::Mat res;
+  if (!model.Predict(im, &res)) {
+    std::cerr << "Failed to predict." << std::endl;
+    return -1;
+  }
+
+  cv::imwrite("style_transfer_result.png", res);
+  std::cout << "Visualized result saved in ./style_transfer_result.png" << std::endl;
+
+  return 0;
+}
diff --git a/examples/vision/generation/anemigan/python/README.md b/examples/vision/generation/anemigan/python/README.md
@@ -0,0 +1,70 @@
+# AnimeGAN Python部署示例
+
+在部署前，需确认以下两个步骤
+
+- 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)  
+- 2. FastDeploy Python whl包安装，参考[FastDeploy Python安装](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
+
+本目录下提供`infer.py`快速完成AnimeGAN在CPU/GPU，以及GPU上通过TensorRT加速部署的示例。执行如下脚本即可完成
+
+```bash
+# 下载部署示例代码
+git clone https://github.com/PaddlePaddle/FastDeploy.git
+cd FastDeploy/examples/vision/generation/anemigan/python
+# 下载准备好的测试图片
+wget https://bj.bcebos.com/paddlehub/fastdeploy/style_transfer_testimg.jpg
+
+# CPU推理
+python infer.py --model animegan_v1_hayao_60  --image style_transfer_testimg.jpg  --device cpu
+# GPU推理
+python infer.py --model animegan_v1_hayao_60 --image style_transfer_testimg.jpg  --device gpu
+```
+
+## AnimeGAN Python接口
+
+```python
+fd.vision.generation.AnimeGAN(model_file, params_file, runtime_option=None, model_format=ModelFormat.PADDLE)
+```
+
+AnimeGAN模型加载和初始化，其中model_file和params_file为用于Paddle inference的模型结构文件和参数文件。
+
+**参数**
+
+> * **model_file**(str): 模型文件路径
+> * **params_file**(str): 参数文件路径
+> * **runtime_option**(RuntimeOption): 后端推理配置，默认为None，即采用默认配置
+> * **model_format**(ModelFormat): 模型格式，默认为Paddle格式
+
+
+### predict函数
+
+> ```python
+> AnimeGAN.predict(input_image)
+> ```
+>
+> 模型预测入口，输入图像输出风格迁移后的结果。
+>
+> **参数**
+>
+> > * **input_image**(np.ndarray): 输入数据，注意需为HWC，BGR格式
+
+> **返回** np.ndarray, 风格转换后的图像，BGR格式
+
+### batch_predict函数
+> ```python
+> AnimeGAN.batch_predict函数(input_images)
+> ```
+>
+> 模型预测入口，输入一组图像并输出风格迁移后的结果。
+>
+> **参数**
+>
+> > * **input_images**(list(np.ndarray)): 输入数据，一组图像数据，注意需为HWC，BGR格式
+
+> **返回** list(np.ndarray), 风格转换后的一组图像，BGR格式
+
+## 其它文档
+
+- [风格迁移 模型介绍](..)
+- [C++部署](../cpp)
+- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)
diff --git a/examples/vision/generation/anemigan/python/infer.py b/examples/vision/generation/anemigan/python/infer.py
@@ -0,0 +1,43 @@
+import cv2
+import os
+import fastdeploy as fd
+
+
+def parse_arguments():
+    import argparse
+    import ast
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", required=True, help="Name of the model.")
+    parser.add_argument(
+        "--image", type=str, required=True, help="Path of test image file.")
+    parser.add_argument(
+        "--device",
+        type=str,
+        default='cpu',
+        help="Type of inference device, support 'cpu' or 'gpu'.")
+    return parser.parse_args()
+
+
+def build_option(args):
+    option = fd.RuntimeOption()
+    if args.device.lower() == "gpu":
+        option.use_gpu()
+    else:
+        option.set_paddle_mkldnn(False)
+    return option
+
+
+args = parse_arguments()
+
+# 配置runtime，加载模型
+runtime_option = build_option(args)
+fd.download_model(name=args.model, path='./', format='paddle')
+model_file = os.path.join(args.model, "model.pdmodel")
+params_file = os.path.join(args.model, "model.pdiparams")
+model = fd.vision.generation.AnimeGAN(
+    model_file, params_file, runtime_option=runtime_option)
+
+# 预测图片并保存结果
+im = cv2.imread(args.image)
+result = model.predict(im)
+cv2.imwrite('style_transfer_result.png', result)
diff --git a/fastdeploy/vision.h b/fastdeploy/vision.h
@@ -55,6 +55,7 @@
 #include "fastdeploy/vision/segmentation/ppseg/model.h"
 #include "fastdeploy/vision/sr/ppsr/model.h"
 #include "fastdeploy/vision/tracking/pptracking/model.h"
+#include "fastdeploy/vision/generation/contrib/animegan.h"
 
 #endif