forked from PaddlePaddle/FastDeploy
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'PaddlePaddle:develop' into gbd_android
- Loading branch information
Showing
15 changed files
with
759 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # 第三方库依赖 | ||
|
|
||
| FastDeploy当前根据编译选项，会依赖如下第三方依赖 | ||
|
|
||
| - OpenCV: 当ENABLE_VISION=ON时，会自动下载预编译OpenCV 3.4.16库 | ||
| - ONNX Runimte: 当ENABLE_ORT_BACKEND=ON时，会自动下载ONNX Runtime库 | ||
| - OpenVINO: 当ENABLE_OPENVINO_BACKEND=ON时，会自动下载OpenVINO库 | ||
|
|
||
| 用户在实际编译时，可能会根据自身需求集成环境中已有的第三方库，可通出如下开关来配置 | ||
|
|
||
|
|
||
| - OPENCV_DIRECTORY: 指定环境中的OpenCV路径，如 `-DOPENCV_DIRECTORY=/usr/lib/aarch64-linux-gnu/cmake/opencv4/` | ||
| - ORT_DIRECTORY: 指定环境中的ONNX Runtime路径， 如`-DORT_DIRECTORY=/download/onnxruntime-linux-x64-1.0.0` | ||
| - OPENVINO_DIRECTORY: 指定环境中的OpenVINO路径， 如`-DOPENVINO_DIRECTORY=//download/openvino` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # ERNIE 3.0 模型部署 | ||
|
|
||
| ## 模型详细说明 | ||
| - [PaddleNLP ERNIE 3.0模型说明](https://github.com/PaddlePaddle/PaddleNLP/tree/release/2.4/model_zoo/ernie-3.0) | ||
|
|
||
| ## 支持的模型列表 | ||
|
|
||
| | 模型 | 结构 | 语言 | | ||
| | :---: | :--------: | :--------: | | ||
| | `ERNIE 3.0-Base`| 12-layers, 768-hidden, 12-heads | 中文 | | ||
| | `ERNIE 3.0-Medium`| 6-layers, 768-hidden, 12-heads | 中文 | | ||
| | `ERNIE 3.0-Mini`| 6-layers, 384-hidden, 12-heads | 中文 | | ||
| | `ERNIE 3.0-Micro`| 4-layers, 384-hidden, 12-heads | 中文 | | ||
| | `ERNIE 3.0-Nano `| 4-layers, 312-hidden, 12-heads | 中文 | | ||
|
|
||
| ## 支持的NLP任务列表 | ||
|
|
||
| | 任务 Task | 是否支持 | | ||
| | :--------------- | ------- | | ||
| | 文本分类 | ✅ | | ||
| | 序列标注 | ❌ | | ||
|
|
||
| ## 导出部署模型 | ||
|
|
||
| 在部署前，需要先将训练好的ERNIE模型导出成部署模型，导出步骤可参考文档[导出模型](https://github.com/PaddlePaddle/PaddleNLP/tree/release/2.4/model_zoo/ernie-3.0). | ||
|
|
||
| ## 下载微调模型 | ||
|
|
||
| ### 分类任务 | ||
|
|
||
| 为了方便开发者的测试，下面提供了在文本分类[AFQMC数据集](https://bj.bcebos.com/paddlenlp/datasets/afqmc_public.zip)上微调的ERNIE 3.0-Medium模型，开发者可直接下载体验。 | ||
|
|
||
| - [ERNIE 3.0 Medium AFQMC](https://bj.bcebos.com/fastdeploy/models/ernie-3.0/ernie-3.0-medium-zh-afqmc.tgz) | ||
|
|
||
| ## 详细部署文档 | ||
|
|
||
| - [Python部署](python) | ||
| - [C++部署](cpp) | ||
| - [Serving部署](serving) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| PROJECT(infer_demo C CXX) | ||
| CMAKE_MINIMUM_REQUIRED (VERSION 3.10) | ||
|
|
||
| option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.") | ||
| set(THIRD_LIBS "") | ||
| include(gflags.cmake) | ||
| include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake) | ||
|
|
||
| include_directories(${FASTDEPLOY_INCS}) | ||
|
|
||
| add_executable(seq_cls_infer_demo ${PROJECT_SOURCE_DIR}/seq_cls_infer.cc) | ||
| target_link_libraries(seq_cls_infer_demo ${FASTDEPLOY_LIBS} ${THIRD_LIBS}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # ERNIE 3.0 模型C++部署示例 | ||
|
|
||
| 在部署前，需确认以下两个步骤 | ||
|
|
||
| - 1. 软硬件环境满足要求，参考[FastDeploy环境要求](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md) | ||
| - 2. 根据开发环境，下载预编译部署库和samples代码，参考[FastDeploy预编译库](../../../../docs/cn/build_and_install/download_prebuilt_libraries.md) | ||
|
|
||
| 本目录下提供`seq_cls_infer.cc`快速完成在CPU/GPU的文本分类任务的C++部署示例。 | ||
|
|
||
|
|
||
| ## 文本分类任务 | ||
|
|
||
| ### 快速开始 | ||
|
|
||
| 以下示例展示如何基于FastDeploy库完成ERNIE 3.0 Medium模型在CLUE Benchmark的[AFQMC数据集](https://bj.bcebos.com/paddlenlp/datasets/afqmc_public.zip)上进行文本分类任务的C++预测部署。 | ||
|
|
||
| ```bash | ||
| # 下载SDK，编译模型examples代码（SDK中包含了examples代码） | ||
| wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-gpu-0.4.0.tgz | ||
| tar xvf fastdeploy-linux-x64-gpu-0.4.0.tgz | ||
|
|
||
| cd fastdeploy-linux-x64-gpu-0.4.0/examples/text/ernie-3.0/cpp | ||
| mkdir build | ||
| cd build | ||
| # 执行cmake，需要指定FASTDEPLOY_INSTALL_DIR为FastDeploy SDK的目录。 | ||
| cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/../../../../../../fastdeploy-linux-x64-gpu-0.4.0 | ||
| make -j | ||
|
|
||
| # 下载AFQMC数据集的微调后的ERNIE 3.0模型以及词表 | ||
| wget https://bj.bcebos.com/fastdeploy/models/ernie-3.0/ernie-3.0-medium-zh-afqmc.tgz | ||
| tar xvfz ernie-3.0-medium-zh-afqmc.tgz | ||
|
|
||
| # CPU 推理 | ||
| ./seq_cls_infer_demo --device cpu --model_dir ernie-3.0-medium-zh-afqmc | ||
|
|
||
| # GPU 推理 | ||
| ./seq_cls_infer_demo --device gpu --model_dir ernie-3.0-medium-zh-afqmc | ||
|
|
||
| ``` | ||
|
|
||
| 运行完成后返回的结果如下： | ||
| ```bash | ||
| [INFO] /paddle/FastDeploy/examples/text/ernie-3.0/cpp/seq_cls_infer.cc(93)::CreateRuntimeOption model_path = ernie-3.0-medium-zh-afqmc/infer.pdmodel, param_path = ernie-3.0-medium-zh-afqmc/infer.pdiparams | ||
| [INFO] fastdeploy/runtime.cc(469)::Init Runtime initialized with Backend::ORT in Device::CPU. | ||
| Batch id: 0, example id: 0, sentence 1: 花呗收款额度限制, sentence 2: 收钱码，对花呗支付的金额有限制吗, label: 1, confidence: 0.581852 | ||
| Batch id: 1, example id: 0, sentence 1: 花呗支持高铁票支付吗, sentence 2: 为什么友付宝不支持花呗付款, label: 0, confidence: 0.997921 | ||
| ``` | ||
|
|
||
|
|
||
|
|
||
| ### 参数说明 | ||
|
|
||
| `seq_cls_infer_demo` 除了以上示例的命令行参数，还支持更多命令行参数的设置。以下为各命令行参数的说明。 | ||
|
|
||
| | 参数 |参数说明 | | ||
| |----------|--------------| | ||
| |--model_dir | 指定部署模型的目录， | | ||
| |--batch_size |最大可测的 batch size，默认为 1| | ||
| |--max_length |最大序列长度，默认为 128| | ||
| |--device | 运行的设备，可选范围: ['cpu', 'gpu']，默认为'cpu' | | ||
| |--backend | 支持的推理后端，可选范围: ['onnx_runtime', 'paddle', 'openvino', 'tensorrt', 'paddle_tensorrt']，默认为'onnx_runtime' | | ||
| |--use_fp16 | 是否使用FP16模式进行推理。使用tensorrt和paddle_tensorrt后端时可开启，默认为False | | ||
|
|
||
| ## 相关文档 | ||
|
|
||
| [ERNIE 3.0模型详细介绍](https://github.com/PaddlePaddle/PaddleNLP/tree/release/2.4/model_zoo/ernie-3.0) | ||
|
|
||
| [ERNIE 3.0模型导出方法](https://github.com/PaddlePaddle/PaddleNLP/tree/release/2.4/model_zoo/ernie-3.0) | ||
|
|
||
| [ERNIE 3.0模型Python部署方法](../python/README.md) |
Oops, something went wrong.