Build Python Version Failed

[pangsg]

Branch/Tag/Commit

main

Docker Image Version

nvcr.io/nvidia/pytorch 22.09-py3

GPU name

A100

CUDA Driver

Driver Version: 510.85.02

Reproduced Steps

docker run -it --gpus all nvcr.io/nvidia/pytorch:22.09-py3 /bin/bash

cd FasterTransformer-main
mkdir build
cmake -DSM=80 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_MULTI_GPU=ON ..
make -j12

Error Info:
n file included from /workspace/FasterTransformer-main/src/fastertransformer/kernels/cutlass_kernels/cutlass_preprocessors.cc:20:
/workspace/FasterTransformer-main/src/fastertransformer/cutlass_extensions/include/cutlass_extensions/gemm/kernel/mixed_gemm_B_layout.h:12:10: fatal error: cutlass/layout/matrix.h: No such file or directory
   12 | #include "cutlass/layout/matrix.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/fastertransformer/kernels/cutlass_kernels/CMakeFiles/cutlass_preprocessors.dir/build.make:76: src/fastertransformer/kernels/cutlass_kernels/CMakeFiles/cutlass_preprocessors.dir/cutlass_preprocessors.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:3536: src/fastertransformer/kernels/cutlass_kernels/CMakeFiles/cutlass_preprocessors.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  1%] Linking CUDA device code CMakeFiles/cuda_driver_wrapper.dir/cmake_device_link.o
[  1%] Linking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o
[  1%] Linking CXX static library ../../lib/libcuda_driver_wrapper.a
[  1%] Built target cuda_driver_wrapper
[  1%] Linking CXX static library ../../../lib/libnvtx_utils.a
[  1%] Built target nvtx_utils
[  1%] Linking CUDA device code CMakeFiles/logger.dir/cmake_device_link.o
[  1%] Linking CXX static library ../../../lib/liblogger.a
[  1%] Built target logger
[  2%] Linking CUDA device code CMakeFiles/cuda_utils.dir/cmake_device_link.o
[  2%] Linking CXX static library ../../../lib/libcuda_utils.a
[  2%] Built target cuda_utils
[  2%] Linking CUDA device code CMakeFiles/custom_ar_kernels.dir/cmake_device_link.o
[  2%] Linking CUDA device code CMakeFiles/transpose_int8_kernels.dir/cmake_device_link.o
[  2%] Linking CUDA static library ../../../lib/libcustom_ar_kernels.a
[  2%] Linking CUDA device code CMakeFiles/bert_preprocess_kernels.dir/cmake_device_link.o
[  2%] Built target custom_ar_kernels
[  2%] Linking CUDA device code CMakeFiles/add_residual_kernels.dir/cmake_device_link.o
[  2%] Linking CUDA static library ../../../lib/libtranspose_int8_kernels.a
[  3%] Linking CUDA static library ../../../lib/libbert_preprocess_kernels.a
[  3%] Built target transpose_int8_kernels
[  3%] Linking CUDA static library ../../../lib/libadd_residual_kernels.a
[  3%] Built target bert_preprocess_kernels
[  3%] Built target add_residual_kernels
[  4%] Linking CUDA device code CMakeFiles/activation_kernels.dir/cmake_device_link.o
[  4%] Linking CUDA static library ../../../lib/libactivation_kernels.a
[  4%] Built target activation_kernels
[  4%] Linking CUDA device code CMakeFiles/unfused_attention_kernels.dir/cmake_device_link.o
[  4%] Linking CUDA static library ../../../lib/libunfused_attention_kernels.a
[  4%] Built target unfused_attention_kernels
[  4%] Linking CUDA device code CMakeFiles/layernorm_kernels.dir/cmake_device_link.o
[  4%] Linking CUDA static library ../../../lib/liblayernorm_kernels.a
[  4%] Built target layernorm_kernels
make: *** [Makefile:136: all] Error 2

[byshiue]

Have you run

git submodule init && git submodule update

?

[pangsg]

Have you run

git submodule init && git submodule update

?

Sry, No. Does it matter?

[pangsg]

I have add git submodule init && git submodule update

another error raised

error: pathspec '3rdparty/cutlass' did not match any file(s) known to git
CMake Error at CMakeLists.txt:67 (message):
  git submodule update --init 3rdparty/cutlass failed with 1, please checkout
  cutlass submodule

[byshiue]

Can you remove the full project and follow https://github.com/NVIDIA/FasterTransformer/blob/main/docs/gpt_guide.md#build-the-fastertransformer to try again?

[pangsg]

Can you remove the full project and follow https://github.com/NVIDIA/FasterTransformer/blob/main/docs/gpt_guide.md#build-the-fastertransformer to try again?

Sure, I'll have a try

[pangsg]

Thanks for your help! I have built it successfully