CircleCI automated build checks and CUDA/CPU Dockerfile base

Summary: Add CircleCI integration for the CUDA and CPU backend builds. We need this: some recent w2l PRs have broken the build, and incentives unfortunately aren't in place to import a PR, patch the diff, and run tests ourselves. Until all of our tests run on internal CI resources (work in progress, almost there), this will sanity check that the PR builds and give a signal to open source contributors as to whether or not their changes build. If this works well and contributions increase, we'll add dedicated AWS GPU resources to run flashlight tests direclty through CircleCI; this is a first step. Core changes to Dockerfiles: - Update ArrayFire verison to v3.6.4, add recursive submodule update - Take advantage of Docker caching and create more intermediate steps for recovering more quickly on failed builds - Create `Dockerfile-CUDA-Base` and `Dockerfile-CPU-Base` - This facilitates creating an environment in which we can test flashlight from scratch without pre-building. We want CircleCI to do the build and succeed/fail, so stop building directly in the Dockerfile, else CircleCI will output a generic error with the Dockerfile environment failing to initialize - `Dockerfile-CUDA` now inherits directly from `Dockerfile-CUDA-Base`, same with CPU In order for the CircleCI build to run properly, Docker Hub assets need to be updated, which I'll do concurrently with landing this diff. Will be added to wav2letter in a separate diff pending these working Reviewed By: tlikhomanenko Differential Revision: D15512063 fbshipit-source-id: b54826f97aed7efeed4733c128df76d583fea451
danthe3rd · May 29, 2019 · 4199d96 · 4199d96
1 parent 5b02b78
commit 4199d96
Show file tree

Hide file tree

Showing 5 changed files with 180 additions and 134 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -0,0 +1,29 @@
+ # Copyright (c) Facebook, Inc. and its affiliates.
+ # All rights reserved.
+ #
+ # This source code is licensed under the BSD-style license found in the
+ # LICENSE file in the root directory of this source tree.
+
+version: 2.0
+
+jobs:
+  build:
+    docker:
+      - image: flml/flashlight:cuda-base-latest
+    steps:
+      - checkout
+      - run:
+          command: |
+            mkdir -p build && cd build
+            cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CUDA
+            make -j8 && make install
+  build-cpu:
+    docker:
+      - image: flml/flashlight:cpu-base-latest
+    steps:
+      - checkout
+      - run:
+          command: |
+            export MKLROOT=/opt/intel/mkl && mkdir -p build && cd build
+            cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CPU
+            make -j8 && make install
diff --git a/Dockerfile-CPU b/Dockerfile-CPU
@@ -1,97 +1,19 @@
 # ==================================================================
 # module list
 # ------------------------------------------------------------------
-# Ubuntu           16.04
-# OpenMPI          latest     (apt)
-# cmake            3.10       (git)
-# MKL              2018.4-057 (apt)
-# arrayfire        3.6.2      (git, CPU backend)
-# MKLDNN           4bdffc2    (git)
-# Gloo             b7e0906    (git)
 # flashlight       master     (git, CPU backend)
 # ==================================================================
 
-FROM ubuntu:16.04
+FROM flml/flashlight:cpu-base-latest
 
-RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \
-    apt-get update && \
-    DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
-        build-essential \
-        ca-certificates \
-        wget \
-        git \
-        vim \
-        emacs \
-        nano \
-        htop \
-        g++ \
-        # for MKL
-        apt-transport-https \
-        # for arrayfire CPU backend
-        # OpenBLAS
-        libopenblas-dev libfftw3-dev liblapacke-dev \
-        # ATLAS
-        libatlas3-base libatlas-dev libfftw3-dev liblapacke-dev \
-        # ssh for OpenMPI
-        openssh-server openssh-client \
-        # OpenMPI
-        libopenmpi-dev libomp-dev && \
 # ==================================================================
-# cmake 3.10 (for MKLDNN)
-# ------------------------------------------------------------------
-    apt-get purge -y cmake && \
-    # for cmake
-    DEBIAN_FRONTEND=noninteractive $APT_INSTALL zlib1g-dev libcurl4-openssl-dev && \
-    cd /tmp && wget https://cmake.org/files/v3.10/cmake-3.10.3.tar.gz  && \
-    tar -xzvf cmake-3.10.3.tar.gz  && cd cmake-3.10.3  && \
-    ./bootstrap --system-curl && \
-    make -j8 &&  make install && cmake --version && \
-# ==================================================================
-# MKL https://software.intel.com/en-us/mkl
-# ------------------------------------------------------------------
-    cd /tmp && wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \
-    apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \
-    wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list && \
-    sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' && \
-    apt-get update && DEBIAN_FRONTEND=noninteractive $APT_INSTALL intel-mkl-64bit-2018.4-057 && \
-    export MKLROOT=/opt/intel/mkl && \
-# ==================================================================
-# arrayfire with CPU backend https://github.com/arrayfire/arrayfire/wiki/
-# ------------------------------------------------------------------
-    cd /tmp && git clone --recursive https://github.com/arrayfire/arrayfire.git && \
-    cd arrayfire && git checkout v3.6.2  && \
-    mkdir build && cd build && \
-    CXXFLAGS=-DOS_LNX cmake .. -DCMAKE_BUILD_TYPE=Release -DAF_BUILD_CUDA=OFF -DAF_BUILD_OPENCL=OFF -DAF_BUILD_EXAMPLES=OFF && \
-    make -j8 && make install && \
-# ==================================================================
-# MKLDNN https://github.com/intel/mkl-dnn/
-# ------------------------------------------------------------------
-    cd /tmp && git clone https://github.com/intel/mkl-dnn.git && \
-    cd mkl-dnn && git checkout 4bdffc2 && mkdir -p build && cd build && \
-    cmake .. -DMKLDNN_USE_MKL=FULL && \
-    make -j8 && make install && \
-# ==================================================================
-# Gloo https://github.com/facebookincubator/gloo.git
-# ------------------------------------------------------------------
-    cd /tmp && git clone https://github.com/facebookincubator/gloo.git && \
-    cd gloo && git checkout b7e0906 && mkdir build && cd build && \
-    cmake .. -DUSE_MPI=ON && \
-    make -j8 && make install && \
-# ==================================================================
-# config & cleanup
+# flashlight with CPU backend
 # ------------------------------------------------------------------
-    ldconfig && \
-    apt-get clean && \
-    apt-get autoremove && \
-    rm -rf /var/lib/apt/lists/* /tmp/* ~/*
-
+# Setup and build flashlight
 RUN mkdir /root/flashlight
+
 COPY . /root/flashlight
 
-# ==================================================================
-# flashlight with CPU backend
-# ------------------------------------------------------------------
-RUN export MKLROOT=/opt/intel/mkl && \
-    cd /root/flashlight && mkdir -p build && cd build && \
-    cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CPU && \
+RUN export MKLROOT=/opt/intel/mkl && cd /root/flashlight && mkdir -p build && \
+    cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CPU && \
     make -j8 && make install
diff --git a/Dockerfile-CPU-Base b/Dockerfile-CPU-Base
@@ -0,0 +1,85 @@
+# ==================================================================
+# module list
+# ------------------------------------------------------------------
+# Ubuntu           16.04
+# OpenMPI          latest     (apt)
+# cmake            3.10       (git)
+# MKL              2018.4-057 (apt)
+# arrayfire        3.6.4      (git, CPU backend)
+# MKLDNN           4bdffc2    (git)
+# Gloo             b7e0906    (git)
+# ==================================================================
+
+FROM ubuntu:16.04
+
+RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
+        build-essential \
+        ca-certificates \
+        wget \
+        git \
+        vim \
+        emacs \
+        nano \
+        htop \
+        g++ \
+        # for MKL
+        apt-transport-https \
+        # for arrayfire CPU backend
+        # OpenBLAS
+        libopenblas-dev libfftw3-dev liblapacke-dev \
+        # ATLAS
+        libatlas3-base libatlas-dev libfftw3-dev liblapacke-dev \
+        # ssh for OpenMPI
+        openssh-server openssh-client \
+        # OpenMPI
+        libopenmpi-dev libomp-dev && \
+# ==================================================================
+# cmake 3.10 (for MKLDNN)
+# ------------------------------------------------------------------
+    apt-get purge -y cmake && \
+    # for cmake
+    DEBIAN_FRONTEND=noninteractive $APT_INSTALL zlib1g-dev libcurl4-openssl-dev && \
+    cd /tmp && wget https://cmake.org/files/v3.10/cmake-3.10.3.tar.gz  && \
+    tar -xzvf cmake-3.10.3.tar.gz  && cd cmake-3.10.3  && \
+    ./bootstrap --system-curl && \
+    make -j8 &&  make install && cmake --version && \
+# ==================================================================
+# MKL https://software.intel.com/en-us/mkl
+# ------------------------------------------------------------------
+    cd /tmp && wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \
+    apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \
+    wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list && \
+    sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' && \
+    apt-get update && DEBIAN_FRONTEND=noninteractive $APT_INSTALL intel-mkl-64bit-2018.4-057 && \
+    export MKLROOT=/opt/intel/mkl && \
+# ==================================================================
+# arrayfire with CPU backend https://github.com/arrayfire/arrayfire/wiki/
+# ------------------------------------------------------------------
+    cd /tmp && git clone --recursive https://github.com/arrayfire/arrayfire.git && \
+    cd arrayfire && git checkout v3.6.4  && git submodule update --init --recursive && \
+    mkdir build && cd build && \
+    CXXFLAGS=-DOS_LNX cmake .. -DCMAKE_BUILD_TYPE=Release -DAF_BUILD_CUDA=OFF -DAF_BUILD_OPENCL=OFF -DAF_BUILD_EXAMPLES=OFF && \
+    make -j8 && make install && \
+# ==================================================================
+# MKLDNN https://github.com/intel/mkl-dnn/
+# ------------------------------------------------------------------
+    cd /tmp && git clone https://github.com/intel/mkl-dnn.git && \
+    cd mkl-dnn && git checkout 4bdffc2 && mkdir -p build && cd build && \
+    cmake .. -DMKLDNN_USE_MKL=FULL && \
+    make -j8 && make install && \
+# ==================================================================
+# Gloo https://github.com/facebookincubator/gloo.git
+# ------------------------------------------------------------------
+    cd /tmp && git clone https://github.com/facebookincubator/gloo.git && \
+    cd gloo && git checkout b7e0906 && mkdir build && cd build && \
+    cmake .. -DUSE_MPI=ON && \
+    make -j8 && make install && \
+# ==================================================================
+# config & cleanup
+# ------------------------------------------------------------------
+    ldconfig && \
+    apt-get clean && \
+    apt-get autoremove && \
+    rm -rf /var/lib/apt/lists/* /tmp/* ~/*
diff --git a/Dockerfile-CUDA b/Dockerfile-CUDA
@@ -1,64 +1,19 @@
 # ==================================================================
 # module list
 # ------------------------------------------------------------------
-# Ubuntu           16.04
-# CUDA             9.2
-# CuDNN            7-dev
-# arrayfire        3.6.2    (git, CUDA backend)
-# OpenMPI          latest   (apt)
 # flashlight       master   (git, CUDA backend)
 # ==================================================================
 
-FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04
+FROM flml/flashlight:cuda-base-latest
 
-RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \
-    rm -rf /var/lib/apt/lists/* \
-           /etc/apt/sources.list.d/cuda.list \
-           /etc/apt/sources.list.d/nvidia-ml.list && \
-    apt-get update && \
-    DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
-        build-essential \
-        ca-certificates \
-        cmake \
-        wget \
-        git \
-        vim \
-        emacs \
-        nano \
-        htop \
-        g++ \
-        # ssh for OpenMPI
-        openssh-server openssh-client \
-        # OpenMPI
-        libopenmpi-dev libomp-dev \
-        # nccl: for flashlight
-        libnccl2 libnccl-dev && \
 # ==================================================================
-# arrayfire https://github.com/arrayfire/arrayfire/wiki/
-# ------------------------------------------------------------------
-    cd /tmp && git clone --recursive https://github.com/arrayfire/arrayfire.git && \
-    cd arrayfire && git checkout v3.6.2 && \
-    mkdir build && cd build && \
-    CXXFLAGS=-DOS_LNX cmake .. -DCMAKE_BUILD_TYPE=Release -DAF_BUILD_CPU=OFF -DAF_BUILD_OPENCL=OFF -DAF_BUILD_EXAMPLES=OFF && \
-    make -j8 && \
-    make install && \
-# ==================================================================
-# config & cleanup
+# flashlight with GPU backend
 # ------------------------------------------------------------------
-    ldconfig && \
-    apt-get clean && \
-    apt-get autoremove && \
-    rm -rf /var/lib/apt/lists/* /tmp/* ~/*
-
+# Setup and build flashlight
 RUN mkdir /root/flashlight
+
 COPY . /root/flashlight
 
-# ==================================================================
-# flashlight with GPU backend
-# ------------------------------------------------------------------
-# If the driver is not found (during docker build) the cuda driver api need to be linked against the
-# libcuda.so stub located in the lib[64]/stubs directory
-RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 && \
-    cd /root/flashlight && mkdir -p build && \
+RUN cd /root/flashlight && mkdir -p build && \
     cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CUDA && \
     make -j8 && make install
diff --git a/Dockerfile-CUDA-Base b/Dockerfile-CUDA-Base
@@ -0,0 +1,55 @@
+# ==================================================================
+# module list
+# ------------------------------------------------------------------
+# Ubuntu           16.04
+# CUDA             9.2
+# CuDNN            7-dev
+# arrayfire        3.6.4    (git, CUDA backend)
+# OpenMPI          latest   (apt)
+# flashlight       master   (git, CUDA backend)
+# ==================================================================
+
+FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04
+
+RUN APT_INSTALL="apt-get install -y --no-install-recommends" && \
+    rm -rf /var/lib/apt/lists/* \
+           /etc/apt/sources.list.d/cuda.list \
+           /etc/apt/sources.list.d/nvidia-ml.list && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive $APT_INSTALL \
+        build-essential \
+        ca-certificates \
+        cmake \
+        wget \
+        git \
+        vim \
+        emacs \
+        nano \
+        htop \
+        g++ \
+        # ssh for OpenMPI
+        openssh-server openssh-client \
+        # OpenMPI
+        libopenmpi-dev libomp-dev \
+        # nccl: for flashlight
+        libnccl2 libnccl-dev && \
+# ==================================================================
+# arrayfire https://github.com/arrayfire/arrayfire/wiki/
+# ------------------------------------------------------------------
+    cd /tmp && git clone --recursive https://github.com/arrayfire/arrayfire.git && \
+    cd arrayfire && git checkout v3.6.4 && git submodule update --init --recursive && \
+    mkdir build && cd build && \
+    CXXFLAGS=-DOS_LNX cmake .. -DCMAKE_BUILD_TYPE=Release -DAF_BUILD_CPU=OFF -DAF_BUILD_OPENCL=OFF -DAF_BUILD_EXAMPLES=OFF && \
+    make -j8 && \
+    make install && \
+# ==================================================================
+# config & cleanup
+# ------------------------------------------------------------------
+    ldconfig && \
+    apt-get clean && \
+    apt-get autoremove && \
+    rm -rf /var/lib/apt/lists/* /tmp/* ~/* && \
+
+    # If the driver is not found (during docker build) the cuda driver api need to be linked against the
+    # libcuda.so stub located in the lib[64]/stubs directory
+    ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1