[build] Incremental build for docker images

New version of incremental build, which will hopefully will satisfy everyone.

Primary motivation for this change is to allow to build images for cluster test and validator during local development in order to try cluster test with those images. We were using build-aws so far, however, it now takes about 8-10 minutes to build, which is longer(!) then cluster test run itself, so some change is required in this area.

This version does not modify docker files in the repo, which I think was main concern previously.

Also, with help of previous refactorings, this change introduces support for incremental compilation for _all_ libra images, that use libra-build

How it works?

Idea is same as previously - it compiles libra code in the build container and copies over artifacts into target image.

It creates temporary Dockerfile from the source one, by sed'ing call to build-common. This obviously limits incremental compilation to only containers that executes build-common as a build step.

Nice thing that this is fairly self-contained change that enables incremental compilation on all libra images, for example incremental compilation of validator can be done like this:

```
./docker/validator/build.sh --incremental
```

Closes: aptos-labs#4763
Approved by: rexhoffman

.gitignore

@@ -43,3 +43,7 @@ language/tools/vm-genesis/genesis/vm_config.toml
 
 # Move Build Output
 move_build_output/
+
+# Docker incremental build temporary files and directories
+target-out-docker
+**/Dockerfile.tmp

docker/build-common.sh

@@ -9,11 +9,12 @@ set -e
 export RUSTFLAGS="-Ctarget-cpu=skylake -Ctarget-feature=+aes,+sse2,+sse4.1,+ssse3"
 cargo build --release -p libra-management -p libra-node -p cli -p config-builder -p libra-key-manager -p safety-rules -p db-bootstrapper -p backup-cli "$@"
 
-rm -r target/release/{build,deps,incremental}
+rm -rf target/release/{build,deps,incremental}
 
-strip /libra/target/release/config-builder
-strip /libra/target/release/cli
-strip /libra/target/release/db-backup
-strip /libra/target/release/db-bootstrapper
-strip /libra/target/release/db-restore
-strip /libra/target/release/libra-management
+STRIP_DIR=${STRIP_DIR:-/libra/target}
+strip "$STRIP_DIR/release/config-builder"
+strip "$STRIP_DIR/release/cli"
+strip "$STRIP_DIR/release/db-backup"
+strip "$STRIP_DIR/release/db-bootstrapper"
+strip "$STRIP_DIR/release/db-restore"
+strip "$STRIP_DIR/release/libra-management"

docker/libra-build.sh

@@ -6,6 +6,10 @@ set -e
 DOCKERFILE=$1
 TAG=$2
 
+RESTORE='\001\033[0m\002'
+BLUE='\001\033[01;34m\002'
+DOCKERFILE_BUILD_RE='^RUN ./docker/build-common.sh$'
+
 if [ -z "$DOCKERFILE" ] || [ -z "$TAG" ]
 then
   echo "Usage libra-build.sh <Docker_file> <Local_tag> <Other_args>"
@@ -20,6 +24,49 @@ if [ "$https_proxy" ]; then
     PROXY=" --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy"
 fi
 
+if [ "$1" = "--incremental" ]; then
+  shift
+  if ! grep -E "$DOCKERFILE_BUILD_RE" "$DOCKERFILE" > /dev/null; then
+    echo "It looks like $DOCKERFILE does not support incremental compilation"
+    exit 1
+  fi
+  echo "Using incremental compilation"
+  if ! docker ps > /dev/null; then
+    echo "Docker seem not to be working, try installing docker?"
+    exit 1
+  fi
+  TOOLCHAIN=$(cat "$DIR/../rust-toolchain")
+  CONTAINER=builder-rust-${TOOLCHAIN}
+  OUT_TARGET="$DIR/../target-out-docker"
+  if ! (docker ps | grep "$CONTAINER" > /dev/null); then
+    if (docker ps -a | grep "$CONTAINER" > /dev/null); then
+      echo "${BLUE}Build container exists, but is not running, starting it${RESTORE}"
+      docker start "$CONTAINER" || (printf "Failed to start build container, try\n\tdocker rm -f %" "$CONTAINER"; exit 1)
+    else
+      echo "${BLUE}Build container does not exist, setting up new one${RESTORE}"
+      IMAGE=circleci/rust:${TOOLCHAIN}-buster
+      mkdir -p "$OUT_TARGET"
+      docker run -u root -d -t -v "$OUT_TARGET:/out-target" -v "$DIR/..:/libra" -w /libra --name "$CONTAINER" "$IMAGE" sh > /dev/null
+      docker exec -i -t "$CONTAINER" apt-get update
+      docker exec -i -t "$CONTAINER" apt-get install -y cmake curl clang git
+      echo "${BLUE}Container is set up, starting build${RESTORE}"
+    fi
+  else
+    echo "${BLUE}Running build in existing container.${RESTORE}"
+    echo "${BLUE}Tip:${RESTORE}: If you see unexpected failure, try docker rm -f $CONTAINER"
+  fi
+  CONTAINER_TARGET=/target
+  docker exec -e "STRIP_DIR=$CONTAINER_TARGET" -i -t "$CONTAINER" ./docker/build-common.sh --target-dir "$CONTAINER_TARGET"
+  echo "${BLUE}Build is done, copying over artifacts${RESTORE}"
+  docker exec -i -t "$CONTAINER" sh -c 'find /target/release -maxdepth 1 -executable -type f | xargs -I F cp F /out-target'
+  TMP_DOCKERFILE="$DOCKERFILE.tmp"
+  trap 'rm -f $TMP_DOCKERFILE' EXIT
+  sed -e "s+$DOCKERFILE_BUILD_RE+RUN mkdir -p /libra/target/release/; cp target-out-docker/* /libra/target/release/+g" \
+      "$DOCKERFILE" > "$TMP_DOCKERFILE"
+  DOCKERFILE=$TMP_DOCKERFILE
+  echo "${BLUE}Starting docker build process${RESTORE}"
+fi
+
 for _ in seq 1 2; do
   if docker build -f $DOCKERFILE $DIR/.. --tag $TAG \
     --build-arg GIT_REV="$(git rev-parse HEAD)" \