Document adapting custom Bazel rules for remote execution.

PiperOrigin-RevId: 200060887

site/_layouts/documentation.html

@@ -230,6 +230,7 @@ <h3>Extending Bazel</h3>
                   <li><a href="/versions/{{ site.version }}/skylark/skylint.html">Linter</a></li>
                   <li><a href="/versions/{{ site.version }}/skylark/performance.html">Optimizing Performance</a></li>
                   <li><a href="/versions/{{ site.version }}/skylark/deploying.html">Deploying Rules</a></li>
+                  <li><a href="/versions/{{ site.version }}/remex-rules-guidelines.html">Guidelines for Remote Execution</a></li>
                 </ul>
               </li>
 

site/docs/remex-rules-guidelines.md

@@ -0,0 +1,168 @@
+---
+layout: documentation
+title: Adapting Bazel Rules for Remote Execution
+---
+
+# Adapting Bazel Rules for Remote Execution
+
+Remote execution allows Bazel to execute actions on a separate platform, such as
+a datacenter. A [gRPC protocol](https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteexecution/v1test/remote_execution.proto)
+is currently in development. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm),
+an open-source project that aims to provide a distributed remote execution
+platform. This document is intended for Bazel users writing custom build and
+test rules in Skylark who want to understand the requirements for Bazel rules in
+the context of remote execution.
+
+This document uses the following terminology when referring to different
+environment types or *platforms*:
+
+*   **Host platform** - where Bazel runs.
+*   **Execution platform** - where Bazel actions run.
+*   **Target platform** - where the build outputs (and some actions) run.
+
+## Overview
+
+When configuring a Bazel build for remote execution, you must follow the
+guidelines described in this document to ensure the build executes remotely
+error-free. This is due to the nature of remote execution, namely:
+
+*   **Isolated build actions.** Build tools do not retain state and dependencies
+    cannot leak between them.
+
+*   **Diverse execution environments.** Local build configuration is not always
+    suitable for remote execution environments.
+
+This document describes the issues that can arise when implementing custom Bazel
+build and test rules for remote execution and how to avoid them. It covers the
+following topics:
+
+*  [Invoking build tools through toolchain rules](#invoking-build-tools-through-toolchain-rules)
+*  [Managing dependencies](#managing-dependencies)
+*  [Managing platform-dependent binaries](#managing-platform-dependent-binaries)
+*  [Managing configure-style WORKSPACE rules](#managing-configure-style-workspace-rules)
+
+## Invoking build tools through toolchain rules
+
+A Bazel toolchain rule is a configuration provider that tells a build rule what
+build tools, such as compilers and linkers, to use and how to configure them
+using parameters defined by the rule's creator. A toolchain rule allows build
+and test rules to invoke build tools in a predictable, preconfigured manner
+that's compatible with remote execution. For example, use a toolchain rule
+instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local
+variables that may not be set to equivalent values (or at all) in the remote
+execution environment.
+
+Toolchain rules currently exist for Bazel build and test rules for
+[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch
+ain.bzl),
+[Rust](https://github.com/bazelbuild/rules_rust/blob/master/rust/toolchain.bzl),
+and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst),
+and new toolchain rules are under way for other languages and tools such as
+[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub).
+If a toolchain rule does not exist for the tool your rule uses, consider [creating a custom toolchain rule](https://docs.bazel.build/versions/master/toolchains.html#creating-a-toolchain-rule).
+
+## Managing implicit dependencies
+
+If a build tool can access dependencies across build actions, those actions will
+fail when remotely executed because each remote build action is executed
+separately from others. Some build tools retain state across build actions and
+access dependencies that have not been explicitly included in the tool
+invocation, which will cause remotely executed build actions to fail.
+
+For example, when Bazel instructs a stateful compiler to locally build _foo_,
+the compiler retains references to foo's build outputs. When Bazel then
+instructs the compiler to build _bar_, which depends on _foo_, without
+explicitly stating that dependency in the BUILD file for inclusion in the
+compiler invocation, the action executes successfully as long as the same
+compiler instance executes for both actions (as is typical for local execution).
+However, since in a remote execution scenario each build action executes a
+separate compiler instance, compiler state and _bar_'s implicit dependency on
+_foo_ will be lost and the build will fail.
+
+## Managing platform-dependent binaries
+
+Typically, a binary built on the host platform cannot safely execute on an
+arbitrary remote execution platform due to potentially mismatched dependencies.
+For example, the SingleJar binary supplied with Bazel targets the host platform.
+However, for remote execution, SingleJar must be compiled as part of the process
+of building your code so that it targets the remote execution platform. (See the
+[target selection
+logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f
+0c89aa8d/tools/jdk/BUILD#L115).)
+
+Do not ship binaries of build tools required by your build with your source code
+unless you are sure they will safely run in your execution platform. Instead, do
+one of the following:
+
+*   Ship or externally reference the source code for the tool so that it can be
+    built for the remote execution platform.
+
+*   Pre-install the tool into the remote execution environment (for example, a
+    toolchain container) if it's stable enough and use toolchain rules to run it
+    in your build.
+
+## Managing `configure`-style WORKSPACE rules
+
+Bazel's `WORKSPACE` rules can be used for probing the host platform for tools
+and libraries required by the build, which, for local builds, is also Bazel's
+execution platform. If the build explicitly depends on local build tools and
+artifacts, it will fail during remote execution if the remote execution platform
+is not identical to the host platform.
+
+The following actions performed by `WORKSPACE` rules are not compatible with
+remote execution:
+
+*   **Building binaries.** Executing compilation actions in `WORKSPACE` rules
+    results in binaries that are incompatible with the remote execution platform
+    if different from the host platform.
+
+*   **Installing `pip` packages.** `pip` packages  installed via `WORKSPACE`
+    rules require that their dependencies be pre-installed on the host platform.
+    Such packages, built specifically for the host platform, will be
+    incompatible with the remote execution platform if different from the host
+    platform.
+
+*   **Symlinking to local tools or artifacts.** Symlinks to tools or libraries
+    installed on the host platform created via `WORKSPACE` rules will cause the
+    build to fail on the remote execution platform as Bazel will not be able to
+    locate them. Instead, create symlinks using standard build actions so that
+    the symlinked tools and libraries are accessible from Bazel's `runfiles`
+    tree. Do not use [`repository_ctx.symlink`](https://docs.bazel.build/versions/master/skylark/lib/repository_ctx.html#symlink)
+    to symlink target files outside of the external repo directory.
+
+*   **Mutating the host platform.** Avoid creating files outside of the Bazel
+    `runfiles` tree, creating environment variables, and similar actions, as
+     they may behave unexpectedly on the remote execution platform.
+
+If an external dependency executes specific operations dependent on the host
+platform, we recommend splitting those operations between `WORKSPACE` and build
+rules as follows:
+
+*   **Platform inspection and dependency enumeration.** These operations are
+    safe to execute locally via `WORKSPACE` rules, which can check which
+    libraries are installed, download packages that must be built, and prepare
+    required artifacts for compilation. For remote execution, these rules must
+    also support using pre-checked artifacts to provide the information that
+    would normally be obtained during host platform inspection. Pre-checked
+    artifacts allow Bazel to describe dependencies as if they were local. Use
+    conditional statements or the `--override_repository` flag for this.
+
+*   **Generating or compiling target-specific artifacts and platform mutation**.
+    These operations must be executed via regular build rules. Actions that
+    produce target-specific artifacts for external dependencies must execute
+    during the build.
+
+To more easily generate pre-checked artifacts for remote execution, you can use
+`WORKSPACE` rules to emit generated files. You can run those rules on each new
+execution environment, such as inside each toolchain container, and check the
+outputs of your remote execution build in to your source repo to reference.
+
+For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl)
+and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl),
+the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains/cpus/py).
+For local execution, files produced by checking the host environment are used.
+For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304)
+on an environment variable allows the rule to use files that are checked into
+the repo. The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84\)
+that can run both locally and remotely, and perform the necessary processing
+that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730).

site/docs/remote-caching.md

@@ -374,6 +374,7 @@ separate platform, such as a datacenter.  You can try remote execution with
 [Buildfarm], an open source project that aims to provide a distributed remote
 execution platform.
 
+[Adapting Bazel Rules for Remote Execution](/remex-rules-guidelines.html)
 [WebDAV module]: http://nginx.org/en/docs/http/ngx_http_dav_module.html
 [docker image]: https://hub.docker.com/r/buchgr/bazel-remote-cache/
 [GitHub]: https://github.com/buchgr/bazel-remote/