Add 'Extensibility For Native Rules' design doc

--
Reviewed-on: https://bazel-review.googlesource.com/#/c/4250/5
MOS_MIGRATED_REVID=129431153

site/designs/_posts/2016-08-04-extensibility-for-native-rules.md

@@ -0,0 +1,154 @@
+---
+layout: contribute
+title: Extensibility For Native Rules
+---
+# Extensibility For Native Rules
+
+**Status**: Reviewed, not yet implemeted
+
+**Author**: [Dmitry Lomov](mailto:[email protected])
+
+## Motivation
+
+There is a number of requests that require Skylark API to access functionality
+of native rules from Skylark rules. Typical scenarios can be illustrated by the
+following "sandwich":
+
+```python
+bread_library(name = "top", …)
+java_library(name = "meat", deps = [":top", …], …)
+bread_library(name = "bottom", deps = [":meat", …])
+```
+
+Here bread\_library is a rule written in Skylark. Here we need three things:
+
+* implementation of bread\_library should be able to produce jar files and other
+  artifacts in the same way as native java\_library rule would; in other words
+  the implementation should be able to delegate to the a native implementation
+* java\_library should allow depending on bread\_library; importantly, that
+  dependence should be *meaningful*, that is if bread\_library produces Java
+  artifacts, such as jar files, java\_library should be able to compile against
+  those like it would against a java\_library dependency
+* bread\_library should be able to depend on java\_library; it should be able to
+  access all information it needs; when delegating to native implementation,
+  there should be a simple way to pass information from a dependency to to the
+  native implementation
+
+In this document we present some ideas about what how this all might look like,
+and suggest some practical steps we can take to get there.
+
+## Extensible Native Rules
+
+This proposal assumes [Declared Providers
+](/designs/declared-providers.html)
+are implemented. Here is how the implementation of bread\_library might look
+like:
+
+```python
+# Implementation of a rule that transpiles to Java and invokes a native
+# compilation
+def _bread_library_impl(ctx):
+    bread_sources = [f for src in ctx.attrs.src for f in src.files]
+    generated_java_files = _invoke_bread_transpiler(ctx, bread_sources)
+    # lang.java.provider is a declared provider for Java
+    java_deps = [target[lang.java.provider] for target in ctx.attrs.deps]
+    # create a native compilation action
+    java_p = lang.java.compile(ctx,
+                srcs = generated_java_files,
+                # information about dependencies is just a lang.java.provider
+                deps = java_deps,
+                ...)
+    # java_p is a lang.java.provider representing the result of compilation
+    # action we return that provider and immediately java_libary rule can depend
+    # on us
+    return [java_p, ...]
+
+# Implementation of a rule that compiles to JVM bytecode directly
+def _scala_library_impl(ctx):
+    # collect dependency jars to pass to the compile action
+    dep_jars = [dep[java.lang.provider].jar for dep in ctx.attrs.deps]
+    jar_output = ctx.new_file(...)
+    ... construct compilation actions ...
+    # build a provider that passes all transitive information
+    transitive_p = lang.java.transitive(
+                      [dep[java.lang.provider] for dep in ctx.attrs.deps])
+    java_p = lang.java.provider(
+                transitive_p,
+                jar = jar_output,
+                # update transitive information that we care about
+                transitive_jars =
+                    transitive_p.transitive_jars | set(jar_output),
+                    ... whatever other information is needed ...)
+    # return java.lang.provider
+    return [java_p, ...]
+```
+
+The provider is the glue ("butter") that connects Skylark rules to native rules
+and also to the native rule implementations exposed to Skylark. Note how the
+native rule implementation (lang.java.compile) both consumes the entire
+providers from dependencies and returns the provider that needs to be returned
+from the rule. `lang.java.transitive` is a function that passes all the
+transitive information correctly from dependencies. The [existing '.java'
+provider](http://www.bazel.io/docs/skylark/lib/JavaSkylarkApiProvider.html)
+becomes the same thing as lang.java.provider.
+
+Note: for the sake for this document we are placing things in lang.java. There
+are other alternatives to this, e.g. "magical" .bzl files from which
+java\_provider and java\_compile function are exported.
+
+## How to get there
+
+Our current native rules are not as neat as described above. Making them
+extensible in one go is a difficult and long term project (or rather, projects:
+one for each language). Here is a suggested steps for extensibility of
+particular language implementation (we continue to use Java as a running
+example).
+
+### Phase 1: Expose native compilation actions
+
+At this step, lang.java.provider is a *black box*. Skylark rules cannot
+construct the lang.java.provider directly: the only way to create it is to
+invoke lang.java.compile function.
+
+Native implementations of Java rules are rewritten so that they can link to deps
+that return lang.java.provider and that they return lang.java.provider.  The
+implementation of the provider can just be a bag of all providers that Java
+rules normally return - since that bag is not openable by Skylark, we can
+refactor it later without much difficulty .
+
+Native compilation function (java.lang.compile) is pretty much
+JavaLibrary.create refactored so that it gets its dependent providers not from
+attributes but from a list of bags. JavaLibrary.create just collects the bags
+from deps and passes those to that function.
+
+At the end of this phase, implementing code generators (and code generating
+aspects) such as java\_proto\_library becomes possible. This also covers many
+(most?) use cases where people use macros to delegate to native rule
+implementations
+
+No huge refactoring of language rule implementation is needed, but the stage is
+set for gradual opening up in the future.
+
+### Phase 1a: Implementing JavaSkylarkApiProvider on top of black box
+
+(Optional) As lang.java.provider is just a bag of existing providers, it is easy
+to just implement everything in 'target.java' on top of it, if desired.
+
+### Phase 2: Evolving the API and opening up
+
+The next step in the API evolution is making the black box provider less black.
+This means introducing a constructor for lang.java.provider as well as accessors
+to fields.
+
+The API can be designed gradually and thoughtfully, only exposing the things we
+need and adding carefully: as an example sequence first just java libraries,
+then resources, then JNI, then support for tests. Existence of
+lang.java.transitive is crucial at this stage as it allows merging of transitive
+information from dependencies that is not yet exposed to Skylark.
+
+As API exposure gradually progresses, the exposed Skylark API reaches parity
+with internal API.
+
+Through the execution of this phase, more and more use cases are covered, and at
+the end the rules are fully extensible.
+