Made shading rules accessible and configurable for jvm_binary.

jvm_binary() targets may now specify a shading_rules argument,
which accepts a list of shading rules, which can be any of:

    shading_relocate()
    shading_exclude()
    shading_relocate_package()
    shading_exclude_package()

The order of rules in the list matters, as typical of shading
logic in general.

These rules are powerful enough to take advantage of jarjar's more
advanced syntax, like using wildcards in the middle of package
names. E.g., this syntax will now work:

    # Destination pattern will be inferred to be
    # `[email protected].@2`
    shading_relocate('com.*.foo.bar.**')

Which can also be done by:

   shading_relocate_package('com.*.foo.bar')

I also added the ability to change the default
`__shaded_by_pants__` prefix.

    # `__my_prefix__.com.foo.bar.@1`
    shading_relocate_package('com.foo.bar', shade_prefix='__my_prefix__.')

The rules are implemented by `Shading.Relocate`, `Shading.Exclude`,
`Shading.RelocatePackage`, and `Shading.ExcludePackage`.

`Relocate` is the most generic, and acts as the base-class. It is
essentially a factory for the Rule nametuple that previously
existed in `Shader`.

Rather than build off of the pre-existing `shade_class`,
`shade_package`, `exclude_class`, and `exclude_package`, I made
`Relocate` and friends more powerful, extensible and (I hope)
intuitive concepts, and refactored the existing functions to use
the new classes instead.

They are wrapped in a Shading object rather than in the previous
Shader object to keep the objects that make it into BUILD file
aliases separate from those that do not. I also wistfully imagine
a day when we can register BUILD file aliases recursively,
which would let us use syntax like `Shading.Relocate` instead of
shading_relocate, which would both be more consistent with things
like `Duplicate` and `Skip` today, without polluting the global
namespace. Today this is not possible, because while you *can*
register `Shading`, and then access `Shading.*` in BUILD files,
they don't make it into the BUILD dictionary when we generate docs.

Testing Done:
Added tests to tests/python/pants_test/java/jar:shader and tests/python/pants_test/java/jar:shader-integration
Test project under testprojects/src/java/org/pantsbuild/testproject/shading and testprojects/src/java/org/pantsbuild/testproject/shadingdep.

CI went green: https://travis-ci.org/gmalmquist/pants/builds/78593423
CI went green: https://travis-ci.org/gmalmquist/pants/builds/78809269
CI went green: https://travis-ci.org/gmalmquist/pants/builds/79349294
CI went green: https://travis-ci.org/gmalmquist/pants/builds/79354035
CI went green: https://travis-ci.org/gmalmquist/pants/builds/79369976

Bugs closed: 2121

Reviewed at https://rbcommons.com/s/twitter/r/2754/

3rdparty/BUILD

@@ -122,6 +122,11 @@ jar_library(name='easymock',
               jar('org.easymock', 'easymock', '3.3.1')
             ])
 
+jar_library(name='gson',
+	    jars = [
+	      jar(org='com.google.code.gson', name='gson', rev='2.3.1')
+	    ])
+
 jar_library(name='guava-testlib',
             jars=[
               jar('com.google.guava', 'guava-testlib', '18.0')

examples/src/java/org/pantsbuild/example/readme.md

@@ -416,6 +416,70 @@ After building our `hello` example, if we check the binary jar's contents, there
     org/pantsbuild/example/hello/world.txt
     $
 
+Shading
+-------
+
+Sometimes you have dependencies that have conflicting package or class names. This typically occurs
+in the following scenario: Your jvm_binary depends on a 3rdparty library A (rev 1.0), and a 3rdparty
+library B (rev 1.3). It turns out that A happens to also depend on B, but it depends on B (rev 2.0),
+which is backwards-incompatible with rev 1.3. Now B (1.3) and B (2.0) define different versions of
+the same classes, with the same fully-qualified class names, and you're pulling them all onto the
+classpath for your project.
+
+This is where shading comes in: you can rename the fully-qualified names of the classes that
+conflict, typically by applying a prefix (eg, `__shaded_by_pants__.org.foobar.example`).
+
+Pants uses jarjar for shading, and allows shading rules to be specified on `jvm_binary` targets with
+the `shading_rules` argument. The `shading_rules` argument is a list of rules. Available rules
+include: <a pantsref='bdict_shading_relocate'>`shading_relocate`</a>,
+<a pantsref='bdict_shading_exclude'>`shading_exclude`</a>,
+<a pantsref='bdict_shading_relocate_package'>`shading_relocate_package`</a>, and
+<a pantsref='bdict_shading_exclude_package'>`shading_exclude_package`</a>.
+
+The order of rules in the list matters, as typical of shading
+logic in general.
+
+These rules are powerful enough to take advantage of jarjar's more
+advanced syntax, like using wildcards in the middle of package
+names. E.g., this syntax works:
+
+    :::python
+    # Destination pattern will be inferred to be
+    # [email protected].@2
+    shading_relocate('com.*.foo.bar.**')
+
+Which can also be done by:
+
+   :::python
+   shading_relocate_package('com.*.foo.bar')
+
+The default shading prefix is `__shaded_by_pants__`, but you can change it:
+
+    :::python
+    shading_relocate_package('com.foo.bar', shade_prefix='__my_prefix__.')
+
+You can rename a specific class:
+
+    :::python
+    shading_relocate('com.example.foo.Main', 'org.example.bar.NotMain')
+
+If you want to shade everything in a package except a particular file (or subpackage), you can use
+the <a pantsref='bdict_shading_exclude'>`shading_exclude`</a> rule.
+
+    :::python
+    shading_exclude('com.example.foobar.Main') # Omit the Main class.
+    shading_exclude_package('com.example.foobar.api') # Omit the api subpackage.
+    shading_relocate_package('com.example.foobar')
+
+Again, order matters here: excludes have to appear __first__.
+
+To see an example, take a look at `testprojects/src/java/org/pantsbuild/testproject/shading/BUILD`,
+and try running
+
+    :::bash
+    ./pants binary testprojects/src/java/org/pantsbuild/testproject/shading
+    jar -tf dist/shading.jar
+
 Further Reading
 ---------------
 

src/python/pants/backend/jvm/BUILD

@@ -30,6 +30,7 @@ python_library(
     'src/python/pants/base:build_file_aliases',
     'src/python/pants/goal',
     'src/python/pants/goal:task_registrar',
+    'src/python/pants/java/jar:shader',
   ],
 )
 
@@ -64,4 +65,4 @@ python_library(
     ':artifact',
     'src/python/pants/base:validation',
   ],
-)
+)

src/python/pants/backend/jvm/register.py

@@ -52,6 +52,7 @@
 from pants.base.build_file_aliases import BuildFileAliases
 from pants.goal.goal import Goal
 from pants.goal.task_registrar import TaskRegistrar as task
+from pants.java.jar.shader import Shading
 
 
 def build_file_aliases():
@@ -86,6 +87,10 @@ def build_file_aliases():
       'jar_rules': JarRules,
       'Repository': Repository,
       'Skip': Skip,
+      'shading_relocate': Shading.Relocate.new,
+      'shading_exclude': Shading.Exclude.new,
+      'shading_relocate_package': Shading.RelocatePackage.new,
+      'shading_exclude_package': Shading.ExcludePackage.new,
     },
     context_aware_object_factories={
       'bundle': Bundle.factory,

src/python/pants/backend/jvm/targets/jvm_binary.py

@@ -291,6 +291,7 @@ def __init__(self,
                deploy_excludes=None,
                deploy_jar_rules=None,
                manifest_entries=None,
+               shading_rules=None,
                **kwargs):
     """
     :param string main: The name of the ``main`` class, e.g.,
@@ -300,9 +301,6 @@ def __init__(self,
       ``'hello'``. (By default, uses ``name`` param)
     :param string source: Name of one ``.java`` or ``.scala`` file (a good
       place for a ``main``).
-    :param sources: Overridden by source. If you want more than one source
-      file, use a library and have the jvm_binary depend on that library.
-    :param resources: List of ``resource``\s to include in bundle.
     :param dependencies: Targets (probably ``java_library`` and
      ``scala_library`` targets) to "link" in.
     :type dependencies: list of target specs
@@ -316,9 +314,10 @@ def __init__(self,
       deploy jar.
     :param manifest_entries: dict that specifies entries for `ManifestEntries <#manifest_entries>`_
       for adding to MANIFEST.MF when packaging this binary.
-    :param configurations: Ivy configurations to resolve for this target.
-      This parameter is not intended for general use.
-    :type configurations: tuple of strings
+    :param list shading_rules: Optional list of shading rules to apply when building a shaded
+      (aka monolithic aka fat) binary jar. The order of the rules matters: the first rule which
+      matches a fully-qualified class name is used to shade it. See shading_relocate(),
+      shading_exclude(), shading_relocate_package(), and shading_exclude_package().
     """
     self.address = address  # Set in case a TargetDefinitionException is thrown early
     if main and not isinstance(main, string_types):
@@ -349,7 +348,8 @@ def __init__(self,
       'deploy_jar_rules': FingerprintedField(deploy_jar_rules or JarRules.default()),
       'manifest_entries': FingerprintedField(ManifestEntries(manifest_entries)),
       'main': PrimitiveField(main),
-      })
+      'shading_rules': PrimitiveField(shading_rules or ()),
+    })
 
     super(JvmBinary, self).__init__(name=name,
                                     address=address,
@@ -369,6 +369,10 @@ def deploy_excludes(self):
   def deploy_jar_rules(self):
     return self.payload.deploy_jar_rules
 
+  @property
+  def shading_rules(self):
+    return self.payload.shading_rules
+
   @property
   def main(self):
     return self.payload.main

src/python/pants/backend/jvm/tasks/BUILD

@@ -70,12 +70,12 @@ python_library(
     'src/python/pants/base:exceptions',
     'src/python/pants/base:workunit',
     'src/python/pants/ivy',
-    'src/python/pants/java/distribution',
     'src/python/pants/java:executor',
     'src/python/pants/java:util',
     'src/python/pants/java/jar:shader',
     'src/python/pants/backend/jvm/subsystems:jvm_tool_mixin',
     'src/python/pants/util:dirutil',
+    'src/python/pants/util:memo',
   ],
 )
 
@@ -290,6 +290,11 @@ python_library(
     ':jar_task',
     '3rdparty/python/twitter/commons:twitter.common.collections',
     'src/python/pants/backend/jvm/targets:jvm',
+    'src/python/pants/base:exceptions',
+    'src/python/pants/java/jar:shader',
+    'src/python/pants/util:contextutil',
+    'src/python/pants/util:fileutil',
+    'src/python/pants/util:memo',
   ],
 )
 

src/python/pants/backend/jvm/tasks/bootstrap_jvm_tools.py

@@ -20,10 +20,10 @@
 from pants.base.exceptions import TaskError
 from pants.ivy.ivy_subsystem import IvySubsystem
 from pants.java import util
-from pants.java.distribution.distribution import DistributionLocator
-from pants.java.executor import Executor, SubprocessExecutor
+from pants.java.executor import Executor
 from pants.java.jar.shader import Shader
 from pants.util.dirutil import safe_mkdir_for
+from pants.util.memo import memoized_property
 
 
 class ShadedToolFingerprintStrategy(IvyResolveFingerprintStrategy):
@@ -75,11 +75,10 @@ def register_options(cls, register):
     super(BootstrapJvmTools, cls).register_options(register)
     register('--jvm-options', action='append', metavar='<option>...',
              help='Run the tool shader with these extra jvm options.')
-    cls.register_jvm_tool(register, 'jarjar')
 
   @classmethod
   def subsystem_dependencies(cls):
-    return super(BootstrapJvmTools, cls).subsystem_dependencies() + (DistributionLocator,)
+    return super(BootstrapJvmTools, cls).subsystem_dependencies() + (Shader.Factory,)
 
   @classmethod
   def global_subsystems(cls):
@@ -142,12 +141,9 @@ def _bootstrap_tool_classpath(self, key, scope, tools):
     targets = list(self._resolve_tool_targets(tools, key, scope))
     return self._bootstrap_classpath(key, targets)
 
-  @property
+  @memoized_property
   def shader(self):
-    if self._shader is None:
-      jarjar = self.tool_jar('jarjar')
-      self._shader = Shader(jarjar, SubprocessExecutor(DistributionLocator.cached()))
-    return self._shader
+    return Shader.Factory.create(self.context)
 
   def _bootstrap_shaded_jvm_tool(self, key, scope, tools, main, custom_rules=None):
     targets = list(self._resolve_tool_targets(tools, key, scope))

src/python/pants/backend/jvm/tasks/bundle_create.py

@@ -66,9 +66,9 @@ def execute(self):
     for target in self.context.target_roots:
       for app in map(self.App, filter(self.App.is_app, [target])):
         basedir = self.bundle(app)
-        # NB(Eric Ayers): Note that this product is not housed/controlled under .pants.d/  Since 
+        # NB(Eric Ayers): Note that this product is not housed/controlled under .pants.d/  Since
         # the bundle is re-created every time, this shouldn't cause a problem, but if we ever
-        # expect the product to be cached, a user running an 'rm' on the dist/ directory could 
+        # expect the product to be cached, a user running an 'rm' on the dist/ directory could
         # cause inconsistencies.
         jvm_bundles_product = self.context.products.get('jvm_bundles')
         jvm_bundles_product.add(target, os.path.dirname(basedir)).append(os.path.basename(basedir))
@@ -123,7 +123,11 @@ def add_jars(target):
       # Add external dependencies to the bundle.
       for basedir, external_jar in self.list_external_jar_dependencies(app.binary):
         path = os.path.join(basedir, external_jar)
-        verbose_symlink(path, os.path.join(lib_dir, external_jar))
+        destination = os.path.join(lib_dir, external_jar)
+        verbose_symlink(path, destination)
+        if app.binary.shading_rules:
+          self.shade_jar(binary=app.binary, jar_id=os.path.basename(external_jar),
+                         jar_path=destination)
         classpath.add(external_jar)
 
     bundle_jar = os.path.join(bundle_dir, '{}.jar'.format(app.binary.basename))

src/python/pants/backend/jvm/tasks/jvm_binary_task.py

@@ -13,6 +13,12 @@
 
 from pants.backend.jvm.targets.jvm_binary import JvmBinary
 from pants.backend.jvm.tasks.jar_task import JarTask
+from pants.base.exceptions import TaskError
+from pants.java.jar.shader import Shader
+from pants.java.util import execute_runner
+from pants.util.contextutil import temporary_dir
+from pants.util.fileutil import atomic_copy
+from pants.util.memo import memoized_property
 
 
 class JvmBinaryTask(JarTask):
@@ -37,6 +43,10 @@ def prepare(cls, options, round_manager):
     round_manager.require('jar_dependencies', predicate=cls.is_binary)
     cls.JarBuilder.prepare(round_manager)
 
+  @classmethod
+  def subsystem_dependencies(cls):
+    return super(JvmBinaryTask, cls).subsystem_dependencies() + (Shader.Factory,)
+
   def list_external_jar_dependencies(self, binary, confs=None):
     """Returns the external jar dependencies of the given binary.
 
@@ -61,25 +71,60 @@ def monolithic_jar(self, binary, path, with_external_deps):
     """
     # TODO(benjy): There's actually nothing here that requires 'binary' to be a jvm_binary.
     # It could be any target. And that might actually be useful.
-
     with self.context.new_workunit(name='create-monolithic-jar'):
       with self.open_jar(path,
                          jar_rules=binary.deploy_jar_rules,
                          overwrite=True,
-                         compressed=True) as jar:
+                         compressed=True) as monolithic_jar:
 
         with self.context.new_workunit(name='add-internal-classes'):
-          with self.create_jar_builder(jar) as jar_builder:
+          with self.create_jar_builder(monolithic_jar) as jar_builder:
             jar_builder.add_target(binary, recursive=True)
 
         if with_external_deps:
+          # NB(gmalmquist): Shading each jar dependency with its own prefix would be a nice feature,
+          # but is not currently possible with how things are set up. It may not be possible to do
+          # in general, at least efficiently.
           with self.context.new_workunit(name='add-dependency-jars'):
             for basedir, external_jar in self.list_external_jar_dependencies(binary):
-              external_jar_path = os.path.join(basedir, external_jar)
-              self.context.log.debug('  dumping {}'.format(external_jar_path))
-              jar.writejar(external_jar_path)
+              jar_path = os.path.join(basedir, external_jar)
+              self.context.log.debug('  dumping {}'.format(jar_path))
+              monolithic_jar.writejar(jar_path)
+
+        yield monolithic_jar
+
+      if binary.shading_rules:
+        with self.context.new_workunit('shade-monolithic-jar'):
+          self.shade_jar(binary=binary, jar_id=binary.address.reference(), jar_path=path)
 
-        yield jar
+  @memoized_property
+  def shader(self):
+    return Shader.Factory.create(self.context)
+
+  def shade_jar(self, binary, jar_id, jar_path):
+    """Shades a jar using the shading rules from the given jvm_binary.
+
+    This *overwrites* the existing jar file at ``jar_path``.
+
+    :param binary: The jvm_binary target the jar is being shaded for.
+    :param jar_id: The id of the jar being shaded (used for logging).
+    :param jar_path: The filepath to the jar that should be shaded.
+    """
+    self.context.log.debug('Shading {} at {}.'.format(jar_id, jar_path))
+    with temporary_dir() as tempdir:
+      output_jar = os.path.join(tempdir, os.path.basename(jar_path))
+      rules = [rule.rule() for rule in binary.shading_rules]
+      with self.shader.binary_shader_for_rules(output_jar, jar_path, rules) as shade_runner:
+        result = execute_runner(shade_runner, workunit_factory=self.context.new_workunit,
+                                workunit_name='jarjar')
+        if result != 0:
+          raise TaskError('Shading tool failed to shade {0} (error code {1})'.format(jar_path,
+                                                                                     result))
+        if not os.path.exists(output_jar):
+          raise TaskError('Shading tool returned success for {0}, but '
+                          'the output jar was not found at {1}'.format(jar_path, output_jar))
+        atomic_copy(output_jar, jar_path)
+        return jar_path
 
   def _mapped_dependencies(self, jardepmap, binary, confs):
     # TODO(John Sirois): rework product mapping towards well known types

src/python/pants/java/jar/BUILD

@@ -13,6 +13,8 @@ python_library(
   name='shader',
   sources=['shader.py'],
   dependencies=[
-    'src/python/pants/util:contextutil'
+    'src/python/pants/util:contextutil',
+    'src/python/pants/java/distribution',
+    'src/python/pants/subsystem',
   ]
 )