Integrate the Android SDK, android-library

This is the 2nd part of splitting up pantsbuild#2040. This follows up
on pantsbuild#2467 (adding unpack_libraries task and android_library).
This adds the google and android m2 repos from the SDK into
ivysettings.xml.

AndroidLibraries are unpacked once per artifact in UnpackLibraries.
The unpack .class files are packed into the android_binary's
classes.dex file. The android_library BUILD files allow for
include/exclude patterns, those patterns are filtered against
during the repack in DxCompile. This allows us several advantages:

  * each artifact is unpacked only once, no matter how many
    dependees or include patterns
  * therefore each class file has a static path, so duplicate
    class files (allowed in the build but illegal to pass to
    dx tool) are easily detected
  * Version conflicts are detected by class name(which is how it
    is done within the dx tool) and can raise a useful exception.

Processing the resources in AaptGen is moved off of the
codegen backend. The gentarget for AaptGen is the
AndroidBinary. Each android_library in the transitive
closure needs to be compiled against the binary's target_sdk.
Libraries are passed to the aapt tool once for every SDK
version they support.

This also adds lots of testing for big parts of the android
pipeline and a couple of new example projects to show the
new functionality.

Problems/Followups

* Duplicated code in UnpackJars and UnpackLibraries, including
  fingerprint handling. Pulling up an Unpack base class or refactor
  to FileUtil or something (this includes Fingerprint strategy)
* No invalidation framework for aapt_gen yet. It doesn't currently
  distinguish between libraries that need to be processed by
  multiple SDKs.

References pantsbuild#1390
References pantsbuild#10

Testing Done:
Travis passed

Bugs closed: 1866

Reviewed at https://rbcommons.com/s/twitter/r/2528/

3rdparty/android/BUILD

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+android_dependency(name='android-support-v4',
+  jars = [
+    jar(org='com.android.support', name='support-v4', rev='22.0.0'),
+  ],
+)
+
+android_dependency(name='appcompat-v7',
+  jars = [
+    jar(org='com.android.support', name='appcompat-v7', rev='22.0.0'),
+  ],
+)
+
+android_dependency(name='google-play-services',
+  jars=[
+    jar(org='com.google.android.gms', name='play-services', rev='7.0.0'),
+  ],
+)

build-support/ivy/ivysettings.xml

@@ -5,27 +5,43 @@ Licensed under the Apache License, Version 2.0 (see LICENSE).
 -->
 
 <ivysettings>
+
   <settings defaultResolver="pants-chain-repos"/>
 
+  <properties environment="env" />
   <property name="m2.repo.relpath" value="[organisation]/[module]/[revision]"/>
   <property name="m2.repo.pom" value="${m2.repo.relpath}/[module]-[revision].pom"/>
   <property name="m2.repo.artifact"
             value="${m2.repo.relpath}/[artifact](-[classifier])-[revision].[ext]"/>
   <property name="m2.repo.dir" value="${user.home}/.m2/repository" override="false"/>
 
+  <property name="env.ANDROID_HOME" value="/please-export-your-ANDROID_HOME" override="false"/>
+  <property name="android.repo.dir" value="${env.ANDROID_HOME}/extras/android/m2repository"/>
+  <property name="google.repo.dir" value="${env.ANDROID_HOME}/extras/google/m2repository"/>
   <resolvers>
+
     <chain name="pants-chain-repos" returnFirst="true">
       <!-- By default ivy does not require metadata (or successful metadata downloads).
            This can lead to jars downloading without their transitive deps which leads
            to confusing failures later when classpaths are constructed and used.
            We setup the maven central resolver to require successful pom downloads here. -->
       <ibiblio name="maven-central" m2compatible="true" descriptor="required"/>
 
+      <!-- Android SDK Support Library, found in reference to the ANDROID_HOME variable. -->
+      <ibiblio name="local.android"
+               m2compatible="true"
+               root="file://${android.repo.dir}/"/>
+      <!-- Google Libraries for the Android SDK, found in reference to ANDROID_HOME. -->
+      <ibiblio name="local.google"
+               m2compatible="true"
+               root="file://${google.repo.dir}/"/>
+
       <!-- The mvn standard local filesystem repo/cache -->
       <filesystem name="local.m2" m2compatible="true" local="true" checkmodified="true">
         <ivy pattern="${m2.repo.dir}/${m2.repo.pom}"/>
         <artifact pattern="${m2.repo.dir}/${m2.repo.artifact}"/>
       </filesystem>
+
     </chain>
   </resolvers>
 </ivysettings>

examples/src/android/example_library/AndroidManifest.xml

@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+Licensed under the Apache License, Version 2.0 (see LICENSE).
+-->
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+          package="org.pantsbuild.examples.example_library">
+    <uses-sdk android:minSdkVersion="9" />
+
+    <application>
+        <meta-data
+                android:name="org.pantsbuild.examples.example_library.ExampleLibrary" />
+    </application>
+
+</manifest>

examples/src/android/example_library/BUILD

@@ -0,0 +1,32 @@
+# coding=utf-8
+# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+android_library(
+  name='example_library',
+  manifest='AndroidManifest.xml',
+  libraries=['3rdparty/android:android-support-v4'],
+  include_patterns=[
+    'android/**/*.class',
+  ],
+  dependencies=[
+    ':gms-library',
+    ':resources',
+  ],
+)
+
+android_resources(
+  name='resources',
+  manifest='AndroidManifest.xml',
+  resource_dir='res'
+)
+
+android_library(
+  name='gms-library',
+  libraries=['3rdparty/android:google-play-services'],
+  include_patterns=[
+    '**/*.class',
+  ],
+  dependencies = [
+  ]
+)

examples/src/android/example_library/res/values/strings.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+Licensed under the Apache License, Version 2.0 (see LICENSE).
+-->
+<resources>
+    <string name="appName">Hello Pants</string>
+    <string name="hello">Hello world!</string>
+</resources>

examples/src/android/hello_with_library/BUILD

@@ -0,0 +1,20 @@
+# coding=utf-8
+# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+android_binary(
+  name='hello_with_library',
+  sources=rglobs('main/src/*.java'),
+  manifest='main/AndroidManifest.xml',
+  dependencies = [
+    'examples/src/android/example_library',
+    ':resources',
+  ],
+)
+
+android_resources(
+  name='resources',
+  manifest='AndroidManifest.xml',
+  resource_dir='main/res'
+)
+

examples/src/android/hello_with_library/main/AndroidManifest.xml

@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+Licensed under the Apache License, Version 2.0 (see LICENSE).
+-->
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="org.pantsbuild.examples.hello_with_library"
+    android:versionCode="1"
+    android:versionName="1.0" >
+
+    <uses-sdk
+        android:minSdkVersion="8"
+        android:targetSdkVersion="21" />
+
+    <application
+        android:allowBackup="true"
+        android:icon="@drawable/ic_launcher_hdpi"
+        android:label="@string/appName">
+        android:theme="@style/AppTheme" >
+        <activity
+            android:name="org.pantsbuild.examples.hello_with_library.HelloLibrary"
+            android:configChanges="orientation|keyboardHidden|screenSize"
+            android:label="@string/appName">
+            android:theme="@style/FullscreenTheme" >
+            <intent-filter>
+                <action android:name="android.intent.action.MAIN" />
+
+                <category android:name="android.intent.category.LAUNCHER" />
+            </intent-filter>
+        </activity>
+    </application>
+
+</manifest>

examples/src/android/hello_with_library/main/res/values/strings.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+Licensed under the Apache License, Version 2.0 (see LICENSE).
+-->
+<resources>
+    <string name="appName">Hello Library</string>
+    <string name="library_greeting">This string is part of the example library!</string>
+</resources>

examples/src/android/hello_with_library/main/src/org/pantsbuild/examples/hello_with_library/HelloLibrary.java

@@ -0,0 +1,30 @@
+// coding=utf-8
+// Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
+// Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+package org.pantsbuild.examples.hello_with_library;
+
+import android.app.Activity;
+import android.content.res.Resources;
+import android.os.Bundle;
+import android.support.v4.app.Fragment;
+import android.widget.TextView;
+
+public class HelloLibrary extends Activity {
+
+  @Override
+  public void onCreate(Bundle savedInstanceState) {
+    super.onCreate(savedInstanceState);
+
+    TextView textView = new TextView(this);
+
+    String text = getResources().getString(R.string.hello);
+    textView.setText(text);
+
+    // Toy demonstration of using an android_library comprised of a jar and associated resources.
+    String greeting = getResources().getString(R.string.library_greeting);
+    textView.setText(greeting);
+
+    setContentView(textView);
+  }
+}

src/python/pants/backend/android/tasks/BUILD

@@ -31,7 +31,6 @@ python_library(
   dependencies = [
     'src/python/pants/backend/android/targets:android',
     'src/python/pants/backend/android/tasks:aapt_task',
-    'src/python/pants/base:build_environment',
     'src/python/pants/base:exceptions',
     'src/python/pants/base:workunit',
     'src/python/pants/util:dirutil',
@@ -42,11 +41,8 @@ python_library(
   name = 'aapt_gen',
   sources = ['aapt_gen.py'],
   dependencies = [
-    '3rdparty/python/twitter/commons:twitter.common.collections',
     'src/python/pants/backend/android/targets:android',
     'src/python/pants/backend/android/tasks:aapt_task',
-    'src/python/pants/backend/codegen/tasks:code_gen',
-    'src/python/pants/backend/core/tasks:common',
     'src/python/pants/backend/jvm/targets:java',
     'src/python/pants/base:address',
     'src/python/pants/base:build_environment',
@@ -60,6 +56,7 @@ python_library(
   name = 'aapt_task',
   sources = ['aapt_task.py'],
   dependencies = [
+    'src/python/pants/backend/android/targets:android',
     'src/python/pants/backend/android/tasks:android_task',
   ],
 )
@@ -71,10 +68,9 @@ python_library(
     'src/python/pants/backend/android/targets:android',
     'src/python/pants/backend/android/tasks:android_task',
     'src/python/pants/backend/core/tasks:common',
+    'src/python/pants/backend/jvm/tasks:unpack_jars',
     'src/python/pants/backend/jvm/tasks:nailgun_task',
     'src/python/pants/base:exceptions',
-    'src/python/pants/base:workunit',
-    'src/python/pants/util:dirutil',
   ],
 )
 

src/python/pants/backend/android/tasks/aapt_builder.py

@@ -9,10 +9,8 @@
 import os
 import subprocess
 
-from pants.backend.android.targets.android_binary import AndroidBinary
 from pants.backend.android.targets.android_resources import AndroidResources
 from pants.backend.android.tasks.aapt_task import AaptTask
-from pants.base.build_environment import get_buildroot
 from pants.base.exceptions import TaskError
 from pants.base.workunit import WorkUnit
 from pants.util.dirutil import safe_mkdir
@@ -28,72 +26,70 @@ class AaptBuilder(AaptTask):
   target's resource files. The output is an unsigned .apk, an Android application package file.
   """
 
-  @staticmethod
-  def is_app(target):
-    return isinstance(target, AndroidBinary)
-
   @classmethod
   def product_types(cls):
     return ['apk']
 
+  @classmethod
+  def package_name(cls, binary):
+    return '{0}.unsigned.apk'.format(binary.manifest.package_name)
+
   @classmethod
   def prepare(cls, options, round_manager):
     super(AaptBuilder, cls).prepare(options, round_manager)
     round_manager.require_data('dex')
 
-  def render_args(self, target, resource_dir, inputs):
-    args = []
-
-    # Glossary of used aapt flags. Aapt handles a ton of action, this will continue to expand.
+  def _render_args(self, binary, resource_dirs, dex_files):
+    # Glossary of used aapt flags.
     #   : 'package' is the main aapt operation (see class docstring for more info).
     #   : '-f' to 'force' overwrites if the package already exists.
     #   : '-M' is the AndroidManifest.xml of the project.
-    #   : '-S' points to the resource_dir to "spider" down while collecting resources.
+    #   : '--auto-add-overlay' automatically add resources that are only in overlays.
+    #   : '-S' points to each dir in resource_dirs, aapt 'scans' them in order while
+    #            collecting resources (resource priority is left -> right).
     #   : '-I' packages to add to base "include" set, here the android.jar of the target-sdk.
     #   : '--ignored-assets' patterns for the aapt to skip. This is the default w/ 'BUILD*' added.
     #   : '-F' The name and location of the .apk file to output.
     #   : additional positional arguments are treated as input directories to gather files from.
-    args.extend([self.aapt_tool(target.build_tools_version)])
+    args = []
+    args.extend([self.aapt_tool(binary)])
     args.extend(['package', '-f'])
-    args.extend(['-M', target.manifest.path])
-    args.extend(['-S'])
-    args.extend(resource_dir)
-    args.extend(['-I', self.android_jar_tool(target.manifest.target_sdk)])
+    args.extend(['-M', binary.manifest.path])
+    args.append('--auto-add-overlay')
+    for resource_dir in resource_dirs:
+      args.extend(['-S', resource_dir])
+    args.extend(['-I', self.android_jar(binary)])
     args.extend(['--ignore-assets', self.ignored_assets])
-    args.extend(['-F', os.path.join(self.workdir,
-                                    '{0}.unsigned.apk'.format(target.manifest.package_name))])
-    args.extend(inputs)
+    args.extend(['-F', os.path.join(self.workdir, self.package_name(binary))])
+    args.extend(dex_files)
     logger.debug('Executing: {0}'.format(' '.join(args)))
     return args
 
   def execute(self):
     safe_mkdir(self.workdir)
-    targets = self.context.targets(self.is_app)
-    with self.invalidated(targets) as invalidation_check:
+    binaries = self.context.targets(self.is_android_binary)
+    with self.invalidated(binaries) as invalidation_check:
       invalid_targets = []
       for vt in invalidation_check.invalid_vts:
         invalid_targets.extend(vt.targets)
-      for target in invalid_targets:
-        # 'input_dirs' is the folder containing the Android dex file.
-        input_dirs = []
-        # 'gen_out' holds resource folders (e.g. 'res').
-        gen_out = []
+      for binary in invalid_targets:
+
+        dex_files = []
         mapping = self.context.products.get('dex')
-        for basedir in mapping.get(target):
-          input_dirs.append(basedir)
+        for dex in mapping.get(binary):
+          dex_files.append(dex)
 
-        def gather_resources(target):
-          """Gather the 'resource_dir' of the target."""
-          if isinstance(target, AndroidResources):
-            gen_out.append(os.path.join(get_buildroot(), target.resource_dir))
+        resource_deps = self.context.build_graph.transitive_subgraph_of_addresses([binary.address])
+        resource_dirs = [t.resource_dir for t in resource_deps if isinstance(t, AndroidResources)]
 
-        target.walk(gather_resources)
-        args = self.render_args(target, gen_out, input_dirs)
-        with self.context.new_workunit(name='apk-bundle', labels=[WorkUnit.MULTITOOL]) as workunit:
+        # Priority for resources is left to right, so reverse the collection order (DFS preorder).
+        args = self._render_args(binary, reversed(resource_dirs), dex_files)
+        with self.context.new_workunit(name='apk-bundle',
+                                       labels=[WorkUnit.MULTITOOL]) as workunit:
           returncode = subprocess.call(args, stdout=workunit.output('stdout'),
                                        stderr=workunit.output('stderr'))
           if returncode:
             raise TaskError('Android aapt tool exited non-zero: {0}'.format(returncode))
-    for target in targets:
-      apk_name = '{0}.unsigned.apk'.format(target.manifest.package_name)
-      self.context.products.get('apk').add(target, self.workdir).append(apk_name)
+    for binary in binaries:
+      apk_name = self.package_name(binary)
+      self.context.products.get('apk').add(binary, self.workdir).append(apk_name)