Skip to content

Commit

Permalink
Add ReDex step in android_binary build rule
Browse files Browse the repository at this point in the history 
Summary: Add a redex step as an optional part of the android_binary build rule.

Test Plan:
# Unittest and integration test added in the diff
# Test locally on my laptop when building a redex enabled android_binary target in fbandroid/native/redex-stable:

  zwei@zwei-mbp: ~/fbsource/fbandroid ⚡ TRACE=3 TRACEFILE=~/fbandroid/trace.log NO_BUCKD=1 ~/local/buck/bin/buck build --no-cache native/redex-stable/test/samples/counter:counter_redex                   [16:16:43]
  ::: '.nobuckcheck' file is present. Not updating buck.
  Not using buckd because NO_BUCKD is set.
  [-] PROCESSING BUCK FILES...FINISHED 0.4s [100%] 🐳  New buck daemon
  [-] DOWNLOADING... (0.00 B/S AVG, TOTAL: 0.00 B, 0 Artifacts)
  [-] BUILDING...FINISHED 8.7s [100%] (46/46 JOBS, 46 UPDATED, 44 [95.7%] CACHE MISS)

# Sandcastle test is coming.

Reviewed By: marcinkosiba

fbshipit-source-id: 328ba55
  • Loading branch information
@thezhangwei @facebook-github-bot
thezhangwei authored and facebook-github-bot committed Feb 27, 2017
1 parent d7eff16 commit 7fe53c9
Show file tree
Hide file tree
Showing 23 changed files with 753 additions and 21 deletions.
89 changes: 89 additions & 0 deletions docs/rule/android_binary.soy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,55 @@
{call buck.rule}
{param status: 'FROZEN' /}
{param overview}
<p>
An <code>android_binary()</code> rule is used to generate an Android APK.
</p>

<h2 id="redex-overview">ReDex</h2>
<p>
Buck makes it easy to run <a href="http://fbredex.com/">ReDex</a> as part
of <code>android_binary()</code>. ReDex is an Android bytecode optimizer
that can help reduce the size of your release builds. Please refer to
<a href="http://fbredex.com/">ReDex</a> for instructions on how to create a ReDex executable.
</p>
<p>
Currently, the path to ReDex must be
specified as a path (or build target) in <code>.buckconfig</code>:
</p>
{literal}<pre class="prettyprint lang-ini">
[android]
# In this example, we assume that you have ReDex packaged as a single
# executable file named redex-bin. Read "Creating redex-bin" below on how
# to create this file. Note that the expectation is that you check
# redex-bin into your project.
redex = third-party/native/redex/redex-bin

# Alternatively, you could create a genrule(), such as
# //tools/redex:redex, that wraps ReDex. It must have
# `executable = True` to ensure the genrule() is runnable.
# In this case, you would specify the build target instead
# of the file path.
# redex = //tools/redex:redex
</pre>{/literal}

<p>
If a ReDex executable is set properly in <code>.buckconfig</code>, then it
is possible to run ReDex as part of building an <code>android_binary()</code>,
but the following conditions must hold:
</p>
<ul>
<li><code>redex = True</code> must be set on the <code>android_binary()</code>.
<li><code>package_type = 'release'</code> must be set on the <code>android_binary()</code>.
</ul>

<p>
Other ReDex configuration options on <code>android_binary()</code> include:
</p>

<ul>
<li><code><a href="#arg/redex_config">redex_config</a></code>
<li><code><a href="#arg/redex_json_args">redex_json_args</a></code>
</ul>
{/param}

{param args}
Expand Down Expand Up @@ -240,6 +288,40 @@ An <code>android_binary()</code> rule is used to generate an Android APK.
{/param}
{/call}

{call buck.arg}
{param name : 'redex' /}
{param default : 'False' /}
{param desc}
<p>
Set this to <code>True</code> to run ReDex on the APK generated by this rule.
</p>
{call .redexLink /}
{/param}
{/call}

{call buck.arg}
{param name : 'redex_config' /}
{param default : 'None' /}
{param desc}
<p>
Path to a ReDex config file. (This is passed as via the <code>--config</code> option to ReDex.)
Note this may be a generated file.
</p>
{call .redexLink /}
{/param}
{/call}

{call buck.arg}
{param name : 'redex_extra_args' /}
{param default : '[]' /}
{param desc}
<p>
A list of additional arguments to pass to the ReDex binary.
</p>
{call .redexLink /}
{/param}
{/call}

{/param} // close args

{param examples}
Expand Down Expand Up @@ -282,3 +364,10 @@ android_binary(
{/param}
{/call}
{/template}

/***/
{template .redexLink}
<p>
See the <a href="#redex-overview">ReDex overview</a> for more on using ReDex in Buck.
</p>
{/template}
55 changes: 47 additions & 8 deletions src/com/facebook/buck/android/AndroidBinary.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

import com.facebook.buck.android.FilterResourcesStep.ResourceFilter;
import com.facebook.buck.android.ResourcesFilter.ResourceCompressionMode;
import com.facebook.buck.android.redex.ReDexStep;
import com.facebook.buck.android.redex.RedexOptions;
import com.facebook.buck.io.ProjectFilesystem;
import com.facebook.buck.jvm.java.AccumulateClassNamesStep;
import com.facebook.buck.jvm.java.HasClasspathEntries;
Expand Down Expand Up @@ -200,6 +202,9 @@ enum RelinkerMode {
private final SourcePathRuleFinder ruleFinder;
@AddToRuleKey
private final Optional<SourcePath> proguardJarOverride;
@AddToRuleKey
private final Optional<RedexOptions> redexOptions;

private final String proguardMaxHeapSize;
@AddToRuleKey
private final Optional<List<String>> proguardJvmArgs;
Expand Down Expand Up @@ -265,6 +270,7 @@ enum RelinkerMode {
Optional<Integer> proguardOptimizationPasses,
Optional<SourcePath> proguardConfig,
Optional<Boolean> skipProguard,
Optional<RedexOptions> redexOptions,
ResourceCompressionMode resourceCompressionMode,
Set<NdkCxxPlatforms.TargetCpuType> cpuFilters,
ResourceFilter resourceFilter,
Expand All @@ -287,6 +293,7 @@ enum RelinkerMode {
this.proguardJarOverride = proguardJarOverride;
this.proguardMaxHeapSize = proguardMaxHeapSize;
this.proguardJvmArgs = proguardJvmArgs;
this.redexOptions = redexOptions;
this.proguardAgentPath = proguardAgentPath;
this.keystore = keystore;
this.keystorePath = keystore.getPathToStore();
Expand Down Expand Up @@ -543,6 +550,18 @@ public ImmutableList<Step> getBuildSteps(

SourcePathResolver resolver = context.getSourcePathResolver();
Path signedApkPath = getSignedApkPath();
final Path pathToKeystore = resolver.getAbsolutePath(keystorePath);
Supplier<KeystoreProperties> keystoreProperties = Suppliers.memoize(() -> {
try {
return KeystoreProperties.createFromPropertiesFile(
pathToKeystore,
resolver.getAbsolutePath(keystorePropertiesPath),
getProjectFilesystem());
} catch (IOException e) {
throw new RuntimeException();
}
});

ApkBuilderStep apkBuilderCommand = new ApkBuilderStep(
getProjectFilesystem(),
context.getSourcePathResolver().getAbsolutePath(resourcesApkPath),
Expand All @@ -554,8 +573,8 @@ public ImmutableList<Step> getBuildSteps(
packageableCollection.getPathsToThirdPartyJars().stream()
.map(resolver::getAbsolutePath)
.collect(MoreCollectors.toImmutableSet()),
resolver.getAbsolutePath(keystorePath),
resolver.getAbsolutePath(keystorePropertiesPath),
pathToKeystore,
keystoreProperties,
/* debugMode */ false,
javaRuntimeLauncher);
steps.add(apkBuilderCommand);
Expand All @@ -564,27 +583,47 @@ public ImmutableList<Step> getBuildSteps(
// the output non-deterministic. So use an additional scrubbing step to zero these out.
steps.add(new ZipScrubberStep(getProjectFilesystem(), signedApkPath));

Path apkToAlign;
Path apkToRedexAndAlign;
// Optionally, compress the resources file in the .apk.
if (this.isCompressResources()) {
Path compressedApkPath = getCompressedResourcesApkPath();
apkToAlign = compressedApkPath;
apkToRedexAndAlign = compressedApkPath;
RepackZipEntriesStep arscComp = new RepackZipEntriesStep(
getProjectFilesystem(),
signedApkPath,
compressedApkPath,
ImmutableSet.of("resources.arsc"));
steps.add(arscComp);
} else {
apkToAlign = signedApkPath;
apkToRedexAndAlign = signedApkPath;
}

boolean applyRedex = redexOptions.isPresent();
Path apkPath = context.getSourcePathResolver().getRelativePath(getSourcePathToOutput());
ZipalignStep zipalign = new ZipalignStep(
Path apkToAlign = apkToRedexAndAlign;

// redex
if (applyRedex) {
Path proguardConfigDir = enhancementResult.getAaptPackageResources()
.getPathToGeneratedProguardConfigDir();
Path redexedApk = apkPath.getParent().resolve(apkPath.getFileName().toString() + ".redex");
apkToAlign = redexedApk;
ImmutableList<Step> redexSteps = ReDexStep.createSteps(
getProjectFilesystem(),
resolver,
redexOptions.get(),
apkToRedexAndAlign,
redexedApk,
keystoreProperties,
proguardConfigDir
);
steps.addAll(redexSteps);
}

steps.add(new ZipalignStep(
getProjectFilesystem().getRootPath(),
apkToAlign,
apkPath);
steps.add(zipalign);
apkPath));

buildableContext.recordArtifact(apkPath);
return steps.build();
Expand Down
83 changes: 81 additions & 2 deletions src/com/facebook/buck/android/AndroidBinaryDescription.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
import com.facebook.buck.android.NdkCxxPlatforms.TargetCpuType;
import com.facebook.buck.android.ResourcesFilter.ResourceCompressionMode;
import com.facebook.buck.android.aapt.RDotTxtEntry.RType;
import com.facebook.buck.android.redex.RedexOptions;
import com.facebook.buck.cli.BuckConfig;
import com.facebook.buck.cxx.CxxBuckConfig;
import com.facebook.buck.dalvik.ZipSplitter.DexSplitStrategy;
import com.facebook.buck.event.PerfEventId;
Expand All @@ -43,11 +45,14 @@
import com.facebook.buck.rules.BuildRule;
import com.facebook.buck.rules.BuildRuleParams;
import com.facebook.buck.rules.BuildRuleResolver;
import com.facebook.buck.rules.CellPathResolver;
import com.facebook.buck.rules.Description;
import com.facebook.buck.rules.Hint;
import com.facebook.buck.rules.ImplicitDepsInferringDescription;
import com.facebook.buck.rules.SourcePath;
import com.facebook.buck.rules.SourcePathRuleFinder;
import com.facebook.buck.rules.TargetGraph;
import com.facebook.buck.rules.Tool;
import com.facebook.buck.rules.coercer.BuildConfigFields;
import com.facebook.buck.rules.coercer.ManifestEntries;
import com.facebook.buck.rules.macros.ExecutableMacroExpander;
Expand All @@ -57,6 +62,7 @@
import com.facebook.buck.util.MoreCollectors;
import com.facebook.buck.util.RichStream;
import com.facebook.infer.annotation.SuppressFieldNotInitialized;
import com.google.common.base.Joiner;
import com.google.common.base.Preconditions;
import com.google.common.base.Suppliers;
import com.google.common.collect.ImmutableList;
Expand All @@ -66,6 +72,7 @@
import com.google.common.collect.Ordering;
import com.google.common.util.concurrent.ListeningExecutorService;

import java.util.Arrays;
import java.util.EnumSet;
import java.util.List;
import java.util.Locale;
Expand All @@ -74,12 +81,18 @@
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;

public class AndroidBinaryDescription
implements Description<AndroidBinaryDescription.Arg>, Flavored {
public class AndroidBinaryDescription implements
Description<AndroidBinaryDescription.Arg>,
Flavored,
ImplicitDepsInferringDescription<AndroidBinaryDescription.Arg> {

private static final Logger LOG = Logger.get(AndroidBinaryDescription.class);

private static final String SECTION = "android";
private static final String CONFIG_PARAM_REDEX = "redex";

/**
* By default, assume we have 5MB of linear alloc,
* 1MB of which is taken up by the framework, so that leaves 4MB.
Expand All @@ -100,6 +113,7 @@ public class AndroidBinaryDescription
private final JavaOptions javaOptions;
private final JavacOptions javacOptions;
private final ProGuardConfig proGuardConfig;
private final BuckConfig buckConfig;
private final CxxBuckConfig cxxBuckConfig;
private final ImmutableMap<TargetCpuType, NdkCxxPlatform> nativePlatforms;
private final ListeningExecutorService dxExecutorService;
Expand All @@ -110,10 +124,12 @@ public AndroidBinaryDescription(
ProGuardConfig proGuardConfig,
ImmutableMap<TargetCpuType, NdkCxxPlatform> nativePlatforms,
ListeningExecutorService dxExecutorService,
BuckConfig buckConfig,
CxxBuckConfig cxxBuckConfig) {
this.javaOptions = javaOptions;
this.javacOptions = javacOptions;
this.proGuardConfig = proGuardConfig;
this.buckConfig = buckConfig;
this.cxxBuckConfig = cxxBuckConfig;
this.nativePlatforms = nativePlatforms;
this.dxExecutorService = dxExecutorService;
Expand Down Expand Up @@ -283,6 +299,7 @@ public <A extends Arg> BuildRule createBuildRule(
args.optimizationPasses,
args.proguardConfig,
args.skipProguard,
getRedexOptions(params.getBuildTarget(), args, resolver),
compressionMode,
args.cpuFilters,
resourceFilter,
Expand Down Expand Up @@ -363,6 +380,64 @@ public boolean hasFlavors(ImmutableSet<Flavor> flavors) {
return true;
}

@Override
public Iterable<BuildTarget> findDepsForTargetFromConstructorArgs(
BuildTarget buildTarget,
CellPathResolver cellRoots,
Arg constructorArg) {
ImmutableSet.Builder<BuildTarget> deps = ImmutableSet.builder();
if (constructorArg.redex.orElse(false) &&
getPackageType(constructorArg).isBuildWithObfuscation()) {
// If specified, this option may point to either a BuildTarget or a file.
Optional<BuildTarget> redexTarget = buckConfig.getMaybeBuildTarget(
SECTION,
CONFIG_PARAM_REDEX);
if (redexTarget.isPresent()) {
deps.add(redexTarget.get());
}
}
return deps.build();
}

private Optional<RedexOptions> getRedexOptions(
BuildTarget buildTarget,
Arg arg,
BuildRuleResolver sourcePathResolver) {
boolean redexRequested = arg.redex.orElse(false);
if (!redexRequested) {
return Optional.empty();
}

PackageType packageType = getPackageType(arg);
boolean buildWithObfuscation = packageType.isBuildWithObfuscation();
if (!buildWithObfuscation) {
List<PackageType> supported = Arrays.stream(PackageType.values())
.filter(PackageType::isBuildWithObfuscation)
.collect(Collectors.toList());
throw new HumanReadableException("Requested running ReDex for %s but the package type %s " +
"is not compatible with that options. Consider changing it to %s.",
buildTarget,
packageType,
Joiner.on(", ").join(supported));
}

Optional<Tool> redexBinary =
buckConfig.getTool(SECTION, CONFIG_PARAM_REDEX, sourcePathResolver);
if (!redexBinary.isPresent()) {
throw new HumanReadableException("Requested running ReDex for %s but the path to the tool" +
"has not been specified in the %s.%s .buckconfig section.",
buildTarget,
SECTION,
CONFIG_PARAM_REDEX);
}

return Optional.of(RedexOptions.builder()
.setRedex(redexBinary.get())
.setRedexConfig(arg.redexConfig)
.setRedexExtraArgs(arg.redexExtraArgs)
.build());
}

@SuppressFieldNotInitialized
public static class Arg extends AbstractDescriptionArg implements HasTests {
public SourcePath manifest;
Expand Down Expand Up @@ -429,6 +504,10 @@ public static class Arg extends AbstractDescriptionArg implements HasTests {
public Optional<Boolean> enableRelinker;
public ManifestEntries manifestEntries = ManifestEntries.empty();
public BuildConfigFields buildConfigValues = BuildConfigFields.empty();
public Optional<Boolean> redex;
public Optional<SourcePath> redexConfig;
public ImmutableList<String> redexExtraArgs = ImmutableList.of();

public Optional<SourcePath> buildConfigValuesFile;
public Optional<Boolean> skipProguard;
public ImmutableSortedSet<BuildTarget> deps = ImmutableSortedSet.of();
Expand Down
Loading

0 comments on commit 7fe53c9

Please sign in to comment.