Skip to content

Commit

Permalink
Updates for documentation followon from Radical site redesign
Browse files Browse the repository at this point in the history 
Updates to the concepts page, Invoking Pants and the 3rdparty JVM page.

Testing Done:
Staged at:
http://pantsbuild.github.io/staging/zundel/first_concepts.html
http://pantsbuild.github.io/invoking.html
http://pantsbuild.github.io/staging/zundel/3rdparty_jvm.html

Bugs closed: 3310

Reviewed at https://rbcommons.com/s/twitter/r/3794/
  • Loading branch information
@ericzundel
ericzundel committed May 1, 2016
1 parent e415a5a commit 1dd44b0
Show file tree
Hide file tree
Showing 4 changed files with 188 additions and 127 deletions.
39 changes: 18 additions & 21 deletions examples/src/java/org/pantsbuild/example/3rdparty_jvm.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,33 +47,30 @@ And your Java code might have:
"Round Trip" Dependencies
-------------------------

Depending on your workspace's relation with the rest of the world, you
might want to look out for "round trip" dependencies. You can publish an
artifact *near* generated from your workspace's source code and consume
a third-party artifact *far* that depends on *near*. If you're not
careful, you might depend on two versions of the *near* code: the local
source code and an artifact you published a while ago.

When consuming such third-party artifacts, ensure that your source dependencies
have `provides` clauses (*near*), and then add the source dependencies
explicitly when you depend on the binary copy of the *far* dependency:
It is possible for your code to exist as source in the repo but
also be published as a binary to an external repository. If you happen to pull in any
third party artifacts, they may express a dependency on the published
version of the artifact. This means that the classpath will contain
both the version in the repo compiled from source and an older version
that was previously published. In this case, you want to be sure that
when pants always prefers the version built from source.

Fortunately, the remedy for this is simple. If you add a `provides=`
parameter that matches the one used to publish the artifact, pants
will always prefer the local target definition to the
published jar.

:::python
jar_library(name='far',
jars=[
jar(org='org.archie', name='far', rev='0.0.18'),
]
dependencies=[
# including the local version of source manually will cause the binary
# dependency to be automatically excluded:
'util/near',
]
)
jar_library(name='api',
sources = globs('*.java'),
provides = artifact(org='org.archie',
name='api',
repo=myrepo,)
)

Controlling JAR Dependency Versions
-----------------------------------


**If you notice a small number of wrong-version things,** then in a JVM
target, you can depend on a `jar` that specifies a version and sets
`force=True` to *force* using that version:
Expand Down
9 changes: 9 additions & 0 deletions examples/src/java/org/pantsbuild/example/from_maven.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,12 @@ Pants: `run.jvm --jvm-debug`
**Run one test in the debugger**<br>
Maven: `-Dtest=com.foo.BarSpec -Dmaven.surefire.debug=true test`<br>
Pants: `test.junit --jvm-debug --test=com.foo.BarSpec`

**Build a binary package**<br>
Maven: `package`<br>
Pants: `binary`<br>

**Look at dependent projects or artifacts**<br>
Maven: `dependency:analyze`<br>
Pants: `depmap`<br>
Pants: `resolve.ivy --open`<br>
192 changes: 96 additions & 96 deletions src/docs/first_concepts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,47 @@ Pants Concepts

To use Pants effectively, it helps to understand a few concepts:

Goals and Targets
-----------------

**Goals** are the "verbs" of Pants.<br>
When you invoke Pants, you name goals on the command line to say what
Pants should do. For example, to run tests, you would invoke Pants with
the `test` goal. To create a bundle--an archive containing a runnable
binary and resource files--you would invoke Pants with the `bundle`
goal.

**Targets** are the "nouns" of Pants, things pants can act upon.<br>
You annotate your source code with `BUILD` files to define these
+ **Goals** help users tell pants what actions to take
+ **Tasks** are the pants modules that run actions
+ **Targets** describe what files to take those actions upon
+ **Target types** define the types of operations that can be performed on
a target
+ **Addresses** describe the location of a target in the repo

Goals and Tasks
---------------

**Goals** are the "verbs" of Pants. When you invoke Pants, you name
goals on the command line to say what Pants should do. For example, to
run tests, you would invoke Pants with the `test` goal. To create a
bundle--an archive containing a runnable binary and resource
files--you would invoke Pants with the `bundle` goal.

**Tasks** actually perform the work. When you invoke a goal, it finds
all the tasks needed to make that goal work. Sometimes, there is a one
to one mapping between a task name and a goal name, but often multiple tasks
are registered in a goal. For example, the `junit`
task runs tests for JVM projects. It is registered in the `test`
goal and sometimes referred to as `test.junit`.

In a nutshell, goals are the way users specify tasks. Sometimes goals
are helpful groupings of tasks to make it easier for users to tell
pants what they want to do.

Tasks can depend on other tasks. When you invoke a goal in
Pants, it builds a list of dependent tasks that need to be executed
first. So, when you invoke the `test` goal, it will also invoke the
tasks in the `gen`, `resolve` and `compile` goals, because tasks for
code generation, artifact resolution, and compilation need to occur
before a test can be run. To find out more about how task dependencies
are computed, read about *product types* in
[[Developing a Pants Task|pants('src/docs:dev_tasks')]].

Targets
-------

**Targets** are the "nouns" of Pants, things pants can act upon. You
annotate your source code with `BUILD` files to define these
targets. For example, if your `tests/com/twitter/mybird/` directory
contains JUnit tests, you have a `tests/com/twitter/mybird/BUILD` file
with a `junit_tests` target definition. As you change your source code,
Expand All @@ -23,34 +52,25 @@ E.g., if you refactor some code, moving part of it to a new directory,
you'll probably set up a new `BUILD` file with a target to build that
new directory's code.

When you invoke Pants, you specify goals and targets: the actions to
take, and the things to carry out those actions upon. Together, your
chosen goals and targets determine what Pants produces. Invoking the
`bundle` goal produces an archive; invoking the `test` goal displays
test results on the console. Assuming you didn't duplicate code between
folders, targets in `tests/com/twitter/mybird/` will have different code
than those in `tests/com/twitter/otherbird/`.

Goals can "depend" on other goals. For example, there are `test` and
`compile` goals. If you invoke Pants with the `test` goal, Pants "knows"
it must compile tests before it can run them, and does so. (This can be
confusing: you can invoke the `test` goal on a target that isn't
actually a test. You might think this would be a no-op. But since Pants
knows it must compile things before it tests them, it will compile the
target.)

Targets can "depend" on other targets. For example, if your `foo` code
imports code from another target `bar`, then `foo` depends on `bar`. You
specify this dependency in `foo`'s target definition in its `BUILD`
file. If you invoke Pants to compile `foo`, it "knows" it also needs to
compile `bar`, and does so.


Target Types
------------

Each Pants build target has a *type*, such as `java_library` or
`python_binary`. Pants uses the type to determine how to apply goals to
that target.
**Target Types** describe the kind of target to be operated on. Each
Pants build target has a *type*, such as `java_library` or
`python_binary`. Tasks choose to work on particular targets by
selecting the target types they are interested in from the build
graph.

For a list of all Target types (and other things that can go in `BUILD`
files), see the <a href="build_dictionary.html">BUILD Dictionary</a>.
The following list describes the most common target types:

**Library Targets**<br>
To define an "importable" thing, you want a library target type, such as
Expand All @@ -60,12 +80,12 @@ library target's code should list the library target in its

**Binary Targets**<br>
To define a "runnable" thing, you want a `jvm_binary` or `python_binary`
target. A binary probably has a `main` and dependencies. (We encourage a
target. A binary probably has a `main` and dependency libraries. (We encourage a
binary's main to be separate from the libraries it uses to run, if any.)

**External Dependencies**<br>
Not everything's source code is in your repository. Your targets can
depend on `.jar`s or `.eggs`s from elsewhere.
Not everything is source code is in your repository. Your targets can
depend on `.jar`s or `.whl`s from elsewhere.

**Test Targets**<br>
To define a collection of tests, you want a `junit_tests` or
Expand All @@ -74,48 +94,71 @@ code it tests. This isn't just logical, it's handy, too: you can
compute dependencies to figure out what tests to run if you change some
target's code.

For a list of all Target types (and other things that can go in `BUILD`
files), see the <a href="build_dictionary.html">BUILD Dictionary</a>.
Addresses
---------
**Addresses** describe the location of a target in the build
graph. The address has two parts: the directory to the BUILD file
and the name of the target within that BUILD file.

For example, if you have a `tests/com/twitter/mybird/BUILD` file
with a `junit_tests(name='test-flight)` target definition, you would
write the address as:

tests/com/twitter/mybird:test-flight

You use an address whenever you specify a target to build on the
command line or when one target depends on another in a BUILD file.
To find out more about addresses, see
[[Target Addresses|pants('src/docs:target_addresses')]].


What Pants Does
---------------

When you invoke Pants, you specify goals (actions to take) and targets
(things to act upon).

**Pants plans a list of goals.** You specify one or more goals on the
command line. Pants knows that some goals depend on others. If you
**Pants plans a list of tasks.** You specify one or more goals on the
command line. Pants finds the tasks needed to complete that goal.
Pants knows that some tasks depend on others. If you
invoke Pants with, say, the `test` goal to test some code, Pants knows
it must first compile code; before it can compile code, it needs to
resolve artifact dependencies and generate code from IDL files (e.g.,
Thrift). Pants thus generates a topologically-sorted list of goals, a
Thrift). Pants thus generates a topologically-sorted list of goals and
tasks to perform, a
*build execution plan*. This plan might look something like

> resolve-idl -\> gen -\> resolve -\> compile -\> resources -\> test
:::bash
./pants --explain test src/java::
Goal Execution Order:

bootstrap -> imports -> unpack-jars -> deferred-sources -> gen
-> jvm-platform-validate -> resolve -> compile -> resources -> test

Pants does *not* consider targets while planning; some of these goals
Pants does *not* consider targets while planning; some of these tasks
might thus turn out to be no-ops. E.g., Pants might plan a `gen`
(generate code) goal even if you don't, in fact, use any generated code.
(generate code) task even if you don't, in fact, use any generated code.

**Pants computes a target dependencies graph.** It starts with the
target[s] you specify on the command line. It notes which targets they
depend on, which targets those targets depend on, which targets *those*
targets depend on, and so on.

**Pants then attempts to carry out its planned goals.** It proceeds goal
by goal. If it has a problem carrying out one goal, it does not continue
to the other goals. (Thus, if you attempt to test targets *A* and *B*,
**Pants then attempts to carry out its planned tasks.** It proceeds goal
by goal. If it has a problem carrying out one task, it does not continue
to the others. (Thus, if you attempt to test targets *A* and *B*,
but there's a compilation error in *A*, then Pants won't test *B* even
if it compiled fine.)

For each goal, Pants attempts to apply that goal to all targets in its
computed dependency tree[s]. It starts with depended-upon targets and
works its way up to depending targets. Each Pants target has a type;
Pants uses this to determine how to apply a goal to that target. In many
cases, applying a goal to a target is a no-op. In the more interesting
cases, Pants does something. It probably invokes other tools. For
example, depending on the code in the relevant targets, that "compile"
goal might invoke `javac` a few times and `scalac`.
For each task in the plan, Pants executes that task to all
targets in its computed dependency tree[s]. It starts with
depended-upon targets and works its way up to depending targets. Each
Pants target has a type; Pants uses this to determine how to apply a
task to that target. In many cases, applying a task to a target is a
no-op. In the more interesting cases, Pants does something. It
probably invokes other tools. For example, depending on the code in
the relevant targets, that "compile" goal might invoke `javac` a few
times and `scalac`.

Pants caches things it builds. Thus, if you change one source file and
re-build, Pants probably doesn't "build the world." It just builds a few
Expand All @@ -125,49 +168,6 @@ change. (It *can* surprise you if you `touch` a file, start a compile,
and nothing happens. If you want to, e.g., see `Foo.java`'s compile
warnings again, instead of using `touch`, you might append a newline.)

Why Choose Pants?
-----------------

As your engineering organization grows past 100 people, your codebase grows even faster.
As your codebase grows, your tools need to scale with it. If you keep buiding everything, builds
get slower. If you don't build everything, then you need to know which parts to build. You want
to decompose your code into parts, each part quickly buildable.

You need a way to keep track of which parts of your code depend on which other parts.

* To build *this* app, which code must be built? (And which can be ignored?)
* If you change *this* low-level "common" code, which other code is affected?

Pants eases this organization into quick-compiling pieces. Pants models code modules (known as
"targets") and their dependencies in `BUILD` files—in a manner similar to Google's [internal build
system](http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html).
It builds only the parts of the codebase you actually need, ignoring the rest.

Pants builds faster because it builds less. It helps you organize your code into small chunks and it
only builds the chunks you need. As your codebase grows and those chunks grow, it's relatively easy
to subdivide them into more chunks. Pants supports local and distributed caching so it doesn't
re-build things it doesn't have to. If you make a small change, Pants can do a quick incremental
build.

Pants features include:

- Builds [Java, Scala](http://pantsbuild.github.io/JVMProjects.html), and
[Python](http://pantsbuild.github.io/python-readme.html).
- Generates code from [thrift](http://pantsbuild.github.io/ThriftDeps.html), Protocol
Buffers.
- Supports a distributed cache.
- Fetches external dependencies from Maven (JVM) repos and Pypi-like Python package repos.
- Runs tests, spawns REPLs, builds deployable packages.
- Runs on Linux and Mac OS X.

Where Pants is missing a feature you need, adding that feature may be easier than you thought.
You can extend Pants with plugins written in Python. Support for new languages is especially
straightforward. Pants' active developer community is eager to integrate your improvements to Pants.

If your codebase is growing beyond your toolchain's capacity but you're reluctant to divide it
into totally separate projects, you might want to give Pants a try. It may be of particular
interest if you have complex dependencies, generated code, and custom build steps.

Next Step
---------

Expand Down
Loading

0 comments on commit 1dd44b0

Please sign in to comment.