Skip to content

Commit

Permalink
Rename the dependees goal to dependents. (pantsbuild#17397)
Browse files Browse the repository at this point in the history
Also fix various casual uses of the term dependees in
comments and docs.
  • Loading branch information
benjyw authored Oct 29, 2022
1 parent 147df94 commit 3aa9c7b
Show file tree
Hide file tree
Showing 12 changed files with 54 additions and 51 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ If a user makes a request to the engine that does not have a corresponding `Quer

`Param`s are eventually used as positional args to `Rule`s, but it's important to note that the `Param`s in a `Rule` instance's identity/memoization-key will not always become the positional arguments to _that_ `Rule`: in many cases, a `Param` will be used by a `Rule`'s transitive dependencies in order to produce an output value that becomes either a positional argument to the `Rule` as it starts, or the result of a `Get` while a coroutine `Rule` runs.

The `Param`s that are available to a `Rule` are made available by the `Rule`'s dependees (its "callers"), but similar to how `Rule`s are not called by name, neither are all of their `Param`s passed explicitly at each use site. A `Rule` will be used to compute the output value for a `DependencyKey`: i.e., a positional argument, `Get` result, or `Query` result. Of these usage sites, only `Query` specifies the complete set of `Params` that will be available: the other two usages (positional arguments and `Get`s) are able to use any Param that will be "in scope" at the use site.
The `Param`s that are available to a `Rule` are made available by the `Rule`'s dependents (its "callers"), but similar to how `Rule`s are not called by name, neither are all of their `Param`s passed explicitly at each use site. A `Rule` will be used to compute the output value for a `DependencyKey`: i.e., a positional argument, `Get` result, or `Query` result. Of these usage sites, only `Query` specifies the complete set of `Params` that will be available: the other two usages (positional arguments and `Get`s) are able to use any Param that will be "in scope" at the use site.

`Params` flow down the graph from `Query`s and the provided `Param`s of `Get`s: their presence does not need to be re-declared at each intermediate callsite. When a `Rule` consumes a `Param` as a positional argument, that `Param` will no longer be available to that `Rule`'s dependencies (but it might still be present in other subgraphs adjacent to that `Rule`).

Expand All @@ -61,7 +61,7 @@ The goals of `RuleGraph` construction are:
If either of the goals were removed, `RuleGraph` construction might be more straightforward:

1. If rather than being type-driven, `Rule`s called one another by name, you could statically determine their input `Params` by walking the call graph of `Rule`s by name, and collecting their transitive input `Params`.
2. If rather than needing to compute a minimum set of `Param` inputs for the memoization key, we instead required that all usage sites explicitly declared all `Param`s that their dependencies might need, we could relatively easily eliminate candidates based on the combination of `Param` types at a use site. And if we were willing to have very large memoization keys, we could continue to have simple callsites, but skip pruning the `Params` that pass from a dependee to a dependency at runtime, and include any `Params` declared in any of a `Rule`s transitive dependees to be part of its identity.
2. If rather than needing to compute a minimum set of `Param` inputs for the memoization key, we instead required that all usage sites explicitly declared all `Param`s that their dependencies might need, we could relatively easily eliminate candidates based on the combination of `Param` types at a use site. And if we were willing to have very large memoization keys, we could continue to have simple callsites, but skip pruning the `Params` that pass from a dependent to a dependency at runtime, and include any `Params` declared in any of a `Rule`s transitive dependents to be part of its identity.

But both of the goals are important because together they allow for an API that is easy to write `Rule`s for, with minimal boilerplate required to get the inputs needed for a `Rule` to compute a value, and minimal invalidation. Because the identity of a `Rule` is computed from its transitive input `Param`s rather than from its positional arguments, `Rule`s can accept arbitrarily-many large input values (which don't need to implement hash) with no impact on its memoization hit rate.

Expand All @@ -87,8 +87,8 @@ The construction algorithm is broken up into phases:
2. [live_param_labeled](https://github.com/pantsbuild/pants/blob/3a188a1e06d8c27ff86d8c311ff1b2bdea0d39ff/src/rust/engine/rule_graph/src/builder.rs#L749-L754) - Run [live variable analysis](https://en.wikipedia.org/wiki/Live_variable_analysis) on the polymorphic graph to compute the initial "in-set" of `Params` used by each node in the graph. Because nodes in the polymorphic graph have references to all possible sources of a particular dependency type, the computed set is conservative (i.e., overly large).
- For example: if a `Rule` `x` has exactly one `DependencyKey`, but there are two potential dependencies to provide that `DependencyKey` with input `Param`s `{A,B}` and `{B,C}` (respectively), then at this phase the input `Param`s for `x` must be the union of all possibilities: `{A,B,C}`.
- If we were to stop `RuleGraph` construction at this phase, it would be necessary to do a form of [dynamic dispatch](https://en.wikipedia.org/wiki/Dynamic_dispatch) at runtime to decide which source of a dependency to use based on the `Param`s that were currently in scope. And the sets of `Param`s used in the memoization key for each `Rule` would still be overly large, causing excess invalidation.
3. [monomorphize](https://github.com/pantsbuild/pants/blob/3a188a1e06d8c27ff86d8c311ff1b2bdea0d39ff/src/rust/engine/rule_graph/src/builder.rs#L325-L353) - "Monomorphize" the polymorphic graph by using the out-set of available `Param`s (initialized during `initial_polymorphic`) and the in-set of consumed `Param`s (computed during `live_param_labeled`) to partition nodes (and their dependees) for each valid combination of their dependencies. Combinations of dependencies that would be invalid (see the Constraints section) are not generated, which causes some pruning of the graph to happen during this phase.
- Continuing the example from above: the goal of monomorphize is to create one copy of `Rule` `x` per legal combination of its `DependencyKey`. Assuming that both of `x`'s dependencies remain legal (i.e. that all of `{A,B,C}` are still in scope in the dependees of `x`, etc), then two copies of `x` will be created: one that uses the first dependency and has an in-set of `{A,B}`, and another that uses the second dependency and has an in-set of `{B,C}`.
3. [monomorphize](https://github.com/pantsbuild/pants/blob/3a188a1e06d8c27ff86d8c311ff1b2bdea0d39ff/src/rust/engine/rule_graph/src/builder.rs#L325-L353) - "Monomorphize" the polymorphic graph by using the out-set of available `Param`s (initialized during `initial_polymorphic`) and the in-set of consumed `Param`s (computed during `live_param_labeled`) to partition nodes (and their dependents) for each valid combination of their dependencies. Combinations of dependencies that would be invalid (see the Constraints section) are not generated, which causes some pruning of the graph to happen during this phase.
- Continuing the example from above: the goal of monomorphize is to create one copy of `Rule` `x` per legal combination of its `DependencyKey`. Assuming that both of `x`'s dependencies remain legal (i.e. that all of `{A,B,C}` are still in scope in the dependents of `x`, etc), then two copies of `x` will be created: one that uses the first dependency and has an in-set of `{A,B}`, and another that uses the second dependency and has an in-set of `{B,C}`.
4. [prune_edges](https://github.com/pantsbuild/pants/blob/3a188a1e06d8c27ff86d8c311ff1b2bdea0d39ff/src/rust/engine/rule_graph/src/builder.rs#L836-L845) - Once the monomorphic graph has [converged](https://en.wikipedia.org/wiki/Data-flow_analysis#Convergence), each node in the graph will ideally have exactly one source of each `DependencyKey` (with the exception of `Query`s, which are not monomorphized). This phase validates that, and chooses the smallest input `Param` set to use for each `Query`. In cases where a node has more that one dependency per `DependencyKey`, it is because given a particular set of input `Params` there was more than one valid way to compute a dependency. This can happen either because there were too many `Param`s in scope, or because there were multiple `Rule`s with the same `Param` requirements.
- This phase is the only phase that renders errors: all of the other phases mark nodes and edges "deleted" for particular reasons, and this phase consumes that record. A node that has been deleted indicates that that node is unsatisfiable for some reason, while an edge that has been deleted indicates that the source node was not able to consume the target node for some reason.
- If a node has too many sources of a `DependencyKey`, this phase will recurse to attempt to locate the node in the `Rule` graph where the ambiguity was introduced. Likewise, if a node has no source of a `DependencyKey`, this phase will recurse on deleted nodes (which are preserved by the other phases) to attempt to locate the bottom-most `Rule` that was missing a `DependencyKey`.
Expand Down
8 changes: 4 additions & 4 deletions docs/markdown/Using Pants/advanced-target-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,15 +148,15 @@ Piping to other Pants runs
To pipe a Pants run, use your shell's `|` pipe operator and `xargs`:

```bash
./pants dependees helloworld/util | xargs ./pants list
./pants dependents helloworld/util | xargs ./pants list
```

You can, of course, pipe multiple times:

```bash
# Run over the second-degree dependees of `utils.py`.
❯ ./pants dependees helloworld/utils.py | \
xargs ./pants dependees | \
# Run over the second-degree dependents of `utils.py`.
❯ ./pants dependents helloworld/utils.py | \
xargs ./pants dependents | \
xargs ./pants lint
```

Expand Down
2 changes: 1 addition & 1 deletion docs/markdown/Using Pants/concepts/source-roots.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ from base.models.user import User
from base.util.math import add_two
```

Note that even though the projects live in different top-level folders, you are still able to import from other projects. If you would like to limit this, you can use `./pants dependees` or `./pants dependencies` in CI to track where imports are being used. See [Project introspection](doc:project-introspection).
Note that even though the projects live in different top-level folders, you are still able to import from other projects. If you would like to limit this, you can use `./pants dependents` or `./pants dependencies` in CI to track where imports are being used. See [Project introspection](doc:project-introspection).

### Config:

Expand Down
22 changes: 11 additions & 11 deletions docs/markdown/Using Pants/project-introspection.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Pants provides several goals to provide insights into your project's structure.
> For example:
>
> ```bash
> $ ./pants dependees project/util.py | xargs ./pants test
> $ ./pants dependents project/util.py | xargs ./pants test
> ```
>
> See [Advanced target selection](doc:advanced-target-selection) for more info and other techniques to use the results.
Expand Down Expand Up @@ -89,29 +89,29 @@ helloworld/main.py:lib
helloworld/translator/translator.py:lib
```

`dependees` - find which targets depend on a target
---------------------------------------------------
`dependents` - find which targets depend on a target
----------------------------------------------------

The `dependees` goal finds all targets that directly depend on the target you specify.
The `dependents` goal finds all targets that directly depend on the target you specify.

```bash
❯ ./pants dependees //:ansicolors
❯ ./pants dependents //:ansicolors
helloworld/main.py:lib
```

You can specify a file, which will run on the target(s) owning that file:

```
❯ ./pants dependees helloworld/translator/translator.py
❯ ./pants dependents helloworld/translator/translator.py
helloworld/greet/greeting.py:lib
helloworld/translator:lib
helloworld/translator/translator_test.py:tests
```

To include transitive dependees—meaning targets that don't directly depend on your target, but which depend on a target that does directly use your targetuse `--transitive`:
To include transitive dependents — meaning targets that don't directly depend on your target, but which depend on a target that does directly use your targetuse `--transitive`:

```bash
❯ ./pants dependees --transitive helloworld/translator/translator.py
❯ ./pants dependents --transitive helloworld/translator/translator.py
helloworld:lib
helloworld:pex_binary
helloworld/main.py:lib
Expand All @@ -122,7 +122,7 @@ helloworld/greet:lib
To include the original target itself, use `--closed`:

```bash
❯ ./pants dependees --closed //:ansicolors
❯ ./pants dependents --closed //:ansicolors
//:ansicolors
helloworld/main.py:lib
```
Expand Down Expand Up @@ -221,10 +221,10 @@ $ ./pants peek --exclude-defaults helloworld/util:tests
> 📘 Piping other introspection commands into `./pants peek`
>
> Some introspection goals, such as `filter`, `dependencies` and `dependees` emit a flat list of target addresses. It's often useful to expand each of those into a full JSON structure with detailed properties of each target, by piping to `./pants peek`:
> Some introspection goals, such as `filter`, `dependencies` and `dependents` emit a flat list of target addresses. It's often useful to expand each of those into a full JSON structure with detailed properties of each target, by piping to `./pants peek`:
>
> ```bash
> ./pants dependees helloworld/main.py:lib | xargs ./pants peek --exclude-defaults
> ./pants dependents helloworld/main.py:lib | xargs ./pants peek --exclude-defaults
> [
> {
> "address": "helloworld:lib",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -711,7 +711,7 @@ This is tracked by <https://github.com/pantsbuild/pants/issues/10917>.

If you have any custom fields that act like the dependencies field, but do not subclass `Dependencies`, there are two new mechanisms for better support.

1. Instead of subclassing `StringSequenceField`, subclass `SpecialCasedDependencies` from `pants.engine.target`. This will ensure that the dependencies show up with `./pants dependencies` and `./pants dependees`.
1. Instead of subclassing `StringSequenceField`, subclass `SpecialCasedDependencies` from `pants.engine.target`. This will ensure that the dependencies show up with `./pants dependencies` and `./pants dependents`.
2. You can use `UnparsedAddressInputs` from `pants.engine.addresses` to resolve the addresses:

```python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ async def demo(...) -> Foo:
)
```

Pants will include your special-cased dependencies with `./pants dependencies`, `./pants dependees`, and `./pants --changed-since`, but the dependencies will not show up when using `await Get(Addresses, DependenciesRequest)`.
Pants will include your special-cased dependencies with `./pants dependencies`, `./pants dependents`, and `./pants --changed-since`, but the dependencies will not show up when using `await Get(Addresses, DependenciesRequest)`.

`SourcesField`
--------------
Expand Down
32 changes: 17 additions & 15 deletions src/python/pants/backend/project_info/dependents.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,41 +85,43 @@ def find_dependents(
known_dependents = dependents


class DependeesSubsystem(LineOriented, GoalSubsystem):
name = "dependees"
class DependentsSubsystem(LineOriented, GoalSubsystem):
name = "dependents"
help = "List all targets that depend on any of the input files/targets."
deprecated_options_scope = "dependees"
deprecated_options_scope_removal_version = "2.23.0.dev0"

transitive = BoolOption(
default=False,
help="List all transitive dependees. If unspecified, list direct dependees only.",
help="List all transitive dependents. If unspecified, list direct dependents only.",
)
closed = BoolOption(
default=False,
help="Include the input targets in the output, along with the dependees.",
help="Include the input targets in the output, along with the dependents.",
)


class DependeesGoal(Goal):
subsystem_cls = DependeesSubsystem
class DependentsGoal(Goal):
subsystem_cls = DependentsSubsystem
environment_behavior = Goal.EnvironmentBehavior.LOCAL_ONLY


@goal_rule
async def dependees_goal(
specified_addresses: Addresses, dependees_subsystem: DependeesSubsystem, console: Console
) -> DependeesGoal:
dependees = await Get(
async def dependents_goal(
specified_addresses: Addresses, dependents_subsystem: DependentsSubsystem, console: Console
) -> DependentsGoal:
dependents = await Get(
Dependents,
DependentsRequest(
specified_addresses,
transitive=dependees_subsystem.transitive,
include_roots=dependees_subsystem.closed,
transitive=dependents_subsystem.transitive,
include_roots=dependents_subsystem.closed,
),
)
with dependees_subsystem.line_oriented(console) as print_stdout:
for address in dependees:
with dependents_subsystem.line_oriented(console) as print_stdout:
for address in dependents:
print_stdout(address.spec)
return DependeesGoal(exit_code=0)
return DependentsGoal(exit_code=0)


def rules():
Expand Down
12 changes: 6 additions & 6 deletions src/python/pants/backend/project_info/dependents_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

import pytest

from pants.backend.project_info.dependents import DependeesGoal
from pants.backend.project_info.dependents import rules as dependee_rules
from pants.backend.project_info.dependents import DependentsGoal
from pants.backend.project_info.dependents import rules as dependent_rules
from pants.engine.target import Dependencies, SpecialCasedDependencies, Target
from pants.testutil.rule_runner import RuleRunner

Expand All @@ -26,7 +26,7 @@ class MockTarget(Target):

@pytest.fixture
def rule_runner() -> RuleRunner:
runner = RuleRunner(rules=dependee_rules(), target_types=[MockTarget])
runner = RuleRunner(rules=dependent_rules(), target_types=[MockTarget])
runner.write_files(
{
"base/BUILD": "tgt()",
Expand All @@ -50,7 +50,7 @@ def assert_dependents(
args.append("--transitive")
if closed:
args.append("--closed")
result = rule_runner.run_goal_rule(DependeesGoal, args=[*args, *targets])
result = rule_runner.run_goal_rule(DependentsGoal, args=[*args, *targets])
assert result.stdout.splitlines() == expected


Expand All @@ -62,7 +62,7 @@ def test_normal(rule_runner: RuleRunner) -> None:
assert_dependents(rule_runner, targets=["base"], expected=["intermediate:intermediate"])


def test_no_dependees(rule_runner: RuleRunner) -> None:
def test_no_dependents(rule_runner: RuleRunner) -> None:
assert_dependents(rule_runner, targets=["leaf"], expected=[])


Expand All @@ -80,7 +80,7 @@ def test_transitive(rule_runner: RuleRunner) -> None:


def test_multiple_specified_targets(rule_runner: RuleRunner) -> None:
# This tests that --output-format=text will deduplicate which dependee belongs to which
# This tests that --output-format=text will deduplicate which dependent belongs to which
# specified target.
assert_dependents(
rule_runner,
Expand Down
2 changes: 1 addition & 1 deletion src/python/pants/engine/target.py
Original file line number Diff line number Diff line change
Expand Up @@ -2697,7 +2697,7 @@ class SpecialCasedDependencies(StringSequenceField, AsyncFieldMixin):
dedicated field.
This type will ensure that the dependencies show up in project introspection,
like `dependencies` and `dependees`, but not show up when you call `Get(TransitiveTargets,
like `dependencies` and `dependents`, but not show up when you call `Get(TransitiveTargets,
TransitiveTargetsRequest)` and `Get(Addresses, DependenciesRequest)`.
To hydrate this field's dependencies, use `await Get(Addresses, UnparsedAddressInputs,
Expand Down
1 change: 1 addition & 0 deletions src/python/pants/goal/run_tracker.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ class RunTracker:
"check",
"count-loc",
"dependees",
"dependents",
"dependencies",
"export-codegen",
"filedeps",
Expand Down
Loading

0 comments on commit 3aa9c7b

Please sign in to comment.