diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-and-target-api.md b/docs/markdown/Writing Plugins/rules-api/rules-api-and-target-api.md index bda28da79d3..5ff2acd2919 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-and-target-api.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-and-target-api.md @@ -53,7 +53,7 @@ if target.has_fields([PythonSourceField, PythonTestsTimeoutField]): As explained in [Concepts](doc:target-api-concepts), subclassing `Field`s is key to how the Target API works. -The `Target` methods `[MyField]`, `.has_field()` and `.get()` understand when a `Field` is subclassesd, as follows: +The `Target` methods `[MyField]`, `.has_field()` and `.get()` understand when a `Field` is subclassed, as follows: ```python >>> docker_tgt.has_field(DockerSourceField) @@ -70,7 +70,7 @@ This allows you to express specifically which types of `Field`s you need to work ### A Target's `Address` -Every target is identifed by its `Address`, from `pants.engine.adddresses`. Many types used in the Plugin API will use `Address` objects as fields, and it's also often useful to use the `Address` when writing the description for a `Process` you run. +Every target is identified by its `Address`, from `pants.engine.addresses`. Many types used in the Plugin API will use `Address` objects as fields, and it's also often useful to use the `Address` when writing the description for a `Process` you run. A `Target` has a field `address: Address`, e.g. `my_tgt.address`. @@ -79,9 +79,9 @@ You can also create an `Address` object directly, which is often useful in tests - `project:tgt` -> `Address("project", target_name="tgt")` - `project/` -> `Address("project")` - `//:top-level` -> `Address("", target_name="top_level")` -- `project/app.py:tgt` -> \`Address("project", target_name="tgt", relative_file_name="app.py") +- `project/app.py:tgt` -> `Address("project", target_name="tgt", relative_file_name="app.py")` - `project:tgt#generated` -> `Address("project", target_name="tgt", generated_name="generated")` -- `project:tgt@shell=zsh`: `Address("project", target_name="tgt", parameters={"shell": "zsh"})` +- `project:tgt@shell=zsh` -> `Address("project", target_name="tgt", parameters={"shell": "zsh"})` You can use `str(address)` or `address.spec` to get the normalized string representation. `address.spec_path` will give the path to the parent directory of the target's original BUILD file. @@ -90,7 +90,7 @@ How to resolve targets How do you get `Target`s in the first place in your plugin? -As explained in [Goal rules](doc:rules-api-goal-rules), to get all of the targets specified on the command line by a user, you can request the type `Targets` as a parameter to your `@rule` or `@goal_rule`. From there, you can optionally filter out the targets you want, such as by using `target.has_field()`. +As explained in [Goal rules](doc:rules-api-goal-rules), to get all the targets specified on the command line by a user, you can request the type `Targets` as a parameter to your `@rule` or `@goal_rule`. From there, you can optionally filter out the targets you want, such as by using `target.has_field()`. ```python from pants.engine.target import Targets @@ -192,7 +192,7 @@ from pants.engine.rules import Get, rule @rule async def demo(...) -> Foo: ... - direct_deps = await Get(Targets, DependenciesRequest(target.get(Dependencies)) + direct_deps = await Get(Targets, DependenciesRequest(target.get(Dependencies))) ``` `DependenciesRequest` takes a single argument: `field: Dependencies`. The return type `Targets` is a `Collection` of individual `Target` objects corresponding to each direct dependency of the original target. @@ -210,7 +210,7 @@ from pants.engine.rules import Get, rule @rule async def demo(...) -> Foo: ... - transitive_targets = await Get(TransitiveTargets, TransitiveTargetsRequest([target.address]) + transitive_targets = await Get(TransitiveTargets, TransitiveTargetsRequest([target.address])) ``` `TransitiveTargetsRequest` takes an iterable of `Address`es. @@ -272,7 +272,7 @@ from pants.engine.rules import Get, rule @rule async def demo(...) -> Foo: ... - sources = await Get(HydratedSources, HydrateSourcesRequest(target[SourcesField]) + sources = await Get(HydratedSources, HydrateSourcesRequest(target[SourcesField])) ``` `HydrateSourcesRequest` expects a `SourcesField` object. This can be a subclass, such as `PythonSourceField` or `GoPackageSourcesField`. @@ -317,7 +317,7 @@ async def demo(...) -> Foo: ) ``` -If the provided `SourcesField` object is already a subclass of one of the `for_sources_types`—or it can be generated into one of those types—then the sources will be hydrated; otherwise, you'll get back a `HydratedSources` object with an empty snapshot and the field `sources_type=None`. +If the provided `SourcesField` object is already a subclass of one of the `for_sources_types`—or it can be generated into one of those types—then the sources will be hydrated; otherwise, you'll get back a `HydratedSources` object with an empty snapshot and the field `sources_type=None`. `SourceFilesRequest` also accepts the `enable_codegen` and `for_source_types` arguments. This will filter out any inputted `Sources` field that are not compatible with `for_sources_type`. @@ -373,7 +373,7 @@ Normally, your rule should simply use `tgt.get()` and `tgt.has_field()` instead To create a `FieldSet`, create a new dataclass with `@dataclass(frozen=True)`. You will sometimes directly subclass `FieldSet`, but will often subclass something like `BinaryFieldSet` or `TestFieldSet`. Refer to the instructions in [Common plugin tasks](doc:common-plugin-tasks). -List every `Field` that your plugin will use as a field of your dataclass. The types hints you specify will be used by Pants to identify what `Field`s to use, e.g. `PythonSourceField` or `Dependencies`. +List every `Field` that your plugin will use as a field of your dataclass. The type hints you specify will be used by Pants to identify what `Field`s to use, e.g. `PythonSourceField` or `Dependencies`. Finally, set the class property `required_fields` as a tuple of the `Field`s that your plugin requires. Pants will use this to filter out irrelevant targets that your plugin does not know how to operate on. Often, this will be the same as the `Field`s that you listed as dataclass fields, but it does not need to be. If a target type does not have registered one of the `Field`s that are in the dataclass fields, and it isn't a required `Field`, then Pants will use a default value as if the user left it off from their BUILD file. diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-concepts.md b/docs/markdown/Writing Plugins/rules-api/rules-api-concepts.md index 38dccc7d5e8..908143313d6 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-concepts.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-concepts.md @@ -36,12 +36,12 @@ async def run_shellcheck(target: Target, shellcheck: Shellcheck) -> LintResult: You do not call a rule like you would a normal function. In the above examples, you would not say `int_to_str(26)` or `run_shellcheck(tgt, shellcheck)`. Instead, the Pants engine determines when rules are used and calls the rules for you. -Each rule should be pure; you should not use side-effects like `subprocess.run()`, `print()`, or the `requests` library. Instead, the Rules API has its own alternatives that are understood by the Pants engine and which work properly with its caching and parallelism. +Each rule should be pure; you should not use side effects like `subprocess.run()`, `print()`, or the `requests` library. Instead, the Rules API has its own alternatives that are understood by the Pants engine and which work properly with its caching and parallelism. The rule graph -------------- -All of the registered rules create a rule graph, with each type as a node and the edges being dependencies used to compute those types. +All the registered rules create a rule graph, with each type as a node and the edges being dependencies used to compute those types. For example, the `list` goal uses this rule definition and results in the below graph: @@ -140,7 +140,7 @@ async def call_fibonacci(...) -> Foo: > > Currently, you can only give a single input. It is not possible to do something like `Get(OutputType, InputType1(...), InputType2(...))`. > -> Instead, it's common for rules to create a "Request" data class, such as `PexRequest` or `SourceFilesRequest`. This request centralizes all of the data it needs to operate into one data structure, which allows for call sites to say `await Get(SourceFiles, SourceFilesRequest, my_request)`, for example. +> Instead, it's common for rules to create a "Request" data class, such as `PexRequest` or `SourceFilesRequest`. This request centralizes all the data it needs to operate into one data structure, which allows for call sites to say `await Get(SourceFiles, SourceFilesRequest, my_request)`, for example. > > See for the tracking issue. diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-file-system.md b/docs/markdown/Writing Plugins/rules-api/rules-api-file-system.md index d0fa8ea4025..26f39c3eb29 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-file-system.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-file-system.md @@ -90,11 +90,11 @@ from pants.engine.rules import Get, rule @rule async def demo(...) -> Foo: ... - digest = await Get(Digest, PathGlobs(["**/*.txt", "!ignore_me.txt"]) + digest = await Get(Digest, PathGlobs(["**/*.txt", "!ignore_me.txt"])) ``` - All globs must be relative paths, relative to the build root. -- `PathGlobs` uses the same syntax as the `sources` field, which is roughly Git's syntax. Use `*` for globs over just the current working directory, `**` for recursive globs over everything below (at any level the current working directory, and prefix with `!` for ignores. +- `PathGlobs` uses the same syntax as the `sources` field, which is roughly Git's syntax. Use `*` for globs over just the current working directory, `**` for recursive globs over everything below (at any level the current working directory), and prefix with `!` for ignores. - `PathGlobs` will ignore all values from the global option `pants_ignore`. By default, the engine will no-op for any globs that are unmatched. If you want to instead warn or error, set `glob_match_error_behavior=GlobMatchErrorBehavior.warn` or `GlobMatchErrorBehavior.error`. This will require that you also set `description_of_origin`, which is a human-friendly description of where the `PathGlobs` is coming from so that the error message is helpful. For example: @@ -131,7 +131,7 @@ from pants.engine.rules import Get, rule @rule async def demo(...) -> Foo: ... - paths = await Get(Paths, PathGlobs(["**/*.txt", "!ignore_me.txt"]) + paths = await Get(Paths, PathGlobs(["**/*.txt", "!ignore_me.txt"])) logger.info(paths.files) ``` @@ -211,7 +211,7 @@ async def demo(...) -> Foo: - It is okay if multiple digests include the same file, so long as they have identical content. - If any digests have different content for the same file, the engine will error. Unlike Git, the engine does not attempt to resolve merge conflicts. -- It is okay if some of the digests are empty, i.e. `EMPTY_DIGEST`. +- It is okay if some digests are empty, i.e. `EMPTY_DIGEST`. `DigestSubset`: extract certain files from a `Digest` ----------------------------------------------------- diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-goal-rules.md b/docs/markdown/Writing Plugins/rules-api/rules-api-goal-rules.md index 982274b2e02..6434eda31d9 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-goal-rules.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-goal-rules.md @@ -8,7 +8,7 @@ updatedAt: "2022-07-25T14:45:07.539Z" --- For many [plugin tasks](doc:common-plugin-tasks), you will be extending existing goals, such as adding a new linter to the `lint` goal. However, you may instead want to create a new goal, such as a `publish` goal. This page explains how to create a new goal. -As explained in [Concepts](doc:rules-api-concepts), `@goal_rule`s are the entry-point into the rule graph. When a user runs `./pants my-goal`, the Pants engine will look for the respective `@goal_rule`. That `@goal_rule` will usually request other types, either as parameters in the `@goal_rule` signature or through `await Get`. But unlike a `@rule`, a `@goal_rule` may also trigger side-effects (such as running interactive processes, writing to the filesystem, etc) via `await Effect`. +As explained in [Concepts](doc:rules-api-concepts), `@goal_rule`s are the entry-point into the rule graph. When a user runs `./pants my-goal`, the Pants engine will look for the respective `@goal_rule`. That `@goal_rule` will usually request other types, either as parameters in the `@goal_rule` signature or through `await Get`. But unlike a `@rule`, a `@goal_rule` may also trigger side effects (such as running interactive processes, writing to the filesystem, etc) via `await Effect`. Often, you can keep all of your logic inline in the `@goal_rule`. As your `@goal_rule` gets more complex, you may end up factoring out helper `@rule`s, but you do not need to start with writing helper `@rule`s. @@ -91,7 +91,7 @@ If your goal's purpose is to emit output, it may be helpful to use the mixin `Ou from pants.engine.goal import Goal, GoalSubsystem, Outputting from pants.engine.rules import goal_rule -class HelloWorldSubsystem(Outputting, GoalSubystem): +class HelloWorldSubsystem(Outputting, GoalSubsystem): name = "hello-world" help = "An example goal." @@ -108,13 +108,13 @@ async def hello_world( ### `LineOriented` mixin (optional) -If your goal's purpose is to emit output—and that output is naturally split by new lines—it may be helpful to use the mixin `LineOriented`. This subclasses `Outputting`, so will register both the options `--output-file` and `--sep`, which allows the user to change the separator to not be `\n`. +If your goal's purpose is to emit output -- and that output is naturally split by new lines -- it may be helpful to use the mixin `LineOriented`. This subclasses `Outputting`, so will register both the options `--output-file` and `--sep`, which allows the user to change the separator to not be `\n`. ```python from pants.engine.goal import Goal, GoalSubsystem, LineOriented from pants.engine.rules import goal_rule -class HelloWorldSubsystem(LineOriented, GoalSubystem): +class HelloWorldSubsystem(LineOriented, GoalSubsystem): name = "hello-world" help = "An example goal.""" @@ -168,9 +168,9 @@ See [Rules and the Target API](doc:rules-api-and-target-api) for detailed infor > > This will not work because the engine has no path in the rule graph to resolve a `PythonDistribution` type given the initial input types to the rule graph (the "roots"). > -> Instead, request `Targets`, which will give you all of the targets that the user specified on the command line. The engine knows how to resolve this type because it can go from `Specs` -> `Addresses` -> `Targets`. +> Instead, request `Targets`, which will give you all the targets that the user specified on the command line. The engine knows how to resolve this type because it can go from `Specs` -> `Addresses` -> `Targets`. > -> From here, filter out the relevant targets you want using the Target API (see [Rules and the Target API](doc:rules-api-and-target-api). +> From here, filter out the relevant targets you want using the Target API (see [Rules and the Target API](doc:rules-api-and-target-api)). > > ```python > from pants.engine.target import Targets @@ -200,4 +200,4 @@ async def hello_world(console: Console, specs_paths: SpecsPaths) -> HelloWorld: `SpecsPaths.files` will list all files matched by the specs, e.g. `::` will match every file in the project (regardless of if targets own the files). -To convert `SpecsPaths` into a [`Digest`](doc:rules-api-file-system), use `await Get(Digest, PathGlobs(globs=specs_paths.files))`. \ No newline at end of file +To convert `SpecsPaths` into a [`Digest`](doc:rules-api-file-system), use `await Get(Digest, PathGlobs(globs=specs_paths.files))`. diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-installing-tools.md b/docs/markdown/Writing Plugins/rules-api/rules-api-installing-tools.md index 9a9df406c4b..1f7a7acbbf4 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-installing-tools.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-installing-tools.md @@ -35,7 +35,7 @@ async def demo(...) -> Foo: docker_bin = docker_paths.first_path if docker_bin is None: raise OSError("Could not find 'docker'.") - result = await Get(ProcessResult, Process(argv=[docker_bin.path, ...], ...) + result = await Get(ProcessResult, Process(argv=[docker_bin.path, ...], ...)) ``` `BinaryPaths` has a field called `paths: tuple[BinaryPath, ...]`, which stores all the discovered absolute paths to the specified binary. Each `BinaryPath` object has the fields `path: str`, such as `/usr/bin/docker`, and `fingerprint: str`, which is used to invalidate the cache if the binary changes. The results will be ordered by the order of `search_path`, meaning that earlier entries in `search_path` will show up earlier in the result. diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-process.md b/docs/markdown/Writing Plugins/rules-api/rules-api-process.md index d8183ab991d..e5ca29c69fa 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-process.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-process.md @@ -61,7 +61,7 @@ from pants.engine.rules import Get, rule @rule async def partial_env(...) -> Foo: relevant_env_vars = await Get(EnvironmentVars, EnvironmentVarsRequest(["RELEVANT_VAR", "PATH"])) - .. + ... ``` ### Output Files @@ -78,7 +78,7 @@ To use a timeout, set the `timeout_seconds: int` field. Otherwise, the process w > > By default, a `Process` will be cached to `~/.cache/pants/lmdb_store` if the `exit_code` is `0`. > -> If it not safe to cache your `Process`—usually the case when you know that a process accesses files outside of its sandbox—you can change the cacheability of your `Process` using the `ProcessCacheScope` parameter: +> If it's not safe to cache your `Process` -- usually the case when you know that a process accesses files outside its sandbox -- you can change the cacheability of your `Process` using the `ProcessCacheScope` parameter: > > ```python > from pants.engine.process import Process, ProcessCacheScope, ProcessResult @@ -90,7 +90,7 @@ To use a timeout, set the `timeout_seconds: int` field. Otherwise, the process w > description="Not persisted between Pants runs ('sessions').", > cache_scope=ProcessCacheScope.PER_SESSION, > ) -> .. +> ... > ``` > > `ProcessCacheScope` supports other options as well, including `ALWAYS`. diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-testing.md b/docs/markdown/Writing Plugins/rules-api/rules-api-testing.md index c083b3e1fed..d39363a2ee0 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-testing.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-testing.md @@ -126,7 +126,7 @@ async def find_needle_in_haystack(find_needle: FindNeedle) -> TargetsWithNeedle: We can write this test: ```python -from pants.engine.addresses import address +from pants.engine.addresses import Address from pants.engine.fs import EMPTY_DIGEST, Snapshot from pants.engine.target import HydratedSources, HydrateSourcesRequest, Target, Sources from pants.testutil.rule_runner import MockGet, run_rule_with_mocks @@ -206,7 +206,7 @@ After setting up your isolated environment, you can run `rule_runner.request(Out ### Setting up the `RuleRunner` -First, you must set up a `RuleRunner` instance and activate the rules and target types you'll use in your tests. Set the argument `target_types` with a list of the `Target` types used in in your tests, and set `rules` with a list of all the rules used transitively. +First, you must set up a `RuleRunner` instance and activate the rules and target types you'll use in your tests. Set the argument `target_types` with a list of the `Target` types used in your tests, and set `rules` with a list of all the rules used transitively. This means that you must register the rules you directly wrote, and also any rules that they depend on. Pants will automatically register some core rules for you, but leaves off most of them for better isolation of tests. If you're missing some rules, the rule graph will fail to be built. @@ -289,7 +289,7 @@ def second_rule_runner() -> RuleRunner: def test_example3(second_rule_runner: RuleRunner) -> None: - second_rule_runner.write_files(..) + second_rule_runner.write_files(...) ... ``` @@ -492,7 +492,7 @@ def test_one_target_one_source(rule_runner: RuleRunner) -> None: } ) result = rule_runner.run_goal_rule(Filedeps, args=["project/example.ext"]) - assert result.stdout.splitlines() = ["project/BUILD", "project/example.ext"] + assert result.stdout.splitlines() == ["project/BUILD", "project/example.ext"] ``` Unlike when testing normal `@rules`, you do not need to define a `QueryRule` when using `rule_runner.run_goal_rule()`. This is already set up for you. However, you do need to make sure that your `@goal_rule` and all the rules it depends on are registered with the `RuleRunner` instance. @@ -543,7 +543,7 @@ To read any files that were created, use `get_buildroot()` as the first part of ```python from pathlib import Path -from pants.base.build_environment import get_buildroot() +from pants.base.build_environment import get_buildroot from pants.testutil.pants_integration_test import run_pants, setup_tmpdir def test_junit_report() -> None: diff --git a/docs/markdown/Writing Plugins/rules-api/rules-api-tips.md b/docs/markdown/Writing Plugins/rules-api/rules-api-tips.md index c9b1e49c1f8..c0b454213b1 100644 --- a/docs/markdown/Writing Plugins/rules-api/rules-api-tips.md +++ b/docs/markdown/Writing Plugins/rules-api/rules-api-tips.md @@ -19,9 +19,9 @@ Every time your rule has `await`, Python will yield execution to the engine and Okay: ```python -from pants.core.util_rules.determine_source_files import SourceFilesRequest, SourceFiles +from pants.core.util_rules.source_files import SourceFilesRequest, SourceFiles from pants.engine.fs import AddPrefix, Digest -from pants.engine.selectors import Get, MultiGet +from pants.engine.internals.selectors import Get @rule async def demo(...) -> Foo: @@ -32,9 +32,9 @@ async def demo(...) -> Foo: Better: ```python -from pants.core.util_rules.determine_source_files import SourceFilesRequest, SourceFiles +from pants.core.util_rules.source_files import SourceFilesRequest, SourceFiles from pants.engine.fs import AddPrefix, Digest -from pants.engine.selectors import Get, MultiGet +from pants.engine.internals.selectors import Get, MultiGet @rule async def demo(...) -> Foo: @@ -136,7 +136,7 @@ if __name__ == "__main__": main() ``` -Once you've identified the smallest combination of backends that fail and you have updated `pants.toml`, you can try isolating which rules are problematic by commenting out `Get`s and the parameters to `@rule`s. +Once you've identified the smallest combination of backends that fail, and you have updated `pants.toml`, you can try isolating which rules are problematic by commenting out `Get`s and the parameters to `@rule`s. Some common sources of rule graph failures: