Skip to content

Commit

Permalink
Helm deployment chart field (pantsbuild#19234)
Browse files Browse the repository at this point in the history 
Adds a new `chart` field to the `helm_deployment`
  • Loading branch information
@alonsodomin
alonsodomin authored Jun 5, 2023
1 parent 46b9518 commit c7b74fd
Show file tree
Hide file tree
Showing 5 changed files with 144 additions and 34 deletions.
40 changes: 28 additions & 12 deletions docs/markdown/Helm/helm-deployments.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,23 @@ name: example
version: 0.1.0
```
```python src/deployment/BUILD
helm_deployment(name="dev", sources=["common-values.yaml", "dev-override.yaml"], dependencies=["//src/chart"])
helm_deployment(
name="dev",
chart="//src/chart",
sources=["common-values.yaml", "dev-override.yaml"]
)

helm_deployment(name="stage", sources=["common-values.yaml", "stage-override.yaml"], dependencies=["//src/chart"])
helm_deployment(
name="stage",
chart="//src/chart",
sources=["common-values.yaml", "stage-override.yaml"]
)

helm_deployment(name="prod", sources=["common-values.yaml", "prod-override.yaml"], dependencies=["//src/chart"])
helm_deployment(
name="prod",
chart="//src/chart",
sources=["common-values.yaml", "prod-override.yaml"]
)
```
```yaml src/deployment/common-values.yaml
# Default values common to all deployments
Expand All @@ -63,8 +75,8 @@ env:
There are quite a few things to notice in the previous example:
* The `helm_deployment` target requires you to explicitly define as a dependency which chart to use.
* We have three different deployments that using configuration files with the specified chart.
* The `helm_deployment` target requires you to explicitly set the `chart` field to specify which chart to use.
* We have three different deployments using different sets of configuration files with the same chart.
* One of those value files (`common-values.yaml`) provides with default values that are common to all deployments.
* Each deployment uses an additional `xxx-override.yaml` file with values that are specific to the given deployment.

Expand Down Expand Up @@ -112,7 +124,7 @@ spec:
```
```python src/deployment/BUILD
# Overrides the `image` value for the chart using the target address for the first-party docker image.
helm_deployment(dependencies=["src/chart"], values={"image": "src/docker"})
helm_deployment(chart="src/chart", values={"image": "src/docker"})
```
> 📘 Docker image references VS Pants' target addresses
Expand Down Expand Up @@ -140,7 +152,7 @@ Value files
It's very common that Helm deployments use a series of files providing with values that customise the given chart. When using deployments that may have more than one YAML file as the source of configuration values, the Helm backend needs to sort the file names in a way that is consistent across different machines, as the order in which those files are passed to the Helm command is relevant. The final order depends on the same order in which those files are specified in the `sources` field of the `helm_deployment` target. For example, given the following `BUILD` file:

```python src/deployment/BUILD
helm_deployment(name="dev", dependencies=["//src/chart"], sources=["first.yaml", "second.yaml", "last.yaml"])
helm_deployment(name="dev", chart="//src/chart", sources=["first.yaml", "second.yaml", "last.yaml"])
```

This will result in the Helm command receiving the value files as in that exact order.
Expand All @@ -159,7 +171,11 @@ src/deployment/last.yaml
And also the following `helm_deployment` target definition:

```python src/deployment/BUILD
helm_deployment(name="dev", dependencies=["//src/chart"], sources=["first.yaml", "*.yaml", "dev/*-override.yaml", "dev/*.yaml", "last.yaml"])
helm_deployment(
name="dev",
chart="//src/chart",
sources=["first.yaml", "*.yaml", "dev/*-override.yaml", "dev/*.yaml", "last.yaml"]
)
```

In this case, the final ordering of the files would be as follows:
Expand All @@ -185,7 +201,7 @@ Inline values are defined as a key-value dictionary, like in the following examp
```python src/deployment/BUILD
helm_deployment(
name="dev",
dependencies=["//src/chart"],
chart="//src/chart",
values={
"nameOverride": "my_custom_name",
"image.pullPolicy": "Always",
Expand All @@ -200,7 +216,7 @@ Inline values also support interpolation of environment variables. Since Pants r
```python src/deployment/BUILD
helm_deployment(
name="dev",
dependencies=["//src/chart"],
chart="//src/chart",
values={
"configmap.deployedAt": "{env.DEPLOY_TIME}",
},
Expand Down Expand Up @@ -240,7 +256,7 @@ helm_artifact(
```python src/deploy/BUILD
helm_deployment(
name="main",
dependencies=["//3rdparty/helm/jetstack:cert-manager"],
chart="//3rdparty/helm/jetstack:cert-manager",
values={
"installCRDs": "true"
},
Expand Down Expand Up @@ -284,7 +300,7 @@ run_shell_command(
)

helm_deployment(
dependencies=["//src/chart"],
chart="//src/chart",
post_renderers=[":vals"],
)
```
Expand Down
8 changes: 8 additions & 0 deletions src/python/pants/backend/helm/dependency_inference/deployment.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
from pants.engine.addresses import Address
from pants.engine.engine_aware import EngineAwareParameter, EngineAwareReturnType
from pants.engine.fs import Digest, DigestEntries, FileEntry
from pants.engine.internals.native_engine import AddressInput
from pants.engine.rules import Get, MultiGet, collect_rules, rule
from pants.engine.target import (
DependenciesRequest,
Expand Down Expand Up @@ -152,6 +153,11 @@ class InferHelmDeploymentDependenciesRequest(InferDependenciesRequest):
async def inject_deployment_dependencies(
request: InferHelmDeploymentDependenciesRequest,
) -> InferredDependencies:
chart_address = None
chart_address_input = request.field_set.chart.to_address_input()
if chart_address_input:
chart_address = await Get(Address, AddressInput, chart_address_input)

explicitly_provided_deps, mapping = await MultiGet(
Get(ExplicitlyProvidedDependencies, DependenciesRequest(request.field_set.dependencies)),
Get(
Expand All @@ -161,6 +167,8 @@ async def inject_deployment_dependencies(
)

dependencies: OrderedSet[Address] = OrderedSet()
if chart_address:
dependencies.add(chart_address)
for imager_ref, candidate_address in mapping.indexed_docker_addresses.values():
matches = frozenset([candidate_address]).difference(explicitly_provided_deps.includes)
explicitly_provided_deps.maybe_warn_of_ambiguous_dependency_inference(
Expand Down
30 changes: 30 additions & 0 deletions src/python/pants/backend/helm/dependency_inference/deployment_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,36 @@ def test_deployment_dependencies_report(rule_runner: RuleRunner) -> None:
assert set(dependencies_report.all_image_refs) == set(expected_container_refs)


def test_inject_chart_into_deployment_dependencies(rule_runner: RuleRunner) -> None:
rule_runner.write_files(
{
"src/mychart/BUILD": "helm_chart()",
"src/mychart/Chart.yaml": HELM_CHART_FILE,
"src/mychart/values.yaml": HELM_VALUES_FILE,
"src/mychart/templates/_helpers.tpl": HELM_TEMPLATE_HELPERS_FILE,
"src/deployment/BUILD": "helm_deployment(name='foo', chart='//src/mychart')",
}
)

source_root_patterns = ("src/*",)
rule_runner.set_options(
[f"--source-root-patterns={repr(source_root_patterns)}"],
env_inherit=PYTHON_BOOTSTRAP_ENV,
)

deployment_addr = Address("src/deployment", target_name="foo")
tgt = rule_runner.get_target(deployment_addr)
field_set = HelmDeploymentFieldSet.create(tgt)

inferred_dependencies = rule_runner.request(
InferredDependencies,
[InferHelmDeploymentDependenciesRequest(field_set)],
)

assert len(inferred_dependencies.include) == 1
assert list(inferred_dependencies.include)[0] == Address("src/mychart")


def test_inject_deployment_dependencies(rule_runner: RuleRunner) -> None:
rule_runner.write_files(
{
Expand Down
40 changes: 39 additions & 1 deletion src/python/pants/backend/helm/target_types.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@
from dataclasses import dataclass

from pants.backend.helm.resolve.remotes import ALL_DEFAULT_HELM_REGISTRIES
from pants.base.deprecated import warn_or_error
from pants.core.goals.package import OutputPathField
from pants.core.goals.test import TestTimeoutField
from pants.engine.internals.native_engine import AddressInput
from pants.engine.rules import collect_rules, rule
from pants.engine.target import (
COMMON_TARGET_FIELDS,
AllTargets,
AsyncFieldMixin,
BoolField,
Dependencies,
DescriptionField,
Expand All @@ -34,7 +37,7 @@
generate_multiple_sources_field_help_message,
)
from pants.util.docutil import bin_name
from pants.util.strutil import help_text
from pants.util.strutil import help_text, softwrap
from pants.util.value_interpolation import InterpolationContext, InterpolationError

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -385,6 +388,39 @@ def all_helm_artifact_targets(all_targets: AllTargets) -> AllHelmArtifactTargets
# -----------------------------------------------------------------------------------------------


class HelmDeploymentChartField(StringField, AsyncFieldMixin):
alias = "chart"
# TODO Will be made required in next release
required = False
help = help_text(
f"""
The address of the `{HelmChartTarget.alias}` or `{HelmArtifactTarget.alias}`
that will be used for this deployment.
"""
)

def to_address_input(self) -> AddressInput | None:
if self.value:
return AddressInput.parse(
self.value,
relative_to=self.address.spec_path,
description_of_origin=f"the `{self.alias}` field in the `{HelmDeploymentTarget.alias}` target {self.address}",
)

warn_or_error(
"2.19.0.dev0",
"chart address in `dependencies`",
softwrap(
f"""
You should specify the chart address in the new `{self.alias}` field in
{HelmDeploymentTarget.alias}. In future versions this will be mandatory.
"""
),
start_version="2.18.0.dev1",
)
return None


class HelmDeploymentReleaseNameField(StringField):
alias = "release_name"
help = "Name of the release used in the deployment. If not set, the target name will be used instead."
Expand Down Expand Up @@ -489,6 +525,7 @@ class HelmDeploymentTarget(Target):
alias = "helm_deployment"
core_fields = (
*COMMON_TARGET_FIELDS,
HelmDeploymentChartField,
HelmDeploymentReleaseNameField,
HelmDeploymentDependenciesField,
HelmDeploymentSourcesField,
Expand All @@ -511,6 +548,7 @@ class HelmDeploymentFieldSet(FieldSet):
HelmDeploymentSourcesField,
)

chart: HelmDeploymentChartField
description: DescriptionField
release_name: HelmDeploymentReleaseNameField
namespace: HelmDeploymentNamespaceField
Expand Down
60 changes: 39 additions & 21 deletions src/python/pants/backend/helm/util_rules/chart.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,16 @@
PathGlobs,
Snapshot,
)
from pants.engine.internals.native_engine import AddressInput
from pants.engine.rules import Get, MultiGet, collect_rules, rule
from pants.engine.target import DependenciesRequest, ExplicitlyProvidedDependencies, Target, Targets
from pants.engine.target import (
DependenciesRequest,
ExplicitlyProvidedDependencies,
Target,
Targets,
WrappedTarget,
WrappedTargetRequest,
)
from pants.util.frozendict import FrozenDict
from pants.util.logging import LogLevel
from pants.util.ordered_set import OrderedSet
Expand Down Expand Up @@ -136,13 +144,13 @@ async def _merge_subchart_digests(charts: Iterable[HelmChart]) -> Digest:
return await Get(Digest, AddPrefix(merged_digests, "charts"))


async def _get_subcharts(
subtargets: Targets, *, description_of_origin: str
async def _find_charts_by_targets(
targets: Iterable[Target], *, description_of_origin: str
) -> tuple[HelmChart, ...]:
requests = [
*(
Get(HelmChart, HelmChartRequest, HelmChartRequest.from_target(target))
for target in subtargets
for target in targets
if HelmChartFieldSet.is_applicable(target)
),
*(
Expand All @@ -154,7 +162,7 @@ async def _get_subcharts(
description_of_origin=description_of_origin,
),
)
for target in subtargets
for target in targets
if HelmArtifactFieldSet.is_applicable(target)
),
]
Expand All @@ -179,7 +187,7 @@ async def get_helm_chart(request: HelmChartRequest, subsystem: HelmSubsystem) ->
)

subcharts_digest = EMPTY_DIGEST
subcharts = await _get_subcharts(
subcharts = await _find_charts_by_targets(
dependencies, description_of_origin=f"the `helm_chart` {request.field_set.address}"
)
if subcharts:
Expand Down Expand Up @@ -287,22 +295,32 @@ def debug_hint(self) -> str | None:

@rule(desc="Find Helm deployment's chart", level=LogLevel.DEBUG)
async def find_chart_for_deployment(request: FindHelmDeploymentChart) -> HelmChart:
explicit_dependencies = await Get(
ExplicitlyProvidedDependencies, DependenciesRequest(request.field_set.dependencies)
)
explicit_targets = await Get(
Targets,
Addresses(
[
addr
for addr in explicit_dependencies.includes
if addr not in explicit_dependencies.ignores
]
),
)
targets = []
address_input = request.field_set.chart.to_address_input()
if address_input:
address = await Get(Address, AddressInput, address_input)
wrapped_target = await Get(
WrappedTarget, WrappedTargetRequest(address, address_input.description_of_origin)
)
targets.append(wrapped_target.target)
else:
explicit_dependencies = await Get(
ExplicitlyProvidedDependencies, DependenciesRequest(request.field_set.dependencies)
)
explicit_targets = await Get(
Targets,
Addresses(
[
addr
for addr in explicit_dependencies.includes
if addr not in explicit_dependencies.ignores
]
),
)
targets.extend(explicit_targets)

found_charts = await _get_subcharts(
explicit_targets, description_of_origin=f"the `helm_deployment` {request.field_set.address}"
found_charts = await _find_charts_by_targets(
targets, description_of_origin=f"the `helm_deployment` {request.field_set.address}"
)

if not found_charts:
Expand Down

0 comments on commit c7b74fd

Please sign in to comment.