Skip to content

Commit

Permalink
fix: Fix the rendering flow of env, vars and args (osl-incubator#41)
Browse files Browse the repository at this point in the history
* add template doc

* fix: Fix the rendering flow of env, vars and args

* fix some tests

* remove obsolete poetry config

* remove obsolete poetry config
  • Loading branch information
xmnlab authored May 21, 2023
1 parent 1b642d9 commit 3235d98
Show file tree
Hide file tree
Showing 9 changed files with 291 additions and 92 deletions.
5 changes: 3 additions & 2 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ jobs:
miniforge-variant: Mambaforge

- name: Install dependencies
run: poetry install
run: |
poetry config virtualenvs.create false
poetry install
- name: Run smoke tests using simple makim file
run: makim smoke-tests.simple
Expand Down Expand Up @@ -121,7 +123,6 @@ jobs:
- name: Install dependencies
run: |
poetry config virtualenvs.create false
poetry config experimental.new-installer true
poetry install
- name: Test documentation generation
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ jobs:
- name: Install deps
run: |
poetry config virtualenvs.create false
poetry config experimental.new-installer true
poetry install
- name: Run semantic release (for tests)
Expand Down
10 changes: 5 additions & 5 deletions .makim.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ groups:
action: store_true
env:
MAKIM_FILE: ./tests/.makim-simple.yaml
VERBOSE_FLAG: '{{ "--verbose" if args.verbose_mode else "" }}'
shell: bash
run: |
export VERBOSE_FLAG='{{ "--verbose" if args.verbose_mode else "" }}'
makim --makim-file $MAKIM_FILE --help
makim --makim-file $MAKIM_FILE --version
makim --makim-file $MAKIM_FILE clean $VERBOSE_FLAG
Expand All @@ -91,14 +91,14 @@ groups:
help: Test makim using a complex makimfile
env:
MAKIM_FILE: ./tests/.makim-complex.yaml
VERBOSE_FLAG: '{{ "--verbose" if args.verbose_mode else "" }}'
args:
verbose-mode:
help: Run the all the tests in verbose mode
type: bool
action: store_true
shell: bash
run: |
export VERBOSE_FLAG='{{ "--verbose" if args.verbose_mode else "" }}'
makim --makim-file $MAKIM_FILE --help
makim --makim-file $MAKIM_FILE --version
makim --makim-file $MAKIM_FILE default.lint --verbose
Expand Down Expand Up @@ -135,14 +135,14 @@ groups:
help: Test makim with containers-sugar
env:
MAKIM_FILE: .makim-complex.yaml
VERBOSE_FLAG: '{{ "--verbose" if args.verbose_mode else "" }}'
args:
verbose-mode:
help: Run the all the tests in verbose mode
type: bool
action: store_true
shell: bash
run: |
export VERBOSE_FLAG='{{ "--verbose" if args.verbose_mode else "" }}'
cd ./tests/
makim --makim-file $MAKIM_FILE containers.run --verbose
Expand All @@ -155,9 +155,9 @@ groups:
action: store_true
env:
MAKIM_FILE: ./tests/.makim-unittest.yaml
VERBOSE_FLAG: '{{ "--verbose" if args.verbose_mode else "" }}'
shell: bash
run: |
export VERBOSE_FLAG='{{ "--verbose" if args.verbose_mode else "" }}'
makim --makim-file $MAKIM_FILE --help
makim --makim-file $MAKIM_FILE --version
makim --makim-file $MAKIM_FILE tests.test-1 $VERBOSE_FLAG
Expand All @@ -178,9 +178,9 @@ groups:
action: store_true
env:
MAKIM_FILE: ./tests/.makim-env.yaml
VERBOSE_FLAG: '{{ "--verbose" if args.verbose_mode else "" }}'
shell: bash
run: |
export VERBOSE_FLAG='{{ "--verbose" if args.verbose_mode else "" }}'
makim --makim-file $MAKIM_FILE --help
makim --makim-file $MAKIM_FILE --version
makim --makim-file $MAKIM_FILE global-scope.test-var-env-file $VERBOSE_FLAG
Expand Down
6 changes: 1 addition & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,13 @@ groups:
help: if not set, the clean dependency will not be triggered.
dependencies:
- target: clean
if: \${\{ args.clean == true \}\}
if: {% raw %}{{ args.clean == true }}{% endraw %}
run: |
echo "build file x"
echo "build file y"
echo "build file z"
```
PS: in `\${\{ args.clean == true \}\}`, this `\` is just for escaping it for
the documentation here, you should use it with no escape `\`.

Some examples of how to use it:
* run the `build` target:
Expand Down
2 changes: 1 addition & 1 deletion conda/dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ channels:
- conda-forge
dependencies:
- python 3.8.1 # min version supported
- poetry
- poetry >=1.3
- nodejs # used by semantic-release
# https://github.com/ionrock/cachecontrol/issues/292#issuecomment-1536120527
- urllib3 <2
130 changes: 130 additions & 0 deletions docs/template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Makim Template

**Makim** config files are powered by the Jinja2 template engine, allowing
you to use Jinja2 tags for added flexibility in your Makim config files.

This page focuses on explaining the options offered directly by **Makim**,
rather than covering all the possibilities with Jinja2.

**Makim** provides three different variable options that can be combined:
`vars`, `env`, and `args`.

Additionally, the `env` and `vars` option has three different scopes:
**global**, **group**, and **target**.
We will discuss each of them in the following sections.

## Variables Scopes

Before delving into the different variable options, let's discuss `env`
and `vars` scope, as it is essential for the subsequent sections.
The `args` attribute just works in the *target* scope.

As mentioned earlier, **Makim** `env` and `vars` has three scopes:
**global**, **group**, and **target**.

The order of their rendering is crucial. First, the *global* scope is processed.
In the *group* scope, any variable defined globally is accessible via the `env`
variable (e.g., `{% raw %}{{ env.my_global_env }}{% endraw %}`).
However, any variable defined in the *global* scope will be overridden by a
variable with the same name in the *group* scope. The same applies to the
*target* scope, where any variable defined in the *global* or *group* scope
will be overridden by a variable defined in the *target* scope.

Moreover, `env` is a bit more complex, as its value can be defined in two
different ways: either through the `env` attribute in the `.makim.yaml`
file or from an environment file specified in the `env-file` attribute.
First, the `env-file` is loaded into memory, and then the variables
defined in the `env` attribute are loaded. In other words, any variable
defined in the file for the given `env-file` will be overridden by a
variable of the same name defined in the `env` attribute. This process
also respects the order of scopes.

PS: **Makim** utilizes system environment variables as the initial scope for
the variables.

## Different Variable Options

**Makim** offers three variable options within the `makim` config file:
`env`, `vars`, and `args`.

* `args` allows users to pass parameters via the CLI (command line interface).
It can also be used for target dependencies when parameters need to be passed
to the dependency. However, this option is not available in the system context
(the commands executed defined by `run` attribute), it is only accessible
within the Makim config file.
* `vars` is a convenient way to define reusable variables in the code. For
example, if you frequently use a command in the `run` section, you can define
a variable inside `vars` to make the Makim file more readable. Like `args`,
this option is not available in the system context; it is only accessible
within the Makim config file.
* `env` is used to define environment variables. Any environment variable can
be accessed via the `env` variable in the template (e.g.,
`{% raw %}{{ env.myenvvar }}{% endraw %}`) or directly as an environment
variable within the `run` section, as shown in the example below:

```yaml
...
groups:
group1:
targets:
target1:
...
env:
MYVAR: 1
run: |
echo $MYENV
```
## Order of Variable Rendering
One crucial point to keep in mind is the order of variable rendering
within the Makim config file.
`vars` is primarily used for `run` section, so they have the lowest
precedence. In another word, you can use `env` or `arg` to create
`vars`, but not the opposite way.

`env` however, can be defined also in the system scope, so it has the
highest rank in the precedence. So, you shouldn't define a `env` that
depends on a variable defined by `vars` or `args`. If you need to set
your environment variable with a value from a `vars` or `args`, you
should do it in the `run` section.

In the following example, it shows a correct way to use all the
different options of variables, respecting the scopes and rendering order:

```yaml
...
env:
MY_GLOBAL_ENV: 1
vars:
MY_GLOBAL_VAR: "my global env is {% raw %}{{ env.MY_GLOBAL_ENV }}{% endraw %}"
groups:
group1:
help: "group"
env:
MY_GROUP_ENV: 2
vars:
MY_GROUP_VAR: "my group env is {% raw %}{{ env.MY_GROUP_ENV }}{% endraw %}"
targets:
target1:
help: "target 1"
env:
MY_TARGET_ENV: 3
args:
my-target-arg:
help: "target arg"
type: string
default: "{% raw %}{{ env.MY_TARGET_ENV }}{% endraw %}"
vars:
MY_TARGET_VAR: "my group env is {% raw %}{{ env.MY_GROUP_ENV }}{% endraw %}"
run: |
echo "{% raw %}{{ env.MY_GLOBAL_ENV}}{% endraw %}"
echo "{% raw %}{{ env.MY_GLOBAL_VAR}}{% endraw %}"
echo "{% raw %}{{ env.MY_GROUP_ENV}}{% endraw %}"
echo "{% raw %}{{ env.MY_GROUP_VAR}}{% endraw %}"
echo "{% raw %}{{ env.MY_TARGET_ENV}}{% endraw %}"
echo "{% raw %}{{ env.MY_TARGET_VAR}}{% endraw %}"
echo "{% raw %}{{ env.my_target_arg}}{% endraw %}"
```
Loading

0 comments on commit 3235d98

Please sign in to comment.