Unable to apply multiple modules at once.

[RyanW8]

Describe the bug

Our terragrunt file structure is the following, top level is the application name, 2nd level is account & 3rd is the region. E.g.
Top Level = example_app
2nd Level = example_account
3rd Level = example_region
Would look like

example_app/
  project.hcl
  terragrunt.hcl
  example_account/
    account.hcl
    example_region/
      region.hcl
      terragrunt.hcl

We define what the application terraform should look like at the application/top level like so

terraform {
 ...
}
locals {
  project_locals = read_terragrunt_config(find_in_parent_folders("project.hcl")).locals
  account_locals = read_terragrunt_config(find_in_parent_folders("account.hcl")).locals
  region_locals  = read_terragrunt_config("region.hcl").locals
}
...

child terragrunt.hcl files are located at app_name/account_name/region_name/terragrunt.hcl. And are simple terragrunt files that have an include block for the root terragrunt configuration (defined at the application/top level). In our root terragrunt configuration we define the above locals for all children, basic things like project/app level locals, account level & region level locals.

The issue is we are unable to use the run-all command to apply multiple accounts/regions at any given time as the terragrunt run-all <operation> command first validates/tries to run the root level terragrunt file which is invalid on it's own as the account & region level locals do not exists in that context.

Running plan/apply operation at the region level works correctly as it finds the child terragrunt.hcl

Expected behavior

Be able to run the run-all command from the top/application level

[RyanW8]

Looks like skip = true would solve this nicely but it doesn't seem to be functioning as intended at the minute #704. Issue open for a very long time too ):

[denis256]

Hi,
I wondered why to have in the project root terragrunt.hcl file?
The include path can reference different file names:

include "root" {
  path = find_in_parent_folders("root.hcl")
}

https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#find_in_parent_folders

[pmaterer]

This appears to still be an issue, and it seems using the structure in this repo and from the docs doesn't work with Terragrunt.

I'm using:

❯ terragrunt --version                                
terragrunt version 0.66.9

Here is my directory layout:

~/tmp/example on ☁️  materer-dev (us-east-1) 
❯ tree                   
.
├── dev
│   ├── account.hcl
│   └── key
│       └── terragrunt.hcl
├── prod
│   ├── account.hcl
│   └── key
│       └── terragrunt.hcl
└── terragrunt.hcl

5 directories, 5 files

And the files:

File: terragrunt.hcl
Path: ./terragrunt.hcl

Contents:
locals {
  account_info = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  profile = local.account_info.locals.profile
}

generate "provider" {
    path = "provider.tf"
    if_exists = "overwrite_terragrunt"
    contents = <<EOF
provider "aws" {
    region = "us-east-1"
    profile = "${local.profile}"
}
EOF
}

----------------------------------------

File: account.hcl
Path: ./prod/account.hcl

Contents:
locals {
    profile = "materer-prod"
}

----------------------------------------

File: terragrunt.hcl
Path: ./prod/key/terragrunt.hcl

Contents:
include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "tfr:///terraform-aws-modules/kms/aws?version=3.1.0"
}

inputs  = {
    description = "Test key"
}

----------------------------------------

File: account.hcl
Path: ./dev/account.hcl

Contents:
locals {
    profile = "materer-dev"
}

----------------------------------------

File: terragrunt.hcl
Path: ./dev/key/terragrunt.hcl

Contents:
include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "tfr:///terraform-aws-modules/kms/aws?version=3.1.0"
}

inputs = {
    description = "Test key"
}

----------------------------------------

And the error I get:

❯ terragrunt run-all init
ERRO[0000] Error: Error in function call
      
ERRO[0000]   on /Users/pmaterer/tmp/example/terragrunt.hcl line 2, in locals: 
ERRO[0000]    2:   account_info = read_terragrunt_config(find_in_parent_folders("account.hcl")) 
ERRO[0000]                                              
ERRO[0000] Call to function "find_in_parent_folders" failed: ParentFileNotFoundError: Could not find a account.hcl in any of the parent folders of /Users/pmaterer/tmp/example/terragrunt.hcl. Cause: Traversed all
the way to the root..
 
ERRO[0000] Error processing module at '/Users/pmaterer/tmp/example/terragrunt.hcl'. How this module was found: Terragrunt config file found in a subdirectory of /Users/pmaterer/tmp/example. Underlying error: /Users/pmaterer/tmp/example/terragrunt.hcl:2,41-64: Error in function call; Call to function "find_in_parent_folders" failed: ParentFileNotFoundError: Could not find a account.hcl in any of the parent folders of /Users/pmaterer/tmp/example/terragrunt.hcl. Cause: Traversed all the way to the root.. 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

[BeyondEvil]

That a fundamental command doesn't work with their own example repo and the total lack of response in this thread, really puts me off Terragrunt.

Given this doesn't work, really throws a wrench in any sane CI configuration.

[yhakbar]

Hey, I'm sorry about that @BeyondEvil .

We're taking that comment seriously. If you'd like, join our Discord and chat with me personally about the issue that you're facing so we can get you resolution quickly. I'm reviewing this issue now to get caught up.

[yhakbar]

We chatted in Discord, and @BeyondEvil 's issues have been resolved. I'll post an update here shortly to address what we discussed, and how we'll be taking action to improve the state of our examples in the future.

[BeyondEvil]

I have to say, I'm SUPER IMPRESSED!

During my 15+ years in this industry and 10+ years in retail sales before that, I don't think I've ever experienced this level of "customer service".

11/10 would recommend! ❤️

[yhakbar]

To recap our discussion, with a bit of extra context:

The example repo currently uses a terragrunt.hcl file at the root of the repository for globally shared configurations. This is idiomatic Terragrunt (find_in_parent_folders defaults to looking for a parent terragrunt.hcl file), and is something that has been supported since very early on, but has become a bit of an anti-pattern due to issues like this. The issue arises due to the fact that Terragrunt will look for all files named terragrunt.hcl and consider them Units when performing a run-all.

In the past, this has been mitigated by having Terragrunt ignore any terragrunt.hcl files found in the current directory during a run-all, but this confuses a different set of users that run terragrunt run-all plan/apply within a unit directory expecting a plan/apply to run, and not seeing anything happen. Without doing a lot of extra filesystem lookups, Terragrunt doesn't know by default whether a terragrunt.hcl file is the root of a project, or where a unit can be found.

We fixed @BeyondEvil 's configurations by having them rename the root terragrunt.hcl file to root.hcl, and updating all references of find_in_parent_folders() to find_in_parent_folders("root.hcl"). I also mentioned that if this wasn't desirable, adding the following would signal to Terragrunt that it could ignore the root terragrunt.hcl file:

terragrunt run-all --terragrunt-exclude-dir . plan

To prevent new Terragrunt users experiencing this kind of issue going forward, I'll be cutting a PR to the example repo to rename the root terragrunt.hcl file to root.hcl, and updating all references of find_in_parent_folders accordingly. New users should get configurations that work out of the box, and we dropped the ball here.

In addition, to prevent the proliferation of the current idiomatic configuration, and to make it clear to users that they're following an anti-pattern, I'll be proposing the addition of a strict control to Terragrunt so that folks stop using a root terragrunt.hcl file, and use something like root.hcl instead. Once we have good reason to believe that the majority of users are no longer using this pattern, we can chose to make it an error to pass terragrunt.hcl to find_in_parent_folders.

Given that this has been idiomatic for a very long time, this may take some time. It will likely not make the cut for Terragrunt 1.0, but it's something we can start emitting warnings on soon so that folks adjust their configurations in anticipation of a 2.0.

[yhakbar]

@RyanW8 + @pmaterer ,

Are you alright with the current state of things? I'm hoping that my explanation above can help you out as well. If so, we can close this issue out.

[pmaterer]

@yhakbar

Appreciate the followup! I actually already implemented a root.hcl. I must've seen it mentioned somewhere else... But everything is working smoothly now.

Thanks again!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for raising this issue.