[ISSUE] Can not update zone_id from non-auto to auto-az

[tejas-angelone]

Describe the issue

The zone_id for the job cluster isn't getting updated from the non-auto availability zone to auto AZ. Whereas, the update works for non-auto AZs only. So, once a cluster is created with non-auto AZ you can switch to auto.

Issue started from CLI v0.220.0

Steps to reproduce the behavior

1. Create a resource config as below and deploy the bundle

job_clusters:
    - job_cluster_key: job_cluster
      new_cluster:
        spark_version: 13.3.x-scala2.12
        node_type_id: i3.xlarge
        aws_attributes:
          zone_id: ap-south-1a
        autoscale:
            min_workers: 1
            max_workers: 4

2. Update the zone_id: auto and deploy the bundle

Expected Behavior

zone_id must update to auto

Actual Behavior

The zone_id doesn't get updated from ap-south-1a (any non-auto AZ) to auto

OS and CLI version

Databricks CLI v0.220.0

Is this a regression?

Works in Databricks CLI v0.219.0

[dmarcus30]

Having same issue here

[ribugent]

In our case we noticed this regression today, we could update the definitions and then update manually on the web, but in our case the number of job clusters is insane and we'll lose too much time.

Unfortunately downgrading to 0.219 in our case doesn't seem to work due to the usage of complex variables.

EDIT: I ended up modifying one by one during 20 minutes all job clusters after adjusting the configuration files.

[rpinzon]

Having same issue here.

I also tried using a policy with the auto zone_id without specifying a value in the bundle, but it ended up creating a cluster with the first non-auto zone_id available.

[FernandoOLI]

I found an alternative.

Before deploying, run this command:
databricks bundle destruction

Deleting all jobs and their permissions and in the AWS attribute settings, make zone_id: auto explicit.

When running the deploy command again
databricks bundle deployment

The clusters were created with thezone_id auto