Investigate using of create_before_destroy option for node pools

added teamCloud workflowNeeds Refinement labels

changed title from Enable create-before-destroy for node pools to Investigate using of create_before_destroy option for node pools

Refinement notes:

check if there will be any downtime
check if it is possible at all (i.e. having the second node pool with the same name)

check if it is possible at all (i.e. having the second node pool with the same name)

Mmm. Yes. We might want to have something like local.pool_generation or similar appended to the pool name which we can bump if there's a change necessitating a pool re-creation.

I finally got some time for experiments and I got promising results (with no efforts, love it).

So I created the cluster with our module in my test env, and I used lates google provider. When I tried to change node type in my project I got this

  # module.cluster.google_container_node_pool.cluster-pool-1[0] will be updated in-place
  ~ resource "google_container_node_pool" "cluster-pool-1" {
...
          ~ machine_type                = "g1-small" -> "n1-standard-1"
...
    }

Plan: 0 to add, 1 to change, 0 to destroy

I applied and all works like a charm

After that I tried the same in our https://gitlab.developers.cam.ac.uk/uis/devops/devhub/gitlab-runner-infrastructure project (not applied obviously, just plan)

  # module.cluster.google_container_node_pool.cluster-pool-1[0] must be replaced
...
          ~ machine_type      = "e2-standard-4" -> "n1-standard-1" # forces replacement
...
    }

Plan: 1 to add, 0 to change, 1 to destroy.

The only difference is that my project is on google v5 provider, and runner-infra is on