How to use kubernetes_service data provider in Terraform?

Question

I have simple workflow. Create a cluster, install ingress with helm and once it's ready, get load balancer's IP address to update CF dns records. It's all in one script.

After creating nginx-ingress controller, I was trying to get IP address of load balancer.

resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  ...
}

data "kubernetes_service" "this" {
  metadata {
    name = "ingress-nginx-controller"
    namespace = "ingress-nginx"
  }
  depends_on = [ helm_release.nginx_ingress ]
}

This never worked for me and the reason was that it was trying to connect to localhost. Then I found this limitation.

WARNING When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources. Please refer to this section of Terraform docs for further explanation.

and moved kubernetes provider to another module. However, I have now a new problem.

module "cluster" {
  source = "./modules/cluster"
  
  cluster_node_count = var.cluster_node_count
  age_path = var.age_path
}


module "network" {
  source = "./modules/network"
  
  host = module.cluster.host
  token = module.cluster.token
  cluster_ca_certificate = module.cluster.cert
  cloudflare_email = var.cloudflare_email
  cloudflare_api_key = var.cloudflare_api_key
  main_domain = var.main_domain
  cluster_subdomains = var.cluster_subdomains

  depends_on = [module.cluster]
}

This also doesn't work and giving

Providers cannot be configured within modules using count, for_each or depends_on.

Without depends_on the network module is called at the same time as cluster, which of course leads to fail, since for the network module cluster should be ready.

So it's kind of dead-loop which I am not sure how to break.

Answer 1

The provider documentation you quoted here seems to have a mistake. It talks about using the two providers together in the same module but it seems like it really means in the same configuration.

The problem arises because Terraform needs to configure a provider in order to plan with it, and so everything inside a provider "kubernetes" block must be known at the time of planning. This means you can't use anything in there which isn't created yet.

That documentation is trying to tell you that you'll need to split the configuration that declares your Kubernetes cluster from all of your configurations that configure or make use of the Kubernetes cluster, and then you can terraform apply the creation of the cluster first, and then separately terraform apply your other configurations that interact with it using the kubernetes provider.