Terraform: Provider configuration with modules and feature flags

Question

I want to use Terraform to manage many resources on many environments.

I chose to create a workspace for each environments.

I also use an object to enable or not module on each environement:

variable "feature_flags" {
  type = object({
    powerdns          = bool
    ovhiam            = bool
    security_groups   = bool
  })
  default = {
    powerdns         = false
    ovhiam           = false
    security_groups  = false
  }
}

Each feature is then splitted into module:

module "pdns" {
  count = var.feature_flags.powerdns ? 1 : 0
  source = "./modules/pdns"

  providers = {
    powerdns = powerdns.pdns
  }

  dns_zones     = "${var.dns_zones}"
}

My problem comes when I need to apply on an environment (and workspace) that has a disabled module.

provider "powerdns" {
  alias           = "pdns"
  server_url      = "${var.pdns_api_url}"
  insecure_https  = false
}

Even when not needed, provider config is called, and failed as the parameter is not defined.

 Error: Error setting up PowerDNS client: Error while creating client: No URL provided
│ 
│   with provider["registry.terraform.io/pan-net/powerdns"].pdns,
│   on providers.tf line 15, in provider "powerdns":
│   15: provider "powerdns" {
│ 

What would be the best practices to support this use case?

Thanks