K8s cluster in peered VPC can't reach RDS cluster

Question

I have two K8s clusters in the same VPC that I want to connect to an MySQL Aurora RDS cluster. One of the clusters can reach the RDS cluster just fine. The other, however, cannot. I'll call these eks-cluster-working and eks-cluster-broken.

I have a security group allowing traffic to the cluster:

Type	Protocol	Port range	Source	Description
MYSQL/Aurora	TCP	3306	sg-1 (eks-cluster-sg-working)	This rule works
MYSQL/Aurora	TCP	3306	sg-2 (eks-cluster-sg-broken)	This one does not

Both eks-cluster-working and eks-cluster-broken have the same default security groups that AWS creates when making an EKS cluster and they are both on the same K8s version (1.18). The one exception is that eks-cluster-working has an extra security group for the load balancer service since it's hosting a web service and the other one (eks-cluster-broken) is not. Both clusters have outbound traffic rules that can reach 0.0.0.0/0.

I have zero firewalls, firewall policies and network firewall rule groups set up in the region the clusters are hosted in.

This is the Terraform configuration I have set up for peering from my VPCs to the default VPC.

resource "aws_vpc_peering_connection" "default_to_environment" {
  count       = local.num_environment_vpcs
  peer_vpc_id = data.aws_vpc.environment[count.index].id
  auto_accept = true
  vpc_id      = var.vpc_default_id

  tags = {
    Name      = "Peer from default to ${data.aws_vpc.environment[count.index].tags["Name"]}"
    tf_module = "vpc"
  }
}

resource "aws_route_table" "r" {
  vpc_id = var.vpc_default_id

  # route = local.route_table_routes

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "igw-<number>"
  }

  dynamic route {
    for_each = [for r in local.route_table_routes : {
      cidr_block                = r.cidr_block
      vpc_peering_connection_id = r.vpc_peering_connection_id
    }]
    content {
      cidr_block                = route.value.cidr_block
      vpc_peering_connection_id = route.value.vpc_peering_connection_id
    }
  }

  tags = {
    Name = "default to environment VPCs"
  }
}

Expected result

I expect eks-cluster-broken to be able to connect to the RDS cluster just like eks-cluster-working can.

Question

What can cause this? Any suggestions as to how to go about debugging this?

Answer 1

For my Peering Connection, I went to:

"Actions" -> "Edit DNS Settings" -> "Allow accepter VPC (vpc-<id>) to resolve DNS of requester VPC (vpc-<id-default>) hosts to private IP"

and checked it. (It was unchecked before). I had to wait a little while before this took effect.