NodeCreationFailure: Instances failed to join the kubernetes cluster

Question

I'm setting up an EKS cluster using https://github.com/ManagedKube/kubernetes-ops

and, based on that code, when I do a terraform apply I'm getting a NodeCreationFailure error.

Specifically:

│ Error: error waiting for EKS Node Group (staging:staging-ng12022040211190144860000000d) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│   * i-0d5c42031eaf061ff, i-0d9a073240fd29c45: NodeCreationFailure: Instances failed to join the kubernetes cluster

Admittedly, that's not much code, but does anyone have any pointers on where to start debugging?

Answer 1

I incurred the same problem during one of my deployments.

This happens due to the Networking issue of the EKS cluster. Assuming that you are working in AWS, follow the following steps to debug this problem:

1. Check if your security groups are correctly configured correctly to your Nodes.
2. If the EKS is present in the public sub-nets, check its connectivity to internet gateway and is able to access the internet.
3. If Eks is present in the private sub-net, check the connectivity with NAT gateway and that NAT gateway is able to access the internet.

Answer 2

I had a similar issue in the past where the nodes couldn't join the cluster. The cause of this was a mismatch in security groups, the nodes and the EKS control plane weren't allowed to talk to each other and therefore timed out.

Answer 3

I was getting the same issue, it was solved by creating another nat gateway and choosing public subnet and attaching that new nat gateway in routable to my private subnet.

Answer 4

AWS has listed a set of reasons it could make Node groups to fail joining a cluster. Refer, https://repost.aws/knowledge-center/eks-worker-nodes-cluster I was able to understand that one of my security groups rejected the comms between Control Plane and the worker node. This troubleshooting was possible using System Automation - TroubleshootingEKSNode.

Answer 5

In the event you are using a NAT instance instead of a NAT gateway, you need to make sure that the NAT instance is provisioned before the cluster. Usually terraform will handles these dependencies automatically, but in this case it is not aware of the depenedency since the NAT instance is just an EC2 instance that wouldn't typically need special status. You can enforce the provison order with

module "eks" {
  source = "./modules/eks"
  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets
  tags            = var.tags
  depends_on = [module.vpc, module.fck_nat] # This is the key point, where fck_nat is your nat instance module
}