1

I'll start off by saying I'm not really an infrastructure person, but am being required to maintain it until we hire someone more suited to it, so apologies if I have any terminology wrong.

We have a kubernetes cluster (AKS)in azure. Currently we are running only 1 server in the cluster, with 4 namespaces configured within it, representing each environment. To properly support our production environment, we ideally want at least 3 servers, with the ability to scale appropriately. However, whenever we try to scale it up, services deployed to different server nodes cannot communicate with one another.

The unfortunate thing is I have no idea even how to start begin debugging this. I know we have some custom nginx rules that microsoft gave us, but even setting allow everything as the first rule does not seem to resolve this issue. How do I begin debugging this? What information would even be helpful for this issue?

Improve this question
5
  • 1
    I'm not very knowledgeable with azure, but I think each node has a kind of firewall on azure level, like security groups on aws, and you need to open the inbound ports for k8s, depending on you k8s setup there's too much options to give more guidance without details on the cluster configuration
    – Tensibai
    Oct 26, 2018 at 18:55
  • 4 namespaces? How many services have been deployed? What does kubectl get nodes and get ns return? Please provide some more information regarding the topology and k8s version. It is not clear to me whether you use the managed k8s or whether you deployed it yourself. Are you sure that all svc run? What apps should communicate with each other?
    – 030
    Oct 26, 2018 at 21:40
  • @030 kubectl get nodes returns 1 node currently. It returns 3 when we have the scaling turned up to 3 on the azure portal. get ns returns 9 namespaces, our 4 environments (dev, test, uat, prod), azure-system, default, kube-public, kube-system, and tiller
    – Marshall Tigerus
    Oct 29, 2018 at 17:41
  • @030 additionally, we hired a contractor that set up this environment, so I'm not sure everything he did. He provided us documentation that includes kubectl commands that were run to set things up, but I think he was just following advice from microsoft and things he googled, as he didn't seem to have any expertise in kubernetes.
    – Marshall Tigerus
    Oct 29, 2018 at 17:42
  • @Tensibai the firewall settings seemed to be at a higher level, but I've been messing with them to try to get things working.
    – Marshall Tigerus
    Oct 29, 2018 at 17:43

2 Answers 2

Reset to default
1

With help from a Microsoft Engineer, we were able to resolve our issue.

The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on. This indicated a problem with communication between nodes. The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.

we had a security rule that restricted all traffic that was not on our specifically approved list.

One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip. However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.

We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure. This resolved the issue.

If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.

Improve this answer
0

What about creating a new managed k8s cluster and start from scratch? This is what I do if the architecture is a maze. Managed k8s on azure and gcp is a couple of clicks away.

Improve this answer
3
  • We have a support ticket oustanding to microsoft, but if they don't get back to us soon, I'm going to try this.
    – Marshall Tigerus
    Oct 30, 2018 at 16:37
  • You could already start. You could leave the current cluster running and create a new managed cluster next to it without destroying the existing one.
    – 030
    Oct 30, 2018 at 22:15
  • more of a matter of time than anything else. I have a lot of other responsibilities to take care of, and technically this is microsoft's problem at the moment.
    – Marshall Tigerus
    Oct 31, 2018 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.