Kubernetes pods scaling doesnt increase throughput

Question

I have a simple Django web application running on AWS EC2 instance (2 vCPU, 4Gi). For that type of deployment I have a performance (measured with Hey)

$ hey -t 30 -z 1m https://mydomain.com/my/endpoint/

about 30 reqs/s

When deployed with Kubernetes onto EKS cluster (same docker image) I have about 15 reqs/s for 1 pod with limits

    resources:
      requests:
        cpu: "500m"
        memory: "2Gi"
      limits:
        cpu: "1"
        memory: "4Gi"

I expect that scaling pods would double the performance but nothing happens (request rate stays the same)

$ kubectl scale deploy/myapp --replicas=2

I'm running Django app behind gunicorn like this (adjusting number of workers did not help)

gunicorn --config gunicorn.py myapp.wsgi:application
# gunicorn.py
import os

# tried different values from 1 to 5
# for 1 vCPU allocated for the pod it should be 3 as per docs
# (2 x $num_cores) + 1
# https://docs.gunicorn.org/en/stable/design.html#how-many-workers
workers = int(os.getenv('GUNICORN_WORKERS_APP', default=4))
daemon = False
bind = '0.0.0.0:8080'
max_requests = 2000

So couple of questions here:

• how to properly debug this type of issue and what's the toolset I can use?
• how service does load balancing to pods and is there any overhead introduced?
• is there an overhead for resource management (cpu allocation/throttling)?
• how performance tuning is done for k8s-driven applications?

Answer 1

What does 'kubectl top node' return for your node usage?

Kubernetes is pretty low overhead and can certainly handle more than 15req/s. Does your application rely on a database? If so, where is that database living? Is that single node also running any other pods?

As others noted, an APM would be a helpful tool to diagnose unexpected performance issues.