We recently moved our
GraphQL API from a big stateful monolithic app to a stateless design running on kubernetes in Google Cloud.
Our production cluster is composed of:
- 6x n1-standard-2 (2 vCPUs, 7.5 GB memory)
- 6x nodes over 3 instance groups (regions)
On top of the kubernetes system pods, we have:
- 2x api-service per node (12 total)
- 1x nginx service per node (6 total)
- + a single redis instance shared by all api-services (1 total)
We developed load tests using locust that graphql queries on the cluster. The test runs a master & slaves processes that run queries over our endpoint to mimic user load.
We tried different configurations, but we seem to hit a wall at arround 25 requests per second. So far, we tried:
- To help make sure our test wasn't the culprit, we ran more slaves running the test from more computers (very powerful 8-cores machines)
- The nodes were at roughly 25% CPU during testing, so we tried doubling the pod count to no avail
- We tried starting a second redis instance to see if redis was the bottleneck - no changes on RPS
- Our SQL server load is super low (~15% cpu, plenty of ram available, no IO bottleneck)
- Our pods are 100% stateless, so they don't even have mounted disks, no IO bottleneck
- Internal network traffic is slow... no Network bottleneck
- Our api-service containers (dotnetcore) have plenty of threads available (Max ThreadCount is at default 32768 and we only see 5-6 threads used at any time during "max load"
- Our api-service code is non blocking, using async calls everywhere we can
- We DID get some redis timeout errors under load though, but we couldn't find why.
slowlogreturned 1 query once in a while
So, we're starting to exhaust our theories on why this setup can't scale any higher. We are starting to doubt our actual test now and investigating that direction, but we haven't found anything suspicious yet.