I have a load testing application (jMeter) that is calling my service that is hosted in Kubernetes.
I have a service hosted in Kubernetes that seems to have limits placed on how many calls it can take. This service uses Istio (both with a Side Car and using the Istio Ingress Gateway). But I cannot seem to figure out what is limiting the calls to it.
The primary action for my service (that is hosted in Kubernetes and uses Istio) is to enqueue a message to a RabbitMQ server. This takes a bit of time, but is not CPU nor memory intensive.
When I load test my service (using jMeter), neither the CPU nor memory get high enough to trigger my Horizontal Pod Autoscaling.
But, the throughput is way too low for what this service should be able to do.
My guess is that istio is throttling the calls to my service somehow, but I cannot seem to find what is causing it.
What I have tried
I made a destination rule like this:
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: my-testing-rule namespace: my-testing-namespace spec: host: my-service-testing-host.svc.cluster.local trafficPolicy: connectionPool: http: http2MaxRequests: 5000
I did this in hopes of upping the max connections, but it seemed to have no effect when I re-run the load test. (Note: the name, namespace and host in the example above have been changed from the actual values.)
I tried removing Istio from the picture, and it overloaded my cluster when I fired up the load test. I could not longer send kubectl commands and the cluster management tools would not respond. (Apparently some throttling is good.)
But this is what leads me to think that Istio is throttling things.
What configurations in the default setup of Istio will throttle traffic?