In Istio, what controls throughput to a service?

Question

I have a load testing application (jMeter) that is calling my service that is hosted in Kubernetes.

Summary

I have a service hosted in Kubernetes that seems to have limits placed on how many calls it can take. This service uses Istio (both with a Side Car and using the Istio Ingress Gateway). But I cannot seem to figure out what is limiting the calls to it.

Details

The primary action for my service (that is hosted in Kubernetes and uses Istio) is to enqueue a message to a RabbitMQ server. This takes a bit of time, but is not CPU nor memory intensive.

When I load test my service (using jMeter), neither the CPU nor memory get high enough to trigger my Horizontal Pod Autoscaling.

But, the throughput is way too low for what this service should be able to do.

My guess is that istio is throttling the calls to my service somehow, but I cannot seem to find what is causing it.

What I have tried

---

Destination Rule

I made a destination rule like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-testing-rule
  namespace: my-testing-namespace
spec:
  host: my-service-testing-host.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 5000 

I did this in hopes of upping the max connections, but it seemed to have no effect when I re-run the load test. (Note: the name, namespace and host in the example above have been changed from the actual values.)

---

No Istio

I tried removing Istio from the picture, and it overloaded my cluster when I fired up the load test. I could not longer send kubectl commands and the cluster management tools would not respond. (Apparently some throttling is good.)

But this is what leads me to think that Istio is throttling things.

Question

What configurations in the default setup of Istio will throttle traffic?

Answer 1

You can use Envoy Filter to apply limit rate. In the below example, you apply a limit rate to a workload based on workloadSelector labels. The main example values are that you have a bucket of 1000 tokens. Each connection takes "one token" and in the bucket we "return" 10 tokens per 5 second. The limit rate is enabled and applied to all connection towards selected workload through local_rate_limit_enabled and local_rate_limit_enforced.

References that helped me a lot to applied the limit rate to our project was:

Limit Rate Example and Documentation

    apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: my-limit-rater
  namespace: mynamespace
spec:
  workloadSelector:
    labels:
      app: "myapp"
      component: "mycomponent"
      instance: "test"
      version: "1.0.0"
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: 'envoy.filters.network.http_connection_manager'
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: http_local_rate_limiter
              enable_x_ratelimit_headers: DRAFT_VERSION_03
              token_bucket:
                max_tokens: 1000
                tokens_per_fill: 10
                fill_interval: 5
              filter_enabled:
                runtime_key: local_rate_limit_enabled
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              filter_enforced:
                runtime_key: local_rate_limit_enforced
                default_value:
                  numerator: 100
                  denominator: HUNDRED