0

Context

Our production Kubernetes clusters configured to use multiple traefik pods (deployment of several replicas) and HPA (Horizontal pod autoscaler) as well as cluster node autoscaler.

We notice a strange behaviour a little bit later after deploying the autoscaler: Some of our customer losing packets (request timeout) - Our customers initiate a request to the load balancer, there are no 400\500 error from traefik pods (we use datadog to monitor all requests) but from the client side, some of the requests just end with timeout (This is from customers report).

Assumption

Since the issue happened with other customers (customers send http requests to our service) since we introduce the HPA - we suspect that our autoscaling configuration make traefik evicted pods drop the TCP connections while scaling up or down the traefik pods.

For the sake of the conversation please exclude other networking assumptions (even though it can be completely related).

HTTP requests ( customer <-> production k8s clusters)

The http request\response: customer initiate and POST\GET\PUT request -> K8s load balancer (traefik) route the request to our service by namespace -> service connects to database -> service respond to customer (assuming with the same tcp connection as initiated)

Specificaitons

The autoscaler and traefik deployment (some of the deployment not shared - only relevant lines):

---
apiVersion: apps/v1
kind: Deployment
  labels:
    app: traefik
  name: traefik
  namespace: ingress-traefik
spec:
  replicas: 2
  selector:
    matchLabels:
      app: traefik
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: traefik
    spec:
      containers:
      - args:
        - --configfile=/config/traefik.toml
        image: traefik:1.7.9-alpine
        imagePullPolicy: IfNotPresent
        name: traefik
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8080
          name: dashboard
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 400Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      serviceAccountName: traefik-ingress-controller
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: traefik-hpa
  namespace: ingress-traefik
spec:
  maxReplicas: 8
  minReplicas: 4
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: traefik
  metrics: 
  - type: Resource 
    resource: 
      name: cpu 
      targetAverageUtilization: 40
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 40 

Currently to try and solve the "lost packets" issue we removed this configuration.

Questions

  1. If anyone experience the same thing using HPA and traefik1.7 in Kubernetes cluster?
  2. Is there recommended configuration for traefik pod (Liveness, Readiness and Startup Probes) using HPA?
  3. What is the best practice configuration for traefik and HPA in K8s cluster?

All the best to all of you out there reading this post

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.