Increase Kubernetes Nginx Ingress limits for large images in Docker registry

Question

When pushing images to a self hosted Docker registry installed using their official helm chart, this works for some images. But for one image, I'm not able to push two layers. Docker retry it several times, without success.

3a6fab8d5d23: Retrying in 2 seconds

The logs of the Kubernetes pod gave me the following error:

unknown error reading request payload: unexpected EOF

Both layers were relatively large (252.5 MB/ 393.8 MB).

What I already tried on the Ingress

I had the Kubernetes Nginx Ingress behind my registy. Using the direct NodePort without Ingress, it works perfectly. So it seems that some limits of the Ingress break the upload. The following annotations were applied:

ingress:
  enabled: true
  hosts: 
    - my-docker-registry.domain
  tls:
    - hosts:
      - my-docker-registry.domain
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"

This was taken from the Nginx Ingress documentation, where a Docker Registry behind Nginx acts as example. Are there any other limits I need to increase so that my registry will properly work behind the Ingress?

Background

I want to install the Component Pack for IBM Connections. IBM has a very special way to deliver this Kubernetes based application: You get a ~4GB Zip file. It contains all images and helm charts as well as a shell script, which tags and pushes all those images to a custom registry. The cnx-ingress image is relatively large:

:~/Downloads/IC-ComponentPack-6.0.0.8# l microservices_connections/hybridcloud/images/cnx-ingress.tar 
-rw------- 1 daniel daniel 739M Mär 29 13:18 microservices_connections/hybridcloud/images/cnx-ingress.tar

Answer 1

I learned that it's not required to set the ingress class to nginx as I did in kubernetes.io/ingress.class: nginx. All annotations prefixed with nginx.ingress.kubernetes.io apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is

nginx.ingress.kubernetes.io/proxy-body-size: "0"

because the default value is way too small for Docker image layers. Without this annotation, we get 413 Request Entity Too Large errors.

The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:

defaults
    # ...
    retries 3
    timeout connect     1000
    timeout client      1000
    timeout server      1000

According to HAProxys time formats, this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems.