Argo workflows health checks failing in GKE

Question

I am trying to expose my argo workflows deployment to a domain but I keep getting Server Error when I access the domain: argo.example.com. I setup a GKE ingress and been trying to fix the health checks by redirecting http to https, which is why I added a custom FrontendConfig and BackendConfig. However, I see the health check for "/" passing. See configuration below.

argo-server command args:

...
- args:
    - server
    - --namespaced
    - --auth-mode
    - server
    - --secure
    - "true"
...

ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argo-ingress
  annotations:
    networking.gke.io/managed-certificates: "argo-cert"
    networking.gke.io/v1beta1.FrontendConfig: argo-frontend-config
spec:
  rules:
    - host: argo.example.com
      http:
        paths:
          - backend:
              service:
                name: argo-server-expose
                port:
                  number: 2746
            path: /*
            pathType: ImplementationSpecific

Frontend:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: argo-frontend-config
  namespace: argo
spec:
  redirectToHttps:
    enabled: true

Backend:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: argo-backend-config
  namespace: argo
spec:
  healthCheck:
    checkIntervalSec: 30
    timeoutSec: 5
    healthyThreshold: 1
    unhealthyThreshold: 2
    type: HTTPS
    requestPath: /
    port: 2746

Service:

apiVersion: v1
kind: Service
metadata:
  name: argo-server-expose
  namespace: argo
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/backend-config: '{"ports": {"http":"argo-backend-config"}}'
spec:
  type: NodePort
  ports:
    - name: http
      port: 2746
      protocol: TCP
      targetPort: 2746
  selector:
    app: argo-server

Answer 1

Do not use GKE ingress, it's a nightmare. Use nginx-controller helm package instead. You will have to configure something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: http
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  rules:
    - host: argo.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argo-system-argo-workflows-server
                port:
                  number: 2746
  tls:
  - hosts:
      - argo.example.com
    secretName: argo-tls-cert