1

I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.

I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.

I created an issuer:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - selector: {}
      http01:
        ingress:
          ingressClassName: nginx

and annotated the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /api/(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: backend
            port:
              number: 8080
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: frontend
            port:
              number: 3000
  tls:
  - hosts:
    - example.com
    secretName: frontend-ingress-cert

This creates the cm-acme-http-solver:

NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m

This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:

Name:         frontend-ingress-cert-1-1328491808-3444115809
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2024-04-02T15:07:31Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  frontend-ingress-cert-1-1328491808
    UID:                   e85e8c53-5965-4be4-87fa-50359217539f
  Resource Version:        10398552697
  UID:                     73e25122-6340-406f-aa68-632e92adec4b
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
  Dns Name:           example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
  Key:      <redacted>>.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
  Solver:
    http01:
      Ingress:
        Ingress Class Name:  nginx
    Selector:
  Token:     <redacted>>
  Type:      HTTP-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/<redacted>>': Get "http://example.com/.well-known/acme-challenge/<redacted>>": EOF
  State:       pending
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:

❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
/app $ wget http://example.com/.well-known/acme-challenge/<redacted>>
Connecting to mydomain.com (51.158.131.xxx:80)
wget: error getting response: Resource temporarily unavailable

Nginx logs this:

2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: "GET /.well-known/acme-challenge/<redacted>> HTTP/1.1" while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80

1 Answer 1

0

For anyone running into the same problem: cert-manager's issue tracker has an excellent discussion about the problem dating back to 2018.

The solution for me was to install compumike's hairpin-proxy to enable PROXY protocol support for internal-to-LoadBalancer traffic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.