Issue getting HTTPS working for subdomain despite “Certificate issued successfully”

Question

Part of my struggle to learn this, I think, is that cert-manager is updated pretty regularly so a lot of the tutorials, blog posts, questions, and courses I have read or taken are no longer relevant.

For example, when I initially started learning Kubernetes you had to make certificate.yaml that looked similar to this:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: test-domain-com
spec:
  secretName: test-domain-com
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: test.domain.com
  dnsNames:
    - test.domain.com
  acme:
    config:
      - http01:
          ingressClass: nginx
        domains:
          - test.domain.com

This doesn't appear to be the case anymore since it seems to be extrapolated from the issuer.yaml and ingress.yaml.

Anyway, I have a certificate that "issued successfully", but still HTTPS is not working on the application when I navigate to it.

issuer.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: test-domain-com
spec:
  secretName: test-domain-com
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: test.domain.com
  dnsNames:
    - test.domain.com
  acme:
    config:
      - http01:
          ingressClass: nginx
        domains:
          - test.domain.com

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: ingress
  namespace: default
spec:
  tls:
    - hosts:
        - test.domain.com
      secretName: test-domain-com
  rules:
    - host: test.domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service
              servicePort: 5000

Certificate Logs

$ kubectl describe certificates test-domain-com
Name:         test-domain-com
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"test-domain-com","namespace":"de...
API Version:  cert-manager.io/v1alpha2
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-01-30T17:49:12Z
  Generation:          6
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  ingress
    UID:                   d31883d9-4388-11ea-84ec-469fd0fd8fec
  Resource Version:        832245
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/default/certificates/test-domain-com
  UID:                     d3222905-4388-11ea-84ec-469fd0fd8fec
Spec:
  Dns Names:
    test.domain.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  test-domain-com
Status:
  Conditions:
    Last Transition Time:  2020-01-30T18:12:41Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-04-29T17:12:40Z
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Requested  43m   cert-manager  Created new CertificateRequest resource "test-domain-com-566504257"
  Normal  Issued     42m   cert-manager  Certificate issued successfully

Describe Secret

$ kubectl describe secret test-domain-com
Name:         test-domain-com
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: test.domain.com
              cert-manager.io/certificate-name: test-domain-com
              cert-manager.io/common-name: test.domain.com
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-staging
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:   0 bytes
tls.crt:  3566 bytes
tls.key:  1675 bytes

ca.crt is empty and not entirely sure if it should or shouldn't be. Looks like there is a Github issue about it.

Errors in cert-manager log

W0130 19:13:11.357728       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0130 19:14:21.811114       1 controller.go:230] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="error decoding CA private key" "error"="error decoding private key PEM block" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
E0130 19:14:21.811257       1 controller.go:131] cert-manager/controller/webhook-bootstrap "msg"="re-queuing item  due to error processing" "error"="error decoding private key PEM block" "key"="cert-manager/cert-manager-webhook-tls" 
rce_kind"="Challenge" "resource_name"="test-domain-com-566504257-1006673842-2382236481" "resource_namespace"="default" "type"="http-01" 
E0130 19:16:21.262484       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="test-domain-com-566504257-1006673842-2382236481" "resource_namespace"="default" "type"="http-01" 
E0130 19:16:42.616965       1 controller.go:156] ingress 'default/cm-acme-http-solver-6xj55' in work queue no longer exists
E0130 19:16:42.837328       1 sync.go:59] cert-manager/controller/orders "msg"="failed to update status" "error"=null "resource_kind"="Order" "resource_name"="test-domain-com-566504257-1006673842" "resource_namespace"="default" 
E0130 19:16:42.837364       1 controller.go:131] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on orders.acme.cert-manager.io \"test-domain-com-566504257-1006673842\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/test-domain-com-566504257-1006673842" 
E0130 19:16:43.687987       1 controller.go:193] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"test-domain-com-566504257-1006673842-2382236481\" not found"  

The only thing that jumped out to me was:

"msg"="error decoding CA private key" "error"="error decoding private key PEM block"

Tried to dumping browser cache, incognito mode, etc... still comes up HTTP or "Not Secure" with "Fake LE Intermediate X1" certificate when navigating to HTTPS.

So what did I screwup/overlook this time and how do I resolve this?

Answer 1

Ok figured it out. It was due to using staging like so:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: nginx

Changed it prod like so and that cleared up the issue:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

Apparently that is the expected behavior when using staging according to this well written and accurate tutorial.

That being said I didn't initially think it worked. Even after a hard reload and empty cache + hard reload, HTTPS still wasn't working. Tried incognito and it worked, so closed all my browser instances, reopened, and HTTPS worked.

And also, certificate.yaml was not necessary at all.