Part of my struggle to learn this, I think, is that cert-manager
is updated pretty regularly so a lot of the tutorials, blog posts, questions, and courses I have read or taken are no longer relevant.
For example, when I initially started learning Kubernetes you had to make certificate.yaml
that looked similar to this:
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: test-domain-com
spec:
secretName: test-domain-com
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
commonName: test.domain.com
dnsNames:
- test.domain.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- test.domain.com
This doesn't appear to be the case anymore since it seems to be extrapolated from the issuer.yaml
and ingress.yaml
.
Anyway, I have a certificate that "issued successfully", but still HTTPS is not working on the application when I navigate to it.
issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: test-domain-com
spec:
secretName: test-domain-com
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
commonName: test.domain.com
dnsNames:
- test.domain.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- test.domain.com
ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/add-base-url: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
cert-manager.io/cluster-issuer: "letsencrypt-staging"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
name: ingress
namespace: default
spec:
tls:
- hosts:
- test.domain.com
secretName: test-domain-com
rules:
- host: test.domain.com
http:
paths:
- path: /?(.*)
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
backend:
serviceName: api-cluster-ip-service
servicePort: 5000
Certificate Logs
$ kubectl describe certificates test-domain-com
Name: test-domain-com
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"test-domain-com","namespace":"de...
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2020-01-30T17:49:12Z
Generation: 6
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: ingress
UID: d31883d9-4388-11ea-84ec-469fd0fd8fec
Resource Version: 832245
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/default/certificates/test-domain-com
UID: d3222905-4388-11ea-84ec-469fd0fd8fec
Spec:
Dns Names:
test.domain.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: test-domain-com
Status:
Conditions:
Last Transition Time: 2020-01-30T18:12:41Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2020-04-29T17:12:40Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Requested 43m cert-manager Created new CertificateRequest resource "test-domain-com-566504257"
Normal Issued 42m cert-manager Certificate issued successfully
Describe Secret
$ kubectl describe secret test-domain-com
Name: test-domain-com
Namespace: default
Labels: <none>
Annotations: cert-manager.io/alt-names: test.domain.com
cert-manager.io/certificate-name: test-domain-com
cert-manager.io/common-name: test.domain.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-staging
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
ca.crt: 0 bytes
tls.crt: 3566 bytes
tls.key: 1675 bytes
ca.crt
is empty and not entirely sure if it should or shouldn't be. Looks like there is a Github issue about it.
Errors in cert-manager
log
W0130 19:13:11.357728 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
E0130 19:14:21.811114 1 controller.go:230] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="error decoding CA private key" "error"="error decoding private key PEM block" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager"
E0130 19:14:21.811257 1 controller.go:131] cert-manager/controller/webhook-bootstrap "msg"="re-queuing item due to error processing" "error"="error decoding private key PEM block" "key"="cert-manager/cert-manager-webhook-tls"
rce_kind"="Challenge" "resource_name"="test-domain-com-566504257-1006673842-2382236481" "resource_namespace"="default" "type"="http-01"
E0130 19:16:21.262484 1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="test-domain-com-566504257-1006673842-2382236481" "resource_namespace"="default" "type"="http-01"
E0130 19:16:42.616965 1 controller.go:156] ingress 'default/cm-acme-http-solver-6xj55' in work queue no longer exists
E0130 19:16:42.837328 1 sync.go:59] cert-manager/controller/orders "msg"="failed to update status" "error"=null "resource_kind"="Order" "resource_name"="test-domain-com-566504257-1006673842" "resource_namespace"="default"
E0130 19:16:42.837364 1 controller.go:131] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on orders.acme.cert-manager.io \"test-domain-com-566504257-1006673842\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/test-domain-com-566504257-1006673842"
E0130 19:16:43.687987 1 controller.go:193] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"test-domain-com-566504257-1006673842-2382236481\" not found"
The only thing that jumped out to me was:
"msg"="error decoding CA private key" "error"="error decoding private key PEM block"
Tried to dumping browser cache, incognito mode, etc... still comes up HTTP or "Not Secure" with "Fake LE Intermediate X1" certificate when navigating to HTTPS.
So what did I screwup/overlook this time and how do I resolve this?