0

I'm attempting to set up my own bare-metal Kubernetes cluster composed of a control plane (VPS 149.62.186.95) and a worker node (VPS 149.62.189.139). Both my VPS are running Ubuntu 22.04 with 2 vCPUs and 4GB vRAM.

After installing Kubernetes (kubeadm) on both, I ran the following commands on my control plane:

kubeadm init

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Weave deamonset
kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

# baremetal nginx ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

I edited my DNS records as follows:

  • registry.rottigni.tech -> 149.62.186.95 (control plane)
  • sandbox.rottigni.tech -> 149.62.186.95 (control plane)

I generated SSL certs for these domains on my control plane and stored them in Kubernetes secrets.

Then, I applied my changes using kubectl apply -R -f . inside my repository. Here's the most relevant file, ingress.yml:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-ingress-class
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lr-labs-ingress
  namespace: lr-labs
  annotations:
    # cert-manager.io/issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx-ingress-class
  tls:
    - hosts:
        - registry.rottigni.tech
      secretName: registry-ssl
    - hosts:
        - sandbox.rottigni.tech
      secretName: sandbox-ssl

  rules:
    - host: registry.rottigni.tech
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: registry-rottigni
                port:
                  number: 5000

    - host: sandbox.rottigni.tech
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sandbox-rottigni
                port:
                  number: 80

At this point, everything seems to be working on the kubectl side, according to the following output:

Every 2.0s: kubectl get pods,deploy,svc,ing,endpoints,node -A                                                                                                                                                                           

NAMESPACE       NAME                                                READY   STATUS      RESTARTS        AGE
ingress-nginx   pod/ingress-nginx-admission-create-wn7xr            0/1     Completed   0               7h58m
ingress-nginx   pod/ingress-nginx-admission-patch-dtzxf             0/1     Completed   0               7h58m
ingress-nginx   pod/ingress-nginx-controller-86968d59df-vdv9f       1/1     Running     0               6h52m
kube-system     pod/coredns-5dd5756b68-h4jwb                        1/1     Running     0               7h58m
kube-system     pod/coredns-5dd5756b68-jrsdq                        1/1     Running     0               7h58m
kube-system     pod/etcd-epsilon-rottigni-tech                      1/1     Running     0               7h58m
kube-system     pod/kube-apiserver-epsilon-rottigni-tech            1/1     Running     0               7h58m
kube-system     pod/kube-controller-manager-epsilon-rottigni-tech   1/1     Running     0               7h58m
kube-system     pod/kube-proxy-4v8bc                                1/1     Running     0               7h58m
kube-system     pod/kube-proxy-8hpjj                                1/1     Running     0               7h57m
kube-system     pod/kube-scheduler-epsilon-rottigni-tech            1/1     Running     0               7h58m
kube-system     pod/weave-net-gvzk5                                 2/2     Running     1 (7h58m ago)   7h58m
kube-system     pod/weave-net-tbr92                                 2/2     Running     0               7h57m
lr-labs         pod/registry-rottigni-5c6cfd877f-qqxsw              1/1     Running     0               7h54m
lr-labs         pod/sandbox-rottigni-9668f7447-dv7js                1/1     Running     0               7h54m

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           7h58m
kube-system     deployment.apps/coredns                    2/2     2            2           7h58m
lr-labs         deployment.apps/registry-rottigni          1/1     1            1           7h54m
lr-labs         deployment.apps/sandbox-rottigni           1/1     1            1           7h54m

NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      7h58m
ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   7h58m
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      7h58m
kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       7h58m
lr-labs         service/registry-rottigni                    NodePort    10.106.231.17    <none>        5000:31673/TCP               7h54m
lr-labs         service/sandbox-rottigni                     NodePort    10.111.173.239   <none>        80:31607/TCP                 7h54m

NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.rottigni.tech,sandbox.rottigni.tech   149.62.189.139   80, 443   7h54m

NAMESPACE       NAME                                           ENDPOINTS                                            AGE
default         endpoints/kubernetes                           149.62.186.95:6443                                   7h58m
ingress-nginx   endpoints/ingress-nginx-controller             10.44.0.2:443,10.44.0.2:80                           7h58m
ingress-nginx   endpoints/ingress-nginx-controller-admission   10.44.0.2:8443                                       7h58m
kube-system     endpoints/kube-dns                             10.32.0.2:53,10.32.0.3:53,10.32.0.2:53 + 3 more...   7h58m
lr-labs         endpoints/registry-rottigni                    10.44.0.3:5000                                       7h54m
lr-labs         endpoints/sandbox-rottigni                     10.44.0.1:80                                         7h54m

NAMESPACE   NAME                         STATUS   ROLES           AGE     VERSION
            node/epsilon-rottigni-tech   Ready    control-plane   7h58m   v1.28.2
            node/zeta-rottigni-tech      Ready    <none>          7h57m   v1.28.2

One strange thing is that my ingress has IP 149.62.189.139, which is my worker node IP (not control plane IP). Is this the correct behavior?

Despite this, I'm unable to reach my services from the public DNS through my nginx ingress controller. I can, however, correctly reach both services using NodePort from my control plane and worker node SSH sessions, eliminating any application errors.

It appears that my ingress isn't facing the internet on the correct IP. I also checked my ingress-nginx-controller logs, which seem fine:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.5
  Build:         f503c4bb5fa7d857ad29e94970eb550c2bc00b7c
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0105 14:11:47.972914       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0105 14:11:47.973187       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0105 14:11:47.983447       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.5" state="clean" commit="506050d61cf291218dfbd41ac93913945c9aa0da" platform="linux/amd64"
I0105 14:11:48.470371       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0105 14:11:48.511293       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0105 14:11:48.529226       7 nginx.go:260] "Starting NGINX Ingress controller"
I0105 14:11:48.539963       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a5edeb27-12a0-4576-ab70-19d8cffb4dfe", APIVersion:"v1", ResourceVersion:"338", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0105 14:11:49.635029       7 store.go:440] "Found valid IngressClass" ingress="lr-labs/lr-labs-ingress" ingressclass="nginx-ingress-class"
I0105 14:11:49.636090       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"lr-labs", Name:"lr-labs-ingress", UID:"5663167b-9b94-4cb6-8670-5e5e1076bfaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1159", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0105 14:11:49.638242       7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/registry-ssl"
I0105 14:11:49.641278       7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/sandbox-ssl"
I0105 14:11:49.731218       7 nginx.go:303] "Starting NGINX process"
I0105 14:11:49.731681       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0105 14:11:49.732611       7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0105 14:11:49.733509       7 controller.go:190] "Configuration changes detected, backend reload required"
I0105 14:11:49.736538       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-685dff585b-2s8jp"
I0105 14:11:49.849528       7 controller.go:210] "Backend successfully reloaded"
I0105 14:11:49.849813       7 controller.go:221] "Initial sync, sleeping for 1 second"
I0105 14:11:49.850019       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-86968d59df-vdv9f", UID:"7ac8215f-175f-4496-90a1-dd63d6acad4e", APIVersion:"v1", ResourceVersion:"6819", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0105 14:12:41.662980       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-86968d59df-vdv9f"
I0105 14:12:41.663007       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
10.32.0.1 - - [05/Jan/2024:14:36:35 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.81.0" 81 0.000 [] [] - - - - 479859864f23ba0523fb9c29c1932d56
10.32.0.1 - - [05/Jan/2024:14:38:30 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - e262c16d9495ef00f412f0ba73dbb453
10.32.0.1 - - [05/Jan/2024:15:25:27 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - 9b662b33bb59be1846863af658f251ab
10.32.0.1 - - [05/Jan/2024:16:08:32 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - a54eda51703800c9a1726a9e1187988d

I even see some manually initiated curl requests for debugging directly to the nginx service using the Host HTTP header.

Additionally here my working curl commands to my cluster service resources:

root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# kubectl get svc -A
NAMESPACE       NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      8h
ingress-nginx   ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   8h
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      8h
kube-system     kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       8h
lr-labs         registry-rottigni                    NodePort    10.106.231.17    <none>        5000:31673/TCP               8h
lr-labs         sandbox-rottigni                     NodePort    10.111.173.239   <none>        80:31607/TCP                 8h
root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.106.231.17:5000
root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:80
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:443
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.111.173.239:80
<html>
  <head>
    <style>
      html {
        background: url(./Cheddar.jpg) no-repeat center center fixed;
        -webkit-background-size: cover;
        -moz-background-size: cover;
        -o-background-size: cover;
        background-size: cover;
      }

      h1 {
        font-family: Arial, Helvetica, sans-serif;
        background: rgba(187, 187, 187, 0.5);
        width: 6em;
        padding: 0.5em 1em;
        margin: 1em;
      }
    </style>
  </head>
  <body>
    <h1>Cheddar</h1>
  </body>
</html>root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s#

Could someone please help me debug this problem? Am I missing an important step for this kind of production deployment?

Thank you everybody!

UPDATE:

Given the following situation:

kubectl get svc,ing -A
NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      23h
ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   23h
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      23h
kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       23h
lr-labs         service/registry-rottigni                    NodePort    10.106.231.17    <none>        5000:31673/TCP               23h
lr-labs         service/sandbox-rottigni                     NodePort    10.111.173.239   <none>        80:31607/TCP                 23h

NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.rottigni.tech,sandbox.rottigni.tech   149.62.189.139   80, 443   23h

I am able to reach nginx from internet like this:

But I can't like this:

Improve this question
0

1 Answer 1

Reset to default
0

Basically you did everything correctly. The chosen way to make your service reachable from the outside does not allow you to use specific, low, ports in order to keep isolation between services. You have two options:

  1. use a service of type LoadBalancer which will allocate a separate IP for the service, providing isolation between services and allowing you a specific port like 80 or 443
  2. modify the ingress deployment and use a hostPort for the nginx containers, you can then use the node IPs where a pod is running. This can lead to a lot of headaches (f.e. when updating the deployment) but might be an easy solution to get started
Improve this answer
2
  • Thanks for you answer, I fixed my problem using MetalLB with a super basic config. I misunderstood the ingress resource behaviour because of his IP address… I still don’t understand why the ingress resource has an IP assigned since its only purpose is to provide an nginx configuration to the ingress-nginx-controller service. Everything is now working as expected, thanks anyway for your help!
    – Lorenzo Rottigni
    Commented Jan 9 at 7:48
  • If you are using MetalLB and It is configured correctly it should provide you with a dedicated IP as it is a LoadBalancer implementation. Please accept the answer if it solved your issue.
    – Thomas
    Commented Jan 9 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.