Is there some metadata that I can apply, or some DNS that I can query, such that when a pod is started, the addreses of all the other pods in that statefulset will be somehow made available to the newly started pod?
I want to run a stateful application (Riak) where every node (pod) in the Riak cluster needs to be aware of the addresses of every other node (pod).
Using the kubernetes statefulset controller, pods are started one by one.
I also wonder how I can somehow listen for some event in each Riak pod so that as instances are started, all the preceeding instances will be made aware of the new instance?
Following on from the information provided by @kaizenCoder - I modified the service so it had a configuration of :
spec: clusterIP: None
I then scaled up to two replicas, exec'd to one of the containers and ran
dig again as mentioned.
root@grafana-0:/usr/share/grafana# dig SRV grafana.default.svc.cluster.local ; <<>> DiG 9.10.3-P4-Debian <<>> SRV grafana.default.svc.cluster.local ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27002 ;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;grafana.default.svc.cluster.local. IN SRV ;; ANSWER SECTION: grafana.default.svc.cluster.local. 5 IN SRV 0 50 55555 10-1-0-77.grafana.default.svc.cluster.local. grafana.default.svc.cluster.local. 5 IN SRV 0 50 55555 10-1-0-78.grafana.default.svc.cluster.local. ;; ADDITIONAL SECTION: 10-1-0-78.grafana.default.svc.cluster.local. 5 IN A 10.1.0.78 10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77 ;; Query time: 0 msec ;; SERVER: 10.96.0.10#53(10.96.0.10) ;; WHEN: Fri Jun 28 12:12:32 UTC 2019 ;; MSG SIZE rcvd: 372
Much better - but unlike @kaizenCoder - I don't get back a name for each of the pods - it just uses the IP address - the names don't match - I would have expected the first pod to have had a name of
grafana-0.grafana.default.svc.cluster.local - that nodes hostname is correct though -
Any suggestions on how to get the DNS to match the hostname ?
Update again - more weirdness - the other pods don't seem to be resolvable by hostname.
root@grafana-0:/usr/share/grafana# ping grafana-1 ping: unknown host
Presumably hostname only resolves in the pod as hostname resolves to 127.0.0.1 ... Which makes it a bit worthless tbh.
I feel like I've made some progress but still missing a piece. When a container starts up, how can software within it resolve :
- It's own fully qualified name
- The fully qualified names of all the other statefulset pods
I could do something like this to determine the node name and the other node names but it feels ugly.
root@grafana-0:/usr/share/grafana# dig SRV *.default.svc.cluster.local | grep 0-77 *.default.svc.cluster.local. 5 IN SRV 0 11 55555 10-1-0-77.grafana.default.svc.cluster.local. 10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77 root@grafana-0:/usr/share/grafana# ip addr | grep eth0 | grep inet inet 10.1.0.77/16 scope global eth0