0

Is there some metadata that I can apply, or some DNS that I can query, such that when a pod is started, the addreses of all the other pods in that statefulset will be somehow made available to the newly started pod?

I want to run a stateful application (Riak) where every node (pod) in the Riak cluster needs to be aware of the addresses of every other node (pod).

Using the kubernetes statefulset controller, pods are started one by one.

I also wonder how I can somehow listen for some event in each Riak pod so that as instances are started, all the preceeding instances will be made aware of the new instance?


Following on from the information provided by @kaizenCoder - I modified the service so it had a configuration of :

spec:
  clusterIP: None

I then scaled up to two replicas, exec'd to one of the containers and ran dig again as mentioned.

root@grafana-0:/usr/share/grafana# dig SRV grafana.default.svc.cluster.local

; <<>> DiG 9.10.3-P4-Debian <<>> SRV grafana.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27002
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grafana.default.svc.cluster.local. IN  SRV

;; ANSWER SECTION:
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-77.grafana.default.svc.cluster.local.
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-78.grafana.default.svc.cluster.local.

;; ADDITIONAL SECTION:
10-1-0-78.grafana.default.svc.cluster.local. 5 IN A 10.1.0.78
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 28 12:12:32 UTC 2019
;; MSG SIZE  rcvd: 372

Much better - but unlike @kaizenCoder - I don't get back a name for each of the pods - it just uses the IP address - the names don't match - I would have expected the first pod to have had a name of grafana-0.grafana.default.svc.cluster.local - that nodes hostname is correct though - grafana-0.

Any suggestions on how to get the DNS to match the hostname ?


Update again - more weirdness - the other pods don't seem to be resolvable by hostname.

root@grafana-0:/usr/share/grafana# ping grafana-1
ping: unknown host

Presumably hostname only resolves in the pod as hostname resolves to 127.0.0.1 ... Which makes it a bit worthless tbh.

I feel like I've made some progress but still missing a piece. When a container starts up, how can software within it resolve :

  1. It's own fully qualified name
  2. The fully qualified names of all the other statefulset pods

I could do something like this to determine the node name and the other node names but it feels ugly.

root@grafana-0:/usr/share/grafana# dig SRV *.default.svc.cluster.local | grep 0-77 
*.default.svc.cluster.local. 5  IN      SRV     0 11 55555 10-1-0-77.grafana.default.svc.cluster.local.
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77
root@grafana-0:/usr/share/grafana# ip addr | grep eth0 | grep inet
    inet 10.1.0.77/16 scope global eth0
  • "I then scaled up to two replicas". Is grafana a StatefulSet, Deployment or something else? – kaizenCoder Jun 29 at 1:30
  • To clarify further, from what I've seen you will get the IPs if you are using a Deployment (by default). Only StatefulSets will provide a Pod name with the ordinal index. – kaizenCoder Jun 29 at 1:56
  • @kaizenCoder - grafana is just something I happened to have laying around. The real reason I'm looking for this metadata is for distributed Elixir or Riak nodes. Each node that starts up needs to be aware of its own FQDN and that of the other nodes currently active in the set – bryan hunt Jun 30 at 16:19
  • Maybe this is a bug - I've just been reading github.com/kubernetes/kubernetes/issues/57671 and it seems people have had similar issues. – bryan hunt Jun 30 at 16:27
0

Yes you can by doing a DNS SRV lookup. Here's an example:

$ kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV jira-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> SRV jira-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;jira-headless.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-0.jira-headless.default.svc.cluster.local.
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-1.jira-headless.default.svc.cluster.local.

;; ADDITIONAL SECTION:
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

;; Query time: 6 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Fri Jun 28 00:18:14 UTC 2019
;; MSG SIZE  rcvd: 143

In the example above I have a headless Service called jira-headless. If you pay attention to ADDITIONAL SECTION: you'll notice it provides the SRV record in the following format:

_service._proto.name. TTL class SRV priority weight port target.

Which maps to the Pods (along with their IPs) backing the Service:

jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.