kubernetes statefulset - communicate addresses of other pods when pod launches?

Question

Is there some metadata that I can apply, or some DNS that I can query, such that when a pod is started, the addreses of all the other pods in that statefulset will be somehow made available to the newly started pod?

I want to run a stateful application (Riak) where every node (pod) in the Riak cluster needs to be aware of the addresses of every other node (pod).

Using the kubernetes statefulset controller, pods are started one by one.

I also wonder how I can somehow listen for some event in each Riak pod so that as instances are started, all the preceeding instances will be made aware of the new instance?

---

Following on from the information provided by @kaizenCoder - I modified the service so it had a configuration of :

spec:
  clusterIP: None

I then scaled up to two replicas, exec'd to one of the containers and ran dig again as mentioned.

root@grafana-0:/usr/share/grafana# dig SRV grafana.default.svc.cluster.local

; <<>> DiG 9.10.3-P4-Debian <<>> SRV grafana.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27002
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grafana.default.svc.cluster.local. IN  SRV

;; ANSWER SECTION:
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-77.grafana.default.svc.cluster.local.
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-78.grafana.default.svc.cluster.local.

;; ADDITIONAL SECTION:
10-1-0-78.grafana.default.svc.cluster.local. 5 IN A 10.1.0.78
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 28 12:12:32 UTC 2019
;; MSG SIZE  rcvd: 372

Much better - but unlike @kaizenCoder - I don't get back a name for each of the pods - it just uses the IP address - the names don't match - I would have expected the first pod to have had a name of grafana-0.grafana.default.svc.cluster.local - that nodes hostname is correct though - grafana-0.

Any suggestions on how to get the DNS to match the hostname ?

---

Update again - more weirdness - the other pods don't seem to be resolvable by hostname.

root@grafana-0:/usr/share/grafana# ping grafana-1
ping: unknown host

Presumably hostname only resolves in the pod as hostname resolves to 127.0.0.1 ... Which makes it a bit worthless tbh.

I feel like I've made some progress but still missing a piece. When a container starts up, how can software within it resolve :

1. It's own fully qualified name
2. The fully qualified names of all the other statefulset pods

I could do something like this to determine the node name and the other node names but it feels ugly.

root@grafana-0:/usr/share/grafana# dig SRV *.default.svc.cluster.local | grep 0-77 
*.default.svc.cluster.local. 5  IN      SRV     0 11 55555 10-1-0-77.grafana.default.svc.cluster.local.
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77
root@grafana-0:/usr/share/grafana# ip addr | grep eth0 | grep inet
    inet 10.1.0.77/16 scope global eth0

Answer 1

Yes you can by doing a DNS SRV lookup. Here's an example:

$ kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV jira-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> SRV jira-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;jira-headless.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-0.jira-headless.default.svc.cluster.local.
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-1.jira-headless.default.svc.cluster.local.

;; ADDITIONAL SECTION:
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

;; Query time: 6 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Fri Jun 28 00:18:14 UTC 2019
;; MSG SIZE  rcvd: 143

In the example above I have a headless Service called jira-headless. If you pay attention to ADDITIONAL SECTION: you'll notice it provides the SRV record in the following format:

_service._proto.name. TTL class SRV priority weight port target.

Which maps to the Pods (along with their IPs) backing the Service:

jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188