Kubernetes deployment with multiple containers

Question

I have two containers, worker and dispatcher. I want to be able to deploy N copies of worker and 1 copy of dispatcher to the cluster. Is there a deployment that will make this work?

dispatcher needs to be able to know about and talk to all of the worker instances (worker is running a web server). However, the external world only needs to talk to dispatcher.

As a stretch goal, I'd like to be able to scale N up and down based on demand.

Questions:

1. Should I be using a Deployment or a StatefulSet? or something else?
2. Do I actually need multiple deployments? Can containers from multiple deployments talk to each other?
3. How do I dynamically scale this?

It seems like I can get partway there with a single Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      deploy: myapp
  template:
    metadata:
      labels:
        deploy: myapp
    spec:
      containers:
        - name: worker
          image: localhost/worker:latest

I expect this will give me 3 workers, but 0 dispatchers. However, if I add dispatcher to the containers list, I would expect to get 3 of those too, which is not what I want.

Answer 1

I have two containers, worker and dispatcher. I want to be able to deploy N copies of worker and 1 copy of dispatcher to the cluster. Is there a deployment that will make this work?

Both a Deployment or Statefulset will work.

dispatcher needs to be able to know about and talk to all of the worker instances (worker is running a web server).

This has to do with Kubernetes Networking plugins and will depend on which plugin your cluster was configured with. Most clusters I have used by default allow traffic across the entire cluster.

However, the external world only needs to talk to dispatcher.

The main ways to do this would be a Service or an Ingress both are explained here.

I'd like to be able to scale N up and down based on demand.

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Should I be using a Deployment or a StatefulSet? or something else?

Does your container have state? If so then use a stateful set. The Kubernetes page explains why to use a statefulset https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

Do I actually need multiple deployments? Can containers from multiple deployments talk to each other?

These things are not really related. You would use multiple deployments so you can configure multiple workloads differently.