Should worker processes of web applications be placed in separate containers?

Question

My setup is based on a Flask app that has a PostgreSQL and Redis database. In addition, a worker process is started to handle asynchronous tasks such as sending emails. I use Gunicorn to create multiple app processes. Passing of jobs from the app processes to the worker process is done via Redis.

The question is, should the worker process be started in a separate container despite being based on the same code? Currently I use a start script with the relevant bit:

gunicorn --bind=0.0.0.0:8000 --workers=3 manage:app &
python -u manage.py run_worker

Also if they are to be separated, how can I ensure that they reuse the common file system container? Additionally, when scaling either Gunicorn with the main app or the worker processes, should additional instances of the container be created or the number of processes within a container be increased?

Related: What are the advantages of dockerizing nginx and php in different containers?

Answer 1

I can think of only one case where overhead of running workers in separate containers is justified: if your setup uses docker swarm for clustered deployment. This way you will get all of the HA benefits.

Otherwise I don't see a reason to complicate such tasks, especially if they must use strictly the same codebase (which leads me to believe that they share one domain of concerns, so to say). It might seem elegant, but I just don't see enough benefits

p.s. any split of such kind will force you to manage interconnection and complicate setup for local development