7

My setup is based on a Flask app that has a PostgreSQL and Redis database. In addition, a worker process is started to handle asynchronous tasks such as sending emails. I use Gunicorn to create multiple app processes. Passing of jobs from the app processes to the worker process is done via Redis.

The question is, should the worker process be started in a separate container despite being based on the same code? Currently I use a start script with the relevant bit:

gunicorn --bind=0.0.0.0:8000 --workers=3 manage:app &
python -u manage.py run_worker

Also if they are to be separated, how can I ensure that they reuse the common file system container? Additionally, when scaling either Gunicorn with the main app or the worker processes, should additional instances of the container be created or the number of processes within a container be increased?

Related: What are the advantages of dockerizing nginx and php in different containers?

6

I can think of only one case where overhead of running workers in separate containers is justified: if your setup uses docker swarm for clustered deployment. This way you will get all of the HA benefits.

Otherwise I don't see a reason to complicate such tasks, especially if they must use strictly the same codebase (which leads me to believe that they share one domain of concerns, so to say). It might seem elegant, but I just don't see enough benefits

p.s. any split of such kind will force you to manage interconnection and complicate setup for local development

  • I guess the complexity can be handled well in my case as I already employ docker-compose. Nonetheless, your point is valid since the load can currently be handled with a single instance and no need to make things more complex than they have to be – Moritz Jan 16 '18 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.