My setup is based on a Flask app that has a PostgreSQL and Redis database. In addition, a worker process is started to handle asynchronous tasks such as sending emails. I use Gunicorn to create multiple app processes. Passing of jobs from the app processes to the worker process is done via Redis.
The question is, should the worker process be started in a separate container despite being based on the same code? Currently I use a start script with the relevant bit:
gunicorn --bind=0.0.0.0:8000 --workers=3 manage:app &
python -u manage.py run_worker
Also if they are to be separated, how can I ensure that they reuse the common file system container? Additionally, when scaling either Gunicorn with the main app or the worker processes, should additional instances of the container be created or the number of processes within a container be increased?
Related: What are the advantages of dockerizing nginx and php in different containers?