While this may be marked "opinion-based", I hope that raise some reliable and tested patterns in allowing developers to run tasks on deployed instances. The deployment and operating principles are:
- There is only one way to deploy applications: our CI/CD pipeline.
- A single pipeline of tools acts on behalf of the developers and operators to test that the application is production ready and then conducts the deployment.
- In principle only changes to code produce changes to deployed state.
- Corollary: everything is code - tests, infrastructure, configuration, etc are all expressed as code.
- Infrastructure and instances are immutable
- interactive shell access to production instances is in principle denied.
- Metrics and configuration are agreed up front with the developers. Configuration is applied before the bake of an image.
- Logs, monitoring and instrumentation is visualised in an common dashboard.
- Admin and batch processes are defined before deployment and should be well-understood and agreed upon by ops and dev teams.
The problem
There are some workloads which will require batch tasks to be run in an unpredictable way - we cannot predict at what frequency (bye, cron) and with what parameters they will need to be run.
The simplest way to solve this problem is to allow access to the production instances to allow someone to interactively run the script. However this breaks our design principles.
Although there are some other ways around this (👀 Rundeck), I would like answers related to patterns.
What pattern of ad-hoc processing works when your instances are immutable?
