I have multiple servers running in data centers around the world. Each server has its own configuraiton database, and cli command needs to be executed to update in-memory cache.
I want to automate this system. I have following requirements:
- Single source of truth (i.e. git repository)
- No drift in the system (inserts/updates/deletes are applied to databases as required)
- Feedback that everything was executed correctly (i.e. if some db replication of cli commands fails, I want to know that)
Please note I would like to store configuration as "Desired state", not as migrations to database which insert additional records.
Right now process if following:
1. Create commmit to source-of-truth repository
2. Connect to database
3. Update configuration in database (this is usually lots of records)
4. Ensure configuration is applied correctly (select and verify)
5. Connect to service via ssh
6. Execute command to reload configuration
7. Verify if configuration is reloaded (i.e. run command to dump config and grep new configuraiton entries)
8. Repeat steps 2-7 for multiple datacenters.
I've evaluated some existing tooling, and I cannot find anything good:
- Infrastructure management tools like terraform are not good for managing contents of databases, and they cannot really call scripts
- Configuration management tools like ansible are more focused on executing scripts, rather than synchronizing state - i.e. I can apply DB migraiton, but this is very cumbersome (btw, I don't want to configure server instances, but software running on these server instances).
Can you suggest how to automate this system?