How to implement simple distributed configuration system

Question

I have multiple servers running in data centers around the world. Each server has its own configuraiton database, and cli command needs to be executed to update in-memory cache.

I want to automate this system. I have following requirements:

1. Single source of truth (i.e. git repository)
2. No drift in the system (inserts/updates/deletes are applied to databases as required)
3. Feedback that everything was executed correctly (i.e. if some db replication of cli commands fails, I want to know that)

Please note I would like to store configuration as "Desired state", not as migrations to database which insert additional records.

Right now process if following:

1. Create commmit to source-of-truth repository
2. Connect to database
3. Update configuration in database (this is usually lots of records)
4. Ensure configuration is applied correctly (select and verify)
5. Connect to service via ssh
6. Execute command to reload configuration
7. Verify if configuration is reloaded (i.e. run command to dump config and grep new configuraiton entries)
8. Repeat steps 2-7 for multiple datacenters.

I've evaluated some existing tooling, and I cannot find anything good:

• Infrastructure management tools like terraform are not good for managing contents of databases, and they cannot really call scripts
• Configuration management tools like ansible are more focused on executing scripts, rather than synchronizing state - i.e. I can apply DB migraiton, but this is very cumbersome (btw, I don't want to configure server instances, but software running on these server instances).

Can you suggest how to automate this system?

Answer 1

You can use Flyway for database migrations and you could wrap actual migrations as releases.

So something like you have a Flyway container running as a Cron job and if there is new release - it would be applying migrations from the new releases, otherwise it would just use existing migrations - meaning not change the state.

Then you can use Reliza Hub - the tool I am developing - to manage rollout of these migrations to different instances - see a sample here on how promotion logic may work - https://worklifenotes.com/2023/05/12/how-to-promote-software-to-production-with-reliza-hub/

Reliza Hub can also give you Slack / MS Teams notifications of the releases in progress and if there were errors.

You could also add your additional verification steps in the end of each cronjob, although Flyway is already capable to provide feedback if migrations passed or failed.

I hope this gives basic idea on how to automate your process - but feel free to reach out to me with follow up questions.