Scaling a stateful service which has primary/secondary architecture by setting a property

Question

I have a java web stateful service which connects to mongodb and currently it is being deployed as a jar directly on the VM. This service has a property in properties file i.e. primary.

If we set primary to true then it is primary instance of the application and on this instance mongodb aggregation will happen.

If we set primary to false then it is secondary instance of the application and on this instance no mongodb aggregation will happen but it will serve all read and write queries.

Now we are migrating towards k8s deployment.

Challenges are :

How do we set the value of primary during the k8s pod deployment having multiple replicas ?

How do we handle the failure of primary instance of application pod ?

I am open for any architectural changes suggestions in order to facilitate better deployment structure.

Thanks in advance.

Answer 1

You are looking for the leader election pattern. This can be implemented in many different ways. It typically means that an instance take a lock in some form of persistent storage, e.g. a PostgreSQL database, Redis or in etcd itself.

Also see Java Leader Election Pattern and Leader Election in distributed systems