Deploy two independently hosted services in production at once without any downtime

Question

I have two different services. Let's say Service A and Service B. Service B is dependent on Service A. Both code bases are hosted in separate repo in Bitbucket and code bases are maintained by different teams. Service A has three backend servers and Service B has seven backend servers. I use Jenkins as a CI for code deployment. So, whenever we merge the code in master branch, Jenkins start the deployment. The process is same for Service A and Service B but they have two different Jenkins server.

Now, problem is that if Service A team deploy something in production independently with major changes in code base. Then whole application will break. Because Service B is not ready yet to compatible with the code changes of Service A. Let's say, parameters of a function has been changed where Service B is expecting a different parameter.

I am looking for a solution, where Service A code merge in master gonna happen only it is time for deployment of Service B. So, every deployment of Service B, it first deploys the code of Service A and then follows with Service B. This whole deployment will take a lot of time. Main application should not be down by this time (because once Service A is deployed then user has to wait for Service B deployment, otherwise they will face errors).

So, I am thinking if it's possible where new deployment will happen in a separate instance When deployment is completed in that instance then it will replace the previous instance. So, code will be updated in main production without any down time or error.

What is the recommended way to achieve this ? Can I do something with Ansible in this case ?

Answer 1

I think you're taking the problem on the wrong side, you may start a whole new infrastructure aside and then switch loadbalancers from blue to green, loosing sessions, breaking ongoing transactions etc.

The proper way to achieve what you're doing is coding differently, when A relasease a breaking API ni version n, a should be able to continue to answer call to api in version n-1. the usual method is to have the version in the uri or an header, for exemple:

<a-host>/api/1/getCustomers and <a-host>/api/2/getCustomerList (for exemple if n is 2). With that, you can do a rolling update of B, with backends still using api v1 and new ones using api v2.

Answer 2

I will make the following assumptions:

• Service B communicates with Service A using any Application layer protocol (such as HTTP)
• Service B is the only consumer of Service A
• Users only interacts with Service B

Solution with down time

When upgrading Service A with breaking changes, move these changes to another "version".

The goal is for you to proactively create a new interface version (i.e. API version) that will implement the breaking changes. In some situation, this might require heavy refactoring. In other (depending if your code base was properly layered), not. Then you can start implementing "contract" tests between Service A and B to make sure A doesn't break the contract with B.

The problem here is that when deploying this solution, for a short while you will not have access to neither API.

Solution without down time

Make the same change as the solution above, but also consider a deployment strategy where Service A.1 is always up while we introduce Service A.2. When no requests to A.1 is incoming, it may go down to leave only A.2 alive to serve all the requests. This is the blue-green deployment strategy.

This is more a cherry on the top if your overall solution requires minimal downtime deployment.