Blue Green Deployment DB transactions management

Question

Imagining I have 2 sets of environment(Blue and Green)

           +
           |
           |
           |
           v
   +-------+---------+
   |     Proxy/LB    |
   |                 |
   +--+-------------++
      |             |
+-----++          +-+----+
|      |          |      |
|APP v1|          |APP v2|
|Blue  |          |Green |
+--+---+          +--+---+
   |                 |
+--+---+          +--+---+
| DB   |          | DB   |
| Blue | DB Sync  | Green|
|      <---------->      |
+------+          +------+

The version Blue is currently running and green is deployed and all DB replication completed. Now I want switch traffic to green environment. In this case, what is the best way to manage the DB transactions?(I have one thought below)

• Forward the new traffic to green and wait for existing blue's environment's transactions complete. Once all are done, switch traffic completely to green(UPDATE: Can I achieve this with Nginx?)

If above is true, I'd have to do little extra work to forward only new traffic to green.

Or what is the best way of doing Blue Green deployment in this case?

PS:

• We are using Nginx proxy
• All servers are bare metal, no cloud!
• 2 standalone DB setups(Postgresql with master-slave)

Answer 1

As per Martin Flowers' definition of Blue-Green deployment

The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.

Steps

You've said that DBs are synced using master-slave topology. So, to switch traffic from one instance to another, we have to make steps:

• Switch web-traffic from APP v1 to APP v2
• Shutdown deprecated DB Blue
• Promote DB Green to master mode
• Start new DB Blue v2 in slave mode (you may ask - why shutdown DB Blue instead of using master-master replication or other ways to sync DB Blue v1 with DB Green in master mode. See explanation in DB replication)

Graceful switch web-traffic

To switch web-traffic from APP v1 APP v2 - just use nginx reload - it will graceful end all working connection and move traffic from App v1 to App v2 There are many scripting solutions for switch blue/green using shell/python scripts

E.g.:

• meappy/docker-nginx-blue-green: Docker blue green upstreams for nginx with ngx_http_perl_module

• Simple 0-Downtime Blue Green Deployments: Python simple 0-Downtime Blue Green Deployments example

The logic is simple: add green upstream in nginx config, use it as default and send reload to nginx:

Sample nginx.conf for reverse proxy/load balancer

http {

upstream appv1 {
    zone appv1 64K;
    server 10.10.0.1:80;
}

upstream appv2 {
    zone appv2 64K;
    server 10.10.0.3:80;
}

server {
...
location / {
    proxy_pass http://$appv1;
}
}

}

Replace proxy_pass http://$appv1; with proxy_pass http://$appvv;

And then

nginx -s reload

Stop DB Blue

systemctl stop postgresql

Promote DB Green to master mode

• Trigger the promotion using your defined trigger in $PGDATA/recovery.conf

touch $PGDATA/failover

• Remove failover trigger from Green DB

cd $PRIMARY_DATA
rm -f recovery.* failover  

• Make sure that hot stanby mode is on on Green DB in master mode

cat postgresql.conf | grep '#hot_standby = on'

DB replication

Make a backup of master Green DB

psql -c "SELECT pg_start_backup('Streaming Replication', true)" postgresql://postgres@GreenDB/postgres

Send a backup to Blue DB

rsync -a $PG_DATA_Green_DB/ $BlueDB_IP:$PG_DATA_Blue_DB/ --exclude postmaster.pid --exclude postmaster.opts 

See also

• Evolution of Fault Tolerance in PostgreSQL

DB Migration (More complex but more robust database migration scheme)

You may use more complex scheme for migration using Ansible and `Pglupgrade tool:

See for details:

• Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part I)

• Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part II)

P.s some definitions.

Just to...

... sort the buyers from the spyers, the needy from the greedy...

And blue/green deployment vs canary deployment vs A/B test

Definitions, kinds of deployments and tests

(see details in these answers )

• Blue-Green Deployment - When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.

• A/B Testing - Two versions of an application are running at the same time. A portion of requests go to each. Developers can then compare the versions.  

• Canary Release - A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.

• Feature flagging - The action of "configuring" (cold, or even hot) which functionality is (not)available for which (group) of users. If you also do something like "feature flagging" you can deploy first, measure soundness of your release in backwards compatibility/bug perspective, and release new functionality gradually to different users, or vice versa (scale down or even rollback functionality and/or binaries). Feature flagging allows for splitting availability of functionality from deployment of binaries, and gives much more fine-grained decision making then only "deploy/rollback"

P.P.S Blue/Green vs Canary

source

• Both blue-green and canary releases solve the same purpose

• Although both of these terms look quite close to each other, they have subtle differences. One put confidence in your functionality release and the other put confidence the way you release.

Blue-Green Deployment

When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.

1. It is more about the predictable release with zero downtime deployment.

2. Easy rollbacks in case of failure.

3. Completely automated deployment process

4. In cloud environment where it is easier to script & recreate infrastructure, blue/green deployment is preferred as it allows the infrastructure to be in sync with the automation

Canary Release

A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.

1. The canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure.

2. It is about to get an idea of how new version will perform (integrate with other apps, CPU, memory, disk usage, etc).

See also illustration (source):

Blue/Green:

Canary:

Answer 2

As promised, here is a solution using haproxy with sticky cookies. Note, that this approach requires support on the application side (as only your application layer may know once transactions are completed).

To achieve this, first - start with haproxy configuration of the following kind:

    global
        daemon
        maxconn 256
    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    frontend http-in
        bind *:80
        default_backend servers

    backend servers
        balance roundrobin
        cookie SID insert indirect preserve
        server server1 blue-server-hostname:8000 cookie sblue

Note that this configuration is pretty generic and taken from haproxy documentation - from here: https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#2.5

The part that is used to achieve stickiness is cookie instruction that is added in the end, detailed documentation here: https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4-cookie

So, with this line:

cookie SID insert indirect preserve

we tell haproxy to insert cookie called SID which may only have value sblue for now (this value is defined in last line of configuration). And we also say that if this cookie is supplied by the back-end server, we don't want haproxy to overwrite it (achieved by insert and preserve keywords).

The next part is to add some application level logic to set SID cookie to client and make its value configurable on the application side. You want to set SID cookie to sblue for your old code and to sgreen for your new code.

Now, when you do deployment, at first you have only blue instance, all your SID cookies are set to sblue and haproxy directs all your traffic to the only blue instance.

Next, you add your green instance, and after that you add it to haproxy by adding the following line to the end of configuration above:

server server2 green-server-hostname:8000 cookie sgreen

Now any request which gets to your new green instance would have its cookie set to sgreen and would stick to the new instance.

Finally, you want to engage your application-level configuration on the old blue instance to set cookie to sgreen to clients once their transaction is over (basically, at the point where application knows it's safe to do so). Then any new requests from such clients would be directed to green instance. At some point all your requests would be transitioned to green - and you can detach blue instance at that point.

This kind of approach (or its variations) would be the most graceful and controlled by your application-layer transaction logic.