13

Consider this scenario (any comparison with real world situations is purely by accident):

  • 3:07 am: incoming support call "Something in production went down, I need your help!".
  • 3:12 am: connected to the system (logon accepted) ... and no time for coffee.
  • 3:15 am: lucky you, right away you could spot the issue via some error message somewhere.
  • 3:17 am: use your SCM toolbox to grab the code, fix the issue, test it, great ... my fix works!
  • 3:20 am: get in touch with the DevOps-team to ship the fix and to get production running again.
  • 3:21 am: red flag ... "To respect , we need 2 more eyes to get approval for this fix".
  • 3:22 am: ggggrrrreat, now what, who else can we call (= wake up some manager)?

If you implemented some approval procedure similar to my answer to "What are possible implementations (or examples) of the four-eyes principle?", then you're out of luck ... here are your choices:

  • Your fix will be stuck (read: production will be down) until 2 more eyes got involved.
  • You figure out a way to get around the missing eyes.

So how to implement the four-eyes principle for emergency fixes? ... So that you get production up and running asap, i.e. around 3:25 am ... And so that you can also close the call (and go back to where you came from)?

  • You did contact a team, which means they should have already blessed the patch with respect to the approbation principles in place. I really start to hate those rhetorical questions :( just my opinion, don't bother with it too much – Tensibai Mar 8 '17 at 19:58
  • @Tensibai how can one possibly "have blessed some patch" (or fix) upfront, not knowing what the cause of the problem was when "you were contacted"? Also, can you be more specific about the rhetorical? Not a good fit, something else? – Pierre.Vriens Mar 8 '17 at 20:09
  • I mean you say you were able to contact the team at 3:20, which means it's not just you pushing the fix. I use rhetorical as 'hypothetical case, based from experience or not, where you already know which answer you're waiting for'. More or less my concerns on meta I feel alone not interested by this 'generic principles' Q/A so I'm probably wrong. What I'm sure off is that I won't visit twice a Q/A about generic principles if I was an outlander of this beta. – Tensibai Mar 8 '17 at 20:18
  • I may say the same about the generic Jenkins questions if they were coming now – Tensibai Mar 8 '17 at 20:20
  • Commitments won't be computed until public beta, for now in private we're building what we 'committed' users feel are exemplary questions for this site, and I think we have bunch of work in the remaining 12 days, or I may just have to go through the 7 stages of grief – Tensibai Mar 8 '17 at 21:00
8

In the SCM-world where I'm mostly familiar with, the above scenario is typically addressed by what's called the "abbreviated-approval list procedure.

Here is a blueprint of it:

  • Define your business hours, say from 8 am to 6 pm.
  • Define a complete approval list of (say) 3 levels of approval (for roles X, Y and Z).
  • Define an abbreviated approval list of (say) only 1 level of approval (only for roles X).
  • Planned changes always require all approvals from the complete approval list.
  • For Unplanned changes, the complete approval list is used also to gather the required approvals, provided the approvals are to be issued during the defined business hours.
  • For any approvals of unplanned changes that are to be issued outside the defined business hours:
    • Only the approvals from the abbreviated approval list (such as role X above) are required to authorize the change. And after the authorization by the abbreviated approval list is given, the deployment of the change (in the target environment) will actually be performed.
    • But additional post-approvals will be needed afterwards (within a reasonable amount of hours/days), i.e from all roles contained in the complete approval list (such as role Y and Z above), which are not also contained in the abbreviated approval list (such as role X above). And if within the (upfront) agreed amount of hours/days not all post-approvals have been issued (e.g because the fix worked "this" time, but was only like a temporary fix), then the change might be subject to a rollback. While there is at least 1 outstanding post-approval, the change is marked as "waiting post approvals".

With such solution in place, the call can be closed around 3:23 am ... since there will be no more red flag at 3:21 am ... ggggrrreat, time for a beer to celebrate my fix to get production going again (instead of coffee) ... and fingers crossed the outstanding post approvals will come in soon ...

3

In the case of off-hours emergency fixes, it's more practical to require less sign-off for changes than your normal procedure. Generally, you can deploy a fix and then do post-approvals the next business day. If the fix is not approved, it can be reverted and replaced with a permanent solution.

During an outage situation, the number one priority should be to restore service. If your organization does not recognize this relaxed process during an outage, then yes, your only option is to start waking more people up for sign off.

  • I agree with your recommendation, which seem to be similar to my own recommendation (answer). Can you think of some example in the SCM world that you are familiar with, and HOW you'd implement it there? If so, can you expand on that in your answer? – Pierre.Vriens Mar 9 '17 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.