What processes or tools enable Segregation of Duties when engineers both deploy and run code?

Question

In highly regulated environments, such as the Financial Services sector, Segregation of Duties is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.

Traditionally this has meant that Developers develop code and then hand it over to Operations however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

• In Google's Site Reliability Engineering, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.

• In the "You Build It, You Run It" model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy Sarbanes Oxley Section 404: Management Assessment of Internal Controls:

(a) Rules Required. The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--

(1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and

(2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.

(b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.

Based upon the comments it is important to call out a couple of assumptions I am making:

• I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value. This would be as opposed to commercial financial services which has a different transaction value profile.
• A financial institution's online offering will be made up of many components that have differing risk considerations:
  • Move Money - Moving money between accounts or transfers between accounts of different owners. An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.
  • Customer Acquisition - Less "Risky" as it has low transaction volumes compared to Move Money but still needs consideration.
  • Internet Banking - Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.
• Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.

TL;DR: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the Securities and Exchange Commission's regulations.

Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.

Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.

Answer 1

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

---

1. Architecture

---

Here are the highlights of how I'd answer your question:

• All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the library structure.
• For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: the server gets exclusive update authority to the entire library structure. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...
• The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a software change package.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the lifecycle of a software change package (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

• Only the server will execute the required (and preconfigured) steps.
• The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.
• The approvals can only be given by users who have a role that allows them (= permission) to issue such approvals.

---

2. Roles and permissions

---

The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

Step 1: Configure the permissions (in the security system)

• Define security entities in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

  • PrmUnit, used for getting permission to request a Promote to say Unit-testing.
  • PrmQA, used for getting permission to request a Promote to say Qa-testing (let's assume this is the highest level of testing).
  • PrdEnduser, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.
  • PrdRelmgnt, used by release managers to authorize an Activation in production (= the last/highest level in the library structure).

• Define groups of users in your security system. A few samples (add as many similar ones to fit your own needs):

  • GrpDevs, which (say) corresponds to your developers (probably more then just 1).
  • GrpEnduser, which (say) corresponds to your end-users (at least 1, preferably with more similar users).
  • GrpRelMgnt, which (say) corresponds to your release managers (at least 1, preferably a few more users).

• Grant permissions, also using your security system, to allow access to selected "security entities" for selected "groups of users". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

  • Group GrpDevs gets access to (only!) security entity PrmUnit.
  • Group GrpEnduser gets access to (only!) security entity PrdEnduser.
  • Group GrpRelMgnt gets access to (both!) security entity PrmQA and PrdRelmgnt.

Step 2: Configure the workflow (in the SCM system)

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.

Here are some examples of how you'd configure your SCM system to make some magic happen:

• If a user has access to PrmUnit, then such user is allowed to request a Promote to Unit-testing. Obviously, the users in group GrpDevs are the users authorized for this (note: not, e.g, the users in group GrpRelMgnt).
• If a user has access to PrmQA, then such user is allowed to request a Promote to QA-testing. Obviously, the users in group GrpRelMgnt are the users authorized for this (note: not, e.g, the users in group GrpDevs, or in group GrpEnduser).
• If a user has access to PrdEnduser, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group GrpRelMgnt to even be able to review a change). Obviously, the users in group GrpEnduser are the (only) users authorized for this.
• If a user has access to PrdRelmgnt, then such user is allowed to authorize an Activation in production (= the last/highest level in the library structure).

---

3. Expect the unexpected, and be ready for it

---

The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.

To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like

What happened when and why, and which authorized user actually approved it ... upfront?

But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.

---

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.

Answer 2

Answer based on my knowledge of French "Internal controls" regulation, kind of equivalent to the SEC regulations you point to, I assume linking here to a French legal text would not be really useful and I know of no good translation of it.

In an ideal 'You build it, you run it' model, everyone in the team will be accountable for the change. The risk assessment can't be enforced by a separation of duties and the only way I know to keep conforming with regulation is to have a periodic short cycle audit of transactions along with an unalterable action tracking to get back to the person who did the release.
This mean all logs of transaction and actions are pushed to a restricted area the team doesn't have access to, a change in the what is logged "should" be catch by functional tests the team has no access to and at worse will be catch by the audit and tracked to its author.

This is not applicable to all products, at time of writing in France any company allowed to emit money (mainly banks), has to ensure every transaction will be recorded and thus can't take the risk of missing a transaction.
On the other hand, they have no legal obligation to track any commercial offer or risk evaluation when someone ask for a loan, and thus the products handling this customer selection and computing the fees which will be in the offer are easier to fit in a post-release audit model.

Main idea is that the release model has to be tweaked according to risk assessment obligations.

A related resource is the ISO27001 norm.

Answer 3

IMHO, Developers & Operations could be represented by nothing more than just two git repositories for the same codebase, with distinct permissions model each, so that teams will not interfer in the work of each other, at all.

Let's call them Dev.mygithub.co & ops.mygithub.co, just as an example.

The idea is, that Developers are free to create/branch/merge to their heart's content -git is providing full traceability and that is what matters here- meanwhile, at the moments that regulatory framework implies review effort a Pull Request can be raised, for the merging to happen in a controlled manner.

Taking that concept to its next level, a develop branch can be propagated towards the remote Ops' production via yet another Pull Request act. That last part has to happen by operations' hands and eyes, since they have a responsibility to vet it into production and they choose the level of review.

Such scheme allows infinite flexibility, full traceability, ability to catch issues early on via a variety of processes, separation of concerns and a very reasonable user experience in the process!

N.B. The model described above may be followed even if Ops & Dev overlap totally!

Answer 4

in order, topmost is most expensive:

• distinct dev and ops sites & methods to carry over work from one to the other => many sites
• distinct dev and ops systems & methods as above => 1 joint site
• distinct dev and ops git/vcs repositories & related methods => 1 system
• distinct dev and ops git/vcs branches (protected) & related methods => 1 repo, many branches

Depending on what you do, some solutions are better than others; for example, if you have the need to serve two teams with distinct roles within them and each having ownership within and provide full traceability, you are hovering over first three.

In short, anything that enforces that one guy or gal cannot take the ball all alone and run with it, as s/he crosses a distinct explicit boundary among dev & ops.

Now, depending on the subsequent level of risk, that boundary might be enforced or not. Choices, choices.