18

In highly regulated environments, such as the Financial Services sector, Segregation of Duties is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.

Traditionally this has meant that Developers develop code and then hand it over to Operations however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

  • In Google's Site Reliability Engineering, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.

  • In the "You Build It, You Run It" model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy Sarbanes Oxley Section 404: Management Assessment of Internal Controls:

(a) Rules Required. The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--

(1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and

(2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.

(b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.

Based upon the comments it is important to call out a couple of assumptions I am making:

  • I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value. This would be as opposed to commercial financial services which has a different transaction value profile.
  • A financial institution's online offering will be made up of many components that have differing risk considerations:
    • Move Money - Moving money between accounts or transfers between accounts of different owners. An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.
    • Customer Acquisition - Less "Risky" as it has low transaction volumes compared to Move Money but still needs consideration.
    • Internet Banking - Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.
  • Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.

TL;DR: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the Securities and Exchange Commission's regulations.

Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.

Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.

  • Main idea behind a devops organization beingto put everyone in the Team accountable for what happens in production, there can't be separation of duties. This mainly means this kind of organization can't be really used when there's regulatory needs for this separation. – Tensibai Mar 18 '17 at 13:29
  • @Tensbai I fundementally disagree with the assertion that DevOps is incompatible with Segregation of Duties. The laws are not prescriptive as to the manner of the controls, nor are the regulators imposing a predefined process on banks and financial services. It is largely down to the organisation to identify what appropriate is and be totally transparent with the regulators and their appointed auditors. As an example both ING and Barclays have adopted DevOps practices to allow them to accelerate their ability to deliver value to their customers. – Richard Slater Mar 18 '17 at 14:16
  • Yes, devops on subjects not bound to regulatory separation, and they took advantage of the automation on a traditional silo based org for subjects restricted (which are in fact very few). They just have two kind of orgs depending on which kind of operations the software will do – Tensibai Mar 18 '17 at 14:26
  • There is no such thing as "Regulatory Separation" the statutes/laws and regulatory bodies don't impose separation upon financial institutions they impose a management responsibility to have "Appropriate Controls" to manage financial risk. In the same way that Agile took software development from long cycles to small cycles, DevOps is taking operations into small cycles, DevOps in Financial Services needs to find a way to take Segregation of Duties into small cycles, by for example creating a CD pipeline that enforces "appropriate controls" such as peer review and approval based promotion. – Richard Slater Mar 18 '17 at 14:49
  • 1
    @Pierre.Vriens yes the broad question is in the title, I have tried to expand upon it by making some assumptions. Roles are likely to be part of the solution as are things like Break-Glass and Privileged Account Management. Roles and Responsibilities are an interesting concept in DevOps/Agile as where once upon a time you had a Java Developer, F/E Developer, Designer, PM, Build Engineer, Release Manager and Ops Engineer - now you have a group of people who can wear multiple hats - cross-functional teams made up of "Engineers" who may specialise but ultimately share responsibility,. – Richard Slater Mar 18 '17 at 19:31
8

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).


1. Architecture


Here are the highlights of how I'd answer your question:

  • All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the library structure.
  • For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: the server gets exclusive update authority to the entire library structure. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...
  • The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a software change package.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the lifecycle of a software change package (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

  • Only the server will execute the required (and preconfigured) steps.
  • The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.
  • The approvals can only be given by users who have a role that allows them (= permission) to issue such approvals.


2. Roles and permissions


The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

Step 1: Configure the permissions (in the security system)

  • Define security entities in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

    • PrmUnit, used for getting permission to request a Promote to say Unit-testing.
    • PrmQA, used for getting permission to request a Promote to say Qa-testing (let's assume this is the highest level of testing).
    • PrdEnduser, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.
    • PrdRelmgnt, used by release managers to authorize an Activation in production (= the last/highest level in the library structure).
  • Define groups of users in your security system. A few samples (add as many similar ones to fit your own needs):

    • GrpDevs, which (say) corresponds to your developers (probably more then just 1).
    • GrpEnduser, which (say) corresponds to your end-users (at least 1, preferably with more similar users).
    • GrpRelMgnt, which (say) corresponds to your release managers (at least 1, preferably a few more users).
  • Grant permissions, also using your security system, to allow access to selected "security entities" for selected "groups of users". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

    • Group GrpDevs gets access to (only!) security entity PrmUnit.
    • Group GrpEnduser gets access to (only!) security entity PrdEnduser.
    • Group GrpRelMgnt gets access to (both!) security entity PrmQA and PrdRelmgnt.

Step 2: Configure the workflow (in the SCM system)

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.

Here are some examples of how you'd configure your SCM system to make some magic happen:

  • If a user has access to PrmUnit, then such user is allowed to request a Promote to Unit-testing. Obviously, the users in group GrpDevs are the users authorized for this (note: not, e.g, the users in group GrpRelMgnt).
  • If a user has access to PrmQA, then such user is allowed to request a Promote to QA-testing. Obviously, the users in group GrpRelMgnt are the users authorized for this (note: not, e.g, the users in group GrpDevs, or in group GrpEnduser).
  • If a user has access to PrdEnduser, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group GrpRelMgnt to even be able to review a change). Obviously, the users in group GrpEnduser are the (only) users authorized for this.
  • If a user has access to PrdRelmgnt, then such user is allowed to authorize an Activation in production (= the last/highest level in the library structure).


3. Expect the unexpected, and be ready for it


The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.

To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like

What happened when and why, and which authorized user actually approved it ... upfront?

But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.


PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.

  • That's sounds like a basic implementation of top-down risk assessment, I don't get how it address the question on how this could be implemented in a devops manner where devs would have the rights to trigger the ´deploy' switch. Is the idea that you can't do it in a devops organization? – Tensibai Mar 20 '17 at 20:36
  • @Tensibai "if" devs would have the auth (role) of (eg) final approval for prod (which they typically do NOT have in such organisations), then such server (started task) would start the deployment. And as per the title of the question, I think this is "a" possible answer. One could question however if this is what we would call a DevOps organization, but I do know that auditors really like this kind of "configurable" segragation of duties (eg: four-eyes and variations of that). Maybe Richard can help us with his viewpoint on this? – Pierre.Vriens Mar 20 '17 at 20:52
  • 1
    I agree auditors like it absolutely, I just missed how this relate/fit with the ´explosion' of access, which auditor usually doesn't like when the list contains more than 6 to 7 persons. Saying it doesn't fit is an absolutely valid answer IMHO. – Tensibai Mar 20 '17 at 21:00
  • 1
    Thank you for putting so much time into an answer. I am actually thinking of implementing a 3-person rule, in that, one developer writes the code, a different developer reviews the code and a third person presses the release button to deploy the code. The other consideration is because this is part of a company wide Agile/DevOps adoption the development teams are quite small, with the net effect of small groups of people having production access to thin slice of production, this seems to be favourable from a risk perspective. – Richard Slater Mar 21 '17 at 8:01
  • 1
    @Pierre.Vriens Can't upvote twice, great extension :) – Tensibai Mar 21 '17 at 17:00
5

Answer based on my knowledge of French "Internal controls" regulation, kind of equivalent to the SEC regulations you point to, I assume linking here to a French legal text would not be really useful and I know of no good translation of it.

In an ideal 'You build it, you run it' model, everyone in the team will be accountable for the change. The risk assessment can't be enforced by a separation of duties and the only way I know to keep conforming with regulation is to have a periodic short cycle audit of transactions along with an unalterable action tracking to get back to the person who did the release.
This mean all logs of transaction and actions are pushed to a restricted area the team doesn't have access to, a change in the what is logged "should" be catch by functional tests the team has no access to and at worse will be catch by the audit and tracked to its author.

This is not applicable to all products, at time of writing in France any company allowed to emit money (mainly banks), has to ensure every transaction will be recorded and thus can't take the risk of missing a transaction.
On the other hand, they have no legal obligation to track any commercial offer or risk evaluation when someone ask for a loan, and thus the products handling this customer selection and computing the fees which will be in the offer are easier to fit in a post-release audit model.

Main idea is that the release model has to be tweaked according to risk assessment obligations.

A related resource is the ISO27001 norm.

  • Interesting answer and very relevant as many European banks do indeed operate in France. Is there any chance you could expand on what 'Emit Money' means, i.e. is that just cash out of an ATM or does it include balance transfers. In this case linking to the statutes would be valuable as it gives a pointer to the relevant laws, regardless of the language they are in. – Richard Slater Mar 21 '17 at 7:51
  • @RichardSlater In brief, any company working with money, could be an investment only company as well as loan brokers along the traditional banks. Mostly anything which have a financial impact somewhere is concerned (out of few exception which the authority can give under come circonstances). The legal "list" in French is here but even in French it is not always obvious. – Tensibai Mar 21 '17 at 8:28
  • I am making the assumption that the link to the ISO standard should actually be ISO27001:2013 – Richard Slater Mar 21 '17 at 18:17
  • @Richard indeed, seems the French to English link hasn't been updated on Wikipedia. I'll update later (or if you wish , feel free to edit) – Tensibai Mar 21 '17 at 18:30
0

IMHO, Developers & Operations could be represented by nothing more than just two git repositories for the same codebase, with distinct permissions model each, so that teams will not interfer in the work of each other, at all.

Let's call them Dev.mygithub.co & ops.mygithub.co, just as an example.

The idea is, that Developers are free to create/branch/merge to their heart's content -git is providing full traceability and that is what matters here- meanwhile, at the moments that regulatory framework implies review effort a Pull Request can be raised, for the merging to happen in a controlled manner.

Taking that concept to its next level, a develop branch can be propagated towards the remote Ops' production via yet another Pull Request act. That last part has to happen by operations' hands and eyes, since they have a responsibility to vet it into production and they choose the level of review.

Such scheme allows infinite flexibility, full traceability, ability to catch issues early on via a variety of processes, separation of concerns and a very reasonable user experience in the process!

N.B. The model described above may be followed even if Ops & Dev overlap totally!

  • 1
    Surely, this same control could be achieved through pull requests and post-commit hooks that ensured that developers could commit freely, however, merge commits could only be made by an approved group of people. Equally that same post-commit hook could ensure that of the authors of the commits that made up the pull request did not include the person making the pull request. – Richard Slater Mar 21 '17 at 18:14
  • @RichardSlater: the reason you'd may wish to have two distinct repositories is that you have the dual need to both allow developers to merge -when they freely swap code in developer mode- as well as block most developers from merging code when it is to go towards production (modulo SysOps, i.e.. the so called "approved group of people"). – fgeorgatos Jun 24 '17 at 8:04
  • Again you can achieve that with post-commit hooks and pull requests, not to mention GitHub Enterprise allows protected branches. – Richard Slater Jun 24 '17 at 9:05
0

higher is more expensive:

  • distinct dev and ops sites & methods to carry over work from one to the other
  • distinct dev and ops systems & methods as above
  • distinct dev and ops git/vcs repositories & related methods
  • distinct dev and ops git/vcs branches (protected) & related methods

Depending on what you do, some solutions are better than others, for example if you have the need to serve two teams with distinct roles within them and each having ownership within and provide full traceability, you are hovering over first three.

In short, anything that enforces that one guy or gal cannot take the ball all alone and run with it, and s/he crosses a distinct explicit boundary among dev & ops. Now, depending on the level of risk, that boundary might be enforced or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.