8

Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?":

This environment usually freezes during the tests.

My question: what are sample implementations of a frozen environment? I.e. what can you do to technically enforce that nobody (except if allowed by an authorized user like a release manager) will be able to change anything in such frozen environment.

Clarifications:

  • I'm not talking about what (I think) is called "frozen periods" during (eg) year-end processing in banks. That is about not being allowed to apply any (repeat any) changes to production environments, to reduce the risk of new changes/fixes being introduced that may impact the year-end processing.

  • Assume that users who are allowed to approve/apply changes anyway (such as the release manager in my example), will only do so in exceptional cases. Such as where during testing a high severity issue is encountered, for which deferring a fix to a next release is not an option (since it would production at risk if the release would be activated without such fix).

  • This could just be about suspending any automated update during the time of the test. The point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This could mean having a guard to avoid a test team to require an update on a dependency under test.

Improve this question
0

3 Answers 3

Reset to default
3

My answer in the context of the question is a very expensive test environment (like mainframes or very large telecom equipment, for example) which are expected to be shared by multiple users for multiple tests, even simultaneously.

In many cases such equipment has a software management system responsible for, among other things, controlling software (un)installation/upgrades/downgrades/etc. Which should have some mechanism for blocking such operations (which would be, if I understood the question correctly, equivalent to freezing the environment) based on some more or less programmable policies.

In such case a specific policy could be developed exactly to support the desired freezing schedule necessary for the test environment. Ideally automated, accepting freezing/thawing triggers from the external sources which could be the test execution wrappers or CI systems. And maybe with human-triggered overrides, if deemed necessary.

Of course, such feature of the software management system would be helpful when testing other components of the equipment, but maybe not for tests of the software management system itself.

Improve this answer
2
  • This answer is exatly about what I'm used to (mainframes), where we do these kind of things for at least 1,5 decade or so already (before "DevOps" was born). I wonder if it would make sense to add my own answer here (to further expand on this answer, how we do this with CMN/ZMF for eg "banks"), or just take it to a new (self answered) question. What do you think?
    – Pierre.Vriens ♦
    Mar 18, 2017 at 8:10
  • Probably a separate one would be better - mine is mostly an opinion as I didn't experience such environment directly, I only discussed about it now and then with friends who did.
    – Dan Cornilescu
    Mar 18, 2017 at 13:25
4

TeamCity has a Shared Resources build feature which allows you to define a resource which multiple Build Definitions depend upon it. Build Definitions can either require a Read Lock or a Write Lock, you can also define if these locks are exclusive or allow a degree of parallelism.

If we make the following assumptions about a shared environment named PreProd:

  • A Shared Resource exists named "PreProd".
  • All build definitions, such as deployments, that make any change to that environment take an exclusive Write Lock on "PreProd".
  • All build definitions, such as non-restrictive tests, that perform read-only operations on the environment take a non-exclusive Read Lock on "PreProd".
  • TeamCity is the only process that can do anything on PreProd, albeit perhaps via another tool.

Therefore the following are true:

  • When a deployment is happening then nothing else can use PreProd, they will be queued.
  • When a test is running, any deployment would be queued until the tests complete.

You can use a similar mechanism with Jenkins using the Exclusion plugin. In fact, you could build this functionality into any process using locking or semaphore - for example Apache ZooKeeper or HashiCorp Consul.

Improve this answer
5
  • Good description of ´standard' implementations of what I did put in the ´guard' idea :)
    – Tensibai
    Mar 17, 2017 at 22:01
  • merci, any links to share/add to TeamCity itself, to learn more about it?
    – Pierre.Vriens ♦
    Mar 18, 2017 at 7:51
  • 1
    Absolument, I have added hyperlinks to both TeamCity and Jenkins in the answer, I can personally recommend the TeamCity course on Udemy.
    – Richard Slater
    Mar 18, 2017 at 9:23
  • merci for the extra update! PS: why not also include the course link in your answer also, eg via a PS at the end? That way, if some day a moderator comes along and starts deleting comments, it's not of risk of getting lost (it happens to me quite a lot at some other SE site).
    – Pierre.Vriens ♦
    Mar 18, 2017 at 9:40
  • I'm cautious about posting links to commercial content in answers on SE sites in general. I want to be helpful in that I recommend this course, however, as it is the only course I have taken it's very much my own opinion.
    – Richard Slater
    Mar 18, 2017 at 10:30
0

This sounds like an anti pattern to me. I believe everybody or nobody should have access to all the environments.

If users are subverting the process then I would take a serious look at the process to try and ensure it's not getting in people's way.

Implementing an automated mechanism that enforces a specific state is also useful for encouraging people to do things the right way. This can be via Config Management or destroying any immutable instance if someone SSHs to it

Improve this answer
5
  • Please check the 2nd "note" I added. Does it help to have you reconsider your first paragraph? Apart from that, I don't understand how the remaining part of your answer actually answers the "how to" part of my question. Maybe there is something I don't understand in it, so can you help me better understand your answer please? PS: Don't worry, I rarely downvote answers (if I do, I do leave comments ....).
    – Pierre.Vriens ♦
    Mar 13, 2017 at 20:33
  • Your have an xy problem meta.stackexchange.com/questions/66377/what-is-the-xy-problem. I'm not answering how to implement your solution I'm answering how I would solve your problem
    – Robo
    Mar 14, 2017 at 6:14
  • @Robo I disagree, the question on the how makes sense, without any subversion or whatever. This could just be suspending any automated update during the time of the test. You're seeing it as manual action, the point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This mean having a guard to avoid a test team to require an update on a dependency under test.
    – Tensibai
    Mar 14, 2017 at 9:33
  • That was not clear from your question. In that case you need some kind of lock.
    – Robo
    Mar 14, 2017 at 9:39
  • Please note that I just integrated (most of) @Tensibai 's comment as an extra clarification (last bullet). Hope "it" helps it make my question more clear. Maybe you (Robo) want to also review your answer as per that extra clarification? PS: merci Tensibai ...
    – Pierre.Vriens ♦
    Mar 15, 2017 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.