6

When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here:

How to avoid a regular user can only test in production?

There are typically the environments like unit test, system test, integration test, volume test, user acceptance test, etc (I bet there are yet another dozen of names like those). But they all relate to environments in which something differs from how things currently work in the real production environments. So none of these is what my question is about.

To make this a less abstract question, consider this sample below:

Can non-SE employees only test SE sites in production?

I run into this scenario over and over (earlier today it happened again ...):

  • I discover some facility on an SE site that gets my attention (that I want to start using).
  • I start looking around in help pages, etc, and either don't find what I'm looking for, or what I find is not 100 accurate or clear to me.
  • I just try to start to use it, see what happens, hoping I will find the answer to what I couldn't find in the help pages (by just using that facility in some SE site ... read: by testing in production).
  • I then just wait and see what happens, which may be things like:
    • I learn about something new/interesting and it works great.
    • I get accused of trying to "fuzzle the system".
    • I get corrected by a mod because of something I did (which I shouldn't have done).
    • I discover a bug, which I either report (on meta.SE), or which I decide to consider as "a bug becomes a feature" (read: I don't want to disclose this handy feature).

However, doing some of those "experiments in production" may damage your reputation (not really your "points", but your real reputation).

So as a good SE citizens, I would be happy to do those experiments in an environment that is designed just for that: a none-production environment designed for users like me who want to do such experiments.

Voilà, if you were asked to configure test environments for SE sites, what would you do (or recommend SE management) about this DevOps issue?

PS: to my knowledge, at this very moment, there is no such SE environment that I can use right now to go check if a user that I "invite right now" for DevOps (while it is in private beta), and which I know did NOT commit to DevOps, will actually be able to login to the DevOps site. So instead of discussing it with mods on some other site, I already ... tested it in production. And dispite what some moderator try to make me believe ... it did work (and the invited user now also has an account on DevOps.SE).

  • 1
    If you're talking about the private beta not being really private, well it's documented here (This doesn't answer the overall question about test in prod, but this sched some ligths) – Tensibai Mar 10 '17 at 8:46
  • 1
    This question could use some linguistic editing. I find it hard to understand. – Assaf Lavie Mar 10 '17 at 9:06
  • @AssafLavie : feel free to edit my question. I'll review it afterwards, and either (a) accept or (b) improve (pretty sure no rollback), ok? If you do, make sure I don't need a dictionary to understand words that are a bit tougher to know/learn if English is not the language you learned on your mother's knees (Dutch shouldn't be any problem for me though). – Pierre.Vriens Mar 10 '17 at 9:29
  • 1
    I'm afraid I can't really help because I don't understand the question. – Assaf Lavie Mar 10 '17 at 9:37
  • @AssafLavie now I'm a bit confused what you wanted to say with "linguistic" in your first comment (I though it was about typos / grammar errors I made). Can you check my comment below the existing answer, does that help "you" somehow also? – Pierre.Vriens Mar 10 '17 at 9:49
6

A Sandbox could be part of the solution

To bring material to the subject you can check the questions with tag on MetaSE, there are sanboxes for:

  • Q/A formatting.
  • Comments formatting.
  • Chat.

There's a feature request about "Can we get a Stack Exchange sandbox?" which has no status (neither declined, pending, review nor in-progress meta-tag).

So I assume the answer (to the specific sample in the question) is that there's no way to test out of real SE sites for now and my recommendation to SE would be to accept this feature request.

When is a sandbox appropriate?

I don't think it's always needed to give users a sandbox, and it can be overkill sometimes. Defects should be spotted on Q/A environments, and when they're not, well, either you can know there's a glitch by a fine monitoring of your logs, or you will just never know until someone tells you.

Sanboxes come at an additional cost

Even if you deploy a sandbox for your users to try and confirm a bug, nothing tells you they will inform you of a bug/problem and you had to go through adding a public environment with dummy or anonymised data with no idea of it's ROI.

So It boils down to: "You can create a sandbox for your users but it adds complexity to your infrastructure for a hardly measured return"

Recommendation

My personal opinion would be to put the efforts in log analysis to spot unwanted behaviors instead.

  • merci for this answer, for which part of it seems to me like an answer to a question like "where somewhere in SE-space can I do this kind of tests". Such question clearly belongs on a meta site. Good to know that part of the answer, but that's not "my" real question here. My question is summarized in my title, somehow rephrased near the end with the "Voilà ..." at the end. IMO the "sandbox" tag you used is the beginning of a possible answer. If I'd be your customer I'd probably say "hang on, I know that from when I was a child, what does that mean in a DevOps context"? Wanna retry? – Pierre.Vriens Mar 10 '17 at 9:46
  • Edited to be more precise on my position. – Tensibai Mar 10 '17 at 10:03
  • 2
    Interesting update! and actually you hit a very important extra item that goes with such a sandbox thing: the data you'll need for such sandbox. You mind if I slightly change the formatting of some words you used which IMO are extremely important in the context of answering my question? If you don't like my edits, just rollback afterwards ... – Pierre.Vriens Mar 10 '17 at 10:21
  • OK, your turn again ... feel free to further improve / expand if needed, or just a rollback if you don't like it at all ... Curious to see if none-sandbox related answers will be added later on ... BTW, maybe "you" should also answer that meta question, eventually with a link to around here, to include another example of why such sandbox sites could/should be used also ..; PS: just completed yet another test in production. The result of it is a brand new user that got registered to join the private DevOps party, So the scenario mentioned in my PS works as a charm! – Pierre.Vriens Mar 10 '17 at 11:00
  • @Pierre.Vriens sounds ok, but I'm not sure of what you mean could/should (and for the new user, as already said, it's expected, the only thing blocked is direct access to http://devops.stackechange.com when you're not registered, passing though the area51 link works) – Tensibai Mar 10 '17 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.