0

In a CD environment with new features hidden behind feature flags, there must be a way to ensure that code being pushed to production is effectively zero change?

Particularly in a change management controlled environment, it would be ideal to not treat code deployment as a change - only the feature switch on. To be able to do this, is there a way to prove that all code changes are protected behind feature flags? And that the effective flow is zero change. Feels like something that could and should exist. But maybe others solve this differently?

Improve this question
5
  • Perhaps I'm misunderstanding, but this seems like the reason why you would use automated testing for regression. If you have sufficient test coverage in your automated regression suite and the only changes are behind a feature flag, all of the automated tests would pass. They would fail if there was an unexpected change outside of the feature flag.
    – Thomas Owens
    Commented Mar 13, 2023 at 12:11
  • Well, yes, maybe. That was my initial thinking, but it kind of raised more questions than answers for me. In a microservices setup, where you want to keep each microservice independently deployable, there's 3 levels you could test this. Unit tests - but they're being changed with the code, so too likely to break at the same time. Component/service/contract tests - they always seem the least reliable, as they're in the no-mans land between tech/dev and qa/product. Or Acceptance tests on the whole system - too slow and brittle. Hence why I was hoping SA could help
    – james-tru
    Commented Mar 13, 2023 at 23:11
  • I want to think this through a little more. Are you concerned with the case where a feature flag affects more than one service's behavior? That seems to be the problematic case. Although I also think you may be missing one level of tests - you have some tests that test a service in isolation and other tests that test the integration between two or more services (but not yet at an acceptance level). I guess I'm struggling with why static analysis would be helpful, especially since I don't see how the tool would look across service boundries to find the types of issues you're concerned with.
    – Thomas Owens
    Commented Mar 13, 2023 at 23:36
  • So, say we have a simple microservice that had a new feature added: if(handle_passover_flag) { do_something_slightly_different_on_passover_if_its_awednesday() } else { do_normal_thing_except_on_wednesdays()} where the new feature is a pretty subtle edge case. It's the kind of thing component tests often miss. The person writing the new feature could accidentally change some existing code like common.utilities.is_weird_day(). Static analysis could compile with the flag as false and then see that the compiled old code and new code were the same and only promote if it is
    – james-tru
    Commented Mar 14, 2023 at 22:34
  • I don't think this is feasible. First, you're limited to compiled languages. And then you need to take the time to do multiple compilations, which would increase in number as you increase the number of feature flags, which adds time to your build pipeline. You wouldn't need complex analysis at this point - just a diff. But I also don't think you need component tests - if there was a change to common.utilities.is_weird_day(), then your unit tests should fail if an expected case is no longer properly managed.
    – Thomas Owens
    Commented Mar 15, 2023 at 12:41

1 Answer 1

Reset to default
0

You've highlighted the great conundrum of feature flags!

I don't believe it would be possible to check this with static-analysis alone as it requires some knowledge of application behaviour (even if the understanding is "what is a feature flag", as they often look like things that aren't feature flags).

In order to prove everything is hidden by feature flags, you'd need to be able to run a test suite aginst the application with all flags switched off.

Whether you also need to test "on" states and combinations of "on" states depends a lot of how you are using feature flags. When you use them to hide a feature in development it's very different to when you make a system user-configurable with flags, or have multiple values per flag.

More flags and flag values = way more combinations. It quickly becomes possible to only test "selected combinations".

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.