Kill Switches and Other Ops Use Cases for Feature Flags
We support permanent flags in Harness Feature Flags, and as we evolve this feature, we want teams to see the value in permanently wiring controls into your product that can be used instantly as kill switches in any of your environments.
If you’ve only come to feature flags recently, you may not know that feature flags as a concept has existed for quite a long time, and can easily be seen as an evolution of app config systems.
Because of this, one of the least used - but most powerful - use cases for feature flags can be ops and SREs teams. Plenty of folks never thought about this potential, but it can be a way in which feature flags can most transform your engineering organization.
Not Just Front-End and User Changes!
When thinking about feature flags, it’s important to say right away that they are not limited to customer-facing changes and frontend services. Here at Harness, we have support for all major backend languages with our server SDKs, and we use Feature Flags to refactor, help remove tech debt, and roll out API changes - and a whole lot more.
One way to think about feature flags as it relates to ops would be as an app config layer. What are the major levers of your app that you may want to turn off and on, or dial up and down?
Feature flags don’t have to be something you take out of the code after a fixed period of time. We support permanent flags in Harness Feature Flags, and as we evolve this feature, we want teams to see the value in permanently wiring controls into your product that can be used instantly as kill switches in any of your environments.
A feature flags dashboard could easily be an ops dashboard, with buttons to turn off backup systems, features in certain regions, certain features based on load, or anything else you may want kill switch or override capabilities for.
Two Example Scenarios
Let’s take a look at two real world examples to help illustrate this idea:
Scenario #1 for Kill Switches
At a previous job, we had logs data that would need to go into storage from the short term cache, so we had a system to collect and store them. During peak loads or other periods of instability (particularly upstream with AWS), we would often see this storage become either very expensive or very unreliable. So, we would turn this service off during these periods, but this was manual and involved a series of AWS commands and database changes to both disable and re-enable later on. By wiring this up to a feature flag, we were able turn this feature off and on instantly in a much more lightweight, visible, and easy to govern way.
Scenario #2 for Kill Switches
If you run your application in multiple regions that have different data laws, or have certain tenants (think government or finance customers) with special regulations, it can be a mess showing a button in one place but not another. Wiring key features up as permanent feature flags allows you to easily turn something on for all your North American users, but not your European users where there may be different legal standards around data, or to disable it only in your high-compliance cluster where it may impact customers’ SOC2 needs. You can move all of this control out of the complexity of application code. Alternatively, you can run multiple app variations and use flags to make it easy for everyone to see and audit these critical controls.
If you’ve never used feature flags for these purposes before, a fair question is, “Where do I even start?”
It may be useful to look at your team's operations runbooks. If you don’t have runbooks, is there a place you keep uncommon or rare incident reports? Start there.
See if any of these scenarios can be controlled by a flag (or by a series of flags). Turning a flag off is easier, safer, more governed, and more visible than shelling into AWS and making a change that way - or having to create a new PR, and merging and deploying in a hurry.
Auditing these processes for opportunities to move things to flags may give you one or two quick wins that you can expand on from there.
Much is made of the power of flags when it comes to testing, to deploying more safely, and to rolling out changes and reducing risk. However, we believe teams don’t think enough about how flags can transform your ability to do ops and support your application’s various configurations and availability needs. We’d like to see that change!
If you have any examples, questions, or feedback around using feature flags for these types of needs, please get in touch! We’d love to help you.
And in the meantime, feel free to download our free, ungated eBook on feature flags to learn more on the topic.