Harness Adds 8 New Features to Redefine Resiliency with AI-Powered Chaos Engineering | Harness Blog

Chaos engineering is the science of injecting faults and verifying the steady state of the system. In this article, we will not delve into the concepts of chaos engineering but will look into a more modern implementation approach called continuous resilience.  Traditionally chaos engineering has been known for verifying the resilience of critical systems and services in production. Recently, chaos engineering has been used to ensure resilience in the entire SDLC spectrum. Ensuring resilience in all stages of SDLC is the most efficient way to deliver the maximum availability of business-critical services to the end users. 

Chaos engineering as a concept is well understood by most of the audience. Some of the challenges today in chaos engineering area are related to implementing and scaling the practice across the organization, measuring the success of such chaos experimentation efforts, and knowing what it takes to get to that final milestone of resilience in terms of timelines and efforts. The reason for these challenges is, traditionally, chaos engineering has been taught as an exercise of careful planning and running of a set of experiments in production using the GameDay approach. The success of GameDay plays into the hands of a few individuals who are responsible for designing and orchestrating the execution from time to time and are not automated like other regular quality or performance tests. 

Continuous Resilience Approach

In modern chaos engineering practice, developers and QA teams share the chaos experiment development. The tests are automated in all environments and are run by all personas: Developers, QA teams, and SREs. The focus on resilience is built into every gate of SDLC, which leads us to the term Continuous Resilience. In the continuous resilience approach, we expect most chaos experiment runs to happen in the pipelines. However, continuous resilience is NOT just running chaos in pipelines; it is about automating the chaos experiment runs in all environments - Dev, QA, Pre-Prod, and Prod, though at various degrees of rigor. 

GameDays can still be used along with automated chaos engineering experimentation, especially in critical systems where resilience needs to be tested on a need basis. GameDays also provide a means to validate documentation, recovery procedures, and train engineers on incident response best practices.

Basic Tenets of Continuous Resilience^TM

When compared to the GameDay approach of chaos engineering, the continuous resilience approach is built around the following tenets:

• Chaos Experiment Development using Chaos Hubs
• Adoption of the new resilience metrics
• Integration into your SDLC or pipelines
• Security Governance for chaos experimentation

‍

‍

1. Chaos Experiment Development using Chaos Hubs

Well-written chaos experiments are the building blocks of a successful chaos engineering practice. These experiments must be upgraded or modified continuously to keep up with software or infrastructure configuration changes. This is why chaos experimentation is called chaos “engineering.” The best practice would be to manage the lifecycle of chaos code similar to that of the regular software code - this means that the chaos experiment code is developed and maintained in a source code repository, tested for false positives and negatives, tested in the dev environment, and promoted for use in larger environments.  

Chaos experiments should be easily tunable, importable, exportable, and shareable by multiple members of a team or by multiple teams of an organization. Chaos Hubs or centralized repositories of chaos experiments with the above features should be utilized for chaos experiment development and maintenance. 

In this method, you actively involve the QA team members and developers in writing the chaos experiment development and not limiting it to just the SREs, as in the traditional GameDay approach. 

2. Adoption of Resilience Metrics

A rollout of chaos practice will usually be associated with business goals such as “% decrease in production incidents” or “% decrease in recovery times,” etc. These metrics are more or less tied to production systems and are handled by the Ops teams or SREs. For example, you cannot measure uptimes and recovery times in QA test beds or pipelines. In the continuous resilience approach, where all personas are involved in chaos experimentation, the metrics to measure should be relevant to both Developers/QA teams/QA pipelines and Pre-production/Production/SREs. For this need, two new metrics are suggested:

- Resilience Scores

- Resilience Coverage

Resilience Score: The experiment may contain one or more faults against one or more services. The resilience score can be tied to a chaos experiment or a service. 

Resilience scores can be used in pipelines, QA systems, and production by all the personas.

Resilience Coverage: While the resilience score covers the actual resilience of a service or an experiment, resilience coverage shows how many more chaos experiments are needed to declare the entire system as checked for resilience. It is similar to code coverage in software testing. Together with resilience scores, resilience coverage gives complete control over the resilience measurement. Resilience coverage applies to a service or a system. 

‍

The resilience coverage metric has to be used to create a practical number of chaos experiments, as it can be argued that the number of possible chaos experiments that can be created against a service can be overwhelming.

3. Integrations into the SDLC or pipelines

Automation of chaos experiments is critical for the achievement of continuous resilience. Resilience coverage always starts at a low number, typically <5%, where the experiments are very safe to run or run against critical services. In both cases, the assurance of resilience is essential when new code is introduced via the pipelines. Hence, these experiments must be inserted into the regular pipelines that verify the sanity or quality of the rollout process. As the chaos experiments are automated, with every development cycle, the resilience coverage will be increased in steps, eventually reaching 50% or more in a few months or quarters, depending on the dev time the dev/qa teams spend. 

Suppose a pipeline reaches 80% resilience coverage with >80% resilience scores. In that case, the risk of a resilience issue or outage happening in production from known sources is mitigated to a large extent, leading to improved reliability metrics such as MTTF (Mean Time To Fail). 

Pipeline Policies can be set up to mandate the insertion of chaos experiments into the pipeline.

Automating pipelines and measuring the coverage through policies mandates the development of chaos experiments by team members who know the product best, i.e., developers and QA team members. SREs can then use these experiments to enhance production-grade environments. This is the organic process in which an organization's chaos engineering practice matures from Level 1 to Level 4. 

Chaos integration into the pipelines can be done in many ways. The following are some options:

1. Choose deployment pipelines that deploy to the system test bed or pre-production and run the chaos experiment suite after the changes are deployed. If the resulting resilience scores or resilience coverage is unsatisfactory, take appropriate action, such as manual inspection and approval of the pipelines or automatic rollback of the changes. 
2. Choose your feature flag pipelines that deploy the feature flags where a new feature is being rolled out into pre-production or production and run the chaos experiment suite on the environment where the flags are enabled. If the resulting resilience scores or resilience coverage is not satisfactory, then take appropriate action such as manual inspection and approval of the pipelines or automatic rollback of the flags. 
3. It is common to insert the basic resilience tests in the lowest environments, such as dev pipelines that deploy the code to the dev-test environment. However, the resilience coverage is expected to be low as the dev environments are maintained with minimal service configurations. 

4. Secure Chaos Experiment Orchestration

Chaos experimentation is generally perceived as disruptive because it may cause unwanted delays if the services are brought down unexpectedly. For example, someone bringing down all nodes of a Kubernetes cluster in a QA environment through a chaos experiment does not help in finding a weakness, but it can cause enormous interruptions for the QA process and its timelines. Similarly, chaos experimentation processes should be associated with the required guard rails or security governance policies.

Examples of security governance around chaos experiments:

• Allow node faults only on production-infrastructure-1
• Do not allow network loss faults on prod-2 between 9 AM and 9 PM
• Allow only the admin-group to inject node restarts
• Do not allow the users-group to inject network faults on qa-2 infrastructure

With security governance policies like those above, chaos experimentation becomes more practical at scale, and automation can increase resilience coverage. 

Get started with Harness Chaos Engineering

Harness Chaos Engineering is built with the above building blocks needed to roll out the Continuous Resilience^TM approach of chaos engineering. It comes with many out-of-the-box faults, security governance, chaos hubs, the ability to integrate with CD pipelines and Feature Flags, and many more. 

Get a demo

‍Objective: This blog aims to provide an understanding of Chaos Engineering, its importance, and best practices for testing the resilience of your Kubernetes Well-Architected Framework.

What is Chaos Engineering?

Chaos Engineering is the discipline of performing experiments on software systems to build confidence in their ability to withstand turbulent and unexpected conditions. In other words, failures are intentionally injected into applications to strengthen resilience. By proactively introducing controlled chaos, you can identify weaknesses and prevent catastrophic failures.

A formal definition:

“Chaos engineering(https://en.wikipedia.org/wiki/Chaos_engineering) is the discipline of performing experiments on software to build confidence in the system's capability to withstand turbulent and unexpected conditions.”

In practice, this means creating and managing chaos in a system by injecting faults to discover vulnerabilities before they become critical failures.

Why is Chaos Engineering Important?

Incidents and outages are inevitable and can lead to significant financial losses. A prime example is one of the AWS outages in November 2020, caused by an issue in Amazon Kinesis. This failure affected a wide range of services, including Amazon ECS, EKS, CloudWatch, Lambda, and others, costing millions during the critical Black Friday sales week.

Chaos Engineering allows organizations to proactively test their systems’ weaknesses by reproducing incidents or injecting faults in a controlled manner. For instance, companies like Amazon and Netflix have reported up to a 45% reduction in downtime after adopting Chaos Engineering practices.

What is a Kubernetes Well-Architected Framework?

Kubernetes is a widely used container orchestration framework that provides best practices for building highly available and resilient applications. Here are key design patterns recommended for ensuring resilience:

Kubernetes Design Patterns:

1. Liveness Probe: Verifies the container's health using checks (HTTP, Command, TCP) to ensure the container is restarted if it becomes unresponsive or deadlocked.
2. Resource Requests and Limits: Developers must set CPU and memory requests and limits to prevent resource exhaustion (e.g., out-of-memory errors), ensuring application stability.
3. Observability and Monitoring: Use observability tools like Prometheus or Datadog to monitor Kubernetes metrics, create alerts, and address anomalies.
4. Node/Pod Affinity: Ensures pods are scheduled across nodes or availability zones for performance and reliability.
5. Horizontal Scaling: Automatically scales application pods based on resource consumption (CPU, Memory) during high traffic, optimizing both performance and cost.
6. Handling Voluntary Disruptions: This refers to intentional actions like maintenance or node restarts. It's essential to test how your application handles these disruptions.
7. Security (Certificates): Applications should handle HTTPS interactions and manage SSL/TLS certificate expiration to prevent downtime.
8. Mapping Service Dependencies: Understanding service dependencies is crucial to identifying bottlenecks during failure scenarios in micro-service architectures.

How Harness Chaos Engineering Enhances Kubernetes Resilience

Harness Chaos Engineering helps you automate resilience testing in your Kubernetes clusters, identifying potential weaknesses in your architecture.

Kubernetes Experiment Suite

Harness offers an experiment suite designed to test the resilience of Kubernetes design patterns. This suite contains 40+ recommended experiments categorized into the following for Kubernetes:

a) Voluntary Disruptions

• Experiments: Pod Delete, Container Kill, Node Drain, Node Restart.
• Use Cases: Test scenarios like node/pod affinity, SLAs, and failover in the event of a node failure. Verify how your application behaves during maintenance or restarts.

b) Scalability

• Experiments: Pod and Node CPU Hog/Stress, Pod and Node Memory Hog/Stress, Pod Disk Fill, Pod and Node I/O Stress.
• Use Cases: Verify the impact of resource exhaustion (CPU, memory) on application performance. Test Horizontal Pod Autoscaling and ensure applications scale appropriately during stress.

c) Service Dependency

• Experiments: Pod and Node Network Latency, Pod and Node Black Hole/Network Loss.
• Use Cases: Test how applications handle network latency, blackholes, and other communication failures. Ensure proper error handling and assess the impact of component failures.

d) Security

• Experiment: Time Chaos (Certificate Expiration).
• Use Cases: Simulate certificate expiration to test application response. Ensure your monitoring tools trigger alerts and that your application gracefully handles the scenario.

How Harness Chaos Engineering Automates Chaos Testing in Kubernetes

Harness Chaos Engineering simplifies and scales chaos testing for Kubernetes environments with features such as:

1. Automatic Service Discovery & Application Maps

Harness’s discovery agent automatically identifies services within your Kubernetes namespaces, providing an application map to show traffic flows between services.

2. Auto-Creation of Experiments

After service discovery, Harness auto-generates chaos experiments at different levels of complexity (Basic, Intermediate, Advanced):

• Basic Experiments (Only a Few): Includes Pod Delete and Network Loss experiments.
• Intermediate Experiments (Moderate): Includes Pod Delete, Network Loss, and CPU experiments.
• Advanced Experiments (Maximum): Includes Basic and Intermediate experiments along with Memory experiments.

3. Out-of-the-Box Kubernetes Faults

Harness offers a wide range of built-in chaos faults for Kubernetes (reference documentation), ensuring you can test for various failure scenarios.

‍

4. Resilience Probes

Probes validate your hypothesis about the system’s resilience. For example, Is your application responsive when an application pod is deleted?

5. Resilience Score

The resilience score quantifies how resilient your environment is after running chaos experiments. It’s based on the weight given to each fault and the success rate of probes, providing a clear picture of your system's reliability.

Conclusion

Chaos Engineering is essential for building resilient systems, particularly in Kubernetes environments. By proactively injecting faults, organizations can prevent failures and ensure high availability, ultimately leading to reduced downtime and better customer experiences.

For more detailed use cases and technical documentation, please refer to the Chaos Engineering Docs.

Sign up for FREE to experience the ease of resilience verification using chaos experiments. The free plan allows you to run a few chaos experiments at no charge for an unlimited time, boosting Chaos Engineering for the community.

Harness' Chaos Engineering ROI Calculator helps estimate business losses from outages and evaluates the ROI of chaos engineering practices. By simulating failures and optimizing recovery, it improves system reliability and reduces downtime, providing a clear financial benefit to organizations.