Test your app’s resilience under load using Grafana K6 and Harness Resilience Testing—best practices and real-world Kubernetes example.
TL;DR
- Resilience testing measures how reliably your business services perform when system components fail temporarily under high load, revealing issues that performance testing alone cannot detect.
- Load testing validates throughput and response times at scale, but combining it with chaos engineering reveals how your application behaves when critical dependencies fail during peak traffic.
- Grafana K6 generates realistic user traffic patterns while Harness Resilience Testing injects controlled failures such as network latency, pod deletions, and API errors to simulate production incident scenarios.
- Resilience probes define your steady-state hypothesis by continuously monitoring service latency, replica counts, and query performance throughout chaos experiments, producing a quantifiable resilience score.
In the real world, high traffic doesn’t arrive alone, it brings chaos with it.
Imagine your eCommerce site during a Black Friday sale. Users are flooding in, carts are filling up, and payment gateways are firing requests by the second. Your team has tested for load, and everything looks good in isolation. But what happens if the checkout service experiences latency, or the cart service pod restarts mid-transaction? Will customers still be able to check out? Will your system fail gracefully—or just fail?
This is where combining load testing with chaos engineering becomes critical.
Load testing validates performance under pressure. Chaos engineering ensures your app behaves reliably when key components fail, slow down, or misbehave—at the exact moment your system is under stress. By running both together, you move beyond just preparing for scale—you prepare for reality.
Let’s walk through a specific step by step guide!
Summary
Goal: Measure the resilience score of the Cart service under real-world load by injecting chaos experiments and tracking system health via resilience probes.
Sample Application: Boutique ECommerce Portal
Load Test Tool: Grafana K6, deployed locally at runtime.
Chaos Experiments: Network Loss, Network Latency, Pod Deletes, and HTTP API Status code errors.
Deployment and Test Setup
As shown above, the policies and experiments are configured on the control plane, and the constructed chaos experiments are sent to the target hosts or clusters on the execution plane. The first step is to get access to the Harness platform.
Get Access to the Platform
Sign up for a free Chaos Engineering on Harness plan that is fully featured and allows you to run a certain number of chaos experiments per month.
Otherwise, if you are a current Harness customer, someone (harness-account-admin) would have invited you to join an existing account at Harness that allows you access to the Chaos Engineering product.
Once you get access through either of the above methods, you can view/create and run chaos experiments along with the load tests. Before we get into the details of setting up and running the specific chaos experiments, it is essential to understand the roles and permissions required to do the proper setup.
We are considering three types of users for this resilience test setup.
- Harness Account Administrator (harness-acc-admin)
- Kubernetes and Application Administrator (service-admin)
- Application developer / Performance tester (service developer)
First, harness-acc-admin configures the required roles for the service developer to create and run the chaos experiments.
Second, the service admin must set up a service account on the target application cluster, which will be used to run the load generation and chaos experiments.
Once the above two roles and permissions are set up, you can create and run the chaos experiments and load generation tasks as an application developer, QA engineer, or performance tester.
Before showing the actual chaos experiments, let's explore the application and the test scenarios in more detail.
The Application and Test Scenario
OnlineBoutique is a sample application we will use in this resilience testing under load scenario. It consists of multiple microservices that interact with each other through simple APIs and store transaction data in a database. The architecture is shown below:
In the above diagram, the load generation is done using the Grafana K6 load generation tool. The OnlineBoutique can be set up on various platforms. In the current example, we have set up the application on a Kubernetes platform.
For resilience testing with Harness Resilience Testing CMS Push Instructions: Testing application resilience under load with Grafana K6
Target Post Slug: testing-application-resilience-under-load-with-grafana-k6
Field 1: Title
How to Test Application Resilience Under Load with K6 and Chaos Engineering
Field 7: Key Takeaways
<ul>
<li>Resilience testing measures how reliably your business services perform when system components fail temporarily under high load, revealing issues that performance testing alone cannot detect.</li>
<li>Load testing validates throughput and response times at scale, but combining it with chaos engineering reveals how your application behaves when critical dependencies fail during peak traffic.</li>
<li>Grafana K6 generates realistic user traffic patterns while Harness Resilience Testing injects controlled failures such as network latency, pod deletions, and API errors to simulate production incident scenarios.</li>
<li>Resilience probes define your steady-state hypothesis by continuously monitoring service latency, replica counts, and query performance throughout chaos experiments, producing a quantifiable resilience score.</li>
</ul>
Field 20: FAQ
<h2>Frequently Asked Questions</h2>
<h3>What does application resiliency mean?</h3>
<p>Application resiliency is the ability of software to maintain core functionality during unplanned disruptions like component failures, network latency, or sudden traffic spikes. Resilient systems detect failures automatically, recover gracefully through fallback mechanisms, and minimize user impact without causing full outages. Unlike simple uptime, resiliency focuses on how well your application adapts when things go wrong.</p>
<h3>How to improve application resiliency?</h3>
<p>Improve resiliency by regularly testing systems under realistic failure conditions using chaos engineering. Inject faults like pod restarts, network delays, and resource exhaustion while monitoring service health with resilience probes. Automate recovery processes, implement retry logic and circuit breakers in your code, and validate resilience continuously in your CI/CD pipeline rather than waiting for production incidents to reveal weaknesses.</p>
<h3>What is the difference between load testing and resilience testing?</h3>
<p>Load testing validates performance under high traffic volumes by measuring throughput, response times, and resource utilization at scale. Resilience testing injects failures during that load to measure how well your system detects issues, recovers, and maintains service continuity. Combining both approaches reveals whether your application can handle Black Friday traffic levels while a key microservice crashes or experiences latency.</p>
<h3>How resilient should your load tests be?</h3>
<p>Resilience testing under load should simulate realistic failure scenarios your production environment faces: network latency between microservices (50-200ms delays), pod restarts during deployments, API timeouts from downstream services, and resource exhaustion. Target resilience scores above 80% where critical probes pass even when chaos is injected under peak load, ensuring your most important user journeys remain functional during incidents.</p>
<h3>What tools do I need for resilience testing on Kubernetes?</h3>
<p>You need three components: a load generator like Grafana K6 to create realistic traffic patterns, a chaos engineering platform like Harness to inject controlled faults (pod deletes, network chaos, resource stress), and observability tools like Prometheus or Datadog to monitor service health via resilience probes that validate your steady-state hypothesis throughout experiments.</p>
Field 19: Structured Data Field (SEO)
{ "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "What does application resiliency mean?", "acceptedAnswer": { "@type": "Answer", "text": "Application resiliency is the ability of software to maintain core functionality during unplanned disruptions like component failures, network latency, or sudden traffic spikes. Resilient systems detect failures automatically, recover gracefully through fallback mechanisms, and minimize user impact without causing full outages. Unlike simple uptime, resiliency focuses on how well your application adapts when things go wrong." } }, { "@type": "Question", "name": "How to improve application resiliency?", "acceptedAnswer": { "@type": "Answer", "text": "Improve resiliency by regularly testing systems under realistic failure conditions using chaos engineering. Inject faults like pod restarts, network delays, and resource exhaustion while monitoring service health with resilience probes. Automate recovery processes, implement retry logic and circuit breakers in your code, and validate resilience continuously in your CI/CD pipeline rather than waiting for production incidents to reveal weaknesses." } }, { "@type": "Question", "name": "What is the difference between load testing and resilience testing?", "acceptedAnswer": { "@type": "Answer", "text": "Load testing validates performance under high traffic volumes by measuring throughput, response times, and resource utilization at scale. Resilience testing injects failures during that load to measure how well your system detects issues, recovers, and maintains service continuity. Combining both approaches reveals whether your application can handle Black Friday traffic levels while a key microservice crashes or experiences latency." } }, { "@type": "Question", "name": "How resilient should your load tests be?", "acceptedAnswer": { "@type": "Answer", "text": "Resilience testing under load should simulate realistic failure scenarios your production environment faces: network latency between microservices (50-200ms delays), pod restarts during deployments, API timeouts from downstream services, and resource exhaustion. Target resilience scores above 80% where critical probes pass even when chaos is injected under peak load, ensuring your most important user journeys remain functional during incidents." } }, { "@type": "Question", "name": "What tools do I need for resilience testing on Kubernetes?", "acceptedAnswer": { "@type": "Answer", "text": "You need three components: a load generator like Grafana K6 to create realistic traffic patterns, a chaos engineering platform like Harness to inject controlled faults (pod deletes, network chaos, resource stress), and observability tools like Prometheus or Datadog to monitor service health via resilience probes that validate your steady-state hypothesis throughout experiments." } } ]}{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What does application resiliency mean?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Application resiliency is the ability of software to maintain core functionality during unplanned disruptions like component failures, network latency, or sudden traffic spikes. Resilient systems detect failures automatically, recover gracefully through fallback mechanisms, and minimize user impact without causing full outages. Unlike simple uptime, resiliency focuses on how well your application adapts when things go wrong."
}
},
{
"@type": "Question",
"name": "How to improve application resiliency?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Improve resiliency by regularly testing systems under realistic failure conditions using chaos engineering. Inject faults like pod restarts, network delays, and resource exhaustion while monitoring service health with resilience probes. Automate recovery processes, implement retry logic and circuit breakers in your code, and validate resilience continuously in your CI/CD pipeline rather than waiting for production incidents to reveal weaknesses."
}
},
{
"@type": "Question",
"name": "What is the difference between load testing and resilience testing?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Load testing validates performance under high traffic volumes by measuring throughput, response times, and resource utilization at scale. Resilience testing injects failures during that load to measure how well your system detects issues, recovers, and maintains service continuity. Combining both approaches reveals whether your application can handle Black Friday traffic levels while a key microservice crashes or experiences latency."
}
},
{
"@type": "Question",
"name": "How resilient should your load tests be?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Resilience testing under load should simulate realistic failure scenarios your production environment faces: network latency between microservices (50-200ms delays), pod restarts during deployments, API timeouts from downstream services, and resource exhaustion. Target resilience scores above 80% where critical probes pass even when chaos is injected under peak load, ensuring your most important user journeys remain functional during incidents."
}
},
{
"@type": "Question",
"name": "What tools do I need for resilience testing on Kubernetes?",
"acceptedAnswer": {
"@type": "Answer",
"text": "You need three components: a load generator like Grafana K6 to create realistic traffic patterns, a chaos engineering platform like Harness to inject controlled faults (pod deletes, network chaos, resource stress), and observability tools like Prometheus or Datadog to monitor service health via resilience probes that validate your steady-state hypothesis throughout experiments."
}
}
]
}
PUSH ORDER
- Field 1: Title
- Field 7: Key Takeaways
- Field 20: FAQ
- Field 19: Structured Data Field (SEO)
After pushing, verify the live page renders all updated fields correctly.
haos Engineering, the following two things are essential:
- Chaos faults and targets: This defines what chaos faults to be injected. In our examples, we will run a K6 load generator using one chaos experiment and run another parallel chaos experiment that does one of the following:
- Cause network latency between Checkout and PaymentServices
- Cause network loss between Frontend and CheckoutServices
- Injects an HTTP error code between Frontend and CartService
- Delete a pod on the ProductCatalog service
- The meaning of resilient state: In chaos engineering terminology, this is also called the "Steady State Hypothesis." In Harness Resilience Testing, resilient states are defined using Resilience Probes. We are considering the following probes.
- Check if the queries per second on the frontend are less than 2000ms at any time.
- Check if the latency between the user and frontend is less than 20ms at any time.
- Check if the number of queries per second on the CartService is less than 1000ms at any time.
- Check if the pod replica count is the same (2) before and after the chaos execution.
- Check if the Kubernetes nodes are healthy during the chaos execution.
- Check if the latency between checkout and frontend is less than 20ms at any time.
- Check if the latency between the CartService and frontend is less than 20ms at any time.
Step-by-Step Guide
The first steps are signing up at Harness and connecting your application to the Harness control plane.
Step 1: Signup
Sign up at Harness to access the fully featured FREE Chaos Engineering plan. Alternatively, you may have received an invitation from someone in your organization who can access the Harness trail or enterprise plan.
In either case, identify your role and others with admin permissions on the Harness control plane and your application’s host/cluster.
Step 2: RBACs and Permissions
The Chaos Admin role is the easiest way to access the overall Chaos administrative functions. You should also have admin access to the target application cluster to set up the Harness Delegate in the next step, run chaos experiments, and load tests later.
Step 3: Setup a Delegate
Install the Harness Delegate on one of your Kubernetes clusters with network access to the cluster on which the application (in this case, the OnlineBoutique) is running. For help with the Delegate installation, see this link.
Step 4: Discover and Connect your Application
Harness Resilience Testing provides an easy process for onboarding a chaos agent onto a Kubernetes cluster. Follow this link to discover the resources and automatically create basic chaos experiments.
Once the control and execution planes are set, specific steps for chaos experiments and load tests are followed.
Step 5: Create Resilience Probes
We are using three kinds of resilience probes:
- Prometheus probe to check the latency of the services.
- HTTP probe to check if a service is responding to the requests as expected.
- Command probe that is custom-written for checking the node and replica checks.
Example Prometheus Probe:
The Prometheus probe uses Prometheus query to compute the latency, as shown below.
name: boutique-frontend-latency-checktype: promProbepromProbe/inputs: endpoint: http://XX.XX.XX.XX:9090/ query: avg_over_time(probe_duration_seconds{job=\"prometheus-blackbox-exporter\", instance=\"frontend.boutique.svc.cluster.local:80\"}[60s:1s])*1000 comparator: type: float criteria: <= value: "50"runProperties: probeTimeout: 10s interval: 2s attempt: 1 probePollingInterval: 3s initialDelay: 1smode: Continuous
Example Command Probe:
A command probe that checks the number of pod replicas:
name: productcatalog-svc-replica-checktype: cmdProbecmdProbe/inputs: command: kubectl get pods -l app=productcatalogservice -n boutique --no-headers | wc -l comparator: type: float criteria: == value: "1"runProperties: probeTimeout: 10s interval: 2s attempt: 1 probePollingInterval: 3s initialDelay: 1smode: Continuous
The list of resilience probes for checking the resilient state of various services is shown below.
Step 6: Create K6 Load Experiment
The K6 load generator experiment is available as a native chaos fault in the Enterprise ChaosHub of Harness Resilience Testing. It is configured for the load parameters, as shown below.
Step 7: Create chaos experiments for various scenarios
The chaos experiments required for test scenarios are constructed from the native chaos faults like:
- Network Latency
- Network Loss
- API Error Code and
- Pod Delete
The experiments are created using the above faults and tuning them for the targeted services such as Frontend, Checkout, and ProductCatalog in the OnlineBoutique All the resilience probes created above can be attached to each experiment so that the resilient state is checked with each experiment.
Step 8: Run the load test and chaos test in parallel
The Harness Resilience Testing pipeline capability is used to run parallel chaos experiments. "Chaos Step" pulls a chaos experiment from the user's project into the pipeline stage, as shown below.
The pipelines can be run manually or through an API.
Observing the resilience test results
From the execution view, you can get the resilience score and the percentage of successful resilience probes when chaos is injected under load. A sample execution view is shown below.
The summary of resilience probes is shown in the Resilience tab of the pipeline execution view.
In summary, chaos experiments and Grafana K6 load tests are created using the out-of-the-box faults and can be run parallel using the pipelines.
Run Your First Chaos + Load Test Now
Ready to see how your systems behave when it matters most?
Sign up for Harness Resilience Testing and start combining real-world failure scenarios with load testing using Grafana K6. Whether you're preparing for your next peak traffic event or hardening your microservices for long-term resilience, this integrated approach helps you move from theory to reality—fast.
🔧 Get Started Free with Harness Resilience Testing
📚 Need help setting up? Check out the Docs
🚀 Already onboard? Try injecting your first chaos experiment with load today and discover your resilience score.
