Security Testing | Harness Blog Primary Category

August 25, 2026

Technical

Time to Read

I've sat in enough incident bridges as the product person in the room to know exactly what it feels like to be the interruption. You're heads down, you've got three tabs of logs open, you think you're close, and then a message drops in the channel: "hey, any update for leadership?" That's me. I'm sorry. I've also been the guy who had to type up that response himself, at Intuit, at Hulu, pulling together what happened from a Slack scroll and a fuzzy memory two hours after the fact.

Both of those jobs are bad, not because the people doing them are bad at it, but because a human shouldn't have to be the status-tracking layer for a system that already knows what happened.

The response starts before anyone touches anything

Before you ever see a summary, there's a runbook doing the unglamorous part. The moment an incident opens, a set of runbooks fires automatically: a Slack or Teams channel gets created, a notification thread goes out, a video bridge spins up, and the Jira and ServiceNow tickets get created, all within seconds. I've shown that flow enough times that I can tell when it's landed just by watching someone's face when they check the ticket timestamp against the clock on the wall.

Those runbooks aren't a fixed menu either. Underneath, they're Harness pipelines, so if you want a runbook that runs a shell script or a Python job against your own infrastructure instead of just posting a message, you can build that. You can also route on severity, so a Sev4 doesn't spin up the same ceremony as a Sev0, and a runbook can escalate to create that Jira ticket only once someone bumps the severity up.

Nobody should be the scribe

I was working with a customer not long ago and said something I've said probably two hundred times at this point: in most incident response processes, there's a dedicated scribe, or the on-call engineer is doing double duty as one, typing while they're also trying to fix the thing. Every company I've ever worked at or sold into runs some version of this. Somebody's got a doc open, or a template in Confluence, and they're the one responsible for capturing the timeline while everyone else works the problem.

I was in a workshop with an enterprise team a while back and someone on their side put it plainly: they used to have a major incident manager on every single P1 and P2, highly scripted, and every five minutes there had to be a status update. That was the job. Then their team got smart about it and moved off that model, but the underlying problem didn't go away, it just got redistributed onto whoever was closest to the keyboard.

Here's the part people don't expect the first time they see it: the moment there's a video bridge tied to that incident, whether it's Zoom, Teams, or Google Meet, our bot joins it on its own. No one invites it. It also sits in the Slack or Teams channel listening to chat. From there it's pulling key events out of both the talking and the typing, and every line gets reprocessed for clarity so the transcript reads like something a person would write, not a raw caption feed. Say "I think this is related to the deploy we did yesterday" out loud on the bridge and it lands in the record the same as if you'd typed it.

That's what an incident summary that updates itself is for. It removes a job nobody signed up for, and it was never a dashboard feature to me.

Automated incident response is the practice of having your incident platform capture the response record itself: creating the channel and bridge, filing tickets, transcribing the bridge and chat into a running timeline, and generating the postmortem, so responders never stop debugging in order to document what they are doing.

The compliance story that sticks with me

I had a call with a customer's compliance lead who described their postmortem process, and I still think about it. They meant well, but the execution was painful. They had a standard template, but by the time they sat down to write the postmortem, the details were scattered across Slack, Confluence, and Jira, and they'd inevitably miss something. She told me flat out that an auditor once picked an incident at random to check their RCA documentation, and they didn't have it.

That's the real cost here, a team getting caught without the record they were supposed to have, because assembling that record by hand after the fact is hard to do well every single time. Capture the timeline, the key events, and the decisions as they happen instead of reconstructing them from memory a week later, and that gap closes.

Naming things honestly

Small thing, but it says something about how I think about this. Every account I've worked with names their post-incident process differently, and when we built the auto-generated version, I had a real opinion about what to call it. I've never used the word "postmortem" in my career. I try to stay as close as possible to what we're actually trying to do, which is review the incident and find the opportunity to improve, not conduct an autopsy. The tab in the product still says postmortem more often than not because that's the term people search for and expect, but I'll take that fight with product marketing over calling it something someone died from.

I bring this up because it matters to me that the thing writing your incident history doesn't just capture what happened, it captures it the way your team talks about it. A summary that gets the facts right but the tone wrong doesn't get used.

Root cause doesn't come from one place

The summary and the postmortem are the visible part. Underneath, we pull in the same things a good on-call engineer would go check by hand, and stitch them into one picture instead of leaving that correlation to whoever's fast enough to think of it at 2am:

Whatever your runbooks turned up. If a runbook pulled logs or ran a script as part of the response, that output becomes part of the incident's context instead of a result sitting in a separate tab someone has to remember to go read.
ServiceNow change records. One customer put this well during a working session: their CAB-approved changes were well recorded, because that process forces the paper trail, but deployments were a black hole, because nobody was reliably logging what actually shipped and when. We pull ServiceNow change records directly and also generate our own from Harness deployments and the pull requests behind them, so both sides of that gap get covered.
The deployments themselves. When something in a deployment lands close in time to the incident, we surface it as a likely cause, with the pull requests behind it and a plain read of what the change was trying to fix. If nothing in that window explains it, we widen the look-back instead of stopping when the obvious window comes up empty.

None of this replaces the engineer's judgment. It means they start from "here's what changed and when" instead of starting from nothing.

What changes when you turn this on

I've watched this work in real accounts, so this isn't hypothetical:

The incident record has a summary field that updates itself as the incident progresses, built from the chat, the bridge, and the tickets, so nobody has to remember to log it.
When someone says something like "I think this is related to the deploy we did yesterday" on the call, that gets pulled out and attached to the record as a key event instead of evaporating in a Zoom transcript nobody rewatches.
When you close the incident, you already have a postmortem, decorated with the actual timeline, not a blank template staring back at you.
Your ticketing tools stay in sync. I've never worked anywhere, and never sold into an account that ran just ServiceNow or just Jira. Everybody runs both, and somebody has always been the poor soul manually updating each one as the incident moves.
You can run most of this straight from Slack if that's where your team already lives. Add a key event, kick off a runbook, hand the incident to someone else, close it out, all without opening a browser tab.

One customer runs their whole comms process off a custom tab they built themselves, updating it by hand and pushing status emails out to a distribution list on a "pretty routine basis." I asked to see what that email looked like, because I wanted to know how much of that work was busywork versus judgment. Almost all of it was busywork. The judgment was "is this still ongoing," and everything after that was transcription.

The action items don't just sit there

This part matters as much to me as the summary does. Action items get captured the same way, straight out of the incident chat and the postmortem discussion, not just the "let's fix the runbook" kind but the "we should add a metric for this next time" kind that everybody says out loud and then forgets by Monday.

I had a customer tell me their teams stopped using action item tracking inside the product entirely. It worked fine, but it became a second place to check, and their team lead didn't want people tracking incident actions in one tool and everything else in Jira. So we sync those action items straight into Jira or ServiceNow, wherever the team already lives, and we don't stop at capturing them. We complete them. If the action is something the platform can execute, it runs it, closes it out, and logs that it happened, instead of leaving a checkbox for someone to remember three sprints from now.

Why I care about this specifically

I've planted my flag on AI-powered incident response, automation, and communications, and I mean that. I've worked through this with more customers than I can count and I still get a little charge out of it every time, because I know exactly whose Saturday this gets back: the engineer who had to stop debugging to type an update, and the on-call lead who had to write a postmortem nobody read carefully enough to catch the missing detail until an auditor did.

Ops work has always attracted people who are good under pressure and bad at asking for help writing it down afterward. The future of this job is automation, and it should be built by people who've carried the pager, not around them. That's the whole reason I'm still doing this after all these years and all these customer conversations: the person who used to own the status update gets to go back to being an engineer.

See how Harness AI SRE keeps everyone in the loop during your next incident.

‍

Frequently Asked Questions

What does an automated incident summary actually contain?

A running record of the incident built from chat, the video bridge, and the linked tickets: the timeline, key events called out by responders, the deployments and change records in the window, and the decisions made.

Do you still need a scribe on incident calls?

No. Harness AI SRE's Scribe Agent joins the bridge without an invitation and monitors the Slack or Teams channel, capturing key events from speech and text. Responders stay on the problem instead of splitting attention.

How are automated postmortems different from a template?

A template is empty when you open it. An automated postmortem already contains the timeline, key events, and root-cause candidates gathered during the incident, so writing it means reviewing and adding judgment.

Can automated incident response satisfy an audit?

It addresses the common failure, which is missing RCA documentation for an incident an auditor picks at random. Capturing the record as the incident happens produces a consistent trail rather than one reassembled from memory.

Do action items sync to Jira and ServiceNow?

Yes, both. Action items captured from incident chat and the postmortem sync to whichever system the team already uses, and items the platform can execute are run and closed automatically.

Automated Incident Response: Nobody Should Be the Scribe

August 24, 2026

Technical

Time to Read

Resilience Testing (RT) Agents scan your CD pipelines and Kubernetes workloads for resilience risk before it reaches production, then generate and run the chaos experiments or load tests needed to confirm it.

What We're Announcing

We're announcing a major update to our Resilience Testing called RT Agents. Instead of handing you a testing tool and a blank page, our agents now do the first part of the work, often the hardest part, for you. They continuously analyze your deployment pipelines and infrastructure to predict where resilience risks are hiding, tell you exactly what kind of testing is needed to prove it, and recommend next steps. From there, you decide where to take action. When you do, the agents help you get there: generating the chaos experiments and load tests, running them, and reading the results, so you don't have to start from scratch.

Resilience Testing no longer has to start from zero. Harness finds the risk and helps you act on it, directly in your existing pipeline.

Key Features

Passive risk detection in your Harness Continuous Delivery (CD) pipelines

If you're a Harness CD user, this shows up as Agentic Resilience inside the pipeline you already run. RT Agents continuously read your deployment configuration, manifests, and pipeline history to flag resilience risk as it's introduced, before it reaches production. There's nothing to instrument and nothing to run. Detection happens automatically as part of your existing CD workflow, so risk gets caught at the moment it's created instead of months later during an incident.

Simplified onboarding for Kubernetes apps

Getting resilience insight into your Kubernetes workloads no longer requires a lengthy chaos experiment or load test setup process. Onboard your applications through a streamlined flow that connects to your existing cluster and infrastructure metadata, and the agents start surfacing the resilience risks specific to your environment right away. No faults injected, nothing touched in production.

In-product dashboards for Resilience Insights

Every risk the agents detect rolls up into dashboards built for tracking resilience posture over time, not just point-in-time results. You get a consolidated view of risk across services and pipelines, so teams and leadership can see whether resilience is trending in the right direction.

Resilience Scores for your Load Tests

Load testing results now come with a Resilience Score: a single, trackable number that reflects how your services hold up under load, beyond raw throughput and latency numbers. It's built to make load test outcomes easier to compare over time and easier to explain to stakeholders who don't work with performance data day to day.

Java, JavaScript, and Python support for Load Test scripting

You can now write Load Test scripts in the language your team already uses. That removes a common barrier to adoption: teams don't need to learn a new scripting language just to write a load test, so load testing fits more naturally into how your engineers already work.

Free Plan: Try It Out

You don't need to take our word for it. Harness offers a free plan for resilience testing, so you can put it to work in your own environment before committing to anything. It includes a hosted experience with the core capabilities you'd expect: an extensive fault library, a centralized control plane, native integration with Harness pipelines, and enterprise features like RBAC and hosted logging, all at no cost. It's a way to validate the resilience of your own services without the limits of a typical trial account.

If you're new to Resilience Testing, this is the easiest way to see what RT Agents find in your own pipelines. Sign up, connect your infrastructure, and let the agents show you where your risk actually is.

Signup for free.

‍

FAQ

Does RT AI Agents touch production or inject faults during detection?

No. Detection is passive: the agents read your existing deployment configuration, manifests, and pipeline history. Nothing is instrumented and nothing runs against your services until you choose to generate and run a chaos experiment or load test.

Do I need to set up a chaos experiment or load test before RT Agents can find risk?

No. The agents analyze your CD pipelines and Kubernetes workloads on their own and tell you what kind of test would confirm the risk. You only run a test once you decide to act on what they find.

What's included in the free plan for Resilience Testing?

A hosted experience with the full fault library, a centralized control plane, native Harness pipeline integration, and enterprise features like RBAC and hosted logging, all at no cost.

Which languages can I use to write load test scripts?

Java, JavaScript, and Python, alongside Harness's existing load testing options. Use whichever your team already works in instead of learning a new scripting language.

What is a Resilience Score?

A single, trackable number attached to each load test that reflects how a service holds up under load, beyond raw throughput and latency numbers, so teams can compare results over time and report them to stakeholders.

Find Resilience Risks Automatically, Then Confirm Them

August 10, 2026

Technical

Time to Read

Manual intake eats the first 10 to 30 minutes of an incident. Harness AI SRE runs it as automation and gives those minutes back to your responders.

The first minutes go to setup

When an alert fires, most teams spend 10 to 30 minutes on coordination before anyone looks at the problem. The pattern repeats on every incident:

An alert arrives from PagerDuty, Datadog, or another monitor
Someone opens a Jira or ServiceNow ticket
Someone spins up a Slack channel and pulls in responders
Someone starts a Zoom bridge and pastes the link
Someone hunts down the runbook or the last incident's notes

None of that work changes between incidents, and the delay compounds on every high-severity page.

Runbooks turn intake into a triggered workflow

In AI SRE, a runbook is an automated workflow that runs when a trigger fires. You build it in a visual workflow editor instead of writing glue code, with more than 50 pre-built integrations for its action steps. When the trigger condition is met, the runbook executes its steps in order and records each one.
A single intake runbook can:

Create a ticket in Jira or ServiceNow
Open a Slack channel and invite the right responders
Start a Zoom bridge and post the join link to the channel
Set severity, category, and impacted-service fields on the incident
Write every action to the incident timeline as it happens

Setup that used to take 15 minutes runs in seconds, and the timeline leaves an ordered record for the post-incident review.

What sets the trigger off

A runbook fires on any of three signals:

An inbound webhook from a monitoring or alerting tool. On the intake side, AI SRE ships ingestion templates for more than 20 alert sources, including Datadog, PagerDuty, BigPanda, Grafana, New Relic, Sentry, and Opsgenie, plus a generic template for any tool that posts JSON.
Manual creation of an incident or alert, or an on-demand run of the runbook.
A change to an existing incident, such as severity, status, or impacted services.

You scope which alerts launch a runbook with conditions. A trigger can filter on fields, tags, or thresholds through a rule builder or a CEL expression, so a Sev3 CPU blip and a Sev1 outage take different paths. Inside the runbook, a conditional step branches further: when severity is critical, page the secondary on-call team; otherwise the standard logging path runs.

The difference: response sits on your change data

Most incident tools begin with no context. They register that an alert fired and little else.
Platform teams already produce the answer. You build artifacts, deploy them through pipelines, and merge pull requests, and that record is the first place to look when something breaks. AI SRE reads it directly through the Deploy Change Investigator, which connects three streams:

Build events carrying artifact version, commit SHA, and repository
Deploy events carrying the services, environment, and versions that shipped
The pull requests merged into the deployed branch

Once those are linked, the platform can name the deploy that was live when the incident started and diff the pull requests between the last healthy release and the current one. An AI agent reasons over that change history alongside recent feature-flag changes and related alerts. It then surfaces the deploys, PRs, and flags most likely behind the incident, each with a confidence level. It answers the first question in any incident, what changed, from data you already generate.

Runbook actions can act on that answer, too. A step can trigger a Harness pipeline directly (for example, to roll back the suspect deployment), so the same workflow that opened the incident can start the fix.

How to set one up

Build the runbook. Lay out the intake steps in the visual editor and pick the integration each step uses.
Add triggers. Point the runbook at the webhook sources and incident events that should launch it, then add conditions so only the right alerts qualify.
Add actions. Define what runs on trigger: create the ticket, open the channel, start the bridge, set the fields, page the on-call. Every action lands on the timeline.

Start with intake, then keep going

Automating intake removes the fixed coordination cost from the start of every incident and gets responders to investigation in seconds. Because AI SRE sits on the build, deploy, and change data your pipelines already produce, it does more than open the channel faster. It hands the team a first read on what changed.
Human-in-the-loop stays the default. Automation clears the setup and surfaces the likely cause; people decide and drive the resolution.

See how Harness AI SRE automates incident intake or build your first runbook today.

Automate Incident Intake and Start Response in Seconds

August 10, 2026

Technical

Time to Read

An A/B testing framework works best when it lives inside your CI/CD pipeline, not outside of it.
Feature flags, progressive delivery, and guardrail metrics help teams test safely without slowing releases down.
Automation makes it easier to detect regressions early, roll back fast, and keep experiment debt under control.

Most engineering teams still tie A/B tests to major releases, waiting weeks or even months to validate impact while users suffer through broken features. Feature flag interdependencies create hidden technical debt at scale, making big-bang experiments even riskier. Instead: decouple deployment from release entirely.

The most effective A/B testing framework lives inside your CI/CD pipeline, not bolted on afterward. Deploy code with features disabled, then use progressive delivery and real-time metrics to prove impact before full rollout. This approach treats experimentation as a first-class deployment capability, with automated verification wired directly into your release process.

Ready to ship 50x faster with built-in experimentation? Harness Feature Management & Experimentation unifies feature flags, release monitoring, and A/B testing in a single platform.

What is an A/B testing framework?

An A/B testing framework is a tool that lets teams compare a control experience to a variation, divide traffic between the two, measure the results, and then act on those results.

That framework is more than just a dashboard when it comes to software delivery. It has:

Traffic distribution
Flags for features or controls for runtime
Rules for targeting and defining experiments
Tracking exposure
Business and reliability metrics
Logic for promotion and rollback
Rules about who can start, change, or end an experiment

The main point is simple: deployment shouldn't have to mean release. You should be able to send code safely, keep a feature hidden, and then slowly show it to real users when you're ready to learn from them.

Why traditional A/B testing breaks down in modern delivery

Traditional A/B testing often assumes a slower release cycle. Teams launch a change, wait for a handoff, then run an experiment after the fact. That model creates friction for engineering teams that deploy often.

It also creates avoidable problems:

Experiments get delayed until after the release windows
Teams rely on manual traffic changes
Business metrics and system health live in different tools
Rollback decisions take too long
Stale flags and one-off test logic pile up over time

Modern delivery needs something tighter. If teams are shipping daily or several times a day, the A/B testing framework has to move at the same pace. That means the workflow for building, testing, releasing, measuring, and rolling back should live in one delivery process instead of being spread across disconnected tools.

Architecture of an A/B testing framework inside CI/CD

The biggest mistake teams make is treating A/B testing like a separate system that sits beside the pipeline. That usually turns experimentation into another manual checkpoint. Velocity drops. Ownership gets fuzzy. Learning slows down.

A stronger setup keeps experimentation inside CI/CD from the start.

Separate the control plane from the data plane

Your control plane is where you define the experiment: feature flags, audience rules, rollout percentages, mutual exclusion rules, and success criteria. Your data plane is where the experiment runs: flag evaluations, telemetry, exposure events, and outcome tracking.

Keep those concerns separate. That gives teams a cleaner operating model:

Changes to targeting rules do not require a redeploy
Applications keep serving users even if the control plane is unavailable for a short period
Experiment logic stays consistent across environments
Release decisions are driven by data instead of manual guesswork

Wire experiments directly into pipeline stages

A practical A/B testing framework follows the same rhythm every time:

Create the feature flag and experiment definition.
Deploy the code with the feature turned off.
Run smoke, regression, and integration checks.
Expose the feature to a small audience.
Measure business impact and guardrail metrics.
Promote, pause, or roll back based on what the data says.

This flow mirrors how modern pipelines move from commit to validation to progressive rollout. As exposure increases, teams rely on real-time metrics to decide whether to continue ramping, hold steady, or roll back.

This is where CI/CD orchestration matters. Teams using pipeline as code can keep those steps visible, repeatable, and reviewable. And when teams need more flexibility across different targets and environments, capabilities such as Powerful Pipelines and Deploy Anywhere help keep the process consistent.

Store experiment rules as code

If experiment settings are stored in spreadsheets or in tribal knowledge, they will drift.

Store flag definitions, targeting rules, metric choices, and rollout policies in version control alongside the application and delivery config.

That gives you:

A reviewable history of every change
Cleaner audit trails
Repeatable rollout patterns
Fewer ad hoc experiments
Less cleanup pain later

This is also the right place to enforce policy. Policy as code in CI/CD helps teams standardize approval logic, experiment guardrails, and release controls without forcing every decision through manual review. For broader controls across delivery workflows, DevOps Pipeline Governance fits naturally here.

How an A/B testing framework works in 6 steps

A clean framework does not have to be complicated. It just has to be disciplined.

1. Start with a real hypothesis

Do not test just because you can. Define what you believe will happen and why. For example, reducing the number of fields on signup will increase completed registrations without increasing fraudulent signups.

That kind of hypothesis gives the experiment direction. It also tells you what to measure.

2. Choose one primary metric and a small set of guardrails

Every experiment needs a main outcome metric. It also needs a few metrics that tell you whether the change is causing harm somewhere else.

A solid setup usually includes:

One primary business metric
Two to four guardrail metrics
Optional diagnostic metrics for deeper analysis

3. Build the control and variation

Keep the comparison clean. If you change too many variables at once, the result gets muddy fast. This is one reason feature flags work well in delivery pipelines. They let teams isolate a change, ship it safely, and expose only the behavior they want to test.

4. Split traffic deliberately

Traffic allocation should be planned, not improvised. Early ramps should stay small until the change proves it is stable. From there, teams can widen exposure in steps. If your environment supports more advanced routing, traffic routing for A/B testing, canary, and blue-green deployments can make that progression much easier to manage.

5. Run the experiment long enough to learn something real

Ending a test too early is one of the fastest ways to make a bad decision look smart.

Set expectations up front for:

Minimum sample size
Minimum test duration
Stop conditions for regressions
Stop conditions for a clear winner

6. Decide what happens next

A useful A/B testing framework does not stop at analysis. It tells the pipeline what to do next. That action might be:

Promote the change to a larger audience
Keep the test running
Pause and investigate
Roll back automatically
Retire the variation and document the result

Best practices for A/B testing in DevOps workflows

The best experimentation programs are not just fast. They are predictable.

Here are the practices that matter most:

Write the hypothesis before the rollout starts. If the team cannot explain what success looks like, the test is not ready.
Use feature flags as release controls. Deploy first, then decide who sees the change and when.
Ramp exposure gradually. Start small, then increase only after the data looks healthy.
Define business metrics and system metrics together. Conversion means very little if latency, errors, or support tickets spike.
Make rollback automatic where possible. Teams should not need to watch dashboards by hand to know when something is going wrong.
Document the result even when the test fails. A losing experiment can still save the team from shipping the wrong thing.
Clean up flags on purpose. Every experiment should have an owner, a review date, and an end state.

This is also where CI performance matters. Faster feedback loops make experiments easier to run and easier to trust. Teams can shorten validation time with tools such as Test Intelligence, Incremental Builds, and Analytics & Insights.

For teams thinking through rollout patterns more broadly, it helps to understand where blue-green, canary, and other deployment strategies fit into the overall delivery model.

Metrics that matter in an A/B testing framework

Not every metric belongs in the decision path. A useful A/B testing framework usually groups metrics into three buckets.

Primary metrics

These answer the main business question.

Examples include:

Conversion rate
Signup completion
Revenue per visitor
Feature adoption
Retention

Guardrail metrics

These tell you whether the experiment is causing damage while chasing the primary goal.

Examples include:

Error rate
Latency or response time
Crash rate
Checkout failures
Support volume
Abandonment rate

Diagnostic metrics

These help the team understand why a change won or lost.

Examples include:

Click-through rate
Time on task
Drop-off by step
Session depth
API-level health signals

Teams also need a clean way to view these signals together. That is where CD Visualize DevOps Data can help delivery teams line up rollout activity with operational outcomes.

AI-powered automation for safer experimentation

Automation matters because humans are slow at repetitive judgment calls, especially during live rollouts.

In a modern A/B testing framework, automation should help with a few specific jobs:

Correlate exposure data with health and performance signals
Flag regressions during percentage ramps
Enforce promotion criteria consistently
Trigger rollback when guardrails fail
Reduce the need for teams to babysit dashboards

This is especially useful during progressive delivery, where the question is not just “Did the feature work?” But also, “Should we widen exposure right now?”

For that part of the workflow, AI-Assisted Deployment Verification is a natural fit. It helps teams move from manual inspection to a more consistent release decision model.

Common A/B testing mistakes in CI/CD

Even the best teams can get caught in some common traps. Most of these mistakes aren't made with bad intentions. They happen when you move too quickly, read early signals too confidently, or think of experimentation as a one-time launch task instead of an ongoing process.

Starting without a clear hypothesis

The analysis will be vague if the team isn't sure what they want to happen. A good hypothesis should say what the change is, who it is for, and what effect it will have. Without that, teams often gather a lot of information but still can't figure out if the test worked.

Only measuring business lift

A variant can make conversions better while quietly hurting performance, reliability, or workflows that come after it. That's why experiments need both outcome metrics and guardrail metrics. If latency, error rates, or support volume go the wrong way, the apparent win may not be worth sending.

Stopping tests too soon

When the numbers start to go in the right direction, it's easy to pick a winner. It's much harder to say that the sample is still too small. Before launching the test, teams should agree on the minimum sample size, runtime, and stop conditions so that the test doesn't end on a whim.

Carelessly running experiments that overlap

When users end up in tests that don't match, the data gets messed up quickly. Rules about mutual exclusion are important. User bucketing that is consistent is also important. When experiments overlap and there aren't clear lines between them, teams can't figure out which change caused the result.

Forget about cleaning up

Flags that are never taken down cause confusion, branch logic, and technical debt. There should always be an exit plan for an experiment. That should include an owner, a date for making the decision, and a clear next step: promote the winner, remove the losing path, and get rid of flag logic that isn't needed anymore.

Ship faster with confidence

Adding an A/B testing framework to your CI/CD pipeline isn't just about adding another tool. It's about making decisions about releases with better signals and less guesswork. When teams separate deployment from release, they have more space to safely test changes. They can learn faster without making every release a risk by using feature flags, progressive delivery, metrics, and rollback logic together. That's what a modern framework is really worth. It helps teams ship, measure, and respond all at once.

Harness supports that model throughout the delivery lifecycle with tools for testing efficiency, managing rollouts, verifying, and governance. To go from basic flagging to a more complete experimentation workflow, you should start by making your rollout process easy to measure, repeat, and control.

How to build an A/B testing framework for modern CI/CD pipelines

August 5, 2026

Technical

Time to Read

Most teams find out their system has a weak point the hard way, when it's already down in production. Harness Resilience Testing (formerly known as Chaos Engineering) exists to flip that around. It lets you intentionally inject controlled faults into your services and infrastructure, observe how the system responds, and fix what breaks before a real incident occurs. It comes with 200+ built-in faults, probes, and actions across Kubernetes, cloud platforms, Linux, and Windows, plus an AI Reliability Agent and an MCP server so you can run and analyze experiments from your IDE instead of jumping into a separate UI every time.

In this blog, we’ll cover two updates to the Resilience Testing documentation that help close that gap. First, we’ve brought the Chaos Hub directly into the docs. Second, we’ve introduced a Prompt Libraryin natural language rather than that lets you interact with Harness MCP using natural language instead of memorizing commands.

Here's a walkthrough of both features

Chaos Hub, without leaving the docs

Go to the Resilience Testing docs, click Chaos Hub, then Enterprise Hub, and you get the full catalog of fault, probe, and action templates in one place. Filter by infra type (Kubernetes, GCP, whatever you're running), browse probe templates, and check what action templates exist before you build an experiment instead of guessing.

Prompt Library Integration

The bigger update is the Prompt Library, under the AI section in the sidebar. If you have connected Harness MCP to your IDE or Claude, you already know the hard part isn't the connection; it's knowing what to ask for. The Prompt Library gives you a set of pre-built prompts for common resilience workflows, each with input fields for your org and project name that get inserted into the prompt text as you type. Copy the finished prompt into Cursor, Claude, or wherever you run MCP, and go.

‍

Three worth trying first:

Resilience coverage map

Audits every service in your project for chaos coverage and flags the ones with none. Fill in org and project, paste into your IDE, and it returns experiments grouped by target service, which services have zero experiments, and a priority ranking for what to fix first.

Useful as a first pass on any project you inherited or haven't touched in a while.

Service resilience risk scan

Ranks every service by unmitigated risk and surfaces AI-generated experiment recommendations you haven't acted on. Running this against a payment/banking service set returned a resilience score breakdown by service, a critical/high-risk tier, a list of recommended experiments, and a flag for anything with no new experiments in the last 30 days. It also proposed a sprint backlog from the output, which is the part worth stealing, even if you ignore everything else.

Pod failure experiment design

This one goes further than reporting; it builds and runs an experiment. Give it org, project, environment, infra type, infra name, and target deployment, and it will confirm the target is active, create an HTTP probe, and show you the full experiment configuration before saving. And then it will create and run your chaos experiment. Check out this video to see it in action.

Try it today

Pick one service in your project with no chaos experiments and no MCP history against it. Run the resilience coverage map, then the risk scan, and see what comes back. You'll probably find something closer to the Bank of Anthos result than you'd expect.

More prompts are getting added to the library as we build them out. If there's a specific resilience workflow you keep doing by hand, that's usually a sign it belongs in the library next.

New to Harness Resilience Testing? Explore our documentation to get started.

‍

Chaos Hub in docs, Prompt Library for MCP: what's new in Resilience Testing

August 4, 2026

Engineering

Time to Read

Most teams know how to run a disaster recovery test. Few know how to run a DR program. The gap between the two is what separates organizations that survive incidents from those that scramble through them.

A single test gives you a snapshot. A program gives you a trajectory. It is proof that your resilience is improving over time, evidence for auditors, and the operational muscle to recover predictably when something goes wrong.

If you haven't read the earlier posts in this series, start with our introduction to disaster recovery testing for the fundamentals, and the step-by-step DR testing guide for the operational playbook. This post builds on both, focusing on best practices, automation, and metrics that show whether your program is actually working.

From Ad Hoc Testing to a Mature DR Program

Most DR programs evolve through four stages. In the ad hoc stage, tests happen reactively, usually after an incident or before an audit. In the scheduled stage, tests follow a calendar but still rely heavily on manual coordination. In the automated stage, recovery procedures are codified and validated through pipelines. In the continuous stage, resilience is measured constantly, and improvements compound.

The DR program maturity curve, from reactive testing to continuous measurement.

Mature programs aren't just faster. They're cheaper, more auditable, and more reliable under pressure. They produce documented evidence on demand, surface configuration drift before it causes outages, and turn DR from a quarterly fire drill into a steady operational practice.

The rest of this blog walks through the practices and metrics that get you there.

Establish a Risk-Aligned Testing Schedule

Testing every system at the same frequency wastes resources and burns out teams. The most effective programs match testing depth and cadence to business risk.

Risk-aligned testing pyramid. Higher-impact systems get more rigorous testing.

Critical services

Quarterly or semiannual operational tests that exercise the full recovery path
Monthly automated validations to catch drift between major exercises
Annual full-scale simulations that include cross-functional teams and external dependencies

Moderate tiers

Semiannual tabletop exercises or simulations
Quarterly restore tests in isolated environments to verify backup integrity
Targeted failover drills for individual components or services

Lower tiers

Periodic tabletop exercises focused on process and communication
Component-level checks tied to deployment or configuration changes

Stagger tests across business units to avoid resource bottlenecks. If every team runs a full failover in the same week, no one gets meaningful results. The goal is consistent coverage across the year, not a flurry of activity right before an audit.

The principle underneath all of this is simple. Invest more testing rigor in systems where downtime hurts the most. A risk-aligned schedule keeps your highest-impact systems sharp without exhausting the teams that maintain them.

Continuously Incorporate Lessons Learned

Every test produces data. Mature programs turn that data into changes: runbook updates, dependency fixes, process improvements. Immature programs file the report and move on.

Three practices separate the two:

Maintain a living knowledge base of incidents and test outcomes, including playbook revisions, updated dependency maps, and contact list changes
Convert lessons into measurable action items with named owners and due dates. Vague "we should improve X" notes don't get resolved
Use regression tests to confirm fixes hold and that environmental changes haven't reintroduced old issues

The discipline here matters. Tests that don't produce closed action items are theater. Tests that close items but don't verify the fixes worked are wishful thinking. The combination of capture, act, and verify is what builds genuine resilience over time.

Another underrated practice: share lessons across teams. A failure mode discovered in one service often exists in others. Building a culture where DR learnings circulate widely turns each test into organizational improvement, not just team improvement.

Leverage Automation and Modern Tooling

Automation is what makes the difference between a DR program you can sustain and one that quietly atrophies. Manual coordination doesn't scale, and tests that depend on hero effort eventually stop happening.

Modern programs build automation across four layers:

The four layers of DR automation. Each layer makes the layer above it more effective.

Recovery as Code

Codify your recovery procedures using infrastructure-as-code templates, configuration management tools, and CI/CD integration. When recovery steps live in version control, they're reviewable, repeatable, and resistant to drift. Treat your DR pipelines like any other production code, with tests, reviews, and change management.

Automated Backup Verification

Backups that haven't been restored are unverified. Automate routine restores into isolated sandboxes and run integrity checks on the restored data. The point isn't just to confirm files exist. It's to confirm the data is usable for recovery.

Chaos Engineering

Apply chaos engineering selectively to validate failure domains and surface hidden service-to-service couplings. Start with controlled experiments in non-production environments, then graduate to production once teams trust the process. The goal is to find weaknesses in your schedule, not in the attacker's or the cloud provider's.

Observability and Multicloud Orchestration

Use observability platforms to capture metrics, logs, and traces during tests. This is your evidence trail for audits and your debugging trail for after-action reviews. For hybrid and multicloud environments, adopt orchestration tools that coordinate failovers across clusters, regions, and providers. The operational complexity is too high to manage by hand.

How Harness Brings This Together

Building each of these capabilities separately is expensive and slow. Harness Resilience Testing consolidates chaos testing, load testing, and disaster recovery testing into a single platform that plugs into your existing pipelines.

Instead of stitching together separate tools for chaos experiments, load generation, and failover validation, teams orchestrate the full resilience workflow in one place. Recovery steps run as pipeline stages. Chaos experiments and load tests share the same environment, evidence trail, and reporting. The Harness Resilience Testing documentation walks through how to set this up end-to-end, including integration with existing CI/CD workflows.

The result is a DR program that fits naturally into how engineering teams already work. That is the single biggest predictor of whether a program gets sustained over time.

Align with Security, Compliance, and Legal

DR testing isn't just an engineering concern. It intersects with security policies, compliance frameworks, and legal obligations, and mature programs treat those teams as partners, not afterthoughts.

Map your test evidence to the control frameworks that apply to your business. ISO 22301, NIST SP 800-34, HIPAA, and PCI DSS all have specific requirements for documented testing, evidence retention, and remediation tracking. Aligning your evidence capture with these requirements up front saves enormous time at audit.

Ensure data handling in test environments complies with privacy and retention policies. Production data in non-production systems is a common audit finding, even when used for legitimate recovery validation. Use synthetic or properly masked data when possible.

Coordinate with legal and communications teams for customer-facing test scenarios and any required external notifications. If a test could trigger customer-visible behavior or contractual obligations, the conversation needs to happen before the test, not after.

Finally, don't forget SaaS and managed services. Many critical workloads depend on third-party providers whose recovery capabilities you can't directly control. Validate that contractual recovery promises actually hold by testing them. Verify contact paths, escalation procedures, and any vendor-side failover commitments.

Track Metrics That Actually Matter

If you can't measure your DR program, you can't improve it. The right metrics turn DR from a series of one-off exercises into a function with clear performance signals.

Track these across every test cycle:

Five DR metrics that signal whether your program is actually working.

Recovery Time Performance

Time to initiate recovery and time to restore services, measured against your RTO targets. Trend these over time. A program is improving if the gap between the target and the actual closes. Persistent gaps point to runbook problems, automation gaps, or unrealistic targets that need revisiting.

Data Loss vs. RPO

Measure actual data loss against your RPO and check for any integrity discrepancies in restored data. RPO is often treated as theoretical, just a metric on a slide deck. Real testing turns it into an operational number you can defend.

Automation Coverage

Track the number of manual interventions per test and the trend in that count. Manual steps are where tests slow down and where they break under stress. A healthy program steadily replaces manual coordination with automated workflows.

Remediation Health

Defect recurrence rates and closure times for action items tell you whether your lessons-learned process is actually working. If the same issues keep surfacing across tests, the after-action discipline isn't yet in place. If items take months to close, ownership and prioritization need attention.

Customer Impact Indicators

During tests, capture error rates, latency, and any degradation in user experience. Even in controlled exercises, these indicators reveal blind spots in your recovery design. They show you places where the system technically recovers, but the customer experience suffers.

Review these metrics regularly. Quarterly is typical for most teams. Dashboards help, but the discussion matters more than the visualization. Trends matter more than snapshots. And every metric should connect to a specific improvement initiative. Tracking numbers nobody acts on is just noise.

Resilience Is a Compounding Discipline

A mature DR program isn't built in a single quarter. It's built through small improvements that compound over time: sharper runbooks, more automation, faster recovery times, cleaner audit evidence. Each cycle should make the next one easier and more revealing.

The best programs treat resilience as a competitive advantage, not compliance overhead. They recover faster than their competitors. They demonstrate trust to customers, regulators, and insurers with documented evidence. They give engineering teams the confidence to ship faster because the safety net is real.

If you're just starting out, return to the basics with our introduction to disaster recovery testing, or work through the operational playbook in the step-by-step DR testing guide. And whenever you're ready to consolidate chaos testing, load testing, and DR testing into a single platform, Harness Resilience Testing is built to make that consolidation straightforward.

Disaster Recovery Testing Best Practices: How to Build a Metrics-Driven Resilience Program in 2026

July 21, 2026

Technical

Time to Read

Native pipeline step - appears alongside Build, Test, and Deploy in the step palette
50+ evaluation metrics - OOTB metrics such as faithfulness, safety, hallucination, task completion, or create your own custom metrics
Eval Suites with blocking or advisory pass strategies.

Building agents is easy. Trusting them is hard.

At Harness, we're building software delivery agents across our platform. Getting to a working prototype was fast, in many cases, in a weekend. But building an agent that performs at production-grade, enterprise-scale was a different problem entirely. And getting to a point where we could actually trust that agents would work for our customers the way we expected every time was harder than anything else.

We shipped them to production. And we learned something that every team that builds agents eventually learns.

Existing Tools Can’t Help With AI Evals

As we dug into why this is so much harder than traditional software, we kept hitting the same five walls.

Failures are silent. When traditional software fails, it crashes. You get an error code, a stack trace, and a log entry. When an agent fails, nothing crashes. It returns a confident, plausible, completely wrong answer. No alert fires.

Output is non-deterministic. Traditional software gives you the same output for the same input. Agents don't. Run the same prompt twice, get different results. You can't write assertEqual for a summarization agent.

There's no debug mode. Stack traces tell you exactly why traditional code broke. With agents, you can't trace why it chose one answer over another. The reasoning is opaque. The decision path is invisible.

Quality is a spectrum, not a binary. Traditional tests either pass or fail. Agent quality is: Did it complete the task? Is the tone professional? Is it faithful to the source? Is it relevant? Is it safe? There is no single "pass."

Maintenance is a moving target. Traditional software: fix the code, ship the patch. Agents: the model drifts, the prompts change, the context window shifts, and the LLM version upgrades silently. A fix today can break tomorrow without anyone touching the code.

Your DORA metrics don't measure agent faithfulness. Your test suites pass while quality silently degrades. And your team has no way to A/B test prompts or swap models without shipping blind.

Harness AI Evals: The Details

Harness AI Evals makes core agent quality measurable, enabling agent-aware quality gates in your CI/CD pipelines. With AI Evals, changes to your agent (or underlying model) are evaluated first against your standard data sets and model outputs. The output is evaluated across many dimensions, including correctness, performance, and safety. Teams can then use the scores as quality gates in their CD pipelines to simplify release decisions. Then, production data can be fed back in, improving your testing based on real inputs.

Two evaluation modes, one continuous loop

Offline evaluation (pre-deploy)

Run your agents against golden datasets before they ship. A golden dataset is a curated set of test cases - inputs your agent will receive, paired with the expected outputs or context it should use to respond. Think of it as your ground truth: the known-good answers your agent should produce or stay faithful to.

Score every response using 50+ built-in metrics. For example:
Is the response grounded in the retrieved context (faithfulness)?
Did the agent call the right tools with the right arguments (tool correctness)?
Is the output safe from prompt injection? Did it complete the task?

Is the tone appropriate?

You can also define custom rubrics in natural language or write your own scoring logic in Python.

Compare prompt variants and model versions side by side. Gate your release pipeline: if scores drop below the threshold, the deploy is blocked. Not a script. A native Harness pipeline step.

Online evaluation (post-deploy)

Offline evaluation is just the beginning. It helps you test the agent before you ship, preferably as part of a CI/CD pipeline.

Online evaluation takes it further. It scores the output of your agent against real scenarios coming from your customers, using the same metrics. Instead of getting scores for synthetic data you generated in a dataset, you're scoring how the agent actually behaved against real user inputs. That's how you learn how your agent operates in real life.

And then you can add those real scenarios back into your datasets, so you can use them in subsequent offline evaluations. You're continuously enriching the datasets, ensuring that as you progress with development, the output improves and definitely doesn't regress.

Here's a walkthrough of how it works:

‍

Our first round of manual testing took one to two weeks every release cycle. Now I've put AI Evals in as a release gate. Whenever there's a deployment, it evaluates whether anything broke. What took days takes minutes. The sign-off isn't someone opening a sheet of 500 cases anymore. It's based on the pass rate. Score above threshold? Ship it. Below? It doesn't go out.

- Chetan Sinha, Software Engineer, Harness QPE Team

Governance-ready from day one

AI Evals inherits the full Harness platform:

RBAC on every eval, dataset, and suite
OPA policy governance on eval configurations
Full audit trails on who changed what threshold and when
Secrets management for model API keys via existing connectors
SSO and data residency out of the box
Centralized registry for prompts, agents, MCP tools, and skills with versioning
Pipeline execution link - every eval run links back to the pipeline execution that triggered it

Getting started

Getting started is designed to be fast. A guided onboarding flow walks you through setup. In-built templates for common patterns (prompt injection detection, correctness checks, RAG quality) let you plug and play without writing scoring logic from scratch. And you can synthesize entire datasets from a single description using AI, so you're not hand-writing hundreds of test cases to get started.

Harness AI Evals is the first native quality gate for AI in CI/CD. Score your agents before deploy, monitor them after, and ship with confidence every release.

Request to start the beta!(/demo/ai-evals)

FAQs

How is this different from an observability tool like LangSmith or Braintrust?

Those tools connect your evals to observability, you export traces, run evaluations separately, and interpret the results outside your pipeline. AI Evals runs as a native step in your Harness pipeline, right alongside Build, Test, and Deploy. A quality regression fails the build the same way a failed unit test does. No scripts to wire in, no glue code to maintain.

Do I need one system for testing before launch and another for monitoring in production?

No. Offline evaluation (pre-deploy) and online evaluation (post-deploy) share the same metrics and the same datasets. The scoring logic you use to test an agent before it ships is the exact same logic that scores it in production. That means no gap between what you validated and what you're actually measuring once real users are involved.

What kinds of problems can it actually catch?

Things like hallucinations, unsafe or off-policy responses, incomplete task execution, wrong tone, and incorrect tool usage — scored across 50+ built-in metrics, or your own custom rubrics if you need something specific to your use case. It also evaluates multi-step agent behavior, not just the final answer: did the agent reason through the task correctly and call the right tools along the way?

Ship AI Agents You Can Trust

July 15, 2026

Technical

Time to Read

Runbook best practices haven't changed that much at their core: a good runbook is actionable, accessible, accurate, authoritative, and adaptable. These five attributes separate a runbook your team relies on from one they ignore. What has changed is what happens after you write it. With Harness AI SRE, your runbooks don't just guide responders — they execute automatically, file tickets, trigger rollbacks, and post updates to the incident timeline without anyone manually following a checklist.

What Is a Runbook?

A runbook is a step-by-step guide for performing a task in a system, whether you're seeing it for the first time or coming back after months away. You reach for it during on-call rotations, service disruptions, or when onboarding a teammate.

This article covers runbooks for software systems and incident response automation — not airplanes or surgery.

When to Use a Runbook

Runbooks earn their place whenever a process is too nuanced or variable to fully automate. Even with strong SRE automation, some steps still need human judgment. Runbooks cover that gap — giving you structure without assuming automation handles everything.

Common use cases include:

Investigating or stabilizing an incident before a full root cause analysis
Running complex business processes, like generating a monthly billing report
Handling repetitive but critical dev tasks, like setting up a test environment

Runbook Best Practices: The Five Attributes of a Good Runbook

1. Actionable

A runbook should tell you what to do next. Each task should be:

Clear, concise, and goal-oriented
Written for whoever will use it — new hires, mid-level engineers, or senior SREs
One completable step at a time, with no compound instructions

When someone needs deeper context, link out to reference docs. Keep the runbook focused on action.

Good: SSH into the database server and run tail -f /var/log/db.log

Bad: Log in to the database server, edit the config file, and restart the process.

For incident runbooks, add a follow-up step like an RCA or retrospective so what you learn makes it back into the runbook and your wider operations.

2. Accessible

A runbook nobody can find during an outage might as well not exist.

Make runbooks easy to find:

Associate them with alerts or services
Tag them with metadata: type (incident, maintenance, onboarding), creation and last-update timestamps, author or owner, linked systems
Make them searchable from Slack, your terminal, or your incident tool

In AI SRE, runbooks are pinned to incident types or attached to alert rules so they surface automatically — the right runbook appears at the moment it's needed, with no searching required.

3. Accurate

Outdated runbooks lose people's trust. Lead an engineer down the wrong path once and they won't come back.

Keep runbooks accurate:

Make updates lightweight, via PRs, comments, or an edit button
Track both last-updated and last-used timestamps
Have engineers validate steps before publishing, and copy-paste commands rather than retyping them
Link usage history, like associated incidents or alerts, where you can

AI SRE logs every runbook execution step by step — inputs, outputs, and status — tied directly to the incident timeline. When a step fails, it shows up in the timeline rather than going unnoticed, making it easy to trace what needs updating.

4. Authoritative

One process, one runbook, no duplicates.

When multiple versions exist, consolidate them and archive the outdated copies. If a section needs to be reused across processes, link to it instead of copying it.

Add a simple way to flag problems. If someone hits a conflicting or misleading step, they should know how to report it.

5. Adaptable

Systems change constantly, and runbooks have to keep up.

Assign clear ownership per runbook or section
Open contributions to the team where it makes sense
Build runbook updates into retrospectives and deployment checklists
Call out the runbooks that save time or prevent an incident
Automate the high-confidence sections once you trust them

Treat a broken runbook like a broken test and fix it right away.

Spotting Stale Runbooks

Signs a runbook has gone stale:

A last-updated timestamp older than 12 months
No recent use, or no link to a recent incident
Feedback or comments flagging problems

If it's outdated but still needed, update it. If the system it documents is gone, archive it: mark the title with [ARCHIVED] and move it to a separate folder.

Runbooks in Harness AI SRE

A runbook in AI SRE is a set of steps that execute during an alert or incident. Each step acts on a connected system or on the incident record, and its result is posted to the incident timeline. The same runbook that pages the on-call can also file the ticket and run the Harness pipeline that ships the fix.

This is the part a static runbook document cannot do: it can tell a responder to roll back, but it cannot run the deploy itself. Harness AI SRE closes that gap — transforming your runbook automation from a reference document into an active participant in incident resolution.

How a Runbook Is Built

Each runbook is an ordered chain of steps. A step does one of four things:

Runs an action against a connected system
Sets a field on the incident
Branches on a condition
Loops over a list

Steps take typed inputs and pass their outputs to later steps. If a step fails, an error path runs. You build runbooks in a visual editor.

Actions a Step Can Call

AI SRE includes built-in actions that a step can call without custom integration work. They cover the systems an incident touches:

Communication: Slack, Microsoft Teams, Google Chat, Zoom, email, SMS
Ticketing and paging: Jira, ServiceNow, PagerDuty, Opsgenie, Jira Service Management
Automation: run a Harness pipeline, toggle a feature flag, set a Harness connector, post an incident review, resolve an alert

Running a Harness Pipeline as a Step

AI SRE has a native step that executes a Harness pipeline. You give the step a pipeline and its input YAML, and it runs your rollback or hotfix deploy inside the incident response. The step checks the caller's pipeline-execute permission, optionally waits for the run to finish, and posts the execution link and status to the incident timeline.

Because Harness owns the CI/CD pipeline, the runbook reaches it directly — no separate integration to configure.

Getting the Right Runbook to the Incident

Two mechanisms put a runbook in front of responders without anyone searching for it (a key incident response automation principle in Harness AI SRE):

Pinned runbooks: Pin runbooks to an incident type, and they appear for one-click execution whenever that type of incident opens.
Alert-rule attachment: Attach a runbook to an alert rule with its inputs pre-filled, and it runs automatically when the alert fires.

A runbook can also be set to trigger on incident lifecycle events through a rule condition.

Tracking What Ran

Every runbook execution is logged step by step, with its inputs, outputs, and a status of running, success, or failed. The record is tied to the incident timeline, so a responder can see what ran, when, and what it returned. A step that fails shows up in the timeline rather than going unnoticed.

Bottom Line

Runbooks are an operational safety net. They cut cognitive load and pass institutional knowledge to whoever's on call. Automation keeps growing, but plenty of situations still need a human in the loop — and that human needs clear, current instructions.

Get the five runbook best practices right and your team recovers faster with less on-call stress. Pair them with Harness AI SRE and those runbooks stop being documents people read — they become automated workflows that execute the moment an incident opens, reducing MTTR and keeping your team focused on the work that actually requires human judgment.

How to Build Runbooks That Work — and Automate Them with Harness AI SRE

June 9, 2026

Technical

Time to Read

Your Harness pipelines, logs, and deployment approvals are now a sidebar panel away inside VS Code.

The Harness VS Code Extension is live on the VS Code Marketplace today, no .vsix download, no manual install. Search "Harness" in the Extensions view, and you're a click away from real-time CI/CD visibility without leaving your editor.

Screenshot related to: The Harness VS Code Extension is live on the VS Code Marketplace today, no .vsix download, no manual install.

Everything Software Delivery in One Panel

Capability	What it does
Pipeline monitoring	Live status for active runs, with automatic git context detection, executions for your current branch and commit surface automatically.
Log viewer	Click any pipeline step to open its logs in a dedicated editor tab, syntax-highlighted. Failed steps are flagged immediately.
Inline approvals	Approve or reject Harness native, Jira, and ServiceNow deployment gates directly in the editor. No navigating to the UI.
AI-assisted debugging	Ask IDE-integrated Cursor, GitHub Copilot, or Claude about a failure. Pipeline context (name, status, execution ID, URL) is injected automatically. No copy-pasting.

Ask Your AI. It Already Has the Context.

When a pipeline fails, the default loop is: open Harness UI, find the execution, read the logs, copy the relevant output, open your AI assistant, paste, and ask. That's four context switches before you've started fixing anything.

The extension collapses that into one step. An input sits at the bottom of the Harness panel. Type your question, select Claude Code, GitHub Copilot, or Cursor from the dropdown, and the extension packages the current execution context automatically before sending.

‍

Screenshot illustrating: Ask Your AI. It Already Has the Context.

‍

What makes the context useful, not just present, is the Harness Software Delivery Knowledge Graph. The Knowledge Graph is a structured data model that connects every entity across your SDLC: pipelines, services, deployments, environments, artifacts, policy results, and more. When the extension sends your AI tool the execution context for a failing pipeline, it's pulling from that graph. So Claude Code, Copilot, or Cursor isn't just reading a raw log dump. It's receiving structured, relationship-aware data about what ran, what it depends on, and where it broke. That's the difference between an AI that can technically answer a question about your pipeline and one that can accurately answer it.

‍Claude Code responses appear directly in the Harness sidebar (CLI mode) or open the Claude Code panel with the prompt pre-loaded (extension mode). Click Configure MCP in the AI footer to wire up your Harness credentials: project scope or global, your choice.

‍GitHub Copilot is auto-detected when the extension is installed. Context and prompt open in Copilot Chat, ready to go.

‍Cursor is auto-detected when you're running inside Cursor. For the simplest setup, install the Harness plugin from the Cursor marketplace. OAuth authentication, no manual configuration.

Install in Two Minutes

Install:

Open the Extensions view (Ctrl+Shift+X), search "Harness", and click Install. Or from the terminal:

code --install-extension harness-inc.harness-vscode

Connect your account:

Click the Harness icon in the Activity Bar → run Harness: Configure API Key → enter your instance URL and Personal Access Token. Your Account ID is extracted from the PAT automatically.

Select your org and project. Pipelines load immediately.

‍Requirements: VS Code 1.85.0+, active Harness account.

Watch it in action

Watch the walkthrough from our very own Luis Redda.

Stay in VS Code. Your Pipelines Will Follow.

The context-switching loop (open Harness, find the execution, copy the log, switch to your AI tool, paste, and ask) doesn't have to be part of how you work. Pipeline status, logs, approvals, and AI-assisted debugging all live in the same panel as your code. Install the extension, connect your account, and the next time something breaks, you'll already be where you need to be.‍

For more information, checkout the docs.

From Commit to Approval, Without Leaving VS Code

June 1, 2026

Partners

Time to Read

AI agents are only as good as the context they operate in. That's not a design philosophy. It's a practical constraint. An AI agent that doesn't understand how the underlying software delivery entities relate to each other, or what the data actually means, will get things wrong. In software delivery, wrong looks like a botched deployment, a misread failure, or an approval granted when it shouldn't have been, which directly affects your users.

Today, we're announcing that the Harness MCP Server is in the official Claude Connectors Directory, making Harness discoverable and connectable for every team using Claude. But the announcement isn't really about the directory listing. It's about what Harness + Claude can actually do in your delivery system.

Screenshot related to: Today, we're announcing that the Harness MCP Server is in the official Claude Connectors Directory, making

What You Can Do with Claude and Harness

Claude can work across the full Harness delivery platform:

Capability	What Claude can do
Pipeline execution	Trigger and monitor builds across GitHub, GitLab, Bitbucket, or Harness Code
Deployment management	Promote services across environments with approval gate verification
Failure diagnosis	Pull structured execution context and surface root cause analysis
Approval workflows	Retrieve pending approvals and take governed delivery actions
Environment state	Query what's deployed where, in real time
Security posture	Review SBOMs, vulnerability scan results, and SSCA compliance status
Resilience testing	Initiate chaos experiments and retrieve structured results
Cost signals	Surface cloud cost anomalies tied to deployment activity

‍

Screenshot illustrating: What You Can Do with Claude and Harness

All of it is grounded in the Knowledge Graph, not raw API responses, but a structured model of your delivery system that Claude can reason over precisely.

The Problem With Giving AI Agents Raw API Access

MCP lets AI models call external tools by reading API descriptions and deciding which to invoke. That flexibility is useful. But when you're building an agent that needs to reason across an entire software delivery lifecycle, CI, CD, security scans, approvals, feature flags, cost signals, and environments, raw API access creates a deep reliability problem.

Consider a question a platform engineering lead might ask:

‍"Show me the pipelines with the highest failure rate over the last 30 days, and for each one, tell me which services they deploy and whether any of those services have open critical vulnerabilities."

That question spans four domains: pipeline execution history, service-to-pipeline relationships, environment state, and security scan results. An agent working off raw APIs has to discover which APIs exist across each domain, call them in the right order, paginate correctly, infer how field names correspond across systems, and synthesize the results without misinterpreting nested objects or guessing at relationships.

The result is 5+ sequential LLM calls, hundreds of thousands of input tokens, high latency, and an agent that had to guess at every join. Guessing is where hallucinations happen.

What the Harness + Claude Integration Changes

The Harness Software Delivery Knowledge Graph is a purpose-built model of everything that happens after code is written: builds, test runs, deployments, approvals, security scans, environment states, feature flags, infrastructure changes, cost signals, and rollbacks. Not as raw data but as a connected, typed, semantically annotated graph of entities and relationships.

Every field in the graph carries metadata that tells an agent exactly how to use it: whether a value is a number or a string, whether it can be aggregated or only filtered, what its unit is, and how it joins to related entities. Cross-module relationships, between a pipeline and the services it deploys, between a deployment and the security scan results for that artifact, between an environment change and the cost anomaly that followed, are explicitly declared, not inferred.

This is the difference between an agent that can access your delivery system and one that understands it.

When Claude connects to Harness via MCP, it doesn't receive a set of API endpoints. It's getting access to a structured model of your entire delivery organization, one where the relationships are known, the data types are enforced, and the agent can construct precise queries rather than guessing at field semantics.

‍The practical effect with Harness + Claude: that same cross-domain question above becomes 2–3 structured queries against a known schema. The agent selects the right entity types from the graph, generates queries with exact fields and declared relationships, and returns a deterministic answer. No guesswork. No hallucinated field names. No silent wrong answers.

What This Looks Like in Practice

Debugging a failed pipeline without context switching

A build has failed. Normally, you'd open the Harness UI, navigate to the execution, copy the relevant logs, paste them into a conversation, and wait for analysis. The AI reasons over whatever you managed to capture.

With the Harness MCP connection active in Claude, you ask what failed. Claude doesn't just pull logs; it queries the Knowledge Graph to understand the structure of that pipeline, which stage failed, what services were involved, whether similar failures have occurred before, and what changed since the last successful run. The answer it surfaces reflects the full delivery context, not just the stack trace you happened to copy.

Screenshot illustrating: Debugging a failed pipeline without context switching

Promoting a deployment through governed gates

Your team is ready to move a service from staging to production. Claude checks the current environment state, verifies that required approval gates have been satisfied, confirms the security scan passed for the artifact version you're promoting, and initiates the deployment — with every action running through your existing RBAC policies and logged for audit.

The agent isn't guessing about whether conditions are met. It's querying a graph where those conditions are modeled as typed relationships with known states. The answer is deterministic because the data is structured to make it so.

This Is Not AI Without Guardrails

The natural question when Claude can trigger pipelines and manage deployments: what stops it from doing something it shouldn't?

The same controls that govern everything else in Harness. Every action taken through the MCP server runs through your existing RBAC permissions, OPA policy enforcement, approval gates, and audit logging. Claude operates with exactly the permissions you have, nothing more. Every action is tracked. Nothing bypasses the governance layer.

The Knowledge Graph reinforces this: because Harness AI understands your delivery system structurally, it also understands the constraints within it. Approval gates aren't just optional steps the agent might skip; they're modeled as typed relationships with state. The agent can't promote past a gate that hasn't cleared because the graph reflects that clearly.

Speed and governance aren't a tradeoff. They coexist by design.

Why the Claude Connectors Directory Matters

The Claude Connectors Directory is a curated, reviewed set of integrations. Anthropic evaluates each server before listing it. Being approved is a signal of trust that carries weight for enterprise teams deciding which AI integrations to enable.

It also means discoverability at scale: engineering teams using Claude for DevOps workflows will find Harness natively. One-click OAuth connection, no API key management, no manual configuration.

This fits a broader pattern. The Google Cloud partnership brought Harness into Google's AI ecosystem through Vertex AI and Gemini CLI. The Cursor plugin brought it into the IDE. The Claude Connectors Directory brings it into conversational AI. In each case, the goal is the same: wherever developers are doing their best thinking and wherever AI is being asked to help with software delivery, Harness should be present with the right context for that AI to act reliably.

Getting Started

If you're already a Harness customer:

Open Claude and then the Connectors page
Search for Harness in the MCP directory
Authenticate with OAuth, no API keys, no manual configuration
Start asking Claude about your pipelines, deployments, and delivery workflows

If you're new to Harness, sign up for free and connect from day one. Detailed steps are listed in the documentation.

The Harness Connector gives Claude the ability to act in your delivery system. The Knowledge Graph gives it the understanding to act well. Together, that's what reliable AI in software delivery actually looks like.

Software Delivery Context, Now Inside Claude

May 29, 2026

Technical

Time to Read

The 10 Best Feature Flag Tools for 2026

Releasing new software used to be a big deal. You would set aside a Saturday night, wake up the on-call engineer, push the code, and hope that nothing broke before Monday morning.

Then came feature flags, which changed everything without anyone noticing.

Feature flags let you separate deployment from release, so you can send code to production in a dormant state and turn it on for users when you're ready. No more 1 a.m. maintenance windows. We don't have to ship every feature in a release together anymore, or scramble to pull one back with a hotfix. Just code in production, off by default, and ready when you say so.

But the tools have improved a lot. Feature flag tools these days are more than just on/off switches. The best ones have flag management, progressive delivery, real-time release monitoring, A/B testing, and AI-driven guardrail metrics all built right into your CI/CD pipeline. That changes how a release looks, how a rollback feels, and how confident your team is when they ship.

Here's a look at the best feature flag tools available, along with what each one does well and what to look for when picking the right one for your team.

What Feature Flag Tools Really Do

A feature flag, or feature toggle, is a conditional block in your code that controls whether a new feature is active for a given user. Wrap a flag around a checkout page redesign, and you can push the code to production while keeping the new flow hidden from 99% of users. Set it to 1% as a canary, monitor your metrics, and gradually increase the rollout percentage if everything looks good.

Feature flag tools handle the whole lifecycle: creating flags, targeting users, rolling them out incrementally, monitoring their impact, and retiring flags once they've served their purpose.

Modern platforms add a few more layers on top of that:

Progressive delivery. Instead of releasing everything at once, release features to bigger groups of users over time, based on performance metrics.
Experimentation. Use proper sample size calculations and significance testing to run statistically sound A/B tests.
Release monitoring. Find out how feature exposure affects error rates, latency, and business KPIs in real time.
Governance. RBAC, audit trails, and approval workflows for organizations operating in regulated industries.

The toggle itself isn't worth much. The safety net around it is.

What to Look for in a Feature Flag Tool

Before you start looking at different tools, make sure you know what your team really needs. Some questions you should ask are:

Does it work with the CI/CD pipeline you already have? Your developers will work around a flag platform that is outside of your delivery workflow, not with it.

Can it connect flag exposure to your observability stack? You don't want three dashboards to cross-reference when something breaks at 3 a.m. You want one screen that tells you which feature caused the spike.

Will it scale with your traffic and your team? When you have millions of users, SDK performance, evaluation latency, and offline fallback are all important.

Does it cover governance for regulated environments? In healthcare, fintech, or anything touching PII, RBAC, approval workflows, immutable audit trails, and Policy as Code aren't optional.

How does it handle flag lifecycle management? Stale flags are technical debt. The best platforms include ownership assignment, sunset policies, and dashboards that surface flag age and usage frequency.

With those criteria in mind, here are the best tools to consider.

The 10 Best Feature Flag Tools

1. Harness Feature Management & Experimentation (FME)

Harness FME is a developer-first platform that brings feature management, A/B testing, and release monitoring into one unified system. Built on the combined Split and Harness lineage, FME is designed for enterprise teams that want experimentation baked into their CI/CD pipeline not bolted on as a separate workflow.

What makes FME stand out:

Unified flags and experimentation. Feature management and A/B testing share the same flag, SDK, and data pipeline. No parallel systems to reconcile.
AI-driven release monitoring. Release monitoring automatically connects flag exposure to error rates, latency, and business KPIs. You know which feature broke something right away, not hours later.
Warehouse-native experimentation. Run analysis directly on your Snowflake, BigQuery, or Databricks data, so experiment results live alongside the rest of your business intelligence.
Automated rollback and progressive delivery. If p95 latency climbs 10% for 84 seconds, FME handles the rollback automatically while you sleep.
Enterprise governance. RBAC, SAML federation, immutable audit logs, and approval workflows for regulated industries.

Best for: Enterprise engineering teams that want a single platform for feature flags, experimentation, and release monitoring, with deep CI/CD integration.

2. LaunchDarkly

LaunchDarkly is one of the oldest feature flag platforms on the market. It's a popular choice for teams that want a flag-first product with mature SDK support for most major languages.

Some of its strengths are that it has a lot of SDK support, good targeting options, and a long history of managing features. Some teams may prefer other vendors for bundled analytics or warehouse-native analysis. Teams that do a lot of A/B testing often use LaunchDarkly with a separate analytics or stats engine, which makes things more complicated.

Best for: Teams whose primary need is feature flag management, with separate tooling for testing and observability.

3. Statsig

Statsig has become a popular platform for product-led growth teams. Statsig is a popular platform for product-led growth teams because it has a free tier that includes feature flags, experimentation, and product analytics all in one place.

The platform's statistical engine is good. It can do sequential testing and has a good way of testing for significance. With warehouse-native mode, you can analyze your own data infrastructure. Statsig is still growing in enterprise governance, but its RBAC and audit features aren't as strong as those found in regulated industries.

Best for: Product-led growth teams that want flags, experiments, and analytics in one system without heavy enterprise requirements.

Ownership note: Statsig announced in September 2025 that it would join OpenAI. OpenAI said Statsig would continue operating independently and serving current customers, so buyers may want to watch how the roadmap evolves under new ownership.

4. Optimizely Feature Experimentation

Optimizely's roots are in web-based A/B testing, and it brings that history of experimentation into its feature flag product. The platform's statistical methods are well-established, and marketing teams that have used other Optimizely products are likely to choose it.

The downside is that you can see where Optimizely came from in some places. The product is more useful for web and front-end use cases and less useful for the kind of deep backend, infrastructure-level flag management that engineering teams often need. More developer-native tools tend to work better for product engineering teams that only work on products.

Best for: Marketing-engineering hybrid teams already invested in the Optimizely ecosystem who want to extend it to product feature testing.

5. PostHog

PostHog is an open-source platform that bundles product analytics, feature flags, experimentation, and session replay together. It's a popular pick for early-stage companies that want a lot of capability without paying for multiple platforms.

The all-in-one approach works well at a smaller scale. As you grow, you may find that specialized tools go deeper on individual capabilities particularly enterprise-level flag management and statistical rigor. The self-hosted option is a meaningful advantage for teams with strict data residency requirements.

Best for: Startups and growth teams that want product analytics and feature flags in one place, with a self-hosting option.

6. Flagsmith

Flagsmith is a feature flag platform that is completely open source and can be hosted in the cloud or on your own server. It's a good choice for teams that need open-source flexibility (or strict self-hosting) but don't want to lose the polished product experience.

The platform does a good job of covering the basics, like targeting, segmentation, multivariate flags, and SDK support for most languages. It's not as heavy as enterprise platforms when it comes to advanced experimentation, AI-driven release monitoring, and deeply automated guardrails.

Best for: Teams with privacy requirements, self-hosting mandates, or a strong preference for open-source software.

7. Unleash

Unleash is another open-source option with a strong following in Kubernetes-native shops. It's known for being straightforward to set up, easy to understand, and well-suited to teams that want full control over their tooling.

Like Flagsmith, Unleash handles flag management well but doesn't extend as far into experimentation or release intelligence. If your team primarily needs to safely gate features and host the platform yourself, Unleash is a solid choice.

Best for: Open-source-first teams, especially those running Kubernetes infrastructure.

8. ConfigCat

ConfigCat markets itself as a simple, inexpensive feature flag service with clear prices and an easy setup. A lot of small to medium-sized teams choose it because they want to manage flags without the extra work that comes with a bigger platform.

The product includes the basics, such as targeting, segmentation, percentage rollouts, and connections to popular tools. It wasn't made to be a testing platform, so teams that need statistical analysis will have to use it with something else.

Best for: Small-to-midsize teams that want light-weight, budget-friendly flag management without enterprise complexity.

9. GrowthBook

GrowthBook is an open-source feature flag platform originally built around warehouse-native experimentation. The premise: your experiment data is already in BigQuery, Snowflake, or Redshift, so it should be analyzed there rather than piped to a separate vendor.

For data teams that have invested heavily in their warehouse, GrowthBook is a strong fit. The statistical methods are rigorous. Bayesian and frequentist options, sequential testing, CUPED variance reduction, and the open-source model gives you full control over the platform.

Best for: Data teams that want serious warehouse-native experimentation with open-source control.

10. AWS AppConfig

AWS AppConfig is Amazon's native configuration and feature flag service for teams operating entirely within the AWS ecosystem. It integrates cleanly with Lambda, ECS, EKS, and EC2, and runs as a fully managed service under your existing AWS account.

The trade-off is depth. AppConfig treats flags as part of broader application configuration. It isn't a purpose-built platform for experimentation or release intelligence. Teams that need advanced targeting, A/B testing, and release monitoring at the level of a dedicated tool will outgrow it quickly.

Best for: AWS-native teams with modest flag requirements who want to stay within the AWS ecosystem.

How to Pick the Right Feature Flag Tool for Your Team

Once you've narrowed down your list, here are a few things to think about.

Match the tool to your scale. A platform that works for a 10-person startup probably won't work for a business with 500 engineers, and the other way around. Check how well the SDK works when it's under load, how deep the governance is, and how the platform handles thousands of flags across hundreds of services.
Look for pipeline-native integration. If turning on a flag means a developer has to stop what they're doing and do something else, that flag won't be used as much. The best platforms let you manage flags like GitOps and trigger updates with CLI commands or pipeline steps.
Build in flag hygiene from day one. Old flags are a type of technical debt. Look for dashboards that show the lifecycle of a project, policies about when to end a project, and who is responsible for what. Amazon requires flag removal tasks to be done when the task is created, which is a good idea to copy.
Plan for governance before you need it. RBAC, audit trails, approval workflows, and policy-as-code may seem like too much for a small project, but they cost a lot to add later. Get the governance bench set up early.
Run a two-week pilot with one team before rolling out company-wide. You can learn more about a platform in two weeks with just one engineering team than you can with a dozen vendor demos. Don't just look at how well it works on its own; make sure it fits with your current tools.
Tie your tool choice to KPIs. You should be able to measure the tool you choose by how often it is deployed, how often it fails to change, how long it takes to recover, and (ideally) how it affects business outcomes for specific experiments. It's hard to explain why you spent the money if you can't connect it to those numbers.

Stop Guessing and Start Shipping with Confidence

Feature flag tools started as a clever way to ship code that wasn't quite ready without breaking production. They've grown into something much larger: the foundation for safer releases, faster experimentation, and a development culture where shipping doesn't feel like gambling.

The best platforms bring feature flags, progressive delivery, real-time monitoring, and AI-driven guardrails together in one place integrated with your CI/CD pipeline so every release becomes a controlled experiment rather than a leap of faith.

Harness Feature Management & Experimentation brings flags, experimentation, and release monitoring into a single enterprise-grade platform, with AI-driven guardrails and deep CI/CD integration built in. Every deployment becomes a measurable, recoverable experiment instead of a gamble.

Feature Flag Tools Compared: 10 Best Platforms for Safer Releases

May 27, 2026

Technical

Time to Read

The Problem with Running Playwright at Scale

Playwright has become the industry standard for end-to-end testing. Most engineering teams already have suites (sometimes hundreds of specs) running against their applications.

Writing the tests isn't the hard part anymore. Running them reliably, at CI speed, with meaningful feedback when things break: that's where teams still struggle.

The numbers tell the story:

50% of pull requests encounter at least one flaky test failure (Slack Engineering, 2022)
95 minutes: the p95 wait time for test results before Slack's CI pipeline rework
Multiple moving parts to self-host Playwright at scale: CI runners with browser dependencies, Docker images, shard configuration, retry logic, and compute scaling
Zero automated root-cause analysis: when tests fail on traditional grids, engineers get raw logs and screenshots, nothing more

Teams at Google, Dropbox, and Spotify have each built dedicated internal systems just to manage test flakiness and infrastructure. That's engineering investment that should go toward the product.

Bring Your Playwright Suites to Harness. No Rewrites.

Harness AI Test Automation now lets you bring your existing Playwright projects and run them natively on the platform.

Your playwright.config, your spec files, your package.json scripts stay in your repo, exactly where they live today. Point Harness at your project root, and we run your suite using your config, extending it with reporters and trace settings that power AI triage and the Tests tab. No code changes required.

Why this matters:

Teams have invested months, often years, building and stabilizing their Playwright suites. A testing platform shouldn't ask you to throw that away and start over. Your stable tests stay exactly as they are. Tests that are flaky or hard to maintain can gradually evolve into AI-generated intent-based tests when you're ready, but there's no rewrite tax to get started.

What Changes When Playwright Runs Inside Your Pipeline

No Infrastructure to Manage

Run in the cloud with parallel workers. No grid to configure, no nodes to scale, no browser images to maintain. Need to test an application behind a firewall? Secure tunnels handle private apps without exposing your network.

AI Failure Triage, Not Raw Logs

When a test fails, Harness automatically classifies it: regression, flaky, performance, or environment issue. You get the failure location, retry patterns, likely root cause, and a recommended fix. No more sifting through stack traces to figure out if the problem is real.

Engineers spend time fixing problems, not investigating whether the problem is real.

AI Assertions via Harness SDK

Some assertions are hard to express in code. "Does this page look correct?" "Is the checkout flow in a valid state?" "Does the error message make sense for this scenario?"

With the Harness SDK, you can add AI-powered assertions directly into your Playwright scripts. Hard-to-write assertions become simple natural-language questions. No complex selector logic, no brittle pixel comparisons. Your scripts stay in Playwright. The assertions just get smarter.

Tests as a First-Class Quality Gate

Playwright runs are native pipeline steps, not a service bolted onto your CI. If tests fail, the pipeline fails. Code is blocked from production. Every deployment is validated, every result is tied to a specific commit.

No context switching to an external dashboard. Results live in the pipeline's Tests tab, alongside your build and deploy stages.

Shared Visibility Across the Team

When Playwright runs locally, one developer's test results are invisible to the rest of the team. Failures get investigated in isolation. Patterns go unnoticed. Knowledge stays siloed.

On Harness, every execution is visible to every developer. Teams can review each other's test runs, spot recurring failures together, and build a shared understanding of test health across the entire suite.

Full Commit-to-Deploy Visibility

Test results are connected to the commit that triggered them and the deployment they validated. When something breaks in production, you can trace back through the exact test run, the exact code change, and the exact environment, all in one place.

How This Differs from External Test Execution Services

Most external test execution services solve one problem well: running browsers at scale. But they leave you to stitch together the rest. CI integration, reporting, triage, and quality gating are your responsibility.

With native pipeline integration:

Results live where engineers already work. No switching between your CI tool and a separate test dashboard.
Quality gates are automatic. Tests block deployments by default, not by custom webhook configuration.
AI triage is built in. You don't need a separate observability tool to understand why tests failed.
No per-session pricing. Run as many parallel workers as your pipeline needs.
A path forward. Scripts that are flaky or unmaintainable today can graduate to intent-based AI tests without migrating to a different vendor or rewriting your suite.

Capability	Self-hosted Playwright	BrowserStack / LambdaTest	Harness AI Test
Infrastructure management	You own it	Managed	Managed
AI failure triage	None	None	Built-in
Pipeline-native quality gates	Manual	Webhook	Native
Per-session pricing	N/A	Yes	No

Playwright for Execution, AI for Everything Else

This isn't about choosing between scripted tests and AI. It's about using each where it's strongest.

Playwright delivers the reliable, repeatable execution your Harness CI/CD pipeline demands. Harness AI layers intelligence on top: triaging failures so you don't waste cycles investigating, generating assertions that would be painful to hand-code, and eventually creating new test cases from your requirements and code.

Bring your Playwright suite to Harness AI Test Automation. Connect your repo, point us at your project root, and run your first execution in minutes -- with AI failure triage included.

Interested to try this out. Please reach out to ait-interest@harness.io

Security Testing Agent Blogs

Featured Blogs

Software Delivery Context, Now Inside Claude

What You Can Do with Claude and Harness

The Problem With Giving AI Agents Raw API Access

What the Harness + Claude Integration Changes

What This Looks Like in Practice

Debugging a failed pipeline without context switching

Promoting a deployment through governed gates

This Is Not AI Without Guardrails

Why the Claude Connectors Directory Matters

Getting Started

Latest Blogs

The response starts before anyone touches anything

Nobody should be the scribe

The compliance story that sticks with me

Naming things honestly

Root cause doesn't come from one place

What changes when you turn this on

The action items don't just sit there

Why I care about this specifically

Frequently Asked Questions

What does an automated incident summary actually contain?

Do you still need a scribe on incident calls?

How are automated postmortems different from a template?

Can automated incident response satisfy an audit?

Do action items sync to Jira and ServiceNow?

Automated Incident Response: Nobody Should Be the Scribe

What We're Announcing

Key Features

Passive risk detection in your Harness Continuous Delivery (CD) pipelines

Simplified onboarding for Kubernetes apps

In-product dashboards for Resilience Insights

Resilience Scores for your Load Tests

Java, JavaScript, and Python support for Load Test scripting

Free Plan: Try It Out

FAQ

Does RT AI Agents touch production or inject faults during detection?

Do I need to set up a chaos experiment or load test before RT Agents can find risk?

What's included in the free plan for Resilience Testing?

Which languages can I use to write load test scripts?

What is a Resilience Score?

Find Resilience Risks Automatically, Then Confirm Them

The first minutes go to setup

Runbooks turn intake into a triggered workflow

What sets the trigger off

The difference: response sits on your change data

How to set one up

Start with intake, then keep going

Automate Incident Intake and Start Response in Seconds

What is an A/B testing framework?

Why traditional A/B testing breaks down in modern delivery

Architecture of an A/B testing framework inside CI/CD

Separate the control plane from the data plane

Wire experiments directly into pipeline stages

Store experiment rules as code

How an A/B testing framework works in 6 steps

1. Start with a real hypothesis

2. Choose one primary metric and a small set of guardrails

3. Build the control and variation

4. Split traffic deliberately

5. Run the experiment long enough to learn something real

6. Decide what happens next

Best practices for A/B testing in DevOps workflows

Metrics that matter in an A/B testing framework

Primary metrics

Guardrail metrics

Diagnostic metrics

AI-powered automation for safer experimentation

Common A/B testing mistakes in CI/CD

Starting without a clear hypothesis

Only measuring business lift

Stopping tests too soon

Carelessly running experiments that overlap

Forget about cleaning up

Ship faster with confidence

How to build an A/B testing framework for modern CI/CD pipelines

Chaos Hub, without leaving the docs

Prompt Library Integration

Resilience coverage map