How to Build Runbooks That Work — and Automate Them with Harness AI SRE | Harness Blog

Harness AI is starting 2026 by doubling down on what it does best: applying intelligent automation to the hardest “after code” problems, incidents, security, and test setup, with three new AI-powered capabilities. These updates continue the same theme as December: move faster, keep control, and let AI handle more of the tedious, error-prone work in your delivery and security pipelines.

What’s New in Harness AI:

• Human-aware incident analysis that correlates conversations with changes
• AI-driven API naming that reduces security noise
• Natural-language auth script generation for faster AST onboarding
• AppSec agent for querying security data and generating policies

Human-Aware Change Agent for AI SRE

Harness AI SRE now includes the Human-Aware Change Agent, an AI system that treats human insight as first-class operational data and connects it to the changes that actually break production. Instead of relying only on logs and metrics, it listens to real incident conversations in tools like Slack, Teams, and Zoom and turns those clues into structured signals.​

• The AI Scribe captures key decisions, timestamps, symptoms, and “right before this happened…” moments from live conversations, filtering out unrelated chatter.​
• The Change Agent uses these human signals to drive a change-centric investigation across deployments, feature flags, config, infra changes, and ITSM records, then produces evidence-backed hypotheses such as, “This checkout deployment changed retry behavior 12 minutes before the incident, and latency spiked immediately after.”​​

By unifying human observations with the software delivery knowledge graph and change intelligence, teams get a much faster path from “what are we seeing?” to “what changed?” to “what should we roll back or fix safely?” The result is shorter incidents, clearer ownership, and a teammate-like AI that reasons about both people and systems in real time.​ Learn more in the announcement blog post.

AI-Powered API Naming for Cleaner Security Signals

Effective application security starts with knowing what you actually have in production. Traditional API naming based on regex heuristics often leads to over-merged or under-merged API groups, noisy inventories, and false positives across detection workflows.​

This month, API naming in our Traceable product gets a major upgrade with AI-powered API semantics:

• API naming is now powered by LLMs that understand intent, behavior, and functional semantics, not just URL or path similarity. The result is more stable, meaningful API groupings that reflect how your services actually behave.​
• The LLM-driven results were baselined against custom naming rules from advanced users and achieved >98.7% average match in internal benchmarking.​
• With cleaner API groupings, teams see reduced false positives across vulnerability detection, AST, and runtime protection, and a less noisy API inventory that’s easier for security and platform teams to act on.​

For security leaders trying to tame API sprawl, this is a foundational improvement that boosts signal quality across the entire platform.​

AI-Based Auth Script Generation: Faster, Safer API Security Testing Setup

Authentication setup has been one of the most consistent sources of friction for application security testing. Manual scripting, validation cycles, and back-and-forths often create bottlenecks — and a broken auth script can quietly invalidate an entire scan run.​

To solve this, all API Security Testing customers now get AI-based Authentication Script Generation:

• Generate auth scripts by simply describing the scenario in natural language; AI produces a ready-to-use script in a few seconds, which you can refine, edit, or use as a base for existing scripts.​
• The feature works alongside existing flows, so teams can keep using form-based or code-based auth with identical behavior while layering in AI where it helps most.​​

The result is less time lost to brittle auth setup, faster onboarding for new apps, and fewer failed scans due to script errors.​

You can find implementation details and examples in the docs. 

Chat with AppSec Agent: Security Data, in Plain Language

Security and platform teams often know the question they want to ask: “Where is this component used?” “Which exemptions are still pending?” , but answering it requires hopping across dashboards and stitching together filters by hand.​

The new AppSec Agent makes this dramatically easier by letting you query AppSec data using natural language.

Here's what it does:

• In Harness STO, you can ask about security issues and exemptions, then drill into issue-level insights from STO results without manually navigating views or composing complex filters. Questions like “Approve all valid pending exemptions in this project of issue type secret” become a single prompt instead of a multi-step workflow.​
• In Harness SCS, you can query for artifacts, code repos, SBOMs, chain-of-custody, and compliance results, then even generate OPA policies with a single prompt to block components based on license risk or vulnerable packages. For example, “Create an OPA policy to block the deployment of components licensed under the GPL-3.0 license” or “Help me identify whether the chalk and xz-utils components are present in any of the artifacts in this project” are fully supported.​
• The AppSec Agent is available across all production environments and integrates directly with Harness Security Testing Orchestration (STO) and Software Supply Chain Assurance (SCS).

This is a big step toward making AppSec data as queryable and collaborative as the rest of your engineering stack. Learn more in the docs.​

How This Fits the Harness AI Vision

Harness AI is focused on everything after code is written — building, testing, deploying, securing, and optimizing software through intelligent automation and agentic workflows. January’s updates extend that vision across:

• Security and AppSec: higher-fidelity API grouping, fewer false positives, and faster AST onboarding with AI-generated auth.
• SRE and Operations: human-aware incident response that unifies human and machine signals into a single, change-driven flow.
• Governance and Compliance: consistent with December’s AI governance updates, all of these capabilities inherit Harness’s approach of policy-aware AI, auditability, and RBAC-aligned actions.

Teams adopting these features can ship changes faster, investigate less, and focus more of their time on the work that actually moves the business — while Harness AI quietly handles the complexity in the background.

Checkout Event: Harness at RSAC

DevOps and SREs have dashboards to monitor services and product performance, and now it’s time for us to understand how to measure these teams’ performance as well.

One significant challenge every company faces these days is measurement. There’s a need to understand what’s happening at every level inside the company and product, from how customers are using the application, to the quality and efficiency of the code, and down to the team’s performance.

When it comes to measuring DevOps and SRE teams, we’re faced with a whole new challenge. They’re in charge of the delivery funnel, and it’s their job to measure and make sure it’s working as it should. From the developers who write the code, to the tools that test and deploy it, and down to the way the product behaves in the real world.

While DevOps and SREs measure performance, making sure every step in the application lifecycle is functional, we need to understand how to measure them in return. We told you it’s a challenge, but we also have some good news – It’s possible, as long as you focus on what’s important.

What Should We Measure?

DevOps and SREs have to stay on top of everything that’s happening inside the application. They need to have a real-time monitoring system, that will help them see the application uptime, load time, number and success of API calls, CPU process threads, memory usage and other metrics.
And while it is their job to make sure everything is up and running, we want to make sure they’re doing it as expected. To do so, we also need to look across the entire application and workflow to find the answers and data about every parameter we’re interested in.
These usually include:

• We want to make sure that dev teams are delivering faster than before, by looking at the cycle time.
• We need to know that fast deployments are not hurting the quality of the code, which can be measured by the availability of the product.
• We want to monitor the product’s quality by looking at the rollback percentage.
• And of course, we want to make sure our users and customers are happy, which can be done by looking at the rollback percentage or complaints sent.

Each one of these parameters contains a world of metrics and calculations, and trying to monitor all of them is like trying to photograph an entire fireworks show on the Fourth of July: you can do it, but you’re missing the point.

To help us help ourselves, we need to narrow down what we’re looking at. And since our goal is to measure our own teams and operations, it’s easier to take a step back and have a broader look at everything. Now, let’s turn these analogies into practices.

Borrowing Google’s Focal Points

DevOps and SREs have to monitor a lot of different aspects of the application, but that doesn’t mean that we need to monitor every single one of them as well. Furthermore, it doesn’t mean we need a number of dashboards just to understand whether the team is doing their job or not.

To narrow this down, we can adopt Google’s approach to measuring its SRE teams. The company encapsulates all of the elements needed to monitor DevOps and SRE into three essential measurements, each with its own baseline:

Service-Level Objective (SLO)

In Google, the Service-Level Objective (SLO) is a number or a percentage that indicates system availability. It’s an indication of whether the system is running as it should, and whether the product is stable or not.

This number will help us understand the state and quality of our product, as well as ensure the quality of our code as we push deployments faster. Part of the responsibilities of DevOps and SRE teams is to maintain application reliability and functionality. This metric clearly represents how successful the team is in accomplishing that goal.

Service-Level Indicator (SLI)

This metric measures the failures per request, by calculating request latency, the throughput of requests per second, or failures per request as measured over time. It connects to the SLO number that was determinate, and helps evaluate if the team is within its SLO.

With visibility into metrics like system availability, it’s easier to understand when errors and failures occur. The logical next step is seeing what causes errors and failures. It will allow us to monitor how reliable our service is, and we will be able to do so by looking at the same stats that the DevOps and SRE teams are measuring.

Service-Level Agreement (SLA)

A Service-Level Agreement (SLA) is an agreement between you and your users/customers, that indicates the availability of the services and products. Unlike SLOs and SLIs, this is a loose metric that can change according to the service you provide, or the customer you’re providing it to.

The SLAs should derive from the SLOs, since you want to make sure you have your own definition and understanding of the system availability, before you make contracts and promises with your customers. Metrics-wise, monitoring the SLAs will help us understand whether DevOps and SREs are keeping up with the numbers they set up for themselves.

Final Thoughts

You want your product to be good, your customers to be happy and your company to succeed. But how will you know if you’re on the right track without attaching the correct numbers and metrics to it?

It’s a challenge to understand how to monitor DevOps and SREs, but you need to be able to measure everything that’s a part of your product – and these teams are a big part of it.

While each company has its own set of requirements, methods and team structures, focusing on monitoring SLOs, SLIs and SLAs will help you understand what metrics to focus on, and how your teams are performing.

Contact a Harness expert

Get a demo

Checkout AI-SRE

Learn more: Effective strategies for proactive incident prevention in SRE, Guide to capacity planning for site reliability engineering

• SRE codifies reliability through SLIs, SLOs, and error budgets, balancing deployment speed with system stability through measurable targets.
• AI-powered CD and GitOps platforms automate verification, rollbacks, and policy enforcement, reducing toil while accelerating incident recovery.
• Start with SLOs for one critical service, add intelligent rollbacks, then scale with policy-as-code guardrails for safe, rapid delivery.

A single second of latency can cost e-commerce sites millions in revenue, while just minutes of downtime trigger customer churn that takes months to recover. Modern users expect instant responses and seamless experiences, making reliability a competitive feature that directly impacts business outcomes.

Site Reliability Engineering treats operations as a software problem rather than a manual discipline. SRE applies engineering principles to achieve measurable reliability through automation. 

Ready to implement SRE practices with AI-powered deployment automation? Explore how Harness Continuous Delivery provides intelligent verification and automated rollbacks that transform reliability from theory into practice.

What Is Site Reliability Engineering (SRE)?

Site Reliability Engineering (SRE) was born at Google to scale services for billions of users, providing concrete frameworks for balancing speed with stability.

SRE: Engineering Discipline That Codifies Operations

Instead of relying on manual processes and undocumented institutional knowledge, SRE codifies operational work through automation, monitoring, and measurable reliability targets. SRE teams write code to manage infrastructure, automate incident response, and build systems that automatically recover when possible.

The Language of Reliability: SLIs, SLOs, and Error Budgets

The engineering approach of SRE relies on three fundamental concepts that quantify reliability.

• Service Level Indicators (SLIs) measure what users actually experience, such as page load times or checkout success rates. 
• Service Level Objectives (SLOs) set specific targets for these metrics, such as "99.9% of requests complete within 200ms." 
• Error budgets represent the acceptable failure rate that remains after meeting your SLO. 

When you burn through your error budget too quickly, it signals time to slow down deployments and focus on reliability improvements rather than new features.

Why SRE Matters for Microservices and High-Frequency Releases

Microservices architectures create cascading failure scenarios that traditional operations can't handle at scale. SRE addresses these challenges through:

• Progressive delivery strategies, like canary releases, detect 87% of service-impacting issues before full rollout, limiting the impact of failures.
• Automated rollbacks reduce recovery time from an average of 57 minutes with manual processes to just 3.7 minutes, preventing widespread outages.
• AI-driven verification shortens mean time to detection by 47% and resolution by up to 63% by automatically correlating metrics, logs, and traces under real traffic conditions.
• Error budgets provide the framework teams need to balance speed with safety, enabling daily or hourly deployments while maintaining service availability targets.

The Origins of SRE

SRE began at Google around 2003 when Ben Treynor Sloss, a software engineer, was asked to run a production team. Instead of hiring more system administrators, he approached operations as an engineering problem. As Sloss famously put it, "SRE is what happens when you ask a software engineer to design an operations team."

Google enforced a strict operational work limit for SREs, ensuring time for automation projects. These principles spread industry-wide through foundational SRE texts, starting with the 2016 publication of "Site Reliability Engineering: How Google Runs Production Systems." Today, SRE principles integrate seamlessly with cloud-native and GitOps patterns, enhancing tools like Argo CD with reliability guardrails rather than replacing existing investments.

Core SRE Principles

High-performing teams don't choose between speed and safety. They achieve both through disciplined engineering practices. The core principles of SRE make this balance measurable, repeatable, and scalable.

Reliability Through Measurable Targets

How do you know when you're reliable enough? When is it safe to deploy versus when you should pause? Error budget policies answer these questions with concrete thresholds that trigger escalating responses:

• At 64% budget consumption within a four-week rolling window, tighten approval processes and require additional review for risky changes
• At 100% budget exhaustion, halt all non-critical deployments until the service recovers within its SLO targets
• Monthly budget resets with full audit trails showing which services consumed the budget and why
• Policy as Code enforcement ensures consistent application across all services without subjective exceptions
• Automated remediation triggers canary rollbacks or traffic shifts when budget burn correlates to specific microservices

This approach transforms error budgets from reactive limits into proactive reliability controls.

Automation-First Mindset

Eliminating toil is fundamental to SRE success. This means reducing manual, repetitive work that scales linearly with service growth. Google limits SRE teams to 50% operational work, forcing automation investments.

Here's how to reduce toil systematically:

• Measure toil percentage of each SRE's time monthly, targeting under 50% initially and driving toward 20%.
• Automate deployment verification with AI-powered health checks that connect to your observability tools.
• Implement automated rollback triggers when anomalies are detected, eliminating manual intervention during incidents.
• Create golden path templates with continuous delivery platforms that let developers self-serve without writing custom scripts.
• Track and celebrate toil elimination wins. Treat deleted work as engineering victories.

The goal isn't zero toil. It's ensuring valuable engineering work always outweighs the mundane.

Controlled Risk and Safety Nets

SRE embraces controlled risk through progressive delivery strategies like canary deployments and blue-green releases. These approaches expose changes to small user populations first, detecting issues before full rollout. Automated rollbacks serve as primary safety nets. When anomalies are detected, systems revert to known-good states without human intervention. This combination of gradual exposure and rapid recovery enables higher deployment frequency while maintaining reliability targets.

Key SRE Practices

Essential practices in Site Reliability Engineering address the core challenges every SRE faces: reducing deployment anxiety, accelerating incident recovery, and preventing issues before they impact users.

Incident Management: From Chaos to Learning

Effective incident response follows the three Cs: coordinate, communicate, and control. 

Here's how to implement structured incident management:

• Assign clear roles during incidents (incident commander, communications lead, operations lead) to reduce response time and prevent confusion.
• Align response time expectations with service criticality: 5 minutes for user-facing systems and 30 minutes for less critical services.
• Pre-write runbooks and escalation paths to eliminate decision latency during production outages.
• Enrich alerts with context by using systems that automatically correlate alerts with recent deployments, service ownership, and probable root causes, reducing MTTR by up to 85%.
• Conduct blameless postmortems immediately after incidents, documenting impact, root causes, and follow-up actions without individual blame.
• Capture specific contributing factors, detection gaps, and assign action items with owners and deadlines. Treat each incident as valuable learning that prevents future occurrences.

When postmortems become a cultural practice, organizations see faster recovery times with measurable improvements.

Progressive Delivery and Automated Rollbacks

Progressive delivery transforms risky big-bang releases into controlled, measurable rollouts. Modern canary deployments shift traffic incrementally while automated systems verify each step and trigger instant rollbacks when needed.

Here's how modern progressive delivery works in practice:

• Start small and grow gradually: Deploy to 10% traffic, then 25%, then 50%, and finally 100% while checking SLIs at each gate.
• Enable AI to select your metrics: Automated verification connects to Datadog, New Relic, Dynatrace, and Prometheus without writing complex analysis templates.
• Trigger instant rollbacks: Anomaly detection identifies issues within seconds and reverts automatically.
• Verify under real traffic: Production validation catches problems that staging environments miss.
• Reduce blast radius: Progressive traffic shifting limits the impact of failures to small user populations.

Observability: The Foundation of Reliable Systems

Focus monitoring on the four golden signals: latency, traffic, errors, and saturation. This approach detects regressions under real traffic conditions by integrating metrics from application performance monitoring, logs from centralized aggregation, and traces from distributed systems. Focus alerts on user-impacting symptoms rather than internal system states. This unified observability approach enables teams to validate changes against actual user experience and catch issues before customers notice them. Begin by instrumenting these four signals across your most critical services.

SRE vs. DevOps: What's the Difference?

Teams often ask how SRE differs from DevOps, especially when both disciplines focus on improving software delivery. While DevOps emerged as a cultural movement to break down silos between development and operations, SRE provides the engineering discipline and measurable frameworks to operationalize reliability at scale.

How SRE and DevOps Work Together

In practice, SRE and DevOps work together rather than compete. Teams implementing comprehensive SRE automation report 82% faster incident response and 47% fewer change failures. SRE operationalizes DevOps principles through platform engineering and GitOps:

• Platform engineering builds the infrastructure highways (internal developer platforms and golden paths).
• SRE acts as the traffic control system (defining SLO thresholds, error budgets, and verification criteria).
• GitOps handles declarative deployment mechanics while SRE provides governance guardrails.

The breakthrough happens when SRE policies become enforceable guardrails within platform tooling. Policy-as-code transforms SRE requirements like freeze windows and SLO gates into automated checkpoints that GitOps workflows execute without manual intervention. Organizations combining SRE and platform engineering see measurable improvements in uptime and recovery time. Development teams deploy more frequently while experiencing fewer customer-visible incidents.

Building an SRE Team

When deployments happen multiple times per day, manual verification becomes impossible and deployment anxiety spreads across engineering teams. Building the right SRE team means assembling engineers who can automate reliability work and eliminate toil.

Essential Skills: Engineers Who Automate Reliability

Look for engineers who blend coding skills with operational experience. These people can write Python or Go scripts to automate deployment checks, understand how services fail across networks, and know which metrics actually matter when things go wrong. They build safety features directly into applications, like circuit breakers that stop bad requests from spreading, or feature flags that let you turn off broken features instantly. Most importantly, they treat reliability problems as engineering challenges that need permanent fixes, not just quick patches.

Team Topologies: Central, Embedded, and Hybrid Models

SRE team structure fundamentally comes down to where reliability expertise lives in your organization:

• Central SRE teams build shared platforms, define policy standards, and create automation that scales across services. Think observability frameworks, deployment verification, and incident response tooling.
• Embedded SREs work directly within product teams, coaching developers on reliability practices and implementing service-specific improvements.
• Hybrid models combine both approaches. A small central team establishes reliability standards and provides AI-powered verification platforms, while embedded SREs implement and adapt these practices for their specific services.

Research across 145 organizations shows that hybrid SRE models report 87% better knowledge sharing and 79% improved operational efficiency compared to single-model approaches. Choose your structure based on organization size, service count, and reliability maturity. Startups often start embedded, enterprises lean central, but most successful organizations evolve toward hybrid models as they scale.

Getting Started with SRE

Learning how to implement SRE best practices doesn't require transforming your entire organization overnight. The most successful adoptions follow three focused steps: select a critical service and establish reliability targets, implement intelligent rollback capabilities, and create self-service guardrails. This approach proves value quickly while building confidence for broader SRE adoption across your microservices architecture.

Pick One Service and Define Your First SLOs

Choose one business-critical application that's actively developed and provides comprehensive monitoring and metrics. Define SLOs from your users' perspective: 99.95% availability, 95th percentile latency under 200ms, or error rates below 0.1%. Use a four-week rolling window for evaluation and document your error budget policy with specific actions when budgets are exhausted. 

Implement Intelligent Rollback Capabilities

Treat AI-powered rollback as your first must-have milestone. It immediately reduces release risk and builds confidence for high-frequency deployments. Context-aware platforms can detect anomalies instantly and trigger self-healing responses without human intervention, turning a potential 15-minute manual recovery into a 30-second intelligent response.

Codify Guardrails with Policy as Code

Policy as Code transforms operational rules into version-controlled artifacts that run in your CI/CD pipeline. Use tools like Open Policy Agent to enforce security baselines, block risky configuration changes, and verify deployment rules before production. Create reusable pipeline templates that embed these policies, allowing teams to self-serve while maintaining compliance. 

A 90-Day SRE Adoption Plan

Breaking down SRE adoption into focused sprints makes the transformation manageable and delivers measurable improvements. This phased approach builds reliability practices incrementally without disrupting daily operations.

• Days 1-30: Define 3-4 customer-facing SLIs, set realistic SLOs (start with 99.9%), and establish clear incident roles with escalation policies.
• Days 31-60: Deploy canary strategies with automated health checks, integrate observability tools for real-time verification, and enable automated rollback on anomaly detection.
• Days 61-90: Implement error budget policies that gate risky changes, introduce blameless postmortem templates, and create self-service deployment templates.
• Ongoing: Track toil reduction percentage, MTTR improvements, and SLO achievement rates to measure progress and justify continued investment.

Common Pitfalls and How to Avoid Them

• Pitfall: Alerts tied to raw error rates instead of meaningful SLO breaches create noise that exhausts teams and influences turnover.

• How to avoid: Tie alerts to SLO breaches and burn rate consumption (such as 2% of your error budget in one hour) rather than arbitrary thresholds. This ensures alerts fire only when customer experience suffers, not when internal metrics fluctuate.
• Pitfall: Custom bash scripts for each service create technical debt that compounds with scale and becomes impossible to maintain consistently.

• How to avoid: Use reusable templates and centralized policies to codify best practices once and apply them everywhere. This eliminates the burden of maintaining service-specific scripts.
  
  
• Pitfall: Creating and maintaining service-specific monitoring scripts for deployment verification consumes significant SRE time and creates inconsistency.

• How to avoid: Leverage AI-powered platforms to automatically generate verification profiles that connect to your observability tools, eliminating manual script creation while ensuring reliable rollback procedures.

SRE Tools and Technologies

Traditional SRE tools force teams to choose: comprehensive features or operational simplicity. Modern platforms eliminate this tradeoff by integrating observability, delivery automation, and AI-powered verification into unified workflows that scale reliability practices without scaling headcount.

Observability: From Dashboard Watching to Automated Correlation

Enterprise observability suites like Datadog, New Relic, and Dynatrace automatically correlate metrics across services, while Prometheus and Grafana provide the open-source foundation for time-series collection and visualization. OpenTelemetry has become foundational for unified instrumentation, enabling teams to collect metrics, logs, and traces without vendor lock-in while supporting automated anomaly detection.

GitOps and Delivery: From Argo Sprawl to Centralized Control

Argo CD excels at declarative infrastructure changes and deployments, but managing multiple instances across teams creates "Argo sprawl" and coordination nightmares. Enterprise control planes solve this by centralizing visibility and orchestrating multi-stage promotions while preserving your GitOps investments. These platforms add policy-as-code governance, drift detection, and release coordination that eliminates manual handoffs between teams and environments.

AI-Powered Automation: From Manual Verification to Instant Rollbacks

Deployment anxiety stems from slow detection and manual rollback processes that extend outages. AI-assisted verification automatically analyzes metrics from your observability tools, compares against stable baselines, and triggers rollbacks within seconds of detecting regressions. Combined with golden-path templates and policy-as-code, these tools enable developer self-service while reducing incident response times by up to 82% and eliminating the manual toil that burns out SRE teams.

From Principles to Practice with AI for SRE

SRE transforms reliability from reactive firefighting into proactive engineering. When SLOs gate your releases, error budgets balance speed with safety, and AI-powered verification runs automatically, and deployment anxiety disappears.

Modern SRE implementation connects your observability tools directly to deployment pipelines through intelligent automation. Harness Continuous Delivery & GitOps eliminates manual verification toil, detecting regressions and rolling back in seconds instead of minutes.

Ready to transform your deployment process from anxiety-inducing to confidence-building? Explore Harness Continuous Delivery & GitOps to see how AI-powered verification and automated remediation deliver reliability at scale.