Harness Blog

• Structured, conditional debugging, using verbosity controls, targeted variable inspection, and context-aware output, dramatically reduces troubleshooting time and noise in Ansible playbooks.
• Integrating best practices such as reusable debug roles, failure taxonomies, and CI/CD-friendly log formatting enables teams to automate root cause analysis and artifact collection, making incidents easier to resolve and less likely to recur.
• Embedding these debugging patterns into your delivery pipelines with platforms like Harness unlocks AI-powered verification, automated rollbacks, and scalable governance, transforming manual debugging insights into proactive, automated deployment safety nets.

Infrastructure as Code (IaC) has transformed how teams manage environments, but let’s be honest: when something breaks, debugging can feel like searching for a needle in a YAML haystack.

If you’re working with Ansible, you already know its power: agentless automation, declarative playbooks, and consistent deployments. But even the most elegant playbooks can fail due to syntax issues, variable conflicts, or unexpected runtime behavior.

That’s where a structured debugging approach and modern platforms like Harness come in. With solutions like Harness Infrastructure as Code Management, teams gain visibility, governance, and control over IaC workflows at scale, enabling faster, more reliable troubleshooting.

Why Debugging Ansible Playbooks Matters

Ansible is designed for simplicity, but real-world environments are anything but simple. Debugging is essential because:

• Playbooks can fail silently or behave unexpectedly
• Variables and templates can produce inconsistent results
• Infrastructure differences across environments can cause drift
• Dependencies between tasks and roles can create hidden issues

Common challenges include variable interpolation problems, connection errors, and inconsistent execution across hosts.

Without proper debugging, these issues can slow down deployments, introduce risk, and waste engineering time.

How Ansible Executes Playbooks

Before debugging, you need to understand how Ansible works:

• A playbook contains plays
• Each play targets hosts
• Plays include tasks
• Tasks execute modules

When something fails, it’s crucial to pinpoint where in this hierarchy the issue occurs. Understanding execution flow helps you isolate problems faster and avoid guesswork.

How to Troubleshoot and Debug Ansible Playbooks

Step 1: Start with a Syntax Check

The fastest way to catch issues? Validate your YAML before execution.

ansible-playbook --syntax-check playbook.yml

This catches:

• YAML formatting errors
• Missing colons or indentation issues
• Invalid structure

Syntax errors are one of the most common causes of failure, so never skip this step.

Step 2: Increase Verbosity for More Insight

Ansible provides built-in verbosity flags that reveal what’s happening under the hood:

ansible-playbook playbook.yml -v
ansible-playbook playbook.yml -vv
ansible-playbook playbook.yml -vvv
ansible-playbook playbook.yml -vvvv

Each level gives you deeper visibility:

• -v: Basic task info
• -vv: Includes variable values
• -vvv: Task-level debugging
• -vvvv: Connection-level details

This is often the quickest way to identify where things go wrong.

Step 3: Use the Ansible Debug Module

The debug module is your best friend when troubleshooting.

It allows you to:

• Print variable values
• Inspect task outputs
• Track execution flow

Example:

- name: Debug variable
  debug:
    var: my_variable

You can also display custom messages:

- name: Print message
  debug:
    msg: "Deployment started"

This helps you verify that variables are set correctly and tasks are executing as expected.

Step 4: Run in Check Mode (Dry Run)

Want to test changes without impacting systems?

Use check mode:

ansible-playbook playbook.yml --check

This simulates execution and shows what would change without actually applying it.

It’s ideal for:

• Testing updates safely
• Validating logic
• Preventing unintended changes

Step 5: Use the Ansible Playbook Debugger

Ansible includes an interactive debugger that triggers when tasks fail.

You can enable it like this:

- name: Example task
  command: /bin/false
  debugger: on_failed

When a task fails, the debugger allows you to:

• Inspect variables
• Modify arguments
• Re-run tasks

This eliminates the need to rerun the entire playbook repeatedly.

Step 6: Validate Variables and Templates

Many issues stem from incorrect variables or templating.

Common problems include:

• Undefined variables
• Incorrect variable precedence
• Jinja2 template errors

Use debug statements to inspect variables:

- debug:
    msg: "{{ my_variable }}"

Or check if variables exist:

when: my_variable is defined

Understanding variable behavior is critical for reliable playbooks.

Step 7: Inspect Registered Variables

Registered variables capture task results, which can be incredibly useful.

Example:

- name: Run command
  command: ls
  register: result

- debug:
    var: result

This shows:

• Command output
• Exit status
• Execution metadata

It’s especially useful when debugging conditional logic or failures.

Step 8: Troubleshoot Connection Issues

Connection problems are another common culprit.

To debug:

• Use -vvvv for connection logs
• Verify SSH access
• Check inventory configuration

You can also enable:

ANSIBLE_KEEP_REMOTE_FILES=1

This keeps temporary scripts on remote machines for inspection.

Step 9: Break Down Complex Playbooks

Large playbooks are harder to debug.

Best practice:

• Split playbooks into smaller roles
• Test individual components
• Run tasks step-by-step

You can use:

ansible-playbook playbook.yml --step

This lets you execute tasks interactively, one at a time.

Step 10: Use Logging and Monitoring

Logging is essential for long-term debugging.

You can configure logging in Ansible.cfg:

[defaults]
log_path = /var/log/ansible.log

Logs help you:

• Track execution history
• Identify recurring issues
• Debug production failures

Step 11: Apply Conditional Debugging

Sometimes, you only want debug output under certain conditions.

Example:

- debug:
    msg: "Variable is set"
  when: my_variable is defined

This reduces noise while still providing useful insights.

Step 12: Common Errors and How to Fix Them

Let’s look at frequent issues:

1. YAML Syntax Errors

• Cause: Incorrect indentation
• Fix: Use linting tools and syntax checks

2. Undefined Variables

• Cause: Missing or misspelled variables
• Fix: Use debug and validation

3. Task Failures

• Cause: Incorrect module usage
• Fix: Check module documentation and outputs

4. Connection Failures

• Cause: SSH or inventory issues
• Fix: Verify credentials and host access

5. Idempotency Issues

• Cause: Tasks not designed to be repeatable
• Fix: Use proper module parameters

Step 13: Build a Systematic Debugging Workflow

The most effective teams follow a structured approach:

1. Run syntax check
2. Execute with verbosity
3. Add debug statements
4. Validate variables
5. Use check mode
6. Inspect logs
7. Use a debugger for failures

This ensures you move from simple checks to deeper analysis without wasting time.

How Harness Improves Ansible Debugging at Scale

While native Ansible tools are powerful, they can become difficult to manage at scale, especially across multiple environments and teams.

That’s where Harness Infrastructure as Code Management comes in.

With Harness, teams can:

• Centralize IaC workflows
• Visualize deployments and failures
• Enforce governance and compliance
• Detect drift across environments
• Debug faster with unified insights

Instead of chasing errors across logs and CLI outputs, Harness provides a single pane of glass for managing and troubleshooting infrastructure.

Best Practices for Debugging Ansible Playbooks

To avoid recurring issues, follow these best practices:

• Keep playbooks simple and modular
• Use descriptive task names
• Validate inputs early
• Leverage check mode before deployment
• Use version control for playbooks
• Add debug statements during development
• Monitor logs consistently

Consistency and structure are key to reliable automation.

Move From Guessing To Knowing, Then Automate It

Structured Ansible debug and delivery practices transform reactive firefighting into proactive problem-solving. When you standardize verbosity levels, conditional output, and failure analysis, you cut mean time to resolution and stop scrolling through endless logs hunting for clues.

Those same debugging patterns that help you isolate failures manually become the intelligence that drives automated pipeline decisions. Smart platforms use your codified failure conditions, rollback triggers, and health checks to make deployment decisions without human intervention. Teams that systematize their debugging insights can focus on building features instead of babysitting deployments.

Ready to turn those debugging skills into automated safety nets? Harness Infrastructure as Code Management delivers AI-powered verification, automatic rollbacks, and GitOps workflows that eliminate the guesswork from production deployments.

FAQ: Fast Answers To Common Ansible Debug Questions

These five questions address the debugging bottlenecks that turn 5-minute fixes into hour-long investigations. Each answer provides tested patterns that cut time-to-resolution.

How do I choose between -vv and -vvvv without flooding logs?

Use -vv for task-level details and connection info. Reserve -vvv for module arguments and package manager chatter. Use -vvvv only when you need SSH transport debugging. In CI, pair verbosity with --limit to constrain output and keep log budgets manageable.

What's the safest way to print secrets without exposing them in CI logs?

Never use debug: var= with secret variables directly. Instead, print only non-sensitive metadata, such as key names or lengths. Use Ansible Vault for encryption and use your CI platform's secret management. Harness handles secret injection safely in pipeline runs.

How do I debug a task that only fails on one host in a 59-host inventory?

Use --limit hostname to isolate the problematic host. Enable the task debugger with ANSIBLE_ENABLE_TASK_DEBUGGER=True for interactive inspection. Compare gathered facts between working and failing hosts to identify configuration drift or missing dependencies.

Why does my changed_when always show changed even when nothing updates?

Shell and command tasks default to changed: true regardless of whether there are actual changes. Define explicit changed_when conditions, such as changed_when: result.rc == 0 and 'updated' in result. stdout. Use --check and --diff modes to preview changes before execution.

How do I persist and attach Ansible logs and diffs as CI artifacts for PRs?

Structure output with JSON formatting and redirect to files. Archive stdout, stderr, and diff outputs as build artifacts. Harness pipelines can capture these automatically and attach them to PR checks. Harness CD extends this with AI verification and automated rollback based on deployment artifacts.

What are DevOps solutions?

DevOps solutions are the tools and platforms that help teams build, test, secure, deploy, monitor, and manage software throughout the software delivery lifecycle. While DevOps is often associated with CI/CD, modern DevOps software supports a much broader set of capabilities, including infrastructure automation, security, observability, testing, and developer self-service.

Organizations can adopt individual point tools for specific functions or use integrated DevOps platforms that bring multiple capabilities together. Point tools typically focus on a single capability, while integrated DevOps platforms combine multiple functions into a unified experience.

Quick Facts

Engineering teams have more DevOps solutions to choose from than ever before. CI/CD platforms, infrastructure automation tools, security scanners, observability platforms, and developer productivity solutions all promise faster software delivery. Yet for many organizations, adding tools has not necessarily made delivery simpler.

As software delivery environments grow, so does the operational burden of managing integrations, permissions, workflows, and governance across multiple systems. By 2027, most organizations will shift from multiple point solutions to unified platforms to streamline application delivery, reversing where the majority sat in 2023.

Choosing a DevOps stack is no longer just a tooling decision. It is an architectural decision that affects developer productivity, operational efficiency, governance, and the ability to scale software delivery over time. This guide explores the different types of DevOps solutions and the criteria teams should use when evaluating the right stack for their needs.

What to look for in DevOps software and what are DevOps tools worth keeping?

The capabilities listed above do not carry equal weight. Most teams already have access to CI/CD tools, security scanners, monitoring platforms, and infrastructure automation frameworks. The real question is whether those capabilities work together to improve software delivery.

When evaluating DevOps solutions, focus on six areas:

Beyond features, consider the long-term operational impact of each option. Integration maintenance, onboarding effort, licensing costs, and platform administration all contribute to the total cost of ownership. A tool that solves one problem today can create additional complexity as teams, applications, and delivery requirements grow.

Quick Tip: The lowest-cost DevOps tool isn't always the most cost-effective option. As teams scale, integration, maintenance, platform administration, and operational overhead can outweigh initial licensing savings.

Having the right capabilities is only part of the decision. Teams must also determine whether those capabilities should come from a unified platform or a collection of specialized devops software tools.

Types of DevOps solutions for modern software delivery in 2026

Most organizations choose between two approaches: adopting a unified DevOps platform or assembling a best-of-breed toolchain. The right choice depends on factors such as team size, operational complexity, compliance requirements, and internal engineering resources.

The same trade-offs apply when evaluating open source and commercial solutions. Open source tools often provide flexibility and community-driven innovation but may require additional expertise to deploy, integrate, and maintain. Commercial platforms typically offer enterprise support, built-in integrations, and streamlined administration in exchange for licensing costs.

Deployment models also influence tool selection. Cloud-native solutions are often preferred for scalability and faster adoption, while on-premise deployments remain common in highly regulated industries with strict security, compliance, or data residency requirements.

The best DevOps solution is not defined by a single category. It depends on how well the chosen approach aligns with your team's delivery model, governance needs, and long-term operational strategy. The same DevOps approach rarely works equally well across organizations. A startup focused on shipping quickly faces a different set of constraints than an enterprise managing hundreds of developers, compliance requirements, and complex delivery pipelines.

How do you evaluate the best DevOps tools for your team?

The same DevOps solution can be a great fit for one organization and a poor fit for another. Team size, delivery complexity, and operational requirements often have a greater impact on tool selection than feature lists.

Regardless of company size, engineering leaders should evaluate every DevOps solution against a few practical questions:

• Will this reduce operational complexity or add to it?
• How much effort will be required to onboard teams and maintain integrations?
• Can it scale alongside our applications, teams, and delivery processes?
• What are the long-term costs, including licensing, administration, and platform maintenance?

Vendor lock-in should also be part of the evaluation process. The deeper a tool becomes embedded in deployment pipelines, security controls, and developer workflows, the more difficult and costly it becomes to replace. Many DevOps initiatives run into trouble not because teams chose the wrong solution category, but because critical considerations such as workflow design, governance, and developer adoption were overlooked during implementation.

Common DevOps solution mistakes to avoid

Even well-intentioned DevOps initiatives can create new challenges when tooling decisions are made in isolation.

• Adding tools to solve every new problem. More tools often mean more integrations, handoffs, and maintenance overhead. Fix: consolidate capabilities where possible and evaluate the operational cost of every addition.
• Treating developer experience as an afterthought. Complex workflows and fragmented tooling slow adoption and create friction for engineering teams. Fix: prioritize solutions that simplify day-to-day development and deployment tasks.
• Leaving governance until later. Retrofitting security, compliance, and access controls is often more difficult than building them into delivery workflows from the start. Fix: evaluate governance requirements alongside functionality.
• Selecting tools before defining delivery workflows. Technology rarely fixes unclear processes. Teams that start with tooling often end up redesigning workflows later. Fix: establish how software should move from development to production before evaluating solutions.

Reducing operational overhead is one reason many organizations are rethinking fragmented DevOps toolchains in favor of platform-based approaches.

How Harness fits into your DevOps stack

As software delivery becomes more complex, many organizations are looking for ways to improve engineering efficiency without adding operational overhead. DORA research (State of AI-assisted Software Development 2025) finds that software delivery performance predicts organizational performance and employee well-being, reinforcing the need for tools that help teams deliver software reliably and at scale.

Harness brings key software delivery capabilities together in a unified, AI-powered platform. Teams can automate build and test workflows with Harness CI, streamline deployments using Harness CD, and gain visibility into engineering productivity and delivery metrics through AI DLC Insights.

Cost efficiency is becoming equally important. According to the FinOps Foundation, 45% of organizations spending more than $100 million annually on cloud report that AI and machine learning are having a rapidly increasing impact on their FinOps practices. Harness Cloud & AI Cost Management (CACM) helps teams understand, optimize, and govern cloud spending alongside their software delivery workflows, reducing the need to manage disconnected tools across the engineering ecosystem.

How do Harness DevOps solutions perform in practice?

Organizations evaluating DevOps solutions often face the same challenge: balancing delivery speed, governance, visibility, and operational overhead. The following examples show how different teams approached those challenges.

How did Ancestry replace 80-plus Jenkins instances with one governed pipeline?

Ancestry managed software delivery across more than 80 Jenkins instances, with each team following a different deployment process and governance practice. After adopting Harness CI/CD, the company onboarded 350 systems in its first year, increased deployment frequency 3x, and achieved an 80-to-1 reduction in the effort needed to roll a change out across every pipeline.

“Harness now empowers Ancestry to implement new features once and then automatically extend those across every pipeline, representing an 80-to-1 reduction in developer effort.”

Ken Angell, Principal Architect, Ancestry

Source: Ancestry adds consistency and governance to cut downtime

How did United Airlines put governance in developers' hands without slowing delivery?

United Airlines needed stronger governance across software delivery without slowing development teams. Choosing Harness for CI and CD let the airline shift security and governance left, giving developers self-service deployment within guardrails instead of waiting on manual review. United reported 75% efficiency gains and cut CI build times for one application from 22 minutes to under 5.

“By choosing Harness for CI and CD, we were able to give the governance policies to the developers and create the guardrails we needed. Harness gives us a platform rather than just a DevOps tool.”

Ratna Devarapalli, Director of IT, Architecture, Platform Engineering and DevOps, United Airlines

Source: United Airlines accelerates deployments with Harness

How did Tyler Technologies save $1.2 million a year on cloud costs?

Tyler Technologies, the largest SaaS vendor solely focused on the U.S. public sector, ran client test environments around the clock even when most sat idle outside business hours. Reorganizing its cloud estate by client time zone and activity pattern and applying Harness Cloud Cost Management's AutoStopping let Tyler power down idle environments automatically. The result: $1.2 million in annualized cloud cost savings.

“Cloud AutoStopping opened up new possibilities for cloud cost management. We saw how reorganizing our deployments by geography, function, and use patterns could unlock game-changing savings.”

Chris Camire, Senior Manager of Technical Services, Tyler Technologies

Source: Tyler Technologies reaches $1.2M annualized cost savings with Harness Cloud Cost Management

Choose the DevOps solution that fits your delivery model, not the longest feature list

The capabilities matter less than how well they fit together. A long feature list does not tell you whether a tool will reduce operational complexity or add to it, and the gap between those two outcomes is where most DevOps initiatives succeed or stall.

Map your own delivery workflow first, then evaluate DevOps solutions against integration, governance, scalability, and total cost, not a checklist of capabilities. 

See how Harness brings CI, CD, security, and cost management onto one AI-powered platform.

FAQs about DevOps solutions

What are the most commonly used DevOps tools?

Common DevOps tools include CI/CD platforms, source code repositories, infrastructure-as-code (IaC) tools, observability platforms, security scanners, and cloud cost management solutions. Popular examples include GitHub, GitLab, Jenkins, Terraform, Kubernetes, Datadog, and Harness.

What is the best DevOps software for small teams?

The best DevOps software for small teams is typically easy to adopt, requires minimal administration, and supports multiple stages of the software delivery lifecycle. Many smaller organizations prefer integrated platforms to reduce the overhead of managing multiple tools and integrations.

How do you build a DevOps stack from scratch?

Start by defining your software delivery workflow. Then identify the capabilities needed to support it, including source control, CI/CD, infrastructure automation, security, observability, and cost management. Select tools that integrate well together and can scale as your requirements evolve.

What is the difference between DevOps tools and a DevOps platform?

DevOps tools typically solve a specific problem, such as source control, testing, or monitoring. A DevOps platform brings multiple software delivery capabilities together in a unified environment, reducing integration complexity and improving visibility across workflows.

How do you know when it is time to switch your DevOps solution?

It may be time to reevaluate your DevOps solution if teams are spending significant effort maintaining integrations, onboarding new tools, addressing visibility gaps, or managing operational complexity. Frequent workflow bottlenecks and growing governance requirements are also common indicators.

What are the limits of consolidating onto one DevOps platform?

A unified platform trades some flexibility for governance and simplicity; teams that need a specific best-of-breed tool for a narrow use case may find a platform's built-in version less specialized. The right test is whether the platform covers enough of your delivery lifecycle to retire the point tools it replaces, not whether it matches every feature in isolation.

Runbook best practices haven't changed that much at their core: a good runbook is actionable, accessible, accurate, authoritative, and adaptable. These five attributes separate a runbook your team relies on from one they ignore. What has changed is what happens after you write it. With Harness AI SRE, your runbooks don't just guide responders — they execute automatically, file tickets, trigger rollbacks, and post updates to the incident timeline without anyone manually following a checklist.

What Is a Runbook?

A runbook is a step-by-step guide for performing a task in a system, whether you're seeing it for the first time or coming back after months away. You reach for it during on-call rotations, service disruptions, or when onboarding a teammate.

This article covers runbooks for software systems and incident response automation — not airplanes or surgery.

When to Use a Runbook

Runbooks earn their place whenever a process is too nuanced or variable to fully automate. Even with strong SRE automation, some steps still need human judgment. Runbooks cover that gap — giving you structure without assuming automation handles everything.

Common use cases include:

• Investigating or stabilizing an incident before a full root cause analysis
• Running complex business processes, like generating a monthly billing report
• Handling repetitive but critical dev tasks, like setting up a test environment

Runbook Best Practices: The Five Attributes of a Good Runbook

1. Actionable

A runbook should tell you what to do next. Each task should be:

• Clear, concise, and goal-oriented
• Written for whoever will use it — new hires, mid-level engineers, or senior SREs
• One completable step at a time, with no compound instructions

When someone needs deeper context, link out to reference docs. Keep the runbook focused on action.

Good: SSH into the database server and run tail -f /var/log/db.log

Bad: Log in to the database server, edit the config file, and restart the process.

For incident runbooks, add a follow-up step like an RCA or retrospective so what you learn makes it back into the runbook and your wider operations.

2. Accessible

A runbook nobody can find during an outage might as well not exist.

Make runbooks easy to find:

• Associate them with alerts or services
• Tag them with metadata: type (incident, maintenance, onboarding), creation and last-update timestamps, author or owner, linked systems
• Make them searchable from Slack, your terminal, or your incident tool

In  AI SRE, runbooks are pinned to incident types or attached to alert rules so they surface automatically — the right runbook appears at the moment it's needed, with no searching required.

3. Accurate

Outdated runbooks lose people's trust. Lead an engineer down the wrong path once and they won't come back.

Keep runbooks accurate:

• Make updates lightweight, via PRs, comments, or an edit button
• Track both last-updated and last-used timestamps
• Have engineers validate steps before publishing, and copy-paste commands rather than retyping them
• Link usage history, like associated incidents or alerts, where you can

AI SRE logs every runbook execution step by step — inputs, outputs, and status — tied directly to the incident timeline. When a step fails, it shows up in the timeline rather than going unnoticed, making it easy to trace what needs updating.

4. Authoritative

One process, one runbook, no duplicates.

When multiple versions exist, consolidate them and archive the outdated copies. If a section needs to be reused across processes, link to it instead of copying it.

Add a simple way to flag problems. If someone hits a conflicting or misleading step, they should know how to report it.

5. Adaptable

Systems change constantly, and runbooks have to keep up.

• Assign clear ownership per runbook or section
• Open contributions to the team where it makes sense
• Build runbook updates into retrospectives and deployment checklists
• Call out the runbooks that save time or prevent an incident
• Automate the high-confidence sections once you trust them

Treat a broken runbook like a broken test and fix it right away.

Spotting Stale Runbooks

Signs a runbook has gone stale:

• A last-updated timestamp older than 12 months
• No recent use, or no link to a recent incident
• Feedback or comments flagging problems

If it's outdated but still needed, update it. If the system it documents is gone, archive it: mark the title with [ARCHIVED] and move it to a separate folder.

Runbooks in Harness AI SRE

A runbook in AI SRE is a set of steps that execute during an alert or incident. Each step acts on a connected system or on the incident record, and its result is posted to the incident timeline. The same runbook that pages the on-call can also file the ticket and run the Harness pipeline that ships the fix.

This is the part a static runbook document cannot do: it can tell a responder to roll back, but it cannot run the deploy itself. Harness AI SRE closes that gap — transforming your runbook automation from a reference document into an active participant in incident resolution.

How a Runbook Is Built

Each runbook is an ordered chain of steps. A step does one of four things:

• Runs an action against a connected system
• Sets a field on the incident
• Branches on a condition
• Loops over a list

Steps take typed inputs and pass their outputs to later steps. If a step fails, an error path runs. You build runbooks in a visual editor.

Actions a Step Can Call

AI SRE includes built-in actions that a step can call without custom integration work. They cover the systems an incident touches:

• Communication: Slack, Microsoft Teams, Google Chat, Zoom, email, SMS
• Ticketing and paging: Jira, ServiceNow, PagerDuty, Opsgenie, Jira Service Management
• Automation: run a Harness pipeline, toggle a feature flag, set a Harness connector, post an incident review, resolve an alert

Running a Harness Pipeline as a Step

AI SRE has a native step that executes a Harness pipeline. You give the step a pipeline and its input YAML, and it runs your rollback or hotfix deploy inside the incident response. The step checks the caller's pipeline-execute permission, optionally waits for the run to finish, and posts the execution link and status to the incident timeline.

Because Harness owns the CI/CD pipeline, the runbook reaches it directly — no separate integration to configure.

Getting the Right Runbook to the Incident

Two mechanisms put a runbook in front of responders without anyone searching for it (a key incident response automation principle in Harness AI SRE):

• Pinned runbooks: Pin runbooks to an incident type, and they appear for one-click execution whenever that type of incident opens.
• Alert-rule attachment: Attach a runbook to an alert rule with its inputs pre-filled, and it runs automatically when the alert fires.

A runbook can also be set to trigger on incident lifecycle events through a rule condition.

Tracking What Ran

Every runbook execution is logged step by step, with its inputs, outputs, and a status of running, success, or failed. The record is tied to the incident timeline, so a responder can see what ran, when, and what it returned. A step that fails shows up in the timeline rather than going unnoticed.

Bottom Line

Runbooks are an operational safety net. They cut cognitive load and pass institutional knowledge to whoever's on call. Automation keeps growing, but plenty of situations still need a human in the loop — and that human needs clear, current instructions.

Get the five runbook best practices right and your team recovers faster with less on-call stress. Pair them with Harness AI SRE and those runbooks stop being documents people read — they become automated workflows that execute the moment an incident opens, reducing MTTR and keeping your team focused on the work that actually requires human judgment.

---
‍Key Takeaway: Today, we're launching the public beta of the Harness CLI: the single, officially supported command-line tool for the entire Harness platform. It replaces the older per-module CLIs with one binary, one grammar, and one auth flow across pipelines, CD, code, artifacts, IaCM, feature flags, governance, and audit. Designed for secure DevSecOps and enables terminal workflows for developers and deterministic execution for AI agents.

---

For most developers, the terminal isn't just a tool: it's home. It's where builds get triggered, deployments get approved, execution logs streamed, and the hard problems of shipping software get worked out one command at a time. That reliance isn't going away. If anything, with AI agents now doing real work alongside developers, the terminal matters more than ever, because agents live there too.

But the terminal experience across Harness has been fragmented until now. A CLI per module. Different flags. Different auth patterns. Different output shapes. Every new capability meant another binary on your PATH and another set of conventions to learn.

That's why we're introducing the Harness CLI 3.0, now in public beta - a single, official command line for the entire Harness platform, designed from the ground up to be fast for humans, drivable by agents, and forward-compatible with every product Harness will ever ship.

Why one CLI, and why now

Harness has grown into a platform of 15+ products. That growth was good for customers. However, for terminals, it meant: a different CLI for each module, different flag conventions, different auth patterns, different output shapes.

At the same time, the persona driving the terminal is changing. Half the shell commands that hit a Harness API in an average customer account this week weren't typed by a human - they were produced by a coding agent working on behalf of one. Those agents don't tolerate CLI inconsistency the way humans do: an agent that can't confidently predict what a command will output can't reliably chain it into a workflow.

So we rebuilt. One official CLI. One grammar. First-class support for humans and agents.

Meet ./harness

“harness” is now the command-line tool for Harness. It's a single binary. It ships from the same team that ships the platform. It's Apache-2.0 open source at github.com/harness/cli.

Every command follows the same shape:

harness <verb> <noun> [identifier] [flags]

Six core verbs - list, get, create, update, delete, execute - plus push/pull for artifacts. Every module in Harness plugs its resources ("nouns") into that grammar. Learn one command, and you've learned all of them.

Ask the binary what it knows:

Built-in discovery - the CLI teaches itself

Forgot the noun? Wondering what verbs it supports? Ask the CLI.

harness list module              # every module currently loaded
harness get  module pipeline     # domain model, nouns, and guides for a module
harness list noun                # every registered resource type
harness get  noun pipeline       # fields, aliases, and commands for one noun
harness list noun      # the entire noun × verb matrix at a glance

The list noun view is the most useful command in the CLI. It prints every resource type Harness supports, the module it lives in, and every verb the CLI provides on it in one screen. Every command supports --help at every level. Every mistype gets a Levenshtein-based "did you mean…?" suggestion.

Nothing is hidden. Nothing fails silently. The CLI is a self-describing surface, out of the box.

The full software delivery lifecycle, in your terminal

Here's what shipped in the beta today:

• Core: Authentication, profile management, install, version, discovery.
• Platform: Orgs, projects, users, roles, connectors, secrets, delegates.
• Pipeline: CI/CD pipelines, executions, logs, triggers, templates, freezes.
• CD: Services, environments, infrastructure, and service overrides.
• Harness Code: Repositories, pull requests, branches, commits, and tags.
• Harness Artifact Registry: Harness Artifact Registry - push and pull across every major package format.
• IaCM: Terraform and OpenTofu workspaces and operations.
• Harness Feature Management: Feature Management & Experimentation - flags and targeting definitions.
• Governance: OPA policies, policy sets, and policy evaluations.
• Audit: The full audit trail across every resource and action.

10+ products and capabilities (more to come). One install. One upgrade train. The day a new Harness product ships, its resources light up in the CLI you already have - no new tool on your PATH, no new auth flow, no new flag conventions.

Why "Built for Humans and Agents" matters

Half the shell commands hitting a modern DevOps platform aren't typed by a person anymore. They're emitted by a coding agent: Claude Code, Cursor, Copilot CLI, Codex, working on someone's behalf. Every one of those agents has been fighting the same problem: a CLI they can't confidently predict, output they can't reliably parse, and permissions they can't guarantee.

The Harness CLI is designed to close that gap.

For humans:

• One grammar to learn. Six verbs. Twelve modules - streamlined scripting from the terminal.
• Live-API tab-completion- real execution ids, real pipeline names, in real time.
• Interactive TUI for auth and scope. Beautiful tables. Piped output that plays well with jq, awk, xargs, and everything else you already know.

For agents:

• A closed grammar that an agent can enumerate exhaustively.
• A self-describing surface - harness list noun --format json returns the entire action graph as structured data.
• Stable output contracts - every command supports --format, and every list command supports table | json | jsonl | csv | tsv | markdown. 
• Deterministic exit codes - no ambiguity about whether a command succeeded.
• The same audit trail as the UI - every CLI call, whether typed by a human or emitted by an agent, flows through the same Harness RBAC and audit path.

What does that mean for your team

Beyond a nicer terminal, the Harness CLI unlocks concrete business outcomes:

• Ship faster. From harness auth login to a live production deploy in under sixty seconds.
• Cut the platform tax. Replace the wrapper scripts, glue tools, and one-off Python helpers most teams maintain across their CI systems with a single, supported binary.
• Stay audit-clean. One audited path in and out of Harness - for developers, for scripts, and for every AI agent your organization deploys.
• Future-proof your automation. New Harness modules add up automatically in the CLI you already have. Your CI never needs a "CLI upgrade day" again.
• AI-ready by default. Every AI agent, every runbook agent your platform team writes - they all get first-class access without you writing a single line of adapter code.

A tour of the CLI, by the user workflow

Here's how the CLI feels in five real workflows.

1. Auth once, work everywhere

Named profiles make it easy to hold multiple accounts side-by-side, and the same auth story works locally and in CI.

2. Discover the surface without leaving the terminal

harness list noun is the single most useful discovery command. It prints every resource type, the module it lives in, and every verb the CLI supports on it. This powers the Agents to perform operations consuming less tokens and reduced context window.

3. Ship a build, watch it, debug it

Pipelines are the core execution primitive. The full lifecycle - trigger, watch, inspect steps, tail logs, abort if needed - is a handful of commands.

# List and inspect
harness list pipeline
harness get  pipeline deploy-checkout
harness get  pipeline:summary deploy-checkout
# Run
harness execute pipeline deploy-checkout
# Follow up
harness list execution --filter pipeline_id=deploy-checkout --limit 5
harness list execution_step <execution-id>
harness list execution_log  <execution-id>
# Approvals live inline
harness list    approval_instance
harness execute approval_instance:approve <instance-id>
# Abort if needed
harness execute execution:abort <execution-id>

Or push and scan an artifact:

harness push artifact:docker internal-oci/checkout:v2.31.4
harness push artifact:helm   internal-oci/checkout-chart:1.4.0
harness push artifact:npm    internal-npm/@acme/checkout-sdk:3.2.0
harness execute artifact_version:firewall_scan internal-oci/checkout:v2.31.4
harness pull artifact internal-oci/checkout:v2.31.4

Same grammar. Same auth. Same output formats. Every module.

4. UI capabilities in the CLI: Add --ui

Sometimes you don't want JSON, you want to browse. Every paged list command and a handful of get commands support --ui, which drops you into an interactive terminal UI: scroll, filter, drill in, and pick without leaving the shell.

# Browse — paged list browser, works on every list command with paging
harness list pipeline --ui
harness list execution --ui
harness list connector --ui
harness list secret --ui
harness list audit_event --ui
harness list artifact_version --ui
harness list feature_flag --ui

# Pick — interactive resource pickers on selected get commands
harness get project --ui   # org-aware project picker
harness get workspace --ui   # IaCM workspace picker
harness get artifact_version --ui   # artifact + version picker

# Watch — live log viewer for a pipeline execution
harness get execution_log <pipeline-id>/<execution-id> --ui

‍

5. Self-upgrading

The CLI upgrades itself.

harness install cli                    # upgrade to the latest
harness install cli --version v1.2.3   # pin a specific version
harness install cli --check            # is a new version available?
harness install module <name>          # install an external module (e.g., har)

6. Built for agents from day one

The self-describing surface, the stable output contracts, the closed grammar, the deterministic exit codes, the --help everywhere - every one of those decisions was tested against a real coding agent driving the CLI end-to-end. 

7. Open source‍

Apache 2.0 at github.com/harness/cli. Every spec file, every command definition, every release tag. Reproducible builds, SBOMs, and Cosign signatures ship with every release.

Open source and open by design

The Harness CLI is fully open source under Apache 2.0. Every spec file, every command definition, every release tag is public on GitHub at github.com/harness/cli.

We think that matters for three reasons:

• Trust. When you pipe our installer into sh, you should be able to read what it does. Reproducible builds, SBOMs, and Cosign signatures ship with every release.
• Verifiability. Your security team can match the binary to the source.
• Extensibility. The plugin SDK works best when the core is open. The Artifact Registry already ships as an external module (harness-har) using the exact plugin pattern we'll open to the community next.

We fully expect (and welcome!) a global community of contributors - filing issues, proposing new nouns, writing spec files for their own tools. The specs are declarative YAML and adding a new command doesn't require a Go compiler. For more information, refer to the Command Reference Wiki.

Easy to get started

Sixty seconds from zero to the first pipeline run.

# 1. Install
curl -fsSL https://raw.githubusercontent.com/harness/cli/main/install.sh  | sh

# 2. Log in
harness auth login

# 3. See what you can do
harness list noun

# 4. Run something real
harness list pipeline
harness execute pipeline <your-pipeline-id>

# 5. Hand your agent the keys
harness list noun --format json

macOS (Intel and Apple Silicon) and Linux (amd64 and arm64) are supported today. Homebrew, yum, and Windows Installer are on the roadmap.

Feedback shapes the roadmap

This is a public beta. 

• GitHub Issues: harness/cli/issues for bugs, feature requests, and "what does this flag do?" questions.
• GitHub Discussions: harness/cli/discussions for ideas and design conversations.

If you build something interesting with the CLI - an agent runbook, a CI consolidation, a jq one-liner that saved your on-call - send it our way. We're collecting show-and-tell submissions for the GA launch later this year.

We've spent nine years making Harness the platform enterprises trust to ship software. The next chapter - where developers and AI agents operate that platform together, at the same terminal, with the same guarantees- starts today.

One binary. Six verbs. Every product. Built for Humans and Agents.

Use Harness CLI today

• Now available on GitHub at https://github.com/harness/cli 
• Command Reference available here

‍

‍

‍

FAQs

What is the Harness CLI?
It's the single, officially supported command-line tool for the entire Harness platform, replacing the previous per-module CLIs. It's one binary covering pipelines, CD, Harness Code, artifacts, IaCM, feature flags, governance, and audit.

Why did Harness replace the per-module CLIs?
As Harness grew to 15+ products, each module shipped its own CLI with different flags, auth patterns, and output shapes. That fragmentation got harder to manage for both humans and, increasingly, AI agents that need predictable behavior to chain commands reliably.

What's the core command structure?
Every command follows the pattern harness <verb> <noun> [identifier] [flags]. There are six core verbs (list, get, create, update, delete, execute) plus push/pull for artifacts. Every module plugs its resources into that same grammar.

Why is it described as "built for humans and agents"?
Half the shell commands hitting a modern DevOps platform today come from coding agents (Claude Code, Cursor, Copilot CLI, Codex, etc.) rather than people. The CLI is designed with a closed, enumerable grammar, a self-describing surface (harness list noun --format json), stable output contracts (table | json | jsonl | csv | tsv | markdown), and deterministic exit codes, so agents can predict and parse output reliably. Every call, human or agent, flows through the same Harness RBAC and audit path.

Is the CLI open source, and how do I get started?
Yes, fully open source under Apache 2.0 at github.com/harness/cli. Then harness auth login and harness list noun to explore. macOS and Linux are supported today; Homebrew, yum, and Windows are on the roadmap.

‍

In the fast-paced world of modern software delivery, compliance is often a bottleneck. While our existing OPA-based Policy as Code feature has long empowered teams to encode complex authorization checks and enforce granular governance across their DevOps workflows, we know that starting from a blank page can be daunting. Security and governance teams struggle to keep up with the volume of releases, while developers often find the initial setup of these policies to be time-consuming.

Today, we are thrilled to announce a significant leap forward in automated governance: Policy Packs.

What Are Policy Packs?

Policy Packs are a curated library of pre-written Rego policies designed to align your software delivery lifecycle (SDLC) with the most popular compliance frameworks.

By providing out-of-the-box policies, we are eliminating the primary barrier to automated governance: the need to write and maintain complex Rego code from scratch. With Policy Packs, you can adopt industry-standard guardrails by adapting our out of the box policies from the policy packs with zero to little customization. This will allow your teams to focus on shipping features rather than writing policy.

Comprehensive Framework Coverage

Our Policy Packs initiative covers the frameworks that matter most to your business and your auditors:

• SOC 2: Focuses on trust service criteria like security, availability, and processing integrity. Our policies help enforce peer-reviewed pull requests and automated change management gates.
• NIST: Specifically targeting SP 800-53 and 800-171, these policies cover configuration management, system integrity, and supply chain risk management via automated SBOM generation and SLSA provenance.
• PCI DSS: Designed for organizations handling payment card data, these policies enforce secrets management, network segmentation, and mandatory vulnerability patching.
• HIPAA & HITRUST: For those dealing with protected health information (PHI), these packs provide automated data leakage prevention and secure API monitoring.

Turning Framework Requirements into DevOps Controls

Compliance is no longer just a "point-in-time" audit; it’s a continuous process. Policy Packs map technical events directly to framework controls, providing the evidence your GRC teams and auditors need.

Common Compliance Challenges in DevOps

In our work with industry leaders, like those in highly regulated industries such as healthcare, finance, or insurance, we’ve seen that compliance is often a manual, high-friction process that slows down software delivery. Two of the most common challenges teams face are:

• The "audit readiness" blind spot: Teams often struggle to prove that their pipelines are consistently secure. Without automated guardrails, compliance is only checked during point-in-time audits, leaving gaps that are hard to identify and remediate.
• The manual approval bottleneck: Many organizations rely on manual checkpoints to satisfy framework requirements. This frustrates developers who want to move fast, leading to "compliance fatigue" where processes are bypassed or ignored.

Harness Policy Packs address these challenges by shifting governance left, embedding compliance checks directly into your CI/CD pipelines so that validation happens automatically with every commit.

Example Policy: Enforcing Separation of Duties

A classic requirement for frameworks like SOC 2 and NIST is "Separation of Duties." In a modern DevOps workflow, this means the person who writes and commits the code cannot be the same person who approves the deployment to production.

To enforce this, a compliance policy in your pipeline would verify the identity of the commit author against the identity of the deployment approver. If the system detects that the author and the approver are the same individual, the policy automatically blocks the deployment. This ensures that every production change has been independently peer-reviewed, providing auditors with a tamper-proof guarantee that your internal controls are working as intended without requiring manual intervention from your GRC team.

Why This Matters

1. Shift governance left: Catch violations before they ever reach production by enforcing policies during the save, run, or step phases of your pipeline.
2. Accelerate audit readiness: Move from "preventing risk" to "proving compliance" with a tamper-proof evidence vault that stores all builds, scans, and approvals.
3. Reduce risk: Mitigate legal and audit risks by using policies that are directly mapped to specific framework controls.
4. Universal coverage: Whether you are using CI, CD, Feature Management, or Cloud & AI Cost Management, our one framework provides cross-module coverage across the entire platform.

Get Started Today

The journey to automated compliance doesn't have to start with a blank page. Get access to our policy packs in the repository here to get started. You can leverage our native Git integration for OPA rego policies to fork the policies from the repository linked above and import them into your account.  

Get ready to stop audit delays before they start. 

The open-source landscape has witnessed another highly automated, ecosystem-level subversion. On June 17, 2026, a critical software supply chain attack struck the Mastra AI framework - a popular open-source TypeScript ecosystem used widely to build AI agents, workflows and RAG pipelines. By exploiting a compromised contributor account, threat actors successfully mass-published 144 malicious packages under the official @mastra npm scope.

The packages themselves contained no malicious code within their repositories; instead, they were altered at the registry level to pull in a weaponized transitive dependency called easy-day-js. Any developer workstation, CI/CD runner or cloud environment executing a routine installation during the compromise window was immediately exposed to a sophisticated cross-platform information stealer. This article delivers a comprehensive technical teardown of the attack mechanics and its stealthy execution pipeline.

Preface

Modern application engineering moves at the speed of automated dependency resolution. Rather than writing utility functions from scratch, software teams routinely orchestrate architectures that pull hundreds of third-party open-source components during active builds and deployment pipelines. This reliance sets up a structural trust chain: developers trust the package registry, the registry trusts the maintainer's cryptographic identity and downstream environments trust that updates are safe and authentic.

However, this architecture exposes a massive, interconnected attack surface. When an adversary manages to compromise an upstream account or subvert a single verification step, the entire downstream distribution network turns into an automated malware delivery pipeline. The Mastra incident highlights a growing shift where threat actors stop targeting production firewalls directly and focus heavily on poisoning the automated software supply chain.

Introduction

At its core, the Mastra supply chain compromise was designed to abuse default package installation behavior to execute arbitrary code, bypass standard static scanners, harvest sensitive host credentials and establish long-term persistence across multiple operating systems. 

What makes this attack structurally advanced is its combination of trust exploitation and defensive evasion:

• Provenance Deception - While the Mastra ecosystem typically publishes official releases via automated CI pipelines backed by SLSA provenance attestations, the attacker used a hijacked contributor token to publish directly to the npm registry. By stripping the provenance attestations, the malicious versions bypassed traditional verification layers while remaining structurally identical to legitimate software updates.
• Transitive Infiltration - The core source code within Mastra's public GitHub repository remained entirely clean. The anomaly was introduced solely inside the published registry tarballs by altering the package.json configurations to demand an external, attacker-controlled module.
• Volatile Loading & Self-Deletion - The first-stage payload was engineered as a transient dropper. It disabled local transport security, fetched a secondary backdoor payload into memory, spawned it as an isolated system process and then deleted its own source files from disk to eliminate obvious post-incident footprints.

Deepdive Into The Mastra Exploit

The campaign was executed within an intensive, automated 88-minute window. Rather than engineering a complex repository exploit, the attacker capitalized on a dormant contributor account ehindero whose publishing access to the @mastra npm scope had not been explicitly revoked. Let's break down the exploit lifecycle step-by-step.

Initial Access & The ehindero Account Compromise

Every supply chain attack requires an initial wedge to subvert the trust architecture. In this campaign, the breach did not stem from a flaw in Mastra’s core source code, but rather from an authentication gap on a historical contributor account by the name of ehindero. Security analysis indicated the account takeover occurred through a combination of two common supply chain vulnerabilities as discussed below.

• The contributor account lacked enforced Multi-Factor Authentication on the npm registry. Threat actors cross-referenced public data breaches to match historical password reuse, successfully logging directly into the publisher profile via automated scripting.
• Further forensic tracking suggested that an active personal access token with broad scope write permissions had accidentally been preserved inside an unencrypted local configuration environment on a legacy workstation.

The Transitive Dependency & Caret Range Trick

The attack relied heavily on establishing a convincing upstream dependency. On June 16, 2026, the threat actor published a "bait" version of a typosquatted package named easy-day-js, mimicking the popular dayjs library. This initial version, v1.11.21, was completely clean and byte-for-byte identical to legitimate components, designed purely to evade early automated registry profiling.

The following day, the attacker published easy-day-js@1.11.22 - this time embedding a malicious postinstall script execution block. Simultaneously, using the hijacked contributor credentials, the attacker mass-republished 144 packages across the @mastra/* scope. In each of these packages, the attacker injected exactly one line given below into the published package.json:

"dependencies": {
  "easy-day-js": "^1.11.21"
}

Because npm interprets the caret ( ^ ) range to mean any minor or patch update up to the next major version, any fresh invocation of npm install for a Mastra package automatically resolved, downloaded and integrated the malicious v1.11.22 payload. 

Bypassing Trusted Publishing Infrastructure

Mastra's legitimate delivery pipeline relies on OIDC-based short-lived publishing tokens. However, the npm registry still allowed manual token authentication paths. By utilizing a long-lived personal token belonging to the compromised account, the adversary circumvented the official GitHub Actions release loop. Although the resulting packages lacked the standard SLSA provenance attestations generated by the framework’s CI runner, consumption policies at the developer level rarely block packages simply due to missing attestations, allowing the unauthorized code to execute freely.

Detached Execution & Stealth Mechanics

When a system triggers the installation of the tainted package, the postinstall lifecycle hook automatically invokes an obfuscated script named setup.cjs. The loader performs the following actions:

• Disables TLS Verification - It overrides the environment configuration with process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0", forcing the Node runtime to accept unverified, self-signed SSL certificates from the attacker's infrastructure.
• Drops Operational Logs - It creates temporary tracking files under the host's temp directory named .pkg_history and .pkg_logs.
• Fetches Stage 2 - The loader initiates an outbound HTTP request to the attacker's Command and Control (C2) endpoint, 23.254.164.92:8000, to retrieve an encrypted secondary payload.
• Detached Spawning - The second stage is written to a randomized 12-hex-character .js file and executed as a detached child process, giving it an independent process group that outlives the parent npm install command.
• Forensic Self-Deletion - The moment the second stage initializes in memory, the loader invokes fs.rmSync(__filename, { force: true }), completely removing the installer script from the local node_modules directory to blind basic static security utilities.

Cross-Platform Info-Stealing & Persistence

The secondary payload operates as a comprehensive cross-platform credential harvester. It systematically searches the host environment for sensitive telemetry:

• Browser Data Extraction - Targets local storage and credentials from Google Chrome, Microsoft Edge and Brave browsers.
• Cryptocurrency Exfiltration - Targets more than 160 browser-based cryptocurrency wallet extensions including MetaMask, Phantom, Coinbase and Binance to drain active sessions.
• Credential Harvesting - Scans local user spaces for cloud configuration tokens like .aws/credentials, environment variables, database keys and SSH profiles.

To ensure long-term control, the malware checks the host OS and installs specific persistence payloads. All harvested data is packaged and exfiltrated to the secondary C2 server located at 23.254.164.123.

Attacker compromises active/former contributor account (ehindero)
↓
Attacker publishes clean bait version easy-day-js@1.11.21
↓
Attacker uploads weaponized version easy-day-js@1.11.22 with postinstall hook
↓
Attacker uses compromised token to mass-publish 144+ @mastra packages
↓
Injected dependency ("easy-day-js": "^1.11.21") added directly to registry tarballs
↓
Developer or CI/CD runner executes npm install for a Mastra package
↓
Registry resolves caret range ^1.11.21 to the malicious v1.11.22
↓
postinstall lifecycle script triggers setup.cjs hook automatically
↓
Loader disables TLS verification and writes local tracking logs
↓
Second-stage payload downloaded from C2 infrastructure (23.254.164.92)
↓
Payload executed as a detached child process to survive parent completion
↓
setup.cjs executes self-deletion routine to erase forensic footprint
↓
Second stage harvests browser data, 160+ crypto wallets and password managers
↓
Cross-platform persistence established via LaunchAgents, systemd or Registry Run keys
↓
Stolen host credentials and secrets exfiltrated to C2 server (23.254.164.123)

Ecosystem Impact: The AI Pipeline Risk Factor

The blast radius of this campaign is significantly amplified by Mastra's core function as an AI development framework. Unlike generic utility libraries, AI orchestration frameworks are explicitly deployed in data-rich environments. They sit at the intersection of production software loops and deep enterprise backends, routinely interacting with proprietary vector databases, LLM clusters and extensive data integration pipelines.

Consequently, a compromise within this specific ecosystem does not just threaten basic web infrastructure but also exposes high-value LLM API tokens, production database credentials and training data connection points. With a weekly download volume exceeding 1.1 million across the framework and the foundational @mastra/core package pulling roughly 918K downloads alone, the potential data exposure across developer endpoints and automated AI build servers represents a systemic risk to enterprise AI security models.

Compromised Packages

The entire compromise in the npm registry stems from the weaponized library easy-day-js, specifically version v1.11.22. This served as the malicious transitive dependency, resulting in the compromised Mastra AI framework outlined in the table below.

‍

Remediations 

Defending against registry-level poisoning requires moving beyond passive perimeter filtering to implement proactive environment isolation and strict runtime controls. Organizations should immediately deploy the following protective protocols:

• Neutralize install-time droppers by globally disabling automatic package script execution. Run npm config set ignore-scripts true across all developer workstations and CI environments.
• Commit strict lockfiles like package-lock.json and pnpm-lock.yaml to version control and utilize deterministic installation flags like npm ci or pnpm install --frozen-lockfile within CI pipelines to prevent automated caret resolutions from pulling unverified upstream releases.
• Configure internal package consumption policies to strictly reject public registry updates that lack verified SLSA provenance attestations or match anomalies where a package shifts from a trusted CI publisher workflow to a manual personal token publish.
• If an infection vector is discovered, treat the host or runner as entirely compromised. Immediately rotate all connected cloud credentials, source control keys, database strings and LLM API keys exposed within that environment.
• Use Software Composition Analysis platforms such as Harness Supply Chain Security (SCS) to continuously inventory dependencies, enforce policy gates, detect malicious packages and block compromised artifacts before they enter build and release pipelines

According to Harness’s analysis of the npm attacks, organizations should treat CI/CD pipelines as critical security infrastructure, combining SBOM visibility, policy enforcement, provenance validation and automated dependency risk analysis to prevent trusted publishing systems from becoming malware distribution channels. Read more about it here.

How Harness Supply Chain Security Helps

Harness SCS helps you quickly detect and contain compromised dependencies like the Mastra AI packages before they impact your pipelines. With real-time visibility into your SBOMs and dependency graph, you can identify affected versions, trace their usage across builds and environments and block them using OPA policies. This ensures malicious packages never propagate through your CI/CD or AI workflows.

Detect Compromised Packages

Harness SCS enables instant search across all repositories and artifacts to quickly identify if compromised package versions exist in your environment. The moment such a malicious package is disclosed, you can pinpoint its presence and assess impact across your entire supply chain in seconds.

Block Compromised Packages

Harness AI streamlines response to incidents like the Mastra AI compromise through simple natural-language prompts. With a single prompt, you can generate OPA policies to block affected versions of Mastra packages, for example, across all pipelines, preventing malicious packages from entering builds or deployments. As new compromised versions emerge, these policies can be quickly updated to maintain strong preventive controls across your SDLC. SCS customers can use this OPA policy to detect and block the affected versions.

Track & Remediate Issues with Developers

Harness SCS automatically detects compromised versions across both production and non-production environments. Teams can track remediation, assign fixes and monitor progress through to deployment, ensuring exposed credentials and vulnerable dependencies are addressed quickly. This end-to-end visibility helps contain the impact and prevents compromised packages from persisting in your supply chain.

Next Steps In The Face Of Supply Chain Attacks

The easy-day-js campaign highlights how quickly a malicious package can expose high-value secrets when embedded deep within registries and CI runners. Given its role in managing dependencies and packages across projects, the impact extends beyond code to API keys, prompt data and downstream systems, often bypassing traditional security checks.

Defending against such attacks requires more than reactive fixes. Teams need real-time visibility into dependencies, the ability to enforce policies to block compromised versions and continuous tracking to ensure remediation is complete across all environments. Harness SCS enables teams to quickly identify where affected package versions are used, prevent them from entering new builds and ensure fixes are consistently rolled out.

With these controls in place, organizations can limit credential exposure, contain threats early and secure their supply chain against attacks like the Mastra packages compromise.

A new report from LeadDev and Harness makes one thing clear: AI coding tools have fundamentally changed how much code organizations are producing. What has not changed nearly fast enough is how that code gets released.

The State of AI-Driven Software Releases 2026 report, based on responses from 500424 engineers across industries and company sizes, puts real numbers behind a problem that engineering leaders have been feeling for a while. AI is accelerating the code creation side of the SDLC. The downstream side, getting that code safely and confidently into production, is struggling to keep pace.

Here are three findings that stand out.

1. Code review is becoming the new bottleneck

57% of organizations still require a manual, human-in-the-loop review for every single line of AI-generated code, regardless of risk level. Among that group, 38% are spending more time on code review than before AI tools arrived. Meanwhile, 32% of respondents saw their release sizes grow after introducing AI-generated code.

The math does not work. AI is producing more code, often in larger pull requests, while review capacity stays flat. The bottleneck that used to sit at the code generation stage has simply moved downstream.

The answer is not to remove humans from the process entirely. It is to be smarter about where human judgment is required. Feature flags change the equation here in a practical way: when AI-generated code ships behind a flag that is off by default, teams can deploy continuously without requiring every line to be perfectly validated before it touches production. The review still happens, but it is no longer a gate on the entire release. Changes can go live in a controlled state, exposed to a limited audience or no one at all, until the team is confident enough to turn them on. That decoupling of deployment from release is what makes it possible to keep pace with AI-generated output without sacrificing oversight.

2. The guardrails gap is real, and it is growing

Only 49% of organizations have specific guardrails in place for AI-generated code. That means roughly half of teams are shipping AI-assisted code with the same review and validation processes they used before AI tools existed. The industry went through a decade of work to build DevOps discipline, continuous delivery, and quality gates into the SDLC. The rush to AI has created pressure to skip that rigor on the release side.

The numbers shift significantly by company size. Vulnerability detection is in use at 44% of large enterprises, but only 16% of smaller companies. Smaller organizations are moving faster with less protection, which compounds as AI-generated output increases and as AI-powered product behavior becomes harder to predict at runtime.

Progressive delivery is the practical guardrail that works at AI speed. Rather than trying to catch every risk before deployment, progressive rollouts expose changes to a small percentage of users first, then expand based on real signals. If something degrades, a feature flag kill switch stops the exposure immediately without requiring a full rollback. Teams that adopt this approach can move faster, not slower, because the blast radius of any individual change is controlled from the start. For AI-powered features specifically, where behavior can drift in ways that are difficult to predict in testing, that kind of runtime control is not optional. It is the safety layer that makes safe shipping possible.

3. More experimentation, less measurement

58% of organizations say they are running more experiments than before, which is genuinely good news. AI coding tools are helping teams build and test more ideas with real users, and that increased experimentation is one of the strongest signals that teams are adapting well to higher code velocity.

The challenge is that 52% of respondents cited a lack of clear metrics as their biggest challenge when working with AI-generated code. Only 29% of organizations are actually measuring the impact of AI tools on their teams at all. Running more experiments without the infrastructure to interpret results and make confident decisions is not a learning system. It is noise.

The teams getting the most value from increased experimentation are the ones connecting feature rollout directly to measurement. That means defining success metrics before a flag turns on, monitoring guardrail metrics during rollout, and having clear criteria for whether to expand, iterate, or stop. Experimentation only compounds in value when teams can close the loop from release to evidence to decision. Without that structure, more exaperiments just means more uncertainty.

What comes next?

The report contains much more data that paints a picture of an industry at a real transition point. AI has changed the pace of software creation, but creating code faster is not the same as releasing better software faster. The teams pulling ahead are treating the release layer with the same discipline they have applied to code generation: progressive delivery, controlled exposure, automated guardrails, and experimentation connected to real decisions.

Feature flags, progressive rollouts, and experimentation are not optional safeguards for AI-driven development. They are the foundational layer that makes AI velocity sustainable.

Want the full picture? Download the State of AI-Driven Software Releases 2026 report for the complete data, including how organizations are adapting their guardrails, what progressive delivery practices the leading teams have adopted, and what the path forward looks like.

When we launched Autonomous Worker Agents, the message we led with was simple: governance is inherited, not integrated. Agents don't get security bolted on after the fact. They inherit the OPA policies, RBAC, and audit trails already running your production pipelines. This post is about the layer underneath that promise: isolation.

We let an Autonomous Worker Agent run shell commands and call APIs inside our pipelines. Then we sat down and asked the uncomfortable question: what happens the moment it goes rogue? This post is about the four walls we put around the agent so that a break-in stays a break-in and never becomes a breach.

‍

OUR STARTING ASSUMPTION

The agent is already compromised. Not might be...already is. Everything below follows from that one sentence.

Key takeaways

• The agent is the threat model. An agent that runs tools based on a model's output is one bad prompt away from doing whatever an attacker wants. We don't try to make that impossible. We assume it's already happened and ask what the agent would still be unable to do.
• Defense comes in layers. A hardened image removes the tools an exploit needs. Isolation keeps the agent away from the secrets. A broker means it never holds a real key. A proxy bounds where data can go. Four layers, each doing one job.
• No single layer is load-bearing. Each layer assumes the one before it already failed. A break-in has to beat all four, one after another, and every one is enforced by the kernel or the network,. never by the model's judgment.
• Proven, not promised. We replayed a real CVSS-9.0 breach against our own image. The step that used to dump 709 live secrets now returns 33 variables and zero usable credentials. Every layer ships with a test that has to pass first.

‍

The old way breaks on contact with the real world

The first wave of agent platforms all looked the same. Take a capable model, hand it some tools, hand it some secrets, and let it run. It's fast to build. It sounds reasonable in a design review. It's also exactly how we built our first internal agent.

Then we actually looked at what that agent does all day. It reads a prompt, makes a plan, and runs tools (a shell, a file editor, an HTTP client) based on the output of a model that is itself reading whatever it's been pointed at: a README, a webpage, the result of the last tool call, a dependency's install script. None of that text is trustworthy by default. And the model is the thing deciding what in it counts as “information” versus what counts as “instructions.” That call is exactly what models are worst at making.

Here's what that looks like in practice. A README that says, as part of setup, run env > .config and continue, doesn't read like an attack on a model. It reads like a setup. The model isn't being careless. It's doing exactly what it was built to do, which is follow the text in front of it. The text just wasn't trustworthy in the first place.

That's not a bug we can patch inside the model. It's a mistake in how we built the system around it. We were treating an agent like trusted code when its actual job is to read untrusted input and act on it. Once we saw it that way, the fix was obvious: stop pretending otherwise.

Same starting point for both columns below: a tool result with text the agent was never supposed to follow. What differs is everything built around the agent.

The naive shape vs. the hardened shape.

The Reframe: Stop asking if the model will behave

Our governing assumption is blunt: the process running the agent is fully under an attacker's control, the same as remote code execution, running as the agent's own user. We don't treat that as a remote possibility we're guarding against. We treat it as the starting condition.

Everything below follows from that one sentence. We stopped asking “Will the model refuse the malicious instruction?” We've already assumed it won't. The question that actually matters is: if the agent turned hostile this exact instant, what would the operating system, the filesystem, and the network physically refuse to do on its behalf?

That question, asked four times, is where the four layers of isolation come from. Isolation is the foundation: the part that has to hold even when everything else fails. It's not the whole story: separately, we control what an agent is allowed to touch, and we read the prompts and responses themselves for injection and sensitive data. Those are their own posts. This one is about the walls.

Here's the bigger picture those three fit into, so you can see where this post sits before we go deep on it:

The bigger picture: isolation is the deterministic foundation; identity/authorization and behavioral controls sit above it.

1. Isolation (this post)

Kernel, OS, network: deterministic. The agent can't reason its way out.

Image hardening

Process isolation

Secret isolation

Network isolation

2. Identity & Authorization

Two levels: who defines the permission surface, and what runs within it.

Operator level - RBAC: who creates the surface

• MCP connectors: creates + tool scope
• Agent creation: creates + tool scope
• Pipeline runs: who can execute

Agent level - authorization: what runs within that surface

• Scoped tokens: short-lived
• Allowed tools: connector-level
• OPA - runtime: pipeline policy

3. Behavioral Controls

Content and call inspection: the only probabilistic layer.

• Injection scan: LLM gateway
• PII + toxicity: LLM gateway
• Call security: MCP gateway
• OPA policies: gateway

Not load-bearing on its own. Designed to fail safely into 01 and 02.

Four threats. Four layers. No overlap in the job.

Working from that one assumption, a concrete list of things a hostile agent would try falls out almost by itself. Each one gets exactly one layer of defense, built to answer it and nothing else. Below is the map, running from the outermost edge of the container inward to the network path out.

Walking the chain, outside in

Each section below stands on its own. If you only care about one threat, say, where your secrets actually live, jump straight to it. Read straight through, and you get the whole chain, from the edge of the container to the only network path out.

Layer 1: Image hardening: An empty toolbox

The threat

Break into a normal container and you'll find a fully stocked workshop waiting. A compiler to build a payload. A package manager to pull one down. Setuid binaries to climb privileges. Network tools to probe around and call home. A general-purpose base image hands an attacker all of it, for free, the moment they're inside.

The defense

We strip the image down to the studs at build time and the stripping is permanent, not a setting someone can flip back:

• No compiler. The build toolchain, C compiler, linker, make, is purged. Nothing left to compile a second-stage payload with.
• No package manager. The agent can't install its way to a capability the image doesn't already ship. (Package metadata stays, so vulnerability scanners and SBOM tooling still see exactly what's installed.)
• No setuid binaries. Every setuid and setgid bit in the image is cleared, closing the classic local privilege-escalation path.
• Read-only at runtime. The recommended way to run it locks the root filesystem read-only, with small writable scratch areas mounted separately, turns off privilege escalation, and drops every Linux capability the container doesn't need. The image ships ready for that posture and our published pod spec sets it.

In plain terms

Picture breaking into a workshop and finding the pegboard empty. No tools on the wall, no machine plugged in, nothing to build with. You can be standing right there, inside the room, and still unable to do anything.

Stripped at build time, irreversibly, compiler removed, package metadata only, setuid bits cleared, network utilities removed.

‍

Layer 2: Process isolation: Three users, one container

The threat

Even with no tools to build with, a compromised agent will still look at what's sitting around it: another process's environment variables, its memory, the files holding the keys. If everything inside the container runs as one user, all of that is just… readable.

The defense

Inside the single container, three distinct, unprivileged users run three distinct processes: the agent, the credential broker, and the egress proxy. The container starts as root only long enough to lay down permissions and network rules, then drops each process to its own user and never returns to root again.

The user running the agent is in none of the broker's groups, so the kernel denies it even the ability to traverse into the secret directory: every attempt comes back as a hard permission error, not a polite refusal. The broker's own source code is locked down the same way, so a compromised agent can't even read the logic it's trying to defeat.

In plain terms

Same building, different locks. Three tenants live under one roof, but tenant A's key was never cut to open tenant B's door. The building super hands out keys exactly once, at move-in, then leaves for good. Nobody can get a new key issued later, not even the super.

Kernel-enforced separation: not a convention, a permission error.

Layer 3: Secret isolation: The agent never holds a key

The threat

Credentials are the actual prize here: API keys, cloud tokens, platform tokens, registry passwords. In the naive design, they sit in the agent's own environment, and the agent can reach them the instant it wants to. One env dump, or a config file generated “with values baked in,” and they're gone.

The defense

We take the secrets away from the agent completely. The moment the container starts, before the agent process even exists, a classification pass runs over every environment variable. Anything that looks like a secret, caught by a generic naming convention, so a brand-new connector needs zero custom code, gets swapped, inside the agent's own environment, for a unique, single-use placeholder. The real value moves into a store that only the broker can read.

So the agent boots up and sees something like this, and nothing more:

One placeholder, one destination. Replays against any other host fail.

ANTHROPIC_API_KEY  = hbroker-Up4k…r3Q   # placeholder, not a key

ANTHROPIC_BASE_URL = http://127.0.0.1:8443/proxy/api.anthropic.com

The real key is nowhere in the agent's world, ever. When the agent makes its call, it builds a normal-looking auth header with the placeholder and sends it to the in-container broker. The broker, and only the broker, swaps it for the real key on the way out, and only at that exact moment.

The agent container holds zero cloud-credential bytes: the same model covers credentials that can't ride in a header. Cloud SDK credentials, Google service-account keys, AWS and Azure managed identity, get signed into the request body instead, so the broker stands in as a metadata server and mints short-lived credentials on demand. No key file, no service-account JSON, nothing.

In plain terms

It's the valet-key trick hotels use. The key you hand the valet starts the car and opens the door, but it won't touch the glovebox or the trunk. Even a dishonest valet holding that exact key can't drive off in the car parked next to yours, it simply doesn't turn that ignition.

Layer 4: Network isolation: Traffic can't leave the path

The threat

A compromised agent with open network access is a data-exfiltration tool, full stop. Phone home to an attacker's server. Push data to a public mirror. Tunnel it out over whatever port happens to be open.

The defense

All outbound traffic from the container goes through an in-container proxy running an allow-list. Approved hosts pass. Everything else is denied by default. The agent's traffic and the broker's traffic ride separate listening ports, so the two zones are never confused with each other.

A proxy only matters if the agent can't route around it, so three independent mechanisms keep it on that path:

• Set for the agent: not left to its discretion as a setting it could choose to ignore.
• An always-on guard inspects every command and file write the agent issues, and blocks anything that tries to disable or sidestep the proxy: unsetting proxy variables, a “no-proxy” flag, stripping the environment through a helper, mutating it from inside an interpreter, or even a quiet bypass written into a shell startup file.
• A kernel-level firewall rule pins the agent's user to the proxy and rejects every other destination, off-container hosts, the broker's private port, and cloud metadata endpoints. The capability that installs it is used once by root at startup, then dropped for good. The agent can never get it back, and can't edit, list, or flush the rule that contains it.

Real APIs occasionally echo a request's own credentials back inside an error body, so the broker scrubs responses on the way back too. A stray 401 can't hand the agent the very key the broker just injected.

In plain terms

One exit, and it's guarded. The fire doors are welded shut, the windows don't open, and the guard's own keys get taken away once the building is set up for the day, so nobody , not even the guard, can prop a door open later.

Three independent locks on one gate: not three settings, three different enforcement mechanisms.

Red-teaming our own design: Proof, not promises

None of this means much as an assertion, so we tested it the way an attacker would. We took a real, high-severity breach chain and replayed it against our own image, logged in as the agent.

Every other move in that chain failed the same way: each one stopped cold by a different layer.

Every layer is a release gate

Each isolation layer has its own end-to-end test that runs against the freshly built image, and the full breach-chain pentest above runs on top of all of them. Every check lives in the build pipeline ahead of the publish step, an image that regresses any single layer simply never ships, on either supported CPU architecture.

• Container smoke — Hardened image boots and is intact
• Process-isolation smoke — Three users separated; agent denied on secret files; egress ports pinned
• Secret-isolation test — Agent's environment holds only placeholders, never real secret bytes
• Per-host scoping test — A credential is injected only for its own host; off-target replays denied
• Egress-policy test — The allow-list is enforced and the deny-all default holds
• Credential-routing test — Each connector's traffic is routed and brokered correctly
• Proxy-bypass-guard test — Attempts to disable or route around the egress proxy are blocked
• Containment pentest — The full breach-chain replay above stays contained
• Image vulnerability scan — The published image clears the CVE policy

The proof isn't a slide from one audit done once. It runs on every release, and a single failure blocks the publish.

How it fits together

Security engineers have a name for this shape: the Swiss cheese model. Slice one piece of cheese and hold it to the light, there are holes in it. Every real-world control has gaps somewhere; that's just true, and pretending otherwise is how people get hurt. But stack four slices together, each with holes in different, unrelated places, and there's no longer a straight line through all four at once. None of these “slices” are guesswork, either. Each is enforced by a different mechanism, the build, the kernel, the filesystem, the network, so a gap in one almost never lines up with a gap in the next.

Attempt 1: finds a gap in image hardening, hits a solid wall at process isolation.

Attempt 2: gets three layers deep, still never reaches the prize.

A compromise of any single layer just lands in the next one. A tool that gets past the hardened image still can't read the secrets. An environment dump that gets read finds only placeholders. A placeholder that gets grabbed can't be aimed anywhere useful. And a connection that tries to phone home is pinned to the proxy and denied. That's the entire point of building it as four layers instead of betting everything on one wall.

The secure shape is the default

There's no separate hardening mode to opt into. You write an ordinary Agent step and pass your secrets the way you already do, under their normal names. The container does the rest before your agent process ever starts:

agent:
  step:
    run:
      container:
        image: harness-ai-agent:latest
      env:
        # Pass the real secret under the name the SDK already reads.
        # The engine swaps it for a placeholder before the agent starts;
        # The broker holds the real bytes and injects them only at egress.
        AWS_BEARER_TOKEN_BEDROCK: <+secrets.getValue("bedrock_token")>
        HARNESS_API_KEY: <+secrets.getValue("harness_token")>

That's the whole interface: no proxy wiring, no per-connector code, no placeholder plumbing on your side. The container classifies your secrets, swaps them for placeholders, writes a per-session network policy, drops to the three unprivileged users, and runs your agent, which never sees a single real credential.

Two things live in the platform, not the step: the three-user split and the read-only, capability-dropped runtime are set once in our published pod spec, and the kernel egress pin turns on wherever the runner can grant it. You don't wire any of it up per pipeline. It's how the image is meant to run.

‍

FAQs

Why layers instead of one strong control?

Because any single control can fail, and an agent running tools on untrusted output gives an attacker a lot of chances to find that failure. Layering means a break in one boundary gets caught by the next one. And each layer is enforced by a different mechanism – the build, the kernel, the filesystem, the network – so a gap in one doesn't line up with a gap in the next.

Where do the secret bytes actually live, and who can read them?

With the broker process, in a store owned by the broker's user and readable only by it. The user running the agent is in none of the broker's groups, so the kernel denies it access outright. The agent's own environment only ever holds placeholders.

Is the egress proxy mandatory, or can the agent route around it?

It's set for the agent rather than left to its discretion, and an always-on guard blocks the obvious ways to disable or bypass it. On top of that, a kernel firewall rule pins the agent's traffic to the proxy and rejects everything else – the same control enforced one level down, where the agent can't touch it.

How does a new connector get secured? Do you write code for each one?

No. Secrets are recognized by a generic naming convention, so a new connector's key gets placeholder-swapped and host-scoped automatically, with zero custom wiring required.

What about prompt injection itself – doesn't that defeat all of this?

Isolation doesn't try to stop the injection – it makes a successful one boring. A hijacked agent still can't read a secret, can't reach a host off the allow-list, and can't turn the container into a toolkit. The blast radius is bounded by the system, not by the model's judgment. Catching the injection payload in the content itself is a separate job, handled by the LLM gateway – that's the subject of the next post in this series.

‍

We shipped 62 features in June: roughly one every twelve hours! That pace isn't an accident. It's what happens when AI is writing more of the code, generating more of the tests, and clearing more of the review queue, and the rest of the delivery pipeline has to keep up.

This month's list touches almost every stage of that pipeline: builds that start faster, tests that write themselves, security scanning built for the volume of code AI agents now produce, feature flags that update without a redeploy, and on-call tooling that beats PagerDuty and OpsGenie on setup time alone. And on June 30, the biggest release of the month landed on top of all of it. Here's everything we shipped.

This month's biggest ships

Autonomous Worker Agents: Every step in a Harness pipeline, testing, security, deployment, and remediation, can now run as a reasoning agent instead of a fixed script, with the same governance and audit trails enterprises already trust for human-run deployments. It's the difference between AI helping write a script and AI running the step.

Parallel pipeline execution using a Directed Acyclic Graph: Pipelines can now declare dependencies between stages and run every independent one in parallel automatically, unlocking fan-out, fan-in, and diamond-shaped flows that sequential pipelines couldn't express. A convert-to-DAG API rewrites existing pipelines automatically, so the migration is a button, not a rebuild.

AI Engineering Insights: Measures what most engineering orgs are still guessing at: AI coding agent adoption over time, AI-committed code percentage, spend per developer, and PR cycle time for AI-assisted developers versus everyone else. It's the metric layer for the AI productivity paradox, not another dashboard of vanity stats.

‍

Here are the details:

Every pipeline step can now be an agent

On June 30, Harness introduced Autonomous Worker Agents, a platform for building and safely running AI agents that handle the work between writing code and shipping it to production. Every step in a pipeline, testing, security, deployment, and remediation, can now run as a reasoning agent instead of a fixed script, governed by the same scoped credentials, OPA policy enforcement, approval gates, and audit trails enterprises already use for human-run deployments.

Agents pull context from the Harness Knowledge Graph, connecting services, pipelines, deployments, incidents, and policies, and act through the Harness MCP Server, so a developer working in Cursor, Claude Code, or another editor can hand a task to a Worker Agent and get the result back without leaving their tool. A new Agent Marketplace lets teams find, clone, and customize both Harness-managed and community-built agents, so the agent one team builds to solve a problem becomes the starting point for the next team that hits the same wall. Worker Agents are available now to all Harness customers and work with any LLM provider. Read the full announcement here.

Builds that start faster and cache smarter

Docker Layer Caching and Build Cache now both support Azure Blob Storage as a backend, joining S3 and GCS. Teams running on Azure no longer have to route their caching through a different cloud to get the speed benefit. Find more details in the CI release notes.

CI build initialization on Kubernetes infrastructure now sends only the fields required to start a build, shrinking the initialization payload. Pipelines with a large number of steps start faster and more reliably as a result.

A new Test Management Dashboard lists every test, including flaky and quarantined ones, with health status and last-run results in one place, so no one has to go hunting through build logs to find the test that's been silently failing all week.

Small but real: the lite-engine HTTP client now handles 3XX and 4XX responses from the log service correctly, so a redirect or an error response during log streaming no longer takes the build down with it.

AI writes the tests, coverage tells you if it's enough

AI Test Automation now runs Playwright test suites natively, in beta, as a first-class execution engine. Teams bring their existing Playwright suites and run them on the platform with zero infrastructure to stand up or maintain. Learn more about running Playwright tests on Harness.

Watch the demo:

Existing tests can now be converted into natural language and downloaded, so teammates who don't work in code, PMs, auditors, and new hires can review or document test coverage without reading a test file. More on that in the AI Test Automation release notes.

And Harness Code Repository now shows what percentage of your codebase is actually covered by tests, surfacing untested paths before they turn into incidents.

Deployments that scale without more YAML

Pipelines can now declare dependencies between stages with a dependsOn field. Harness runs every independent stage in parallel the moment its predecessors finish, unlocking fan-out into multiple parallel deployments, fan-in to a single approval gate, and diamond-shaped flows where build, test, and infra stages converge before promotion, patterns a sequential pipeline simply couldn't express. A new convert-to-DAG API rewrites an existing sequential pipeline automatically, so migration doesn't mean a rewrite.

Templates now support custom floating tags, not just the built-in stable pointer. Template owners can create tags like prod, canary, or qa, each pointing at a different template version, and repoint any of them independently without touching a single line of consumer YAML. That means you can promote a new template version to canary, let it bake, then move prod once it's validated, with different environments running different versions at the same time.

A handful of other pipeline and delivery upgrades this month: the Copy command in Command steps can now preserve directory structure so same-named files in different subfolders stop overwriting each other on target hosts; Kubernetes Dry Run steps now accept kubectl flags like --server-side and --force-conflicts, so approval gates reflect what a real deployment will do; the Artifactory connector supports OIDC for credential-free authentication with short-lived JWTs; AWS OIDC connectors now tag sessions with environment identifiers, letting you restrict production secrets to production pipelines even on a shared delegate pool; Istio traffic routing steps support AND/OR match logic across route rules; approval steps can show details to non-approvers without granting them approval rights; and issue-comment webhook triggers now resolve to the latest commit message instead of the PR description, bringing them in line with every other trigger type. Find the full rundown in the continuous delivery release notes.

Security built for how much code AI agents write now

A new AI-powered SAST engine, in beta, validates findings in context to cut triage effort and catch vulnerabilities that traditional static scanners miss, purpose-built for a world where AI-generated code is multiplying the volume of things that need checking. Learn more about AI SAST.

‍

API Security Testing scans that get interrupted by timeouts or infrastructure issues can now resume from the point of failure instead of restarting from scratch. Scans also picked up a real-time validation summary tab, plus the ability to label and rename them so a security team running dozens of scans a week can actually tell them apart. Details are in the scan documentation.

‍

A new API Inspector automatically analyzes OpenAPI specs for security, design, and data validation issues before deployment, catching problems while they're still cheap to fix.

Approvers reviewing exemption requests can now adjust the requested duration instead of only accepting or rejecting it outright, which means fewer rejected requests that just get resubmitted with a smaller ask.

Teams can now bulk onboard Bitbucket repositories and run SBOM and security scans automatically, no manual pipeline setup required, and role-based permissions expanded across more supply chain workflows. Learn more about Bitbucket onboarding.

Harness AI now generates OPA policies directly, trained on OPA and REGO best practices, and can explain existing policies in plain language, so writing policy-as-code no longer requires deep REGO expertise on the team.

Rounding out security: issue detection policies can now scope to specific span attributes to cut noise, API exclusion rules gained span-attribute filtering for the same reason, a new Applications view groups related APIs and AI assets into business-centric categories like payments or order processing so large API inventories stay navigable, the older API Activity dashboard is being retired in favor of these newer views, and Role Assignment API endpoints now validate request payloads up front, returning clear errors instead of failing downstream.

Feature flags that update without a redeploy

Feature flag definitions can now be changed through a pipeline step that applies structured, atomic updates, batching multiple configuration changes into one consistent operation instead of stitching together raw patch calls. Find more details here.

And a new family of thin SDKs, built for browsers, mobile apps, and edge or serverless JavaScript, delegates flag evaluation to a remote evaluator in the cloud instead of computing it on-device. Rollout rules and segment definitions stay server-side; the SDK only gets the result. Learn more in the feature flag release notes.

Finding out what AI is actually doing to your engineering org

A new AI Engineering Insights capability tracks AI coding agent adoption across teams over time, measures output with metrics like AI-committed code percentage and lines generated, and monitors spend per developer and per commit. It also compares AI-assisted and non-AI developer performance head-to-head on PR cycle time and work items delivered, with a leaderboard for drilling into individual patterns. Read more in the AI DLC Insights release notes.

Cloud and AI cost management gets more precise

The Cloud and AI Cost Management overview page got a redesign: new widgets for top spenders, optimization impact, and service breakdown, plus support for AWS and GCP cost adjustments.

You can now connect OpenAI and Anthropic accounts directly through a guided three-step wizard with secure API key storage and live connection testing, currently behind a feature flag. Grouping AI traces by service in Cost Explorer now opens a drawer with run history, a span-level timeline, and cost attribution down to the individual trace. And a new Asset Governance tab lets you set cost preferences separately for AWS, GCP, and Azure, also behind a feature flag.

The rest of this month's cost updates were accuracy and access fixes: anomaly filters no longer silently ignore a perspective that's been deleted, nodepool savings calculations now use the same cost basis as the monthly total instead of overstating savings, overview anomaly counts now include both resource-level and cost-category anomalies, only users with the right permission can edit or delete anomaly alerts, recommendation savings labels are localized to your preferred cost type, repeated commitment renewal events now collapse into a single indicator instead of a wall of icons, and filter messaging and recommendations navigation both got cleaned up. Full details are in the cost management release notes.

The developer portal gets better at knowing what you actually have

New integrations auto-discover and ingest entities from your existing tools, no manual YAML, no custom scripts. Unlike frontend-only plugins, this data feeds into scorecards, workflows, aggregation rules, and the knowledge graph itself.

Bitbucket Cloud entities in the catalog now show pull request history and commit activity right on the entity page, and Dynatrace entities show monitor and SLO data the same way, so developers get observability context without leaving the portal. More on that in the IDP release notes. 

Environment Management now shows a clear, current state for every environment, so platform engineers know what's actually happening without cross-referencing three other tools.

Infrastructure as code, managed like a real registry

Module Registry 2.0, in beta, lets teams publish IaC modules as immutable artifacts, auto-sync new versions on publish, and govern them with Supported, Warning, and Deprecated controls across project, org, and account scopes. Self-service, with guardrails, instead of a spreadsheet tracking who's on which version. Learn more about Module Registry.

Drift detection can now run on any schedule you set, and workspaces can be given a TTL so short-lived infrastructure, PR environments, and feature branches tears itself down automatically instead of quietly running up a cloud bill. Find more details on ephemeral workspaces.

Database changes that ship like code

Database DevOps now supports Harness Code Repository natively as a schema source, so teams no longer need a separate Git connector just to point at their own repo.

And Database DevOps can now authenticate to Amazon RDS and Aurora using AWS IAM through Harness's delegate, currently in beta and behind a feature flag, cutting down on credential management overhead and closing off a class of stored-secret risk.

Artifacts that clean up after themselves

Harness Artifact Registry now supports policy-based lifecycle rules that automatically delete or protect artifacts based on version count, age, or download activity. Administrators set the policy once instead of manually pruning old builds every quarter. Learn more in the Artifact Registry release notes.

Resilience testing before production finds the gap for you

Resilience Testing picked up Linux and Windows chaos experiment templates for the Chaos Step, audit trails with YAML diff visualization for the image registry, better probe timeout and retry handling, and the ability to copy output variables straight from the timeline view. Find the full list in the Chaos Engineering release notes.

Incident response that outpaces PagerDuty and OpsGenie

Escalation policies can now target a single rotation inside a schedule instead of paging every responder across it. If an incident belongs to "IT West Critical," only that rotation gets paged. Rotations, schedules, and users can all be mixed as targets across different levels of the same policy, so each escalation step reaches exactly the right people.

On-call configuration can now be imported by picking the exact services and groups you want, instead of pulling an entire account in one shot, and it works across PagerDuty, OpsGenie, and xMatters. That's a direct answer to the biggest complaint about migrating off those tools: an all-or-nothing import that forces a weekend cutover.

Runbooks can now be duplicated in one click, chain items, action configurations, and metadata all carried over intact, with the copy opening automatically so you can start tailoring it instead of rebuilding from scratch.

The pattern

None of these features exists in isolation. AI is compressing how fast code gets written, which means the pressure shows up everywhere downstream: builds need to start faster, tests need to generate themselves, security scanning needs to handle more volume without more false positives, and incident response needs to route the right alert to the right rotation the first time. Autonomous Worker Agents is the next turn of that same screw: instead of just helping teams keep up with AI-written code, Harness is now putting AI directly into the pipeline steps that ship, secure, and operate it. Sixty-two features in thirty days, plus a new agent platform on top, is what closing the velocity gap looks like in practice. We'll be back next month with more of it.

‍

Harness AI Security provides a unified control plane for AI discovery, risk visibility, and runtime protection, helping organizations operationalize key requirements of the EU AI Act. Instead of relying on manual audits or fragmented tooling, teams get continuous insight into how AI systems are built, exposed, and used, along with the evidence needed to demonstrate compliance.

By combining AI asset discovery, risk classification, data flow visibility, and runtime enforcement, Harness enables customers to proactively identify high-risk systems, prevent unsafe integrations, and continuously monitor AI behavior in production. This approach aligns directly with the EU AI Act's focus on transparency, traceability, and ongoing risk management.

Harness helps in the following areas of the EU AI Act

AI System Inventory

Harness automatically discovers all AI assets—AI APIs, Agents, MCP servers, MCP tools, resources, prompts, and AI backends by analyzing live network traffic. The centralized inventory provides a real-time breakdown of discovered assets by type, call volume trends, and sensitive data exposure across your environment. Security and compliance teams gain a single, continuously updated source of truth for every AI component in use, without requiring manual cataloging or developer-submitted forms.

Risk Identification & Classification (Article 6)

Once assets are discovered, Harness derives a risk score for each based on policy violations, known vulnerabilities, exposure level (internal vs. external), and sensitive data flow. This scoring helps teams prioritize remediation efforts and demonstrate that high-risk AI systems have been identified and assessed, which is a core expectation under the EU AI Act's risk-based framework.

Prohibited Use Cases (Article 5)

Harness detects shadow AI vendors, unapproved MCP servers, and undocumented AI APIs surfacing in your environment. The Third Party view surfaces AI APIs grouped by vendor (e.g., OpenAI, Google, Anthropic) so teams can identify integrations that haven't undergone procurement or security review. This is directly relevant to the EU AI Act's prohibition on certain AI use cases and its requirements around supply chain transparency for AI systems.

Data Governance & Quality (Article 10)

Harness monitors sensitive data flows across all discovered AI assets, identifying where PII and regulated data enters and exits AI systems. The platform classifies data sensitivity automatically by analyzing asset metadata and observed traffic patterns, giving teams a continuous view of which assets handle sensitive information and surfacing misuse risks before they become compliance incidents.

Technical Documentation & Auditability (Articles 11, 16)

Harness automatically generates schemas for AI APIs, MCP tools, resources, and prompts by analyzing real network traffic with no manual documentation effort required. Each asset detail page captures the asset's type, dependencies, call volume, risk posture, and data flows in one place. This detail provides compliance teams with the structured technical records required under Articles 11 and 16 without burdening engineering teams with additional documentation.

Logging & Traceability (Article 12)

Harness captures all AI interactions, including AI API calls, MCP tool invocations, database calls, and non-AI API calls, in a centralized data lake with seven-day standard retention and 30-day retention for threat activity. This complete, queryable record of AI system behavior supports both routine audit needs and forensic investigations, directly satisfying Article 12's requirements for logging and traceability of high-risk AI systems.

Accuracy, Robustness & Security (Article 9)

The AI Firewall (beta) provides runtime enforcement against the most common AI-layer threats: prompt injection attacks, PII leakage in model responses, excessive model usage, and unauthorized model access. Together, these controls address the robustness and security requirements of Article 9, helping organizations demonstrate that their AI systems have active protections in place rather than passive policies.

Post-Market Monitoring (Article 72)

Harness continuously discovers new AI assets, shadow AI usage, and emerging sensitive data risks in production as your environment evolves. Real-time alerts are triggered for new vulnerabilities and compliance violations, with native integrations into SOC/SIEM workflows for rapid response. This ongoing monitoring capability aligns directly with the EU AI Act's post-market surveillance requirements, ensuring compliance doesn't end at deployment.

Bottom line: Harness AI Security provides the visibility, controls, and audit evidence layer required to operationalize EU AI Act compliance at scale and oversee AI system security. (Article 14)

AI is writing more of the code. Software delivery, the work between writing code and running it in production, is where most of the day still goes. Building, testing, scanning, deploying, remediating, and operating still require the same, if not more, effort as before AI. 

Today, we're introducing Autonomous Worker Agents for software delivery: the platform for enterprises to build and safely run AI agents that handle the work between writing code and shipping it to production. 

Autonomous Worker Agents execute as pipeline steps and produce auditable outputs. Their memory is the organization: services, pipelines, deployments, incidents, policies, all connected through the Harness Knowledge Graph, and their capability is powered by the Harness MCP. They operate in production and support the deployment, security, remediation, and validation of your code.

They join Harness Expert Agents, which have been available to customers for some time, to form a complete AI layer across the platform.

Each agent runs as a step inside a Harness pipeline, on customer-controlled infrastructure, with full governance: scoped credentials, OPA policy enforcement, approval gates, and complete audit trails.

Safe to Run in Production

Autonomous Worker Agents are invoked as pipeline steps or independently. They inherit the governance Harness pipelines already provide. Instead of trying to teach an AI agent a massive list of corporate rules, the agent operates entirely within the constraints of your existing software delivery pipelines.

• OPA Policies that gate production deployments gate the agents.
• RBAC that controls who can push to production controls who can trigger an agent.
• Approval Gates apply before an agent's fix ships, just as they do before any release.

Safety is architected in as well. Workloads execute on Harness Delegates, lightweight runtimes installed inside the customer's own Kubernetes cluster or VPC. An agent that "shouldn't be able to merge to main" cannot merge to main, even if its prompt asks it to. The architecture enforces it.

‍

We built RiskSentinel, a Harness Autonomous Worker Agent, to demonstrate that governed AI can move beyond identifying security issues to safely remediate them while maintaining enterprise controls, auditability, and compliance. When building with Harness, what stood out most was how intuitive the experience was — it enabled our team to move from an initial idea to a production-ready agent in just four days, allowing us to focus on solving a real enterprise challenge rather than the underlying platform. That combination of developer experience and enterprise-ready capabilities is what will enable organizations to confidently scale AI across software delivery.
‍- Ratna Devarapalli, Director IT, United Airlines

Six additional controls make Autonomous Worker Agents production-safe.

1. Sandboxing

Agents are run containerized, with non-root execution (UID 65534, "nobody"). Their filesystem is read-only except for the workspace. Network access is configurable per agent: unrestricted, restricted to allowed MCP servers, or fully disabled.

An agent that produces a malicious bash command has nowhere to send the data.

2. Scoped Credentials

When a pipeline triggers, Harness mints an ephemeral scoped token. Its scope is the intersection of the agent's permissions and the triggering user's RBAC.

Token deletes on completion. TTL as a failsafe. MongoDB TTL index as final backstop.

3. Policy Enforcement

OPA policies, the same framework Harness customers use to govern deployments, apply to agents. Policies govern the agent at runtime and during configuration. 

4. Audit Trails

Every execution is captured in the Harness Audit Trail. This includes a full provenance chain: who or what triggered the agent, template version, every action taken, and final outcome.

Prompts and reasoning chains are sanitized before persistence: secrets stripped, and PII is stripped.

5. Cost Tracking

Token consumption and costs are surfaced per execution, per agent, and per pipeline. Running totals are shown live in the step header.

6. Chaining

Agents are architected to run within pipelines and can be naturally composed into multi-step workflows.

• Sequential: Agent B consumes Agent A's output.
• Parallel: agents run simultaneously.
• Conditional: an agent runs only if a previous step meets a condition.
• Matrix: same agent across repos, environments, or services.

Output handoff happens via pipeline expressions and shared workspace files.

Three ways to create an agent

Using YAML

A Worker Agent is defined in a single file. Here's a complete agent that reviews every pull request for security issues:

agent:
   group:
     steps:
       - name: Run Code Coverage Agent
         id: runCodeCoverageAgent
         if: <+Always>
         run:
           container:
             image: pkg.harness.io/vrvdt5ius7uwygso8s0bia/harness-agents/harness-ai-agent:latest
           env:yam
             ANTHROPIC_MODEL: ${{inputs.model_name}}
             PLUGIN_HARNESS_CONNECTOR: ${{inputs.llm_connector.id}}
             PLUGIN_MAX_TURNS: "150"
             PLUGIN_MCP_FORMAT: harness
             PLUGIN_MCP_SERVERS: <+connectorInputs.resolveList(<+inputs.mcp_connectors>)>
             PLUGIN_TASK: |
               Autonomous Harness Code Coverage Agent; no prompts. Resolve branch/repo/clone_url/account/org/project/execution strictly: input -> env -> MCP, never guess; branch must exist via SCM MCP or fail.
               Use /harness first, else $HARNESS_WORKSPACE; if repo missing, clone (SCM MCP preferred, git fallback) and checkout resolved branch.
               Detect language/test/coverage stack, run baseline coverage (overall + per-file), and target >=90% overall and >=80% per-file.
               Add meaningful tests for critical uncovered paths (happy/edge/error/boundary); allow only minimal production testability tweaks.
               Re-run full tests + coverage + lint + build; all must pass before continuing.
               Review full diff (SCM MCP preferred, git diff fallback); allow only tests + minimal testability tweaks (+ COVERAGE.md only if it already exists; never create it).
               Build report with overall before->after, per-file before/after for touched files, and key improvements.
               Stage files one-by-one only; never use git add -A or git add .; verify staged diff is clean and in-scope.
               Create exactly one commit: "Code coverage: automated test additions by Harness AI"; push plain to origin <branch> (no pull/rebase/merge/force).
               If push fails, print rejection, git reset --hard HEAD~1, exit non-zero; never commit unrelated changes, never weaken existing tests, never log secrets.

YAML frontmatter on top. Natural language below ---. The same convention Jekyll, Hugo, and AI agent definitions across the industry use.

Save the file, commit it to the repo, and the agent is live, governed, and in the catalog. Every PR triggers it. Every run is audited. Every action is scoped by RBAC. From a blank file to a live governed agent in minutes.

The Harness pipeline engine handles container runtime, scoped credentials, MCP server integration, audit logging, and cost tracking.

Using the UI

The Harness Agent Builder is a simple form for configuring your Agents. Define your prompts in plain English, referencing Harness constructs through common expressions. This experience makes it easy to see what you need to provide and set up your agent in minutes.

All agent definitions are stored in Harness. Their reference in pipelines can be managed in Git. Approval gates apply. Pipeline Branch-based versions let teams test new agent behavior in feature branches before merging to main.

"We built an agent that handles log analysis directly inside Harness. No tool switching, no context loss. The ability to stay on one platform and have the agent surface what's happening and review it for us was the biggest immediate win. We're planning to use it in production."
- Mandy Pearce, Senior Engineer, Cloud Automation, Verint

Create with MCP

Using your favorite coding agent, you can connect to Harness over the MCP. The MCP bridges the AI Coding agents’ inner-loop context and the outer-loop context and the constructs in Harness. 

Agents as Pipeline Steps

Most software delivery workflows have more than one step. Autonomous Worker Agents compose with shell scripts, plugins, approval gates, and other agents to make full pipelines.

Referencing an Agent in a Pipeline

pipeline:
 stages:
   - steps:
       - name: Feature Agent
         template:
           uses: ca_feature_triage_agent@1.0.2
       - name: Plan Agent
         template:
           uses: ca_work_planning_agent@1.0.2
       - name: Build Feature Agent
         template:
           uses: ca_builder_agent@1.0.2

uses: references a Worker Agent template by name and version. The agent runs as one step alongside everything else a Harness pipeline can run.

Sequential: Output Handoff

Agent B consumes Agent A's output. The pipeline expression ${{ steps.<agent_id>.output }} carries the result forward.

pipeline:
 stages:
   - steps:
       - name: spec design
         parallel:
           steps:
             - name: Feature Agent
               template:
               uses: ca_feature_triage_agent@1.0.2
             - name: PR Body
               template:
                 uses: pr_body_writer
                 with:
                   artifactPath: ${{featureagent.output.artifact}}
                   issueKey: cds-1234

Parallel

Multiple agents run simultaneously:

 parallel:
           steps:
             - name: Feature Agent
               template:
                 uses: ca_feature_triage_agent@1.0.2
             - name: PR Body
               template:
                 uses: pr_body_writer
                 with:
                   artifactPath: ${{featureagent.output.artifact}}
                   issueKey: cds-1234

Step Groups

A Step Group bundles agents and deterministic steps into a single reusable unit:

         group:
           steps:
             - name: feature anaylzer
               template:
                 uses: feature_ingester_agent@1.0.2
             - name: work planner
               template:
                 uses: ca_work_planning_agent@1.0.4

Save the group as a template. Reference it from any pipeline. The PR Autofix workflow ships as a Step Group template.

Conditional and Matrix

An agent runs only when a condition is met:

  - steps:
         group:
           steps:
             - name: feature ingest
               template:
                 uses: feature_ingester_agent
             - name: work planner
               template:
                 uses: ca_work_planning_agent
     name: Spec Driven Development
     if: <+OnPipelineSuccess>

The same agent runs across multiple targets:

             - name: work planner
               template:
                 uses: ca_work_planning_agent
               strategy:
                 fail-fast: true
                 for:
                   iterations: 3

Approval gates, failure strategies, retry policies, and rollback work the same way they do for any other pipeline step.

‍

Introducing the Harness Agent Marketplace

The Harness Agent Marketplace is where teams discover, install, fork, customize, and publish Autonomous Worker Agents.

Three publisher tiers anchor it:

• Harness Managed: Built and maintained by Harness. SLA-backed. Versioned. Pinnable (e.g., harness.autofix@1.2).
• Harness Certified: Partner-built. Reviewed and certified by Harness engineering and security. Examples: dependency vendors with their own scanning agents, cloud providers with cloud-specific deployment agents.
• Community: Published by the broader Harness community. Validated for schema, no secrets in prompt. Enterprise accounts can restrict via OPA policy. Allow only Managed and Certified in production, for instance.

Harness Managed Agents

With today’s launch, Harness has pre-built agents for the most requested use cases. Here are some examples of what’s currently available: 

CI Autofix

Reads build logs from a failed PR build, identifies the root cause, commits a fix to the PR branch, re-triggers the build, and repeats until the build passes or the configured max-turns limit is reached.

Manifest Remediator

Analyzes failed Kubernetes deployments. Identifies whether the issue is the manifest, the cluster, or the workload. Fixes manifest issues. Used by teams managing dozens of services across multiple clusters.

Code Review

Reviews PR diffs across security, quality, and test coverage. Outputs structured findings with severity ratings and concrete remediation. Grounded in the Harness Knowledge Graph, the agent knows which services are production-critical, which have had recent incidents, and which historical anti-patterns have caused outages.

Feature Flag Cleanup

Reads code, config, and flag-system state to identify feature flags that are fully rolled out or fully off. Once it validates removal is safe, the agent generates a cleanup PR. With this agent, the status of your experiments automatically informs you when flags are cleaned up, reducing flag debt and the drudgery of cleaning up old flags.

Code Coverage

Reads coverage reports, identifies untested lines, branches, and functions, and generates tests to close gaps. Used when a team has inherited a codebase with weak coverage and needs to lift it before a release.

IaCM Remediation

Fixes configuration drift, security findings, and cloud cost issues by editing infrastructure configurations.

‍

Bring Your Own Model

Autonomous Worker Agents are model-agnostic. Connect LLM providers through Harness connectors:

• OpenAI: Direct to Provider
• Anthropic: AWS Bedrock, Direct to Provider 

The model can be specified at three levels: in the agent template, at the pipeline step level (overriding the template), or at the account level via environment variable defaults. Switch models per agent, per environment, or per pipeline without changing agent logic.

Three reasons this matters:

1. Cost. Different models have different price points. Routing high-volume work through cheaper models is a common pattern.
2. Compliance. Some teams require AWS-routed Bedrock for billing consolidation, VPC routing, or Bedrock-specific compliance attestations.
3. Future-proofing. Model leaders change. The enterprise decides which model today, which model tomorrow.

Getting Started

Autonomous Worker Agents are available today for all Harness customers. Learn more about Harness Autonomous Worker Agents or request a demo to see them in production.

Visit the in-app Harness Marketplace in app to try out any of the Worker Agents. Add it to your pipeline and watch it run. 

Federal agencies need more than a modernization tool, – they need an “automated” early warning software delivery system. The Canary Partner Program is how Harness and its partners deliver one.

‍

Miners carried canaries underground for one reason: early warning. The moment the canary went silent, you knew something was wrong before it was too late to act. That’s exactly what’s been missing from federal software delivery — and exactly what the Harness Canary Partner Program is built to provide.

Across the Department of Defense, Civilian Agencies, and the Intelligence Community, we keep hearing the same thing: harness AI, harness automation, prioritize governance and security. Agencies aren’t just modernizing,  they’re racing to get ahead of risk before it surfaces in production, before it shows up in an audit, and before it becomes a breach. 

Today, we’re opening the Canary Partner Program to federal-focused partners who can help them do that.

‍

Why federal, and why now

Federal IT modernization is accelerating. Agencies aren't asking whether to modernize their software delivery anymore. They're asking how, and they're asking fast. The Administration's AI and cloud-first directives, RMF framework requirements, zero trust mandates, continuous ATO expectations, are all all of it is landing at once. We intend to be the platform agencies depend on, and the Canary Partner Program is how we scale that across the federal ecosystem.

We're already seeing it firsthand. Harness is deployed across federal civilian and defense agencies, where teams are reducing deployment lead times, increasing release frequency, and enforcing security policies at scale. The demand is growing faster than our direct team can serve alone, and that's exactly why we built this program.

‍

What the Canary Partner Program actually looks like

The Canary Partner Program is built around a co-sell motion that puts Harness resources behind partners from the first opportunity. Unlike traditional tiered partner models with steep upfront certification requirements and a long runway before seeing any return, partners work alongside our federal sales, solutions engineering, and professional services teams from day one, — whether they're a government-focused reseller, systems integrator, or consulting firm looking to grow their federal practice. The engagement model is flexible: co-sell, resell, or managed services, based on what each opportunity calls for.

What partners get access to:

• A self-service partner portal with role-based training for sales, presales, and implementation
• Environments and federal compliance resources aligned to FedRAMP, NIST, RMF, and Zero Trust frameworks
• Deal registration protections and competitive incentives that reward both partner-sourced and partner-influenced pipeline
• Co-marketing resources, federal campaign kits and joint go-to-market support across civilian, defense, and intelligence communities

The enablement is built around the full chain of custody that agencies are increasingly requiring partners to demonstrate: code to artifact to deploy to runtime, with traceability at every step.

‍

Come join us on this mission

The Harness Canary Partner Program is open now, and this is just the start. We have more federal news coming, and the partners who are in early will be best positioned to move with us as the program grows. If you're ready to build something that lasts in the federal market, we'd like to do it together.

Reach out or apply at www.harness.io/partners.

‍

AI coding security risks emerge the moment your assistant suggests `npm install suspicious-package` and your team accepts without question. In production environments, AI-generated code recommendations bypass traditional review workflows, introducing vulnerable dependencies at a pace human oversight cannot match. One accepted suggestion can pull in dozens of transitive dependencies, each a potential supply chain entry point.

This is not about slowing developers down. It is about giving platform and security teams a scalable control point while developers keep moving fast.

‍

How AI Assistants Accelerate Dependency Risk

AI coding tools operate on pattern recognition trained across millions of public repositories. When a developer asks for authentication logic, the assistant suggests popular packages based on usage frequency, not security posture. The tool has no visibility into CVE databases, package maintainer history, or recent compromise patterns. It recommends what worked statistically, not what remains safe operationally.

This creates volume problems traditional security gates cannot address. A team of ten engineers using AI assistance can introduce 50 new external dependencies per sprint. Manual security review of each package, its maintainers, and its transitive tree becomes a bottleneck that development velocity simply routes around. The dependencies enter `package.json`, pass CI checks that only verify build success, and deploy to production before anyone evaluates supply chain risk.

‍

The npm Supply Chain Attack Surface

Recent npm ecosystem compromises demonstrate how attackers exploit this acceleration. Package maintainers get compromised through credential theft or social engineering. Attackers publish malicious updates to widely-used packages. AI assistants continue recommending these packages based on historical popularity metrics. Development teams install them automatically as part of normal workflow.

The May 2026 TanStack supply chain attack illustrates the current threat landscape. Attackers published malicious npm packages impersonating TanStack libraries, targeting developer credentials, secrets, and CI/CD-related access tokens. Because TanStack packages are widely used across React ecosystems, AI coding assistants readily suggested these typosquatted variants based on name similarity and perceived popularity. Teams relying on AI suggestions without registry-level controls had no automated way to prevent these packages from entering their dependency trees. The attack specifically exploited the trust developers place in AI-generated recommendations, harvesting credentials that could enable deeper supply chain compromise.

The older ‘event-stream’ incident followed a similar pattern. A legitimate package with millions of weekly downloads received a malicious update that harvested cryptocurrency wallet credentials. The compromise remained undetected for weeks because the package maintained its reputation score and continued appearing in AI-generated suggestions.

‍

Why Traditional Security Checks Miss AI-Introduced Dependencies

Standard vulnerability scanning happens too late in the development lifecycle. Most teams run security checks after code reaches staging or pre-production environments. By this point, vulnerable dependencies have already integrated into application logic, created transitive dependency chains, and potentially exposed sensitive data in development environments.

SCA Tells You What Risk Exists. Dependency Firewalls Stop It From Entering.

This is the critical distinction. Software Composition Analysis (SCA) tools scan your codebase and report which vulnerable packages you already have. They are reactive: they tell you about risk after it exists in your environment. Dependency firewalls are preventive: they stop risky packages at the registry boundary before they ever reach your codebase, builds, or pipelines.

SCA scanning remains valuable for ongoing visibility. But when AI coding tools introduce dependencies at high velocity, you need a control that operates before installation, not after. The registry boundary is that control point.

Static analysis tools detect known CVEs but miss zero-day vulnerabilities and recently compromised packages. The gap between package compromise and CVE publication creates a window where AI assistants continue recommending dangerous dependencies while security databases report clean status. Teams operating on daily or weekly vulnerability scan schedules remain exposed to supply chain attacks that evolve hourly.

License compliance presents another blind spot. AI coding tools suggest packages based on functionality, not licensing terms. A developer receives a suggestion for an AGPL-licensed package when building a proprietary commercial application. The licensing conflict only surfaces months later during audit preparation, requiring expensive refactoring or license negotiation.

The Speed-Versus-Security Tension

Development teams adopt AI assistance specifically for velocity gains. Asking developers to manually verify every AI-suggested dependency contradicts the efficiency goal that justified the AI tool investment. This creates a cultural pressure where security verification becomes the exception rather than the rule.

‍

Why the Registry Boundary Is the Natural Control Point

The upstream proxy is where external dependencies enter your organization. Every npm install, pip install, or maven dependency resolution that reaches out to a public registry passes through this layer. This makes it the natural enforcement point for open-source governance.

Think of it the same way you think about network firewalls. You do not let arbitrary external traffic into your internal network without evaluation. The same logic applies to software packages. The upstream proxy fetches and caches artifacts from external registries, so placing policy evaluation at this layer means every external package gets assessed before it becomes available to any developer, any build, or any pipeline in your organization.

A dependency firewall at the registry boundary evaluates external packages before they enter your organization's artifact ecosystem. Rather than scanning for vulnerabilities after installation, it blocks retrieval of packages that fail security, compliance, or policy checks. When a developer or AI assistant attempts to install a package, the request routes through the firewall, which evaluates:

- Known vulnerabilities: CVSS severity scores checked against your defined threshold.

- Package age and stability: Newly published packages can be flagged or blocked.

- License compatibility: Alignment with organizational compliance requirements.

- Custom policy rules: Organization-specific policies written in Rego for nuanced control.

- Transitive dependency risk: Security posture of the complete dependency tree.

Packages that fail evaluation are not cached and are not available for download. Developers receive immediate feedback about why the package was blocked. This shifts security decisions from post-integration remediation to pre-installation prevention.

‍

How Harness Artifact Registry Implements Dependency Governance

Harness Artifact Registry implements dependency firewall capabilities as part of its upstream proxy architecture. When configured as your primary package source, it evaluates external dependencies against configurable policies before caching them for internal use.      

The Flow

Here is how it works in practice:

‍

- **Blocked** versions are not cached and are not available for download. The install fails with a clear policy violation message.

- **Warning** versions are cached and available for download, but flagged for visibility. Teams can review warnings on the dashboard and decide whether to tighten policy.

- **Passed** versions are cached normally and available without restriction.

Policy Configuration

The platform supports configurable policy sets that you apply to your upstream proxy registries:

•  CVSS severity threshold: Block packages with known vulnerabilities above your defined severity level (e.g., block all Critical and High CVEs).
• License policy: Block or warn on packages with specific license types (GPL, AGPL, or any license incompatible with your distribution model).
• Package age policy: Flag or block packages published within a configurable window (e.g., packages less than 30 days old receive a warning).
• Custom Rego policies:Write organization-specific rules for nuanced evaluation logic that goes beyond built-in checks.

Policy sets group-related rules and can be applied across multiple registries. This means security rules defined once apply consistently, whether developers are pulling JavaScript packages, Java libraries, or container images. For a deeper look at how this fits into a unified artifact management strategy, Harness AR handles the full lifecycle from ingestion to deployment.

Visibility and Governance

The Dependency Firewall dashboard provides visibility into policy evaluations, showing which packages were blocked, which received warnings, and which passed. This supports both incident response and continuous policy refinement based on actual development patterns.

Integration with CI/CD pipelines ensures build environments use the same controlled package sources as local development. A package that passes firewall evaluation in development remains available in CI without re-fetching from public registries. This consistency eliminates scenarios where local and build environments reference different package versions or bypass controls.

For organizations managing multiple package formats (npm, Maven, Docker, Helm), Harness AR provides unified policy management across registries, reducing policy management overhead while maintaining comprehensive supply chain governance.

Operational Implementation

Initial firewall deployment involves cataloging currently used dependencies and establishing baseline policies. Most organizations start with blocking known high-severity CVEs and expand policy coverage incrementally. This prevents disrupting existing workflows while building the approved package catalog that development teams and AI tools can safely reference.

Learn more about Harness Artifact Registry for more information about implementing these controls.

‍

Building Sustainable AI-Assisted Development Workflows

Dependency firewalls enable rather than restrict AI coding tool adoption. When developers know that package suggestions route through security evaluation, they can accept AI recommendations with appropriate confidence. The firewall handles security verification automatically, removing the burden of manual package vetting from development workflows.

This creates a sustainable balance between velocity and governance. AI assistants continue accelerating development by suggesting relevant packages. The dependency firewall ensures those suggestions meet organizational security standards before integration. Development teams focus on building features while platform teams maintain supply chain integrity through policy rather than through manual review queues.

Organizations implementing dependency firewalls report faster incident response when supply chain compromises occur. Instead of searching codebases for usage of a newly compromised package, firewall logs immediately identify which projects requested it and whether the request was approved. Remediation becomes targeted rather than organization-wide.

The investment in dependency firewall infrastructure pays forward as AI coding tools become more capable. Future assistants generating entire microservices will introduce even more dependencies even faster. The control point established now scales to handle that acceleration without requiring fundamental workflow changes or security architecture redesign.

If AI is accelerating how your teams write code, Harness Artifact Registry helps ensure the dependencies entering that code are governed before they reach builds, pipelines, or production. The registry boundary is where supply chain security starts.

Key takeaway: The Harness MCP Server now connects directly inside Google Antigravity. Developers can link Harness in under two minutes and give the agent structured, real-time access to their pipelines, execution history, services, environments, and policies, without leaving the editor. What makes it reliable isn't the connection itself. It's the Harness Software Delivery Knowledge Graph underneath, which gives the agent the context to act accurately, fast, and within your guardrails.

‍

AI has made the inner loop faster than ever. Inside Antigravity, you can write, refactor, and test code in seconds. But the moment a change needs to be built, deployed, or debugged in production, you leave the editor entirely, back to juggling pipelines, approvals, scan results, and failed runs across a half-dozen browser tabs. That gap between fast code and slow delivery is the part AI hasn't fixed yet.

The Harness MCP integration closes that gap. Connect once, and Antigravity gains direct access to your Harness delivery environment. The agent now understands your delivery system the same way it understands your codebase. So you can ask it to list pipelines, explain a failure, or trigger a deployment, and it acts on live Harness context instead of generic knowledge.

Connect Antigravity to Harness in a Few Steps

There's no YAML to write and no manual server config. You generate a Personal Access Token in Harness, open Antigravity's Customizations panel, add the Harness MCP server, and paste the token. That's the entire setup.

1. Generate a PAT. In Harness, go to Account Settings → Personal Access Tokens and scope the token to the org, project, and pipelines you want the agent to reach.
2. Open Customizations. In Antigravity, go to Settings → Customizations to configure default behaviors, skills, and MCP server connections.
3. Add the Harness MCP server. Click + MCP Servers, search Harness, select it, and paste your PAT.
4. Start building. The agent now operates with your Harness account, org, and project context. Describe what you need and it acts on real pipeline and execution data.

Settings → Customizations → Add MCP Servers, search “Harness,” connect, done.

All The Software Delivery Use Cases Within Antigravity

Once Harness is connected, you interact with your delivery system the same way you interact with your code, in plain language, from the same window.

Ask it what's in your delivery environment

Start simple: "Can you list pipelines in <Your Project> project in the default org in my Harness account?" The agent resolves the project identifier, pages through every pipeline, and returns a structured report (names, identifiers, creation times, descriptions, and tags) with links straight back to the Harness UI.

‍

Drill into execution history

From there, ask about recent activity: "List out my recent executions in this project." The agent reads the execution history, converts raw timestamps and durations into something readable, and lays out every run, including the one that came back ApprovalRejected, so you can see exactly what happened and when.

‍

Triggering a deployment, with a human in the loop

This is where most "AI in delivery" stories get nervous, and where the design matters most. When you ask the agent to run something, it doesn't just act. It shows you the exact tool it wants to call and the arguments it will send, then waits for your approval.

‍

Approve it, and the run goes through. The agent triggers the pipeline using the Harness run action and returns the live execution (pipeline ID, status, trigger type, and a link to the execution in Harness), so you can follow it from the same chat.

‍

This Is Not AI Without Guardrails

The natural question, once an agent can trigger pipelines, what stops it from doing something it shouldn't? The same controls that govern everything else in Harness.

Why Context Beats Raw API Access

MCP lets a model call external tools by reading API descriptions and deciding which to invoke. That flexibility is useful, but when an agent needs to reason across an entire delivery lifecycle (CI, CD, security scans, approvals, environments, cost signals), raw API access creates a reliability problem. The agent has to discover which endpoints exist, call them in the right order, paginate correctly, and infer how fields relate across systems. Every inferred join is a place to guess. Guessing is where hallucinations happen.

The Harness Software Delivery Knowledge Graph removes the guesswork. It's a purpose-built model of everything that happens after code is written (builds, test runs, deployments, approvals, scans, environment states, feature flags, infrastructure changes, cost signals, and rollbacks) represented as a connected, typed, semantically annotated graph. Every field carries metadata telling the agent how to use it, and relationships between entities are explicitly declared, not inferred.

This is the difference between an agent that can access your delivery system and one that understands it.

When Antigravity connects to Harness via MCP, it isn't handed a list of endpoints. It gets a structured model of your delivery organization, where relationships are known, data types are enforced, and the agent can construct precise queries rather than guessing at field semantics. The same controls apply structurally, too: an approval gate isn't an optional step the agent might skip; it's a typed relationship with state. The agent can't promote past a gate that hasn't cleared, because the graph reflects that clearly. Speed and governance aren't a tradeoff; they coexist by design.

Software Delivery Context At Your Fingertips

If you're already a Harness customer, you're a couple of minutes away from having the software delivery control in Antigravity. New to Harness? Sign up for free and connect from day one. For enterprise onboarding and design-partner access, contact your Harness account team.

The Harness connection gives the agent the ability to act in your delivery system. The Knowledge Graph gives it the understanding to act well. Together, that's what reliable AI in software delivery actually looks like, now available wherever you build, including inside Antigravity.

‍

When a CI pipeline runs on cloud infrastructure, the build machine is ephemeral. It spins up, executes your build, and disappears. During that window, you have zero visibility into how much CPU and memory your pipeline actually consumes.

This blind spot creates real problems. Teams over-provision VMs "just in case," wasting compute spend. Others under-provision and deal with silent OOM-kills or CPU throttling — the only clue being a cryptic exit code 137. Without historical resource profiles, there's no data-driven way to right-size pipelines or catch regressions introduced by dependency upgrades.

We built CPU and Memory Insights to solve this. It gives you real-time and historical visibility into resource consumption during every Harness CI Cloud build — with zero configuration and zero impact on build performance.

Why Resource Visibility Matters

Consider a typical scenario: your build takes 12 minutes on a Large machine (4 vCPU, 8GB RAM). Is it CPU-bound during compilation? Memory-bound during docker build? Or is it I/O-bound pulling dependencies? Without metrics, you're guessing.

With CPU and Memory Insights, you can:

• Right-size your machines — see that a "Large" build peaks at 30% CPU and safely downgrade to "Medium," cutting your cloud spend.
• Debug failures faster — watch the memory ramp leading to an OOM kill and pinpoint which step caused it.
• Detect regressions — compare P90 CPU across builds to catch when a dependency update made things worse.

How It Works

The system collects resource metrics from inside the ephemeral VM, streams them in real-time to the Harness platform, and renders interactive charts in the execution view.

Architecture

Harness CI Cloud uses a multi-layered architecture for pipeline execution. The metrics flow is overlaid on the same path used for build orchestration:

The key insight: lite-engine is the only component running inside the VM — it's the only one with access to actual resource utilization. But it has no persistent storage. Everything must be streamed out before the VM is destroyed.

Data Collection

When a VM is provisioned for your build, lite-engine starts a background process that samples system metrics every second:

• CPU utilization — aggregate percentage across all cores
• Memory usage — total and available, in GB
• Disk I/O — read and write throughput in bytes/sec

Each sample is written as a single JSON line (NDJSON format) to the Harness Log Service using a dedicated stream key. This is the same battle-tested infrastructure that powers step-level log streaming — we reuse its real-time SSE transport, blob storage, and access control. No new infrastructure needed.

Real-Time Streaming

The metrics stream opens during VM setup and closes during VM destroy, giving continuous coverage regardless of how many steps run or fail in between. The stream is independent of step execution — there are no gaps between steps.

During execution, the UI connects via Server-Sent Events (SSE) to receive metrics as they're collected. For completed builds, the same data is available from blob storage. The UI handles both transparently — same visualization whether you're watching a live build or reviewing a historical one.

Summary Statistics

When the VM is destroyed, lite-engine computes a final summary before closing the stream:

• Peak CPU — maximum utilization observed
• Average CPU — mean utilization across the entire stage
• P90 CPU — 90th percentile utilization (useful for right-sizing decisions)
• Total Disk I/O — cumulative bytes read and written

The frontend also computes P50, P90, P95, and P99 percentiles client-side, which means you get full statistics even for in-progress executions.

What You See in the UI

Click the resource indicator button in the execution view (it shows your platform and size, e.g., "Linux (Large)"). A drawer opens with three charts:

CPU Usage

An area chart showing utilization percentage over time, with a P90 reference line. The stats bar shows total cores, peak utilization, average, and percentiles (P50/P90/P95/P99).

Memory Usage

An area chart with dual Y-axes: percentage on the left, GB on the right. Helps you understand both relative and absolute consumption at a glance.

Disk I/O

A line chart showing read and write throughput in MB/s. Useful for identifying I/O-bound steps like image pulls or large file operations.

A stage selector dropdown at the top lets you switch between stages in multi-stage pipelines.

Cross-Platform Support

CPU and Memory Insights works across all Harness Cloud infrastructure:

layer normalizes platform-specific differences. Whether the underlying OS reports per-core or aggregate CPU, or uses different disk I/O naming conventions, the metrics are always presented consistently: aggregate CPU as a single percentage, memory in GB, and disk throughput as a delta rate.

Performance Impact

Resource collection runs with negligible overhead:

For long-running builds, the frontend intelligently downsamples to 120 data points for chart rendering while preserving visual accuracy — peaks and valleys are maintained using the LTTB (Largest-Triangle-Three-Buckets) algorithm.

Reliability

Builds can end in many ways: graceful completion, timeout, infrastructure failure, or force-kill. We handle all of them:

• Happy path: lite-engine writes the summary and closes the stream on VM destroy.
• Crash path: The platform-level cleanup phase independently closes the metrics stream if lite-engine didn't. This runs regardless of how the VM terminated.

This dual-closure approach ensures metrics data is never orphaned — you always get at least the raw timeline, even if the summary couldn't be computed.

What's Next

We're continuing to invest in resource intelligence for CI builds:

• Step-level attribution — correlating resource spikes with specific pipeline steps to pinpoint exactly which step is expensive.
• Automated right-sizing recommendations — using historical P90 data to suggest optimal machine sizes for your pipelines.
• Resource threshold alerts — notifying you when builds consistently approach memory limits, before they OOM-kill.
• Build-over-build comparison — overlaying metrics from the current build against previous runs to visualize the resource impact of code changes.

Get Started

CPU and Memory Insights is enabled by default for all pipelines running on Harness CI Cloud no setup required.

To explore the feature:

• Open any pipeline execution running on a Harness Cloud machine.
• Click the resource indicator in the stage execution header (for example, Linux (Large)).
• Open the insights drawer to view real-time and historical CPU and memory usage for your build.

No YAML changes. No additional agents. No configuration needed.

Use this visibility to quickly identify resource bottlenecks, right-size your build infrastructure, and improve overall CI efficiency.

Ready to optimize your builds? Try it in your next pipeline run or learn more in the Harness CI documentation.