Cloud spend is up 40% year-over-year. Your CFO wants answers.

So your team does the thing everyone does — pulls up the console, starts hunting. Orphaned resources. That RDS instance nobody's touched since Q2. A dev environment three engineers have forgotten exists, quietly billing $800 a month.

You find some waste. You kill it. You send a report. Everyone breathes again.

Until next quarter.

Here's the problem with that reflex — you're asking the wrong question.

"Where are we wasting money?" sounds responsible. But "where are we leaving savings on the table?" is the one that actually changes your trajectory. You're not overspending. You're under-saving. That distinction changes what you measure, what you automate, and ultimately what your cloud bill looks like twelve months from now.

The Reactive Trap: Why Traditional FinOps Misses the Mark

Most FinOps programs run in a reactive mode. A budget alert fires, an exec asks an uncomfortable question, or a surprise bill lands. Teams scramble, find the obvious stuff, ship a report, return to delivery work. Until next time.

This is cost management as damage control. It finds something — these exercises usually do. But they systematically miss everything that didn't make enough noise to trigger a review.

Reserved Instances are a clear example. A reactive team reviews commitment coverage quarterly, maybe monthly if they're disciplined. A proactive team treats it as a continuous process — analyzing utilization patterns, forecasting demand shifts, adjusting before inefficiency compounds. The gap isn't a few percentage points. Over a year, that difference on compute alone can be 15% savings versus 35%.

Cloud Cost Savings Strategy: Building a Proactive Framework

Proactive cloud cost optimization runs on three things: continuous visibility, automated governance, and team-level accountability.

Continuous visibility means treating cost data like application performance data. You wouldn't wait for a service to degrade before checking latency. Every resource should map to a team, service, and environment. When something spikes, you should know within hours, not when the monthly bill arrives.

Dashboards aren't visibility — operational discipline is. Before a developer provisions a new database, they should see the projected monthly cost. When a team's weekly spend jumps 20%, someone should be asking why before the week ends.

Automated governance goes further than guardrails. Budget limits stop runaway costs, but savings automation optimizes what's already running. Idle resource detection, snapshot lifecycle management, rightsizing that executes after approval — these are what separate optimizing once from optimizing continuously.

Here's the mental shift: governance isn't a constraint on engineering. When cost controls are baked in, teams move faster because they're not second-guessing decisions. When finance knows optimization runs systematically, they stop sending nervous emails.

FinOps Best Practices: Accountability Beyond Finance

The biggest obstacle isn't technology. It's org structure.

Most companies centralize cloud cost management in a FinOps team or finance. That team generates reports and recommendations. Engineering teams receive them, nod, and file them under "next sprint." Sometimes that sprint never comes.

That structure almost guarantees under-saving. The people who can change architecture and resource allocation aren't tracking savings opportunities. The people tracking opportunities don't have enough context to know what's worth doing now versus later.

What works: embed cost accountability at the team level. Platform teams own the infrastructure baseline — commitments, shared services, networking architecture. Service teams own their marginal costs — compute, storage, data transfer. Both groups have budgets, both have optimization targets, both report on them in the same cadence they use for reliability and delivery. Not as cost-cutting pressure. As engineering ownership.

This also surfaces trade-offs a centralized team can't see. A 10% cost increase might be pure waste for one team and completely justified for another. Only the team building the service knows which is which.

Maximize Cloud Savings Through Preventive Optimization

Reactive FinOps asks: what can we clean up? Proactive FinOps asks: what are we about to waste?

Developers over-provision for testing and forget to scale down. A reactive team writes a runbook, sends a Slack reminder, adds it to the wiki. A proactive team changes the deployment template — test environments scale down automatically after hours, resources created without tags hit an approval workflow. The waste never happens because the system prevents it.

Storage lifecycle policies archive cold data before it accumulates. Commitment analysis runs daily, catching utilization shifts before they hurt savings rates. Anomaly detection flags unexpected resource creation within the hour.

The cloud cost governance model shifts from periodic audits to continuous validation. A pull request that increases projected spend past a threshold needs cost justification before merge. A service blowing its cost budget gets the same escalation as a service blowing its error budget. Cost feedback runs as tight as your CI/CD pipeline.

Harness CCM: Engineering Proactive Cloud Cost Optimization

Harness Cloud Cost Management is built around this proactive model. Not just tracking what you spent — identifying what you should be saving and providing automation to capture those savings continuously.

Visibility maps to how engineering teams actually work. Every resource ties to a service, team, and environment through automatic tagging and Kubernetes label support, with a unified view across AWS, Azure, and GCP. Costs break down by workload, not just account or region.

Budget tracking adjusts dynamically. Teams set budgets tied to their delivery roadmap and the platform forecasts against actual deployment patterns. Anomaly detection surfaces cost increases before they compound, with enough context to separate signal from noise.

Governance enforces optimization without blocking delivery. Idle resource detection finds underutilized instances, and policy controls define what happens next — notify, require approval, or automate rightsizing. Commitment optimization analyzes Reserved Instance and Savings Plan coverage on an ongoing basis, not just when someone remembers to run a report.

Recommendations quantify trade-offs rather than just flagging problems. Teams see the projected savings from rightsizing a specific instance, the risk of reducing commitment coverage, the benefit of moving a workload to spot instances. Informed decisions instead of guesses.

Cost data integrates into deployment pipelines and operational dashboards alongside reliability and delivery metrics — part of how engineering works, not a separate initiative.

Moving from Reactive to Proactive FinOps

This isn't a project with an end date. It's a change in how teams think about cloud spend.

Teams with strong cost visibility make better architecture decisions during planning, not during post-mortems. They pick storage tiers, compute types, and networking patterns based on actual trade-offs instead of defaults. Finance gets predictability — when optimization runs continuously, cost trajectories stabilize and budget conversations stop being about explaining overruns. Platform teams get time back from repetitive cost hygiene and focus on architecture and reliability.

The under-saving vs overspending paradigm isn't semantic. It's a different operating model. Every dollar left on the table through missed savings is a dollar not funding new capabilities, better reliability, or faster delivery.

Start by measuring what you're not saving. Check your Reserved Instance coverage. Calculate the gap between actual commitment utilization and where it should be. Find resources running overnight that could be scheduled off. Quantify what's available through rightsizing and what it would actually take to capture it.

The number will probably be uncomfortable. It should be. That's what reactive FinOps actually costs — not what you're wasting, but the savings you never went looking for.

Learn more about Cloud Cost Management, check out Harness implementation docs, and see the CCM roadmap.

Dek: A self-propagating npm worm poisoned hundreds of packages and spread through legitimate publishing workflows. The incident exposes a critical distinction for software supply chain security: build provenance can be valid even when the source entering the build is malicious.

Based on public reporting available as of August 5, 2026, at 10:19 a.m. Central Time. Package and version counts may change as researchers continue investigating.

What happened

On August 4, a self-propagating npm worm spread from packages in the widely used keyv and cacheable ecosystem into packages controlled by multiple unrelated organizations.

SafeDep confirmed 2,234 poisoned versions across 444 package names and reported that the worm reached 12 organizations in less than four hours. Four foundational affected packages—keyv, flat-cache, file-entry-cache, and cacheable-request—collectively receive approximately 1.88 billion downloads per month.

Each malicious release added an npm preinstall hook that executed before installation completed. The payload searched developer machines and CI/CD environments for GitHub, npm, cloud, HashiCorp Vault, Kubernetes, database, and other credentials. It could also inspect GitHub Actions runner memory and use stolen npm publishing access to modify and republish additional packages.

Researchers also identified persistence mechanisms involving VS Code and Claude Code project configuration. Opening an affected repository in those tools could trigger malicious code independently of a normal package installation.

One of the most important response details is counterintuitive: the malware installed a monitor that could execute an attacker-controlled command when a stolen GitHub token stopped working. Teams therefore need to locate and remove that persistence mechanism before revoking or rotating credentials.

Why this matters

ChainDrop did not rely only on unsigned artifacts or an obviously unauthorized release process.

Some malicious packages were published through legitimate GitHub Actions and npm OpenID Connect trusted-publishing workflows. Those releases retained valid SLSA provenance because the authorized build system had, in fact, produced them.

The provenance was not technically false. It accurately identified the workflow that built and published the artifact. The failure occurred earlier: malicious source or workflow inputs had already entered the trusted process. As Socket summarized, provenance attested to build integrity, not source integrity.

That distinction matters for engineering and security leaders. Signing and attestation are important controls, but they cannot independently answer several essential questions:

• Was the source change reviewed and approved?
• Did the release correspond to an expected commit, pull request, and tag?
• Were workflow files or project configuration modified unexpectedly?
• Should this dependency be allowed to execute an install script?
• Did the build behave consistently with the package’s intended purpose?
• What identities and secrets could the build runner access?

A delivery system can be cryptographically verifiable and still faithfully build malware.

The business consequences extend beyond one poisoned dependency. Credentials taken from a developer workstation or runner can create access to source repositories, registries, cloud environments, clusters, secret stores, and downstream release workflows. What starts as a dependency incident can quickly become an identity compromise, a release-integrity problem, and an organization-wide response effort.

How teams can reduce the risk of similar incidents

Teams should treat provenance as one layer in a broader evidence chain rather than as a standalone trust decision. Provenance confirms how an artifact was produced, not whether the underlying source code is trustworthy.

Govern source and workflow changes. Protect branches, require independent review for release-related code, and apply additional scrutiny to package manifests, lifecycle scripts, CI workflows, editor tasks, and AI-agent configuration. A valid signature should not override an unexpected or unreviewed change.

Correlate releases with source events. Before permitting publication or promotion, verify that an artifact maps to an approved commit, pull request, tag, workflow version, and expected build identity. Unexpected patch releases or artifacts without corresponding source activity should be investigated.

Restrict install-time execution. Disable npm lifecycle scripts in CI where they are unnecessary. Where scripts are required, isolate their execution, monitor network and process activity, and explicitly allow only known behavior.

Use ephemeral, least-privilege identities. Prefer short-lived credentials with narrowly scoped permissions. A build that only needs to read dependencies should not have standing access to publish packages, modify repositories, enumerate cloud secrets, or reach production infrastructure.

Isolate runners. Use disposable build environments, minimize credentials present in memory, restrict outbound connectivity, and rebuild runners after suspected exposure. Long-lived, broadly privileged runners increase both the value of stolen credentials and the worm’s opportunity to persist.

Gate artifact publication and promotion. Evaluate more than signature status. Policies should consider source review, dependency risk, suspicious new scripts, unusual package behavior, scanner findings, and the relationship between the source repository and published artifact.

For this incident specifically, organizations should first hunt for the reported token-monitoring persistence, editor hooks, package IOCs, and unexpected releases. Credential revocation and rotation should then occur from a known-clean environment, followed by rebuilding affected systems and downstream artifacts.

How Harness could help

Harness positions software supply chain security as a combination of dependency governance, security testing, policy enforcement, artifact controls, and delivery evidence not merely the presence of an attestation.

Harness Supply Chain Security supports assessing software supply chain posture, governing open source dependencies using software bills of materials, and managing artifact promotion with SLSA attestations. Harness messaging also emphasizes workflows for rapidly responding to vulnerable or compromised components.

The broader Harness platform can enforce organizational policies through Open Policy Agent, apply security gates within delivery pipelines, use fine-grained access controls, and maintain audit trails. Harness Artifact Registry is designed to centralize artifacts and dependencies while enforcing quality and security standards.

Applied to an incident like ChainDrop, those capabilities could help teams:

• govern which open source dependencies and package versions are permitted
• enforce a minimum package-age policy to delay adoption of newly published dependency versions
• require security and policy checks before publishing or promoting artifacts
• correlate attestations with approved delivery processes
• limit problematic packages through automated policy enforcement
• identify where affected components are used
• preserve audit evidence for investigation and remediation

These controls would not make provenance infallible or guarantee prevention. They could, however, strengthen the surrounding guardrails, reduce exposure, improve visibility, and accelerate remediation when trust signals conflict.

The bigger trend

ChainDrop reflects three converging trends.

First, software supply chain attacks increasingly target identities rather than only source code. npm tokens, GitHub credentials, cloud identities, and CI/CD permissions let attackers move through the same trusted paths maintainers use.

Second, developer environments now include more executable surfaces. Package lifecycle hooks, IDE tasks, automation files, and AI coding-agent configuration can all become persistence or propagation mechanisms.

Third, security decisions increasingly require combined evidence. Provenance, code review, dependency intelligence, workflow history, policy results, and runtime behavior each answer different questions. None is sufficient by itself.

Harness take

ChainDrop shows that “signed” and “safe” are not synonyms. Modern software delivery needs to establish continuity of trust from the proposed source change through the workflow, artifact, promotion decision, and runtime—not simply verify the identity of the pipeline that produced the final package.

Closing

ChainDrop does not make provenance unimportant. It clarifies its proper role.

Attestations remain valuable evidence about how software was produced, but trustworthy delivery requires controls before and after the build: reviewed source, governed workflows, constrained identities, policy-gated artifacts, isolated execution, and evidence from runtime behavior.

Harness helps organizations connect these controls across the software delivery lifecycle so that a valid signature becomes one input to a risk decision—not the entire decision.

Key takeaways

• SafeDep confirmed 444 affected npm package names and 2,234 poisoned versions as of its August 4 analysis.
• The worm executed during package installation and targeted developer, CI/CD, cloud, registry, Vault, and Kubernetes credentials.
• Legitimate OIDC publishing workflows produced artifacts with valid SLSA provenance after malicious inputs entered the pipeline.
• Teams should remove the reported token-revocation persistence before rotating credentials.
• Provenance should be combined with source governance, policy gates, dependency controls, least-privilege identities, and runtime evidence.

FAQ

What is the ChainDrop npm worm?

ChainDrop is the name used for a self-propagating npm supply chain campaign that inserts credential-stealing malware into packages and uses compromised publishing access to infect additional packages.

How can malware have valid SLSA provenance?

Provenance verifies information about the build process and identity that produced an artifact. When an authorized workflow builds already-compromised source, the resulting attestation can be valid even though the artifact is malicious.

What should teams do if they installed an affected package?

Treat the relevant developer machine or build runner as potentially compromised. Hunt for the published persistence mechanisms and indicators first, then revoke and rotate exposed credentials from a clean environment and rebuild affected systems and artifacts.

Manual intake eats the first 10 to 30 minutes of an incident. Harness AI SRE runs it as automation and gives those minutes back to your responders.

The first minutes go to setup

When an alert fires, most teams spend 10 to 30 minutes on coordination before anyone looks at the problem. The pattern repeats on every incident:

• An alert arrives from PagerDuty, Datadog, or another monitor
• Someone opens a Jira or ServiceNow ticket
• Someone spins up a Slack channel and pulls in responders
• Someone starts a Zoom bridge and pastes the link
• Someone hunts down the runbook or the last incident's notes

None of that work changes between incidents, and the delay compounds on every high-severity page.

Runbooks turn intake into a triggered workflow

In AI SRE, a runbook is an automated workflow that runs when a trigger fires. You build it in a visual workflow editor instead of writing glue code, with more than 50 pre-built integrations for its action steps. When the trigger condition is met, the runbook executes its steps in order and records each one.
A single intake runbook can:

• Create a ticket in Jira or ServiceNow
• Open a Slack channel and invite the right responders
• Start a Zoom bridge and post the join link to the channel
• Set severity, category, and impacted-service fields on the incident
• Write every action to the incident timeline as it happens

Setup that used to take 15 minutes runs in seconds, and the timeline leaves an ordered record for the post-incident review.

What sets the trigger off

A runbook fires on any of three signals:

• An inbound webhook from a monitoring or alerting tool. On the intake side, AI SRE ships ingestion templates for more than 20 alert sources, including Datadog, PagerDuty, BigPanda, Grafana, New Relic, Sentry, and Opsgenie, plus a generic template for any tool that posts JSON.
• Manual creation of an incident or alert, or an on-demand run of the runbook.
• A change to an existing incident, such as severity, status, or impacted services.

You scope which alerts launch a runbook with conditions. A trigger can filter on fields, tags, or thresholds through a rule builder or a CEL expression, so a Sev3 CPU blip and a Sev1 outage take different paths. Inside the runbook, a conditional step branches further: when severity is critical, page the secondary on-call team; otherwise the standard logging path runs.

The difference: response sits on your change data

Most incident tools begin with no context. They register that an alert fired and little else.
Platform teams already produce the answer. You build artifacts, deploy them through pipelines, and merge pull requests, and that record is the first place to look when something breaks. AI SRE reads it directly through the Deploy Change Investigator, which connects three streams:

• Build events carrying artifact version, commit SHA, and repository
• Deploy events carrying the services, environment, and versions that shipped
• The pull requests merged into the deployed branch

Once those are linked, the platform can name the deploy that was live when the incident started and diff the pull requests between the last healthy release and the current one. An AI agent reasons over that change history alongside recent feature-flag changes and related alerts. It then surfaces the deploys, PRs, and flags most likely behind the incident, each with a confidence level. It answers the first question in any incident, what changed, from data you already generate.

Runbook actions can act on that answer, too. A step can trigger a Harness pipeline directly (for example, to roll back the suspect deployment), so the same workflow that opened the incident can start the fix.

How to set one up

1. Build the runbook. Lay out the intake steps in the visual editor and pick the integration each step uses.
2. Add triggers. Point the runbook at the webhook sources and incident events that should launch it, then add conditions so only the right alerts qualify.
3. Add actions. Define what runs on trigger: create the ticket, open the channel, start the bridge, set the fields, page the on-call. Every action lands on the timeline.

Start with intake, then keep going

Automating intake removes the fixed coordination cost from the start of every incident and gets responders to investigation in seconds. Because AI SRE sits on the build, deploy, and change data your pipelines already produce, it does more than open the channel faster. It hands the team a first read on what changed.
Human-in-the-loop stays the default. Automation clears the setup and surfaces the likely cause; people decide and drive the resolution.

See how Harness AI SRE automates incident intake or build your first runbook today.

FAQs

What is a runbook in incident response?

A runbook is an automated workflow that runs when a trigger fires. Instead of writing custom glue code, you build it in a visual workflow editor on top of pre-built integrations, and it executes a defined set of steps in order: creating tickets, opening Slack channels, starting bridges, and more, every time it's triggered.

What triggers an incident runbook to run?

A runbook fires on one of three signals: an inbound webhook from a monitoring tool like Datadog or PagerDuty, manual creation of an incident or an on-demand run, or a change to an existing incident such as a severity or status update. Conditions let you scope which alerts actually launch the runbook, so low-severity blips and major outages can follow different paths.

How does AI SRE reduce incident response time?

AI SRE automates the coordination work that normally eats the first 10 to 30 minutes of an incident (ticket creation, Slack channel setup, bridge links, and field tagging) and runs it in seconds instead. It also connects build, deploy, and pull request data through the Deploy Change Investigator, so responders get a first read on likely causes instead of starting from zero.

What is the Deploy Change Investigator?

It's the part of AI SRE that links three data streams: build events, deploy events, and merged pull requests, so the platform can identify which deploy was live when an incident started and diff the changes since the last healthy release. An AI agent reasons over that history, along with feature-flag changes and related alerts, to surface likely causes with a confidence level.

Which monitoring tools integrate with AI SRE?

AI SRE ships ingestion templates for more than 20 monitoring and alerting sources, including Datadog, PagerDuty, BigPanda, Grafana, New Relic, Sentry, and Opsgenie, plus a generic template for any tool that posts JSON.

Can a runbook trigger a rollback automatically?

A runbook step can trigger a Harness pipeline directly. For example, to roll back a suspect deployment so the same workflow that opens the incident can also kick off the fix. That said, human-in-the-loop remains the default: automation gets responders to the starting line and surfaces likely causes, but people make the call and drive resolution.

Does automating incident intake replace human responders?

No. Automation handles the repeatable coordination work (tickets, channels, bridges, timelines), so responders can start investigating immediately. Decisions and resolution still sit with people; AI SRE's job is to remove setup time, not judgment.

Quick question: can you envision how Terraform scalability issues you when your third team tries to apply changes at the same time, locks the state file, and someone's Friday evening turns into a debugging session? Maybe that hits a little too close to home. The first 500 resources feel manageable. The next 2,000 teach you what "workspace limitations" actually mean in production.

When infrastructure code grows past a certain threshold, the tools and workflows that worked for small teams stop working. State files balloon into multi-megabyte JSON blobs. Plan times stretch from seconds to minutes. What used to be a quick `terraform apply` becomes a risky operation requiring coordination across time zones. The infrastructure as code bottleneck isn't hypothetical. It's the moment your team starts avoiding changes because the process has become unpredictable.

Why Terraform Scalability Issues Emerge

Most scalability problems stem from decisions made when the codebase was small. A single monolithic state file works fine for 10 resources. For a thousand, it creates a serialization chokepoint. Every change requires locking the entire state, even if you're only modifying a security group in one region.

Remote state backends help, but they introduce their own failure modes. S3-backed state with DynamoDB locking works until someone's network hiccups mid-apply and leaves a stale lock. Cleaning up requires manual intervention and tribal knowledge about which lock IDs matter. At scale, these incidents compound. What was once an edge case becomes Tuesday.

Module sprawl accelerates the problem. Teams copy-paste working configurations, tweaking variables for each environment. Over time, you end up with dozens of nearly identical modules that diverged in subtle ways. When a breaking change hits a core provider, you're patching every fork individually. Version drift becomes a compliance risk.

State Management at Scale Breaks Assumptions

Terraform state management at scale exposes the implicit contract between your code and reality. State files aren't just metadata. They're the single source of truth about which resources exist, their current configuration, and the dependency graph connecting them. When that file grows past a certain size, read performance degrades. Plan operations slow down because Terraform has to parse and validate every resource, even if you're only touching one.

Splitting state helps, but it shifts complexity. You trade a monolith for a distributed system of loosely coupled state files. Now you need a mental model of which workspace controls which resources. Documentation becomes critical because the implicit coupling between workspaces isn't enforced by the tooling. A misconfigured variable in one workspace can cascade into another through shared data sources or implicit dependencies.

Remote backends introduce latency. If your state is in S3 and your CI runners are in another region, every read adds round-trip time. Multiply that by hundreds of resources and the cumulative effect is noticeable. Teams start optimizing for fewer plan runs, which ironically makes each run riskier because more changes accumulate between tests.

Collaboration Friction Increases with Team Size

When five engineers share a single workspace, coordination is informal. Someone shouts in Slack before running an apply. By the time you have fifteen people across three continents, informal processes collapse. The infrastructure code management problem becomes an access control and workflow problem simultaneously.

Without proper guardrails, junior engineers can accidentally destroy production resources. The blast radius of a bad apply is proportional to the size of the state file. In a monolithic setup, one mistake can ripple across hundreds of resources. Policy enforcement becomes necessary, but Terraform's native tooling offers limited help. You need something outside the core workflow to validate plans before they execute.

Terraform workspace limitations compound this. Workspaces were designed for environment separation, not collaborative workflows. They don't enforce approval gates, policy checks, or audit trails. If you need those capabilities, you're building them yourself or adopting a platform that provides them natively.

When Standard Tooling Stops Scaling

The breaking point varies by team, but the symptoms are consistent. Plan times stretch past acceptable thresholds. State lock contention increases. Engineers start working around the tooling instead of with it. Someone proposes splitting the monorepo into per-team repositories, which solves the immediate pain but fragments governance.

This is the moment teams evaluate Terraform alternatives or consider an OpenTofu migration. OpenTofu offers a community-governed fork with a stable open-source foundation, which matters if licensing uncertainty is a concern. But switching engines doesn't solve workflow problems. A poorly structured codebase remains poorly structured regardless of which binary executes it.

The real question isn't which IaC engine to use. It's whether you're managing infrastructure code or merely running it. If your workflow is `git clone,` `terraform init,` `terraform plan,` and `terraform apply` in a loop, you're missing the orchestration layer that enables teams to scale safely.

How an IaC Management Platform Addresses Scale

An IaC management platform like Harness IaCM treats infrastructure code as a first-class workflow problem, not just a tooling problem. It provides the orchestration, governance, and collaboration features that become non-negotiable past a certain scale.

Harness IaCM supports OpenTofu, Terraform, and Terragrunt, which means you're not locked into a single engine. The choice of execution runtime becomes an implementation detail. What matters is the management layer: default pipelines for plan and apply operations, workspace templates that enforce consistency, and variable sets that reduce configuration drift.

State management moves from a manual coordination problem to a platform-managed concern. Harness handles backend configuration, workspace isolation, and access control through policy. Drift detection runs continuously, flagging resources that diverged from code without requiring manual audits. When someone needs to troubleshoot, they're working from a unified interface with full audit history, not SSH'ing into CI runners and reading logs.

The Module and Provider Registry centralizes reusable components. Instead of maintaining dozens of forked modules, teams publish versioned modules to a shared registry. Consumers reference specific versions, which prevents breaking changes from propagating unexpectedly. Dependency management becomes explicit and trackable.

Policy enforcement integrates directly into the workflow. You can block applies that violate security policies, fail compliance checks, or exceed cost thresholds. These gates trigger before any changes reach production, which shifts risk left in the development cycle. Engineers get immediate feedback instead of discovering issues during post-deployment audits.

The Harness IaCM product page details the full feature set, and the IaCM documentation walks through real implementation patterns. If you're curious about where the platform is headed, the roadmap shows upcoming capabilities around enhanced governance and multi-cloud orchestration.

Moving Beyond the Bottleneck

The infrastructure as code bottleneck isn't about Terraform's limitations, but the gap between what basic tooling provides and what production-scale operations require. Small teams can manage with scripts and conventions. At scale, you need structured workflows, enforced policies, and visibility into who changed what and why.

Guardrails don't slow teams down. They enable speed by reducing the cognitive load of coordinating changes across distributed systems. When engineers trust that the platform will catch obvious mistakes before they reach production, they ship with confidence. The alternative is a culture of fear where every apply feels like defusing a bomb.

If your team is feeling the pain of Terraform scalability issues, the solution isn't to abandon infrastructure as code. It's to adopt tooling that matches the complexity of what you're building. 

Start by auditing your current workflow: Where do engineers waste time? What manual steps could be automated? Which failure modes recur most often? The answers will point toward the specific capabilities that matter most for your context.

Modern IaC governance and compliance aren't optional at scale. They're the difference between infrastructure that evolves safely and infrastructure that becomes unmaintainable. The teams that figure this out early gain a compounding advantage. Those who wait pay an escalating tax on every change.

At Harness, we build an AI-powered software delivery platform, and test result data is core to how we help engineering teams ship faster. The table that stores it started small: one row per record, all the context right there on the row. Simple, readable, and it worked. Until it didn't.

This is the story of how we refactored it, what we learned, and what I'd tell you to watch for in your own systems.

What We're Working With

Within Harness Continuous Integration, we built a backend service, the Test Intelligence service (TI Service), that powers three critical features:

• Test Intelligence: Automatically selects only the tests that need to run based on code changes.
• Test Management: Tracks test trends, flakiness patterns, and health metrics.
• Flaky Test Detection: Identifies unreliable tests that pass and fail inconsistently.

These features help engineering teams ship faster by reducing test execution time and improving test reliability - but they only work if we can process and analyze test results at a massive scale.

Every time a CI pipeline runs, it produces test results in standard formats like JUnit XML: which tests ran, which passed or failed, how long each took, and any output they produced. Each report belongs to a build, each build belongs to a pipeline, each pipeline belongs to a project, and so on up to the account level. A busy organization can produce thousands of builds per day, with reports ranging from a handful of records to tens of thousands.

Why the Original Design Made Sense

When you're building a new product with evolving requirements, simplicity wins. The first version of our report table used a flat, denormalized approach: one row per test result, with all the context stored as text strings directly on each row. Every level of the hierarchy lived right there on the row. If you've worked with document databases, this pattern looks familiar. It's essentially how you'd model a collection in a NoSQL store: every record is self-contained, carrying all the context it needs.

In the early days, before you know exactly what queries you'll need to support, this approach has real advantages. Inserts are dead simple: one row, all the data, done. Reads don't need joins. The schema is easy to reason about because it is the data. When you have thousands of rows, queries are fast, and the duplication barely matters. This is a perfectly valid design until the data grows. And ours did.

The Cracks

As the data scaled to millions of rows, the flat design started working against us. For a pipeline running 10,000 tests, every row carried the same hierarchical scope strings; that's 10,000 copies of identical context. Here's what started breaking:

• Every summary was a full scan. Want a count of passed vs. failed for a build? Scan every row, aggregate on the fly. No pre-computed totals anywhere.
• String comparisons on everything. Queries filtered on text columns across millions of rows. No integer keys, no index-friendly joins. Structural identifiers like account, project, and pipeline names were stored as repeated strings on every row instead of integer references. String comparisons take more CPU cycles and are less cache-friendly than integer comparisons - at millions of rows, queries were roughly 10x slower than they needed to be.
• API latency grew with the payload. The write handler inserted one row per test result inline in the HTTP request. A report with 50,000 results meant 50,000 INSERT statements before the API could respond.
• Massive duplication. About 93% of the storage on each row was repeated scope strings - the same values, over and over.
• Hot and cold data mixed together. Test output (stdout, stderr - sometimes very large) lived alongside lightweight metadata. Queries that only needed a count still had to wade past blob-sized columns.

These are smells. Individually, they're manageable. Together, at scale, they compound into something that's hard to patch.

But here's the thing: we only saw these clearly because we stress-tested. Before we call something "production-ready" at Harness, we load-test every API and processing path, pushing each to its limits. Not just typical load, but burst traffic - what happens when a thousand pipelines finish at once? What happens when a single report has 100,000 test results instead of 100? This approach comes from experience. We've seen systems fall over under real-world load that never appeared in testing. So we don't guess. We measure. We break things in test environments so they don't break in production. Load testing tells you exactly where your ceilings are and which to raise first.

The Principles We Followed

Before jumping into a rewrite, we defined the principles that would guide every decision. These apply to any system that ingests high-volume data behind an API.

1. Keep API calls bounded. Never do O(N) work in the request path. Our old handler's latency and memory usage scaled with the number of records - processing each test meant keeping all that data in memory during the request. When multiple large reports came in concurrently, the service would crash with out-of-memory errors. Even when it didn't crash, slow-running requests would block others, causing API calls to hang and eventually timeout after minutes. The fix: make the API do a fixed number of operations and return immediately, whether the report has 100 records or 100,000. Store the input, return fast, process later.
2. Offload variable work to workers. Everything proportional to data size runs in a background worker pool, decoupled from the API via a queue. Producer and consumer scale independently.
3. Bound memory explicitly. Every component gets a hard cap. Process in fixed-size chunks and release between chunks. If memory usage is proportional to input size, you will eventually run out.
4. Stream, don't buffer. Decode one record at a time. Our old path deserialized the full payload, then re-serialized it for storage- holding the data in memory twice. The new path streams directly from the HTTP body to compressed storage with roughly 32 KB of fixed overhead.
5. Pre-aggregate at write time. A counter increment during a write is negligible. A full scan during a read is not. We maintain running totals directly on each execution row - summary queries become a single indexed lookup.
6. Use integer foreign keys over string matching. Normalize once, join cheaply forever. A 4-byte integer comparison is orders of magnitude faster than comparing variable-length strings, and storage per row dropped from roughly 400 bytes to about 28.
7. Design for horizontal scaling. Vertical scaling has hard limits. We designed for horizontal: stateless workers, distributed coordination via locks with TTL, and work that can be split across instances. When load increases, add instances, not resources.

What Changed - The Big Picture

The architectural shift boils down to one idea: separate the "accept" from the "process."

The Data Model Shift

The flat table with repeated strings became a set of normalized tables joined by integer foreign keys.

Key Moves on Writes

• Normalized schema: Small tables joined by integer foreign keys, replacing one wide table with repeated strings.
• Delta processing: Each upload is processed as a delta and merged incrementally—no double-counting, concurrent-safe.
• Auto-scaling worker pool: Workers scale up with load and down after idle. Distributed locks prevent duplicate processing.

Key Moves on Reads

• Pre-aggregated summaries: Counters maintained at write time mean summary queries hit a single row, no scanning.
• Hybrid tiered reads: Small reports are read from compressed blobs and streamed through memory one record at a time. Large reports are stored as columnar files and queried by an analytical engine that only reads the columns needed—a count query never touches output columns.
• Memory-bounded pagination: Instead of loading a full report to paginate, we use a two-pass approach: a lightweight first pass collects just enough metadata to filter and sort, then a second pass reads full data only for the requested page. Memory stays bounded regardless of report size.

Performance Improvements

What to Watch For in Your Own Systems

If you're running a service that ingests detail data and serves aggregated views, here's a quick checklist:

• Are your summary queries scanning all detail rows? If getting a count requires touching every record, you need pre-aggregation.
• Is your API latency proportional to input size? If bigger payloads mean slower responses, you're doing too much work inline.
• Are you repeating the same strings on every row? That's a normalization opportunity—and the savings compound fast.
• Are you buffering entire payloads into memory? Stream when you can. Your memory profile should be constant, not proportional.
• Do your workers scale with demand? If it takes minutes to ramp up, you're leaving throughput on the table during burst load.

These patterns aren't unique to the report data. They show up anywhere you have high-cardinality detail tables behind an API - logs, events, metrics, audit trails.

And one more thing: this isn't the last refactor. At 10x or 100x the current scale, new bottlenecks will surface in different places, and the solutions will look different. That's fine. The goal is to define the requirements you need to support right now, find the right way there—even if that means a refactor—and leave room to evolve. No design is forever. Solve today's problem well, and grow from there.

‍

--- A more technical deep dive ---

‍

How It Actually Happened

Working with AI

We used AI extensively throughout this refactor, not just for writing code, but as a design partner. The process involved many back-and-forth iterations: propose a solution, challenge it with edge cases, refine, and repeat. The key difference from working solo was that we pushed every proposed design to handle 100% of cases, not just the 80% we might have settled for without that collaboration.

We went through 3-4 different refactoring designs before landing on the final approach. Each iteration surfaced assumptions that didn't hold or trade-offs we hadn't considered. The AI helped us explore those alternatives more thoroughly than we would have on our own. That said, AI didn't eliminate the hard parts. Testing remained a challenge, so we planned comprehensive unit and integration tests up front before starting implementation.

Design Philosophy

Our main goal was to preserve API contracts - same inputs, same outputs - while completely changing how the internals worked. We also made a conscious decision that API response time should be bounded and predictable. This led to two key strategies:

1. If the work can be done asynchronously, push it to a worker. The API stores the input and returns immediately. Background workers handle the variable processing.
2. If the work needs to be done synchronously, reconsider the data format. Sometimes we keep data in formats that are easy to query ad hoc but expensive to compute on every request. If you're aggregating the same raw data on every API call, you're doing it wrong. Pre-process it asynchronously and store a ready-to-serve report instead.

We also chose to process reports incrementally. Reports arrive over time - sometimes in chunks from parallel test runners, sometimes from retries. Instead of waiting for everything to arrive before processing, we merge incrementally.

Key Technical Choices

• Hybrid storage: Small reports as compressed blobs directly in the database and large reports as columnar files in object storage.
• In-process analytical engine: We chose an embedded analytical engine (DuckDB) instead of an external database because it eliminated a network hop and let us read columnar files directly from signed object storage URLs.
• Incremental aggregation and merge: Process each upload as a delta. When a new step report arrives or updates, we compute just that step's contribution and merge it into the running totals. This keeps memory bounded.
• Uniform summary storage: We store summaries for builds and steps in the same table, using a special marker value to distinguish them, keeping the query layer simple.

What We Learned from Scale Testing

We ran the system through heavy load testing - pushing services to their limits - and several issues surfaced that we wouldn't have caught otherwise.

Database insert performance degraded with table size. 

Tables that receive heavy writes slow down as they grow, especially if they have indexes and foreign keys. Every insert validates constraints, updates indexes, and writes to the database's write-ahead log (a sequential record of all changes for crash recovery). At high concurrency, workers contend for locks on the index and the write-ahead log, resulting in reduced throughput. Our solution was a staging pattern: workers insert into a separate staging table with no indexes or foreign key constraints. They fire and forget. A single background worker periodically batch-processes rows from the staging table into the main table.

ID lookups for new objects became a bottleneck. The insert-and-query pattern in the same transaction was extremely heavy on the database. We switched to a query-first pattern: first, select all existing IDs in a batch. Then insert only the objects that are missing. Finally, query again to get IDs only for the newly inserted rows.

Worker auto-scaling was too slow. During a traffic burst, it took almost 6 minutes to ramp from minimum to maximum workers. We changed the algorithm to calculate how many workers were actually needed (based on queue depth and current capacity) and spin them all up at once. Scaling became immediate.

Migration Strategy

Using feature flags, we controlled the rollout per customer, directing data to both the old and new systems during the transition. The strategy was simple: redirect writes first, then reads. This prevented data loss. Once we had confidence in the new system's correctness, we shifted read traffic over.

Closing Thought

Refactors aren't set in stone. You can refactor the refactor if needed, which we did during this exercise. Don't be afraid of it, but do it when it's actually needed - when the system tells you it's time, not because the architecture feels imperfect. In some cases, it's needed. This was one of them. Learn more about Harness CI.

‍

Most teams think infrastructure problems start during provisioning. They're wrong. Infrastructure breaks after deployment—in what the industry calls "day 2 operations."

You provision an EC2 instance. It works. Then someone makes a console change at 2 a.m. during an incident. Or configuration drifts between what Terraform declares and what's actually running. Or Ansible doesn't see the infrastructure dependencies it's trying to configure. And suddenly, your infrastructure is in an undocumented state that nobody fully understands.

The core problem is simple: most organizations run multiple tools—Terraform, Ansible, CI/CD platforms—each managing infrastructure in isolation. There's no single system thinking about correctness end-to-end. State is defined once but never reconciled. Control is assumed at every layer but enforced nowhere.

This is where infrastructure control planes become essential.

The Day 2 Operations Problem: When Infrastructure Diverges from Reality

Here's what happens in most organizations. Day one looks great: infrastructure gets provisioned, policies are in place, everything deploys successfully. Day two is where it falls apart.

First, console changes creep in. A firefighter needs to test something quickly or fix an incident at 3 a.m., so they click a change directly into AWS or GCP instead of going through the pipeline. That's the spark. But it's not the real problem.

The real problem is that nothing in your tool chain was watching for it in the first place. Your Terraform management tool has no idea what happened. Your CI/CD platform sees no change. Your Ansible configuration assumes a specific state that's no longer true. The systems operate in silos.

Then comes the second wave: incidents occur, you make changes to fix them, but you don't remediate those changes back into your Infrastructure as Code. So the same incident pops back up weeks later. Your infrastructure state file doesn't reflect reality. Cost balloons unexpectedly. Compliance checks fail because nobody knows what's actually deployed.

This is infrastructure drift, and it's the symptom of a broken control model.

The Broken Model: Control Assumed, Never Enforced

Think about the infrastructure lifecycle in most organizations:

Coding — You write Infrastructure as Code using Terraform, OpenTofu, or CloudFormation.

Provisioning — Terraform applies your code, but it's isolated. It has no awareness of what happens next.

Configuration — Ansible configures the infrastructure, but it has no awareness of infrastructure dependencies or what Terraform did.

Deployment — CI/CD deploys your application, often blind to the infrastructure configuration happening in parallel.

Governance — Policy tools like Open Policy Agent or Sentinel scan each step independently, but nothing ties policies across the entire workflow.

The pattern is consistent: state is defined but never enforced across the system end-to-end.

Most teams assume control exists at each layer. Security assumes policies are enforced. DevOps assumes CI/CD has visibility into infrastructure. Platform teams assume nothing will diverge from the declared state. But assumption is not enforcement. When things break—and they will—there's no unified system to catch it.

The Solution: A Control Plane That Continuously Reconciles State

A control plane sits across provisioning, configuration, and deployment. It does two critical things: it continuously reconciles desired state against actual state, and it enforces policy before anything is allowed to move to the next stage.

Concretely, this means:

Continuous Reconciliation — Drift is caught in hours or minutes, not at the next quarterly review. If someone changes an EC2 instance size in the console, your system detects it immediately and alerts you or automatically remediates it.

Policy Enforcement at Design Time — Policies aren't a side process security runs sporadically. They're baked into every stage. If someone tries to provision an oversized instance that violates cost policy, the system blocks it before Terraform plan even runs—like a linter catching a syntax error.

Unified Ownership and Context — When an infrastructure change moves through provisioning, configuration, and deployment, ownership and context are preserved. If you define RBAC at the provisioning stage, that same RBAC applies at the configuration and deployment stages. There's no handoff where context is lost.

Only Compliant Infrastructure Reaches Production — This is the line that resonates with security and compliance leaders. Non-compliant infrastructure literally cannot reach production. It's binary, enforceable, and auditable.

From Silos to Unified Automation: The Practical Impact

Here's what this looks like in practice. You're using Terraform to provision infrastructure, Ansible to configure it, and Jenkins or GitHub Actions to deploy applications. Normally, these tools never talk to each other. You manually coordinate changes across them.

With a unified infrastructure control plane, everything happens in one place. You provision with Terraform, configure with Ansible, and deploy with CI/CD—all in a single orchestrated pipeline. Policies run at every stage. Cost estimation happens before approval. Security scanning is mandatory. Drift is detected automatically.

The result: one platform team can realistically stand behind hundreds of policies and review every change in real time, instead of a handful of reviewers trying to catch everything that slips through.

Where AI Agents Accelerate the Model

This is where Harness AI becomes powerful. Harness brings together context across your code, infrastructure, configurations, deployments, policies, and the broader software delivery lifecycle giving AI a complete understanding of how your systems operate.

Harness AI already includes specialized platform agents and AI-powered capabilities like DevOps Agent and AppSec Agent that help teams reduce manual effort, make better decisions, and automate complex software delivery workflows. With more AI agents and capabilities coming, Harness is continuing to expand what teams can safely delegate to AI.

What makes Harness AI different for infrastructure is its deep, unified context. It does not operate from isolated prompts, schemas, or point-in-time tool queries. It understands the relationships between code, infrastructure, deployments, policies, dependencies, and organizational standards.

That context enables Harness AI to deliver results that are more reliable, explainable, and governed, helping engineering teams move faster and remediate infrastructure without sacrificing control.

The Path Forward: Treat Infrastructure Like Product

The north star most platform and engineering leaders are chasing is simple to state, hard to do: treat infrastructure as code like a product.

This means infrastructure goes through the same golden path pipeline as application code. Policy checks. Cost estimation. Approvals. Audit trails. Full visibility. Not a side process, but the same process.

When infrastructure follows this model, day 2 operations stop being a crisis. Drift is caught and remediated automatically. Costs are controlled before they spike. Compliance is enforced, not aspirational. And teams can move faster because the governance is built in, not bolted on.

The infrastructure that breaks after deployment isn't a technical problem—it's a control problem. Fix the control model, and everything else follows.

Ready to fix your infrastructure control model? The teams that treat infrastructure as a governed product, not a manual process, are the ones that scale safely. Learn how to unify Terraform, Ansible, and CI/CD governance in one control plane.

FORM SUBMISSION THANK YOU MESSAGE

Thank You Message Copy:

"Thanks for downloading! You've taken the first step toward infrastructure that's governed, not fragile. Check your email for the complete guide plus templates for building your own unified infrastructure pipeline. Bonus: See how AI agents can automate drift remediation and cost control while your team focuses on innovation. Questions? Our infrastructure experts are ready to help—reach out anytime."

Alternative (Shorter):

"Got it! Your guide is on the way. While you're reading, explore how a unified control plane catches infrastructure drift in minutes, not months. Watch our live demo showing policy enforcement, cost estimation, and automated remediation in action. Or schedule a walkthrough with our team to see how this works for your infrastructure."

Here's the uncomfortable truth about the Mythos era: knowing about a vulnerability and being able to neutralize it are two entirely different problems.

AI models like Mythos are finding vulnerabilities 10x faster than humans ever could. Project Glasswing participants discovered over 10,000 high and critical vulnerabilities in their applications. Firefox alone had 271 previously unknown zero-days exposed by Mythos. That's the good news.

The bad news? Most organizations aren't unprepared from a security standpoint. They're unprepared from an engineering standpoint. The bottleneck that matters isn't discovery—it's everything that comes after. When you're drowning in vulnerabilities you can't prioritize, remediate, and deploy fixes for, finding more vulnerabilities doesn't reduce risk. It just makes you more aware of how exposed you are.

This is where the real challenge lies: closing the gap between detection and deployment has become the defining challenge of the AI era.

The Discovery Isn't the Problem – Response Is

For years, the security industry has focused on finding vulnerabilities faster. Better SAST tools. More sophisticated scanning. AI-powered analysis. And it worked—SAST tools have been mainstream for two decades. Most organizations have multiple scanners running.

But here's what actually happens when you turn on a SAST tool: you instantly have thousands of vulnerabilities in your backlog. Then what?

You start with CVSS ratings, filtering for critical. You triage, asking which are real, which matter, who owns the code. You assign tickets to developers. They may or may not look at them. They may or may not know how to fix them. You may have to provide security training. Meanwhile, the vulnerability sits in a queue.

Once a developer actually fixes it in code, your security team thinks it's solved. But it's not. If this is a critical application with high DevOps maturity, the fix might deploy in hours. But for most organizations with average DevOps maturity or non-critical applications? That fix takes days or weeks to reach production.

So the real timeline isn't measured in hours. It's measured in days, weeks, sometimes months from discovery to production deployment.

Now introduce Mythos or another frontier LLM scanner. You're finding 10x the vulnerabilities. But you haven't hired 10x more security staff. Your developers haven't multiplied. Your CI/CD pipeline hasn't suddenly gotten faster. Everything after discovery remains the same. You've just made the backlog catastrophically larger.

This is the core problem of the Mythos era: AI accelerated discovery, but it didn't accelerate your ability to respond to that discovery.

The Real Framework: Security at Machine Speed

If machines are finding vulnerabilities at machine speed, your security program must respond at machine speed. This requires rethinking your entire vulnerability response lifecycle across five stages:

1. Understand Exposure — The moment a zero-day drops, you need to know immediately if you're affected and where. This means comprehensive software composition analysis (SBOM) and code analysis across all your applications. When Log4J hit, organizations without visibility took weeks to understand their exposure. With proper instrumentation, it should take hours or minutes.

2. Prioritize Vulnerabilities — Not all vulnerabilities are equal, but CVE scoring doesn't reflect your actual risk. You need exploitability analysis (EPSS), reachability analysis (is the vulnerable code path actually called?), and AI reasoning to cut through noise. Organizations report 90% noise reduction using these techniques.

3. Make Remediation Easier — AI-generated fixes are table stakes now. But the real acceleration comes from auto-generated pull requests with validated fixes. Developers don't need to understand the vulnerability or how to fix it—they just review the PR and click accept.

4. Protect Production Immediately — Don't wait for code fixes to deploy. Use virtual patching in your WAF to block exploitation while you fix the underlying issue. This applies every day, every year, not just during zero-day crises.

5. Verify and Prove Remediation — Automated audit trails prove you've taken action. This is critical for compliance, incident response, and forensics.

A global financial services institution implemented this framework. They had over 1,000 microservices and performed bi-weekly security patch cycles. Before automation, each patch took five days per application and 25 hours of engineering time. After implementing security response automation, patch time dropped to under two hours with nearly zero human effort. That's a 98% improvement. More importantly, if a zero-day dropped, they could respond almost instantly instead of scrambling for days.

The Dual Nature of the Problem: Security + Engineering

This is where most security vendors miss the mark. They focus entirely on the left side of the problem: finding vulnerabilities faster. But the real bottleneck isn't on the security side, it's on the engineering side.

This is both a security problem and an engineering problem. Your security team needs better visibility and prioritization. Your engineering team needs faster, easier ways to consume security findings, build fixes, and deploy them. Your DevOps team needs automated patching and virtual patching capabilities integrated into CI/CD.

The winning organizations treating this as a cross-functional problem. Security initiates, but engineering owns execution. The best customer example came from the engineering side of the house—they wanted faster deployment velocity and realized security response was the constraint. When engineering and security align around speed, that's when real progress happens.

What's Next: AI Agents Close the Loop

The next frontier is AI agents orchestrating the entire response. Imagine this flow:

• A zero-day agent monitors threat intelligence feeds and automatically flags new vulnerabilities in your dashboard when they're disclosed
• A triage agent automatically prioritizes vulnerabilities, applying EPSS, reachability analysis, and false positive filtering
• A remediation agent generates pull requests with validated fixes that developers can merge directly
• A virtual patch agent immediately creates and deploys WAF rules to block exploitation

This isn't science fiction. Organizations are running this today. The key is that each agent has full context—knowledge of your code, infrastructure, configurations, and deployment pipelines. Context is what makes AI agents deterministic and reliable instead of hallucinating random suggestions.

The Window Is Closing

The timeline for action is compressed. Frontier LLM capabilities like Mythos are currently held back—expensive, restricted access, re-released under pressure. But that won't last. Within months to a year, similar capabilities will become generally available. When that happens, attackers will have access to the same tools.

That's when your response speed becomes a competitive advantage. Not weeks, not days. Hours. Minutes. Seconds in some cases.

The organizations that move now—that build security response automation, that integrate security into their CI/CD pipelines, that treat this as an engineering problem alongside a security problem—will be positioned to respond. Everyone else will be scrambling.

The Mythos era isn't about panicking over a flood of zero-days. It's about fundamentally rethinking how fast your organization can respond to any vulnerability, anytime, anywhere. That's security at machine speed. That's what survival looks like.

Ready to close the gap between detection and deployment? Security response speed is your new competitive advantage. Learn how to build a vulnerability response framework that matches the pace of AI discovery.

FORM SUBMISSION THANK YOU MESSAGE

Thank You Message Copy:

"Thanks for downloading! You've taken the first step toward vulnerability response at machine speed. Check your email for the complete Mythos readiness assessment (just 11 questions) plus our vulnerability remediation playbook. See how a global FSI reduced patch cycles from 5 days to 2 hours with security response automation. Your security team can do the same. Questions? Our experts are ready to help you design your response framework."

Alternative (Shorter):

"Got it! Your assessment is on the way. While you wait, explore how security response automation closes the gap between zero-day disclosure and production deployment. Watch our live demo showing AI-powered triage, auto-remediation, and virtual patching in action. Or schedule a walkthrough with our team to see your vulnerability response velocity transformed."

Q2 | May – July 2026‍

Companion post: [Q2 26 CD & GitOps update →]

‍

Q2 2026 advances Harness Pipeline with DAG-based execution, smarter looping strategies, enhanced governance tooling, and a wave of improvements in reliability and developer experience. The quarter's headline feature — DAG pipelines — fundamentally changes how you model complex multi-stage workflows by letting stages declare explicit dependencies rather than relying on sequential order. Combined with improved cron scheduling, template label management, and a new pipeline dry-run API, Q2 gives teams greater control over how their pipelines are structured and validated. See the [Q1 2026 Pipeline update] for prior context.

‍

---

‍

Pipeline Execution

‍

DAG Pipelines (Beta)

‍

Harness now supports Directed Acyclic Graph (DAG) pipelines, enabling you to define explicit dependencies between stages instead of relying on sequential or parallel stage order. Each stage declares the stages it must complete before it starts via the dependsOn field, so independent execution paths progress as soon as their dependencies finish — without waiting for unrelated stages to complete. This unlocks execution patterns that are common in CI but historically difficult to model in CD: fan-out/fan-in topologies, diamond-shaped dependency graphs, and multi-path workflows where different service groups deploy in parallel but converge on a shared integration step.

Learn more about DAG pipelines →

‍

Pipeline Chaining with Looping Strategies

‍

Pipeline chaining now supports looping strategies when a parent pipeline calls a child pipeline, so you can run a child pipeline multiple times across a matrix or repeat configuration. Previously, chaining was limited to a single child pipeline invocation per stage. Teams that deploy to multiple environments or regions by parameterizing a shared child pipeline can now drive those invocations directly from the parent's looping strategy rather than duplicating stages.

‍Learn more about pipeline chaining →

‍

Matrix Exclusions Evaluated at Loop Start

‍

Matrix looping strategies now evaluate and exclude conditions at the beginning of the loop, removing excluded combinations before iterations start, rather than skipping them mid-execution. This change eliminates "skipped" iterations from cluttering your execution view and makes matrix execution counts predictable — the number of running iterations you see matches the number that will actually do work. 

Learn more about matrix looping strategies →

‍

Barrier Cycle Detection

‍

Harness now detects barrier reference cycles and enforces unique barrier references within a stage or step group, preventing circular barrier dependencies that could deadlock a pipeline. Pipelines with cycle configurations will fail validation at save time rather than silently deadlocking at runtime — a much faster feedback loop for teams using barriers to synchronize parallel deployments.

Learn more about barriers →

‍

Cron Triggers with AND Semantics

‍

Cron triggers now support AND semantics for complex scheduling where both a date range and a day-of-week condition must be satisfied simultaneously. For example, 0 3 16-22 * 1 executes at 3:00 AM UTC only on Mondays between the 16th and 22nd of the month. This enables precise scheduling for monthly maintenance windows, release cycles, and compliance-driven deployment schedules without scripting workarounds. 

Learn more about cron triggers →

‍

Governance & Policy

‍

Harness AI for OPA Policies

‍

OPA policy authoring now uses the unified Harness AI agent for enhanced, context-aware policy generation. Powered by specialized skills trained on OPA and REGO best practices, the AI assistant helps you write policies without deep REGO knowledge and generates plain-language descriptions of existing policies. Teams can iterate on governance rules faster, with less reliance on REGO specialists.

Learn more about building policies with Harness AI →

‍

Policy as Code: 6-Month Evaluation Data Retention

‍

Harness now retains Policy as Code evaluation data for 6 months by default. If your account has a pipeline data retention policy that exceeds 6 months, your OPA evaluations retention automatically aligns with the longer window. You can download evaluation data for compliance purposes or request a custom retention period. This closes a gap for compliance teams that needed evaluation history for audit trails beyond short-term retention windows. 

Learn more about policy evaluations retention →

‍

Policy as Code: Cloud Storage for Evaluation Inputs

‍

Large OPA evaluation inputs (pipeline YAML, Terraform plans, IaCM stacks) are now stored in cloud storage (GCS/S3) rather than the database, optimizing database performance for accounts with high evaluation volumes. Evaluation results and metadata remain in the database for fast access. A new signed URL API endpoint (GET /api/v1/evaluations/evaluation-input-signed-url/{id}) lets you retrieve input data from cloud storage when needed. 

Learn more about how evaluation data is stored →

‍

Executions Management Page

‍

A new Executions Management page gives you account-level visibility into all queued and running pipeline executions. You can see queue positions, monitor execution status across organizations and projects, and abort individual or bulk executions from Account Settings > Security and Governance > Executions Management. This is the operational view platform teams have needed for managing execution backlogs and diagnosing concurrency issues without diving into individual project views.

Learn more about Executions Management →

‍

Templates & Input Sets

‍

Template Overrides

‍

Templates now support configurable overrides, letting template owners designate specific advanced settings that callers can override without editing or duplicating the template. The template owner marks settings as overridable in the template's allowedOverrides list — including conditional execution, failure strategy, looping strategy, delegate selectors, and policy enforcement — and each caller that references the template can supply its own value for any allowed setting via templateOverrides in YAML. Any setting not listed in allowedOverrides remains locked to the template's value. A common use case: a stage template used across 50 pipelines sets a default conditional execution rule, and a single pipeline that needs different behavior overrides just that one setting instead of maintaining a duplicate template. Supported for step, stage, step group, and pipeline templates. 

Learn more about template overrides →

‍

Template Labels

‍

Harness now supports template labels for referencing template versions using semantic names instead of fixed version numbers. Labels let you create stable, human-readable pointers like stable, latest, or team-approved that you update as your template evolves, so pipelines that reference a label automatically pick up new template versions without requiring each pipeline to be updated. 

Learn more about template labels →

‍

Pipeline Dry Run Validation API

‍

A new dry run validation API endpoint lets you validate pipeline YAML changes while editing files in Git before committing them to your repository. The endpoint performs YAML schema validation, template expansion, and OPA policy evaluation without executing the pipeline — providing the same validation feedback you get from the Harness UI editor to Git-native workflows and CI checks. Available at POST /pipeline/api/v1/orgs/{org}/projects/{project}/dry-run. 

Learn more about pipeline YAML validation →

‍

Tag Filtering with AND/OR Logic

‍

Pipeline execution filtering by tags now supports AND/OR logic, enabling Matches Any (OR) or Matches All (AND) filtering. Teams using tags to label executions by environment, team, or release train can now run more expressive queries across their execution history without filtering results manually after fetching.

‍

Git Experience

‍

Git Tags for Pipeline and Input Set Sources

‍

Pipeline YAML and input sets stored in Git can now be referenced using Git tags in addition to branch names during trigger-based executions. Use the $tag:<tag-name> format in the Pipeline Reference Branch or Input Set Source fields (for example, $tag:v1.0.0), or resolve tag names dynamically from trigger expressions. This enables version-controlled pipeline configurations that align with your release workflow — triggering on a Git tag runs the pipeline and input set version that matches that tag, making rollbacks to prior configurations as simple as re-running against the original tag. Requires delegate version 26.04.89002 or later. Learn more about Git tag support →

‍

Git Experience Monitoring & Observability

‍

A new Webhooks monitoring section gives you full visibility into the health of your Git Experience. The Events tab shows the complete history of webhook events processed by Harness, with per-event payloads and troubleshooting details for synchronization failures. The new Observability tab goes further — it surfaces repository synchronization health across all Git-backed repositories (showing webhook coverage and sync status at a glance) and tracks Git provider API rate-limit consumption per connector, so you can identify which connectors are approaching rate limits before they cause sync failures. 

Learn more about monitoring Git Experience →

‍

Input Set Caching

‍

The Harness Git cache now extends to remote input sets, joining remote pipelines and templates. Input sets stored in Git are cached and served from the cache in the UI, reducing load times when opening or editing input sets for large repositories. A new refresh-and-get API endpoint lets you programmatically clear and refresh the cache for a specific branch, returning the refreshed entity in a single call — the API equivalent of the Reload from Git UI action. 

Learn more about Git caching →

‍

Git-Based Pipeline YAML in Dynamic Stages

‍

Harness now supports Git-based pipeline YAMLs in Dynamic Stages, allowing you to execute pipeline YAMLs stored in Git repositories in addition to inline and runtime-provided YAML. You can optionally specify a commit hash to pin execution to a specific version of the file — useful for audit trails and reproducible pipeline runs.

‍

Notifications & Observability

‍

Multi-Line String Variables

‍

String variables at pipeline and stage scope now support a multi-line input mode. With this feature enabled, any variable marked as multi-line expands to a resizable text area wherever users are prompted for that variable's value — in the run pipeline dialog, in input sets, and in the pipeline editor. This removes the friction of editing long structured values like YAML configuration blocks, lists of IP addresses, or shell script snippets through a single-line input. Learn more about variables →

‍

"Waiting for User Action" Pipeline Notification Event

‍

Harness now fires a "Waiting for User Action" notification event whenever a pipeline pauses for user input— such as Approval steps, Manual Interventions, or runtime input requests. You can configure these notifications at both the pipeline level and through centralized notification rules, enabling on-call routing, Slack alerts, or webhook integrations that fire precisely when a human action is blocking a deployment. 

Learn more about pipeline notifications →

‍

What's Next

‍

Q3 FY27 will continue expanding pipeline execution models, deepen template and Git Experience integrations, and bring further improvements to the governance and observability layer. Explore the Harness Developer Hub for full documentation.

‍

‍

Q2 | May – July 2026

Welcome back to the quarterly update series! If you've been following along, Q1 2026 brought AI-powered continuous verification, expanded deployment platform support, and GitOps workflow enhancements. Q2 builds on those foundations with progressive Kubernetes canary rollouts, native AI agent deployments, sharper verification controls, and a wave of GitOps improvements — from one-click application rollback to finer-grained RBAC — that make self-service operations a reality for platform teams.

‍

Deployments

Kubernetes

Progressive Canary for Kubernetes

Harness now supports progressive canary deployments as a dedicated subtype of the Kubernetes Canary strategy. Instead of a single canary phase, you can define percentage-based rollout stages — for example, 25%, 50%, and 100% — with verification or approval gates between each phase. Harness maintains two Deployments and shifts replicas between them, keeping the total pod count within a fixed budget throughout the rollout. This gives platform teams a repeatable, observable path for shipping changes with automated quality gates at each step rather than a binary canary-or-full-traffic decision. Learn more about Kubernetes progressive canary deployments →

Kubernetes Blue-Green: Scale-Up on Rollback

When a Kubernetes Blue-Green deployment fails and triggers a rollback, Harness now automatically scales up the previous (stable) deployment after the services are swapped back. Previously, the stable deployment could be sitting at zero replicas during the rollout, meaning traffic routed to it immediately after the service swap would hit unscaled pods. The automatic scale-up eliminates this gap, ensuring traffic reaches active pods as soon as the rollback completes. Learn more about Blue-Green Scale Up on rollback →

Kubernetes Dry Run: Additional kubectl Flags

The Kubernetes Dry Run step now accepts additional kubectl flags — including --server-side and --force-conflicts — so the dry run validation command matches the actual kubectl arguments used at deploy time. Teams using server-side apply can now ensure that their dry-run output accurately reflects what will happen at deployment, making approval gates and policy checks more reliable. Learn more about the Kubernetes Dry Run step →

Istio Traffic Routing: AND/OR Match Logic

Istio traffic routing steps now support configurable AND/OR match logic for route rules. The new Match all rules option lets you require that every configured rule (URI, headers, method, port) match before a request is routed, rather than routing on any single match. This gives teams with complex traffic shaping requirements finer control over which requests reach canary or stage deployments. Learn more about the Traffic Shifting step →

Kubernetes Connector: Client Credentials OIDC Grant

The Kubernetes cluster connector now supports the client credentials OIDC grant type for machine-to-machine cluster access. In addition to the existing password grant, you can now authenticate with a client ID and client secret alone — suitable for scenarios such as AKS clusters fronted by Microsoft Entra ID, where a human user credential is neither available nor appropriate. Existing connectors are unaffected.

‍

AWS

Bulk Post-Production Rollback

When a bad release reaches multiple infrastructure targets simultaneously, you no longer need to roll back each one individually. From the service dashboard, you can now select multiple infrastructures where a service is deployed and roll them back together — choosing the target execution for each infrastructure before confirming the rollbacks. This compresses what was previously a tedious per-environment operation into a single coordinated action. Learn more about rollback deployments →

ASG MixedInstancesPolicy Support

AWS Auto Scaling Group deployments now support MixedInstancesPolicy, enabling the use of spot instances and automatic fallback under capacity constraints. You can configure the policy in the ASG configuration JSON to allow AWS to select from multiple instance types, and Harness automatically detects and updates the launch template version within the MixedInstancesPolicy during deployments. This opens cost optimization options for teams running non-critical workloads on spot capacity without sacrificing deployment automation. Learn more about MixedInstancesPolicy →

AWS CDK on ECS Delegates

AWS CDK steps can now run on ECS-based delegates, removing the requirement for a Kubernetes delegate runtime to execute CDK synth and deploy steps. Teams running hybrid or ECS-only delegate fleets can now use CDK provisioning without having to stand up a dedicated Kubernetes delegate. Learn more about AWS CDK on ECS delegates →

Smart CDK Diff: Skip Deploy When No Changes

Harness now supports automatic skipping of AWS CDK Deploy steps when a preceding CDK Diff step detects no infrastructure changes. This prevents pipelines from executing a no-op CDK deploy when only the application code has changed, but the infrastructure definitions have not, reducing end-to-end pipeline runtime. Learn more →

ECS Rolling: Skip Application Auto Scaling

ECS Rolling deployments now support a Skip application auto scaling option that tells Harness to bypass all AWS Application Auto Scaling API calls for that deployment. This is designed for teams that manage auto-scaling externally (outside of Harness service definitions), and for scenarios where many ECS services are deployed in parallel, and the volume of scaling API calls risks hitting AWS rate limits. Learn more →

OIDC Environment Session Tags for AWS

AWS connectors with OIDC authentication now include environment identifiers as session tags in OIDC tokens. This lets you enforce environment-specific IAM policies — for example, restricting access to production secrets to pipelines running in production environments — while sharing a single delegate pool across environments. You no longer need separate connectors or delegates to achieve environment-level IAM isolation. Learn more about OIDC environment-based session tags →

‍

Google Cloud

MIG Blue-Green: Pre-Traffic Staging Deployment

Google Managed Instance Group (MIG) Blue-Green deployments now expose a staging deployment step before the traffic shift so that you can test and validate the new MIG version — including running pre-traffic checks and approval gates — before any live traffic is shifted to it. This closes a gap for teams that need to exercise the new MIG at scale before committing to the shift. Learn more about MIG Blue-Green deployments →

OCI Helm Charts from Google Artifact Registry

You can now use OCI-based Helm charts stored in Google Artifact Registry (GAR) as a manifest source for Helm deployments, expanding the set of OCI Helm registries Harness supports natively alongside ECR, Docker Hub, and others. Learn more about Helm chart cloud providers →

‍

AI Agent Deployments

Harness now natively supports deploying AI agent workloads as a dedicated deployment type in Harness CD — the first CI/CD platform to offer first-class pipeline automation for AI agent services.

This release adds out-of-the-box support for two agent runtimes: **AWS Agent Core** and **Google Agent Runtime**. You can model an AI agent service in Harness the same way you would any other service — define the artifact, configure the infrastructure, and attach a deployment pipeline. From there, all the same pipeline primitives apply: approval gates before a new agent version goes live, canary phases to validate agent behavior under partial traffic, automatic rollback when a verification step fails, and OPA policy checks before any agent deployment reaches production.

For platform teams managing a growing portfolio of AI agent workloads, this means agent deployments stop being one-off automation scripts and become first-class, auditable, policy-governed releases alongside the rest of your software. Explore AI Agent Deployments →

Traditional (SSH / WinRM)

Shell Script Step: Harness ID Tokens

The Shell Script step now supports Harness ID tokens via named identities. You declare one or more named identities on the step, and Harness generates an independent OIDC ID token for each, injecting it into the script as an environment variable at runtime. Scripts can then authenticate as the workload itself rather than through a connector, enabling fine-grained, per-script identity for sensitive automation tasks. Learn more about Harness ID tokens for the Shell Script step →

Preserve Directory Structure in Command Step Copy

The Copy command in the Command step now includes a Preserve Directory Structure option that maintains the original directory hierarchy when copying config files to target hosts. This prevents silent data loss from overwriting when multiple config files share the same name but reside in different subdirectories — a common pattern in multi-environment or multi-region SSH/WinRM deployments. Learn more about preserving directory structure →

‍

Connectors

Artifactory OIDC Authentication

The Artifactory connector now supports OIDC authentication, enabling credential-free federated authentication with JFrog Artifactory using short-lived JWT tokens. This eliminates the need to store static Artifactory credentials as Harness secrets and integrates Artifactory into the same keyless authentication model already available for AWS, GCP, and Azure connectors. Learn more about Artifactory OIDC authentication →

‍

Approvals

Approval Steps Inside Container Step Groups

Container Step Groups run all steps inside a single Kubernetes pod, which previously meant any pipeline that paused for human approval risked hitting the pod's 24-hour TTL before the approver responded. Harness now supports Harness Approval steps natively within Container Step Groups, and the pod's TTL automatically extends to match the stage timeout if it exceeds 24 hours. This removes the ceiling on containerized deployment workflows that span multiple time zones, require cross-team sign-off, or sit between canary and full-traffic phases. Note that only Harness Approval steps are supported inside Container Step Groups — Jira, ServiceNow, and Custom Approval types are not. Learn more →

Approval Visibility for Non-Approvers

Approval steps now support configurable visibility for non-approvers, letting users without approval permissions view step details during pipeline execution while keeping approval actions disabled. This is particularly useful for on-call engineers who need to monitor the approval status of an active deployment without approval authority. Configured at the project level under Default Settings. Learn more about approval visibility →

‍

Email Step

Email Step: Continue Send

The Email step now supports a non-blocking send mode, allowing pipeline execution to continue immediately after the email is dispatched, without waiting for delivery confirmation from the mail server. This removes an unnecessary blocking dependency for notification workflows where timely pipeline progress matters more than confirmed delivery. Learn more →

Verification

Git Experience for Monitored Services

Harness now supports Git Experience for monitored services, enabling version control and code review workflows for verification configurations. You can store monitored service configurations in your Git repository alongside the application code they verify, treat changes to health sources and SLOs as reviewable pull requests, and roll back verification configurations with standard Git operations. Learn more about Git Experience for monitored services →

AI Verify: New Configurable Properties

The AI Verify (v1) step now supports two additional **configurable properties** beyond the existing `deploymentStartTime`:

• dataCollectionDuration — Override the default data collection interval. Accepts a positive integer (number of minutes, maximum 60).
• customVerificationStartTime — Shift the verification window start to an earlier timestamp so that data from your health source already recorded between the deployment start and the Verify step execution is included in the analysis. Useful when:
  
  • Your observability tool (such as Dynatrace) begins recording immediately at service start
  • You have additional steps between your deployment step and the Verify step
  • Deployment and verification run in separate pipeline stages

Learn more about AI Verify configurable properties →

Per-Metric Sensitivity Override in AI Verify

The AI Verify step now supports per-metric sensitivity overrides, allowing you to set different sensitivity levels for individual metrics rather than applying a single threshold across the entire verification run. This means you can configure strict sensitivity for latency metrics and relaxed sensitivity for noisy metrics like memory usage — within the same Verify step — rather than tuning the global threshold to a compromise value. Learn more about per-metric sensitivity override →

CloudWatch: Full Metrics Insights Query Support

The CloudWatch health source now fully supports the complete CloudWatch Metrics Insights SQL syntax, including WHERE and ORDER BY clauses. You can now filter metrics by dimension values — for example, isolating metrics for a specific ALB target group, EC2 instance, or EKS namespace — directly in the query field. This is particularly useful for Continuous Verification comparisons: configure separate metric definitions with WHERE clauses targeting each group's dimensions to compare control and test traffic during blue/green or canary deployments. Learn more about CloudWatch health sources →

‍

‍

GitOps

Application Rollback

You can now roll back a GitOps application to a previous deployment directly from the Harness UI. A new History & Rollback tab on the application details page shows the full deployment history and lets you select a version to roll back to. Rollback is also available as a dedicated pipeline step, making it composable with approval gates and automated verification in your GitOps pipelines. Learn more about syncing and rolling back GitOps applications →

Applications in Any Namespace

GitOps now supports Applications in Any Namespace. When you configure a cluster-scoped agent, you can specify applicationNamespaces to allow ArgoCD Application resources to be created in namespaces outside the agent's install namespace. This removes a longstanding constraint for multi-tenant clusters where different teams need application isolation without per-team agents. Learn more about applications in any namespace →

Force-Delete Unreachable Applications

You can now force-delete a GitOps application from Harness when the underlying ArgoCD application is unreachable — for example, when the agent or ArgoCD project has been deleted. The Delete from Harness option removes the application record without requiring connectivity to the agent, unblocking cleanup workflows that previously got stuck when infrastructure was partially torn down. Learn more about managing GitOps applications →

Fine-Grained RBAC for Application Resource Actions

GitOps now enforces finer-grained RBAC, separating application-level operations (create, edit, delete) from Kubernetes resource actions (sync, restart, delete pod). This lets you grant operators the ability to restart pods or sync applications without granting them the ability to delete the application definition itself — a meaningful privilege boundary in shared platform environments. Learn more about managing GitOps access →

Squash and Merge in PR Pipelines

GitOps PR pipelines now support squash-and-merge when closing a GitOps pull request. Instead of a standard merge commit, Harness will squash all commits on the PR into a single commit before merging, keeping the release repository history clean and readable. This is especially useful for teams with many incremental commits per deployment that want a one-commit-per-release history in their release repo. Learn more about PR pipelines →

Selective Sync for GitOps Applications

The GitOps Sync step now supports selective sync — letting you choose a specific subset of resources to sync rather than syncing all resources in the application. Useful for large applications where you need to apply a targeted change without touching unrelated resources. Learn more about syncing GitOps applications →

Update Release Repo: Succeed When No Files Changed

The GitOps Update Release Repo (URR) step now supports an option to succeed even when no files are changed in the PR. Previously, the step always failed when there was nothing to commit, which blocked pipelines where the deployment state was already current. Teams using URR in pipelines that may or may not produce a change can now configure the step to treat a no-op commit as a success rather than a failure. Learn more about the Update Release Repo step →

Harness AI for GitOps

Harness AI now supports GitOps entities and pipeline stages. When creating or troubleshooting GitOps Applications or ApplicationSets, the AI can diagnose common setup errors and suggest remediations — including manifest syntax errors, incorrect service or environment types, missing GitOps clusters on linked environments, incomplete manifests, and connectivity issues with Git or infrastructure connectors. Learn more about Harness GitOps →

Zero Trust Service for Agent Communication

The GitOps agent now supports the Zero Trust Service (ZTS) for agent-to-SaaS communication, routing task parameters through the ZTS validation flow. This brings GitOps agent communication in line with the Zero Trust security posture available across other Harness services. Learn more about installing the GitOps agent →

ArgoCD 3.3.10 Bundled in GitOps Agent

The GitOps agent now bundles ArgoCD 3.3.10 (upgraded through the quarter from 3.3.9), incorporating the latest security fixes, stability improvements, and Helm and Kustomize updates from the ArgoCD project. Learn more about the GitOps agent →

‍

Repository Templates Across Scopes

You can now select repository templates from different scopes (account, organization, or project) when configuring GitOps repositories. Teams with shared repository templates defined at the account or org level no longer need to duplicate them at the project level to make them available during repository configuration. Learn more about GitOps repository credentials templates →

---

What's Next

Q3 will continue to expand Kubernetes-native deployment capabilities, streamline management of large-scale GitOps setups, and extend AI-assisted operations across more CD workflows. Explore the Harness Developer Hub for full documentation.

‍

AI has fundamentally changed software development.

Developers are writing more code than ever. AI coding assistants can generate features, tests, documentation, and infrastructure configurations in minutes. Engineering organizations are seeing meaningful productivity gains as AI becomes embedded throughout the software development lifecycle.

But there is a catch.

Security teams now face a difficult reality: application security was already struggling to keep pace with software delivery before AI arrived.

Now the gap is widening.

During a recent discussion on AI SAST, Rennye Shen, Senior Director of Product Marketing at Harness, summarized the challenge clearly:

"If you were struggling before with the pace of software development, you're probably going to break once your development organization adopts these tools en masse."

The problem isn't that AI created new security challenges.

The problem is that AI amplified existing ones.

Why Traditional SAST Is Struggling to Keep Up

Most organizations already run some combination of static application security testing (SAST), software composition analysis (SCA), and additional application security tools.

Yet many security leaders face the same outcomes:

• Thousands of findings
• Growing vulnerability backlogs
• Low remediation rates
• Frustrated developers
• Limited pipeline coverage

The issue isn't visibility.

The issue is action.

Security teams can find vulnerabilities. What they struggle with is helping developers resolve them quickly enough to keep pace with delivery demands.

As AI accelerates development velocity, this challenge becomes harder.

More code means more opportunities for vulnerabilities. Larger commits, faster release cycles, and increased deployment frequency create pressure on security workflows that were already stretched thin.

The Four Reasons SAST Programs Fail

1. Developers Don't Trust Findings

False positives remain one of the biggest obstacles to effective application security. When developers repeatedly investigate findings that turn out to be non-issues, trust erodes quickly. Security teams may view a 30% false-positive rate as acceptable. Developers do not. Once trust is lost, remediation rates decline.

2. Vulnerabilities Don't Get Fixed

Finding vulnerabilities is not the same as fixing them. Most developers are measured on shipping software, not performing security analysis. Without clear remediation guidance, security findings often become another item in an already overloaded backlog.

3. Business Logic Vulnerabilities Are Missed

Traditional SAST solutions excel at pattern matching. They identify known coding mistakes, insecure functions, and common weaknesses. But modern applications increasingly fail in more subtle ways. Business logic vulnerabilities, authorization flaws, and complex data flow issues often evade traditional scanning approaches because they require deeper contextual understanding.

4. Security Cannot Match Development Velocity

Many organizations only scan a fraction of their repositories and pipelines. The reason is simple. Traditional security tooling requires tuning, maintenance, and integration effort. As application portfolios grow, coverage often declines. The result is security visibility that scales more slowly than software delivery itself.

AI SAST Is Not One Technology

One of the most important lessons for buyers is that AI SAST is not a single category. There are two fundamentally different approaches emerging.

LLM-Native Security Testing

This approach uses large language models directly to analyze source code.

Its strengths include:

• Deep reasoning capabilities
• Better understanding of business logic
• Ability to discover novel vulnerabilities
• Rich contextual analysis

However, there are tradeoffs.

LLMs are probabilistic systems.

The same code may produce different results across scans. Hallucinations can occur. Findings may be harder to validate and explain.

AI-Assisted SAST

This model starts with a traditional deterministic scanning engine and applies AI to improve results.

Benefits include:

• Consistent findings
• Repeatable scans
• Easier workflow integration
• Greater auditability

The downside is that many implementations provide incremental improvements rather than fundamentally new detection capabilities.

This creates an important tension:

Reasoning versus reliability.

Three Myths About AI SAST

Myth #1: AI Eliminates False Positives

No security technology eliminates false positives completely. AI can reduce noise significantly. But uncertainty does not disappear. Instead, uncertainty changes form. Traditional scanners may generate false positives. LLM-based systems may introduce hallucinations. Security leaders should be skeptical of any claim that promises perfect accuracy.

Myth #2: AI Will Replace Traditional SAST

The future is unlikely to be either-or. Deterministic scanning remains valuable for CI/CD pipelines, compliance requirements, and auditability. LLM reasoning is valuable for code generation workflows and deeper vulnerability analysis. Most organizations will benefit from combining both approaches.

Myth #3: AI Is Always Correct

AI systems often sound confident. Confidence should never be confused with accuracy. Human validation remains essential, particularly when security decisions impact production systems. Trust and explainability will remain critical requirements for AI-driven security tools.

Five Questions Every Buyer Should Ask

When evaluating AI SAST platforms, focus on outcomes rather than marketing claims.

1. Can it secure AI-generated code?

The most important security shift may occur before code reaches a repository.

Can security testing happen directly within AI coding workflows?

2. Does it actually reduce noise?

Ask vendors for measurable evidence.

How much reduction in false positives do customers experience?

3. Can it detect business logic vulnerabilities?

Modern applications require contextual analysis.

Security tools must move beyond pattern matching.

4. Does it improve remediation?

Detection without remediation provides limited value.

Evaluate how effectively developers can fix identified issues.

5. Are findings explainable?

Security teams, developers, and auditors all need understandable evidence.

Trust requires transparency.

The Emerging Hybrid Security Model

The most practical future is neither fully deterministic nor fully probabilistic.

Instead, platform teams should expect hybrid architectures.

Deterministic engines provide repeatability and governance.

LLM-based systems provide reasoning and contextual understanding.

Together, they address different parts of the software delivery lifecycle.

This mirrors a broader trend across platform engineering.

The goal is not simply to find more problems.

The goal is to operationalize security at the speed of software delivery.

What Platform Teams Should Do Next

Three priorities stand out:

Shift Security Earlier

As AI-generated code becomes more common, security controls should move closer to code creation itself.

Optimize for Remediation

The best security finding is the one that gets fixed. Measure remediation effectiveness, not vulnerability volume.

Prioritize Developer Experience

Security programs succeed when developers trust and adopt them. Reducing friction matters as much as improving detection.

Conclusion

AI is not replacing application security.It is forcing application security to evolve.

Organizations that continue relying solely on traditional SAST approaches will find it increasingly difficult to keep pace with AI-driven software development. The winners will be teams that combine reliable security automation with intelligent AI-assisted workflows that developers actually use.

Ready to see how Harness helps platform teams secure software delivery while maintaining developer velocity? Explore Harness Security Testing Orchestration and AI-powered software delivery solutions.

FAQ

What is AI SAST?

AI SAST combines artificial intelligence with static application security testing to improve vulnerability detection, reduce noise, and accelerate remediation.

How is AI SAST different from traditional SAST?

Traditional SAST primarily relies on deterministic pattern matching. AI SAST adds reasoning, contextual analysis, and automated remediation capabilities.

Can AI SAST eliminate false positives?

No. AI can reduce false positives but cannot eliminate them entirely.

What is LLM-native application security testing?

LLM-native testing uses large language models directly to analyze source code and identify vulnerabilities.

Why is AI-generated code creating security challenges?

AI accelerates code production, increasing development velocity and creating more opportunities for vulnerabilities to enter software pipelines.

Can AI SAST detect business logic flaws?

Some AI-based approaches can analyze complex application behavior and identify business logic vulnerabilities that traditional tools may miss.

Will AI replace traditional application security testing?

Most experts expect hybrid approaches that combine deterministic scanning with AI-powered reasoning.

Everyone talks about moving faster with AI. But here's what nobody mentions: AI amplifies both good and bad practices alike.

You can build code 10x faster with AI agents. But if your testing, operational controls, and failure mode analysis aren't keeping up, you'll deploy bugs 10x faster too. Speed without guardrails is just a faster path to production incidents.

The uncomfortable truth is this: knowing you can use AI agents to build software and actually knowing how to do it safely and maintain operational control are two entirely different problems. Most organizations are optimizing for the first without solving for the second.

This is the defining challenge of the agentic era. How do you gain the speed that AI agents promise without compromising the quality and operational risk you're expected to deliver?

The Four Pillars of Agentic Engineering Excellence

After six months of redesigning their SDLC around AI-driven development, Harness identified four core pillars that enable both speed and safety. These aren't aspirational. They're now standard practice across their engineering organization.

1. Spec-Based Development: Make Context Explicit

Traditional product requirements documents live outside code. Specs exist in Confluence. Design docs languish in Google Drive. Tech specs might be on a wiki. The result is that context is fragmented, outdated, and inaccessible when you need it.

With agentic development, context is everything. AI agents need to understand product vision, technical constraints, design decisions, and test requirements. Scattered documentation defeats that purpose.

The solution is moving specs into code repositories. They should be versioned, searchable, and always in sync with implementation. This includes:

• Product specs: Domain knowledge, competitive landscape, roadmap, finished and in-progress features
• Tech specs: Design docs, resilience patterns, API contracts, scalability requirements
• UI specs: Mock implementations in code (using tools like Claude or Figma), shared styles, design system inheritance
• Test specs: Test cases, performance targets, SLA requirements, operational constraints

When specs live in code, AI agents can access them directly. More importantly, they stay current. Engineers reference them during implementation. Leaders can experiment with UI mockups in 30 minutes instead of weeks. Context becomes a core asset, not an afterthought.

2. Agent-Ready Architecture: Know When to Delegate

Not every problem needs an agent. The instinct to use agents everywhere will kill your ROI fast.

The right pattern is: use agents for open-ended, multi-step tasks that require dynamic decision-making and reasoning. Code review? Good candidate for an agent. Analyzing a Jira ticket to generate a full solution? Excellent. Routing a request to the right microservice? No. Use a traditional API.

At Harness, they found that roughly 15-20% of work is truly agent-driven (fully autonomous from Jira to PR), while 80% is AI-assisted (developers using agents as tools). Both are valuable.

The architecture looks like traditional microservices with agent endpoints (using Model Context Protocol). Services are still built for scalability, resilience, and multi-tenancy. But where there's ambiguity or complex reasoning, agents can operate on context and collaborate across systems.

The key is scope. Each agent has a specific role with bounded permissions. A code review agent doesn't implement changes. A testing agent doesn't modify production. A specification agent doesn't decide architecture. Granular, specialized agents are faster and cheaper than bloated omniscient agents trying to do everything.

3. Multi-Layer Verification: Test Everything Differently

Traditional testing doesn't work for agentic systems. You need six layers:

1. Front-end verification: Visual tests (Vitest, Playwright), UI correctness
2. Product and API testing: Unit, integration, and functional tests for services
3. Agent efficacy testing: Does the agent produce correct outputs? Baseline the agent's behavior against known-good examples
4. Data verification: Cross-check that training data and validation data are correct. This is critical for agents that hallucinate.
5. Continuous efficacy monitoring: The most crucial layer. Just like regression tests catch regressions in code, efficacy tests catch regressions in agent quality. If you change a prompt or context, your agent's output might degrade. Continuous monitoring catches this automatically.
6. Agentic feedback loops: Monitor agent behavior in production, not just in tests

‍

The difference is layer 5. With AI agents, quality isn't static. Change a system prompt slightly? The agent's outputs change. Update the context knowledge base? Efficacy drops. You need continuous verification that catches regressions in real time, just like you do with code tests.

4. Operational Excellence: Inspect Before Launch, Monitor Always

Most organizations surface operational concerns after development. That forces expensive rework and delays launches.

Operational Readiness Reviews (ORR) shift this left. During design review and test specification, teams ask: What about GDPR compliance? How do we ensure availability? What's the data residency strategy? What are our SLAs?

Instead of surprises at launch, ORR creates a checklist with documented evidence. Load tests prove you hit 200 TPS targets. Game days show incident response works. Security audits validate compliance. By launch, you don't wonder if you're ready. You know.

Then, post-launch, teams meet weekly for operational reviews. Dashboards show service health, customer issues, and RCAs from incidents. Engineering leadership meets separately for cross-calibration on operational excellence.

This discipline isn't just about preventing incidents. It's about continuously learning from operations and feeding that back into architecture and design.

The Results: Speed Without Sacrificing Safety

After six months of implementing these four pillars, Harness shipped 23% more features, reduced incidents by 44%, and achieved 45% AI ROI. More importantly, they now operate with predictable velocity and confidence in operational control.

That's not just faster. It's smarter.

Why This Matters

The agentic era doesn't have to be a choice between speed and safety. Teams that treat agentic development as a discipline (with specs, verification, architecture discipline, and operational rigor) gain speed and control.

Teams that just turn agents loose and hope for the best will gain speed and lose control. They'll ship faster. They'll also cause incidents faster. They'll accumulate technical debt faster. The cost of speed without structure compounds quickly.

The question isn't whether to use AI agents. You will. The question is whether you'll engineer for the agentic era with the same rigor you'd apply to critical systems. Agentic systems are critical systems now. They're making architectural decisions, generating code, and running production.

The organizations winning in the agentic era aren't the ones using the newest models or the most agents. They're the ones who slowed down just enough to build the guardrails, specs, and operational discipline that lets them move fast safely.

That's engineering excellence. That's the new baseline.

Ready to engineer for the agentic era safely? Learn how to implement spec-driven development, multi-layer verification, and operational excellence in your organization. The four pillars of agentic engineering aren't optional. They're table stakes.

Form Submission Thank You Message

Thank You Message Copy (Full Version):

"Thanks for downloading! You've just unlocked the framework that got Harness to 23% more features shipped with better quality. Check your email for the complete four-pillar implementation guide plus templates for spec-driven development, agent governance policies, and operational readiness reviews. See exactly how multi-layer verification catches AI regressions before they hit production. Your team can achieve the same results. Questions? Reach out anytime."

Thank You Message Copy (Short Version):

"Got it! Your guide is on the way. While you're reading, explore how Harness teams reduced incidents by 44% using operational readiness reviews and continuous efficacy monitoring for AI systems. Watch our live demo showing spec-driven development in action and how to chain agents safely. Or book a session with our engineering experts to design your agentic SDLC framework."

Most teams find out their system has a weak point the hard way, when it's already down in production. Harness Resilience Testing (formerly known as Chaos Engineering) exists to flip that around. It lets you intentionally inject controlled faults into your services and infrastructure, observe how the system responds, and fix what breaks before a real incident occurs. It comes with 200+ built-in faults, probes, and actions across Kubernetes, cloud platforms, Linux, and Windows, plus an AI Reliability Agent and an MCP server so you can run and analyze experiments from your IDE instead of jumping into a separate UI every time.

In this blog, we’ll cover two updates to the Resilience Testing documentation that help close that gap. First, we’ve brought the Chaos Hub directly into the docs. Second, we’ve introduced a Prompt Libraryin natural language rather than that lets you interact with Harness MCP using natural language instead of memorizing commands.

Here's a walkthrough of both features

Chaos Hub, without leaving the docs

Go to the Resilience Testing docs, click Chaos Hub, then Enterprise Hub, and you get the full catalog of fault, probe, and action templates in one place. Filter by infra type (Kubernetes, GCP, whatever you're running), browse probe templates, and check what action templates exist before you build an experiment instead of guessing.

Prompt Library Integration

The bigger update is the Prompt Library, under the AI section in the sidebar. If you have connected Harness MCP to your IDE or Claude, you already know the hard part isn't the connection; it's knowing what to ask for. The Prompt Library gives you a set of pre-built prompts for common resilience workflows, each with input fields for your org and project name that get inserted into the prompt text as you type. Copy the finished prompt into Cursor, Claude, or wherever you run MCP, and go.

‍

Three worth trying first:

Resilience coverage map

Audits every service in your project for chaos coverage and flags the ones with none. Fill in org and project, paste into your IDE, and it returns experiments grouped by target service, which services have zero experiments, and a priority ranking for what to fix first.

Useful as a first pass on any project you inherited or haven't touched in a while.

Service resilience risk scan

Ranks every service by unmitigated risk and surfaces AI-generated experiment recommendations you haven't acted on. Running this against a payment/banking service set returned a resilience score breakdown by service, a critical/high-risk tier, a list of recommended experiments, and a flag for anything with no new experiments in the last 30 days. It also proposed a sprint backlog from the output, which is the part worth stealing, even if you ignore everything else.

Pod failure experiment design

This one goes further than reporting; it builds and runs an experiment. Give it org, project, environment, infra type, infra name, and target deployment, and it will confirm the target is active, create an HTTP probe, and show you the full experiment configuration before saving. And then it will create and run your chaos experiment. Check out this video to see it in action.

Try it today

Pick one service in your project with no chaos experiments and no MCP history against it. Run the resilience coverage map, then the risk scan, and see what comes back. You'll probably find something closer to the Bank of Anthos result than you'd expect.

More prompts are getting added to the library as we build them out. If there's a specific resilience workflow you keep doing by hand, that's usually a sign it belongs in the library next.

New to Harness Resilience Testing? Explore our documentation to get started.

‍

Modern software releases rarely consist of application code alone. A new feature might require additional database tables, modified columns, new indexes, updated constraints, or reference data changes. While many organizations have successfully automated application deployments, database schema migrations often remain disconnected from the rest of the software delivery process.

This separation creates deployment bottlenecks, increases operational overhead, and introduces unnecessary risk.

Application teams may deploy code through automated Continuous Delivery (CD) pipelines, while database changes are managed through separate processes, manual approvals, or standalone tooling. The result is a fragmented release workflow where application and database changes must be done manually. In this article, we'll explore how Harness Continuous Delivery (CD) and Harness Database DevOps (DBDevOps) enable teams to deploy database and application changes through a unified pipeline, improving release velocity, visibility, governance, and deployment safety.

Why Database Schema Migrations Should Be Part of Your CD Pipeline?

Applications and databases are now a days mostly tightly coupled and an application release may depend on:

• New tables
• Additional columns
• Updated indexes
• Modified constraints
• New database objects

When application deployments and database schema migrations happen independently, teams often encounter several challenges.

Manual Coordination

Application and database teams must coordinate release schedules and deployment windows.

Deployment Risk

Applications may be deployed before required database changes are available, resulting in runtime failures or broken functionality.

Limited Visibility

Teams often struggle to answer questions such as:

• Which database schema migrations have been deployed?
• Which environments are running the latest schema version?
• What changes were included in a release?

Complex Rollbacks

• Application rollbacks are generally straightforward.
• Database rollbacks can become significantly more complicated when deployments are managed through separate workflows.
• A unified deployment process helps eliminate these challenges by treating database schema migrations as a first-class part of software delivery.

What a Unified Deployment Pipeline Looks Like

Instead of managing application and database releases separately, teams can orchestrate both through a single deployment pipeline.

This approach ensures database and application changes remain synchronized throughout the deployment lifecycle.

Deploying Database Schema Migrations and Applications Together

Let's examine a practical example using Harness Database DevOps and Harness Continuous Delivery. The deployment stage consists of three primary phases:

1. Apply database schema migrations
2. Deploy the application
3. Verify deployment health and initiate rollback when necessary

Phase 1: Apply Database Schema Migrations

The deployment begins by applying database changes using the Harness Database DevOps schema deployment step.

- step:
    type: DBSchemaApply
    name: Apply Database Schema
    identifier: Apply_Database_Schema
    spec:
      connectorRef: account.harnessImage
      migrationType: Liquibase
      dbSchema: bookkeeper
      dbInstance: goodinstance
      tag: v1.0.0

The DBSchemaApply step executes the database schema migration associated with the specified release version. By versioning database changes and deploying them through a pipeline, teams gain:

• Consistent database schema migration across environments
• Controlled promotion between development, staging, and production
• Deployment traceability
• Improved governance and auditability

Most importantly, database changes are deployed automatically as part of the release process rather than through a separate operational workflow.

Phase 2: Deploy the Kubernetes Application

After the database schema migration completes successfully, the pipeline proceeds with the application deployment.

- step:
    name: Deploy Application
    identifier: rolloutDeployment
    type: K8sRollingDeploy
    timeout: 10m
    spec:
      skipDryRun: true
      pruningEnabled: false

The K8sRollingDeploy step performs a rolling deployment to Kubernetes, gradually replacing existing application instances with the new version while maintaining service availability. Because the database schema migration has already been applied, the application can safely consume:

• Newly created tables
• Updated columns
• Additional indexes
• New constraints
• Modified database structures

This sequencing helps eliminate compatibility issues that commonly occur when application and database deployments are managed separately.

Automated Rollback for Safer Releases

One of the most valuable aspects of a unified deployment pipeline is coordinated rollback. If verification identifies an issue, the stage can automatically trigger rollback actions.

failureStrategies:
  - onFailure:
      errors:
        - AllErrors
      action:
        type: StageRollback

The rollback workflow executes the Kubernetes rollback step:

rollbackSteps:
  - step:
      type: K8sRollingRollback
      name: Rollback Rollout Deployment

This enables teams to quickly restore the previous application version and minimize production impact. Combined with database deployment visibility and rollback strategies, organizations can significantly reduce deployment risk and improve recovery times.

Benefits of Combining Continuous Delivery and Database DevOps

• Faster Release Velocity: Database and application changes move through a single automated workflow without waiting for separate deployment windows.
• Safer Database Schema Migrations: Version-controlled database schema migrations help ensure consistency across environments.
• Improved Visibility: Teams gain centralized insight into deployments, migration history, and release status.
• Better Governance: Approval workflows, policies, and compliance requirements can be enforced directly within deployment pipelines.
• Reduced Deployment Risk: Verification and rollback become built-in parts of every release.
• Stronger Collaboration: Developers, platform engineers, and database teams work from the same deployment process rather than maintaining separate release workflows.

Bringing Database Schema Migration into Modern Continuous Delivery

Database schema migration should not be treated as a separate operational activity. Modern software delivery requires application code and database changes to move together through a secure, automated, and governed deployment process. Harness Database DevOps enables teams to:

• Automate database schema migrations
• Integrate database deployments into CI/CD pipelines
• Manage version-controlled database changes
• Apply governance and compliance controls
• Support rollback and recovery workflows
• Deploy applications and databases together through a unified pipeline

By combining Harness Continuous Delivery and Harness Database DevOps, organizations can streamline releases while reducing deployment risk and improving operational efficiency.

Conclusion

Application releases and database schema migrations are two parts of the same deployment. Managing them separately creates unnecessary complexity, increases risk, and slows delivery.

By orchestrating database schema migrations, Kubernetes deployments, verification, and rollback within a single pipeline, teams can build a more reliable and efficient release process. Harness Database DevOps and Continuous Delivery provide the foundation for modern software delivery, enabling organizations to deploy applications and database changes together with greater confidence, visibility, and control. Explore how Harness Database DevOps can transform your delivery process today.

FAQs

1. What is a database schema migration?

A database schema migration is a version-controlled change to a database structure. Examples include creating tables, adding columns, modifying indexes, updating constraints, or introducing new database objects. Database schema migrations ensure that database changes can be tracked, reviewed, and deployed consistently across environments.

2. Why should database schema migrations be included in a Continuous Delivery pipeline?

Including database schema migrations in a CD pipeline ensures application and database changes are deployed together. This reduces deployment risk, improves consistency across environments, simplifies release management, and provides better visibility into the overall deployment process.

3. How does Harness Database DevOps support database schema migration?

Harness Database DevOps enables teams to automate database schema migrations through pipelines, manage version-controlled database changes, enforce governance policies, support rollback workflows, and orchestrate database deployments alongside application deployments within a unified release process.

‍

Why is Artifact Registry cleanup hard to get right

AI is changing artifact management in two ways at once. Every AI-generated pull request, dependency update, and automated build creates more container images, packages, and Helm charts than ever before. Registries are growing faster than engineering teams can manage them, driving up storage costs and leaving thousands of stale artifacts behind. At the same time, the cost of deleting the wrong artifact has never been higher. One mistaken cleanup policy can remove a production image that's still serving traffic or a package that hundreds of downstream applications still depend on.

One enterprise SaaS customer we spoke to run more than 270 microservices on a single Docker registry. A single age-based cleanup rule, set to delete artifacts older than 30 days, removed production images that were still serving traffic. The next deployment failed. An incident occurred. The rule was disabled. Cleanup never restarted. The registry kept growing at 2 to 3 TB per month.

That story is becoming increasingly common as release velocity accelerates. Across Docker registries, npm, Maven, Helm, and Python repositories, traditional cleanup strategies weren't designed for the volume and pace of AI software delivery.

• Age-based rules like "delete artifacts older than 30 days" remove production artifacts that haven't been redeployed but are still actively running.
• Count-based rules like "keep last 10 versions" silently drop deployed versions when rapid release cadence pushes them out of the keep window.
• Tag-based patterns fail when teams share tags across environments (the same release tag applied to both staging and production).

There's a fourth failure mode that's harder to see. Traditional lifecycle rules only look at the artifact itself: version count, age, and download activity. They have no awareness of the broader SDLC. The same artifact can be 60 days old, downloaded exactly once, and still be the exact image running live in a production pipeline. In a world where AI-generated pull requests are creating more builds than any team can manually track, that blind spot grows fast. Harness Artifact Registry's lifecycle rules are designed to 

close it. Rather than evaluating artifacts in isolation, they can tie in metadata across different stages of the SDLC, including CI/CD, so that an artifact currently deployed in a pipeline is automatically protected from deletion, no extra rule required. The registry doesn't just know what it's storing. It knows what's running.

Teams are left with an impossible choice: disable cleanup and let storage costs grow unchecked, or automate cleanup and risk deleting artifacts that production still depends on. In a world where most code is generated by AI, neither approach scales. Artifact lifecycle management has to become policy-driven, predictable, and safe by default.

Lifecycle Rules is built around fixing this exact problem. Two rule types. Three scope levels. Dry-run preview before any rule fires. Soft-delete with one-click recovery. An explicit attachment model that prevents the hidden cascading behavior most registries suffer from.

This guide walks through the feature as a narrative: one DevOps engineer's first day setting up cleanup rules across five package types and learning how the hierarchy works. The engineer is fictional; the registry behavior, screenshots, and execution data are real, captured from a working Harness Artifact Registry instance.

The feature is currently behind the HAR_ARTIFACT_LIFECYCLE_POLICY feature flag. Contact Harness Support to enable it on your account. Once on, you'll be looking at exactly what's shown below.

The mental model, in 60 seconds

Before the walkthrough, the building blocks:

Two rule types. Three scopes. Always reversible. That's the whole feature.

Day one. Maya opens the Lifecycle Rules tab.

Maya is a DevOps engineer on a fictional e-commerce platform. Her team has Docker, npm, Maven, Python, and Helm registries. Releases are tagged 1.0.0, 1.1.0, 1.2.0. Dev and PR builds use 0.9.0-dev, 1.3.0-pr-100, and similar. The registries have been growing for months. Today Maya gets the ticket: set up automated cleanup, but don't break anything.

Maya opens the Harness UI, navigates to Artifact Registry → Registries Settings → Lifecycle Rules, and clicks + New Lifecycle Rule.

She knows the rule of any cleanup operation: protect what matters before you delete anything. Her first rule won't delete a thing. It will mark her formal releases as untouchable.

Maya creates her first rule: a Retention rule

She fills in:

Retention rule with literal version chips. Each chip is one exact version. Predictable matching, no surprises.

She uses literal version strings, not 1.*. Maya knows that wildcards are greedy. 1.* would also match 1.3.0-dev, which would protect dev builds she actually wants cleaned up later. Exact strings are predictable.

She submits. The rule lands in the table. There's no schedule attached, because Retention rules don't run on cron. Instead they evaluate passively, every time a Cleanup rule tries to delete something. If the version matches, the deletion is blocked.

Retention always wins. If a Retention rule matches an artifact, no Cleanup rule, no manual delete, nothing within the lifecycle system can remove that artifact while the rule exists.

Now Maya goes after the dev builds

With her safety net in place, Maya is ready to actually delete something. She targets her npm registry first because the team's @shop/cart-sdk package has accumulated dev builds from months of feature branches.

She clicks + New Lifecycle Rule again, this time as a Cleanup rule:

NPM cleanup rule configured with strattest07 registry, NPM type, dev-version patterns, and Keep last 1.

Maya uses two patterns instead of one. She learned from experience that stacking narrow patterns is more reliable than reaching for one clever glob. 0.* catches versions like 0.9.0-dev and 0.9.1-dev. *dev* catches anything with "dev" in the name. Both together cover the dev-build space cleanly.

She submits. The rule is saved but hasn't run yet. Cleanup rules execute on schedule, but Maya isn't going to wait for tomorrow's cron. She's going to dry-run it right now.

The dry-run is where you build trust

This is the part Maya's old registry didn't have. From the rules table, she clicks the three-dot menu next to cleanup npm devs and selects Dry Run.

Within seconds, she lands on the execution detail:

Dry-run result. Stable 1.0.0 and 1.1.0 versions are PROTECTED by the Retention rule. Older dev versions are flagged as WOULD BE DELETED. No artifacts were actually touched.

Maya scans the result:

• 1.0.0 and 1.1.0: PROTECTED, because her Retention rule matched.
• The older dev versions: WOULD BE DELETED.
• At the top: an estimated storage reclaim number she can show her manager.

Nothing was actually deleted. This is the safety net. Maya can change her patterns, change the keep count, change the registry scope, and re-run the dry-run as many times as she wants. Each dry-run uses the same evaluation logic as a real execution. The only difference is the state mutation.

Maya stacks criteria for an even stronger safety net

For her Helm registry, Maya wants to be extra careful. Helm charts deploy production infrastructure. A wrong delete here means real downtime.

She creates a Cleanup rule with two criteria stacked:

• Keep last 1 version
• Delete artifacts older than 1 day

The two criteria combine with AND logic. An artifact must satisfy both conditions to be deleted.

She dry-runs it:

Helm dry-run with stacked AND criteria. No results found because every chart was pushed that morning. The age criterion blocked the deletion entirely.

No results found. Estimated savings: 0 bytes.

Why? Because Maya pushed all her Helm charts that same morning. Keep-last-1 alone would have flagged plenty for deletion, but the age criterion saw all the charts as less than 1 day old and blocked everything. Nothing gets deleted until every guard agrees.

This is exactly the conservative behavior she wants for Helm. She saves the rule confidently knowing it won't act until artifacts are genuinely old.

Rule of thumb. Use a single criterion when you want aggressive cleanup. Stack two or three with AND logic when you want a safety net.

Now her manager walks over

"Maya, the org admin team wants all releases tagged 1.0.0, 1.1.0, and 1.2.0 protected across every project. They don't want each team setting their own retention rules. Can we centralize that?"

Yes. This is what scoped rules are built for.

HAR has three scope levels for lifecycle rules:

1. Account level: for CISO, compliance, central governance.
2. Organization level: for platform teams, business unit governance.
3. Project level: for service teams, day-to-day operations.

But here's the key design choice from the docs:

"Rules are reusable policies that are explicitly attached to registries. A rule does nothing until attached. This explicit attachment model eliminates hidden cascading behavior."

In plain words: rules created at the org or account level don't automatically take effect in child scopes. The child scope admin sees the rule, decides to opt in, and explicitly attaches it. This means central governance doesn't surprise project teams, and project teams can't accidentally weaken what central governance has defined.

The org admin creates the rule at org scope

Maya's org admin switches to the organization scope using the top breadcrumb, then runs the same New Lifecycle Rule flow Maya used. Same form. Same fields. Only the breadcrumb above the form is different.

Org-scope rules tab. The breadcrumb shows Account / Organization, and the rules list contains the org-level Retention rule. From here, the rule is visible to all child projects but does not automatically protect anything.

The rule now lives at org scope. The next step is Maya's: she has to opt in.

Maya sees the org rule, but it isn't doing anything yet

Back in her project, Maya goes to the Lifecycle Rules tab and ticks the Show policies from parent scope box.

Project rules list with parent-scope visibility enabled. The org rule appears with the ORGANIZATION badge in the Created In column. There's no toggle on the right, only project-owned rules are toggleable from a child scope.

The org rule appears in her list with an ORGANIZATION badge in the Created In column. It's visible. But Maya wants to verify whether it's actually protecting anything yet.

She re-runs the dry-run on cleanup npm devs:

Dry run with the org rule visible but unattached. Every version, including the 1.0.0 and 1.1.0 releases, shows WOULD BE DELETED. The org rule is passive.

Every version shows WOULD BE DELETED. Including the 1.0.0 and 1.1.0 releases that should be protected by the org rule. The org rule is visible but passive. Maya hasn't attached it yet.

Maya attaches it

She clicks on the org rule row. An Edit form opens, but only one field is editable: which registries this rule attaches to in her project. Criteria, patterns, and schedule are all inherited from org scope and locked.

Attach form for the inherited org rule, opened from project scope. The project admin can add their registries to the attachment list. Criteria and patterns are inherited from the parent and cannot be changed here.

She adds her project's registries to the attachment list and saves.

Now she re-runs the dry-run on cleanup npm devs one more time. This time, 1.0.0 and 1.1.0 show PROTECTED. The protection is coming from the org-level rule, attached at the project scope.

Rules list after attachment. A chain icon next to the org rule name indicates it is actively attached to one or more registries in this scope.

The model in one sentence: authorship stays at the parent scope, attachment is opt-in by the child scope. Org admins define what counts as a release. Project admins decide which of their registries the definition applies to. Nobody can weaken anybody else's rule.

Maya schedules her rules to run overnight

The dry runs are passing. The hierarchy is in place. It's time to let the system run for real.

Maya edits each Cleanup rule and walks through to the Schedule step. She picks Daily, sets the timezone, and lets the cron expression auto-generate.

Schedule step. Timezone selector at the top, frequency tabs, and a generated cron expression. Custom mode lets you write any cron.

She schedules npm at 01:00 UTC, python at 01:15 UTC, maven at 01:30 UTC, spreading them out to keep the execution log clean. Helm and the generic bundles she leaves running with their safety-net criteria, knowing they'll keep returning "no results" until the artifacts are genuinely old enough.

Final state before overnight scheduled execution. Five Cleanup rules covering all package types, plus one attached Org Retention rule. Saved Storage starts at zero.

She closes her laptop.

The next morning

Maya opens the Lifecycle Rules tab and looks at the Saved Storage card at the top:

After overnight execution. 5.59 KB saved, 7 artifacts cleaned, 5 executions in the last 30 days. Real artifact cleanup, automated.

5.59 KB saved. 7 artifacts cleaned. 5 executions in the last 30 days. These are small numbers because this is a demo project. In production scale, with a registry growing at 2 to 3 TB monthly, those numbers add up fast.

She switches to the Lifecycle Rule Executions tab:

Executions tab with five SCHEDULED runs and several DRY RUN entries, all showing SUCCESS. Notice the rules that returned zero deletions (helm charts and bundles) still ran successfully. They just had nothing to delete, thanks to AND-logic safety.

Every execution is logged. Rule name, registry, type (Scheduled or Dry Run), status, registries affected, packages affected, versions deleted, storage reclaimed, timestamp. The execution log doubles as a complete compliance audit trail. Every deletion traceable to a specific rule run.

She drills into the python cleanup execution:

Drill-in view of the python execution. Two versions show SOFT DELETED, while 1.0.0 and 1.1.0 show PROTECTED by the Retention rule.

Two versions show SOFT DELETED. Two versions show PROTECTED. Exactly what Maya designed: dev builds cleaned, releases retained.

And then somebody asks for the deleted version back

"Maya, I was about to test a regression against 0.9.0.dev1. Can you get it back?"

Soft delete is the whole reason Maya can sleep through the night. Every deletion in lifecycle rules is reversible during the recovery window (configured at the account level).

She navigates to Registries → pinym → shop-reco-engine. The artifact view has two tabs: Available and Deleted. She clicks Deleted.

Artifact view, Deleted tab open. The three-dot menu reveals Restore and Delete options. One click brings the version back to Available.

She clicks the three-dot menu on 0.9.0.dev1. The dropdown shows Restore and Delete. She clicks Restore.

After restore. Confirmation toast reads 'Artifact version restored successfully' and the version is back in the Available state. The metadata, digest, and size are preserved.

"Artifact version restored successfully." The version is back. Same metadata, same digest, same size. As far as the downstream consumer is concerned, it never went away.

The safety promise. There is no hard-delete option in lifecycle rules. Even the most aggressive Cleanup rule can only soft-delete. Within the recovery window, every deleted version is one click away from restoration.

What Maya accomplished in one day

By the end of day one, Maya:

1. Set up automated cleanup across five package types.
2. Protected her formal releases with a Retention rule, so the cleanup can't touch them.
3. Verified every rule with dry-runs before letting them fire.
4. Adopted the org admin's centralized protect rule without losing project autonomy.
5. Scheduled real overnight execution and watched it work.
6. Demonstrated one-click recovery the moment somebody asked for an artifact back.

She didn't break production. She didn't delete the deployed image. She didn't have to manually triage which builds to keep. The system did what the docs promised, and her engineering org has a paper trail to show compliance if anyone asks.

The same machinery scales from a single demo project to a 270-service real customer registry. Same Retention rule pattern. Same Cleanup rule pattern. Same explicit-attachment hierarchy from project to organization to account scope.

As software delivery accelerates, artifact registries can no longer be passive storage systems. They need to actively govern what stays, what gets removed, and what is safe to promote. Lifecycle Rules are one part of a trusted software supply chain, working alongside provenance, vulnerability scanning, promotion policies, and artifact signing to ensure every artifact in your registry exists for a reason. By automatically retaining what matters, removing what doesn't, and enforcing policy before storage becomes operational risk, teams can deliver software faster without sacrificing security, reliability, or trust.

We invite you to sign up for a demo and see firsthand how Harness Artifact Registry delivers high-performance artifact distribution with built-in security and governance at scale.

Frequently asked questions

What's the difference between a Retention Rule and a Cleanup Rule?

A Cleanup Rule deletes artifact versions on a schedule based on age, version count, or download activity. A Retention Rule protects matching artifacts from any deletion. It's always evaluated first, and a match always wins over a deletion attempt. Use Retention for releases you never want touched, and Cleanup for dev, snapshot, and stale builds.

Does soft delete mean my artifact is gone forever?

No. Every artifact deleted by a Lifecycle Rule enters a recovery window. During that window, the artifact can be restored in one click from the Deleted tab of the artifact's detail view. There is no hard-delete option in Lifecycle Rules at all.

How do rules at the Organization or Account scope apply to my Project's registries?

They don't apply automatically. A rule created at any scope is visible to child scopes when you tick "Show policies from parent scope," but it only acts on registries it's explicitly attached to. The child-scope admin attaches the rule by selecting which of their registries it should apply to. Criteria and patterns inherited from the parent scope cannot be modified at the child scope.

Can I preview what a rule will delete before it actually runs?

Yes. Every Cleanup Rule has a Dry Run action in its three-dot menu. The dry run uses identical evaluation logic as the real run but mutates no data. It produces a per-version table showing PROTECTED versions and WOULD BE DELETED versions plus an estimated storage reclaim.

What happens if I stack multiple retention criteria on one Cleanup Rule?

Multiple criteria combine with AND logic. An artifact must satisfy every enabled condition to be deleted. This is conservative by design and is the main safety mechanism against accidental over-deletion. Use a single criterion when you want aggressive cleanup, and stack two or three when you want a safety net.

Which package types does this work with?

The feature works across all package types supported by Harness Artifact Registry: Docker, npm, Maven, Python, Helm, NuGet, RPM, Cargo, Go, Generic, and others. Filter fields adapt to the package type (Maven shows Group ID patterns, Docker shows tag patterns, etc.).

How do I get access to Lifecycle Rules?

Lifecycle Rules are currently behind the HAR_ARTIFACT_LIFECYCLE_POLICY feature flag. Contact Harness Support to enable it on your account.

Next steps

If your registry has been growing without a cleanup policy, the safest place to start is exactly where Maya did:

1. Identify your release version patterns (the artifacts you never want touched).
2. Create a Retention Rule with those literal version strings at your project scope.
3. Create a Cleanup Rule targeting dev or snapshot builds in one registry.
4. Run a Dry Run before saving a schedule.
5. Let it run for one week, then review the Executions tab to see what was cleaned up.

For full reference documentation, see the official Harness docs. The Harness CLI used to push artifacts in this guide is open source. 

If you set up Lifecycle Rules and something surprises you, the Harness team wants to hear about it. Reach out through the community Slack, the developer forum, or open an issue on GitHub.

Cloud cost visibility at scale usually works great… until it suddenly doesn’t.

At first, everything feels manageable. You can track spend by service. You know which team owns which resources. Reports are clean, and the numbers make sense.

Then one day, there’s a $47,000 spike spread across three AWS accounts that no one noticed for eleven days. Leadership wants answers. Engineering wants context. And your carefully designed tagging strategy? It turns out half the resources aren’t tagged correctly anymore.

This isn’t about carelessness. It’s about scale.

The systems and processes that work for one account, a few teams, and predictable workloads simply don’t hold up in a fast-growing, distributed, multi-cloud environment. What worked when you had 50 engineers doesn’t scale to 500. Tagging strategies that worked for ten microservices fall apart at a hundred. Manual reviews that felt reasonable in year one become overwhelming operational debt by year three.

Cloud cost management doesn’t fail because teams don’t care. It fails because the model never evolved.

Why Traditional Cloud Cost Visibility Breaks Down

Many organizations still treat cloud spending visibility the way they treated on-prem infrastructure: centralized reports, periodic reviews, and reconciliation after the invoice arrives.

But the cloud doesn’t behave like a static data center.

Infrastructure is provisioned with API calls. Workloads scale up and down automatically. Teams deploy multiple times a day. In that environment, monthly reporting isn’t just slow — it’s disconnected from reality.

1. Cloud Cost Allocation Challenges Multiply Fast

Cost allocation usually starts simple.

You tag an EC2 instance with a team name.
You assign an S3 bucket to a product line.
You map Kubernetes namespaces to cost centers.

Easy enough.

Then things get complicated.

Shared services support multiple teams. A single database might power ten applications. Load balancers route traffic across services owned by different squads. Now your allocation model depends on custom logic, judgment calls, and manual adjustments.

At scale, cloud cost allocation challenges aren’t just about missing tags. They’re about unclear ownership and constantly shifting boundaries.

Without strong FinOps governance and automated enforcement, tagging degrades over time. Trust in the numbers erodes. And once teams stop trusting the data, infrastructure cost transparency disappears.

2. Multi-Cloud Environments Create Fragmented Visibility

Multi-cloud cost governance sounds great in theory. In practice, it’s messy.

AWS, Azure, and GCP all have different pricing models, billing exports, and discount structures. Reserved Instances don’t map cleanly to Committed Use Discounts. Credits and savings plans behave differently. Even basic service naming varies.

Maintaining true cloud spending visibility across providers requires more than dashboards — it requires normalization and context.

Without a unified view, engineers working in AWS don’t see how their choices impact Azure costs. Data teams running BigQuery jobs aren’t aware of the downstream effect on shared networking or storage costs. Everyone optimizes within their silo, but total spend keeps growing.

Enterprise cloud cost optimization can’t happen in fragments. It requires shared visibility across environments.

3. Reactive Reporting Comes Too Late

By the time finance flags a cost spike, the root cause is buried.

The change that triggered it may be three sprints old. The engineer who made it might not even remember. The workload has already scaled, dependencies have grown, and what started as a small inefficiency is now baked into production.

Traditional cloud cost monitoring tools often operate at the billing layer. They tell you what changed — but not why.

Was the spike caused by a misconfigured NAT gateway? An inefficient query? A new feature launch? Increased traffic? The invoice doesn’t know.

Cloud cost visibility at scale requires linking cost signals to engineering context — deployments, configuration changes, usage patterns — before those signals turn into major overruns.

How to Restore Cloud Cost Visibility at Scale

The fix isn’t just better reports. It’s treating cost visibility as an engineering capability.

Cost needs to live inside workflows, not in a finance slide deck.

1. Automate and Deepen Cost Allocation

At scale, allocation can’t be a quarterly cleanup exercise.

You need automated rules that continuously map resources to teams, services, and business units. When something falls outside those rules, it should be flagged immediately — not discovered weeks later during reconciliation.

Strong FinOps governance means allocation is proactive, not reactive.

When teams see near real-time cost impact — through showback or chargeback — behavior changes. Engineers start asking better architectural questions. Optimization becomes part of daily decision-making, not an annual mandate.

2. Use Context-Aware Anomaly Detection

Not every spike is a problem.

A 200% increase during a product launch might be expected. A 200% increase on a quiet Tuesday afternoon probably isn’t.

Effective anomaly detection understands seasonality, traffic patterns, deployment schedules, and baseline behavior. It integrates with observability and CI/CD systems so that when costs change, teams can immediately see what else changed at the same time.

Cloud cost monitoring tools are most valuable when they connect cost signals directly to engineering activity.

3. Make Governance Proactive, Not Punitive

Waiting for an invoice to exceed budget doesn’t protect you.

Instead, set budget thresholds at the team and project level. Trigger approval workflows when provisioning exceeds expected spend. Automatically identify idle or non-compliant resources before they accumulate real cost.

Done right, governance doesn’t slow engineers down. It simply makes cost constraints visible early, when they’re still easy to manage.

How Harness Cloud Cost Management Supports Visibility at Scale

Harness Cloud Cost Management approaches cloud cost visibility at scale as a continuous engineering discipline, not a monthly accounting ritual.

It combines allocation, anomaly detection, and policy enforcement in a way that aligns with how modern teams actually work.

Unified Cost Visibility Across Clouds and Kubernetes

Harness brings AWS, Azure, GCP, and Kubernetes data into a single, consistent view. Engineers can analyze spend by team, namespace, workload, or service without switching dashboards or reconciling inconsistent billing exports.

This unified approach strengthens multi-cloud cost governance while improving infrastructure cost transparency across the organization.

Cost Allocation That Matches Real Ownership

Harness automates allocation using tags, labels, hierarchies, and usage data that reflect how teams structure their work. Shared services and multi-tenant infrastructure are distributed based on actual consumption — not static assumptions.

When allocation logic changes, it updates system-wide. No spreadsheets. No manual reconciliation.

Context-Driven Anomaly Detection

Harness correlates cost spikes with deployments, configuration changes, and infrastructure events. Instead of simply highlighting that spend increased, it surfaces the engineering activity likely responsible.

That’s what makes cloud cost monitoring tools actionable — not just informative.

Governance That Enables Speed

Harness supports budget policies, approval workflows, and automated lifecycle management that prevent overruns without introducing bottlenecks.

Teams keep their autonomy. Finance keeps predictability. Everyone shares visibility.

Learn more about how Harness approaches cost visibility and governance at:
https://www.harness.io/products/cloud-cost-management

Detailed implementation guidance is available at:
https://developer.harness.io/docs/cloud-cost-management

Visibility Creates a Cost-Aware Engineering Culture

Cloud cost visibility at scale isn’t really about dashboards.

It’s about alignment.

When engineers see cost impact in context, when allocation reflects real ownership, and when guardrails are proactive instead of reactive, cost awareness becomes part of the culture.

Without that, organizations fall into a cycle of surprise invoices, reactive firefighting, and growing tension between finance and engineering.

Visibility alone won’t solve cloud cost problems. But without it, enterprise cloud cost optimization is almost impossible.

If your current cloud cost management strategy still revolves around monthly reports and manual reconciliation, the real question isn’t whether you need better dashboards.

It’s whether your engineers have the cost signals they need at the exact moment they make decisions that drive spend.