Harness Blog

• Traditional web application firewalls (WAF) are no longer sufficient for protecting modern, API-first applications. WAAP provides unified, comprehensive runtime security for applications & APIs.
• A WAAP platform should automate policy creation, reduce alert fatigue, and integrate seamlessly into CI/CD pipelines, eliminating ticket-ops bottlenecks and reducing developer and security toil.
• Best practices such as continuous API discovery, API testing, and application-layer threat detection & response enable platform teams to operationalize security at scale without sacrificing delivery speed or developer autonomy.

When APIs now handle the majority of web traffic, protecting only HTML requests creates a blind spot that puts your organization at risk. Traditional WAFs weren't built for this reality and miss API-specific vulnerabilities that define modern attack vectors. Your security platform needs complete visibility into every API endpoint, authentication flow, and data exchange.

A web application and API protection (WAAP) platform should unify API discovery, API testing, API protection, bot & abuse protection, and cloud-scale WAF capabilities into a single platform. Instead of bolting security on after deployment, WAAP should provide standard runtime security control that’s also integrated into your software delivery lifecycle. Harness Web Application & API Protection delivers this unified approach without reintroducing the ticket-ops bottlenecks that slow your teams down.

What Is WAAP?

Web Application and API Protection (WAAP) is a modern cybersecurity approach designed to protect web applications and APIs from a wide range of threats.

Instead of relying on a single layer of defense like a network firewall, WAAP combines multiple security technologies into a unified solution to safeguard modern cloud- and AI-native designs.

At its core, WAAP protects against:

• Application-layer attacks (like SQL injection and cross-site scripting)
• API-specific exploitation and data exposures
• Bots, abuse, and automated threats
• Distributed Denial-of-Service (DDoS) attacks

In simple terms:

WAAP ensures that the apps and APIs powering your business stay secure, available, and trustworthy.

Why WAAP Matters More Than Ever

The need for WAAP isn’t just theoretical. It’s driven by real shifts in how applications are built and operated today.

1. The Explosion of APIs

APIs are now the backbone of digital ecosystems. They allow different systems to communicate, power mobile apps, and enable integrations between platforms.

However, APIs are often:

• Poorly documented
• Exposed to the public internet
• Lacking strong authentication

This makes them a prime target for attackers.

2. Cloud and Microservices Architecture

Modern applications are no longer monolithic. They’re built using microservices distributed across cloud environments.

While this improves scalability and flexibility, it also:

• Increases the number of endpoints
• Makes security visibility more difficult
• Expands the attack surface significantly

3. Sophisticated Cyber Threats

Attackers today use automation, AI, and large-scale bot networks to exploit vulnerabilities faster than ever.

They’re not just targeting infrastructure. They’re targeting:

• Login forms
• Payment systems
• Data flows
• User sessions

4. Limitations of Traditional Security

Older security tools, such as network firewalls and basic WAFs, were designed for simpler environments. They often:

• Rely on static rules and attack signatures
• Lack context about user behavior
• Struggle to detect modern attack patterns

WAAP fills this gap by offering adaptive, intelligent, and application-aware protection.

The Core Components of WAAP

WAAP isn’t a single tool. It’s a collection of advanced security capabilities working together. Let’s explore each one in more detail.

1. Web Application Firewall (WAF)

A Web Application Firewall (WAF) is one of the foundational layers of WAAP.

It inspects incoming HTTP and HTTPS traffic and filters out malicious requests before they reach your application. But modern WAAP goes far beyond simple rule matching.

They now incorporate:

• Behavioral analysis
• Machine learning-powered detections
• Context-aware filtering

This allows them to detect not only known threats but also suspicious patterns that may indicate new or evolving attacks.

For example, if a user suddenly submits hundreds of unusual requests in a short time, a modern WAAP can flag and block that behavior, even if it doesn’t match a known attack signature.

2. API Security

APIs are one of the most critical and vulnerable parts of modern systems.

WAAP provides dedicated API security that goes beyond traditional protections by focusing on how APIs operate.

Key capabilities include:

• API Discovery
  Identifies all APIs in your environment, including undocumented or “shadow” APIs that developers may have deployed without formal tracking as well as “zombie” APIs” that were abandoned.
• API Schema Validation
  Ensures that incoming requests follow the expected structure, preventing attackers from sending malformed or malicious data.
• Authentication and Authorization Enforcement
  Verifies that only authorized users and systems can access specific API endpoints.
• API Attack  Prevention
  Prevents requests that exploit API functionality or extract sensitive data, including risks defined in the OWASP API Security Top 10.

This WAAP layer is crucial to application security because many breaches today occur  as a result of poorly secured APIs.

3. Bot & Abuse Protection

Bots and automation account for a significant portion of internet traffic, and not all of them are friendly.

WAAP includes advanced bot & abuse protectionthat can distinguish between:

• Acceptable bot behavior (search engines, monitoring tools)
• Undesirable  bot behavior (scrapers, account-targeting, transaction fraud)

Instead of blocking all automated traffic, WAAP uses techniques like:

• Behavioral analysis
• Device fingerprinting
• Interaction patterns

This allows it to:

• Block malicious bot behavior
• Challenge suspicious activity (e.g., insert CAPTCHA dynamically into an application flow)
• Allow legitimate automation

4. DDoS Protection

Distributed Denial-of-Service (DDoS) attacks aim to flood your system with traffic until it crashes or becomes unavailable.

WAAP provides robust DDoS protection by:

• Detecting abnormal traffic spikes
• Filtering malicious requests
• Distributing traffic across infrastructure
• Automatically scaling defenses

This ensures that your application remains accessible, even under heavy attack conditions.

5. Application Threat Intelligence and Analytics

One of WAAP’s most powerful features is its ability to provide deep visibility into application threats.

Instead of simply blocking attacks, WAAP helps you understand:

• Where attacks are coming from
• What methods are being used
• Which API endpoints are being targeted
• How frequently attacks occur

This data enables teams to:

• Improve application runtime security
• Identify complex attack chains
• Respond faster to security incidents

How WAAP Works in Practice

Let’s walk through a simplified example of how WAAP operates behind the scenes.

1. A user (or bot) sends a request to your application
2. WAAP intercepts the request before it reaches your servers
3. The request is analyzed using multiple layers:
   
   • Signature-based detection
   • Behavioral analysis
   • Machine learning models
4. WAAP assigns a risk score to the request
5. Based on that score:
   
   • Safe requests are allowed
   • Suspicious requests are challenged
   • Malicious requests are blocked

All of this happens in real time, often within milliseconds, ensuring both security and performance.

WAAP vs. Traditional Security Tools

Understanding the difference between WAAP and older tools helps clarify its value.

Traditional Security Tools

• Focus on protecting networks rather than applications
• Use static, rule-based detection
• Provide limited visibility into API traffic
• Struggle with modern cloud environments and AI systems

WAAP

• Focuses on application-layer security
• Protects applications, APIs, and AI
• Uses adaptive, intelligent detection methods
• Designed for distributed cloud- and AI-native systems

In essence:

WAAP is built for modern application architectures, while other security tools were built for traditional infrastructure.

Common Threats WAAP Protects Against

WAAP is designed to defend against a wide range of modern threats, including:

• Injection Attacks - Attackers insert malicious code into inputs to manipulate databases or systems.
• Cross-Site Scripting (XSS) - Malicious scripts are injected into web pages to steal user data or hijack sessions.
• API Exploitation - Attackers misuse APIs to extract sensitive data or bypass security controls.
• Credential Stuffing & Account Takeovers (ATO) - Automated bots use stolen credentials or target account logins to compromise identities and gain unauthorized access.
• DDoS Attacks - Flooding systems with traffic to disrupt availability. 
• Zero-Day Attacks - Previously unknown vulnerabilities that haven’t yet been patched.

WAAP’s behavioral and AI-driven detection helps identify these even before signatures exist.

Benefits of Implementing WAAP

Adopting WAAP provides several key advantages:

• Comprehensive Protection - A unified solution that covers multiple attack vectors.
• Enhanced Visibility - Clear insights into traffic, usage patterns, and threats.
• Reduced Complexity - Eliminates the need for multiple disconnected tools and point solutions.
• Improved User Experience - Minimizes false positives, ensuring legitimate users aren’t blocked and availability is maintained.
• Scalability - Designed to grow with your applications and handle global traffic.

Who Needs WAAP?

WAAP is essential for any organization that relies on web applications or APIs, especially:

• SaaS companies
• E-commerce platforms
• Financial institutions
• Healthcare providers
• Enterprises with digital services

If your business operates online, WAAP is no longer optional and is a critical defense layer.

Best Practices for Implementing WAAP

Platform teams can't protect what they can't see, and the scale of the problem is often shocking. Most organizations discover they have three times as many APIs as they thought once they enable continuous discovery.

These enterprise API protection best practices turn visibility gaps and manual processes into automated guardrails that scale with your delivery velocity.

• Inventory everything first - Deploy continuous API discovery across all clusters and regions to eliminate shadow endpoints.
• Map sensitive data flows early - Identify which APIs handle PII, authentication, and critical business functionality before attackers do.
• Fail builds on high-severity issues: Integrate API testing into CI pipelines with contextual remediation guidance for developers.
• Enforce policies as code - Embed org-wide security rules in templates and gateway configurations, not manual reviews.
• Start with controlled rollouts - Operationalize protection for 7-10 critical services first to prove low friction before expanding.

Smart platform teams treat API protection like any other infrastructure component. Start small, automate the repetitive work, and let unified WAAP capabilities handle the scale while developers focus on shipping features.

Operationalize WAAP With Harness

Platform teams can't afford to treat API security as an afterthought when most incidents start with compromised endpoints. Harness Web Application & API Protection transforms security from a bottleneck into an automated control that fits your delivery pipelines. AI-powered detection cuts through alert noise while continuous API discovery eliminates shadow endpoints across your entire infrastructure.

The right WAAP solution integrates with your existing CI/CD workflows without creating new ticket-ops friction. Teams get runtime protection that adapts to application changes and shift-left testing that catches vulnerabilities before production.

Ready to gain full API visibility and protect applications at the speed of modern delivery? Try Harness Web Application & API Protection and see how unified API discovery, API testing, API protection, bot & abuse protection, and WAF  can eliminate security toil for your application security program.

WAAP In Practice: CI/CD Integration And Governance FAQ

Platform teams need concrete answers about WAAP integration that avoid new approval workflows and maintain engineering’s ability to deliver quickly. These WAAP FAQ responses address deployment scenarios andstrengthening security posture.

How does WAAP integrate with CI/CD without slowing builds?

WAAP runs API testing as part of your existing pipeline stages, not as a separate gate, to detect issues such as those defined in the OWASP API Security Top 10. Tests run against live traffic patterns and API schemas. Builds fail only on high-severity issues like broken authentication or data exposure, following NIST guidance for automated security enforcement without review delays.

What reduces false positives compared to traditional WAF solutions?

WAAP correlates full user journeys and API call chains (the sequence of service-to-service requests) to understand user intent rather than inspecting isolated requests. Machine learning models tune to your actual application configurations and data flows, not generic signature databases. This contextual approach reduces alert noise significantly compared to rule-based WAFs and catches threats that bypass traditional detection methods.

Which deployment options work for hybrid multi-cloud environments?

Any WAAP should support agentless edge routing, in-line integration with API gateways and load balancers, and out-of-band collection via traffic mirroring or eBPF. Teams should be able to combine deployment methods for different services to support their given application architecture. Kubernetes and container environments may also require  native ingress controller and sidecar support for east-west microservices traffic protection. Harness WAAP provides 30+ integrations to support modern designs. 

How do you enforce governance without creating ticket-ops?

Policy-as-code templates can be used to embed WAAP deployment directly into service deployment pipelines. Teams inherit org-wide protections automatically when using approved templates, providingguardrails that scale without approval bottlenecks.

What operational overhead should platform teams expect?

AI-assisted policy generation eliminates the need for signature writing and weekly change windows. Runtime protection automatically adapts to application changes. Platform teams generally see a 60-70% reduction in security-related tickets compared to traditional WAF management. Most operational tasks shift to self-service developer workflows.

How do you measure WAAP ROI and security improvements?

WAAP provides metrics on API discovery coverage, vulnerability remediation time, and blocked attacks. Platform teams track developer velocity through build time impact and ticket reduction. Security posture improves through measurable reductions in exposed APIs, faster incident response, and more efficient compliance audits.

At some point, every engineering team gets asked a version of the same uncomfortable question: "Where exactly are we using this package?"

Maybe it comes after a CVE drops. Maybe it's a compliance audit. Maybe it's just a principal engineer trying to clean up years of accumulated technical debt. Whatever the trigger, the answer is almost always the same: a lot of shoulder shrugging, Slack messages to people who might know, and a mad scramble through repos.

This is what happens when artifact management has no well-defined governance. Packages get pulled, images get built, dependencies pile up, and nobody has a real record of any of it.

Harness Artifact Registry is built with this problem front and centre. Alongside storing and distributing your artifacts, it gives your organization genuine visibility into what's happening with them, through audit trails and compliance-ready reporting.

Let's walk through what that actually looks like.

Every Action, Recorded

The foundation of auditing in Harness Artifact Registry is simple: nothing happens silently.

Every push, pull, deletion, policy evaluation, and quarantine action is logged, along with who or what triggered it and when. That applies to human users, automated pipelines, and service accounts alike. If something touched an artifact, there's a record of it.

Because Harness Artifact Registry is natively integrated into the Harness platform, these logs don't exist in isolation. When a pipeline pulls an image during a deployment, the audit trail connects that download to the specific pipeline run, the environment it deployed to, and the user or trigger that kicked it off. You're not just seeing that something was downloaded. You're seeing the full context around why.

Try it yourself: If you have Harness Artifact Registry set up, you can check the audit trail for your account right now. Navigate to Account Settings > Audit Trail, and filter by module:

Module: Artifact Registry
Resource Type: ARTIFACT_REGISTRY

You'll see every registry-level event with the actor, timestamp, and action. For pipeline-linked events, click through to see the full execution context.

You can also query audit events programmatically via the Harness API:

curl -X POST 'https://app.harness.io/v1/audit-events' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Harness-Account: YOUR_ACCOUNT_ID' \
  -H 'Content-Type: application/json' \
  -d '{
    "filterType": "AuditEvent",
    "modules": ["CORE"],
    "resourceTypes": ["ARTIFACT_REGISTRY"],
    "startTime": 1714003200000,
    "endTime": 1714089600000
  }'

This matters enormously during incident response. If a vulnerability surfaces in a package, you don't need to start guessing or digging through repos manually. You pull up the artifact, check its audit history, and you have a clear map of every pipeline and service that has consumed it. What might otherwise take days to piece together becomes a focused, time-bounded task.

Compliance Without the Scramble

For teams operating in regulated environments, audit trails aren't just useful, they're mandatory.

Harness Artifact Registry is built to support compliance with SOC 2, HIPAA, PCI-DSS, and other regulations out of the box. The audit trail covers the full artifact lifecycle, so when an auditor asks for evidence of access controls, artifact provenance, or quarantine procedures, you're not building that record from scratch. It's already there

‍

Access to artifacts is governed through Harness's role-based access control (RBAC). Harness Artifact Registry supports three pre-built roles, Viewer, Contributor, and Admin, allowing you to define who has permission to push, pull, or manage artifacts at the registry level. Every access event is logged against those permissions, giving you a traceable, defensible record of who was allowed to do what and what they actually did.

What the RBAC roles look like in practice:

These roles can be assigned to individual users, user groups, or service accounts. To check your current role assignments via the Harness CLI:

# List all role assignments for a specific registry project
harness role-assignment list \
  --account-id YOUR_ACCOUNT_ID \
  --org-id YOUR_ORG \
  --project-id YOUR_PROJECT

When Security Meets Auditability

One of the more distinctive things about Harness Artifact Registry is how tightly it integrates with the platform's security modules. Artifact Registry doesn't try to be a scanner or a policy engine on its own. Instead, it plugs directly into Harness Security Testing Orchestration (STO) for vulnerability scanning and Harness Supply Chain Security (SCS) for SBOM generation, policy enforcement, and compliance checks. The result is that security findings flow straight into the artifact record rather than living in a separate tool.

Through integration with Harness Security Testing Orchestration (STO) and Harness Supply Chain Security (SCS), every container image that lands in the registry can be automatically evaluated for vulnerabilities and compliance. If an image fails a security or policy check, can be quarantined automatically, before it can be consumed by any downstream pipeline.

Critically, all of that is auditable. Every quarantine action, every policy evaluation outcome, and every SBOM generated through SCS is logged, timestamped, and tied to the artifact in question. So your audit trail doesn't just tell you who accessed what; it tells you what the security posture of every artifact was at the time it was accessed.

Seeing it in action: On any artifact's detail page, you'll find dedicated tabs for security and supply chain data:

• SBOM tab - Full bill of materials including dependency lists, software suppliers, and package managers (requires the SCS module)
• Vulnerabilities tab - Security test results from STO scans (requires the STO module)
• Deployments tab - Which environments this artifact has been deployed to and how many instances are running (requires the CD module)

For example, to check the supply chain posture of a specific artifact version using the Harness API, you can query its chain of custody:

# Step 1: List artifact sources registered in SCS
curl -X GET 'https://app.harness.io/v1/orgs/{org}/projects/{project}/scs/artifact-sources' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Harness-Account: YOUR_ACCOUNT_ID'

# Step 2: Get the chain of custody for a specific artifact
curl -X GET 'https://app.harness.io/v1/orgs/{org}/projects/{project}/scs/artifacts/{artifact_id}/chain-of-custody' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Harness-Account: YOUR_ACCOUNT_ID'

The chain of custody shows every orchestration event (SBOM generation, policy enforcement, signing) that has occurred for that artifact, giving you a full security timeline.

Understanding Usage: What's Available Now

Beyond the transactional audit log, Harness Artifact Registry gives you practical tools to understand and manage how your artifacts are actually being used.

The Deployments tab on any artifact shows which environments it has been deployed to and how many instances are running, so you can answer "where is this version live?" without leaving the registry. Cleanup policies let you automatically remove artifacts based on age, usage, tags, or custom rules, keeping your registry lean without manual housekeeping.

You can also attach custom metadata to artifact versions, such as build IDs, Git commit SHAs, approval status, and environment tags. This makes it possible to query and reason about your artifacts in ways that reflect your actual workflow, rather than just generic registry metadata.

Custom metadata in practice: You can tag artifact versions with whatever context matters to your team. For example, after a build pipeline runs:

# Example: Harness pipeline step to push with metadata
- step:
    type: BuildAndPushDockerRegistry
    name: Build and Push
    spec:
      connectorRef: my_ar_connector
      repo: my-registry/my-app
      tags:
        - <+pipeline.sequenceId>
      labels:
        build-id: <+pipeline.executionId>
        git-commit: <+codebase.commitSha>
        branch: <+codebase.branch>
        approved-by: <+pipeline.triggeredBy.email>

Once metadata is attached, you can filter and search artifacts by these fields in the UI, making questions like "which version was approved for production?" answerable without digging through pipeline logs.

The Bigger Picture

What's worth stepping back to appreciate is that auditing and reporting in Harness Artifact Registry isn't a separate module or a dashboard you check once a quarter. It's woven into how the registry works. Every interaction generates a record, every security evaluation produces an insight, and all of it is accessible within the same Harness platform your pipelines already run on.

For teams that are new to centralised artifact management, this is one of the most immediately valuable things you gain. Not just control over your artifacts, but actual visibility into them.

And when the next CVE drops, that visibility is going to matter.

New to Harness Artifact Registry? Check out the quickstart guide to get up and running.

What are DevOps technologies?

DevOps technologies are the infrastructure that enables teams to ship code frequently, safely, and reliably. They automate repetitive work, provide visibility into system behavior, enforce governance, and give teams the confidence to deploy multiple times per day without breaking production. The best DevOps technologies aren't the fanciest, they're the ones that actually reduce toil and risk.

How has the DevOps tool stack and technology landscape changed?

2015: Point tools dominated. Teams picked a specialised tool for each stage: Jenkins for CI, Ansible for infrastructure, Splunk for logs, PagerDuty for incidents. Each tool had its own interface, permissions model, and failure modes. Teams owned 8 to 12 tools, context-switching was constant, and integration was manual.

2020: Consolidation began. Cloud-native tools emerged (Kubernetes, GitHub Actions, ArgoCD). Teams started asking: can we reduce tools instead of adding more? The problem: consolidation is hard, tools do not integrate cleanly, and switching costs are high.

2026: Platforms emerge. The trend accelerates. Teams adopt unified platforms that handle CI, CD, infrastructure, security, and observability in one place. The shift is economic: fragmentation costs more in toil and governance gaps than a unified platform costs in licensing.

The core shift: DevOps technologies used to be evaluated individually. Now they are evaluated as ecosystems, and the devops tool stack you build is as important as any individual tool in it.

How AI is reshaping DevOps technologies and the delivery lifecycle

AI coding assistants are changing the math. According to Harness research, 63% of organizations use AI tools like Copilot or Claude to write code. Code arrives faster, but it is also different: AI-generated code has different patterns, edge cases, and failure modes than handwritten code.

72% of organizations have experienced at least one production incident from AI-generated code. That is the AI Velocity Paradox. DevOps technologies must evolve to keep up. The testing, security scanning, and deployment gates that worked for handwritten code may not work for machine-generated code at scale.

DevOps technologies in the AI era need to focus on automated governance, fast rollback, and continuous observability. You cannot safely ship AI code without automated testing that catches AI-specific failure modes, security scanning that covers generated code patterns, deployment gates fast enough to match code volume, and rollback strategies that revert instantly.

What are the most important DevOps technologies and devops engineer tools to invest in?

Not all DevOps technologies are equally important. The devops practices and tools that earn a place in your stack cluster around three pillars.

• Continuous Integration and Delivery (CI/CD). This is where code gets tested and deployed. Invest in platforms that can scale with AI-generated code volume.
• Infrastructure as Code Management. Define infrastructure in code so it is version-controlled, auditable, and repeatable.
• Observability and AI SRE. You cannot manage what you cannot see. Real-time visibility into application performance, errors, and user impact is critical as code volume increases.
• Security testing. Embed scanning into the pipeline rather than bolting it on after. DevOps engineer tools that enforce policy-as-code at every stage prevent the governance gaps that incident postmortems trace back to.

DevOps best practices: evaluating devops practices and tools for your stack

Choosing well is itself one of the core DevOps best practices: the strongest stacks are built on a few deliberate decisions, not an ever-growing pile of tools. When evaluating DevOps technologies, ask five questions.

• Does it reduce toil? Can it automate the repetitive work that is consuming engineering time? If not, it is not worth the cost.
• Does it integrate with your existing stack? Can it work alongside the tools you already use, or does it require ripping out and replacing everything?
• Can it scale? Can it handle 10x more deployments, services, or teams without requiring a complete overhaul?
• Does it enforce governance? Can you apply policy-as-code, audit access, and maintain control across teams without manual coordination?
• What is the total cost of ownership? Factor in licensing, integration, training, and the fragmentation cost of choosing point tools.
  

How Harness approaches the DevOps technology strategy

The challenge

DevOps technology sprawl is real. Teams run 8 to 10 AI tools plus another 20 or more for the delivery pipeline. Each tool has its own logs, permissions, and failure modes. That fragmentation slows everything down: context-switching drains productivity, governance gaps create risk, and incident response is painful because no single tool has the full picture.

The approach

Harness consolidates the delivery platform with unified products: Continuous Integration, Continuous Delivery and GitOps, Infrastructure as Code Management, the Internal Developer Portal, Application Security Testing, AI SRE, and AI-native delivery. The unifying mechanism is the Software Delivery Knowledge Graph: an intelligence layer that connects code, commits, deployments, and outcomes into one source of truth.

The outcome

Teams consolidate their DevOps technology stack, reduce tool sprawl, lower governance risk, and operate at scale. The average team reclaims 10–15 hours per week previously lost to context-switching and integration work.

How have teams modernized their DevOps tool stack?

The shift from fragmented tools to a unified platform shows up in delivery metrics, not just in tooling inventories.

How did Vivun achieve a 300% improvement in DevOps engineer efficiency?

A SaaS company struggled with fragmented DevOps technologies. Each product feature required manual coordination across CI, CD, and infrastructure tools; engineers spent more time integrating than delivering. Consolidating onto Harness transformed the devops practices and tools the team used daily.

“Harness has been the catalyst for faster delivery and more DevOps engineers shipping higher-quality products every day.”

Jon Call, Engineering Manager for SRE, Vivun

Source: Vivun scales DevOps with Harness

How did Citi cut deployment lead time from days to 7 minutes?

A global financial services company with 20,000 or more engineers needed DevOps technologies that could scale and govern delivery across a heavily regulated environment. Their legacy approach was slow and fragile. Moving onto Harness CD reduced deployment lead time from days to minutes.

“Harness CD let us release each change within minutes of a pull request being merged.”

Stefanos Piperoglou, Technical Program Manager, Citi

Source: Citi improves software delivery performance with Harness CD

Different industries, same pattern: when DevOps technologies consolidate onto one governed platform, delivery gets faster and governance gets tighter at the same time.

Invest in the ecosystem, not just the tool

DevOps technologies in 2026 are not evaluated in isolation. The question is whether they reduce toil, integrate cleanly, scale without re-engineering, and enforce governance automatically. That is a different question from whether a tool is powerful. Many powerful tools fail this test because they add integration seams faster than they remove them.

The teams pulling ahead are not using the most tools. They are using the fewest well-integrated ones, built on shared governance, shared data, and shared audit trails. See how Harness brings the full after-code lifecycle onto one AI-native platform.

FAQs about DevOps technologies

Should teams build their own DevOps technology stack or buy a unified platform?

Building offers flexibility but requires ongoing maintenance, integration work, and engineering time. Buying a platform trades some flexibility for speed and governance. Most teams find that buying and customizing is faster and cheaper than building from scratch, particularly for the after-code delivery stages where the integration complexity is highest.

How do you migrate from current DevOps technologies to a new platform?

Migration is a process, not an event. Start with one team or one pipeline. Run it in parallel with existing tools for 2 to 4 weeks to validate. Once confident, gradually move other teams and pipelines over. A big-bang cutover creates risk; incremental migration lets you prove value and build confidence at each step.

What DevOps technologies does every engineering team actually need?

At minimum: CI (to build and test), CD (to deploy), infrastructure automation (to manage environments), and observability (to see what is happening). Everything else is additive. Start with these four and add based on actual pain, not theoretical coverage.

How do DevOps technologies relate to platform engineering?

DevOps technologies are the tools. Platform engineering is the discipline of using those tools to build internal platforms (golden paths, self-service workflows, software catalogs) that make developers more productive. They are complementary: platform engineering determines how the technologies are packaged and delivered to developers.

How often should teams reevaluate their DevOps technology choices?

Annually. Check whether tools are still reducing toil, whether the landscape has shifted, and whether new options would lower risk or cost. You do not need to rip and replace on every review; you need to stay aware of where your integration seams are creating the most friction.

What is the difference between DevOps best practices and DevOps technologies?

DevOps best practices are the principles (automate everything you can, measure what matters, ship small and often). DevOps technologies are the tools that put those principles into practice. Best practices without the right technologies rely on human consistency. Technologies without the right practices create automation of the wrong things.

Ensuring Security in Feature Flagging and Experimentation

With the success of feature flags and experimentation widely touted by influential companies (like AirBnb and Netflix), it would seem that the benefits of nurturing an experimentation culture are a no-brainer. Surely any downsides would be far outweighed by the great insights gained into user behavior, the measurable metrics observed, and the precise monitoring together with exquisite fine tuning and orchestration of feature releases. The whole concept brims with safety.

Why wouldn’t it?

Because, among other principles, software security centers on knowing your technology (what it does and the relevant security requirements) and reducing your attack surface. The Harness platform was built with security by design from the get-go, and there are many interesting articles you can look up like this one about security in the Harness CI/CD pipeline.

For this post, we will hone in on Harness Feature Management and Experimentation (FME), specifically on two vital keys for managing your flags securely:

• FME Thin and standard client-side SDKs: the choice between local and remote feature flag target evaluation
• The Harness FME flag management pipeline steps, including the Feature Flag Cleanup AI agent

These are newly released security features that the Harness FME team is proud to celebrate. This blog explains what these mean for you. Let’s dive in.

Local and Remote Flag Evaluation

Protecting the privacy of your users means knowing where private data resides, and reducing its movement between software components.

Most security models are satisfied when private data stays and stops at the mobile or web client. Any sensitive user attributes (like PII, business context, or session data) never leave the user device. This scenario describes the concept of local feature flag target evaluation, the security premise all client-side standard FME SDKs are built on.

It may be your case, however, that feature flag targeting rules themselves are sensitive. You may rightly not trust the client with them (here’s an elaborative story that illustrates this point and another compelling story about the impossibility of securing the client). If that is your scenario, you can choose remote feature flag target evaluation. User attributes travel over the network to the Remote Evaluator in FME cloud and feature flag treatments are returned. Remote evaluation is invoked by all client-side thin FME SDKs. The flag definitions and targeting rules never leave the FME servers.

Takeaway: While most client-side surfaces are well served by local evaluation, which protects user attributes, remote evaluation is the right tool when those targeting rules themselves carry information that has to stay private. A common pattern is to keep most of the application on the standard SDK and reserve the thin SDK for the surfaces where rule visibility is sensitive. This detailed technical analysis can help you decide on your own strategy.

Local or remote evaluation—now you have the choice. It’s our pleasure to be giving you the key.

Feature Flag Lifecycle and the Cleanup AI Agent

Flags let you catch disasters before they happen. In the early stages of a canary release, triage for a faulty feature variant is as trivial as opening an alert and flicking a kill switch.

With this perception of safety and visibility, it might seem like a convenience to reuse old flags, but that would be a major malpractice (read: very very Bad, with a capital B).

There are many reasons why informed people strongly discourage flag reuse. Besides confusing flag purpose, undermining intra- and inter-team communication (getting colleagues and management mad), and deliberately ignoring the very common reality of long-delayed application updates/old code still active (especially on mobile devices, but also on a server as in this $460 million mistake); flags not removed in a timely manner, once the golden feature variant has been measured and identified, leave over unnecessary code complexity. When everyone has forgotten the unneeded variant paths (instead of removing them), we are left with encumbering tech debt.

Better to keep code simple, and follow good flag lifecycle practices that Harness FME fully supports by providing:

• OPA policies for feature flags
• Feature flag pipeline steps
• Feature Flag Cleanup AI agent

OPA Policies for Feature Flags

OPA policies allow you to enforce standardized naming practices for your feature flags project-, organization-, or account-wide in Harness. Keep the intent of each flag crystal clear to all engineers and stakeholders.

Feature Flag Pipeline Steps

Pipelines, built from standardized pipeline templates, can manage the entire flag lifecycle by using automated Harness pipeline steps for feature flag management. These take a feature flag through smoke and beta testing, canary release, monitored ramping, full GA, removal from code, archival, and deletion. More on removal from code, the feature flag cleanup step, in the next subsection.

Feature Flag Cleanup AI Agent

Harness worker agents are now available in Harness Marketplace and ready to be configured with an AI model of your choice. The Feature Flag Cleanup AI agent runs in a pipeline step that safely removes references to a stale feature flag, keeps the chosen treatment, and commits the cleanup to a given branch in the code repo.

Add the agent step to your pipeline, and the agent will search through your code for feature flag evaluations and conditional execution paths and create a commit to remove these from your code base, leaving just your chosen variant path. You can see in FME when a flag has not received traffic in the past week or month, signaling that the flag can safely be killed, archived, and deleted in Harness FME.

How to add the Feature Flag Cleanup AI Agent step to your pipeline:

1. Create or open a pipeline in Harness.
2. Create or click on a supported stage type (or a containerized step group) for Agent steps.
3. Add the Feature Flag Cleanup agent step to the step group.

1. Configure the LLM Connector, AI model, flag name and code repository parameters.

After the step runs successfully, your team can review the changes and merge the branch. Once the new code is deployed and the flag is no longer receiving traffic, you can safely kick off archival and deletion of the flag in Harness FME. 

Feature flag lifecycle management is automated within your governed pipeline.

Takeaway: Harness enforcement of your naming policies and native-to-Harness feature management pipeline steps allow you to ensure your flags are not reused. Ever.

You have the key tooling in hand to successfully govern and cleanly complete your feature flag lifecycles, and you can effectively do this across your organization.

More Security Best Practices - Completing the List

The focus here was on feature flag security. There is much more on Harness pipeline security, including:

• Security testing orchestration (STO) and application security posture management (ASPM)
• Role-based access control (RBAC) for all Harness resources (including feature flags, but going way beyond)
• Web Application & API Protection‍
• AI Security

We help you to shift security left in your SDLC, secure your CI/CD pipeline, and shield your application in production.

Harness has always put security first by design. Security is our priority. We are proud of that legacy, and we empower you to do the same.

Frequently Asked Questions

Why does security need to be considered when using feature flags?

Feature flag targeting rules or user attributes may contain sensitive information that can be protected by a well-chosen SDK evaluation mode. Flag lifecycle management is best automated and standardized across your teams, including clear flag naming conventions and well-supported cleanup practices.

You may also be interested in learning about setting up a proxy and reverse proxy to harden your network boundaries while allowing FME traffic through.

What tools does Harness provide to strengthen the security posture of my CI/CD pipeline?

Security is a top priority at Harness, and we are rapidly innovating new security tools, integrations, and partnerships. Some of these innovations are listed below:

• Static Application Security Testing (SAST)
• Software Composition Analysis (SCA)
• Supply Chain Security (SCS)
• Security Testing Orchestration (STO)
• Web Application & API Protection‍
• AI Security

How does Harness support you shift left and shield right in the SDLC?

Shifting left means incorporating security best practices early in the software delivery life cycle (SDLC), most preferably in the development process. This approach helps developers identify and fix vulnerabilities as early as possible and reduce costly mistakes and security patches.

That's what the Harness platform does today. Harness brings Application Security Testing directly into the development workflow, surfacing vulnerabilities where they're faster and cheaper to fix, while Supply Chain Security ensures the integrity of artifacts from build to deploy.

As code ships to production, Web Application & API Protection monitors and defends applications and APIs in real time, detecting and blocking attacks as they happen. And critically, findings in production don't disappear into a security team's backlog—they flow back to devs and engineers to remediate issues before the next release.

The result is a closed loop: find vulnerabilities early in code, protect applications in production, resolve incidents lightning fast. All on a single, unified platform.

How is security changing in the AI era?

AI-assisted attacks are surfacing in the AI era. Apple is responding to the changing technology landscape by stepping up security in an unprecedented policy change, in recognition of the security impacts of AI. Harness recently released AI Security to address the security gap that can exist with AI assisted coding.

Where can I learn more about best practices for securing my Harness CI/CD pipeline?

You can keep your knowledge current by watching the Harness Testing & Compliance blogs; the Harness Application Security Testing, Web Application & API Protection, and AI Security product pages; and organizations like Open Worldwide Application Security Project (OWASP) that regularly publishes authoritative Top 10 lists of critical security risks.

What is a DevOps toolchain?

A DevOps toolchain is the connected set of tools your team uses to move software from code to production: source control, CI/CD, security testing, IaC, and observability. DORA 2025 finds that elite teams deploy 182x more frequently than low performers; the difference is not more tools but fewer, better-integrated ones with shared governance.

‍
A new engineer joins a team and asks what is a DevOps toolchain. What comes back is a 22-line inventory: a source host, two CI systems, an IaC engine, a registry, three scanners, a deployment tool, a couple of dashboards, and nobody who can fully explain how they all connect. That inventory is the team's DevOps toolchain, and its length is often mistaken for its strength.

A DevOps toolchain is the set of tools spanning the software delivery lifecycle (source, build, test, security, deployment, and operations) that a team assembles to move software from code to production safely and reliably. A useful DevOps toolchain covers every stage with as few disconnected tools as possible. The goal is not the longest list. It is the smallest unified stack that lets teams ship faster and safer.

Why isn't the longest DevOps toolchain the best one?

Every team needs certain tools: source control, CI, CD, security testing, observability. But the instinct to add a specialised tool for every edge case is what creates sprawl. A team running GitHub, Jenkins, CircleCI, ArgoCD, Terraform, Atlantis, LaunchDarkly, Snyk, Datadog, and PagerDuty is not well-equipped. It is fragmented. Each tool owns its logs, its access model, and its failure modes.

The real cost of a long DevOps toolchain is not the tool licenses. It is the integration toil, the governance gaps, the constant context-switching, and the developer time spent chasing approvals instead of shipping. Harness research (State of AI in Software Engineering 2025) shows 71% of teams say context-switching between tools drains productivity, and 73% of engineering leaders report barely any teams have standardized golden paths.

A DevOps toolchain that scales is not the one with the most tools. It is the one where adding the hundredth team costs about what adding the tenth did, because the path is standardized and governed centrally, not rebuilt each time.

What DevOps automation tools, CI/CD automation, and devops toolchain list do you need?

A functional DevOps toolchain covers these stages. Each stage has multiple options, but the principle is the same: choose DevOps automation tools that integrate well, then consolidate the integration points.

The categories matter less than the integration. A CI tool that shares a policy layer with your CD and GitOps platform and security testing stages is more valuable than three separate tools with no shared context. CI/CD automation is the backbone, but the value compounds when security, cost, and reliability share the same governance layer. The Internal Developer Portal is what surfaces these as golden paths developers self-serve on, rather than ticket queues they wait on.

How does AI change your DevOps toolchain requirements?

AI coding assistants changed the production rate. Developers now produce code significantly faster, and organizations ship faster as a result. But the DevOps toolchain that has to test, secure, and ship that code did not accelerate at the same rate. That mismatch is the AI Velocity Paradox: the build queue grows, the deployment queue grows, the surface area for security scanning expands.

A DevOps toolchain that worked fine for 50 commits a day falls apart at 500. The solution is not to add more tools. It is to consolidate the ones you have so governance, verification, and rollback stay consistent as volume increases.

Teams using AI coding tools most heavily have the highest remediation rates (22%) and longest mean time to recovery (7.6 hours), according to the Harness 2026 State of DevOps Modernization. That is not a tool problem. It is a governance and integration problem.

How Harness simplifies DevOps toolchain consolidation

The challenge

Platform teams are asked to give developers fast, self-service delivery while maintaining governance and reliability. As AI accelerates code output and tools accumulate, the after-code stages (testing, securing, deploying, operating) fragment across products with no shared context or governance. The platform team ends up maintaining integration seams instead of improving delivery.

The approach

Harness is the AI-native Software Delivery Platform that automates and governs everything after code is written. The Software Delivery Knowledge Graph ties each build, deployment, and security event back to the service and commit it came from. On that foundation sit the after-code modules: Continuous Delivery and GitOps, Continuous Integration, the Internal Developer Portal, Infrastructure as Code Management, Application Security Testing, AI SRE, and Cloud and AI Cost Management. Each inherits shared access control, governance, and a single audit trail. Developer-friendly guardrails.

The outcome

Consolidating the after-code stages onto one governed platform reduces governance gaps, accelerates remediation, and cuts the developer toil that sprawl creates. Teams can ship faster and safer as they scale, and adding new teams or services does not require rebuilding the entire DevOps toolchain. See how teams have simplified their toolchains.

How have teams simplified their DevOps toolchains?

Two teams, two different sprawl problems, one pattern: consolidation returns engineering time to the work that requires judgment.

How did Ancestry go from 80 Jenkins instances to governed CI/CD at scale?

Ancestry managed over 80 distinct Jenkins instances: one per team, with no central governance. Consolidating onto Harness let them apply a single pipeline change across all teams instead of editing each instance by hand. The result: an 80-to-1 reduction in pipeline implementation effort, 50% fewer deployment-caused outages, and a 78% reduction in systems-onboarding toil.

“Harness now enables Ancestry to implement new features once and automatically extend those across every pipeline, representing an 80-to-1 reduction in developer effort.”

Ken Angell, Principal Architect, Ancestry

Source: Ancestry adds consistency and governance to cut downtime

How did a UK software company cut manual DevOps tickets by 80 to 90%?

A UK-based software company relied on manual, ticket-based access requests for GitHub, Copilot, and AWS, creating a continuous bottleneck for a small DevOps team. Adopting the Harness Internal Developer Portal turned that manual overhead into self-service workflows with guardrails. Priority projects onboarded in weeks instead of months; the DevOps team refocused on higher-value work.

“We have reduced tickets by 80 to 90%. What took a full-time team to manage manually is now done automatically with appropriate guardrails.”

Principal DevOps Architect, enterprise software company

Source: Enterprise software company reduces DevOps tickets by 80%

Consolidate the DevOps toolchain: CI CD automation and beyond

The best DevOps toolchain is not the longest devops toolchain list. It is the one where fewer, well-integrated DevOps automation tools replace fragmented point solutions, and where adding the hundredth team costs about what adding the tenth did. Start from the governance gaps: find the stages where your audit trails break, where approvals wait on a human, where a deploy needs someone watching a dashboard. Those are the integration seams worth removing.

A unified platform covering the after-code lifecycle with shared governance, golden paths, and AI-native automation is how teams absorb AI-generated code at machine speed without losing control of what ships. 

See how Harness brings the full after-code lifecycle onto one platform.

FAQs about the DevOps toolchain

What is a DevOps toolchain?

A DevOps toolchain is the connected set of tools spanning the software delivery lifecycle: source control, CI, artifact management, security testing, deployment, and monitoring. The goal is not the longest list but the smallest unified stack with shared governance that lets teams ship faster and safer.

What is the difference between a DevOps toolchain and a CI/CD pipeline?

A CI/CD pipeline automates build, test, and deployment. A DevOps toolchain is the broader set of tools spanning planning, coding, security, operations, cost, and reliability. Every pipeline lives inside a toolchain, but a toolchain covers stages a pipeline alone does not.

How many tools should a DevOps toolchain include?

Fewer well-integrated tools scale better than many loosely connected ones. The average team runs 8 to 10 AI tools and up to about 30 across the full SDLC. The goal is sufficient coverage with minimal integration seams and one governance layer across all of them.

What is a golden path, and why does it matter for a DevOps toolchain?

A golden path is a pre-approved, standardized pipeline template that lets teams self-serve within guardrails. New teams onboard onto a consistent, governed process instead of rebuilding their own. 73% of engineering leaders report barely any teams have golden paths, which is the clearest signal of toolchain sprawl.

How do you consolidate a DevOps toolchain without losing specialized capabilities?

Unified platforms consolidate the after-code stages (CI, CD, security, cost, reliability) while maintaining specialized capability in each. The goal is removing integration seams, not eliminating tools you actually need. If a tool solves a real problem and integrates cleanly, keep it. If it adds governance gaps, it is a candidate for consolidation.

How does CI/CD automation fit into a DevOps toolchain?

CI/CD automation is the backbone of the DevOps toolchain: it connects the build, test, and deploy stages into a repeatable flow. The value compounds when CI/CD shares a policy engine and audit trail with security, cost, and reliability tools, rather than running as an isolated pipeline.

What is software release management?

Software release management is the set of practices, tools, and governance that moves code safely from development into production through defined stages (CI/CD, approval gates, progressive deployment, and rollback). DORA 2025 research finds that elite teams recover from failures 24x faster than low performers; release management discipline is the separator.

A software release is a moment. Software release management is the process that leads to it. It spans planning, testing, approvals, deployment, monitoring, and rollback: every controlled step between code and production.

Software release management is the set of practices, tools, and governance that ensures code moves safely from development into production, with clear stages, approval gates, verification checkpoints, and a rollback strategy. The goal is to reduce risk, accelerate delivery, and give teams confidence that they can ship at any time without breaking production.

In practice, release management means your team has a defined process, code does not go to production without approval, you test before release, you can verify that a release is working, and you can roll back quickly if it does not.

What are the stages of the software release pipeline and management process?

Every release follows a path through your release pipeline. The stages differ by organization and risk tolerance, but the pattern is consistent: prepare, validate, approve, deploy, monitor, and be ready to revert.

What is the difference between deploying and releasing software?

These terms are often used interchangeably, but they mean different things.

Deployment is a technical action: moving code from one environment to another. You can deploy code to staging, to a canary, to 5% of users, or to your data center. Deployment is infrastructure-driven.

Release is a business decision: making a feature or fix available to end users. You can deploy a feature without releasing it (using feature flags), or release a feature that was deployed days ago. Release is decision-driven.

In practice: you can deploy rapidly, but releases should be deliberate. That is why feature flags and experimentation have become essential software release tools in modern release management: they let you decouple deployment from release, verify before exposure, and roll back without redeploying.

Why does the AI Velocity Paradox make software release management harder?

AI coding assistants are accelerating code production. Developers using tools like GitHub Copilot write code 63% faster. That is a win until your release pipeline cannot keep up. According to Harness research, 72% of organizations have experienced at least one production incident from AI-generated code. That is the AI Velocity Paradox: faster code, but the safety gates did not accelerate with it.

The math is simple. If code is produced 2x faster but testing and approval stay the same speed, the queue grows, and either releases slow down or safety checks start to skip. Release management becomes the bottleneck.

Key insight: The solution is not to slow down code production. It's to automate your release gates so they can process more code safely, faster.

The best release management tools, software release tools, and software release platform options

Strong release management looks the same everywhere: automation where possible, human judgment where it matters, and speed without recklessness. The right software release platform enforces that discipline.

• Automate testing and validation gates. Run tests automatically before any release; do not let a human forget. Make approval gates automatic too: if a release meets policy, do not wait for a person to click a button.
• Make rollback easy and fast. If a release breaks production, teams should be able to revert in seconds, not hours. Design your release process so rollback is a one-button action.
• Use feature flags for safer releases. Release code to production but keep features off. This gives you time to verify before users see the change, and lets you roll back instantly without redeploying.
• Monitor actively from day one. Do not wait for users to report problems. Watch error rates, latency, and business metrics immediately after release; catch issues in minutes, not hours. AI SRE tools can automate this correlation and remediation.
• Follow the DORA metrics. DORA (DevOps Research and Assessment) tracks four key metrics: deployment frequency, lead time for changes, change failure rate, and mean time to recovery. Strong release management improves all four.
  

How Harness approaches software release management

The challenge

As AI accelerates code production, teams face a choice: slow down releases to maintain safety, or ship faster and accept higher incident rates. The real problem is that release management is fragmented. Testing happens in one tool, approvals in another, deployment in a third, and monitoring in a fourth. That fragmentation slows everything down and creates the governance gaps that incident postmortems trace back to.

The approach

Harness offers a unified software release platform that manages the entire release process: from automated testing through approval gates, deployment strategies, and rollback. It integrates with Continuous Integration so testing happens first, then the Internal Developer Portal for governance and golden paths. The Software Delivery Knowledge Graph ties each release back to the code, the tests, and the business outcome. Feature Management and Experimentation decouples deploy from release. AI SRE monitors and remediates automatically.

The outcome

Teams consolidate release management onto one governed platform, which reduces cycle time, lowers change failure rates, and gives teams confidence to ship faster. Automation handles the routine gates; teams focus on the decisions that matter. Hundreds of engineering teams trust Harness to govern their release processes at scale.

How have teams mastered software release management?

The evidence shows up in delivery metrics, not just in tooling decisions.

How did The Warehouse Group cut release lead time from 120 hours to one hour?

The Warehouse Group, a New Zealand retail enterprise, had a manual release process: approvals were slow, testing was inconsistent, and incidents took hours to roll back. Moving onto Harness CD gave developer squads on-demand deployment with governance enforced through the pipeline. Lead time for changes dropped from 120 hours to 1 hour, a 99% reduction.

“We saw lead time for changes decrease from 120 hours to 1 hour by using Harness as a key part of our path to production. This gain in efficiency is key to supporting our business goals.”

Matt Law, DevOps Chapter Lead, The Warehouse Group

Source: The Warehouse Group reduces change lead time by 99%

How did Ancestry govern release management across 80-plus pipelines?

Ancestry managed a decentralized release process: each team owned its own pipeline with different standards and approval processes. Consolidating onto Harness let them apply a single pipeline change across all teams instead of editing each instance by hand. The result: 50% fewer deployment-caused outages and a governed release process across all teams.

“Harness now enables Ancestry to implement new features once and automatically extend those across every pipeline, representing an 80-to-1 reduction in developer effort.”

Ken Angell, Principal Architect, Ancestry

Source: Ancestry adds consistency and governance to cut downtime

Build a release management process that keeps pace with AI

Software release management is not a bureaucratic layer on top of shipping. It is the mechanism that makes fast, confident shipping possible. As AI tools push more code through your pipeline, the teams that pull ahead are the ones that automated their release gates before the volume arrived.

The components are the same everywhere: a clear release pipeline with defined stages, automated approval gates, feature flags that decouple deploy from release, live monitoring tied to rollback, and DORA metrics that tell you whether it is working. The software release platform you choose determines how much of that you can automate, and how fast you can move when something goes wrong.

FAQs about software release management

How often should a team release?

Deployment frequency depends on risk tolerance and product type. Many successful teams release multiple times per day; others release weekly. The key is that you can release confidently at your chosen cadence without increasing incident rates. DORA metrics are the benchmark: elite teams deploy on-demand.

What is the difference between a release manager and a DevOps engineer?

A release manager owns the release process: planning, approval gates, communication, and rollback decisions. A DevOps engineer builds the infrastructure that makes releases automated and safe. Both roles are essential, though in many teams the responsibilities overlap and are handled by the same person.

How do feature flags help with software release management?

Feature flags let you deploy code without releasing it. You can deploy a new feature to production but keep it switched off, then turn it on gradually (to 1% of users, then 10%, then everyone). If something breaks, you switch it off without redeployment needed. This separates deploy risk from release risk.

What happens if a release goes wrong?

That is what rollback is for. If errors spike or users report problems, you should be able to revert to the previous version in seconds. This is why fast rollback is a non-negotiable best practice, and why automated continuous verification (which catches problems before they reach users) is equally important.

How do you balance speed with safety in software release management?

Automate everything you can: testing, approval gates, deployment verification. Reserve human judgment for the decisions that matter. Automation handles the routine; humans focus on strategy. Teams that automate their release gates first are the ones that can safely absorb faster code production from AI coding tools.

What is a release pipeline and how does it differ from a CI/CD pipeline?

A release pipeline is the end-to-end flow from code merge to production, including approval gates, deployment strategies, and rollback. A CI/CD pipeline is the build-and-deploy automation inside that flow. The release pipeline is broader: it includes the governance, verification, and rollback layers that CI/CD alone does not cover.

• AI assets (prompts, skills, agents, plugins, commands) are sprawling unmanaged across repos. Harness IDP's AI Asset Catalog brings them into the same governed catalog as existing software components (APIs, environments, etc.).
• Four core capabilities: Git-driven auto-discovery, semantic/natural-language search, lineage & ownership mapping, and automated scorecards for risk/compliance.
• Benefits differ by role: developers get reusable, verified building blocks; platform teams get a single source of truth; security/governance teams get automated policy enforcement instead of chasing shadow AI.

‍

Here is a story platform engineering teams know by heart: developers find a shiny new tool, start building at a breakneck pace, and before you know it, the organization is drowning in a massive wave of unmanaged components.

Right now, that exact story is playing out with generative AI.

Developers are spinning up prompts, skills, agents, plugins, and custom commands faster than anyone can keep track. They are forking them, tweaking them, and quietly dropping them across dozens of scattered repositories. Sure, some of them work. But many of them carry real operational and compliance risks. And almost none of them can be found by the next engineer who needs the exact same thing. So everyone starts from scratch which leads to redundancy, wasted effort, and unnecessary complexity.

The reality is that we are looking at a classic case of sprawl, just with a fresh coat of AI paint.

That is exactly why we built the AI Asset Catalog in Harness IDP. We have spent the last few months baking these capabilities directly into our internal developer portal catalog, elevating AI Assets to a first-class entity right next to your standard components, APIs, and environments.

Why an AI Catalog belongs in your developer portal

There is an understandable temptation to treat AI components like they are some kind of alien technology that requires a completely bespoke tooling stack. I would argue the exact opposite.

The fundamental reasons a software catalog exists do not change just because a component uses a large language model. You still need to answer three basic questions: What do we have? Who owns it? Is it safe to use?  Those core questions apply to an AI skill or an autonomous agent just as cleanly as they do to a traditional microservice.

By placing AI assets inside the same developer portal your teams already use, they automatically inherit your existing software governance model. You do not have to stand up, secure, and maintain a separate control plane. Because the AI Asset Catalog runs natively on the Harness platform, your AI components are instantly scoped by your granular role-based access control, and changes are logged in your immutable audit trails.

This unified control plane becomes incredibly important as autonomous agents start acting on your production systems. Through the Harness MCP Server, external coding assistants can already safely discover ownership and platform standards directly from your catalog. The AI Asset Catalog simply extends that exact same auditable model to the very building blocks those agents are built from.

A closer look at the AI Asset Catalog capabilities

The AI Asset Catalog automatically indexes, maps, and scores your internal AI components to make them instantly discoverable. We focused on four core capabilities to keep things simple and highly scannable:

Automated, Git-driven discovery

Manual cataloging is where good ideas go to die because nobody has the spare cycles to keep documentation current. That is why discovery is entirely driven from where your engineers actually live: source control.

With a simple toggle via our GitHub integrations, Harness automatically ingests, de-duplicates, and maps AI assets straight from your repositories. There is no manual upload step and no parallel registry to baby-sit. When an asset changes in Git, the catalog updates in lockstep.

Intelligent content ingestion and semantic search

Simply indexing a text file is not the same as actually understanding its purpose. Harness AI reads and parses the artifacts that describe your assets, including instruction files, agent.md profiles, and READMEs, to interpret exactly what a component does.

This deep parsing powers an intuitive natural language search. Instead of playing keyword guessing games, a developer can type a plain question like, "Is there an approved skill to analyze my codebase?" The portal instantly surfaces verified items along with cleanly formatted execution constraints, meaning teams can understand the intent and health of an asset before they ever decide to consume it.

Lineage and team ownership mapping

Modern AI architectures are highly compositional. A single plugin bundles multiple skills, an agent triggers specific commands, and a command relies on a highly tuned prompt. When those invisible links break, debugging turns into an absolute nightmare.

The catalog visually charts these parent-child relationships automatically. It maps precisely how prompts and skills roll up into specific plugins, while enforcing explicit team ownership. When an asset misbehaves, you do not waste hours on a wild goose chase; you immediately know the exact blast radius and the exact team to page, slashing triage and support times.

Automated AI asset scorecards

Enabling developer reuse is fantastic, but it is only safe if you can separate reliable, compliant assets from experimental code. Scorecards bring our established software maturity and governance patterns straight to the AI playground.

Our out-of-the-box checks evaluate every single AI asset against essential dimensions: structural integrity, risk maturity, confidence levels, popularity, and data classification compliance. Out-of-policy components are flagged proactively, stopping compliance violations before they escape into production environments. Because these scorecards hook into our broader platform reporting, engineering leaders get a true company-wide view of AI maturity without a separate reporting headache.

Who feels the difference?

The value of a centralized AI catalog looks a bit different depending on your day-to-day role:

• Application Developers: You can stop wasting hours rewriting a prompt or an execution skill that a peer team down the hall already perfected. Natural language search gives you verified, ready-to-use building blocks with clear operational guidelines, making reuse the path of least resistance.
• Platform and DevOps Engineers: You finally get a single pane of glass to track exactly what agents and plugins exist across the enterprise. You can replace chaotic AI sprawl with a clean, searchable source of truth that defines clear ownership and code dependencies.
• Security and Governance Leaders: You can trade your late-night anxiety about shadow AI components and unverified code leaks for automated guardrails. Scorecards proactively flag out-of-policy items and seamlessly enforce your corporate risk matrices.

The big picture

The AI Asset Catalog is not just a shiny standalone tool. It is a foundational part of our goal to make Harness IDP the definitive control plane for both human developers and autonomous AI agents.

Google's DORA research regularly reminds us that while AI code generation tools are making coding faster, actual software delivery throughput remains stubbornly flat because teams get bogged down in downstream execution, testing, and security bottlenecks. Only about 30% of engineering time is spent actually writing code. We want to fix that chokepoint across the entire lifecycle.

Simply put, the catalog handles the question of what assets you have and whether they are safe to use. Our Knowledge Agent assists engineers by executing complex workflows, and our MCP Server grounds external LLMs in your internal architecture and governance standards. Underneath it all sits the exact same secure, auditable platform you already trust to ship code safely every single day.

If you are already running Harness IDP, getting started is incredibly straightforward. You just plug in your existing GitHub or Bitbucket repositories, turn on automated discovery, and watch the catalog map out your AI ecosystem. From there, you can roll out scorecards, assign clear team ownership, and let your developers innovate with total confidence.

‍

FAQs

How does the catalog discover AI assets — do I need to manually upload anything?

No manual step is required. It ingests directly from source control (GitHub/Bitbucket) and stays in sync automatically as Git repos change.

What kinds of AI components does it track?

Prompts, skills, agents, plugins, and custom commands, including their parent-child relationships (e.g., which skills roll up into which plugins).

How does it decide if an asset is safe or compliant to use?

Through automated scorecards that check structural integrity, risk maturity, confidence, popularity, and data classification, flagging out-of-policy assets before they reach production.

Does this require a separate governance or security tool?

No, it inherits the existing Harness platform's RBAC, Open Policy Agent policy layer, and audit trails, so there's no new control plane to stand up.

How is this different from the Harness MCP Server or Knowledge Agent?

They're complementary: the catalog answers "what assets exist and are they safe," the Knowledge Agent executes workflows, and the MCP Server grounds external LLMs/coding assistants in your internal architecture and standards.

• Agent development today is too risky. Only ~8% of organizations have gotten agentic AI into production. Most of the rest are stuck deploying agents by hand or with one-off CI jobs. This slows teams down and, for those who push forward anyway, leaves no record of who deployed what or how its quality was evaluated.
• Deployment is where that risk actually lives. Building an agent has gotten easy; getting it into production safely hasn't. Fixing that starts with the deployment step itself: predictable, repeatable, and owned by the platform.
• Harness makes agent deployment safe and easy. Agent Deployments in Harness CD brings tested, out-of-the-box pipeline steps for platforms like Google's Agent Runtime and Amazon Bedrock AgentCore, so teams get governed, auditable agent releases without hand-rolled scripts, using the RBAC, policy engine, and audit trail they already trust for the rest of their software delivery.
• Agents don't ship alone. Harness coordinates the agent alongside what it depends on: backend services, database changes, and prompt or model config. There’s one release governed by one tool.

‍

Overview

Teams building agents have converged on something that looks a lot like the software development lifecycle, but reshaped around a system whose output isn't deterministic: prototype an agent against a framework, evaluate it against a dataset of expected behavior, deploy it somewhere real, observe how it behaves against live traffic, and feed what you learn back into the next prototype. Call it the agent development lifecycle (Agent DLC). 

Most of that lifecycle borrows tooling that already existed - a framework like LangGraph or Google's ADK for the prototyping stage, an eval platform for the evaluation stage. This post is about one stage of that lifecycle: deployment, and the decisions behind how we help users deploy their agents reliably with Harness Continuous Delivery.

Because deployment is when an agent shifts from being safely under test to being exposed to production, decision-making and safety are critical. Organizations need to ensure that only good versions of agents are actually released, policies are adhered to, and a dependable audit trail is created. 

Further complicating agent deployments is the fact that agents rarely stand alone and are often updated alongside changes to data, configuration, front ends, and companion services. Deploying an agent is not enough. We have to orchestrate its changes with everything else in a release.

This article covers how agent deployments are different, how to govern them, and what release orchestration with agents looks like. In short, how to make agent deployments both safe and easy.

Where deployment sits in the agent lifecycle

Something genuinely new shows up in the deployment stage of the Agent DLC. Rather than packaging an agent as a generic container and hosting it the way any other service gets hosted, AWS and Google both shipped purpose-built, managed runtimes for agents specifically - Bedrock AgentCore and GCP Agent Runtime. 

Unlike a general compute product with an agent tutorial bolted on, these runtimes are shaped around what an agent actually needs - session and memory primitives, identity scoped to the agent rather than the pod, versioned "runtime revisions" instead of arbitrary deploys.

That's the piece that benefits from dedicated automation support. It's worth spending a minute on what changes when the target is one of these runtimes instead of a Kubernetes cluster, before getting into what we actually decided. The shift to managed runtimes means teams either build this operational muscle themselves or get it from a platform.

Kubernetes vs. Native agent runtime: what actually changes

None of this is an argument that Kubernetes is the wrong place to run an agent - plenty of teams will keep doing exactly that, and it's on our roadmap as a deployment target for this same agent-service model. It's a genuinely different set of trade-offs, not a strictly better or worse one.

This operational surface (session stores, readiness probes, traffic routing) is exactly what a native runtime (and what Harness's deployment step) absorbs for the team. The first two rows are the reason a native runtime exists at all - session and memory management is genuinely hard to get right underneath an agent, and both clouds decided it was worth building once, centrally, rather than leaving every team to rebuild it next to their pod. 

The rows below that are the downstream consequence: once the platform owns session, memory, and isolation, it ends up owning versioning and traffic control too, because those all have to agree with each other underneath.

The trade Kubernetes gives up in exchange for that control is exactly the operational surface a native runtime absorbs for you: you're not sizing replica counts, standing up your own state store, or writing readiness probes for something whose "readiness" is closer to a language-model call than a TCP health check. Whether that trade is worth it depends entirely on how much of that control a given team actually wants to keep exercising, which is the real reason we're not treating native runtimes as the only supported target going forward.

A working example: the Academic Research Agent

Let's take an example of an Academic Research Agent - a LangGraph agent that searches academic papers and journals, synthesizes findings across sources, and drafts a literature-review section for a researcher to approve. It's been working in a notebook. Getting it live means three things: register it as a service, define where it runs, and put a pipeline in front of it that can promote it safely.

Deployable Entity - The Agent Service

Does a customer think about their Academic Research Agent as one thing, or as two different things depending on which cloud it happens to run on? We bet on one thing. An agent's name, its config and secrets, its purpose - those don't change depending on where it's deployed. This eliminates the need to maintain multiple deploy scripts for what is functionally one agent.

What changes is the shape of the cloud underneath it: the image reference and agent framework on Google's side, the execution role on AWS's. So the service definition keeps a single outer identity with the cloud-specific pieces contained inside it, rather than asking someone to maintain what is functionally the same agent as two separate service definitions.

We know this model works well. Our existing Kubernetes deployment type has separate infrastructure kinds per cloud (GCP, Azure, direct) underneath a single deployment type. What's different here is applying that same idea one layer higher, at the service itself, because what defines an agent - its name, purpose, and configuration - doesn't change across clouds, even though the infrastructure underneath it does get changed.

We have registered our Academic Research Agent as an Agent Service on Harness. Now, the question is about the target platform configuration, which involves defining your infrastructure.

Infrastructure: What does the agent need on a specific cloud?

The infrastructure definition is where the cloud-specific configuration lives. For the Academic Research Agent on AWS AgentCore, that means:

• A connector Harness uses to call AWS's deploy APIs
• A region, and the VPC subnets and security groups the agent should run inside
• A reference to an AgentCore Gateway - required only if traffic shifting matters

The Gateway is the one worth pausing on. It isn't automatically part of an AgentCore deployment - it's additional infrastructure the team provisions up front, specifically so traffic shifting has something to act on. If the Academic Research Agent's infrastructure skips it, deployments still work; every promotion is just a direct cutover instead of a gradual one, because there's nothing underneath to hold a partial split.

If the target is Google's Agent Runtime, the infrastructure definition asks for less than AWS's, because traffic shifting doesn't need a separate resource to act on; it's native to how GCP serves revisions.

For the Academic Research Agent here, that means:

• A connector Harness uses to call Google Cloud's deploy APIs
• A project and a region
• A networking type - left at None by default, or set to a private connectivity mode if the agent needs to reach resources inside a VPC rather than over the public internet

Deployment via pipeline: Simplifying deploy, shift, rollback

Underneath, the two clouds don't agree on how traffic splitting actually works. On Google's runtime, a percentage split is native to how revisions are served - the platform already speaks in those terms. AWS has no equivalent primitive on the runtime itself, which is why the Academic Research Agent's pipeline needs that Gateway from the infrastructure section: our traffic-shift step reads the gateway rule's current routing action and rewrites it, switching between a direct route and a weighted split depending on whether the requested split is a clean cutover or a partial one. Two different cloud mechanics, one authored concept on our side - a target revision and a percentage, so the pipeline for the Academic Research Agent reads the same shape it would if it were deployed to Google instead.

Rolling back the Academic Research Agent never creates anything new. It re-points traffic - or, without a Gateway configured, flips the runtime endpoint directly - back to whatever was live before. That target resolves automatically from what the deploy step actually did; nobody authors it by hand.

The rollback step automatically resolves its target to what the deploy step produced.

Orchestrating the Release - Multiple Agents and Backend Service

Key thing to note: Multiple agents can now be deployed together, along with other backend services, in a release.

Governance: Delivering Safely 

Everything so far has been about one agent. In practice, it's never just one - the Academic Research Agent ships, and a few months later, the same research org builds a Grants Compliance Agent, owned by a different team, and to save on infrastructure, the two agree to share the same AWS AgentCore Gateway for traffic shifting. That's a completely reasonable thing to do, and it's exactly the point where governance stops being optional: a careless traffic-shift call from one agent's pipeline shouldn't be able to touch the other's routing rule, and "who deployed what, to which backend, and when" needs one answer across every agent, not a different answer per team.

We didn't build a separate governance layer for agents. An Agent Service and its infrastructure definition are first-class Harness resources, so the same three mechanisms that already govern every other deployment type apply here without modification:

RBAC: Registering an Agent Service, editing its infrastructure, triggering a deploy or rollback - scoped the same way as any other service and environment in Harness. Two agents can share a Gateway while sitting in different projects with different owners, because RBAC lives at the Harness resource level, not the cloud API level, where a shared Gateway would otherwise blur that line.

Policy as Code: Every AI Agent pipeline execution is a plan that Harness can evaluate against OPA policies before it runs, the same as any other deployment type. That's what actually protects a shared Gateway - a policy can require a traffic-shift step only to touch rules the deploying agent owns, block an overly-permissive execution role, or enforce a minimum instance count before a full cutover. Same policy engine, pointed at a new deployment type.

Approvals and audit trail: Production promotions carry the same approval step regardless of cloud target. Every deploy, shift, and rollback across every agent lands in one execution history - so "what changed on the shared Gateway, and who approved it" has one answer, not one per team.

We reused the existing model instead of deferring it, so a team's first agent and their fiftieth are governed the same way - nothing to retrofit once there's more than one.

What's next

This phase covers deployment. Two extensions are already on the roadmap.

Kubernetes as a third, agent aware deployment target:  Alongside Google’s Agent Runtime and Amazon Bedrock AgentCore, we plan to let you deploy the Academic Research Agent directly to your own Kubernetes cluster - define an Agent Service with Kubernetes as the target, create a Kubernetes infrastructure definition, and get the same progressive-delivery shape through deployment strategies like canary and blue-green, rather than a separate set of primitives built just for agents.

Evaluation gates and observability, wired into the same pipeline: The outcome this phase already produces - a revision, an endpoint, a traffic state - is exactly what the next phase needs as input: a quality gate before promoting the Academic Research Agent, a validation check after, and visibility into how it's actually behaving in production. That's next on the roadmap, using eval provider connectors - Harness AI Evals, Braintrust, LangSmith, Arize, Langfuse, and others as the ecosystem grows.

FAQs

Q: What is Harness Agent Deployment?

A: Agent Deployments is a capability in Harness Continuous Delivery that brings tested, out-of-the-box pipeline steps to deploying AI agents. Instead of scripting a deploy by hand, teams register their agent as a first-class Harness service and get governed, repeatable releases.

Q: How do I deploy an AI agent to Amazon Bedrock AgentCore?

A: Register the agent as an Agent Service in Harness, define an AWS AgentCore infrastructure target (region, VPC/security groups, and optionally an AgentCore Gateway if you need traffic shifting), then run it through a Harness CD pipeline, which handles packaging, deployment, and — if a Gateway is configured — progressive traffic rollout.

Q: How do I deploy an AI agent to Google Agent Runtime?

A: Same model as Bedrock, with a lighter infrastructure definition — a connector, project, and region, plus an optional private networking mode. Traffic shifting doesn't need a separate resource the way AgentCore does, because Google Agent Runtime natively supports revision-based traffic splitting.

Q: Can I deploy AI agents to Kubernetes with Harness?

A: Yes. However, as of the writing of this blog, agent-specific support has not been added for Kubernetes, and the deployment is treated as a standard K8s artifact. We intend to add agent-aware Kubernetes support in the near future.

Q: What's the difference between deploying an agent to a managed runtime vs. Kubernetes?

A: Managed runtimes (Bedrock AgentCore, Google Agent Runtime) handle session/memory management and per-session execution isolation natively — something you'd otherwise have to stand up yourself on Kubernetes (a Redis store, custom readiness probes, etc.). It's a genuine tradeoff, not a strictly better option: Kubernetes gives you more control over that operational surface; the managed runtimes take that control away from you in exchange for not having to build it.

Q: Does Harness support canary or blue/green deployments for AI agents?

A: Harness supports progressive traffic shifting for agents — a target revision plus a percentage split, authored the same way regardless of which cloud you're deploying to, even though the two clouds implement traffic splitting differently under the hood (native revision-split on Google's side, a Gateway routing rule on AWS's).

Q: How does rollback work for AI agent deployments?

A: Rollback re-points traffic to whichever revision was live before, rather than creating a new one — the rollback target resolves automatically from what the deploy step produced, so nobody has to author it by hand.

Q: Can I govern and audit AI agent deployments the same way I govern application deployments?

A: Yes. An Agent Service and its infrastructure are ordinary Harness resources, so RBAC, OPA policy-as-code, and audit trails apply without any separate configuration for agents. This is also what lets two teams safely share underlying infrastructure (like an AgentCore Gateway) without one team's deploy affecting the other's.

Q: What frameworks does Harness support for AI agent deployment?

A: All major agent development frameworks eg. CrewAI, LangGraph, Agents SDK, ADK and custom frameworks as well.

• Agents fail silently with no errors, no bad status codes, so standard observability misses them. AgentTrace observes, evaluates, and acts on agents in one pipeline.
• Production failures can be one-click exported into eval cases, closing the loop between real failures and CI testing.
• Core layers (harness-sdk, harness-evals) are open-sourced under Apache 2.0 and work with any OTel backend, no Harness platform required.

AI agents fail differently from the software we spent the last two decades learning to monitor. We hear some version of the same story from teams shipping agents to production: an agent starts producing wrong answers. Not obviously broken: confident, well-formatted, plausible wrong. The logs are clean, latency looks healthy, and error rates sit at zero. Nothing flags a problem. A user eventually does.

None of the standard tooling was built to catch this. Our observability stack assumes misbehaving software leaves evidence: an exception, a timeout, a bad status code. Agents break that assumption: a hallucination returns HTTP 200, and a run that took fourteen needless tool calls looks identical to a clean one. A wave of LLM-observability tools has grown up to help, but almost all of it stops at observing, it shows you what the agent did, not whether it was any good, and they can't step in while a run is going wrong.

Closing that gap is the idea behind AgentTrace. It isn't a product you adopt; it's the framework Harness uses internally to observe, evaluate, and govern the AI agents across our own platform. It runs as a single pipeline: collect, filter, evaluate, act, so the system doesn't just record what an agent did; it can score whether the work was any good and intervene when it isn't. Today, we're describing how the framework works, and open-sourcing the two layers any team can run on their own stack — harness-sdk and harness-evals, under Apache 2.0.

Why observing isn't enough

Harness AgentTrace is a framework used by Harness to observe, evaluate, and govern AI agents by connecting production monitoring with evaluation metrics. It functions by allowing production failures to be converted into regression test cases, effectively closing the loop between identifying agent errors and preventing them in future releases.

The three gaps make plain tracing insufficient for agents.

1. It doesn't score quality. A distributed trace tells you an LLM call took 340ms and returned 200. It can't tell you whether the response was grounded in the context provided, whether the agent chose the right tools in the right order, or whether a correct answer came through a fragile path that breaks on the next input. Quality is invisible to timing and status codes.

2. The unit of work isn't a request. A microservice trace ends when the request returns. An agent only makes sense across two levels: a run — every model call, tool call, and state transition in a single execution — and a session — every run in one user interaction, so you can see behavior evolve or degrade across turns. Traditional tracing gives you neither.

3. Observing is passive. Even when tracing surfaces a bad run, it can only tell you after the fact. It has no way to intervene — to block a runaway tool call, cap a request about to blow a budget, or redirect a prompt headed somewhere it shouldn't. Watching and acting are different jobs, and agents in production need both.

So the requirement isn't “better tracing.” It's a framework that observes, judges, and acts — and connects them, so what you learn from one run shapes the next. That's what AgentTrace is.

How it's built: one framework, three tiers, two planes

AgentTrace is one pipeline of four stages, deployed across three tiers, connected by two planes. Start with the whole picture:

AgentTrace can run as a standalone Gateway or as a worker inside your existing gateway. Telemetry flows up as OTLP; config and policy are pulled back down — no redeploy.

The pipeline is the same everywhere it runs:

• Collect produces OpenTelemetry spans using Harness semantic conventions — a shared vocabulary for agent interactions (tool calls, LLM invocations, user messages, retrieval). The conventions are what make everything downstream possible: a filter can redact PII because the convention defines where user input lives; a runtime check can track cost because the convention defines where token counts live.
• Filter transforms spans in-flight before they leave the process — redaction, sampling, enrichment — as standard OTel SpanProcessors.
• Eval detects conditions from span data: a cost threshold breached, a security-sensitive tool call, a trajectory going in circles.
• Act enforces a response: block the call, redirect the agent, emit a warning, adapt strategy.

One principle holds throughout: evals detect, actions enforce. Neither does both — an eval emits a decision, an action responds to it. That separation is what keeps the framework composable.

Because Filter, Eval, and Act can run in-process, the framework does what passive tracing can't: redact PII before a span ever leaves the process, warn when a trajectory starts looping, or block a tool call that breaches policy — while the run is still happening. Much of AgentTrace's value at the client tier is exactly this: guardrails that act on the live run, not dashboards you read afterward.

A note on the word “eval,” because it does double duty. In the runtime pipeline above, an eval is an in-flight guardrail that watches a live run. In harness-evals (below), an eval is an offline quality score you run in CI or against stored traces. Same idea — judge the agent — at two speeds: one guards the run in progress, the other grades runs after the fact and gates releases.

The same pipeline runs at three tiers, each with a different data window and latency budget: client-side, in the agent runtime, for guardrails that can't afford a network hop; on a platform agent, an inline intermediary that handles cross-agent concerns like budgets and rate limits (more on this below); and server-side, in the Harness platform, for aggregate patterns no single client can see — cross-session anomalies, account cost trends, fleet-wide degradation.

Two planes connect the tiers, and keeping them separate is deliberate. The data path is pure OpenTelemetry: telemetry flows up via OTLP, which means a customer running only stock OTel SDKs — no Harness client code — still gets server-side collection, storage, and evals. The control path flows down: dynamic configuration and server-side eval decisions, with no client redeployment. Data flows up, decisions flow down, and the two never share a transport.

The loop: turning production failures into regression gates

One capability the framework unlocks is worth calling out on its own, because it's what most teams are trying to build by hand.

A run lands in analytics within seconds. When a user reports a problem, an engineer pulls the exact run and sees, span by span, where it went wrong — the real execution record, not a sampled approximation. The natural next move is to flag it and move on. On the Harness platform, a different move is one click away: Export to Dataset. A production run that revealed a failure — a hallucination, a wrong tool selection, an inefficient path — is promoted into a golden evaluation case, with its input/output pair extracted and retrieved context preserved.

That one action closes a loop most teams close by hand: a production failure becomes an evaluation case, the case joins your eval suite, the suite gates the next release in CI, the next release is observed in production, and the next failure feeds the suite again. If your CI suite only contains failures you thought to write in advance, it will always lag production. Export to Dataset means every failure you investigate becomes a permanent regression gate — over time the suite reflects what actually breaks, not what someone imagined might.

None of this is a novel idea — it's what good teams already do with error reporting and regression tests. The gap was that nothing connected the agent observability layer to the eval layer with a shared data model. AgentTrace makes them one system with one run identity running through both.

What we're open-sourcing

Two layers of the framework are available today under Apache 2.0 — the two you need to run this yourself, on any stack, against any backend.

harness-sdk (harness/otel-python-sdk) is the collection runtime. The Python SDK auto-instruments OpenAI, Anthropic, and LiteLLM with no code changes — wrap your process with a CLI command (harness-instrument python app.py) and set an environment variable. Output is standard OTLP, so it works with any OTel-compatible backend. Every instrumented run produces a tree of typed spans — LLM calls, tool invocations, retrieval, orchestration — with token counts, latency, cost, and model attribution. It's more than collection: a plugin model adds filter hooks (SpanProcessors) and control hooks that can block, so Collect, Filter, and Act all live here. Node.js, Go, and Java packages are in active development; those teams can export via Langtrace or any OTel SDK today.

harness-evals (harness/harness-evals) is the evaluation layer — our opinion, in code, on how agent quality should be scored: correctness, groundedness, safety, trajectory, and performance, each a transparent 0.0–1.0 metric with an explicit threshold and pass/fail. It gates CI through exit codes, absolute score floors, and baseline regression checks; plugs into pytest; reads production traces back in via OTEL and Langfuse importers; and complements DeepEval and RAGAS by adding trajectory, MCP tool-evaluation, and reliability metrics. The opinionated design choices — why trajectory is a first-class dimension, why safety never averages into a composite score — are documented in the repo.

Together they are the loop in two packages: harness-sdk captures the run, harness-evals scores it, and a shared run identity ties a production failure to the test case it becomes and the CI result that gates the next deploy.

Where it goes next: enforcement at the edge

The client SDK sees one agent process. Some guardrails are inherently cross-agent — budget caps that span teams, rate limits across sessions, model routing, spend that must survive a restart — and none can live in-process. They belong on the platform-agent tier: the same Collect → Filter → Eval → Act pipeline, but inline on the network path between your agents and their LLM providers. This is the part we're actively building; the design is settled enough to describe.

It splits into two roles — a platform agent that intercepts and enforces, and the AgentTrace Gateway, a decision service that evaluates — because the component that decides shouldn't be the one that acts. On each outbound LLM call, the platform agent intercepts the request (synchronous or streaming), hands its context to the Gateway synchronously on a tight budget (a <10 ms target, to keep first-token latency negligible), and the Gateway returns one decision: allow, block, route to a different model, or warn. The platform agent enforces it and records a span up the same OTLP data path.

Two properties make this worth the complexity. It covers agents that never adopted the SDK: because enforcement is on the network path, any agent whose calls route through the platform agent gets observability and cost/rate enforcement with zero code changes — add the SDK later for in-process guardrails on top. And it fails open: if the Gateway is unreachable, traffic passes straight through with an annotated span rather than blocking. It starts narrow — routing, cost, and rate limiting first — with content-aware guardrails like PII and prompt-injection detection layered on as it matures.

What's actually different

We're not the first team to build LLM observability, and we won't be the last — LangSmith, Langfuse, Helicone, and others have been at parts of this longer than we have. What's different is that AgentTrace doesn't stop at observing. It scores quality and it acts: in-process guardrails on a live run, enforcement at the edge, and a loop where a production failure becomes the test that gates the next release — observability, evaluation, and guardrails on one data model instead of three tools you wire together yourself.

We're open-sourcing the runtime and the evaluation layer because a way of measuring agent quality only becomes a shared standard if anyone can run it — you can't build a common vocabulary for agent quality when the only people who can use your metrics are your customers. The foundational layers are open. Any team, any stack, any backend.

What's available now

harness-sdk and harness-evals are on PyPI under Apache 2.0:

pip install harness-sdk

pip install harness-evals

‍

With extras for LLM auto-instrumentation and OTLP export:

pip install "harness-sdk[anthropic,openai,litellm]"

pip install "harness-evals[llm,otlp]"

If you're shipping agents and don't have a good answer to “how do we know this is working in production,” start there: harness-evals in your CI pipeline and harness-sdk sending traces to any OTel backend gives you eval gating and production visibility without touching the Harness platform. If you're already on Harness, the platform wires the two together and adds the Trace Viewer, run and session views, human annotation, Export to Dataset, analytics, and CI pipeline gating on top, the full loop, managed. 

The docs cover the parts we got right. We'll be honest about the parts we got wrong and we expect to find some.

‍

FAQs

How is AgentTrace different from existing LLM observability tools like LangSmith or Langfuse?

Most tools stop at observing — showing what an agent did. AgentTrace also scores whether the work was good (via evals) and can intervene in real time (via guardrails/actions), unifying observability, evaluation, and enforcement on one data model instead of three separate tools.

What exactly gets open-sourced, and what stays Harness-platform-only?

harness-sdk (collection/instrumentation) and harness-evals (offline quality scoring) are open-sourced under Apache 2.0 and work standalone. The Harness platform adds the Trace Viewer, run/session views, human annotation, Export to Dataset, analytics, and CI gating on top — the "full loop, managed."

Do I need to change my code to use it?

For harness-sdk, no, it auto-instruments OpenAI, Anthropic, and LiteLLM by wrapping your process with a CLI command and setting an environment variable. For agents that never adopt the SDK, the upcoming platform-agent tier can still enforce guardrails at the network level with zero code changes.

What's the difference between a "run," a "session," and the two meanings of "eval"?

A run is every model/tool call in a single execution; a session is every run in one user interaction. "Eval" means two things at two speeds: an in-flight guardrail watching a live run (runtime pipeline) versus an offline quality score run in CI or against stored traces (harness-evals).

What happens if the guardrail/decision service (the Gateway) goes down — does it block all agent traffic?

No, it fails to open. If the Gateway is unreachable, traffic passes straight through with an annotated span rather than blocking, so enforcement issues don't create an availability outage.

• Native pipeline step - appears alongside Build, Test, and Deploy in the step palette
• 50+ evaluation metrics - OOTB metrics such as faithfulness, safety, hallucination, task completion, or create your own custom metrics
• Eval Suites with blocking or advisory pass strategies.

Building agents is easy. Trusting them is hard.

At Harness, we're building software delivery agents across our platform. Getting to a working prototype was fast, in many cases, in a weekend. But building an agent that performs at production-grade, enterprise-scale was a different problem entirely. And getting to a point where we could actually trust that agents would work for our customers the way we expected every time was harder than anything else.

We shipped them to production. And we learned something that every team that builds agents eventually learns.

Existing Tools Can’t Help With AI Evals

As we dug into why this is so much harder than traditional software, we kept hitting the same five walls.

Failures are silent. When traditional software fails, it crashes. You get an error code, a stack trace, and a log entry. When an agent fails, nothing crashes. It returns a confident, plausible, completely wrong answer. No alert fires.

Output is non-deterministic. Traditional software gives you the same output for the same input. Agents don't. Run the same prompt twice, get different results. You can't write assertEqual for a summarization agent.

There's no debug mode. Stack traces tell you exactly why traditional code broke. With agents, you can't trace why it chose one answer over another. The reasoning is opaque. The decision path is invisible.

Quality is a spectrum, not a binary. Traditional tests either pass or fail. Agent quality is: Did it complete the task? Is the tone professional? Is it faithful to the source? Is it relevant? Is it safe? There is no single "pass."

Maintenance is a moving target. Traditional software: fix the code, ship the patch. Agents: the model drifts, the prompts change, the context window shifts, and the LLM version upgrades silently. A fix today can break tomorrow without anyone touching the code.

Your DORA metrics don't measure agent faithfulness. Your test suites pass while quality silently degrades. And your team has no way to A/B test prompts or swap models without shipping blind.

Harness AI Evals: The Details

Harness AI Evals makes core agent quality measurable, enabling agent-aware quality gates in your CI/CD pipelines. With AI Evals, changes to your agent (or underlying model) are evaluated first against your standard data sets and model outputs. The output is evaluated across many dimensions, including correctness, performance, and safety. Teams can then use the scores as quality gates in their CD pipelines to simplify release decisions. Then, production data can be fed back in, improving your testing based on real inputs.

Two evaluation modes, one continuous loop

Offline evaluation (pre-deploy)

Run your agents against golden datasets before they ship. A golden dataset is a curated set of test cases - inputs your agent will receive, paired with the expected outputs or context it should use to respond. Think of it as your ground truth: the known-good answers your agent should produce or stay faithful to.

Score every response using 50+ built-in metrics. For example:
Is the response grounded in the retrieved context (faithfulness)?
Did the agent call the right tools with the right arguments (tool correctness)?
Is the output safe from prompt injection? Did it complete the task? 

Is the tone appropriate? 

You can also define custom rubrics in natural language or write your own scoring logic in Python.

Compare prompt variants and model versions side by side. Gate your release pipeline: if scores drop below the threshold, the deploy is blocked. Not a script. A native Harness pipeline step.

Online evaluation (post-deploy)

Offline evaluation is just the beginning. It helps you test the agent before you ship, preferably as part of a CI/CD pipeline.

Online evaluation takes it further. It scores the output of your agent against real scenarios coming from your customers, using the same metrics. Instead of getting scores for synthetic data you generated in a dataset, you're scoring how the agent actually behaved against real user inputs. That's how you learn how your agent operates in real life.

And then you can add those real scenarios back into your datasets, so you can use them in subsequent offline evaluations. You're continuously enriching the datasets, ensuring that as you progress with development, the output improves and definitely doesn't regress.

Here's a walkthrough of how it works:

‍

Our first round of manual testing took one to two weeks every release cycle. Now I've put AI Evals in as a release gate. Whenever there's a deployment, it evaluates whether anything broke. What took days takes minutes. The sign-off isn't someone opening a sheet of 500 cases anymore. It's based on the pass rate. Score above threshold? Ship it. Below? It doesn't go out.

- Chetan Sinha, Software Engineer, Harness QPE Team

Governance-ready from day one

AI Evals inherits the full Harness platform:

• RBAC on every eval, dataset, and suite
• OPA policy governance on eval configurations
• Full audit trails on who changed what threshold and when
• Secrets management for model API keys via existing connectors
• SSO and data residency out of the box
• Centralized registry for prompts, agents, MCP tools, and skills with versioning
• Pipeline execution link - every eval run links back to the pipeline execution that triggered it

Getting started

Getting started is designed to be fast. A guided onboarding flow walks you through setup. In-built templates for common patterns (prompt injection detection, correctness checks, RAG quality) let you plug and play without writing scoring logic from scratch. And you can synthesize entire datasets from a single description using AI, so you're not hand-writing hundreds of test cases to get started.

Harness AI Evals is the first native quality gate for AI in CI/CD. Score your agents before deploy, monitor them after, and ship with confidence every release.

Request to start the beta!(/demo/ai-evals)

FAQs

How is this different from an observability tool like LangSmith or Braintrust?

Those tools connect your evals to observability, you export traces, run evaluations separately, and interpret the results outside your pipeline. AI Evals runs as a native step in your Harness pipeline, right alongside Build, Test, and Deploy. A quality regression fails the build the same way a failed unit test does. No scripts to wire in, no glue code to maintain.

Do I need one system for testing before launch and another for monitoring in production?

No. Offline evaluation (pre-deploy) and online evaluation (post-deploy) share the same metrics and the same datasets. The scoring logic you use to test an agent before it ships is the exact same logic that scores it in production. That means no gap between what you validated and what you're actually measuring once real users are involved.

What kinds of problems can it actually catch?

Things like hallucinations, unsafe or off-policy responses, incomplete task execution, wrong tone, and incorrect tool usage — scored across 50+ built-in metrics, or your own custom rubrics if you need something specific to your use case. It also evaluates multi-step agent behavior, not just the final answer: did the agent reason through the task correctly and call the right tools along the way?

• AI agents never stop changing. Prompts go stale, models improve, and guardrails need tuning. Managing those changes through code deployments is too slow for the pace AI demands.
• AI configs give you a governed runtime layer. Change a prompt, swap a model, or adjust parameters in production without redeploying. Every change is targeted, approved, versioned, and auditable.
• AI tuning becomes a release and learning loop. Expose a new behavior to a controlled audience, measure the impact, then expand, iterate, or roll back with confidence.
• The same discipline you apply to software, now applied to AI. Harness AI Config Management brings targeting, experimentation, approvals, and audit trails to the layer where AI behavior actually lives.

Overview

AI agents don't stop evolving when they ship to production. Teams continuously optimize for better accuracy, lower cost, faster response times, stronger safety, and higher customer satisfaction. That means updating prompts, changing models, and introducing new guardrails far more frequently than traditional code releases. Yet most teams are still managing those changes through deployments, environment variables, or manual processes.

That is the gap Harness AI Config Management is designed to close. It is a governed runtime configuration system that lets teams change prompts, models, routing, and behavior without redeploying code.

Harness Feature Management & Experimentation already helps teams control features in production, gating who sees a capability, governing the release strategy, and measuring impact. Harness AI Config Management extends that same discipline to agent behavior. Teams can manage prompts, model selection, and the inference parameters that control how the model behaves, all as runtime configurations that are targeted, measurable, and governed.

Feature flags answer: who should see this AI capability?

AI Configs answer: how should that capability behave?

Experiments answer: which behavior delivered the best outcome?

The Rise of AI Runtime Behavior 

The industry is transitioning from deterministic software to probabilistic/AI-driven software. This evolution is transforming a number of dimensions and dramatically changing what runtime management must address.

Together, these differences mean that the operational surface area for AI agents is significantly wider and less forgiving than traditional software. Costs can spike without warning, failures are harder to detect, and the number of stakeholders making changes continues to grow. 

The Ungoverned AI Problem

In most organizations today, AI changes happen without governance:

• Prompts are edited in notebooks, spreadsheets, or config files with no versioning
• Model switches happen without cost impact analysis or governance
• No pre-prod verification as changes go straight to production
• No rollback path when things go wrong
• No audit trail for compliance

The result is a standoff. Platform teams can't give product teams the freedom to iterate because the risks of ungoverned changes are too high. Product teams can't move at the pace AI requires because every change bottlenecks through engineering and deployment. Harness AI Config Management is designed to break that standoff.

A working example: the AI Support Agent

Imagine a team shipping a new AI Support Agent.

The team deploys the agent behind a feature flag, limiting access to internal users and a small beta cohort. To control agent behavior at runtime, the team uses AI configs to manage the prompt, temperature, token limits, retrieval threshold, and fallback message without touching code.

Before promoting anything to production, the team tunes parameters in pre-production environments using environment-level definitions and targeting. Changes are reviewed and governed through Harness before they go live.

With the agent running in production, the team creates two AI config variations: one optimized for concise answers and one for detailed troubleshooting. Those variations are targeted to different user cohorts, and impression data feeds directly into an experiment. The team now has real evidence of which behavior drives better outcomes before deciding whether to roll out further, iterate, or roll back.

What Harness AI Config Management adds

Harness AI Config Management is built on Harness Configs, a first-class runtime configuration layer in Harness FME. It uses the same config model, the same delivery path, and the same governance patterns teams already rely on for production software.

Environment-specific values with shared structure

Harness AI Configs use a two-level model. Teams define schema, default values, and variations once, then manage targeting rules and live values per environment. Developers and AI teams can iterate freely in development and staging while keeping production stable. When a configuration is ready, it can be promoted across environments without a redeployment.

Variations, targeting, and experimentation readiness

Harness AI Config Management supports multiple variations per config. A team might compare two prompts, route different customer cohorts to different model choices, or test parameter changes against a controlled audience. Targeting rules determine which users, accounts, or environments receive each variation, using the same rule builder as Harness feature flags.

This is what turns AI tuning from a guessing exercise into a structured release and learning loop. Teams can shift AI behavior for a limited group, watch quality and business metrics, then decide whether to expand, iterate, or roll back.

SDK delivery without a separate AI path

Developers call getConfig() to resolve the right configuration for a given target and get back type-safe accessors for string, number, and boolean values. For AI use cases, that means resolving a prompt, model, temperature, or parameter at runtime without redeploying. There is no separate AI SDK. AI configs build on the same delivery layer as every other config, giving teams one consistent way to manage runtime behavior.

Governance where AI changes happen

The most important part of Harness AI Config Management is not that teams can change AI behavior faster. It is that they can do it with governance.

Harness FME applies RBAC, granular permissions, approvals, OPA policy evaluation, version history, and audit logs to every config change. Sensitive production changes can require policy checks before they propagate. That matters because AI behavior changes carry real customer impact. A model swap can affect cost and latency. A prompt update can shift output quality in ways that are hard to detect without the right tooling.

Changes to AI configs require the same release discipline as code. 

Ready to see Harness AI Config Management in action?

Harness AI Config Management helps teams manage the behavior behind AI features with the same discipline they apply to modern software releases: targeted rollout, experimentation, approvals, policy, auditability, and rollback.

Ready to see how Harness helps teams govern AI behavior without slowing iteration? Sign up today and get started with a free account!

FAQ

What are AI Configs?

AI Configs are governed runtime configurations for AI-powered product experiences. They can include the prompt, the model, inference parameters such as temperature and token limits, and other AI behavior parameters.

How are AI Configs different from feature flags?

Feature flags control access. They decide who sees a feature and when. AI Configs control behavior. They decide which prompt, model, parameter, threshold, or fallback a user receives once the AI capability is available.

Why not manage prompts and models in code?

Hardcoding prompts and model parameters makes every behavior change depend on a deployment, slowing iteration and hiding important production decisions inside application logic. AI Configs externalize those decisions into a governed runtime layer.

How do AI Configs help with governance?

Harness FME can apply RBAC, approvals, OPA policy checks, version history, and audit logging to AI config changes. 

Can teams experiment with AI Configs?

Yes. AI Configs support variations and targeting, which creates the foundation for testing different prompts, models, or parameters with controlled audiences. Teams can compare outcomes such as quality, latency, cost, satisfaction, reliability, and business impact.

Do AI Configs require a separate AI SDK?

No. AI Configs build on the same Configs SDK delivery model. Developers can resolve the right config for a target at runtime and use type-safe accessors for the values their application needs.

What kinds of teams should use AI Configs?

AI engineers, ML engineers, prompt engineers, product managers, SREs, platform teams, data scientists, and experimentation teams can all benefit. The common need is controlled, measurable, governed iteration on AI behavior at runtime.

More teams are building AI agents today. Engineers deploy them into customer-facing production environments, product teams integrate them into customer workflows, platform teams build them for internal use, and even sales, marketing, and support teams are creating agents for their own operations.

Shipping a software update usually follows a known process: build it, test it, deploy it with a script. That process works. It's how the modern software delivery industry was built, and it works because application code is deterministic — test it once and you’ll get the same result next time.

Agents break this model. An agent’s underlying language model decides how to complete a task. But that flexibility comes at a cost. Building an agent is easy. Delivering one safely to production is not. 

Today, Harness is extending its platform to cover the full agent development lifecycle. With Harness Agent DLC, you can now build, test, deploy, operate, and govern agents through the same platform you already use for your applications — with the same controls, pipelines, and governance you apply to everything else you ship.

AI agents behave differently. They dynamically select tools, invoke models, coordinate with other agents, and adapt their execution paths based on context. Each time an agent runs, it can make different choices even with the same input. That makes their behavior inherently unpredictable and can lead to significant differences in cost, latency, reliability, or risk. Traditional testing and governance playbooks no longer apply. 

According to Gartner®, “only 8% of organizations have agentic AI in production”. Agents aren’t making it to production because organizations can't apply the same security, governance and quality guardrails they rely on for traditional software. The risk is too high to ignore. A rogue agent in production can expose customer data, violate a compliance policy, or make unauthorized decisions. In financial services, healthcare, and travel, that's not a bad week. It's a regulatory event or a headline. Incidents are no longer reproducible on demand. Teams building agents need a way to ship them with the same confidence they ship everything else.

Build

You build agents with the same coding tools you already use. Harness Continuous Integration (CI) builds them into deployable artifacts, just as it builds any other service. No new build system to learn. The agent is just another standard microservice that goes through CI.

Test

Testing an AI agent requires different tools versus traditional software. With software, an input generates a predictable output. AI agents are inherently non-deterministic. A single prompt can yield multiple correct, yet entirely different, outputs. A response might be factually correct but delivered in the wrong tone or return a plausible wrong answer. 

Traditional testing falls short in this new reality where a test is no longer a pass fail but a scored spectrum. Teams need testing capabilities built for agents that can be integrated cleanly into their governed delivery pipelines.

Harness AI Evals (NEW) makes the core agent quality measurable, allowing for agent-aware quality gates in your CI/CD pipelines. With AI Evals, changes to your agent (or underlying model) are evaluated first against your standard data sets and model outputs. The output is evaluated across many dimensions, including correctness, performance, and safety. Teams can then use the scores as quality gates in their CD pipelines to simplify release decisions. Then, production data can be fed back in, improving your testing based on real inputs. Learn more.

Harness AI Test Automation (AIT) validates that the agent works within an application’s chat interface using AI assertions. Instead of writing brittle code, testers use plain English assertions to describe a good response. AIT emulates a real user via the browser, requiring no API hooks or direct model access.

You cannot isolate agent validation from how you deliver the rest of your system. Natively binding backend logic evaluation with user-centric UI testing inside your delivery lifecycle ensures your agents are actually improving, not just running and returning answers.

Store

Unlike traditional software, AI agents are composed of many independently evolving artifacts, including agent definitions, prompts, skills, MCP servers, models, and policies. These artifacts are developed, evaluated, and updated independently, yet together determine how an agent behaves in production. As they're built, evaluated, and reused across teams, the Agent DLC needs more than a repository. It needs a trusted system of record that manages the complete release definition of an AI agent.

Harness Artifact Registry provides a centralized registry for AI and software artifacts, preserving their versions, provenance, dependencies, and promotion history. By governing the complete set of artifacts behind every agent release, teams can confidently compose trusted AI agents, reuse approved components, and promote reproducible agent deployments.

Deploy

Most AI agents in production run either as containers on Kubernetes or on managed runtimes like Amazon Bedrock AgentCore. Harness Continuous Delivery (CD) already covers the Kubernetes path with canary releases, progressive rollout, automated rollback, approval gates, and policy guardrails. Until today, deploying to a managed runtime typically meant a separate cloud-specific workflow, outside your pipelines and governance.

Agent Deployments (NEW) extend that same governance to managed agent runtimes, starting with Amazon Bedrock AgentCore and Google’s Agent Runtime. Agent Deployments now run as a stage in the same Harness CD pipeline as your services, with no new scripts or tooling required. Because the agent deploy is a stage in that pipeline rather than a separate workflow, release orchestration can sequence backend, frontend, and agent code into a single release, instead of running the agent deploy on its own, separate from the other two.

In addition, the same OPA policy-as-code framework that governs your backend and frontend now governs your agent deployments. You can now cap CPU or memory consumption, restrict which models agents can call, or block anything outside an approved list. Agent Deployments also integrate directly with Harness AI Evals and other LLM evaluation frameworks, such as Deepeval. This closes the gap between an agent that deploys successfully and one that runs reliably in production. Learn more.

Operate

Once an agent ships, the same platform that manages your services manages your agents. Harness already handles cost visibility, runtime release management, and experimentation. All of these capabilities now extend to cover agents.

AI Cost Management extends the cost visibility, attribution, and governance you rely on for cloud spend to every agent, model, and provider, so you know what agents cost and are better able to keep that spend under control.

AI Configs (NEW) support the release and management of prompts and model changes at runtime, backed by the same feature flagging infrastructure your teams already rely on. Run controlled experiments to see which AI behavior performs best, then promote the winner with confidence. If it doesn’t improve the customer experience, roll back instantly without redeploying. Learn more.

Govern

Building, testing, and deploying AI agents safely into production is only part of the challenge. Organizations also need to know which agents exist, who owns them, and whether an agent already solves the problem they're about to tackle. Without that visibility, organizations risk unnecessary cost and operational complexity.

AI Asset Catalog (NEW), part of the Harness Internal Developer Portal, makes sure every agent in your organization has an owner. It auto-discovers and registers your agents, skills, and plugins from your source code repositories. Each AI asset is stored with its full instruction set and linked to its owner and dependencies. Developers can easily discover what agents and skills already exist before building new ones, cutting down on duplicate work and sprawl. With out-of-the-box and custom scorecard checks, platform teams can define and enforce standards so AI assets are governed just like any other software component.

Agent Security: Woven Into Every Stage

Security runs through every stage of the Agent DLC — build, test, deploy, operate, and govern — and requires a different approach than traditional software security. Agents reason and act at runtime in ways no static scan can anticipate. They dynamically expand their attack surface by connecting to tools and APIs, spawn sub-agents without a human in the loop, and inherit trust from every model and dependency they touch. Traditional security wasn't built for any of that.

New agent security capabilities shift left to limit what agents can do before they ship, and shields right to enforce policy on agents already running in production. 

Shift-left

• Primitive Scanning (NEW): scans skills and prompts with Harness SAST and packaged models with Harness Security Testing Orchestration to catch misconfigurations and other risks before an agent is built or deployed.
• AIBOM (NEW): Harness Supply Chain Security extends the traditional SBOM to capture every model, tool, and dependency an agent was built with.
• Harness AI Testing (NEW): tests agents for adversarial inputs, unexpected behaviors, and policy violations across the OWASP Top 10 for LLMs.

Shield-right

• Agent Discovery and Posture Management (NEW): Harness AI Discovery continuously surfaces agents as they’re invoked, maps how they connect and orchestrate work, and assesses their posture across your org.
• Harness AI Firewall (NEW): enforces policy continuously against prompt injection, tool misuse, and data exfiltration while the agent runs.
  
  Learn more details in our blog.

AgentTrace Provides End-to-End Visibility Across Every Execution

As organizations move AI agents into production, those agents are increasingly reviewing code, remediating security issues, assisting customers, and automating complex workflows. Yet most organizations have little visibility into how those agents actually operate.

Harness AgentTrace (NEW) captures execution at the run level and across full sessions, so teams can understand both what happened in a single agent run and how behavior evolves across an entire user interaction.

With AgentTrace, organizations can understand how an agent behaved and what influenced its path, identify performance bottlenecks and failure points, compare execution quality across models and prompts, and establish governance for AI systems running in production.

AgentTrace serves as the layer that connects every stage of Harness Agent DLC, providing the telemetry and audit trail consumed across Harness products.

Today, Harness is open-sourcing the foundational components of AgentTrace, including harness-sdk and harness-evals (the open-source SDK on which Harness AI Evals is built), so developers can use the same tracing primitives in their own AI applications. Learn more.

One platform supporting the Agent DLC

The challenge organizations face today is extending the same software development lifecycle to their agent delivery. That's the truth behind what’s driving Gartner's finding that only 8% of organizations have agentic AI in production.

Today, Harness is launching our Agent DLC to support our customers in successfully delivering AI agents. Everything you’ve done for software delivery over the last decade — governance, orchestration, security, testing — you can now do for agents in the same platform.

Talk to our team about what shipping your first agent through Harness Agent DLC would look like. Book a demo.

Traditional security was built for software that sits still. Agents don't. 

Agentic applications break the assumptions traditional security was built on. The traditional SDLC assumes artifacts are static once deployed, attack surfaces are known and inventoried, humans control every decision and gate, and supply chain risk ends at code and dependencies. Agents violate all four. They reason and act at runtime in ways no static scan can fully anticipate. They dynamically discover and connect to tools, MCP servers, and APIs, expanding the attack surface continuously. They spawn sub-agents and chain across systems without a human in the loop. And they inherit trust from every model, tool, and API they connect to, making supply chain risk exponentially wider and harder to contain.

Security tools built for the traditional SDLC weren’t designed for any of this.

Agent Security: A New Discipline 

Securing agents requires a fundamentally different approach than securing traditional software. Not a new tool bolted onto existing practice, but a new discipline built around how agents actually work - one that shifts left to constrain what agents can do before they're deployed, and shields right to enforce policy and maintain visibility while they run.

Today, Harness announced the availability of the industry's first DevSecOps platform for the Agent Development Lifecycle (Agent DLC) - purpose-built for both developing and operating agents securely.

Shift-Left for secure agent development

You can't predict everything an agent will do once deployed. But what an agent does at runtime isn't random. The skills you expose, the tools you wire up, the models you select, and the prompts you write all shape it. Shift-left for agent development isn't about catching every runtime risk before it happens; it's about limiting what agents can do once they're live.

Primitive Scanning analyzes the new design-time decisions that agents introduce, helping catch exposure and misconfiguration before any agent is built or deployed. It extends Harness SAST with skill and prompt scanning, and Harness Security Testing Orchestration with model scanning.

AIBOM with Harness Supply Chain Security gives you a complete inventory of every model, tool, and dependency used to build an agent, expanding the traditional SBOM to cover the full AI supply chain.

AI Testing tells you whether agents are safe by testing them for adversarial inputs, unexpected behaviors, and policy violations across the OWASP Top 10 LLM and Agentic AI risks.

AIBOM inventories the AI components used to build an agent, including models, libraries, frameworks, datasets, prompts, skills, tools, and services.

Shield-Right for secure agent runtime 

Shift-left narrows what agents can do, but once deployed, agents operate in an open world. They encounter inputs, tool responses, and chain behaviors that no pre-deployment check can fully anticipate. Protecting agents in production requires continuous visibility into what agents are running and what they're doing.

Agent Discovery expands AI Discovery beyond individual AI assets by introducing an agent SPM capability to the platform that continuously surfaces agents as they spin up, maps how AI assets connect and chain, and assesses their posture across your organization. Where shift-left defines the boundaries, Agent Discovery tells you what's operating within them.

AI Firewall protects all AI assets, including agents at runtime, enforcing policy continuously against prompt injection, tool misuse, and data exfiltration through agentic chains - after the agent is live and as long as it runs.

AgentTrace brings observability throughout the entire agent activity session, providing a full audit trail across prompts, reasoning, tool calls, and outputs. This is especially important for agentic operations to have confidence that the non-deterministic systems are acting as expected. 

Agent Discovery continuously identifies active AI agents, maps their dependencies and interactions, and surfaces associated risk and sensitive-data exposure.

The Agent DLC Needs Dedicated Security

Blindly applying traditional AppSec to agent development doesn't work. The assumptions don't hold, the tools don't reach, and the attack surface keeps moving. What's needed isn't a patch on existing practice; it's a purpose-built approach that secures agents at every stage of how they're actually built and run.

That's what Harness offers. Shift-left to define the boundaries. Shield-right to hold them. A single platform that treats agent security not as an afterthought, but as a first-class discipline across the full Agent DLC.

Agents are already in production. The question is whether your security posture is built for them.

Talk to our team to see how Harness secures the Agent DLC.

• Infrastructure as Code solved infrastructure provisioning, but it did not solve database schema change delivery.
• Database changes are fundamentally different because they modify persistent production state.
• Separate application and database deployment workflows increase operational risk.
• Modern platform teams need version-controlled database schema change delivery with rollback and governance built in.
• AI will accelerate software delivery, making unified delivery workflows increasingly important.

‍

For more than a decade, Infrastructure as Code (IaC) has transformed how engineering organizations build and operate systems.

Infrastructure became programmable, provisioning became repeatable, and configuration became version-controlled.

Teams gained the ability to automate environment creation, enforce policy consistently, and scale infrastructure operations far beyond what manual processes could support.

Yet one critical part of software delivery never fully made the same transition, the database.

As Mrina Sugosh, Senior Product Marketing Manager at Harness, explained during a recent discussion on modern software delivery: infrastructure evolved into a declarative system while database delivery largely remained procedural.

That gap is becoming increasingly difficult to ignore.

Infrastructure Became Programmable. Database Changes Didn't.

Infrastructure as Code introduced a powerful operating model. Instead of manually configuring servers and cloud resources, teams could define desired state in code and let pipelines handle deployment. Tools like Terraform, OpenTofu, AWS CloudFormation, AWS CDK, and Terragrunt let organizations standardize infrastructure management and retire the fragile, hand-run processes that came before.

Platform engineering itself emerged from this shift. Version-controlled infrastructure, reproducible environments, and codified policy became foundational capabilities for modern engineering organizations.

Databases followed a different path. Schema changes, migrations, rollback logic, and data management frequently stayed outside that same delivery framework. The result is a split operating model: infrastructure is automated, applications are automated, and database changes are still done by hand. That separation is where the risk lives.

The Hidden Gap in Modern Software Delivery

Modern systems are no longer simple applications running on servers. They're interconnected ecosystems of applications, databases, APIs, queues, caches, event systems, and infrastructure services, each one depending on the others behaving correctly.

Yet many organizations still deploy these components through separate workflows. A database engineer manages schema changes, a DevOps engineer manages infrastructure, an application team manages deployments, and success depends on those people coordinating rather than those systems coordinating.

Wyatt Munson, Product Education Engineer at Harness, described what that looks like in practice: sitting shoulder to shoulder with a software engineer during a deployment, one person ready to roll back the application and the other ready to roll back the database, both watching for the moment to click at the same time. It worked. But a release that hinges on two people and their timing isn't a delivery model that scales.

Why Database Delivery Is Fundamentally Different

Many organizations assume database delivery should behave like IaC, but the reality is more complicated. Infrastructure provisioning is generally additive and reconstructable: if a virtual machine fails, you recreate it; if a Kubernetes cluster is misconfigured, you rebuild it.

Database changes operate differently because they carry persistent business state. They represent years of accumulated data, application assumptions, indexing strategies, access patterns, and operational dependencies. A schema migration isn't provisioning infrastructure. It's modifying a live system, and even seemingly small changes can ripple outward:

• Dropping a column removes information permanently.
• Index changes affect query performance.
• Schema migrations can impact application behavior.
• Data migrations may create consistency issues across services.

As Munson put it: "At the end of the day, it's the data that's the most important." Losing infrastructure is disruptive. Losing data is catastrophic.

The GitHub Incident That Exposed the Problem

The risks of database delivery aren't theoretical. One example discussed during the webinar was a GitHub production database migration. A migration removed a column believed to be unused, but parts of the application still referenced that column through a separate ORM path. After the change deployed, GitHub saw elevated error rates across pull requests, push operations, notifications, webhooks, and API traffic.

The failure wasn't really the migration itself. It was that the database change and the application that depended on it were validated separately rather than together. The migration was checked as a migration and the application was checked as an application, but the two were never exercised as one system before the change reached production. So the dependency stayed invisible until it broke in front of customers.

This is the gap a unified delivery model is built to close, and it closes it in two places:

1. Catch it in pre-prod. Had the schema change and the application deployed together into a production-like environment, spun up on demand with Infrastructure as Code, the same errors that surfaced in production would have shown up there first, against real application behavior, where they cost nothing.
2. Gate it before prod. A destructive change like dropping a production column should pass through a policy gate before it ever reaches production, held for explicit review and ideally backed by proof that it cleared QA without breaking anything.

Three Requirements for Modern Database Delivery

1. Version Control Database Changes

Database changes should be managed like application code, through version control and GitOps. That gives every change traceability, auditability, and a documented history: who made it, why, and when it shipped. The same properties that make application code reviewable and repeatable apply just as well to a schema migration.

2. Automate Validation and Rollback

Database delivery needs more than deployment automation. Before a change reaches production it should go through:

• Validation against a realistic environment, so problems surface in pre-prod, not prod.
• Schema and migration testing as part of the pipeline.
• Environment promotion, the same staged path application code follows.
• A defined reverse path: a reviewed rollback where the change can be cleanly undone, a tested forward-fix where it can't.

The goal is to remove uncertainty before a change ships, not to rely on an emergency undo after it's already in production.

3. Unify Delivery Workflows

The biggest opportunity is bringing infrastructure, application, and database delivery into a single operational framework. Instead of coordinating separate workflows by hand, platform teams orchestrate them through one pipeline, so application deployment, infrastructure provisioning, and database migration run as a single coordinated motion, governed consistently rather than stitched together by hand.

From Infrastructure as Code to Delivery as Code

The next phase of platform engineering extends the code-driven model beyond infrastructure to delivery itself. Call it Delivery as Code: the principles that made infrastructure programmable, applied to how every change ships, databases included.

In practice that means three things.

Database Pipelines

Database changes move through controlled promotion workflows rather than ad hoc scripts, so each one is reviewed, tested, and promoted environment to environment like application code. 

Policy Enforcement

Governance gets codified through policy engines such as Open Policy Agent, so the rules that used to live in someone's head become enforceable at the pipeline:

• No dropping production tables
• No violating naming standards
• No deploying unapproved schema changes to prod

AI-Assisted Operations

AI begins to absorb the repetitive operational work, helping draft migration definitions and validation checks so engineers spend less time on boilerplate and more on the changes that actually need judgment.

The goal isn't replacing engineers. It's removing the repetitive operational toil so they can focus on higher-value work.

Building a Unified Delivery Platform

Platform teams are facing a new reality. AI-generated code is increasing development velocity dramatically, and more changes are entering delivery systems than ever before. When application deployment is automated but database delivery stays manual, the database becomes the bottleneck, and the fix isn't more coordination meetings. It's a unified delivery model where:

• Infrastructure changes are version-controlled.
• Database changes are version-controlled.
• Governance is codified.
• Testing is automated.
• Rollbacks are built into workflows.

That's a delivery system that can scale alongside modern software development. Infrastructure as Code was the first step. Database DevOps is the next one.

Conclusion

Infrastructure as Code fundamentally changed software delivery by making infrastructure programmable.

But modern systems are more than infrastructure.

Applications, databases, policies, and delivery workflows must operate as a coordinated system.

As software delivery accelerates—particularly with the rise of AI-generated code—database delivery can no longer remain a separate operational process.

Platform teams that unify infrastructure, application, and database workflows will be better positioned to deliver software faster, safer, and with greater operational confidence.

Ready to see how Harness helps platform teams bring database delivery into the modern software delivery lifecycle? Explore Harness Database DevOps and Infrastructure as Code Management.

FAQ

What is Database DevOps?

Database DevOps applies DevOps principles to database changes, including version control, automation, testing, governance, and CI/CD workflows.

Why isn't Infrastructure as Code enough for databases?

Infrastructure as Code manages infrastructure state but does not address schema changes, data migrations, rollback workflows, or database governance.

Why can database changes be riskier than infrastructure changes?

Database changes modify persistent production data. Unlike infrastructure resources, databases often cannot be recreated without business impact.

How does Database DevOps improve software delivery?

It enables automated testing, version-controlled changes, rollback capabilities, and coordinated application and database deployments.

What role does GitOps play in database delivery?

GitOps provides version control, auditability, and workflow automation for database schema changes and migration management.

Can Harness DB DevOps be integrated into CI/CD pipelines?

Yes. Database DevOps integrates natively with Harness Continuous Delivery, so schema management, validation, and rollback run as steps in the same pipeline as your application deployments. If you run a different CI/CD tool, those same capabilities can be invoked from your existing pipelines instead.

How does Harness AI help database delivery?

Harness has an AI powered schema authoring capability, which can assist with migration generation, so engineers spend less time on boilerplate and more on the changes that need real judgment. Governance and approval controls stay in place regardless of how a change was authored.

Database migrations are rarely just about changing schema. In real production systems, every migration has to preserve three things at the same time: application availability, data consistency, and compatibility across versions. That is the hard part.

When teams say they want “100% uptime,” what they usually mean is no planned downtime during deployments and no user-visible interruption while the application and database are evolving. That goal is realistic, but only if the migration strategy is designed around compatibility from the start.

The Real Objective: Compatibility First, Cleanup Later

A migration should never assume that the new application version is the only code touching the database. During a rolling deployment, blue-green cutover, or staged rollout, both versions may run side by side for a period of time.

That creates a simple rule: “Every schema change must be safe for the current app version and the next app version.” If your schema is not designed for this overlap, you introduce:

• Runtime failures
• Data inconsistencies
• Forced downtime

The solution is not complex tooling, it's the correct migration strategy. Since the safe database migration is not just “correct” - it must be compatible across versions.

• Old application must continue to work
• New application must start working immediately
• Database must support both during rollout
• Database must ensure both versions of the application always see the same data, must ensure data integrity even while writing in it.

This is the foundation of zero downtime database migration.

Migration Patterns that Work in Production

‍

‍

These patterns follow the same rule: never break existing reads or writes during transition.

The Safest Rollout Sequence

This is the real test of a safe migration. The old version may still:

• Read a deprecated column,
• Expect a legacy enum value,
• Or write a field that the new version no longer prefers.

The new version may:

• Depend on a new column,
• Expect a new constraint,
• Or write into a split schema.

To support both, design the transition so that:

• The old app still sees valid data,
• The new app can start using new structures immediately,
• And the schema does not force an all-or-nothing deployment.

That is why additive changes and compatibility windows matter more than raw speed. This aligns with real-world pipelines where schema and application changes are decoupled but coordinated.

Keeping Old and New Structures in Sync

During migration, both schema versions may remain active simultaneously.

This creates a synchronization window where:

• Old application versions may still write legacy fields,
• New versions may write updated structures,
• And both new and old representations must remain consistent.

Common synchronization approaches include:

• Database triggers,
• Change data capture (CDC),
• Tansactional dual writes,
• Event-driven synchronization pipelines.

Without synchronization safeguards, post-migration edits can cause data divergence between old and new schemas.

A Practical Deployment Pattern That Mirrors Real Pipelines

Your pipeline example reflects the same deployment philosophy: application rollout, schema application, and controlled progression are separated into explicit steps rather than collapsing everything into one risky event. That is exactly the kind of sequencing needed for production-safe migrations. 

In a mature release process, a database migration stage should be treated as a release gate, not a side effect. The schema change should happen only when the release pipeline has proven that the next application version can coexist with the previous one.

That is how you preserve uptime without gambling on runtime behavior.

What Is the Expand-and-Contract Pattern?

The expand-and-contract pattern is a phased migration strategy used to evolve database schemas safely without downtime. It works in three stages:

1. Expand - Introduce new schema structures without removing old ones.
2. Migrate - Keep old and new structures synchronized while applications transition.
3. Contract - Remove deprecated schema only after all traffic has migrated.

This allows both old and new application versions to operate safely during deployment.

Best Practices for Zero Downtime Database Migrations

Key best practices for zero downtime database migrations include:

• Use the expand-and-contract pattern
• Avoid destructive changes during active deployments
• Ensure backward and forward compatibility and ensure both versions of the app will always see the same data.
• Run idempotent, incremental backfills
• Monitor and validate data consistency continuously

These practices minimize risk and ensure smooth production rollouts.

Frequently Asked Questions

1. How can I migrate a database without downtime?

To migrate a database without downtime, use a phased, backward-compatible approach:

• First, apply non-breaking schema changes (add columns, avoid deletions)
• Perform background data backfills in small batches
• Release the updated service version utilizing synchronized dual-write operations.
• Gradually shift reads to the new schema
• Remove old schema only after all traffic is migrated

This approach ensures continuous availability during the migration process. 

Application-layer dual writes alone do not guarantee consistency. Failures between writes, retries, or partial transaction completion can still introduce divergence between old and new structures. In relational systems, teams often use triggers, CDC pipelines, or transactional synchronization to reduce drift risk during migration windows.

2. What is backward compatible schema design in database migrations?

Backward compatible schema design means structuring database changes so that existing application versions continue to function without modification.

For example:

• Adding a nullable column is backward compatible
• Renaming or dropping a column is not

This is critical during rolling deployments where multiple application versions interact with the database simultaneously.

3. What are the risks of database schema changes in production?

Common risks of database schema changes in production include:

• Breaking compatibility with running application versions
• Causing downtime due to locks or blocking queries
• Data inconsistency can also occur when edits continue in the old schema after migration, but synchronization mechanisms fail to propagate those updates into the new structure.
• Making rollback difficult or unsafe

These risks can be mitigated by using safe migration patterns, staged rollouts, and compatibility-first database design. For example, if a trigger, CDC stream, or synchronization process misses an update, the old and new representations may diverge silently.