We covered [why AI agents are different from traditional software (the link to the initial blog), the silent failures, the plausible-sounding wrong answers, and the gap that pushed us to build Harness AI Evals.
But how do you actually test something that's non-deterministic by design? You can't write assertEqual on a summarization agent. So what does AI testing look like in practice, and how do the pieces come together to catch a regression before it ships? Let's get into it.
One system, one source of truth
Offline and online evaluation share the same metrics and datasets. The scoring logic you define for pre-deploy testing is the exact same scoring logic that runs against production traffic. There's no drift between what you test before shipping and what you measure after. One system handles both, so your quality bar is consistent across the entire lifecycle.
And it's built on an open-source evaluation SDK (Apache 2.0), so you're never locked in.
How it works in practice
You define your quality bar with four building blocks:
Target - the agent you're evaluating. A prompt template, an HTTP endpoint, or static outputs from a previous run.
Dataset - your golden test cases. Input + expected output + context. Versioned. Domain-grouped. Grows from production over time.
Metrics - what "good" means. 47+ built-in metrics, custom LLM-as-Judge rubrics, deterministic checks, safety and policy compliance scoring (including prompt injection detection), and trajectory evaluation for multi-step agent logic (step reasoning, tool usage, final outcome).
Threshold - how strict? All-must-pass (blocking) or weighted composite (advisory).
These compose into Evals. Evals bundle into Suites. Suites attach to your release pipeline as a native quality gate.
Catching a regression before it ships
How a caught regression looks
Take a real example from our Customer Support Agent eval. A customer asks: "Are the smart light bulbs compatible with Alexa?" The agent responds: "I couldn't find any smart light bulbs in our current product listings." Looks polite, sounds reasonable. But it dodged the question entirely. Here's how AI Evals scored it:
Deploy blocked. The agent was polite (0.95) but didn't actually answer the question - the response wasn't relevant to the input, so NoTyposMetric (which evaluates relevance + grammar) flagged it. Without this gate, that response ships, customers leave, and nobody knows why.
Evals as a first-class pipeline stage
Other platforms connect evaluation to observability. But enforcement? That's still a script you wire into CI yourself. Raise an error, configure branch protection, and hope someone doesn't skip it.
Harness AI Evals runs as a native pipeline step. Same gates you already use. Fails the build on regression thresholds. Surfaces item-level results in the pipeline execution. Datasets, metrics, and thresholds live as config alongside your pipeline. No scripting. No glue code.
Enterprise-ready from day one
AI Evals inherits the full Harness platform:
- RBAC on every eval, dataset, and suite
- OPA policy governance on eval configurations
- Full audit trails on who changed what threshold and when
- Secrets management for model API keys via existing connectors
- SSO and data residency out of the box
- Centralized registry for prompts, agents, MCP tools, and skills with versioning
- Pipeline execution link - every eval run links back to the pipeline execution that triggered it
- Token tracking - input/output token spent per run, so LLM spend can be known by the owner
What's coming next
- Git-backed eval configurations (connect any repo, config fetched at runtime) - actively shipping
- Observe (Traces, Sessions, Annotations) scoped to individual targets - actively shipping
- Auto-generated datasets from production traces
- Drift detection with automated rollback
- Git-backed eval configurations (connect any repo)
- Pre-built suites for common agent patterns
- Human annotation queues
Here's a walkthrough of how it works:
Join the beta
We're building this with design partners. If you're running AI agents in production and want to shape what enterprise-grade evaluation looks like, we want to hear from you.
- Where does it hurt? Which production AI use case is your biggest pain right now?
- What's your current state? Are you collecting, attributing, governing, or optimizing?
- Which tools and providers would you like us to prioritize?
If you're interested in evaluating the product and helping shape the roadmap, let's talk.
Request to start the beta! (https://www.harness.io/demo/ai-evals)
%2520copy.png)
All this author’s posts
Shibam Dhar is a developer Relations professional with years of experience advancing developer experience, education, and community engagement.
All this author’s posts
Uri is a General Manager at Harness and focuses on products such as Infrastructure as Code, Artifact Registry, AI Test Automation, and AI Evals.
