From defining quality to catching regressions: how Harness AI Evals scores AI agents before and after deployment.
We covered [why AI agents are different from traditional software (https://www.harness.io/blog/introducing-ai-evals), the silent failures, the plausible-sounding wrong answers, and the gap that pushed us to build Harness AI Evals.
But how do you actually test something that's non-deterministic by design? You can't write assertEqual on a summarization agent. So what does AI testing look like in practice, and how do the pieces come together to catch a regression before it ships? Let's get into it.
One system, one source of truth
Offline and online evaluation share the same metrics and datasets. The scoring logic you define for pre-deploy testing is the exact same scoring logic that runs against production traffic. There's no drift between what you test before shipping and what you measure after. One system handles both, so your quality bar is consistent across the entire lifecycle.
And it's built on an open-source evaluation SDK (Apache 2.0), so you're never locked in.
How it works in practice
You define your quality bar with four building blocks:
Target - the agent you're evaluating. A prompt template, an HTTP endpoint, or static outputs from a previous run.
Dataset - your golden test cases. Input + expected output + context. Versioned. Domain-grouped. Grows from production over time.
Metrics - what "good" means. Choose from over 50 built-in metrics spanning deterministic checks, safety and policy compliance, prompt injection detection, and trajectory evaluation for multi-step agents. You can also create custom LLM-as-Judge rubrics tailored to your use case.
Threshold - how strict? All-must-pass (blocking) or weighted composite (advisory).
These compose into Evals. Evals bundle into Suites. Suites attach to your release pipeline as a native quality gate.
Catching a regression before it ships
For our Customer Support Agent, we built an eval around a dataset of representative customer questions and expected answers. For each test case, AI Evals sends the input to the agent, captures its response, and evaluates that response against the quality signals we care about before allowing the change to move forward.
For this example, we chose three metrics because each tests a different part of the customer experience:
- TaskCompletionMetric checks whether the agent actually fulfils the customer's request.
- NoTyposMetric checks the quality and correctness of the generated response.
- PolitenessMetric checks whether the response maintains the professional, courteous tone expected from a support agent.
The point isn't to optimise for one score. A support response can be perfectly polite while still being unhelpful, or complete the task while delivering a poor customer experience. Testing multiple dimensions helps catch those failures before they reach production.
How a caught regression looks
Take one test case from the Customer Support Agent eval.
Customer: "Are the smart light bulbs compatible with Alexa?"
Expected answer: "Yes."
Agent response: "I couldn't find any smart light bulbs in our current product listings."
At first glance, the response looks reasonable. It's grammatically sound and polite. But it doesn't answer the customer's compatibility question.
| Metric | Score | Threshold | Verdict |
|---|---|---|---|
| TaskCompletionMetric | 0.200 | ≥ 0.200 | Pass |
| NoTyposMetric | 0.500 | ≥ 0.800 | Fail |
| PolitenessMetric | 0.950 | ≥ 0.800 | Pass |
Deploy blocked
This is exactly why evaluating an agent across multiple quality dimensions matters. Politeness alone would make this response look excellent at 0.95, while TaskCompletionMetric only just clears its threshold. The failed metric exposes a response-quality issue that would otherwise be easy to overlook.
Evals as a first-class pipeline stage
Other platforms connect evaluation to observability. But enforcement? That's still a script you wire into CI yourself. Raise an error, configure branch protection, and hope someone doesn't skip it.
Harness AI Evals runs as a native pipeline step. Same gates you already use. Fails the build on regression thresholds. Surfaces item-level results in the pipeline execution. Datasets, metrics, and thresholds live as config alongside your pipeline. No scripting. No glue code.
Enterprise-ready from day one
AI Evals inherits the full Harness platform:
- RBAC on every eval, dataset, and suite
- OPA policy governance on eval configurations
- Full audit trails on who changed what threshold and when
- Secrets management for model API keys via existing connectors
- SSO and data residency out of the box
- Centralized registry for prompts, agents, MCP tools, and skills with versioning
- Pipeline execution link - every eval run links back to the pipeline execution that triggered it
- Token tracking - input/output token spent per run, so LLM spend can be known by the owner
What's coming next
- Git-backed eval configurations (connect any repo, config fetched at runtime) - actively shipping
- Observe (Traces, Sessions, Annotations) scoped to individual targets - actively shipping
- Auto-generated datasets from production traces
- Drift detection with automated rollback
- Git-backed eval configurations (connect any repo)
- Pre-built suites for common agent patterns
- Human annotation queues
Here's a walkthrough of how it works:
Join the beta
We're building this with design partners. If you're running AI agents in production and want to shape what enterprise-grade evaluation looks like, we want to hear from you.
- Where does it hurt? Which production AI use case is your biggest pain right now?
- What's your current state? Are you collecting, attributing, governing, or optimizing?
- Which tools and providers would you like us to prioritize?
If you're interested in evaluating the product and helping shape the roadmap, let's talk.
Request to start the beta! (https://www.harness.io/demo/ai-evals)
