Harness AI Evals tests AI agent quality in CI/CD, using golden datasets and quality gates to catch behavioral regressions before production.
Traditional software fails loudly: a test breaks, an API error, a container crashes. AI applications do not. An agent can return 200 OK, respond fast, pass every infrastructure check, and still hand a customer a wrong answer.
A support agent telling customers the wrong billing terms or return rules at scale is not a bug ticket. It is a brand, support-cost, and compliance problem that a traditional pipeline waves straight through to production. So the question for CI/CD becomes:
How do you decide whether an AI application is actually good enough to ship?
To test this, Harness AI Evals ran inside the deployment pipeline of a simple e-commerce support agent as a blocking quality gate:
The setup
- Dataset: 32 golden support scenarios (payments, orders, shipping, returns, cancellations, support, promotions, security).
- Metrics: Answer Relevancy, Task Completion, and Toxicity. Each scores a different part of the response. (See How do you actually test an AI agent? for how these are defined.)
- Gate: blocking, 70% pass threshold. Below the bar, no production deploy.
The first version failed the gate
An early run passed only ~65% of cases. Nothing was broken in the traditional sense: the build succeeded, the service deployed to QA, and the endpoint was live. But the behavior was not good enough, so the blocking step stopped the deploy. A 65% → FAIL number is only useful if you can see why, and the failed cases showed the pattern immediately:
The first row is the dangerous kind. Relevancy is 1.0, yet Task Completion is 0.3 because the fact was wrong. This is the regression that survives traditional testing: the API works, the reply reads naturally, and the customer gets bad information. The other rows are communication failures. The agent often had the right information but buried it, skipped part of it, or answered a nearby question instead.
From a vague complaint to an action list
Without evaluation, feedback is just "the agent needs improvement." The failed cases made it concrete, and five of the eight recurring failures were traced to incorrect, missing, or incomplete knowledge:
- Wrong payment and billing facts
- Missing cancellation and customer-service details
- Incomplete shipping information
- Inconsistent gift-card answers
- Indirect responses to yes/no questions
The tempting shortcut is to lower the bar until the gate turns green. That defeats the purpose. The honest path is to fix the application: correct the knowledge, and tune prompts so the agent leads with a direct answer and refuses to reveal its instructions. After iterating, later runs reached 75% and 78% against the same 70% bar, held it across repeated runs, and the production deploy continued.
AI is non-deterministic, so design for it
The same target, dataset, and config did not always produce identical results, with one comparison showing 24 passing cases versus 22. Sometimes the agent itself varied. Asked "Do you sell gift cards?", it scored Task Completion 1.0 in one run and 0.2 in another, answering "yes" once and "no" the next time. That is a real consistency problem, not noise.
Two rules follow:
- Trust repetition, not luck. An improvement counts only once it holds across repeated runs.
- Read a red gate before reacting. A failure can mean a genuinely bad answer (block it) or a broken judge or target (an infrastructure issue, not quality).
Why it belongs in the pipeline
Running the eval inside the delivery pipeline, rather than a separate tool, is the point. The eval and the deployment gate are one blocking step next to build and deploy, completing end to end in roughly seven to eight minutes. No glue code shuttling scores from an external dashboard, no second system deciding whether the build ships. The quality signal and the delivery decision are the same decision, made in the same place.
Traditional pipelines ask whether the app compiled, tests passed, and the service is healthy. AI applications add one more: is it behaving well enough to put in front of users? Because the most dangerous AI regression does not crash anything. It looks healthy while confidently giving your customer the wrong answer, and that is exactly what to catch before it ships.
Try it in your own pipeline
Harness AI Evals lets teams define golden datasets, pick the metrics that matter, and turn results into a blocking quality gate inside the pipeline, so a build only ships if the AI behaviour clears the bar. The fastest way to see it work is to watch the AI Evals demo and try it against your own agent.
Want to go deeper first?
- Explore the AI Evals product for what it does and how it fits your pipeline.
- Introducing Harness AI Evals for what it is and why it exists.
- How do you actually test an AI agent? for a closer look at how the evaluation works.
