AI agents don't crash, they return plausible-sounding wrong answers. AgentTrace monitors every LLM call, tool call, and decision as an execution graph, the observability stage of your Agent DLC.
A hallucination returns HTTP 200. A run that took fourteen needless tool calls looks identical to a clean one. Three gaps make plain tracing insufficient for agents.
A distributed trace tells you a call took 340ms and returned 200. It says nothing about whether the answer was right.
AgentTrace assembles every span into an execution graph that AI Evals scores for correctness, groundedness, and safety.
A microservice trace ends when the request returns. An agent's behavior only makes sense across a run and a multi-turn session.
AgentTrace groups every model call, tool call, and turn into sessions, so you can see behavior evolve or degrade over a conversation.
Even when tracing surfaces a bad run, it can only tell you after the fact. It can't act while a run is going wrong.
AgentTrace's runtime layer closes the loop: every investigated failure becomes a permanent regression gate for the next release.
Agents emit OpenTelemetry spans using Harness semantic conventions, a shared vocabulary for tool calls, LLM invocations, and retrieval. Data flows in through the existing Harness data platform, no new infrastructure required.
A single LLM call. Model, prompt tokens, completion tokens, temperature.
A tool or function call. Tool name, arguments, and result.
A vector search or document lookup. Query, number of results, source.
Agent / custom step. Top-level invocations, orchestration steps, and any user-defined step.
Every failure becomes a test
the loop turns production incidents into release gates
Investigate. The full span tree for any execution: input, output, latency, tokens, and cost for every step.
Annotate. Quality labels (Correct, Incorrect, Borderline) and free-text rationale, with a taxonomy customizable per team.
Export to Dataset. One click promotes a production failure into a golden eval case that gates the next CI release.
AI Evals and Agent DLC don't track your agents separately, they build directly on what AgentTrace already sees. Every eval score links straight back to the run that produced it.
Agents emit standard OpenTelemetry spans via the open source harness-sdk. Works with OpenAI, Anthropic, LangChain, or raw HTTP, no proprietary format to adopt.
Runs on the Harness data platform you already have: existing gateway, Kafka, and query services. Nothing new to provision or operate.
Every span carries a parent ID, reconstructing the complete causal chain from user query through reasoning, retrieval, and response.
All trace data lives in a high-volume, columnar analytics store, queryable via HQL through query-service. Any module that can issue HQL can access agent telemetry.
Auth, RBAC, and scope resolution inherited from NG Manager, the same governance already protecting your pipelines.
Score.trace_id is the join key that connects every eval score back to the trace that produced it, one data model across your Agent DLC, not two.
They're two halves of the same system. AgentTrace captures what happened: the execution graph. AI Evals judges whether it was good: the quality scores. Score.trace_id is the join key that links every eval score back to the trace that produced it, so production metrics and offline evals show up on one screen.
Depends on how they run. The harness/ai container emits OTel spans automatically with zero instrumentation code. Custom agents can use the AgentTrace SDK with native support for OpenAI, Anthropic, LangChain, and raw HTTP. Any OTel-compatible agent can also send spans directly to the OTLP endpoint.
Traces, spans, sessions, and eval scores live in a high-volume, columnar analytics store. Human annotations live in Postgres, linked back to traces by trace_id. Traces typically land in the analytics store within about 15 seconds of the agent execution.
Yes, the collection and evaluation layers are open source under Apache 2.0. harness-sdk (github.com/harness/otel-python-sdk) auto-instruments OpenAI, Anthropic, and LiteLLM and exports standard OTLP to any backend. harness-evals (github.com/harness/harness-evals) scores correctness, groundedness, safety, trajectory, and performance and gates CI through pytest. On Harness, the platform wires the two together and adds the trace viewer, human annotation, Export to Dataset, and analytics on top.