Observability · Agent DLC
Observability for AI Agents with AgentTrace
AI agents don't crash, they return plausible-sounding wrong answers. AgentTrace monitors every LLM call, tool call, and decision as an execution graph, the observability stage of your Agent DLC.
How it works
One pipeline, four stages
Agents emit OpenTelemetry spans using Harness semantic conventions. Data flows in through the existing Harness data platform — no new infrastructure required.
.png)
One pipeline, four stages — evals detect, actions enforce.
what it captures
Every execution is a tree of typed spans
Each span carries input, output, latency, start time, and a parent span ID that forms the tree. LLM spans additionally carry token counts and model attribution.
.png){kind=icon}
llm A single LLM call: model, prompt tokens, completion tokens, temperature.
tool A tool or function call: tool name, arguments, and result.
retrieval A vector search or document lookup: query, number of results, source.
agent / chain / custom Top-level invocations, orchestration steps, and any user-defined step.
.png)
Investigate, annotate, export
Every production failure can become a regression gate
The trace viewer is for investigation, not monitoring: come here when something went wrong. Annotate traces with quality labels and rationale, then export any bad run into a golden eval case in one click.
.png)
Investigate The full span tree for any execution: input, output, latency, tokens, and cost for every step.
Annotate Quality labels (Correct, Incorrect, Borderline) and free-text rationale.
Export to Dataset One click promotes a production failure into a golden eval case that gates the next CI release.
The foundation that powers Agent DLC
One trace layer, every AI product
AI Evals and Harness products supporting Agent DLC don't track your agents separately, they build directly on what AgentTrace already sees. Every eval score links straight back to the run that produced it.
.png){kind=icon}
OTel-native ingestion
Agents emit standard OpenTelemetry spans via the open source harness-sdk. Works with OpenAI, Anthropic, LangChain, or raw HTTP — no proprietary format to adopt.
.png){kind=icon}
Zero new infrastructure
Runs on the Harness data platform you already have: existing gateway, Kafka, and query services. Nothing new to provision or operate.
.png){kind=icon}
Full execution graph
Every span carries a parent ID, reconstructing the complete causal chain from user query through reasoning, retrieval, and response.
.png){kind=icon}
HQL-queryable
All trace data lives in a high-volume columnar analytics store, queryable via HQL through query-service. Any module that can issue HQL can access agent telemetry.
.png){kind=icon}
RBAC and tenant scoping
Auth, RBAC, and scope resolution inherited from NG Manager — the same governance already protecting your pipelines.
.png){kind=icon}
Shared model with AI Evals
Score.trace_id is the join key that connects every eval score back to the trace that produced it — one data model across your Agent DLC, not two.
The observability gap
Standard tooling wasn't built to catch this
A hallucination returns HTTP 200. Three gaps make plain tracing insufficient for agents.
Doesn't score quality
.png){kind=icon}
A distributed trace tells you a call took 340 ms and returned 200. It says nothing about whether the answer was right.
.png){kind=icon}
AgentTrace assembles every span into an execution graph that AI Evals scores for correctness, groundedness, and safety.
Wrong unit of work
A microservice trace ends when the request returns. An agent's behavior only makes sense across a run and a multi-turn session.
AgentTrace groups every model call, tool call, and turn into sessions, so you can see behavior evolve or degrade over a conversation.
Observing is passive
Even when tracing surfaces a bad run, it can only tell you after the fact. It can't act while a run is going wrong.
AgentTrace's runtime layer closes the loop: every investigated failure becomes a permanent regression gate for the next release.
Frequently asked questions
Common questions about AgentTrace
How does AgentTrace relate to AI Evals?
They're two halves of the same system. AgentTrace captures what happened: the execution graph. AI Evals judges whether it was good: the quality scores. Score.trace_id is the join key that links every eval score back to the trace that produced it, so production metrics and offline evals show up on one screen.
Do I need to instrument my agents?
Depends on how they run. The harness/ai container emits OTel spans automatically with zero instrumentation code. Custom agents can use the AgentTrace SDK with native support for OpenAI, Anthropic, LangChain, and raw HTTP. Any OTel-compatible agent can also send spans directly to the OTLP endpoint.
Where is trace data stored, and how fast does it arrive?
Traces, spans, sessions, and eval scores live in a high-volume, columnar analytics store. Human annotations live in Postgres, linked back to traces by trace_id. Traces typically land in the analytics store within about 15 seconds of the agent execution.
Is any of this available outside the Harness platform?
Yes. The collection and evaluation layers are open source under Apache 2.0. harness-sdk (github.com/harness/otel-python-sdk) auto-instruments OpenAI, Anthropic, and LiteLLM and exports standard OTLP to any backend. harness-evals (github.com/harness/harness-evals) scores correctness, groundedness, safety, trajectory, and performance and gates CI through pytest.
Get started
See what your AI agents actually did
Start capturing execution graphs today — no new infrastructure required.