Learn how we tackle data trust in DevOps. Discover our multi-layered validation approach combining AI evals, API checks, and product verification to ensure reliable AI-powered software delivery insights.
TL;DR
- Multi-layered validation: Trust is built by verifying results across AI evaluation criteria, schema traversal, API truth sources, and direct product inspection.
- Engineering workflow: Manual checks were replaced by automated, repeatable validation using tools that define scenarios, extract HQL, and generate comparative reports.
- Production-first mindset: Evaluation scenarios now use real-world production data, including edge cases such as retries and null values, to reflect data freshness and the true operational context.
- Shift-left evaluation: Testing is now integrated into the development lifecycle, ensuring scenarios, evaluation metrics, and expectations are defined before building capabilities.
Modern software delivery generates an enormous amount of operational data. Pipeline executions, builds, deployments, services, environments, artifacts, security findings, and other resources are continuously created across the software delivery lifecycle. The challenge is not simply collecting this information. It is understanding how these resources relate to one another and using that context to answer meaningful questions.
The Harness SDLC Knowledge Graph brings this context together by representing software delivery data as entities and relationships. Instead of requiring users to navigate multiple product views or construct queries manually, the Knowledge Graph can interpret natural-language questions, translate them into HQL (Harness Query Language), traverse relevant entities and relationships, and return an answer.
“Which pipelines failed the most in the last 30 days?”
This sounds like a simple question. But producing a useful answer requires much more than generating a syntactically valid query. The system needs to identify the right entities, traverse the right relationships, retrieve the right data, account for the freshness of that data, and ultimately return an answer that agrees with what the underlying systems and the user actually see.
This leads to an important engineering question:
How do you know that a Knowledge Graph answer is correct?
Our initial approach was straightforward: ask a question, inspect the generated HQL, look at the response, and determine whether the result appeared correct. That approach worked until it didn't. A seemingly reasonable answer could depend on an unavailable entity, an unpopulated relationship, stale data, or a result that differs from the underlying Harness API or product experience.
More importantly, we realized that our goal was not simply to prove that individual queries worked. We wanted to establish confidence in the Knowledge Graph's ability to consistently represent software delivery data and produce answers that held up across multiple validation paths. That became the foundation for our approach to evaluation.
A Knowledge Graph answer is only as trustworthy as the path behind it
One of the first questions we tested was:
“Which pipelines have the highest failure rate?”
The Knowledge Graph generated the query, retrieved pipeline execution data, and identified pipelines with the highest failure rates.
We then asked a follow-up:
“What are the common failure points?”
The response looked reasonable. However, when we inspected the generated HQL, we discovered that the query depended on relationships that were not yet available in the graph. The answer was plausible. The underlying query path was not sufficient to support it.
That changed how we thought about validation. Instead of asking only:
“Does this answer look right?”
We started asking:
“What needs to be true for this answer to be right?”
That gave us a useful model for thinking about Knowledge Graph evaluation:
Question → HQL → Entities → Relationships → Data → Source of truth → Product experience
Every layer in that chain can introduce a different class of failure. A query can execute successfully while referencing incomplete schema information. A relationship can exist while not being populated. The graph can contain the right information while being behind the source system. And even when the underlying data is correct, the final response may still fail to address the user’s question. Validation, therefore, needed to happen at multiple levels.
1. Start with the schema
The first question is simple:
Does the query reference entities that actually exist?
We started with 36 evaluation prompts covering scenarios such as pipeline failure analysis, build-time optimization, flaky test identification, security compliance, and deployment root-cause analysis. For each prompt, we inspected the generated HQL and verified that the required entities were available.
The first pass validated 19 of the 36 prompts, or approximately 53%. The remaining scenarios depended on entities that were not yet available, including entities required for test analysis, cache-miss analysis, and error-pattern analysis.
This gave us an important baseline, but it also exposed a limitation: Entity availability does not indicate whether the query can retrieve the required data.
2. A registered relationship is not necessarily a usable relationship
Consider a question such as:
“Which build stage takes the longest in my pipeline?”
A generated query might find a build execution entity and then traverse the relationship that connects builds to stages. Structurally, that query may be completely valid. But the existence of a relationship in the schema does not guarantee that the relationship is populated with usable runtime data.
During validation, we found cases where relationships were registered, but the upstream ingestion path had not yet populated the corresponding data.
This led us to explicitly separate three checks:
Entity validation: Does the required entity exist?
Relationship validation: Can the required relationship be traversed?
Data validation: Does that traversal actually return the expected information?
This distinction is important because schema availability and data availability are different problems.
3. A valid query can still produce the wrong answer
The next failure mode was even more subtle. For one pipeline-failure scenario, the Knowledge Graph returned 21 failures for a pipeline over a selected period. The corresponding Harness API returned 29.
The HQL was valid. The entities existed. The relationships worked. The result still differed.
The reason was ingestion lag: at the time of validation, the Knowledge Graph was eight executions behind the source API.
This exposed an important principle:
Query validity is not the same as answer correctness.
A system can successfully execute a query and still return an incomplete or inconsistent result with the source system.
It also taught us that freshness needs to be treated explicitly. An ingestion delay does not necessarily mean that a query or the graph is broken. Different systems can have different consistency and freshness characteristics. What matters is understanding those characteristics and accounting for them when evaluating the result.
4. Establish a source of truth
Once we identified these cases, we changed our validation model. Instead of reviewing a Knowledge Graph response in isolation, we began treating each result as something that should be verified against an authoritative source whenever one was available.
Where the underlying information was exposed through a Harness API, we programmatically compared the Knowledge Graph result with the API response.
The workflow became:
- Generate the HQL for the natural-language question.
- Execute the HQL against the Knowledge Graph.
- Query the corresponding Harness REST API.
- Compare the results.
Depending on the scenario, we compared several dimensions:
| Dimension | Question |
|---|---|
| Entity match | Are the same resources returned? |
| Count match | Do aggregated values agree? |
| Rank match | Is the ordering consistent? |
| Data availability | Does the Knowledge Graph contain the information required to answer the question? |
This comparison surfaced problems that were difficult to identify by inspecting the final response alone. We found scenarios where queries returned the correct pipelines but ordered them incorrectly, included deleted pipelines, or double-counted failures associated with manually retried executions. These were not HQL syntax problems. There were correctness problems.
5. Validate against the product, not just the API
APIs provide an excellent source of truth when the information required for validation is exposed programmatically. But they do not always expose everything a user sees. For scenarios where the required information was not available through an API, we validated the result directly against the Harness product.
We followed the same path a user would take: navigate to the relevant module, locate the resource or execution, apply the appropriate filters and time range, and compare the product's display with the Knowledge Graph result.
This gave us two complementary validation paths:
API-backed validation: Programmatically compare Knowledge Graph results with the corresponding API response.
Product-backed validation: Compare the result with the information presented in the Harness product.
The two approaches catch different classes of problems. API comparisons can reveal differences in counts, ordering, and data freshness. Product validation can reveal missing resources, incorrect associations, scope differences, or relationships that do not reflect how users actually work with resources in Harness.
At this point, an important pattern emerged:
Confidence did not come from any single validation method. It came from the agreement between multiple independent checks.
That idea became central to the next stage of our work.
Introducing AI evals
As our evaluation scenarios grew, another problem became apparent. Manual validation was useful for debugging individual scenarios, but insufficient to continuously measure the capability's overall quality.
We needed a repeatable way to answer:
How well does the Knowledge Graph perform across a representative set of questions?
This is where AI evals became useful. An AI eval provides a structured way to run a capability against a defined set of scenarios and measure the results against explicit evaluation criteria.
For our Knowledge Graph use cases, we think about an evaluation in four parts:
Dataset → Target → Metric Set → Evaluation
Dataset: What are we testing?
The dataset contains the scenarios we want to evaluate. Each item represents a question or task and the information needed to evaluate the resulting response. For example: “Which pipelines failed the most in the last 30 days?” The important property of the dataset is repeatability.
Instead of asking a question manually and deciding whether the response looks right, we can run the same scenario again after a schema change, an ingestion change, or an update to the underlying Knowledge Graph behavior.
Target: What are we evaluating?
The target defines the system that receives the evaluation input. For a Knowledge Graph scenario, the target can be the Knowledge Graph experience exposed through the Unified Agent. The same evaluation model can be used for other types of targets as well, including prompts, agents, or precomputed outputs.
Metrics: What does a good answer mean?
One of the biggest advantages of an eval is that it lets us define what “good” actually means. Depending on the scenario, we can evaluate dimensions such as Correctness, Groundedness, Answer Relevance, and Actionability. These metrics can be configured with thresholds and weights.
For example, a Knowledge Graph evaluation can combine a G-Eval-based metric with an Answer Relevancy metric, with separate thresholds and weights. This provides a measurable view of different dimensions of response quality.
Evaluation: Run it repeatedly
Once the dataset, target, and metric set are defined, the evaluation can be executed repeatedly. The system runs the target against the evaluation dataset, applies the configured metrics to each result, and produces an evaluation report. We can measure the overall pass rate, metric-level scores, the number of successful and failed items, score trends across runs, and changes in specific quality dimensions.
Combining AI evals with Knowledge Graph validation
AI evals solve one part of the problem: they provide a structured, repeatable mechanism for measuring response quality. Knowledge Graph validation solves another: it helps us understand the underlying reasons for success or failure.
Together, they create a folded validation approach where multiple signals reinforce each other:
- Define scenario
- Create an evaluation dataset
- Run against the Knowledge Graph target
- Measure correctness, groundedness, relevance, and other metrics
- Validate the graph path: entities and relationships
- Compare with Harness APIs, where available
- Validate against the product where required
- Investigate failures across query, schema, relationships, ingestion, and data
- Fix and re-run
The AI eval indicates how well the response meets the defined criteria. The underlying validation helps explain whether that result is supported by the graph, source data, and product experience. That combination is much more powerful than relying on either approach alone.
Turning evaluation into an engineering workflow
As the evaluation set grew, manually repeating these checks became increasingly difficult. We therefore built /validateKG, a Claude Code skill that automates much of the validation workflow.
For a scenario, the workflow can:
- Read the expected behavior.
- Extract the generated HQL.
- Execute the query against the Knowledge Graph.
- Query the corresponding Harness API where available.
- Compare the returned entities and data.
- Produce a validation report.
Where complete API validation is unavailable, the automated result can be supplemented with product-backed verification. The important part is not the automation itself. It is the ability to turn a validation scenario into something repeatable, measurable, and easy to run again.
What this validation gives us
The most important outcome of this work is not simply a higher evaluation score. It is greater confidence in the Knowledge Graph itself.
We deliberately avoid relying on a single validation mechanism. A Knowledge Graph scenario can be evaluated from several perspectives:
- AI evaluation measures response quality.
- Graph validation verifies that the required entities and relationships exist and can be traversed.
- API-backed validation compares the result with an authoritative source when available.
- Product-backed validation verifies that the result reflects what a user can actually see in Harness.
- Data validation helps account for freshness and ingestion behavior.
Each path catches issues that the others may miss. A response can be relevant but rely on an invalid graph path. A valid graph query can return stale data. An API comparison can confirm the values while missing a product-level scope issue. Product validation can reveal an association that is technically valid but does not align with how the resource is represented to users.
By combining these checks, we moved from asking whether an answer looked correct to building evidence that it was correct across multiple layers.
This folded validation approach has strengthened the Knowledge Graph as its coverage has expanded. It gives us a repeatable way to identify gaps, understand failure modes, and build confidence before making new scenarios available to customers.
The result is not simply a Knowledge Graph that can answer more questions.
It is a Knowledge Graph with a growing body of evidence behind those answers.
What we learned
Production data exposes problems that happy-path tests do not
Early testing often relies on controlled examples. Production data is different. It contains null values, deleted resources, retries, unexpected data shapes, changing relationships, and other edge cases that are difficult to reproduce in simple test environments. Representative evaluation scenarios, therefore, matter as much as the evaluation framework itself.
Freshness is part of correctness
The 21-versus-29 execution example also changed how we think about correctness. A difference does not always mean that one system is wrong. It can indicate different characteristics of freshness or consistency. That means the evaluation needs sufficient context to distinguish between a genuinely incorrect result and an expected difference due to data propagation.
Product context matters
Knowledge Graph quality is not only about matching individual values. The graph also needs to represent how resources relate to one another in the product. Validating against the Harness experience helps us verify scopes, associations, statuses, resources, and operational context from the perspective that matters most: the user.
Evaluation needs to be part of development
One of the most useful changes we made was moving validation earlier in the development process. Our workflow now starts by defining the scenario and expected behavior, identifying the authoritative source, deciding how the scenario will be evaluated, and only then building or refining the capability. This makes evaluation part of feature development rather than a final check before release.
Where we are today
The validation approach has expanded beyond the initial pipeline scenarios. We have applied it across Pipelines, Continuous Delivery, Continuous Integration, Security Testing Orchestration, Software Supply Chain Assurance, and Platform, using Harness APIs for programmatic comparison where available and direct product validation where necessary.
Across the initial evaluation set, 22 of 27 scenarios were validated through API-backed and product-backed checks, representing approximately 81% validation coverage at that stage. The remaining scenarios were partially validated, where the required APIs were not yet available.
This is important because the metric is not being presented as a generic “AI accuracy” score. It represents how much of our scenario set we were able to validate through the available programmatic and product-level evidence.
As coverage expands, the same methodology gives us a consistent way to validate new entities, relationships, and use cases. The goal is not simply to increase the number of questions the Knowledge Graph can answer. The goal is to expand that coverage without compromising confidence in the answers.
What comes next
Broader Knowledge Graph coverage
As new entities and relationships become available, the same evaluation methodology can be applied to identify gaps earlier and reduce the time required to validate new scenarios.
Anomaly validation
Some questions require more than validating an aggregate. Consider: “Why did my build time increase last week?” A useful evaluation needs to determine whether the system correctly identifies the underlying change, not simply whether the reported build time is accurate.
Recommendation validation
Recommendation-oriented scenarios introduce another challenge. If the system recommends changing a deployment timeout, for example, we need to evaluate whether that recommendation is supported by the operational context and available historical evidence.
Continuous evaluation
Today, many evaluations operate as development or release gates. Running them continuously could help identify regressions caused by schema changes, ingestion issues, or changes in underlying APIs before they become visible to users.
Conclusion
Building a Knowledge Graph that can answer questions is one problem. Building one that users can trust is another.
Our biggest lesson has been that trust cannot come from a single successful query or a single evaluation score. It comes from validating the complete path behind an answer and looking at that answer from multiple perspectives.
- Does the required entity exist?
- Can the necessary relationships be traversed?
- Is the underlying data populated and sufficiently fresh?
- Does the result agree with an authoritative source?
- Does it reflect what users see in the product?
- Does the final response meet the expected quality bar?
By combining AI evals with schema, relationship, data, API, and product validation, we turned these questions into a repeatable engineering process.
That process has given us increasing confidence as we expand Knowledge Graph coverage across Harness. New entities and relationships can be evaluated systematically, failures can be traced back to their source, and supported scenarios can be made available with a clearer understanding of their reliability.
The goal is not simply to build a Knowledge Graph that can answer more questions.
The goal is to build one whose answers we can explain, validate, and trust.
And that is ultimately what makes the Knowledge Graph a stronger foundation for natural-language access to software delivery data.
Learn more about the Harness Knowledge Graph.
