Large language models are effective at security review for the same reason they are effective at many other tasks: they reason rather than pattern match. In plain terms, a traditional scanner checks code against a list of known bad patterns, the way a spell checker flags a misspelled word, regardless of what the sentence means. An LLM can instead follow the program's logic: trace a piece of attacker-controlled input through several layers of application code to determine whether it is reachable. That is a genuine advantage over many traditional static application security testing (SAST) tools, which can only catch what someone has already written a rule for.
That reasoning ability comes at a cost. Point the same model at the same, unchanged repository twice, and the two scans can return different findings, even though nothing in the code changed. Give it an open ended instruction like "find the vulnerabilities in this codebase" and it will spend an unpredictable amount of time and compute doing so, and the resulting report will vary in shape and detail from run to run. For a scanner meant to run automatically on every commit, that is a real problem: a security backlog that reshuffles itself with no corresponding code change is difficult for any team to trust or act on.
None of this is a reason to avoid using an LLM for security scanning. It is a reason you cannot simply point a model at a repository and call the result a scanner. What we built in Harness Security Test Orchestration (STO) is the layer that sits around the model, the part that decides what the model is asked to do and when, and turns a powerful but unpredictable reasoning engine into a scan that can run on every pull request and produce output you can trust and compare over time. In practical terms, that means four things: where the scan is triggered, how its scope is controlled, how its reasoning is constrained at each step, and how its output is standardized once the scan finishes.
What this delivers: the benchmark results
We benchmarked incremental scanning against full re-scans on three intentionally vulnerable test applications commonly used for security tool evaluation: dvpwa, govwa, and vulpy. Each repository was scanned three ways using the same model (GPT-5.5, with a 258k token context window per session): a full scan at the latest commit (HEAD), a full scan at an older base commit, and an incremental scan spanning the same base to HEAD range.
Incremental scanning significantly improves scan efficiency and accuracy by focusing the model on code changes, resulting in dramatic reductions in cost and time while maintaining consistent, validated findings.
On dvpwa, the full scan cost $26.20 and took 50 minutes; the incremental scan covering the same range cost $4.51 and took 16 minutes. It also surfaced more findings overall (20 versus 16): each retained finding kept the same title, description, and evidence carried over from the prior scan, not just a coincidentally similar count. govwa showed the same pattern: 83% cheaper, 72% faster, and a stable finding count (11 versus 12). On both repositories, the incremental scan caught every finding the full scan did; none were missed.
Because incremental scanning runs continuously rather than as a one-off, the savings compound: modeling 10 scans as one full scan plus nine incremental scans, dvpwa and govwa both come out 75% cheaper than running 10 full scans, and that gap widens further as scan count grows.
Where this runs: pipeline step versus one-off scan
The scanner itself is currently built on top of OpenAI Codex Security with a Mythos agent plugin and is invoked as a named "skill," a pre-defined task the underlying agent knows how to run (for example, security-scan, deep-security-scan, or security-diff-scan). Harness STO controls when that invocation happens and how much of the repository it covers. There are two ways this gets triggered:
- As a one-off, manually triggered scan. A user, or a separate pipeline, can also invoke a full deep-security-scan directly. Typical reasons include a compliance audit, a pre-release review, or onboarding a new repository for the first time when there is no scan history to build on yet.
- As a step inside an existing pipeline. A security scan stage runs automatically on every pull request or merge into a protected branch, and the pipeline can be configured to block the build if findings of a chosen severity or higher are detected. Because this runs continuously, the scan must be fast and inexpensive enough not to become the slowest part of someone's CI/CD process.
Both entry points run the same underlying four-phase workflow (threat model, finding discovery, validation, and attack path analysis). What differs is how much of that work the orchestration layer decides to redo on each run.
Incremental versus full scans: controlling scope, not just the prompt
A full scan runs all four phases from a blank slate: it builds or regenerates a threat model for the repository, ranks every file in scope by how much security relevant surface area it exposes, deep reviews the highest ranked files, validates every candidate finding using the strongest method available (for example a proof of concept, a test harness, or a debugger trace), and finally runs attack path analysis and severity scoring on whatever survives validation.
Redoing all of that on every commit is expensive and also part of why the results vary. The more code the model has to reason over from scratch, the more room there is for its conclusions to differ between runs.
An incremental scan changes what the model is asked to do, not what it is capable of doing:
- It loads the results of the previous scan as context, retrieved from the API, rather than starting from scratch.
- It revalidates every finding carried over from the prior scan against the current state of the code. A finding is never simply assumed to still be true. If the model can no longer confirm it, because the vulnerable code was fixed, refactored, or removed, that finding is dropped rather than carried forward indefinitely.
- It scopes new discovery, the most computationally expensive part of the pipeline (this is the step where the model actually reads and reasons over code, looking for new problems), to only the files that changed since the last scan, rather than re-ranking and re-reviewing the entire repository.
Validation runs against both newly discovered vulnerabilities and everything carried forward from the prior scan, and severity scoring and reporting proceed the same way regardless of which mode produced a given finding.
The point of this design is not that incremental scanning trades accuracy for speed. It reframes the task from "rediscover everything in this repository" to "confirm what was already found, and look carefully at what changed." That reframing is also what keeps the finding list stable between scans of unchanged code.
Focusing on the LLM: what each phase does for you
The single most effective way to get consistent, useful output from an LLM scanner is not a clever system prompt. We want to avoid asking one open ended question across the entire codebase at once. The workflow is instead split into four narrow phases, each with one job and a clear handoff to the next, so the model is never asked to hold in mind "understand this entire repository and find every bug in it" all at once.
Here's what that looks like for the OpenAI Codex Security integration, one of the engines we orchestrate. Other engines we support, like Mythos Agent, are structured differently, since the reasoning approach isn't identical across backends. But the underlying principle, narrow scope beats one big open question, holds regardless of which engine is doing the reasoning.
1. Threat model. Before looking for a single issue, the model builds a picture of what actually matters in your repository: what is exposed to the outside world, what data is sensitive, and who a realistic attacker would be. It also decides upfront what counts as Critical versus Low for your specific application, so severity ratings later on are not just a guess made in the moment. The output is a readable summary of what the scan considers risky in your codebase, and why.
2. Finding discovery. The model works through your code and records every candidate issue it finds, along with the evidence supporting it. It is deliberately told not to bundle similar-looking issues together, so two separate problems that happen to look alike do not get collapsed into one and quietly lost.
3. Validation. This is where the model checks its own homework. Rather than taking a candidate finding at face value, it tries to prove it: by building a small test, tracing the code path, or reproducing the behavior, using whatever method provides the strongest evidence. Only findings that can be backed up with real evidence move forward. Anything it cannot confirm is marked as unproven rather than dropped silently or reported as fact.
4. Attack path analysis. For everything that survives validation, the model has to argue against reporting it before it can argue for it: is this actually reachable by an attacker, does it matter in the context of this application, is it in scope at all? Only what survives gets a final severity rating and makes it into your report. This is the step that keeps the final list from being cluttered with things that are technically true but not actually worth your team's time.
You do not need to write or tune any of this yourself. It is built into how the scan runs. What you get at the end is a report where every finding has already been through a check for plausibility, evidence, and relevance before it reaches you.
Normalizing the output: making scans comparable to one another
A scan is only useful inside a pipeline if you can compare it to the last one. You need to know whether a given finding was fixed, whether something new just appeared, and whether your backlog is actually shrinking. That only works if the scan's output follows a consistent structure every time, even though the findings themselves vary.
A few things make that possible:
- Every finding looks the same on paper. Regardless of which run produced it, a finding always comes with the same basic information: what the problem is, where it is, how serious it is, and why. The model fills in the details, but it does not get to change the shape of the report.
- Every finding has a paper trail. Each one is tracked through discovery, validation, and final review, with a record of why it was reported, ruled out, or held for follow-up. That means a finding's status is never just a final yes or no with no explanation.
- The report format itself is checked before it ships. A separate, non AI check confirms the report has all the right sections and details before it is delivered. If something is missing, it gets fixed before you see it.
- Findings get a stable identity across scans. Once a finding is identified, it keeps the same identity from one scan to the next. That is what lets us say "this issue from last week is still open" or "this one just appeared," and it is also what makes incremental scanning's carry forward logic possible in the first place, and what lets us directly measure finding stability across scans rather than only measuring cost and speed.
The takeaway
This orchestration layer does not make the underlying model deterministic, and that was never the goal. The goal is to give the model a narrow, well defined task at every stage: a limited scope, a concrete rubric to check its own work against, and a fixed output format, instead of one large open ended question. That combination is what makes an LLM based scanner in Harness STO practical to run continuously as a part of CI/CD pipeline: fast and inexpensive enough to run on every commit, structured enough to compare across scans, and transparent about where its own assumptions, such as expecting changes to be pull request sized, begin to break down.
