Your AI code spend is soaring, here's how leading engineering orgs are proving the ROI | Harness Blog

AI coding tools have gone from novelty to default in the span of about two years. Adoption is no longer the question; spend is. Every engineering leader now faces some version of the same question from finance: we know we're spending more on AI, but what are we actually getting for it?

That question was the center of a recent panel discussion hosted in partnership with Harness, featuring Martin Reynolds (Field CTO, Harness), Claire Knight (Director of Engineering, n8n), and Lawrence Jones (Product Engineer leading AI, Incident.io), moderated by Amanda Sopkin (Datadog). Here are the biggest takeaways for anyone trying to build a real measurement framework instead of guessing.

Stop measuring activity. Start measuring outcomes.

The panel converged quickly on one point: raw activity metrics (lines shipped, PRs merged, tokens burned) are noisy and easy to game. As Claire put it, teams that "shipped quicker" often also shipped more defects, which isn't obviously a win.

Lawrence's team at Incident.io took a more rigorous approach. After a deep dive across roughly 50 engineers, cohorted by tenure and adoption timing, they found that every dollar spent on AI tooling would have cost roughly $3.50 to achieve the same output through hiring. But he was careful to note the number only became trustworthy once it was checked against qualitative reality: daily observations of non-engineers building internal tools, PRs resolving customer requests in 15 to 20 minutes, and large-scale refactors that simply weren't feasible before. Numbers without ground-truth context, he warned, are "deeply suspect."

The message: pair quantitative throughput metrics (PRs shipped, cycle time, defect rate) with qualitative signal, and treat any metric that moves without a corresponding shift in real-world behavior as a red flag.

Attribution is the foundation everything else is built on

Martin argued that granular cost attribution isn't optional; it's the prerequisite for every other kind of AI ROI analysis. Harness's internal approach ties spend to the specific ticket, PR, test run, or workflow stage that generated it, so a team can say "this feature cost $15 in AI spend and shipped successfully to production" rather than looking at an undifferentiated monthly bill.

That level of detail changes conversations fast. Martin's example: a $1,000 bug fix that used a frontier model for a task that would've taken five minutes without it. Without attribution, that's invisible in a lump-sum invoice. With it, it's an immediate flag for model-routing and task-matching improvements.

Claire extended this beyond coding spend to the growth team, where AI accelerated the entire experimentation loop, from generating hypotheses to synthesizing feature-flag data to evaluate them. The lesson: the efficiency gains aren't only in token counts. They show up in how fast teams can close the loop between an idea and a validated answer.

You need a baseline, and it's later than you think

Both Lawrence and Martin flagged the same structural problem: most organizations don't have a clean "before AI" baseline to compare against, because the industry has changed so much in the last 12 to 24 months that "before" keeps moving. Lawrence's advice: measure across the entire software lifecycle, not just one stage. A team might be spending more time in planning and less in shipping, a real efficiency gain that a single-dimension metric (like PR count) would completely miss, or even penalize.

Martin's related point: treat the full path from idea to production (including testing, security scanning, resiliency checks, and provisioning) as one measurable pipeline. AI adoption in one stage can just create a new bottleneck downstream ("we're shipping more code, but it's all stuck in test").

Budgeting for a moving target

On the question of planning for rising AI costs, the panel was candid that unpredictability is the real challenge, not the price trend itself. Lawrence pointed to model price collapses, citing an 80% price cut on a frontier model that made previously uneconomical use cases viable almost overnight, as a preview of Jevons paradox playing out in real time: cheaper tokens don't necessarily mean lower total spend, because usage expands to fill the headroom.

That unpredictability is why Lawrence's customers are increasingly asking for flat, predictable pricing over metered billing; a VP of Finance does not want to watch a usage graph that looks like a hockey stick. Martin agreed governance is coming, comparing the current moment to the early cloud-cost-management era, just compressed from a decade into months.

Where should budget ownership sit?

There was genuine, productive disagreement here. Lawrence argued individual developers shouldn't be handed unconstrained personal AI budgets, because they lack the organizational context to judge what's "sensible" spend: a developer can blow through what amounts to a senior contractor's daily rate on a single frontier-model session without any signal that it was a bad call. His preferred model: teams negotiate a budget with the business based on their actual use case (e.g., an AI-heavy team gets a different watermark than a team doing routine feature work), and individuals operate within that, with the ability to burst through when justified.

Martin's view leaned further toward decentralization: pushing budget visibility and responsibility down to the individual level, similar to how cloud spend democratized over time, so people internalize cost the way they would any other resource.

Both agreed governance tooling is where this gets solved. Martin mentioned Harness's approach of merging AI token spend, infrastructure spend, and developer efficiency data into a single view, sliced by team, ticket, model, or use case. Claire's team is earlier in that journey, currently piping OpenTelemetry data with custom headers to capture token usage, with finance integration still a work in progress.

The risk nobody's naming: are we automating the wrong process?

Claire raised what might be the most important point of the session: most organizations are applying AI to accelerate the existing software development lifecycle (the same steps, just faster) rather than asking whether those steps still make sense at all. Many process steps exist because coordination and building were expensive. If AI makes building cheap, some of that process may no longer be earning its keep. Her advice: don't just chase the next bottleneck with more AI; periodically ask whether the pipeline itself needs to be redesigned.

What to actually do with this

If there's a synthesis of the panel's advice, it's this:

• Attribute spend granularly (by ticket, feature, model, and lifecycle stage) before trying to calculate ROI at all.
• Measure the whole lifecycle, not just coding throughput, so gains in one stage don't hide new bottlenecks in another.
• Pair quantitative metrics with qualitative checks. If the numbers move but the ground truth doesn't, question the numbers.
• Use deliberately counterbalancing metrics (speed vs. defect rate, throughput vs. time-to-customer-impact) to avoid optimizing for a single, gameable number.
• Decide budget ownership deliberately. Neither pure centralization nor pure individual autonomy fits every team; align it to how much organizational context a given team actually has.
• Periodically revisit the process itself, not just the tooling layered on top of it.

The panel's clearest consensus: there's no universal ROI formula yet, and anyone claiming otherwise is probably selling something. But the organizations getting real signal are the ones treating this like a measurement science problem, with baselines, attribution, and healthy skepticism of their own metrics, rather than a vibes-based budget negotiation.

For more information, you can watch the full webinar here or go to the Harness website to learn more about the Cost Management Agent.