What engineering leaders are struggling with that vendors aren't solving

Ask an engineering leader whether AI made their team more efficient, and you'll usually get a pause before the answer. Not because they don't believe it. Because nobody ever told them what "efficient" was supposed to mean, or gave them a number to check it against.

Jonas Kröger, senior director of customer solutions at Upsun, spends his time talking to the people making these calls, and the gap he keeps running into isn't a tooling gap. It's that most of the promises being made to engineering leaders right now can't actually be checked.

The metric nobody has

"There are leaders concerned about cost spend, yes, but also about the promise of efficiency gains versus the reality," Kröger said. "Most of the teams we talk to today don't measure their efficiency. So how can you promise an efficiency gain if you have nothing to prove it?"

That's not a knock on those teams. It's just true of most engineering orgs, AI or not: cycle time, review turnaround, incident rate, few of them were being tracked consistently before agents showed up. Which means the "before" half of every "X% more efficient" claim is usually missing. Nobody's lying. There's just nothing to compare against.

More output isn't the same as more efficient

Even when teams do have a number to point to, it's often the wrong one. Kröger walked through why pull request volume, a metric plenty of teams reach for first, doesn't hold up.

"Is the output okay? We have more PRs. Is that more efficient? Not sure," he said. "Releasing more features more often doesn't necessarily mean you're more efficient. It's just more of the wrong things. So it might actually be less efficient, because suddenly your support or incident count goes up."

More code shipped faster isn't automatically progress. If it's the wrong code, or code nobody had time to think through properly, the volume just moves the cost downstream, into support tickets and incident calls, where it's harder to trace back to the thing that caused it.

An old answer to a new question

The fix Kröger points to isn't a new AI-specific dashboard. It's an old one most teams already know and rarely use consistently.

"The whole DORA metrics have been around for a while to measure software teams," he said, "and maybe there needs to be a connection to these things that is more obvious." 

DORA (deployment frequency, lead time, change failure rate, and recovery time) was built to answer exactly this question for software delivery in general. Most vendor pitches skip straight past it and go looking for a metric that sounds native to AI. Kröger's read is closer to the opposite: connect the new activity to a framework that already has years of shared meaning behind it, rather than inventing a new one nobody's agreed on yet.

The cost conversation nobody's finished having

Spend is the other place where the gap between promise and proof shows up. "How can I control spend?" is going to be an objection for sure, Kröger said, "because especially right now if you throw in your API key and you have no way of capping it, controlling it, having visibility, that is going to be a topic."

And underneath the cost question is a murkier one: does spending more actually buy better output? "They're lost," Kröger said of most organizations trying to answer that. "They don't know. And who does?" That uncertainty is a big part of why large organizations keep spending anyway, he suggested, since the fear of falling behind outweighs not yet having proof the spend is paying off.

What leaders are actually asking for

None of this means engineering leaders are hesitant to adopt AI. Kröger's read is closer to the opposite: the ones furthest along aren't asking whether to use agentic workflow;, they're stuck on how to scale what's already working past one team, and nobody's selling them help with that part.

Kröger doesn't lean on product differentiation to make his case. Differentiation, in his view, isn't going to come from the agent or the model underneath it, since that layer looks relatively similar everywhere. Instead it comes from the enablement and services wrapped around it: understanding where a team is actually stuck, and building the onboarding and workflow support that gets them unstuck. 

That's a service problem as much as a product one, and it's the part that gets skipped when a pitch leads with capability instead of with the question of how a team will know it worked.

That gap, between confident product promises and the unglamorous work of proving and scaling them, is exactly where Upsun Dispatch™ is trying to meet engineering leaders: cost visibility built in from day one, human-in-the-loop as a default rather than an afterthought, and a narrow, honest starting point instead of a promise to solve everything at once.

• New here? Read the introduction to Upsun Dispatch.
• Want to help build it? Apply to the founding design partner cohort.