The AI trust dial: from local agents to autonomous software factory

There are many conversations about the use of AI, particularly how engineering teams are using it in their coding workflows. Manual work is being replaced by agent-driven automation, and human value increasingly lies in the higher-order work: writing specs, thinking through architecture, steering the direction, exercising taste, and reviewing the output.

In this context, the constant debates around model selection, productivity, and KPIs are just part of a larger question: to what extent can the generated output be trusted? And what can you do upfront to trust more by default?

Fabien Potencier, co-founder and CTPO of Upsun, has been thinking about this problem for a long time. Having watched the industry move through several waves of AI adoption, from autocomplete to copilots to agents running in the cloud, he has seen the same tension emerge at every stage: the gap between what an agent can do and what a team is actually comfortable letting it do unsupervised.

Why should we focus on trust?

When an agent takes actions with real consequences, humans either over-supervise everything mechanically, approving actions they are no longer actually reading, or they go to the opposite extreme: what Fabien Potencier calls "yolo mode", giving up on oversight entirely and letting the agent do whatever it wants by default.

He describes it best: "At first, you don't trust the LLMs. You want to be in the loop to make sure you can validate everything the agent is doing." But validating everything can quickly snowball if you don't establish shared context, defined skills, and clear guardrails first, leaving you facing the same problems over and over.

In this scenario, trust is all the context you build over time: giving the agent the right understanding of what you want to do and how you want to do it, so that it can earn the autonomy it is given. It is the deliberate work of building foundations that makes meaningful autonomy possible, and the prerequisite for everything that comes after.

So, can we move from full supervision to meaningful autonomy without compromising quality or the long-term maintainability and security of what gets produced?

It's a dial, not a switch

The framing Fabien Potencier uses to explain this is that trust is a dial, not a switch. A switch is binary: on or off, supervised or autonomous. A dial, on the other hand, is graduated; it moves in increments.

At first, the dial is turned all the way toward human oversight, meaning every action requires approval and review. At that stage, every decision has a human in the loop and, as the track record builds, the dial can gradually turn.

"The trust is earned because you've done the work," Fabien Potencier explains, "and at some point you can remove some of the gates because you have the confidence that it's going to be good every single time."

But that confidence does not come from the agent simply repeating the same task many times. It needs the human doing retrospective work after each run: providing feedback, tweaking prompts, refining skills, so what was learned gets encoded into the process itself.

The goal is to progressively move anything that can be made deterministic into scripts and skills that work the same way every time. Once something is encoded, you are not trusting the agent to figure it out again; you are trusting a process you built and know will hold.

The human steps back not because they have been replaced, but because they have built enough guardrails and encoded enough of the process to know when their attention is actually needed and when it is not.

The actual challenge

This change seems pretty straightforward, but the difficulty is not technical; it is behavioral. Most teams struggle to move the dial at all because they simply don't trust output they didn't produce themselves. Consequently, the engineer becomes, in Fabien Potencier's words, a "human checklist": not a decision-maker or architect, but someone who simply checks boxes one after another while the agent does the actual work.

Fatigue becomes unavoidable. "The fatigue comes from having to approve every single step in all the conversations with agents," he says. He argues that this is not a sustainable model and does not yield better outcomes; it simply leaves engineers tired.

"Instead of trying to move as fast as possible from day one, we should first slow down, understand how agents work, and how to make them better; it's all about context management. It takes time; it's not a science yet, so people need to invest time to learn how to do it," Fabien Potencier says.

The compounding effect

As trust accumulates, so does knowledge. "Every single run makes the next one better," he says, "because we have shared skills and prompts." When agents run on shared infrastructure rather than individual machines, the skills refined after each run, the prompts improved through each retrospective, and the context built up over time belong to the whole team, not just the person who ran the workflow. 

The workflows improve, the context deepens, and the agent gets better at understanding not just the task, but also the standards and conventions of the specific team it is working with.

This compounding effect separates teams getting value from agents from those stuck in supervised mode. The result isn't just code, but a system that understands their codebase and can be trusted to apply their standards consistently.

What this means for engineering leaders

The trust dial is not a product feature; it's a way of thinking about the relationship between human judgment and automated systems, a relationship that every engineering team will have to develop deliberately or stumble into reactively.

Building trust incrementally, encoding what works, and earning autonomy through evidence rather than assumption is the prerequisite for everything that comes next: moving agents off individual machines, running workflows across the whole team, and eventually operating the kind of autonomous software factory where the agent does its work and the human focuses on what only they can do.