Agent security starts with where the agent runs, not how it behaves

When engineering teams evaluate AI agents, the first questions are usually about capability. Which model performs best? How much faster can it write code? What's the return on investment? Security, if it enters the conversation at all, tends to come later.

Patrick Dawkins, Principal Software Engineer at Upsun, thinks that's backward.

Over the past year, he's been building the infrastructure that enables AI agents to operate safely within engineering teams. His conclusion isn't that AI models are inherently unsafe. It's that most teams are running them in environments that were never designed for them.

The problem with personal machines

When an agent runs on a developer's laptop, it typically needs three things to do anything useful, and having all three at once is where the risk begins. Security researcher Simon Willison calls this the lethal trifecta: giving an agent access to untrusted data, the ability to produce side effects, and access to sensitive information like credentials or customer data. "You're in a danger zone if you have those three together," Dawkins says.

The challenge is that useful agents usually need all three. A code review agent has to read repository data, post comments back to GitHub or GitLab, and authenticate with repository credentials. Those capabilities are exactly what make it useful, and what make it risky.

Most teams running agents on personal laptops are in this situation right now, and the majority do not know it.

What nobody can see

The bigger issue isn't just what agents can access; it's what nobody else in the organization can see.

When an agent runs on a developer's machine, its activity is invisible. There is no central audit log showing what it read, which tools it called, what it wrote to disk, or what it sent over the network. If a prompt injection attack causes the agent to exfiltrate data or take an unintended action, the investigation starts from almost nothing. "You don't see if customer data is being sent to the wrong place," Dawkins says. "We don't share customer data with AI models at all, but secrets, credentials — it starts to feel wrong."

Agents running locally can access credentials, SSH keys, environment variables, and other sensitive files that already exist on a developer's machine. They may not be instructed to use them. But they are accessible, and an agent operating under a malicious or poorly constructed prompt can reach them.

The shift to cloud-based tooling over the past decade was partly about solving exactly this problem for infrastructure. Engineering teams moved away from running servers on individual machines because governance, visibility, and incident response became unmanageable at scale. The same logic applies to agents. Most teams have not made that connection yet.

Guarantees before guardrails

Dawkins draws a distinction that most security conversations around agents miss: the difference between guarantees and guardrails.

Guarantees are structural and enforced at the system level. Giving an agent a read-only OAuth token means it cannot write to that system regardless of what it is instructed to do. Scoping network access, applying the principle of least privilege across every tool and API the agent can reach, and running it in an ephemeral sandbox that is destroyed after each run are all guarantees. Those protections still apply even if the model gets something wrong.

Guardrails are different. Labeling untrusted input so the model treats it with appropriate skepticism is a guardrail. Filtering agent output to detect and strip credentials before anything is posted anywhere is a guardrail. Prompt hardening to reduce susceptibility to injection attacks is a guardrail. Dawkins is clear that these measures can meaningfully reduce risk, but probabilistic controls are not the same as structural ones. "They are not guarantees," he says.

Most teams reach for guardrails first because they are easier to implement incrementally. Dawkins argues that the guarantees need to come first. Guardrails that sit atop a structurally permissive system are not a foundation. They are a patch.

What good actually looks like

The goal is not to make agents less capable; it is to constrain the blast radius of what can go wrong. An agent that runs in a structurally sound environment will still hit edge cases and produce unexpected output. The question is whether the system around it is designed to contain that before it becomes a problem.

An agent that runs in an isolated, ephemeral sandbox with scoped credentials, produces output that passes through a validation layer before reaching any human or external system, and requires explicit human approval for any action with persistent consequences is meaningfully safer than one running unsupervised on a laptop. The capabilities are the same. The difference is what the failure mode looks like.

"You need to make it observable, traceable, controllable, interruptible," Dawkins says. "All of those things to have some confidence."

That confidence does not come from trusting the model to behave correctly under all conditions. It comes from designing the system so that the consequences of misbehavior, whether from a hallucination, a prompt injection, or a misconfigured tool, are contained before they propagate. 

That's a different engineering problem from the one most teams are focused on today, but it's the one that will determine whether AI agents become production infrastructure or remain experimental tools.