AI wrote the bug. You need production data to reproduce it

A pull request lands with a function an AI assistant generated. It reads cleanly, passes the existing tests, and gets merged. Three weeks later it breaks, but only for accounts with a specific billing history. Nobody wrote a test for that condition because nobody was thinking about it when the code went in. Now someone has to debug logic they didn't reason through, against a failure they can't picture without the exact data that triggered it.

Debugging someone else's code has always worked this way. AI makes it the default.

Why AI-written code fails differently

Key takeaway: the code isn't necessarily worse, but the reviewer's intuition for where it breaks is thinner.

When developers write code themselves, they build a mental model of its edge cases as they go, because they work through those cases while writing. AI-generated code can be just as correct on its face, but the person reviewing it starts from the output rather than the reasoning behind it. They can check that it does what it's supposed to. They're less equipped to predict how it behaves under conditions nobody explicitly considered, because they didn't make those judgment calls.

The effects show up in delivery data too. DORA's 2024 Accelerate State of DevOps Report found that a 25% increase in a team's AI adoption was associated with an estimated 7.2% drop in delivery stability and a 1.5% drop in throughput, even as individual productivity rose. DORA points to larger batch sizes and weaker testing as the likely causes. Both make edge cases easier to miss. The same report found that 39% of respondents have little or no trust in AI-generated code.

The reproduction problem gets harder

Key takeaway: debugging logic you didn't reason through, whether a person or an AI wrote it, depends more on real conditions and less on intuition.

When the person fixing a bug didn't reason through the original code, guessing at the failure condition is harder. They can't fall back on "I bet it's that edge case." What's left is the data: the specific account and input state that triggered the failure. Without it, debugging AI-generated code means reading unfamiliar logic and hoping to stumble on the right test case.

The way code is being written makes this worse. GitClear's 2025 analysis of 211 million changed lines found that as AI assistants spread, 2024 became the first year that copy/pasted lines outnumbered moved (refactored) lines, and duplicated blocks of five or more lines rose roughly eightfold. More duplicated code means more places for an edge case to hide.

Isn't this just a code review problem?

Key takeaway: review can check whether code does what it's supposed to. It rarely catches conditions nobody thought to check for.

A more careful reviewer, or a stricter policy for AI-generated code, looks like the obvious fix. It helps, but it doesn't close the gap. To review for an edge case, you have to suspect it exists first, and that's the intuition a reviewer working from someone else's output tends to lack.

More scrutiny produces more careful guessing. It doesn't give the reviewer information they didn't already have. To close the gap, you need to see the condition that breaks the code.

What production-state branching gives you here

Key takeaway: cloning production puts the failing condition in front of the person debugging, so they don't have to reconstruct it from a hunch.

A branch that clones production, with the same services, configuration, and data, cuts out most of the guessing. The account with the billing history that triggered the bug is already in the preview environment. Nobody has to rebuild it by hand or wait for the failure to happen again.

Copying production data raises a fair question about personal data. A sanitization script in your deploy hook can anonymize personal data on preview environments while keeping the billing history the bug depends on. The sanitize databases docs have working examples.

From there, it's the same workflow you'd use for any hard-to-reproduce bug. Branch off production, open a tunnel to the cloned services, and reproduce the failure against the data that triggered it. Our walkthrough, From bug report to verified fix in a single workflow, covers each step. What's different here is how much the debugger relies on that data, because they're working from someone else's output rather than their own model of the logic.

Want the failing conditions in front of you when you debug? Start a 15-day free trial and clone your first environment.

Frequently asked questions (FAQ)

Does this mean AI-generated code is buggier than code written by hand? Not necessarily. What differs is how much intuition the person maintaining it has for its edge cases, and that makes testing against real conditions more important.

Why does production data matter more for AI-written code specifically? Because the usual fallback, the author's memory of the edge cases they considered, is thinner or missing. Real data stands in for intuition the reviewer doesn't have.

Does this apply to all AI-assisted code, or only fully AI-generated features? It applies anywhere the person debugging didn't reason through the original logic, whether that's a fully AI-generated function or human-written code someone else now maintains. AI-assisted development just makes that situation far more common.

How do I trace where the bug came from? git blame shows you the diff, but not how that diff came about. If the change came through Upsun Dispatch, each run keeps a record of the rest. The run detail in the Dispatch console shows the prompts, tool calls, and findings from that run. For Jira or Linear triggered runs with plan approval turned on, it also shows who approved the plan and any revisions. Pair that with a production-state branch and you have both halves: why the code was written that way, and the data that breaks it.