• Docs
  • Talk to an expert
Blog
Blog
BlogProductCase studiesNewsInsights
Blog

Microsoft was taken down by its own fix. Expect more of that.

securityAIdeploymentAgentic SDLC
20 November 2026
Fred Plais
Fred Plais
CEO
Share

Nov 20, 2026 · Fred Plais

Over the last few weeks, I've had the strange feeling that everything is breaking at once.

On November 12, Microsoft 365 went down for almost a day. Outlook, Teams, Exchange, Copilot. So many people spent that day refreshing status pages. A few days later it was Workday. Then the FAA.

What struck me most was Microsoft's own explanation. No attacker, no AI. A bug introduced by a fix they shipped back in July, dormant for months, then amplified by the very systems that were supposed to recover from failures.

If that can happen at Microsoft, it can happen anywhere. And I think we're about to put a lot more pressure on our production systems than ever before.

Not every outage is an attack

Let's be careful here. These incidents don't share one cause. Microsoft was a latent bug. Workday was reportedly a disaster-recovery test that reached production. The FAA had a communications infrastructure failure. And in parallel, real attacks on companies and their suppliers keep piling up.

But there's one category we don't talk about enough: outages caused by the fixes themselves. Outages generated by the good guys protecting the system. How ironic?

A critical vulnerability is disclosed, so you patch, fast. You upgrade a dependency, restart services, change a config. Sometimes the patch brings a regression. Sometimes the deployment fails. Sometimes something nobody thought about breaks. You made the system safer, and for a few hours it's down.

That's exactly what Microsoft's July fix turned into. And it's the tension I want to talk about: the faster we have to change production, the more those changes need to be safe.

AI makes finding vulnerabilities cheap

For years, finding a serious vulnerability was hard work. You needed rare people, a lot of time, and some luck. Scanners helped, but they only found what their rules told them to look for.

LLMs change that. They can read a huge codebase, follow dependencies, trace execution paths and spot weird interactions between components. All day, every day, for a fraction of what a human team costs.

It's already happening, and very close to home. Fabien Potencier (Upsun’s Chief Product & Technology Officer), who also created Symfony, wrote about it recently: "A few years ago, we used to have one security report a Quarter, now it’s at least one per day". On Symfony alone, Fabien and the community fixed as many security issues this year as in the previous ten years combined. And Symfony is not an exception: Google's Big Sleep agent, for example, found an exploitable bug in SQLite, which runs pretty much everywhere.

These tools still make mistakes. Lots of false positives, things they miss, and you still need a human to check. But they get better every few months.

And of course, attackers have the same tools.

So my concern is simple: we may soon find vulnerabilities much faster than we can fix them.

Finding the bug is the easy part

When a critical vulnerability goes public, the clock starts ticking. But finding it is only step one.

First you need to know if you're even affected. Which apps use that component, in which version, and can the vulnerable code actually be reached? Then you need a fix. Sometimes it's a one-line dependency bump, sometimes it changes across several systems. Then you test it: does it close the hole, and does the app still work with its database, its services, its integrations? And then you deploy it. Not on one laptop. Potentially on hundreds or thousands of production environments, on different versions, in different regions.

That's where things get stuck in real life. The vulnerability is found but nobody owns it. The patch is written but waits a week for review. It passes the tests and breaks in production. Three environments get updated and forty don't.

If you can find a vulnerability in ten minutes but need three weeks to ship a safe fix, you're exposed for three weeks. And if you rush it and take production down, you've just swapped one problem for another.

Why we're building an AI Software Factory

Coding assistants have made writing code a lot faster. Great. But writing code was never the real bottleneck. Shipping it safely is. Everything that happens after the code is written is where we focus at Upsun.

Upsun Dispatch runs the engineering side. When a vulnerability lands, it helps find the affected repos, opens the remediation tasks, has agents propose the patches and runs the reviews, sending the risky changes to the right people.

Upsun Cloud runs the result. Every fix gets its own preview environment, a real copy of production with its data services, so you can check it works before it ships. Once approved, it rolls out across your environments in a way you can watch, and roll back if needed.

Humans stay in the loop for anything critical. But the whole chain gets faster, more consistent, and it scales to a full fleet of apps instead of depending on one heroic engineer at 2am.

We know this problem first-hand. A few weeks ago, PHP2Shell hit: two critical flaws in PHP itself. PHP gave advance notice on October 20, and by that afternoon we had WAF rules blocking exploit attempts in every Upsun Cloud region, about 22 hours before the official fix even existed. By the next day we were blocking around 1,700 malicious requests per hour.

When PHP shipped the patch on October 21, we rolled it out ourselves. Tested in a separate region first, replacing only PHP, without touching customer code or configuration. Every current PHP version was patched across all regions about four and a half hours after the release, and the older versions were done overnight. As the team wrote at the time: "Customers didn't need to do anything." (full story here)

That's what I mean by fast, safe remediation. Now imagine doing that every week, for every framework you run, on your own.

The one number I'd want to know

We've spent years measuring engineering teams on deployment frequency, lead time, how fast features ship. That's all still valid. But there's one number I think every CTO should know now: how long does it take you to go from a vulnerability being disclosed to a verified fix running everywhere in production?

Not how fast one engineer writes a patch. How fast the whole organization moves.

In Europe, this won't stay optional for long. The Cyber Resilience Act will require software vendors to report actively exploited vulnerabilities and to keep shipping security updates for as long as they support a product. Fixing fast, and safely, is becoming a legal requirement.

Better scanners won't get you there. You need orchestration, solid testing, environments you can reproduce, automated deployment and clear ownership. Software delivery is a security capability now.

A bumpy road to a better place

I think the next couple of years will be bumpy. More emergency patches, more late-night upgrades, more incidents. But the same AI that finds the bugs can help us fix them, and I believe we can come out of this with software that's a lot safer than what we have today.

It won't happen just because the models get better, though. It will happen if we build the systems that turn what they find into fixes that actually ship.

So here's my question for you: if a critical vulnerability hit your main framework tomorrow morning, how long until every production environment is patched? And how confident are you that nothing breaks along the way?

If you don't love the answer, let's talk.

Fred Plais
Cofounder & CEO Upsun

Stay updated

Subscribe to our monthly newsletter for the latest updates and news.

Deploy possibility.
Try Upsun for free.

Build with DispatchDeploy on Cloud