Breaking things fast: A new Approach to QA and testing

This post is based on Greg Qualls, Director of Product Marketing, presentation, "Accelerating QA and Testing," at SymfonyCon 2024. We utilized AI tools for transcription and to enhance the structure and clarity of the content.

Before we dive in, I have over 18 years of experience in sales. If, at times, I sound like I'm trying to sell you something, please forgive me. I promise I'm not. I just want to share ideas that might help you think differently about testing and QA, particularly if you're working with or considering Upsun.

Also, full disclosure: I'm really good at two things: breaking stuff and asking what some might call "dumb questions." Today, we're going to explore why both of these skills are actually valuable in software development.

The cost of waiting

Let's start with a sobering statistic: it's 30 times more expensive to fix a bug after it reaches production than it is to catch it earlier in the development process.

This isn't just a made-up number. The reason the cost multiplies so dramatically is because production defects create ripple effects far beyond the code itself. They impact your customers, damage your brand reputation, erode user trust, and force your team to spend valuable time firefighting instead of building new features.

This is why testing matters. But here's where things get interesting.

The philosophy of breaking things fast

In 2012, Mark Zuckerberg shared a core value in his "Hacker Way" philosophy: "Move fast and break things. Unless you are breaking stuff, you're not moving fast enough."

I'd like to adapt this for our context: Move fast and break things. Unless you're breaking stuff, you're not really testing or moving fast.

The goal of testing should be to break things as quickly as possible, to determine if something works, so we can either fix it or proceed with building the next feature. This is what we really love doing: creating features, developing functionality, writing code that matters.

Two fundamental questions

When you think about it, all testing really boils down to two fundamental questions:

1. Is it broken?

This covers your unit testing, functional testing, system testing, and exploratory testing. You're asking: Does this work the way it should?

2. When will it break?

This encompasses load testing, penetration testing, scalability testing, and spike testing. You're asking: what are the limits of this system?

In both cases, you're trying to answer these questions as fast as possible.

The problem with traditional workflows

Most development workflows follow a familiar pattern: local development → dev environment → staging → production. You test at each stage, iterate, fix issues, and eventually see those green checkmarks that mean you're ready to deploy.

But this process can be painfully slow.

My testing journey

Let me share a personal story. Back in 2002-2003, I was the "Web Master" at Eastern New Mexico University. Everything was pure HTML—no JavaScript, no CSS (CSS came later, and I thought it was revolutionary).

My testing process was beautifully simple: write the code, FTP it to the server, visit the page on enmu.edu, and see if it worked. If it didn't, I'd fix it and upload again. We'd now call this "testing in production," but back then, there was no distinction—everything was production. There were no dev environments, no staging environments. Just my computer and the live server.

The hidden benefits of testing in production

Now, before you panic, hear me out. What if you could test in production? Think about the benefits:

• Immediate feedback – You know instantly if something works
• Real-world data – You're using actual servers, databases, and services
• No data sanitization needed – Everything is already where it should be
• Quick discovery – Environment-specific issues reveal themselves immediately
• Proper performance testing – You're testing against real conditions

The one big problem

Of course, there's one massive issue with testing in production: users.

Those pesky users who complain when the site goes down. Then your boss finds out. Then you're in a performance review explaining what happened. And before you know it, you're on LinkedIn posting "Open to work."

Therefore, we cannot simply test in production. Or can we?

The solution: replicate production

The answer is to replicate production so accurately that testing there is essentially the same as testing in production, minus the users.

However, replicating production properly is an incredibly complex process. You need to:

• Clone not just code, but data layers and services
• Handle networking pieces like DNS and SSL certificates
• Use infrastructure as code and Git-based workflows
• Manage access control and permissions
• Ensure everything syncs perfectly

Many companies struggle with this. I once spoke with a large university that needed three weeks to spin up a development environment. Three weeks! By the time they had everything configured and the data synced, production had already undergone significant changes.

Full-Stack preview environments

This is where Upsun's approach becomes interesting. The platform offers full-stack cloned environments—complete replicas of production that you can spin up instantly.

When you create a branch in Upsun, you're not just branching the code. Behind the scenes, the platform replicates:

• The entire codebase
• All databases and data
• Services and configurations
• Network settings
• Infrastructure setup

Everything except the users. 

Breaking things fast in practice

Earlier at SymfonyCon, a live demo featured two developers attempting to decouple a monolithic Symfony application into a Next.js frontend with a Symfony backend. It was a 30-minute demonstration on fire; the kind of live demo where things don't quite work at first.

They deployed. It didn't work. They changed something and deployed again. Still didn't work. They iterated rapidly, testing and breaking things as fast as possible. Eventually, they found the issue: in the configuration file, they had written "npm" instead of "npx"—one character difference.

Here's the remarkable part: they transitioned from a monolithic application to a fully decoupled architecture in 30 minutes, thanks to their ability to iterate rapidly. They knew that if it worked in their development environment, it would work in production, because the environments were identical.

The collaboration challenge

Of course, development is rarely a solo endeavor. You have coworkers, those other developers who, let's be honest, are never quite as good as you are.

In traditional workflows, when multiple developers push to the same development environment, merge conflicts are inevitable. Someone has changed something, and now your code no longer works. You have that conversation: "Greg, what are you doing? I'm working on this! You need to install this service first!"

Multiple isolated environments

The better solution is to give every developer their own production-like environment. You work on your code, they work on theirs, completely isolated.

When you're ready, you push to production. Then your colleague pulls down your changes—not just the code, but all the data, database updates, service configurations, and caching. Everything syncs automatically. They test their code against this new state, and the cycle continues.

Everyone works separately but collaboratively, moving seamlessly because you're all developing in production-like environments.

The real-world benefits

When you can break things faster—especially with platforms like Upsun—you gain:

• Faster testing cycles – You can attempt ambitious changes, like decoupling an entire application, in under an hour.
• Faster development cycles – When leadership asks for something "as soon as possible," you can iterate rapidly, trying new languages, frameworks, services, or machine learning libraries to see what works.
• Higher code quality – You test and iterate until you're confident, knowing that if it works here, it will work in production.
• Reduced risk – You catch issues before users do, avoiding those uncomfortable performance reviews about site downtime and lost revenue.

And here's something for the business side: proper testing and QA environments have been proven to show a 40% increase in customer satisfaction scores. When you trust your testing, you ship better code, which leads to happier customers.

Three practical takeaways

Whether or not you use Upsun, here are three principles for breaking things faster:

1. Change your mindset

Too many developers feel they need to get everything perfect on the first try. Instead, embrace failure as learning. Try it. See if it works. Try again. You'll learn more from breaking things than from getting it right immediately.

From that live demo earlier, I'll never forget: if you build your Next.js app with npx, make sure you run the command also uses npx, not npm. I learned that by watching someone else fail—and that lesson stuck.

2. Leverage testing environments

Please don't walk out of here thinking, "Greg said we should just test in production!" If you tell your leadership that, they'll ask, "Who's Greg?" and you'll be in trouble.

You need testing environments. But they must be as close to production as possible. Think through all the variables: infrastructure, data, services, networking, and access control. Cover as many as you can.

3. Automate everything

It's not just about breaking things, it's about breaking things fast. Automate your environment creation. Set up CI/CD pipelines. Make it so testing environments spin up instantly and deployments happen automatically when tests pass.

The faster you can break and fix, the sooner you can get back to building new features and moving your product forward.

Final thoughts

Testing doesn't have to be a chore. When you have the right tools and mindset, it becomes part of the creative process, a way to explore possibilities, push boundaries, and learn quickly.

The goal isn't to avoid breaking things. The goal is to break them as fast as possible in safe environments, learn from those breaks, and ship code you can trust.

So go ahead: break things. Just break them fast, break them safely, and learn something from each break.

Questions about Upsun or implementing these practices? Visit the Upsun website or check out our free trial to see full-stack preview environments in action.