You Cannot Guess Your Way to Success
Microsoft runs tens of thousands of controlled experiments every year. Here’s what they’ve found: only about a third produce positive results. Another third show no measurable impact. The final third actually make things worse. And when they asked experts to predict which experiments would succeed, those experts were wrong 96.1% of the time.
Let that sink in. The people closest to the product, with the most context and the most experience, are wrong about what will work almost every single time.
Jeff Bezos has said that Amazon’s success “is a function of how many experiments we do per year, per month, per week, per day.” Not how smart the experiments are. How many. Because if even experts can’t predict what works, the only winning strategy is to test more.
This isn’t just big-company wisdom. Researchers at Duke University studied over 35,000 startups and found that firms adopting A/B testing saw 30 to 100% performance improvement after one year. Ten percent more page views, five percent higher likelihood of VC funding, nine to eighteen percent more product launches. And the most interesting finding: A/B testing freed founders to think about radical changes, not just incremental ones. When testing is cheap, you stop playing it safe.
The research is overwhelming and it all points in the same direction: the companies that test the most win. Not because they’re smarter. Because they learn faster.
The Bottleneck Was Never Measurement
Most companies have figured out how to measure experiments. Analytics tools, feature flags, A/B testing platforms. The infrastructure for learning from experiments is mature.
The bottleneck has always been building the variants.
Think about how experimentation typically works in a product organization. You have an idea. You write a ticket. It goes into the backlog. It gets prioritized against everything else. If it survives sprint planning, an engineer builds it. Then you test it. The cycle from hypothesis to result might be weeks or months.
And that cycle time creates a brutal filter. Only the ideas that seem most likely to succeed survive prioritization. The weird ideas, the long shots, the “this probably won’t work but what if it does” bets get killed before they’re ever tested. Not because they’re bad ideas. Because engineering time is too expensive to spend on low-probability experiments.
Most companies release maybe four times a year. Each release contains a handful of bets. Those bets were chosen not because they’re the best ideas, but because they’re the safest ones. The ones that could be justified in a sprint planning meeting. The ones where someone could build a business case.
And we just established that experts are wrong 96% of the time about what will work. So we’re carefully selecting our four annual bets using judgment that is almost never correct. That’s the current state of product development at most companies.
AI Changes the Math
Here’s what happens when AI collapses the cost of building experiment variants to near zero.
You can swing at pitches you would have let pass. The ideas that were too weird, too niche, too speculative to justify engineering time? You can now build and test them in a weekend. The cost of being wrong dropped so dramatically that the rational strategy flipped. It used to be rational to be selective. Now it’s rational to be prolific.
The Koning study at Duke found that A/B testing freed founders to think about radical changes. AI amplifies that by a hundred times. It’s not just that you can test faster. It’s that the entire category of “ideas worth testing” explodes. Things that would never have survived a prioritization meeting can now be built and shipped before the next standup.
The gap between companies that release four times a year and companies that run four experiments a day is about to become unbridgeable. Not because the fast companies are smarter. Because they’re learning at a rate that makes traditional product development look like guessing. Which, statistically, it is.
The Delighter Economy
This is where it gets fun. And I mean genuinely fun, the kind of fun that made you want to work in product in the first place.
When the cost of building drops to near zero, you can start building things that don’t make traditional financial sense. The long-tail, niche, delightful features that perfectly fit a specific persona’s workflow. Things that would never survive an ROI analysis but make customers fall in love with your product.
I work in the regulatory compliance space. Not exactly known for whimsy. But here’s what becomes possible when you can build fast:
A 16-bit RPG that trains new professionals on regulatory structures. Instead of handing someone a manual and hoping they read it, you drop them into a retro game where they learn the common gotchas of compliance auditing by playing through scenarios. Is it strictly necessary? No. Does it make financial sense on a spreadsheet? Absolutely not. Would new hires love it? Would it make your onboarding unforgettable? Would it make your company the one people talk about at conferences? Yes, yes, and yes.
Or consider how auditors actually work in the field. They walk around facilities with a clipboard and paper, scribbling notes. What if you could scan those handwritten notes and automatically link them back to the specific regulatory requirements for the country they’re auditing in? That’s a feature that lives at the intersection of “deeply understanding how your users actually work” and “AI makes it buildable now.” It’s not a feature a product committee would greenlight. It’s a feature a PM who has watched auditors work would dream up and ship in a weekend.
These aren’t enterprise features. They’re delighters. And delighters are what separate products people tolerate from products people love. Think about the difference between Snapchat and Facebook. Facebook had more features, more users, more money. Snapchat had delight. It had personality. It had the feeling that someone who actually used the product built it. That’s what these long-tail features create, and AI makes them economically viable for the first time.
Both Things Are True
Here’s where I want to be direct about the stakes, because both things are true simultaneously.
This is the most fun I’ve had in product in years. Building creative things that customers love, shipping ideas that would have died in a backlog, seeing the look on someone’s face when they encounter something unexpected and delightful in a regulatory compliance tool of all things. The ability to experiment freely, to take creative swings, to build the thing you’ve always wanted to build but could never justify. That’s genuinely exciting.
And if you’re not doing this, you’re cooked.
That’s not hyperbole. The companies that embrace high-volume experimentation with AI are going to compound learning at a rate their competitors cannot match. Every experiment teaches you something about your customers. Every delighter builds loyalty that’s hard to quantify but impossible to replicate. Every weird idea you test and ship successfully widens the gap between you and the company that’s still debating whether to put it on the roadmap.
The Microsoft data proves it. Most ideas fail. The winners aren’t the ones with the best ideas. They’re the ones who test the most ideas. AI just made testing almost free. The companies that internalize this will be wildly successful. The companies that don’t will be outlearned.
What This Means for Product People
If you’re a PM, this changes your job in the best possible way.
You stop being the person who decides which four bets to make this quarter. You become the person who generates hypotheses fast enough to keep up with the pace of testing. Your value isn’t in picking winners. It’s in understanding your customers deeply enough to generate hypotheses worth testing. And then testing them. And then testing more.
The PM who knows how auditors actually work, who has watched them walk around with clipboards, who understands the gap between their daily reality and the tools they’ve been given, that PM generates better hypotheses than any amount of market research. Customer proximity isn’t just a nice-to-have anymore. It’s the engine that feeds the experimentation machine.
And the experiments themselves become the product strategy. You don’t need a quarterly roadmap when you’re shipping and learning daily. The roadmap emerges from what the experiments tell you. Build, ship, measure, learn, repeat. The strategy writes itself if you’re paying attention to the data.
The Next Question
The experimentation advantage raises an uncomfortable follow-up: how far do we take this?
If AI can build experiment variants, can it also generate the hypotheses? Can it run the experiments autonomously? Can it optimize without human involvement? At what point does the PM stop directing the experimentation and start just watching it happen?
That’s a question I’m still working through, and it’s worth its own article. But the short answer is: the further we push automation, the more important the human layer becomes for the things automation can’t do. Taste. Judgment. Knowing that an auditor uses a clipboard. Understanding why a 16-bit RPG would make a compliance professional smile.
AI can test a thousand variants. It can’t tell you which problems are worth solving. That’s still your job. And it’s a better job than writing tickets.