The Problem With Asking a Fake Customer What a Real One Thinks
Synthetic data has an origin story, and it starts with a $6.4 billion write-down. Thoma Bravo bought Medallia for that price in 2021. Five years later, the private equity firm handed the company to its creditors. Voice of Customer software, outpaced by AI. But the bigger headline is that, apparently, synthetic data is about to replace the slow, expensive work of asking real customers what they actually think.
This is a bad idea. Not because synthetic data is worthless, but because the industry is skipping a step.
Before you decide to simulate your customer, ask yourself whether you’re actually listening to the real one.
Why synthetic data has everyone excited about fake customers
The pitch is easy to like. Ninety five percent of market researchers say they’re already using synthetic data or plan to within 12 months, filling data gaps and simulating audience segments faster than a traditional panel ever could. One CX leader has even floated “synthetic NPS” as its own product category. I’d love to see that methodology.
I get the appeal: No survey fatigue. No sample size limits. No waiting three weeks for a panel to provide feedback that nobody wanted to provide in the first place. You can run a thousand scenarios before lunch.
If speed is your only goal, synthetic data wins every time.
But speed was never the hard part of customer listening. The actual listening part was.
The synthetic data case study everyone’s citing actually proves the point
CVS Health is the example that keeps coming up, and it’s a genuinely impressive build. The company has rolled out roughly 100,000 agentic AI “twins” that simulate customer and patient reactions before anything goes live. For years, CX teams did this the old fashioned way: an empty chair at the conference table, reserved for “the customer,” because at least somebody had to be in the room speaking for the person paying the bills. VP Sri Narasimhan’s team replaced that chair with 100,000 digital stand-ins.
Those “twins” were built on 2.9 million consented responses from more than 400,000 participants, across more than 200 behavioral scenarios, modeled on real individuals rather than tidy averages. CVS Health did not skip the step of talking to real customers. They did an enormous amount of it, then used that foundation to extend the reach of what they’d already heard.
And even with all that real data underneath, the models run at 85 to 95 percent accuracy, which means CVS still checks every result against live research before trusting it. Narasimhan’s rule is the whole ballgame: when the model produces something unexpected, you don’t shrug it off. You go look closer, because it’s either a new insight or a crack in the foundation.
That’s not a company that trusts synthetic data (or even AI) alone. That’s a company that trusts real data enough to build something powerful on top of it, and then keeps checking its own work because the stakes are real.
Where synthetic data accuracy actually breaks down
Two independent studies tested whether AI could stand in for real human respondents, and the results should give anyone pause before they cut real listening out of the loop.
One had GPT-4o try to replicate consumer choice data. It handled broad patterns and general rank ordering fine. But it fell apart on the nuanced behavioral insight that actually drives business decisions, and it couldn’t produce market segments with any real meaning behind them. A model that can rank preferences but can’t tell you why is a model that can describe the weather but not predict the storm.
A separate study tested whether large language models could replace human survey respondents on political and lifestyle questions. Performance varied from model to model, and no single one reliably outperformed the others. The researchers’ conclusion wasn’t “pick the best model.” It was “combine several, because even the best one still needs to hedge against being wrong.”
Ipsos put it well: the quality of synthetic data depends entirely on the real human data that built it, and on the judgment of the people maintaining it. (Again, this can be said for AI in general too.) Take away the real data, or let it go stale, and the synthetic layer doesn’t know it’s wrong. It just keeps producing confident answers to yesterday’s questions.
What a model can’t do that a person can
Every synthetic data system, no matter how sophisticated, only knows how to respond within the scenarios it was built on. It can’t surprise you. It can’t hand you the complaint nobody thought to script, or the offhand comment that turns out to be the whole story. Real customers do that constantly, usually when you least expect it, and it’s often the most valuable thing they give you.
Synthetic data also doesn’t know when the world changed. A rate hike, a policy shift, or a bad headline may shift responses measurably but the model doesn’t know this on its own. Without a real, current pipeline of human feedback feeding it, a synthetic model keeps answering with total confidence about a customer who no longer exists.
This connects directly to being Responsive
In my WISER framework, the R stands for Responsive, and I’ve always defined that as more than fast reply times. Being Responsive means you actually hear what customers tell you and you act on it, publicly and specifically enough that they know they were heard.
That only works if what you’re responding to is real.
You cannot be Responsive to a customer who doesn’t exist. You can only be responsive to a pattern a model predicted that customer might say. Those are not the same thing, and customers can tell the difference, even when they can’t articulate why an experience suddenly feels hollow.
The real question isn’t synthetic data versus real data
Most companies reaching for synthetic data aren’t doing it because they’ve concluded it’s superior. They’re doing it because their real listening is too slow, too expensive, or producing response rates too low to be useful. Synthetic data is being adopted as a workaround for a broken front door, not as an upgrade to a healthy one.
Fix the front door first. Make it easy and fast for real customers to tell you the truth, and you’ll have something worth building on. Skip that step and build a simulation instead, and you’re optimizing your entire business around a customer who was never in the room.
CVS Health didn’t get to 100,000 twins by skipping real listening. They got there by doing more of it than almost anyone else, then multiplying it.
That’s the model worth copying: real listening first, synthetic data as the multiplier after. Not the shortcut.
—
Feedback To Action™
Stop drowning in data and satisfaction scores that don’t spark change. Feedback to Action turns customer feedback into a clear roadmap for improvement, helping you see exactly what to applaud, correct, and track to deliver remarkable experiences.
Most companies collect endless customer feedback, but few actually understand it. Surveys give you numbers – not nuance. They tell you what’s happening, but not why.
That’s where Feedback to Action comes in.
This short, voice-powered feedback tool asks just a few open-ended questions – and listens deeply. Using conversational AI, it uncovers what customers truly feel about your brand, in their own words. The result: clear, actionable insights you can implement immediately.
