Synthetic research:an overview, criticisms included

    Synthetic research is the use of language models in place of human respondents. It is a young category with real uses and real problems, and most of what is written about it is written by people selling it. This page is an attempt at the version a competitor could link to without wincing — including the criticisms, argued rather than dismissed.

    Where it came from

    Two things happened at once. Traditional research got harder: response rates to surveys fell for decades, panel quality became a standing complaint, and recruiting a genuinely representative group became slow and expensive enough that most product decisions were simply made without one.

    At the same time, language models became good enough at holding a described perspective that people started asking them to answer as somebody else. The step from 'answer as a 58-year-old rural retiree' to 'answer as a thousand of them, weighted' is short, and by the early 2020s several teams had taken it.

    The academic version arrived alongside the commercial one. Papers on 'silicon sampling' and algorithmic fidelity found that models could reproduce some population-level patterns from public opinion datasets, sometimes strikingly well, and also that the fit varied a great deal by group and by topic. Both halves of that finding matter, and only the first half tends to appear in marketing.

    The three shapes it takes

    These are genuinely different products that get called the same thing, and they answer different questions.

    Qualitative interview simulation

    A small number of described personas are interviewed in depth, in conversation. The output reads like a discovery interview transcript. It is useful early, when the question is still being formed and you want language, objections and framings rather than numbers. Its weakness is that a fluent transcript is extremely convincing regardless of whether it is right.

    Population read-out

    A weighted cross-section of a country is put to a single question and the result is reported per segment. The output is a distribution rather than a conversation. It is useful when the decision is about a broad audience and you need to know who splits. Its weakness is that it models a population rather than a market, and it cannot say anything about the people who would actually buy.

    Agent-driven product testing

    An agent uses the product — clicks through a flow, fills in a form, tries to complete a task — and reports where it stopped. Closer to automated usability testing than to research. It catches concrete breakage well, and it does not experience confusion the way a first-time human user does.

    The criticisms, stated properly

    These are not strawmen and they are not answered below. They are the reasons a serious researcher is sceptical of this category, and every one of them applies to Reynard as much as to anyone else.

    • A language model reflects its training data, not a population. Whatever was written down, in the languages and by the kinds of people who write things down, is what the model has absorbed. Weighting adjusts the arithmetic on top of that; it does not reach back and change what the model knows. Groups that are underrepresented in text are underrepresented in the model, and they are frequently the same groups that are hardest to reach in conventional research — so the method is weakest exactly where the need is greatest.
    • There is no published accuracy standard for the category. No agreed benchmark, no shared test set, no independent body checking claims. Vendors publish their own validations, choose their own questions and report their own results, and there is no reason to expect that the runs that went badly were published. In the absence of a standard, an accuracy percentage on a vendor's website is a marketing asset, not a measurement.
    • Fluency is mistaken for validity. Simulated responses read well. They have texture, hesitation, personality and quotable phrases, and none of that is evidence. A transcript that sounds like a real person is exactly as convincing when it is wrong, which makes this category unusually easy to be fooled by — including by your own product.
    • Models converge on the middle. Asked to represent a group, a model tends to produce that group's most typical, most legible, most publicly discussed opinion. Real populations contain far more internal disagreement and far more idiosyncrasy than a simulation reproduces, and the flattening is systematic rather than random.
    • Training data ages. A model's sense of what people think is fixed at a point in time, while the thing you are asking about is now. On anything that moved recently — a price level, a norm, a technology, a public mood — the simulation is answering from a slightly earlier world without signalling that it is doing so.
    • It cannot be audited the way a sample can. With fieldwork you can inspect the sampling frame, the response rate and the weighting scheme, and someone else can rerun it. A simulation's behaviour depends on a model whose weights you cannot inspect and whose version may change under you.
    • There is a structural incentive problem. The people best placed to evaluate this method are the people selling it. That is not an accusation of dishonesty; it is a reason to treat every published result in the category, including ours, as provisional until someone with nothing to gain has checked it.

    What it is nevertheless good for

    The criticisms above rule out one specific use: synthetic research as evidence. They do not rule out the use most people actually have for it, which is deciding what to do next when the alternative is deciding from the opinions of four colleagues.

    Compared to that alternative — not to a properly designed study, which is better than both — a weighted simulation is a real improvement. It is structured, it covers groups nobody in the room belongs to, it produces the objection you had not thought of, and it costs an amount that lets you run it on decisions that would never have justified research at all.

    The correct place for it is upstream: narrowing options, finding the misreading, deciding which two of six ideas are worth a real test. Downstream, where a number is going into a business case or a board pack, it has no standing and should not be used.

    Where Reynard sits in this

    Reynard is the second shape: a population read-out. A weighted cross-section of a market — the Netherlands on CBS data plus eight value groups, the United States on Census Bureau data plus eight value segments — is given a scenario, and the result comes back per segment with the reasoning behind it.

    That choice has consequences both ways. It means a run covers the whole population rather than the people you would have recruited, and that a large segment cannot be drowned out by a vocal small one. It also means the panel is a model of a country and not of your customers, and that every criticism above about training data and convergence applies in full.

    On the accuracy question specifically: Reynard publishes no figure, because none has been measured in a way worth standing behind. /accuracy sets out the validation intended to produce one, written down in advance so that the design cannot be adjusted after the result is known.

    Keep reading

    Run the free check first

    One check costs nothing and needs no account. You see the format of the output before you decide whether it is worth paying for a full run.

    Start the free check