Synthetic Users Research for India: Test Before You Trust

No. Synthetic users research works for drafting: discussion guides, survey wording and early hypotheses. It does not work for decisions about demand, price, concepts or regional audiences, because a synthetic user produces a predicted answer, not an observed one. The table below, which is our recommendation, sorts common tasks.

For context, in the User Interviews survey of 150 researchers (May 2026), 46% use synthetic users for survey or screener design, 34% for usability testing and 32% for early-stage research, while 21% see no appropriate use.

Because a language model is built to produce a likely, agreeable answer, not a true one. Three faults follow: it flatters your idea, it treats everything as important, and it gives you less variety than real people do.

Sycophancy is the tendency to give the answer the asker seems to want. In a side-by-side comparison of a real and a synthetic learner, the real learner said they had not finished every course and had completed three. The synthetic user replied, "Yes, I completed all the courses I mentioned." Asked what matters in online learning, synthetic users listed seven engagement factors with no ranking, because, in the article's words, "Synthetic users seem to care about everything." A product decision needs the ranking.

The spread problem shows in a published comparison. In a Journal of Marketing study (Arora, Chakraborty and Nishimura, 2025) that replicated a refrigerated dog-food concept test, 605 human respondents were matched with 605 synthetic users. As reported by Nielsen Norman Group, synthetic users captured general trends but not the size of the effects or the variability. Purchase likelihood averaged 1.66 for humans against 1.58 for synthetic users on a 5-point scale, and the synthetic answers had a consistently lower standard deviation. That is one study in one category, not a rate you can apply to India.

One vendor publishes 85–92% parity, its own score for how closely synthetic and real answers align, measured across thematic overlap, depth and qualitative alignment (as of October 2026). Our inference: theme overlap is the easiest part to match. A tool can list the right themes and still get the order, the strength and the dissent wrong, and those are what you decide on.

Thirty answers from one model are also not thirty independent observations. That is our inference, but it is why repeating a synthetic run does not add the confidence that thirty real interviews do.

Substantially worse. On everyday questions that depend on which Indian region you come from, eight leading language models scored only 13.4% to 20.9% accuracy in INDICA, a benchmark of regional cultural commonsense questions. They also leaned towards North and Central Indian answers.

INDICA collected human answers from five Indian regions across 515 questions in eight areas of daily life. Only 39.4% of questions drew the same answer in all five regions. India's "typical" answer barely exists, and a persona labelled "Indian, 30, urban" gets pulled towards one region's default.

INDICA tests cultural knowledge, not purchasing behaviour, so it shows the risk rather than the size of your error. A second study shows how the risk behaves. Using 10,000 responses from the World Values Survey (a long-running international attitudes survey) across 12 countries, Ziaei (2026) measured how much of the human spread, meaning how widely answers vary between people, models kept. In this September 2026 preprint (not yet peer reviewed), the most accurate model kept about 0.50 of human spread overall, 0.11 for Nigeria and 0.70 to 0.87 for WEIRD countries (Western, educated, industrialised, rich and democratic). Training the model to follow instructions cut spread from 1.22 to 0.59.

Market research in India is about segments: tier-2 cities against metros, the Hindi belt (north and central Hindi-speaking states) against the South, first-time against repeat buyers. A model that flattens spread erases the differences you are paying to find. Our inference is that Hinglish (mixed Hindi and English), regional languages and low-income households are where you should expect the weakest results, so test them first.

Run it on a past India study whose real results you hold, then score it on five checks. Our estimate is about 90 minutes in total: 30 minutes to set up, 30 to run the tool and 30 to score the five checks. The test tells you what this tool does on your audience, which no vendor claim can.

Set-up. Pick a completed study the tool has never seen. Write a one-page answer sheet: the top five themes, any percentages or ratings, and three findings that surprised you. Give the tool the same audience brief and the same questions, and ask for as many synthetic users as you had real participants.

The thresholds below are our recommendation, not a published standard.

Does it get the rank order right?

Pass if the real study's top three themes all appear in the tool's top five. Fail if any is missing. A tool that misses the main theme cannot be trusted to find the next one.

Does it get the size right?

Pass if, on at least four of five questions, the tool's figure is within 10 percentage points (or one point on a five-point scale) of the real one. Fail otherwise.

Does it show real disagreement?

Pass if, on each key question, at least one synthetic user disagrees with the majority and gives a reason you recognise from the real data. Fail if the answers are near-identical.

Does it find what surprised you?

Pass if it raises at least one of your three recorded surprises without being prompted. Fail if it raises none. Surprises are what discovery research exists to find, so a tool that misses all three is draft-only for discovery work.

Does it tell your segments apart?

Run the same questions twice: once as an English-speaking metro persona and once as your real segment, such as a Hindi-speaking tier-2 buyer. Take 5 answers from each (10 in all), remove the labels and shuffle them. A colleague who knows the audience sorts them into the two segments. Pass at 8 or more correct; fail at 7 or fewer.

Reading the result. Five passes on one study earns the tool a wider drafting role. It does not earn the right to decide, because one study is one sample. Any fail means draft-only for that audience.

Only when a wrong answer is cheap. The break-even rule is our model, not a published figure: buy the real study when your chance of the synthetic answer being wrong exceeds the saving divided by the cost of a wrong decision.

Break-even chance of error = (cost of real study − cost of synthetic tool) ÷ cost of a wrong decision

Assumptions: the real study gives the right answer, and a wrong synthetic answer leads to a wrong decision costing the full amount.

Take assumed figures for illustration, which you should replace with your own: a real study at ₹2,00,000, a tool at ₹20,000 and a wrong launch decision at ₹25,00,000. Then (2,00,000 − 20,000) ÷ 25,00,000 = 1,80,000 ÷ 25,00,000 = 7.2%. If you believe there is more than a 7.2% chance the synthetic answer is wrong, the real study is cheaper in expectation.

We have no India-specific error rate, so test your own tool with the five checks above before assuming one. For a discussion guide, a wrong answer costs one revised question, and synthetic users win easily.

Label it at the source and write the rule down. Nielsen Norman Group advises against presenting synthetic findings as real-user research, calling it "an unethical way to share what you've learned". In the User Interviews survey, 79% fear stakeholders will over-rely on AI findings, and 62.7% say their organisation has no guidance on synthetic users.

Put this line on every slide or document that draws on synthetic users: "Hypotheses generated with synthetic users; not evidence from real participants." Keep synthetic percentages out of decks, and attach no recommendation to a claim until real participants have confirmed it.

Turn the three claims you most want to be true into questions a real person can disprove, then test them in one-on-one conversations. We recommend in-depth interviews (IDIs): one-on-one sessions of 30 to 120 minutes, online or in person. Our guide to the in-depth interview method covers the brief and the plan, and sample size in qualitative research explains how to size each segment.

Here is how we run it. You send a brief with your objective, audience, geography, sample size and timeline. We check feasibility and send a tailored proposal and quote, because we price each project on its method, audience and recruitment difficulty rather than from a fixed rate card. We then recruit by city, run identity checks before invitation, and schedule sessions in the participant's language, including languages other than English. We can supply moderators. Recordings, transcripts, data exports and reports are agreed before the study begins.

We also offer a free pilot, so you can compare a small set of real conversations with what your tool told you before committing budget. Email your brief to support@qualfacto.io, or message us on WhatsApp for quick questions. We aim to reply within one business day (Monday to Friday, 10:00 to 18:00 Indian Standard Time, IST).