Think Aloud Protocol: Time Budget, Script and Language Check

No measurable change in performance has been found. A 2011 meta-analysis (a statistical pooling of earlier studies) of 94 studies and nearly 3,500 participants put the effect of thinking aloud on performance at r = −0.03. Here r is a correlation, and a value near 0 means no relationship.

The same analysis found two things that matter for your script. Asking people to explain or describe their reasoning did change performance. And every kind of verbal reporting made tasks take longer.

A think aloud protocol is the set of rules around the talking: what participants are asked to say, how the moderator prompts, and how the recording is analysed. It comes from the protocol analysis of K. Ericsson and H. Simon (Ericsson and Simon, 1980, Psychological Review 87(3)), and Clayton Lewis brought it into usability work at IBM (Lewis, 1982, IBM Research Report RC-9265).

Our recommendation follows from this. During the task, ask only "What are you thinking?" and move "Why did you do that?" to after the task. The meta-analysis measured performance on set tasks. A usability session adds a moderator and a live product, so treat the result as strong reassurance for the method, not a guarantee for every session.

With a 60-minute session, 10 minutes of set-up and debrief, and tasks that take 4 minutes in silence, about 10 tasks fit with concurrent think aloud. We would plan 8. The same hour fits 12 silent tasks, or 6 if participants explain afterwards while watching a replay.

The time penalty comes from a randomised comparison (participants were assigned at random to think aloud or work silently) by MeasuringU: 423 participants across 10 remote, unmoderated (no facilitator present) studies run between January and September 2022 on tasks for travel, real-estate and restaurant websites.

• Participants who thought aloud took about 17% longer across all attempts, and about 20% longer on successful completions.
• Nine of the ten studies showed longer times; three were individually significant at p < .10, and across all ten the pattern was significant (sign test, p = .03). Here p is the chance of seeing a result at least this strong if thinking aloud had no effect.
• Completion times varied about 27% more (standard deviation, how widely times spread around the average) in the think aloud groups.

MeasuringU tested unmoderated sessions only. For moderated sessions we use the higher 20% as a cautious planning assumption, because we expect prompts to add time.

Our model (the inputs are assumptions you replace with your own):

Tasks that fit = (session minutes − overhead minutes) ÷ (silent task minutes × time factor)

Time factor: 1.0 for silent testing, 1.20 for concurrent think aloud, and 2.0 for retrospective think aloud from video. The 2.0 follows from replaying a recording that lasts as long as the task, and MeasuringU also describes it as doubling the time per task.

Here 50 minutes is the 60-minute session less 10 minutes for consent, the script, a practice task and a debrief. Using the 17% figure instead gives 4.68 minutes per task and still only 10 tasks.

Why plan 8 and not 10? Because time varies more. Eight tasks at 4.8 minutes take 38.4 minutes. MeasuringU found completion times spread about 27% more. As a stress test, we assume a slow participant takes 27% longer than average: 38.4 × 1.27 = 48.8 minutes, still inside the 50. This is our assumption, not a measured overrun. Ten tasks would not survive that.

Think aloud task times overstate how long people normally take, so do not report them as speed benchmarks.

Choose concurrent think aloud unless talking would disturb what you need to measure. Retrospective versions suit tasks that need silent concentration or where timing matters, and they cost more time.

If you want the numbers for retrospective from memory, pilot it. We would not commit a full study to a time estimate we have not tested.

For how many people each method needs, see our guide to user experience (UX) research methods and sample sizes.

Read a short brief, run a practice task on something unrelated, and prompt with one neutral question after a pause. The script below takes about 45 seconds to read out.

Brief to read out:

"Thank you for joining. Today we are testing the app, not you, so nothing you do can be wrong. As you work through each task, please say out loud what you are trying to do, what you expect to happen, and anything that surprises or confuses you. Say it even when it feels obvious. If you go quiet, I will ask what you are thinking. That does not mean you are doing anything wrong. I will not be able to help with the tasks, but I will answer your questions at the end. Before we start, let's try one practice task."

Practice task: "How many windows are there in your home?" It is unrelated to your product, so the participant learns to narrate without the pressure of a real task.

Prompt bank:

• After a pause of a few seconds: "What are you thinking now?"
• Before a tap or choice: "What do you expect to happen next?"
• After a visible hesitation: "Tell me more about that."
• Not during the task: "Why did you tap that?" (asking for reasons changes behaviour, as the 2011 meta-analysis found) and "Should that button be bigger?" (it invites a design opinion).

Keep the useful talk and let the rest go. Useful comments state a task intention ("I want to…"), interpret the screen ("I think this does…"), mark an expectation gap ("That's not what I expected") or note performance ("That took longer than I thought"). Style preferences, guesses about other users and design instructions are less useful (UXmatters). When a participant offers a design idea, do not discuss it or praise it. Note it, carry on, and ask about it at the end.

If someone cannot do it: show them how once, remind them gently, and if it still interferes with the task, let them work silently and ask your questions at set points or afterwards (UXmatters).

Run the session in the language the participant uses when they use apps, and let them mix languages mid-sentence. The moderator must follow every sentence without help.

The evidence on language is thin. A 2022 preprint comparing Chinese non-native English speakers with native English speakers found similar amounts and types of verbalisation, and similar problems found. That is encouraging for participants who are fluent in English. It covers one language group and sessions held in English, so it does not show how sessions in Hindi, Tamil, Bengali or any other Indian language will go.

Our recommendations:

• Ask the screener which language the participant uses for apps day to day, and use that language for the session.
• Use the practice task as the language check. If the moderator needs a translator at that point, replace the participant.
• Translate the guide and pilot-test it before fieldwork, and avoid live translation. Each extra language needs its own moderator and guide, so keep a study to one or two languages. Our page on multilingual research explains the cost of each added language.
• Run one pilot session first, and treat it as a test of the language, the script and the time budget together.

We recruit participants for usability and app testing, by city and by household and demographic profile, and check each participant's identity before inviting them. We run sessions in languages other than English. We can supply the moderator.

Studies are run by us or by research partners, depending on the project. Recordings and transcripts come with the study depending on its scope, which is agreed before it starts.

Put these five checks in your brief. Each one has a pass and a fail.

To scope the study, use our one-page market research brief template. Then email your brief to support@qualfacto.io. We aim to reply within one business day, Monday to Friday, 10:00 to 18:00 Indian Standard Time (IST).

We offer a free pilot, and pricing is a tailored proposal for each study. Email your brief to support@qualfacto.io.

If you want to take part in studies yourself: to be considered for suitable studies, register as a participant; registration does not guarantee an invitation, and invitations, rewards and requirements depend on each study.