Topic:

Think-aloud protocols

This topic page includes introductory information, a list of readings, and questions to guide your
reading and prepare you for class discussion; it may also include an individual or group
assignment, which may or may not be graded.

Introduction
Think-aloud protocols have been around for more than 40 years. They were intended as a way to
gather information about the cognition of human agents dealing with ill-structured problems. Ill-
structured problems contrast with well-structured problems; the latter type of problem is one
where the beginning state, end states, and available moves are all well known or articulated up-
front. Think of playing tic-tac-toe. Ill-structured problems lack some of that certainty; in fact, while
doing an ill-structured problem, one might be uncertain about exactly what the problem is.

In its most straight-forward form, the interviewer or facilitator asks the participant (sometimes
called the subject) to work through a task and to talk all the while about what she is thinking
while she is doing it. The participant is told to ignore the facilitator and not to ask the facilitator
for assistance. The facilitator is present only to remind the participant to keep thinking aloud (e.g.,
by saying Keep thinking aloud). In theory, this should result in the participant sharing what is
happening in her cognition from moment to moment without her editing it or attempting to
shape it to make herself look good (addressing two concerns with qualitative interviewing).

The major work on protocols is Ericsson and Simon (1993); that book is on reserve at the library
and is also available electronically through the library. An earlier article by them (1980) discusses
some of the main theoretical concerns with collecting recollections, which is what qualitative
interviews (about which we have already read) do. That article is available on T-Square.

Readings for this topic

van den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent
think-aloud protocols: Testing the usability of an online library catalogue. Behaviour &
Information Technology, 22(5), 339351. http://doi.org/10.1080/0044929031000
Optional: Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review,
87(3), 215251.

Reading questions
Read the following questions and comments and think about them as you read van den Haak et al.:
The authors refer to face validity. Look this up at Wikipedia or elsewhere. Why do you
think that protocol analysis has high face validity?
They also refer to other kinds of validity, including predictive validity, congruent validity,
and reliability. Think about what each of these terms means. Why do the authors and other
usability folks care about these characteristics of their research methods?
In what ways do the authors suggest that think-aloud protocols may be reactive?
Make sure you understand the difference between concurrent think-aloud (CTA) and
retrospective think-aloud (RTA). See p. 340 for definitions.

2015 Brian N. Larson

Topic: Think-aloud protocols

Page 1

At p. 341, the authors discuss reasons that RTA might be better than CTA, and reasons why
it might not be. Make sure you understand the widely accepted advantages and
disadvantages of RTA. Note that the authors propose to test some of these views with their
study.
Note the organization of the authors presentation of their own study. After their lengthy
introduction, they carefully describe their methods, present the results they obtained, and
then discuss the implications of those results. This is what is sometimes called the IMRaD
pattern: Introduction, Method, Results, and Discussion. You may want to consider it for
your own projects in this class.
On page 344, the instructions for the participant for the study in this article are quoted.
Locate and identify them. You may want to use a similar instruction if you choose to use
CTA for your projects in this class.
Note how the authors organized the problems their participants had/found into categories:
Layout problems, data entry problems, comprehensiveness problems, feedback problems.
Think about your own projects in this class: What categories of problems might you
anticipate users of the communications you are examining to have?
When the authors (p. 349) say that CTA resulted in identifying significantly more problems
detected by means of observation only, what do they mean? Contrast the fashion in which
RTA proved more useful.
The authors overall conclusion is a bit perplexing (p. 350): the results of this study
indicate that concurrent and retrospective think-aloud protocols can be regarded as
equivalent, but clearly different evaluation methods. A strong, and new argument in favour
of RTA protocols is that they may be less susceptible to the influence of task difficulty, both
in terms of reactivity and in terms of completeness of the verbalisations. What to they
mean by regarded as equivalent, but clearly different evaluation methods? Can you
rephrase that statement to make it sound less oxymoronic?

Classroom activities for this topic

Observe a think-aloud protocol in action.

Assignments for this topic

Post notes regarding the in-class think-aloud protocol to your groups forum before the
next class after the TAP.

Works cited

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: verbal reports as data (Rev. ed.). MIT
Press.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215251.

2015 Brian N. Larson

Topic: Think-aloud protocols

Page 2