Chezy Ofir, Itamar Simonson, & Song-Oh Yoon

The Robustness of the Effects of

Consumers’ Participation in Market
Research: The Case of Service
Quality Evaluations
The authors propose that participation in market research can determine consumers’ experiences with and
evaluations of marketing services/products. Building on and extending a prior finding that expectation to evaluate a
service (or product) leads to more negative evaluations, this research investigates the robustness, process, and
consequences of asking consumers to form evaluations of services/products. The results of several field studies
(conducted in Israel, Korea, and the United States) show that (1) the effect of expecting to evaluate on the service
provider’s performance evaluation is enduring and lasts beyond the immediate aftermath of the service encounter;
(2) the effect of study participation on the perception of marketers reflects true perceptual change rather than
“made-up” perceptions—it requires supporting evidence and is eliminated if cognitive load interferes in the
production of that evidence; and (3) the robust bias produced by expecting to evaluate cannot be eliminated by
causing participants to consider the task effect on their evaluations or the impact of their feedback on the evaluated
marketers. The authors discuss the broader implications of this research for understanding the impact of
participation in market research on consumers’ experiences and inputs.

Keywords: market research, service expectations, customer satisfaction, task effects, consumer experience

btaining customer input is the main task of market vices, design more effective advertisements). Regardless of

O research and is essential for the development of

effective marketing strategies. Thus, marketers use
various research techniques to find out how customers eval-
uate service quality and satisfaction, the importance of
attributes, and how consumers’ brand perceptions change
over time. At the same time, managers are often skeptical of
the value of consumer studies and whether the firm should
revise its current strategies, offerings, and communications
program on the basis of consumers’ evaluations and feed-
back. A key goal of the current research is to assist man-
agers and marketing researchers in understanding the fac-
tors that determine the validity of research findings and the
limitations of the data produced by the answers consumers
provide to survey questions, in general, and questions
regarding customer satisfaction, in particular.
When providing feedback to marketers, customers
might be aware that the researchers are trying to learn from
their responses and are using the information they provide
to improve performance (e.g., offer better products and ser-
Chezy Ofir is Academic Director of Davidson Research Center, School of
Business Administration, Hebrew University of Jerusalem (e-mail:
msofir@mscc.huji.ac.il). Itamar Simonson is Sebastian S. Kresge Profes-
sor of Marketing, Graduate School of Business, Stanford University
(e-mail: Itamars@stanford.edu). Song-Oh Yoon is Assistant Professor of
Marketing, Korea University Business School, Korea University (e-mail:
soyoon@korea.ac.kr). The authors thank the anonymous JM reviewers for
their helpful comments and acknowledge the support of the Kmart and
Davidson Research Centers at the Hebrew University.
whether customers receive monetary compensation for their
time and input, it is reasonable to assume that, in general,
they try to be helpful and provide useful inputs.
There are positive aspects to consumers’ desire to be
helpful. However, such a cooperative tendency also means
that consumers’/respondents’ perceptions of what their role
is and what being helpful entails might be key determinants
of how they perform their task. Even more important, the
mere participation in market research while or before expe-
riencing a product/service can affect customers’ experiences
and subsequent satisfaction (e.g., Dholakia and Morwitz
2002; Mandel and Nowlis 2008; Morwitz, Johnson, and
Schmittlein 1993; Ofir and Simonson 2001, 2007).
Although prior research has illustrated that participation
in market research can affect consumers’ perceptions, little
is known about the robustness, underlying mechanisms, and
boundaries of these effects. Furthermore, a question that
naturally arises is whether marketing researchers can con-
trol or eliminate such unintended effects of participation in
studies on consumers’ perceptions and evaluations by alert-
ing them to the possibility that the mere performance of a
task can affect their experience and satisfaction. The current
research examines these questions in the context of an
important and commonly used market research task—con-
sumers’ evaluations of service quality. Prior research has
shown that when consumers know in advance that they will
be asked to evaluate a service provider, they tend to evaluate
the service more negatively, thus misrepresenting the true
satisfaction of other customers (Ofir and Simonson 2001).

© 2009, American Marketing Association Journal of Marketing

ISSN: 0022-2429 (print), 1547-7185 (electronic) 105 Vol. 73 (November 2009), 105–114
 We use this finding to gain a broader understanding of
the problems that may arise by asking customers to provide
feedback and inform marketers, and we examine whether
such undesirable consequences of market research dissipate
over time and can be eliminated by revising task instruc-
tions. The results of several field studies (in Israel, Korea,
and the United States) show that (1) the negative effect of
expecting to evaluate on the service provider’s evaluation is
enduring and lasts beyond the immediate aftermath of the
service encounter; (2) the effect of study participation on
the perception of marketers reflects true rather than “made-
up” perceptions—it requires supporting evidence and is
eliminated if cognitive load interferes in the production of
that evidence; and (3) the robust bias produced by expecting
to evaluate cannot be eliminated by causing participants to
consider the task effect on their perceptions or the impact of
their feedback on the evaluated marketers. We discuss the
theoretical implications of the findings with respect to use
of market research and interpretation of consumers’ inputs
to marketers.
The Effect of Participation in
Market Research on Service
Evaluations: Duration, Boundaries,
and Underlying Process
A great deal of research in both psychology and marketing
has examined the role and consequences of demand charac-
teristics and demand artifacts (e.g., Orne 1962). Much of
this research has focused on experimental findings, espe-
cially the impact of such effects on the internal validity and
generalizability of laboratory studies. However, the notion
that experimental tasks and related cues may influence sub-
sequent perceptions and behavior is not limited to lab
experiments and the familiar demand effects; they may also
apply to forms of actual market research and other con-
sumer initiatives (e.g., Sherman 1980). For example, cus-
tomers who are shown an advertisement for a new product
and asked if they intend to buy it may, in some cases,
assume that they are expected to respond favorably to the
advertised product; in turn, this may affect their true evalua-
tions of the evaluated new products.
Such effects of participation in market research can
have a major impact on consumer behavior, regardless of
whether consumers have well-defined hypotheses regarding
their research roles. Even when there are no prior research
hypotheses and/or respondents discern the wrong hypothe-
ses, demand biases may be produced as long as respondents
act on what they believe to be the research hypotheses or
the purpose of the study. Furthermore, whereas guessing the
precise hypothesis may often be difficult, participants in
customer research and many other studies can have general
notions about what certain tasks entail (e.g., paying close
attention to details). In particular, when asked for their
inputs regarding the performance of marketers and the ser-
vices or products they offer, respondents will try to get
ready and be prepared to provide well-reasoned, construc-
tive feedback.
106 / Journal of Marketing, November 2009
In this research, we examine the impact of a common
customer research task in which a consumer is told in
advance to evaluate a service provider and then subse-
quently provides his or her feedback (Ofir and Simonson
2001). Considering the importance of delighting customers
and enhancing customer loyalty, marketers routinely mea-
sure customers’ service evaluations and satisfaction. Fur-
thermore, in many cases, customers know in advance that
they will be asked to evaluate, either based on prior experi-
ence or because they receive the evaluation form at the
beginning of the service encounter (e.g., when checking in
to a hotel).
When told before a service encounter about an upcom-
ing evaluation task, many consumers are likely to expect
that they should provide constructive suggestions for
improvement. Merely reporting that “everything is fine”
may be viewed as not doing a proper job and taking the
easy way out. Conversely, identifying areas that need
improvement can (1) help enhance marketers’ performance
and (2) provide the customer’s point of view to a marketer
and/or to customer researchers regarding issues or areas in
which customers expect better performance. Indeed, most
participants in a pilot study with 25 supermarket shoppers,
who were asked about the purpose of providing store
evaluations, referred to service improvement aspects
(details of this pilot study are available on request).
Customers who believe that they should identify defi-
ciencies may be more inclined, perhaps unconsciously, to
look for problems during the service encounter and to make
sure they have something to say and are able to support and
illustrate their feedback. In other words, to fulfill their role,
such customers may look for negative aspects. Such a
process might be unconscious, as is typically implied by the
notion of motivated reasoning (Kunda 1990), or conscious,
when respondents are aware of both their search for things
to improve and their resulting focus on negative perfor-
mance aspects.
Ofir and Simonson (2001) show that expectation to
evaluate a service or product produces more negative
evaluations, which reflects the internalization of respon-
dents’ perceived role, referred to as “negativity enhance-
ment” (i.e., the prominence of negative information), and
their “role expectations.” However, this prior research does
not address questions that are particularly important for our
understanding of the impact of marketing research, in gen-
eral, and service evaluations, in particular, on consumers’
experiences and perceptions. Specifically, prior research
does not enable us to determine whether the apparent
prominence of negative aspects reflects real changes in con-
sumers’ perceptions of service providers. In particular, true
changes in perceptions, rather than mere transient artifacts,
imply that the negative evaluations associated with expect-
ing to evaluate will persist well beyond the research context
and could be observed even several days or weeks follow-
ing the service experience. This question is important for
marketers because a lasting effect would indicate that the
harm created by participation in market research can influ-
ence long-term relationships with customers.
It is also important to determine whether the negative
bias of (expected) service evaluations can be eliminated by
revising the task. Specifically, it might be possible to
decrease the bias produced by participation in market
research by alerting consumers to the notion that expecting
to evaluate may affect their evaluations. The success of such
a debiasing manipulation is far from certain. Especially if
the bias produced by participation in market research is
unconscious, eliminating it is likely to be challenging when
consumers are unaware that they judge the service provider
more negatively simply because they were asked to provide
their evaluation. Conversely, if consumers recognize that
they tend to be too critical, the mere mention of such task
effects may diminish the resulting bias.
Our assumption is that though market research partici-
pants try to be helpful and to offer constructive feedback,
they are unlikely to fabricate or make up evidence when
none exists. That is, although expecting to evaluate leads
consumers to pay more attention to negative service aspects
(Ofir and Simonson 2001), they are unlikely to recognize
that this negative focus is due to their task (e.g., Nisbett and
Wilson 1977). Accordingly, customers are less likely to
report more negative evaluations if their ability to support
such evaluations is constrained (e.g., if they are under cog-
nitive load), and we do not expect them to manipulate the
valence of information to fit their presumed task.
In summary, we predict that the negative effect of
expecting to evaluate will endure beyond the immediate
aftermath of the service encounter and be resistant to cor-
rection or debiasing. Furthermore, consistent with the
assumption that the more negative (expected) evaluations
are based on the actual perceptions created during service
evaluations, when consumers lack the cognitive resources
needed to perform the task and gather (mostly negative) evi-
dence, the negative bias of expected evaluations will be less
pronounced. In addition, we predict that the negative bias of
expected evaluations will be resistant to change. The fol-
lowing hypotheses summarize our predictions:
H1: The negative effect of expecting to evaluate on service
evaluations extends beyond the immediate aftermath of
the service encounter.
H2: Restricting the cognitive resources during the service
encounter decreases the impact of expecting to evaluate
on evaluations.
H3: The negative effect of expecting to evaluate on service
evaluations persists even after survey participants are
alerted to a possible task effect; only explicit information
regarding the bias produced by the task decreases the
effect.
In Study 1, we test H1 by examining whether partici-
pants who expect to evaluate a store still hold more negative
evaluations of that store after the service encounter. In
Study 2, we test H2 by contrasting the effect of expecting to
evaluate under normal conditions with the effect of expect-
ing to evaluate under a cognitive-load condition. Finally, in
Study 3, we test H3 by examining the effect of (1) asking
respondents to consider whether there is an effect of expect-
ing to evaluate and what that effect might be and (2)
informing them about the negative effect of expecting to
evaluate.
Study 1: Does the Negative Effect
of Expecting to Evaluate Extend
Beyond the Service Encounter?
Although prior research has established that expectation to
evaluate generates more negative service evaluations imme-
diately after the experience (Ofir and Simonson 2001), we
still do not know whether that effect lasts beyond the ser-
vice encounter and has longer-term implications. Beyond its
practical implications, this question is important from a
theoretical perspective because it can provide information
as to whether the negative bias of expected evaluations is
genuine and reflects true evaluations or whether it is merely
a short-term effect that does not reflect true changes in
internal representation of the service experience. To exam-
ine this question, in Study 1, we varied both the task condi-
tion (expecting to evaluate or not expecting) and the timing
of the postservice evaluation (both immediately after the
shopping experience and again 4–5 days later or just 4–5
days later). This design enabled us to determine the robust-
ness of the negative bias of expected evaluations.
Method
The respondents were 120 consumers who were inter-
viewed immediately after shopping at a supermarket and/or
4–5 days after shopping at that supermarket. They were ran-
domly assigned to one of four conditions (n = 30 per
group), including two experimental groups and two control
groups. Shoppers assigned to the two experimental condi-
tions were informed before entering the supermarket that
they would later be asked several questions about customer
service. In one experimental group (Condition 1), con-
sumers were interviewed immediately after their shopping
trip and again, by telephone, 4–5 days later. In the second
experimental group (Condition 2), consumers were inter-
viewed by telephone only once, 4–5 days after shopping.
Respondents in the control groups were not informed about
the upcoming evaluation task before entering the store and
were interviewed either immediately after finishing their
shopping (Condition 3) or 4–5 days later (Condition 4). As
we explain subsequently, these control groups enabled us to
test two rival accounts for the results: (1) that more negative
delayed evaluations were produced by expressing more
negative evaluations immediately after shopping and (2)
that the more negative evaluations reflected the delay rather
than the effect of expecting to evaluate.
We used the same questionnaire in all four groups.
Specifically, we included the following measures: (1) listing
of thoughts that come to mind while shopping at the store;
(2) six items pertaining to service, including employee
courtesy, employee willingness to help, professionalism of
employees, waiting time, convenience in shopping, and
quality of service; (3) two overall satisfaction items, satis-
faction with the shopping trip and satisfaction with the ser-
vice; and (4) intention to recommend the store to friends
(all measures, except for thought listing, used seven-point
scales). As a manipulation check, we also asked respon-
dents if they had expected to be interviewed about their ser-
vice experience at the store.
Effects of Consumers’ Participation in Market Research / 107
Results
As we expected, respondents in the two control groups did
not expect to be questioned about their supermarket experi-
ence (average ratings in both groups were under 1.5 on a
seven-point scale), whereas those in the expected-
evaluation groups did expect to be questioned (average rat-
ings were greater than 6).
As Table 1 shows and consistent with prior research, the
evaluations of participants who expected to evaluate and
were interviewed immediately after shopping were signifi-
cantly lower than the evaluations of those in the control
group (for the combined service measure: MExpect = 4.72
versus MControl = 5.37; t(58) = 4.39, p < .0001; for the com-
bined satisfaction measure: MExpect = 4.58 versus MControl =
5.35; t(58) = 3.09, p < .003; and for recommendation:
MExpect = 4.60 versus MControl = 5.13; t(58) = 1.96, p < .05).
Moreover, the service evaluations and likelihood of rec-
ommendation by the same respondents (i.e., Condition 1)
after a delay of 4–5 days were significantly lower than their
own evaluations immediately after shopping (for the service
index: M = 4.14; paired-t(29) = 5.63, p < .0001; for recom-
mendation: M = 4.20; paired-t(29) = 2.35, p < .03). Further-
more, the difference between the number of positive and the
number of negative recalled aspects of the visit also indi-
cated an evaluation deterioration over time: (MDelay = –1.07
versus MImmediate = –.77; paired-t(29) = 2.07, p < .05). The
overall satisfaction index was (lower but) not significantly
lower after 4–5 days of delay (MDelay = 4.35 versus
MImmediate = 4.58; t(29) = 1.37, p > .1).
Next, we tested two rival accounts. First, it is possible
that the negative delayed evaluations were produced by the
more negative immediate (postshopping) evaluations of
participants who had expected to evaluate. However, the
evaluations of those in a second experimental group, who
expected to evaluate but were interviewed only 4–5 days
after shopping, were statistically indistinguishable from the
evaluations of those who expected to evaluate and were
interviewed at both times (see Table 1) (for service
[delayed]: MDelay after immediate = 4.14 versus MDelay only =
4.04; t(87) < 1; for satisfaction [delayed]: MDelay after immedi-
ate = 4.35 versus MDelay only = 4.27; t(87) < 1; for recom-
mendation [delayed]: MDelay after immediate = 4.20 versus
MDelay only = 4.00; t(87) < 1; and for the recall measure:
MDelay after immediate = –1.07 versus MDelay only = –1.03;
t(87) < 1).
Second, a rival account for the more negative delayed
evaluations is that they reflect the delay rather than the
effect of expecting to evaluate. However, the evaluations of
participants in the delayed-evaluation control group were
significantly more positive than the evaluations of those in
both expected-evaluation (delayed experimental) groups
(see Table 1). Because delayed evaluations in Conditions 1
and 2 were virtually identical and not significantly different,
we averaged and compared them with the delayed evalua-
tions of the control (Condition 4) (for service: t(87) = 8.59,
p < .001; for satisfaction: t(87) = 6.33, p < 001; and for rec-
ommending the store: t(87) = 6.74, p < .001).
Discussion
The results of Study 1 show that the (negative) effect of
expecting to evaluate on subsequent evaluations lasts
beyond the service encounter and has a longer-term effect
of consumers’ perceptions. Indeed, though tentative and
requiring replication, the results indicate that the negative
effect may even become stronger after time has passed. This
finding is consistent with findings in prior research showing
an increase in the extremity of attitude over a duration of
time, which provides an opportunity to process evidence
consistent with the initial attitude (Sadler and Tesser 1973;
Tesser and Conlee 1975).
The temporal robustness of the effect of expecting to
evaluate suggests that the resulting (more negative) evalua-
tions reflect true changes in the internal representation of
the store experience rather than an attempt to fulfill a pre-
conceived evaluation role. In particular, expecting to evalu-
ate the store’s service appears to change a consumer’s
actual shopping experience and to promote a more thorough
evaluation process, which might account for the stable
effect on evaluations. If this conjecture is correct, we would
expect conditions that limit the shoppers’ ability to evaluate
to reduce the effect of expecting to evaluate on evaluations.
We test this prediction in Study 2 using cognitive load to
control the available cognitive resources (Drolet, Luce, and
Simonson 2009; Ward and Mann 2000).

TABLE 1
Study 1 Means

Expected
Evaluations: Expected
Immediate and Evaluations: Only Control: Immediate Control: Delayed
Delayed Evaluations Delayed Evaluations Evaluations Evaluations
(Condition 1) (Condition 2) (Condition 3) (Condition 4)
Service index (1–7) 4.72 → 4.14 4.04 5.37 5.35
Satisfaction index (1–7) 4.58 → 4.35 4.27 5.35 5.33
Recommendation (1–7) 4.60 → 4.20 4.00 5.13 5.30
Recall index: number of
positive aspects –
number of negative
aspects –.77 → –1.07 –1.03 .03 –.27

108 / Journal of Marketing, November 2009

 Study 2: Using Cognitive Load
Timing to Test the Process
Underlying the Effect of Market
Research on Consumers’ Service
Evaluations
A question that arises is whether the expectation to evaluate
creates a desire to report more negative (and constructive)
evaluations and whether consumers report such negative
evaluations regardless of the actual information or evidence
they gather about the service at issue. That is, because con-
sumers who expect to evaluate are determined to be con-
structively negative, they will find something negative to
report regardless of the actual evidence they collect. Alter-
natively, it is possible that reporting negative evaluations
requires supporting evidence, and a failure to collect such
evidence will diminish the tendency to be more negative.
We examine this question in Study 2 using a manipulation
of cognitive load (Drolet and Luce 2004; Drolet, Luce, and
Simonson 2009).
We examine the effect of cognitive load that is applied
either before or after the evaluation task is performed.
Specifically, in addition to the standard expected-evaluation
and control groups, some supermarket shoppers were asked
to recall product names either before entering the store
(after being told about the evaluation task) or immediately
after shopping (before reporting their evaluations). We
expected that imposing cognitive load before shopping
would constrain consumers’ ability to gather the negative
evidence necessary for supporting negative evaluations. If
such evaluations are reported only when pertinent evidence
is gathered online, the cognitive load will diminish or elimi-
nate the effect of expecting to evaluate.
However, even if cognitive load mitigates such negative
evaluations, it may still have a neutralizing effect for other
reasons, such as a more general effect of distraction, even
without influencing the ability to gather (negative) evidence
during the shopping experience. Accordingly, we contrasted
the effect of cognitive load before entering a supermarket
with the same cognitive-load manipulation that occurs
immediately after exiting the supermarket but before the
evaluation task.
Method
Participants were 120 supermarket shoppers, who received
no compensation. They were randomly assigned to one of
four conditions. The expected-evaluation group was
informed of the upcoming service evaluation task before
they entered the store, whereas the control group was not.
Respondents in the cognitive-load-before-shopping group
were also informed of the upcoming evaluation task. How-
ever, they were then told that they would be participating in
a second, unrelated study that examined people’s ability to
recall product names. These respondents were then given a
minute and a half to memorize nine brand-product names,
and their recall was tested after they exited the store and
completed all other measures. Respondents assigned to the
cognitive-load-after-shopping group were informed of the
upcoming evaluation task before they entered the store and
then completed their shopping visit. When exiting the store,
they were informed of a second, unrelated product recall
study and were asked to memorize the same nine names
(followed by the store evaluation task). Thus, all four
groups of respondents—control, expected evaluation, cog-
nitive load before shopping, and cognitive load after shop-
ping—were interviewed immediately after they existed the
store.
Specifically, respondents were asked to rate the super-
market on nine dimensions, all on seven-point scales. Six
items assessed specific dimensions of service quality,
including politeness of store employees, employees’ will-
ingness to assist shoppers, employees’ professionalism,
length of wait at the checkout, convenience of product dis-
play, and level of service (Cronbach’s α = .75). Two items
measured satisfaction—overall satisfaction with the service
and overall satisfaction with the shopping visit (r = .86, p <
.0001)—and one item asked participants about their likeli-
hood to recommend the store to friends. Respondents were
also asked whether, while shopping at the store, they paid
more attention to positive (7) or negative (1) aspects. A
manipulation check confirmed that those in the control
group did not expect to be interviewed after shopping (M =
1.4 on a seven-point scale), whereas those in the expected-
evaluation, cognitive-load-before, and cognitive-load-after
groups expected to be reinterviewed (higher than M = 6.6
on a seven-point scale).
Results
As a replication of prior studies, shoppers in the expected-
evaluation group evaluated the supermarket more negatively
than those in the control group (for service: MExpect = 4.44
versus MControl = 5.46; for satisfaction: MExpect = 4.07 ver-
sus MControl = 5.40; and for recommendation: MExpect =
4.27 versus MControl = 5.77; ps < .001). The main test
focused on the contrast between the effect of cognitive load
before and after shopping. The results indicated that cogni-
tive load before entering the store statistically significantly
decreased the negative effect of expecting to evaluate,
whereas cognitive load after completing the store visit did
not.
Specifically, all average ratings were significantly lower
in the cognitive-load-after group than in the cognitive-load-
before and control groups for service (MLoad_after = 4.85,
MLoad_before = 5.14, and MControl = 5.46), for satisfaction
(MLoad_after = 4.35, MLoad_before = 4.87, and MControl =
5.40), and for recommendation (MLoad_after = 5.20,
MLoad_before = 6.10, and MControl = 5.77). Conversely, the
evaluations of the cognitive-load-before group and control
group were similar on all aspects (all ps >.1). Finally,
although the ratings of the cognitive-load-after group were
significantly lower than those of the control and cognitive-
load-before groups, they were higher than the ratings in
the expected-evaluation (no-load) group for service
(MLoad_after = 4.85 versus MExpect = 4.44; t(116) = 3.06, p <
.01) and recommendation (MLoad_after = 5.20 versus
MExpect = 4.27; t(116) = 3.60, p < .01).
Next, we examined the effect of cognitive load and its
timing on the participants’ stated focus of attention. Corre-
Effects of Consumers’ Participation in Market Research / 109
sponding to the evaluation findings, the cognitive-load-
before group reported significantly greater focus on positive
aspects than the cognitive-load-after group (MLoad_before =
5.10 versus MLoad_after = 4.30; F(1, 116) = 18.5, p < .001)
and was statistically indistinguishable from the control
(MLoad_before = 5.10 versus MControl = 5.13; F(1, 116) < 1,
p > .1). Conversely, the cognitive-load-after group was sta-
tistically indistinguishable from the expected-evaluation
group (MLoad_after = 4.30 versus MExpect = 4.27; F(1, 116) <
1, p > .1) and significantly more negative than the control
(MLoad_after = 4.30 versus MControl = 5.13; F(1, 116) =
20.03, p < .001).
Discussion
The main finding of Study 2 is that cognitive load before
the online evaluation (i.e., before the actual shopping
experience) greatly attenuates the effect of expecting to
evaluate, whereas cognitive load after the experience does
not. Furthermore, shoppers who received the name-recall
task (i.e., cognitive load) before shopping subsequently
reported that their focus of attention was more positive than
those in the expected-evaluation group and the cognitive-
load-after group. Evidently, cognitive load before shopping
limits the production of negative evaluations during the
actual experience. However, after the negative evaluation
has been formed, cognitive load does not inhibit shoppers
from reporting their already-formed evaluations.
These results indicate that expectation to evaluate tends
to cause a shopper to pay attention to negative aspects of the
store’s service when cognitive resources are available. Fur-
thermore, the findings of Study 2 imply that shoppers do
not “fabricate” negative evaluations just to comply with
their task, and they report such evaluations only if they are
able to gather supporting evidence during the shopping
experience.
We extended Study 2 in a follow-up study with 200
supermarket shoppers, who were randomly assigned to one
of four groups: control, expecting to evaluate, expecting to
evaluate + cognitive load after shopping, or just cognitive
load after shopping. We designed the fourth condition to
test whether cognitive load even without expecting to evalu-
ate produces more negative evaluations. Another objective
of the follow-up study was to contrast the content of recall
from the shopping visit across the four experimental
groups. As in Study 1, we computed for each group a recall
measure by subtracting the average number of positive
aspects recalled from the average number of negative
aspects recalled.
The store evaluation results in the control, expected-
evaluation, and expected-evaluation + load-after groups
were similar to those found in Study 2. We replicated the
basic effect (for service: MControl = 4.79 versus MExpecting =
4.06; t(196) = 8.06, p < .0001; for satisfaction: MControl =
4.97 versus MExpecting = 4.05; t(196) = 8.80, p < .0001; and
for recommendation: MControl = 4.56 versus MExpecting =
3.58; t(196) = 7.27, p < .0001). More important, the evalua-
tion ratings in the cognitive-load-after-shopping group
(without expected evaluation) were either statistically indis-
tinguishable from or higher than the control group (for ser-
110 / Journal of Marketing, November 2009
vice: MLoad_Only_After = 4.86 versus MControl = 4.79; t(196) <
1, p > .1; for satisfaction: MLoad_Only_After = 5.11 versus
MControl = 4.97; t(196) = 1.34, p > .1; and for recommenda-
tion: MLoad_Only_After = 4.96 versus MControl = 4.56; t(196) =
2.97, p < .003) and significantly more positive than those in
the expected-evaluation + cognitive-load-after-shopping
group (for service: MLoad_Only_After = 4.86 versus
MExpecting&Load_after = 4.28; t(196)= 6.40, p < .0001; for sat-
isfaction: MLoad_Only_After = 5.11 versus MExpecting&
Load_after = 4.28; t(196) = 7.94, p < .0001; and for recom-
mendation: MLoad_Only_After = 4.96 versus MExpecting&
Load_after = 3.84; t(196) = 8.30, p < .0001). Corresponding to
these results, the recall measure (number of positive less
negative aspects) was .84 in the control, –1.22 in the
expected-evaluation group (t(196) = 5.14, p < .0001), –1.28
in the expected-evaluation + load-after-shopping group, and
1.08 in the cognitive-load-after-shopping group (t(196) =
5.89, p < .0001). We obtained no significant difference
between the control and the load-only-after-shopping
groups (shopping groups: t(196) < 1, p > .1). Finally, the
pattern of results for the focus-of-attention measure (i.e.,
the explicit measure regarding the focus to positive versus
negative aspects of the supermarket) paralleled the evalua-
tion results for the four groups (MControl = 4.54, MExpecting =
3.46, MLoad_Only_After = 4.82, and MExpecting&Load_after =
3.84). Thus, the results of the follow-up study confirmed
that (1) cognitive load alone does not produce more nega-
tive store evaluations and (2) the valence of recall in the
cognitive-load-only-after-shopping group was similar to the
control group and significantly more positive than the
expected-evaluation group.
Another follow-up study, with 120 bookstore shoppers,
focused on the mediating role of consumers’ “role expecta-
tions.” As the pilot study we mentioned previously shows,
most consumers are likely to assume that their role is to
offer constructive feedback, which highlights areas on
which the service provider should improve its performance.
Accordingly, the study examined whether consumers’ per-
ceptions regarding the type of feedback that would be most
helpful to the store mediates the effect of expecting to eval-
uate. In addition to the same measures of service, satisfac-
tion, and recommendation as in Studies 1 and 2, we mea-
sured respondents’ perceptions of relative helpfulness of
positive versus negative feedback to the store using a seven-
point scale (1 = “negative feedback,” and 7 = “positive feed-
back”). In line with Baron and Kenny’s (1986) mediation
test, the results indicated that the consumers’ naive “role
expectation” theory mediates the effect of expecting to eval-
uate on service evaluations. First, as a replication of the
results of Study 1 and 2, there was a standard negative bias
of expected evaluations on all three measures (for service
quality: MExpect = 4.71 versus MControl = 5.22; t = 2.87, p <
.01; for satisfaction: MExpect = 4.85 versus MControl = 5.32;
t = 2.10, p < .05; and for recommendation: MExpect = 4.73
versus MControl = 5.36; t = 2.63, p < .01). Second, shoppers
who were expected to evaluate before entering the store
reported that negative versus positive feedback is more
helpful to the store manager in improving future service
(MExpect = 3.43 versus MControl = 4.01; t = 2.00, p < .05).
However, the significant main effect of expectation on
evaluation decreased significantly or marginally signifi-
cantly when we included both the task condition and the
perceived helpfulness of negative feedback as predictors.
The previously significant impact of expecting to evaluate
on overall satisfaction was no longer significant when we
included the perceived helpfulness of negative feedback as
a predictor (t = –1.43, p > .1). In addition, Sobel tests show
that the negative effect of expecting to evaluate on service
quality, satisfaction, and recommendation decreased signifi-
cantly when we added perceived helpfulness of negative
feedback as a predictor variable (Sobel’s Z = 1.77, p < .07;
Sobel’s Z = 1.81, p < .07; and Sobel’s Z = 2.05, p < .05,
respectively). As prior research (e.g., Ofir and Simonson
2001, Study 3) has shown, despite the apparent mediating
impact of role expectations, the prominence of negative
information (i.e., negativity enhancement) has an indepen-
dent, potentially greater impact on the resulting negative
evaluations.
Specifically, the prominence of negative information is
likely to interact with and be magnified by role expecta-
tions. Thus, as the results of Studies 1 and 2 indicate, con-
sumers do not just “make up” negative feedback to be help-
ful. Instead, they appear to internalize their market research
task, which unconsciously affects their shopping experi-
ence, the information they attend to, their recall, and subse-
quent enduring store evaluations.
A question that arises is, Can marketers seek customer
feedback and encourage consumers to pay attention to what
they like and dislike about their service without creating a
negative bias, which greatly undermines the value of the
feedback? That is, can the expected-evaluation task be
revised in such a way that the negative bias in resulting
evaluations is eliminated or reduced? In Study 3, we exam-
ine the impact of three revised expected-evaluation instruc-
tions that call the respondents’ attention to the possible
impact of their task and feedback.
Study 3: Can Predictions of Market
Research Task Effect Correct the
Negative Bias of Expecting to
Evaluate?
Study 3 examines the impact of three revised expected-
evaluation tasks, which call attention to the role of the
evaluation task and could potentially decrease or even
eliminate the effect. The evidence presented so far suggests
that the negative bias of expected evaluations is robust and
not easy to reverse (with the exception of cognitive load
before the shopping task). The question that arises is
whether calling study participants’ attention to the effect of
the evaluation task on their own evaluations and on the
store’s response to these evaluations can eliminate the nega-
tive bias of expected evaluations. Accordingly, the three
revised tasks tested in Study 3 include (1) explicitly inform-
ing shoppers that other consumers who were asked to eval-
uate tended to provide more negative evaluations; (2) pro-
viding more implicit information and asking those in the
expected-evaluation task to predict the effect of expecting
to evaluate on evaluations; and (3) asking respondents, after
they are informed about the upcoming evaluation but before
they enter the store, whether they expect the store to listen
to their feedback and consider it important. This third task
variation enables us to assess the impact of consumers’ per-
ceptions regarding the impact of their service evaluations.
We refer to the first revised task as “known effect,” to the
second as “predicted effect,” and to the third as “feedback
use.” Consistent with the previous findings, we expect the
shoppers’ negative bias of expected evaluations to be rather
resistant to change, except when they are explicitly told
about the “bias.”
Method
Participants were 175 supermarket shoppers who were ran-
domly assigned to one of five conditions (n = 35 in each
condition). The expected-evaluation (Condition 1) and con-
trol (Condition 2) groups were the same as in the prior stud-
ies. Respondents in the known-effect group (Condition 3)
were also told of the upcoming evaluation task. However,
they were then informed of a previous study that included
two groups, one that had been told about the evaluation task
and one that had not, which showed that the former group’s
evaluations were significantly lower than the latter’s.
Respondents in the predicted-effect group (Condition 4)
were also provided with a description of the previous study.
However, instead of being told the outcome, they were
asked for their opinions as to whether the subsequent satis-
faction with the store was the same in both groups or
whether one of the two tended (after the store visit) to be
more satisfied than the other. Respondents assigned to the
feedback-use group (Condition 5) answered (immediately
after being told about the subsequent evaluation task) two
questions on seven-point scales: (1) “To what extent do you
expect the supermarket to listen to customers’ feedback?”
and (2) “To what extent do you believe that the supermar-
ket’s management regards customers’ feedback as important
and useful?”
When exiting the store, all respondents were asked the
same questions. They rated the supermarket on nine seven-
point scales, including (1) six items that were designed to
assess specific dimensions of service quality (politeness of
store employees, employees’ willingness to assist shoppers,
employees’ professionalism, length of wait at the checkout,
convenience of product display, and level of service; Cron-
bach’s α = .86); (2) two overall satisfaction measures (with
service and the shopping visit; r = .81, p < .0001); and (3)
their likelihood to recommend the store to friends. Respon-
dents were also asked whether, while shopping at the store,
they paid more attention to positive (7) or negative (1)
aspects. A manipulation check confirmed that participants
in the control group did not expect to be interviewed after
shopping (M = 1.35 on a seven-point scale), whereas those
in the expected-evaluation conditions did expect to be rein-
terviewed (higher than M = 6.5 on a seven-point scale).
Results
As Table 2 shows, consistent with the prior studies, the
evaluation ratings in the expected-evaluation group were
Effects of Consumers’ Participation in Market Research / 111
 TABLE 2
Study 3 Means
Standard Expected
Control Evaluation
Service index (1–7) 5.60 4.56
Satisfaction index
(1–7) 5.72 4.40
Recommendation
(1–7) 5.40 3.91
Recall index:
number of positive
aspects – number
of negative
aspects .46 –1.17
lower than in the control (for service: MControl = 5.60 versus
MExpecting-no debias = 4.56; t(170) = 5.23, p < .0001; for satis-
faction: MControl = 5.72 versus MExpecting-no debias = 4.40;
t(170) = 4.96, p < .0001; and for recommendation:
MControl = 5.40 versus MExpecting-no debias = 3.91; t(170) =
4.01, p < .0001). Although the ratings were also slightly
lower in the known-effect group (i.e., the group told explic-
itly about the negative bias of expected evaluations) than in
the control, these differences were statistically insignificant
(see Table 2). The results suggest that the known-effect
group did not exhibit the bias, whereas the expected-
evaluation condition, without any debiasing procedure,
exhibited the bias.
The other two debiasing procedures failed to overcome
the effect of expecting to evaluate. Thus, asking consumers
to predict the direction of the expected-evaluation effect did
not diminish the negative bias. Specifically, the evaluations
were similar to those of the expected-evaluation group, and
the differences were insignificant (for service: MExpecting-no
debias = 4.56 versus MPredicted = 4.49; t(170) < 1; for satis-
faction: MExpecting-no debias = 4.40 versus MPredicted = 4.47;
t(170) < 1; and for recommendation: MExpecting-no debias =
3.91 versus MPredicted = 3.23; t(170) = 1.85, ps >.05). These
results were observed regardless of the predicted effect (i.e.,
both the majority who expected the task to have a negative
effect and those who expected a positive or no effect).
Furthermore, asking respondents before they entered the
store whether they expected management to listen to their
feedback and find it useful did not debias the subsequent
evaluations (i.e., the negative bias was still observed) (for
service: MExpecting-no debias = 4.56 versus MFeedback use =
4.46; t(170) < 1; for satisfaction: MExpecting-no debias = 4.40
versus MFeedback use = 4.33; t(170) < 1; and for recommen-
dation: MExpecting-no debias = 3.91 versus MFeedback use = 4.06;
t(170) < 1, ps > .1). As in the predicted-effect group, there
was no significant difference between those within the
feedback-use group who expected the store management to
use the customer feedback and those who did not (p > .1).
Finally, as Table 2 shows, the recall measures corresponded
to the pattern of ratings in the various conditions, such that
lower ratings were associated with a higher percentage of
negative versus positive recalled aspects.
112 / Journal of Marketing, November 2009
Expected: Expected: Expected:
Known Effect Predicted Effect Feedback Use
5.25 4.49 4.46
5.21 4.47 4.33
5.03 3.23 4.06
.74 –1.11 –.66
Discussion
The results of Study 3 further show the robustness of the
negative bias created when consumers are told in advance to
form evaluations of marketers’ performance. The only
group who received explicit information regarding that bias
did not show it, whereas other procedures that were likely
to enhance consumers’ task involvement and cause them to
consider possible task effects failed to correct the negative
bias of expected evaluations. That is, both the predicted-
effect group and the feedback-use group were indistinguish-
able in their subsequent store and service evaluations from
the standard expected-evaluation task. In real market
research applications, a company cannot and should not
lead respondents to address previously observed biases,
because such explicit instructions greatly undermine the
potential value of customer feedback and may simply
reflect demand effects. Thus, the observed bias produced
by participation in a rather commonly used service/
performance evaluation task is difficult to debias. These and
the previously reported results also highlight the process
mechanisms that generate this robust bias.
General Discussion
This research examines the scope, boundaries, and mecha-
nisms underlying the impact of participation in market
research on the evaluation of marketers’ performance.
Although our focus is on consumer feedback regarding ser-
vice performance, our findings have broader implications
regarding some unintended consequences of market
research with respect to customer loyalty and perceptions of
service quality. In particular, a primary determinant of the
value of market research is the degree to which the inputs
consumers provide in market studies/surveys can help mar-
keters improve their performance, such as offering better
products or better service. Thus, if consumers who partici-
pate in market research studies evaluate marketing perfor-
mance differently from those who do not, (1) the provided
inputs may be misleading, and (2) the impact of participa-
tion in market research may linger beyond the task and
affect subsequent attitudes and purchase behavior.
Indeed, Study 1 shows not only that consumers who
expect to evaluate a store perceive it more negatively (Ofir
and Simonson 2001) but also that the bias persists after
several days, regardless of whether service evaluations are
reported immediately after shopping or after a few days (or
at both times). Indeed, there was preliminary evidence that
the negative bias may exacerbate over time. This finding
indicates that more negative expected evaluations are not
made up but rather reflect the true resulting perceptions of
the market research participants.
The subsequent studies reinforce this conclusion and
demonstrate how difficult it is to correct the biased evalua-
tions. Study 2 shows that cognitive load while shopping
interferes with the production of the negative bias, which
indicates that the resulting perceptions are “real” and
require supporting evidence, which consumers under cogni-
tive load fail to obtain. Conversely, cognitive load after
shopping but before reporting store evaluations does not
correct the negatively biased perceptions.
Study 3 further shows how challenging it is to debias
market research participants; only participants explicitly
told about the specific negative bias observed in a prior
study managed to avoid it. Conversely, asking respondents
to consider the task effect or whether the store would listen
to their feedback failed to debias. Overall, these results indi-
cate that the negative bias of study participants is robust,
enduring, and difficult to debias.
We should recognize that most market research studies
often involve somewhat unnatural tasks that consumers do
not perform except when providing inputs to marketers. For
example, outside the context of market research, consumers
rarely rate brands, service, or attribute importance on a
seven-point or any other scale. Furthermore, although con-
sumers may spontaneously evaluate their product satisfac-
tion or the performance of a service provider, the knowl-
edge that participants prepare such evaluations as inputs to
marketers who want to improve their offerings is likely to
influence how these consumers approach the evaluation
task. Without an understanding of how this knowledge
affects the feedback-generation process and its resulting
content, marketers may often be unable to determine
whether and in what way to use consumers’ inputs.
Each marketing research task may trigger different pro-
cesses, perceptions, and naive role theories that affect the
reliability of consumer inputs. In some cases, consumer role
perceptions and theories regarding their task may not inter-
fere with accurate reporting of the true underlying states, if
such exist. For example, certain findings derived from labo-
ratory choice tasks (e.g., Dhar 1997; Huber, Payne, and
Puto 1982; Kivetz 2005) appear to replicate under real-
world conditions. However, in many other cases, con-
sumers’ beliefs about what they are supposed to do can play
a key role in how the task is performed, leading to unreli-
able and potentially misleading results. In particular, con-
sidering the growing evidence that consumer preferences
are often susceptible to seemingly irrelevant influences and
their evaluations of consumption experiences are fuzzy and
imprecise, perceived task roles can be especially influential.
Further research might examine other key marketing
research procedures in depth to determine what, if any, task
(mis)perceptions they produce and the resulting impact on
the reliability of the findings of such studies. Specifically,
the same three steps we employed in this research can be
followed. First, researchers need to identify the beliefs, if
any, that study participants form regarding their role and
how it should be performed. Second, researchers can exam-
ine the process and valence consequences of these beliefs.
Third, moderators of the observed effects need to be
investigated.
Furthermore, a key question that needs to be addressed
is whether and in what ways researchers might debias
participants in marketing studies. In general, the record of
debiasing procedures in terms of lasting effectiveness is
rather mixed (e.g., Fischhoff 1982). In the case of the effect
of expecting to evaluate, the simplest debiasing tactic is not
to forewarn consumers about the upcoming evaluation task
and merely interview them at the conclusion of the experi-
ence. This approach assumes that the control group pro-
vides unbiased and more accurate feedback, whereas those
who expect to evaluate are biased and report incorrect
assessments. However, a limitation of relying on the control
group (i.e., a nonforewarned group) is that people may
mindlessly and superficially report a high level of satisfac-
tion; conversely, the more careful evaluation of those in the
expected-evaluation group might more accurately reflect
consumers’ true long-term responses to the evaluated ser-
vice and the likelihood of switching to other service
providers. Determining unambiguously which condition is
more biased or less informative can be challenging and may
depend on the specific goals of the research.
For the current discussion, we continue to assume that
the more negative evaluations among those who expected to
evaluate represent a bias and that forewarning consumers
about the task is considered beneficial for obtaining careful
and accurate evaluations. Under these assumptions, a sim-
ple debiasing approach might be to tell participants that the
sole purpose of the research is to receive their honest and
unbiased opinion, considering both positive and negative
aspects they encounter. A downside of this approach is that
it might stimulate the production of hypotheses about the
objectives of the study and the way participants are
expected to perform. Alternatively, the researchers could
inform participants explicitly about the previously docu-
mented biases that might be created when consumers expect
to evaluate; that is, in addition to informing participants
about the upcoming task, researchers can forewarn them
about specific mistakes that expected evaluators tend to
make (as we did in Study 3).
As managerial reliance on marketing research tech-
niques increases, the limited understanding of the way
study participants perceive their roles and how such percep-
tions affect their behavior and reported evaluations becomes
more of an issue. For example, the task of portraying brands
as types of people may not be something that consumers
perform spontaneously in everyday life, but such tasks are
becoming increasingly popular in marketing research. Gain-
ing insights into consumers’ lay beliefs about this and other
common market research tasks is essential to understand
whether and in what ways market research findings can be
best employed.
Effects of Consumers’ Participation in Market Research / 113
REFERENCES
Baron, Reuben M. and David A. Kenny (1986), “The Moderator-
Mediator Variable Distinction in Social Psychological
Research: Conceptual, Strategic, and Statistical Considera-
tions,” Journal of Personality and Social Psychology, 51 (6),
1173–82.
Dhar, Ravi (1997), “Consumer Preference for a No-Choice
Option,” Journal of Consumer Research, 24 (2), 215–31.
Dholakia, Utpal M. and Vicki G. Morwitz (2002), “The Scope and
Persistence of Mere-Measurement Effects: Evidence from a
Field Study of Customer Satisfaction Measurement,” Journal
of Consumer Research, 29 (2), 159–67.
Drolet, Aimee and Mary Frances Luce (2004), “The Rationalizing
Effect of Cognitive Load on Emotion-Based Trade-Off Avoid-
ance,” Journal of Consumer Research, 31 (1), 63–77.
———, ———, and Itamar Simonson (2009), “When Does
Choice Reveal Preference? Moderators of Heuristic Versus
Goal-Based Choice,” Journal of Consumer Research, 36
(June), 137–47.
Fischhoff, Baruch (1982), “Debiasing,” in Judgments Under
Uncertainty: Heuristics and Biases, Daniel Kahneman, Paul
Slovic, and Amos Tversky, eds. Cambridge, UK: Cambridge
University Press, 422–44.
Huber, Joel, John W. Payne, and Christopher Puto (1982), “Adding
Asymmetrically Dominated Alternatives: Violations of Regu-
larity and the Similarity Hypothesis Source,” Journal of Con-
sumer Research, 9 (1), 90–98.
Kivetz, Ran (2005), “Promotion Reactance: The Role of Effort-
Reward Congruity,” Journal of Consumer Research, 31 (4),
725–36.
Kunda, Ziva (1990), “The Case for Motivated Reasoning,” Psy-
chological Bulletin, 108 (3), 480–98.
Mandel, Naomi and Stephen M. Nowlis (2008), “The Effect of
Making a Prediction About the Outcome of a Consumption
114 / Journal of Marketing, November 2009
Experience on the Enjoyment of That Experience,” Journal of
Consumer Research, 35 (1), 9–20.
Morwitz, Vicki G., Eric Johnson, and David Schmittlein (1993),
“Does Measuring Intent Change Behavior?” Journal of Con-
sumer Research, 20 (1), 46–61.
Nisbett, Richard and Timothy Wilson (1977), “Telling More Than
We Can Know: Verbal Reports on Mental Processes,” Psycho-
logical Review, 84 (3), 231–59.
Ofir, Chezy and Itamar Simonson (2001), “In Search of Negative
Customer Feedback: The Effect of Expecting to Evaluate on
Satisfaction Evaluations,” Journal of Marketing Research, 38
(May), 170–82.
——— and ——— (2007), “The Effect of Stating Expectations
on Customer Satisfaction and Shopping Experience,” Journal
of Marketing Research, 44 (February), 164–74.
Orne, Martin T. (1962), “On the Social Psychology of the Psycho-
logical Experiment: With Particular Reference to Demand
Characteristics and Their Implications,” American Psycholo-
gist, 17 (11), 776–83.
Sadler, Orin and Abraham Tesser (1973), “Some Efforts of
Salience and Time upon Interpersonal Hostility and Attraction
During Social Isolation,” Sociometry, 36 (1), 99–112.
Sherman, Steven J. (1980), “On the Self-Erasing Nature of Errors
of Prediction,” Journal of Personality and Social Psychology,
39 (2), 211–21.
Tesser, Abraham and Mary C. Conlee (1975), “Some Effects of
Time and Thought on Attitude Polarization,” Journal of Per-
sonality and Social Psychology, 31 (2), 262–70.
Ward, Andrew and Traci Mann (2000), “Don’t Mind if I Do This:
Disinhibited Eating Under Cognitive Load,” Journal of Person-
ality and Social Psychology, 78 (4), 753–63.