12 Bowden (JG/d) 11/7/02 10:06 am Page 322

HEALTH POLICY AND PLANNING; 17(3): 322330 Oxford University Press 2002

How to do (or not to do) . . .

Methods for pre-testing and piloting survey questions:

illustrations from the KENQOL survey of health-related quality
of life
A BOWDEN,1 JA FOX-RUSHBY,1 L NYANDIEKA2 AND J WANJAU2
1Health Policy Unit, London School of Hygiene and Tropical Medicine, UK, and 2Centre for Public Health Research,

Kenya Medical Research Institute, Nairobi, Kenya

Part of the assertion that any survey researcher can make about the validity of their results needs to contain
an analysis of questions and their responses from the respondents viewpoint. Claims concerning the valid-
ity, reliability and sensitivity of health-related quality of life measures tend to be based on the quantitative
approach of psychometrics, which fails to identify when respondents: misinterpret questions; do not recall
the information requested; or give answers that present themselves in a better or worse light. The paper
presents some approaches to pre-testing and piloting survey questionnaires to check the interpretation of
survey questions, using illustrations from the KENQOL project. The paper describes: how the intended
referential and connotative meaning of each question was established; the criteria to judge the appropriate-
ness of each question; the methods used to make those judgements; and the process of reviewing questions
based on findings. The role of piloting is highlighted, and further reading is suggested for readers wishing
to develop a model for their own investigation.

Key words: survey, pre-testing, piloting, Kenya, health-related quality of life

Introduction We were in the process of developing a survey tool to

Questionnaire-based surveys are a very common approach to
data collection used by researchers and the interpretation of
findings may be used to inform policy or public debate. It is
not uncommon to find reports in the media saying a recent
survey showed that X% of people think . . . . But do they?
How much trust can we place in the findings of surveys?
Answering questions on a survey requires several thought
processes: respondents have to interpret the question,
retrieve the necessary information and then decide what
answer to give and all this before the researcher gives their
own interpretation to the responses.
Part of the claim that any survey researcher can make about
the validity of their results needs to contain an analysis of the
questions from the respondents viewpoint. This is a salient
starting point either for those developing their own question-
naire or for those using an existing questionnaire in a new
context (either in a new population or new language). Failing
to investigate the interpretation of questionnaire items may
result in misinterpretations (by respondents and researchers),
falsified answers, missing responses (to the questionnaire as a
whole or to particular items), and possibly an offended
respondent who chases away the interviewer and encourages
others to refuse interviews.
measure the self-perceived and locally defined health of
people in Makueni District (Kenya), and reviewed practices
in the development and translation of such measures as part
of this process. We noticed that claims concerning the valid-
ity, reliability and sensitivity of health-related quality of life
(HRQL) measures tended to be based on the quantitative
approach of psychometrics. Such quantitative techniques are
unlikely to identify when respondents misinterpret ques-
tions, fail to recall the information requested or give answers
that present themselves in a better light. Indeed the interpre-
tation of questionnaire items has often been overlooked
(or at least not reported) in the development and trans-
lation of HRQL measures (Fox-Rushby and Parker 1995;
Bowden and Fox-Rushby 2000). It is also the case that well-
known texts in survey research often fail to address issues
of how to pre-test survey questions in sufficient detail
(Moser and Kalton 1992; Fowler 1993; Grosh and Glewwe
2000).
The aim of this paper is to present a variety of examples to
help researchers improve the validity,1 reliability2 and sensi-
tivity3 of their survey instruments prior to undertaking a
household survey. The paper first provides a brief context to
our research before proceeding to the range of methods used
to pre-test and pilot the survey instrument. The discussion
12 Bowden (JG/d) 11/7/02 10:06 am Page 323

How to do (or not to do) . . . 323

section focuses on the generalizability of our approach to Pre-testing the survey questions
testing questions for surveys and introduces the reader to a
Pre-testing is the main chance for researchers to gauge the
range of literature for further reading.
meaning attributed to survey questions before it is too late,
i.e. before a substantial investment is made in the wrong
Context of research questions or in questions where the researcher cannot be
sure about what is being asked. It is therefore imperative that
The overall aim of our research is to develop a culturally
the researchers undertake these interviews, rather than
relevant generic measure of health to measure the impact of
assigning the task to field assistants, because they know the
interventions designed to reduce disease and/or improve
aim of the research. The pre-testing involved a number of
health in Kenya. We have been working amongst the Kamba
steps including:
of Makueni District in Eastern Kenya and the Maragoli of
Vihiga District in Western Kenya. The conceptualization and
establishing the intended referential and connotative
operationalization of health emerged from extensive
meaning of each question;
periods of participant observation by anthropologists, in
agreeing a set of criteria to judge the appropriateness of
addition to qualitative research by others (see Fox-Rushby et
survey questions;
al. 1997; Fox-Rushby, for the KENQOL Group, 2000). The
selecting the methods for judging appropriateness and
definition of health adopted focused on the subjective per-
undertaking research;
ceptions of the positive and negative aspects of health that
reviewing questions for inclusion, revising (the question or
comprised contentment, cleanliness, corporeal capacity,
intended meaning) or exclusion.
co-operation and completeness of an intending individual
within the intentional worlds of Makueni and Vihiga (Fox-
Rushby, for the KENQOL Group, 2000). Step 1: Establishing the intended referential and
connotative meaning of each question
The KENQOL survey questionnaire was developed to
It is important to ascertain whether respondents are inter-
access those aspects of health that a member of the com-
preting the question as intended to ensure appropriate
munity could respond to in public4 to an interviewer who was
interpretations are placed on the data. Therefore we pre-
not necessarily known to them personally. This paper focuses
pared descriptions of the intended referential and connota-
on our work amongst the Kamba and reflects the develop-
tive meaning for each of the survey questions drawn up.
ment of the household survey in the Kamba language.5 The
There were several advantages to this: it served as an account-
aim of this part of our research was to investigate: how
able approach to assessing content validity; it facilitated
respondents understood the KENQOL survey questions;
clarification of our own intentions in asking questions; it
the types of information retrieved and over what time
ensured we assessed the same hypotheses as there was less
period; the types of socially acceptable answers that might
variation in our own interpretation of the meaning of the
be given and why; in addition to evaluating whether they
original questions (either over time or between researchers);
performed the task the way the researcher expected. We
and it was expected to be helpful in the future assessment of
tested a total of 150 potential health measurement survey
the conceptual equivalence of the Kamba language version
questions.
with other language versions (see Herdman et al. 1997, 1998).
Box 1 provides an example of the type of description we
The field site was located in Makueni District in Eastern
attached to a question.
Kenya. It is approximately halfway between Nairobi and
Mombasa, just north of the main trunk road. The Kamba
population (part of the Central Bantu group) is the majority
ethnic group in this region (99%) and traditionally they are
pastoralists. Currently 50% of the population are aged
014 years but infant mortality rates are high, at 96 per 1000
live births in 1995 (Kenya Government 2000). Rainfall is Box 1. Sample meaning for a KENQOL survey question
scarce in the area, with an annual precipitation of less than
500 mm and this, combined with high daytime temperatures,
leads to a hot, dry environment. Agriculture is unreliable and Do you feel that there are some people who could sabo-
food deficits are often experienced. Health facilities are rela- tage the progress of people in your neighbourhood at the
tively scarce, although they can be reached by matatu from moment?
the main Nairobi/Mombasa highway; the walk to the road can
take up to 2 hours. We want to find out an individuals current views about
whether they feel that their ability to improve them-
A days bus journey is required to reach hospital facilities in selves and their lives (collectively as a community)
Nairobi. The major diseases are malaria, respiratory tract may be adversely affected by people within the com-
infections, diarrhoeal diseases, skin ulcers, intestinal worms munity in some way. It is possible this may include a
and urinary tract infections (Kenya Government 2000). range of ideas such as violence and witchcraft and we
Incomes are low compared with many other areas in Kenya, leave it up to the respondent to decide if they are being
with many people living below the poverty line and involved threatened.
in a daily struggle for survival and development.
12 Bowden (JG/d) 11/7/02 10:06 am Page 324

324 How to do (or not to do) . . .

Table 1. Criteria selected for judging appropriateness of survey questions and investigative method employed

1 Criteria 2 Method

Expanded Targeted Group Expert

interviews methods methods reviews

1 No negative questions, and therefore no double negative answers

2 There are not two questions in one
3 Level of language is not too high/old fashioned/unusual in any sense
4 The question is simple and grammatically correct
5 The question is free from jargon
6 Singular and plural you is clear
7 The same idea is not contained in another question
8 The question is sensitive to measure change over time
9 Captures current views
10 Reflects local issues
11 Meaning and interpretation of question is clear
12 The question makes sense to everyone
13 A yes is clear and unambiguous in meaning
14 A no is clear and unambiguous in meaning
15 Time period is clear
16 Conception(s) of health it represents is specified
17 How the question relates to: time of year; head of household; individual/
homestead/wider family/community
18 Understanding of how it relates to severity of HRQL

Step 2: Criteria to judge the appropriateness of survey organize the expanded interviews we divided the set of survey
questions questions into groups of between three and five questions. In
each interview6 the respondent was first asked the KENQOL
Having developed the survey question and agreed on the
survey question and asked to respond either yes, no or
intended meaning, we developed a series of criteria for evalu-
dont know as they chose. The probes used were divided into
ating the appropriateness of the survey questions. This facili-
three types and used in separate interviews:
tated an agreed and standardized way of making decisions
about which questions to include and enabled different
(1) Comprehension of key phrases within the question. We
researchers to bring their own judgements together in a
asked respondents to talk about the kinds of things they
comparable way. Column 1 of Table 1 outlines the range of
thought about when answering the question. Examples of
criteria we adopted. It can be seen that the issues focus largely
key phrases included the time frame the respondent
on the style of language, attributed meaning, clarity of the
considered and who respondents included when the
question and consistency of interpretations.
word friends or neighbours was used (see Box 2 for an
example).
(2) Applicability of the questions: to people of different ages,
Step 3: Methods for judging appropriateness of survey
gender and socioeconomic groups (see Box 3 for an
questions
There are a wide variety of techniques available to
researchers wanting to test survey questions. We adopted
four principal types (described in Forsyth and Lessler 1991): Box 2. Comprehension of key phrases in the question
expanded interview techniques, targeted methods, group
methods and expert evaluation. Others approaches are
addressed in the discussion. Table 1 (column 2) indicates Do you feel safe at the moment?
which technique(s) addresses which type of issue and this
section outlines each method. Of these techniques, the The question intended to access concerns about per-
expanded interviews took most of our time. sonal safety and safety of belongings. The probe ques-
tions asked respondents to give examples for why
someone (a) will feel safe and (b) may not feel safe. It
Expanded interviews
emerged that the question not only addressed notions
Expanded interviews begin by one or more of the intended of personal safety and safety of belongings, but also
survey questions being asked and answered by the respon- having enough food, being properly taken care of and
dent, followed by a set of probe questions to investigate the having money for school fees.
thought processes involved in answering the question(s). To
12 Bowden (JG/d) 11/7/02 10:06 am Page 325

How to do (or not to do) . . . 325

Box 3. Applicability of questions for respondents of different ages

We asked respondents who in the family can and cannot be asked the question, At the moment are you always worry-
ing about getting food? The text below demonstrates some of our findings with respect to this question.

Interview with an elderly man:

Interviewer: Can children worry about getting food?

Respondent: No, since it is their fathers fault they are fed.

Interviewer: Could a young unmarried man answer this question?

Respondent: A young unmarried man isnt responsible for that. You will find that in most cases he comes at night and
gets in his room and sleeps. Maybe he has passed to his friends home and had something to eat. The old mamas cant
worry since they are just looked after by their children they brought up.

Interviewer: Is there anyone who might say no to this question?

Respondent: Old and young people.

example); by level of sensitivity of the question; and
social desirability of responses (see Box 4).
Box 4 has a particularly interesting response because it
also shows how sensitive the questioning on sensitivity
needs to be. It might at first appear that there are no prob-
lems with the question Ivindani yii nutonya kwiyikalya
wi mutheu undu utonya kwenda? (At the moment, are
you clean? meaning in a hygienic and moral sense),
but further probing highlights some potential problems.
(3) The meaning of a yes or no response needed to be clear
and as intended (see Box 5).
The respondents for the expanded interviews were volun-
teers willing to expend more time than the data collection

Box 4. Addressing the sensitivity of the question

We asked respondents directly about the sensitivity of questions. We needed to know whether a question was so sensi-
tive that people might purposefully provide the wrong answer. Alternatively they may provide the right answer, but this
makes the respondent or their family uncomfortable in some way. The transcript below is a translation from an inter-
view that addressed sensitivity.

Interviewer: At the moment, are you clean [hygienically and morally]?

Respondent: Yes

Interviewer: When we ask this question in the interview, we want to be sure that we are not embarrassing the person we
are asking. Do you think this question might cause embarrassment if it is asked in an interview?

Respondent: The question is good and cannot cause embarrassment in an interview.

Interviewer: Do you think people will feel they can answer this question truthfully in an interview?

Respondent: Not all people can answer truthfully. This is because if one feels unclean or has done wrong, he wouldnt
want to be known to have wronged.

Interviewer: If someone answers no to this question, how will it reflect on the family of that person?

Respondent: One who answers no would be thought to be unclean, but not the whole family.
12 Bowden (JG/d) 11/7/02 10:06 am Page 326

326 How to do (or not to do) . . .

Box 5. Testing the meaning of a yes or no response

By undertaking a relatively simple interview we could quickly establish whether a yes or no response could be inter-
preted clearly and as intended. This particular question was dropped because it became clear that we could not be sure
that a no meant worse health.

Interviewer: Do you get enough information about current events in Kenya?

Respondent: No.

Interviewer: If someone answers no to this question, is he/she in a better or worse position than someone who answers
yes?

Respondent: The one who answers yes is in a better position.

Interviewer: Why is this?

Respondent: Information about current events in Kenya can assist people in this area by helping them develop, such as
knowing different methods of farming, knowing about disease and outbreaks and ways of combating them, and also
improvement of educational facilities.

Interviewer: What other reasons are there that someone could answer yes?

Respondent: Those who own radios and are literate. People can get access to this information from radios, newspapers,
meetings and friends.

Interviewer: What other reasons are there that someone could answer no?

Respondent: Those who dont have radios and dont read newspapers. Everyone doesnt want to have access to infor-
mation about Kenya. Those who do are mostly interested in politics and current affairs. Those who arent are the old,
illiterate and those not interested in politics.

itself in helping us learn about the questions. They were
selected purposively to include equal proportions of males
and females, as well as being roughly representative of the
different generations in the community. In total each ques-
tion was the focus of between four and eight interviews with
a range of people from the local population.
Targeted methods
Targeted methods focus on a particular component of the
response process, to gain a detailed insight into the respon-
dents comprehension and preparation of a response and
to evaluate whether they performed the task the way the
researcher intended. We used three types of targeted
methods:
People with known (and opposite) characteristics of inter-
est were asked to complete specific questions to assess how
likely it was that the questionnaire would be interpreted in
the same way by people with particular characteristics. This
was important in being able to interpret any future differ-
ences/similarities across groups. Therefore we contacted
(through the help of our field assistants): people with and
without sickness/illness; the relatively wealthy and very
poor; and those with harmonious relationships compared
with those in disputes.
As the questionnaire was intended to be able to measure
change we asked eight people for one of two types of hypo-
thetical considerations of the questionnaire. For example,
members of a youth-based water and agriculture project
were asked to imagine how they might have answered the
questions before and after the project, and other key
informants were asked how they might have completed
the questionnaire during and after malaria. The task was
followed up with a discussion of the answers most likely to
change and why.
To address which conception of health people felt the
question represented, as well as helping further investi-
gations into the meaning and interpretation of the ques-
tion, respondents were asked to sort the Kikamba
questions into groups, either under the Kikamba words
used to translate the English word health7 that each ques-
tion was most closely related to, or to devise their own
categories and group them. Each question was placed on a
separate card for ease of sorting. The task was followed up
with an interview about the meanings of the Kikamba
words used to translate the English word health and
reasons for groupings.
12 Bowden (JG/d) 11/7/02 10:06 am Page 327

How to do (or not to do) . . . 327

Box 6. Group discussion on people in your homestead

Many of the KENQOL questions ask respondents about people in their homestead,i for example,

Do people in your homestead listen to your advice/suggestions at the moment?

When you are sick, do people in your homestead help you in getting treatment?
At the moment is it easy for people in your homestead to find transport to go for treatment when they are sick?

To check who people thought about when answering these questions, we provided several groups of respondents with
an example family (see below) and initiated a discussion about who different people in that family would think about
when they were asked specific questions.

Example family: Headed by a married male (mzee) and female (mama), who had three sons. The first son is married
to one wife with a child and they live just outside the compound. The second son is married with two wives (polygy-
nous marriage) and both wives have children. The third son lives away in Nairobi.

The discussion revealed that people would make reference to different people in their response. In the example above,
the mzee would consider seven adults, whereas the first-born son thinks of his wife and his wife thinks only of her
husband. To complicate matters further still, we discovered that a person could alter the group of people they thought
about depending on the question that was being asked, even if each question used identical phrases for people in your
homestead. We noted all the instances in which this occurred.

i The phrase musyi kwaku/kwenyu for homestead was used after finding that the word family was interpreted very
differently across people.

Group methods
Group methods also focus on a particular component of the
response process, but a group of people are addressed, rather
than an individual, to encourage greater discussion around
the issues involved. There were a number of key issues within
the survey questions that were repeated often, such as time
frames, the use of singular or plural you and terms such as
neighbours, friends and family. Individual interviews
were very helpful in understanding interpretations but did
not give us the opportunity to see opposing suggestions
debated. We therefore held group discussions on focused
issues. The group discussions lasted 12 hours, and were led
by two moderators. Each interview was recorded and later
transcribed. Each group involved people with similar ages
and gender, and the sum of groups captured the range of ages
and gender in the area. Box 6 gives an example of our
approach to finding out how people interpreted the word
homestead.
Expert evaluation
Expert evaluation involved no interaction with potential
survey respondents, but rather those who could offer particu-
lar guidance on specific issues. The experts included two
groups of people: external advisors including Kikamba Bible
translators and Kamba university lecturers on sociology; and
the field assistants living locally in Makueni who had assisted
the KENQOL research group for several years. The external
language advisors reviewed the written survey questions and
focused particularly on their interpretation of the meaning of
questions and issues 16 in Table 1. The local field assistants
also reviewed the written Kikamba questions and gave their
views on issues 16 as well as 1317. They were very helpful
in their criticisms and suggestions, and once we had been able
to take down their advice, it was also easier for them to inter-
view others (as opposed to discussing their own ideas with
interviewees).
Step 4: Reviewing questions for inclusion, revising the
question or intended meaning, or dropping questions
The field transcripts and results from each research method
were collated and summarized by the authors for each survey
question. Reviews of questions were supported by a dis-
cussion between the authors and field assistants. Following
these discussions, decisions were made concerning the action
required for each question including:
acceptance of the original question and meaning;
acceptance of the original question, with a change to the
meaning;
changing the question (slightly) but keeping the meaning
the same;
dropping the question;
writing a new question (and new intended meaning).
Any changes were tested through additional expanded
interviews and any new questions also went through various
translation checks. Prior to accepting any new or changed
questions all data on the question were once again collated
and reviewed prior to being taken forward to piloting.
12 Bowden (JG/d) 11/7/02 10:06 am Page 328

328 How to do (or not to do) . . .

Piloting into other languages. There has been some use of qualitative
techniques to assess the content validity of original (e.g.
Moser and Kalton (1992) refer to piloting as the dress
Saxena et al. 1998), but more usually the adapted versions of,
rehearsal. Questions are placed together as it is expected that
generic HRQL instruments (e.g. Chwalow et al. 1992; Lam et
they will appear on the final questionnaire and the dynamics of
al. 1994), as well as the applicability of generic instruments to
the survey as a whole are investigated. The KENQOL group
specific disease groups (e.g. Tazaki et al. 1998). These investi-
undertook piloting in two stages. The first focused on ques-
gations tend to be relatively unstructured, and can be
tions from each sub-section (n = 6) of the survey. The second
described as cursory investigations. In general there is
involved straight runs through the entire survey. One of the
too much emphasis placed on quantitative assessments of
authors was present at each of these interviews, and a field
reliability, validity and sensitivity. The quantitative assess-
assistant conducted the interview in Kikamba. At both stages
ment is a rather closed approach, that ignores the respon-
of piloting, the interviewer noted down the response to each
dents own interpretation of questions.
question (yes, no, dont know, no response). They also noted
any other information the respondent gave, such as asking for
We therefore adopted a qualitative approach to the assess-
clarification of the question, or the respondent providing
ment of validity, reliability and sensitivity. This made both the
additional information after responding yes, no or dont
researchers and the respondents interpretation of questions
know. In addition, the authors noted any body language,
more open and accountable, and ensured that the final survey
pauses before a response, emotional reactions to questions etc.
questions were more reliable and valid and able to detect
At the end of the pilot survey interviews, each respondent was
change as revisions were made. Earlier we defined a reliable
asked for their opinion about the questions, including:
measure as one which yields the same results in repeated
trials (Carmine and Zeller 1979). A question may be un-
what they thought about the questions in general;
reliable due to bad wording, causing a person to understand
whether any of the questions seemed to be strange or
the question differently on different occasions (De Vaus
unusual;
1995). This paper has described procedures for assessing the
their opinion on the order of questions;
reliability of questions during the pre-testing stage, for
how appropriate the response categories were;
example identifying questions that cannot elicit a double
whether the correct procedures are used to address:
negative for a response, do not contain high level or old fash-
(a) married and unmarried people,
ioned language, and are free from jargon. A valid measure is
(b) old and young people;
one which measures what is intended (Carmine and Zeller
whether any questions should not be asked in the survey;
1979), and methods described in this paper which helped to
to point out any questions they do not want to answer, or
address this included ensuring that the questions capture
think they should not be asked;
current views, reflect local issues and the conception of health
any problems they thought we might encounter asking
each question represents is clear. Similarly, during pre-testing
people the questions:
the sensitivity to questions over time was investigated. De
(a) alone,
Vaus (1995) explains that instead of this procedure some
(b) with others listening (e.g. young people were asked
people include a large number of indicators and at a later
whether they might have inhibitions answering ques-
stage eliminate those that do not seem to be working.
tions in front of parents);
However, he explains: this seems to be the wrong way of
whether any questions appeared to be asking the same
doing things since we will end up by defining the concept in
thing;
terms of the indicators which worked. If this is done then the
their opinion on what changes we could make to improve
indicators may not represent the concepts of the theory we
this survey in terms of:
set out to test and as such the research can end up having
(a) introduction,
little relevance to the original research question (De Vaus
(b) questions;
1995: 54).
which questions might be difficult for someone to talk
about in front of an interviewer who comes from the same
However, we recognize that the process was labour intensive
area as the respondent;
and took a considerable amount of time (around 6 months of
whether it matters who interviews people;
researcher and 12 months of field assistants time). In our case
who would be a good interviewer in terms of age, gender
we judged this to be an important part of the process and
and any other characteristics.
outcome of our research. First, we are intending that this
measure becomes used across many different diseases as
Based on feedback during piloting, several additional
an outcome measure used to judge the effectiveness and
revisions were made to the KENQOL questions. Each
efficiency of health-improving interventions. It is therefore
change was followed up by further piloting (and pre-testing
important to us that the people whose health we are trying to
where necessary) until the final version of the KENQOL
measure (and improve) are integrated into this process, and
survey was agreed.
it is often the case that generic HRQL instruments are devel-
oped and used for years following their development. Sec-
ondly, it provided valuable comparative data for explaining
Discussion
any similarities and dissonances in measured health in Vihiga
We have been concerned about the procedures used in the district, Kenya. Related to this is the possibility that others
development of HRQL measures as well as their translations would want to translate the instrument for use elsewhere, and
12 Bowden (JG/d) 11/7/02 10:06 am Page 329
How to do (or not to do) . . .
we wanted to improve the basis from which judgements are
made about the equivalence of future translations.
There are other techniques that are available to researchers
to test their survey questions that we did not use. The
KENQOL researchers adopted some cognitive assessment
survey methodologies, but there are others that may be more
appropriate for other research groups (Jobe and Mingay
1989):
Concurrent think-aloud interviews: respondents think
aloud when answering questions, and responses are
probed.
Retrospective think-aloud interviews: respondents answer
all question first and then are asked how they arrived at
their answers.
Confidence ratings: respondents relate the degree of confi-
dence they have in the accuracy of their answers.
Paraphrasing: respondents repeat the questions in their
own words.
Response latency: measurement of the elapsed time
between the questions and respondents answers.
The KENQOL group also adopted some methods recognized
as participatory or rapid rural appraisal (PRA and RRA)
approaches, for example the use of anthropologists and local
leaders as key informants (Chambers 1983). Other useful
RRA and PRA approaches might include a well-being (or
HRQL) ranking exercise of local households with local
participants, and linking this to likely responses to questions
and analysis of difference based on responses to questions
(see Chambers 1997).
Whilst the approach we have taken to the qualitative assess-
ment of meaning attached to the KENQOL survey questions
is unique amongst generic HRQL research, many of the ideas
were drawn from a variety of sources, including survey
research (e.g. Moser and Kalton 1992; Fowler 1993), trans-
lation studies (e.g. Larson 1984; Barnwell 1992), the cognitive
sciences (e.g. Jobe 1990; Forsyth and Lessler 1991) as well as
the health measurement literature (e.g. Guillemin et al. 1993;
Herdman et al. 1997, 1998). De Vaus (1995) is particularly
useful because of the links made between investigating ques-
tionnaire interpretation and the reliability and validity of a
questionnaire. This gives an indication of the applicability of
our approach to the design of survey questions more gener-
ally. As Tourangeau (1984) states [in] the respondents task
. . . there is considerable room for error. Respondents may
misunderstand the question or the response categories; they
may forget or misremember the crucial information; they may
misjudge the information they do recall; and they may mis-
report their answer (Tourangeau 1984: 7374).
Finally, to help researchers give greater attention to investi-
gating whether respondents interpret questions in ways
researchers expect (or hope), we give a brief introduction to
a variety of different types of source texts. Most social
research texts address questionnaire design, but few provide
detail on investigations into question meanings. One text that
is useful in this context is Oppenheim (2000). For an excellent
insight into the many pitfalls associated with eliciting the
329
required information through household survey, readers
should consider some of the literature from demographic
studies; some examples include Caldwell and Igun (1997),
Ewbank (1981), Bledsoe and Pison (1994). We found cogni-
tive research assessment a particularly inspiring approach as
it captures approaches given to a variety of psychological
processes involved in information retrieval. Some of this
literature has already been cited above, but readers might
also like to consider Lessler and Tourangeu (1989) and the
proceedings from a seminar on the Cognitive Aspects of
Survey Methodology (Sirken et al. 1999). Researchers in the
field of marketing have also been considering these issues for
some time, consider for example Gendall and Hoek (1990)
and Kumar et al. (1999).
Conclusions/recommendations
The investigations described in this paper should not be seen
as an unusual step in the development and testing of survey
questionnaires particularly those that will be used repeat-
edly. The validity and reliability (and possibly sensitivity) of
any survey is important to establish prior to accepting results
and conclusions. Each of these ideas can begin to be explored,
and improved, through the use of qualitative methods in pre-
testing and piloting questionnaires. Indeed, evidence that a
survey tool has content validity is required, rather than
simply relying on unsubstantiated assertions by researchers,
and only once this has been demonstrated is it appropriate to
adopt quantitative assessment methods for further analysis.
Finally, the knowledge gained about the interpretation of
survey questions will also help in judging the equivalence
of any future translations of the survey questionnaire.
Endnotes
1 Validity determines that an instrument is measuring what was
intended. It concerns the crucial relationship between concept and
indicator. The objective is to ensure that the indicator represents the
intended (and only the intended) concept (Carmines and Zeller
1979).
2 Reliability concerns the extent to which an experiment, test
and any measuring procedure yields the same results on repeated
trials (Carmines and Zeller 1979).
3 The ability of a measure to detect change.
4 Because most household interviews happen outside the home
with many people around. To ask to interview people privately in a
household survey tends to arouse a lot of suspicion.
5 Our household survey eventually consisted of 112 health
measurement questions and 50 socioeconomic and demographic
questions. We achieved a total of 550 interviews a response rate of
81.3% and a mean item response rate of 98.1% (SD 3.6).
6 All interviews were undertaken in Kikamba, the local lan-
guage of the Kamba population we were working amongst. The
Kikamba versions of the KENQOL questions were used for all
survey testing, although the English versions are reported here. The
pre-test interviews were written up in full in English to facilitate
participation by the non-Kikamba speaking researchers.
7 wailu, wianie, utheu, useo, wiwa nesa, uima.
References
Barnwell K. 1992. Bible Translation: An introductory course in trans-
lation principles (3rd edition). Dallas, TX: Summer Institute of
Linguistics International.
12 Bowden (JG/d) 11/7/02 10:06 am Page 330
330 How to do (or not to do) . . .
Bledsoe C, Pison G. 1994. Introduction. In: Bledsoe C, Pison G (eds).
Nuptiality in Sub-Saharan Africa: Contemporary anthropo-
logical and demographic perspectives. Oxford: Clarendon Press,
pp. 122.
Bowden A, Fox-Rushby J. 2000. A critical review of the adaptation
and use of generic HRQL measures amongst populations
outside of North America, western/southern/northern Europe,
Australia and New Zealand. Quality of Life Research 9: 313.
Caldwell JC, Igun AA. 1971. An experiment with census-type age
enumeration in Nigeria. Population Studies 25: 287302.
Carmines EG, Zeller RA. 1979. Reliability and validity assessment.
London: Sage Publications.
Chambers R. 1983. Rural development: putting the last first. Harlow,
UK: Longman.
Chambers R. 1997. Whose reality counts? Putting the first last.
London: Intermediate Technology.
Chwalow AJ, Lurie A, Bean K et al. 1992. A French version of the
Sickness Impact Profile (SIP): stages in the cross-cultural vali-
dation of a generic quality of life scale. Fundamental Clinical
Pharmacology 6: 31926.
De Vaus. 1995. Surveys in social research (4th edition). London:
Routledge.
Ewbank DC. 1981. Age misreporting and age-selective undernumer-
ation: sources, patterns, and consequences for demographic
analysis. Committee on Population and Demography. Report
No. 4. Washington, DC: National Academy Press.
Forsyth BH, Lessler JT. 1991. Cognitive laboratory methods: a tax-
onomy. In: Biemer PP, Groves SM, Lyberg LE, Mathiowetz
NA, Sudan S (eds). Measurement errors in surveys. New York:
John Wiley, pp. 393418.
Fowler FJ. 1993. Survey research methods (2nd edition). London:
Sage Publications.
Fox-Rushby J, Parker M. 1995. Culture and the measurement of
health related quality of life. European Review of Applied Psy-
chology 45: 25763.
Fox-Rushby J, Johnson K, Mwanzo I et al. 1997. Creating an instru-
ment to assess lay perceptions of HRQL: options and impli-
cations. Quality of Life Research 6: 633.
Fox-Rushby J, for the KENQOL Group. 2000. Operationalising con-
ceptions of health amongst the Wakamba and Maragoli of
Kenya: the basis of the KENQOL instrument. Quality of Life
Research 9: 316.
Gendall P, Hoek J. 1990. A question of wording. Marketing Bulletin
1: 2536.
Grosh M, Glewwe P. 2000. Designing household survey question-
naires for developing countries: lessons from 15 years of the
Living Standard Measurement Survey (Volume 1). Washington,
DC: World Bank.
Guillemin F, Bombardier C, Beaton D. 1993. Cross-cultural adap-
tation of health-related quality of life measures: literature
review and proposed guidelines. Journal of Clinical Epidemiol-
ogy 46: 141732.
Herdman M, Fox-Rushby JA, Badia X. 1997. Equivalence and the
translation and adaptation of health-related quality of life ques-
tionnaires. Quality of Life Research 6: 23747.
Herdman M, Fox-Rushby JA, Badia X. 1998. A model of equival-
ence in the cultural adaptation of HRQL measures: a univer-
salist approach. Quality of Life Research 7: 32335.
Jobe JB. 1990. Research on questionnaire design: perspectives from
other disciplines. American Journal of Epidemiology 132: 824.
Jobe JB, Mingay DJ. 1989. Cognitive research improves question-
naires. American Journal Public Health 79: 10535.
Kenya Government. 2000. Makueni District Development Report
19972001. Nairobi: Office of the Vice President and Ministry of
Planning and National Development.
Kumar V, Aaker DA, Day GS. 1999. Essentials of marketing
research. Chichester, UK: John Wiley.
Lam CL, Van Weel C, Lauder IJ. 1994. Can the Dartmouth COOP/
WONCA charts be used to assess the functional status of
Chinese patients? Family Practice 11: 8594.
Larson ML. 1984. Meaning based translation: a guide to cross-lan-
guage equivalence. Lanham, MD: University Press of America
Lessler JT, Tourangeau R. 1989. Vital health statistics. Series 6.1.
Washington, DC: National Center for Health Statistics.
Moser CA, Kalton G. 1992. Survey methods in social investigation
(2nd edition). Aldershot, UK: Gower.
Oppenheim AN. 2000. Questionnaire design, interviewing and
attitude measurement. Buckingham, UK: Open University
Press.
Saxena S, Chandiramani K, Bhargava R. 1998. WHOQOL-Hindi: A
questionnaire for assessing the quality of life in health care set-
tings in India. National Medical Journal of India 11: 1605.
Sirken M, Jabine T, Willis G, Martin E, Tucker C. 1999. A New
Agenda for Interdisciplinary Survey Research Methods: Pro-
ceedings of the CASM II Seminar. Hyattsville, MD: National
Center for Health Statistics. [http://www.cdc.gov/nchs/data/
casm2pro.pdf, 2nd August 2001]
Tazaki M, Nakane Y, Endo T et al. 1998. Results of a qualitative and
field study using the WHOQOL instrument for cancer patients.
Japanese Journal of Clinical Oncology 28: 13441.
Tourangeau R. 1984. Cognitive sciences and survey methods. In:
Jabine TB (ed). Cognitive aspects of survey methodology: build-
ing a bridge between disciplines. Washington, DC: National
Academy Press, pp. 73100.
Acknowledgements
We thank the following organizations for the funding of this survey:
UNDP/World Bank/WHO Special Programme for Research and
Training in Tropical Disease, the UK Economic and Social Research
Council (Ref. H52427005994) and the Health Economics and
Financing Programme (funded by the Department for International
Development). We also thank all our field assistants, especially
Justus Mutunga, Jeremiah Nyamai and Alex Muvea, for their help,
guidance and enthusiasm. Finally we thank all the people who
participated in the preparation for this survey and made us welcome
in their homes.
Biographies
Annabel Bowden, Ph.D. (Health Geography), was a Research Fellow
at the London School of Hygiene and Tropical Medicine (LSHTM)
at the time of this study. She was involved with the KENQOL project
for 5 years and was based in Kenya for 1 year during the household
survey. She also has a wider interest in the applicability of
adapted/translated versions of generic health-related quality of life
measures. She is now based with the Lewin Group (UK).
Julia Fox-Rushby, Ph.D. (Economics), is a Senior Lecturer at the
LSHTM. She has written numerous academic papers on the cost-
effectiveness of health interventions across the world, specializing
particularly in maternal and child health, malaria and, more recently,
vaccine preventable disease. She has also been involved over the past
15 years in developing a number of non-disease specific measures of
health-related quality of life as a member of the EuroQol group,
advisor to the WHOQOL group and as principal investigator of the
KENQOL group.
Lilian Nyandieka, MA (Sociology), is a Research Officer at the
Center for Public Health Research, a department of the Kenya
Medical Research Institute (PO Box 20752, Nairobi, Kenya). She has
been involved in the KENQOL project for 3 years and is also
researching social and economic aspects of malaria.
John Wanjau, BA (Anthropology), is an Assistant Research Officer
at the Center for Public Health Research, Kenya Medical Research
Institute. He has been involved in the KENQOL project for 3 years.
Correspondence: Julia Fox-Rushby Ph.D., Health Policy Unit,
London School of Hygiene and Tropical Medicine, Keppel Street,
London WC1E 7HT, UK. Tel: +44 (0) 20 7927 2267, Fax: +44 (0) 20
7637 5391, Email: Julia.Fox-Rushby@lshtm.ac.uk