Developing Data Collection Tools

[insert date]
Qualitative Methods
in Evaluation of Public
Health Programs
Session 4
The evaluation process
Learning objectives
At end of this session, participants
will be able to:
 Identify specific tools for qualitative data collection
 Describe the structure and components
of qualitative data collection tools
 Formulate sets of questions that can address specific
evaluation components in data collection instruments
 Demonstrate use of probes to elicit
in-depth responses
 Design a tool with logical flow of questions
Qualitative evaluation
Qualitative evaluation data collection tools:
 Are written outlines of topics to be
covered during a focus group discussion
or in-depth interview
 Are checklists to ensure that all important topics are
covered during discussion
or interview
 Provide flexible direction and documentation of a
central aspect of the evaluation

**What are some tools used

in qualitative data collection?
Data collection tools
Examples

 Focus group discussion guide

 In-depth interview guide
 Observation checklist/guide
 Topic guide
 Others…

Unstructured and Semistructured

Structure of
qualitative
evaluation
Data
collection
tools
Structure of qualitative evaluation
Data collection tools
Combine structure with flexibility
 Evaluator must have a sense of areas to explore, and sufficiently
permit topics to
be covered in the order most
suited to the interviewee
 Ensure subject coverage through
literature review, assessing study
aims, and engaging larger teams
Exploratory
 Evaluator uses a range of probes and other techniques to achieve
depth of answer in terms
of penetration, exploration, and explanation
Structure
Interactive
 Allow responses to be fully probed and explored—allows the
evaluator to be responsive to relevant issues raised
spontaneously by
the interview
Generative
 New knowledge or thoughts are
likely to be created
 Varies depending on
evaluation question
 It is likely that the participant will
at some point direct themselves, or be directed
by the researcher, down unexplored avenues
of thought
Techniques
for achieving
flexibility
Techniques for achieving flexibility

Establish subjects to be explored; ask different kinds of

questions:
 Ensure all aspects of evaluation questions
are asked and answered
 Use probes: Standard expressions that encourage
participants to keep talking on
the subject; provide examples and details
 Use follow-up questions: Ask interviewee to elaborate on
key concepts, themes, ideas,
and events
 Use prompts: Help participant recall
Techniques for framing questions
 Content mapping
 Content mining
 Using enabling
and projective techniques
o E.g., vignettes

 “Tree-and-branch model”;
issues specified for follow-up
 “Rivers and channels model” follows “channels” or
themes wherever they lead
Content mapping questions

Designed to:

 Open the conversation on an evaluation

topic(s)/question

 Engage participants in identifying what is

important and relevant
to them
Content mapping questions (2)

Example: You are conducting a formative

evaluation to understand
why use of skilled birth attendants is
low in an area.

What question can be asked to open

up a conversation on the use of facility-based
delivery during a focus group discussion (FGD)
with pregnant women
in a community?
 Example content mapping questions
Type of content mapping question Examples

Ground mapping Describe what pregnant women in

 To open up a subject this community do when they are
in labor.

Dimension mapping What happens next?

 Focus the participant
more narrowly on
particular topics; signposts,
for chronological order

Perspective-widening questions What other factors would

 Understand participant’s perspective influence their decision?
fully
Content mapping
Information can be used to map out factors that influence
use of facility-based delivery services in a community.

Accessibility

Personal/
Community
individual
norms
choices

Use of facility-
based delivery
services
Content mining questions
 Explore the details within given dimensions
or topics
 Understand meaning from the participant’s perspective

Example: Assuming our opening questions identified

individual/personal, service,
and community factors:

What can your questions seek

to explore in details?
 Example content mining questions
Type
Amplifying probes: Used to encourage
participants to elaborate
Examples
You said that you went to the traditional
birth attendant to deliver. Can you tell me
a little more about the kinds of services
that you go to the TBA for?

Exploratory probes: Exploring How did your approach change when you
consequences also helps to illuminate received that service?
experiences and behaviors

Explanatory probes: Exploring issues in What was it about that experience that
depth requires a high degree of precision made you decide to ...?
and clarity

Clarifying probes: Clarifies details and Can you please explain how you went
sequences about seeking this kind of service from
the pharmacy?
Content mining
Questions to draw details from all dimensions of the
subject matter.

Accessibility

Availability Affordability

Time/money;
??? ???
social cost
Pair activity
 Split into pairs
 Select one evaluation question for the case study
 Write out two interview questions for the selected evaluation
question and specify participant
o (Participants can be women, men, program officers, etc.)
 For each interview question, provide two probes

Groups A: Content mapping questions

Groups B: Content mining questions

20 minutes for group work;

20 minutes to share
Using enabling and projective
Techniques
These aid expression and help refine views:
 To pinpoint specific aspects of attitudes
 Identify differing views
 Explore boundaries
 Expose the roots of beliefs
or opinions

Technique examples:
 Vignettes
 Card sorting
 Showing written material, video, or a picture
 Mapping emergent issues
 Photovoice
Using vignettes
The guide provides participants with a story describing
different scenarios related to a person or households’
experiences with the intervention being evaluated.
 Ask questions to reveal perceptions, values,
and social norms in a community
 Allows researchers to get details that might otherwise be
challenging to ask about
 Might be used as an icebreaker or a
stand-alone technique
 Discuss example…; frame other questions
Other useful techniques

Card-sorting Written material

Stages of the topic guide
Ordering information on the tool

Instructions Order of Summary/ ending

for use questions

 How to introduce  Introduction  Reassurance of

interview;  Core part of confidentiality;
summary of study evaluation contacting for
objectives; questions follow-up, if
confidentiality,  Winding needed, etc.
privacy, etc. down
Stages of the topic guide
From general to specific
Stages of the topic guide
Combining different tools
for data collection
 Tools are designed for the specific method being employed
(focus group discussion (FGD), in-depth interview [IDI], etc.)
 When triangulating, you need tools for
each method
 Sometimes guides go with a more structured tool
o May be called different names (screener, recruitment
questionnaire, screening questionnaire)
o Collect predefined set of information about background
characteristics of study participants (sex, age, educational
background, etc.)
Tips for preparing data collection tools

 Avoid leading questions

o “Tell me how the program helped you
stop smoking.”
 Avoid double-barreled questions
o “What did you eat and what did your child eat?”

 Avoid using dichotomous questions (yes/no)

o “Did you eat breakfast today?”

 Be sure questions are clear

o “Have you discussed contraception?”

 Avoid asking “why” questions

o “Why didn’t you bring up contraception
with your husband?”
Tips for preparing data collection tools

 Instead of dichotomous lead-in questions, use

presupposition lead-in questions to create rapport and
delve deeper
o “Have you had any sexual relationships
with men?” (dichotomous)
• How to reword?

 Role-playing questions
 Pretest your tools and refine based
on experience
 Limit number of questions!
o 6–10 well written questions can easily
take 1–1.5 hours
Special considerations:
Gender
Depending on the topic, one may need to consider
indirect probes more or probe indirectly to get at
sensitive issues.
 E.g., underreporting of women’s economic activity;
men’s overreporting
of number of sexual partners or underreporting of
violence
against women
References
Burgess, R.G. (1984). In the field: An introduction to field
research. London, United Kingdom: Unwin Hyman.

Roller, M., & Lavrakas, P.J. (2015). Applied qualitative

research design: A total quality framework approach. New
York, NY, USA: Guilford Press.

Ritche, J., & Lewis, J. (2003). Qualitative research

practice: A guide for social science students and
researchers. Thousand Oaks, CA, USA: Sage Publications,
Ltd.
This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of
MEASURE Evaluation cooperative agreement
AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the
Carolina Population Center, University of North Carolina at Chapel Hill in
partnership with ICF International; John Snow, Inc.; Management Sciences
for Health; Palladium; and Tulane University. Views expressed are not
necessarily those of USAID or the United States government.

www.measureevaluation.org