Mixed Methods Research: State

of the Art
(What Has Developed In Mixed
Methods)
by John W. Creswell, Ph.D.
Department of Educational Psychology,
University of Nebraska-Lincoln
Co-editor, Journal of Mixed Methods Research, and
Co-Director, Office of Qualitative and Mixed Methods
Research
Please do not duplicate or use these slides without the express
permission of the author.

The Story of Mixed Methods in the Last 20

Years

Highlight four developments

Why I can tell the story

First generation mixed methods reflect on my role in story

Balanced background in quan and qual
5 books on research methods
Applied research methodologist
Focused on education, social and human sciences
Journal editor

What I am trying not to do

Fix the history

Discourage discourse
Speak for all disciplines
Limit the discussion to a US perspective
Cover all developments

Four Developments (last 20 years)

Increasing interest in and advocacy for mixed methods

Evolving understanding of what is mixed methods research
Emerging philosophical perspectives as a foundation for mixed
methods
Developing designs and innovative techniques

Development #1: Increasing Interest in and

Advocacy for Mixed Methods
Key Developments

Research movement
Federal funding interest
Discipline interest
International interest
Books
Journals
Conferences
Critics

Mixed Methods as
a Movement

The emergence of mixed

methods as a third
methodological movement
in the social and behavioral
sciences began during the
1980s. (p. 697)
-Tashakkori &
Teddlie (2003)

New
NIH Awards Using
Mixed Methods

New NIH Awards

Using the Term "Mixed Methods"

Number of New Awards

45
40
35
30
25
20
15
10
5
0
1998

2000

2002

2004

2006

2008

Year

Development of NIH guidelines (1999)

Combining qualitative and quantitative
methods has gained broad appeal in
public health research. (1999)
- Mentioned several

approaches for combining

qualitative and
quantitative research
- Advanced considerations for deciding
what model to use (literature
available, prior studies,
realistic design, expertise)
- Suggested to describe each
method thoroughly

NIH Summer Institute:

The Design and Conduct of Qualitative and Mixed-Method Research in Social Work and Other Health
Professions
August 4 -8, 2004
Table of Contents
Course Objectives
Dates and Location
Background
Concept
Institute Co-Chairs
Faculty/Presenters
Speaker Presentations
Eligibility Requirements
Number of Participants
Costs
Application Process and Receipt Date
Additional Information

Social Work Summer Institute

on Mixed-Methods
Research

Course Objectives
To provide a thorough grounding in the design and conduct of qualitative and mixed method research to social work researchers and
other health professionals interested in developing competence in the planning, design, and execution of these techniques and
become more successful in the NIH extramural research program.

National Science Foundation Workshop,

2003, Includes Mixed Methods Research

-Provided general
guidelines for devel
qualitative research
projects
-Several papers
addressed mixed
methods research

Books on
Mixed Methods
Research

14+ books on mixed methods

research
have been written
since 1988

10

Two New Books From

Sage Publications

11

A New
Journal

Journals
Devoted to
Mixed
Methods:
Fieldwork
Quality and
Quantity
Mixed
Method
Approaches

Editors: John W. Creswell

and Abbas Tashakkori
Managing Editor: Vicki L.
Plano Clark
Email: vpc@unlserve.unl.edu

12

Conferences
Mixed

Methods Conference, Cambridge,

England, July 21-24, 2008
Proposed mixed methods conference,
Sydney, Australia, 2009
Discipline conferences with mixed methods
papers

13

Critics (Creswell, 2007)

Is there a post-positivist leaning to mixed methods?

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10, 42-61.
Giddings, L. S. (2006). Mixed-methods research: Positivism dressed in drag?
Journal of Research in Nursing, 11(3), 195-2003.
Holmes, C. A. (2007). Mixed(up) methods, methodology and interpretive
frameworks. Contributed paper for the Mixed Methods Conference, Cambridge,
University, July 10-12, 2006.

What are the limits for mixed methods research?

Sale, J. E. M., Lohfeld, L. H., Brazil, K. (2002). Revisiting the quantitativequalitative debate: Implications for mixed-methods research. Quality and
Quantity, 36, 43-53.
Leahey, E. (2007). Convergence and confidentiality? Limits to the
implementation of mixed methodology. Social Science Research, 36, 149-158.

Is there a dominant discourse in mixed methods research?

Freshwater, D. (2007). Reading mixed methods research: Contexts for criticism.

Journal of Mixed Methods Research, 1(2), 134-146.

14

Development #2: Evolving Understanding

of What is Mixed Methods Research
Key Developments

The emergence of diverse perspectives - Two strands to

mixing to methodology to mixed methods within designs
Incorporation of perspectives into current definitions
Questions arise about what is mixing?
Recent assessment of the value of mixed methods research (or
mixing)

15

Framework for Viewing Perspectives on

Mixed Methods

Quantitative Data

Qualitative Data

Mixed Methods

Method
Paradigm
Perspective

Methodology
Research Design
Procedures

16

Mixed methods is
a Method, a Design, a Methodology
Identifying a research problem
Reviewing the literature
Identifying a purpose
and stating questions
Collecting data
Analyzing and interpreting data
Reporting and evaluating the study
17

Mixed Methods within Designs

Ethnography

Qual ----- Quan

Case Study Research

Qual----- Quan

Narrative research

Qual-----Quan

Experimental
research

Qual---- Quan

18

What is this Method Called?

Multi-method
Triangulation
Integrated
Combined
Quantitative and qualitative methods
Multi-methodology
Mixed methodology
Mixed-method
Mixed research
Mixed methods
19

A Definition
Mixed methods research is a methodology for conducting research
that involves collecting, analyzing, and integrating (or mixing)
quantitative and qualitative research (and data) in a single study or a
longitudinal program of inquiry. The purpose of this form of research is
that both qualitative and quantitative research, in combination, provide
a better understanding of a research problem or issue than either
research approach alone.
Aspects of this definition:
Collecting both quantitative and qualitative data
Mixing the data (from merging in which both quan and qual
loses identity to keeping them distinct but connected)
Single vs multiple-phase project
Raises issues: how, what, where, why, value (of mixing)

20

Mixing (How)
Converge
data:
Qual

Results

Quan

Connect data:
Qual

Quan

Results

Embed the data:

Quan data
Qual data

21

Mixing
What

Data (Methods)
Qualitative and quantitative research
Philosophical assumptions

Where

Throughout the process of research

Data collection
Data analysis
Interpretation
22

Mixing Why? (Bryman, Qualitative

Research, 2006)

Validity to corroborate quan and qual data

Offset offset weaknesses of quan and qual and draw on strengths
Completeness more comprehensive account that qual or quan alone
Process quan provides outcomes; qual, the processes
Different question quan and qual answer different questions
Explanation qual can explain quan results or vice-versa
Unexpected results surprising results from one, other explains
Instr development qual employed to design instr, then instr tested
Sampling one approach facilitates sampling from other approach
Credibility both approaches enhance integrity of findings
Context qual provides context; quan provides general.
Illustration qual data helps develop depth for quan data
Utility more useful to practitioners
Confirm quan tests qual generated hypotheses
Diversity of views relationship and meaning; researcher/participant views
Enhancement augmenting or building on one form of data with the other

23

Value of Mixing

What does mixed methods research provide that either quantitative

or qualitative research alone does not provide?
Indicators of yield (OCathain, Murphy, & Nicholl, JMMR, 2007)

Integration as a measure of yield

Publications as a measure of yield

My speculations:

Results are useful

Stronger impact on policymakers
Cost-effective
Better solutions through interdisciplinary input
More evidence for a study

24

Development #3: Emerging Philosophical

Perspectives as a Foundation for Mixed
Methods Research
Key developments:
There are many stances taken by researchers.
Purist stance - Paradigms are incompatible (fixed, unmoveable),
paradigms relate to methods, therefore mixing is impossible.
Dialectic stance - Multiple paradigms are possible and can be
used, but honor and make explicit the paradigms.
Single paradigm stance - There is a single paradigm for mixed
methods research: pragmatism, transformative.
Design stance - Multiple paradigms can be used relate them
to your design.
Community stance source of belief systems come out of
shared meanings and understanding

25

Philosophical Assumptions Behind Mixed

Methods Research
Worldview or philosophy (e.g., attitudes and beliefs about knowledge, such
as constructivism, post-positivism)

Theoretical lens (e.g., feminist, racial)

Methodological approach (e.g.,

mixed methods)

experimental, survey, ethnography,

Methods of data collection (e.g.

interviews, focus groups)

Adapted from Crotty M. (1998).

26

Four Worldviews
Postpositivism
Determination
Reductionism
Empirical observation and
measurement
Theory verification

Constructivism
Understanding
Multiple participant meanings
Social and historical
construction
Theory generation

Advocacy/Participatory
Political
Empowerment issue-oriented
Collaborative
Change-oriented

Pragmatism
Consequences of actions
Problem-centered
Pluralistic
Real-world practice oriented

27

Purist Stance
Smith &
Heshusius,
(1986)

Incompatibility thesis
The contention of this
paper is that the claim of
compatibility, let alone one
of synthesis, cannot be
sustained. (p. 4)

28

Dialectic Stance Greene (2007)

Paradigms philosophical assumptions,

knowledge, methodology, values
Paradigms are different but are
social constructions not sacrosanct
Differences should be respected,
intentionally used together
Used together, tensions achieve
dialectical discovery of enhanced,
reframed, or new understandings.

29

Single Paradigm Stance

Pragmatism (Tashakkori & Teddlie, 2003)

13 writers endorse
Based on Dewey, Rorty, Pierce, others (Americans)
Focus on research question
What works, consequences
Multiple methods large toolkit
Transformative (Mertens, 2003)
Transform lives of underrepresented, marginalized groups
Incorporate this focus into all phases of research

30

Design Stance (Creswell & Plano Clark,

2007)

Multiple paradigms can be used

Different paradigms can provide the foundation for different
phases of research
Often post-positivist paradigm provides the foundation for
quantitative aspects of research
Often constructivist/advocacy paradigm influences
qualitative aspects of research
Often pragmatism paradigm influences both qualitative and
quantitative aspects of research
This does not suggest that certain paradigms
are linked to certain methods (paradigm-method fit)

31

Community Stance (Morgan, 2007)

Espouses the pragmatic approach

Shifts conversation to the importance of
shared meanings and joint actions between
researchers.
To what extent are two people (or two research
fields) satisfied that they understand each other,
and to what extent can they demonstrate the
success of shared meaning by working together
on common projects. (Morgan, 2007, p. 67)
32

Development #4: Developing Designs and

Innovative Techniques
Key Developments

Evaluation designs
Notation
Typologies, and a parsimonious set
Diagrams of procedures
Complex evaluation models
Reconceptualizing designs
The practice of research
Emergence of innovations in procedures
Unusual blends
Methodological issues
Data analysis techniques
Presentation techniques

33

Early Designs
(Greene, Caracelli,
& Graham, 1989)
Triangulation
(convergence,
corroboration)
Complementarity
(clarify results from
one method to the
other)
Development
(one method informs
the other method)
Initiation (paradox,
contradiction)
Expansion (extends
breadth of inquiry)

34

Preliminary Design Considerations (Morse,

1991)
Approach

Type

Purpose

Limitations

Resolutions

QUAL + quan

Simultaneous

Enrich description
of sample

Qualitative
sample

Utilize normative
data for
comparison of
results

QUAL

Sequential

Test emerging H,
determine
distribution of
phenomenon in
population

Qualitative
sample

Draw adequate
random sample
from same
population

QUAN + qual

Simultaneous

To describe part
of phenomena
that cannot be
quantified

Quantitative
sample

Select appropriate
theoretical sample
from random
sample

QUAN

Sequential

To examine
unexpected
results

Quantitative
sample

Select appropriate
theoretical sample
from random
sample

quan

qual

35

Parsimonious Designs (Creswell & Plano Clark, 2007)

Concurrent Mixed Methods Designs
Triangulation Design

QUAN
QUAN
Data
Data&&
Results
Results

Interpretation
Interpretation

QUAL
QUAL
Data
Data&&
Results
Results

Embedded Design
QUAN
QUAN
Pre-test
Pre-test
Data
Data&&
Results
Results

Intervention

qual
qual
Process
Process

QUAN
QUAN
Post-test
Post-test
Data
Data&&
Results
Results

Interpretation
Interpretation

36

Sequential Designs Mixed Methods Designs

Explanatory Design
QUAN
QUAN
Data
Data&&
Results
Results

Following up

qual
qual
Data
Data&&
Results
Results

Building to

quan
quan
Data
Data&&
Results
Results

Interpretation
Interpretation

Exploratory Design
QUAL
QUAL
Data
Data&&
Results
Results

Interpretation
Interpretation

Sequential Embedded Design

BeforeBeforeintervention
intervention

qual
qual

QUAN
QUAN

Intervention
Intervention
Trial
Trial

AfterAfterintervention
intervention

qual
qual

Interpretation
Interpretation

37

Helpful Tips for Drawing this

Visual of the Design:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Give a title to the visual model.

Choose either horizontal or vertical layout for the model.
Draw boxes for quantitative and qualitative stages of data
collection, data analysis and interpretation of the study results.
Use capitalized (QUAN) or small letters (quan) to designate priority
of quantitative and qualitative data collection and analysis.
Use single-headed arrows to show the flow of procedures in the
design.
Specify procedures for each quantitative and qualitative data
collection and analysis stage.
Specify expected products or outcomes of each quantitative and
qualitative data collection and analysis procedure.
Use concise language.
Make your visual diagram simple.
Size your visual diagram to one page.

Source: Ivankova, N. et al. (2006)

38

Example of
A Diagram
39

Complex Mixed Methods

Evaluation Designs
(Nastasi, Hitchcock,
Sarkar, Burkholder,
Varjas & Jayasena, 2007)
Study of mental health
interventions for youth
in Sri Lanka

Concurrent,
Sequential,
Recursive
Qual/Quan
Phases
(Nastasi et al.,
2007)

Reconceptualizing Designs

Three ways to look at mixed methods designs:

Typological approach (Bryman, 1988; Creswell & Plano Clark, 2007)

Systems theory (Maxwell & Loomis, 2003)

Synergistic approach - Hall & Howard (in press), A Synergistic

Approach: Conducting Mixed Methods Research with Typological and
Systemic Design Considerations

Combined effect greater than separate parts

Position of equal value of qual and quan
Ideology of difference protect paradigm differences
Researcher as collaborator and relationship with design skilled in
both quan and qual or part of interdisciplinary team

42

The Practice of Conducting A Mixed

Methods Study

Cordon & Hirst (in press) Implementing a mixed methods approach

to explore the financial implications of death of a life partner
Social Policy Researchers, UK

Why they chose mixed methods

Reasons for their choice of design
Sampling decisions
Concept map
Discussions among team members

43

Technique - Designing a Quantitative

Survey Based on Qualitative Findings
QUAL data
analysis
Quotes
Codes
Themes

Quan data analysis

instrument development
Items on a survey
Variables on a survey
Scales on a survey

44

Technique- Matrix with Quantitative and Qualitative Data

Count
Adj.Count**
Row Pct
Column Pct

Patients
N=2

Physicians
N=4

Medical Assistants
N=4

Familiarity
With the
Form

13
6.5
41.9
5.8

17
4.25
27.4
3.0

19
4.75
30.7
5.3

49
15.5
100.0

Reactions to
the Form

23
11.5
22.2
10.2

100
25.0
48.3
17.4

61
15.3
29.5
16.9

184
51.75
100.0

Use for
Managing
Depression

67
33.5
38.6
29.8

177
44.25
51.0
30.7

36
9.0
10.4
10.0

280
86.75
100.0

Changes to
the Form

115
57.5
37.5
51.1

196
49.0
32.0
34.0

187
46.8
30.5
51.7

498
153.3
100.0

Situational
Use of the
Form

7
3.5
8.9
3.1

86
21.5
54.4
14.9

58
14.5
36.7
16.1

151
39.5
100.0

225
112.5
100.0

576
144.0
100.0

361
90.3
100.0

Themes

45

Technique Raising and Addressing

Methodological Issues in Designs

Creswell, Plano Clark, Garrett (in press)

Contradictory findings
Data merging
Sample selection
Sample size
Introducing bias
Time

46

Technique Unusual Blends of Data

Collection

Qualitative

Quantitative

Post-modern philosophy
and
Longitudinal data and
Qualitative data and
Qualitative themes
matched

Quantitative data

Qualitative discourse
data

Longitudinal data
Secondary survey data
With items on survey to
produce new quan
variables
Survey data

47

Other Developments:
International

emphases
Discipline differences
Research team practices
The emergence of bilingual nomenclature
(will employ this thinking in the workshop this
afternoon as we title projects)

48

Four Developments (last 20 years)

Increasing interest in and advocacy for mixed methods

Evolving understanding of what is mixed methods research
Emerging philosophical perspectives as a foundation for mixed
methods
Developing designs and innovative techniques

49

References
Bryman, A. (1988). Quantity and quality in social research. London: Routledge.
Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research
6(1), 97-113.
Cordon, A., & Hirst, M. (in press). Implementing a mixed methods approach to explore the financial
implications of death of a life partner. Journal of Mixed Methods Research.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand
Oaks, CA: Sage.
Creswell, J. W., Plano Clark, V. L., & Garrett, A. L. (in press). Methodological issues in conducting mixed
methods research designs. In M. Bergman (Ed.), Advances in mixed methods research. London: Sage.
Crotty M. (1988) The foundations of social research: Meaning and perspective in the research process.
London: Sage.
Freshwater, D. (2007). Reading mixed methods research: Contexts for criticism. Journal of Mixed Methods
Research, 1(2), 134-146.
Giddings, L. S. (2006). Mixed-methods research: Positivism dressed in drag? Journal of Research in Nursing,
11(3), 195-2003.
Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco: Jossey-Bass.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method
evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255274.
Hall, B., & Howard, K. (in press). A synergistic approach: Conducting mixed methods research with
typological and systemic design considerations. Journal of Mixed Methods Research.
Holmes, C. A. (2007). Mixed(up) methods, methodology and interpretive frameworks. Contributed paper for
the Mixed Methods Conference, Cambridge, University, July 10-12, 2006.
Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10, 42-61.
Ivankova, N. V., Creswell, J. W., & Stick, S. (2006). Using mixed methods sequential explanatory design:
From theory to practice. Field Methods, 18(1), 320.
Leahey, E. (2007). Convergence and confidentiality? Limits to the implementation of mixed methodology.
Social Science Research, 36, 149-158
Maxwell, J., & Loomis, D. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C.
Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 241-271). Thousand Oaks,
CA: Sage.

References (Contd)
Mertens, D. M. (2003). Mixed methods and the politics of human research: The transformative-emancipatory
perspective. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research
(pp. 135164). Thousand Oaks, CA: Sage.
Morgan, D. L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining
qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76.
Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40,
120123.
Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, K., & Jayasena, A. (2007). Mixed methods in
intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164-182.
National Institutes of Health (2004). NIH Summer Institute: The design and conduct of qualitative and mixedmethod research in social work and other health professions, August 4-8, 2004.
National Institutes of Health. Office of Behavioral and Social Sciences Research (1999). Qualitative methods in
health research: Opportunities and considerations in application and review. Washington D.C.: NIH.
National Science Foundation (2003). Workshop on scientific foundations of qualitative research. Washington
D.C.: NSF.
OCathain, A., Murphy, E., & Nicholl, J. (2007). Integration and publications as indicators of yield from mixed
methods studies. Journal of Mixed Methods Research 1(2), 147-153.
Plano Clark, V. L., & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage.
Sale, J. E. M., Lohfeld, L. H., Brazil, K. (2002). Revisiting the quantitative-qualitative debate: Implications for
mixed-methods research. Quality and Quantity, 36, 43-53.
Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The end of the quantitative-qualitative
debate among educational inquirers. Educational Researcher, 15(1), 4-12.
Tashakkori, A., & Teddlie, C. (Eds.). (2003a). Handbook of mixed methods in social and behavioral research.
Thousand Oaks, CA: Sage.

Mixed Methods Research: State

of the Art
(What Has Developed In Mixed
Methods)
by John W. Creswell, Ph.D.
Department of Educational Psychology,
University of Nebraska-Lincoln
Co-editor, Journal of Mixed Methods Research, and
Co-Director, Office of Qualitative and Mixed Methods
Research
Please do not duplicate or use these slides without the express
permission of the author.

52