The Qualitative

Assessment and Review

Instrument (QARI)

PART 3:
META-AGGREGATION
Its history
Its philosophy
A worked example

Meta-aggregation
A structured, and process driven approach to
systematic review drawing on the classical
understandings and methods associated with
systematic review of quantitative literature as
practiced by the Cochrane and Campbell
Collaboration
Based on an a-priori protocol
Established, answerable question
Explicit criteria for inclusion
Documented review methods for searching, appraisal,
extraction and synthesis of data

Meta-aggregation: History
Participative consensus project involving
Australian academics
The outcome = an aggregative approach
to the synthesis of qualitative evidence
that:
emphasised the complexity of interpretive and
critical understandings of phenomena;
recognised the need to ensure that the process
is practical and usable;
balanced utility of the outcomes with the
complexity of the material;
was grounded in pragmatism.

Meta-aggregation: A pragmatic
approach
The philosophy of pragmatism - Charles Peirce, William James,
John Dewey
the value of any thought lies in its practical use and
consequences. It is the focus on practical consequences that
characterizes meta-aggregation as a synthesis approach
Central to pragmatism were communities of inquiry or groups of
people with a shared interest, problem or issue trying to resolve doubt
through critical reasoning.

Research articles
Actions are assessed in the light of their practical consequences.
Aggregation concentrates on the original researchers findings
(processed data) and summarizes common and competing findings to
produce cross-study generalizations that lead to recommendations
for action The utility of an outcome is important.

Meta-aggregation: a worked
example
Review Topic:
Obstacles to the implementation of Evidence-Based
Practice in Belgium: a context-specific study
Question formulation:
Setting: Belgium (Belgian health care system)
Population: Health Care Practitioners
(Phenomena of) Interest: Obstacles toward the
implementation of Evidence-Based Practice
Comparison: (additional literature search on strategies
developed by countries with other systems)
Evaluation: Experiences and perceptions

Meta-aggregation: a worked
example
Search strategy:
Major databases: Medline, CINAHL, Psychinfo, Embase, Social Sciences
abstracts and ERIC (1990-May 2008).
Where possible we used a methodological filter for qualitative
research.
Other information sources:
Federal research actions-database from the Belgian governmental department
of science
Consultation of Belgian experts in qualitative research methods and/or
Evidence-Based Practice to check on any other published material that could be
of use for the synthesis.
Screening of references

Keywords:
Evidence-Based Practice (Medical Subject Heading (MeSH) term), including
Evidence-Based Dentistry, Nursing and Medicine.
Additional keywords: evidence-based combined with the geographical notion
Belgian, Flemish/Flanders or Walloon.

Meta-aggregation: a worked
example
Inclusion criteria:
Study type: Qualitative, empirical research papers. Opinion
pieces and descriptive articles were excluded.
Study participants: All types of health care practitioners
e.g. physicians, dentists, nurses, physiotherapists,
psychologists
Topic of interest: Obstacles to the implementation of EBP.
Context: the Belgian health care system.
Outcome of interest: Experiences and/or perceptions from
participants.
8 studies met the inclusion criteria

Meta-aggregation: a worked
example

Critical appraisal of selected studies:

No criteria based quality appraisal, although the
software QARI provides a standard critical appraisal
checklist.
limited amount of studies found
the majority of the selected studies were written by the lead
reviewer, who would also be one of the appraisers.

An overall judgment approach was used instead. This

approach has been proven to deliver the same outcome
(Dixon-Woods, 2007). However, it tends to be less
explicit about potential reasons for exclusion.
An evaluation based on the JBI-critical appraisal
instrument is recommended for other review teams
opting for a meta-aggregative approach to synthesis.

Meta-aggregation: a worked
example

85 different findings were found

The findings were classified in 9 major categories:

(1) evidence is hard to implement

(2) decision making processes are influenced by practitioner variables
(3) decision-making processes are influenced by practitioner variables
(4) commercial/financial interests affect EBP
(5) governmental regulations influence the process of implementation,
(6) EBP is more feasible for practitioners working in a scientific setting
compared to those working in other settings
(7) physicians display of power hampers other professionals to put
evidence-based practice into effect
(8) a lack of knowledge and skills hinders the implementation of
evidence-based practice
(9) attitudes of health care practitioners hinder the implementation of
EBP.

Meta-aggregation: a worked
example
These categories were further analysed to
produce 4 synthesized statements
Synthesis 1: Evidence might have a limited role in decision-making
processes in daily practice, if the importance of the scientific
component is not stressed (categories 1, 2 and 3).
Synthesis 2: Aspects other than quality of care will steer the EBP
agenda, if governmental regulations and economic interests are contraproductive for delivering the best possible care (categories 4 and 5).
Synthesis 3: Although EBP is intended to serve all practitioners, some
health care providers will benefit less from EBP than others, if inequity
issues between practitioners are not solved and support for field
workers is not established (categories 6 and 7).
Synthesis 4: A lack of competences will hinder the implementation of
EBP, if gaps in knowledge and skills are not being filled and efforts to
change contra-productive attitudes are not undertaken (categories 8
and 9).

Meta-aggregation: a worked
example

Meta-aggregation: a worked
example

Implications for practice and policy

Consider educating patients and exploring potential information

channels that influence patients opinions to create space for
well-informed decisions (synthesis 1).
Provide easy and free access to well-structured, compact and
relevant information targeted to a particular discipline and
consider helpdesks. Screen information, control its quality and
translate it to the field (synthesis 1).
Consider updating the Belgian nomenclature and reimbursement
system to bring it in line with the latest evidence (synthesis 2).
Consider incentives for those practitioners who are engaging
themselves for the implementation of EBP in daily practice, to
keep them motivated and to prevent from creating a negative
spiral in which practitioners tend to believe that efforts to
improve practice automatically lead to a loss of income
(synthesis 2).

Meta-aggregation: a worked
example
Implications for practice and policy
Enhance communication and cooperation between
physicians and other partners in health care, via their
professional groups and journals (synthesis 3)
Consider programs tailored to the needs of specific
disciplines and stimulate multidisciplinary education
should be stimulated to create mutual understanding
across disciplines (synthesis 3, 4).
Consider direct access to allied health services to
increase autonomy (synthesis 3)
Integrate EBP in the basic curricula (synthesis 4).

The Qualitative
Assessment and Review
Instrument (QARI)
Designed to provide a
systematic process mirroring
that taken for systematic
reviews of quantitative
research, whilst being
sensitive to the nature of
qualitative data.

Starting a synthesis project

(or review of qualitative evidence)

Critical Appraisal of Evidence

arising out of
qualitative research

Data Extraction
Reduce
The findings of many studies into a single
document
Summarise
Methods
Phenomena
Findings

The units of extraction in this process are specific

findings and illustrations from the text that
demonstrate the origins of the findings.
In QARI a finding is defined as: Aconclusion reached
by the researcher(s) and often presented as themes
or metaphors.

Levels of credibility
Unequivocal - relates to evidence beyond reasonable doubt which may
include findings that are matter of fact, directly reported/observed and not
open to challenge
Credible - those that are, albeit interpretations, plausible in light of data and
theoretical framework. They can be logically inferred from the data. Because
the findings are interpretive they can be challenged.
Not Supported - when 1 nor 2 apply and when most notably findings are
not supported by the data

Meta-Synthesis
The aim of meta-synthesis is to assemble findings; categorise
these findings into groups on the basis of similarity in meaning;
and to aggregate these to generate a set of statements that
adequately represent that aggregation. These statements are
referred to as synthesised findings - and they can be used as a
basis for evidence based practice.

In QARI, a synthesised finding is defined as anoverarching description of

a group of categorised findings that allow for the generation of
recommendations for practice.

Can be stated propositionally as if-then statements - for

example: If students are advised reconsider their choice of
study , their relatives will sometimes feel as if they are not
involved. (a somewhat awkward and eccentric form)
We prefer the declamatory form that emphasizes the probability
of the claim: Relatives of students that are advised to
reconsider their choice of study may feel as if they are not
involved if strategies to include them are not pursued.