Dr.C.

Susila,Msc (N),PhD, MBA,

Principal
Billroth College of Nursing
Chennai
Mr Smith is 64 years old and recently diagnosed with atrial
fibrillation (AF), a condition associated with a high risk of
stroke. You wish to know if prescribing warfarin will reduce his
risk of stroke?

How will you answer this?

Conduct a trial by own?

Search and appraise a relevant RCT?

Conduct a literature review?

 WE NEED SOME EVIDENCE TO PRACTICE

 Healthcare decisions should be informed by the best
available research evidence.
◦ Many guidelines rely on previously published research reviews
◦ Practice guideline quality is dependent on rigorous review methods
and high quality primary studies
 Primary studies Randomized controlled trials
 Systematic reviews Evidence based practice and \
clinical practice guidelines
 EBM (quick & dirty)  Systematic Review
Steps Steps
1. Question (PICO)? 1. Question (PICO)
2. Find the best evidence? 2. Find the best evidence x 2+
3. Appraise? 3. Appraise x 2+
4. Synthesized? 4. Synthesize
5. Apply? 5. ---

Time: 120 seconds Time: 6 months+, team

1 - 20 articles < 2,000 articles
This patient survives! This patient is dead

Find a systematic review (and appraise it quickly)!

Reviews
 Uses transparent procedures to find, evaluate and
synthesize the results of independent studies.
 Procedures are explicitly defined in advance, to assure
that the process is transparent and can be replicated.
 This process is also designed to minimize bias.
What is a systematic review?
“A systematic review is a review in which there is a comprehensive
search for relevant studies on a specific topic, and those identified are
then appraised and synthesized according to a predetermined and
explicit method.”* (*Klassen et al. Guides for reading and interpreting
systematic reviews. Arch Pediatr Adolesc Med 1998;152:700-704.)

A systematic review attempts to collate all empirical evidence that fits

pre-specified eligibility criteria in order to answer a specific research
question. It uses explicit, systematic methods that are selected with a
view to minimizing bias, thus providing more reliable findings from
which conclusions can be drawn and decisions made (Antman 1992,
Oxman 1993)
 Why we need systematic reviews?

• Minimise the impact of bias/errors

• Can help to end confusion

• Highlight where there is not sufficient evidence

• Combining findings from different studies can highlight new

findings

• Can mitigate the need for further trials

 Why we need systematic reviews?

• Health care providers, researchers and policy makers are

inundated with unmanageable amounts of information
– Over 20 million citations in PubMed
– Approx. 75 to 100 RCTs published daily
– Usually impossible to consider all relevant individual primary
research studies in a decision making context
• Enable practitioners to keep up to date with evidence
accumulating in field and to practice evidence-based medicine
• Protocols in neurology units - 80% still
recommend bed rest after LP
• Systematic review of 10 trials of bed rest after
spinal puncture
– no change in headache with bed rest
– Increase in back pain

Serpell M, BMJ 1998;316:1709–10

 A systematic review must have:
◦ Clear inclusion and exclusion criteria
◦ Explicit search strategy
◦ Systematic coding and analysis of included studies
◦ Meta-analysis (where possible)
 Why not traditional reviews?
• ‘Unscientific’ rarely pre-specify or make methods to explicit

• Rarely transparent or reproducible

• Usually qualitative, subjective, opinions of individual

• Often incomplete, filing cabinet or MEDLINE review

• Difficult to make sense across groups of studies, especially when

conflicting based on qualitative reading alone
Traditional review Systematic review

 Subjective  Objective
 Narrow question
 Broad question
 Methods clearly defined
 Methods unclear  Studies appraised
 Who undertakes systematic reviews?

• Multidisciplinary teams
– Clinicians
– Health services researchers
– Information scientists
– Statisticians
– Health Economists
– Patient and public involvement
 What are the steps in performing a systematic review?
Define research/review question
In consultation/collaboration with the clinical community,
commissioners and patient/public representatives
Develop review protocol
Pre-specify the type of studies to be included, the methods of
collating, appraising and analysing data
Identify relevant studies
Develop a comprehensive search strategy and undertake
systematic searches of the literature
Assess eligibility
Select those studies which meet the pre-defined inclusion
criteria
Data extraction /checking
Develop data extraction from into which study information and outcome
data can be extracted, checked & verified
Study assessment/appraisal
Assess the quality and validity of the included studies using the pre-
defined method.
Synthesis
Narratively and/or statistically summarise/describe the data, exploring
similarities and differences between studies.
Knowledge translation
Review details and results are disseminated to relevant target audiences
using appropriate formats
?
Delay or not delay?
• A clearly defined, focused systematic review begins
with a well formulated research question.
• Delayed surgeries are better than immediate surgeries?
• Recurrence of injury and return to former activity is
related to time of surgery?
• W question should be focused.
• How are these two questions different?
– Does homework improve student achievement?
– In high school students who are below average performers,
does daily homework lead to better performance on unit
tests, compared to no homework?
 A research question should be:
◦ Feasible
◦ Interesting
◦ Novel
◦ Ethical
◦ Relevant
 A well-established format for structuring research
questions is known by the acronym PICO.
◦ Patient or Population
◦ Intervention or Indicator
◦ Comparator or Control
◦ Outcome
 Population
Amongst adults with acute ACL injuries, does
Intervention
early reconstructive surgery compared with
Control
delayed reconstructive surgery lead to
Outcome 1
favourable return to former activity and/or
Outcome 2
risk of recurrent knee injury?
 The research question guides the author in working
through many stages of the systematic review process
◦ Defining inclusion and exclusion criteria
◦ Searching the literature
◦ Selecting studies
◦ Extracting data
◦ Analyzing and presenting results
 Develop review protocol
Pre-specify the type of studies to be included, the
methods of collating, appraising and analysing
data
• Forces you to read and understand the background
• Makes you to formulate a focused question
• Makes you to plan your information retrieval strategy
• Makes you to think through and describe
inclusion/exclusion criteria clearly
• Makes you to think about the data you want to collect
and the methods you will use to analyze them
 One of the features that distinguish a
systematic review from a traditional review is
the pre-specification of inclusion and
exclusion criteria.
 Inclusion criteria are a combination of
• Aspects of the research question
 Population
 Intervention
 Comparison
 Outcome
• Study type
 Randomized controlled trials
 Observational studies
Develop a comprehensive search strategy and undertake systematic
searches of the literature

 The goal of the literature search is to discover all studies that meet the
inclusion criteria
◦ Search comprehensively
 Terminology
 Databases
◦ Search for grey literature
 Not commercially published
◦ Search for unpublished studies
 Reduce risk of publication bias
 The search strategy should be designed to identify the
maximum number of studies relevant to the research
question.
◦ The search strategy should be systematic, transparent and
reproducible.
◦ Database specific controlled vocabulary terms and all relevant
text words should be included in the search strategy.
 The Cochrane Collaboration recommends searching the
following databases (at minimum):
◦ PubMed
◦ EMBASE
◦ Cochrane Central Register of Controlled Trials
 Interdisciplinary databases
◦ Scopus
◦ Web of Science
 Specialized databases
◦ CINAHL Plus
◦ PsycINFO
 Search cited and citing references
◦ Scopus
◦ Web of Science
 Hand search selected journals and conference
proceedings
 Conduct author searches for recent articles
written by topic experts
 The inclusion of unpublished and grey literature may
minimize the potential effects of publication bias.
◦ Publication bias
 Occurs when the outcome of an experiment or research study
influences the decision about whether—or how quickly—the
manuscript may be published
 Grey literature refers to academic, business, government
or industry print or electronic literature that is not
controlled by commercial publishers.
◦ Conference proceedings
◦ Research reports
◦ Government reports
◦ Dissertations, theses
◦ Research monographs
 Conference proceedings
◦ EMBASE
◦ Scopus
◦ Web of Science
◦ Google
 Clinical trials
◦ ClinicalTrials.gov
◦ Centerwatch.com
◦ EU Clinical Trials Register
◦ ISRCTN Registry
◦ OpenTrials
◦ WHO International Clinical Trials Registry Platform
 Compile search results using reference
management software (EndNote or
Mendeley)
 Remove duplicate records
 Document the search process
◦ Databases
◦ Dates searched
◦ Search strategies
◦ Limits (date ranges, publication types, language restrictions)

 Identification of studies meeting inclusion
criteria should be done independently by two
review authors.
 Review titles and abstracts of retrieved citations.
 Review full text of studies which are found to meet the
inclusion criteria.
 Keep a record of reasons for inclusion or exclusion.
 Tools for sorting
◦ Reference management software
 EndNote
◦ New web application
 Rayyan
◦ Excel
 Extract reported findings from selected studies using a
data extraction form.
◦ Extraction forms and approaches should be determined by the
needs of the specific review.
◦ At least two review authors should independently extract data
from study reports.
 Data Extraction

• Be clear what information you want from the studies:

– Study details
– Data for your analysis

• Information need to be collected relating to:

– Methodology
– Population
– Interventions being compared
– Outcomes evaluated
 Give consideration to….

• What effect or measures you are going to calculate

– What data do you need to do this?
• How are you planning to group studies for the analysis?
– By intervention?
– By study design?
• What information do you need to extract to enable you to
organise and analyse the way you want?
• REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW
IT!
 How much to extract??

• Level of judgement is required

– Sufficient to describe studies
– Sufficient to allow you to undertake the planned analysis
– Sufficient so you do not need to return to the full text
papers
• However
– You need to limit unnecessary detail
 Data extraction software?
• Main considerations are probably
• What are you familiar with?
• What package best suits your data?
• How many included studies do you have?
• Word
• Excel
• Access
• EPPI reviewer
• COEVIDENCE
• Consistency/Standardisation is essential
 Things to consider
• Are you including more than one study design?
• You may need separate forms for each study design
• However, you are still answering the same question, so make sure the core
information extracted is the same

• Have one or a few studies reported data differently from the others?
• Will the data still be useful?
• Should you include it?
• Make sure the core information extracted is the same

• You may need to update the form, or have more than one form
• Any changes need to be agreed and made consistently
 Stay on track……
• Be careful about collecting ‘extra’ data
• It is very tempting to collect data that are not directly relevant to the
review question
• The data needed to answer the review question should have already been
decided (REMEMBER YOUR PROTOCOL)
• Collect data for good reasons – stay focused and don’t get side-tracked

• Time and effort to collect, only to find it is not useful

 Following the full text review, assess the selected
studies for risk of bias and study quality.
Assess eligibility
Select those studies which meet the pre-defined
inclusion criteria
Quality Assessment & Critical Appraisal
• Why bother????
• What are we trying to achieve?
• Not all published and unpublished literature is
rigorous!
– being in a journal doesn’t mean it is good
• Quality may be used as an explanation for
differences in study results or to guide
interpretation of findings, strength of inferences
Quality Assessment & Critical Appraisal
• Quantitative studies
– Internal validity
– Bias: selection; performance; detection; attrition; reporting
– External validity
• Move away from checklists/numerical scores to domain based
assessment
– Cochrane Risk of Bias - RCTs
– QUADAS 2 – diagnostic accuracy
– ROBIS for systematic reviews
 Quality Assessment & Critical Appraisal
• Qualitative studies
• Three broad categories
– Rigour: has a thorough and appropriate approach been applied
to key research methods in the study?
– Credibility: are the findings well presented and meaningful?
– Relevance: how useful are the findings to you and your
organisation?
 CASP: Critical Appraisal Skills
Programme Checklists

 Critically Appraised Topics:

generic systematic reviews
(ACP Journal club)

 SIGN: Scottish Intercollegiate

Guidelines Network
 CASP appraisal checklist

1. Clear aims of research (goals, why it is

important, relevance)
2. Appropriate methodology
3. Sampling strategy
4. Data collection
5. Relationship between researcher and
participants
6. Ethical issues
7. Data analysis
8. Findings
9. Value of research (context dependent)
 Tools for assessing risk of bias
◦ ROBIS: Risk of Bias in Systematic Reviews
◦ ROBINS-I tool: Risk of Bias in Non-randomized studies
of Interventions
◦ RoB 2.0 tool (revised tool for Risk of Bias in randomized
trials
 The findings from individual studies are aggregated to
produce a type of evidence synthesis appropriate to the
type of data within the review.
◦ Narrative synthesis – findings are summarized and explained
in words
◦ Quantitative/statistical synthesis – data from individual studies
are combined statistically and then summarized
(meta‐analysis)
 Building up; putting together; making a whole
out of the parts; the combination of separate
elements of thought into a whole; reasoning
from principles to a conclusion
 Instead
of/alongside
meta-analysis
 Potential bias in
presentation
 Lack of a take
home message
 Partly informed by methodological work in qualitative
synthesis
◦ Tabulation
◦ Groupings and clusters
◦ Vote counting as a descriptive tool
◦ Examination of moderator variables (elements of e.g. setting,
population)
 Rodgers et al Evaluation 2009 15 49-72
 Tables and figures are used to present included studies
and their findings in a systematic and clear format.
◦ Flow diagram
◦ Summary of findings table
◦ Forest plot
 PRISMA Flow diagram
◦ Depicts the flow of information through the different phases of a
systematic review
◦ Documents the number of studies that remain after each stage of
the selection process
◦ Maps the number of studies identified, included and excluded,
and the reasons for being excluded
◦ PRISMA Flow Diagram Generator
 Summary of findings table
◦ Provides key information concerning the quality of evidence
◦ Depicts the magnitude of effect of the interventions
◦ Illustrates the sum of available data on all important outcomes for
a given comparison

Cochrane
Handbook
 Forest Plot
 A graphical display designed to illustrate the relative strength
of treatment effects in multiple quantitative scientific studies
addressing the same question (meta-analysis)

Names of fictional studies on left; odds ratios and confidence intervals on right; odds ratios
(squares proportional to weights used in meta-analysis); summary measure (center line of
diamond); associated confidence intervals (lateral tips of diamond); solid vertical line of no
effect
 Statement of findings, discussion and
conclusions
◦ Information on all important outcomes, including
adverse outcomes
◦ Quality of evidence for each outcome
◦ How values and preferences may bear on balance of
benefits, harms, and costs of interventions
 Decisions about whether and when to update a
systematic review
◦ The currency of the question asked
◦ The need for updating to maintain credibility
◦ The availability of new evidence
◦ Whether new research or new methods will affect the
findings

When and how to update systematic reviews, BMJ

(2016)
 Cochrane Collaboration
◦ Cochrane Library
 Cochrane protocols, systematic reviews, other reviews and trials
◦ Guides and handbooks
 Cochrane Handbook for Systematic Reviews of Interventions
 Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
 Cochrane Standards for conduct and reporting of new reviews of
interventions
 GRADE Handbook
◦ Cochrane tools for assessing risk of bias
 ROBIS: Risk of Bias in Systematic Reviews
 ROBINS-I tool: Risk of Bias in Non-randomized studies of Interventions
 RoB 2.0 tool (revised tool for Risk of Bias in randomized trials
◦ Methodological Expectations of Cochrane Intervention Reviews
 GRADE Working Group
◦ GRADE (Grading of Recommendations,
Assessment, Development and Evaluation)
 Method of grading quality of evidence and
strength of recommendations in guidelines
 GRADEpro software can be used to create
Summary of Findings tables
 PRISMA (Preferred Reporting Items for Systematic
Reviews and Meta-Analyses)
◦ PRISMA checklist
 27 items for inclusion in systemic reviews and meta-
analyses
◦ PRISMA flow diagram
 Diagram depicting flow of information and number of
records through phases of a systematic review
 Prospero
◦ International prospective register of health-related
systematic reviews
 Software
◦ Software for Systematic Reviewing
◦ Software Programs for Preparing and Maintaining
Systematic Reviews
◦ Systematic Review Toolbox
 Use a reference manger to sift and store
 Keep all citations retrieved
◦ Add in those you can’t download
 Use to de-duplicate results
 Sift citations for inclusion/exclusion
◦ Can use codes/notes
 Dates – YYYYMMDD
 Version numbering
◦ v0.1 = first draft
◦ v1.0 = final version
◦ v1.1 = minor amendments to final version
◦ v2.0 = major revision

 Avoid using draft, draft 1, final final, etc.

 Clear naming convention
◦ E.g. Date_project_title
 20/11_Autism HTA_resultsv1.2.doc
 FMS Systematic review group
◦ Informal monthly session where methods are
discussed and issues can be raised
◦ fiona.beyer@ncl.ac.uk or jenni.hislop@ncl.ac.uk
◦ alternatively you have my contact details

 MSc in Public Health and Health Services

Research (~October 2016)
◦ 10 credit module ‘Introduction to systematic reviewing and
critical appraisal’
◦ pghealth@newcastle.ac.uk
• How do you capture qualitative distinctions between studies?
• Primary studies are reported poorly
• “Apples and oranges”
• Nearly all reviews include “flawed” studies to some extent
(e.g., a randomized design with attrition)
• Publication bias is always an issue
• Analysis of between- study differences is correlational or
“interocular”
Abrami, P.C., Wade, A. & Bernard, R.M. (2005). Evidenced-based Practice: Bridging the gap between educational
Cooper, H., Nye, B., Charlton, K., Lindsay, J.,& Greathouse, S. (1996). The effects of summer
vacation on achievement test scores: A narrative and meta-analytic review. Review of
Educational Research, 66, 227-268
Cooper, H., Robinson, J.C. & Patall, E.A. (2006). Does Homework Improve Academic
Achievement?: A Synthesis of Research, 1987-2003. Review of Educational Research.
Ennett, ST, Tobler, NS, Ringwalt, CL & Flewelling, RL(1994).How effective is drug abuse
resistance education? A meta-analysis of Project DARE outcome evaluations. American
Journal of Public Health, 84,1394-401.
Hanushek, E.A. 1989. "The Impact of Differential Expenditures on School Performance."
Educational Researcher 18(4): 45-65.
Hedges,L.V., R.D. Laine, and R. Greenwald. 1994. "Does Money Matter? A Meta-Analysis of
Studies of the Effects of Differential School Inputs on Student Outcomes." Educational
Researcher 23(3): 5-14
Glass, Gene V., and Mary Lee Smith. (1979) "Meta-Analysis of Research on Class Size and
Achievement." Educational Evaluation and Policy 1, 2-16.
Paterson, B. L., Thorne, S. E., Canam, C., Jillings, C. (2001). Meta-Study of Qualitative Health
Research. Thousand Oakes, CA: Sage
Shadish, W.R., Matt, G.E., Navarro, A.M., & Phillips, G. (2000). The effects of psychological
therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin,
126, 512-529.

Smith, Mary Lee, and Gene V. Glass. (1980) "Meta-Analysis of Research on Class Size and
Its Relationship to Attitudes and Instruction.“American Education Research Journal,
17,419-433
Whitehead, MF & Becker, BJ (2000). The Young Child’s Post-Divorce Adjustment: A Meta-
Analysis with implications for parenting arrangements. Journal of Family Therapy, 14, 5-26.
Thank you