Critical Appraisal

By
Dr. Hassan Bin Usman
Assistant Professor
Learning objectives
Define the term critical appraisal and discuss its
relevance to medical practice, policy and research.

To provide a brief overview of the critical appraisal

process.

Discuss the purpose of critical appraisal.

Identify research questions and study types from

published papers and abstracts
Why read critically?
Most doctors/people read journal articles by scanning the
abstracts
Abstracts rarely tell the whole story, do not contain enough
details or nuances, and are frequently biased
Abstracts are the appetizers that should get your interest up,
but it cannot be the main course (Methods, Results, and
Conclusions)
The rest of the article contains the important nuances
Beware of reading only abstracts, especially if you plan to
change your practice based on the study!
Why read critically?
Need to be able to evaluate the quality of published
studies

Understand the ideal features of a study

Identify study errors and biases

Appropriateness of data analysis methods

statistics
Appraisal

Increases the effectiveness of reading by enabling you

to exclude research studies that are too poorly
designed to inform practice.

This frees time to concentrate on a more systematic

evaluation of those studies that cross the quality
threshold and then to extract their salient points.
 Why is this necessary?

So much information is now easily available, or

forced upon us, that it is not possible to read or
study it all.

Selection is required.

Criteria for selection need to be established.

Identify Relevant Literature.

There are many ways of finding literature.

Remember to ask a librarian they are the experts.

Selectivity is the key to successful critical appraisal.

 What is critical appraisal?

Critical appraisal is a systematic approach to:

- reading,
- understanding,
- interpreting,
- identifying the limitations of, and
- deciding upon the usefulness of results of
scientific papers.
 Critical Appraisal- Definition
Can be defined as

the assessment of methodological quality

Application of rules of evidence to a study to assess the

validity of the data, completeness of reporting, methods
and procedures, conclusions, compliance with ethical
standards, etc. The rules of evidence vary with
circumstances
(Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University
Press)
 What is Critical
Appraisal?
Definition contd..
Critical appraisal is the process of
carefully and systematically analyzing the
research paper to judge its trustworthiness, and
its value and relevance in a particular context.

Critical appraisal is the process of

systematically examining research evidence to
assess its validity, results, and relevance before
using it to inform a decision (Hill and Spittlehouse,
2001, p.1).
Critically Appraise What You Read

Separating the wheat from the chaff.

Time is limited you should aim to quickly stop reading

the dross.

Others contain useful information mixed with rubbish.

Simple checklists enable the useful information to be

identified.
 How do I appraise?
Mostly common sense.

You dont have to be a statistical expert!

Checklists help you focus on the most

important aspects of the article.

Different checklists for different types of

research.

Will help you decide if research is valid and

relevant.
 Brief General
guidelines/Questions to Ask
One should keep these basic questions in
mind before reading a topic

Is it of interest?
Why was it done?
How was it done?
What has been found?
What are the implications?
What else is of interest?
General guidelines/Questions to
Ask
Is it of interest?
Title, abstract, source.

Why was it done?

Introduction.
Should end with a clear statement of the purpose of the study.
The absence of such a statement can imply that the authors
had no clear idea of what they were trying to find out.
Or they didnt find anything but wanted to publish!
General guidelines/Questions to
Ask
How was it done?
Methods.
Brief but should include enough detail to enable
one to judge quality.
Must include who was studied and how they were
recruited.
Basic demographics must be there.
An important guide to the quality of the paper.
General guidelines/Questions to
Ask
What has it found?
Results.
The data should be there not just statistics.
Are the aims in the introduction addressed in the
results?
Look for illogical sequences, bland statements of
results.
Flaws and inconsistencies.
All research has some flaws this is not nit picking,
the impact of the flaws need to be assessed.
General guidelines/Questions to
Ask
What are the implications?
Discussion.
The whole use of research is how far the results can
be generalised.
All authors will tend to think their work is more
important than the rest of us!
What is new here?
What does it mean for health care?
Is it relevant to my patients?
General guidelines/Questions to
Ask
What else is of interest?
Introduction / discussion.
Useful references?
Important or novel ideas?
Even if the results are discounted it doesnt
mean there is nothing of value.
Classification of Epidemiological
Studies
Main topics/research questions in
Public health
Although there are numerous public health research projects
conducted every year, almost all addressing one of five main
research questions:

1. How frequent is a disease, condition or risk factor? (i.e. how common is

this disease or condition in a specific population?)
2. What is the aetiology of a certain condition or disease? (i.e. are there
factors associated with a particular condition or disease?)
3. Does an intervention work? (i.e. what are the effects of an intervention?)
4. Does a test accurately classify groups with a condition or disease? (i.e.
How accurate is a sign, symptom or test in predicting a condition or
disease?)
5. What is the prognosis of a person with a certain condition or disease?
(i.e. can the risk of disability and death be predicted?)
Clinical Questions and Study
Design
There are 5 fundamental types of clinical questions:

1) therapy
2) harm
3) differential diagnosis
4) diagnosis
5) prognosis

It is important to correctly identify the category of

study, because, to answer your question, you must find
an appropriately designed study. (Guyatt, 2008)
 diferent research designs
require evidence from
Diferent clinical questions
Study Category Suggested Best Method of Investigation

Therapy RCT>cohort>case control>case series

Diagnosis prospective, blind comparison to a gold standard

Etiology/Harm RCT>cohort>case control>case series

Prognosis cohort>case control>case series

Prevention RCT>cohort>case control>case series

Clinical Exam prospective, blind comparison to a gold standard

Cost Economic Analysis

Medical Library Association. MLANET, Education, Web-based Learning. Hp. Nov, 2001.
Web-based Courses: EBM and the Medical Librarian.
Available: http://www.mlanet.org/education/web/web_courses.html 10 Apr. 2005.
Types of Studies
Anatomy of a Scientific Article

Abstract
Introduction
Materials and
Methods
Results
Discussion
Conclusion
 Critically Appraise What You Read
Critical appraisal is most easily carried out using structured check
lists.

Simple checklists enable the useful information

to be identified.

Different checklists are available like:

CONSORT checklist for trials
STROBE checklist for case control/cohort and cross sectional
studies
AGREE checklist for clinical guidelines
Checklists
Checklists for particular types of literature are a
quick and easy way of learning critical appraisal.

They all have 3 stages: (Australian checklist)

Basic questions.
Essential appraisal.
Detailed appraisal
Formal Critical Appraisal - CASP
Useful source of checklists that over many study
designs is Critical Appraisal Skills Program
(CASP)

CASP developed by Public Health Resource Unit

to bring about improvement in health and health
care
 Critical Appraisal Skills Program (CASP)
making sense of evidence

10 questions to help you make sense of qualitative research

This assessment tool has been developed for those unfamiliar with
qualitative research and its theoretical perspectives. This tool presents a
number of questions that deal very broadly with some of the principles or
assumptions that characterize qualitative research. It is not a definitive
guide and extensive further reading is recommended.

Questionnaire attached link

CASP Checklists are available on:
(http://www.casp-uk.net/#!casp-tools-checklists/c18f8 )
 Critical Appraisal
Check list
based on widely used checklist by Prof Jane Hall, Prof David
Australia, Public Health 2004
 Critical Appraisal
Check list
based on widely used checklist by Prof Jane Hall, Prof
David Australia, Public Health 2004
Structure of scientific
publications:
Title
Summary/Abstract
Key words
Main text (IMRAD)
Introduction
Methods
Results and
Discussion
Conclusions
References
Introduction
Reading a scientific article

Subject selection
Literature search
Title of the scientific article:
Helps the reader to decide whether this matches with the
subject
First impression of the articles content
What is the research question?
Identifying the Research Question (RQ) is the first and
most important part of critical appraisal

Directly related to choice of study type

Needs to be appropriate to answer the RQ
Ideal study type may not always be feasible or ethical
E.g. intervention versus natural experiment
i.e. a study assessing association between exposure
to a known harmful substance and a new disease
need to use natural experiment (or observational)
design
Stating the Research Question
Not able to judge suitability of study type if RQ not
clearly stated in literature
Ideal RQ contains the following information
Population under study
Intervention (or exposure / study factor)
Comparison group (unexposed or controls )
Outcome (or disease)

i.e. PICO
Example of PICO
Do women aged 60 years and over, who have taken
hormone replacement therapy (HRT) for a year or more,
have a higher risk of ovarian cancer than women aged
60 years and over who have never taken HRT?

Population under study

Intervention (or study factor or exposure)
Comparison group
Outcome (or disease)
Example of PICO
Do women aged 60 years and over, who have taken
hormone replacement therapy (HRT) for a year or more,
have a higher risk of ovarian cancer than women aged
60 years and over who have never taken HRT?

Population under study

Intervention (or study factor or exposure)
Comparison group
Outcome (or disease)
Critical Appraisal- Abstract

Abstract:
Same basic structure as the article
Essential points of the article in shortened form
Not a substitute for the article
Important to know if the abstract has been able to
summarize the aims, methods, results and
conclusions
Critical Appraisal- Introduction
Introduction:
To familiarize the reader with the subject matter
Current state of knowledge presented with reference to
the recent literature
The necessity of the study should be clearly laid out
Findings of the studies cited should be given in detail
with numerical results
Avoid phrases such as inconsistent findings, or
"somewhat better"
Critical Appraisal- Introduction
contd..
A good publication backs up its central statements with references
to the literature

Proceeds from general to specific

Should explain clearly what question the study is intended to answer

Choice of appropriate study design

Should include rationale and aims/objectives of the study

Critical Appraisal- Methods
Methods:
Section resembles a cookbook with procedures as recipes that
can be followed to repeat the study
Can be divided into sub-sections

Should describe:
All stages of planning
Composition of the study sample
Execution of the study
Statistical methods
Critical Appraisal- Methods
contd..
Most important element of methods is Study Design
Choice of study design should be explained and depicted in clear terms
Inclusion and exclusion criteria
Sample size calculation
Response rate (80% is good, 30% is no or only slight power)
Rate of loss to follow-up
Non-participation rate
Information on missing values
Ethical review
Written informed consent
Critical Appraisal- Methodology
contd..
Questions to think about:

Was a study protocol written before the study commenced?

Was the investigation preceded by a pilot study?

Are location and study period specified?

Is it explained how measurements were conducted?

Critical Appraisal- Methodology
contd..
Are the instruments and techniques, e.g. measuring devices, scale
of measured values, laboratory data, and time point, described in
sufficient detail?

Were the measurements made under standardized and thus

comparable conditions in all patients?

What kind of scale the variables are being measured (e.g. eye color,
nominal; tumor stage, ordinal; bodyweight, Metric)?

Was there a careful power calculation before the study started?

 Critical Appraisal- Methodology
contd..
Levels of internal validity
1. Were there enough subjects in the study?
2. Was a control group used?
3. Were the subjects randomly assigned?
4. Was a pretest used?
5. Was the study started prior to the intervention or event?
6. Was the outcome measured in an objective and reliable way?

6 x yes = very high (A)

5 x yes = high (A)
4-3 x yes = limited (B)
2 x yes = low (C)
1-0 x yes = very low (D)
Critical Appraisal- Results
Results:
Findings should be presented clearly and objectively,
i.e. without interpretation

Should first be formulated descriptively, stating

statistical parameters such as case numbers, mean
values, measures of variation, and confidence intervals.

Includes comprehensive description of the study

population
Critical Appraisal- Results contd..
Second sub-section describes relationship between
characteristics, or estimates the effect of a risk factor,
and may include calculation of appropriate statistical
models

Statistical significance in the form of p values,

comprehensive description of the data and details on
confidence intervals and effect sizes are strongly
recommended

Tables and figures with self-explanatory data

Two methods of assessing the role
of chance
P-values (Hypothesis Testing)
use statistical test to examine the null
hypothesis
associated with p values - if p<0.05 then
result is statistically significant

Confidence Intervals (Estimation)

estimates the range of values that is likely to
include the true value
 P values
P stands for probability - how likely is the result to have
occurred by chance?

P value of less than 0.05 means likelihood of results

being due to chance is less than 1 in 20 = statistically
significant.

P values and confidence intervals should be consistent

Confidence intervals provide a range of values

 Confidence intervals
(CI)
The range of values within which the true
value in the population is found

95% CI 95% confident the population lies

within those limits

Wide CI = less precise estimates of effect

 Critical Appraisal- Discussion
Discussion:
Comparison of the findings with the status quo (available
literature)

Questions to keep in mind:

How has the study added to the body of knowledge on the given
topic?

What conclusions can be drawn from the results?

Will the findings of the study lead the author to reconsider or change
his/her own professional behavior?
 Critical Appraisal- Discussion
contd..
Do the findings suggest further investigations?

Does the study raise new, unanswered questions?

What are the implications of the results for science, clinical

routine, patient care, and medical practice?

Are the findings in accord with those of the majority of earlier

studies? If not, why might that be?

Do the results appear plausible from the biological or medical

viewpoint?
Critical Appraisal- Study
Limitations
Critical analysis of the study's limitations:
Check what are the sources of bias (random or systematic)

When comparing groups should know whether there is any intergroup

difference in the composition of participants lost to follow-up

Missing values should be discussed

Results that do not attain statistical significance must also be

published.
Critical Appraisal- Conclusion
Conclusion:
The interpretations should follow logically from the results

Avoid conclusions that are supported neither by one's own data nor
by the findings of others

Study can attain objectivity only if the possibility of erroneous or

chance results is admitted

Inclusion of non significant results contributes to the credibility of

the study
Critical Appraisal- Conclusion
contd..
"Not significant" should not be confused with "no
association

Significant results should be considered from the

viewpoint of biological and medical plausibility
 Critical Appraisal- References
Reference:
Should be presented in the journal's standard style

Reference list must include all sources cited in the text,

tables and figures of the article

Should be up to date and help the reader to explore the

topic further

Do these reflect fairly and appropriately the current state of

knowledge?
Critical Appraisal- Acknowledgements and conflict of interest

Acknowledgements and conflict of interest

Must provide information on any sponsors of the study

Any potential conflicts of interest, financial or otherwise,

must be revealed in full
Critical appraisal
questionnaires
Examples
Examples of Check lists
Examples of Check lists- Standard
appraisal
1. questions
Did the study address a clearly focused issue?
2. Is the sample size justified?
3. Is the design appropriate to the stated aims?
4. Are the measurements likely to be valid and reliable?
5. Are the statistical methods described?
6. Did untoward events occur during the study?
7. Were the basic data adequately described?
8. Do the numbers add up?
9. Was the statistical significance assessed?
10. What do the findings mean?
11. Are important effects overlooked?
12. What implications does the study have for your practice?
Examples of Check lists- Appraisal of a
controlled study
1. Did the study address a clearly focused issue?
2. Were subjects randomly allocated to the experimental and control
group? If not, could this have introduced bias?
3. Are objective inclusion / exclusion criteria used?
4. Were groups comparable at the start of the study?
5. Are objective and validated measurement methods used and were they
similar in the different groups? (misclassification bias)
6. Were outcomes assessed blind? If not, could this have introduced bias?
7. Is the size of effect practically relevant?
8. Are the conclusions applicable?
Examples of Check lists- Appraisal of a cohort
/ panel study
1. Did the study address a clearly focused issue?
2. Was the cohort / panel recruited in an acceptable way? (selection
bias)
3. Was the cohort/ panel representative of a defined population?
4. Was a control group used? Should one have been used?
5. Are objective and validated measurement methods used and were
they similar in the different groups? (misclassification bias)
6. Was the follow up of cases/subjects long enough?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?
Examples of Check lists- Appraisal of a
case-control study
1. Did the study address a clearly focused issue?
2. Were the cases and controls defined precisely?
3. Was the selection of cases and controls based on external, objective and
validated criteria? (selection bias)
4. Are objective and validated measurement methods used and were they
similar in cases and controls? (misclassification bias)
5. Did the study incorporate blinding where feasible? (halo-effect)
6. Was there data-dredging?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?
Examples of Check lists- Assessment of
a survey
1.Did the study address a clearly focused issue?
2.Was the sample size justified?
3.Could the way the sample was obtained introduce
(selection)bias?
4.Is the sample representative and reliable?
5.Are the measurements (questionnaires) likely to be
valid and reliable?
6.Was the statistical significance assessed?
7.Are important effects overlooked?
 Group Activity
Read abstracts 1 to 5. For each abstract:

a) What was the research question?

b) Was the study descriptive, analytic or observational?
c) Was the study design used by the researcher the best
one to answer the research question? Why?
d) Identify the study factor and the outcome factor.
e) What did the researchers conclude from the study?
f) In your own words, briefly summarise the main results
of the study
 Abstract 1
Parkes G. Greenhalgh T. Griffin M. Dent R. Effect on smoking quit rate of telling patients their lung age:
the Step2quit randomised controlled trial. BMJ, 336(7644):598600, 2008.
Abstract
OBJECTIVE: To evaluate the impact of telling patients their estimated spirometric lung age as an
incentive to quit smoking. DESIGN: Randomised controlled trial. SETTING: Five general practices in
Hertfordshire, England. PARTICIPANTS: 561 current smokers aged over 35. INTERVENTION: All
participants were offered spirometric assessment of lung function. Participants in intervention group
received their results in terms of "lung age" (the age of the average healthy individual who would
perform similar to them on spirometry). Those in the control group received a raw figure for forced
expiratory volume at one second (FEV1). Both groups were advised to quit and offered referral to local
NHS smoking cessation services. MAIN OUTCOME MEASURES: The primary outcome measure was
verified cessation of smoking by salivary cotinine testing 12 months after recruitment. Secondary
outcomes were reported changes in daily consumption of cigarettes and identification of new
diagnoses of chronic obstructive lung disease. RESULTS: Followup was 89%. Independently verified
quit rates at 12 months in the intervention and control groups, respectively, were 13.6% and 6.4%
(difference 7.2%, P=0.005, 95% confidence interval 2.2% to 12.1%; number needed to treat 14).
People with worse spirometric lung age were no more likely to have quit than those with normal lung
age in either group. Cost per successful quitter was estimated at 280 pounds sterling (366 euros,
$556). A new diagnosis of obstructive lung disease was made in 17% in the intervention group and
14% in the control group; a total of 16% (89/561) of participants. CONCLUSION: Telling smokers their
lung age significantly improves the likelihood of them quitting smoking, but the mechanism by which
this intervention achieves its effect is unclear. TRIAL REGISTRATION: National Research Register
N0096173751
 Abstract 2
Jerrett M. Shankardass K. Berhane K. Gauderman WJ. Kunzli N. Avol E. Gilliland F. Lurmann F.
Molitor JN. Molitor JT. Thomas DC. Peters J. McConnell R. Trafficrelated air pollution and asthma
onset in children: a prospective cohort study with individual exposure measurement.
Environmental Health Perspectives 116(10):14338, 2008.
Abstract
BACKGROUND: The question of whether air pollution contributes to asthma onset remains
unresolved. OBJECTIVES: In this study, we assessed the association between asthma onset in
children and trafficrelated air pollution. METHODS: We selected a sample of 217 children from
participants in the Southern California Children's Health Study, a prospective cohort designed to
investigate associations between air pollution and respiratory health in children 1018 years of
age. Individual covariates and new asthma incidence (30 cases) were reported annually through
questionnaires during 8 years of followup. Children had nitrogen dioxide monitors placed outside
their home for 2 weeks in the summer and 2 weeks in the fallwinter season as a marker of traffic
related air pollution. We used multilevel Cox models to test the associations between asthma and
air pollution. RESULTS: In models controlling for confounders, incident asthma was positively
associated with traffic pollution, with a hazard ratio (HR) of 1.29 [95% confidence interval (CI),
1.071.56] across the average withincommunity interquartile range of 6.2 ppb in annual
residential NO2. Using the total interquartile range for all measurements of 28.9 ppb increased
the HR to 3.25 (95% CI, 1.357.85). CONCLUSIONS: In this cohort, markers of trafficrelated air
pollution were associated with the onset of asthma. The risks observed suggest that air pollution
exposure contributes to newonset asthma.
Note: for this activity, hazard ratio can be considered to be similar to relative risk.
 Abstract 3
Zucchetto A. Talamini R. Dal Maso L. Negri E. Polesel J. Ramazzotti V. Montella M.
Canzonieri V. Serraino D. La Vecchia C. Franceschi S. Reproductive, menstrual, and other
hormonerelated factors and risk of renal cell cancer. International Journal of Cancer
123(9):22136, 2008.
Abstract
A role of hormonerelated factors in renal cell cancer (RCC) etiology has been
hypothesized, but the epidemiological evidence is inconsistent. The present study aimed
at evaluating the effect of reproductive, menstrual and other genderspecific variables on
RCC risk among women. This study is part of a larger hospitalbased, casecontrol study
on RCC risk, conducted in northern, central and southern Italy. Cases were 273 women,
below age 80, with histologically confirmed, incident RCC. Controls were 546 women
hospitalized for acute, nonneoplastic conditions, frequencymatched to cases by age and
center. Odds ratios (OR) and 95% confidence intervals (CI) were computed using multiple
logistic regression models. RCC risk was inversely related to age at first birth (OR = 0.7,
95% CI 0.51.0, for 25 years vs. <25 years). Hysterectomy was found to double RCC risk
(OR = 2.3, 95% CI 1.34.2). A negative association of borderlinestatistical significance
emerged for age at menarche, whereas, no associations were found between RCC risk and
parity, menopausal status, age at menopause and use of hormone replacement therapy
or oral contraceptives. Our findings give support to a role of hysterectomy in increasing
RCC risk without corroborating, however, a major role of female hormonerelated factors.
 Abstract 4
Khameneh ZR. Sepehrvand N. Survey of hepatitis B status in
hemodialysis patients in a training hospital in Urmia, Iran. Saudi Journal of
Kidney Diseases & Transplantation. 19(3):4669, 2008.
Abstract
To evaluate the prevalence of HBV infection in chronic hemodialysis
patients at our dialysis center of Urmia's Taleqni Hospital, Urmia, Iran, we
studied crosssectionally the hepatitis surface antigen (HbsAg) status in
blood samples of 167 active chronic hemodialysis patients at our center
with enzyme linked immunosorbant assay (ELISA). The mean frequency
of HbsAg+ was 6.58%, which was higher in patients less than 50 years
old than in those above 50 years (9.3% vs. 5.3%, respectively), in males
than females (10.5% vs. 2.5%, respectively), and in those on three times
dialysis than twice per week (7.1% vs. 0.0%, respectively). We did not
find a significant relationship between the factors of: age, sex, being
resident in city or village, duration of the therapy, history of blood
transfusion, marital status, job status, history of kidney transplantation,
and prevalence of HbsAg+.
 Abstract 5
Lemstra M, Neudorf C, Opondo J. Health disparity by neighbourhood income. Canadian
Journal of Public Health Revue Canadienne de Sante Publique 97(6): 4359, 2006.
Abstract
BACKGROUND: Canadian cities are becoming more segregated by income. As such,
investigation is required into the magnitude of health disparity between low, average
and highincome neighbourhoods in order to quantify the level of health disparity at
the scale of an urban city. METHODS: A crosssectional ecological study design was
used to review all hospital discharges, physician visits, medication utilization, public
health information and vital statistics for an entire city by neighbourhood income
status. Postal code information was used to identify six existing contiguous residential
neighbourhoods in the city of Saskatoon that were defined as lowincome cutoff
neighbourhoods (N=18,228). There were two comparison groups: all other Saskatoon
residents (N=184,284) and the five most affluent neighbourhoods in Saskatoon
(N=16,683). FINDINGS: Statistically significant differences in health care utilization by
neighbourhood income status were observed for suicide attempts, mental disorders,
injuries and poisonings, diabetes, chronic obstructive pulmonary disease, coronary
heart disease, chlamydia, gonorrhea, hepatitis C, teen birth, low birthweight, infant
mortality and allcause mortality. The rate ratios increased in size when comparing
lowincome neighbourhoods to highincome neighbourhoods. No clear trend was
observed for stroke or cancer.
Questions ??
Thank You