EVIDENCE BASE MEDICINE

THERAPY & HARM

J.EKO WAHONO. R.dr.SpS.M.Kes
SMF I.P.Saraf
RSUD Dr.Soetomo
Surabaya

1
 Principal of critical appraisal-
primary research
• Involves 3 overall question

– What is the PICO of the study, and is it closed

enough to your PICO ?
– How well was the study done ?
– What do the results mean and could they
have been due to chance ?

2
 Critical appraisal Tx
Your question
(PICO)

Is the study question Study What do the

the same
result mean ?
as your question ?

How well was

The study done ?

3
Wheter wearing elastic stockings on long-haul
flights helps to prevent deep vein trombosis
(DVT)

• Population/problem = passengers on long-

haul flights

• Intervention = wearing elastic

compression stockings

• Comparator/control = no elastic stockings

• Outcome = symptomless DVT

4
 Critical appraisal Tx DVT
• Clinical question :
– In passengers on long-haul flights, does
wearing elastic compression stockings,
compared to not wearing elastic stockings,
prevent DVT ?
• Search terms :
– Based on the clinical question (PICO) 
– (flight*OR travel*) AND stockings*AND (DVT
OR trombosis)
5
 V.I.A
V : Valid

I : Important

A : Applicable

ebm tx 10 6
VALID …?

ebm tx 10 7
 REPORTS OF INDIVIDUAL STUDIES
ARE THE RESULTS OF THIS INDIVIDUAL STUDY VALID?

ebm tx 10 8
 Critical appraisal Tx DVT
• Search result :
– PubMed Clinical Queries therapy,broad), 20
hits(referring to 5 studies and several reviews,
including 1 recent Cochrane review)
– For exercise :
• Scurr et al (2001). Frequency and prevention of symptomless DVT
in long-haul flights:a randomised trial
• The Lancet 357:1485-1489
• Authors’conclusion :
– Wearing of elastic compression stockings during long-haul air
travel is associated with a reduction in symptomless DVT

9
 Analyse
• How do we know that the results are valid
and real ?
• Wearing elastic stockings is an
intervention --- RCT  ok
• In the real life 
– straight to the Cochrane systematic review

10
 Question 1 : Is the PICO of the study
close enough to your PICO ?
• If you find a study that will answer your
clinical question 
– Study PICO match your PICO or not !
• Example your PICO
–P = In patients with rheumatoid arthritis
–I = Does taking anti-inflammatory drugs
–C = no treatment or simple analgetics
–O = Increase or reduce fatique

11
 If study PIC = your PIC
but study O # your O

• Study O = reduce joint pain

• Your O = fatique
• However :
– The study may report on some measure of
fatique as secondary outcome
– Please decide quickly wheter to critize or not

12
 Question 2 :
How well was the study done ?
• The quality of an epidemiological study
– Internal validity
• Free from bias & confounding factors
– Bias
• The degree to which the result is skewed away
from the truth
– Selection bias
– Treament bias
– Measurement bias
• To overcome bias  RCT & Blinding
13
 Question 2 :
How well was the study done ?
• The quality of an epidemiological study
– Internal validity
• Free from bias & confounding factors
– Confounding factors
• Patients features & causal factors
• To overcome CF 
– Both Group are closely matched/similar
– The management of the group is the same

14
How well was bias & confounding factors
were avoided ?
• Check each stage of the study How
fairly were :
– the subjects recruitted (the “P”)
– the subjects allocated to groups ( the I and C )
– the study group maintained through equal
management and follow up of subjects ( the I
and C )
– the outcome measured ( the O )

15
 Stucture of a comparative health
care research study
Study Aim Study methods Critical appraisal
question

P Fair recruitment Large enaugh sample R

Subject representative of the + randomly
target pop

I Fair allocation Randomly allocated A

Adjust confounding
(statitical
adjustment/matching)
C Fair maintenance Manage grup equally M
Follow up all subjects

O Fair measurement Measure outcome M

Valid & unbiased outcome Blinded B
measure Objective measure o

16
 Steps in critical appraisal
of primary research - RAMMBO
• Recruitment :
– Were the subjects representative of the target pop ?
• Allocation or adjustment
– Was the tx allocation concealed before randomisation and were the
groups comparable at the start of the trial
• Maintenance :
– Was the comparable status of the study groups maintained through
equal management and adequate follow up ?
• Measurement
– Were the outcomes measured with
– Blinded subjects and assessors, and/or
– Objective outcomes ?

17
 Recruitment
Were the subjects representative of target population ?

• If the subject are not representative 

– difficult to know to which pop may be applicable
• The best way to ensure – study group
representative is to :
– Recruit potential subjects
• Sequentially
• At random from population
– Only apply exclusion criteria – relevant for study
methods
• Excluding deaf people from study requiring subjects to listen
the music

18
 Recruitment
Were the subjects representative of target population ?

• Prefer large study,

– because small study group 
• imprecise estimate of the effects !
• Continuous outcomes  50 – 100 pt
• Binary outcomes :
– Common event  hundreds pt
– Rare event  thousands pt

19
 Recruitment
Were the DVT trial subjects representative of target population ?

• Inclusion/exclusion criteria
– For RCT 
• difficult random sampling due to inform consent
– Clear idea who they do represent
– Describe
• the severity,
• duration and/or
• risk level of the patients recruited

20
 Recruitment
Were the DVT trial subjects representative of target population ?

• Volunteers
– Were recruited by placing advertisements in paper
– Passengers
• > 50 yo
• Economy class
• At least 8 hours flight within 6 weeks
– Various exclusions
• Size of study groups
– 231  116 received stockings & 116 no
– This seem small
• As a 10% DVT rate – 12 events

21
 Allocation
Were the study groups comparable ?
• It is vital the groups are matched
– except for the interventions ( or
exposure/other indicator)
• Ways in which groups could differ
– Age
– Sex
– Smoker/nonsmoker
– Disease severity
• Random allocation
22
 Allocation
were the DVT study groups comparable ?

• The paper states –

– “Volunteers were randomised by sealed
envelopre to one of two groups”

23
 Allocation
Characteristic of DVT study groups
No stockings Stockings
Number 116 115
Pre-study
Age 62(56-68) 61(56-66)
Females 61(53%) 81(70%)
Varicose veins 41 45
Hb 142 140
During study
Hours flying 22 24
Day of stay 17 16

24
P < 0.01
 Maintenance
was the comparable status of the study groups
maintained through equal management and adequate
follow-up ?
• Once comparable groups have been set up  stay a
that way
• Equal management
• Unequal tx invalidates result !
– In a trial of vit E in preterm infants
– Vit E appeared to prevent retrolental fibroplasia ?
– It was not !
– Control groups  100% O2
– Tx groups  not 100% O2
• because the babies were removed from O2 for freq
dose of vit E

25
 Maintenance
Adequate follow-up
• Inevitably, some subjects drop out, change
groups or variously lost to follow up during
study  uncomparable groups !
• Check :
– Subject at start = at the end
– Subject are analysed in the groups that they
stated out in ( Intention-to-treat principle )

26
 Measurement
Were the outcomes measured with Blinded subjects
and accessorrs and/or Objective measures ?

• Measurement bias
– Human tendency to unfairly “nudge” results
– Can be overcome by
• Blinding
• Objective measurement

27
 ….Measurement
Blinding

• Best  Double-blind trial

• Moderate  Single blind
• Worst  Not blinded

• Placebo effect

28
Important …?

29
ARE THE VALID RESULTS OF THIS
INDIVIDUAL STUDY IMPORTANT?

ebm tx 10 30
Table 5.3. Measure of effect size

ebm tx 10 31
 1. WHAT IS THE MAGNITUDE OF
THE TX EFFECT ?
in clinical journals as

• the relative risk reduction (RRR)

• = ( |CER − EER| / CER ).

• In this example, the RRR = (5.7% − 4.3%) / 5.7%

= 25%,
• we can say 
– that statin therapy 
– decreased the risk of stroke by 25%
relative to those who received placebo.
ebm tx 10 32
 STUDI CAPRIE
ASA vs CLOPIDOGREL dgn p = 0.043

ebm tx 10 33
 NNT
Number Needed to Treat
• The inverse of the ARR = (1/ARR)

is a whole number and has the useful property of telling us

 the number of patients that we need to treat (NNT)

 with the experimental therapy for

 the duration of the trial in order to

 prevent one additional bad outcome.

ebm tx 10 34
 NNT
Number Needed to Treat

• In our example, the NNT is 1/1.4% = 72 ,

– means 
– we need to treat 72 people with a statin (rather than
placebo)
– for 5 years
– to prevent 1 additional person from suffering a stroke.

• for this tiny treatment effect means

• that we would have

– to treat over 7 million patients
– for 5 years to prevent one additional bad event!

ebm tx 10 35
 NNH
Number Needed to Harm
= the number needed to cause harm to one more patient (NNH) from
the therapy.
• The NNH = 1/ARI.
• In the statin study,
 0.03% of the control group experienced rhabdomyolysis
 0.05% of statin group .

• This (ARI) absolute risk increase of |0.03% − 0.05%| = 0.02%

• NNH = 1 / 0.02% = 5000

• This means that we’d need
• to treat 5000 patients with a statin for 5 years
• to cause 1 additional patient to have rhabdomyolysis.
ebm tx 10 36
 Question 3:
What do the result mean ?
• Outcome measures
– Binary
– Continuous

• Are the results real and relevent ?

• Assesing change :
– P-values (hypothesting)
– Confidence interval (estimation)
37
 Outcome measures for binary
outcomes
Measure Meaning Example
Relative risk RR – RR = 0.1/0.15
Risk of outcome how many times more = 0.67
in the Tx likely event in the tx
group/risk in the group relative to control
control group group
RR < 1
RR = 1 – no diff between 2
groups The Tx –
RR < 1 – the tx reduces decrease the
the risk of event risk of death
RR > 1 – the Tx increass
the risk of the event

38
 Outcome measures for binary
outcomes
Measure Meaning Example

ARR ARR = 0 – no diff ARR = 0.15 – 0.10 =

Risk of event in ARR +  the Tx is 0.05 (5%)
the control group beneficial The absolute benefit
– risk event in the ARR -  harmfull of Tx is a 5%
tx group reduction in the death
rate

39
 Outcome measures for binary outcomes
RRR

Measure Meaning Example

RRR RRR  RRR =

ARR/risk of event Reduction in the rate ARR / Risk of event
in the control of event in the Tx control group
group group relative to
control group

40
 Outcome measures for binary outcomes
NNT

Measure Meaning Example

NNT NNT =
= 1 / ARR The number of pts
we need to treat in
order to prevent bad
event

41
 P-values
• Are a measure of the probability that the
result is purely due to chance

42
 Confidence interval (CIs)
• More informative > P – value
• An estimate of the range of value that likely to
include the real value
• 95% means :
– The range of values that have a 95% chance of
including the real value
• If the 95% CI for the diff between Tx & Control
group
– Small
– No overlap the “no effect “ point

The result is real ( that is, with a P-value < 0.05)

43
 ……Confidence interval (CIs)
= point estimate
Diff between
Tx vs Control
= CI

Null hypothesis
(no effect)

A B C D

A : statistically sig result (p < 0.05 ) but low precision

B : statistically sig result (p < 0.05 ) with high precision
C : not statistically sig result (p > 0.05 ) with low precision
D : not statistically sig result (no effect ) with high precision

44
 Intervention
= usesful if …
• The 95% CI includes clinically important tx
effects
• Statistically sig
– Relates to the size of the effect and the 95%
CI in relation to the Null hypothesis
• Clinical importance
– Relates to the size of effect and the 95% CI in
relation to a minimum effect that would be
considered to be clinically importantce

45
 ……Clinical importance
= point estimate

= CI
Minimum clinical
Important diff

Null hypothesis
(no effect)

A B C D

A : Diff is statistically sig and clinical importance

B : Diff is not statistically sig but is clinical importance
C : Diff is statistically sig but not clinical importance
D : Diff is not statistically sig and not clinical importance 46
APPLICABLE …?

ebm tx 10 47
 ARE THE VALID, IMPORTANT RESULTS OF THIS
INDIVIDUAL STUDY APPLICABLE TO OUR PATIENT?

• If the evidence • To apply evidence,

valid and important,
integrate the evidence
with our
consider  clinical experience
and expertise,
 pt’s values and
apply it !! preferences.
to our own patient.  The guides for doing
this are in Table 5.5.

ebm tx 10 48
1. Is our patient so different from those in
the study that its results cannot apply?
• use our clinical expertise to • This isn’t sensible
decide approach
– if our patient is so different – because most differences
– that its results don’t apply. – to be quantitative
• they have different
– ages,
• One approach – degrees of risk of the
outcome event,
– responsiveness to the
– fit all the inclusion criteria therapy
for the study and – rather than qualitative
• (total absence of
– reject it if our patient responsiveness to
doesn’t fit each one. treatment or risk of event

ebm tx 10 49
1. Is our patient so different from those in
the study that its results cannot apply?
• a far more appropriate • There are only a few
approach is occasions when this might
• consider  our be the case:
patient’s
– different
– sociodemographic
pharmacogenetics,
features
– absent immune
– or pathobiology are so
responses, comorbid
different from those in
conditions that prohibit
the study
the treatment, and the
– discard its results and like.
– resume our search for
relevant evidence

ebm tx 10 50
Table 5.5

ebm tx 10 51
 Sometimes treatments appear
to produce qualitative differences in the responses of subgroups
of
patients so that they appear to benefit some subgroups
but not others

• For example,
– early trials of aspirin for – If you think
TIAs showed • that the tx you’re examining
may work in a qualitatively
different way among
– large benefits for different subgroups of
• men patients,
• but none for women;
– you should refer to the
– subsequent trials and guides in Table 5.6.
systematic reviews showed

• that this was a chance

finding and
• that aspirin is efficacious in
women.
–
ebm tx 10 52
2.Is the treatment feasible in our setting?

• if the tx is feasible in • Is the treatment

our practice setting.
• Statin therapy is
• Can our pts, or health
care system pay for
– the treatment,
– its administration,
– and the required
monitoring?
available in our setting?
currently
– “free” for use with
certain conditions in
some regions and
countries,
– but, in others, patients
would be required to pay
for it themselves.

ebm tx 10 53
 3. What are our patient’s potential
benefits and harms from the therapy?
• If the study is applicable & feasible,

• estimate our pts’s benefits & risks of

therapy.

• There are two general approaches to doing

this 
ebm tx 10 54
 3. What are our patient’s potential
benefits and harms from the therapy?

much quicker approach

• works entirely from the NNT and NNH in the

study.

• Note that, with both approaches, we assume

– that the relative benefits and risks of
therapy are the same for patients with high
and low PEERs.
ebm tx 10 55
 “likelihood of being helped and harmed”
(LHH)
• we found that :
• LHH = (1/NNT):(1/NNH)
• the ARR = 1.4% and
= (1/72):(1/5000)
• the NNT = 72.
= 70*
• he has
– a 1 in 72 chance of being
helped by a statin and • Its mean
– a stroke being prevented. – that statin therapy
– is 70 times more likely to
• Similarly, looking at his risk help him > than to harm
of harm from Table 5.3, him

• he has
– a 1 in 5000 chance of
experiencing harm
– (e.g. rhabdomyolysis) with
statin therapy. 
ebm tx 10 56