Studying Diagnosis

Universitaire Masterstudie Evidence Based Practice AMC-UvA
Module 2
Epidemiology and Evidence Based Practice: Designs Lecture 6
Probabilistic research II: Studying diagnosis
1
Learning objectives Lecture 6

Student can describe and explain the analysis of research studying diagnosis, i.e. sensitivity/specificity, predictive values, likelihood ratios, ROC-curves Student can name methodological issues concerning internal and external validity in research studying diagnosis
2
Agenda
1. Intro diagnostics 2. Studying Diagnosis: analysis (sensitivity/specificity, predictive values, likelihood ratios, ROC-curves) 3. Studying Diagnosis: methodology (cross-sectional design)
3
1. Intro diagnostics
Probabilistic research
Predictive relation between one or more determinants and one (diagnosis) or more (prognosis) outcomes Descriptive No interest in confounding Data collection reflects practice Prognosis, Diagnosis
5
Diagnosis
Predicting the presence, type, severity of disease based on patients profile
Individual prediction based on clinical profile Preferably multiple determinants (i.e. tests) Often one outcome Transversal
Diagnostic reasoning
Descriptive - pattern recognition - hypothesis testing Probabilistic - rational and quantitative - from pre-test to post-test probabilities
7
Thomas Bayes (1702-1761)

Estimating probabilities from new data using pre-existing knowledge Bayes theorem:
Post-test probability = Pre-test probability * X
8
Clinical decision-making
Will the results of the test move a decision across the test (1) or treatment (2) threshold?
Do not test Do not treat Test and, depending on the results, treat (or not) Do not test, but treat now!
pre-test probability
Multistage, multidimensional diagnostic process
Therapy
Patient Outcomes
Presenting symptoms
History & Examination
Further Testing 1
Further Testing 2
No therapy
Patient Outcomes
Cave: focus on single disease
True positives [p*seA]
Treat
risk reduction as observed in trial + risk side effects risk side effects, no treatment effect
False positives [(1-p)*(1-specA)] Test A False negatives [p*(1-seA)] True negatives [(1-p)*spA]
Treat
No Treat risk untreated No Treat none
Test A or B?
True positives [p*seB] False positives [(1-p)*(1-specB)] Test B False negatives [p*(1-seB)] p = prevalence se = sensitivity sp = specificity
Treat
risk reduction as observed in trial + risk side effects risk side effects, no treatment effect
Treat
No Treat risk untreated
True negatives [(1-p)*spB]
No Treat none
Sutton et al. Integration of meta-analysis and economic decision modelling for evaluating tests. MDM 2008
1. Classical diagnostic thinking

Uni- or multivariable approach:
probability of disease being present given clinical profile one or multiple pieces of information estimate independent contribution (weight) uni- or multivariable analysis clinical decision rules Goal: better prediction/discrimination
Tests can be
history physical examination blood tests imaging procedures questionnaires etcetera
2.
Studying Diagnosis: analysis
14
Diagnostic accuracy
The extent to which the test results reflect the true state The ability of a test to discriminate among patients with and without the suspected disease
How accurate is the test?
Reference test
Patient t n ie t a P la m r o N
TP True Positive
Normal
FP False Positive Threshold
t s e t x e d n I
FN False Negative
TN True Negative
Basic design
Patients with a suspected disease
Index test Reference test (gold standard) Compare test results

17
Clinical problem
Patient with chest pain suggestive for acute myocardial infarction (AMI) Does this patient have an AMI?
18
Clinical problem
Diagnostic value of creatine kinase (CK) measurement Does CK measurement distinguish between those with and without myocardial infarction?
19
Anatomy of the study

Target population: patients with chest pain Index test: CK measurement Target condition: acute myocardial infarction Final diagnosis based on WHO criteria (reference standard):
clinical outcome ECG-changes enzym values autopsy
20
Our example
Patients with chest pain CK measurement WHO criteria for AMI Cross-classification
21
Results of CK study
AMI Present high (>80) low 215 ab cd 15 230 114 130 129 360
22
Absent 16 231
CK
Measures of accuracy
sensitivity 215 / 230 = 93% < Pr(T+|D+) > specificity 114 / 130 = 88% < Pr(T-|D-) >
AMI Present high (>80) low 215 15 230 Absent 16 114 130 231 129 360
23
CK
Cut-off value
Accuracy: Sens/Spec
dichotomous test outcomes depend on cut-off value (trade-off; FP/FN) independent of disease prevalence Se/Sp of a single test may vary widely across studies Se/Sp: from disease status to test result (testing the test)
25
Measures of accuracy
PPV 215 / 231 = 93% < Pr(D+|T+) > NPV 114 / 129 = 88% < Pr(D-|T-) >
26
CK
Prevalence
Prevalence: percentage of patients with the target disorder at a certain point in time
formula: (TP + FN) / N
reference positive index test positive negative TP FN TP+FN / standard negative FP TN FP+TN TP+FP FN+TN N
Effect of prevalence on measures of outcome
D-dimer test in GP practice (low prevalence DVT) D-dimer test in academic hospital (high prevalence DVT)
High prevalence
DVT D-dimer positive negative 215 15 230 no DVT 16 114 130 231 129 360
prevalence = 230 / 360 = 64% Sens = 215 / 230 = 0.93 Spec = 114 / 130 = 0.88 PPV = 215 / 231 = 0.93 NPV = 114 / 129 = 0.88
Low prevalence
DVT D-dimer positive negative 215 15 230 no DVT 248 1822 2070 463 1837 2300
prevalence = 230 / 2300 = 10% Sens = 215 / 230 = 0.93 Spec = 1822 / 2070 = 0.88 PPV = 215 / 463 = 0.46 NPV = 1822 / 1837 = 0.99
Accuracy: PVs
post-test probabilities dichotomous test outcomes depend on Se/Sp dependent upon prevalence PVs: from test result to disease status (testing the patient)
31
Estimating post-test probabilities

Unknown prevalence:
- SpPin/SnNout
Known prevalence:
- directly use PVs (similar prevalence) - calculate PVs using Se/Sp (prevalence not similar) - use likelihood ratios (LR +/-)
32
Unknown prevalence: ruling out the target disorder

If a test has a sufficiently high Sensitivity, a Negative result rules out the target disorder the rule of SnNout low FN rate
e.g. breast cancer screening
TP FN
FP TN
Unknown prevalence: ruling in the target disorder

If a test has a sufficiently high Specificity, a Positive result rules in the target disorder the rule of SpPin low FP rate
TP FN
FP TN
Likelihood ratios
Direct link from pre-test probabilities to post-test probabilities Applicable in situations with more than two test outcomes
35
Likelihood ratios
Summarize predictive power of a test result in a single measure Likelihood ratio of a positive and negative test result How more often a positive test result occurs in persons with compared to those without the target condition
36
Positive likelihood ratio

Likelihood ratio of a positive test result
How more likely a positive test result is in persons with the target condition compared to those without the target condition
37
Positive likelihood ratio

LR + = Pr(T + | D +) 215 / 230 = = 7.6 Pr(T + | D ) 16 / 130
38
CK
Negative likelihood ratio

Likelihood ratio of a negative test result
How less likely a negative test result is in persons with the target condition compared to those without the target condition
39
Negative likelihood ratio

LR = Pr(T | D +) 15 / 230 = = 0.07 Pr(T | D ) 114 / 130
40
CK
93% LR+ = 7.6
64%
41
Advantages of likelihood ratios

Relatively stable (function of Se/Sp) Still useful when there are more than two test outcomes
42
CK is a continuous measurement
Dichotomisation of CK (high vs low) means loss of information Higher values of CK are more indicative of myocardial infarction
43
Results of CK study
MI CK 280 80-279 40-79 1 - 39 Total Present 97 118 13 2 230 Absent 1 15 26 88 130 Total 98 133 39 90 360
44
Likelihood ratios
Stratum-specific likelihood ratios in case of more than two test results
Pr(T = x | D +) Pr(T = x | D )
LR (T = x ) =
45
Stratum-specific likelihood ratios

LR (CK 280) = Pr(CK 280 | D +) 97 / 230 = = 54.8 Pr(CK 280 | D ) 1 / 130
MI CK 280 80-279 40-79 1 - 39 Total Present 97 118 13 2 230 Absent 1 15 26 88 130
46
Stratum-specific likelihood ratios

MI CK 280 80-279 40-79 1 - 39 Total Present 97 118 13 2 230 Absent 1 15 26 88 130 LR 54.83 4.45 0.28 0.01 360
47
Bayes rule
Post-test odds for disease = Pre-test odds for disease * Likelihood ratio
48
Bayes rule
odds = probability / (1 probability)
Odds( D + ) =
Pr( D + ) 1 Pr( D +)
probability = odds / (1 + odds)
Pr ( D +) =
Odds( D + ) 1 + Odds( D + )
49
Bayes rule: patient with CK between 80-279

Pre-test probability = 0.5 Pre-test odds = 0.5 / (1-0.5) = 1 LR(CK 80-279) = 4.45 Post-test odds = 1 * 4.45 = 4.45 Post-test probability = 4.45 / (1+4.45) = 0.82
50
Bayes rule: patient with CK lower than 40

Pre-test probability = 0.5 Pre-test odds = 0.5 / (1-0.5) = 1 LR(CK < 40) = 0.013 Post-test odds = 1 * 0.013 = 0.013 Post-test probability = 0.013 / (1+0.013) = 0.013
51
Probability of MI after CK
Pre-test prob. CK 280 80 - 279 40-79 1-39 LR 50% Post test prob. 54.83 4.45 0.28 0.013 98% 82% 22% 1%
52
Probability of MI after CK
Pre-test prob. CK LR 5% 50% Post test prob. 280 80 - 279 40-79 1-39 54.83 4.45 0.28 0.013 74% 19% 1% 0% 98% 82% 22% 1%
53
Classification of LR values
>10 and <0.1: large and conclusive changes 5-10 and 0.1-0.2: moderate 2-5 and 0.5-0.2: small (but sometimes important) 1-2 and 0.5-1: small (and rarely important)
54
ROC-curve
ROC stands for Receiver Operating Characteristic ROC-curve shows the pairs of sensitivity and specificity that correspond to various cut-off points for the continuous test result
55
Cut-off value
MI absent
MI-patients
TN
FP CK measurement
56
Cut-off value
MI absent
MI-patients
FN
TP
CK measurement
57
Change in cut-off value
MI absent
MI present
FP Spec
CK measurement
58
Change in cut-off value
MI absent
MI present
FN Sens
CK measurement
59
Change in cut-off value and effect on Se/Sp

Cut-off 9999 280 80 40 1 Sensitivity 0.0% 42.2% 93.5% 99.1% 100.0% Specificity 100.0% 99.2% 87.7% 67.7% 0.0%
60
ROC-curve CK
100% 80% Sensitivity 60% 40% 20% 0% 0% 20% 40% 60% 1-specificity 80% 100%
61
Cut-off: 80
Cut-off: 280
ROC-curve
AUC 0.91
62
ROC-curve
Shows the effect of different cut-off values on sensitivity and specificity Better tests have curves that lie closer to the upper left corner Area Under the ROC-Curve (AUC) is a single measure of test performance (0-1, higher is better)
63
So far,
Accuracy: how close to the true state? - measures of prediction (Se/Sp, PVs, LRs) Accuracy: distinguishing between patients - measures of discrimination (DOR, ROC with AUC)
64
3.
Studying Diagnosis: methodology
65
Basic design
Patients with a suspected disease
Index test Reference test (gold standard) Compare test results

66
67
Whiting P et al. Ann Intern Med 2004;140:189-202
2. More modern diagnostic thinking

Determine the most likely role and position of a new test within the test-treatment pathway relative to current practice
- replacement, add-on, triage
Compare differences in:

- test safety etc. - Se/Sp - treated population - management following positive (TP/FP) and negative (TN/FN) test results - treatment effects - patient outcomes
Roles of tests and positions in existing diagnostic pathways
Bossuyt et al. BMJ 2006;332:1089-92

Copyright 2006 BMJ Publishing Group Ltd.
69
Test evaluation research
70
Van den Bruel et al. J Clin Epidemiol 2007;60:1116-22
Further reading
Knottnerus JA, Buntinx F (Ed.). The evidence base of clinical diagnosis. Theory and methods of diagnostic research, 2e edition. Blackwell Publishing, 2009
Newman TB, Kohn MA. Evidence-based diagnosis. Cambridge University Press, 2009

Studying Diagnosis

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Studying Diagnosis

Încărcat de

Drepturi de autor:

Formate disponibile

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Learning objectives Lecture 6

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Thomas Bayes (1702-1761)

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Multistage, multidimensional diagnostic process

History & Examination

Cave: focus on single disease

True positives [p*seA]

No Treat risk untreated No Treat none

No Treat risk untreated

True negatives [(1-p)*spB]

Universitaire Masterstudie Evidence Based Practice AMC-UvA

1. Classical diagnostic thinking

Universitaire Masterstudie Evidence Based Practice AMC-UvA

Universitaire Masterstudie Evidence Based Practice AMC-UvA