UNIVERSITATEA BABE-BOLYAI CLUJ-NAPOCA FACULTATEA DE PSIHOLOGIE I TIINE ALE EDUCAIEI Catedra de Psihologie Clinic i Psihoterapie

ANALIZA CURBELOR ROC

CS III dr. Sebastian Pintea

Introducere
ROC (Receiver Operating Characteristic) Analysis Utilizat n analiza proprietilor diagnostice ale testelor (modul n care testele discrimineaz ntre populaia clinic i non-clinic) Rezultatele la test se raporteaz la rezultatul unui diagnostic valid (golden standard)

Introducere
Istoricul utilizrilor n domeniul militar, n analiza imaginilor radar (WW II) n domeniul medical, ncepnd din anii 60 (mai ales n imagistica medical) n chimie (ex. Dac un marker proteic este prezent sau nu)
n psihologia clinic: n evaluarea unor teste diagnostice sau de screening pt: Dificulti n comprehensiunea limbajului (Shapiro, Solari & Petscher, 2008) Dizabiliti neuropsihologice (Horwitz et al., 2008; O'Brien et al., 2007) Depresie (Benazzi, 2008; Serrano-Duenas & Serrano, 2008; Stafford, Berk & Jackson, 2007; Ballesteros et al., 2007; Walsh et al., 2006) Tulburare obsesiv-compulsiv (Ivarsson & Larsson, 2008) Tulburare bipolar (Parker et al., 2008) Suicid (Jokinen, Nordstrom & Nordstrom, 2008) Demen (Chiu et al., 2008; Giaquinto & Parnetti, 2006) Risul de dropout n diferite intervenii precum terapia cognitivcomportamental pentru insomnie (Ong, Kuo & Manber, 2008)

Indicatori ai curbelor ROC

Fig. 1. Patru posibile categorii de subieci atunci cnd intersectm un diagnostic valid
cu un clasificator

Tabelul 1. The confusion matrix

DIAGNOSTIC TEST Pozitiv Negativ Total Pozitiv TP FN D+ Negativ FP TN DTotal T+ Tn

(1) Sensitivity = TP/D+ (2) Specificity = TN/D(3) Positive likelihood ratio = Sensitivity / (1-Specificity) (4) Negative likelihood ratio = (1-Sensitivity) / Specificity (5) Positive predictive value = TP/T+ (6) Negative predictive value = TN/T(7) Accuracy = (TP+TN)/n

Spaiul ROC (interpretarea punctelor)

Sensitivity 1.0 0.8 0.6 0.4 0.2 0.0
0.0

0.2

0.4 0.6 1 - Specificity

0.8

1.0

Utilitatea analizei curbelor ROC

Determinarea capacitii unui test de a dscrimina ntre grupuri (+, -) Alegerea unei valori prag (cut-off point) optim Compararea performanei a dou sau mai multe teste

Determinarea capacitii unui test de a discrimina ntre grupuri

AUC area under the curve Cu ct AUC corespunztoare unui test este mai mare, cu att performana lui discriminativ e mai bun Diagonala spaiului ROC graficul unui test care determin cele dou categorii diagnostice n mod aleator H0: AUC= 0.50 Streiner and Cairney (2007):
AUC ntre 0.50 i 0.70: AUC ntre 0.70 i 0.90: AUC peste 0.90: acuratee redus a testului acuratee moderat acuratee ridicat

Alegerea unei valori prag (cut-off point) optim

Valoarea prag optim este dat de cel mai nord-vestic punct trasat de curba testului n spaiul ROC Valoarea prag optim este cea care maximizeaz suma TP + TN n decizie, se ia n calcul i costul unui fals pozitiv, i costul unui fals negativ (exemple?) Alte studii recomand:
Specificitate minim de 95% (Westin, 2001) n screening, senzitivitate minim de 80% (Sharifi et al., 2008) (puterea testului)

Compararea performanei a dou sau mai multe teste

Compararea ca performan global Compararea pe anumite intervale de senzitivitate sau specificitate Compararea la valorile lor prag
B

Compararea ca performan global

Softuri pentru analiza curbelor ROC

AccuROC, Analyse-it, CMDT, GraphROC, MedCalc, mROC, ROCKIT and SPSS Stephan et al. (2003): analiza comparativ a softurilor Concluziile autorilor: Doar Analyse-It, AccuROC i MedCalc au dovedit o performan bun Fiecare soft are propriile neajunsuri Doar GraphROC poate s compare curbe la o anumit valoare a senzitivitii sau specificitii O analiz ROC adecvat inclusiv cu reprezentare grafic nu poate fi fcut cu un singur soft Autorii recomand Analyse-It, AccuROC i MedCalc, dar i acestea cu anumite limitri

ROC ANALYSIS: AN APPLICATION IN CLINICAL PSYCHOLOGY Depression (DSM) BDI vs ATQ

Objectives (1) to evaluate the diagnostic performance for BDI and ATQ (2) to establish the optimal cut-off point of each measure

(3) to test if there is a difference between the two scales regarding their mean diagnostic performance

Sample
Data were pooled from participants in two separate studies. The sample consisted of 50 patients aged between 12 and 18 who met criteria for MDD according to the DSM-IV. Inclusion criteria required that patients (children and adolescents aged from 12 to 18) meet criteria for current principal diagnosis of MDD as per the Diagnostic and Statistical Manual of Mental Disorders (4th edition, text revision; DSM-IV-TR, American Psychiatric Association, 2000). Exclusion criteria included a number of concurent psychiatric disorders, current substance abuse, mental retardation, organic brain sindrome; we also excluded participants who were in some concurent form of psychotherapy, who were receiving psychothropic medication or who needed to be hospitalized because of imminent suicidal risk. Patients were recruited by local advertisements and by referrals from clinics within the Pediatric Psychiatry Clinic in Cluj-Napoca from March 2007 until June 2008. A number of 50 other voluntary adolescents from several high schools in Cluj-Napoca were also included.

Procedure
Patients were assessed prior to random assignment to three treatment groups [(1) cognitive-behavioral psychotherapy, (2) medication; (3) cognitivebehavioral psychotherapy and medication]. After detailed description of the study to all participants, informed consent was obtained. Screening instruments were completed individually before entering treatment and afterwards for 16 weeks until all treatment conditions are completed.

Measures
Beck Depression Inventory (BDI; Beck et al., 1979) is a 21item self-report inventory measuring current characteristic symptoms of depression (e.g., sadness, fatigue, social withdrawal, irritability, hopelessness etc.).
Automatic Thoughts Questionnaire (ATQ; Hollon & Kendal, 1980) is a 15-item questionnaire assessing the frequency of negative thoughts experienced by depressives. All items consist of different self related automatic thoughts (e.g. I am worthless; Future is dark; I feel helpless), frequently identified in patients with MDD.

Results
Table 2. The AUC for ATQ scale
Area under the ROC curve (AUC) Standard error 95% Confidence interval z statistic Significance level P (Area=0.5) 0.909 0.0307 0.835 to 0.957 13.338 0.0001

Table 3. The AUC for BDI scale

Area under the ROC curve (AUC) Standard error 95% Confidence interval z statistic Significance level P (Area=0.5) 0.996 0.00655 0.955 to 0.996 75.682 0.0001

Results
Table 4. Criterion values and coordinates of the ROC curve for ATQ
Criterion Sensitivity 95% CI Specificity 95% CI

+LR 1.00 1.11 1.16 1.19 1.28 1.32

-LR

>=15 >15 >16 >18 >20 >21

100.00 100.00 100.00 100.00 100.00 100.00

92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0

0.00 10.00 14.00 16.00 22.00 24.00

0.0 - 7.2 3.4 - 21.8 5.8 - 26.7 7.2 - 29.1 11.5 - 36.0 13.1 - 38.2

0.00 0.00 0.00 0.00 0.00

>23
>24 >25 >26 >27 >28 >29 >31 >33 >34 * >35

100.00
100.00 100.00 100.00 98.00 98.00 96.00 96.00 94.00 94.00 90.00

92.8 - 100.0
92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 89.3 - 99.7 89.3 - 99.7 86.3 - 99.4 86.3 - 99.4 83.4 - 98.7 83.4 - 98.7 78.2 - 96.6

32.00
42.00 48.00 50.00 58.00 60.00 62.00 66.00 68.00 70.00 70.00

19.5 - 46.7
28.2 - 56.8 33.7 - 62.6 35.5 - 64.5 43.2 - 71.8 45.2 - 73.6 47.2 - 75.3 51.2 - 78.8 53.3 - 80.5 55.4 - 82.1 55.4 - 82.1

1.47
1.72 1.92 2.00 2.33 2.45 2.53 2.82 2.94 3.13 3.00

0.00
0.00 0.00 0.00 0.034 0.033 0.065 0.061 0.088 0.086 0.14

Results
Table 5. Criterion values and coordinates of the ROC curve for BDI scale
Criterion Sensitivity 95% CI Specificity 95% CI

+LR 1.00 1.14 1.25 1.39

-LR

>=0 >0 >1 >2

100.00 100.00 100.00 100.00

92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0

0.00 12.00 20.00 28.00

0.0 - 7.2 4.6 - 24.3 10.0 - 33.7 16.2 - 42.5

0.00 0.00 0.00

>3
>4 >5 >6 >8 >9 >10 >14 >15 >17 >18 >21 * >22 >23 >24

100.00
100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 92.00 90.00 86.00

92.8 - 100.0
92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 92.8 - 100.0 80.7 - 97.7 78.2 - 96.6 73.3 - 94.2

40.00
42.00 56.00 60.00 64.00 70.00 82.00 84.00 88.00 90.00 94.00 96.00 96.00 98.00 100.00

26.4 - 54.8
28.2 - 56.8 41.3 - 70.0 45.2 - 73.6 49.2 - 77.1 55.4 - 82.1 68.6 - 91.4 70.9 - 92.8 75.7 - 95.4 78.2 - 96.6 83.4 - 98.7 86.3 - 99.4 86.3 - 99.4 89.3 - 99.7 92.8 - 100.0

1.67
1.72 2.27 2.50 2.78 3.33 5.56 6.25 8.33 10.00 16.67 25.00 23.00 45.00

0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.083 0.10 0.14

Results
Table 6. Pairwise comparison of ATQ and BDI AUC's ATQ ~ BDI Difference between areas 0.0868

Standard error
95% Confidence interval z statistic Significance level

0.0299
0.0283 to 0.145 2.907 P = 0.004

Results
Figure 4. The ROC curves of ATQ and BDI
1.0

0.8

Se nsi tivi ty

Source of the Curve ATQ BDI Reference Line

0.6

0.4

0.2

0.0 0.0 0.2 0.4 0.6 0.8 1.0

1 - Specificity