Intra-class Correlation Coefficient

Psychologists commonly measure various characteristics by having a rater assign scores to observed
people, or events. When using such a measurement technique, it is desirable to measure the extent to which two
or more raters agree when rating the same set of things. This can be treated as a sort of reliability statistic for
the measurement procedure.
Continuous Ratings, Two Judges
For example, suppose that we have two judges rating the aggressiveness of each of a group of children on a
playground. f the judges agree with one another, then there should be a high correlation between the ratings
given by the one judge and those given by the other. !ccordingly, one thing we can do to assess inter"rater
agreement is to correlate the two judges# ratings. $onsider the following ratings %they also happen to be ran&s'
of ten subjects(
)ubject * + , - . / 0 1 2 *3
4udge * *3 2 1 0 / . - , + *
4udge + 2 *3 1 0 . / - , * +
5ere is the dialog window in )P)). $lic& on !naly6e, $orrelate, 7ivariate(
The Pearson correlation is impressive, r 8 .2/-. f our scores are ran&s or we can justify converting
them to ran&s, we can compute the )pearman correlation coefficient or 9endall#s tau. For these data Spearman
rho is .2/- and Kendall's tau is .1/0.
We must, however, consider the fact that two judges scores could e highl! correlated with one
another ut show little agreement. $onsider the following data(
)ubject * + , - . / 0 1 2 *3
4udge - *3 2 1 0 / . - , + *
4udge . 23 *33 13 03 .3 /3 -3 ,3 *3 +3
The correlations between judges - and . are identical to those between * and +, but judges - and .
obviously do not agree with one another well. 4udges - and . agree on the ordering of the children with respect
to their aggressiveness, but not on the overall amount of aggressiveness shown by the children.
:ne solution to this problem is to compute the intraclass correlation coefficient. For the data above, the
intraclass correlation coefficient between 4udges * and + is .2/0+ while that between 4udges - and . is .3.,..
What if we have more than two judges, as below; We could compute Pearson r, )pearman rho, or
9endall tau for each pair of judges and then average those coefficients, but we still would have the problem of
high coefficients when the judges agree on ordering but not on magnitude. We can, however, compute the
intraclass correlation coefficient when there are more than two judges. For the data from three judges below,
the intraclass correlation coefficient is .11+*.
)ubject * + , - . / 0 1 2 *3
4udge * *3 2 1 0 / . - , + *
4udge + 2 *3 1 0 . / - , * +
4udge , 1 0 *3 2 / , - . + *
The intraclass correlation coefficient is an inde" of the reliailit! of the ratings for a t!pical, single
judge. We employ it when we are going to collect most of our data using only one judge at a time, but we have
used two or %preferably' more judges on a subset of the data for purposes of estimating inter"rater reliability.
)P)) calls this statistic the single measure intraclass correlation.
The Intraclass Correlation Coefficient
$lic& !naly6e, )cale, <eliability !nalysis.
)coot all three judges into the tems box.
$lic& )tatistics. !s& for an ntraclass correlation coefficient, Two"Way <andom model, Type 8 !bsolute
!greement.
$ontinue, :9.
5ere is the output. =ou are loo&ing for the intraclass correlation coefficient, which have bolded.
****** Method 1 (space saver) will be used for this analysis ******
Intraclass Correlation Coefficient
Two-way Random ffect Model (!bsolute !"reement #efinition)$
%eople and Measure ffect Random
Single Measure Intraclass Correlation = .6961*
&'())* C(I($ +ower , ()''- .pper , (&/)0
1 , 210()))) #1 , (03 -()) 4i"( , ()))) (Test 5alue , ()))) )
!vera"e Measure Intraclass Correlation , (-67)
&'())* C(I($ +ower , (10-) .pper , (&-/0
1 , 210()))) #1 , (03 -()) 4i"( , ()))) (Test 5alue , ()))) )
*$ 8otice that the same estimator is used whether the interaction effect
is present or not(