Documente Academic
Documente Profesional
Documente Cultură
Notation
There are N persons taking a test that consists of k items. A score X ji is given to
the jth person on the ith item.
i items
1
1
X 11
X 12
X 1k
X ji
Pj
M
N
P1
X N1
X N2
T1
T2
Ti
X Nk
PN
Tk
2 RELIABILITY
If the model is SPLIT, k1 items are in part 1 and k 2 = k k1 are in part 2. If the
number of items in each part is not specified and k is even, the program sets
k1 = k 2 = k 2 . If k is odd, k1 = k + 1 2 . It is assumed that the first k1 items are
in part 1.
1 6
wj
W=
j =1
Pj =
ji
i =1
Pj = Pj k
Ti =
X ji w j
j =1
k
G=
X ji w j
i =1 j =1
G =
G
Wk
Ti = Ti W
RELIABILITY 3
w j X 2ji WTi 2
j =1
Si =
W 1
M =G W
For the split model:
Mean Part 1
k1
M1 =
i =1
Mean Part 2
k
M2 =
Ti
i = k1 +1
Scale Variance
S 2p
6 !
1
=
W 1
j =1
W T
k
Pj2 w j
i =1
"#
##
$
4 RELIABILITY
S 2p1
1
=
W 1
w
X
!
k1
j =1
W T
k1
ji
i =1
i =1
"#
##
$
Variance Part 2
1
S 2p2 =
W 1
N k 2 k 2 "#
w j X ji W Ti ##
! j =1 i = k +1 i = k +1 $
1
Item-Total Statistics
Scale Mean if the ith Item is Deleted
~
M i = M Ti
~
Si2 = S 2p + Si2 2 cov X i , P
cov X i , P =
1
W 1
P X
N
j =1
k
ji w j
Tl Ti
l =1
RELIABILITY 5
S
k 1
Ai =
1
k 2
2
l
~
Si2
l =1
l i
Ri =
cov X i , P Si2
~
Si Si
Sum of Squares
df
Between
people
Pj2 w j
W 1
j =1
k
Within
people
k G 2 Wk
w j X 2ji
i =1 j =1
Pj2 w j
1 6
W k 1
j =1
Between
measures
i =1 j =1
Total
W G 2 Wk
k 1
i =1
k
Residual
w j X 2ji
Pj2 w j
j =1
Ti2 W G 2 Wk 1W 161k 16
i =1
w j X 2ji G 2 Wk
i =1 j =1
Wk 1
6 RELIABILITY
Each of the mean squares is obtained by dividing the sum of squares by the
corresponding degrees of freedom. The F ratio for between measures is
F=
MS between measures
,
MS residual
61 67
df = k 1, W 1 k 1
SS between measures
,
MS within people
df = k 1
Note: Data must be ranks for the Friedman test and a dichotomy for the Cochran test.
SS between measures
SS total
SS nonadd = M 2 D ,
df = 1
RELIABILITY 7
where
8 1Wk 6
"
= 3T G 8 "# w 3 P G 8 ##
#$ !
#$
!
k
i =1
j =1
T P X w G P w
= w 3 P G 8 X 3T G 8"##
!
$
M=
i =1
ji
2
j
j =1
k G SS bet. meas
j =1
ji
j =1
i =1
61 6
df = W 1 k 1 1
F=
MS nonadd
MS balance
2 1
61 6 7
df = 1, W 1 k 1 1
B$ = M D ,
and the power to transform to additivity is
$
p$ = 1 BG
8 RELIABILITY
Scale Statistics
Reliability coefficient alpha (Cronbach 1951)
S
k
A=
1 S
k 1
k
2
i
i =1
2
p
A1 =
SS
k1
1
k1 1
k1
2
i
2
p1
i =1
k
A2 =
2
Si2
k2
1
k2 1
S2
i = k +1 p 2
1
1 2
S p S 2p1 S 2p 2
2
R=
S p1S p 2
Y=
2R
1+ R
RELIABILITY 9
G=
2 S 2p S 2p1 S 2p2
S 2p
ULY =
R 2 + R 4 + 4 R 2 1 R 2 k1k 2 k 2
2 1 R 2 k1k 2 k 2
Basic ComputationsMethod 2
Items with zero variance are deleted from the scale and from k , k1 , and k 2 . The
inverses of matrices, when needed, are computed using the sweep operator
described by Dempster (1969). If V < 10 30 , a warning is printed and statistics
that require V 1 are skipped.
%K 1
W 1 X X w
K
=&
KKr S S
K'
li
vij
lj
WTi Tj , i , j = 1, K , k
l =1
vij
Si Si
, where Si2 =
ij i j
rij =
vij
W 1
10 RELIABILITY
Scale Variance
k
S 2p =
Si2 + 2
i =1
vij
i< j
S p21 =
k1
Si2 + 2
i =1
k1
ij
i< j
S p22 =
k1
Si2 + 2
i = k1 +1
ij
i = k1 +1 j > i
Scale StatisticsMethod 2
Alpha Model
Estimated Reliability
S
k
1
k 1
S
k
i =1
2
i
2
p
k Corr
1 6
1 + k 1 Corr
RELIABILITY 11
where
k
rij
1 6 i <
j
2
Corr =
k k 1
Split Model
Correlation between Forms
k1
ij
i =1 j = k1 +1
S p1S p2
Guttman Split-Half
k1
v
G=
ij
i =1 j = k1 +1
S 2p
Alpha and Spearman-Brown equal and unequal length are computed as in method 1.
12 RELIABILITY
2
i
L1 = 1
i =1
S p2
2k
k 1
L2 = L1 +
L3 =
2
ij
i< j
2
Sp
k
L1
k 1
k
4
L4 =
ij
i< j
S p2
2 max
i
L5 = L1 +
2
ij
j i
S p2
k
L6 = 1
2
i
3 8
S p2 ; where i2 = V 1
i =1
CV = var =
1
k
S
i =1
2
i
1
ii
RELIABILITY 13
True Variance
TV = cov =
vij
1 6 i<
j
2
k k 1
Error Variance
EV = var cov
R$ = cov var
S
k
A=
1 S
k 1
k
2
i
i =1
2
p
1 6
1 6
2+ W 3 A
A$ =
W 1
where A is defined above.
14 RELIABILITY
k 1 k + 16 12 k 36
= 1W 16 1
121k 16 k 1k + 16 2 1W 16 log L
2
2
where
L=
log UC = log V
4
9k 14var + 1k 16cov9
CV = var +
1
k
3T G 8
i
i =1
RELIABILITY 15
Error Variance
EV = MS within people
All mean squares are calculated as described later in the analysis of variance table.
True Variance
TV = var +
1
k
3T G 8
i
EV
i =1
R$ =
1 6 3T G 8
1
var + 3T G 8
k
1
cov
k 1 k
i =1
i =1
Re l =
kR$
1 + k 1 R$
1 6
Re l =
3 + W 3 Re l
W
16 RELIABILITY
6
2 = W 1 1
1 6 12k 36 log L
121 k 162 k 1 k + 36 2 371W 16
k k +1
where
L=
k
k 1
2
1
4var + 1k 16cov9 var cov + k 3Ti G 8
i =1
df = k 1 k + 36 2 3
log UC = log V
1
log C = log4 var + 1 k 16cov9 var cov +
T
G
3
8
k 1
k
i =1
k 1
RELIABILITY 17
Additional StatisticsMethod 2
Descriptive and scale statistics and Tukeys test are calculated as in method 1.
Multiple R 2 if an item is deleted is calculated as
2
~
Ri2 = 1 i2
Si
i2 =
4V 9ii
1
Sum of Squares
df
1
1W 16 k S
!
k
Between
people
Within
people
i =1
k 1
"# 2
v # + 1k 16 v
#$
ij
i< j
W T
T
1W 161k 16 S 2
k 1v
k
!
i
i =1
i =1
Between SS + Within SS
2
T 2 = WY B 1Y
k 1
2
i
1 6
W k 1
i =1
Total
W 1
i< j
Residual
ij
"
1W 161k 16 k Si2 2
vij # + 1W 16SS bet. people
k
! i =1 k 1 i < j ##$
k
Between
measures
2
i
ij
i< j
"#
##$
1W 161k 16
Wk 1
18 RELIABILITY
where
T T "#
T T #
Y=
##
M
!T T #$
B = CVC
1
k 1
F=
W k +1
1W 161k 16 T
Mean =
i =1
T T
Variance =
1k 16
k
i =1
i =1
Maximum = max Ti
i
Minimum = min Ti
i
Ratio =
Maximum
Minimum
, with df = k 1, W k + 1
RELIABILITY 19
v
Mean =
k 1 k 16
ij
i< j
1 6
!
1
Variance =
k k 1 1
vij2
i< j
1 6 v
k k 1
ij
i< j
"#
##
$
Ratio =
Maximum
Minimum
Inter-Item Correlations
Same as for inter-item covariances, with vij being replaced by rij .
If the model is split, statistics are also calculated separately for each scale.
20 RELIABILITY
measurements, for example, the average rating for k judges, or the average score
for a k-item test.
(1) =
2p
2p + 2w
Estimate
ICC (1) =
MS BP - MSWP
.
MS BP + ( k - 1) MSWP
In general,
-1
< ICC(1) 1.
k -1
Confidence Interval
RELIABILITY 21
For 0 < < 1, a (1- )100% confidence interval for (1) is given by
Fp / w - F / 2 ,W -1,W ( k -1)
Fp / w + ( k - 1) F / 2 ,W -1,W ( k -1)
where
MS BP
MSWP
Fp / w =
and F ,v1 ,v2 is the upper point of a F-distribution with degrees of freedom v1
and v2.
Hypothesis Testing
The test statistic F (1) for H0 : (1) = 0 , where 1 > 0 0 is the hypothesized
value, is
F (1) = Fp / w
1- 0
.
1 + ( k - 1) 0
Under the null hypothesis, the test statistic has an F-distribution with
W - 1, W ( k - 1) degrees of freedom.
Average Measure Intraclass Correlation
The average measure intraclass correlation is defined as
(k ) =
2p
2p + 2w / k
Estimate
22 RELIABILITY
ICC ( k ) =
MS BP - MSWP
.
MS BP
Confidence Interval
Fp / w - F / 2 ,W -1,W ( k -1)
Fp / w
Hypothesis Testing
F ( k ) = Fp / w 1 - 0 .
Under the null hypothesis, the test statistic has an F-distribution with
W - 1, W ( k - 1) degrees of freedom.
RELIABILITY 23
( A,1,r )
%K
K
=&
KK
K'
2p
2
p
+ 2m + 2pm + 2e
2p
2
p
+ 2m + 2e
Estimate
ICC ( A,1, r ) =
MS BP
MS BP - MS Res
.
+ ( k - 1) MS Res + k ( MS BM - MS Res ) / W
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
where
(aMS )
! k -1
BM
(bMS Res ) 2
(W - 1)( k - 1)
"#
#$
24 RELIABILITY
a=
k ICC ( A,1, r )
W (1 - ICC ( A,1, r ))
and
b = 1+
k ICC ( A,1, r ) (W - 1)
.
W (1 - ICC ( A,1, r ))
Hypothesis Testing
F ( A,1,r ) =
MS BP
a0 MS BM + b0 MS Res
where
a0 =
k 0
W (1 - 0 )
and
b0 = 1 +
k 0 (W - 1)
.
W (1 - 0 )
Under the null hypothesis, the test statistic has an F-distribution with W - 1, (01)
degrees of freedom.
(01) =
(a0 MS BM + b0 MS Res ) 2
(a MS
! k -1
0
BM )
(b MS Res ) 2
+ 0
(W - 1)( k - 1)
"# .
#$
RELIABILITY 25
( A,k ,r )
%K
K
=&
KK
K'
2p
2
p
2
+ ( m
+ 2pm + 2e ) / k
2p
2
p
2
+ ( m
+ 2e ) / k
Estimate
ICC ( A, k , r ) =
MS BP
MS BP - MS Res
.
+ ( MS BM - MS Res ) / W
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
W ( MS BP - F / 2 ,W -1,v MS Res )
< ( A,k ,r )
F / 2 ,W -1,v ( MS BM - MS Res ) + W MS BP
<
The test statistic for H0 : ( A,k ,r ) = 0 , where 1 > 0 0 is the hypothesized value,
is
26 RELIABILITY
F ( A, k ,r ) =
MS BP
c0 MS BM + d 0 MS Res
where
c0 =
0
W (1 - 0 )
and
d0 = 1 +
0 (W - 1)
.
W (1 - 0 )
Under the null hypothesis, the test statistic has an F-distribution with W - 1, (0k )
degrees of freedom.
(0k ) =
(c0 MS BM + d 0 MS Res ) 2
(c MS
! k -1
BM )
(d MS Res ) 2
+ 0
(W - 1)( k - 1)
"# .
#$
( C ,1,r )
%K
K +
=&
KK
K' +
2
p
2
pm
2
p
2
p
2
p
2
e
+ 2e
Estimate
RELIABILITY 27
ICC (C ,1, r ) =
MS BP - MS Res
.
MS BP + ( k - 1) MS Res
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
Fp / r =
MS BP
.
MS Res
Hypothesis Testing
The test statistic for H0 : ( C ,1,r ) = 0 , where 1 > 0 0 is the hypothesized value,
is
F ( C ,1,r ) = Fp / r
1- 0
.
1 + ( k - 1) 0
Under the null hypothesis, the test statistic has an F-distribution with
W - 1,(W - 1)( k - 1) degrees of freedom.
Type C Average Measure Intraclass Correlation
The type C average measure intraclass correlation is defined as
28 RELIABILITY
( C , k ,r )
%K
K
=&
KK
K'
2p
2
p
+ ( 2pm + 2e ) / k
2p
2
p
+ 2e / k
Estimate
ICC (C , k , r ) =
MS BP - MS Res
.
MS BP
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
Hypothesis Testing
F ( C ,k ,r ) = Fp / r (1 - 0 ) .
Under the null hypothesis, the test statistic has an F-distribution with
W - 1,(W - 1)( k - 1) degrees of freedom.
RELIABILITY 29
Two-Way Mixed Effects Model: People Effects Random, Measures Effects Fixed
Let Xji be the response to the i-th measure given by the j-th person, i = 1, , k, j =
1, , W. Suppose that Xji can be expressed as Xji = + pj + mi + pmji + eji, where pj
is the people effect which is normal distributed with zero mean and a variance of
2p , mi is considered as a fixed effect, pmji is the interaction effect which is normal
distributed with zero mean and a variance of 2pm , and eji is the error effect which
is again normal distributed with zero mean and a variance of e2 . Denote m2 as
the expected measure square of between measures effect mi.
Let MS BP and MS Res be the respective between-people Mean Squares and
Residual Mean Squares. Formulas for these quantities can be found on page 479 of
SPSS 7.5 Statistical Algorithms by dividing the corresponding Sum of Squares with
its degrees of freedom.
Type A Single Measure Intraclass correlation
The type A single measure Intraclass correlation is defined as
%K - / (k - 1)
K + + ( +
=&
KK
K' + +
2
p
( A,1,m)
2
p
2
p
2
pm
2
m
2
p
2
m
2
pm
2
e
2
e)
Estimate
ICC ( A,1, m) =
MS BP
MS BP - MS Res
.
+ ( k - 1) MS Res + k ( MS BM - MS Res ) / W
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
30 RELIABILITY
A (1- )100% confidence interval for ( A,1,m) is the same as that for ( A,1,r ) , with
ICC ( A,1, r ) replaced by ICC( A,1, m) .
Hypothesis Testing
The test statistic for H0 : ( A,1,m) = 0 , where 1 > 0 0 is the hypothesized value,
is the same as that for ( A,1,r ) , with the same distribution under the null hypothesis.
Type A Average Measure Intraclass Correlation
The type A average measure intraclass correlation is defined as
( A,k ,m)
%K
K
=&
KK
K'
2p - 2pm / ( k - 1)
2
p
+ ( 2m + 2pm + 2e ) / k
2p
2
p
+ ( 2m + 2e ) / k
Estimate
%KNot estimable
MS - MS
ICC ( A, k , m) = &
K' MS + ( MS - MS
BP
BP
BM
Res
Res ) / W
Confidence Interval
A (1- )100% confidence interval for ( A,k ,m) is the same as that for ( A,k ,r ) ,
with ICC ( A, k , r ) replaced by ICC ( A, k , m) . Notice that the hypothesis test is not
available when the interaction effect pmji is present.
Hypothesis Testing
The test statistic for H0 : ( A,k ,m) = 0 , where 1 > 0 0 is the hypothesized value,
is the same as that for ( A,k ,r ) , with the same distribution under the null
RELIABILITY 31
hypothesis. Notice that the hypothesis test is not available when the interaction
effect pmji is present.
Type C Single Measure Intraclass Correlation
The type C single measure intraclass correlation is defined as
( C ,1,m)
%K -
K + (
=&
KK
K' +
2
p
2
p
2
pm / ( k - 1)
2
2
pm + e )
2
p
2
p
2
e
Estimate
ICC (C ,1, m) =
Notice that the same estimator is used whether or not the interaction effect pmji is
present.
Confidence Interval
Fp / r =
Hypothesis Testing
MS BP
.
MS Res
32 RELIABILITY
The test statistic for H0 : ( C ,1,m) = 0 , where 1 > 0 0 is the hypothesized value,
is
1- 0
.
1 + ( k - 1) 0
F ( C ,1,m) = Fp / r
Under the null hypothesis, the test statistic has an F-distribution with
W - 1,(W - 1)( k - 1) degrees of freedom.
Type C Average Measure Intraclass Correlation
The type C average measure intraclass correlation is defined as
%K - / ( k - 1)
K + ( + ) / k
=&
KK
K' + / k
2
p
( C ,k ,m)
2
pm
2
pm
2
p
2
e
2
p
2
p
2
e
Estimate
%K
&K
'
Not estimable
ICC (C, k , m) = MS BP - MS Res
MS BP
Confidence Interval
RELIABILITY 33
Notice that the confidence interval is not available when the interaction effect pmji
is present.
Hypothesis Testing
The test statistic for H0 : ( C ,k ,m) = 0 , where 1 > 0 0 is the hypothesized value,
is
F ( C ,1,m) = Fp / r 1 - 0 .
Under the null hypothesis, the test statistic has an F-distribution with
W - 1,(W - 1)( k - 1) degrees of freedom. Notice that the F-test is not available
when the interaction effect pmji is present.
References
Cronbach, L. J. 1951. Coefficient alpha and internal structure of tests.
Psychometrika, 16: 297334.
Dempster, A. P. 1969. Elements of Continuous Multivariate Analysis. Reading,
Mass.: Addison-Wesley.
Guttman, L. 1945. A basis for analysing test-retest reliability. Psychometrika,
10(4): 255282.
Kristof, W. 1963. The statistical theory of stepped-up reliability coefficients when
the test has been divided into several equivalent parts. Psychometrika, 28: 221
238.
Kristof, W. 1969. Estimation of true score and error variance under various
equivalent assumptions. Psychometrika, 34: 489507.
McGraw, K. O., and Wong, S. P. (1996). Forming Inferences About Some
Intraclass Correlation Coefficients. Psychological Methods, 1, p30-46.
Novick, M. R., and Lewis, C. 1967. Coefficient alpha and the reliability of
composite measurements. Psychometrika, 32: 113.
Shrout, P. E., and Fleiss, J. L. (1979). Intraclass Correlations: Uses in Assessing
Rater Reliability. Psychological Bulletin, 86, p420-428.
34 RELIABILITY
Winer, B. J. 1971. Statistical principles in experimental design, 2nd ed. New York:
McGraw-Hill.