Appropriate Critical Values When Testing for a Single Multivariate Outlier by

Using the Mahalanobis Distance

Kay I. Penny

Applied Statistics, Vol. 45, No. 1. (1996), pp. 73-81.

Stable URL:
http://links.jstor.org/sici?sici=0035-9254%281996%2945%3A1%3C73%3AACVWTF%3E2.0.CO%3B2-D

Applied Statistics is currently published by Royal Statistical Society.

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/about/terms.html. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained
prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in
the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/journals/rss.html.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

The JSTOR Archive is a trusted digital repository providing for long-term preservation and access to leading academic
journals and scholarly literature from around the world. The Archive is supported by libraries, scholarly societies, publishers,
and foundations. It is an initiative of JSTOR, a not-for-profit organization with a mission to help the scholarly community take
advantage of advances in technology. For more information regarding JSTOR, please contact support@jstor.org.

http://www.jstor.org
Fri Aug 24 03:28:52 2007
AppL Statist. (1996)
45, No. 1,pp. 73-81

Appropriate Critical Values whenTesting for a

Single Multivariate Outlier by Using the
Mahalanobis Distance
By KAY I. PENNY?
University of Aberdeen, UK

[Received June 1994. Final revision March 19951

SUMMARY
The Mahalanobis distance is a well-known criterion which may be used for detecting outliers in multivariate
data. However, there are some discrepancies about which critical values are suitable for this purpose.
Following a comparison with Wilks's method, this paper shows that the previously recommended
+
( p ( n - l ) / ( n - p)JFp,,-, are unsuitable, and p(n - 1 ) Fp,,,-l / n ( n - p - 1 pFp,,-,-I) are the correct
critical values when searching for a single outlier.The importance of which critical values should be used is
illustrated when searching for a single outlier in a clinical laboratory data set containing 10 patients and five
variables. The jackknifed Mahalanobis distance is also discussed and the relevant critical values are given.
Finally, upper bounds for the usual Mahalanobisdistance and the jackknifed version are discussed.
Keywords: Critical values; Jackknifed Mahalanobisdistance; Mahalanobis distance; Multivariate outliers

1. Introduction
The methods discussed in this paper are illustrated on a small clinical laboratory data
set. This data set, which is reproduced in Table 1, consists of five liver function
variables for each of 10 patients. The variables contained in the second to sixth
columns of the table are alkaline phosphatase, bilirubin, gamma glutamyl trans-
pepidase, aspartate aminotransferase and alanine aminotransferase respectively.
The Mahalanobis distance D iis suggested in many texts as a method for detecting
outliers in multivariate data. For each of the n observations in a p-variable data set, a
distance value D iis calculated. Let 2 be the sample mean vector and let S be the
sample covariance matrix,

Then

?Address for correspondence: Department of Public Health, Medical School, Polwarth Building, University of
Aberdeen, Foresterhill, Aberdeen, AB9 2ZD, UK.

O 1996 Royal Statistical Society 0035-9254/96/45073

74 PENNY
TABLE 1
Clinical laboratory data and distances

Patient Laboratory variables 0: oh

1 151 13.7 88 336 582 8.09 12223.93
2 115 34.0 80 15 12 7.41 95.04
3 57 11.1 8 21 14 5.45 18.26
4 142 6.0 36 26 16 1.45 1.93
5 78 9.1 10 20 25 2.59 4.17
6 95 5.0 10 21 11 1.32 1.73
7 193 19.0 25 34 11 6.97 54.47
8 101 9.0 127 30 25 7.50 111.80
9 112 5.0 22 19 9 0.77 0.94
10 138 7.0 17 15 6 3.46 6.64

for i = 1, . . ., n. Jackson (1991) suggested the use of @(n - l)/(n -p))FP,,, as

critical values for D? when searching for outliers.
Mardia et al. (1979) and Krzanowski (1988) have discussed the theory behind this
result. We must firstly assume that our data sample is from a multivariate normal
distribution. Krzanowski (1988), section 7.5, showed that

where x has a multivariate normal distribution with mean p and dispersion matrix X.
In practice, we estimate the parameters p and X with the sample mean vector 2 and
the sample covariance matrix S. Replacing Z with S gives

which follows from Mardia et al. (1979), section 3.5. However, this result assumes
that x is independent of S, which is not true as x is one of the observations used to
calculate S. Further complications arise when p is replaced by 2 to give D;, as x is not
independent of 2.
It is commonly suggested that an F-distribution is more appropriate than a x2-
distribution especially when dealing with small sample sizes. However, in practice
@(n - l)/(n -p)) Fp,?,, as we show below, is inappropriate when testing for
multivariate outliers in small samples.
Squared Mahalanobis distances have been calculated for each of the 10 patients
and are displayed in the penultimate column in Table 1. A comparison of the
D?-values with the two critical values mentioned above incorrectly suggests that
none of the patients are significantly outlying (i.e. at the 5% level, x,,,~,
2
= 16.75,
@(n - l)/(n -p)) FP,,,;,ln = 134.46). These critical values are calculated by using
Bonferroni bounds, which implies that the tests are rather cautious. However, this
may be regarded as a suitable approach in the context of identifying an outlier.
In Section 2, a comparison between D? and Wilks's (1963) method for multivariate
outlier detection is made. This leads to the derivation of appropriate critical values,
 CRITICAL VALUES WHEN TESTING FOR OUTLIER 75
which are confirmed by simulations. In Section 3, the jackknifed Mahalanobis
distance is discussed, and appropriate critical values are given. In Section 4 upper
bounds for both the jackknifed and the usual Mahalanobis distance are discussed,
and the paper concludes with a discussion in Section 5.

2. Critical Values for Mahalanobis Distance

Wilks (1963) used so-called scatter ratios to search for outliers in multivariate
normal data. To search for a single outlier, Wilks calculates a scatter ratio

where

and IAl = determinant(A). Ai is calculated as for A with observation i eliminated

from the sample. The most outlying observation is that which has the smallest scatter
ratio R1, where R1 = min{Ri), i.e. the observation whose removal leads to the
greatest reduction in IAl.
Wilks showed that

Critical values for the use of R1 as an outlier test are approximated by using
Bonferroni bounds obtained from the lower 100a/n% points of the above beta
distribution, or equivalently the upper 100a/n% points of

as shown in Caroni and Prescott (1992).

Wilks's criterion for detecting a single outlier is related to D! by the equation

as shown in Barnett and Lewis (1984), section 9.3, or Krzanowski (1988), section 8.3.
Wilks's critical values derived from the beta distribution can be converted by using
equation (2) to upper critical values for use with the squared Mahalanobis distance:

Substitution of equation (1) into equation (3) gives

76 PENNY

which I shall now refer to as Wilks's critical values.

Wilks's critical values and

differ greatly for small n (Table 2).

A simulation study of 10 000 standard multivariate normal (SMVN) data sets has
been carried out for various n and for p = 2 and p = 3. D?-values were calculated for
each observation in each data set. Two methods of determining critical values for a
single outlier were used.
(a) All the Df-results from the 10 000 data sets were pooled, then sorted in
increasing order. The empirical upper a / n % point was then found.
(b) Only the largest Df-value from each data set is considered. These 10 000 values
are sorted in increasing order, and the empirical upper a% point is then
determined.
The results from both of these methods are very similar (equal to at least one
decimal place). These findings confirm that the use of Bonferroni bounds is
acceptable when determining critical values for D!. These results (Table 3) are equal
to Wilks's critical values to one decimal place in most cases. This confirms that

are the appropriate critical values when searching for a single outlier, whereas
(p(n - l)/(n - p)) F,,,,; are incorrect.
Returning to the data example described in Section 1, patient 1 is found to be

TABLE 2
Critical values for Df
 CRITICAL VALUES WHEN TESTING FOR OUTLIER

TABLE 3
Critical values for D: found by simulation

n Critical values found by using method 2

p=2 p=3
5% - 1% 5% 1%

5 3.17 3.19 3.20 3.20

10 6.33 7.00 7.04 7.51
20 9.07 10.70 10.50 11.97
50 12.18 14.42 14.13 16.42
100 14.20 16.89 16.50 19.20
500 18.07 21.19 20.71 24.12

outlying at the 1% significance level when using the correct critical values for D!, i.e.

3. Critical Values for Jackknifed Mahalanobis Distance

The problem of dependence of each xi with 2 and S may be overcome by using
jackknifed means and variances. The jackknifed mean 2(i)and covariance matrix S(,)
are simply the mean vector and covariance matrix of the n - 1 sample observations
excluding xi. Hence, the jackknifed D! may be defined by

Krzanowski (1988), section 8.1, showed that

where 2 and S-I are calculated from a sample of size n, and xi is a further observation
independent of 2 and S.
When calculating D&,xi is a further observation from the n - 1 observations used
in deriving 2(i)and S!,);therefore xi is independent of 2(,)and S(i).Hence, it follows
directly from expression (4) that

Some critical values based on the above F-distribution are given in Table 4.
Jackknifed Mahalanobis distances were calculated for 100 000 SMVN data sets,
and critical values were obtained by using method 1 described in Section 2. The
results are given alongside the theoretical critical values in Table 4. The critical values
obtained by theory and by simulation are very similar.
Jackknifed Mahalanobis distances for each of the 10 patients in the example
78 PENNY
TABLE 4
Critical values for Di0

n p f n - 2) Critical values found by simulation

- p - 1, Fp~n-p-l:~ln
(n -

described in Section 1 are given in the last column of Table 1. The critical values for
D& also finds patient 1 to be an outlier at the 1% significance level, i.e.

4. Upper Bounds for and qi)

4.1. Upper Bounds for 03

Wilks's ratios are distributed as follows:

The smaller the scatter ratio, the more outlying the observation is. Hence, to find a
lower bound for Ri let Ri + 0; then

From the findings in Section 2, we know that

D? is a monotone increasing functions of FP,,,-1; hence, by letting Fp,n-p-l + co,

 CRITICAL VALUES WHEN TESTING FOR OUTLIER

and the upper bound for D? is

f o r n - p - 1 >O.
Simulations of 10000 SMVN data sets confirm these results (Table 5). Note that
this upper bound is achieved when p = n - 2 but is not a tight bound for p < n - 2.
A comparison with Table 2 shows that in many cases the incorrect critical values
calculated from (p(n - l)/(n - p)} F,,,,; substantially exceed these upper bounds,
whereas the critical values calculated from
2
- 1) F p , n-p- 1; a / n
+
n(n - p - 1 pFP,n-p-1; a/n)
are necessarily within these bounds.

4.2. Upper Bounds for Dti)

In Section 3, it was shown that

+ m , then
Dti) is a monotone increasing function of Fp,n-p-l. Hence, as Fp,n-p-l

i.e. an upper bound does not exist for Dti,.

5. Discussion
The example described in Section 1 illustrates the importance of using the
appropriate critical values when using the Mahalanobis distance, as patient 1 goes

TABLE 5
Upper bounds for D: found by simulation

P Boundsfor the following values of n:

5 10 20 50 100

2 3.20 7.92 13.80 21.04 23.22

3 3.20 7.98 15.08 21.05 28.42
4 8.05
6 8.10
8 8.10
10 17.70
18 18.05
40 47.79
46 48.02
48 48.02
98 98.01
80 PENNY

undetected as an outlier when the previously recommended critical values of X;;aln

and Ip(n - l)/(n - p)) FP,,-,; are used.
In Section 2,

are found to be the appropriate critical values for testing for a single outlier by using
D:. Likewise,

are appropriate for use with Dti).

In Section 4, it is shown that upper bounds exist for D?, and in many cases the
incorrect critical values calculated from (p(n - l)/(n - p)) F,,,-,; exceed these
bounds. It is then shown that the jackknifed Mahalanobis distance is unbounded,
and this explains why much greater distances may be found by using the jackknifed
version. However, as the example shows, the D:-value of 8.09 and the D&-value of
12 223.93 for patient 1 are both significant at the 1% level.
Atkinson and Mulira (1993) have shown that the jackknifed Mahalanobis distance
is a monotone function of the usual Mahalanobis distance, i.e.

The ordering of the D:- and ~ t , ~ - v a l u eare

s the same because of the monotone
relationship between the two distances. Hence, both methods will detect the same
outliers in the same order of magnitude when analysing any particular data set. Since
the same monotone relationship also exists between equations (5) and (6), any
outliers detected will be significant at the same level, whichever method is used.
Hence, the additional calculations involved when using the jackknifed version may
be avoided, since the usual Mahalanobis distance gives equivalent results, provided
that the appropriate critical values are used.

Acknowledgements
I wish to thank Ian Jolliffe for the stimulating discussions while this work was in
progress, and also the Editor and referees for their constructive comments and
suggestions. The author is supported by a Co-operative Award in Science and
Engineering studentship funded by the Science and Engineering Research Council,
and Boots Pharmaceuticals, Nottingham.

References
Atkinson, A. C. and Mulira, H.-M. (1993) The stalactite plot for the detection of multivariate outliers.
Statist. Cornput., 3, 27-35.
 CRITICAL VALUES WHEN TESTING FOR OUTLIER 81
Barnett, V. and Lewis, T. (1984) Outliers in Statistical Data, 2nd edn. New York: Wiley.

Caroni, C. and Prescott, P. (1992) Sequential application of Wilks's multivariate outlier test. Appl.

Statist., 41, 355-364.

Jackson, J. E. (1991) A User's Guide to Principal Components. New York: Wiley.
Krzanowski, W. J. (1988) Principles of Multivariate Analysis. Oxford: Oxford University Press.
Mardia, K. V., Kent, J. T, and Bibby, J. M. (1979) Multivariate Analysis. London: Academic Press.
Wilks, S. S. (1963) Multivariate statistical outliers. Sankhya A, 25, 407-426.