Data reconciliation and gross error diagnosis based on

regression
Ricardo Maronna
1
and Jorge Arcas
2
1
National University of La Plata and C.I.C.P.B.A.
Departamento de Matemtica, C.C. 172, La Plata 1900, Argentina;
e-mail: rmaronna@mail.retina.ar
2
National University of La Plata and CONICET
CINDEFI, Calle 50 y 115, La Plata 1900, Argentina
Abstract
In this article we show that the linear reconciliation problem can be repre-
sented by a standard multiple linear regression model. The appropriate criteria
for redundancy, determinability and gross error detection are shown to follow
in a straightforward manner from the standard theory of linear least squares.
The regression approach suggests a natural measure of the redundancy of an
observation. This approach yields also an explicit expression for the probability
of detecting a gross error in an observation, which depends on its redundancy.
The criterion for the detection of gross errors derived from the regression model
is shown to yield the maximum probability of correct outlier identication. We
consider two examples analyzed in the literature to demonstrate how our ap-
proach allows a complete understanding of the main data features.
Key words: Determinability; Gross errors; Least squares; Linear model;
Redundancy.
1
1 Introduction
The problem of estimating the variables involved in a chemical process, subject
to linear balance equations, has been considered by several authors, in particular
Romagnoli and Stephanopoulos (1980), Crowe et al. (1983) and van der Heijden
et al. (1994a and b). For a complete account see Romagnoli and Snchez (2000).
Basic issues are whether: the estimation of an observed value can be improved by
using the other measurements (redundancy), an unobserved value is estimable
from the observed ones (determinability) and whether an observed value is a
gross error.
In this article we show that the observations constrained by the balance equa-
tions may be represented by a linear multiple regression model. This approach
has the advantage that the appropriate criteria for redundancy, determinability
and gross error detection follow in a straightforward manner from the stan-
dard theory of linear least squares: no ad-hoc methods are necessary, and the
procedures are very simple to derive and to apply.
Section 2 develops the regression model. In Section 3 a simple criterion
for the determinability of a variable or of a linear combination of variables is
derived from well-known results in regression theory. In Section 4 a measure of
redundancy is dened, based on the concept of leverage, which is widely used
in regression data analysis. Section 5 deals with gross error detection. The
criterion for the detection of gross errors derived from the regression model is
shown to yield the maximum probability of correct outlier identication. In
Section 6 we give an explicit expression for the probability of detecting a gross
2
error in an observation, which is shown to depend on its redundancy.. Finally
in Section 8 we consider two examples analyzed in the literature to demonstrate
how our approach allows a complete understanding of the main data features.
2 The linear model
Let the state of a plant operating under steady-state conditions can be de-
scribed by the vectors x 1
m
and u 1
n
of measured and unmeasured process
variables, respectively, which satisfy a system of j independent linear balance
equations
A
1
x +A
2
u = 0, (1)
where A
1
and A
2
are j:- and j:-matrices, respectively. The observations
vector is y = x + -, where - is a vector of random errors. It is desired to estimate
x and the elements of u which are estimable, and to detect gross errors in y.
The classical approach to this problem is based on the weighted least squares
estimate of x dened as the solution x of
(y x)
0

1
(y x) = min (2)
subject to (1), where is the covariance matrix of - and in general A
0
denotes
the transpose of A.
We shall now show that the data may be represented by a regression model.
Call A the j (: + :)-matrix and v the : + :-vector dened by
A =[A
1
A
2
] , v =
_

_
x
u
_

_
3
so that (1) is equivalent to Av = 0.
The set of vectors v such that Av = 0 is a linear subspace, called the null
subspace of A, which has dimension = :+:j. Let the vectors b
1
, ..., b
q
be
a basis of this subspace; then any vector v such that Av = 0 can be expressed
as a linear combination of these vectors, and therefore
v =
q

j=1
,
j
b
j
, (3)
for a set of unknown numbers ,
1
, .., ,
q
. Call B the (: + :) matrix whose
columns are the b
j
, and , the -vector with components ,
j
. Then (3) can be
written as v = B,. The matrix B (which is not unique) is easy to compute. In
the matrix languages Matlab and Gauss, B is obtained through the command
null(A). Otherwise B can be computed using the QR orthogonalization pro-
cedure.
Call B
x
and B
u
the :- and :matrices consisting of the rst : and
the last : rows of B, that is
B =[b
1
, ..., b
q
] =
_

_
B
x
B
u
_

_
. (4)
Then v = B, implies
x = B
x
,, u = B
u
,, (5)
and therefore
y = B
x
, + -, (6)
It follows from (6) that y is the response vector of a linear regression model
with predictor matrix B
x
and errors with covariance matrix . It is assumed
4
that the errors are independent and hence that is a diagonal matrix. Hence-
forth var (y) and var (j) will denote the covariance matrix of the random vector
y and the variance of the random variable j, respectively. Then we have
=var(-) =var(y) =diag(o
2
1
, ..., o
2
m
) =
2
, (7)
with =diag(o
1
, ..., o
m
), where o
i
= sd(-
i
) is the standard deviation of -
i
.
The errors in the model have possibly unequal variances. To transform it
to a standard regression model, call y
s
and e the vectors of observations and of
errors standardized to unit variances, with elements
j
s;i
=
j
i
o
i
, c
i
=
-
i
o
i
(i = 1, ..., :), (8)
that is, y
s
=
1
y and e =
1
-; and call C the matrix obtained by dividing
each row of B
x
by the respective o
i
, that is
C =
1
B
x
. (9)
Then (6) is equivalent to
y
s
= C, +e. (10)
Since the c
i
s are independent with unit variances, we have var(e) =var(y
s
) = I
m
where I
m
is the :-dimensional identity matrix. Hence the appropriate estimate
of

, of , is the ordinary least squares estimate dened as solution of
_
_
_y
s
C

,
_
_
_ = min, (11)
where |a| denotes the Euclidean norm of a. A solution is given by

, = C
+
y
s
,
where C
+
denotes the (Moore-Penrose) pseudo-inverse of C. The vectors of
5
reconciled values and of observation residuals are respectively
x = B
x

, = C

, and - = y x. (12)
The reconciliation problem is thus reduced to ordinary least squares estimation,
for which there is a vast literature (e.g., Montgomery et al., 2001; Weisberg,
2005). The existing body of knowledge on ordinary least squares will now be
exploited to handle the reconciliation problem.
Crowe et al. (1983) reduce the original problem to one without unmea-
sured variables by multiplying (1) by a projection matrix whose range is the
null subspace of A
2
(recall that we use the null subspace of A instead). Their
approach is equivalent to the Q-R approach proposed by Snchez and Ro-
magnoli (1996). It can be shown that our approach yields the same results as
theirs (see also Section 8).
3 Determinability
Let r = rank(C) = rank(B
x
); then r _ min(:, ). If r = (full rank), then

, is unique and C
+
= (C
0
C)
1
C
0
. In this case the unmeasured values are
estimated through (5):
u= B
u

,. (13)
If r < , then (11) has innite solutions, but the reconciled values x are the
same for all solutions

,.
In the terminology of linear regression, a parameter is estimable if it has a
linear unbiased estimate. Estimable values are the determinable ones as dened
6
by Romagnoli and Snchez (2000).
If r < , it can be shown that the estimable elements of u correspond to the
null columns of
D =
_
I
q
C
0
(C
0
)
+
_
B
0
u
. (14)
In general, a linear combination of the elements of u, = a
0
u where a
is a given :-dimensional vector is estimable if and only if
Da = 0, (15)
and its estimate is = a
0
u (to simplify the exposition, proofs of all mathematical
statements are deferred to Appendix A).
4 Redundancy
The relationship between the tted values y
s
and observed values y
s
in model
(10) is given by y
s
= C

, = Hy
s
, where
H = CC
+
(16)
is the so called hat matrix (because it relates the observed y
s
to the tted
y
s
hat). Then
j
s;i
=
m

j=1
H
ij
j
s;j
, (17)
and this implies
var( y
s
) = H. (18)
The diagonal element H
ii
of H is called the leverage of the i-th observation
in regression theory, and will be denoted for simplicity as /
i
. The name stems
7
from the fact that an observation with a large /
i
has a high weight in the
determination of the least squares t. The /
i
s fulll
0 _ /
i
_ 1 (19)
and
n

i=1
/
i
= rank (H) = rank (C) . (20)
It follows from (18) that
var( j
s;i
) =
var( r
i
)
var(j
i
)
= /
i
. (21)
According to the terminology of Romagnoli and Snchez (2000), r
i
is redun-
dant if it can be estimated using observations other than j
i
. It is shown that if
/
i
= 1, then j
s;i
= j
s;i
(or equivalently r
i
= j
i
) and hence the i-th measurement
is not redundant and therefore its residual is null. If /
i
< 1, it follows from (21)
that
var( r
i
) = /
i
var(j) < var(j
i
),
and hence r
i
is redundant. Moreover, /
i
measures the reduction in variance
gained with the use of j
j
with , ,= i, and hence /
i
can be taken as a measure of
non-redundancy. Besides, it is shown that 1 /
i
may be interpreted as the
proportion of the variability of j
s;i
contributed by the j
s;j
with , ,= i. For these
reasons we dene the redundancy of observation i as
1
i
= 1 /
i
. (22)
Since C depends on , so do the 1
i
s. This can be seen intuitively by noting
that if for a particular i we let o
i
0, then the corresponding 1
i
0, which
8
is natural since if an observation has no error, the other observations will not
improve on its estimation. However, it can be shown that exact non-redundancy
(i.e., 1
i
being exactly zero) does not depend on .
For the detection of multiple gross errors, Zhang et al. (2001) dene a
quantity that they call the reconciliation precision of variable i, which in our
notation would be expressed as o
i
/
i
.
5 Detection of gross errors
The residuals for model (10) are
e = y
s
C

, =
1
-, (23)
with - dened in (12). Call
o
res
= e
0
e =
m

i=1
c
2
i
(24)
the residual sum of squares. If : = r, then o
res
= 0 and no further analysis
can be performed. It will henceforth be assumed that the number of residual
degrees of freedom / = :r is positive.
Assume the errors -
i
to be normal. It is a standard result of multiple re-
gression theory that then o
res
has a chi-squared distribution with / degrees of
freedom. This fact may be used to carry a global test of t: if o
res
is larger
than say the 0.95 quantile of the chi-squared distribution, then the data do
not support model (10), with j-value less than 0.05.
This test does not signal the specic cause of mist. We shall now estimate
the location of a gross error. The residual vector e has covariance matrix I H,
9
and hence var( c
i
) = 1
i
. Call c
s;i
the residuals standardized to unit variance:
c
s;i
=
c
i
_
1
i
=
-
i
sd ( -
i
)
. (25)
The c
s;i
are standard normal (but not independent!). Call 1 the value of i
that maximizes [ c
s;i
[ . Then 1 estimates the location of the suspect value. Call

,
(I)
the estimate computed without using observation 1. Then we correct the
suspect value by using j
s;I
= c
0
I

,
(I)
(where c
0
i
is the i-th row of C) instead of
j
s;I
= c
0
I

,, and hence using r
I
= o
I
j
s;I
instead of r
I
. It is not necessary to
actually recompute the estimate: it can be shown (Belsley et al., 1980) that
r
I
=
r
I
/
I
j
I
1
I
. (26)
Note that c
s;i
is undened if 1
i
= 0.
When var (-) is diagonal (as is assumed in this article) the standardized
residuals c
s;i
coincide with the transformed residuals 7
i
dened by Mah and
Tamhane (1982, eq. (15)), which have certain optimality properties.
An alternative to the chi-squared test is to use the statistic
T = max
i
[ c
s;i
[ . (27)
Call G the distribution of T under the hypothesis of no gross errors. Then a
new test can be dened, by declaring an observed value of T signicant at level
c if it is larger than the (1 c)-percent point of G.
This procedure has a theoretical justication. Consider the model of a single
gross error, represented by
j
i
=
_

_
b
0
i
,+-
i
for i ,= i
0
b
0
i
,+o
i
+ -
i
for i = i
0
, (28)
10
where i
0
and are unknown and b
i
is the i-th row of B
x
. This is equivalent to
j
s;i
=
_

_
c
0
i
,+c
i
for i ,= i
0
c
0
i
,+ + c
i
for i = i
0
(29)
Then it is shown by Belsley et al. (1980) that the Maximum Likelihood Esti-
mates of i
0
and are 1 and j
s;I
c
0
I

,
(I)
respectively; and that T yields the
Likelihood Ratio Test of the null hypothesis = 0 against the alternative
,= 0 . It can be shown that the estimator 1 is optimal in that maximizes
the probability of correctly choosing i
0
.
The distribution G depends on C, and does not have an explicit form, but
useful bounds can be found. Let

G(t) = (2(t) 1)
m
where is the standard
normal cumulative distribution function. Then Sidk (1967) shows that
G(t) _

G(t) for all t. (30)
Call
j
T
= 1 G(T), j
T
= 1

G(T) (31)
the j-values of the test based on T and its approximation. Then j
T
_ j
T
,
and hence j
T
yields a conservative test. The bound above is accurate if the
rank r is small compared to the number of observations : (which is seldom the
case in chemical applications) but otherwise the test is conservative. The exact
distribution function G may be approximated by Monte Carlo simulation for
each particular C.
The test based on (27) was proposed by Narasimhan and Mah (1987), who
also pointed out the usefulness of Sidks bound (30).
11
Wang and Stephanopoulos (1983) dene a sum of squared residuals which
coincides with o
res
. They estimate the location of the gross error by searching the
observation whose deletion causes the largest decrease in o
res
. That is, for i =
1, .., : call o
(i)
the value of o
res
computed without using the i-th observation.
Then they look for the i such that o
(i)
is a minimum. Is is shown that
o
(i)
= o
res
c
2
s;i
, (32)
and hence minimizing o
(i)
is equivalent to maximizing [ c
s;i
[ . It can also be
shown that the compare vector approach of Van der Heijden et al. (1994b) is
equivalent to using 1 as dened above.
6 Calculation of detection probabilities
Assume model (29). Then the probability that the chi-squared test yields a
j-value less than c (the power of the test) is
= 1 1
k;
_

2
k
(1 c)

, (33)
where ` =
_
1
i0
, / = : r,
2
k
(c) is the c-quantile of the chi-squared dis-
tribution with / degrees of freedom, and 1
k;
is the distribution function of
the non-central chi-squared distribution with / degrees of freedom and non-
centrality parameter `
2
. Table 1 gives for c = 0.05 and a range of values of
/, 1
i
and . It follows that if 1
i
= 0.1, then the probability is very low even
for = 4. In the extreme case 1
i
= 0, we have r
i
= j
i
and hence detecting a
gross error is impossible.
12
In general, an observation with small o
i
will have a large /
i
and hence a low
1
i
. In fact, the i-th row of the matrix C is obtained by dividing the i-th row
of B
x
by o
i
, and this will yield large values of /
i
if o
i
is small. Intuitively, an
observation with a low error variance receives a high weight in weighted least
squares, which makes it more dangerous if it is a gross error.
According to (20), the average of the 1
i
s is 1 r,:. If for instance we have
: = 30 and r = 20 this average is 0.33 and so we cannot hope all redundancies
to be large enough to ensure a high detection probability.
The Table shows to be a decreasing function of /. The reason is that /
is the actual number of squares in o
res
, and if it is large, a single outlier will
cause only a relatively small increase in o
res
.
We now compare the power of the test based on j
T
dened in (31) to that
of the chi-squared test. The distribution of T in (27) seems very dicult to deal
with theoretically, but can be approximated through a Monte Carlo simulation.
We assume model (29) with normally distributed j
i
. Since the distribution of T
depends on C, one should have to run a simulation for each specic C. Since we
want a broad comparison, we have chosen to take C also random with a normal
distribution.
For given and test level c call
S
and
T
the powers of the tests based on
o
res
and j
T
respectively. We have computed these results for c = 0.05, = 3
and 4, : ranging between 10 and 60, and r,: ranging between 0.2 and 0.7
(recall that 1 r,: is the average redundancy). The results can be roughly
13
summarized by saying that j
T
is preferable to o
res
when r,: _ 0.4 or when
r _ 20.
Table 2 displays part of the results. To simplify the output, we show
(
T

S
) ,
S
, the relative gain in power obtained by using j
T
rather than
j
T
.
7 Detecting several gross errors
The former methods are adequate for the detection of a single gross error. Zhang
et al. (2001) and Wang et al. (2004) propose iterative strategies for dealing
with multiple gross errors. However, it must be noted that the fact that these
methods are based on least squares estimation which is sensitive to atypical
obbservations implies that several if not all of the gross errors may remain
undetected. This is a situation that is best handled through the so-called robust
estimates. There is a vast literature on robust reconciliation methods; we cite
Arora and Biegler (2001), zyurt and Pike (2004), Wang and Romagnoli (2003)
among many others. A recent general account of robust statistical methods is
given by Maronna et al. (2006). The advantage of using our regression setup for
robust reconciliation is twofold. First, existing algorithms and software devised
for general regression problems can be readily applied to reconciliation. Second,
and probably more important, the regression setup makes it possible to fully
understand the behavior and especially the limitations of a robust method
when applied to a specic case. This is the subject of a future article.
14
8 Examples
8.1 Example 1
We rst deal with Example 5.3 of (Romagnoli and Snchez, 2000, p. 82), which
has : = 8, : = 7 and j = 7. The rank of C is r = 7, and hence the resid-
ual degrees of freedom are / = 1. Table 3 gives the observed and reconciled
values j
i
and r
i
, the redundancies 1
i
and the standardized residuals c
s;i
. Our
results coincide with those derived by Romagnoli and Snchez using the Q-R
factorization, which is in turn equivalent to the approach by Crowe et al. (1983).
It is seen that ows )
3
, )
5
, )
8
and )
13
have no redundancy. The c
s;i
s cor-
responding to redundant ows dier only in sign. This is due to the fact that
/ = 1 and hence the residual vector remains in an one-dimensional subspace.
This fact implies that if some of these ows contains a gross error, even if the
test gives a signicant j-value it will be impossible to determine the source of the
error. Although this is a very small example, it points out to a basic diculty:
if / is small, the chances to locate a gross error will be small.
The matrix D of (14) has 8 rows. Table 4 shows D after omitting null and
repeated rows. It is seen that it has rank one. Flows )
6
and )
7
are determinable.
It follows from (15) that the sums )
i
+)
j
with i = 1,10,12 and , = 2,9 and the
dierences )
2
)
9
and )
i
)
j
with i, , = 1, 10, 12 are determinable.
15
8.2 Example 2
We now deal with the system in Case 3 of (Romagnoli and Snchez, 2000, p.
231), which has : = 28, : = 34 and j = 31. The rank of C and the residual
degrees of freedom are r = 20 and / = 8. Table 5 displays the results. Streams
13, 17, 20, 25, 30 and 36 are seen to have very low redundancies.
Some groups of streams like 15-16-17 and 30-31-32 have the same values of
the absolute standardized residuals [ c
s;i
[ . This implies that if one element of the
group is a gross error, its eect will be scattered over the whole group, making
the detection of its existence and its location more problematic.
The chi-squared test has j-value 0.025, indicating the existence of gross
errors. The largest [ c
s;i
[ corresponds to stream 26. The corrected value given
by (26) is 16.33. Reprocessing the data using the corrected value does not yield
a signicant j-value.
The matrix D is too large to be displayed here. Its examination readily
shows that streams 23, 33, 34, 35 and 48 are determinable. Inspection shows
also the determinability of the sums of the pairs of streams (3,4) and (55,56)
and of the dierences (10,11), (42,44), (46,47), (49,50), (50,51), (49,51), (58,59)
and (60,62). Actually, a frequently occurring pattern in D is to nd columns
that are equal or dier only in sign. It is easy to program a procedure to nd
those pairs automatically and thus avoid inspection of large matrices.
Our results again coincide with those of Romagnoli and Snchez. But we
wanted to demonstrate how displaying the redundancies and the standardized
residuals gives at a glance the necessary information to understand the main
16
data features, which are complemented by the matrix D.
9 Conclusions
The approach based on regression has yielded a straightforward derivation of the
criteria for redundancy and determinability, and for the detection of gross errors.
This approach allows the denition of the redundancy measure 1
i
, and makes it
straightforward to calculate the probability of detecting the presence of a single
gross error, which is seen to depend on the redundancy. These probabilities
are seen to be rather low in practical cases, especially for observations with
low redundancy. The redundancies and the standardized residuals from the
regression analysis allow at a glance a complete statistical analysis of the data.
10 Appendix A: Proofs of results
For reasons of space, only the essential steps of the proofs are presented.
Proof of (15)
Let = a
0
u = a
0
B
u
, for some :-dimensional vector a. If

, is not unique,
a linear combination of , is estimable (Stapleton, 1995) if and only if it de-
pends on , only through C, (recall that C

, is always unique even if

, is
not). Let d = B
0
u
a. Then the estimability of is equivalent to d = C
0
c for
some :-dimensional vector c. A solution to d = C
t
c is c =(C
0
)
+
d, and hence
B
0
u
a = C
0
(C
0
)
+
B
0
u
a, which is equivalent to (15).
The matrix H
17
It follows from (16) that H veries H = H
0
= H
2
, and hence
/
i
=
m

j=1
H
2
ij
= /
2
i
+

j6=i
H
2
ij
, (34)
which implies that /
i
_ 0 and /
i
(1 /
i
) _ 0, which is equivalent to (19); and
that if /
i
is either 0 or 1, then H
ij
= 0 for , ,= i.
It follows from (17) and (34) that
var
_

j6=i
H
ij
j
s;j
_
var ( j
s;i
)
= 1 /
i
,
and hence 1 /
i
represents the proportion of the variance of j
s;i
supplied by
j
s;j
with , ,= i.
It will now be shown that exact non-redundancy does not depend on .
The image of C is the subspace Im(C) of vectors equal to Ca for some a.
Assume that C is such that /
1
= 1. This is equivalent to H
1j
= 0 for , ,= 1, and
hence to Hv = v for v =(1, 0, 0, .., 0) . This is in turn equivalent to v Im(C) =
Im(B
m
) ; and this property does not depend on .
Optimality of 1
Note that each i
0
in (29) yields a dierent distribution for the observations
y. Assume that i
0
is chosen at random among 1, .., : with equal probabilities.
Estimating i
0
is then a typical problem in discriminant analysis (Seber, 1984). It
is known that the estimator that maximizes the probability of choosing the true
value of i
0
is given by choosing among the : distributions the one that attributes
the highest likelihood to the observed data, and this property is fullled by 1
since it is the Maximum Likelihood estimator.
Proof of (33)
18
If the random vector t is /-variate normal with var (t) = I
k
and expectation
j, then |t|
2
has a non-central chi-squared distribution with / degrees of freedom
and non-centrality parameter `
2
= |j|
2
. Under model (29), y
s
has identity
covariance matrix and expectation C,+g, where g =(q
1
, .., q
m
) with q
i0
= 1
and q
i
= 0 for i ,= i
0
. Hence e= (I
m
H)y
s
has var (e) = I
m
H and expectation
j with j =(I
m
H) g. Note that |j|
2
= 1/
i0
. Since I
m
H is an orthogonal
projection matrix of rank / = : r, it has / unit eigenvalues and : / null
ones. Call a
i
(i = 1, ..., /) the eigenvectors corresponding to the former, and let
t
i
= e
0
a
i
and t
i
= j
0
b
i
, so that
e =
k

i=1
j
s;i
a
i
, |t|
2
= |e|
2
= o
res
and j =
k

i=1
t
i
a
i
.
Then t =(t
1
, ..., t
k
) has var(t) = I
k
and expectation t, where t =(t
1
, .., t
k
)
has |t| = |j| , and hence |t|
2
= o
res
has a non-central chi-squared distribu-
tion.
Proof of (32): The result is proved in (Belsley et al., 1980). By the way,
since o
(i)
_ 0 for all i, it follows that the statistic (27) fullls T
2
_ o
res
.
11 Appendix B: Summary of results on the lin-
ear model
The standard linear model is y = C, +e, where y is an observations :-vector,
C is an : matrix of rank r, , is a vector of unknown parameters, and
e is a random :vector of errors with mean 0 and covariance matrix o
2
I. The
19
least squares estimate (LSE) is the vector

, that minimizes
_
_
_y C

,
_
_
_. It sat-
ises the normal equations C
0
C

, = y. If r = then

, is unique and is given
by

, = (C
0
C)
1
Cy. Otherwise there are innite solutions, which can be ob-
tained by

, = C

y where C

is any generalized inverse. In particular, the

(Moore-Penrose) pseudoinverse yields the solution with minimum norm. If e is
multivariate normal, then the LSE is also the Maximum Likelihood estimate.
In general, the Gauss-Markov theorem states that the LSE has minimum vari-
ances among all unbiased estimates. If r = the covariance matrix of

, is
o
2
(C
0
C)
1
.The vector of tted values is y = C

, = Hy where H = CC
+
(the
hat matrix). The vector of residuals e= y y has mean 0 and covariance matrix
o
2
(I H) . The matrix H is symmetric and idempotent, that is H = H
0
= HH.
If y is normal, then |e,o|
2
is chi-squared with :r degrees of freedom.
12 Notation
A
1
, A
2
: j :- and j :-matrices of constraints
A
0
, A
+
: transpose and (Moore-Penrose) pseudoinverse of A
B : (: + :) matrix of basis null subspace of A
B
x
, B
u
: rst : and last : rows of B
C : result of dividing the rows of B
x
by the o
i
s, with rows c
0
i
e: vector of raw residuals of standard regression model
c
s;i
: residuals standardized to unit variance
D: estimability matrix
20
G,

G: distribution function of T and its conservative approximation
/
i
: diagonal elements of hat matrix H
1: estimate of gross error location
r: rank of C
1
i
: redundancy of observation j
i
o
res
: residual sum of squares with residual degrees of freedom /
T: gross error test statistic
u: :-vector of unmeasured variables
x, x : :-vector of measured variables and its estimate
r
I
: reconciled value
y, - : observations and errors vectors
y
s
, e : y, - standardized to unit variances
, : vector of unknown regression parameters
- : vector of observation residuals
: covariance matrix of -
: power of the chi-squared test
o
i
= sd(-
i
) : standard deviation of -
i
: diagonal matrix with elements o
i
Acknowledgement: This research was partially supported by grants PIP
5505 from CONICET, and PICT 21407 and PAV 120 from ANPCyT, Argentina.
References
21
Arora, N. and Biegler, L.T. (2001). Redescending estimators for data recon-
ciliation and parameter estimation. Computers and Chemical Engineering, 25,
1585-1599.
Belsley, D. A., Kuh, E. and Welsch, R. E. (1980). Regression Diagnostics:
Identifying Inuential Data and Sources of Collinearity. New York: John Wiley
and Sons
Crowe, C. M., Garca Campos, Y. A. and Heymak, A. (1983). Reconciliation
of process ow rates by matrix projection. Part I: Linear case. American
Institute of Chemical Engineering Journal, 29, 881-888.
Mah, R. S. H. and Tamhane, A. C. (1982). Detection of gross errors in
process data. American Institute of Chemical Engineering Journal, 28, 828-
830.
Maronna, R.A., Martin, R.D. and Yohai, V.J. (2006). Robust Statistics:
Theory and Methods. New York: John Wiley and Sons.
Narasimhan, S. and Mah, R. S. H. (1987). Generalized likelihood ratio
method for gross error identication. American Institute of Chemical Engineer-
ing Journal, 33, 1514-1521.
Montgomery, D. C., Peck, E. A. and Vining, G. G. (2001). Introduction to
Linear Regression Analysis, 3rd Edition, New York: John Wiley and Sons.
zyurt, D.B. and Pike, R.W. (2004). Theory and practice of simultaneous
data reconciliation and gross error detection for chemical processes. Computers
and Chemical Engineering, 28, 381-402.
Romagnoli, J. A. and Stephanopoulos, G. (1980). On the rectication of
22
measurement errors for complex chemical plants. Chemical Engineering Science,
35, 1067-1081.
Romagnoli, J. A. and Snchez, M. C. (2000). Data Processing and Recon-
ciliation for Chemical Process Operations, London: Academic Press.
Snchez, M. and Romagnoli, J. (1996). Use of orthogonal transformations
in data classicationreconciliation. Computers and Chemical Engineering, 20,
483-493.
Seber, G. A. F. (1984). Multivariate Observations. New York: John Wiley
and Sons.
Sidk, Z. (1967). Rectangular regions for the means of multivariate nor-
mal distributions. Journal of the American Statistical Association, 62, 626-
633.
Stapleton, J. H. (1995). Linear Statistical Models. New York: John Wiley
and Sons.
van der Heijden, R. T. J. M., Heijnen, J. J., Hellinga, C., Romein, B. and
Luyben, K. Ch. A. M. (1994a). Linear constraint relations in biochemical
reaction systems: I. Classication of the calculability and the balanceability of
conversion rates. Biotechnology and Bioengineering, 43, 3-10.
van der Heijden, R. T. J. M., Romein, B., Heijnen, J. J., Hellinga, C. and
Luyben, K. Ch. A.M. (1994b). Linear constraint relations in biochemical reac-
tion systems: II. Diagnosis and estimation of gross errors. Biotechnology and
Bioengineering, 43, 11-20.
Wang, D and J. A. Romagnoli, J.A. (2003). A framework for robust data
23
reconciliation based on a generalized objective function. Industrial and Engi-
neering Chemistry Research, 42, 3075-3084.
Wang, F., Jia, X: Zheng, S. and Yue, J. (2004). An improved MTNT
method for gross error detection and data reconciliation. Computers and Chem-
ical Engineering, 28, 21892192.
Wang, N. S. and Stephanopoulos, G. (1983). Application of macroscopic
balances to the identication of gross measurement errors. Biotechnology and
Bioengineering, 25, 2177-2208.
Weisberg, S. (2005). Applied Linear Regression, 3rd. Ed.. New York: John
Wiley and Sons.
Zhang, P., Rong, G. and Wang, Y. (2001). A new method of redundancy
analysis in data reconciliation and its application. Computers and Chemical
Engineering, 25 941949.
24
Table 1: Rejection probabilities of the chi-squared test
1
i
resid.deg. fr. /
2 4 6
3 0.8 0.67 0.55 0.48
0.6 0.54 0.43 0.36
0.4 0.38 0.29 0.24
0.2 0.21 0.16 0.13
0.1 0.12 0.10 0.09
4 0.8 0.90 0.83 0.77
0.6 0.80 0.70 0.62
0.4 0.61 0.50 0.43
0.2 0.34 0.26 0.22
0.1 0.19 0.14 0.12
25
Table 2: Simulation results: relative gains by using the approximate test
: r,:
0.3 0.4 0.5 0.6
3 20 0.056 0.008 -0.031 -0.0143
40 0.309 0.263 0.061 0.008
60 0.514 0.375 0.159 -0.008
4 20 0.223 0.183 0.070 0.009
40 0.484 0.400 0.238 0.122
60 0.699 0.561 0.411 0.200
Table 3: Example 5.3 of Romagnoli and Snchez (2000)
Flowrates j
i
r
i
1
i
c
s,i
)
3
115.0663 115.0663 0
)
4
111.6730 109.6203 0.3386 1.284
)
5
53.3700 53.3700 0
)
8
0.8373 0.8374 0
)
11
66.0986 66.8670 0.1267 -1.284
)
13
95.7552 95.7552 0
)
14
116.5318 118.8308 0.3792 -1.284
)
15
77.857 76.9148 0.1555 1.284
26
Table 4: Example 5.3 of Romagnoli-Snchez: matrix D
)
1
)
2
)
6
)
7
)
9
)
10
)
12
0.2 -0.2 0 0 -0.2 0.2 0.2
-0.2 0.2 0 0 0.2 -0.2 -0.2
Table 5: Case 3 of Romagnoli-Snchez
Stream j
i
r
i
1
i
c
s;i
1 70.49 68.64 0.8790 0.5988
2 7.103 7.068 0.0164 0.5988
7 13.04 12.87 0.1024 0.8572
8 35.38 36.52 0.6771 -0.8572
9 53.21 52.05 0.9030 0.5078
12 23.90 23.66 0.2776 0.4833
13 0.0765 0.0767 0 0001 -0.7896
14 54.59 51.98 0.9031 1.146
15 12.78 10.88 0.4899 2.261
16 23.42 22.49 0.2402 2.261
17 0.2378 0.2154 0.0057 2.261
18 8.657 8.470 0.5546 0.6956
19 5.087 5.291 0.3866 -1.091
20 1.740 1.752 0.0046 -1.441
21 0.0255 0.0259 0 0009 -1.091
22 3.113 3.154 0.0773 -1.091
24 5.407 5.288 0.0306 2.261
25 2.898 2.880 0.0047 2.261
26 11.83 12.60 0.1715 -3.106
27 8.197 8.306 0.3313 -0.2528
28 1.364 1.368 0.0135 -0.2528
29 20.94 19.48 0.7375 1.622
30 1.051 1.063 0.0068 -1.637
31 12.58 13.18 0.3423 -1.637
32 4.999 5.230 0.1311 -1.637
36 5.730 5.906 0.0441 -2.785
45 4.250 4.504 0.0637 -2.785
52 16.34 18.74 0.6041 -2.785
27