Documente Academic
Documente Profesional
Documente Cultură
ORNL-4957
V"
3 ^Sb DSlSlbB ^
THE
C-3
NUMERICAL SOLUTION OF
ILL-CONDITIONED SYSTEMS OF
LINEAR EQUATIONS
Michael T. Heath
LIBRARY
LOAN
COPY
OAK
RIDGE
NATIONAL LABORATORY
OPERATED BY UNION CARBIDE CORPORATION FOR THE U.S. ATOMIC ENERGY COMMISSION
Energy Commission, nor any of their employees, nor any of their contractors,
subcontractors, or their employees, makes any warranty, express or implied, or
assumes any legal liability or responsibility for the accuracy, completeness or
usefulness of any information, apparatus, product or process disclosed, or
represents that its use would not infringe privately owned rights.
L_
ORNL-U957
UC-32 Mathematics and Computers
Michael T-
Heath
JULY 1974
Ill
ACKNOWLEDGMENTS
critical review and helpful suggestions, and to Bert Rust for many
discussions of the problems dealt with herein and particularly for his
creative collaboration on the method presented in Section III.
The
CONTENTS
Page
Abstract
Introduction
I.
II.
III.
IV-
V-
Ill-Posed Problems
Measurement
Mathematical Model
Discretization
Ill-Conditioning
^
6
7
Previous Methods
Constraints
Smoothing
11
1*+
Mathematical Programming
15
Constrained Estimation
19
Classical Regression
Suboptimal Estimation
Iterative Procedure
19
20
2U
27
Related Methods
27
29
An Example
31
Bibliography
35
Heath
ABSTRACT
are examined.
An extensive bibliography is
included.
INTRODUCTION
This does
present approach.
ture on regularization.
just enough to indicate some of the flavor and variety of concrete ways
and results from linear algebra and matrix theory are used freely without
specific reference.
matters, one of the several excellent books in this area [20, 35, 59,
8h, 106] should be consulted.
I.
ILL-POSED PROBLEMS
1.
Measurement
obscuration.
Similar circumstances occur in physics [50, 91, 93], meteorology [53, 98?
99, 103, 10U], astronomy [11, 15, hk, kf, 9h], optics [19, 31, 79], geo
physics [^-7, 37, hi, 63, 81], medicine [29], and many other fields.
2.
Mathematical Model
observed data, and the desired unknown function reflects the actual
phenomenon taking place.
J K(s,t)f(t)dt = g(s),
a
Then
side.
lim fK(s,t)sin(nt)dt = 0.
a
severely from blurring and biasing effects and reduces still further
the information content of measured data.
As evidenced
say that the integral operator does not have a bounded inverse.
3-
Discretization
If the weights
for the quadrature formula are denoted by w , j=l, ..., n, and the
mesh points chosen are
g(Si) = ^K(si,t;.)f(tj)wJ,
i-1, ..., m.
3=1
This can be more compactly written in matrix notation as Ax~b, where the
,n
mxn
= K(s.,t.)w.
v i
, ,
b.
l
\
/
X.
3
^m
and b eR
i=l.
. .,
m;
j=l,
..,
n.
= f(t.)
1
=g(s.)
It
approximations:
h.
Ill-Conditioning
tioned problems.
to error.
In
where e eR
When
ll*-*ll2
(i-i)
, , IMI2
where x=A b is the solution to the original system and x=A (b+e)
is the solution to the perturbed system.
Thus, ill-conditioning is
asociated with a large value for Cond (A), since in this case the bound
on the error in x is very large.
is small.
a.' .
db
lj
where ja!.j =A~ . Here again the large magnitudes of the elements of
the inverse matrix indicate great sensitivity of the solution to the
data.
For example,
st
).
some are better than others, they are all poorly conditioned.
11
II.
PREVIOUS METHODS
1.
Constraints
An obvious
a priori.
2.
Smoothing
[51, 52, 87, 88, 101], smoothing [68, 95, 9, 80], regularization
[89, 90], relaxation [76], or stabilization [78, 82].
The central idea of such methods is to replace the minimization
problem
T{!|a*-W|}
12
by the problem
*f {||Ax-b|| +K||x||'},
(2-1)
where I[I|' is possibly a different norm from ||.||, and the positive
scalar A. is a relative weighting factor.
By simultaneously minimizing
the residual and some norm of the solution, oscillations in the solu
lead to the norms ||x| |' = ||a.x|| , and the minimization problem (2.1)
becomes
jn{||Ax-b||2+x||Aix||2|.
mm
x
The normal equations for this least squares problem are given by
T
(A A+\ ATA- )x = A b.
13
subsumes the case where the Euclidean norm is employed for both the
residual and the solution by simply taking A0=I , the nxn identity
matrix.
where X. , X
However, if X. is taken
too large, then the solution is excessively damped and the original
equation to be solved has little influence on it.
The arbitrariness
there is no indication how much the solution could vary and still remain
Ik
3-
<y.
^ 0.
and A A,
If 8,
<DoK* =>.
or
x = V(Df,0)UTb,
t
where D =diag(s
-1
-1
,...,&
,0,...,0).
(2.2)
T,
x. I -|_ v.,
^ uib
i=l
Thus,
the smaller singular values entering into the denominator of the terms
of the expansion tend to greatly magnify any error in the data vector b,
15
gested [27, 28, 100, 8] that the expansion simply be cut off before
the contamination due to the small singular values creeps in.
This
does produce a well behaved solution but again, like the damping factor
X discussed earlier, entails a somewhat arbitrary judgment as to when
to terminate the summation.
k.
Mathematical Programming
It has already been observed that the classical bound (1.1) is exces
sively pessimistic and of little practical value in the presence of
ill-conditioning.
16
-l i . :i
The solution
side conditions.
ii O
C = I
-11
-:ri
17
is one whose unit ball is a polytope, such as the max-norm |||| or the
sum-norm j|> ||,.
|Ax-b| |e|.
T
sw xf
subject to x g0,
Ax gb - IeI,
-Ax s _b - |e|
and
max/w TxtI
,. , ,
subject to x l 0,
Ax ^b + IeI,
-Ax -b + |e|.
18
The
source of uncertainty.
:iw xi or min<w xf
max<
12
It can be
It is therefore the
aim of the next section to develop methods for the quadratic problem
which, like the smoothing and singular value methods discussed earlier,
require little more work than simple matrix inversion techniques but,
unlike those methods, produce error bounds as well.
19
III.
CONSTRAINED ESTIMATION
1.
Classical Regression
defined by
-2
constant.
is a positive read-
From the
ellipsoid E
*
defined by
,
and
min I T I
cp, =
wx .
lo
xeE I
J
n
From the theory of linear regression it is well known that the minimum
residual
, ,T -2
|(Ax-b)T
S"2 (Ax-b)i
^x-o;f
min L
r0 = x ^AX_D^ b
s^
is attained when
20
-2
solution x is (A S
A)
8=[(^2-r0)wT(ATS-2A)tw]*,
the bounds are given by
TA
cp
=wx + 8,
^up
'
cp-,
^lo
TA
=wx-8.
2.
Suboptimal Estimation
problem when cp is estimable; and this is always the case if the rank
r of the matrix A is full, r=n.
the column space of A
-2
-2
A and
behaved, if A
-2
may be well
determined by
Near singularity
21
and severe ill-conditioning are the rule rather than the exception in
allow
solutions obtained to roam far afield from expected results and yet
remain within E
.
n
B = ixeRn :p gx ^ql.
Assuming that the solution must lie in BflEn narrows the interval
[cp-,
,cp
using the l
of least volume.
22
Such a combination is
D=|xeRn :T\.\ (Ax -b)T s"2 (Ax -b) +(1-7)) '\ (x -d)T Q-2 (x -d) Sll,
where 0 ^ 7] == 1. Note that CHE CD, so that D ^ f> whenever CHEn 0.
The ellipsoid D can be represented more compactly in partitioned matrix
form by defining
/l-j\ 2
T = r
' S 2
-2
V " = I ^
o
W
' v = o iio-2
Q
n ^
so that
(
n
D= ^xeR
: (Rx - y) T V 2 (Rx - yM t 2 + n21?
ellipsoids, t =0 giving E
and t = corresponding to C2
Assume
Strategies
The new suboptimal problem has now been reduced to the form
of a classical linear regression
axf T )
max
cp
up
^<w x>
eDl
'
cp,
\Lo
min
T \
w x(
xeDl
Specifi
23
min (.
,1
-2,
* 2
is attained when
x = (rWVvSt .
2
again given by
with
8
[(t2+M-2-P0)wT(rV2R)-1w]s
R V
-2
(3-1)
V_1R =w(K)
x = (U_1,0)z,
T -2
|
P0 = yV
y - z11IQn 0lz,
[, 2
-1
2k
3.
Iterative Procedure
rep,
L lo
, cp
upJ
At least some of these are hopefully sharper than the original values p
and q.
After the
first step any improvements in the bounds can be incorporated into the
estimates p and q and the process repeated.
In practice the
quite be reached.
fit [U2].
After the first step subsequent iterations are not as expensive
the matrix V
m rows, S
Furthermore, since
25
these lower rows are nonzero only on the diagonal, a simpler orthogo
nalization technique, such as Givens transformations, can be employed
to complete the factorisation.
-1
-T
, cp
Initial estimates p and q for the bounds on the solution are needed
to begin the iteration.
gives p S 0.
This automatically
m'
inequalities [7^]
x.
3
min
(Ax). )
\ [ , j. = .
1, ...,n
1 < i S m ) a. .
1.1 I
(5.2)
for the components of the solution vector x, and these can be used to
26
them as T varies.
as E
if T is very large, then C dominates the combination and the new bounds
for the solution cannot be expected to differ much from the original
ones which determine C.
and may be sought by trial and error or determined from detailed know
This is
because at any stage the best previous information is retained for use
in the next step.
Thus,
27
IV.
1.
Related Methods
in the previous section has some important similarities and some impor
tant differences with other techniques for ill-conditioned linear systems.
There is an obvious kinship with the methods (2.1), the parameter t being
analogous to the weighting or damping factor X.
solutions.
(U.l)
H = A1A1-
=x
= ... =x
when
[85, 86], the stochastic extension method of Franklin [22], and the
ridge regression method of Hoerl and Kennard [32-3U].
Another technique
along these lines was developed by Backus and Gilbert in the context of
28
rather than truncate, the singular value expansion for the solution.
He also goes a step further by defining a doubly relaxed least squares
solution, given by
[T
latory behavior of the true solution starts to creep into the iterates,
A number of authors [1, 1+6, 88] have pointed out the relationship
of the damping methods to the Moore-Penrose pseudoinverse summarized
by the formula
= 11m(A A + XI)
X-0
-1
29
by Foster [21].
The idea of splitting the solution space into two subspaces, one
where the problem is well conditioned and the other where it is poorly
conditioned, has been explored in several papers [16, 18, 25, 30]
but without conspicuous practical success, unless one includes the
2.
Throughout this paper the elements of the matrix A have been tacitly
assumed to be sharply defined constants, subject to no uncertainties.
However, in an experimental situation the calibration runs which deter
mine the kernel matrix may be as liable to instrumental error as later
data gathering runs.
by the fact that the rank of the matrix is poorly determined numerically
30
equations.
An approach
closer in spirit to that of Section III is [17]Whatever method is used in solving a system of linear equations,
ideally one would like to know under what conditions the solution it
Too
little attention has been given to this aspect of the problem by pro
posers of techniques for ill-conditioned systems, including the present
author.
research.
31
V.
AN EXAMPLE
6JT/2
8jVz\
8^1/2
.6/172 J, b=
7172
yi/2 /
Z1
2
\3
u2 = 0.8636,
This simple system does not exhibit the
above.
Furthermore,
damped least squares method (!+.l) with H=I and the truncated singular
value expansion (2.2).
of the two methods is traced as the parameter X and the rank k, respec
tively, vary.
from [110].
and 0^Y = 1, mean that the solution is obtained by summing the first k
terms of the singular value expansion plus the fraction y of the k+1
term.
32
3 P
Lk=\)
.4
10
DAMPED
-1
LEAST
SQUARES
(0<X<>)
LINEAR REGRESSION
(X = 0,* = 2)
-3
4
f-
Figure 1. The Loci of Solutions for the Damped Least Squares and
Singular Value Methods as Their Respective Parameters Vary.
33
and the
n
positive quadrant.
Furthermore, no
t=0.
and these were used as initial values for the iterative procedure.
3k
best strategy is to use t=2 for several iterations, then take t=1-5
The initial and terminal stages of this process are shown in Figure 2.
If a very small or zero value for t is used initially, then p,
and q~ assume their optimal values very quickly, but q
beyond its starting value.
at present.
never improves
varied so that all the bounds are improved at about the same rate in order
to obtain the best ultimate result.
radical change in one or two component bounds can cause the approximating
However,
they can be considered a worst case analysis in the sense that neither
the vector p nor q is actually a solution to the problem.
To get even
more precise limits on the constraint region as a whole, one can estimate
T
gives the interval estimate x.. +x? e [2.932, 1+.353]- Adding this to the
previous information gives the final picture shown in Figure 3.
35
ORNL- DWG
Figure 2.
74-3785
36
ORNL- DWG
74 - 3786
5
4
3
2
1
x2 0
-1
HI\i
6
10
-2
-3
-4
-5
Figure 3-
37
BIBLIOGRAPHY
1.
2.
Correction Problems," Ann. Acad. Sci. Fenn. Ser. A VI, 255 (1967),
pp. 1-17-
3-
k.
5.
Gross Earth Data," Philos. Trans. Roy. Soc. London Ser. A, 266
(1970), pp. 123-192.
8.
9.
38
13-
Ik.
15-
17.
18.
19.
20.
21.
22.
25.
39
26.
27.
pp. 205-22U.
28.
29.
15 (1972), pp.
30.
31-
32.
Biased
3U.
35-
37-
38.
39-
152-171.
1+0
40.
kl.
4-2.
1+3.
kk.
1+5.
1+6.
47.
49.
54.
41
55.
57.
58.
59.
60.
Oettli, W., "On the Solution Set of a Linear System with Inaccurate
Coefficients," SIAM J. Numer. Anal., 2 (1965), PP- 115-118.
61.
64.
Payne, L. E., "On Some Non Well Posed Problems for Partial
Differential Equations," Numerical Solutions of Nonlinear
Differential Equations, D. Greenspan, ed., John Wiley and Sons,
66.
67.
42
69-
70.
73.
74.
75-
76.
77.
Scheffe, H., The Analysis of Variance, John Wiley and Sons, Inc.,
New York, 1959-
78.
79.
80.
81.
43
84.
85.
86.
88.
89.
90.
91.
95.
96.
97.
44
98.
99*
102.
104.
105.
106.
108.
109.
110.
111.
112.
45
113-
114.
Melo, 0. T. and C
116.
47
ORNL-4957
UC-32 Mathematics and Computers
INTERNAL DISTRIBUTION
G.
Alsmiller
94.
R.
A.
McLaren
w.
Armstrong
95-
G.
s.
McNeilly
3
4
E.
Arnurius
96.
F.
L.
Miller
J.
Beauchamp
97-
T.
J.
Mitchell
o.
Bowman
98.
G.
w.
Morrison
A.
Brooks
99.
F.
R.
Mynatt
R.
100.
w.
Nestor
101.
M.
L.
Newman
A.
Busing
Carpenter
P.
Carter
102.
R.
v.
O'Neill
103.
M.
R.
Patterson
'104.
R.
w.
Peelle
8-32
33
34
J.
Chang
35
E.
Clifford
Coleman
36
R.
37
38
39
40
R.
41
Coveyou
Crowell
105-
M.
Reeves
106-115-
B.
w.
Rust
s.
K.
Penny
P.
St ansbury
L.
T.
Stewart
s.
L.
Culler
E.
Fields
116.
117.
118.
H.
Fowler
119.
Herman Postma
42-51
E.
Funderlic
120.
J.
G.
Sullivan
52
53
H.
Gard
121.
R.
E.
Textor
A.
Gardiner
122.
J.
N.
Tunstall
54
A.
Giles
55
J.
Gray
123.
124.
56 .
57-81 -
J.
Handley
T.
Heath
125.
126.
D.
Turner
v.
R.
Uppuluri
p.
R.
Vanstrum
s.
B.
Watson
82 .
w.
Henderson
127.
T.
A.
Welton
83 -
F.
Hibbs
128.
G.
w.
Westley
84 -
R.
Hicks
129.
G-
E.
Whitesides
85 .
J.
Hinton
130.
D-
G.
Wilson
Holcomb
131.
132.
G-
Young
A.
Zucker
86 -
D.
87 -
88 -
Householder
K. Johnson
89 .
M.
Larson
90
91
92
93
P.
Leinius
s.
133-135.
136.
137-156.
Maienschein
157-
Laboratory Records
Laboratory Records - Record Copy
B.
Mankin
158.
ORNL
F.
Maskewitz
Patent Office
EXTERNAL DISTRIBUTION
159-
160.
Corvallis, Oregon
97331
Turku, Finland
161.
162.
92038
48
l63
164.
Texas
165.
60440
166.
167.
168.
90007
Massachusetts 02138
169.
170.
171-
Stockholm, Sweden
New York
172.
14850
Tennessee
173.
17^.
37830
37901
175.
176.
Charlottesville, Virginia
Texas
177-
78712
Ohio
178.
44240
179-
37916
180.
47907
181.
182.
10598
183-
22903
78712
184.
185.
53706
87544
186.
187.
Illinois
New York
60637
14226
188.
189.
Connecticut
06520
Murray, Kentucky
42071
kg
190.
191-
192.
193-
19*+
195-
196.
197-
198.
199-
200.
201.
203.
204.
91109
10012
202.
California
92037
Madison, Wisconsin
205-
92037
53706
53706
206.
207.
Jersey 07974
E. R. Hansen, Lockheed Aircraft Corp., Palo Alto, California
94301
208.
209210.
211.
212.
213.
19711
10598
214.
215*
50
216.
217.
218.
219.
Berkeley, CA 94720
220.
221.
222.
Pasadena, California
9H09
223.
224.
225.
Kentucky 42071
G- Kreikebaum, Environmental Systems Corp., Box 2525, Knoxville,
Delaware
19711
Tennessee
226.
37901
227.
91103
228.
229.
9H03
230.
231.
California
232.
10012
233.
234.
235.
236.
237-
60440
Madison, Wisconsin
53706
238.
239-
240.
Virginia
23665
Berkeley, California
94720
51
241.
242.
53706
243-
244.
Knoxville, Tennessee
245-
37916
60439-
246.
247.
Sewanee, Tennessee
Madison, Wisconsin
37375
53706
248.
249.
20742
253.
Indiana 47907
J. M- Rinzel, Bldg. 12A, Room 204l, National Institutes of
Health, Bethesda, Maryland 20014
E. Y. Rodin, Dept. of Appl. Math, and Computer Science, Wash
ington University, St. Louis, Missouri 63130
J. N. Rogers, Division 8321, Sandia Laboratories, P. 0. Box 969,
Livermore, California 9^550
D. J. Rose, Mathematics Dept., University of Denver, Denver,
254.
255-
250.
251-
252.
Colorado
256.
80210
02139
257-
258.
259.
California 94550
H. W. Schmitt, Environmental Systems Corp., Box 2525, Knoxville,
260.
261.
262.
Switzerland
Tennessee
37901
Madison, Wisconsin
Connecticut
06520
Knoxville, Tennessee
263.
53706
37916
264.
265.
37901
37916
52
266.
267.
268.
37916
269.
02139
270.
271.
272.
273.
Pittsburgh, Pennsylvania
New Mexico
15213
87544
40506
275.
Finland
276.
15213
08540
277.
278.
279.
280.
281.
44242
Virginia
23665
Knoxville, Tennessee
282.
37916
53706
283-
284.
285.
Tennessee
Colorado
37901
80210
287.
10598
22903
78712
53
290.
291.
292.
ATTENTION:
293.
296.
Technical Librarian
297.
98124
Technical Librarian
295.
07974
Technical Librarian
294.
Technical Librarian
78712
ATTENTION:
Technical Librarian
10036
ATTENTION:
Technical Librarian
298.
299.
300.
ATTENTION:
Technical Librarian
301.
302.
303.
ATTENTION:
Technical Librarian
Technical Librarian
304.
92112
22202
ATTENTION:
Technical Librarian
ATTENTION:
Technical
Librarian
305.
91103
ATTENTION:
Technical Librarian
306.
307.
308.
Technical Librarian
309.
310.
311.
312.
48104
ATTENTION:
Wisconsin
ATTENTION:
313.
53706
Technical Librarian
ATTENTION:
Technical Librarian
Technical Librarian
Technical Librarian
94305
54
314.
315.
316.
20234
Technical Librarian
Technical Librarian
318.
ATTENTION:
317-
23665
Technical Librarian
20910
Technical Librarian
20390 ATTENTION:
Technical Librarian
319-
320.
321.
322.
323.
324.
California
90278
94205
ATTENTION:
ATTENTION:
Technical Librarian
Technical Librarian
10598
ATTENTION:
Technical Librarian
325.
326.
327-503.