Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    Only $11.99/month after trial. Cancel anytime.

    The Theory of Matrices in Numerical Analysis
    The Theory of Matrices in Numerical Analysis
    The Theory of Matrices in Numerical Analysis
    Ebook560 pages3 hours

    The Theory of Matrices in Numerical Analysis

    By Alston S. Householder

    3.5/5

    ()

    Read preview

    About this ebook

    This text explores aspects of matrix theory that are most useful in developing and appraising computational methods for solving systems of linear equations and for finding characteristic roots. Suitable for advanced undergraduates and graduate students, it assumes an understanding of the general principles of matrix algebra, including the Cayley-Hamilton theorem, characteristic roots and vectors, and linear dependence.
    An introductory chapter covers the Lanczos algorithm, orthogonal polynomials, and determinantal identities. Succeeding chapters examine norms, bounds, and convergence; localization theorems and other inequalities; and methods of solving systems of linear equations. The final chapters illustrate the mathematical principles underlying linear equations and their interrelationships. Topics include methods of successive approximation, direct methods of inversion, normalization and reduction of the matrix, and proper values and vectors. Each chapter concludes with a helpful set of references and problems.
    LanguageEnglish
    PublisherDover Publications
    Release dateJun 18, 2013
    ISBN9780486145631
    The Theory of Matrices in Numerical Analysis

    Related to The Theory of Matrices in Numerical Analysis

    Titles in the series (100)

    View More
    View More

    Related ebooks

    Mathematics For You

    View More
    View More

    Related podcast episodes

    Related articles

    • Article

      Physicists Attack Math’s $1,000,000 Question

      Apr 4, 2017

      Physicists are attempting to map the distribution of the prime numbers to the energy levels of a particular quantum system.

      4 min read

    • Article

      Math After COVID-19

      Apr 28, 2020

      Math After COVID-19
      4 min read

    • Article

      Graph Theory In A Nutshell

      May 5, 2020

      A graph G(V,E) is a finite, non-empty set of vertices V (or nodes) and a set of edges E. If the edges E that have the form of (a,b) are an ordered pairs of vertices, then we have a directed graph. In contrast, if the edges E are unordered pairs of ve

      1 min read

    • Article

      The Universe Began With a Big Melt, Not a Big Bang

      Oct 5, 2017

      There are two tantalizing mysteries about our universe, one dealing with its final fate and the other with its beginning, that have intrigued cosmologists for decades. The community has always believed these to be independent problems—but what if the

      10 min read

    • Article

      Visualising Experimental Results

      Mar 10, 2020

      All the examples in the main part of this tutorial generated results purely within LibreOffice Calc, using the laws of maths. However, it is also useful for creating graphs of experimental results, perhaps combined with equivalent results from simula

      1 min read

    • Article

      The Deepest Uncertainty

      May 16, 2019

      Georg Cantor died in 1918 in a sanatorium in Halle, Germany. A pre-eminent mathematician, he had laid the foundation for the theory of infinite numbers in the 1870s. At the time, his ideas received hostile opposition from prominent mathematicians in

      4 min read

    • Article

      Babbage And Running His Many Engines

      Nov 16, 2021

      Mike Bedford Like most of us, Mike tended to think of the likes of Alan Turing and John von Neumann as the pioneers of computing, so delving further into Charles Babbage’s creations was truly inspiring. Discussion about the first ever computer often

      11 min read

    • Article

      How to Build a Search Engine for Mathematics: The Surprising Power of Neil Sloane’s Encyclopedia of Integer Sequences.: The surprising power of Neil Sloane’s Encyclopedia of Integer Sequences.

      Oct 22, 2015

      On the average summer Saturday, the mathematician Neil Sloane woke up to a crisis. “There are always crises,” he said— albeit crises of the teapot tempest variety. One Saturday over breakfast, he faced an inbox message titled “edits from outer space.

      10 min read

    • Article

      Are Neural Networks About to Reinvent Physics?

      Nov 21, 2019

      Can AI teach itself the laws of physics? Will classical computers soon be replaced by deep neural networks? Sure looks like it, if you’ve been following the news, which lately has been filled with headlines like, “A neural net solves the three-body p

      9 min read

    • Article

      The Deepest Uncertainty: When a hypothesis is neither true nor false.

      Jun 6, 2013

      Georg Cantor died in 1918 in a sanatorium in Halle, Germany. A pre-eminent mathematician, he had laid the foundation for the theory of infinite numbers in the 1870s. At the time, his ideas received hostile opposition from prominent mathematicians in

      4 min read

    • Article

      Microcontrollers In Amateur Radio

      Aug 1, 2022

      My way of teaching about program data has always been a little different than the way most approach the subject. As you may know, pointers in C are a special type of variable that allows you to access data in a very efficient manner. Indeed, many com

      6 min read

    • Article

      How Math’s Most Famous Proof Nearly Broke

      Dec 20, 2018

      Andrew Wiles gave a series of lectures cryptically titled “Modular Forms, Elliptic Curves, and Galois Representations” at a mathematics conference in Cambridge, England, in June 0f 1993. His argument was long and technical. Finally, 20 minutes into t

      8 min read

    • Article

      How Math’s Most Famous Proof Nearly Broke: Andrew Wiles thought he had a solution to an age-old puzzle. Until it began to unravel.

      May 28, 2015

      Andrew Wiles gave a series of lectures cryptically titled “Modular Forms, Elliptic Curves, and Galois Representations” at a mathematics conference in Cambridge, England, in June 0f 1993. His argument was long and technical. Finally, 20 minutes into t

      8 min read

    Related categories

    Reviews for The Theory of Matrices in Numerical Analysis

    3.5/5

    3 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      The Theory of Matrices in Numerical Analysis - Alston S. Householder

      Index

      CHAPTER 1

      Some Basic Identities and Inequalities

      1.0. Objectives; Notation. In this chapter and the next will be developed some of the basic tools and relations that will be utilized repeatedly in subsequent chapters. At the outset a matrix of quite simple form will be introduced. It is the result of subtracting a matrix of rank 1 at most from the identity matrix, and any matrix of this form will be called here an elementary matrix. Such matrices will play a fundamental role in inversion, and also in the reduction of a matrix by similarity transformations to facilitate the evaluation of its characteristic roots. These will be applied immediately to obtain certain factorization theorems, which are of basic importance both for inversion and for equation solving.

      The method of least squares can be interpreted geometrically as the projection of an arbitrary given vector upon a certain subspace; and each step in certain iterative methods can likewise be interpreted as a projection. Hence a general form of a projection operator will be obtained, and the formation of this operator makes some use of the factorization theorems just, obtained. A further application of the factorization theorems will be made in the following section, where certain classical determinantal identities and inequalities will be derived. These will be required in the analysis and derivation of certain iterative methods, and also for more general theoretical considerations.

      In the last two sections of this chapter will be introduced certain polynomials associated with an arbitrary matrix and vector. One class is defined by an algorithm due to Lanczos who introduced it originally as a first step toward finding the characteristic values and vectors of a matrix; indeed, the algorithm is much used for this purpose. But the same algorithm occurs in the method of conjugate gradients for solving a linear system of equations [Chapter 5], and in the least-squares fitting of polynomials to experimental data [Exercise 54]. For this reason, and in order not to break the continuity later on, the Lanczos algorithm is developed here, instead of in Chapter 6 where it might seem more appropriate. Another advantage of having the Lanczos algorithm here is the following: Although the reader is assumed to be acquainted with the Jordan normal form of a matrix, its derivation from the tridiagonal form to which the Lanczos algorithm leads is quite direct. In fact, it is necessary only to verify that a certain matrix that will be exhibited in Chapter 6, does, in fact, transform a tridiagonal matrix to the Jordan form. Hence the reader who wishes to do so may follow this path at the outset to formulate a constructive derivation of the Jordan normal form of an arbitrary matrix. Even the Cayley-Hamilton theorem is a simple consequence, and is not presupposed.

      The polynomials to which the Lanczos algorithm leads, and which may be called the Lanczos polynomials, are closely related to another set which will be called orthogonal, for reasons that will be brought out at the end of the chapter. The two sets can coincide when the matrix is Hermitian. The orthogonal polynomials are also defined implicitly by a certain algorithm to be discussed more fully in Chapter 6, but they will find application in Chapter 3 in connection with the localization of the characteristic roots of a matrix.

      Certain notational conventions will be observed throughout this book, and for the sake of continuity these will be listed here.

      Except for dimensions and indices, or when otherwise indicated, lower case Greek letters represent scalars; lower case Latin letters column vectors; capital letters, Greek or Latin, matrices. However, the notation Cn,i will be used for the number of combinations of n things i at a time. The superscript H (for Hermitian) is for conjugate transpose, T for transpose, C for conjugate, and A for adjoint. In Section 1.3 a general reciprocal will be defined to be denoted by the superscript I. Since the general reciprocal of a nonsingular matrix is the true reciprocal, the same superscript will sometimes be used to denote the inverse. In general ai is the ith column of A, αij the element in position (i, j), but ei is the ith column of the identity I, and

      e = Σei.

      The elements of x are ξi; the elements of y are ηi.

      The matrix | A | has element | αij |, and the vectors | x | has elements | ξi |, that is, the absolute values of the elements of A and x itself;

      Particular matrices are I,

      The determinant of A is δ(A), and the trace of A is τ(Ais the determinant whose elements are taken from rows i, j, . . . and from columns k, l, ⋯ . The proper values of A (these will often be called the roots of A) are λi(A), often ordered

      ∣λ1(A) ∣≥ · · · ≥ ∣ λn(A) ∣;

      the statement λ = λ(A) signifies that λ is one of the proper values; the singular values of A are Ϭi(A) ≥ 0 where

      σi²(A) = λi(AHA);

      the spectral radius of A is

      If λ = λ(A), then x belongs to λ in case x ≠ 0 and

      Ax = λx.

      In that case x is a proper vector. But y ≠ 0 is a princpial vector belonging to λ if for some v,

      (A − λI)vy = 0

      and v is its degree. A proper vector is therefore a principal vector of degree 1.

      1.1. Elementary Matrices. A matrix of the form

      (1)

      will be called elementary. Since

      (2)

      it follows that for

      (3)

      By an element-wise expansion it is easy to verify that the determinant has the value

      (4)

      Let (P, Q) be a nonsingular matrix of order n such that PHQ = 0. Thus the columns of P and Q are bases for mutually orthogonal and complementary subspaces. Then for any vector a and any vector v such that vHa ≠ 0, it is possible to choose σ and u so that for any vector b such that

      (5)

      it is true that

      (6)

      This is to say that a is transformed by E into a vector b which is orthogonal to the space of Q, and whose projection on the space of P is the same as that of a. To see this, one has only to multiply through (5) by PH and QH, apply (6), and observe that u satisfies the equations

      whose matrix (P, Q)H is nonsingular, by hypothesis.

      A matrix, or the transpose of a matrix

      (7)

      will be called an elementary triangular matrix, and

      (8)

      A matrix

      (9)

      will be called an elementary permutation matrix. Evidently

      (10)

      The effect of multiplying a matrix on the left by Iij, i ≠ j, is to interchange ith and jth rows; that of multiplying on the right is to interchange columns.

      An elementary Hermitian (or unitary) matrix is of the form

      (11)

      Such a matrix is Hermitian, unitary, and also involutory

      (12)

      It reflects the space in the hyperplane orthogonal to w. If a and b are vectors of equal length, aHa = bHb, and such that aHb = bHa (i.e., is real), then it is always possible to choose w so that

      Ha = b.

      In fact, if aHb is real, set

      2μw = a − b,

      and obtain the real scalar 2μ by normalizing a − b. It is verified that μ = wHa = −wHb. This means that the reflection is made in the hyperplane bisecting the angle from a to b. Analogously, any components of a can be held fixed and the reflection made in the complementary subspace if b lies in that subspace.

      1.2. Some Factorizations. By applying the orthogonality relations (1.1.7) it is verified directly that

      (1)

      Since any unit lower triangular matrix (i.e., one in which every diagonal element = 1) can be written in the form indicated on the right, it follows that any unit lower triangular matrix can be factored as in (1). Since the transpose of a lower triangular matrix is upper triangular the factorization of a unit upper triangular matrix follows immediately. A unit upper triangular matrix can also be written

      R = I − r2e2T − r3e3T − ...,   eiTrj = 0,   i ≥ j;

      an alternative factorization is

      R = ⋅⋅⋅ (I − r3e3T)(I − r2e2T),

      the transpose yielding an alternative factorization of a unit lower triangular matrix. Any nonsingular triangular matrix can be written as the product of a diagonal matrix by a unit triangular matrix. Hence any nonsingular lower triangular matrix can be factored as a product of elementary Lower triangular matrices and a diagonal matrix.

      Let

      (2)

      be any matrix of n rows and m columns. Certain algorithms will be developed for expressing A in the form

      (3)

      where, if p is the rank of A, then M is a matrix of p linearly independent columns and P a matrix of p linearly independent rows. Let j. That is, the element in position i, the element in position (l, j1) is nonnull, and for some L1−1, possibly the identity, the matrix

      is null in the first j1− 1 columns, and null below the first element in the next column.

      It may be that every row below the first in A(1) is null, in which case the algorithm is complete. If not, and if n = 2, let A(2) = A, then for some i2,

      Hence for some L2−1 the matrix

      is null in the first j1− 1 columns, null below the first element in the next j2− j1 columns, and null below the second element in the next column. Proceed thus until reaching A(p) where either p = n or else all rows below the pth are null. The result is that A is expressed in the form

      A = M′ A(p),

      where

      and is nonsingular, and A(p) is a matrix whose first p rows are linearly independent and all other rows, if any, are null. In the latter event, the product remains the same if the n − p null rows of A(p) are dropped, along with the last n − p columns of M. Call the resulting matrices P and M, respectively, and the required form (3) is obtained. Clearly A itself is of rank p.

      If A is square and nonsingular, it may be that one can take

      i1 = 1, i2 = 2, i3 = 3, · · · ,

      which is to say that no permutation is required at any stage. In that event, M is a unit lower triangular matrix, P upper triangular. When a nonsingular matrix A can be expressed as a product of a unit lower triangular matrix by an upper triangular matrix, the factorization in that form is unique. In fact, unique determinantal expressions will be obtained in Section 1.4 for the elements of the two matrices. But suppose A is nonsingular and

      A = M1P1 = M2P2,

      where M1 and M2 are unit lower triangular matrices, and P1 and P2 are upper triangular matrices. Then

      M2−1M1 = P2P1−1.

      But the product on the left is unit lower triangular, and that on the right is upper triangular. Hence both are diagonal, and, furthermore, in the product on the left each diagonal element = 1. Hence, each product can only be I, and therefore M1 = M2, P1 = P2.

      As a method for evaluating δ(A) when A is square, the method of triangularization is the method of Chió. As a step toward the inversion of A (to be completed by inverting the triangles) it is Gaussian elimination. However, as a practical algorithm it is not to be recommended for either purpose in this simple form unless the exact arithmetic operations are performed throughout (without rounding or truncation). The computational precautions will be discussed later.

      The elementary Hermitians can be used for a different factorization. Again consider any matrix

      of n rows and m be the first nonnull column, and form H1 = H(wis some multiple of e1. Hence let

      A(1) = H1A(0).

      Observe that no permutation is required. If any row below the first in Abe the first column having a nonvanishing component orthogonal to e1. Form H2 = H(worthogonal to e1 is reflected into a multiple of e2. If n = 2, then H2 = I. By continuing thus there is formed eventually a matrix A(p) such that

      and either p = n or else every row below the pth in A(p) is null. The matrix H is, of course, not Hermitian, but it is unitary. If p < n, drop the null rows in A(p) and the corresponding columns of H, calling the resulting matrices R and W, respectively. Then

      (4)

      where R has p linearly independent rows and W has p linearly independent columns.

      In case A is nonsingular, W is unitary and R is an upper triangle. Suppose, in this event, one obtains in any way two factorizations

      A = W1R1 = W2R2,

      where R1 and R2 are both upper triangles, and W1 and W2 are both unitary. Then

      W2HW1 = R2R1−1.

      Hence R2R1−1 is both unitary and upper triangular. But it is easy to verify that such a matrix is diagonal (cf. Exercise 4),

      W2HW1 = R2R1−1 = D,

      and, being unitary and diagonal,

      | D | = I,

      which is to say that the diagonal elements are points on the unit circle. Hence

      Thus when A is nonsingular and factored as in (4), each column of W and each row of R is uniquely determined up to a scalar factor which is a point on the unit circle.

      If A is itself unitary, R will be diagonal. Hence, in particular, every unitary matrix can be expressed as a product of elementary Hermitians and of a diagonal matrix with elements on the unit circle.

      The factorization (4) can be achieved in another way. Suppose the columns of A are linearly independent. The first column of W is a normalized first column of A (note that the normalization is unaffected by the introduction of a point on the unit circle as a scalar factor). The second column of W is a unit vector chosen so that a2, the second column of A, is a linear combination of the two columns of W. The third column of W is a unit vector so chosen that a3 is a linear combination of these three columns of W. It is easily verified that on completing the algorithm the desired factorization will have been achieved. This method is known as Schmidt, orthogonalization.

      1.3. Projections, and the General Reciprocal. Let the columns of the rectangular matrix F be linearly independent. The vector Fx is a vector in the space spanned by the columns of F, and Fx is the orthogonal projection of a vector a on this space in case a − Fx is orthogonal to F:

      FH(a − Fx) = 0.

      Hence Fx is the orthogonal projection of a on the space of F in case

      (1)

      and the inverse indeed exists because of the linear independence of the columns of F (this follows directly from the theorem of corresponding matrices, Section 1.4). Hence

      (2)

      is called the projector for the space of F since, for any a, PFa is the orthogonal projection of a upon the space of F. It is the projector, and not just a projector, since if M is nonsingular, and F is replaced by FM, one obtains the same PF. This can be verified directly. Note that PF is Hermitian, and furthermore that

      (3)

      that is, PF is idempotent. Moreover,

      (I − PF)² = I − PF,   (I − PF)PF = 0,

      and I − PF is the projector for the subspace complementary to that of F.

      Any Hermitian idempotent matrix P is a projector. To prove this, let P have rank p, and express P in the form

      P = FRH,

      where each of F and R has p linearly independent columns. Since P is Hermitian and idempotent,

      P = FRH = RFH = FRHRFH.

      Hence

      FHFRH = FHFRHRFH,

      and since FHF is nonsingular, therefore

      RH = RHRFH.

      Hence

      RHR = RHRFHFRHR.

      Again, RHR is nonsingular, hence

      FHFRHR = I;

      hence

      P = FRHRFH = F(FHF)−1FH = PF,

      and P is a projector.

      Now consider an arbitrary matrix A of rank r, and let

      A = FRH

      where each of F and R has r linearly independent columns. It is legitimate to denote

      PA = PF = F(FHF)−1FH.

      Next consider the matrix defined by

      (4)

      By direct verification it is found that

      AAI = F(FHF)−1FH,

      and also that

      AIA = R(RHR)−1RH

      which is similarly the projector PAH for AH. Moreover

      (5)

      Because of this last property AI is known as the general reciprocal (E. H. Moore). It can be shown to be unique. Formally it solves the least squares problem, given a vector a, to find the vector x such that if

      d = a − Ax,

      then dHd is a minimum, since the required vector Ax is the orthogonal projection of a upon the space of A. Hence the required solution is

      x = AIa,

      and it satisfies

      AHAx = AHa.

      If G is positive definite Hermitian, the product xHGy is a natural generalization of the ordinary scalar product xHy = xHIy. It will be shown in Section 1.4 that any positive definite matrix G can be factored in the form

      (6)

      with B a square, nonsingular matrix. Hence

      xHGy = (Bx)H(By),

      so that the generalized scalar product of x and y is the ordinary scalar product of Bx and By, and x and y can be considered as representing in an oblique coordinate system the vectors whose representations are Bx and By in an orthonormal system. The vectors x and y will be considered orthogonal with respect to G in case

      xHGy = 0.

      If the columns of F are linearly independent, then Fx is the orthogonal projection with respect to G of the vector a in case a − Fx is orthogonal to F with respect to G:

      (7)

      Hence the generalized projector, with respect to G, is

      (8)

      It is idempotent, but not Hermitian. It is unchanged when F is replaced by FM, where M is nonsingular. Evidently

      1.4. Some Determinantal Identities. Suppose A is nonsingular and the factorization

      A = LP

      is possible with L unit lower triangular and P upper triangular. A diagonal factor D can be removed from P so that

      (1)

      where R is unit upper triangular. Determinantal expressions for the elements of L, D, and R will now be obtained.

      Let the matrices be partitioned

      (2)

      with A11, L11, D11 and R11 square matrices of order p. Since L11 and R11 are unit triangles,

      δ(L11) = δ(R11) = 1.

      Moreover, by direct multiplication it follows that

      A11 = L11D11R11.

      Hence

      δ(A11) = δ(D11).

      By letting p range from 1 to n it follows that if

      D = diag (δ1, δ2, ⋯ , , δn),

      then

      (3)

      Next, observe that

      (A11, A12) = L11(D11R11, D11R12).

      Again, since δ(L11) = 1, any pth-order determinant formed from the columns of the matrix on the left is equal to the determinant formed from the corresponding columns of the rectangular matrix on the right. In particular, let the determinants be formed of the first p − 1 columns along with any other. The determinant on the right is again triangular and equal to δ1 δ2 ⋯ δp−1 multiplied by the last element of the adjoined column. Let this be δpρpi, i ≥ p. It follows, on applying (3) that

      (4)

      Analogously one finds

      (5)

      This also demonstrates the uniqueness of the factorization, when it can be made, since the elements of the factor matrices are given by explicit formulas.

      Now consider the factorization (still with A nonsingular)

      A = WP,

      where W is unitary and P upper triangular. This can be written

      (6)

      where D is diagonal and R unit upper triangular. Then

      (7)

      H = AHA = RHDHDR.

      Hence formulas (3) and (4) can be applied to give the diagonal elements of DHD and the elements of R:

      (8)

      (9)

      Now write (6) in the partitioned form

      and consider, in particular,

      (10)

      There is no restriction in assuming D real and nonnegative, since if Δ is diagonal with points on the unit circle, then WΔ is unitary, and Δ can be chosen so that Δ−1D is real and nonnegative. Hence each δi is the square root of the right member of (8). Hence any determinant formed from complete rows of

      is equal to the determinant formed from the same rows of

      divided by

      From (3) one obtains a necessary and sufficient condition for the unique factorization (1), which is that every principal minor in the sequence

      shall be different from zero. In case A is singular the conditions are sufficient to assure factorization, but not uniqueness. The factorization (6) is possible for any A with D and A having the same rank p, as was shown in the derivation of (1.2.4).

      A Hermitian matrix A is said to be positive definite in case

      x ≠ 0 → xHAx > 0;

      it is nonnegative semidefinite in case xHAx ≥ 0 for every x. For non-Hermitian matrices the concept is not defined. If A is positive definite (nonnegative semidefinite) then −A is negative definite (nonpositive semidefinite) and conversely. There will be no occasion to speak of negative definite or nonpositive semidefinite matrices here, hence when a matrix is said to be definite (semidefinite) it will be understood to be positive (nonnegative).

      Any principal submatrix of a semidefinite (definite) matrix is itself semidefinite (definite), for consider

      With η = 0, y ≠ 0, it is apparent that the entire matrix cannot be semidefinite (definite) unless A is. In particular, a definite matrix can have no vanishing diagonal element, and a diagonal element of a semidefinite. matrix can vanish, only if all other elements of the same row and column also vanish. For if α = 0, take y = areal and negative and sufficiently large, 2ηaHa + aHAa can be made negative unless

      Enjoying the preview?
      Page 1 of 1