Lecture Notes on

Computational and Applied Mathematics

(MAT 3310, 3 units)

Academic year 2009/2010: second term

Prepared by
Jun ZOU (2006)a and Patrick CIARLET (2010)b

We have prepared the lecture notes purely for the convenience of our teaching. We gratefully
acknowledge the contribution of Prof. I-Liang CHERN (National Taiwan University, Taipei, Taiwan),
made during his visit to the Dept of Mathematics of CUHK in 2009.
a
Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
b
Dept of Applied Mathematics, Ecole Nationale Superieure de Techniques Avancees, Paris, France

At some locations, we put the symbol .

This corresponds to paragraphs that may
be omitted in a first reading.

Some additional numerical methods

We omit the ~ on top of vectors in this chapter...

5.1

Basic linear algebra

We first recall some basic linear algebra concepts in Cn . By removing the conjugation,
one can derive similar results in Rn , that are alluded to along the way. In particular, we
shall give some more specific real-valued results explicitly, dealing with symmetric and
orthogonal matrices. Let us begin by vectors, and then matrices.
Let n N \ {0}. In Cn , let (, )Cn denote a complex (or hermitian) scalar product 78 ,
and let k kCn denote the associated norm:
p
kakCn = (a, a)Cn a Cn .
As before, see 4.6.2, one can prove the Cauchy-Schwarz inequality
|(a, b)Cn | kakCn kbkCn

a, b Cn .

The set of vectors (qi )i=1,m is called orthogonal if

(qi , qj )Cn = 0 i, j {1, , m}, i 6= j.
The set of vectors (qi )i=1,m is called orthonormal if
(qi , qj )Cn = ij

i, j {1, , m},

where ij is the Kronecker symbol.

An orthonormal basis (qi )i=1,n is a basis of Cn that is moreover orthonormal.
78

It is a sesquilinear form from Cn Cn to C:

1. (a1 + a2 , b)Cn = (a1 , b)2 + (a2 , b)Cn and (b, a1 + a2 )Cn = (b, a1 )2 + (b, a2 )Cn for all a1 , a2 , b Cn ;
2. (a, b)Cn = (a, b)Cn and (a, b)Cn = (a, b)Cn for all a, b Cn , for all C;
that enjoys the following properties (hermitian 3., positive definite 4.):
3. (a, b)Cn = (b, a)Cn for all a, b Cn ;
4. (a, a)Cn 0 for all a Cn , and (a, a)Cn = 0 implies a = 0.
To define a scalar product over Rn , one simply assumes that (, )Rn is a bilinear form from Rn Rn to
R that fulfills properties 3. (symmetric) and 4. (positive definite).

133

From now on, we assume that an orthonormal basis (qi )i=1,n is given.
In this case, any vector b Cn can be expressed as
b=

n
X

bi qi , where bi = (b, qi )Cn i = 1, , n,

i=1

and moreover we can identify the vector b with the column vector

b1

b2
. .
.
.
bn
We have then the identities
(a, b)Cn =

n
X

bi ai and kakCn

v
u n
uX
=t
|ai |2

a, b Cn .

i=1

i=1

So, we can adopt the equivalent notation

(a, b)Cn = b a a, b Cn ,
where b := (b)T is the conjugate transpose vector of the vector b.
Also, the norm can be defined equivalently as

kakCn = a a a Cn .
When it is clear from the context, we shall remove the mentions Cn or Rn from the (complex) scalar products and norms.
Let n, m N \ {0}. A complex-valued matrix A with n rows and m columns is usually
called an n m matrix, and it belongs to the vector space of matrices which is denoted
by Cnm . Its entries are called A := (Aij )i=1,n;j=1,m . Furthermore, if Cn and Cm are
endowed respectively with bases79 (qi )i=1,n and (qj0 )j=1,m , then A represents a unique
linear transformation A from Cm to Cn , namely:
Aqj0

n
X

Aij qi for j = 1, , m.

i=1
79

Those definitions are valid, even when the bases are not orthonormal. More precisely, the notion
of complex scalar product is not needed... However, we assume here for convenience that the bases
(qi )i=1,n and (qj0 )j=1,m are orthonormal respectively in Cn and Cm .

134

Or, equivalently, given a vector a0 Cm which writes a0 =

Pn
i=1 (Aa)i qi has components
(Aa)i =

m
X

a0j qj0 , the vector Aa =

Aij a0j for i = 1, , n.

j=1

Remark 5.1. A (column) vector of Cn can be identified with an n 1 matrix, with

obvious notations.
Given two matrices A Cnm and B Cpn , the matrix product BA belongs to Cpm ,
and it corresponds to a linear transformation C from Cm to Cp : with obvious notations,
C = BA. Furthermore,
(BA)kj =

n
X

Bki Aij for k = 1, , p and j = 1, , m.

i=1

We define the adjoint of A Cnm as the unique matrix A Cmn that fulfills
(u, A v)Cm = (Au, v)Cn

u Cm , v Cn .

(5.40)

As a consequence, the entries of the adjoint A are

(A )ji = Aij , for i = 1, , n and j = 1, , m.
Given two matrices A Cnm and B Cpn , one has: (BA) = A B in Cmp .
In the case of real-valued matrices, one introduces the transpose AT of A, still with the
help (5.40) (with scalar products (, )R ). Its entries are given by
(AT )ji = Aij , for i = 1, , n and j = 1, , m.
Given two matrices A Rnm and B Rpn , one has: (BA)T = AT B T in Rmp .
A matrix is said to be a square matrix when it has the same number of rows and columns,
i.e. n = m. In this case, n is called the order of A. Further, a square matrix is said to
be lower triangular (resp. upper triangular) when Aij = 0 for j > i (resp. i < j). An
upper or lower triangular matrix is simply called a triangular matrix. When the only
non-zero entries of a square matrix occur on the diagonal, that is for indices i = j, the
matrix is said to be a diagonal matrix.
Below, we denote by I the identity matrix of Cnn , that corresponds to the identity
transformation from Cn to Cn , with the same basis, that is, (qj0 )j=1,n = (qi )i=1,n . So,
Iij = ij

i, j {1, , n}.

Let us give a few additional definitions concerning square matrices.

135

Definition 5.1. Let A Cnn .

A is invertible if there exists a matrix, written A1 , such that AA1 = A1 A = I.
A is normal if A A = AA .
A is hermitian if A = A.
A is unitary if A A = A A = I.
Let A Rnn .
A is symmetric if AT = A.
A is orthogonal if AT A = AT A = I.
Remark 5.2. Hermitian, unitary (resp. symmetric, orthogonal) matrices are all normal
matrices.
We remark that unitary matrices Q are automatically invertible, with Q1 = Q . As a
consequence, the column vectors of Q form an orthonormal basis. Indeed, writing

..
..
..
.
.
.

Q = q1 qi qn
,
..
..
..
.
.
.
we find that Q Q = I amounts to (qi , qj ) = ij for i, j = 1, , n.
Also, unitary matrices Q preserve the norm of a vector, namely
kQxk = kxk x Cn .
This is clear since
kQxk2 = (Qx) (Qx) = (x Q )(Qx) = x (Q Q)x = x x = kxk2 .
Moreover, the solution of a unitary linear system is easy to compute. Consider
Find x Cn such that Q x = b
where Q is a n n unitary matrix and b Cn . It is straightforward to find its solution,
namely x = (Q1 Q)x = Q1 b = Q b.
Again, the same properties hold for real valued, orthogonal matrices, with Q1 = QT .
We have the fundamental result below, which we state without proof.
136

Theorem 5.1. Let A be a square matrix.

There exists a unitary matrix Q such that Q1 AQ is triangular.
If A is normal, there exists a unitary matrix Q such that Q1 AQ is diagonal.
If A is symmetric, there exists an orthogonal matrix Q such that Q1 AQ is diagonal.
Next, let us consider eigenvalues and eigenvectors of matrices.
Definition 5.2 (Eigenvalue and eigenvector). Let A be a square matrix of order n.
Then, C is an eigenvalue of A if there exists a non-zero vector x Cn such that
Ax = x.
The vector x is called an eigenvector of A, associated to .
Definition 5.3 (Diagonalizable matrix). A square matrix A of order n is diagonalizable
if there exists a basis (qi )i=1,n of Cn , made of eigenvectors of A. Namely,
Aqi = i qi

i = 1, , n,

where i is the eigenvalue associated to qi , for i = 1, , n.

As a corollary to theorem 5.1, one has the result below.

Corollary 5.1. Let A be a square matrix of order n.

If A is normal, then it is diagonalizable. Moreover, there exists an orthonormal
basis of Cn made of eigenvectors of A.
If A is symmetric, then it is diagonalizable with real eigenvalues. Moreover, there
exists an orthonormal basis of Rn made of eigenvectors of A.
Proof.
Consider that A is normal. Then, according to theorem 5.1, there exists
a unitary matrix Q such that Q1 AQ is diagonal. Let D := Q1 AQ, then we
remark that AQ = QD. Next, denote the diagonal entries of D by Dii = i , for
137

i = 1, , n, and by (qi )i=1,n the vector columns of Q, which form an orthonormal

basis of Cn . Then, if we rewrite AQ = QD column by column, we find simply
Aqi = i qi

i = 1, , n.

Consider that A is symmetric. Then, according to theorem 5.1, there exists an

orthogonal matrix Q such that Q1 AQ is diagonal. Let D := Q1 AQ, then we
remark that the diagonal entries of D are automatically real. Proceeding as before
with the same notations, if we rewrite AQ = QD column by column, we find again
Aqi = i qi

i = 1, , n.

We consider here more specifically real-valued vectors and matrices. Let us recall first
the definition of positive definite matrices.
Definition 5.4 (Positive definite matrix). A square matrix A is positive definite if
(x, Ax) > 0 x 6= 0.
A direct observation is that the diagonal entries of A must be positive80 .

Lemma 5.1. A matrix A is symmetric positive definite if and only if A is symmetric and
all its eigenvalues are strictly positive.
Proof. If A is symmetric, then it is diagonalizable. So, let i be the i-th eigenvalue of
A, we have for some xi 6= 0 that Axi = i xi . As A is positive definite, (xi , Axi ) > 0,
which leads to i kxi k2 > 0, and we have i > 0.
Conversely, if A is symmetric and all its eigenvalues are positive, then we have on one
hand D := Q1 AQ with Q orthogonal and D diagonal. On the other hand, Dii > 0,
for i = 1, , n, since the diagonal entries of D are the eigenvalues of A. We note
that A = QDQ1 = QDQT . Consider next x 6= 0, and set y = QT x (by construction,
y = Q1 x 6= 0). Then we have
(x, Ax) = (x, QDQT x) = (x, QDy) = (QDy)T x = y T DQT x = y T Dy = (y, Dy) > 0.
Hence A is postive definite.
80

Check this property...

138