CH 5

Chapter 5
Inner Product Spaces

5.1 Length and Dot Product in R
n
5.2 Inner Product Spaces
5.3 Orthonormal Bases:Gram-Schmidt Process
5.4 Mathematical Models and Least Square Analysis
Elementary Linear Algebra
R. Larsen et al. (6 Edition)
2/80
5.1 Length and Dot Product in R
n

Length:
The length of a vector in R
n
is given by
2 2
2
2
1
|| ||
n
v v v + + + = v
Notes: Properties of length
( )
( )
( )
( ) v v
v v
v v
v
c c =
= =
=
>
4
0 iff 0 3
1 2
0 1
is called a unit vector.
) , , , (
2 1 n
v v v = v
Notes: The length of a vector is also called its norm.
Elementary Linear Algebra: Section 5.1, p.278
3/80
Ex 1:
(a) In R
5
, the length of is given by

(b) In R
3
the length of is given by
) 2 , 4 , 1 , 2 , 0 ( = v
5 25 ) 2 ( 4 1 ) 2 ( 0 || ||
2 2 2 2 2
= = + + + + = v
1
17
17
17
3
17
2
17
2
|| ||
2 2 2
= = |
.
|
\
|
+ |
.
|
\
|
+ |
.
|
\
|
= v
) , , (
17
3
17
2
17
2
= v
(v is a unit vector)
4/80
A standard unit vector in R
n
:
( )
( ) <
>
=
0 2
0 1

c
c
cv u
u and v have the same direction
u and v have the opposite direction
Notes: (Two nonzero vectors are parallel)
{ } ( ) ( ) ( ) { } 1 , , 0 , 0 , 0 , , 1 , 0 , 0 , , 0 , 1 , , ,
2 1
=
n
e e e
Ex:
the standard unit vector in R
2
:

the standard unit vector in R
3
:
{ } ( ) ( ) { } 1 , 0 , 0 , 1 , = j i
{ } ( ) ( ) ( ) { } 1 , 0 , 0 , 0 , 1 , 0 , 0 , 0 , 1 , , = k j i
5/80
Thm 5.1: (Length of a scalar multiple)
Let v be a vector in R
n
and c be a scalar. Then
|| || | | || || v v c c =
|| || | |
| |
) (
) ( ) ( ) (
|| ) , , , ( || || ||
2 2
2
2
1
2 2
2
2
1
2
2 2
2
2
1
2 1
v
v
c
v v v c
v v v c
cv cv cv
cv cv cv c
n
n
n
n
=
+ + + =
+ + + =
+ + + =
=
Pf:
) , , , (
2 1 n
v v v = v
) , , , (
2 1 n
cv cv cv c = v
6/80
Thm 5.2: (Unit vector in the direction of v)
If v is a nonzero vector in R
n
, then the vector

has length 1 and has the same direction as v. This vector u
is called the unit vector in the direction of v.
|| || v
v
u =
Pf:
v is nonzero 0
1
0 > =
v
v
v
v
1
u =
(u has the same direction as v)
1 || ||
|| ||
1
|| ||
|| || = = = v
v v
v
u
(u has length 1 )
7/80
Notes:
(1) The vector is called the unit vector in the direction of v.

(2) The process of finding the unit vector in the direction of v
is called normalizing the vector v.
|| || v
v
8/80
Ex 2: (Finding a unit vector)
Find the unit vector in the direction of ,
and verify that this vector has length 1.
|
.
|
\
|
= =
+ +

=
14
2
,
14
1
,
14
3
) 2 , 1 , 3 (
14
1
2 ) 1 ( 3
) 2 , 1 , 3 (
|| ||
2 2 2
v
v
1
14
14
14
2
14
1
14
3

2 2 2
= = |
.
|
\
|
+ |
.
|
\
|
+ |
.
|
\
|
v
v

is a unit vector.
) 2 , 1 , 3 ( = v ( ) 14 2 1 3
2
2
2
= + + = v
Sol:
) 2 , 1 , 3 ( = v
9/80
Distance between two vectors:
The distance between two vectors u and v in R
n
is
|| || ) , ( v u v u = d
Notes: (Properties of distance)
(1)
(2) if and only if
(3)
0 ) , ( > v u d
0 ) , ( = v u d
v u =
) , ( ) , ( u v v u d d =
10/80
Ex 3: (Finding the distance between two vectors)
The distance between u=(0, 2, 2) and v=(2, 0, 1) is
3 1 2 ) 2 (
|| ) 1 2 , 0 2 , 2 0 ( || || || ) , (
2 2 2
= + + =
= = v u v u d
11/80
Dot product in R
n
:
The dot product of and
is the scalar quantity
Ex 4: (Finding the dot product of two vectors)
The dot product of u=(1, 2, 0, -3) and v=(3, -2, 4, 2) is
7 ) 2 )( 3 ( ) 4 )( 0 ( ) 2 )( 2 ( ) 3 )( 1 ( = + + + = v u
n n
v u v u v u + + + =
2 2 1 1
v u
) , , , (
2 1 n
u u u = u ) , , , (
2 1 n
v v v = v
12/80
Thm 5.3: (Properties of the dot product)
If u, v, and w are vectors in R
n
and c is a scalar,
then the following properties are true.
(1)
(2)
(3)
(4)
(5) , and if and only if
u v v u =
w u v u w v u + = + ) (
) ( ) ( ) ( v u v u v u c c c = =
2
|| || v v v =
0 > v v 0 = v v 0 = v
13/80
Euclidean n-space:
R
n
was defined to be the set of all order n-tuples of real
numbers. When R
n
is combined with the standard
operations of vector addition, scalar multiplication, vector
length, and the dot product, the resulting vector space is
called Euclidean n-space.
14/80
Sol:
6 ) 8 )( 2 ( ) 5 )( 2 ( ) a ( = + = v u
) 18 , 24 ( ) 3 , 4 ( 6 6 ) ( ) b ( = = = w w v u
12 ) 6 ( 2 ) ( 2 ) 2 ( ) c ( = = = v u v u
25 ) 3 )( 3 ( ) 4 )( 4 ( || || ) d (
2
= + = = w w w
) 2 , 13 ( ) 6 8 , ) 8 ( 5 ( 2 ) e ( = = w v
22 4 26 ) 2 )( 2 ( ) 13 )( 2 ( ) 2 ( = = + = w v u
Ex 5: (Finding dot products)
) 3 , 4 ( ), 8 , 5 ( , ) 2 , 2 ( = = = w v u
(a) (b) (c) (d) (e)
v u w v u ) ( ) 2 ( v u
2
|| || w ) 2 ( w v u
15/80
Ex 6: (Using the properties of the dot product)
Given

39 = u u 3 = v u
79 = v v
) 3 ( ) 2 ( v u v u + +
Sol:
) 3 ( 2 ) 3 ( ) 3 ( ) 2 ( v u v v u u v u v u + + + = + +
Find
254 ) 79 ( 2 ) 3 ( 7 ) 39 ( 3 = + + =
v v u v v u u u + + + = ) 2 ( ) 3 ( ) 2 ( ) 3 (
) ( 2 ) ( 6 ) ( 3 v v u v v u u u + + + =
) ( 2 ) ( 7 ) ( 3 v v v u u u + + =
16/80
Thm 5.4: (The Cauchy - Schwarz inequality)
If u and v are vectors in R
n
, then
( denotes the absolute value of )
|| || || || | | v u v u s
| | v u v u
v u v u
v v u u v u
v u
s
= = =
= =
55 5 11
1 1
5 , 11 , 1 = = = v v u u v u
Ex 7: (An example of the Cauchy - Schwarz inequality)
Verify the Cauchy - Schwarz inequality for u=(1, -1, 3)
and v=(2, 0, -1)
Sol:
Elementary Linear Algebra: Section 5.1, p.285-286
17/80
Note:
The angle between the zero vector and another vector is
not defined.
The angle between two vectors in R
n
:

t u u s s
= 0 ,
|| || || ||
cos
v u
v u
1 cos =
=t u
1 cos
0
=
= u
0 cos
2
<
< < t u
t
0 cos
2
=
=
t
u
0 cos
2
0
>
< <
t
u
u u u u
0 v u < 0 v u = 0 v u >
Opposite
direction
Same
direction
18/80
Ex 8: (Finding the angle between two vectors)
) 2 , 2 , 0 , 4 ( = u ) 1 , 1 , 0 , 2 ( = v
Sol:
( ) ( ) 24 2 2 0 4
2
2 2
2
= + + + = = u u u
1
144
12
6 24
12
|| || || ||
cos = =
=
v u
v u
u
( ) ( ) 6 1 1 0 2
2
2 2
2
= + + + = = v v v
12 ) 1 )( 2 ( ) 1 )( 2 ( ) 0 )( 0 ( ) 2 )( 4 ( = + + + = v u
= t u u and v have opposite directions. ) 2 ( v u =
19/80
Orthogonal vectors:
Two vectors u and v in R
n
are orthogonal if
0 = v u
Note:
The vector 0 is said to be orthogonal to every vector.
20/80
Ex 10: (Finding orthogonal vectors)
Determine all vectors in R
n
that are orthogonal to u=(4, 2).
0
2 4
) , ( ) 2 , 4 (
2 1
2 1
=
+ =
=
v v
v v v u
| |
(
0
2
1
1 0 2 4
t v
t
v =
=
2 1
,
2
R t ,t
t
e
|
.
|
\
|

= ,
2
v
) 2 , 4 ( = u
Let ) , (
2 1
v v = v
Sol:
21/80

Thm 5.5: (The triangle inequality)
n
, then || || || || || || v u v u + s +
Pf:
) ( ) ( || ||
2
v u v u v u + + = +
2 2 2 2
|| || | | 2 || || || || ) ( 2 || ||
) ( 2 ) ( ) (
v v u u v v u u
v v v u u u v u v v u u
+ + s + + =
+ + = + + + =
2
2 2
||) || || (||
|| || || || || || 2 || ||
v u
v v u u
+ =
+ + s
|| || || || || || v u v u + s +
Note:
Equality occurs in the triangle inequality if and only if
the vectors u and v have the same direction.
22/80
Thm 5.6: (The Pythagorean theorem)
n
, then u and v are orthogonal
if and only if
2 2 2
|| || || || || || v u v u + = +
23/80
Dot product and matrix multiplication:
(
(
(
(
=
n
u
u
u
2
1
u
(
(
(
(
=
n
v
v
v
2
1
v
] [ ] [
2 2 1 1
2
1
2 1 n n
n
n
T
v u v u v u
v
v
v
u u u + + + =
(
(
(
(
= =
v u v u
(A vector in R
n

is represented as an n1 column matrix)
) , , , (
2 1 n
u u u = u
24/80
Keywords in Section 5.1:
length:
norm:
unit vector:
standard unit vector :
normalizing:
distance:
dot product:
Euclidean n-space: n
Cauchy-Schwarz inequality: -
angle:
triangle inequality:
Pythagorean theorem:
25/80
5.2 Inner Product Spaces
(1)
(2)
(3)
(4) and if and only if
u v v u , , =
w u v u w v u , , , + = +
v u v u , , c c =
0 , > v v
0 , = v v
0 = v
Inner product:
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on V is a function that associates
a real number <u, v> with each pair of vectors u and v and
satisfies the following axioms.
26/80
Note:
V
R
n
space for vector product inner general ,
) for product inner Euclidean ( product dot
>= <
=
v u
v u
Note:
A vector space V with an inner product is called an inner
product space.
( ) - + , , V
Vector space:
Inner product space: ( ) > < - + , , , , V
Elementary Linear Algebra: Section 5.2, Addition
27/80
Ex 1: (The Euclidean inner product for R
n
)
Show that the dot product in R
n
satisfies the four axioms
of an inner product.
n n
v u v u v u + + + = =
2 2 1 1
, v u v u
) , , , ( , ) , , , (
2 1 2 1 n n
v v v u u u = = v u
Sol:
By Theorem 5.3, this dot product satisfies the required four axioms.
Thus it is an inner product on R
n
.
28/80
Ex 2: (A different inner product for R
n
)
Show that the function defines an inner product on R
2
,
where and .
2 2 1 1
2 , v u v u + = v u
) , ( ) , (
2 1 2 1
v v u u = = v u
Sol:
u v v u , 2 2 , ) (
2 2 1 1 2 2 1 1
= + = + = u v u v v u v u a

w u v u
w v u
, ,
) 2 ( ) 2 (
2 2
) ( 2 ) ( ,
2 2 1 1 2 2 1 1
2 2 2 2 1 1 1 1
2 2 2 1 1 1
+ =
+ + + =
+ + + =
+ + + = +
w u w u v u v u
w u v u w u v u
w v u w v u
) , ( ) (
2 1
w w b = w
Elementary Linear Algebra: Section 5.2, pp.293-294
29/80
Note: (An inner product on R
n
)
0 , ,
2 2 2 1 1 1
> + + + =
i n n n
c v u c v u c v u c v u
v u v u , ) ( 2 ) ( ) 2 ( , ) (
2 2 1 1 2 2 1 1
c v cu v cu v u v u c c c = + = + =
0 2 , ) (
2
2
2
1
> + = v v d v v
) 0 ( 0 0 2 0 ,
2 1
2
2
2
1
= = = = + = v v v v v v v
30/80
Ex 3: (A function that is not an inner product)
Show that the following function is not an inner product on R
3
.
3 3 2 2 1 1
2 v u v u v u + = v u
Sol:
Let ) 1 , 2 , 1 ( = v
0 6 ) 1 )( 1 ( ) 2 )( 2 ( 2 ) 1 )( 1 ( , Then < = + >= < v v
Axiom 4 is not satisfied.
Thus this function is not an inner product on R
3
.
31/80
Thm 5.7: (Properties of inner products)
Let u, v, and w be vectors in an inner product space V, and
let c be any real number.
(1)
(2)
(3)
0 , , = = 0 v v 0
w v w u w v u , , , + = +
v u v u , , c c =
Norm (length) of u:
u u u , || || =
u u u , || ||
2
=
Note:
32/80
u and v are orthogonal if .
Distance between u and v:
) ( = = v u v u v u v u , || || ) , ( d
Angle between two nonzero vectors u and v:
t u u s s = 0 ,
|| || || ||
,
cos
v u
v u
Orthogonal:
0 , = v u
) ( v u
33/80
Notes:
(1) If , then v is called a unit vector.

(2)
1 || || = v
0
1
=
=
v
v

g Normalizin
v
v
(the unit vector in the
direction of v)
not a unit vector
34/80
Ex 6: (Finding inner product)
) ( in s polynomial be 2 4 ) ( , 2 1 ) ( Let
2
2 2
x P x x x q x x p + = =
n n
b a b a b a q p + + + =
1 1 0 0
, is an inner product
? , ) ( = q p a ? || || ) ( = q b ? ) , ( ) ( = q p d c
Sol:
2 ) 1 )( 2 ( ) 2 )( 0 ( ) 4 )( 1 ( , ) ( = + + = q p a
21 1 ) 2 ( 4 , || || ) (
2 2 2
= + + = = q q q b
22 ) 3 ( 2 ) 3 (
, || || ) , (
3 2 3 ) (
2 2 2
2
= + + =
) ( = =
+ =
q p q p q p q p d
x x q p c
35/80
Properties of norm:
(1)
(2) if and only if
(3)
Properties of distance:
(1)
(2) if and only if
(3)
0 || || > u
0 || || = u 0 u=
|| || | | || || u u c c =
0 ) , ( > v u d
0 ) , ( = v u d
v u =
) , ( ) , ( u v v u d d =
36/80
Thm 5.8
Let u and v be vectors in an inner product space V.
(1) Cauchy-Schwarz inequality:

(2) Triangle inequality:

(3) Pythagorean theorem :
u and v are orthogonal if and only if
|| || || || || || v u v u + s +
Theorem 5.5
2 2 2
|| || || || || || v u v u + = +
Theorem 5.6
|| || || || | , | v u v u s
Theorem 5.4
37/80
Orthogonal projections in inner product spaces:
Let u and v be two vectors in an inner product space V,
such that . Then the orthogonal projection of u
onto v is given by
0 v =
v
v v
v u
u
v
) (
) (
=
,
,
proj
Note:
If v is a init vector, then .
The formula for the orthogonal projection of u onto v
takes the following simpler form.
1 || || ,
2
= = v v v
v v u u
v
) ( = , proj
38/80
Ex 10: (Finding an orthogonal projection in R
3
)
Use the Euclidean inner product in R
3
to find the
orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).
Sol:
10 ) 0 )( 4 ( ) 2 )( 2 ( ) 1 )( 6 ( , = + + = ) ( v u
5 0 2 1 ,
2 2 2
= + + = ) ( v v
) 0 , 4 , 2 ( ) 0 , 2 , 1 ( proj
5
10
= =
= v
v v
v u
u
v
Note:
). 0 , 2 , 1 ( to orthogonal is 4) 2, (4, 0) 4, (2, 4) 2, (6, proj = = = v u u
v
39/80
Thm 5.9: (Orthogonal projection and distance)
Let u and v be two vectors in an inner product space V,
such that . Then
0 v =
) (
) (
= <
v v
v u
v u u u
v
,
,
, ) , ( ) proj , ( c c d d
40/80
inner product:
inner product space:
norm:
distance:
angle:
orthogonal:
unit vector:
normalizing:
Cauchy Schwarz inequality: -
triangle inequality:
Pythagorean theorem:
orthogonal projection:
41/80
5.3 Orthonormal Bases: Gram-Schmidt Process
Orthogonal:
A set S of vectors in an inner product space V is called an
orthogonal set if every pair of vectors in the set is orthogonal.
Orthonormal:
An orthogonal set in which each vector is a unit vector is
called orthonormal.

{ }
=
=
= ) (
_ =
j i
j i
V S
j i
n
0
1
,
, , ,
2 1
v v
v v v
{ }
0 ,
, , ,
2 1
= ) (
_ =
j i
n
V S
v v
v v v
j i =
Note:
If S is a basis, then it is called an orthogonal basis or an
orthonormal basis.
42/80
Ex 1: (A nonstandard orthonormal basis for R
3
)
Show that the following set is an orthonormal basis.

)
`
|
.
|
\
|

|
|
.
|
\
|
|
.
|
\
|
=
3
1
,
3
2
,
3
2
,
3
2 2
,
6
2
,
6
2
, 0 ,
2
1
,
2
1
3 2 1
S
v v v
Sol:
Show that the three vectors are mutually orthogonal.
0
9
2 2
9
2
9
2
0 0
2 3
2
2 3
2
0 0
3 2
3 1
6
1
6
1
2 1
= + =
= + =
= + + =
v v
v v
v v
43/80
Show that each vector is of length 1.
Thus S is an orthonormal set.
1 || ||
1 || ||
1 0 || ||
9
1
9
4
9
4
3 3 3
9
8
36
2
36
2
2 2 2
2
1
2
1
1 1 1
= + + = =
= + + = =
= + + = =
v v v
v v v
v v v
44/80
The standard basis is orthonormal.
Ex 2: (An orthonormal basis for )
In , with the inner product
) (
3
x P
2 2 1 1 0 0
, b a b a b a q p + + = ) (
} , , 1 {
2
x x B =
) (
3
x P
Sol:
, 0 0 1
2
1
x x + + = v , 0 0
2
2
x x + + = v , 0 0
2
3
x x + + = v
0 ) 1 )( 0 ( ) 0 )( 1 ( ) 0 )( 0 ( ,
, 0 ) 1 )( 0 ( ) 0 )( 0 ( ) 0 )( 1 ( ,
, 0 ) 0 )( 0 ( ) 1 )( 0 ( ) 0 )( 1 ( ,
3 2
3 1
2 1
= + + = ) (
= + + = ) (
= + + = ) (
v v
v v
v v
Then
45/80
( )( ) ( )( ) ( )( )
( )( ) ( )( ) ( )( )
( )( ) ( )( ) ( )( ) 1 1 1 0 0 0 0
, 1 0 0 1 1 0 0
, 1 0 0 0 0 1 1
3 3 3
2 2 2
1 1 1
= + + = ) ( =
= + + = ) ( =
= + + = ) ( =
v , v v
v , v v
v , v v
46/80
Thm 5.10: (Orthogonal sets are linearly independent)
If is an orthogonal set of nonzero vectors
in an inner product space V, then S is linearly independent.
{ }
n
S v , , v , v
2 1
=
Pf:
S is an orthogonal set of nonzero vectors
0 and 0 i.e. > ) ( = = ) (
i i j i
j i v , v v , v
i c c c
c c c
i i n n
n n
= ) ( = ) + + + (
= + + +
0 , 0 ,
0 Let
2 2 1 1
2 2 1 1
v v v v v
v v v
) ( =
) ( + + ) ( + + ) ( + ) (
i i i
i n n i i i i i
c
c c c c
v , v
v , v v , v v , v v , v

2 2 1 1

t. independen linearly is 0 0 S i c
i i i
= = ) ( v , v
47/80
Corollary to Thm 5.10:
If V is an inner product space of dimension n, then any
orthogonal set of n nonzero vectors is a basis for V.
48/80
Ex 4: (Using orthogonality to test for a basis)
Show that the following set is a basis for .
4
R
)} 1 , 1 , 2 , 1 ( , ) 1 , 2 , 0 , 1 ( , ) 1 , 0 , 0 , 1 ( , ) 2 , 2 , 3 , 2 {(
4 3 2 1
= S
v v v v
Sol:

: nonzero vectors
0 2 2 6 2
0 2 4 0 2
0 2 0 0 2
4 1
3 1
2 1
= + =
= + + =
= + + =
v v
v v
v v
4 3 2 1
, , , v v v v
0 1 2 0 1
0 1 0 0 1
0 1 0 0 1
4 3
4 2
3 2
= + + =
= + + + =
= + + + =
v v
v v
v v
. orthogonal is S
4
for basis a is R S (by Corollary to Theorem 5.10)
49/80
Thm 5.11: (Coordinates relative to an orthonormal basis)
If is an orthonormal basis for an inner
product space V, then the coordinate representation of a vector
w with respect to B is
} , , , {
2 1 n
B v v v =
} , , , {
2 1 n
B v v v = is orthonormal
j i
j i
j i
=
=
= ) (
0
1
, v v
V e w
n n
k k k v v v w + + + =
2 2 1 1
(unique representation)
Pf:
is a basis for V
} , , , {
2 1 n
B v v v =
n n
v v w v v w v v w w ) ( + + ) ( + ) ( = , , ,
2 2 1 1

50/80
i k
k k k
k k k
i
i n n i i i i
i n n i
=
+ + + + =
+ + + =
, , ,
, ) ( ,
1 1
2 2 1 1
v v v v v v
v v v v v w

n n
v v w v v w v v w w ) ( + + ) ( + ) ( = , , ,
2 2 1 1

Note:
If is an orthonormal basis for V and , } , , , {
2 1 n
B v v v =
V e w
Then the corresponding coordinate matrix of w relative to B is
| |
(
(
(
(
) (
) (
) (
=
n
B
v w
v w
v w
w
,
,
,
2
1

51/80
Ex 5: (Representing vectors relative to an orthonormal basis)
Find the coordinates of w = (5, -5, 2) relative to the following
orthonormal basis for .
)} 1 , 0 , 0 ( , ) 0 , , ( , ) 0 , , {(
5
3
5
4
5
4
5
3
= B
3
R
Sol:
2 ) 1 , 0 , 0 ( ) 2 , 5 , 5 ( ,
7 ) 0 , , ( ) 2 , 5 , 5 ( ,
1 ) 0 , , ( ) 2 , 5 , 5 ( ,
3 3
5
3
5
4
2 2
5
4
5
3
1 1
= = = ) (
= = = ) (
= = = ) (
v w v w
v w v w
v w v w
(
(
(
=
2
7
1
] [
B
w
52/80
Gram-Schmidt orthonormalization process:
is a basis for an inner product space V } , , , {
2 1 n
B u u u =
1 1
Let u v =
}) ({
1 1
v w span =
}) , ({
2 2 1
v v w span =
} , , , { '
2 1 n
B v v v =
} , , , { ' '
2
2
n
n
B
v
v
v
v
v
v
1
1
=
is an orthogonal basis.
is an orthonormal basis.
=
= =

1
1

proj
1
n
i
i
i i
i n
n n n n
n
v
v , v
v , v
u u u v
W
2
2 2
2 3
1
1 1
1 3
3 3 3 3

proj
2
v
v , v
v , u
v
v , v
v , u
u u u v
W
= =
1
1 1
1 2
2 2 2 2

proj
1
v
v , v
v , u
u u u v
W
= =
53/80
Sol:
) 0 , 1 , 1 (
1 1
= = u v
) 2 , 0 , 0 ( ) 0 ,
2
1
,
2
1
(
2 / 1
2 / 1
) 0 , 1 , 1 (
2
1
) 2 , 1 , 0 (
2
2 2
2 3
1
1 1
1 3
3 3
= =
= v
v v
v u
v
v v
v u
u v
Ex 7: (Applying the Gram-Schmidt orthonormalization process)
Apply the Gram-Schmidt process to the following basis.
)} 2 , 1 , 0 ( , ) 0 , 2 , 1 ( , ) 0 , 1 , 1 {(
3 2 1
= B
u u u
) 0 ,
2
1
,
2
1
( ) 0 , 1 , 1 (
2
3
) 0 , 2 , 1 (
1
1 1
1 2
2 2
= =
= v
v v
v u
u v
54/80
} 2) 0, (0, 0), ,
2
1
,
2
1
( 0), 1, (1, { } , , { '
3 2 1

= = v v v B
Orthogonal basis
} 1) 0, (0, 0), ,
2
1
,
2
1
( 0), ,
2
1
,
2
1
( { } , , { ' '
3
3
2
2

= =
v
v
v
v
v
v
1
1
B
Orthonormal basis
55/80
Ex 10: (Alternative form of Gram-Schmidt orthonormalization process)
Find an orthonormal basis for the solution space of the
homogeneous system of linear equations.
0 6 2 2
0 7
4 3 2 1
4 2 1
= + + +
= + +
x x x x
x x x
Sol:

(
0 8 2 1 0
0 1 2 0 1
0 6 2 1 2
0 7 0 1 1
. . E J G
(
(
(
(
+
(
(
(
(
=
(
(
(
(
+
=
(
(
(
(
1
0
8
1
0
1
2
2
8 2
2

4
3
2
1
t s
t
s
t s
t s
x
x
x
x
56/80
Thus one basis for the solution space is
)} 1 , 0 , 8 , 1 ( , ) 0 , 1 , 2 , 2 {( } , {
2 1
= = u u B
( )
( ) ( )
( ) 1 , 2 , 4 , 3
0 1, 2, , 2
9
18
1 0, 8, 1,
,
,
0 1, 2, , 2
1
1 1
1 2
2 2
1 1
=
=
) (
) (
=
= =
v
v v
v u
u v
u v
( )( ) { } 1 , 2 , 4 , 3 0 , 1 , 2 , 2 ' = B
(orthogonal basis)
)
`
|
.
|
\
|
|
.
|
\
|
=
30
1
,
30
2
,
30
4
,
30
3
, 0 ,
3
1
,
3
2
,
3
2
' ' B
(orthonormal basis)
57/80
orthogonal set:
orthonormal set:
orthogonal basis:
orthonormal basis:
linear independent:
Gram-Schmidt Process: Gram-Schmidt
58/80
. in all and in all for 0 , if
orthogonal are V space product inner an of and subspaces The
2 2 1 1 2 1
2 1
W W
W W
v v v v = ) (
Orthogonal subspaces:
Ex 2: (Orthogonal subspaces)
. zero is in any vector and in any vector for 0 , because orthogonal are
)
1
1
1 -
span( and )
0
1
1
,
1
0
1
span(
subspaces The
2 1 2 1
2 1
W W
W W
= ) (
(
(
(
=
(
(
(
(
(
(
=
v v
5.4 Mathematical Models and Least Squares Analysis
59/80
Let W be a subspace of an inner product space V.
(a) A vector u in V is said to orthogonal to W,
if u is orthogonal to every vector in W.
(b) The set of all vectors in V that are orthogonal to every
vector in W is called the orthogonal complement of W.
(read perp)
} , 0 , | { W V W e = ) ( e =
w w v v
W
W
Orthogonal complement of W:
{ } ( ) { } 0 (2) 0 (1) = =

V V
Notes:
60/80
Notes:

{ }
W W
W W
V W
V W
=
=

) ( (3)
(2)
of subspace a is (1)
of subspace a is
0
{ }
W W
W W
R y W
x W R V
=
=
=
= =

) ( (3)
) 0 , 0 ( (2)
of subspace a is axis - (1) Then
axis , If
2
2
Ex:
Elementary Linear Algebra: Section 5.4, Addition
61/80
Direct sum:
Let and be two subspaces of . If each vector
can be uniquely written as a sum of a vector from
and a vector from , , then is the
direct sum of and , and you can write .
1
W
2
W
n
R
n
R e x
1
W
1
w
2
W
2
w
2 1
w w x + =
n
R
2 1
W W R
n
=
Thm 5.13: (Properties of orthogonal subspaces)
Let W be a subspace of R
n
. Then the following properties
are true.
(1)
(2)
(3)
n W W = +

) dim( ) dim(
= W W R
n
W W =

) (
1
W
2
W
62/80
Thm 5.14: (Projection onto a subspace)
If is an orthonormal basis for the
subspace S of V, and , then
} , , , {
2 1 t
u u u
V e v
t t W
u u v u u v u u v v ) ( + + ) ( + ) ( = , , , proj
2 2 1 1

Pf:
for W basis l orthonorma an is } , , , { and W proj
2 1 t W
u u u v e
t t W W W
u u v u u v v ) ( + + ) ( = , proj , proj proj
1 1

) proj proj (
, proj , proj
1 1
v v v
u u v v u u v v

=
) ( + + ) ( =
W
W
t t
W W
) , 0 , proj ( , ,
1 1
i
i
W
t t
= ) ( ) ( + + ) ( =

u v u u v u u v
63/80
Ex 5: (Projection onto a subspace)
( ) ( ) ( ) 3 , 1 , 1 , 0 , 0 , 2 , 1 , 3 , 0
2 1
= = = v w w
Find the projection of the vector v onto the subspace W.
{ } : ,
2 1
w w
Sol:
an orthogonal basis for W
{ } ( ) : 0 , 0 , 1 ),
10
1
,
10
3
, 0 ( , ,
2
2
1
1
2 1
)
`
=
w
w
w
w
u u
an orthonormal basis for W
}) , ({
2 1
w w span W =
( ) )
5
3
,
5
9
, 1 ( 0 , 0 , 1 )
10
1
,
10
3
, 0 (
10
6
, , proj
2 2 1 1
= + =
) ( + ) ( = u u v u u v v
W
64/80
Find by the other method:
| |
b b
b
b
v b w w
T 1 T
) (
T 1 T
2 1
) ( proj
) (
, ,
A A A A Ax
A A A x
Ax
A
A cs

= =
=
=
= =
65/80
Thm 5.15: (Orthogonal projection and distance)
Let W be a subspace of an inner product space V, and .
Then for all ,
V e v
W e w v w
W
proj =
|| || || proj || w v v v <
W
|| || min || proj || r o
W
w v v v
w
=
e
W
( is the best approximation to v from W) v
W
proj
66/80
Pf:
) proj ( ) proj ( w v v v w v + =
W W
) proj ( ) proj ( w v v v
W W
By the Pythagorean theorem
2 2 2
|| proj || || proj || || || w v v v w v + =
W W
0 proj proj > = w v v w
W W
2 2
|| proj || || || v v w v
W
>
|| || || proj || w v v v <
W
67/80
Notes:
(1) Among all the scalar multiples of a vector u, the
orthogonal projection of v onto u is the one that is
closest to v. (p.302 Thm 5.9)
(2) Among all the vectors in the subspace W, the vector
is the closest vector to v.

v
W
proj
68/80
Thm 5.16: (Fundamental subspaces of a matrix)
If A is an mn matrix, then
(1)

(2)

(3)
(4)

) ( )) ( (
) ( )) ( (
A CS A NS
A NS A CS
=
=
T
T
) ( )) ( (
) ( )) ( (
T
T
=
=
A CS A NS
A NS A CS
m m T
R A NS A CS R A NS A CS = =

)) ( ( ) ( ) ( ) (
n T n T
R A CS A CS R A NS A CS = =
T
)) ( ( ) ( ) ( ) (
69/80
Ex 6: (Fundamental subspaces)
Find the four fundamental subspaces of the matrix.
(
(
(
(
=
0 0 0
0 0 0
1 0 0
0 2 1
A (reduced row-echelon form)
Sol:
( ) ( ) { } ( )
4
of subspace a is 0 , 0 , 1 , 0 0 , 0 , 0 , 1 span ) ( R A CS =
( ) ( ) ( ) { } ( )
3
of subspace a is 1 , 0 , 0 0 , 2 , 1 span ) ( R A RS A CS = =
T
( ) { } ( )
3
of subspace a is 0 , 1 , 2 span ) ( R A NS =
70/80
( ) ( ) { } ( )
4
of subspace a is 1 , 0 , 0 , 0 0 , 1 , 0 , 0 span ) ( R A NS =
T
(
(
(
=
(
(
(
=
T
0 0 0 0
0 0 1 0
0 0 0 1
~
0 0 1 0
0 0 0 2
0 0 0 1
R A
Check:
) ( )) ( (
T
= A NS A CS
) ( )) ( ( A NS A CS =
T
4
) ( ) ( R A NS A CS
T
=
3
) ( ) ( R A NS A CS
T
=
t s
71/80
Ex 3 & Ex 4:

Let W is a subspace of R
4
and .
(a) Find a basis for W
(b) Find a basis for the orthogonal complement of W.
) 1 0, 0, 0, ( ), 0 1, 2, 1, (
2 1
= = w w
Sol:
2 1
0 0
0 0
1 0
0 1
~
1 0
0 1
0 2
0 1
w w
(
(
(
(
=
(
(
(
(
= R A (reduced row-echelon form)

}) , ({
2 1
w w span W =
72/80
( )
( ) ( ) { } 1 , 0 , 0 , 0 , 0 , 1 , 2 , 1
) (
= A CS W a
is a basis for W
( ) ( ) ( )
( ) ( ) { }

T
T

(
(
(
(
+
(
(
(
(
=
(
(
(
(

=
(
(
(
(
=
= =
W
t s
t
s
t s
x
x
x
x
A
A NS A CS W b
for basis a is 0 , 1 , 0 , 1 0 , 0 , 1 , 2
0
1
0
1
0
0
1
2
0
2

1 0 0 0
0 1 2 1
) (
4
3
2
1
Notes:
4
4
(2)
) dim( ) dim( ) dim( (1)
R W W
R W W
=
= +

73/80
Least squares problem:
(A system of linear equations)
(1) When the system is consistent, we can use the Gaussian
elimination with back-substitution to solve for x
b x = A
1 1 m n n m
(2) When the system is inconsistent, how to find the best possible
solution of the system. That is, the value of x for which the
difference between Ax and b is small.
74/80
Notes:
Least squares solution:
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in R
n
that
minimizes with respect to the Euclidean inner
product on R
n
. Such a vector is called a least squares
solution of Ax = b.
b x A
x b x b b b
b
x b x
x
x
A A
A CS A A A
R
n
R
A CS
A CS
n
= =
e =
e
min
proj
is, That . to possible as close as is
) ) (
(i.e., of space column in the proj
such that in
vector a find to is problem square least The

) (
) (
75/80
( ) ) of subspace a is ( ) (
m
n
n m
R A CS A CS A
R
M A
e
e
e

x
x
b x
x b
x b
x b
b x
T T
T
T
=
=
= e

=
A A A
A A
A NS A CS A
A CS A
proj A
A CS
i.e.
0 )
(
) ( )) ( (
) ( )
Let
) (
(the normal system associated with Ax = b)
) system nt inconsiste an is b ( ) ( b = e x A A CS
76/80
Note: (Ax = b is an inconsistent system)
The problem of finding the least squares solution of
is equal to he problem of finding an exact solution of the
associated normal system .
b x = A
b x
T T
= A A A
77/80
Ex 7: (Solving the normal equations)
Find the least squares solution of the following system

(this system is inconsistent)

and find the orthogonal projection of b on the column space of A.

(
(
(
=
(
(
(
(
=
3
1
0
3 1
2 1
1 1
1
0
c
c
A b x
78/80
Sol:
(
=
(
(
(
=
(
=
(
(
(
=
11
4
3
1
0
3 2 1
1 1 1
14 6
6 3
3 1
2 1
1 1
3 2 1
1 1 1
b
T
T
A
A A
the associated normal system
(
=
(
=
11
4
14 6
6 3
1
0
c
c
A A A
T T
b x
79/80
the least squares solution of Ax = b
(
=
2
3
3
5
x
the orthogonal projection of b on the column space of A
(
(
(
=
(
(
(
(
= =

6
17
6
8
6
1
2
3
3
5
) (
3 1
2 1
1 1
proj x b A
A CS
80/80
orthogonal to W: W
orthogonal complement:
direct sum:
projection onto a subspace:
fundamental subspaces:
least squares problem:
normal equations:

CH 5

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

CH 5

Încărcat de

Drepturi de autor:

Formate disponibile

Chapter 5

Inner Product Spaces

Elementary Linear Algebra: Section 5.3, pp.310-311

= R A (reduced row-echelon form)

Elementary Linear Algebra: Section 5.4, pp.322-323

(i.e., of space column in the proj

vector a find to is problem square least The

S-ar putea să vă placă și