Optimal Control

Lecture Notes
on
Optimal Control
Mikhail Ivanov Krastanov
Institute of Mathematics and Informatics
Bulgarian Academy of Sciences
2007
1 Introduction
There are many examples of calculus of variations and optimal control prob-
lems. Some are easy, others hard. Some can be solved directly by elementary
arguments, others cannot be solved unless one uses the machinery of the con-
temporary methods. One of the ancients problems is the so called isoperi-
metric problem, also known as Didos problem. This problem was inspired
by the story told by Virgil (70-19 BCE) in the Aeneid about the founda-
tion of Carthage. It is more than 2100 years old. It is essentially solved by
Zenodorus (200-140 BCE), who showed that, (i) among all polygons with a
given perimeter P and given number N of sides, the regular polgyon has the
largest area, and (ii) a circle had a larger area than any regular polygon of
the same perimeter. Heron of Alexandria showed in his Catoprics that when
a light emitted by an object is reected by an mirror, it follows a path from
the object to the eye which is the shortest of all possible such paths. In the
works of Fermat (1601-1665) a general principle is formulated. According to
this principle, the light rays follow the fastest paths. This principle explained
the Hero observation about reection, but also the Snellius law of refraction.
Also, Newton had studied in 1685 the determination of the shape of a body
with minimal drag.
It is accepted that the calculus of variations was born in 1696, more
than 300 years ago, in Groningen, a university town in the north of the
Netherlands. In this year, Johann Bernoulli challenged all mathematicians
to solve the so called brachystochrone problem. Solutions of this problem
were proposed by Johann Bernoulli and his brother Jacob Bernoulli, and such
other giants as Newton, Leibniz, Tschirnhaus and lHopital. Euler, (a student
of Bernoulli) and Lagrange (who became interested in such kind of problems
by reading Eulers works) develop a general technique to solve variational
problems. Many other well known mathematicians as Legendre, Hamilton,
Weierstrass, Caratheodory and etc. have contributed in the development of
the theory of calculus of variations.
The contemporary optimal control theory was born in 1961 in the former
Soviet Union with the work on the Pontryagin maximum principle by L. S.
Pontryagin and his group (V. G. Boltyanskii, R. V. Gamkrelidze, and E. F.
Mischenko). The formulation and the proof of this principle is one of the most
important results not only in the optimal control but also in mathematics of
the XX century. They solve a long standing problem for obtaining necessary
optimality conditions for dynamical processes in the presence of constraints
2
on the control and phase variables. This is one of the basic results of the
optimal control theory. This classical version was then improved by other
autors, e.g. L. Cesari [4], F. Clarke [6], M. Hestenes [10], E. B. Lee and L.
M. Larkus [16], L. D. Berkovitz [2], H. Sussmann [26] and etc.
We start with the following simple example:
Example 1.1 Let us x the positive constants C and T, and consider the
following control problem:
I(u(), v()) = T min
x(t) = u, x(0) = 0, x(T) = x
0
,
y(t) = v, y(0) = 0, y(T) = y
0
.
_
u
2
(t) +v
2
(t) C for almost every t [0, T],
where u() and v() are arbitrary measurable functions dened on the interval
[0, T] and satisfying the above written inequality for almost every t [0, T].
Roughly speaking, this control problem means that we want to reach the
point (x
0
, y
0
) from the origin in minimal time by a trajectory of the above
written control system. We set
T
=
_
x
2
0
+y
2
0
C
, u
(t) =
Cx
0
_
x
2
0
+y
2
0
and v
(t) =
Cy
0
_
x
2
0
+y
2
0
for each t [0, T
].
The corresponding trajectory is admissible (x
(), y
()). One can checked

that
x
(0) = 0, x
(T) = x
0
, y
(0) = 0, y
(T) = y
0
and
_
u
2
(t) + v
2
(t) = C
for every t [0, T
]. Moreover, I(u
(), v
()) =
_
x
2
0
+y
2
0
/C.
On the other hand, let u() and v() be arbitrary measurable functions
satisfying the inequality |(u(t), v(t))| C for almost each t from [0, T]. If
we denote by , ) the standard scalar product in IR
n
, we obtain that
x
2
0
+y
2
0
= (x
0
, y
0
), (x
0
, y
0
)) =
_
(x
0
, y
0
),
_
T
0
( x(t), y(t))
_
dt =
=
_
T
0
(x
0
, y
0
), (u(t), v(t))) dt
_
T
0
_
x
2
0
+y
2
0
_
u
2
(t) + v
2
(t) dt
3
_
x
2
0
+y
2
0
_
T
0
C dt = C T
_
x
2
0
+y
2
0
= C
_
x
2
0
+y
2
0
I(u(), v()).
Thus
I(u(), v())
_
x
2
0
+y
2
0
C
and hence, (u
(), v
()) is a solution of our problem.

Here we shall present a couple of simple versions of the Pontryagin maxi-
mum principle, under technical assumptions that will be considerably weak-
ened later.
1.1 A classical version for xed time problems
We assume that the following conditions hold true:
C1. n and m are positive integers;
C2. X is an open subset of R
n
;
C3. U is a closed subset of R
m
;
C4. t
0
and t
1
are xed real numbers such that t
0
t
1
;
C5. the following continuous maps
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X U [t
0
, t
1
] (x, u, t) L(x, u, t)
are given;
C6. for each (u, t) U [t
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x L(x, u, t) R
are continuously dierentiable with respect to x, and their partial
derivatives with respect to the x coordinate are continuous functions
of (x, u, t);
4
C7. x
0
and x
1
are given points from X;
C8. the set of all trajectory-control pairs T (T
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
] is the set of all pairs (x(), u()) such that
a) [t
0
, t
1
] t u(t) U is a measurable bounded map;
b) [t
0
, t
1
] t x(t) X is an absolutely continuous map;
c) x(t) = f(x(t), u(t)) for almost every t [t
0
, t
1
];
C9. for each element (x(), u()) fromT (T
[t
0
,t
1
]
(X, U, f) the functional I(x(), u())
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t), t) dt
C10. the reference trajectory-control pair (x
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) = x
1
;
c) I(x
(), u
()) I(x(), u()) for all (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) = x
1
;
Theorem 1.2 Assume that the data m, n, X, U, t
0
, t
1
, f, L, x
0
, x
1
satisfy
the conditions C1-C7, T (T
[t
0
,t
1
]
(X, U, f) and I are dened by C8-C9 and
(x
(), u
()) satises C10. Dene the Hamiltonian H to be the function

X U R
n
R R (x, u, p, p
0
, t) H(x, u, p, p
0
, t) R
given by
H(x, u, p, p
0
, t) := p
0
L(x, u, t) +
n
j=1
p
j
f
j
(x, u, t).
Then there exists a pair (,
0
) such that
E1. [t
0
, t
1
] (t) is an absolutely continuous map;
E2.
0
0, 1;
E3. ((t),
0
) ,= (0, 0) for every t [t
0
, t
1
];
E4. the adjoint equation holds, i. e.

i
(t) = p
0
L
x
i
(x
(t), u
(t), t)
n
j=1
p
j
f
j
x
i
(x
(t), u
(t), t)
for almost every t [t
0
, t
1
];
5
E5. the Hamiltonian maximization condition holds, that is
H(x
(t), u
(t), (t),
0
, t) = max H(x
(t), u, (t),
0
, t) : u U
0
, t
1
].
Remark 1.3 Since
H
p
i
(x, u, p, p
0
, t) = f
i
(x, u, t), i = 1, . . . , n,
the adjoint equation together with the equation of C8.c are called Hamiltons
equations, i.e. the pair (x
(t), u
(t)) satises the following system of equa-

tions:
d x
i
(t)
dt
=
H
p
i
(x
(t), u
(t), (t),
0
, t), i = 1, . . . , n,
d
i
(t)
dt
=
H
x
i
(x
(t), u
(t), (t),
0
, t), i = 1, . . . , n.
Remark 1.4 The map is referred to in the control theory as the adjoint
vector. The number
0
is the abnormal multiplier. The adjoint equation
says that
d
i
(t)
dt
=
0
L
x
i
(x
(t), u
(t), t)
n
j=1
j
(t)
f
i
x
i
(x
(t), u
(t), t).
In particular, if ((t),
0
) = (0, 0) for one value of t, then
0
= 0, so the
adjoint equation reduces to
d
i
(t)
dt
=
n
j=1
j
(t)
f
i
x
i
(x
(t), u
(t), t).
which is a linear homogeneous time-varying dierential equation for . There-
fore the fact that () = 0 implies that (t) = 0 for all t, so ((t),
0
) = (0, 0)
for all t. Hence the nontriviality condition can be stated by saying for some
t rather than for all t.
6
Remark 1.5 If we set
h(t) = H(x
(t), u
(t), (t),
0
, t),
then, formally, the derivative of h is given by
h(t) =
n
i=1
H
x
i
((t)) x
i
(t) +
n
j=1
H
u
j
((t)) u
j
(t) +
n
k=1
H
p
k
((t)) p
k
(t)+
+
H
t
((t)) =
n
j=1
H
u
j
((t)) u
j
(t) +
H
t
((t)),
where we have written (t) = (x
(t), u
(t), (t),
0
, t) and used the fact that
x
() and p() satisfy the Hamiltons equations. If we disregard the fact that
u
j
(t) need not exist (because u
j
is not assumed to be dierentiable or even
continuous), and if we pretend that
H
u
j
((t)) must vanish because the func-
tion u H(x
(t), u, (t),
0
, t) has a maximum at u = u
(t), then we end

up with
h(t) =
H
t
(x
(t), u
(t), (t),
0
, t))
This leads to the conjecture that if the dynamical law f and the Lagrangian
L do not depend on t then the function [t
0
, t
1
] t H(x
(t), u
(t), (t),
0
)
is constant. In fact, this conjecture is true.
1.2 A classical version for variable time interval prob-
lems
We assume that the following conditions are satised:
n
;
m
;
X U (x, u) f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X U (x, u) L(x, u)
are given;
7
C5. for each u U the maps
X x f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X x L(x, u) R
are continuously dierentiable, and their partial derivatives with re-
spect to the x coordinate are continuous functions of (x, u);
C6. x
0
and x
1
are given points from X;
C7. the set of all trajectory-control pairs T (T(X, U, f) is the set given by
T (T(X, U, f) :=
_
_
T (T
[t
0
,t
1
]
(X, U, f) : t
0
, t
1
R, t
0
t
1
_
,
where for t
0
, t
1
R such that t
0
t
1
, T (T
[t
0
,t
1
]
(X, U, f) is the set of
all pairs (x(), u()) such that
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
[t
0
,t
1
]
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t)) dt
C9. there exist real number t
0
and t
1
and a reference trajectory-control pair
(x
(), u
()) such that

a) x
(), u
() T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) = x
1
;
c) I(x
(), u
()) I(x(), u()) for all (x(), u()) T (T

[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) = x
1
;
0
, t
1
, f, L, x
0
, x
1
satisfy
[t
0
,t
1
]
t
0
, t
1
, (x
(), u
()) satisfy C9. Dene the Hamiltonian H to be the function

X U R
n
R (x, u, p, p
0
, t) H(x, u, p, p
0
) R
8
given by
H(x, u, p, p
0
) := p
0
L(x, u) +
n
j=1
p
j
f
j
(x, u).
0
) such that
E1. [t
0
, t
1
E2.
0
0, 1;
E3. ((t),
0
) ,= (0, 0) for every t [t
0
, t
1
];

i
(t) =
0
L
x
i
(x
(t), u
(t))
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t))
0
, t
1
];
E5. the Hamiltonian maximization condition holds with zero value, that is
0 = H(x
(t), u
(t), (t),
0
) = max H(x
(t), u, (t),
0
) : u U
0
, t
1
].
Remark 1.7 The only dierence between the conclusions of Theorem 1.2
and Theorem 1.6 is that in Theorem 1.6 an additional condition (that the
value of the Hamiltonian vanishes) holds true.
Example 1.1 (continuation) Now, we solve Example 1 by applying The-
orem 1.6. In this case, the conguration space X is R
2
, the control set U is
(u, v)
T
: u
2
+v
2
C
2
, the dynamical law is given by
f(x, y, u) = (u, v),
and the Lagrangian is identically equal to 1. The Hamiltonian H is given
by
H(x, y, u, v, p
x
, p
y
, p
0
) = p
x
u +p
y
v p
0
,
where p
x
and p
y
denote the two components of the adjoint variable p.
Assume that the trajectory-control pair (x
(), y
(), u
(), v
()) belongs
to T (T
[t
0
,t
1
]
(X, U, f) and is a solution of our minimum time problem. Then
9
Theorem 1.6 tells us that there exists a pair (,
0
) satisfying all the condi-
tions of the conclusion, where := (
x
,
y
). The adjoint equation is

x
(t) = 0,

y
(t) = 0.
Therefore the functions
x
and
y
are constant. Let a, b R be such that
x
(t) = a and
x
(t) = b for all t [t
0
, t
1
]. Now, if a and b are both equal to
zero, the function (
x
,
y
) would vanish identically and then the Hamiltonian
maximization condition would say that the function
U (u, v) H(x
(t), y
(t), u, v, 0, 0,
0
) =
0
is maximized by taking u = u
(t) and v = v
(t), a fact that would give

us no information about the values of u
(t) and v
(t). Fortunately, the

conditions of Theorem 1.6 imply that a and b can not both vanish. To see
this, observe that if a = b = 0 then it follows that
x
(t)
y
(t) 0.
But then the nontriviality condition tells us that
0
,= 0. So the value
H(x
(t), y
(t), u
(t),
x
(t),
y
(t),
0
) =
0
, which is not equal to zero. This
contradicts the fact that for our time-varying problem, the Hamiltonian is
supposed to vanish.
According to the Hamiltonian maximization condition, we obtain that for
almost each t [t
0
, t
1
]
au
(t) + bv
(t) = max
_
au +bv :
u
2
+v
2
C
_
.
It could be directly checked that the solution of this optimization problem
is:
u
(t) =
aC
a
2
+b
2
, v
(t) =
bC
a
2
+b
2
. (1)
Taking into account that x(t
0
) = 0, y(t
0
) = 0, x(t
1
) = x
0
and y(t
1
) = y
0
,
we obtain that
x
0
=
a(t
1
t
0
)C
a
2
+b
2
, y
0
=
b(t
1
t
0
)C
a
2
+b
2
.
These two equalities imply that
(a
2
+b
2
)(t
1
t
0
)
2
C
2
a
2
+b
2
= x
2
0
+y
2
0
.
10
Hence,
t
1
t
0
=
_
x
2
0
+y
2
0
C
.
Since the control system is autonomous, we may think without loss of gener-
ality, that t
0
= 0 and T
= t
1
t
0
. The identities x
0
= u
and y
0
= v
imply that
u
=
Cx
0
_
x
2
0
+y
2
0
, v
=
Cy
0
_
x
2
0
+y
2
0
.
One can verify that all conditions E1-E5 of Theorem 1.6 holds true.
Hence, if this problem has a solution (x
(), y
(), u
(), v
()) dened on
[0, T
], then for almost each t [0, T
] the optimal control values u
(t) and
v
(t) have to satisfy (1), and (x
(), y
()) will be the corresponding trajectory

connecting the origin with the point (x
0
, y
0
).
Example 1.8 We consider the so called one dimensional soft landing
problem in which it is desired to nd, for a given point (x
0
, y
0
) R
2
, a
trajectory-control pair of the system
x(t) = y(t), u [1, 1]
y(t) = u(t)
such that (x(), y()) goes from (x
0
, y
0
) to the origin in minimum time.
We solve this problem by applying Theorem 1.6. In this case, the cong-
uration space X is R
2
, the control set U is the interval [1, 1], the dynamical
law is given by
f(x, y, u) = (y, u),
and the Lagrangian is identically equal to 1.
The Hamiltonian H is given by
H(x, y, u, p
x
, p
y
, p
0
) = p
x
y +p
y
u p
0
,
where we are using p
x
and p
y
to denote the two components of the adjoint
variable p.
Assume that the trajectory-control pair (x
(), y
(), u
()) belongs to the

set T (T
[t
0
,t
1
]
(X, U, f) and is a solution of our minimum time problem. Then
11
Theorem 1.6 tells us that there exists a pair (,
0
) satisfying all the condi-
tions of the conclusion. Write (t) := (
x
(t),
y
(t)). The adjoint equation
the implies that

x
(t) = 0,

y
(t) =
x
(t).
Therefore the function
x
is constant. Let
x
(t) = a for all t [t
0
, t
1
]. Then
there exist a constant b such that
y
(t) = b at for all t [t
0
, t
1
].
Now, if a and b are both equal to zero, the function
y
would vanish identi-
cally and then the Hamiltonian maximization condition would say that the
function
[1, 1] u H(x
(t), y
(t), u, 0, 0,
0
) =
0
(t), a fact that would give us no information

about the value of u
(t). Fortunately, the conditions of Theorem 1.6 imply

that a and b can not both vanish. To see this, observe that if a = b = 0 then
it follows that
x
(t)
y
0. But then the nontriviality condition tells us
that
0
,= 0. So the value H(x
(t), y
(t), u
(t),
x
(t),
y
(t),
0
) =
0
, which
is not equal to zero. This contradicts the fact, for our time-varying problem,
the Hamiltonian is supposed to vanish.
Hence, a and b can not vanish simultaneously. The are two possibilities:
either a = 0 or a ,= 0.
Suppose rst that a = 0. Then b ,= 0, and Hamiltonian maximization
tells us that [1, 1] u bu is maximized by u = u
(t). If b > 0, this

implies that u
(t) = 1 for all t. If b < 0, it follows that u
(t) = 1 for all t.

Now assume that a ,= 0. Then (t) := b at is a nonconstant linear
function on t. So, vanishes at most once on the interval [t
0
, t
1
]. If never
vanishes on [t
0
, t
1
], or vanishes at one of the endpoints, then u
(t) is always
equal to 1 (if (t) > 0) always equal to 1 (if (t) < 0). If () = 0 for
some ]t
0
, t
1
[, then u
(t) will be equal to 1 for t < and to 1 for > t

(if changes sign from positive to negative) or u
(t) will be equal to 1 for

t < and to 1 for > t (if changes sign from negative to positive). Thus
we have proved that u
is one of the following four types:

a. constantly equal to 1;
b. constantly equal to 1;
12
c. constantly equal to 1 for t < and to 1 for t > for some ]t
0
, t
1
[;
d. constantly equal to 1 for t < and to 1 for t > for some ]t
0
, t
1
[;
For a problem where the control set U is a compact convex subset of R
m
,
a control u
that takes values in the set of extreme points of U is said to be

a bang-bang control.
Let 1, (x
0
, y
0
) R
2
, and let us consider the following system of
ODEs on the interval [t
0
, t
1
]:
x(t) = y(t), x(0) = x
0
,
y(t) = , y(0) = y
0
.
One can verify that for each t [t
0
, t
1
], its solution is
x(t) = x
0
+y
0
(t t
0
) +
(t t
0
)
2
2
,
y(t) = y
0
+(t t
0
).
By setting t t
0
=
y(t) y
0
2
we obtain that
x(t) = x
0
+
y(t)
2
y
2
0
2
, (2)
i.e. for each t [t
0
, t
1
] the point (x(t), y(t)) belong to the graph Graph f
c
of
the following function:
f
c
(x) = c
y
2
2
,
where c are suitable chosen real parameter.
If there exists 1 and a constant c such that (x
0
, y
0
) Graph f
c
and (0, 0) Graph f

c
, then it can be proved that will be the optimal value

of the control and the path belonging to Graph f
c
and connecting the points

(x
0
, y
0
) and (0, 0) will be the optimal trajectory.
Let there exists no element of the above written family of functions such
that (x
0
, y
0
) and (0, 0) belongs to its graph. We have proved that all optimal
controls are bang-bang and piece-wise constant with at most one switching.
This implies the existence of two dierent elements f
c
1
and f
c
2
of the above
written family of functions family such that (x
0
, y
0
) Graph f
c
1
and (0, 0)
Graph f
c
2
. At the point
(x
s
, y
s
) Graph f
c
1
Graph f
c
2
13
the optimal control switches its value. Then the the optimal trajectories
is the path belonging to Graph f
c
1
and connecting the points (x

0
, y
0
) and
(x
s
, y
s
), and after that is the path belonging to Graph f
c
2
and connecting
the points (x
s
, y
s
) and (0, 0). The corresponding value of the optimal control
is until the switching point (x
s
, y
s
) and after that is .
Notice that the switchings of the optimal control are determined by the
function , which in our example happens to be
y
. That is why this function
is called switching function for this problem.
Notice also, that we have made essential use of all conditions given by
theorem 1.6. In particular, the nontriviality condition was crucial, and the
fact that the value of the Hamiltonian is equal to zero was also decisive.
14
2 Preliminaries
2.1 Some properties of the measurable maps
A subset S of R
n
is said to have zero measure if for each > 0 there is a
countable union of balls B
1
, B
2
, . . . of volume
1
, 2, . . ., respectively, such
that
i=1
i
< and S
_
i=1
B
i
.
The Lebesgue measurable subsets of R
n
are those obtained by countable
intersections, unions and complements, starting from open sets and sets with
measure zero.
Two maps f, g : O S from some set O R
n
into some other set S are
said to be equal almost everywhere (a.e.) if the set
x O : f(x) ,= g(x)
has zero measure. In general, a property is said to hold almost everywhere
provided that it only fails in a set of measure zero. For example, let be a
metric space and f
i
: i = 1, 2, . . . be is a sequence of maps from O R
n
into . Then it is said that f
i
converges to f almost every where if the set
x O : f(x) , g(x)
has measure zero. This will be denoted by f
i
f (a.e.).
Let I be an interval in R and be a metric space. A piece-wise constant
map f : I is one that is constant in each element I
j
of a nite partition
of I into subintervals. A map f : O is a measurable map if there exists
some sequence of piece-constant maps f
i
, i = 1, 2, . . . so that f
i
con-
verges to f almost everywhere. Clearly, each continuous map is measurable,
and in general g f is measurable if f is measurable and g is continuous. Let
be a separable metric space, that is it has a countable dense subset. Then
a map f : I is measurable if and only if for each open set V of the
set f
1
(V ) is Lebesgue measurable.
A map f : I is essentially bounded if it is measurable and there
exists a compact subset K | such that f(t) K for almost all t I.
Assuming that is a separable metric space, f is essentially bounded if and
only if there is a sequence of piece-wise constant maps f
i
converging almost
15
every where to f which are equibounded, that is f
n
(t) K for almost every
t, for some compact K independent of n.
For measurable maps f : I C
n
(C is the set of all complex numbers)
on a nite interval 1 one may dene the Lebesgue integral
_
I
f(t) dt via the
limits of integrals of suitable sequences approximating piece-wise constant
maps. One can also dene integrals of maps on innite intervals in a similar
fashion. An integrable map is one for which
_
I
|f(t)| dt is nite; the map f
is locally integrable if
_
t
1
t
0
|f(t)| dt <
for each t
0
< t
1
, t
0
, t
1
1.
A map f : I C
n
dened on a compact interval I = [t
0
, t
1
] is said to be
absolutely continuous if for each > 0 there exists > 0 such that, for every
k and for every sequence of points
t
0
a
1
< b
1
< a
2
< b
2
< < a
k
< b
k
t
1
so that
k
i=1
(b
i
a
i
) < , it holds that
k
i=1
|f(b
i
) f(a
i
)| < . The map f is
absolutely continuous if and only if there is an integrable map g such that
f(t) = f(t
0
) +
_
t
t
0
g() d (3)
for all t I. An absolutely continuous map f is dierentiable almost every-
where, and
f(t) = g(t)
holds for almost all t, if g is as in (3). If I is an arbitrary, not necessarily
compact, interval, a locally absolutely continuous map f : I C
n
is one
whose restriction to each compact interval is absolutely continuous.
2.2 Norms of matrices
Lemma 2.1 If A is a complex mn matrix, by |A| we denote its operator
norm
|A| = max
x=1
|Ax|,
for which the following properties hold true:
16
N1. for each matrix A, |A| = 0 is equivalent to A = 0;
N2. for each matrix A and for each real number , |A| = [[ |A|;
N3. for each matrices A and B, |A +B| |A| +|B|.
N4. for each matrices A and B, |A B| |A| |B|.
Proof. The property N1 is evident. The equalities
|A| = [[ max
x=1
|Ax| = [[ |A|
show that the property N2 also is true. The relations
|A +B| = max
x=1
|(A +B)x| max
x=1
|Ax| + max
x=1
|Bx| = |A| +|B|
prove the validity of the property N3. For x ,= 0 we have that
|Ax| = |x|
_
_
_
_
_
A
x
|x|
_
_
_
_
_
|x| |A|.
Hence
|A B| = max
x=1
|ABx| |A| max
x=1
|Bx| = |A| |B|.
This completes the proof of N4.
If x C
n
, we denote the Euclidean norm by
|x| =
_
[x
2
1
+x
2
2
+ +x
2
n
.
Then |A| is the the square root of the largest eigenvalue of the following Her-
mitian matrix: A
A, where by A
we have denoted the conjugate transposed

matrix of the matrix A. This follows easily by the denition and
|Ax|
2
= Ax, Ax) = A
Ax, x) .
If we dene the norm
|x|
1
= max
1jn
[x
i
[,
17
then the corresponding
|A|
1
= max
x
1
=1
|Ax|
1
= max
max
1jn
[x
j
[ = 1
max
1im
j=1
a
ij
x
j
max
1im
n
j=1
[a
ij
[ =
n
j=1
[a
i
0
j
[ .
On the other hand, for x
0
= (x
0
1
, . . . , x
0
n
) with x
0
j
= a
i
0
j
/[a
i
0
j
[ we have that
|Ax
0
|
1
= max
1im
n
j=1
a
ij
a
i
0
j
[a
i
0
j
[
=
n
j=1
[a
i
0
j
[ .
Hence,
|A|
1
= max
1im
n
j=1
[a
ij
[ .
If we dene the norm
|x|
2
=
n
i=1
[x
i
[,
then the corresponding
|Ax|
2
=
m
i=1
j=1
a
ij
x
j
i=1
n
j=1
[a
ij
[ [x
j
[ =
n
j=1
m
i=1
[a
ij
[ [x
j
[ =
j=1
max
1kn
m
i=1
[a
ik
[ [x
j
[ = max
1kn
m
i=1
[a
ik
[ |x| =
m
i=1
[a
ik
0
[ |x|.
On the other hand, for x
0
= (x
0
1
, . . . , x
0
n
) with x
0
k
0
= 1 and x
0
j
= 0 for j ,= k
0
we have that
|Ax|
2
=
m
i=1
j=1
a
ij
x
j
=
m
i=1
[a
ik
0
[ .
Hence,
|Ax|
2
= max
1kn
n
i=1
[a
ik
[ .
2.3 Some properties of the initial-value problems
The following result is known as the Gronwall Lemma. It is central to the
proof of uniqueness and well-posedness of ODEs. By interval we mean either
nite either innite interval. By R
+
we denote the set of all nonnegative real
numbers.
18
Lemma 2.2 Assume given an interval I R, a constant c 0, and two
functions , : I R
+
such that is locally integrable and is continuous.
Suppose further that for some t
0
I it holds that
(t) (t) := c +
_
t
t
0
()() d
for all t t
0
, t I. Then it must hold that
(t) ce
_
t
t
0
(s) ds
Proof. Note that (t) = (t)(t) (t)(t) almost everywhere, so (t)
(t)(t) 0 for almost all t We set
(t) := (t)e
_
t
t
0
(s) ds
.
Then is a locally absolutely continuous function, and
(t) = [ (t) (t)(t)]e
_
t
t
0
(s) ds
0.
Hence, is nonincreasing, and thus
(t)]e
_
t
t
0
(s) ds
= (t) (0) = c,
from which the conclusion follows.
Let us consider the following initial value problem (IVP):
x(t) = f(x(t), t), x(t
0
) = x
0
, (4)
where x(t) belongs to the open subset X of R
n
. A solution of this (IVP) is
each locally absolutely continuous map
x : I X
19
such that the integral equation
x(t) = x
0
+
_
t
t
0
f(x(), ) d
holds for all t I.
In order for this integral to be meaningful, we rst remark that f(x(), )
has to be measurable. For that reason, we assume that
f : X I R
n
is a vector function, where I is an interval in R, X is an open subset of R
n
,
and that the following two properties hold:
(H1) f(x, ) : I R
n
is measurable for each xed x X;
(H2) f(, t) : X R
n
is continuous for each xed t I.
Under this hypotheses, the function f(x(t), t) is measurable as a function of
t, for any given continuous function x(). We now state a basic theorem for
existence and uniqueness of solutions of (IVP):
Theorem 2.3 Assume that f : X I R
n
satises the assumptions (H1)
and (H2), where X is open and I R is an interval, and the following two
conditions also hold:
(H3) f is locally Lipschitz on x, i.e. for each x
0
X there exists a real
number > 0 and a locally integrable function : I R
+
such that
the ball B(x
0
, ) of radius centered at x
0
is contained in X and
|f(x, t) f(y, t)| (t)|x y|
for all t I and x, y B(x
0
, ).
(H4) f is locally integrable on t, i. e. for each xed x
0
there is a locally
integrable function : I R
+
such that
|f(x
0
, t)| (t)
for almost all t.
20
Then, for each pair (x
0
, t
0
) I X there is some subinterval J I open
relative to I and there exists a solution x of (IVP) on J with the following
property: If y : J
X is any other solution of the same (IVP), where

J
I, the necessarily
J
J and y x on J
.
The solution x is called the maximal solution of the (IVP) in the interval
I.
The next theorem asserts that the solutions of (IVP) depend continuously
on the initial conditions as well as on the right-hand side of the equation:
Theorem 2.4 Assume that f and h are two maps X I R
n
that satisfy
the assumptions (H1) (H4), where X is open and I = [t
0
, t
1
] R is a
bounded closed interval. Let : I R
+
be an integrable function, be a
positive real number and x : I X be a solution of
x(t) = f(x(t), t)
such that the -neighbourhood of its range:
K = x R
n
: |x x(t)| for some t [t
0
, t
1
]
is included in X. Let
H(t) :=
_
t
t
0
h(x(s), s) ds, t I
C := sup|H(t)| : t I. Assume that
maxC, |x(t
0
) y
0
|

2
e
_
t
1
t
0
(s) ds
and, with g := f +h,
|g(x, t) g(y, t)| (t)|x y| for all x, y X and t I.
Then the solution of the perturbed equation
y(t) = g(y(t), t) = f(y(t), t) +h(y(t), t), y(0) = y
0
exists on the entire interval [t
0
, t
1
], and is uniformly close to the original
solution in the following sense:
max|y(t) x(t)| : t I (|y
0
x
0
| +C).e
_
t
1
t
0
(s) ds
.
21
2.4 Linear dierential equations
Assume that A() is an n n matrix of locally (essentially) bounded (mea-
surable) function on an interval I with values in K (K is R or C). Consider
the following initial value problem
x(t) = A(t)x(t), x(t
0
) = x
0
(5)
Without loss of generality, we may assume that I = R simply by dening
A(t) 0 outside I. Since the right-hand side is globally Lipschiz, there is an
unique solution for all t t
0
. Solving the equation for t < t
0
is equivalent
to solving the reversed-time equation x(t) = A(t)x(t), which has also a
globally Lipshits right-hand side. Thus, the solution of (5) is dened on
(, +). Observe that in the complex case, the resulting system in R
2n
is
again linear, so the conclusions apply for both real and complex equations.
A convenient way to study all solutions of a linear equation, for all possible
initial values simultaneously, is to introduce the matrix dierential equation
X(t) = A(t) X(t), X(t

0
) = E, (6)
where E is the identity matrix, X(t) K
nn
for each t R and by we
have denoted the usual matrix multiplication. Since this can be seen as a
linear equation over K
n
2
, it has a global solution on (, +). We denote
by (t, t
0
) the solution of (6) at time t. This is dened for each real t
0
and t
and is called fundamental solution of (6) associated to A. Observe that the
solution of (5) is then given by (t, t
0
)x
0
. More generally, the solution of a
nonhomogeneous equation
x(t) = A(t)x(t) + b(t), x(t
0
) = x
0
(7)
on the interval [t
0
, t], where b() is any locally integrable n-vector of functions,
is given by
x(t) = (t, t
0
)x
0
+
_
t
t
0
(t, s)b(s) ds.
This formula can be veried by derivation. It is well dened because conti-
nuity of implies that the integrand is locally integrable.
Lemma 2.5 The following properties of hold true:
22
1. The equation (6) says that for all t and t
0
from R
t
(t, t
0
) = A(t)(t, t
0
);
2. The equation (6) says also that for all t and t
0
from R
(t
0
, t
0
) = E;
3. For all t, t
0
and t
1
from R
(t, t
0
) = (t, t
1
) (t
1
, t
0
);
4. For all t and t
0
from R
(t, t
0
)
1
= (t
0
, t),
where by (t, t
0
)
1
we have denoted the inverse matrix of the matrix
(t, t
0
);
5. For all t and t
0
from R
t
(t
0
, t) = (t
0
, t) A(t);
Proof. The identities 1 and 2 are evident. To prove the identity 3 we
x for a moment t
0
and t
1
and dene the following functions F, G : R R
nn
:
F(t) = (t, t
0
) and G(t) = (t, t
1
) (t
1
, t
0
). According to 2, we obtain
that
F(t
1
) = (t
1
, t
0
) = E (t
1
, t
0
) = (t
1
, t
1
) (t
1
, t
0
) = G(t
1
).
Applying 1, we obtain that
d
dt
F(t) =

t
(t, t
0
) = A(t) (t, t
0
) = A(t) F(t)
and
d
dt
G(t) =

t
(t, t
1
) (t
1
, t
0
) = A(t) (t, t
1
) (t
1
, t
0
) = A(t) G(t).
23
Hence, we obtained that F() and G() are solutions of the following IVP:
d
dt
X(t) = A(t) X(t), X(t
1
) = (t
1
, t
0
).
Since this initial value problem has an unique solution, we obtain that F(t) =
G(t) for each real t. This completes the proof of 3.
Applying 3, we obtain that
(t, t
0
) (t
0
, t) = (t
0
, t
0
) = E.
Hence, the matrix (t, t
0
) has an inverse matrix and 4 holds true.
The identity 4 shows us that the derivative of (t
0
, t) exists. By dier-
entiating with respect to t the identity
(t, t
0
) (t
0
, t) = E
we obtain that
t
(t, t
0
) (t
0
, t) + (t, t
0
)

t
(t
0
, t) = 0.
Taking into account that
t
(t, t
0
) = A(t) (t, t
0
)
we obtain according to 3 and 4 that
t
(t
0
, t) = (t, t
0
)
1
A(t)(t, t
0
)(t
0
, t) = (t, t
0
)
1
A(t) = (t
0
, t)A(t),
i.e. the identity 5 also hold true.
It could be directly veried that for each xed real numbers t and t
0
,
(t, t
0
) can be represent by a power series expansion (called the Peano-Baker
formula:
(t, t
0
) = E +
_
t
t
0
A(s
1
) ds
1
+
_
t
t
0
_
s
1
t
0
A(s
1
)A(s
2
) ds
2
ds
1
+
+
_
t
t
0
_
s
1
t
0

_
s
k1
t
0
A(s
1
)A(s
2
) A(s
k
) ds
k
. . . ds
2
ds
1
+ (8)
24
This is obtained formally when solving by iteration for X() the xed
point equation
X(t) = E +
_
t
t
0
A(s)X(s) ds
corresponding to the initial-value problem (6). The series (8) converges
uniformly with respect to t and t
0
. Indeed, since the matrices are locally
bounded, for each interval I there exists a constant K such that |A(s)| < K
for almost all s I. Thus the k-th term in the series is bounded by
K
k
(t t
0
)
k
/k!. The partial sums of the series obtained by dierentiating
each terma with respect to t are simply the products of A(t) by the par-
tial sums of the above series. Hence, the term-wise dierentiated series also
converges. It follows that the derivative of the sum exists and it equals this
term-wise derivative, that is A(t)(t, t
0
). So (t, t
0
) is a solution of (6) in
[t
0
, t] and hence is the fundamental solution matrix introduced earlier.
An advantage of this derivation is that it provides an explicit formula for
the solution in terms of the power series (8). This forms the basis of some
numerical methods for solving of linear systems.
For the case of constant matrix A, we denote the corresponding solution
(t, t
0
) by e
(t t
0
)A
. It could be veried that
e
(t t
0
)A
= E + (t t
0
)A +
(t t
0
)
2
2
A
2
+ +
(t t
0
)
k
k!
A
k
+ +
Clearly,
_
_
_
_
_
e
(t t
0
)A
_
_
_
_
_

k=0
(t t
0
)
k
k!
_
_
_A
k
_
_
_ = e
(t t
0
) |A|
25
3 The transversality conditions
We now move one step further, and present two even more general versions of
the Pontryagin maximum principle. The new versions are still rather simple
as far as technical hypotheses go, but contain a new ingredient, namely, the
transversality condition. This shows up an additional necessary condition for
optimality for a problem in which the terminal constraint, instead of being of
the form x(t
1
) = x
1
considered so far, is of the more general form x(t
1
) S,
where S is a given subset of X.
3.1 Cones
Let X be a nite-dimensional space. Usually, X will be R
n
. By X
we denote
its dual, that is the set of all linear continuous functionals x
: X R. If
X = R
n
, then X
= R
n
. If x X and x
, we denote its scalar product

as follows:
x
, x) =
n
i=1
x
i
x
i
.
By denition, X
= (X
= X.
Let C be a convex subset of X and let x
0
C, we denote by Lin (C) the
intersection of all linear subspaces of X containing the set C x
0
. It is said
that x belong to the relative interior ri C of C i there exists a neighborhood
U of x such that U Lin (C) is contained in C.
A cone in X is a subset K of X, such that:
C1. K ,= ;
C2. K is a union of rays (i.e. whenever x K and > 0, it follows that
x K).
If K is a cone in X, then not necessarily 0 K. In general, a cone need
not be convex. A cone K is convex if and only if it is closed under addition,
that is x
1
+x
2
K whenever x
1
and x
2
K.
Since X is endowed with a topology, we can talk about closed cones.
Clearly, if K is a closed cone, then necessarily 0 K. If K is a cone, by K
we shall denote its closed hull, i.e. K is the set of the limits of all convergent
sequences of elements of the cone K.
26
The dual cone K
of a cone K in the real linear space X is dened as

the set
K
:= x
: x
, x) 0 for all x K .
Clearly, K
is a convex closed cone in X
(the scalar product depends in a

continuous way on his arguments). Also, K
= K
.
Lemma 3.1 Let K be a convex closed cone in X and
x
, x
0
) 0 (9)
for all x
. Then x K.
Proof. Let us assume that x
0
, K. Applying the Hahn-Banach theorem,
we obtain the existence of a real number > 0 and of an element x
such that for all x K

x
, x
0
) x
, x) +. (10)
Let us assume that there exist x K for which x
, x) > 0. But then x K

for each > 0, and hence, x
, x) + whenever +. But this

contradicts to (10). Thus we have obtained that x
. Since K is a
closed convex cone, then 0 K. By setting x = 0 in (10), we obtain
x
, x
0
) x
, 0) + = .
But this contradicts to the assumption (9). The obtained contradiction shows
that the assumption x
0
, K is wrong. This completes the proof of Lemma
3.1.
Since K
is a convex closed cone in X
, we can dene its dual cone K
in X:
K
:=
_
x X, x
, x) 0 for all x
_
.
Clearly, K K
. If K is a convex closed cone, the Lemma 3.1 implies that

K
K, and hence K
= K.
Lemma 3.2 Let the vectors a
0
, a
1
, . . . , a
p
have the following property: if for
some vector x the following equalities hold true: a
i
, x) = 0 for all i =
1, . . . , p, then also a
0
, x) = 0. Then the vector a
0
is a linear combinations
of the vectors a
i
, i = 1, . . . , p.
27
Proof. Without loss of generality, we may think that the vectors a
i
, i =
1, . . . , p, are linearly independent. According to the above written property,
the following linear system of equations
a
i
, x) = 0, i = 1, . . . , p
a
0
, x) = 1
has no solution. Then the vectors a
0
, a
1
, . . . , a
p
are are linearly dependent.
Since the vectors a
i
, i = 1, . . . , p, are linearly independent, it follows that a
0
is a linear combinations of the vectors a
i
, i = 1, . . . , p.
Lemma 3.3 (Farkas lemma) Let the vectors a
0
, a
1
, . . . , a
p
have the fol-
lowing property: if for some vector x the following inequalities hold true:
a
i
, x) 0 for all i = 1, . . . , p, then also a
0
, x) 0. Then the vector a
0
belong to the cone generated by the vectors a
i
, i = 1, . . . , p.
Proof. If a
0
= 0, the Lemma holds true. Let a
0
,= 0. We shall prove
Lemma 3.3 by induction on the number p.
Let p = 1 and a
1
, x) = 0. Then a
1
, x) 0 and a
1
, x) 0. According
to the above written property, we obtain that a
0
, x) 0 and a
0
, x) 0,
i.e. a
0
, x) = 0. Then Lemma 3.3 implies that a
0
=
1
a
1
. If we assume that
a
0
, a
1
) < 0, then 0 |a
0
|
2
= a
0
, a
0
) 0. Hence a
0
= 0 and a
0
, a
1
) = 0.
The obtained contradiction shows that a
0
, a
1
) 0. This and the equality
a
0
=
1
a
1
imply that
1
0. Thus the assertion holds true for p = 1.
Let us assume that the assertion holds true for some positive integer p.
We shall prove its validity for p + 1. As in the case p = 1, we obtain that
a
0
=
1
a
1
+ +
p
a
p
+
p+1
a
p+1
.
Without loss of of generality, we may think that for some positive integer s
with 1 s < p + 1 the following inequalities hold true:
1
0,
2
0, . . . ,
s
0,
s+1
< 0, . . .
p+1
< 0.
We set
c = a
0
s+1
a
s+1

p
a
p
=
p+1
a
p+1
+
1
a
1
+ +
s
a
s
. (11)
Let us x a vector x for which the following inequalities hod true: a
i
, x) 0
for all i = 1, . . . , p. If a
p+1
, x) 0, then a
0
, x) 0 and the inequalities
28
s+1
< 0, . . . ,
p
< 0 and (11) imply that c, x) 0. If a
p+1
, x) > 0, then
the inequalities
p+1
< 0,
1
0, . . . ,
s
0 and (11) imply that c, x) < 0.
According to the inductive assumption, we obtain that there exist non-
negative numbers
1
, . . . ,
p
such that c =
1
a
1
+ +
p
a
p
. Then
a
0
= c +
s+1
a
s+1
+ +
p1
a
p1
+
p
a
p
=
=
1
a
1
+ +
s
a
s
+
+(
s+1
+
s+1
)a
s+1
+ + (
p1
+
p1
)a
p1
+ (
p
+
p
)a
p
.
In this presentation of a
0
is a linear combination of the vectors a
1
, . . . , a
p
and
the number of the negative -s is less than in the rst one. Continuing in the
same way, we can prove the assertion for p+1. This completes the proof.
Lemma 3.4 Let us consider the cone
K = x R
n
: x
i
, x) 0, i = 1, 2, . . . , m .
Then K is a closed convex cone and its dual cone is
K
:=
_
x
R
n
: x
=
m
i=1
i
x
i
,
i
0, i = 1, . . . , m
_
.
Proof. Clearly, K is a convex and closed cone. Let x
be an arbitrary
element from the dual cone K
. Then x
, x) 0 for each element x

K. This means that the inequalities x
i
, x) 0, i = 1, . . . , m, imply the
inequality x
, x) 0. According to the Farkas lemma, we obtain that

x
C :=
_
x
R
n
: x
=
m
i=1
i
x
i
,
i
0, i = 1, . . . , m
_
.
On the other hand, the set C is contained in K
. This completes the proof.
3.2 Boltyanskii tangent cone to a set at a point

Let S be an arbitrary subset of X and let x
0
S. The vector v X is
called tangent vector to S at x
0
i there exists a function : R X such
29
that x
0
+ v + () S for all suciently small > 0 and
()
0 as
0. Clearly, if v is a tangent vector to S at x
0
and > 0, then v is also
a tangent vector to S at x
0
.
Unfortunately, the knowledge of the set of tangent vectors to S at x
0
is not sucient to make some conclusions about the properties of S in a
neighborhood of the point x
0
. For that reason, it is introduced the so called
Boltyanskii tangent cone K
S
(x
0
) to the set S at the point x
0
. This cone can
be characterized in the following way: for each element v
0
from the relative
interior of K
S
(x
0
) there exist a convex cone K
v
0
K
S
(x
0
), a neighborhood
U
v
0
of the origin and a continuous map : K
v
0
U
v
0
S such that
a) v
0
belongs to the relative interior of the cone K
v
0
;
b) Lin K
S
(x
0
) = Lin K
v
0
;
c) (v) = x
0
+v +r(v) where
r(v)
|v|
0 as v 0.
In order to construct a Boltyanskii tangent cone we need the following
version of the implicit function theorem (cf. for example [20], p. 191):
Theorem 3.5 Let x R
n
, y R
k
and g : R
m
R
k
R
n
be a continu-
ous functions satisfying the following condition: There exists a nonsingular
matrix A = (a
ij
)
nn
such that
|g(x, y) Ax| r
_
_
|x|
2
+|y|
2
_
with
r()
0 as 0. Then for each suciently small |y| there exists

x(y) such that g(x(y), y) 0 and
|x(y)|
|y|
0 as y 0.
Remark 3.6 The assumptions of Theorem 3.5 imply that
a) g(0, 0) = 0;
b) if g = (g
1
, g
2
, . . . , g
n
)
T
, then all partial derivatives of the functions g
i
at the point (0, 0) exist and

y
s
g
i
(0, 0) = 0, i = 1, . . . , n, s = 1, . . . , k;
30
c) the matrix
_

x
j
g
i
(0, 0)
_
nn
is a nonsingular matrix.
If all functions g
i
, i = 1, . . . , n, are continuously dierentiable on a neigh-
borhood of the point (0, 0), one can easily veried that the conditions a), b)
and c) imply that the assumptions of the above written theorem hold true.
Proof. The assumptions of the Theorem 3.5 imply that
g(x, y) = Ax +r
1
(x, y), (12)
|r
1
(x, y)| r
_
_
|x|
2
+|y|
2
_
.
Let us dene the following map
f(x, y) = x A
1
g(x, y).
The relation (12) implies that
f(x, y) = A
1
r
1
(x, y), (13)
|f(x, y)|
_
_
_A
1
_
_
_ r
_
_
|x|
2
+|y|
2
_
.
We set
r() = sup
0t
r(t).
Since
r()
0 as 0, for each > 0 there exists > 0 such that

r(t)
t
for each t [0, ]. Let t be an arbitrary element from the interval
[0, ]. Then
r(t)
t
=
1
t
sup
[0,t]
r() sup
[0,t]
r()
.
Hence
r()
0 as 0. Moreover, the very denition of r() implies that

it is an non decreasing function.
Let c := |A
1
|. For each y we set
(y) := inf
_
: cr
_
_
|y|
2
+
2
_

_
. (14)
31
For all suciently small |y|, we have that
cr
_
_
|y|
2
+|y|
2
_
= cr
_
2|y|
_
|y|,
and hence (y) |y|. The very denition (14) implies that for each y there
exists a positive number
(y) such that

(y)
(y) (y) +|y|

2
.
Moreover, (14) implies that
(y) |y|
2
< cr
_
_
|y|
2
+ ((y) |y|
2
)
2
_
cr
_
|y| +[(y) |y|
2
[
_
. (15)
The last inequality follows from
_
|y|
2
+ ((y) |y|
2
) |y|+[(y) |y|
2
[
and from the fact that is a non decreasing function. Since (y) |y|, we
obtain that
cr
_
|y| +[(y) |y|
2
[
_
cr (2|y|) .
Hence,
(y) |y|
2
cr (2|y|) =
(y)
|y|
2c
r (2|y|)
2|y|
+|y|
which proves that
(y)
|y|
0 as y 0. But then, the inequalities (15) imply
that

(y)
|y|
0 as y 0. Hence, for |x|
(y) the following inequalities

hold true:
|f(x, y)| cr
_
_
|x|
2
+|y|
2
_

_
_
(y)
2
+|y|
2
_

(y).
Thus we have proved that the image of the ball
B(
(y)) := x R
n
: |x|
(y)
under the continuous map f is contained in B(
(y)). According to the

Brouwers xed point theorem, we obtain the existence of a point x(y), such
that
x(y) = f(x(y), y), |x(y)|
(y),
32
and hence, g(x(y), y) = 0. Also, the inequality
x(y)
|y|

x(y)
|y|
implies that
x(y)
|y|
0 as y 0.
Example 3.7 Let
S = x R
n
: f
i
(x) = 0, i = 1, 2, . . . , m ,
where f
1
, i = 1, 2, . . . , m are continuously dierentiable functions. Let x
0
S
and the gradients f
i
(x
0
) are linearly independent vectors. Then the Boltyan-
skii tangent cone K
S
(x
0
0
is
K
S
(x
0
) =
_
x R
n
:
_
f
i
(x
0
), x
_
= 0, i = 1, 2, . . . , m
_
.
Proof. By f
(x
0
) we denote the mn-matrix whose rows are the vector-
rows f
i
(x
0
), i = 1, 2, . . . , m. Then the relation x K
S
(x
0
) can be written
as f
(x
0
)x = 0. By
_
f
(x
0
)
_
T
we denote its transpose matrix. We set M =
f
(x
0
)
_
f
(x
0
)
_
T
. If we assume that the m m-matrix M is singular, then
there exists a non vanishing vector y such that My = 0. But,
y, My) =
_
_
f
(x
0
)
_
T
y,
_
f
(x
0
)
_
T
y
_
=
_
_
_
_
_
f
(x
0
)
_
T
y
_
_
_
_
2
=
m
i=1
f
i
(x
0
)y
i
= 0,
i. e.
m
i=1
f
i
(x
0
)y
i
= 0.
The obtained contradiction shows that the matrix M is non singular. Let us
consider the following system of equations:
g
i
(x, y) := f
i
_
x
0
+x +
_
f
(x
0
)
_
T
y
_
_
x, f
i
(x
0
)
_
= 0, i = 1, 2, . . . , m.
One can directly checked that
g
i
(0, 0) = 0, i = 1, . . . , m,
g
i
x
j
(0, 0) = 0, i = 1, . . . , m, j = 1, . . . , n,
33
and the matrix
_
g
i
y
j
(0, 0)
_
nn
is just the matrix M, and hence it is non
singular. According to the Remark after the Theorem 3.5, for all suciently
small |x| there exists a smooth function y(x) with
y(x)
|x|
0, whenever
x 0. We set Q := K
S
(x
0
) and
(x) := x
0
+x +
_
f
(x
0
)
_
T
y(x).
Let us x so small that the solution y(x) is well dened for all x with
|x| . The denition of K
S
(x
0
) implies that
f
i
((x)) = 0, i = 1, . . . , m,
for all x K
S
(x
0
) B
(0) i. e. (x) S. Thus, we have shown that K

S
(x
0
)
is a Boltyanskii tangent cone K
S
(x
0
0
.
Example 3.8 Let
S = x R
n
: f
i
(x) = 0, i = 1, 2, . . . , m, g
j
(x) 0, j = 1, . . . s ,
where f
i
and g
j
, i = 1, 2, . . . , m, j = 1, . . . , s, are continuously dierentiable
functions. Let x
0
S. We set
I(x
0
) = j : g
j
(x
0
) = 0.
Assuming that the gradients f
i
(x
0
), i = 1, 2, . . . , m, are linearly independent
vectors, we have that the Boltyanskii tangent cone K
S
(x
0
) to the set S at the
point x
0
is the set
_
x R
n
:
_
f
i
(x
0
), x
_
= 0,
_
g
j
(x
0
), x
_
< 0, i = 1 . . . , m, j I(x
0
)
_
.
Proof. One can directly checked that
g
i
(0, 0) = 0, i = 1, . . . , m,
g
i
x
j
(0, 0) = 0, i = 1, . . . , m, j = 1, . . . , n,
and the matrix
_
g
i
y
j
(0, 0)
_
nn
singular. According to the Remark after the Theorem 3.5, for all suciently
34
small |x| there exists a smooth function y(x) with
y(x)
|x|
0, whenever
x 0. Let x K
S
(x
0
). We set Q :=
_
x R
n
:
_
f
i
(x
0
), x
_
= 0,
_
g
j
(x
0
), x
_
< |x|, i = 1, . . . , m, j I(x
0
)
_
,
where
=
1
2|x|
max
jI(x
0
)
_
g
j
(x
0
), x
_
< 0
and
(x) := x
0
+x +
_
f
(x
0
)
_
T
y(x).
Clearly, x belong to the relative interior of Q. Let us x so small that
the solution y(x) is well dened for all x with |x| . The denition of Q
implies that
f
i
((x)) = 0, i = 1, . . . , m,
for all x Q B
(0).
Let j be an arbitrary index from the set 1, 2, . . . , s I(x
0
). The con-
tinuity of g
j
and the inequality g
j
(x
0
) < 0 imply that decreasing > 0, if
necessary, we can ensure that g
j
((x)) < 0 for all x Q B
(0).
Let j I(x
0
). Then the mean-value theorem implies that
g
j
((x)) = g
j
_
x
0
+x +
_
f
(x
0
)
_
T
y(x)
_
=
= g
j
(x
0
) +
_
g
j
(x
0
), x
_
+o(|x|) < |x| +o(|x|).
Decreasing again > 0, if necessary, we can ensure that g
j
(x
0
) < 0 for all
x QB
(0). This shows that (x) S, and hence, K

S
(x
0
) is a Boltyanskii
tangent cone K
S
(x
0
0
.
3.3 The maximum principle with transversality condi-
tions for xed time interval problems
n
;
35
m
;
C4. t
0
and t
1
0
t
1
;
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X U [t
0
, t
1
] (x, u, t) L(x, u, t)
are given;
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x L(x, u, t) R
of (x, u, t);
C7. x
0
is a given point of X and S is a given subset of X;
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
[t
0
,t
1
]
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t)) dt
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) = x
1
;
c) I(x
(), u
()) I(x(), u()) for all x(), u() T (T

[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) S;
36
C11. C is a Boltyanskii tangent cone to S at the point x = x
(t
1
).
0
, t
1
, f, L, x
0
, S satisfy
[t
0
,t
1
]
(x
(), u
()) satises C10, x is dened by C11 and C satises C11. Dene

the Hamiltonian H to be the function
X U R
n
R R (x, u, p, p
0
, t) H(x, u, p, p
0
, t) R
given by
H(x, u, p, p
0
, t) := p
0
L(x, u, t) +
n
j=1
p
j
f
j
(x, u, t).
0
) such that
E1. [t
0
, t
1
E2.
0
0, 1;
E3. ((t),
0
) ,= (0, 0) for every t [t
0
, t
1
];

i
(t) = p
0
L(x, u, t)
x
i
(x
(t), u
(t), t)
n
j=1
p
j
f
j
(x, u, t)
x
i
(x
(t), u
(t), t)
0
, t
1
];
H(x
(t), u
(t), (t),
0
, t) = max H(x
(t), u, (t),
0
, t) : u U
0
, t
1
]
E6. the transversality condition holds, that is
(t
1
) C
.
37
tions for variable time interval problems
n
;
m
;
X U (x, u) f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X U (x, u) L(x, u)
are given;
X x f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X x L(x, u) R
C6. x
0
T (T(X, U, f) :=
_
_
T (T
[t
0
,t
1
]
(X, U, f) : t
0
, t
1
R, t
0
t
1
_
,
where for t
0
, t
1
R such that t
0
t
1
, T (T
[t
0
,t
1
]
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
38
[t
0
,t
1
]
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t)) dt
C9. there exist real number t
0
and t
1
and a reference trajectory-control pair
(x
(), u
()) such that

a) x
(), u
() T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) S;
c) I(x
(), u
()) I(x(), u()) for all (x(), u()) T (T

[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) S;
(t
1
).
0
, t
1
, f, L, x
0
, x
1
satisfy
[t
0
,t
1
]
t
0
, t
1
, (x
(), u
()) satisfy C9. Dene the Hamiltonian H to be the function

X U R
n
(x, u, p, p
0
) H(x, u, p, p
0
) R
given by
H(x, u, p, p
0
) := p
0
L(x, u) +
n
j=1
p
j
f
j
(x, u).
0
) such that
E1. [t
0
, t
1
E2.
0
0, 1;
E3. ((t),
0
) ,= (0, 0) for every t [t
0
, t
1
];

i
(t) = p
0
L
x
i
(x
(t), u
(t))
n
j=1
p
j
f
j
x
i
(x
(t), u
(t))
0
, t
1
];
39
E5. the Hamiltonian maximization condition holds with zero value, that
is
0 = H(x
(t), u
(t), (t),
0
) = max H(x
(t), u, (t),
0
) : u U
0
, t
1
].
(t
1
) C
.
3.5 An example
We consider the one-dimensional landing problem in which it is desired
to nd, for a given initial point (x
0
, y
0
) R
2
, a trajectory-control pair
(x(), y(), u()) of the system
x(t) = y(t), u [1, 1],
y(t) = u(t)
such that (x(), y()) goes from (x
0
, y
0
) to some point of the y-axis in minimum
time. This is exactly the same as the soft-landing problem discussed before,
except that now we are not asking to be soft, that is, our terminal condition
is that the terminal position x be zero, but we are not imposing any condition
on the terminal velocity y.
2
law is given by
f(x, y, u) = (y, u)
T
,
and the Lagrangian is identically equal to 1. The terminal constraint is
x(t
1
) S, where
S =
_
(x, y) R
2
: x = 0
_
.
The Hamiltonian H is given by
H(x, y, u, p
x
, p
y
, p
0
) = p
x
y +p
y
u p
0
,
x
and p
y
variable p.
40
Assume that (x
(), y
(), u
()) is a solution of our minimum time prob-

lem, and that (x
(), y
(), u
()) T (T
[t
0
,t
1
]
(X, U, f). Then Theorem 1.6
tells us that there exists a pair (,
0
) satisfying all the conditions of the
conclusion. Write (t) := (
x
(t),
y
(t)). The adjoint equation the implies
that

x
(t) = 0,

y
(t) =
x
(t).
Therefore the function
x
is constant. Let a R be such that
x
(t) = a for
all t [t
0
, t
1
]. Then there must exist a constant b such that
y
(t) = b at for all t [t
0
, t
1
].
Now, if a and b are both equal to zero, the function
y
would vanish identi-
cally and then the Hamiltonian maximization condition would say that the
function
[1, 1] u H(x, y, u, 0, 0, p
0
) = p
0
(t), a fact that would give us no information

whatsoever u
. Fortunately, the conditions of Theorem 3.10 imply that a

and b can not both vanish. To see this, observe that if a = b = 0 then it
follows that
x
(t)
y
0. But then the nontriviality condition tells us
that
0
,= 0. So the value H(x
(t), y
(t), u
(t), 0, 0, p
0
) = p
0
, which is not
equal to zero. This contradicts the fact, for our time-varying problem, the
Hamiltonian is supposed to vanish.
It is clear that the set S itself is a Boltyanskii tangent cone to S at
the terminal point x
(t
1
). Therefore the tranversality condition says that
(t
1
) S
, i.e.
y
(t
1
) = 0.
So, R t
y
(t) = b at is a linear function which is not identically zero
(because a and b do not both vanish) but vanishes at the endpoint t
1
of the
interval [t
0
, t
1
]. Therefore,
y
(t) never vanishes on (t
0
, t
1
). It follows that the
optimal control u
() is either constantly equal to 1 or constantly equal to -1.

Thus we have proved that all optimal controls are bang-bang and constant.
41
4 Separation of sets
If S
1
and S
2
are arbitrary subsets of the real linear space X, and R,
then
S
1
S
2
:= s
1
s
2
: s
1
S
1
, s
2
S
2
,
S
1
:= s
1
: , s
1
S
1
.
In particular, if B is closed unit ball of X centered at the origin (with respect
to some norm in X), then x +rB is the closed ball with radius r centered at
x.
A well known notion is the concept of separability of convex sets. We
present it for the case of cones. Let K
1
and K
2
be convex cones. We say
that K
1
and K
2
are (linearly) separable if there exist nonvanishing x
1
K
1
and x
2
K
2
such that x
1
+x
2
= 0.
Closely related to this property is the property of transversality. This
is a well known property for linear subspaces. One says that two linear
subspaces A
1
and A
2
of a nite-dimensional space X are transversal if the
sum A
1
+A
2
= X.
The general philosophy of transversality theory is that, if two sets S
1
and
S
2
have linear approximations A
1
and A
2
at a point x, then S
1
S
2
looks
locally near x, like A
1
A
2
. Here the transversality of A
1
and A
2
is essential.
For example, if we take X = R
2
, S
1
to be the x-axis, and S
2
to be the set
(x, y)
T
: y = x
2
, we see that S
1
and S
2
intersect at the origin, and their
tangent spaces A
1
and A
2
at the point (0, 0)
T
both coincide with the x-axis.
So A
1
A
2
is a one-dimensional linear subspace, but S
1
S
2
consists of a
single point. Thus, S
1
S
2
does not look at all like A
1
A
2
. This shows that
the principle that S
1
S
2
looks near x like A
1
A
2
can fail if A
1
and A
2
fail to be transversal.
Clearly, for linear subspaces we could equally well use the dierence A
1
A
2
. Using this dierence of sets, the notion of tranversality can be dened
also for cones: we say that K
1
and K
2
are transversal if K
1
K
2
= X.
Thus, the notion of transversality of cones is a natural extension of the well
known notion of transversality of linear subspaces. The relation between
separability and transversality of cones is established in the following:
Lemma 4.1 The convex cones K
1
and K
2
are separable i the cones K
1
and
K
2
are not transversal.
42
Proof. Necessity. Let the convex cones K
1
and K
2
are separable. Then,
there will exist nonvanishing x
1
K
1
and x
2
K
2
such that x
1
+ x
2
= 0.
If we assume that the cones K
1
and K
2
are transversal, then for each point
x X will exist two points, x
1
K
1
and x
2
K
2
, such that x = x
1
x
2
.
Since x
1
K
1
and x
2
K
2
,
x
1
, x) = x
1
, x
1
x
2
) = x
1
, x
1
) +x
1
, x
2
) = x
1
, x
1
) +x
2
, x
2
) 0.
Since x was an arbitrary point of X, we can take the point x and to obtain
that x
1
, x) 0. Thus we have proved that x
1
, x) = 0 for each x X.
But, this is impossible since x
1
is nonvanishing. The obtained contradiction
shows that our assumption (that the cones K
1
and K
2
are transversal) is
wrong. Thus we proved that the cones K
1
and K
2
are not transversal.
Suciency. Now, let the cones K
1
and K
2
are not transversal, i.e.
K
1
K
2
,= X. Then 0 belongs to the boundary of the set K
1
K
2
. (If the
origin belongs to the interior of the set K
1
K
2
, then the fact that K
1
K
2
is a cone implies that K
1
K
2
= X.) Hence, there exists a nonvanishing
element x
such that x
, x) 0 for each x K
1
K
2
. So, x
, x
1
x
2
) 0
for each x
1
K
1
and for each x
2
K
2
. We set x
1
= x
and x
2
= x
.
Let us assume that there exists x
1
K
1
such that x
1
, x
1
) > 0. Since K
1
is a cone, nx
1
K
1
for each positive integer n. Hence for each x
2
K
2
,
0 < nx
1
, x
1
) = nx
, x
1
) x
, x
2
) ,
which is impossible. So, for each x
1
K
1
we have that x
1
, x
1
) 0, i.e .
x
1
K
1
.
Let us assume that there exists x
2
K
2
such that x
2
, x
2
) > 0. Since K
2
is a cone, nx
2
K
2
for each positive integer n. Hence for each x
1
K
1
,
0 < nx
2
, x
2
) = nx
, x
2
) x
, x
1
) ,
which is impossible. So, for each x
2
K
2
we have that x
2
, x
2
) 0, i.e .
x
2
K
2
. Clearly, x
1
and x
2
are nonvanishing and x
1
+x
2
= 0. So, the cones
K
1
and K
2
are separable. This completes the proof.
We now proceed to geometrize the problem by completely eliminating
the optimization aspect, and working instead with a question about separa-
tion of two sets. Precisely, let us say that two sets S
1
and S
2
are separated
at a point x S
1
S
2
if S
1
S
2
= x, i.e. S
1
S
2
contains no points
43
other than x. We say that two sets S
1
and S
2
are locally separated at a point
x S
1
S
2
if there exists a neighborhood U of x such that U S
1
S
2
= x.
Unfortunately, separation of two sets does not imply linear separation of
their approximating cones. This can be see easily by considering the following
example: Let X = R
2
, and take S
1
and S
2
to be the x-axis and the y-axis,
respectively. Then S
1
and S
2
are separated at the origin. On the other hand,
S
1
and S
2
are their own linear approximation at 0. Clearly, S
1
and S
2
are
not linearly separable.
This is the reason to introduce the notion of strong transversality which
is slightly stronger then the property of transversality: it is said that two
convex cones K
1
and K
2
are strongly transversal if they are transversal and
K
1
K
2
,= 0. The relation between transversality and strong transversality
is established in the following
Lemma 4.2 Let K
1
and K
2
be two convex cones. Then the following two
conditions are equivalent:
i) K
1
and K
2
are transversal;
ii) K
1
and K
2
are strongly transversal or K
1
and K
2
are both linear sub-
spaces such that K
1
K
2
= X.
Proof. Clearly, ii) implies i). Assume now that K
1
and K
2
are transversal,
but not strongly transversal. Let x K
2
. Since K
1
K
2
= X, there exist
x
1
K
1
and x
2
K
2
such that x = x
1
x
2
, i.e. x
1
= x+x
2
. But x+x
2
K
2
and x
1
K
1
. Thus, x + x
2
K
1
K
2
and x
1
K
1
K
2
. Since K
1
and K
2
are not strongly transversal, x + x
2
= 0 and x
1
= 0. Since, K
2
is a convex
cone, we obtain that K
2
is a linear subspace.
Let x K
1
. Since K
1
K
2
= X, there exist x
1
K
1
and x
2
K
2
such
that x = x
1
x
2
, i.e. x
2
= x + x
1
. But x + x
1
K
1
and x
2
K
2
. Thus,
x + x
1
K
1
K
2
and x
2
K
1
K
2
. Since K
1
and K
2
are not strongly
transversal, x + x
1
= 0 and x
2
= 0. Since, K
1
is a convex cone, we obtain
that K
1
is a linear subspace.
Hence, X = K
1
+ K
2
. Our assumption implies that K
1
K
2
= 0. So,
the sum is direct.
It is remarkable, that strong transversality of two approximating cones
implies non-separation of the sets (this assertion we shall prove later). Hence,
separation of sets implies that the corresponding approximation cones are not
44
strongly transversal. And if one of these approximationg cones is not a linear
subspace, the Lemma 4.2 implies the stronger conclusion that these cones
are not transversal. But then Lemma 4.1 implies that these approximating
cones are linearly separable. In fact, this is the main idea of the proof of the
Pontryagin maximum principle.
4.1 The maximum principle as a set separation condi-
tion for xed time interval problems
n
;
m
;
C4. t
0
and t
1
0
t
1
;
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X U [t
0
, t
1
] (x, u, t) L(x, u, t)
are given;
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x L(x, u, t) R
of (x, u, t);
C7. x
0
45
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
C9. the reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
) from x
0
for the data X, U, f over the
interval [t
0
, t
1
] is dened by the set of all points y X for each of them
there exists (x
y
(), u
y
()) T (T
[t
0
,t
1
]
(X, U, f) such that x
y
(t
0
) = x
0
and x
y
(t
1
) = y.
C10. the reference TCP (x
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) S;
c) the reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
) from x
0
for the data
X, U, f over the interval [t
0
, t
1
] and S are locally separated at the point
x = x
(t
1
) ;
C11. C is a Boltyanskii tangent cone to S at the point x.
0
, t
1
, f, L, x
0
, S satisfy
[t
0
,t
1
]
(X, U, f) and 1
[t
0
,t
1
]
(X, U, f, x
0
) are dened
by C8-C9, (x
(), u
()) satises C10, x is dened by C10 and C satises

C11.
Assume, moreover, that C is not a linear subspace of R
n
.
Dene the Hamiltonian H to be the function
X U R
n
R (x, u, p, t) H(x, u, p, t) R
given by
H(x, u, p, t) :=
n
j=1
p
j
f
j
(x, u, t).
Then there exists a such that
E1. [t
0
, t
1
E2. (t) ,= 0 for every t [t
0
, t
1
];
46

i
(t) =
n
j=1
p
j
f
j
(x, u, t)
x
i
(x
(t), u
(t), t)
0
, t
1
];
H(x
(t), u
(t), (t), t) = max H(x
(t), u, (t), t) : u U
0
, t
1
]
(t
1
) C
.
tions for variable time interval problems
n
;
m
;
X U (x, u) f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X U (x, u) L(x, u)
are given;
X x f(x, u) = (f
1
(x, u), . . . , f
n
(x, u)) R
n
and
X x L(x, u) R
47
C6. x
0
T (T(X, U, f) :=
_
_
T (T
[t
0
,t
1
]
(X, U, f) : t
0
, t
1
R, t
0
t
1
_
,
where for t
0
, t
1
R such that t
0
t
1
, T (T
[t
0
,t
1
]
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
C8. the reachable set 1(X, U, f, x
0
) from x
0
for the data X, U, f is dened
by
1(X, U, f, x
0
) :=
_
<t
0
<t
1
<+
1
[t
0
,t
1
]
(X, U, f, x
0
)
where 1
[t
0
,t
1
]
(X, U, f, x
0
) is the set of all points y X for each of them
there exists (x
y
(), u
y
()) T (T
[t
0
,t
1
]
y
(t
0
) = x
0
and x
y
(t
1
) = y.
(), u
()), and t
0
and t
1
are such that
a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) S;
c) 1(X, U, f, x
0
) and S are locally separated at the point x =
x
(t
1
) ;
0
, t
1
, f, L, x
0
, x
1
satisfy
[t
0
,t
1
]
(X, U, f) and 1(X, U, f, x
0
) are dened by
C7-C8 and t
0
, t
1
, (x
(), u
()), x and C satisfy C9 and C10.

n
.
X U R
n
(x, u, p) H(x, u, p) R
given by
H(x, u, p) :=
n
j=1
p
j
f
j
(x, u).
48
E1. [t
0
, t
1
0
, t
1
];

i
(t) =
n
j=1
p
j
f
j
x
i
(x
(t), u
(t))
0
, t
1
];
E4. the Hamiltonian maximization condition holds with zero value, that
is
0 = H(x
(t), u
(t), (t)) = max H(x
(t), u, (t)) : u U
0
, t
1
].
(t
1
) C
.
4.3 An example
We consider the one-dimensional landing problem in which it is desired to
nd a trajectory-control pair (x(), y(), u()) of the system
x(t) = y(t), u [1, 1],
y(t) = u(t)
such that (x(), y()) goes from (0, 0) to some point of the set
S :=
_
(x, y)
T
: x y 1
_
in minimum time.
2
law is given by
f(x, y, u) = (y, u),
and the Lagrangian is identically equal to 1. The terminal constraint is
x(t
1
) S.
49
Assume that (x
(), y
(), u
()) is a solution of our minimum time prob-

lem, and that (x
(), y
(), u
()) T (T
[t
0
,t
1
]
(X, U, f). Then Theorem 4.4
tells us that there exists a pair (,
0
) satisfying all the conditions of the con-
clusion. Write (t) := (
x
(t),
y
(t)). Since our control system is autonomous,
we may think without loss of generality that t
0
= 0. The Hamiltonian H is
given by
H(x, y, u, p
x
, p
y
, p
0
) = p
x
y +p
y
u p
0
,
x
and p
y
variable p.
It is clear that the set S itself is a Boltyanskii tangent cone to S at
the terminal point x
(t
1
). Therefore the tranversality condition says that
(t
1
) S
, i.e.
x
(t
1
) = 1,
y
(t
1
) = 1. (16)
The adjoint equation:

x
(t) = 0,
x
(t
1
) = 1

y
(t) =
x
(t),
y
(t
1
) = 1.
and transversality condition (16) imply that
x
(t) 1 and
y
(t) = t
1
1 t
for each t [0, t
1
].
Then the Hamiltonian maximization condition would say that the func-
tion
[1, 1] u (t
1
1 t)u
is maximized by taking
u
(t) =
_
1, t [0, t
1
1);
1, t [t
1
1, t
1
].
It can be checked that using the control u
(t) 1 we can not reach (by

means of trajectories of the considered system starting from the origin) any
points from the set S. Thus, the expression for u
() shows that t
1
must be
greater than 1 in order to reach at last one point from S.
The corresponding trajectory (x
(), y
()
T
can be explicitly calculated:
x
(t) =
_
_
t
2
2
, t [0, t
1
1);
(t
1
1)
2
(2t
1
2 t)
2
2
, t [t
1
1, t
1
].
,
50
y
(t) =
_
t, t [0, t
1
1);
2t
1
2 t, t [t
1
1, t
1
].
Clearly, (x
(t
1
), y
(t
1
)
T
belongs to the boundary of the set S. So, we substi-
tute in the equation x y = 1 the variables x and y by the expressions for
x
(t
1
) and y
(t
1
), respectively, and obtain that
1
2
(t
1
)
2
t
1
= 0,
and hence t
1
= 2.
51
5 The main ideas standing behind the maxi-
mum principle
5.1 The maximum principle as a set separation condi-
tion for xed time interval problems
Let us remind the assumptions and the formulation of the maximum principle
as a set separation condition for xed time interval problems:
n
;
m
;
C4. t
0
and t
1
0
t
1
;
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X U [t
0
, t
1
] (x, u, t) L(x, u, t)
are given;
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x L(x, u, t) R
of (x, u, t);
C7. x
0
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
52
[t
0
,t
1
]
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t)) dt
C9. the reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
) from x
0
for the data X, U, f over the
interval [t
0
, t
1
] is dened by the set of all points y X for each of them
there exists (x
y
(), u
y
()) T (T
[t
0
,t
1
]
y
(t
0
) = x
0
and x
y
(t
1
) = y.
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) S;
c) the reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
) from x
0
for the data
X, U, f over the interval [t
0
, t
1
] and S are locally separated at the point
x = x
(t
1
) ;
0
, t
1
, f, L, x
0
, S satisfy
[t
0
,t
1
]
(X, U, f) and 1
[t
0
,t
1
]
(X, U, f, x
0
) are dened
by C8-C9, (x
(), u
()) satises C10, x is dened by C10 and C satises

C11.
n
.
X U R
n
R (x, u, p, t) H(x, u, p, t) R
given by
H(x, u, p, t) :=
n
j=1
p
j
f
j
(x, u, t).
E1. [t
0
, t
1
0
, t
1
];
53

i
(t) =
n
j=1
p
j
f
j
(x, u, t)
x
i
(x
(t), u
(t), t)
0
, t
1
];
H(x
(t), u
(t), (t), t) = max H(x
(t), u, (t), t) : u U
0
, t
1
]
(t
1
) C
.
The basic idea of the proof is to use needle variations, to combine these
variations into packets of needle variations, to propagate the eect of
these variations to the terminal point of the reference trajectory, and con-
struct an approximating cone to the reachable set. Then a topological
argument will be used to derive, from the separation assumption for the
reachable set and the set, the existence of a hyperplane separating the corre-
sponding tangent cones. By propagating backwards via the adjoint equation
the linear functional that denes this hyperplane, we will get the desired
vector .
In order to avoid technical complications, we will rst do the proof under
the following additional simplifying assumption:
(SA) the reference control u
is continuous.
Assumption (SA) is not needed, but the proof without it is a litle bit
more technical, so we will rst assume (SA), and the explain how to get rid
of this condition.
54
5.2 The variational equation.
Dene
A(t) :=
f
x
(x
(t), u
(t), t) , t [t
0
, t
1
] .
Then, under our hypotheses, including the additional condition (SA), the
map [t
0
, t
1
] t A(t) R
nn
is continuous.
The linear time-varying ordinary dierential equation
v(t) = A(t) v(t) (17)
is called variational equation along the reference trajectory-control pair (x
, u
).
Let us consider the following matrix dierential equation
X(t) = A(t) X(t), X(s) = E, (18)

where E is the identity matrix, X(t) K
nn
for each t R and by we
have denoted the usual matrix multiplication. Since this is a linear equation
over K
n
2
, it has a global solution on [t
0
, t
1
]. We denote by (t, s) the so-
lution of (18) at time t. Then (t, t
0
) is dened for each real s and t from
the interval [t
0
, t
1
], and is called fundamental solution of (18) associated to A.
5.3 Packets of needle variations.
Let us x
NV1. a positive integer k;
NV2. a k-tuple = (
1
, . . . ,
k
) (the variation times) such that
t
0

1
<
2
< <
k
< t
1
;
NV2. a k-tuple u = (u
1
, . . . , u
k
) (the variation controls) of elements of the
control set U.
We denote by BM ([t
0
, t
1
] , U) the set of all bounded measurable maps
u : [t
0
, t
1
] U. For a positive real number , dene
R
k
+
() :=
_
(x
1
, x
2
, . . . , x
k
) R
k
: 0 x
j

_
.
55
Let
:= min (
2
1
,
3
2
, . . . ,
k

k1
, t
1
k
) .
We then dene the packet of needle variations of the reference control u
associated to and to u to be the map

R
k
+
() = (
1
, . . . ,
k
) u
,u,
BM ([t
0
, t
1
] , U)
that associates to each = (
1
, . . . ,
k
) a control u
,u,
dened as follows
u
,u,
(t) :=
_
_
u
(t), if t [t
0
, t
1
]
k
_
j=1
[
j
,
j
+
j
] ,
u
j
, if t [
j
,
j
+
j
] .
5.4 The endpoint map.
Given k, , u as above, we let x
,u,
be, for R
k
+
(), the unique solution
t x(t) of the equation
x(t) = f
_
x(t), u
,u,
(t), t
_
such that x(t
0
) = x
0
.
Uniqueness follows because the map
X [t
0
, t
1
] (x, t) f
_
x(t), u
,u,
(t), t
_
is Lipschitz continuous with respect to x with a bounded Lipschitz constant
as long as (x, t) belongs to a compact subset of X [t
0
, t
1
].
In principle, x
,u,
need not exist on the whole interval [t
0
, t
1
]. The end
point map
c
,u
()
associated to , u, is the map that assigns to each R
k
+
() the terminal
point
c
,u
() := x
,u,
(t
1
) (19)
of the trajectory x
,u,
(t
1
). Precisely, the domain T
,u
of c
,u
is the set of
those R
k
+
() for which x
,u,
exists on the whole interval [t
0
, t
1
], and for
T
,u
, c
,u
() is dened by (19).
56
5.5 The main technical lemma.
Lemma 5.2 Given k, , u as above, write
x
j
:= x
(
j
), u
j
:= u
(
j
).
Then there exists an such that:
I. 0 < ;
II. R
k
+
( ) T
,u
;
III. The endpoint map c
,u
is continuous on R
k
+
( );
IV. The endpoint map c
,u
is is dierentiable at 0 R
k
, and the dierential
Dc
,u
(0) of c
,u
at 0 is the linear map
R
k
= (
1
, . . . ,
k
) ( ) R
n
given by
( ) =
k
j=1
j
(t
1
,
j
)
_
f
_
x
j
, u
j
,
j
_
f
_
x
j
, u
j
,
j
__
,
i.e.
c
,u
() = x
(t
1
)+
k
j=1
j
(t
1
,
j
)
_
f
_
x
j
, u
j
,
j
_
f
_
x
j
, u
j
,
j
__
+o (||)
as 0 via values in R
k
+
.
5.6 The transversal intersection theorem.
Further, the proof uses the so called transversal intersection theorem:
Theorem 5.3 Assume that the following conditions hold
I. X
1
, X
2
and Y are nite dimensional real linear spaces;
II. C
1
and C
2
are convex cones in X
1
and X
2
, respectively;
57
III. U
1
and U
2
are neighborhoods of the origin in X
1
and X
2
, respectively;
IV. F
1
: U
1
Y and F
2
: U
2
Y are continuous maps such that F
1
(0) =
F
2
(0) = 0;
V. L
i
: X
i
Y , i = 1, 2, are linear maps such that
F
i
(x
i
) = L
i
x
i
+o (|x
i
|)
as x
i
tends to the origin via values in C
i
, i.e. each F
i
is dierentiable
at the origin along C
i
with dierentials L
i
;
VI. The sets L
1
(C
1
) and L
2
(C
2
) are strongly transversal.
Then the sets F
1
(U
1
C
1
) and F
2
(U
2
C
2
) are not locally separated at the
origin, i.e. F
1
(U
1
C
1
) F
2
(U
2
C
2
) V ,= 0 for each neighborhood V of
the origin.
Remark 5.4 The statement of Theorem 5.3 involves in item V norms on
the spaces X
1
, X
2
and Y , but the validity of the condition of this item does
not depend on the choice of these norms.
5.7 Application of the transversal intersection theo-
rem.
Fix k, , u as above. Using the fact that C is a Boltyanskii tangent cone
to S at the point x, we pick a neighborhood V of the origin in R
n
and a
continuous map : V C S such that (v) = x + v + o(|v|) as v tends
to the origin with values from C.
Since the image of R
k
+
under the endpoint map c
,u
is contained in the
reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
), the image of C under is contained in S
and the sets 1
[t
0
,t
1
]
(X, U, f, x
0
) and S are locally separated at x, it follows
that the sets c
,u
(R
k
+
) and (C) are locally separated at the point x. Then
the transversal intersection theorem implies that the cones Dc
,u
(0)(R
k
+
) and
C are not strongly transversal. Since the cone C is not a linear subspaces of
R
n
, we can conclude that the cones Dc
,u
(0)(R
k
+
) and C are not transversal.
Therefore there exists a nonzero vector R
n
such that
, v) 0 whenever v Dc
,u
(0)(R
k
+
)
58
and
, v) 0 whenever v C. (20)
The formula for Dc
,u
(0)(R
k
+
) given by the main technical lemma implies
that
(t
1
,
j
)
_
f
_
x
j
, u
j
,
j
_
f
_
x
j
, u
j
,
j
__
Dc
,u
(0)(R
k
+
) for j = 1, . . . , k.
Therefore,
_
, (t
1
,
j
)
_
f
_
x
j
, u
j
,
j
_
f
_
x
j
, u
j
,
j
___
0 for j = 1, . . . , k,
and then
_
, (t
1
,
j
) f
_
x
j
, u
j
,
j
__

_
, (t
1
,
j
) f
_
x
j
, u
j
,
j
__
(21)
for j = 1, . . . , k. Clearly, we can also assume that
|| = 1. (22)
5.8 The compactness argument.
We have already proved that for every choice of k, and u, there exists a
vector for which (20), (21) and (22) hold true.
Given any subset W of [t
0
, t
1
] U, let (W) be the set of all R
n
satisfying (20), (22) and
, (t
1
, ) f (x
(), u, )) , (t
1
, ) f (x
(), u
(), )) (23)
for all (, u) from W.
We have proved that (W) is nonempty whenever W is a nite set
(
1
, u
1
), (
2
, u
2
), . . . , (
k
, u
k
)
whenever t
0

1
<
2
<
k
< t
1
.
Now suppose that W is any nite subset of [t
0
, t
1
] U. Let
W = (
1
, u
1
), (
2
, u
2
), . . . , (
k
, u
k
)
with t
0

1

2

k
< t
1
. Let W
be the set
_
(
1
, u
1
) ,
_
2
+
1
, u
2
_
, . . . ,
_
k
+
k 1
, u
k
__
59
Then (W
) ,= . Pick
(W
). Then
belongs to the unit sphere of

R
n
, which is compact. So, there is a subsequence
s=1
that converges to
a limit . Then (W), and so (W) ,= .
A similar limiting arguments show that (W) ,= and (W) is compact
if W is any nite subset of [t
0
, t
1
] U. Moreover,
(W
1
) (W
2
) (W
s
) ,=
if W
1
, W
2
, . . . , W
s
are nite subsets of [t
0
, t
1
] U.
Let J be the set of all nite subsets of [t
0
, t
1
] U. Then
(W)
WW
is a family of compact subsets of the unit sphere of R
n
having the property
that every nite intersection of members of the family is nonempty. It follows
that
(W) : W J ,= .
Therefore
([t
0
, t
1
] U) ,= .
This means that there exists a R
n
that satises (20), (22) and
, (t
1
, ) f (x
(), u, )) , (t
1
, ) f (x
(), u
(), )) (24)
for all (, u) from [t
0
, t
1
] U.
5.9 The adjoint equation
Let R
n
be such that (20), (22) and (24) hold. We set
(t) := , (t
1
, t)) =
T
(t
1
, t) .
Then satises the adjoint equation. Indeed, we know that
t
(t
1
, t) = (t
1
, t) A(t).
Then
(t) =
d
dt
_
T
(t
1
, t)
_
=
T
t
(t
1
, t) =
T
(t
1
, t)A(t) = (t)A(t).
60
Condition (20) says that
(t
1
)
T
C
.
Condition (22) implies that is nonzero. Finally, condition (24) says that
, (t
1
, ) f (x
(), u, )) , (t
1
, ) f (x
(), u
(), )) (25)
for all (, u) from [t
0
, t
1
] U. Therefore,
H(x
(t), u, (t), t) H(x
(t), u
(t), (t), t)
for all t [t
0
, t
1
].
If the optimal control u
is not continuous, but only bounded and mea-

surable, then the set (x
(t), u
(t), t) : t
0
t t
1
is contained in a compact
set of XU [t
0
, t
1
]. The proof of the main technical depends very strongly
on the continuity of u
. To relax the continuity assumption on u
we need
some facts from the theory of functions of a real variable.
Recall that, if [a, b] t (t) is an integrable function, a Lebesgue point
of is a (a, b) such that
lim
h0
1
2h
_
+h
h
|(t) ()| dt = 0.
It is then a theorem saying that almost every point of [a, b] is a Lebesgue
point of . To avoid the assumption that u
is continuous, the times

j
where the needle variations are made are Lebesgue points of the function
f (x
(t), u
(t), t). This choice is enough to prove the main technical lemma.
Example 5.5 Let us consider the following one-dimensional example:
_
2
0
u
2
(t) + 4x(t) dt min
x(t) = u(t), u 0, x(0) = 0, x(2) = 1.
We solve this problem by applying the maximum principle. In this case,
the conguration space X is R, the control set U is the ray [0, +), the
dynamical law is given by f(x, u) = u and L(x, u) = u
2
4x.
Because we search a minimal value of the integral, the Hamiltonian H is
given by
H(x, u, p, p
0
) = p
0
L(x, u) + pf(x, u) = p
0
(u
2
+ 4x) +pu.
61
Assume that x
(), u
() is a solution of our problem, and that (x
(), u
())
T (T
[0,2]
(X, U, f). Then the maximum principle tells us that there exists a
pair (,
0
) satisfying all the conditions of the conclusion. The adjoint equa-
tion implies that
(t) = 4
0
,
Therefore the function (t) = a + 4
0
t, where a is a constant.
Let
0
= 0. Then 0 ,= (t) = a for all t [0, 2] and the Hamiltonian
maximization condition would say that the function
[0, +) u au
(t). But this is possible only for a < 0 which

implies that u
(t) = 0 for all t from [0, 2]. Since the corresponding trajectory
x
is x
(t) = 0 for all t from [0, 2], we have that x
(2) = 0 ,= 1. Hence,
the trajectory x
is not admissible. The obtained contradiction shows that

equality
0
= 0 is impossible.
So, we can assume that
0
= 1, and hence (t) = a+4t for some constant
a. Now, the Hamiltonian maximization condition would say that the function
[0, +) u u
2
+ (a + 4t)u
(t). Hence,
u
(t) =
_
_
_
0, if a + 4t 0;
1
2
(a + 4t), if a + 4t 0.
The corresponding trajectory x
is
x
(t) =
_
_
0, if t
a
4
;
1
2
a
a
4
+
2
a
4
=
1
2
a
_
t +
a
4
_
+
_
t
2
a
2
16
_
, if t
a
4
.
Since x
(2) = 1, we obtain the following equation with respect to the

unknown constant a:
1 =
1
2
a
_
2 +
a
4
_
+ 4
a
2
16
= a +
a
2
8
+ 4
a
2
16
= 4 + a +
a
2
16
which has two roots a
1
= 4 and a
2
= 12. Since a
2
+ 4t < 0 for t [0, 2],
we obtain that only the value a
1
is admissible. Substituting a by a
1
in the
expressions for u
and x
, respectively, we obtain them explicitly.

62
6 Boltyanskii cones and the proof of the max-
imum principle in some cases
Here we shall consider two important examples of nonlinear sets and shall
calculate their approximating Boltyanskii cones and their corresponding dual
cones, respectively.
6.1 Set dened by a nite set of equalities
Let
S = x R
n
: f
i
(x) = 0, i = 1, 2, . . . , m ,
where f
i
, i = 1, 2, . . . , m are continuously dierentiable functions. Let x
0
S
and their gradients
_
f
i
x
(x
0
)
_
T
, i = 1, 2, . . . , m, are linearly independent
vectors. Then the Boltyanskii tangent cone K
S
(x
0
) to the set S at the point
x
0
is
K
S
(x
0
) =
_
_
_
v R
n
:
__
f
i
x
(x
0
)
_
T
, v
_
= 0, i = 1, 2, . . . , m
_
_
_
and its dual cone is
K
S
=
_
_
_
v
R
n
: x =
m
i=1
i
_
f
i
x
(x
0
)
_
T
,
i
R, i = 1, . . . , m
_
_
_
.
Proof. By
f
x
(x
0
) we denote the mn-matrix whose rows are the vector-
rows
f
i
x
(x
0
), i = 1, 2, . . . , m. So, the relation x K
S
(x
0
) can be written
as
f
x
(x
0
) x = 0. By
_
f
x
(x
0
)
_
T
we denote its transpose matrix. We set
M =
f
x
(x
0
)
_
f
x
(x
0
)
_
T
. If we assume that the mm-matrix M is singular,
then there exists a non vanishing vector y such that My = 0. But,
0 = y, My) =
__
f
x
(x
0
)
_
T
y,
_
f
x
(x
0
)
_
T
y
_
=
_
_
_
_
_
_
_
f
x
(x
0
)
_
T
y
_
_
_
_
_
_
2
,
63
i. e.
m
i=1
_
f
i
x
(x
0
)
_
T
y
i
= 0.
The obtained contradiction shows that the matrix M is non singular. Let us
consider the following system of equations:
g
i
(x, y) := f
i
_
_
x
0
+x +
_
f
x
(x
0
)
_
T
y
_
_
_
x,
_
f
i
x
(x
0
)
_
T
_
= 0, i = 1, 2, . . . , m.
One can directly checked that
g
i
(0, 0) = 0, i = 1, . . . , m,
g
i
x
j
(0, 0) = 0, i = 1, . . . , m, j = 1, . . . , n,
and the matrix
_
g
i
y
j
(0, 0)
_
mm
singular. According to the Implicit function theorem, there exists a neigh-
borhood U of the origin and a continuously dierentiable function y : R
m
such that
g
i
(0, y(0)) = 0, and g
i
(x, (y(x))) = 0, i = 1, . . . , m,
for each point from U.
Dierentiating the last identity with respect to x
j
, we obtain that
0 =

x
j
g
i
(0, 0) =
g
i
x
j
(0, 0) +
m
k=1
g
i
y
k
(0, 0)
y
k
x
j
(0).
The last equalities imply that for each index k = 1, 2, . . . , m,
0 = M
_
y
k
x
_
T
(0).
Since the matrix M is nonsingular, we obtain that
_
y
k
x
_
T
(0) = 0 for
each k = 1, 2, . . . , m,. This and the equality y(0) = 0 imply that
y(x)
|x|
0,
whenever x 0. We set
(v) := x
0
+v +
m
j=1
_
f
j
x
(x
0
)
_
T
y
j
(v).
64
Let us x so small that the solution y(v) is well dened for all v with
|v| . The denition of K
S
(x
0
) implies that
f
i
((v)) = 0, i = 1, . . . , m,
for all v K
S
(x
0
) B
(0) i. e. (v) S. Thus, we have shown that K

S
(x
0
)
is a Boltyanskii tangent cone K
S
(x
0
0
.
Applying Lemmas 3.1 and 3.4, we conclude the proof.
6.2 Set dened by a nite set of equalities and inequal-
ities
Let
S = x R
n
: f
i
(x) = 0, i = 1, 2, . . . , m, g
j
(x) 0, j = 1, . . . s ,
where f
i
and g
j
, i = 1, 2, . . . , m, j = 1, . . . , s, are continuously dierentiable
functions. Let x
0
S. We set
I(x
0
) = j : g
j
(x
0
) = 0.
i
(x
0
), i = 1, 2, . . . , m, are linearly independent
vectors, we have that the Boltyanskii tangent cone K
S
(x
0
) to the set S at
the point x
0
is K
S
(x
0
) = K
S,f
K
S,g
, where
K
S,f
=
_
_
_
v R
n
:
__
f
i
x
(x
0
)
_
T
, v
_
= 0, i = 1, 2, . . . , m
_
_
_
and
K
S,f
=
_
_
_
v R
n
:
__
g
j
x
(x
0
)
_
T
, v
_
< 0, j I(x
0
)
_
_
_
and its dual cone is K
S
= K
S,f
+K
S,g
, where
K
S,f
=
_
_
_
v
R
n
: v
=
m
i=1
i
_
f
i
x
(x
0
)
_
T
,
i
R, i = 1, . . . , m
_
_
_
65
and
K
S,g
=
_
_
_
v
R
n
: v
jI(x
0
)
j
_
g
j
x
(x
0
)
_
T
,
j
0, j I(x
0
)
_
_
_
.
Proof. One can directly checked that
g
i
(0, 0) = 0, i = 1, . . . , m,
g
i
x
j
(0, 0) = 0, i = 1, . . . , m, j = 1, . . . , n,
and the matrix
_
g
i
y
j
(0, 0)
_
nn
singular. As in Example 1, we obtain that for all suciently small |x| there
exists a smooth function y(x) with
y(x)
|x|
0, whenever x 0.
Let S be the unit sphere in R
n
. We set
= max
jI(x
0
), vS
__
g
j
x
(x
0
)
_
T
, v
_
< 0
and
(v) := x
0
+v +
n
i=1
_
f
i
x
(x
0
)
_
T
y
i
(v).
Let us x so small that the solution y(v) is well dened for all v with
|v| . Then
f
i
((v)) = 0, i = 1, . . . , m,
for all v B
(0).
Let j be an arbitrary index from the set 1, 2, . . . , s I(x
0
). The con-
tinuity of g
j
and the inequality g
j
(x
0
) < 0 imply that decreasing > 0, if
necessary, we can ensure that g
j
((v)) < 0 for all x B
(0).
Let j I(x
0
). Then the mean-value theorem implies that
g
j
((v)) = g
j
_
_
x
0
+v +
n
i=1
_
f
i
x
(x
0
)
_
T
y
i
(v)
_
_
=
= g
j
(x
0
) +
__
g
j
x
(x
0
)
_
T
, v
_
+o(|v|) |v| +o(|v|).
66
Decreasing again > 0, if necessary, we can ensure that g
j
(x
0
) < 0 for all
v B
(0).
This shows that (v) S, and hence, K
S
(x
0
) is a Boltyanskii tangent
cone K
S
(x
0
0
.
Applying Lemmas 3.1 and 3.4, we conclude the proof.
tions for xed time intervals problems
n
;
m
;
C4. t
0
and t
1
0
t
1
;
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X U [t
0
, t
1
] (x, u, t) L(x, u, t)
are given;
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x L(x, u, t) R
of (x, u, t);
C7. x
0
67
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
a) [t
0
, t
1
b) [t
0
, t
1
0
, t
1
];
[t
0
,t
1
]
is dened by
I(x(), u()) :=
_
t
1
t
0
L(x(t), u(t), t) dt
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) = x
1
;
c) I(x
(), u
()) I(x(), u()) for all x(), u() T (T

[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) S;
(t
1
).
0
, t
1
, f, L, x
0
, S satisfy
[t
0
,t
1
]
(x
(), u

X U R
n
R R (x, u, p, p
0
, t) H(x, u, p, p
0
, t) R
given by
H(x, u, p, p
0
, t) := p
0
L(x, u, t) +
n
j=1
p
j
f
j
(x, u, t).
0
) such that
E1. [t
0
, t
1
E2.
0
0, 1;
E3. ((t),
0
) ,= (0, 0) for every t [t
0
, t
1
];
68

i
(t) = p
0
L(x, u, t)
x
i
(x
(t), u
(t), t)
n
j=1
p
j
f
j
(x, u, t)
x
i
(x
(t), u
(t), t)
0
, t
1
];
H(x
(t), u
(t), (t),
0
, t) = max H(x
(t), u, (t),
0
, t) : u U
0
, t
1
]
(t
1
) C
.
Proof. We consider the following set of trajectory-control pairs T (T
[t
0
,t
1
]
(X
R, U, (f, L)) which is dened on [t
0
, t
1
] and is the set of all pairs ((x(), y()), u())
such that
a) [t
0
, t
1
b) [t
0
, t
1
] t (x(t), y(t)) X R is an absolutely continuous map;
c) x(t) = f(x(t), u(t), t) and y(t) = L(x(t), u(t), t) for almost every
t [t
0
, t
1
];
Next, we dene the set
S :=
_
(x, y) S R : y y +|x x|
2
_
,
where
y := y
(t
1
), y
(t) :=
_
t
t
0
L(x
(t), u
(t), t) dt.
Let us consider the reachable set 1
[t
0
,t
1
]
(X R, U, (f L), (x
0
, 0)) from
(x
0
, 0) for the data X R, U, (f, L) over the interval [t
0
, t
1
], i.e. we con-
sider the set of all points (x, y) X R for each of them there exists
((x(), y()), u()) T (T
[t
0
,t
1
]
(X R, U, (f, L)) such that x(t
0
) = x
0
, y(t
0
) =
0, and x(t
1
) = x and y(t
1
) = y.
Clearly, ((x
, y
), u
) belongs to the set T (T

[t
0
,t
1
]
(XR, U, (f, L)). More-
over, the optimality of the trajectory-control pair (x
, u
) implies that the

reference TCP ((x
(), y
()), u
()) is such that

a) x
(t
0
) = x
0
, y
(t
0
) = 0 and (x
(t
1
), y
(t
1
)) = ( x, y)

S;
69
b) the reachable set 1
[t
0
,t
1
]
((X R), U, (f, L), (x
0
, 0)) from (x
0
, 0) for
the data X, U, f, L over the interval [t
0
, t
1
] and

S are locally separated at the
point ( x, y) = (x
(t
1
), y
(t
1
)).
Since C is a Boltyanskii tangent cone to S at the point x, there exist a
neighborhood U of the origin and a continuous map : C U S such
that
(v) = x +v +r(v), where
r(v)
|v|
0 as v 0. (26)
We dene the map : C U R
+

S in the following way:
(v, r) =
_
(v), y +r +|(v) x|
2
_
T
.
The estimate (26) implies that
|(v) x|
2
|v|
0 as v 0. (27)
The both estimates (26) and (27) imply that C R
+
is the Boltyanskii
tangent cone to the set

S.
(X R) U R
n
R (x, u, p, t) H(x, u, p, t) R
given by
H(x, u, p, p
0
) := p
0
L(x, u) +
n
j=1
p
j
f
j
(x, u).
Then from the proved Theorem we obtain the existence of a (,
0
) such that
E1. [t
0
, t
1
] (
0
(t), (t)) is an absolutely continuous map;
E2. (
0
(t), (t)) ,= (0,
0) for every t [t
0
, t
1
];

i
(t) =
0
L
x
i
(x
(t), u
(t), t)
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t), t),

0
(t) = 0,
0
, t
1
];
70
H(x
(t), u
(t), (t), t) = max H(x
(t), u, (t), t) : u U
0
, t
1
]
((t
1
),
0
(t
1
)) (C R
+
)
.
The second equation of E3 implies that
0
is a constant. From E5 we ob-
tain that
0
0. So, without loss of generality we may think that
0
0, 1.
The relation E5 implies also that (t
1
) C
. This completes the proof of

the Theorem.
6.4 The maximum principle for a Mayer optimal prob-
lem
n
;
m
;
C4. t
0
and t
1
are real numbers such that t
0
t
1
;
XU[t
0
, t
1
] (x, u, t) f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
and
X x g(x) R
n
are given;
0
, t
1
] the maps
X x f(x, u, t) = (f
1
(x, u, t), . . . , f
n
(x, u, t)) R
n
71
and
X x g(x) R
are continuously dierentiable with rexpect to x, and their partial
of (x, u, t);
C7. x
0
is a given point of X and S is a given subset of X, where
S = x R
n
: f
i
(x) = 0, i = 1, 2, . . . , m, g
j
(x) 0, j = 1, . . . s .
Here we assume that f
i
and g
j
, i = 1, 2, . . . , m, j = 1, . . . , s, are con-
tinuously dierentiable functions.
[t
0
,t
1
]
(X, U, f) dened on
[t
0
, t
1
a) [t
0
, t
1
b) [t
0
, t
1
c) x(t) = f(x(t), u(t), t) for almost every t [t
0
, t
1
];
[t
0
,t
1
]
is dened by
I(x(), u()) := g(x(t
1
)).
(), u
()) is such that

a) (x
(), u
()) T (T
[t
0
,t
1
]
(X, U, f);
b) x
(t
0
) = x
0
and x
(t
1
) S;
c) I(x
(), u
()) J(x(), u()) for all x(), u() T (T

[t
0
,t
1
]
(X, U, f)
such that x(t
0
) = x
0
and x(t
1
) S;
C11. Let x = x
(t
1
). We set
I( x) = j : g
j
( x) = 0.
i
( x), i = 1, 2, . . . , m, are linearly inde-
pendent, we obtain that C is a Boltyanskii tangent cone to S at the
point x = x
(t
1
), where C = C
1
C
2
,
C
1
=
_
_
_
v R
n
:
__
f
i
x
( x)
_
T
, v
_
= 0, i = 1, 2, . . . , m
_
_
_
,
72
C
2
=
_
_
_
v R
n
:
__
g
j
x
( x)
_
T
, v
_
< 0, j I(x
0
)
_
_
_
.
0
, t
1
, f, L, x
0
, S satisfy
[t
0
,t
1
]
(x
(), u

X U R
n
R R (x, u, p, t) H(x, u, p, t) R
given by
H(x, u, p, t) :=
n
j=1
p
j
f
j
(x, u, t)
0
) such that
E1. [t
0
, t
1
E2.
0
0, 1;
0
, t
1
];

i
(t) =
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t), t)
0
, t
1
] (here by v
T
is denoted the transposed vector
of v);
H(x
(t), u
(t), (t), t) = max H(x
(t), u, (t), t) : u U
0
, t
1
]
(t
1
) p
0
_
g
x
( x)
_
T
+C
,
73
where C
= C
1
+C
2
,
C
1
=
_
_
_
v
R
n
: v
=
m
i=1
i
_
f
i
x
( x)
_
T
,
i
R, i = 1, . . . , m
_
_
_
and
C
2
=
_
_
_
v
R
n
: v
jI( x)
j
_
g
j
x
( x)
_
T
,
j
0, j I( x)
_
_
_
.
Proof. Let us dene the set
S :=
_
x S : g(x) g( x) +|x x|
2
_
and consider the reachable set 1
[t
0
,t
1
]
(X, U, f, x
0
) fromx
0
for the data (X, U, f)
over the interval [t
0
, t
1
], i.e. we consider the set of all points x X for each
of them there exists (x(), u()) T (T
[t
0
,t
1
]
(X, U, f) such that x(t
0
) = x
0
and x(t
1
) = x.
Clearly, (x
, u
) belongs to the set T (T

[t
0
,t
1
]
(X, U, f, L). Moreover, the
optimality of the trajectory-control pair (x
, u
) implies that the reference

TCP (x
(), u
()) is such that

a) x
(t
0
) = x
0
and x
(t
1
) = x

S;
b) the reachable set 1
[t
0
,t
1
]
((X, U, f, x
0
) from x
0
for the data X, U, f
over the interval [t
0
, t
1
] and

S are locally separated at the point x.
Since

S = x R
n
: g(x) g( x) +|x x|
2
,
f
i
(x) = 0, i = 1, 2, . . . , m, g
j
(x) 0, j = 1, . . . s, we ob-
tain according to Example 2 that the Boltyanskii tangent cone to

S at the
point x is just
_
_
_
p
0
_
g
j
x
( x)
_
T
, p
0
0
_
_
_
+C
.
Then from the proved Theorem we obtain the existence of a such that
E1. [t
0
, t
1
0
, t
1
];

i
(t) =
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t), t)
74
0
, t
1
] (here by v
T
is denoted the transposed vector
of v);
H(x
(t), u
(t), (t), t) = max H(x
(t), u, (t), t) : u U
0
, t
1
]
(t
1
)
_
_
_
p
0
_
g
j
x
( x)
_
T
, p
0
0
_
_
_
+C
.
Let
(t
1
)
0
_
g
j
x
( x)
_
T
+C
.
Since
0
0, we may think that
0
0, 1. This completes the proof of
the Theorem.
75
7 Necessary conditions in innite horizon op-
timal problems
A simple optimization problem with an innite horizon problem is the fol-
lowing:
_

0
e
rt
L(x(t), u(t)) dt max (28)
subject to
x(t) = f(x(t), u(t)) and x(0) = x
0
. (29)
The state x(t) belongs to an open subset A of R
n
, and the control u(t)
is a bounded measurable function taking its values from a closed subset |
of R
m
. The functions f : A | R
n
and L : A | R are continu-
ous and continuously dierentiable with respect to the state variable. The
corresponding dierentials are denoted by f
x
and L
x
.
A trajectory-control pair (x(t), u(t)), 0 t < is admissible if x() is a
solution of (29) with control u() on [0, ) and if the integral (28) converges.
A trajectory-control pair (x
(t), u
(t)), 0 t < is optimal if it is admissible

and optimal in the set of admissible trajectory-control pairs, i.e. for any
admissible trajectory-control pair (x(t), u(t)), 0 t < , the value of the
integral (28) is not greater then than its value corresponding to (x(), u()).
7.1 Reduction of an innite horizon optimal problem
to a nite horizon optimal problem
We now consider an optimal trajectory-control pair (x
(t), u
(t)), 0 t < ,
and a xed time T > 0. Let us dene
h(z) = e
r(zT)
_

T
e
rt
L(x
(t), u
(t)) dt =
=
_

z
e
rt
L(x
(t z +T), u
(t z +T)) dt.
The following problem (P
T
) will be considered, with state (X, Z) belong-
ing to X R, and control (U, V ) belonging to |
_
1
2
,
_
:
h(Z(T)) +
_
T
0
V (t)e
rZ(t)
L(X(t), U(t)) dt max (30)
76
subject to
_

X(t) = V (t)f(X(t), U(t)), X(0) = x
0
and X(T) = x
(T),
Z(t) = V (t), Z(0) = 0

(31)
Lemma 7.1 The trajectory (x
(t), t) and the control (u
(t), 1), 0 t T,
constitute an optimal trajectory-control pair for the problem (P
T
).
Proof. The trajectory (x
(t), t) and the control (u
(t), 1), 0 t T, verify

equation (31) because (x
(t), u
(t), t [0, T], verify the equation (29). The

corresponding value of the integral (30) is
h(T) +
_
T
0
e
rt
L(x
(t), u
(t)) dt =
_

0
e
rt
L(x
(t), u
(t)) dt
For any admissible trajectory-control pair (X(t), Z(t), U(t), V (t)), 0
t T, the function Z(t) =
_
t
0
V (t) dt is continuous, strictly increasing and
reversible. Let us dene the functions:
s = Z(t), t [0, T], and t = Z
1
(s), s [0, S],
where S = Z(T). For each s [0, S] we set
u(s) = U
_
Z
1
(s)
_
and x(s) = X
_
Z
1
(s)
_
.
Then for each t [0, T] we have x(Z(t)) = X (Z
1
(Z(t))) = X(t) and the
following inequalities holds true:
_
T
0
V (t)e
rZ(t)
L(X(t), U(t)) dt =
_
T
0
Z(t)e
rZ(t)
L(x(Z(t)), u(Z(t))) dt =
=
_
S
0
e
rs
L(x(s), u(s)) ds
x(s) = x(Z(t)) = X(t) = x
0
+
_
t
0
X() d = x
0
+
_
t
0
V ()f(X(), U()) d =
= x
0
+
_
t
0
Z()f(x(Z()), u(Z())) d = x
0
+
_
z(t)
0
f(x(), u()) d,
77
i.e. for each s [0, S]
x(s) = x
0
+
_
s
0
f(x(), u()) d.
The last equality implies that the function x(s) is almost everywhere dif-
ferentiable with derivative f(x(s), u(s)). The nal condition on X(T) gives:
x(S) = x(Z(T)) = X(T) = x
(T).
For s S, the trajectory-control pair (x(s), u(s)) is dened by:
x(s) = x
(s S +T), and u(s) = u
(s S +T).
The trajectory x() is an absolutely continuous function and veries the
equation (29): for s S, this has been shown; for s S, the last denition
gives:
x(s) = x
(s S +T) = f(x
(s S +T), u
(s S +T)) = f(x(s), u(s)).

On the other hand, the denition of the function h implies
h(z(T)) = h(S) =
_

S
e
rt
L(x
(t S +T), u
(t S +T)) dt =
=
_

S
e
rs
L(x(s), u(s)) ds.
Consequently, the value of the criterion (30) of the problem (P
T
) is:
h(z(T)) +
_
T
0
V (t)e
rt
L(X(t), U(t)) dt =
_

0
e
rs
L(x(s), u(s)) ds.
The trajectory-control pair (x(s), u(s)), 0 s < is admissible, and the
optimality of (x
(t), u(t)) implies:

_

0
e
rt
L(x(t), u(t)) dt
_

0
e
rt
L(x
(t), u
(t)) dt.
The left term is equal to the value of the criterion (30) corresponding to
the trajectory-control pair (X(t), Z(t)), U(t), V (t))), 0 t T; analogously,
the right term is equal to the value of the criterion (30) corresponding to the
trajectory-control pair ((x
(t), t), (u(t), 1)), 0 t T. This shows that the

latter is an optimal trajectory-control pair of the problem (P
T
). The proof
of the lemma is complete.
78
7.2 Innite horizon optimal problems and the maxi-
mum principle
Let us dene the following Hamiltonian:
H (x, u, p
0
, p) :=
n
j=1
p
j
f
j
(x, u) + p
0
e
rt
L(x, u).
Theorem 7.2 A necessary condition for (x
(t), u
(t)), 0 t < , to be an
optimal solution of the problem (28) - (29) is that there exist a real number a,
a vector A of R
n
, and an absolutely continuous function (t) : [0, +) R
n
such that
i) a 0, 1;
ii) (a, A) ,= (0,
0);
iii) (t) is the solution of
_
_

i
(t) =
H
x
i
(x
(t), u
(t), a, (t)),
(0) = A;
iv) for almost each t [0, +)
H(x
(t), u
(t), a, (t)) = max

uU
H(x
(t), u, a, (t));
v) for almost each t [0, +)
H(x
(t), u
(t), a, (t)) = ar
_

t
e
r
L(x
(), u
()) d.
Remark 7.3 In fact, the dierential equation from iii) can be written as
follows:
_
_

i
(t) = ae
rt
L
x
(x
(t), u
(t))
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t)),
(0) = A;
(32)
79
Proof. Let ((X(), Y (), Z()), (U(), V ()) be the set of all admissible
trajectory-control pairs, i.e.
a) [0, T] t U(t) U is a measurable bounded map;
b) [0, T] t V (t) [1/2, +) is a measurable bounded map;
c) [0, T] t (X(t), Y (t), Z(t)) X R R is an absolutely
continuous map;
d) for almost every t [0, T]
_
X(t) = V (t)f(X(t), U(t)), X(0) = x

0
and X(T) = x
(T),
Y (t) = V (t)e
rZ(t)
L(X(t), U(t)), Y (0) = 0,
Z(t) = V (t), Z(0) = 0

(33)
Next, we dene the set S := (X, Y, Z) A R R :
X = X
(T), Y +h(Z)

Y +h(

Z) +|Z

Z|
2
+|Z

Z|
2
_
,
where

Z = Z
(T) and
Y := Y
(T), Y
(t) :=
_
t
0
V ()e
rZ()
L(X(), U()) d,
By 1
[0,T]
(X R R), (U [1/2, ), (f, V, L), (x
0
, 0, 0)) we denote the
reachable set from the point (x
0
, 0, 0) for the phase space (X R R, for
the control space U [1/2, ) and right-hand-side (f, V, L) over the interval
[0, T], i.e. we consider the set of all points (X, Y, Z) X R R for each
of them there exists a trajectory-control pair ((X(), Y (), Z()), (U(), V ())
such that X(0) = x
0
, Y (0) = 0, Z(0) = 0, and X(T) = X, Y (T) = Y and
Z(T) = Z.
According to Lemma 7.1, ((X
, Y
, Z
), (U
, V
)) with U
(t) = u
(t),
V
(t) = 1, X
(t) = x
(t) and Z
(t) = t), t [0, T], is the optimal so-

lution of the problem (P
T
). Clearly, ((X
, Y
, Z
), (U
, V
)) is an admis-
sible trajectory control pair for this problem. Moreover, the optimality of
the trajectory-control pair ((X
, Y
, Z
), (U
, V
)) implies that the reference

((X
, Y
, Z
), (U
, V
)) is such that the reachable set 1

[0,T]
(XRR), (U
[1/2, ), (f, V, L), (x
0
, 0, 0)) and the set S are locally separated at the point
(

X,

Y ,

Z).
It can be directly checked that the Boltyanskii tangent cone to S at the
point (

X,

Y ,

Z) is
C =
_
(v
x
, v
y
, v
z
) : v
x
= 0, (1).v
y
+
_
h
z
(

Z)
_
.v
z
0
_
80
and
C
=
_
(v
x
, v
y
, v
z
) : v
y
= , v
z
= h
z
(

Z), 0
_
.
For the problem (P
T
) the Hamiltonian is:
H(X, Y, Z, U, V, P
x
, P
y
, P
z
) =
= V P
y
e
rz
V L(X, U) + V P
z
+V
n
j=1
P
T
x j
f
j
(X, U).
Then the maximum principle of Pontryagin implies that there exist
E1. [0, T] (
T
x
(t),
T
y
(t),
T
z
(t)) is an absolutely continuous map;
E2. (
T
x
(t),
T
y
(t),
T
z
(t)) ,= (
0, 0, 0) for every t [0, T];

_
T
x
(t) =
T
y
(t)e
rt
L
x
(x
(t), u
(t))
n
j=1
T
xj
(t)f
xj
(X
(t), U
(t)),
T
y
(t) = 0,
T
z
(t) = r
y
(t)
T
e
rt
L(X
(t), U
(t))
(34)
H(X
(t), Y
(t), Z
(t), U
(t), V
(t),
T
x
(t),
y
(t)
T
,
T
z
(t)) =
= max
_
H(X
(t), Y
(t), Z
(t), U, V,
x
(t)
T
,
T
y
(t),
T
z
(t))
: U |, V [1/2, +)
for almost every t [0, T].
(
T
x
(T),
T
y
(T),
T
z
(T)) C
.
The second equation of (34) implies that there exists a constant a
T
such
that
T
y
(t) = a
T
for each t [0, T]. Then, according to E5 and to the
expression of C
T
Z
(T) = a
T
h
z
(T) = ra
T
_

T
e
rt
L(x
(t), u
(t)) dt.
81
The Hamiltonian is maximum with respect to (U, V ) on the set U
_
1
2
,
_
at (u
(t), 1). The maximum with respect to V for U = u
(t) gives:
a
T
e
rt
L(x
(t), u
(t)) +
T
x
(t)f(x
(t), u
(t)) +
T
z
(t) = 0, (35)
and the maximum with respect to u for V
1 implies that
a
T
e
rt
L(x
(t), u) +
T
x
(t)f(x
(t), u)
has a maximum on the set U at u = u
(t).
The vector A
T
:=
T
x
(0) veries (a
T
, A
T
) ,= (0,
0). If not, then a

T
= 0
and
T
x
(0) = 0. Since
T
x
(t) is a solution of
T
x
(t) =
T
x
(t)f
x
(x
(t), u
(t)),
starting from the origin, we obtain that
T
x
(T) =

0, and so (a
T
,
T
x
(T)) =
(0,
0). Since
T
z
(T) = 0.a
T
h
z
(T) = 0, we obtain that (
T
x
(T),
T
y
(T),
T
z
(T)) =
(
0, 0, 0), which is impossible. Hence, (a

T
, A
T
) ,= (0,
0). Without loss of gen-

erality, we may think that the norm of (a
T
, A
T
) equal to 1.
All these properties are true for every T > 0. Let T
n
be an increasing
sequence of positive real numbers such that T
n
+ as n . All the
(a
T
n
, A
T
n
) being of norm 1, there exists some sequence
_
a
T
n
,

A
T
n
_
which has
a limit (a, A) ,= (0,
0). Since the conclusion of Theorem does not change if

we multiply a and A by one positive number, we may think that a 0, 1
(the conclusions i) and ii) of the theorem). Let us dene () and () by:
_
(t) = ae
rt
L
x
(x
(t), u
(t))
n
j=1
(t)f
xj
(x
(t), u
(t))
(0) = A;
(t) = ra
_
t
e
rs
L(x
(s), u
(s)) ds
Let us x an arbitrary T
n
, and let
n
x
,
n
y
= a
T
n
and
n
z
be the corre-
sponding
T
x
,
T
y
and
T
z
. For each nonnegative t
n
z
(t) =
n
z
(T
n
)
_
T
n
t
n
z
() d =
= ra
T
n
_

T
n
e
rt
L(x
(), u
()) d
_
T
n
t
ra
T
n
e
rt
L(x
(), u
()) d =
82
= ra
T
n
_

t
e
rt
L(x
(), u
()) d.
Since a
T
n
a as n , the function (t) is the limit of
n
z
(t) for each
t 0.
Also (t) is the limit of
n
x
(); this follows from the denition of () and
of
n
x
() which implies:
(t) = (t, 0)A a
_
t
0
e
rt
(t, s)L
x
(x
(s), u
(s)) ds
and
n
x
(t) = (t, 0)A
T
n
a
T
n
_
t
0
e
rt
(t, s)L
x
(x
(s), u
(s)) ds
with (t, s) the fundamental matrix solution of the linear equations:
z(t) =
n
j=1
z
j
(t) f
xj
(x
(s), u
(s)).
Thus the conclusion iii) and iv) of the Theorem hold true.
According to (35), we have that
a
T
n
e
rt
L(x
(t), u
(t)) +
n
x
(t)
T
f(x
(t), u
(t)) +
n
z
(t) = 0, (36)
Since
n
x
(t) (t),
n
z
(T
n
) (t) as n ), we take a limit in (36)
and obtain conclusion v) of the Theorem. So, the proof is complete.
7.3 Innite horizon optimal problems and transversal-
ity conditions
Obtaining of transversality conditions for innite horizon optimal problems is
an open problem. But, under additional assumptions, the the next corollary
is an example of a result in this direction:
Corollary 7.4 Assuming that L is nonnegative and there exists a neigh-
borhood V of 0 R
n
which is contained in the set of the possible speeds
f(x
(t), u) for u | and for all suciently large t. Then, an optimal solu-
tion in innite horizon veries in addition to the conclusions of the Theorem,
the following transversality condition:
lim
t
(t) = 0.
83
Proof. Since L is nonnegative, conclusions iv) and v) of the Theorem imply
that for all u U
n
j=1
j
(t) f
xj
(x
(t), u) (t).
We dene
Q(t) :=
1
max(1, |(t)|)
(t) and := limsup
t
|Q(t)|.
If is zero, the corollary is veried. Assume that > 0 and consider a
sequence t
n
converging to innity and such that |Q(t
n
)| > /2. Then, for
each suciently large n the set f(x
(t
n
), u) : u |, contains all points
in R
n
of norm less than > 0. Moreover, lim
ninfty
(t
n
) = 0. Then (t
n
) is
not greater than /2 for each suciently large n. Then there exists u
n
|
such that f(x
(t
n
), u
n
) = (2/)Q(t
n
), and we obtain:
n
j=1
j
(t
n
) f
xj
(x
(t
n
), u
n
) = max(1, |(t
n
)|)(2/)|Q(t
n
)|
2
,
n
j=1
j
(t
n
)f
xj
(x
(t
n
), u
n
) > /2 (t
n
).
This is a contradiction. The proof of the corollary is complete.
Example 7.5
I(u) :=
_

0
(1 x(t))u(t) dt max
subject to
x(t) = (1 x(t))u(t), x(0) = 0,
where the control u(t) belongs to the interval (, ), < 1 < .
If u() is an arbitrary admissible control, then the corresponding trajec-
tory x() is:
x(t) = e
_
t
0
u(s) ds
.0 +
_
t
0
e
_
t
s
u() d
u(s) ds =
84
=
_
t
0
e
_
s
t
u() d
d
_
_
s
t
u() d
_
= e
_
s
t
u() d
t
0
= 1e
_
t
0
u() d
.
Then the value of the criterion is:
I(u) :=
_

0
(1 x(t))u(t) dt =
_

0
e
_
t
0
u() d
u(t) dt =
=
_

0
e
_
t
0
u() d
d
_
_
t
0
u() d
_
= e
_
t
0
u() d
0
= 1e
_

0
u() d
.
This shows that any control such that
_
0
u() d = is optimal. For
example, u
(t) = 1 for each t [0, ) is an optimal control. Let us denote

by x
the corresponding trajectory.

Then the maximum of the Hamiltonian
a(1 x
(t))u +(t)(1 x
(t))u = (a +(t))(1 x
(t))u
for u (, ) is reached at u
(t) = 1 if and only if (t) = a. Hence,

a ,= 0. In this example, x
(T) 1 as T , and the set of all possible

speeds (1 x
(t))u : u (, ), contains a given neighborhood of the origin

for all T suciently large if and only if = and = . We have to
mention that we can not apply Corollary 102 for this example, because the
assumption that L takes only nonnegative values is not true.
85
8 Sucient conditions in innite horizon op-
timal problems
8.1 Preliminaries
First we shall prove two lemmas which are important for our exposition:
Lemma 8.1 Let S be a convex subset of R
n
and f be a real-valued concave
function dened on S. If x
0
is an interior point of S, there exists a vector
p R
n
such that for each point x S
f(x) f(x
0
) < p, x x
0
> .
Proof. We set M := (x, z)
T
: x S, f(x) z. Let us x two
arbitrary points (x
1
, z
1
)
T
and (x
2
, z
2
)
T
from M, and let c [0, 1]. Then
cx
1
+(1 c)x
2
S because S is convex. The denition of the set M and the
concavity of f imply that
cz
1
+ (1 c)z
2
cf(x
1
) + (1 c)f(x
2
) f (cx
1
+ (1 c)x
2
) .
So, the point (cx
1
+(1c)x
2
, cz
1
+(1c)z
2
)
T
also belongs to M. This shows,
that M is convex.
The denition of the set M also implies that the point (x
0
, f(x
0
))
T
is a
boundary point of M. Then we can nd a nonvanishing vector (a, b)
T
R
n
R such that
< a, x > + bz < a, x
0
> + bf(x
0
)
for each point (x, z)
T
M. Since (x
0
, z)
T
M for z < f(x
0
) and since x
0
belongs to the interior of S, we obtain that b > 0. We set p := a/b and
complete the proof.
Lemma 8.2 Let x
0
be a point of R
n
, B be an open convex neighborhood
of x
0
, f be a concave function and g be a dierentiable real-valued function
dened on B such that f(x
0
) = g(x
0
) and g(x) f(x) for all x B. Then
for each point x B
f(x) f(x
0
) g
(x
0
)(x x
0
).
86
Proof. Let x be an arbitrary point of B and c be an arbitrary number from
the interval [0, 1]. We set x
c
= x
0
+ c(x x
0
). The convexity of B implies
that x
c
belongs to B. Applying Lemma 1, we nd a vector p such that
f(x
c
) f(x
0
) < p, x
c
x
0
> = c < p, x x
0
> .
Since f(x
0
) = g(x
0
) and g(x
c
) f(x
c
), we obtain that
g(x
c
) g(x
0
) c < p, x x
0
> .
Hence,
g(x
c
) g(x
0
)
c
_
< p, x x
0
> for c > 0;
< p, x x
0
> for c < 0.
Since
d
dc
g(x
c
) = g
(x
0
)(x x
0
), it follows that g
(x
0
)(x x
0
) = <
p, x x
0
>. So, (g
(x
0
))
T
= p, because x
0
belongs to the interior of B.
8.2 Overtaking optimal trajectory-control pairs and

transversality conditions
Let us consider the following innite horizon problem:
I(x
0
, u) =
_

0
L(x(t), u(t), t) dt max (37)
subject to
x(t) = f(x(t), u(t), t) and x(0) = x
0
. (38)
n
is a bounded measurable function taking its values from a compact subset |
of R
m
. The functions f : A U R
+
R
n
and L : A U R
+
R are
continuous and continuously dierentiable with respect to the state variable.
The corresponding dierentials are denoted by f
x
and L
x
.
A trajectory-control pair (x(t), u(t)), 0 t < , is admissible if x() is a
Let (x(t), u(t)), 0 t < , be an admissible trajectory-control pair. For
each T > 0 we set:
I
T
(x
0
, u) :=
_
T
0
L(x(t), u(t), t) dt.
87
(t), u
(t)), 0 t < , is optimal if it is

admissible and optimal in the set of admissible trajectory-control pairs, i.e.
for any admissible trajectory-control pair (x(t), u(t)), 0 t < , the value
of the integral (37) is not greater than its value corresponding to (x(), u()).
Following [3], we say that a trajectory-control pair (x
(t), u
(t)), 0 t <
, is overtaking optimal if it is admissible and for each admissible trajectory-
control pair (x(t), u(t)), 0 t < ,
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) 0,
i.e. for each > 0 there exists T(, u) > 0 such that for each T > T(, u)
I
T
(x
0
, u
) > I
T
(x
0
, u) .
Let us dene the following Hamiltonian H : A | R
n
R
+
R:
H (x, u, p, t) := L(x, u, t) +
n
j=1
p
j
f
j
(x, u, t) (39)
and the maximized Hamiltonian
H
(x, p, t) = max
uU
H(x, u, p, t). (40)
Theorem 8.3 Let the following assumptions hold true: Let (x
(t), u
(t)),
0 t < , be an admissible trajectory-control pair and (t), 0 t < , be
an absolutely continuous function such that the following assumptions hold
true:
A1. The function H
(x, p, t) is well dened for each (x, p, t) A R

n
R
+
and concave with respect to x for each xed point (p, t) R
n
R
+
;
A2. For almost all t [0, +)
d
dt
i
(t) =
L
x
i
(x
(t), u
(t), t)
n
j=1
j
(t)
f
j
x
i
(x
(t), u
(t), t).
H (x
(t), u
(t), (t), t) = H
(x
(t), (t), t) .
88
Then the following transversality condition
liminf
T
n
j=1
j
(T) (x
j
(T) x
j
(T)) 0 (41)
for each admissible trajectory x(t), t [0, ), implies that trajectory-control
pair (x
(t), u
(t)), 0 t < , is overtaking optimal.

Proof. The concavity of the function H
(, p, t) implies that for each

t [0, ) and each x A
H
(x, (t), t) H
(x
(t), (t), t) +
n
j=1
H
x
j
(x
(t), (t), t)(x

j
x
j
(t))
Let (x(t), u(t)), 0 t < , be an arbitrary admissible trajectory-control
pair. Then, taking into account the denition of H
and the assumptions A2

and A3, we obtain that for almost each t [0, +)
L(x(t), u(t), t)+
n
j=1
j
(t)
d
dt
x
j
(t) = L(x(t), u(t), t)+
n
j=1
j
(t)f
j
(x(t), u(t), t) =
= H(x(t), u(t), (t), t) H
(x(t), (t), t)
H
(x
(t), (t), t) +
n
j=1
H
x
j
(x
(t), u
(t), (t), t)(x

j
x
j
(t)) =
= L(x
(t), u
(t), t) +
n
j=1
j
(t)f
j
(x
(t), u
(t), t)+
+
n
j=1
_
L
x
j
(x
(t), u
(t), t) +
n
k=1
k
(t)
f
k
x
j
(x
(t), u
(t), t)
_
(x
j
(t) x
j
(t)) =
= L(x
(t), u
(t), t) +
n
j=1
j
(t)
d
dt
x
(t)
n
j=1
d
dt
j
(t) (x
j
(t) x
j
(t)).
From here we obtain that
L(x
(t), u
(t), t) L(x(t), u(t))

n
j=1
j
(t)
d
dt
(x(t) x
(t))+
+
n
j=1
d
dt
j
(t) (x
j
(t) x
j
(t)) =
n
j=1
d
dt
(
j
(t) (x
j
(t) x
j
(t))).
89
The last inequality implies that
_
T
0
(L(x
(t), u
(t), t) L(x(t), u(t), t)) dt

n
j=1
j
(t) (x
j
(t) x
j
(t))
T
0
=
=
n
j=1
j
(T) (x
j
(T) x
j
(T))
(because x(0) = x
(0) = x
0
). It can be written as follows:
I
T
(x
0
, u
) I
T
(x
0
, u)
n
j=1
j
(T) (x
j
(T) x
j
(T)),
and hence
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) liminf
T
n
j=1
j
(T) (x
j
(T) x
j
(T)) 0
(according to (41)). This completes the proof.
Corollary 8.4 Let the assumptions A1, A2 and A3 hold true. We also
assume that the state variable x(t) and the co-state variable (t), t [0, ),
take nonnegative values, and that
lim
T
n
j=1
j
(T) x
j
(T) = 0.
Then the trajectory-control pair (x
(t), u
(t)), 0 t < , is overtaking opti-

mal.
Proof. As in the proof of Theorem 8.3, we obtain that for each T > 0
I
T
(x
0
, u
) I
T
(x
0
, u)
n
j=1
j
(T) (x
j
(T) x
j
(T)).
Since the state variable x(t) and the co-state variable (t), t [0, ), take
nonnegative values, we obtain that
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) liminf
T
_
_
n
j=1
j
(T) (x
j
(T) x
j
(T))
_
_

90
liminf
T
_
_
n
j=1
j
(T) x
j
(T)
_
_
+ liminf
T
_
_
j=1
j
(T)x
j
(T))
_
_
=
= liminf
T
_
_
n
j=1
j
(T) x
j
(T)
_
_
+ lim
T
_
_
j=1
j
(T)x
j
(T))
_
_
0.
This completes the proof.
assume that
i) lim
T
(T) = 0;
ii) there exists a compact set A
0
such that A A
;
(t), u

mal.
Proof. As in the proof of Theorem 8.3, we obtain that for each T > 0
I
T
(x
0
, u
) I
T
(x
0
, u)
n
j=1
j
(T) (x
j
(T) x
j
(T)),
Since x(T) and x
(T) belong to the compact set A and (T) 0 as

T , then
lim
T
n
j=1
j
(T) (x
j
(T) x
j
(T)) = 0.
Hence for each > 0 there exists T(, u) > 0 such that for each T > T(, u)
I
T
(x
0
, u
) > I
T
(x
0
, u) .
But this means that
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) 0.
91
8.3 Current Hamiltonian and sucient conditions for
innite horizon optimal problems
An important class of innite horizon optimal problems is the following one:
I(x
0
, u) =
_

0
e
rt
L(x(t), u(t)) dt max (42)
subject to
x(t) = f(x(t), u(t)) and x(0) = x
0
. (43)
n
is a bounded measurable function taking its values from a compact subset
| of R
m
n
and L : A U R are contin-
uous and continuously dierentiable with respect to the state variable. The
x
and L
x
.
Then the following necessary condition holds true:
Theorem 8.6 A necessary condition for (x
(t), u
(t)), 0 t < , to be an
optimal solution of the problem (42) - (43) is that there exist a real number a,
a vector A of R
n
, and an absolutely continuous function (t) : [0, +) R
n
such that
i) a 0, 1;
ii) (a, A) ,= (0,
0);
_
_

i
(t) = r
i
(t)
H
c
x
i
(x
(t), u
(t), a, (t)),
(0) = A;
H
c
(x
(t), u
(t), a, (t)) = max

uU
H
c
(x
(t), u, a, (t));
H
c
(x
(t), u
(t), a, (t)) = ra
_

t
e
r(t)
L(x
(), u
()) d.
92
Proof. We set
H (x, u, p
0
, p, t) :=
n
j=1
p
j
f
j
(x, u) + p
0
e
rt
L(x, u).
Since (x
(t), u
(t)), 0 t < , is an optimal solution of the problem

(42) - (43), there exist a real number a, a vector A of R
n
, and an absolutely
continuous function (t) : [0, +) R
n
such that
i) a 0, 1;
ii) (a, A) ,= (0,
0);
_
_

i
(t) =
H
x
i
(x
(t), u
(t), a, (t), t),

(0) =

A;
H(x
(t), u
(t), a, (t), t) = max

uU
H(x
(t), u, a, (t), t);

H(x
(t), u
(t), a, (t), t) = ra
_

t
e
r
L(x
(), u
()) d.
We dene the current value multiplier (t), t [0, ), as follows:
(t) = e
rt
(t).
We dene also the current Hamiltonian H
c
: A | RR
n
R as follows:
H
c
(x, u, a, ) :=
n
j=1
j
f
j
(x, u) + aL(x, u).
It can be directly checked that
e
rt
H
c
(x, u, a, ) = H
_
x, u, a, e
rt
, t
_
.
93
Since (t) = e
rt
(t), we can formulate the necessary optimality condi-
tion in terms of the current Hamiltonian H
c
and current value multiplier
as follows:
To prove iii), one can directly checked that

i
(t) =
d
dt
_
e
rt
(t)
_
= re
rt
(t)+e
rt
(t) = r(t)e
rt
H
x
i
(x
(t), u
(t), a, (t), t) =
= r(t)
H
c
x
i
(x
(t), u
(t), a, e
rt
(t)) = r(t)
H
c
x
i
(x
(t), u
(t), a, (t)).
We also have that (0) = e
r0
(0) = A. So, we have proved iii).
To prove iv), we can write that for almost each t [0, +)
H
c
(x
(t), u
(t), a, (t)) = e
rt
H
_
x
(t), u
(t), a, e
rt
(t), t
_
=
= e
rt
H (x
(t), u
(t), a, (t), t) = e
rt
max
uU
H (x
(t), u, a, (t), t) =
= e
rt
max
uU
H
_
x
(t), u, a, e
rt
(t), t
_
= max
uU
H
c
(x
(t), u, a, (t)).
At last, to prove v), we verify that for almost each t [0, +)
H
c
(x
(t), u
(t), a, (t)) = e
rt
H
_
x
(t), u
(t), a, e
rt
(t), t
_
=
= e
rt
H (x
(t), u
(t), a, (t), t) = rae

rt
_

t
e
r
L(x
(), u
()) d =
= ra
_

t
e
r(t)
L(x
(), u
()) d.
Let us dene the following Hamiltonian

H
c
: A | R
n
R:
H
c
(x, u, p) := L(x, u) +
n
j=1
p
j
f
j
(x, u)
and the maximized Hamiltonian
H
c
(x, p) = max
uU
H
c
(x, u, p).
Then the following assertions can be easily obtained as a corollary of the
results from the previous section:
94
Theorem 8.7 Let the following assumptions hold true: Let (x
(t), u
(t)),
0 t < , be an admissible trajectory-control pair and (t), 0 t < , be
an absolutely continuous function such that the following assumptions hold
true:
A1. The function

H
c
(x, m) is well dened for each (x, p) A R
n
and
concave with respect to x for each xed point (p) R
n
;
d
dt
i
(t) = r
i
(t)

H
c
x
i
(x
(t), u
(t)).
H
c
(x
(t), u
(t), (t)) =

H
c
(x
(t), (t))
and the maximum is achieved at the unique point u
(t).
Then the following transversality condition
liminf
T
n
j=1
e
rT
j
(T) (x
j
(T) x
j
(T)) 0 (44)
for each admissible trajectory x(t), t [0, ), implies that the trajectory-
control pair (x
(t), u
(t)), 0 t < , is overtaking optimal.

Proof. We set (t) := e
rt
(t) and remind the denitions (39) and (40)
of the Hamiltonian H
(x, u, p, t) and the maximized Hamiltonian H
(x, p, t).
Since
H (x, u, p, t) = e
rt

H
c
_
x, u, e
rt
p
_
we obtain that
H
(x, p, t) = max
uU
H(x, u, p, t) = e
rt
max
uU
H
c
(x, u, e
rt
p) = e
rt

H
c
(x, e
rt
p),
we obtain that H
(, p, t) is a concave function for each xed point (p, t).

Next, for almost all t [0, +) we have that
d
dt
i
(t) =
d
dt
_
e
rt
i
(t)
_
= re
rt
i
(t) + e
rt

i
(t) =
95
= re
rt
i
(t) + e
rt
r
i
(t) e
rt

H
c
x
i
(x
(t), u
(t), (t)) =
= e
rt

H
c
x
i
(x
(t), u
(t), e
rt
(t)) =
H
x
i
(x
(t), u
(t), (t), t).

For almost all t [0, +) we have that
H (x
(t), u
(t), (t), t)) = e

rt

H
c
_
x
(t), u
(t), e
rt
(t)
_
=
= e
rt

H
c
_
x
(t), e
rt
(t)
_
= H
(x
(t), (t), t) .
At last, the transversality condition can be written as follows
liminf
T
n
j=1
j
(T) (x
j
(T) x
j
(T)) 0.
Then all assumptions of Theorem 8.3 hold true and we can conclude that
for each positive number T
I
T
(x
0
, u
) I
T
(x
0
, u)
n
j=1
j
(T) (x
j
(T) x
j
(T)) =
= e
rT
n
j=1
j
(T) (x
j
(T) x
j
(T)) 0.
This completes the proof of Theorem 8.7.
assume that the state variable x(t) and the co-state variable (t), t [0, ),
take nonnegative values, and that
lim
T
e
rT
n
j=1
j
(T) x
j
(T) = 0.
(t), u

mal.
96
Proof. As in the proof of Theorems 8.7, we obtain that for each T > 0
I
T
(x
0
, u
) I
T
(x
0
, u) = e
rT
n
j=1
j
(T) (x
j
(T) x
j
(T))
Since the state variable x(t) and the co-state variable (t), t [0, ), take
nonnegative values, we obtain that
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) liminf
T
_
_
e
rT
n
j=1
j
(T) (x
j
(T) x
j
(T))
_
_

liminf
T
_
_
e
rT
n
j=1
j
(T) x
j
(T)
_
_
+ liminf
T
_
_
e
rT
n
j=1
j
(T)x
j
(T))
_
_
=
= liminf
T
_
_
e
rT
n
j=1
j
(T) x
j
(T)
_
_
+ lim
T
_
_
e
rT
n
j=1
j
(T)x
j
(T))
_
_
0.
assume that
i) lim
t
e
rt
(t) = 0;
ii) Let there exists a compact set A
0
such that A A
;
(t), u

mal.
Proof. As in the proof of Theorem 8.7 we obtain that for each T > 0
I
T
(x
0
, u
) I
T
(x
0
, u) e
rT
n
j=1
j
(T) (x
j
(T) x
j
(T)).
Since x(T) and x
(T) belong to the compact set A and e

rT
(T) 0 as
T , then
lim
T
e
rT
n
j=1
j
(T) (x
j
(T) x
j
(T)) = 0.
97
Hence for each > 0 there exists T(, u) > 0 such that for each T > T(, u)
I
T
(x
0
, u
) > I
T
(x
0
, u) .
But this means that
liminf
T
(I
T
(x
0
, u
) I
T
(x
0
, u)) 0.
98
9 Bellman function and the Hamilton-Jacobi-
Bellman equation
9.1 Bellman function and its properties
Let us consider the following optimization problem:
_
T
t
0
L(x(), u(), ) d +g(x(T)) max (45)
subject to
x() = f(x(), u(), ) and x(t
0
) = x
0
. (46)
n
, and the control u(t) is a
bounded measurable function taking its values from a closed subset | of R
m
.
The functions f : A U R
+
R
n
, g : A R and L : A U R
+
R
are continuous with respect to the state variable.
Instead to consider the optimal control problem (45)(46), we consider
the following family T(x, t), x A, t [t
0
, T], of of control problems
I(t, x, u) =
_
T
t
L(x(), u(), ) d +g(x(T)) max (47)
subject to
x() = f(x(), u(), ) and x(t) = x. (48)
A trajectory-control pair (x(t), u(t)), t < T, is admissible for T(x, t),
if x(, t, x, u) is a solution of (48) corresponding to the control u(), [t, T]
and the value of the integral (47) is well dened. We denote by (t, x, T)
the set of all admissible trajectory-control pair (x(), u()), t T. The
corresponding Bellman (value) function V : AR
+
R is dened as follows:
V (x, t) = sup
_
_
T
t
L(x(), u(), ) d +g(x(T)) : (x(), u()) (t, x)
_
.
(49)
Lemma 9.1 For each x A, for each t and s such that t
0
t s T,
and
i) for each admissible pair (x(), u()) (t, x, s),
V (t, x)
_
s
t
L(x(), u(), ) d + V (s, x(s, t, x, u))
99
ii) if the value function V (t, x) is achieved, (i.e. if there exists an admis-
sible pair (x
(s), u
(s)), t s T, such that V (t, x) = I(t, x, u
)),
then
V (t, x) =
_
s
t
L(x
(), u
(), ) d + V (s, x
(s, t, x, u
)).
Proof. Let x A, t and s, t
0
t s T, and (x(), u()) (t, x, s)
are given. Pick any > 0. By the denition of V (s, x(s, t, x, u)), there exists
some admissible pair (y(), v()) (s, x(s, t, x, u), T) such that
V (s, x(s, t, x, u)) I(s, x(s, t, x, u), v).
Let
w : [t, T] =
_
u(t), for [t, s];
v(t), for [s, T].
Then by the denitions of V , we obtain that
V (t, x) I(t, x, w) =
_
s
t
L(x(), u(), ) d +
_
T
s
L(y(), v(), ) d +
+g(y(T)) =
_
s
t
L(x(), u(), ) d +I(s, x(s, t, x, u), v)
_
s
t
L(x(), u(), ) d +V (s, x(s, t, x, u)) .
Taking 0, we complete the proof of i).
We now prove the second part of this lemma. Let there exists an admis-
sible pair (x
(t), u
(t)), t T, such that V (t, x) = I(t, x, u
), i.e.
V (t, x) =
_
T
t
L(x
(), u
(), ) d +g(x
(T)).
Let us assume that ii) is not true. This means that there exists > 0 such
that
V (s, x(s, t, x, u)) >
_
T
s
L(x
(), u
(), ) d +g(x
(T)) +,
and so
>
_
T
s
L(x
(), u
(), ) d +g(x
(T)) V (s, x(s, t, x, u)). (50)

100
According to the denition of V , there exists some admissible pair
(y(), v()) (s, x(s, t, x, u
), T) such that
V (s, x(s, t, x, u
)) I(s, x(s, t, x, u
), v) V (s, x(s, t, x, u
)) . (51)
Let
w : [t, T] =
_
u
(t), for [t, s];

v(t), for [s, T].
According to i), the denitions of V and (50) , we obtain that
V (t, x)
_
s
t
L(x
(), u
(), ) d +V (s, x
(s, t, x, u
))
_
s
t
L(x
(), u
(), ) d +I(s, x(s, t, x, u
), v)
_
s
t
L(x
(), u
(), ) d +V (s, x(s, t, x, u)) >

>
_
s
t
L(x
(), u
(), ) d + V (s, x(s, t, x, u)) +

+
_
T
s
L(x
(), u
(), ) d + g(x
(T)) V (s, x(s, t, x, u)) =

=
_
T
t
L(x
(), u
(), ) d + g(x
(T)) = V (t, x).

The obtained contradiction shows that our assumption is wrong. This com-
pletes the proof of ii).
Lemma 9.2 Let

V : [t
0
, T] A be a function such that for each x A, for
each t and s, t
0
t s T, and
i) for each admissible pair (x(), u()) (t, x, s),
V (t, x)
_
s
t
L(x(), u(), ) d +

V (s, x(s, t, x, u));
ii) there exists an admissible pair (x
(), u
()) (t, x, s), such that
V (t, x) =
_
s
t
L(x
(), u
(), ) d +

V (s, x
(s, t, x, u
));
101
iii) the boundary condition

V (t, x) = g(x) holds for all x A.
Then the Bellman function V is achieved and V =

V .
Proof. Fix any t [t
0
, T] and x A. As a particular case of i) and
using the boundary condition we can conclude that for any admissible pair
(x(), u()) (t, x, T),
V (t, x)
_
T
t
L(x(), u(), ) d + g(x(T)) = I(t, x, u).
Hence,

V (t, x) V (t, x).
By the second property ii) and using the boundary condition we can
conclude that there exists an admissible pair (x
(), u
()) (t, x, T) such

that
V (t, x) =
_
T
t
L(x
(), u
(), ) d + g(x(T)) = I(t, x, u
).
Hence,

V (t, x) V (t, x). This completes the proof.
9.2 The Hamilton-Jacobi-Bellman equation
Denition 9.3 We say that V is a smooth solution of the optimal control
problem (47)(48) on [t
0
, T] A if the following properties hold true:
i) the Bellman function V is continuously dierentiable with respect to
(t, x) on [t
0
, T] A;
ii) for each (t, x) [t
0
, T] A the value function V (t, x) is achieved by
using of a continuous control (that is each all (t, x) [t
0
, T] A there
exists an admissible pair (x
(t), u
(t)), t T, such that u
() is
continuous and V (t, x) = I(t, x, u
).
Proposition 9.4 Assume that there is a smooth solution of the optimal con-
trol problem (47)(48) on [t
0
, T] A, i.e. for all (t, x) [t
0
, T] A the value
function V (t, x) is achieved. Then,
102
i) The Bellman function V satises the so called Hamilton-Jacobi-Bellman
equation (HJB) for all (t, x) [t
0
, T] A
V
t
(t, x)+max
uU
_
L(x, u, t) +
V
x
(t, x)f(x, u, t)
_
= 0 and V (T, x) = g(x);
ii) if there exists an admissible pair (x
(t), u
(t)), t T, such that

u
() is continuous and V (t, x) = I(t, x, u
), then at all pairs (t, x
(t)),
t [t
0
, T], along this trajectory, the maximum in the Hamilton-Jacobi-
Bellman equation is attained at u
(t).
Proof. The nal condition V (T, x) = g(x) holds by the denition V .
According to the condition ii) of Lemma 9.1, for all (t, x) [t
0
, T] A there
exists an admissible pair there exists an admissible pair (x
(), u
()) dened
on [t, T] such that for each s and t, t s T,
V (t, x) =
_
s
t
L(x
(), u
(), ) d + V (s, x
(s)),
Thus the Bellman function evaluated along the optimal trajectory, V (s, x(s))
is absolutely continuous and has derivative
dV
ds
(s, x(s)) = L(x
(s), u
(s), s)
almost everywhere. Since u() is a continuous function, this derivative exist
everywhere. Because
dV
ds
(s, x(s)) =
V
s
(s, x(s)) +
V
s
(s, x(s)) x(s),
we can write the previous identity (using the argument t instead of s) as
follows
V
t
(t, x
(t)) +
V
t
(t, x
(t))f(x
(t), u
(t), t) + L(x
(t), u
(t), t) = 0.
In order, to establish ii), it is only necessary to show that
V
t
(t, x) +
V
x
(t, x)f(x, u, t) + L(x, u, t)) 0 (52)
103
for each control value u at any given point (t, x). Consider any such u and
let u
(s) = u for all s [t, t +]. If is suciently small, then the trajectory
x
() (corresponding to the control u
) will be well dened on [t, t +].

According to the condition i) of Lemma 9.1,
V (t, x)
_
s
t
L(x(), u(), ) d + V (s, x(s))
for each s [t, t +], and hence
h() := V (t +, x
()) V (t, x) +
_
t+
t
L(x(), u(), ) d 0.
Since h(0) = 0, the last inequality implies that h
(0) = lim
0
h() h(0)
=
= lim
0
_
V (t +, x
()) V (t, x)
_
t+
t
L(x(), u(), ) d
_
0,
i.e.
V
t
(t, x) +
V
x
(t, x)f(x, u, t) +L(x, u, t)) 0.
The last inequality completes the proof of ii).
To prove i), it is only needed to show that every possible point (t, x) from
[t
0
, T] A appears as a part of an optimal trajectory. But this is immediate
consequence of the assumption that the problem (47)(48) has an optimal
solution when applied to the initial point (t, x).
With the optimal control problem (47)(48) is closely related the Hamil-
tonian
H(x, u, p, t) = L(x, u, t) +
n
j=1
p
j
f
j
(x, u, t)
seen as a function of (x, u, t) A | [t
0
, T] and p R
n
.
Denition 9.5 Let us consider the control system (48) and let I be a subin-
terval of [t
0
, T]. A function k : I A | is an admissible feedback on I if
for each I and each x A there exists an unique trajectory-control pair
(x(), u()) on J = t I, t such that
x() = x and u(t) = k(t, x(t))
for almost all t J. In this case u is called closed-loop control starting at
(, x).
104
Theorem 9.6 Let us assume that
i) there exists a continuous function : A R
n
[t
0
, T] | satisfying
max
uU
H(x, u, p, t) = H(x, (x, p, t), p, t)
and this is the unique maximum for all (x, p, t) A R
n
[t
0
, T].
ii) There is given a continuously dierentiable function V on [t
0
, T] A
so that k(t, x) = (x, V
x
(t, x), t) is an admissible feedback on t
0
, T.
Then the following two statements are equivalent:
1) V satises the HJB equation on [t
0
, T] A, which can be written as
L(x, k(t, x), t) + V
x
(t, x)f(x, k(t, x), t) + V
t
(t, x) = 0, V (T, x) = g(x);
2) There is a smooth solution of the optimization problem, V is is the
Bellman function, and for each (, x) [t
0
, T] A the closed-loop
control k(, ) starting at (, x) is the unique optimal control for (, x).
Proof. That 2) implies 1) follows from Proposition 9.4. Now, we shall
prove that 1) implies 2). Pick any t and s, t
0
< t < s < T, and any x from
A. Let u
() be the closed-loop control starting at (t, x) up to time s. Let

(x(), u()) be any admissible trajectory-control pair from (t, x, s). Consider
the function B() := V (, x()), [t, s]. This is an absolutely continuous
function, and its time-derivative is
V
t
(, x()) +
V
x
(, x())f(x(), u(), ).
On the other haned, since V satises the HJB equation, we have that
V
t
(, x()) +
V
x
(, x())f(x(), k(, x()), ) + L(x(), k(, x()), ) = 0.
The last two relations imply that
d
dt
V (, x()) =
=
V
x
(, x()) (f(x(), u(), ) f(x(), k(, x()), ))L(x(), k(, x()), ).
105
Because of i), the last expression is less or equal to L(x(), u(), ), and
equal to this when u(t) = u
(t) = k(t, x(t)). So,

V (s, x(s)) V (t, x)
_
s
t
L(x(), u(), ) d
with equality when u = u
. By continuity on t and s, the same holds true

even if t = t
0
or s = T. So, the assumptions of Lemma 9.2 hold. We conclude
that k is indeed optimal for (, x), and it is continuous because is.
Finally, we prove the uniqueness statement. Assume that u ,= u
on a set
S of positive measure. Then B/t is strictly less than L(, x(), u()) on
this set, and hence
V (T, x(T)) V (t, x) <
_
T
t
L(x(), k(, x()), ) d,
that is
V (t, x) > g(x(T))
_
T
t
L(x(), k(, x()), ) d = I(t, x, u).
Thus u can not be optimal.
Let us consider the following innite horizon problem:
V (t, x) := sup
__

t
e
r
L(x(), u()) d : x() is a solution of (54)
_
, (53)
where
x(t) = f(x(t), u(t)) and x(t) = x. (54)
n
is a bounded measurable function taking its values from a closed subset |
of R
m
n
and L : A U R are continu-
ous and continuously dierentiable with respect to the state variable. The
x
and L
x
.
A trajectory-control pair (x(t), u(t)), 0 t < is admissible if x() is a
(t), u
(t)), 0 t < is optimal if it is admissible

and optimal in the set of admissible trajectory-control pairs, i.e. for any
admissible trajectory-control pair (x(t), u(t)), 0 t < , the value of the
integral (53) is not greater then than its value corresponding to (x(), u()).
106
Let the trajectory-control pair (x
(t), u
(t)), 0 t < is optimal. Then

V (t, x) :=
_

t
e
r
L(x
(), u
()) d = e
rt
_

t
e
r(t)
L(x
(), u
()) d =
= e
rt
_

0
e
r
L(x
( +t), u
( +t)) d = e
rt
V (x, 0),
and so
t
V (t, x) =

t
_
e
rt
V (x, 0)
_
= re
rt
V (x, 0) = rV (x, t).
Assuming that V is a smooth function. Then the HJB equation
t
V (t, x
(t)) +

x
V (t, x
(t))f(x
(t), u
(t)) + e
rt
L(x
(t), u
(t))
together with
x
V (t, x
(t)) = (t)
implies the last relation obtained from the Pontryagin maximum principle
for the case a = 1. But, as it is well known, the assumption that V is C
1
does not hold in the general case studied here.
9.3 A linear-quadratic optimal control problem
Let us consider the following linear control system:
x(t) = A(t) x(t) + B(t) u(t). (55)
We assume that A and B are measurable essentially bounded (m.e.b) ma-
trices of suitable dimensions. We denote by |
[t
0
,T]
the set of all measurable
essentially bounded m-valued function u : [t
0
, T] R
m
. We assume that
three matrices of functions are also given:
R is an mm symmetric matrix measurable essentially bounded func-
tion, Q is an n n symmetric matrix measurable essentially bounded
function and S is a constant symmetric matrix;
for each t, R is positive denite;
for each t, Q is positive semidenite;
107
S is positive semidenite.
For each pair of real numbers (t, T) with t
0
t T, for each state x
and each measurable essentially bounded control function u : [t, T] R
m
,
we dene
I(T, t, x, u) :=
_
T
t
u(s)
T
R(s)u(s) ds +
_
T
t
(s)
T
Q(s) (s) ds +(T)
T
S (T),
where () is the corresponding solution of the linear system (55) with the
initial condition (t) = x. Here we have denoted by M
T
the transposed
matrix of the matrix M. The main result of this section is
Theorem 9.7 Pick any t
0
< T. Then there exists an absolutely continuous
symmetric n n matrix function P(t) dened on the interval [t
9
, T], which
satises the Riccati dierential equation
P(t) = P(t)B(t)R
1
(t)B
T
(t)P(t) P(t)A(t) A
T
(t)P(t) Q(t),
P(T) = S
(56)
Let
F(t) := R
1
(t)B
T
(t)P(t). (57)
Then the F-closed loop control staring at (t
0
, x
0
) is the unique optimal control
for the point x
0
on the interval [t
0
, T]. Moreover, for all x R
n
and all
t [t
0
, T]
V (t, x) := x
T
P(t)x,
where the value function V : [t
0
, T] R
n
R is dened as follows:
V (t, x) := inf
_
I(T, t, x, u) : u |
[t
0
,T]
_
.
Proof. The proof follows from Theorem 9.6. Consider the nal value
problem (56) on the interval [t
0
, T]. Let us assume that the corresponding
solution is dened on the interval [, T]. Pick any x R
n
, any control
function u : [t
0
, T] and any t [, T]. Denote by () the corresponding
solution of the linear system (55) with the initial condition (t) = x. Let
P() denote the solution of the ordinary Riccati dierential equation (56).
108
Then
d
dt
_
T
(t)P(t)(t)
_
=

T
(t)P(t)(t) +
T
(t)

P(t)(t) +
T
(t)P(t)

(t) =
=
T
(t)A
T
(t)P(t)(t) +u
T
B
T
(t)P(t)(t) +
T
(t)P(t)B(t)R
1
(t)B
T
(t)P(t)(t)
T
(t)P(t)A(t)(t)
T
(t)A
T
(t)P(t)(t)
T
(t)Q(t)(t)
+
T
(t)P(t)A(t)(t) +
T
(t)P(t)B(t)u(t) =
= (u(t) +R
1
(t)B
T
(t)P(t)(t))
T
R(t) (u(t) +R
1
(t)B
T
(t)P(t)(t))
u(t)
T
R(t)u(t)
T
(t)Q(t)(t)
Integrating, we obtain that
T
(T)S(T) =
T
(T)P(T)(T) =
T
()P()()
+
_
T
_
u(t) + R
1
(t)B
T
(t)P(t)(t)
_
T
R(t)
_
u(t) + R
1
(t)B
T
(t)P(t)(t)
_
dt
_
T
u(t)
T
R(t)u(t)
T
(t)Q(t)(t) dt
Taking into account that () = x, the last equality implies that
I(T, , x, u) = x
T
P()x+
_
T
_
u(s) +R
1
(s)B
T
(s)P(s)(s)
_
T
R(s)
_
u(s) + R
1
(s)B
T
(t)P(s)(s)
_
ds
From this inequality, it is clear that the unique minimum cost control is
that one making u(s) +R
1
(s)B
T
(s)P(s)(s) 0, and that this minimum is
V (x) = x
T
P()x.
Consider again the nal value problem (56). Let P() be its maximal
solution which is dened on some maximal relatively open (with respect to
[t
0
, T]) interval (t
, T]. We choose the positive constant K so that

max
_
(T, t)
T
_
_
T
t
(s, T)
T
Q(s)(s, T) ds +S
_
(T, t), t [t
, T]
_
K,
109
where (t, s) denote the fundamental solution of the problem

X(t) = A(t)X(t)
with X(s) being the identity matrix.
Pick any x R
n
and any t (t
, T]. Then
0 V (t, x) I(T, t, x, 0) =
= x
T
(T, t)
T
_
_
T
t
(s, T)
T
Q(s)(s, T) ds +S
_
(T, t)x K|x|
2
.
Since the norm of a symmetric and nonnegative matrix equals the largest
possible value of x
T
P(t)x with |x| = 1, we conclude that |P(t)| K. From
here, it follows easily that the maximal solution P() is dened on the interval
(, ). In particular, P() is dened on the interval [t
0
, T].
We dene the Hamiltonian as follows:
H(t, x, u, ) := u
T
R(t)u +x
T
Q(t)x +
T
(A(t)x +B(t)u).
We are looking for some unknown matrices C and D satisfying the fol-
lowing equality:
u
T
R(t)u +
T
B(t)u = (u +C)
T
R(t)(u +C) + D.
Then the equalities
u
T
R(t)u +
T
B(t)u = (u +C)
T
R(t)(u +C) + D =
= u
T
R(t)u +
T
C
T
R(t) u +u
T
R(t)C +
T
C
T
R(t) C +D =
= u
T
R(t)u + 2
T
C
T
R(t)u +D.
imply that
2C
T
R(t) = B(t) and hence C =
1
2
R
1
(t)B
T
(t).
Moreover,
D =
T
C
T
R(t) C =
1
4
T
B(t)R
1
(t)R(t) R
1
(t)B
T
(t) =
=
1
4
T
B(t)R
1
(t)B
T
(t)
110
Hence
H(t, x, u, ) =
_
u +
1
2
R
1
(t)B(t)
T
)
T
_
R(t)
_
u +
1
2
R
1
(t)B
T
(t)
_
+D.
Since the matrix R(t) is positive denite, the minimal value of the Hamilto-
nian is obtained for u(t, x, ) =
1
2
R
1
(t)B(t)
T
.
Since V (x) = x
T
P(t)x, we have that V
x
(x) = 2x
T
P(t). Then the feedback
law from Theorem 9.6
k(t, x) = u(t, x, V
x
(x)) = R
1
(t)B(t)
T
Px.
Substituting this feedback in the dynamics (55), we get the closed loop system
x(t) = (A(t) + B(t)F(t))x(t) with F(t) = R
1
(t)B(t)
T
P.
This is again a linear system, so it has a solution for each initial point on
the interval (, ). Moreover, the Hamilton-Jacobi-Bellman equation for
V (x) = x
T
Px is equivalent to the ordinary Riccati dierential equation (56).
This complete the proof.
111
10 On the optimal taxation problem
10.1 Introduction
Economists often resort to game theory to model relationships between rms,
individuals and the government. The game theoretic approach is indispens-
able in situations where decision makers have dierent interests or dierent
policy tools are available at their disposal. Public nance theory provides a
wealth of opportunities to develop models characterized by strategic inter-
actions. Whenever the government makes decisions about tax policy, it has
to take into account the response of the taxpayers in terms of changes in
their behaviour. It is well-known that raising tax rates does not necessarily
lead to an increase in tax revenue. Beyond a certain level of taxation the
revenue in fact falls due to the narrowing of the tax base. Furthermore, if
the objective is not simply raising as much tax revenue as possible but rather
it is the maximization of social welfare, determining the optimal tax policy
becomes more complicated. Early contributions to the theory of taxation
provided arguments in favor of progressive tax schedules under the assump-
tion that all individuals shared the same utility and diered only in the levels
of income. Later on, economists broadened the class of social welfare func-
tions and began paying more attention to eciency considerations. There
are numerous examples of the distortionary eects of taxation. For instance,
the introduction of a tax on capital reduces the amount of savings and thus
decreases future income. Similarly, an individual would normally choose to
work less in the presence of tax on labor income (especially if the schedule is
progressive), as compared to the situation of no tax. The distortions associ-
ated with taxes have important welfare implications and this motivates the
strong interest in public nance theory.
In dynamic models of taxation the problem of choosing the optimal tax
structure is even harder. In addition to the eects arising from redistribution
of income across individuals, one has to take into account the redistribution
between dierent moments of time. A key question is how to tax income from
dierent sources, i.e. whether it is more protable from welfare perspective
to levy higher taxes on income from capital as compared to wage income or
vice versa. A somewhat surprising answer to this question was given by Judd
[11] and Chamley [5]. They independently arrived at the conclusion that the
optimal tax on capital income approaches zero as time goes to innity. In
these two papers, as well as in most of the other contributions, the opti-
112
mal taxation problem is stated formally as a Stackelberg game. Stackelberg
games are characterized by hierarchical structure in the sense that one of the
players (the leader) has an advantage over the other player (the follower).
In optimal taxation models the role of the leader is naturally assigned to
the government and rms and consumers are the followers. It is well known,
however, that open-loop Stackelberg equilibria are generally time inconsis-
tent [7] and without a reliable commitment mechanism they are not very
plausible. In contrast, feedback Stackelberg and Nash equilibria are always
time consistent [1], a desirable property of economic policy.
In this paper we propose a constructive approach to studying dieren-
tial games. A sucient condition for existence of stationary feedback Nash
equilibria is proved. As an application of the proposed approach, we have re-
considered some well-known taxation models in continuous time. We present
conditions ensuring the existence of strategies which constitute stationary
feedback Nash equilibria for each of these models. Unlike most of the litera-
ture, where the feedback equilibria are obtained as solutions to appropriate
Hamilton-Jacobi-Bellman equations, we derive our results from sucient con-
ditions for optimality based on Pontryagins maximum principle. For each
of the models considered we provide examples.
Acknowledgement. The authors would like to express their gratitude
to the anonymous referees for the valuable comments and suggestions which
helped considerably in improving the quality of the paper.
10.2 One-dimensional tax models
10.3 General result
We consider a two-persons dierential game on the interval [0, +). The for-
mulation of the game in a general form, including the assumptions pertaining
to the functions and the data involved are presented below. The evolution
of the state x : [0, ) R
n
is described by the dierential equation
x = f(x, u
1
, u
2
), x(0) = x
0
R
n
. (58)
Each player aims at maximizing a utility functional of the form
J
i
(u
1
, u
2
) =
_

0
e
t
L
i
(x, u
1
, u
2
)dt, i = 1, 2. (59)
113
The functions u
i
, i = 1, 2, are the admissible controls of Player 1 and
Player 2, respectively. For each player we introduce two sets of admissible
controls, i.e for each i = 1, 2, we dene the sets |
o
i
and |
f
i
. The rst
set is the set of so-called open-loop controls. Formally, |
o
i
consists of all
measurable functions u
i
dened on the interval [0, +) whose values belong
to the closed set U
i
R
m
i
. The second one is the set of feedback controls.
Formally, |
f
i
contains all locally Lipschitz functions u
i
: R
n
U
i
. The use of
feedback controls means that each of the players has information concerning
the current state of the system for all t [0, +).
The couple of feedback strategies (u
1
, u
2
) |
f
1
|
f
2
represents a sta-
tionary feedback Nash equilibrium solution for the game (58)(59) if each
of the feedback controls u
1
= u
1
(x) and u
2
= u
2
(x) provides a solution to
the optimal control problem of players 1 and 2, respectively. This means the
following: If we substitute u
1
and u
2
in (58) by u
1
(x) and u
2
(x), respectively,
then the corresponding solution x
of (58) and the functional

J
1
(u
1
, u
2
) =
_

0
e
t
L
1
(x
(t), u
1
(x
(t)), u
2
(x
(t)))dt.
are well dened on [0, +). Furthermore, the following inequality holds true
J
1
(u
1
, u
2
) J
1
(u
1
, u
2
),
where u
1
|
o
1
is an arbitrary admissible control for which the solution x of
the Cauchy problem
x(t) = f(x(t), u
1
(t), u
2
(x(t))), x(0) = x
0
R
n
as well as the criterion
J
1
(u
1
, u
2
) =
_

0
e
t
L
1
(x(t), u
1
(t), u
2
(x(t)))dt
are well-dened. Similarly, for Player 2 we require that J
2
(u
1
, u
2
) J
2
(u
1
, u
2
)
for all u
2
|
o
2
, such that the solution of the respective Cauchy problem and
the corresponding criterion are well dened.
Next, we consider a specic dierential game with scalar state variable x
and scalar controls u
i
, i = 1, 2. This game is important because it contains as
particular cases some of the dynamic taxation models studied further. The
dynamics of the system is determined as follows:
x = f(x)y(u
2
) u
1
(60)
114
u
1
0, x 0, x(0) = x
0
> 0. (61)
The performance criteria of Player 1 and Player 2 are given by the integrals
_

0
e
t
L
1
(u
1
)dt (62)
and _

0
e
t
[L
1
(u
1
) + L
2
(x, u
2
)]dt, (63)
respectively. The following assumptions are made regarding the data:
the functions L
1
(), L
2
(, ), f() and y() are suciently smooth and
y
() < 0, f(0) = 0, f
() > 0 and
L
2
u
2
(, ) 0. (64)
Let the couple of feedback strategies (u
1
, u
2
) be a stationary feedback
Nash equilibrium. The corresponding denition implies that u
1
|
f
1
is a
solution of the following optimal control problem solved by Player 1
_

0
e
t
L
1
(u
1
(t))dt max (65)
x(t) = f(x(t))y(u
2
(x(t))) u
1
(t) (66)
x(0) = x
0
> 0, (67)
x(t) 0, u
1
(t) 0 (68)
with respect to the control variable u
1
|
o
1
, and u
2
|
f
2
is a solution of the
following optimal control problem solved by Player 2
_

0
e
t
[L
1
(u
1
(x(t))) + L
2
(x(t), u
2
(t))]dt max (69)
with respect to u
2
|
o
2
and subject to
x(t) = f(x(t))y(u
2
(t)) u
1
(x(t)) (70)
x(0) = x
0
> 0 (71)
x(t) 0. (72)
To formulate a sucient condition for existence of a stationary feedback
Nash equilibrium, we assume that there exist continuously dierentiable func-
tions u
1
|
f
1
and u
2
|
f
2
that satisfy the following system of ordinary
dierential equations:
f
(x) u
1
(x) = f(x)( f(x)y
( u
2
(x)) u
2
(x)) (73)
115
(x)(f(x)y( u
2
(x)) u
1
(x)) = (x)( f
(x)y( u
2
(x)) + u
1
(x))
d
dx
L
1
( u
1
(x))
L
2
(x, u
2
(x))
x
(74)
for each x > 0, where
(x) :=
u
2
L
2
(x, u
2
(x))
y
( u
2
(x))f(x)
.
Dene the current value Hamiltonian functions for the optimal control
problems (65)(68) and (69)(72) as follows:
H
1
(x, u
1
, ) = L
1
(u
1
) + (f(x)y( u
2
(x)) u
1
)
H
2
(x, u
2
, ) = L
1
( u
1
(x)) + L
2
(x, u
2
) +(f(x)y(u
2
) u
1
(x))
and the maximized Hamiltonian functions as
H
1
(x, ) = max
u
1
0
H
1
(x, u
1
, )
H
2
(x, ) = max
u
2
H
2
(x, u
2
, ).
Now we are ready to formulate a sucient condition for existence of a
stationary Nash equilibrium. The proof is based on sucient conditions for
optimality as presented in [23].
Proposition 10.1 Assume that the following conditions hold true:
(C1) There exist feedback strategies u
i
, i = 1, 2, of the rst and of the second
player, respectively, such that the corresponding trajectory x() of (60)
is well dened on the interval [0, +), the performance criteria for
both players are also well dened and the following equality holds true
lim
t
e
t
( x(t)) x(t) = 0;
(C2) The functions u
1
and u
2
are continuously dierentiable and satisfy the
system of dierential equations (73) (74) for each x > 0;
(C3) The functions H
1
(, ) and H
2
(, ) are concave for any xed 0
and for any xed 0.
116
(C4)
lim
x+
(f(x) xf
(x))((f(x)y( u
2
(x)))
)
xf
(x)
< ;
(C5) there exists > 0 such that for each x > 0 the following equality holds
true
L
1
( u
1
(x))f(x) = ; (75)
Then the pair ( u
1
, u
2
) represents a stationary feedback Nash equilibrium
for the dierential game (60)(63).
Remark 10.2 Assumptions (C1)(C5) imply that the sucient condition
in [23] can be applied. This is the main idea of the proof. In the next sec-
tion we present examples showing that equalities (73) and (74) are sucient
to determine optimal feedback controls constituting stationary feedback Nash
equilibria for the considered dierential games. In fact, equation (73) and
the conditions (C4) and (C5) ( equation (74) and condition (C1)) are related
to the optimal control problem of Player 1 (Player 2). These equalities and
conditions imply the adjoint equation, the condition for maximization of the
Hamiltonian and the transversality condition for the optimal control prob-
lem of Player 1 (Player 2). Condition (C3) is important in the proof of the
sucient optimality condition (cf. [23]).
Proof. First, consider the solution of the optimal control problem of
Player 1. Let us substitute x(t), u
1
(t) and u
2
(x(t)) by x(t), u
1
( x(t)) and
u
2
( x(t)), respectively, in equation (66). The smoothness of its right-hand
side with respect to x implies that this equation has a unique solution for
each starting point. Since f(0) = 0 and f
(x) > 0, it follows from (73)

that u
1
(0) = 0, so x(t) 0 is the solution corresponding to the initial value
x(0) = 0. This fact and the uniqueness of solution imply that the solution
x(t) corresponding to u
1
and u
2
, and starting from x
0
> 0 remains positive
on the time interval, where it is dened.
According to the sucient optimality conditions for the optimal control
problem (65)-(68) (see [23]), the relations
1
:
= L
1
( u
1
) (76)
1
In some of the equations we omit the time argument for better readability.
117
= ( f
( x)y( u
2
( x)) f( x)y
( u
2
( x)) u
2
( x)) (77)
lim
t
e
t
(t) x(t) = 0. (78)
together with condition (C3) imply that the controltrajectory pair ( u
1
, x)
is a solution of this problem.
We set (t) := L
1
( u
1
( x(t))), t [0, +). Then (76) holds true. More-
over, by (C5) it follows that
(t)f( x(t)) = (79)
By taking the time derivative of (79) we obtain
f( x) + f
( x)
x = 0
=
f
( x)
f( x)
(f( x)y( u
2
( x)) u
1
( x))
=
f
( x)
f( x)
_
f( x)y( u
2
( x))
f( x)( f( x)y
( u
2
( x)) u
2
( x))
f
( x)
_
= (f
( x)y( u
2
( x)) +f( x)y
( u
2
( x)) u
2
( x))
The above equation is equivalent to (77).
If the solution x(t), t [0, +), is bounded, then the transversality
condition will be satised. Let us assume that lim
t+
x(t) = +. Inequality
(C4) implies the existence of a real number < and x > 0, such that
(f(x) xf
(x))((f(x)y( u
2
(x)))
)
xf
(x)
< for each x > x. (80)
We set z(t) = x(t)/f( x(t)) > 0, t [0, +). Then there exists T > 0
such that for each t T we have that x(t) > x. For these values of t we
calculate
z =
(f( x) xf
( x))
f( x)
2
x =
(f( x) xf
( x))((f( x)y( u
2
( x)))
)
f( x)f
( x)
z =
(f( x) xf
( x))((f( x)y( u
2
( x)))
)
xf
( x)
z
118
Therefore, according to (80), z(t) z(t) for each t T, i.e. z(t)
z(T)e
(tT)
for each t T. This ensures that the transversality condition
holds since < , and hence
0 lim
t
e
t
(t) x(t) = lim
t
e
t
x(t)
f( x(t))
z(T)e
T
lim
t
e
()t
= 0.
Next, we turn to the optimal control problem of Player 2. Analogously,
the following relations
=
u
2
L
2
( x, u
2
)
y
( u
2
)f( x)
(81)
=
d
dx
L
1
( u
1
( x))

x
L
2
( x, u
2
) (f
( x)y( u
2
) u
1
( x)) (82)
together with condition (C3) in the proposition statement and the transver-
sality condition lim
t
e
t
x(t)(t) = 0 imply that the controltrajectory pair
( u
2
, x) is a solution of the optimal control problem of Player 2.
Let us substitute in equation (70) x(t), u
1
(x(t)) and u
2
(t) by x(t), u
1
(x(t))
and u
2
(x(t)), respectively. We set
(t) = ( x(t)) (83)
and take the derivative with respect to t of (83):
(t) =
( x(t))
x(t) =
( x(t))(f( x(t))y( u
2
( x(t))) u
1
( x(t)))
Because of (74) and the denition of (x), the latter equation is equivalent
to the adjoint equation (82).
Since the trajectory x(t) > 0 for each t 0 (this follows from the unique-
ness of the solution and the initial condition x(0) = x
0
> 0), the assumptions
f(0) = 0 and f
() > 0 imply that f( x(t)) > 0 for each t 0. This and the
assumptions y
() < 0 and
L
2
u
2
(, ) 0 imply that is non-negative. Thus,
the transversality condition is satised because of (C1).
Below we show how the above result can be applied to solve specic
dierential games arising from models of optimal taxation. Both general
solutions and examples are provided for each case.
119
10.4 Linear output tax
Xie [27] considered a simple output tax model to illustrate the fact that some-
times open-loop Stackelberg solutions may be time consistent. In particular,
he linked time consistency with a property of the game which he referred
to as controllability, i.e. the ability of the leader to inuence the co-state
variable in the followers problem through its control. We take the output tax
model and show how it can be solved explicitly with the aid of Proposition
1. First we consider the general case and next we solve the example provided
in [27].
The players in this game are the government and the representative con-
sumer. The consumer maximizes over an innite time interval a utility func-
tional, which depends on consumption c, subject to the capital accumulation
constraint. The instantaneous utility function U() is dierentiable and con-
cave. The production function F(k) which depends on capital k, has the
standard properties F(0) = 0, F
(k) > 0 and F
(k) < 0 for k > 0. The

positive number indicates how the consumer values present consumption
as opposed to the future one.
Formally, the restrictions on the dynamics are described as follows:
k(t) = F(k(t))(1 (t)) c(t)

k(t) 0, c(t) 0, k(0) = k
0
> 0,
where denotes the output tax chosen by the government. The consumer
and the government have utility functionals
_

0
e
t
U(c(t))dt and
_

0
e
t
[U(c(t)) + V (F(k(t))(t))]dt,
respectively. Here the utility function V (), which depends on the collected
tax revenue, is also assumed to be dierentiable and concave. Thus, in this
model social welfare depends both on the amount of collected taxes and on
the consumption of the agent through the functions U and V . Let (c
)
represent a stationary feedback Nash equilibrium of this game. Then c
is
the optimal control of the consumers problem
_

0
e
t
U(c(t))dt max (84)
k(t) = F(k(t))(1
(k(t))) c(t) (85)

120
k(t) 0, c(t) 0, k(0) = k
0
> 0, (86)
where
denotes the output tax which is the solution of the governments

problem of maximizing social welfare. The government solves
_

0
e
t
[U(c
(k(t))) + V (F(k(t))(t))]dt max (87)

with respect to and subject to
k(t) = F(k(t))(1 (t)) c
(k(t)) (88)
k(t) 0, k(0) = k
0
> 0. (89)
Proposition 10.3 Let assumptions (C1) and (C3)-(C5) in Proposition 1
hold true. If there exist continuously dierentiable functions c(k) and (k)
satisfying the system of ordinary dierential equations:
c(k) =
F(k)
F
(k)
( +F(k)
(k)) (90)
V
(F(k) (k))(F
(k) (k) +F(k)
(k))(F(k)(1 (k)) c(k)) =

V
(F(k) (k))( F
(k) + c
(k)) U
( c(k)) c
(k),
(91)
then the couple ( c(k), (k)) represents a stationary feedback Nash equilibrium
for the game (84)-(89).
Proof.
The proof follows directly from Proposition 1 with L
1
(c) = U(c), L
2
(k, ) =
V (F(k)), = 1, u
1
= c, u
2
= and y() = 1. Note that for this problem
equations (90) and (91) are equivalent to equations (73) and (74), respectively
since
(k) = V
(F(k) (k))F(k)(F(k))
1
= V
(F(k) (k)).
Example 1. This example is studied in [27]. Let F(k) = Ak, U(c) =

ln(c), V (Ak) = ln(Ak). We shall show that the controls c(k) = k and
121
(k) =

2A
and the associated trajectory

k(t) = k
0
e
(A3/2)t
solve the game
for = A/. It is straightforward to see that conditions (C3) and (C5) are
satised. We need to verify equations (90) and (91) and to ensure that (C1)
holds true. Equation (90) is obvious. To verify the second equation, call LHS
and RHS the left-hand side and the right-hand side of (91), respectively. We
have
LHS =
4
2
k
2
_
2
_
_
Ak
(2A )
2A
k
_
=
3 2A
k
RHS =
2
k
( A +)
1
k
=
3 2A
k
,
so they are equal. Finally, the transversality condition in the optimal control
problem of the government is also satised for this example since V
(A
k (
k))
k =
2
= const.
With the above optimal controls the dierential equation for capital be-
comes
k(t) = A
k(t)
3
2
k(t),
from where

k(t) = k
0
e
(A3/2)t
. The above solution coincides with that in
[27].
It is interesting to note that Xie in [27] obtained this result using a dier-
ent equilibrium concept. As already mentioned, he was concerned with the
time inconsistency property of open-loop Stackelberg solutions and used this
example as an illustration that sometimes these solution are time-consistent.
In a recent paper Rubio [?] derived conditions leading to equivalence of the
feedback Nash and feedback Stackelberg equilibria under the implicit as-
sumption that the value function is C
1
. Rubio introduced the notion of
orthogonal reaction functions, referring to the property that the mixed
partial derivatives of the utility functions and the right-hand side of the dy-
namics equation with respect to the controls of the two players are zero. He
proved a proposition that for the class of dierential games with orthogonal
reaction functions, if the stationary feedback Nash equilibrium exists, it co-
incides with the stationary feedback Stackelberg equilibrium, independently
of which player acts as a leader. It is easy to see that for the above example
122
Rubios conditions are met and in fact we have a coincidence of open-loop,
feedback Stackelberg and feedback Nash equilibria.
10.5 Redistributive capital income taxation
The model considered in this section was originally proposed by Judd [11]. It
studies the question of how income from capital should be taxed. There are
three types of agents in the model economy capitalists, workers and a gov-
ernment. In the simple setting considered here, capitalists do not work and
receive income only from capital ownership. Capital is used for the produc-
tion of a single good which can be consumed or saved and invested. Workers
supply labor inelastically for which they receive wages and in addition, the
government grants them a lump-sum transfer or subtracts a lump-sum tax
from their labor income. The production technology is described by the
function F(k) with the standard properties that F(0) = 0, F
(k) > 0 and

F
(k) < 0 for positive k. Capital depreciates at a constant rate > 0 and
F(k) k is output net of depreciation. Further, capital is assumed to be
bounded. This assumption is also made in [11] and it seems to hold true for
reasonable specications of the production and utility functions. The static
prot maximization conditions for the competitive rm are:
r = F
(k)
w = F(k) kF
(k),
where r denotes the interest rate and w is the wage paid to the worker. Since
in this setup workers do not save, their consumption equals income. The
workers income is composed of wage earnings plus a government transfer
T (or minus a lump-sum tax, depending on the sign of T) which is determined
as the tax rate times the capital income:
(k, ) = w +T = F(k) kF
(k) + k(F
(k) ).
The problem that the representative capitalist faces is to maximize his
lifetime utility subject to the capital accumulation constraint. The utility
function U(c) is assumed to be increasing in consumption c and concave.
The problem of the government is to nd a tax rule that maximizes wel-
fare which is represented by the weighted sum ( being the weight) of the
instantaneous utilities of the capitalist U(c) and the worker V (). The utility
function V () is also concave.
123
k(t) = (1 (t))f(k(t)) c(t)

k(t) 0, c(t) 0, k(0) = k
0
> 0,
where f(k) := (F
(k) )k is the net of depreciation capital income. The

capitalist and the government have the following utility functionals
_

0
e
t
U(c(t))dt and
_

0
e
t
V ((k(t), (t))) + U(c(t)) dt,
respectively. Let the couple of strategies (c
) represent a stationary feed-

back Nash equilibrium of the above game. This means that c
is the optimal
control of the capitalists problem:
_

0
e
t
U(c(t))dt max (92)
k(t) = (1
(k(t)))f(k(t)) c(t) (93)

k(t) 0, c(t) 0, k(0) = k
0
> 0. (94)
Also,
is the optimal control of the governments problem:

_

0
e
t
[V ((k(t), (t))) + U(c
(k(t)))]dt max (95)
k(t) = (1 (t))f(k(t)) c
(k(t)) (96)
k(t) 0, k(0) = k
0
> 0. (97)
Judd approached the capital income taxation problem as an open-loop
Stackelberg game and arrived at the interesting result that the optimal tax
tends to zero as time goes to innity. Although this nding has become
widely accepted in public economics, the validity of the zero limiting capital
income tax has been questioned in dierent contexts. For example, Kemp et
al. [13] claim that in the original open-loop formulation of Judds model the
optimal tax rate on capital income may not converge to a stable equilibrium
and when a feedback Stackelberg solution is considered, the equilibrium tax
rate could be positive or negative.
Lansing [?] provides a counter-example to the redistributive tax model
examined by Judd, and argues that with logarithmic utility function of cap-
italists the optimal tax is non-zero. However, he claims that the obtained
124
result is specic to this functional form and will not be true in the general
case.
Below we show that under rather general assumptions about the utility
and production functions the optimal tax is dierent from zero when the
solution is sought in the class of feedback Nash equilibria.
Proposition 10.4 Let assumptions (C1) and (C3)-(C5) in Proposition 1
hold true. If there exist continuously dierentiable functions c(k) and (k)
satisfying the system of ordinary dierential equations:
c(k) =
f(k)
f
(k)
( +f(k)
(k)) (98)
V
((k, (k)))
d
dk
(k, (k))((1 (k))f(k) c(k)) =
= V
((k, (k)))[ + F
(k) + c
(k)] U
( c(k)) c
(k),
(99)
the couple ( c(k), (k)) represents a stationary feedback Nash equilibrium
for the game (92)-(97).
Proof.
The proof follows from Proposition 1 with L
1
(c) = U(c), L
2
(k, ) =
V ((k, )), (k, ) = F(k) kF
(k) +f(k) and y() = 1 . It is straight-

forward to see that equation (98) is equivalent to (73). Equation (99) in turn
is equivalent to (74). To show this, note that
(k, ) = f(k), so we have

that
(k) = V
((k, (k)))f(k)(f(k))
1
= V
((k, (k))).
Therefore,
(k) = V
((k, (k)))
d
dk
(k, (k))
Also, note that
V
((k, (k)))
k
(k, (k)) = V
((k, (k)))(f
(k) kF
(k))
so that the right-hand side of (74) becomes
V
((k, ))( f
(k)(1 (k)) + c
(k)) U
( c(k)) c
(k)
V
((k, ))( f
(k) kF
(k))
125
= V
((k, ))( f
(k) + c
(k) + kF
(k)) U
( c(k)) c
(k)
= V
((k, ))( + F
(k) + c
(k)) U
( c(k)) c
(k).
Example 2. Let U(c) =

c
1
1
, F(k) =
k
+ k and V () =

1
1
,
(0, 1). Then, f(k) = k
. Assume further that

0
= ( 1)/. We
shall show that the feedback Nash equilibrium for this example is the pair of
functions
c(k) = k
(k) =

1
k
1
+
1
,
where =
1/
. Let us rst check equations (98) and (99). Equation (98)
for this example is equivalent to
k =
k
k
1
[ +k
( )k
] = k,
and it holds true.
To verify (99), rst note that the workers income is the following linear
function of k:
=

1
k.
In order to have positive income it is necessary that > . By dividing
both sides by V
() > 0 and using the fact that

V
()
V
()
= , condition
(99) simplies to
k
_
k

1
k k
_
= k
1
+
k
_

1
k
_
( )
1
= +

1
_

1
_
(2 )(1 )
1
=
(1 )
126
and nally,
(1 )
1
_
1 2 +

_
=
_

. (100)
The latter gives a relationship between the parameters of the model. This
equation would allow us to determine the constant (and hence ) and to
pin down c(k) and (k).
An important question is when equation (100) has a unique solution. Let
G() = (1 )
1
_
1 2 +

.
If we calculate the derivative G
() we see that
G
() = (1 )
1

2

_

_
1
1
2
< 0,
hence, G() is strictly decreasing and if a solution exists it is unique. Re-
garding existence, since (/, +) and
lim
/
G() = (1 )
> 0
lim
+
G() = (1 )
1
(1 2)
,
a sucient condition for existence of solution will be > 1/2. Clearly,
the corresponding solution belongs to the interval (/, +), i.e.
>

> because of the inclusion

_
1
2
, 1
_
. (101)
Note that a solution of the algebraic equation (100) can also exist for
< 1/2 for suciently small positive values of .
Condition (C5) of Proposition 1 is straightforward to check. Next we
verify the concavity of the maximized Hamiltonians and the transversality
condition.
The Hamiltonian functions for the optimal control problems of the con-
sumer and the government are
H
1
(c, k, ) =
c
1
1
+
_
k

1
k c
_
127
and
H
2
(, k, ) =

1
1
+
(k)
1
1
+((1 )k
k),
respectively. For each xed > 0 and > 0 the maximized Hamiltonians
are
H
1
(k, ) =

1
1
+
_
k

1
k
_
and
H
2
(k, ) =

1
1
+
(k)
1
1
+
k
k,
respectively, and they are concave for (0, 1).
The dynamics of capital corresponding to the optimal controls obeys the
dierential equation
k(t) =
k(t)
k(t).
It can be solved explicitly and we nd that
k(t)
1
=
1 +e
()t
(( )k
1
0
1 + )
( )
.
According to (101), < and from the solution of the dierential equa-
tion for capital it follows that capital is bounded. More specically,
lim
t
k(t)
1
=
1
( )
The fact that

k is bounded ensures that the transversality conditions for
the optimal control problems of the consumer and the government hold true.
Indeed, for the optimal control problem of the consumer the transversality
condition lim
t
e
t
k(t)(t) = 0 is equivalent to lim

t
e
t

k(t)
f(
k(t))
= 0 and it is
satised whenever

k is bounded. Similarly, for the optimal control problem
of the government the transversality condition holds true as well.
It is interesting to explore the behaviour of the capital income tax. Dene
the limiting capital income tax
as the limit of (
k(t)) as t . We
obtain
128

=
( 2) +
( )
.
It can be shown that for dierent values of the parameters of the model in
this example we may have positive or negative limiting capital income taxes.
Negative taxes typically occur when capital is declining over time, so at some
point it becomes optimal from the welfare perspective to grant a subsidy to
capitalists in order to sustain a certain level of production.
We note that for Judds model, as well as for the output tax model,
the condition for orthogonality of the reaction functions in [?] is satised
and therefore, the feedback Nash equilibrium coincides with the feedback
Stackelberg equilibrium. Generally, the computation of feedback Stackel-
berg equilibria is associated with considerable diculties (see [8]), so the
framework developed above for Nash equilibria can be used to obtain also
the Stackelberg equilibria whenever both coincide.
10.6 A model with government bonds
The model considered in this section is proposed by Chamley [5]. In an
inuential article he examined a dynamic taxation problem and showed that
if the utility function is separable in consumption and labour, the optimal tax
on capital income is zero in the long run. This is essentially the same result as
the one obtained by Judd, despite the dierences in the model formulation.
Frankel [9] studied the same problem and reconrmed Chamleys ndings
in the case of separable utility functions. For more general utility functions he
claimed that the capital income tax may be positive or negative depending
on whether the sum of the elasticities of marginal utility with respect to
consumption and labour is increasing or decreasing over time.
The model presented below follows the model studied in [9] with two
dierences. The rst one is our assumption that the government expenditure
g depends on capital k, while in [9] it is assumed that g depends on time.
Still, we retain the assumption that g is given exogenously and hence it is
not a control for the government. The second dierence is condition (105)
below. In [9] the left-hand side of (105) is greater than or equal to zero. But
as it is shown in [15], the existence of solution of the optimal control problem
for the consumer implies that this condition is satised as equality. For that
reason we modify the condition (105) in the form of the usual transversality
condition.
129
Unlike the models discussed in the previous section, now the consumer
chooses not only the consumption path but also how many hours to work (the
labour input l). Also, in the optimal control problem of the representative
agent the state variable is private assets a. Private assets consist of capital k
and government bonds b, that is a = k +b. In our notation r and w stand for
after tax capital and labour income, respectively. We consider the following
dierential game. On the one hand, the consumer maximizes his utility
function under given taxes on the wage and on the income from capital. On
the other hand, the government decides how to tax the wage and the income
from capital in an optimal way so that to achieve the highest utility for the
consumer and at the same time to be able to nance its expenditure g.
a(t) = r(t)a(t) + w(t)l(t) c(t)
k(t) = F(k(t), l(t)) c(t) g(k(t))

c(t) 0, l(t) 0, a(0) = a
0
,
k(t) 0, k(0) = k
0
,
lim
t
e
t
a(t) = 0.
The meaning of the last equality is evident: the present value of the private
assets tends to zero as t tends to innity.
The consumer and the government have the same utility functional of the
following form
_

0
e
t
U(c(t), l(t)dt.
Let ((l
, c
), (r
, w
)) represent a stationary feedback Nash equilibrium

of the above written game under the assumption that all controls depend
only on the state variable a. This assumption can be partly motivated by
the fact that a is the only state variable in the optimal control problem
of the consumer. Moreover, typically the consumer observes only his own
private assets and hence, the decision on his consumption and labour supply
depend on them. Essentially, this assumption is technical. It is important
for obtaining specic solutions of the Chamleys problem.
The aforementioned assumption and the denition of stationary feedback
Nash equilibrium imply that (l
, c
) and (r
, w
) are solutions of the opti-

mal control problem of the consumer and of the government, respectively.
130
Formally, the consumer solves with respect to c and l the following optimal
control problem:
_

0
e
t
U(c(t), l(t))dt max (102)
a(t) = r
(a(t))a(t) + w
(a(t))l(t) c(t) (103)

c(t) 0, l(t) 0, a(0) = a
0
(104)
lim
t
e
t
a(t) = 0 (105)
The government chooses r and w so that to solve:
_

0
e
t
U(c
(a(t)), l
(a(t)))dt max (106)

subject to
a(t) = r(t)a(t) + w(t)l
(a(t)) c
(a(t)) (107)
k(t) = F(k(t), l
(a(t))) c
(a(t)) g(k(t)) (108)

k(t) 0, k(0) = k
0
(109)
a(0) = a
0
. (110)
Note that besides the private assets equation, the optimization problem
of the government takes into account the capital accumulation dynamics. As
usual, F(k, l) denotes the production technology. We shall assume further
that the utility function U(, ) is dierentiable with U
c
(c, l) > 0 and U
l
(c, l) <
0 for each l > 0 and each c > 0.
The existing results in the economic literature are based on analysis of
the open-loop Stackelberg solution of the game in which the optimal controls
are functions only of time. An interesting question is whether the claim for
zero optimal tax on capital is valid when feedback controls are considered.
It turns out that in this case, even if the utility function is separable, the
optimal tax can be dierent from zero.
To nd a stationary feedback Nash equilibrium of Chamleys optimal
taxation problem, we can make use of the approach employed in the solution
of the one-dimensional model. To do this we assume that the after-tax wage
is proportionate to the after-tax income from capital. Before formulating the
131
corresponding sucient condition, we assume the existence of continuously
dierentiable functions r and

l determined by the following relations:
U
c
( c(a),
l(a))f(a) = , (111)
U
l
( c(a),
l(a)) = (112)
for some positive constants and , where
c(a) :=
f(a)
f
(a)
> 0 for each a ,= 0 and f(a) := a r(a). (113)
We set w(a) := f(a) and dene the Hamiltonian functions for the optimal
control problems of the consumer and the government as follows:
H
c
(a, c, l, ) = U(c, l) + (a r(a) + w(a)l c)
H
g
(a, k, r, w, , ) = U( c(a),
l(a)) + (ar +w
l(a) c(a))+
+(F(k,
l(a)) c(a) g(k)).

The maximized Hamiltonians are dened as:
H
c
(a, ) = max
c>0, l0
H
c
(a, c, l, )
H
g
(a, k, , ) = max
r, w>0
H
g
(a, k, r, w, , ).
Now we are in a position to formulate a sucient condition for existence
of a stationary Nash equilibrium:
Proposition 10.5 Assume that the following conditions hold true:
(D1) There exist continuously dierentiable functions r and

l satisfying re-
lations (111) and (112) for each a and for some positive constants
and . Moreover, we set
w(a) := f(a); (114)
(D2) the function H
c
(a, ) is concave with respect to a for all nonnegative

; the function H
g
(a, k, , ) is concave with respect to a and k for
= 0 and for all nonnegative ;
132
(D3) We set
(a, k) :=
(f(a)
(a) c
(a))
f(a)(F
l
(k,
l(a))
(a) c
(a))
and assume that for any a and k 0 the following relations hold true:
(a, k) 0 and
(a, k)( F
k
(k,
l(a)) + g
(k)) =
a
(a, k)(f(a) + w(a)
l(a) c(a))+
+
k
(a, k)(F(k,
l(a)) c(a) g(k)).

(115)
Then the stationary feedback Nash equilibrium of the dierential game
(102)(110) is given by the pairs ( c(a),
l(a)) and ( r(a), w(a)) whenever the

corresponding solution ( a(),
k()) of the ODE system (107)(108) are dened

and bounded on the interval [0, +).
Proof. The proof follows the idea of Proposition 1. According to the
sucient optimality conditions the relations
= U
c
( c( a),
l( a)) (116)
w = U
l
( c( a),
l( a)) (117)
(t) = (t)( f
( a) w
( a)
l( a)) (118)
together with concavity of the maximized Hamiltonian H
c
and the transver-
sality condition imply that ( c,
l, a) is an optimal control-trajectory pair of the

optimal control problem of the consumer.
We set := U
c
( c( a),
l( a)). Thus condition (116) is satised. Next, con-

ditions (111), (112), (114) and (116) imply (117). From (111) and (116) it
follows that for each t [0, +) we have that (t)f( a(t)) = . Take the
time derivative of both sides to obtain
(t) f( a(t)) + (t) f
( a(t))

a(t) = 0
Substituting

a(t) with the right-hand side of (103) for the optimal controls,
yields that the above equation is equivalent to (118). This is so because
f
(a) w(a) f(a) w
(a) = 0 which in turn follows from (114).

The transversality condition holds true because of the equality (105) and
because () and a() are bounded.
133
We now turn to the optimal control problem of the government. Accord-
ing to the sucient optimality conditions the relations
a = 0

l( a) = 0
= U
c
c
( a) U
( a) (r +w
( a) c
( a)) (F
( a) c
( a))
= ( F
k
(
k,
l) + g
k)),
together with concavity of the maximized Hamiltonian function and the ap-
propriate transversality condition imply that ( , w, a,
k) is an optimal control-
trajectory pair of the optimal control problem (106)(110).
We set (t) := 0 and (t) := ( a(t),
k(t)). Then assumption (D3) implies

that (t) 0. Clearly, the rst two sucient conditions are satised. The
co-state equation for becomes
U
c
c
( a) U
( a) = (F
( a) c
( a)).
Using (111), (112), (114) and the sucient conditions for the consumers
problem, it is straightforward to check that this equation holds true for
dened as above. The co-state equation for is satised as well because of
(115):
=
a
( a,
k)
a +
k
( a,
k)
k
=
a
( a,
k)(f( a) + w( a)
l( a) c( a)) +
k
( a,
k)(F(
k,
l( a)) c( a) g(
k))
= ( a,
k)( F
k
(
k,
l( a)) + g
k)) = ( F
k
(
k,
l( a)) + g
k)).
The boundedness of a and

k implies that is also bounded, hence the
transversality condition is veried. Notice that the boundedness of a and k
actually prevents the government debt from becoming innite. Otherwise the
consumer will refuse to hold government bonds knowing that the government
will not pay back.
Remark 10.6 Note that for separable utility functions

l(a) =

l, a constant.
This implies that =

f(a)
= .
134
Below we present an example which shows that for separable utility func-
tions it is possible to obtain taxes dierent from zero if the dependence of
the functions r and w on a is explicitly taken into account.
Example 3. Assuming that the state variables a and k take only positive val-
ues, we are given the following functions and initial data. U(c, l) =
c
1
1
l
2
2
,
where (0, 1). F(k, l) = 2
_
l
1
+ l, a(0) = k(0) = k
0
> 0,
l (0, 1] and g(k) = . We shall show that the controls c(a) =

a,

l = 1,
r(a) =
_
a
1
, w(a) =
_
and the corresponding trajectories

a(t)
1
=

k(t)
1
= 2
_
_
1
_
2
_
_
1
k
1
0
_
e
(1)t
solve the dierential game with = = 1.
Condition (111) for this example is equivalent to ( c(a))
f(a) = 1. Hence,
substituting c(a) we obtain:
_
f(a)
f
(a)
_
f(a) = 1
f(a)
(1)/
= f
(a).
The latter is a dierential equation which we can solve to nd the function
f(a) using the initial condition f(0) = 0. We recall that by denition f(a) =
ar(a). The solution given that initial condition is
f(a) =
_
hence,
r(a) =
_
a
1
.
The computation of w(a) and

l is straightforward. Since the utility func-
tion is separable and the optimal labour input is constant (Remark 10.6),
the function (a, k) depends only on a and (a, k) =
1
f(a)
. Thus, condition
(115) simplies to:
135
F
k
(k,
l)
f(a)
=
f
(a)
f(a)
2
(f(a) + w(a) c(a)).
Using that w(a) = f(a) and the solution for c(a) we obtain:
F
k
(k,
l) = 2f
(a) + .
Thus,
F
k
(k,
l) = 2f
(a) = 2
_
a
1
.
Given that a(t) =

k(t) the equality is satised.
The maximized Hamiltonian function for the optimal control problem of
the consumer is
H
c
(a, ) =

(1)/
1

(
_
)
2
2
+
_
_
+
_
_
2
a
2

1/
_
and it is concave with respect to a for each positive whenever (0, 1/2).
The transversality condition holds because of (105) and the equality
lim
t
e
t
a(t)(t) = lim
t
e
t
_
a(t)
1
= 0
for the function a(t)
1
dened above.
The maximized Hamiltonian for the optimal control problem of the gov-
ernment is
H
g
(a, k, ) =
_
a
_
1
1

1
2
+
_
2
_
+
a
g
_
and it is also concave with respect to a and k for each positive . Given
that a(t) =

k(t) and (t) = (t) = 1/f(a(t)) > 0, the transversality condition
lim
t
e
t
(t)
k(t) = 0 holds true.

It turns out that in the feedback formulation of the optimal taxation
problem the taxes are generally dierent from zero even when the utility
is separable. Note that in this example the government expenditure equals
gross income minus net income (income after taxes) which is exactly the
amount of collected tax revenue. This means that, starting with zero debt,
136
the government does not have an incentive to borrow or accumulate assets
and thus to redistribute income over time. This result may have something
to do with the fact that the utility of the government coincides with that
of the agent, i.e. the government does not derive utility from the collected
taxes or the expenditure it makes. Since it cares only about the welfare of
the consumer, it collects in the form of taxes exactly the amount which is
needed to cover its current outlays.
Finally, we may examine the evolution of the capital income tax for this
particular problem. Before that, it is convenient to state explicitly the before-
tax capital income r(k) using the standard equilibrium condition that r(k) =
F
k
(k, l). The gross return on capital is r(k) = 2
_
k
1
and the net
return is r(a) =
_
k
1
= (1
k
) r(k). Thus, we obtain that
k
=
2 1
2
.
Interestingly, in this example the optimal tax depends only on the in-
tertemporal elasticity of substitution of consumption. It will be negative for
< 1/2.
10.7 Conclusion
In this paper we have reconsidered some well-known optimal taxation models
which previously have been dealt with in the framework of open-loop Stack-
elberg equilibria. The results we obtain by calculating the feedback Nash
equilibria of the tax models are quite dierent from those prevailing in the
economic literature the optimal tax on capital income is not zero in the
long run. Furthermore, for the one-dimensional models the feedback Nash
solution coincides with the feedback Stackelberg one. This means that even
if we stick to the original hierarchical structure of the game, the non-zero tax
result we obtain remains valid. One direction for future work would be to
relax some of the assumptions about the structure of the dierential games
studied here, so that more general specications can be handled.
137
References
[1] Basar, T., Olsder, G.J.: Dynamic Noncooperative Game Theory, SIAM,
1999.
[2] L. D. Berkovitz, Optimal Control Theory, Springer Verlag, 1974.
[3] D. Carlson, A. Haurie, A. Leizarowitz, Innite Orizon Optimal Control:
Deterministic and Stochastic Systems, Springer Verlag, 1991.
[4] L. Cesari, Optimization - Theory and Applications, Springer Verlag,
1983.
[5] Chamley, C.: Optimal Taxation of Capital Income in General Equilib-
rium with Innite Lives, Econometrica 54 (3) (1986), 607-622.
[6] F. Clarke, Optimization and nonsmooth analysis, Canadian Mathe-
matical Society Series of Monographs and Advanced Texts. A Wiley-
Interscience Publication. New York: John Wiley & Sons, Inc. XIII, 308
p., 1983.
[7] Cohen, D., Michel, P.: How Should Control Theory Be Used to Calculate
a Time-Consistent Government Policy? Review of Economic Studies 55
(2) (1988), 263-274.
[8] Dockner, E., Jorgensen, S., Van Long, N., Sorger, G.: Dierential Games
in Economics and Management Science, Cambridge University Press,
2000.
[9] Frankel, D.: Transitional Dynamics of Optimal Capital Taxation,
Macroeconomic Dynamics 2 (1998), 492-503.
[10] M. Hestenes, Calculus of Variations and Optimal Control, Wiley, New
York, 1966.
[11] Judd, D. Redistributive Taxation in a Simple Perfect Foresight Model,
Journal of Public Economics 28 (1985), 59-83
[12] Karp, L., Ho Lee,In : Time-Consistent Policies Journal of Economic
Theory 112 (2003), Issue 2, 353-364.
138
[13] Kemp, L., van Long, N., Shimomura, K. : Cyclical and Noncyclical
Redistributive Taxation International Economic Review 34 (2) (1993),
415-429.
[14] KR M. I. Krastanov, R. Rozenov, On Optimal Redistributive Capital
Income Taxation, in: Large-Scale Scientic Computing, Lirkov, Ivan;
Margenov, Svetozar; Wasniewski, Jerzy (Eds.) Springer Lecture Notes
in Computer Science 4818 (2008), 342349.
[15] Krastanov, M., Rozenov, R.: On Chamleys Problem of Optimal Taxa-
tion, Proceedings of the 37th Spring Conference of the Union of Bulgar-
ian Mathematicians, (2008), 216220.
[16] E. B. Lee, L. Larkus, Foundations of Optimal Control Theory, Wiley,
New York, 1967.
[17] Ph. Michel, On the transversality conditions in innite horizon optimal
problems, Econometrica 50, No. 4, 1982.
[18] L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F.
Mischenko, Mathematische Theorie der optimalen Prozesse, Moskau:
Staatsverlag fur physikalisch-mathematische Literatur, 391 S. (1961)
(in Russian).
[19] L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mis-
chenko, The mathematical theory of optimal processes, Wiley, New York,
1962.
[20] B. Pschenichnii, Convex analysis and extremal problems, Nauka,
Moscow, 1980 (in Russian).
[21] R. T. Rockafellar, Convex analysis, New-Jersey, Princeton University
Press, XVIII, 451 p. (1970).
[22] R. Rozenov, M. I. Krastanov, Dierential Games and Optimal Tax Pol-
icy, Lecture Notes in Computer Science 4310, 353-360, 2007, Springer.
[23] Seierstad, A., Sydsaeter, K.: Sucient Conditions in Optimal Control
Theory International Economic Review 18 (1977), No.2, 367-391.
[24] E. D. Sontag, Mathematical Control Theory, Deterministic Finite Di-
mensional Systems, Second Edition, Springer Verlag, 1998.
139
[25] H. Sussmann, Handouts for the course taught at the Weizmann
Institute in the fall term of the year 2000,
http://www.math.rutgers.edu/ sussmann/weizmann-course-2000.html.
[26] H. Sussmann, New theories of set-valued dierentials and new versions
of the maximum principle of optimal control, in: Nonlinear Control in
the Year 2000, A. Isidori, F. Lamnabhi-Lagarrigue and W. Respondek
Eds., Springer-Verlag, 487526, 2000.
[27] Xie, D.: On Time Consistency: A Technical Issue in Stackelberg Dier-
ential Games Journal of Economic Theory 76 (1997), Issue 2, 412-430.
140
Contents
1 Introduction 2
1.1 A classical version for xed time problems . . . . . . . . . . . 4
1.2 A classical version for variable time interval problems . . . . . 7
2 Preliminaries 15
2.1 Some properties of the measurable maps . . . . . . . . . . . . 15
2.2 Norms of matrices . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Some properties of the initial-value problems . . . . . . . . . . 18
2.4 Linear dierential equations . . . . . . . . . . . . . . . . . . . 22
3 The transversality conditions 26
3.1 Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Boltyanskii tangent cone to a set at a point . . . . . . . . . . 29
3.3 The maximum principle with transversality conditions for xed
time interval problems . . . . . . . . . . . . . . . . . . . . . . 35
3.4 The maximum principle with transversality conditions for vari-
able time interval problems . . . . . . . . . . . . . . . . . . . . 38
3.5 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4 Separation of sets 42
4.1 The maximum principle as a set separation condition for xed
4.2 The maximum principle with transversality conditions for vari-
able time interval problems . . . . . . . . . . . . . . . . . . . . 47
4.3 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5 The main ideas standing behind the maximum principle 52
5.1 The maximum principle as a set separation condition for xed
5.2 The variational equation. . . . . . . . . . . . . . . . . . . . . . 55
5.3 Packets of needle variations. . . . . . . . . . . . . . . . . . . 55
5.4 The endpoint map. . . . . . . . . . . . . . . . . . . . . . . . . 56
5.5 The main technical lemma. . . . . . . . . . . . . . . . . . . . 57
5.6 The transversal intersection theorem. . . . . . . . . . . . . . . 57
5.7 Application of the transversal intersection theorem. . . . . . . 58
5.8 The compactness argument. . . . . . . . . . . . . . . . . . . . 59
141
5.9 The adjoint equation . . . . . . . . . . . . . . . . . . . . . . . 60
6 Boltyanskii cones and the proof of the maximum principle
in some cases 63
6.1 Set dened by a nite set of equalities . . . . . . . . . . . . . 63
6.2 Set dened by a nite set of equalities and inequalities . . . . 65
6.3 The maximum principle with transversality conditions for xed
time intervals problems . . . . . . . . . . . . . . . . . . . . . . 67
6.4 The maximum principle for a Mayer optimal problem . . . . . 71
7 Necessary conditions in innite horizon optimal problems 76
7.1 Reduction of an innite horizon optimal problem to a nite
horizon optimal problem . . . . . . . . . . . . . . . . . . . . . 76
7.2 Innite horizon optimal problems and the maximum principle 79
7.3 Innite horizon optimal problems and transversality condi-
tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
8 Sucient conditions in innite horizon optimal problems 86
8.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
8.2 Overtaking optimal trajectory-control pairs and transversality
conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8.3 Current Hamiltonian and sucient conditions for innite hori-
zon optimal problems . . . . . . . . . . . . . . . . . . . . . . . 92
9 Bellman function and the Hamilton-Jacobi-Bellman equa-
tion 99
9.1 Bellman function and its properties . . . . . . . . . . . . . . . 99
9.2 The Hamilton-Jacobi-Bellman equation . . . . . . . . . . . . . 102
9.3 A linear-quadratic optimal control problem . . . . . . . . . . . 107
10 On the optimal taxation problem 112
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
10.2 One-dimensional tax models . . . . . . . . . . . . . . . . . . . 113
10.3 General result . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
10.4 Linear output tax . . . . . . . . . . . . . . . . . . . . . . . . . 120
10.5 Redistributive capital income taxation . . . . . . . . . . . . . 123
10.6 A model with government bonds . . . . . . . . . . . . . . . . 129
10.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
142

Optimal Control

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Optimal Control

Încărcat de

Drepturi de autor:

Formate disponibile

Lecture Notes

()). One can checked

()) is a solution of our problem.

()) is such that

()) I(x(), u()) for all (x

()) satises C10. Dene the Hamiltonian H to be the function

(t)) satises the following system of equa-

(t), then we end

()) such that

()) I(x(), u()) for all (x(), u()) T (T

()) satisfy C9. Dene the Hamiltonian H to be the function

(t), a fact that would give

(t). Fortunately, the

], then for almost each t [0, T

] the optimal control values u

(t) have to satisfy (1), and (x

()) will be the corresponding trajectory

()) belongs to the

(t), a fact that would give us no information

(t). Fortunately, the conditions of Theorem 1.6 imply

(t). If b > 0, this

(t) = 1 for all t. If b < 0, it follows that u

(t) = 1 for all t.

(t) will be equal to 1 for t < and to 1 for > t

(t) will be equal to 1 for

is one of the following four types:

that takes values in the set of extreme points of U is said to be

and (0, 0) Graph f

, then it can be proved that will be the optimal value

and connecting the points

and connecting the points (x

we have denoted the conjugate transposed

X is any other solution of the same (IVP), where

X(t) = A(t) X(t), X(t

, we denote its scalar product

of a cone K in the real linear space X is dened as

is a convex closed cone in X

(the scalar product depends in a

such that for all x K

, x) > 0. But then x K

, x) + whenever +. But this

is a convex closed cone in X

, we can dene its dual cone K

. If K is a convex closed cone, the Lemma 3.1 implies that

, x) 0 for each element x

, x) 0. According to the Farkas lemma, we obtain that

. This completes the proof.

3.2 Boltyanskii tangent cone to a set at a point

0 as 0. Then for each suciently small |y| there exists

0 as 0, for each > 0 there exists > 0 such that

0 as 0. Moreover, the very denition of r() implies that

(y) such that

(y) (y) +|y|

(y) the following inequalities

(y)). According to the

(0) i. e. (x) S. Thus, we have shown that K

(0). This shows that (x) S, and hence, K

()) is such that

()) I(x(), u()) for all x(), u() T (T

()) satises C10, x is dened by C11 and C satises C11. Dene

()) such that

()) I(x(), u()) for all (x(), u()) T (T