Instructor:

Associate Professor\ Mohammed Alhanjouri

ISLAMIC UNIVERSITY OF GAZA
Faculty of Engineering
Computer Engineering Department

(Graduate Studies)
ECOM 6302: Engineering Optimization
Chapter Three:
Functions of a Several Variables
we examine fundamental concepts and useful methods for
finding minima of unconstrained functions of several
variables.
Functions of a Several Variables
we address the static question. That is, we examine
conditions that allow us (in the most general circumstances
possible) to characterize optima. We use these conditions
to test candidate points to see whether they are (or are not)
minima, maxima, saddlepoints, or none of the above.
in the absence of constraints on x, where x is a vector of design
variables of dimension N and is a scalar objective function.
We typically assume that x
i
(for all i = 1, 2, 3, . . . , N) can take
on any value,
In addition, we shall often find it convenient to assume that
and its derivatives exist and are continuous everywhere,
even though we know that optima may occur at points of
discontinuity in or its gradient,
as in Figure 3.1, a contour plot of Himmelblaus function:
3.1 OPTIMALITY CRITERIA

We examine optimality criteria for basically two reasons:
(1)because they are necessary to recognize solutions and
(2) because they provide motivation for most of the useful methods.

Consider the Taylor expansion of a function of several variables:
by definition a minimum is a point such that all other points in
a neighborhood produce a greater objective value. That is,
3.2 DIRECT-SEARCH METHODS

In this and the following sections of this chapter we consider
the dynamic question for functions of several variables. That
is, we examine methods or algorithms that iteratively produce
estimates of x*, that set of design variables that causes (x) to
take on its minimum value. The methods work equally well
for maximization by letting the objectives be (x).
These methods can be classified into three broad categories
based on the type of information that must be supplied by the
user:
1. Direct-search methods, which use only function values
2. Gradient methods, which require estimates of the first
derivative of (x)
3. Second-order methods, which require estimates of the first
and second derivatives of (x)
We will examine in particular detail three direct search
techniques:
The earliest attempts to solve unconstrained problems by direct
search produced methods modeled essentially on single-
variable methods
For example, in two dimensions, a square pattern such as in
Figure 3.4 is located. Then the best of the five points is
selected as the next base point around which to locate the
next pattern of points. If none of the corner points is better
than the base point, the scale of the grid is reduced and the
search continues.
In higher dimensions this corresponds to evaluating the
performance index at each vertex as well as the centroid of a
hypercube and is called a factorial design pattern. If the
number of variables, that is, the dimension of the search
space, is N, then a factorial search pattern will require 2
N
+ 1
function evaluations per pattern. The number of evaluations,
even for problems of modest dimension, rapidly becomes
overwhelming.
Rule 1 Minimum Straddled

If the selected worse vertex was generated in the previous
iteration, then choose instead the vertex with the next highest
function value.
Rule 2 Cycling

If a given vertex remains unchanged for more than M
iterations, reduce the size of the simplex by some factor. Set
up a new simplex with the currently lowest point as the base
point. Spendley et al. suggest that M be predicted via
Rule 3 Termination Criterion

The search is terminated when the simplex gets small enough
or else if the standard deviation of the function values at the
vertices gets small enough.
This rule requires the specification of a termination parameter.
The best overall choice of parameters was (o, |, ) = (2, 0.25, 2.5)
x
0
In the previous section we examined several methods that
require only objective function values to proceed toward the
solution. These direct methods are important, because very
often in a practical engineering problem this is the only reliable
information available. On the other hand, even the best direct
methods can require an excessive number of function
evaluations to locate the solution.
The methods will be iterative, since the elements of the
gradient will in general be nonlinear functions of the design
variables.
We shall assume throughout that (x), V(x), and V
2
(x) exist
and are continuous.
Newtons Method
It is a second order method.
Let x
(k)
be the current point.
The Taylor expansion of the objective function about
x
(k)
:

The quadratic approximation of f(x) is

We need to find the critical point of the approximation:
The Newton optimization method is



If the function f(x) is quadratic, the solution
can be found in exactly one step.
) ( ) (
) ( 1 ) ( 2 ) ( ) 1 ( k k k k
x f x f x x V V =
+
Modified Newtons Method
Newton method can be unreliable for non-quadratic functions.
The Newton step will often be large when x
(0)
is far from x
*
.
To solve this problem, we add a step length:
) ( ) (
) ( 1 ) ( 2 ) ( ) ( ) 1 ( k k k k k
x f x f x x V V =
+
Quasi-Newton Method
Quasi-Newton methods use a Hessian-like matrix
but without calculating second-order derivatives.
Sometimes, these methods are referred to as the
variable metric methods.
Take the general formula:


When A
(k)
= I (identity matrix), the formula becomes
the formula of the steepest descent method.
When A
(k)
= V
2
f(x
(k)
)
-1
, the formula becomes the
formula of the Newton method.
) (
) ( ) ( ) ( ) 1 ( k k k k
x f x x V =
+
A
Starting from a positive definite matrix, the quasi-
Newton methods gradually build up an approximate
Hessian matrix by using gradient information from the
previous iterations.
The matrix A is kept positive definite; hence the
direction
s
(k)
= - A
(k)
Vf(x
(k)
)
remains a descent direction.
There are several ways to update the matrix A, one of
which is


where A
(0)
= I, o
(k)
= x
(k+1)
x
(k)
and

(k)
= Vf(x
(k+1)
) Vf(x
(k)
).
) (
T
) (
T
) ( ) (
) ( ) (
T
) (
) (
T
) ( ) ( ) (
) ( ) 1 (




k k
k k
k k k
k k k k
k k
+ =
+
A
A A
A A
Example
Minimize using the quasi-Newton
method with x
(0)
= [1, 1]
T
. Stop when ||Vf(x
(k)
)|| 0.0001.

Solution
Iteration 1:
2
2
2
1
10 ) ( x x x f + =
(

= + =
(

= V =
(

= V
(

= =
20 1
2 1

20
2
20
2
1 0
0 1
(0) ) 0 ( ) 1 (
) 0 ( ) 0 ( (0)
) 0 (
) 0 (
s x x
f s
f
A
I A
(

=
=
0.008991 -
0.8991009
0.05044955
) 1 (
(0)
x
Iteration 2:
(

= V =
(

= + =
(

= =
(

=
(

= V V =
0.017998
1.79982 -
0.05005 0.005 -
0.005004 - 1.0004




1.00899 -
0.1009 -

20.1798 -
0.2018 -
20
2
0.17982 -
1.798202

) 1 ( ) 1 ( ) 1 (
) 0 (
T
) 0 (
T
) 0 ( ) 0 (
) 0 ( ) 0 (
T
) 0 (
) 0 (
T
) 0 ( ) 0 ( ) 0 (
) 0 ( ) 1 (
) 0 ( ) 1 ( ) 0 (
) 0 ( ) 1 ( (0)
f s
x x
f f
A
A
A A
A A
Stop
0 || ) ( ||
0
0
0.49955045
0.017998 0.008991 -
1.79982 - 0.8991009

) 2 (
) 2 (
(1)
) 1 ( ) 1 ( ) 2 (
= V
(

=
=
(

+
= + =
x f
x
s x x
Quasi-Newton Methods (DFP)

Quasi-Newton methods are based primarily upon
properties of quadratic functions and they are
designed to mimic Newton method using only first-
order information.
All Quasi-Newton methods generate directions using


A
(k)
is an NN matrix called the metric.
Methods employing this direction generation form are
called variable metric methods because A changes at
each iteration.
) ( ) (
) ( ) ( ) ( k k k
x f x s V = A
The recursion for the estimate to the inverse of the
Hessian is


The matrix A
c
(k)
is formed such that the sequence


approaches H
-1
= V
2
f(x
*
)
-1
.

) ( ) ( ) 1 ( k
c
k k
A A A + =
+
) 1 ( ) 1 ( ) 0 (
,..., ,
+ k
A A A
The DFP formula used to update the matrix A is




where A
(0)
= I, Ax
(k)
= x
(k+1)
x
(k)
and
Ag
(k)
= g(x
(k+1)
) - g(x
(k)
)

=Vf(x
(k+1)
) Vf(x
(k)
).

The DFP formula preserves symmetry and positive
definiteness so that the sequence A
(1)
, A
(2)
, will
also be symmetric and positive definite.
) 1 ( ) 1 (
T
) 1 (
) 1 (
T
) 1 ( ) 1 ( ) 1 (
) 1 ( ) 1 (
) 1 ( ) 1 (
) 1 ( ) (

A A
A A

A A
A A
+ =
k k k
k k k k
k
T
k
T
k k
k k
g g
g g
g x
x x
A
A A
A A