Page |1

TABLE OF CONTENTS TITLE: Chan, Fran Ryan PAGE NO. ---------------------------------------------------Aguirre, Zyrill Mae ----------------------------------------------------Elizan, Jane Josanin ---------------------------------------------------Macutay, Angelo Victor ------------------------------------------------Samson, Jeddlou Yape, Janine ---------------------------------------------------------------------------------------------------------

Page |2

I.

Optimization Techniques (At least 5 methods)

1. Cutting-plane method Theoretical Discussion: In mathematical optimization, the cutting-plane method is an umbrella term for optimization methods which iteratively refine a feasible set or objective function by means of linear inequalities, termed cuts. Such procedures are popularly used to find integer solutions to mixed integer linear programming (MILP) problems, as well as to solve general, not necessarily differentiable convex optimization problems. The use of cutting planes to solve MILP was introduced by Ralph E. Gomory. Cutting plane methods for MILP work by solving a non-integer linear program, the linear relaxation of the given integer program. The theory of Linear Programming dictates that under mild assumptions (if the linear program has an optimal solution, and if the feasible region does not contain a line), one can always find an extreme point or a corner point that is optimal. The obtained optimum is tested for being an integer solution. If it is not, there is guaranteed to exist a linear inequality that separates the optimum from the convex hull of the true feasible set. Finding such an inequality is the separation problem, and such an inequality is a cut. A cut can be added to the relaxed linear program. Then, the current non-integer solution is no longer feasible to the relaxation. This process is repeated until an optimal integer solution is found. Cutting-plane methods for general convex continuous optimization and variants are known under various names: Kelley's method, Kelley-Cheney-Goldstein method, and bundle methods. They are popularly used for non-differentiable convex minimization, where a convex objective function and its subgradient can be evaluated efficiently but usual gradient methods for differentiable optimization cannot be used. This situation is most typical for the concave maximization of

Page |3
Lagrangian dual functions. Another common situation is the application of the Dantzig-Wolfe decomposition to a structured optimization problem in which formulations with an exponential number of variables are obtained. Generating these variables on demand by means of delayed column generation is identical to performing a cutting plane on the respective dual problem.

Sample Problems: Examples 1. Consider the integral optimization problem Maximize x1+x2+x3 Subject to x1+x2 1 x1+x2 1 x1+x2 1 xi 0, an integral for i=1,2,3 introduce slack variables xi, i=4,5,6 to produce the standard form maximize x1+x2+x3 subject to x1+x2+x4=1 x1+x3+x5=1 x2+x3+x4=1 x1+x2+x6=1 xi 0, xi an integer for i=1,2,,6 solving this with the simplex method gives the solution x1=x2=x3=1/2 and the equations

Page |4
1 2 3 1 1 2 1 2 1 2 1 2 4 4 4 4 1 2 1 2 1 2 1 2 5 5 5 5 1 2 1 2 1 2 1 2 1 2 1 6= 2 1 6= 2 1 6= 2 6=

Each of these equations produce the same cutting plane, and with the introduction of a new slack variable x7 it can be written as a new constraints 7 1 4 2 1 5 = 2 1 2

Applications:

Page |5
2. CONJUGATE GRADIENT METHOD Theoretical Discussion: Subject to linear equality and linear inequality constraints its feasible region is a convex polyhedron, which is a set defined as the intersection of finitely many half spaces, each of which is defined by a linear inequality. Its objective function is a realvalued affine function defined on this polyhedron. A linear programming algorithm finds a point in the polyhedron where this function has the smallest or largest value if such a point exists. linear programming is a technique for the optimization of a linear objective function, subject to linear equality and linear inequality constraints. Its feasible region is a convex polyhedron, which is a set defined as the intersection of finitely many half spaces, each of which is defined by a linear inequality. Its objective function is a real-valued affine function defined on this polyhedron. A linear programming algorithm finds a point in the polyhedron where this function has the smallest (or largest) value if such a point exists. Linear programs are problems that can be expressed in canonical form Maximize Subject to Ax And x 0 where x represents the vector of variables (to be determined), c and b are vectors of (known) coefficients, A is a (known) matrix of coefficients, and is the matrix

transpose. The expression to be maximized or minimized is called the objective

function (cTx in this case). The inequalities Ax

b are the constraints which specify

aconvex polytope over which the objective function is to be optimized. In this context, two vectors are comparable when they have the same dimensions. If every entry in the first is less-than or equal-to the corresponding entry in the second then we can say the first vector is less-than or equal-to the second vector. Linear programming can be applied to various fields of study. It is used in business and economics, but can also be utilized for some engineering problems. Industries that use linear programming models include transportation, energy,

Page |6
telecommunications, and manufacturing. It has proved useful in modeling diverse types of problems in planning, routing, scheduling, assignment, and design.

Sample Problems: Examples 1. find the numbers x1 and x2 that maximize the sum x1+x2 subject to the constraints x1 0, 0, and X1+2x2 4 4X1+2x2 12 -X1+x2 1 In this there are two unknowns, and five constraints. All the constraints are inequalities and they are all linear in the sense that each involves an inequality in some linear function of the variables. The first two constraints x1 0 and x2 0, are special. These are called nonnegativity constraints and are often found In linear programming problems. The other constraints are then called the main constraints. The function be maximized or minimized is called the objective function. Here, the objective function is x1+x2 Since there are only two variables, we can solve this problem by graphing the set of points in the plane that satisfies all the constraints called the constraints set and then finding which point of this set maximizes the value of the objective function each inequality of all the half-planes, in the present example, the constraints set is the five sided figure shaded in figure 1

Page |7

We seek the points (x1, x2), that achieves the maximum of x1 +x2 as (x1,x2) ranges over this constraint set. The function x1+x2 is constant on lines with slope 1, for example the line x1+x2=1, and as we move this line further from the origin un and to the right, the value of x1+x2 increases. Therefore, we seek the line of slope -1 that is farthest from the origin and still touches the constraint set. This occurs at the intersection of the lines x1+2x2=4 and 4x1+2x2=12, namely, (x1, x2)=(8/3,2/3). The value of the objective function there is (8/3 +(2/3)=10/3. 2. A company makes two products (X and Y) using two machines (A and B). Each unit of X that is produced requires 50 minutes processing time on machine A and 30 minutes processing time on machine B. Each unit of Y that is produced requires 24 minutes processing time on machine A and 33 minutes processing time on machine B. At the start of the current week there are 30 units of X and 90 units of Y in stock. Available processing time on machine A is forecast to be 40 hours and on machine B is forecast to be 35 hours. The demand for X in the current week is forecast to be 75 units and for Y is forecast to be 95 units. Company policy is to maximise the combined sum of the units of X and the units of Y in stock at the end of the week.

Formulate the problem of deciding how much of each product to make in the current week as a linear program. Solve this linear program graphically.

Solution
Let

Page |8

x be the number of units of X produced in the current week y be the number of units of Y produced in the current week

then the constraints are: 50x + 24y <= 40(60) machine A time 30x + 33y <= 35(60) machine B time x >= 75 - 30 i.e. x >= 45 so production of X >= demand (75) - initial stock (30), which ensures we meet demand y >= 95 - 90 i.e. y >= 5 so production of Y >= demand (95) - initial stock (90), which ensures we meet demand The objective is: maximise (x+30-75) + (y+90-95) = (x+y-50) i.e. to maximise the number of units left in stock at the end of the week It is plain from the diagram below that the maximum occurs at the intersection of x=45 and 50x + 24y = 2400

Solving simultaneously, rather than by reading values off the graph, we have that x=45 and y=6.25 with the value of the objective function being 1.25. Applications: Aerospace: Vehicle Guidance and Control

Page |9
The effectiveness of the CG Conjugate Gradient method for the solution of the optimal aerospace vehicle guidance and control problem is demonstrated. The superiority of the convergence rate of the CG method over the steepest descent method is demonstrated. The search of golden section technique is also discussed and shown to be a very powerful technique for one dimensional optimization. As one dimensional search cannot be carried out exactly accumulation of errors due to this source can produce directions of search which increase rather than decrease the objective functional. In this case it is found useful to go back to the local gradient direction and start the process over again. A number of new basic theorems and algorithmic techniques fundamental to the CG method are presented here and the establishment of a monotonic decreasing sequence for the cost functional which results in the determination of the optimal control is shown. A companion paper applies the basic results developed in this paper to the Mars entry guidance and control problem. Implicit Feedback Collaborative Filtering The need for solving weighted ridge regression (WRR) problems arises in a number of collaborative ltering (CF) algorithms. Often, there is not enough time to calculate the exact solution of the WRR problem, or it is not required. The conjugate gradient (CG) method is a state-of-the-art approach for the approximate solution of WRR problems. 3. MULTIPLIER METHODS Theoretical Discussion: The method of Lagrange multipliers is a powerful tool for solving this class of problems without the need to explicitly solve the conditions and use them to eliminate extra variables. And one of the most common problems in calculus is that of finding maxima or minima of a function, but it is often difficult to find a closed form for the function being extremes. Such difficulties often arise when one wishes

P a g e | 10
to maximize or minimize a function subject to fixed outside conditions or constraints Lagrange multiplier is to provide a strategy for finding the local maxima and minima of a function subject to equality constraints. Maximize f(X,y) Subject to g(x,y)=c. We need both f and g to have continuous first partial derivatives. We introduce a new variable called a lagrange multiplier and study the lagrange function defined by ( , , ) = f(x, y) (g(x, y) c) Where the term may be either added or subtracted if f(x0,y0) for the original constrained problem, then there exest 0 sucg that (x0,y0, 0) is a stationary point for the lagrange function (stationary point ate those where the partial derivatives of are zero i.e problems. Sample Problems: Examples 1. Suppose we wish to maximize f(x,y)=x+y subject to the constraint = 1. The feasible set is the unit circle, and the level sets of f are diagonal lines (with slope -10, so we can see graphically that the maximum occurs at (2 /2 , 2 /2) and that the minimum occurs at ( 2 /2 , 2 /2). Using the method of lagrange multiplier, we have g(x,y)-c= hence ( , , ) = ( , ) Setting the gradient
, ,

= 0. however, not all stationary point yield a solution of the

1,

( ( , )

)=

1)

(x,y,)=0 yield the system of equations =1 =1 2 2 = 0, = 0,

P a g e | 11
= 1 = 0,

Where the last equation is the original constraint The first two equation yield x=-1/2()and y=-1/2(), where substituting into the last equation yields /2)= 2 and f ( 2 /2 , 2 /2)= 2 , Thus the maximum is 2 , which is attained at (2 /2 , 2 /2), and the minimum is 2 , which is attained at ( 2 /2 , 2 /2). 2. Entropy Suppose we wish to find the descrete probability on the points (x1,x2,.,x3) with maximal information entropy. This is the same as saying that we wish to find the least blassed probability distribution on the points(x1,x2,.,xn). In other words, we wish to maximize the Shannon entropy equation. ( 1, 2, , )= log

0.

= 1 so f(2 /2 , 2

For this to be a probability distribution the sum of the probabilities pi at each point xi must equal to 1, so our constants is g(p)=1. ( 1, 2, . , )=

We use lagrange multipliers to find the point of mzximum entropy, p , across all discrete probability distribution p on *x1,x2,,xn+. We require that ( ( 1) = = 0,

Which gives a system of n equation,k=1,,n, such that { log ( 1) = =0

Carrying out the differentiation of these n equations, we get ( 1 2 log ) =0

P a g e | 12
This shows that all pk are equal because they depend on only. By using the constant = 1 we find = 1

Applications: Economics Constrained optimization plays a central role in economics. For example, the choice problem for a consumer is represented as one of maximizing a utility function subject to a budget constraint. The Lagrange multiplier has an economic interpretation as the shadow price associated with the constraint, in this example the marginal utility of income. Other examples include profit maximization for a firm, along with various macroeconomic applications. Control theory In optimal control theory, the Lagrange multipliers are interpreted as costate variables, and Lagrange multipliers are reformulated as the minimization of the Hamiltonian, in Pontryagin's minimum principle.

P a g e | 13
4. LINE-SEARCH METHOD OF OPTIMIZATION Theoretical Discussion: In optimization, the line search strategy is one of the most common iterative approaches on finding a local minimum XN of an objective function f: RN R. The line search approach first finds a descent direction along which the objective function F will be reduced and then computes a step size that decides how far X should move along that direction. The descent direction can be computed by various methods, such as gradient descent, Newton's method and Quasi-Newton method. The step size can be determined either exactly or inexactly. We prepare for the consideration of algorithms for locating a local minimum in the optimization problem with no constrains. All methods have in common the basic structure: in each iteration a direction dn is chosen from the current location xn. The next location, xn+1, is the minimum of the function along the line that passes through xn in the direction dn. Each iteration of a line search method computes a search direction pk and then decides how far to move along that direction. The iteration is given byxk+1 = xk + _kpk where the positive scalar k is called the step length. The success of a line search method depends on effective choices of both the direction pk and the step length_k. A method like "Newton's" method chooses a step, but the validity of that step only goes as far as the Newton quadratic model for the function really reflects the function. The idea of a line search is to use the direction of the chosen step, but to control the length, by solving a one-dimensional problem of minimizing where is the search direction chosen from the position . Note that . If you can compute the gradient, you can effectively do a one-dimensional search with derivatives. Typically, an effective line search only looks toward since a reasonable method should guarantee that the search direction is a descent direction, which can be expressed as . It is typically not worth the effort to

P a g e | 14
find an exact minimum of since the search direction is rarely exactly the right direction. Usually it is enough to move closer. One condition that measures progress is called the Armijo or sufficient decrease condition for a candidate .

Often with this condition, methods will converge, but for some methods, Armijo alone does not guarantee convergence for smooth functions. With the additional curvature condition, If you look at graphs showing iterative searches in two dimensions, you can see the evaluations spread out along the directions of the line searches. Typically, it only takes a few iterations to find a point satisfying the conditions. However, the line search is not always able to find a point that satisfies the conditions. Usually this is because there is insufficient precision to compute the points closely enough to satisfy the conditions, but it can also be caused by functions that are not completely smooth or vary extremely slowly in the neighborhood of a minimum. Sample Problems: 1. Determine the following () = ( 1 1) 0.6( 2 2) ) 4( 3 (5 3) 0.25( 4 4)

log(

where (x1,x2,x3,x4)=(1,1,1,1) and (d1,d2,d3,d4)=(-1,0.6,-4,-0.25). () = 4.65 17.4 log(1 () = 17.4225 log(1 4.65) 1 1 0.6 1 0.6 4 1 4 0.25 1 0.25 4.65 1 4.6 ) log(1 0.6) log(1 4) log(1 0.25)

P a g e | 15

P a g e | 16
This method is solved using Bisection Line- Search Method

2 . Using matlab Find minimum [x^2/2+cos[x],{x,1}] The line search decreased the step size to within tolerance specified by Accuracy and Precision. It was unable to find a sufficient decrease in the function. This runs into problems because the real differences in the function are negligible compared to evaluation differences around the point, as can be seen from the plot. Plot[x^2/2+cos[x],{x,0, . 0004}], plot range up to {1-10^-15,1+10^-15}] Sometimes it can help to subtract out the constants so that small changes in the function are more significant.

findmimum[x^2/2 + cos[x]-1,{x,1}] {1.11022x10^-16, {x to 0.00024197}} In this case, however, the approximation is only slightly closer because the function is quite noisy near 0, as can be seen from the plot. Plot[x^2/2+cos[x]-1,{x,0, . 0004}]

P a g e | 17

Thus, to get closer to the correct value of zero, higher precision is required to compute the function more accurately. This is an example of a problem where the Newton step is very large because the starting point is at a position where the Jacobian (derivative) is nearly singular.

The step size is (not severely) limited by the option. Findrootplot[cos[x Pi],{{x, -5}}

This shows the same example but with a more rigorous step-size limitation, which finds the root near the starting condition.

P a g e | 18
Note that you need to be careful not to set the maximum relative step size too small, or it will affect convergence, especially for minima and roots near zero.

Applications: 1. A NEW CONJUGATE GRADIENT METHOD WITH GUARANTEED DESCENT AND AN EFFICIENT LINE SEARCH A new nonlinear conjugate gradient method and an associated

implementation, based on an inexact line search, are proposed and analyzed. With exact line search, our method reduces to a nonlinear version of the HestenesStiefel conjugate gradient scheme. For any (inexact) line search, the scheme satisfies the descent condition . Moreover, a global convergence result is

established when the line search fulfills the Wolfe conditions. A new line search scheme is developed that is efficient and highly accurate. Efficiency is achieved by exploiting properties of linear interpolants in a neighborhood of a local minimizer. High accuracy is achieved by using a convergence criterion, which we call the approximate Wolfe conditions, obtained by replacing the sufficient decrease criterion in theWolfe conditions with an approximation that can be evaluated with greater precision in a neighborhood of a local minimum than the usual sufficient decrease criterion.Numerical comparisons are given with both L-BFGS and conjugate gradient methods using the unconstrained optimization problems in the CUTE library. 2. APPLICATION OF A PRIMAL-DUAL INTERIOR POINT ALGORITHM USING EXACT SECOND ORDERINFORMATION WITH A NOVEL NON-MONOTONE LINE SEARCH METHOD TO GENERALLY CONSTRAINED MINIMAX OPTIMISATION PROBLEMS This work presents the application of a primal-dual interior point method to

P a g e | 19
minimax optimisation problems. The algorithm differs significantly from previous approaches as it involves a novel non-monotone line search procedure, which is based on the use of standard penalty methods as the merit function used for line search. The crucial novel concept is the discretisation of the penalty parameter used over a finite range of orders of magnitude and the provision of a memory list for each such order. An implementation within a logarithmic barrier algorithm for bounds handling is presented with capabilities for large scale application. Case studies presented demonstrate the capabilities of the proposed methodology, which relies on the reformulation of minimax models into standard nonlinear optimisation models. Some previously reported case studies from the open literature have been solved, and with significantly better optimal solutions identified. We believe that the nature of the nonmonotone line search scheme allows the search procedure to escape from local minima, hence the encouraging results obtained.

P a g e | 20
5. Powell's Method Theoretical Discussion: Discussion The essence of Powell's method is to add two steps to the process described in the preceding paragraph. The vector direction moved over represents, in some sense, the average steps in an

the n intermediate

iteration. Thus the point

is determined to be the point at which the minimum . As before, f is a function of one

of the function f occurs along the vector

variable along this vector and the minimization could be accomplished with an application of the golden ratio or Fibonacci searches. Finally, since the vector was such a good direction, it replaces one of the direction vectors for

the next iteration. The iteration is then repeated using the new set of direction vectors to generate a sequence of points outlined below. Let be an initial guess at the location of the minimum of the function . . In one step of the iteration

instead of a zig-zag path the iteration follows a "dog-leg" path. The process is

Let vectors. Initialize the vectors

for

be the set of standard base

for

and use their transpose

to

form the columns of the matrix U as follows: . Initialize the counter (i) Set (ii) For find the value of . , that minimizes , .

P a g e | 21
and set .

(iii) Set set .

for

and

(iv) Increment the counter (v) Find the value of and set .

. that minimizes ,

(vi) Repeat steps (i) through (v) until convergence is achieved. Algorithm (Powell's Method for Finding a Minimum). To numerically approximate a local minimum of and , where f is a continuous function of n real variables by starting with one point and using the "dog-leg" .

search along the directions of the direction vectors Sample Problems:

1. Find X1 and X2 for the function f (x, y) = cos(x) + sin(y). Use the initial point X0 = (5.5, 2). 1 =[ 0 f (P0 0 ] and P0 = X0 = (5.5, 2). When i = 1 the function 1 1U1) = f ((5.5, 2) = f (5.5 = cos(5.5 has a minimum at 1 = function f (P1 2U2) = f ((3.1415958, 2) = f (3.1415982, 2 = cos(3.1415982) 2(0, 1)) 2) sin(2 2) 1(1, 0)) 1) + sin(2) 1, 2)

2.3584042. Thus P1 = (3.1415958, 2). When i = 2 the

P a g e | 22
has a minimum at 2 = 2.7123803. Thus P2 = (3.1415958, 4.7123803). Set U2 = (P2 P0) and =[ The function f (P0 U2) = f ((5.5, 2) = f (5.5 = cos(5.5 ( 2.3584042, 2.7123803)) 2.7123903 ) sin(2 2.7123803 ) 2.3584042 ) 2.3584042 , 2 0 1 2.3584042 ] 2.7123803

has a minimum at = 0.9816697. Thus X1 = (3.1848261, 4.6626615). Set P0 = X1. When i = 1 the function f (P0 1U1) = f ((3.1848261, 4.6626615) = f (3.1848261, 4.6626615 = cos(3.1848261) the function f (P1 2U2) = f ((3.1848261, 4.7123732) = f (3.1848261 2.71238092) has a minimum at 2 = 0.0078820. Thus P2 = (3.1662373, 4.7337521). Set U2 = (P2-P0) and =[ The function f (P0 U2) = f ((3.1848261, 4.6626615) = f (3.1848261 0.0710906 ) has a minimum at = 0.8035684. Thus X2 = (3.1698887, 4.7197876). = cos(3.1848261 ( 0.0185889, 0.0710906)) 0.0710906 ) sin(4.6626615 0.0185889 , 4.6626615 0.0185889 ) 2.3584042 2.7123803 0.0185889 ] 0.0710906 = cos(3.1848261 2( 2.3584042, 2.7123809)) 2.71238092) 2.35840422, 4.7123732 1(0, 1)) 1) 1)

sin(4.6626615

has a minimum at 1 = 0.0497117. Thus P1 = (3.1848261, 4.7123732). When i = 2

2.35840422) + sin(4.7123732 +

P a g e | 23
2. Determine the minimum of the following function using Powell's method starting at point xI = (2,2). Minimize: y = 2x12 + x22 x1x2.

As shown in the figure, the procedure begins at point xI = (2,2), and step 0 locates the minimum on the contour tangent line sn, xo, by a single variable search along coordinate axis n (= 2) as follows: Step 0. or n=2 s1T = (1,0) s2T = (0,1) xIT = (2,2) xo = xI + I s2

I)

= 2(2)2 + (2 + s1T = (1,0)

1 s1

I)2

- (2)(2 +
I

I)

Using an exact line search, Step 1. or j = 1 x1 = xo +

= -1 and xoT = (2,1). xoT = (2,1)

s2T = (0,1)

P a g e | 24

1)

= 2(2 +

1)2

+ (1)2 - (2 +
1

1)(1)

Using an exact line search, j = 2 x2 = x1 + 2 s2 or

= -7/4 and x1T = (,1). Replace s1 with s2

2)

= 2()2 + (1 +

2)2

- ()(1 +
2

2)

Using an exact line search, Step 2. s2 is replaced with

= - 7/8 and x2T = (1/4, 1/8)

Step 3. Choose

so that y[x2 +

3(x2

- xo)] is a minimum. Let:

3)

= 2(1/4 -

3)2

+ (1/8 3

3) 2

- (1/4 -

3)(1/8

3)

Using an exact line search,

= 1/7 and x3T = (0,0). x3 is the minimum of the

quadratic function, and the procedure ends.

Applications:

P a g e | 25
References:

P a g e | 26
Members Participation:

Aguirre, Zyrill Mae -

Chan, Fran Ryan -

Elizan, Jane Josanin -

Macutay, Angelo Victor -

Samson, Jeddlou -

Yape, Janine -

P a g e | 27