L 17 18 Linear Programming and Simplex

11/3/14
Linear Programs (LPs)
Linear Programming and

the Simplex Method
Contributions from:
Brian Williams
Eric Feron. Sommer
Gentry, Paul Robertson,
and Emilio Frazzoli.
Linear Programs are optimization problems:

Maximize Z=3x1 + 5x2
Subject to: x1 4
2x2 12
3x1 + 2x2 18
x1, x2 0
Prof. Julie Shah

16.410-13
November 3rd, 2014
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
Inputs:
Solutions:
real-valued decision variables x,
An assignment is a feasible solution

if it satisfies all constraints.
linear real-valued objective

function Z (utility, cost), and
An assignment is an optimal solution

if no feasible solution exists with a
higher objective value.
linear inequality constraints.
History
Outline
Linear Programs and the Simplex Method were

invented by George Dantzig (1914-2005) in late 1940s.
Crucial to the economic success of many real-world tasks,
including design, manufacturing, control, distribution and
logistical support.
Many algorithmic advances.
Linear Programs.
Example 1: Maximizing Revenue.
Geometric Representation.
Example 2: Resource Allocation.
Geometric Solutions to Linear Programs.

Algebraic Solutions to Linear Programs.
1979 L. Khachyans adaptation of the ellipsoid algorithm.

N. Karmarkars new interior-point algorithm.
Simplex still the best algorithm for most applications.
Emerging as a powerful tool for real-time applications.
Example 1: Maximize Revenue / Day

Problem: Decide how to split hours between work (x1) and
playing the stock market (x2) to maximize revenue (z).
Geometric Representation
Maximize z=8x1+
x1
Subject to
Revenue: Combined wage and stock profits:

Wage is $8 / hour
# hours spent
Stock Profit is daily % yield =
capital
30
Linear Program:
Objective
Our first LP
objective
function
x2 0
x1 + x2 10
x2 3
x1 0
x1 + x2 10
x2 3
x1, x2 0
x1 + x2 10
x2 3
x1, x2 0
feasible
region
Constraint: Total hours no more than 10 hours/day,

with at most 3 hours on the stock market.
x2 capital
Maximize z=8x1+
3000
Subject to
x2 capital
3000
x2
Constraint
11/3/14
Each inequality constraint

defines a feasible half space.
x1
Each inequality constraint

defines a feasible half space.
infeasible
x1
The intersection of all feasible

half spaces forms a polyhedron,
called the feasible region.
feasible
region
A point, if in the region is called

feasible, otherwise, infeasible.
x1 + x2 10
x1 + x2 10
feasible
x2 0
x2
feasible
region
Answer 1
x1 + x2 10
Answer 2
x1 0
Problem: Decide how to allocate production of

Engines 1 and 2 to maximize profit.
Engine parts are made at three plants: A, B, C.
Plants A, B, C have limited throughput.
Max z=8x1+
objective
function
x1 + x2 10
x2 capital
3000
x2 3
x2
An aircraft company makes two aircraft engines.
A constraint is active if the optimal

solution lies along its boundary.
Answer 1
x2 0
Example 2: Resource Allocation
Engine 1 uses parts produced by A and C.

Engine 2 uses parts produced by B and C.
active
feasible
region
x2 capital
3000
x2 3
An optimal solution is a feasible

point such that no other feasible
point has a greater cost Z.
x1
A constraint is active if the optimal

solution lies along its boundary.
Max z=8x1+
objective
function
x2 0
An optimal solution is a feasible

point such that no other feasible
point has a greater cost Z.
active
x2
x1 0
x1
x2 3
x2
Answer depends upon the
orientation of Zs gradient.
x1 0
Example 2 Continued
Plant
A
Production
Time Per
Engine 1
1
Production
Time Per
Engine 2
0
Production Time
Available
Per Week
4
12
18
Profit/ $300,000
Engine
x1: # of Engine 1s
x2: # of Engine 2s
$500,000
Profit (in $100K):

Constraints:
3x1 + 5x2
s.t. x1 4
2x2 12
3x1+ 2x2 18
11/3/14
Example 2
Z*=36
x2
(0,6)
objective
contours
(4,3)
max Z = 3x1+ 5x2

s.t.
x1 4
2x2 12
3x1+ 2x2 18
x1, x2 0
1.6
x1 0
x1
0
(4,0)
x2 0
Maximize Z = c1 x1 + ... + cn xn
Subject to a11 x1 + ... + a1n xn b1
2x2 12
(2,6)
feasible
region
Linear Program: Standard Form
x1 4
3x1 + 2x2 18
Well-formed Linear Programs
am1 x1 + ... + amn xn bm

x1 0,, xn 0.
where:
Z: Overall measure of performance.
xj: Activity level for product j.
cj: Marginal improvement of Z associated with product j.
bi: Available resource I.
aij: Amount of resource I to produce j.
Other forms:
Arbitrary signs on ci, bj, aij, xi.
Equality constraints.
Ill-formed Linear Programs
x2
x2
x1
x1
Feasible region is a convex polyhedron.
Empty feasible region (no solution).

x2
x2
x1
x1
x1>0
Unbounded feasible region (infinite cost).
and x2>0
Constraint 1 and
(Constraint 2
Non-convex feasible region.
Or Constraint 3)
11/3/14
Property: Solutions are at Corner Points
Outline
x2
Linear Programs.
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
Generate and Test.

The Simplex Method.
Assume:
n decision variables x.
x1
No interior solutions:
An interior optimum x* must satisfy z(x*) = 0,
but z(x*) = c 0.
Solutions only at corner points:
Let x be on a constraint boundary B, but not a corner point.
B is the intersection of at most k < n constraints.
x is in the interior of a lower dimensional (n-k) subspace
not a soln.
Property: Optimal Corner Points

x2
Z=30
Z=36
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
x1
Z=0
x2
corner point infeasible solution

Z=29
corner
point
solutions
Solving LPs by Generate and Test
corner point feasible solution
Z=12
If a corner-point feasible solution has no better neighbor,

then it is optimal.
If there are multiple optimal solutions,
then they are connected by edges of the constraint polytope.
Generating Corner Points

x2
Assume:
n decision variables.
m > n inequality
constraints.
corner
point
solutions
Z=30
corner
point
solutions
Z=36
1
Z=0
Z=29
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
x1
Z=12
incumbent x = nil, incumbent z = -inf;

For each corner-point solution cp,
incumbent x = nil 1 2 3
incumbent z = -inf 0 30 36
If feasible cp and
z at cp > incumbent z,
Then incumbent x = cp, incumbent z = z at cp;
Return incumbent x and z;
3
36
Outline
Linear Programs.
Generate and Test.
The Simplex Method.

x1
x1 > 0, x2 > 0
Constraints 1, 2, 3
A corner point is formed by intersecting n constraints.
! m $
m!
&=
" n % n!(m n)!
# corner points = #
20 variables, 30 constraints = 30+ million points!
11/3/14
Property: Paths to Optimal Corner Points

x2
Z=30
Z=30
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
Z=29
corner
point
solutions
Simplex Method
x2
Z=36
Z=36
Standard Form:
Maximize
z = cTx
Subject to:
Ax b
x0
Z=29
corner
point
solutions
x1
Z=12
Z=0
There exists a non-decreasing path from any corner-point

feasible solution to each optima, i.e., never decreases cost.
Hence, from every non-decreasing neighbor, there exists a
non-decreasing path to every optima.
Current = a feasible corner point (origin);

Repeat:
If current has no better neighbor,
Then return current as an optima,
Else current = neighbor of current that increases cost (the most).
Finding Corner Point Coordinates and Cost
Outline
x1
Z=12
Z=0
Linear Programs.
Solution using Simplex Tableau
Generate and Test.
The Simplex Method.
(2,6)
Subject to: x1 4
2x2 12
3x1 + 2x2 18
x1, x2 0
x4 3
x5
x1
1
Assume:
n decision variables.
m > n inequality
constraints.
x3
x2
A corner point is the intersection of n active constraints.

Point 3 intersects 2x2 12 and 3x1 + 2x2 18.
An active constraint is treated as an equality constraint.

Active 2x2 12 ! 2x2 =12, active 3x1 + 2x2 18 ! 3x1 + 2x2 = 18.
Find coordinates and cost of point by solving active constraints.

Point 3: x2 = 6, x1=2, z=36.
Slack Variables and Constraint Distance

(2,6)
2
x4
3
x5
x1
1
x3
x2
Distance from x to x1=0 is x1.

Distance from x to x2=0 is x2.
Distance from x to x1=4 is x3 = 4-x1.
Distance from x to 2x2=12 is 6-x.
proportional to
x4 = 12-2x2.
Introduce slack variables to measure distance

from point x=(x1, x2) to each constraint.
A constraint is active iff its distance
(slack variable) is 0.
Slack variables form redundant coordinate
systems, each a corner + n variables.
Augmented form replaces inequalities
with distance equations.
Distance from x to 3x1+2x2=18 is
18 3x1 2 x 2
13
proportional to
x5 = 18-3x1-2x2.
Augmented Form Facilitates

Search Over Corner Points
Rewrite problem using x1,x2,x3,x4,x5:
Standard Form:

Subject to: x1 4
2x2 12
3x1 + 2x2 18
x1, x2 0
Maximize
Subject to:
z = cTx
Ax b
x0
Augmented Form:
Maximize: Z
Subject to: Z-3x1-5x2 = 0
x1+x3 = 4
2x2+x4 = 12
3x1+2x2+x5 = 18
x1, x2, x3, x4, x5 0
Maximize
Subject to:
z = cTx
z - cTx = 0
Ax+xT = b
x0
11/3/14
Solutions to Augmented Form
Finding Corner Point Coordinates and

Cost From Augmented Form
Idea: Generate corner point by setting n augmented variables to

zero, activating corresponding constraints; solve.
Augmented Solution
Non-Basic Variables: Set to zero; corresponding constraints are active.

Basic Variables:
Determined by active constraints.
Assignment to original and

slack variables.
L augmented constraints
K augmented variables
Basic Solution
An augmented
corner-point solution
(not necessarily feasible).
L basic variables
n = K-L non-basic variables
= # non-decision variables.
Point (1): At the origin

non-basic
x1 = 0
x2 = 0
Basic Feasible Solution

An augmented corner-point
feasible solution.
basic
x3 = 4
x4 = 6
x5=18
Outline
Linear Programs.
Generate and Test.
The Simplex Method.
Point (3): At (2,6)

non-basic
x4 = 0
x5 = 0
x1+x3 = 4
2x2+x4 = 12
3x1+2x2+x5 = 18
basic
x1 = 2
x2 = 6
x3 = 2
x1+x3 = 4
2x2+x4 = 12
3x1+2x2+x5 = 18
The Simplex Method is a

Greedy Improvement Algorithm
1. Setup and Initialization Steps
2. Improvement Step
Simplex Tableau.
Algebraic Interpretation.
3. Optimality Test (stopping rule)

No?
Yes?
4. Exit
The Simplex Method

1.
2.
3.
4.
Setup: Add slack variables to problem.

Initialize: Start at a corner-point feasible solution.
Improve: Move to a better adjacent feasible solution.
Test Optimality: The corner-point solution is optimal
when no better adjacent corner-point solution can be
found.
(0,6), Z=30
2
(0,0), Z=0
Linear Programs.
Generate and Test.
The Simplex Method.
Simplex Tableau.
(2,6), Z=36
4
Outline
(4,3), Z=29
5
(4,0), Z=12
11/3/14
The Initial Tableau

Standard Form:
Maximize: Z
Subject to: Z-3x1-5x2 = 0
x1+x3 = 4
2x2+x4 = 12
3x1+2x2+x5 = 18
x1, x2, x3, x4, x5 0

Subject to: x1 4
2x2 12
3x1 + 2x2 18
x1, x2 0
Iterating the Tableau: Pivoting
Augmented Form:
x1
x2
-3
-5
r1
x3
x4
r2
2
3
12
18
x1
x2
-3
Choose row r3 as first r1

limiting the increase in r2
x1s variable (min
r3
ratio test b/c); Pivot.
Entries of z positive,
no increase possible.
Solution in column b:
x1=2, x2=6, x3=2,
x4=x5=0, z=36.
r2
1
3
-1
x2
x3
1
b
30
4
1/2
x1
x5
5/2
r3
x4
z
r1
x3
6
1
x4
x5
3/2
36
1/3
-1/3
1/2
-1/3
Generate and Test.

The Simplex Method.
Simplex Tableau.
-5
r1
x3
x1
x2
-3
1
3
b
0
4
12
1
x3
x4
x5
5/2
18
b
30
r2
r3
x5
1
2
r1
x4
1/2
-1
6
1
Outline
Linear Programs.
x2
-3
Best for extremely large problems.
6
1/3
x1
With n variables, each bounded 0, 1, there are 2n

corner point feasible solutions.
Pathological examples force examination of all
2n solutions.
Each pivot involves Gaussian elimination, and takes
m x n time.
Simplex algorithm is exponential in the worst-case,
yet is remarkably fast in practice.
Linear programs can be solved in polynomial time
by interior point methods.
Second Iteration:
z
Simplex Algorithm Analysis
Iterating the Tableau: Pivoting

Choose x1 as column
with the most negative
z
entry in row z.
2. Choose row that first r2

limits the increase in the r3
columns variable
(min ratio test b/c).
3. Eliminate x2 by
pivoting intersection:
Add ci*r2, for some ci,
to rows z, r1, r2, r3, so
that the orange cell is
1, and the rest of
column x2 is 0.
b
0
r3
x5
First Iteration:
1. Choose column
with the most negative
entry in row z.
The Simplex Method

1.
2.
3.
4.
Setup: Add slack variables to problem.

Initialize: Start at a corner-point feasible solution.
Improve: Move to a better adjacent feasible solution.
Test Optimality: The corner-point solution is optimal
when no better adjacent corner-point solution can be
found.
(0,6), Z=30
2
(2,6), Z=36
4
1
(0,0), Z=0
(4,3), Z=29
5
(4,0), Z=12
11/3/14
Step 2: Initialize
Maximize Z subject to:
a) Z -3x1 -5x2
= 0
b)
+x1
+x3
= 4
c)
+2x2
+x4
= 12
d) +3x1 +2x2
+ x5 = 18
Start at vertex (0,0):
Usually feasible.
x1, x2
non-basic;
Z, x3, x4, x5
basic.
x10, x20 are active constraints.
Z, x3, x4, and x5 are easily
determined from x1, x2.
(0,0) centered coordinates,
with axes x1, x2.
Step 3: Improve Which Direction?

(2,6)
3
x5
x4
x1
x1
1
x3
x2x2
5
= 0
= 4
= 12
= 18
Move to an adjacent corner, 2 or 5.

Swap an active constraint:
2: make x2 > 0,
5: make x1 > 0.
3
x5
x4
x1
x2
x3
5
Which corner do we pick?
(2,6)
2
Which corner do we pick?

Any corner that increases utility.
3
4
x1
x2
Heuristic: Select direction xi that increases

utility the fastest.
(2,6)
2
Swap (non-)basic variable.
Step 3: Improve Which Direction?

a) Z -3x1 -5x2
b)
+x1
+x3
c)
+2x2
+x4
d) +3x1 +2x2
+x5

a) Z -3x1 -5x2
= 0
b)
+x1
+x3
= 4
c)
+2x2
+x4
= 12
d) +3x1 +2x2
+ x5 = 18
! dZ
$
argmax #
= ci&
" dxi
%
xi
ci is called the marginal gain.

x2 enters as basic,
x2 0 becomes inactive.
Step 3: Improve How far?

a) Z -3x1 -5x2
=
b)
+x1
+x3
=
c)
+2x2
+x4
=
d) +3x1 +2x2
+ x5 =
0
4
12
18
(2,6)
2
3
x4
How far can we increase x2?
Which variable exits basic?

To the first constraint that x2 hits.
b) x1+x3=4 is independent of x2.
c) 2x2+x4=12 is hit at x2=6.
d) x1+2x2+x5=18 hit at x2=9.
x2 hits c) first
Make c) x2+x4=6 active.
x4 exits basic.
New corner point at (x1=0, x4=0).
x2, x3, x5 are basic.
x5
x1
x2
Find value of x2 when constraint c is hit.

1. Set cs slack variable xn to 0
(x3 for b, x4 for c, x5 for d).
2. Solve for the increasing variable
(x2).
Step 3: Improve Where is the Point?
Maximize Z subject to
a) Z -3x1 -5x2
=
b)
+x1
+x3
=
c)
+2x2
+x4
=
d) +3x1 +2x2
+ x5 =
a) Z -3x1 -5x2
=
b)
+x1
+x3
=
c)
+2x2
+x4
=
d) +3x1 +2x2
+ x5 =
0
4
12
18
(2,6)
2
x4
Solve for corner point 2:

Express Z, x2, x3, x5 as fns of x1,x4.
Diagonalize column x2 using c).
Reformulation
Z=30+3x1-5x4
x1+x3=4
x2+x4=6
3x1-2x4+x5=6
x1
1
Substitute for x2 wth c)

Z=3x1+5(6-x4)
x1+x3=4
x2+x4=6
3x1+2(6-x4)+x5=18
(Z=30, x1=0, x2=6, x3=4, x4=0, x5=6)
Coordinates now (0,6)-centered, (x1, x4) specify coordinate axes.
0
4
12
18
(2,6)
2
x4

Diagonalize column x2 using c).
a) Z -3x1
+5x4
b)
+x1
+x3
c)
+2x2
+x4
d) +3x1
-2x4 +x5
x1
1
= 30
= 4
= 12
= 6
(Z=30, x1=0, x2=6, x3=4, x4=0, x5=6)
11/3/14
Step 4: Test Optimality

(0,6)-centered coordinates (x1,x4=0):
a) Z -3x1
+5x4
b)
+x1
+x3
c)
+2x2
+x4
d) +3x1
-2x4 +x5
= 30
= 4
= 12
= 6
Is (0,6) a maximum?
Increasing x1 increases Z. Why?
Not a maximum.
Step 3: Improve What Direction?

(2,6)
2
3
x4
x1
1
a) Z -3x1
+5x4
= 30
b)
+x1
+x3
= 4
c)
+2x2
+x4
= 12
d) +3x1
-2x4 +x5 = 6
Which corner, 3 or 1?
Increasing x1 increases Z.
Increasing x4 decreases Z.
(2,6)
2
3
x4
x1
x1 enters as basic.
Step 3: Improve How Far?

a) Z -3x1
+5x4
b)
+x1
+x3
c)
+2x2
+x4
d) +3x1
-2x4 +x5
= 30
= 4
= 12
= 6
Increasing x1 hits which constraint first?

Hits b) at x1=4.
c) is independent of x1.
Hits d) at x1=2.
x1 enters and x5 exits basic, d) active.
(2,6)
2
x4
3
x5
x1
4
x3
1. Set constraints slack variable to 0

(x3 for b, x4 for c, x5 for d).
2. Solve for the increasing variable
(x1).

a) Z -3x1
+5x4
b)
+x1
+x3
c)
+2x2
+x4
d) +3x1
-2x4 +x5
= 30
= 4
= 12
= 6
(2,6)
2
x4

Diagonalize column x1 using d).
a) Z
+ 3x4 +x5 = 36
b)
+x3 +2/3x4 -1/3x5 = 2
c)
+2x2
+x4
= 12
d) +3x1
-2x4 +x5 = 6
Step 3: Minimum Ratio Test

Increasing x1 hits which constraint first?
As xi increases, zero or more basic variables xj decrease.
A decrease in xj is limited by
xj + cij*xi = bj
xj 0.
xi = -bj/cij when xj = 0.
The exiting base variable xj is the one with the smallest positive ratio bj/cij.
3
x5
b)
c)
d)
+x1
+x3
= 4
+2x2
+x4
= 12
+3x1
-2x4 +x5 = 6
x4 = 0
x3 = 4 x1 0
x2 = 6
0
x5 = 6 3x1 0
x1 4/1 = 4
x1 6/3 = 2
x1 enters and x5 exits basic; d) becomes active.
Step 4: Test Optimality

a) Z
+ 3x4 +x5 = 36
b)
+x3 +2/3x4 -1/3x5 = 2
c)
+2x2
+x4
= 12
d) +3x1
-2x4 +x5 = 6
(2,6)
2
x4
Is (2,6) a maximum?
Optimal solution!
3
x5
x1
1
(Z=36, x1=2, x2=6, x3=2, x4=0, x5=0)

L 17 18 Linear Programming and Simplex

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

L 17 18 Linear Programming and Simplex

Încărcat de

Drepturi de autor:

Formate disponibile

11/3/14

Linear Programs (LPs)

Linear Programming and

Linear Programs are optimization problems:

Prof. Julie Shah

real-valued decision variables x,

An assignment is a feasible solution

linear real-valued objective

An assignment is an optimal solution

linear inequality constraints.

Linear Programs and the Simplex Method were

Geometric Solutions to Linear Programs.

1979 L. Khachyans adaptation of the ellipsoid algorithm.

Emerging as a powerful tool for real-time applications.

Example 1: Maximize Revenue / Day

Revenue: Combined wage and stock profits:

Constraint: Total hours no more than 10 hours/day,

Each inequality constraint

Each inequality constraint

The intersection of all feasible

A point, if in the region is called

Problem: Decide how to allocate production of

Plants A, B, C have limited throughput.

An aircraft company makes two aircraft engines.

A constraint is active if the optimal

Example 2: Resource Allocation

Engine 1 uses parts produced by A and C.

An optimal solution is a feasible

A constraint is active if the optimal

An optimal solution is a feasible

Profit (in $100K):

max Z = 3x1+ 5x2

Linear Program: Standard Form

Well-formed Linear Programs

am1 x1 + ... + amn xn bm

Ill-formed Linear Programs

Feasible region is a convex polyhedron.

Empty feasible region (no solution).

Ill-formed Linear Programs

Ill-formed Linear Programs

Non-convex feasible region.

Property: Solutions are at Corner Points

Generate and Test.

Algebraic Solutions to Linear Programs.

Property: Optimal Corner Points

corner point infeasible solution

Solving LPs by Generate and Test

corner point feasible solution

If a corner-point feasible solution has no better neighbor,

Generating Corner Points

incumbent x = nil, incumbent z = -inf;

Return incumbent x and z;

Algebraic Solutions to Linear Programs.

A corner point is formed by intersecting n constraints.

20 variables, 30 constraints = 30+ million points!

Property: Paths to Optimal Corner Points

There exists a non-decreasing path from any corner-point

Current = a feasible corner point (origin);

Finding Corner Point Coordinates and Cost

A corner point is the intersection of n active constraints.

An active constraint is treated as an equality constraint.

Find coordinates and cost of point by solving active constraints.

Slack Variables and Constraint Distance

Distance from x to x1=0 is x1.