EE236A (Winter 2012-13)

Lecture 6 Duality

dual of an LP in inequality form variants and examples complementary slackness

61

Dual of linear program in inequality form

we dene two LPs with the same parameters c Rn, A Rmn, b Rm an LP in inequality form minimize cT x subject to Ax b an LP in standard form maximize bT z subject to AT z + c = 0 z0 this problem is called the dual of the rst LP in the context of duality, the rst LP is called the primal problem
Duality 62

Duality theorem
notation p is the primal optimal value; d is the dual optimal value p = + if primal problem is infeasible; d = if dual is infeasible p = if primal problem is unbounded; d = if dual is unbounded

duality theorem: if primal or dual problem is feasible, then p = d moreover, if p = d is nite, then primal and dual optima are attained note: only exception to p = d occurs when primal and dual are infeasible

Duality

63

Weak duality
lower bound property: if x is primal feasible and z is dual feasible, then c T x bT z proof: if Ax b, AT z + c = 0, and z 0, then 0 z T (b Ax) = bT z + cT x cT x + bT z is the duality gap associated with primal and dual feasible x, z weak duality: the lower bound property immediately implies that p d (without exception)
Duality 64

Strong duality
if primal and dual problems are feasible, then there exist x, z that satisfy c T x = bT z , Ax b, A T z + c = 0, z 0

combined with the lower bound property, this implies that x is primal optimal and z is dual optimal the primal and dual optimal values are nite and equal: p = c T x = bT z = d (proof on next page)

Duality

65

proof: we show that there exist x, z that satisfy A 0 cT 0 I bT x z b 0 , 0 x z

0 A

=c

the lower-bound property implies that any solution necessarily satises c T x + bT z = 0 to prove a solution exists we show that the alternative system (p. 55) u 0, t 0, AT u + tc = 0, Aw tb, bT u + c T w < 0

has no solution

Duality

66

the alternative system has no solution because: if t > 0, dening x = w/t, z = u/t gives z 0, AT z + c = 0, Ax b, cT x < bT z

this contradicts the lower bound property if t = 0 and bT u < 0, u satises u 0, A T u = 0, bT u < 0

this contradicts feasibility of Ax b (page 52) if t = 0 and cT w < 0, w satises Aw 0, cT w < 0

this contradicts feasibility of AT z + c = 0, z 0 (page 53)

Duality 67

Primal infeasible problems

if p = + then d = + or d = proof: if primal is infeasible, then from page 52, there exists w such that w 0, A T w = 0, bT w < 0

if the dual problem is feasible and z is any dual feasible point, then z + tw 0, AT (z + tw) + c = 0 for all t 0

therefore z + tw is dual feasible for all t 0; moreover, as t , bT (z + tw) = bT z tbT w + so the dual problem is unbounded above
Duality 68

Dual infeasible problems

if d = then p = or p = + proof: if dual is infeasible, then from page 53, there exists y such that A T y 0, cT y < 0

if the primal problem is feasible and x is any primal feasible point, then AT (x + ty ) b for all t 0

therefore x + ty is primal feasible for all t 0; moreover, as t , cT (x + ty ) = cT x + tcT y so the primal problem is unbounded below
Duality 69

Exception to strong duality

an example that shows that p = +, d = is possible primal problem (one variable, one inequality) minimize x subject to 0 x 1 optimal value is p = + dual problem maximize z subject to 0 z + 1 = 0 z0 optimal value is d =

Duality

610

PSfrag

Summary
p = + d = +
primal inf. dual unb. optimal values equal and attained exception primal unb. dual inf.

p nite

p =

d nite

d =

upper-right part of the table is excluded by weak duality rst column: proved on page 68 bottom row: proved on page 69 center: proved on page 65
Duality 611

Outline

dual of an LP in inequality form variants and examples complementary slackness

Variants
LP with inequality and equality constraints minimize cT x subject to Ax b Cx = d standard form LP minimize cT x subject to Ax = b x0 maximize bT y subject to AT y c maximize bT z dT y subject to AT z + C T y + c = 0 z0

dual problems can be derived by converting primal to inequality form same duality results apply
Duality 612

Piecewise-linear minimization
minimize f (x) = max (aT i x + bi )
i=1,...,m

LP formulation (variables x, t; optimal value is minx f (x)) minimize subject to t A 1 x t b

dual LP (same optimal value) maximize bT z subject to AT z = 0 1T z = 1 z0

Duality 613

Interpretation
for any z 0 with
i zi

= 1, for all x

T f (x) = max(aT x + b ) z (Ax + b) i i i

this provides a lower bound on the optimal value of the PWL problem min f (x) min z T (Ax + b)
x x

bT z if AT z = 0 otherwise

the dual problem is to nd the best lower bound of this type strong duality tells us that the best lower bound is tight
Duality 614

-Norm approximation
minimize LP formulation minimize subject to dual problem maximize bT u + bT v subject to AT u AT v = 0 1T u + 1T v = 1 u 0, v 0 simpler equivalent dual maximize bT z subject to AT z = 0,
Duality

Ax b

t A 1 A 1 x t b b

(1)

(2)
615

proof of equivalence of the dual problems (assume A is m n) if u, v are feasible in (1), then z = v u is feasible in (2):
m

=
i=1

|vi ui| 1T v + 1T u = 1

moreover the objective values are equal: bT z = bT (v u)

if z is feasible in (2), dene vectors u, v by ui = max{zi, 0} + , with = (1 z 1)/(2m) these vectors are feasible in (1) with objective value bT (v u) = bT z vi = max{zi, 0} + , i = 1, . . . , m

Duality

616

Interpretation
lemma: uT v u
1

holds for all u, v

1

therefore, for any z with z

1,

Ax b

z T (Ax b)

this provides a bound on the optimal value of the -norm problem min Ax b
x

min z T (Ax b)
x

bT z

if AT z = 0 otherwise

the dual problem is to nd the best lower bound of this type strong duality tells us the best lower bound is tight
Duality 617

Outline

dual of an LP in inequality form variants and examples complementary slackness

Optimality conditions
primal and dual LP minimize cT x subject to Ax b Cx = d maximize bT z dT y subject to AT z + C T y + c = 0 z0

optimality conditions: x and (y, z ) are primal, dual optimal if and only if x is primal feasible: Ax b and Cx = d y , z are dual feasible: AT z + C T y + c = 0 and z 0 the duality gap is zero: cT x = bT z dT y

Duality

618

Complementary slackness
assume A is m n with rows aT i the duality gap at primal feasible x, dual feasible y , z can be written as c T x + bT z + d T y = (b Ax)T z + (d Cx)T y = (b Ax)T z
m

=
i=1

z i ( bi a T i x)

primal, dual feasible x, y , z are optimal if and only if z i ( bi a T i x ) = 0, i = 1, . . . , m

i.e., at optimum, b Ax and z have a complementary sparsity pattern: zi > 0 = aT i x = bi ,

Duality

aT zi = 0 i x < bi =
619

Geometric interpretation
example in R2
a1 x c a2

T two active constraints at optimum: aT 1 x = b1 , a 2 x = b2

optimal dual solution satises A T z + c = 0, z 0, zi = 0 for i {1, 2}

in other words, c = a1z1 + a2z2 with z1 0, z2 0 geometric interpretation: c lies in the cone generated by a1 and a2
Duality 620

Example
4x 1 5x 2 1 0 2 1 subject to 0 1 1 2 show that x = (1, 1) is optimal second and fourth constraints are active at (1, 1) therefore any dual optimal z must be of the form z = (0, z2, 0, z4) with 2 1 1 2 z2 z4 = 4 5 , z 2 0, z4 0 minimize

x1 x2

0 3 0 3

z = (0, 1, 0, 2) satises these conditions dual feasible z with correct sparsity pattern proves that x is optimal
Duality 621

Optimal set
primal and dual LP (A is m n with rows aT i ) minimize cT x subject to Ax b Cx = d assume the optimal value is nite
> 0} let (y , z ) be any dual optimal solution and dene J = {i | zi

maximize bT z dT y subject to AT z + C T y + c = 0 z0

x is optimal i it is feasible and complementary slackness with z holds: aT i x = bi for i J, aT i x bi for i J, Cx = d

conclusion: optimal set is a face of the polyhedron {x | Ax b, Cx = d}

Duality 622

Strict complementarity
primal and dual optimal solutions are not necessarily unique any combination of primal and dual optimal points must satisfy z i ( bi a T i x ) = 0, in other words, for all i, aT i x < bi , z i = 0 or aT i x = bi , z i > 0 or aT i x = bi , z i = 0 i = 1, . . . , m

primal and dual optimal points are strictly complementary if for all i aT i x < bi , z i = 0 or aT i x = bi , z i > 0

it can be shown that strictly complementary solutions exist for any LP with a nite optimal value (exercise 72)
Duality 623