Lecture: Examples of LP, SOCP and SDP

Zaiwen Wen

Beijing International Center For Mathematical Research

Peking University

http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html
wenzw@pku.edu.cn

Acknowledgement: this slides is based on Prof. Farid Alizadeh and Prof. Lieven
Vandenberghes lecture notes

1/31
Problems with absolute values
X
min ci |xi |, assume c 0
i
s.t. Ax b

Reformulation 1:
X X
min ci zi min ci zi
i i

s.t. Ax b s.t. Ax b
|xi | zi zi xi zi

Reformulation 2: xi = xi+ xi , xi+ , xi 0. Then |xi | = xi+ + xi

X
min ci (xi+ + xi )
i
s.t. Ax+ Ax b, x+ , x 0
2/31
Problems with absolute values

data fitting:
min kAx bk
x

min kAx bk1

x

Compressive sensing

min kxk1 , s.t. Ax = b (LP)

1
min kxk1 + 2 kAx + bk2 (QP, SOCP)
min kAx bk, s.t. kxk1 1

3/31
Quadratic Programming (QP)

min q(x) = x> Qx + a> x + assume Q  0, Q = Q>

s.t. Ax = b
x0

q(x) = kuk2 + 14 a> Q1 a, where u = Q1/2 x + 21 Q1/2 a.

equivalent SOCP

min u0
1
s.t. u = Q1/2 x + Q1/2 a
2
Ax = b
x 0, (u0 , u) Q 0

4/31
Quadratic constraints

q(x) = x> B> Bx + a> x + 0

is equivalent to
(u0 , u) Q 0,
where !
Bx 1 a> x
u = a> x++1 and u0 =
2
2

5/31
Norm minimization problems

Let vi = Ai x + bi Rni .
P
minx i kvi k is equivalent to
X
min vi0
i
s.t. vi = Ai x + bi
(vi0 , vi ) Q 0

minx max1ir kvi k is equivalent to

min t
s.t. vi = Ai x + bi
(t, vi ) Q 0

6/31
Norm minimization problems

Let vi = Ai x + bi Rni and k r

kv[1] k, . . . , kv[r] k are the norms kv1 k, . . . , kvr k sorted in
nonincreasing order
Pk
minx i=1 kv[i] k is equivalent to

r
X
min ui + kt
i=1
s.t. vi = Ai x + bi , i = 1, . . . , r
kvi k ui + t, i = 1, . . . , r
ui 0, i = 1, . . . , r

7/31
Rotated Quadratic Cone

rotated cone w> w xy, where x, y 0, is equivalent to

 
2w
xy x+y

Minimize the harmonic mean of positive affine functions

X
min 1/(a>
i x + i ), s.t. a>
i x + i > 0
i

is equivalent to X
min ui
i
s.t. vi = a>
i x + i
1 ui vi
ui 0
8/31
Logarithmic Tchebychev approximation

min max | ln(a>

i x) ln bi |
x 1ir

Since | ln(a> > >

i x) ln bi | = ln max(ai x/bi , bi /ai x), the problem is
equivalent to
min t
s.t. 1 (a>
i x/bi )t
a>
i x/bi t
t0
geometric means of nonnegative affine functions:

n
!1/n
Y
(a>
i x + bi ) t
i=1

9/31
n=4
max w3
s.t. a>
i x bi 0
4
Y (a> > 2
1 x b1 )(a2 x b2 ) w1
max (a>
i x bi )
i=1 (a> > 2
3 x b3 )(a4 x b4 ) w2
w1 w2 w23
wi 0

This can be extended to products of rational powers of affine

functions

10/31
Robust linear programming
the parameters in LP are often uncertain

min c> x
s.t. a>
i x bi

There can be uncertainty in c, ai , b.

two common approaches to handling uncertainty (in ai , for simplicity)

deterministic model: constraints must hold for all ai Ei
min c> x
s.t. a>
i x bi , for all ai Ei

stochastic model: ai is random variable; constraints must hold

with probability

min c> x
s.t. prob(a>
i x bi )
11/31
deterministic approach via SOCP

Choose an ellipsoid as Ei :

Ei = {ai + Pi u | kuk2 1}, ai Rn , Pi Rnn

Robust LP
min c> x
s.t. a>
i x bi , for all ai Ei

is equivalent to the SOCP

min c> x
s.t. a> >
i x + kPi xk2 bi

since
sup (ai + Pi u)> x = a> >
i x + kPi xk2
kuk2 1

12/31
stochastic approach via SOCP

ai is Gaussian with mean ai , covariance i ( ai N (ai , i ))

a> > >

i x is Gaussian r.v. with mean ai x, variance x i x; hence

bi a>
 
i x
prob(a>
i x bi ) =
k1/2 xk2
Rx 2
where (x) = (1/ 2) et /2 dt is CDF of N (0, 1)

robust LP
min c> x
s.t. prob(a>
i x bi )

is equivalent to the SOCP

min c> x
s.t. a> 1
i x + ()k
1/2
xk2 bi
13/31
SDP Standard Form
S n = {X Rnn | X > = X}, S+
n = {X S n | X  0},
n = {X S n | X  0}
S++

Define linear operator A : S n Rm :

>
A(X) = hA1 , Xi , . . . , hAm , Xi , X S n .

Since A(X)> y = m
P Pm
i=1 yi hAi , Xi = h i=1 yi Ai , Xi, the adjoint of A:

m
X

A (y) = yi Ai
i=1

The SDP standard form:

(P) min hC, Xi (D) max b> y
s.t. A(X) = b s.t. A (y) + S = C
X0 S0
14/31
Facts on matrix calcuation

If A, B Rmn , then Tr(AB> ) = Tr(B> A)

If U, V S n and Q is orthogonal, then hU, Vi = Q> UQ, Q> UQ



If X S n , then U = Q> Q, where Q> Q = I and is diagonal.

Matrix norms: kXkF = k(X)k2 , kXk2 = k(X)k , (X) = diag()

X  0 v> Xv for all v Rn (X) 0 X = B> B

n is S n
The dual cone of S+ +

If X  0, then Xii 0. If Xii = 0, then Xik = Xki = 0 for all k.

If X  0, then PXP>  0 for any P of approriate dimensions

 
X11 X12
If X = > X  0, then X11  0.
X12 22

X  0 iff every principal submatrix is positive semidefinite (psd).

15/31
Facts on matrix calcuation
 
A B
Let U = with A and C symmetric and A  0. Then
B> C

U  0 ( or  0) C B> A1 B  0 ( or  0).

The matrix C B> A1 B is the Schur complement of A in U:

I A1 B
     
A B I 0 A 0
=
B> C B> A1 I 0 C B> A1 B 0 I

If A S n , then x> Ax = A, xx>



If A  0, then hA, Bi > 0 for every nonzero B  0 and

{B  0 | hA, Bi } is bounded for > 0

If A, B  0, then hA, Bi = 0 iff AB = 0

A, B S n , then A and B are commute iff AB is symmetric, iff A

and B can be simultaneously diagonalized
16/31
Eigenvalue optimization

P
minimizing the largest eigenvalue max (A0 + i xi Ai ):
X
min max (A0 + xi Ai )
i

can be expressed as an SDP

and its dual is
min z max hA0 , Yi
hAi , Yi = k
X
s.t. zI xi Ai  A0 s.t.
i hI, Yi = 1
Y0
follows from
max (A) t A  tI

17/31
Eigenvalue optimization

Let Ai Rmn . Minimizing the 2-norm of A(x) = A0 +

P
i xi Ai :

min kA(x)k2
x

can be expressed as an SDP

min t
x,t
 
tI A(x)
s.t. 0
A(x)> tI

Constraint follows from

kAk2 t A> A  t2 I, t 0
 
tI A(x)
0
A(x)> tI

18/31
Eigenvalue optimization

Let k (A) indicate sum

P of the k largest eigenvalues of A. Then
minimizing k (A0 + i xi Ai ):
X
min k (A0 + xi Ai )
i

can be expressed as an SDP

min kz + Tr(X)
X
s.t. zI + X xi Ai  A0
i
X0
since

k (A) t t kz Tr(X) 0, zI + X  A, X  0

19/31
The following problems can be expressed as SDP
P
maximizing sum of the k smallest eigenvalues of A0 + i xi A i

minimizing sum of the k absolute-value-wise largest eigenvalues

P
minimizing sum of the k largest singular values of A0 + i xi Ai

20/31
Quadratically Constrained Quadratic Programming
Consider QCQP

min x> A0 x + 2b>

0 x + c0 assume Ai S n
s.t. x> Ai x + 2b>
i x + ci 0, i = 1, . . . , m

If A0  0 and Ai = B>
i Bi , i = 1, . . . , m, then it is a SOCP

If Ai S n but may be indefinite

D E
>
x Ai x + 2b>
i x + ci = Ai , xx >
+ 2b>
i x + ci

The original problem is equivalent to

min TrA0 X + 2b>

0 x + c0
s.t. TrAi X + 2b>
i x + ci 0, i = 1, . . . , m
>
X = xx
21/31
QCQP

If Ai S n but may be indefinite

   
> > Ai bi X x

x Ai x + 2bi x + ci = > , > := Ai , X
bi ci x 1

X  0 is equivalent to X  xx>

The SDP relaxation is

min TrA0 X + 2b>

0 x + c0
s.t. TrAi X + 2b>
i x + ci 0, i = 1, . . . , m
>
X  xx

Maxcut: min x> Wx, s.t. xi2 = 1

Phase retrieval: |a> >

i x| = bi , the value of ai x is complex

22/31
SDP-Representablity

What kind of problems can be expressed by SDP and SOCP?

Definition: A set X Rn is SDP-representable (or SDP-Rep for
short) if it can be expressed linearly as the feasible region of an
SDP

X = x | there exist u Rk such that for some



X X
mm
Ai , Bj , C R : xi Ai + uj Bj + C  0

i j

23/31
SDP-Representablity

Definition: A function f (x) is SDP-Rep if its epigraph

epi(f ) = {(x0 , x) | f (x) x0 }

is SDP-representable

If X is SDP-Rep, then minxX c> x is an SDP

If f (x) is SDP-Rep, then minx f (x) is an SDP

24/31
A calculusof SDP-Rep sets and functions

SDP-Rep sets and functions remain so under finitely many

applications of most convex-preserving operations.
If X and Y are SDP-Rep then so are
Minkowski sum X + Y

intersection X Y

Affine pre-image A1 (X) if A is affine

Affine map A(X) if A is affine

Cartesian Product: X Y = {(x, y) | x X, y Y}

25/31
SDP-Rep Functions

If functions fi , i = 1, . . . , m and g are SDP-Rep. Then the following are

SDP-Rep
P
nonnegative sum i i fi for i 0

maximum maxi fi

composition: g(f1 (x), . . . , fm (x)) if fi : Rn R and g : Rm R

Legendre transform

f (y) = max y> x f (x)

x

26/31
Positive Polynomials

The set of nonnegative polynomials of a given degree forms a

proper cone

Pn = {(p0 , . . . , pn ) | p0 + p1 t + . . . + pn tt > 0 for all t I}

where I is any of [a, b], [a, ) or (, )

Important fact: The cone of positive polynomials is SDP-Rep

To see this we need to introduce another problem

27/31
The Moment cone

The Moment space and Moment cone

Let (c0 , c1 , . . . , cn )> be such that there is a probability measure F
where Z
ci = ti dF, for i = 0, . . . , n.
I
The set of such vectors is the Moment space

The Moment cone

Z
Mn = {ac | There is a distribution F : ci = ti dF and a 0}
I

28/31
The Moment cone

The moment cone is also SDP-Rep:

The discrete Hamburger moment problem:

I = R, c M2n+1

c0 c1 . . . cn
c1 c2 . . . cn+1
..  0

.. .. ..
. . . .
cn cn+1 . . . c2n
This is the Hankel matrix

29/31
The Moment cone

The discrete Stieltjes moment problem

I = [0, ), c Mm

c0 c1 ... cn c1 c2 . . . cn1
c1 c2 ... cn+1 c2 c3 . . . cn
 0, and ..  0

.. .. .. .. .. .. . .
. . . . . . . .
cn cn+1 . . . c2n cn1 cn . . . c2n1
where m = b n2 c

The Hausdorff moment problem where I = [0, 1] is similarly

SDP-Rep

30/31
Moment and positive polynomial cones

Pn = Mn , i.e., i.e. moment cones and nonnegative polynomials

are dual of each other

If {u0 (x), . . . , un (x)}, x I are linearly independent functions

(possibly of several variables)

The cone of polynomials that can be expressed as sum of

squares is SDP-Rep.

if I is a one dimensional set then positive polynomials and sum of

square polynomials coincide

In general except for I one-dimensional positive polynomials

properly include sum of square polynomials

31/31