Documente Academic
Documente Profesional
Documente Cultură
7. Statistical estimation
experiment design
71
Parametric distribution estimation
y is observed value
l(x) = log px(y) is called log-likelihood function
can add constraints x C explicitly, or define px(y) = 0 for x 6 C
a convex optimization problem if log px(y) is concave in x for fixed y
Statistical estimation 72
Linear measurements with IID noise
yi = aTi x + vi, i = 1, . . . , m
(y is observed value)
Statistical estimation 73
examples
2
/(2 2 )
Gaussian noise N (0, 2): p(z) = (2 2)1/2ez ,
m
m 2 1 X T
l(x) = log(2 ) 2 (ai x yi)2
2 2 i=1
ML estimate is LS solution
Laplacian noise: p(z) = (1/(2a))e|z|/a,
m
1X T
l(x) = m log(2a) |ai x yi|
a i=1
Statistical estimation 74
Logistic regression
random variable y {0, 1} with distribution
exp(aT u + b)
p = prob(y = 1) =
1 + exp(aT u + b)
k
X m
X
= (aT ui + b) log(1 + exp(aT ui + b))
i=1 i=1
concave in a, b
Statistical estimation 75
example (n = 1, m = 50 measurements)
0.8
prob(y = 1)
0.6
0.4
0.2
0 2 4 6 8 10
u
Statistical estimation 76
(Binary) hypothesis testing
randomized detector
Statistical estimation 77
detection probability matrix:
1 Pfp Pfn
D= Tp Tq =
Pfp 1 Pfn
variable T R2n
Statistical estimation 78
scalarization (with weight > 0)
minimax detector
Statistical estimation 79
example
0.70 0.10
0.20 0.10
P =
0.05
0.70
0.05 0.10
0.8
0.6
Pfn
0.4
1
0.2
2 4
3
0
0 0.2 0.4 0.6 0.8 1
Pfp
m
!1 m
X X
x = aiaTi yi a i
i=1 i=1
T 1
Pp
Sn+)
minimize (w.r.t. E= k=1 mk vk vk
subject to mk 0, m1 + + mp = m
mk Z
T 1
Pp
Sn+)
minimize (w.r.t. E = (1/m) k=1 k vk vk
subject to 0, 1T = 1
T 1
Pp
minimize log det k=1 k vk vk
T
subject to 0, 1 =1
k (1 vkT W vk ) = 0, k = 1, . . . , p
2 = 0.5
p
!!
X
L(X, , Z, z, ) = log det X 1+tr Z X k vk vkT z T +(1T 1)
k=1
dual problem
change variable W = Z/, and optimize over to get dual of page 713