Detection: chapter 3 Statistical detection theory I

Natasha Devroye devroye@ece.uic.edu http://www.ece.uic.edu/~devroye Spring 2010

Example of detection

Example of estimation

Goals

infer value of unknown state of nature based on noisy observations

Mathematically, optimally
Noise
Nature Transmission / measurement Processing

phenomenon experiment the ``truth

model of observation or transmission process

decision rule estimation function

model of hypothesis H

mapping from observation space to decisions/ estimates

Detection example 1: digital communications

Noise

Source

Encoder

Channel

Decoder

Destination

10001010100010

Detect?

Detection example 2: Radar communication

Send Receive
Hypothesis

Detect?

Hypothesis

Further examples
Sonar: enemy submarine! Image processing: detect and aircraft from infrared images Biomedicine: cardiac arryhthmia from heartbeat sound wave Control: detect occurrence of abrupt change in system to be controlled Seismology: detect presence of oil deposit

Difference between detection and estimation?

Detection:

Discrete set of hypotheses Right or wrong

Estimation:

Continuos set of hypotheses Almost always wrong - minimize error instead

Estimation example 1: communications

Pulse amplitude modulation (PAM)

Analog source

Sampler

Transmitter

Receiver?

Estimation example 2: Radar

Send Receive
Hypothesis

Estimate?
Hypothesis

Our methods
Will treat everything generally, with a unied mathematical representation Bias towards Gaussian noise Examples mainly drawn from communications / radar

Aside: Classical vs. Bayesian Classial

Hypotheses/parameters are xed, non-random

Bayesian
Hypotheses/parameters are treated as random variables with assumed priors (or a priori distributions)

Course outline

Other useful references: Course Textbook: Fundamentals of Statistical Signal Processing, Volume III, IV Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, 1: Estimation Theory, by Steven M. Kay, Prentice Hall, Signal Detection and Estimation H. Vincent Poor, Introduction to 1993 and (possibly) Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998. Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time Series Analysisreferences: Other useful Carl Helstrom, Trees, Detection, Estimation, and Estimation. It's out ofPart I, so here's my pdf copy. Harry L. Van Elements of Signal Detection Modulation Theory, print, II, III, IV H. Vincent Poor, Introduction to Signal Detection and Estimation Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlightingand Time Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, the main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I Series Analysis will post a pdf version of the slides as they become ready here, but the derivations will be my pdf copy. Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's given in class only. Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M.of slides (highlighting the Notes: I will follow the course textbooks fairly closely, using a mixture Kay, Prentice Hall, 1993 Topics: Estimation Theory: illustrations) and more in-depth blackboard derivations/proofs in class. I main points and with nice General Minimum Variancethe slides as they become ready here, but the derivations will be given in will post a pdf version of Unbiased Estimation, Ch.2, 5 Cramer-Rao Lower Bound, Ch.3 class only. Linear Models+Unbiased Estimators, Ch.4, 6 Maximum Likelihood Estimation, Ch.7 Topics: Estimation Theory: Least squares estimation, Ch.8Unbiased Estimation, Ch.2, 5 General Minimum Variance Bayesian Estimation, Bound, Ch.3 Cramer-Rao Lower Ch.10-12 Detection Theory: Linear Models+Unbiased Estimators, Ch.4, 6 Statistical Detection Theory, Ch.3 Ch.7 Maximum Likelihood Estimation, Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998. Deterministic Signals, Ch.4 Least squares estimation, Ch.8 Random Signals, Ch.5 Bayesian Estimation, Ch.10-12 Statistical Detection Theory 2, Ch.6 Detection Theory: Non-parametric and robust detection Statistical Detection Theory, Ch.3 Deterministic Signals, Ch.4 Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 = Random Signals, Ch.5 max(Exam 2, Final) (20%), Project (15%), Final exam (30%). Statistical Detection Theory 2, Ch.6 Non-parametric and robust detection

1 of 3

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 = 1/11/10 8:50 PM max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

1 of 3

1/11/10 8:50 PM

Estimation: General Minimum Variance Unbiased Estimation

Bias: (expected value of estimator - true value of data)

MVUE:

Estimation: Cramer-Rao lower bound

Lower bound on variance of ANY unbiased estimator! Usage: assert whether an estimator is MVUE benchmark against which to measure the performance of an unbiased estimator feasibility studies Depends on?

Noise
Transmission / measurement Processing

Nature

Estimation: linear models

Whats a linear model and why is it useful?

What can be said? Best Linear Unbiased Estimators (BLUE)

Estimation: Maximum Likelihood Estimation

Alternative to MVUE which is hard to nd in general Easy to compute - very widely used and practical What is the MLE?

Properties?

Estimation: Least Squares

Alternative estimator with no general optimality properties, but nice and intuitive and no probabilistic assumptions on data are made - only need a signal model

Advantages? Disadvantages?

Estimation: Bayesian Estimation

Parameter to be estimated is assumed to be random, according to some prior distribution which models our knowledge of it Bayesian Minimum Mean Squared Error (MMSE):

Applications to Gaussian noise / linear model

Estimation: Bayesian Estimation

General risk functions - arbitrary cost functions

Maximum a posteriori (MAP) estimation

Linear MMSE: constrain estimator to be linear - very practical

Course outline

Other useful references: Course Textbook: Fundamentals of Statistical Signal Processing, Volume III, IV Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, 1: Estimation Theory, by Steven M. Kay, Prentice Hall, Signal Detection and Estimation H. Vincent Poor, Introduction to 1993 and (possibly) Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998. Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time Series Analysisreferences: Other useful Carl Helstrom, Trees, Detection, Estimation, and Estimation. It's out ofPart I, so here's my pdf copy. Harry L. Van Elements of Signal Detection Modulation Theory, print, II, III, IV H. Vincent Poor, Introduction to Signal Detection and Estimation Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlightingand Time Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, the main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I Series Analysis will post a pdf version of the slides as they become ready here, but the derivations will be my pdf copy. Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's given in class only. Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M.of slides (highlighting the Notes: I will follow the course textbooks fairly closely, using a mixture Kay, Prentice Hall, 1993 Topics: Estimation Theory: illustrations) and more in-depth blackboard derivations/proofs in class. I main points and with nice General Minimum Variancethe slides as they become ready here, but the derivations will be given in will post a pdf version of Unbiased Estimation, Ch.2, 5 Cramer-Rao Lower Bound, Ch.3 class only. Linear Models+Unbiased Estimators, Ch.4, 6 Maximum Likelihood Estimation, Ch.7 Topics: Estimation Theory: Least squares estimation, Ch.8Unbiased Estimation, Ch.2, 5 General Minimum Variance Bayesian Estimation, Bound, Ch.3 Cramer-Rao Lower Ch.10-12 Detection Theory: Linear Models+Unbiased Estimators, Ch.4, 6 Statistical Detection Theory, Ch.3 Ch.7 Maximum Likelihood Estimation, Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998. Deterministic Signals, Ch.4 Least squares estimation, Ch.8 Random Signals, Ch.5 Bayesian Estimation, Ch.10-12 Statistical Detection Theory 2, Ch.6 Detection Theory: Non-parametric and robust detection Statistical Detection Theory, Ch.3 Deterministic Signals, Ch.4 Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 = Random Signals, Ch.5 max(Exam 2, Final) (20%), Project (15%), Final exam (30%). Statistical Detection Theory 2, Ch.6 Non-parametric and robust detection

1 of 3

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 = 1/11/10 8:50 PM max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

1 of 3

1/11/10 8:50 PM

Detection: Statistical Detection Theory

Binary hypothesis testing

Hypothesis

Hypothesis

Detection: Statistical Detection Theory

Binary hypothesis testing

Detection: Statistical Detection Theory

Detection: Deterministic Signals

How to detect known signals in noise?

The famous matched lter!

T(x) x[n] X s[n]

Generalized matched lter > 2 hypotheses

Detection: Random Signals

What if s[n] is random?

Key idea behind estimator-correlator:

Linear model simplies things again...

Detection: Statistical Decision Theory II

model for the pdfs under 2 hypotheses are unknown

Uniformly most powerful test Generalized likelihood ratio test Bayesian approach Wald test Rao test

Detection Theory

Binary hypothesis testing

Binary hypothesis testing

Neyman-Pearson hypothesis testing

Example 1

Useful problem 2.1

Example 2

Example 3

Deection coefcient

Receiver Operating Characteristics (ROC)

[different terminology - can you map it to ours?]

Bayesian risk

Bayesian risk

Bayesian risk example

Multiple hypothesis testing

Multiple hypothesis testing

Multiple hypothesis testing example