Digital Signal Processing (ET 4235) Chapter 4: Signal Modeling

Given a signal (set of samples), how can it be modeled using a lter?

error Original (bmp, 780 kB)

1

Parametrized model (jpg 10%, 5 kB)

4 Signal Modeling September 19, 2011

Signal ModelingMotivation
Motivation 1: Efcient transmission/storage
2 1 1 0 1 0 1 0 1 2 1 0 0 5 10 15 time [n] 20 25 30 1 0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30

Direct: store all samples Coded: Model the signal, e.g., sum of sinusoids: Estimate the parameters of the signal,
          

Store the parameters instead of the original samples. Example: GSM speech coding, MP3 audio coding, JPEG image coding, 2
4 Signal Modeling

September 19, 2011

Motivation
Motivation 2: Interpolation/extrapolation

Interpolation/extrapolation requires a model, e.g., bandlimited/lowpass, sum of sinusoids, etc.

1
0.5

0
0

1
0.5

4 time [n]

10

10

15 time [n]

20

25

30

4 Signal Modeling

September 19, 2011

Motivation
Extrapolation of Lena

Horizontal extrapolation:

(In fact, there are stationarity requirements that would prohibit this. . . ) 4
4 Signal Modeling September 19, 2011

Motivation
Example: ionospheric calibration Objects in field of view
see different ionospheric phase and gain Ionosphere
ionosphere phase screen (time varying)

Station beam field of view

geometric delays

Full array aperture

station beamformers

LOFAR station

LOFAR station

x1 (t)

xJ (t)

The ionosphere causes small delays in the reception of signals that modify the apparent direction of astronomical sources. Low frequency radio telescopes can sample the ionosphere in the direction of calibration sources. For other directions, we rely on interpolation. A simple ionospheric model species correlations in phase delay distance between points:
     

as function of

4 Signal Modeling

September 19, 2011

Motivation
Example: Interpolation of correlated variables

Suppose we have samples of random variables

that are correlated. Given a new sample , can we predict the corresponding ?

Pose the model:

, and estimate

by minimizing

If we stack the measured samples in vectors and , then we obtain the estimate
 

(cf. the Wiener lter in Ch. 7.) This is the solution of the Least Squares problem
 

  

     


4 Signal Modeling

September 19, 2011

Motivation
Motivation 3: Filter design

Design a lter that approximates a desired response (e.g., ideal lowpass)

1.4
0.25

N=20
N=20 0.2

1.2 1

0.15 0.1 h[n] 0.05 0 0.05 0.1 10

|H()

0.8 0.6 0.4 0.2 0 1

10 n

15

20

25

30

0.5

0 /

0.5

The design depends on the lter model (FIR, IIR), lter order, error criteria, etc.

4 Signal Modeling

September 19, 2011

Signal models
Signal models

standard input Deterministic: impulse (or known signal)

signal model (lter)

modeled signal observations

Stochastic:

white noise

Models for

Special cases: AR( ), MA( )

 

 %

  $ 


ARMA(

):

 ! "


 

 



&




 %

$ '


4 Signal Modeling

September 19, 2011

Signal models

Model identication

Given observations eters of




and lter order



, nd the param-

such that the modeled output signal



best matches the

observations. For stochastic signals, we will try to match the correlation sequences: .

 

Issues: stability of the lter (in case )

computational complexity for parameter estimation model order selection error criterion for the approximation

4 Signal Modeling

September 19, 2011

Model identication via Least Squares

In the next slides, we consider deterministic input signals (impulse is a lter : LTI, causal, rational. The desired signal has


). The model

Minimize the error depends on the denition of error, and the norm. Least squares:
 !  

The minimization of requires

The resulting

error signal



"

with

P P

" " ) ( )0 ( ' ' 4&3 %&$ 2 # " ! #"

   



equations are nonlinear because of the division by 10

4 Signal Modeling

 

 5 6
$ '


  

5 
  

6
$ #

.


September 19, 2011

Model identication via Least Squares

Alternative that leads to tractable results: consider weighted error

 


   

 

 

 

 

 

modied error signal

Now

is in the nominator, the error equation is linear.



Techniques that are based on this: Pade Approximation, Pronys Method, Shanks Method.

 

11

4 Signal Modeling

September 19, 2011

Model identication via Pade Approximation

Pade approximation

We have

model parameters: Can match

  

signal samples exactly:

How to nd









in terms of the parameters:

 


   


 


   

 

 

 

'




  &


$ ' #




where

, and for

or :

Match exactly with



&


#
  

$ '





12

4 Signal Modeling

September 19, 2011

Model identication via Pade Approximation

Write these equations in matrix form:

. . .

. . .

. . .

. . .

.. . .. ... . .. ... .

Take the submatrix that does not involve the

This is a square The solution is



 
 

. . .

. . .

. . . . . .

. . .

, and rst solve for the

'

..

..

.. ... ... .

  %

. . .

. . .

. . .

matrix equation: .

  

13

4 Signal Modeling

September 19, 2011

Model identication via Pade Approximation

Plug the solution back to nd the

Disadvantages of the Pade approximation

The resulting model doesnt have to be stable Outside the used interval

. . .

. . .

. . .

.. ... .

. . .

. . .

, the approximation may be very bad

 "



If

is singular, then a solution does not always exist that matches all signal ,
 

values. (Sometimes a solution exists of lower order for



 

, i.e., smaller


, but then no guarantee on matching for samples beyond 14

4 Signal Modeling

 

 

.)

September 19, 2011


  

$


to nd

 




Model identication via Pronys Method

..

$


$


, i.e.,



$


$ ' &

  ' &

 ! !

..

Derivation of Pronys Method

For Pade, we rst solved




 

  

 

1 (
!

Prony: solve

In matrix form:



'

..

 

. . .

As before,

. . .

 

. . .

.. ... . .. .

( '

15

 

..

.. ... . .. .

 

. . .

& !

'

4 Signal Modeling

. . .

. . .

September 19, 2011

Model identication via Pronys Method

... .. ... ... . .. ... . .. .

..

..

. . .

. . .

. . .

.. ... . .. .

. . .

. . .

. . .


  

 

1


(Now,

refers to the innite-dimensional matrix.)

This is a Least-Squares problem of an overdetermined system of equations. The solution is .

 % %
   

This assumes that In practice,



 

 

is not singular.

and

are of nite size as determined by the available data. 16

4 Signal Modeling September 19, 2011

Model identication via Pronys Method

matrix


 

is positive (semi)denite by construction. We will see later that this makes (marginally) stable.


 

If

is singular, then this indicates that the lter order can be reduced. is known, we nd

After

This makes

Alternatively, we can nd the numerator by minimizing record. For the error criterion based on Shanks method switches back to


(i.e.,
  

) precisely as in Pades method:



 

 "

. over the entire data

, this does not make a difference.

and minimizes over the entire data.

17

4 Signal Modeling

September 19, 2011

Example: lter design

Suppose we want to design an ideal linear phase lowpass lter:
%

is the lter delay. The corresponding impulse response is

 

otherwise

We will match

values, and choose

Two cases: FIR lter (





Ideal and truncated filter impulse response 0.4 0.2 0 0.2 ideal truncated

10 n

15

), ARMA lter (

).



18

4 Signal Modeling

September 19, 2011

Example: lter design

Pade approximation

Filter coefcients to match:

   

FIR lter (

):

ARMA lter (

and subsequently the numerator coefcients are found as



(Use Matlab

 

 

): solving the Pade equations for

 

otherwise gives



 

to nd the design) 19
4 Signal Modeling September 19, 2011

     ! " $ %   #

Example: lter design

Ideal filter impulse response; design p=0,q=10 0.2 0.4 0.2 0 0.2 ideal p=0,q=10 0.1 0 0.1 0 10 40 50 60 70 n Ideal filter impulse response; design p=q=5 ideal p=q=5 20 30 80 0.2 0 40 50 60 70 80 n Difference of ideal filter impulse response with design p=q=5 10 20 30 Difference of ideal filter impulse response with design p=0,q=10

0.2 0.4 0.2 0 0.2 0.1 0 0.1 0 10 20 30 40 n 50 60 70 80 0.2 0 10 20 30 40 n 50 60 70 80

The ARMA(5,5) lter gives a larger error outside the specied interval. Also, this lter is not linear phase (no symmetry).

20

4 Signal Modeling

September 19, 2011

Example: lter design

Filterdesign using Pade Approximation 10 0 10 magnitude [dB] 20 30 40 50 60 70 ideal p=0,q=10 p=q=5

pi/4

pi/2

3pi/4

pi

The ARMA(5,5) lter does not have a good frequency response in the passband. 21

4 Signal Modeling

September 19, 2011

Example: lter design

Filterdesign using Pade and Prony Approximation 10 0 10 magnitude [dB] 20
0.2 Ideal filter impulse response; Prony design p=q=5

ideal pade, p=q=5 prony, p=q=5 elliptic

0.4 0.2 0

ideal Prony, p=q=5

30 40 50 60 70

40 50 60 70 80 n Difference between ideal filter impulse response and Prony design p=q=5 0.2 0.1 0

10

20

30

0.1

pi/4

pi/2

3pi/4

pi

0.2

10

20

30

40 n

50

60

70

80

The ARMA(5,5) design using Pronys method is much better than the Pade approximation, since is designed to minimize the error over the entire domain


A specialistic design (elliptic lter of 5th order) can still be better, with full control over the passband error and stopband attenuation.

 

22

4 Signal Modeling

September 19, 2011

Model identication via Pronys Method

Alternative writing of the equations

Recall: The solution is

with
 

 %
 

 




Thus, the entries of

and

are
 6
 

 6

The equation . . .

The entries of

cussed later), we arrive at very similar equations. 23

4 Signal Modeling September 19, 2011

$

  

 1 1

can be written as



 



   




'

  

'

..

 

.
  

. . .



. . .


. . .



'



are considered autocorrelations. In a stochastic context (dis-

Model identication via Pronys Method

These equations can also be summarized as
&

  

$ 

$ 

'



These are called the Prony normal equations.

Yet another writing form:

&
  

$ $ $


'

 

. . .

. . .

..



.
  

. . .



'

. . .


. . .

   

   

  

'

'

24

4 Signal Modeling

September 19, 2011

Model identication via Pronys Method

Orthogonality principle

Consider the Least Squares problem is



. After solving, the error vector




 

The orthogonality principle states, in general, that the error vector for the optimal is orthogonal to all the columns of , i.e., colspan


we could take a linear combination of these columns to reduce the error!

  
    



column span of

. If it was not the case,



 

25

4 Signal Modeling

September 19, 2011

Model identication via Pronys Method

The minimum error

At the minimum, the total error is (due to the orthogonality principle)


  

This can further be written as

  
  



&

 

In terms of the autocorrelation sequence,

 6
  

&

1  6

this can be written as

&

 1

 1

'

 

& !

26

4 Signal Modeling

September 19, 2011

Model identication via Pronys Method

This can be combined with the equation


. . .

as
'


 

 

&

   

  

'

..

. . .
  

. . .


. . .

'

 

 





or also as
&

&

where tions.

is a unit vector. These are the augmented normal equa-

27

4 Signal Modeling

September 19, 2011

All-pole modeling
Special case: all-pole modeling

If

, then we have an all-pole model:

#

Recall that the solution is given by solving the overdetermined system (

  

In some cases, this is an accurate physical model (e.g., speech) Even if it is not a valid physical model, it is attractive because it leads to fast computational algorithms to nd (the Levinson algorithm)

 

&

$ '

'

'

..

. .

..

. . .

. . .

. . .

. . .

. . . 28

. . .


. . . . . .

' '

'

4 Signal Modeling

September 19, 2011

All-pole modeling
The normal equations become (premultiply with ..

)
"

 

..

..

. . .

..

. . .

..

..

. . .

. . . . . .

..

..

..

6
 

'

From this structure it is seen that the normal equations become

6 6
 
  

where the autocorrelation sequence is in fact stationary:

 6
   

. . .

. . .

6
 

'

..
 

.
 
  

. . .

. . .


'

 

6
 
  

'

. . .

'

$


29


"

4 Signal Modeling

September 19, 2011

All-pole modeling

For an all-pole model, the matrix

has a Toeplitz structure (constant along

diagonals); it can be efciently inverted using the Levinson algorithm (later in Ch.5).

The numerator will be chosen as

. It can be shown that then

&

Meaning: the autocorrelation sequence of the lter matches the rst

autocorrelation sequence of the specied data (moment matching).



lags of the

30

4 Signal Modeling

September 19, 2011

All-pole modeling with nite data

Suppose we have only
Autocorrelation method

samples,

. What changes?

Here, the entire data set is used, and extended with zeros: . . .

..

. . . .

.. ..

..

. . .

. . .


. . .

:


We will have the same normal equations as before, but with a different . 31 . . .


'

. . .

'

6

  

$

  

$ 

. . .

. . .
 
  

. . .

September 19, 2011

6
 
  

'

'

4 Signal Modeling

All-pole modeling with nite data

Covariance method

If we know that

is not zero for

and

, it is more accurate to omit

those equations that contain an extension with zeros: .. . .. . . . . . .. . . . . . . .. .

 

We will have slightly different normal equations as before, with .

 

. . .

dened as


'



'

$ 


  

The Toeplitz property is lost. 32

4 Signal Modeling September 19, 2011

$


&

. . .

. . .
  

. . .


. . .




'




  

'






Example: pole estimation

Consider the nite data sequence for this signal ( ):
#

. Estimate an AR(1) model

  


  

Autocorrelation method

The normal equations collapse to

 

 '

, where

'

  


  

. . .

 

  


"

. . .

%
  

  


"

'


 

33

4 Signal Modeling

September 19, 2011

Example: pole estimation

Covariance method

The normal equations collapse to

, where


'

%
  

  

  

. . .
%




"

%

  

. . .

%
  




 


"

Thus,

. Set

, then

The pole location is found exactly. 34

4 Signal Modeling September 19, 2011

'

 %


 


Example: channel inversion



Consider a communication channel with a known (estimated) transfer function At the receiver, we wish to equalize (invert) the channel using an equalizer

Further If


is not possible, and we allow for a delay:

%

 

.


:


 

 


 

must be causal and stable, typically FIR.



is not minimum-phase (has zeros outside the unit circle), causal inversion


 

 


 

35

4 Signal Modeling

September 19, 2011

Example: channel inversion

Design of an FIR equalizer

of length


:
%
  

 

! "

Minimize

: .
%
 

%


 !


 !

 

The LS solution of this overdetermined system Also, the corresponding normal equations are . . . . . . . . .

 

"

"

"

. . . . . .


.. . .. . .. . .. .

. . .

. . .


. . .

'


(


. . .

is

, or . . .

 !


 

 

 

where matrix

and

. Because the

 !

6


6


is Toeplitz, it can be inverted efciently. 36

4 Signal Modeling September 19, 2011

"

"