Transform Domain LMF Algorithm for Sparse

System Identification Under Low SNR

Murwan Bashir and Azzedine Zerguine

AbstractIn this work, a transform domain Least Mean

Fourth (LMF) adaptive filter for a sparse system identification, in
the case of low Signal-to-Noise Ratio (SNR), is proposed. Unlike
the Least Mean Square (LMS) algorithm, the LMF algorithm,
because of its error nonlinearity, performs very well in these
environments. Moreover, its transform domain version has an
outstanding performance when the input signal is correlated.
However, it lacks sparse information capability. To overcome this
limitation, a zero attractor mechanism, based on the l1 norm
is implemented to yield the Zero-Attractor Transform-Domain
LMF (ZA-TD-LMF) algorithm. The ZA-TD-LMF algorithm
ensures fast convergence and attracts all the filter coefficients
to zero. Simulation results conducted to substantiate our claim
are found to be very effective.
Index TermsLeast Mean Fourth (LMF), Transform Domain
(TD), Zero-Attractor ZA, Sparse solution

I. I NTRODUCTION
The LMF algorithm [1] is known to perform better than
the LMS algorithm in the case of non-Gaussian noise and
in low SNR environments. However, both algorithms do not
explore the special structure of sparsity that appears in many
systems, e.g., digital transmission channels [2] and wide area
wireless channels. Several approaches have been used to endow adaptive algorithms the ability to recognize such systems.
For example, the work in [3] uses sequential updating, since
the sparse filters are long by nature and most of the elements
are zeros. The proportionate LMF algorithm [4] and PNLMS
algorithm [5] are applied to sparse systems identification,
where the updating power depends on the value of the weight.
The advent of compressive sensing [6] and the least absolute
shrinkage and selection operator (LASSO) [7], are different
approaches appeared by endowing adaptive algorithms with
the ability to recognize sparse structures. By adding l1 norm
regression to the LMS algorithm, the derived sparse-aware
algorithm called zero attractor LMS (ZA-LMS) algorithm [8].
This algorithm fundamentally tries to attract all the weights
to zero, and hence the naming zero attractor. To avoid the
strong bias of the ZA-LMS algorithm when the system is
not sparse, a weighted zero attractor is introduced in [8] to
endow the resultant adaption with the ability to recognize nonzero elements, and apply small attraction to zero for these
group of elements. The sparse LMF algorithms introduced
in [9] proved to outperform their counterpart sparse-aware
LMS algorithms in low SNR environment. However, both the
LMS and the LMF family sparse families inherit the slow
M. Bashir and A. Zerguine are with the Department of Electrical Engineering, King Fahd University of Petroleum & Minerals, Dhahran, 31261, KSA
(e-mail: {g201304570,azzedine}@kfupm.edu.sa).

convergence property in the correlated environment, where the

eigenvalue spread of the autocorrelation matrix of the input
signal is large.
Applying discrete transformation (e.g., DCT and DFT)
accompanied with power normalization to the input is known
to result in more whiter input, shrinks the eigenvalue spread, as
in the case of transform domain LMS algorithm in [10]. Moreover, by endowing the TD-LMS algorithm with zero attractor,
the result is TD-ZA-LMS and TD-WZA-LMS algorithms [11],
where it was shown their convergence is faster in comparison
to their counterpart ZA-LMS and WZA-LMS algorithms.
In this work, we investigate a sparse-aware transform domain LMF algorithm, and assess its performance in a low
SNR environment. We study specifically the TD-ZA-LMF
algorithm, since the l1 penalty added to the LMF cost function
is convex, unlike other penalties, e.g., lp and l0 for example.
Notations: In the following parts of the paper, matrices and
vectors are denoted by capital and lower case boldface letters,
respectively, the superscripts H, T , and -1 denote Hermitian,
transpose, and inverse operators, respectively, and finally, kk1
and E[] denote the l1 norm and the statistical expectation,
respectively.
II. N EW LMF A LGORITHM
A. The Transform Domain LMF Algorithm
Consider a system identification scenario with input ui and
the desired output d(i) defined by
d(i) = wo ui + n(i)

(1)

where w is the optimal filter with length N . The transformed

input vector is defined as
xi = ui T

(2)

where T is an N N transformation matrix and the transformed input is xi . Both xi and ui are of length N . The
transform domain LMF algorithm is given by
T 3
wi = wi1 + 1
i xi e (i)

(3)

wi = TT w
i

(4)

where

where w
i is the time domain weight vector and wi is the
transformed domain weight vector. is the step size, e(i) is the
error, the difference between the desired output and the output
of the adaptive system, and is the power normalization
matrix. Clearly (3) does not exploit any sparsity information
in its recursion. In order to exploit sparsity, we will alter the

cost function to include a sparsity-aware term and then apply

the general gradient algorithm formula:
wi = wi1 1
i

J
wiT

(5)

to yield the proposed TD-ZA-LMF algorithm.

for a small step size scenario. This assumption suggests the

following relation:
E[n2k2
vi viT zi ]
i

= E[n2k2
]E[vi viT ]E[zi ]
i
= E[n2k2
]Rv E[zi ]
i

(12)

Finally, taking into account (12), (11) looks like:



2k2
E[zi ] = I (2k 1)E[ni
]Rv E[zi1 ] E[si ] (13)

B. Zero Attractor TDLMF Algorithm

The cost function of the zero attractor is given by
1 4
e (i) + ZA kTwi k1
(6)
4
where ZA is the zero attraction force (Lagrangian multiplier).
Note that we are transforming back the coefficients in order
to exploit sparsity, since the transformed weight vector is not
sparse. Now,
JZA =

JZA
= e3 (i)xTi + ZA TT sgn(Twi )
wiT

(7)

Clearly the sparsity contribution, E[si ], does not affect the

convergence. Equation (13) is quite similar to that of the LMS
convergence relation introduced in [12], and hence a necessary
condition for the stability of (8) in the mean is that the step
size, , should satisfy the following:
0<<

0<<

T
wi = wi1 + 1 xTi e3 (i) ZA 1
i T sgn(Twi )

where ZA = ZA . In contrast to the algorithm introduced in

[9], this algorithm is designed for Non-Gaussian and correlated
inputs. Moreover, it has the ability to attract all the filter
coefficients to zero.
III. P ERFORMANCE A NALYSIS OF THE TD-ZA-LMF
In this section, we study the convergence in the mean of
the TD-LMF sparse-aware algorithm. We launch the analysis
by using the following general recursion:
(8)

where si is the sparsity penalty term and k N , note that

k = 1 and k = 2 result in the LMS algorithm and LMF
algorithm, respectively.
Defining the weight error vector by zi = wo wi1 and
employing the relation between the zi and e(i), that is,
zTi vi

(9)

equation (8), reads:

zi = zi1 + zTi {n(i) zTi vi }2k1 (i) si

(10)

Ignoring the high power of the viT ui and applying statistical

expectation to (10) results in
E[zi ]

E[zi1 ] + E[vi n2k1

]
i
2k2
T
(2k 1)E[ni
v i v i zi ]

2
n2 T r(Rv )

(15)

where n2 is the variance of noise. and from [13], T r(Rv ) =

M , where M is the number of filter coefficients. The condition
in (15) becomes:
2
0<< 2
(16)
n M
where for sparse filters, M is known to be large, which reflects
in a small step size.

A. Convergence Analysis

wi = wi1 + viT e2k1 (i) si

(14)

and in the case of TD-ZA-LMF, k = 2, this range should be:

substituting (7) into (5) gives

e(i) = n(i)

2
(2k 1)E[n2k2
]T r(Rv )
i

B. Steady State Analysis

The steady-state analysis of the proposed algorithm is
derived in this section. For the case of the TD-ZA-LMF, the
recursion is governed by
wi = wi1 + viT e3 (i) si
viT

(17)

1 xTi .

where
=
Using energy conservation relation
arguments leads to the following [12]:
e i k2 + E
Ekw

|ea (i)|2
|ep (i)|2
e i1 k2 + E
= Ekw
2
kvi k2
kvi k22

(18)

fi = wo wi is the weight error vector, ea (i) =

where w
e i1 and ep (i) = vi w
e i are the posterior and priori errors,
vi w
respectively. At steady state the following condition is always
valid:
e i k22 = Ekw
e i1 k22
Ekw

(19)

the Excess Mean Square Error (EMSE) is defined as

E[si ]

(11)

By modeling the measurement noise as a white Gaussian

process, E[vi n2k1
] = 0. The noise and regressor, using indei
pendence assumption, can be assumed to be independent from
each other at steady state. The higher the noise plant power,
the lower the SNR which implies higher E[vi ] value. But for
very small step size, the effect of E[vi ] will be decreasing
regardless of the noise plant power and the regressor power
at steady state. Hence this independence assumption is valid

EM SE = E|ea (i)|2

(20)

Hence equation (9) becomes:

E

|ea (i)|2
|ep (i)|2
=
E
kvi k22
kvi k22

(21)

the priori and a posteriori errors can be found to be related

through the following recursion:
ea (i) = ep (i) kvi k2 e3 (i) + vi si

(22)

Then:
ksi k2vi vT
|ea (i)|
2 6
2
i
E
+
E
e
(i)kv
k
+
E
i
|
{z
}2
kvi k22
kvi k22
| {z }
A
B
vi si
3
2Eea (i)e (i) + ea (i)
{z
}
|
kvi k22
{z
}
|
C
D
e3 (i)vi si
(23)
2

|ep (i)|
kvi k22

{z

Also, the term D can be shown to be results in

vi si
D = Eea (i)
kvi k22
w
T vT vi si
= E i i 2
kvi k2
T
w
Rv si
= E i
T r(Rv )

Finally, applying the same procedure for term D, E gives:

E = 6n2 w
iT Rv si

In order to simplify (22) the following adopted assumptions

are used:

(31)

Now, substituting the different terms in (22), the EM SE reads

A1: At steady state the error is independent from the input,

and hence the a posterior error.
A2: The measurement noise is independent of the adaption
error, the input and the sparsity enforcing term.

(30)

5n4 T r(Rv )
2 9n2 T r(Rv )
|
{z
}
EM SET DLM F


ksi kRv
1
+ E
w
i T r(Rv )[6n2
]si
T r(Rv )
T r(Rv )
{z
}
|
EM SEsparse

(32)
Moreover, we will also resort to the separation principle
assumption, which states that, at steady-state the adaption error
is independent of the input. Also, we will use the following
relation expectation approximation [14]:
E

EviT vi
Rv
viT vi

=
2
2
kvi k2
Ekvi k2
T r(Rv )

(24)

This approximation is quite helpful in simplifying the analysis,

and suitable for long filters, which is the case for sparse filters.
It should be noted that the TD-LMF algorithm inherits the
same step size conditions for convergence.
We start the steady-state analysis by exploring each term
of the right hand side of (22). By applying the separation
principle, the term A reads:
A = 2 Ee6 (i)Ekvi k22

(25)

Using the following relation between the error, a posterior

error and noise:
e(i) = ea (i) + n(i)

(26)

the term A results in the following:

2

A 45

T r(Rv )n2 Eea (i)2

+ 15

T r(Rv )n2

(27)

The sparsity enforcing at steady-state is independent from the

regressors. However, for the case of the transform domain, we
will assume further that the input is not too non-stationary,
and hence the power normalization matrix is almost constant
and by employing the rational expectation approximation, the
term B lands in:
E

ksi k2vi vT
i

kvi k22

ksi kRv
=E
T r(Rv )

(28)

Using A2 and ignoring the high power terms, C approximates

to
C 6n2 Eea (i)2

(29)

where
=

6n2

1
452 T r(Rv )n4

(33)

If the transformation and power normalization matrices are

set to I, the EM SE for the TD-LMF falls back to that of the
EM SE for the LMF given in [12].
To get more insight of equation (31), we study the EM SE
for the two set of elements in the vector wi , namely, the
non-zero (N Z) elements and zero (Z) elements. The EM SE
exerted for the non-zero elements is
EM SEmN Z =

5n4 T r(Rv )
ksi kRv
+ E
2
2 9n T r(Rv )
T r(Rv )

(34)

where for this group of weights, the sparsity enforcing term si

i . While, for the zero elements,
is independent from the error w
the EM SE is
EM SEmZ

5n4 T r(Rv )
2 9n2 T r(Rv )

Ew
i T r(Rv )(6n2

1
)si (35)
T r(Rv )

Since for this group, the sparsity term si is expected to be

zero, then from (32) and (33), we can see the tension in the
EMSE between the N Z and Z elements. When the sparsity
rate is low (i.e., |Z| >> |N Z|), the overall EM SE will be
lower than for the non-sparse-aware algorithm, which in this
case the TD-LMF.
IV. S IMULATION R ESULTS
In this section, the performance of the propsed TD-ZALMF, ZA-LMF, ZA-LMS and TD-ZA-LMS are evaluated
using Monte Carlo simulations. The results are averaged over
500 runs. The transform used is the Discrete Cosine Transform
(DCT-II). The SNR is set to 5 dB, (n2 = 0.31) and M = 32,
which sets the step size range to 0 < < 0.098. We choose
= 0.005. Two experiments are conducted for the input

ZALMF
ZALMS
TDZALMF
TDZALMS

2
4

MSD (dB)

MSD (dB)

ZALMF
ZALMS
TDZALMF
TDZALMS

10
12
14

6
8
10

16
12
18
14

20
22
1000

2000

3000

4000

5000

6000

7000

8000

iteration

1000

2000

3000

4000

5000

6000

7000

8000

9000

iteration

Fig. 1: MSD curves for white Gaussian input with sparsity rate = 2.

Fig. 2: MSD curves for correlated input with sparsity rate = 2.

signal, Gaussian input and correlated one. For the correlated

1
input, a first-order filter with transfer function 1+0.9z
1 is
sued to generate the correlated samples. The system under
identification has sparsity m = 2, with random location for
the non-zero elements.
The zero-attraction parameters are set to allow all algorithms
to reach the same level of mean-square-deviation (MSD),
which allow a fair comparison of the algorithms. The values
are give in Table I.

performed a bit slower in comparison with the ZA-LMF in

case of white Gaussian input, because of the bias of the power
normalization introduced in the zero-attractor.

Algorithm
ZA-LMS
ZA-LMF
TD-ZA-LMS
TD-ZA-LMF

First Experiment
5.5 105
5 106
4.5 105
9 106

Second Experiment
1 105
9 106
3.1 105
9 106

TABLE I: Zero Attractor values for the different algorithms.

As can be seen from Fig. 1, the LMF based algorithms

reached the steady state faster compared to the LMS based
ones. Because of the bias introduced by the power normalization matrix, the TD-ZA-LMF is slightly slower than the
ZA-LMF.
Figure 2 depicts the ZA-LMF algorithm is better than the
ZA-LMS algorithm when the SNR is low. However, both
algorithms suffer from very slow convergence due to the high
correlated input. More importantly, The transform domain
algorithms, though, are comparatively faster in convergence
and lower in steady state level, and the TD-ZA-LMF is the best
performing. This proves the power of the hybridisation of the
transform domain and the sparse-aware ability in enhancing
the convergence behavior of the LMF algorithm at low SNR.
V. C ONCLUSION
By introducing the Zero Attractor to the TD-LMF algorithm,
the resultant recursion is able to explore sparse structure in correlated environment with low SNR. Compared to the existing
methods, the proposed algorithm improved the convergence
as confirmed by simulations. The proposed algorithm though,

R EFERENCES
[1] E. Walach and B. Widrow, The least mean fourth (lmf) adaptive
algorithm and its family, Information Theory, IEEE Transactions on,
vol. 30, no. 2, pp. 275283, 1984.
[2] W. Schreiber, Advanced television systems for terrestrial broadcasting:
Some problems and some proposed solutions, Proceedings of the IEEE,
vol. 83, no. 6, pp. 958981, 1995.
[3] D. Etter, Identification of sparse impulse response systems using an
adaptive delay filter, in Acoustics, Speech, and Signal Processing, IEEE
International Conference on ICASSP85., vol. 10. IEEE, 1985, pp.
11691172.
[4] Y. Yilmaz and S. S. Kozat, An extended version of the nlmf algorithm based on proportionate krylov subspace projections, in Machine
Learning and Applications, 2009. ICMLA09. International Conference
on. IEEE, 2009, pp. 404408.
[5] D. L. Duttweiler, Proportionate normalized least-mean-squares adaptation in echo cancelers, Speech and Audio Processing, IEEE Transactions on, vol. 8, no. 5, pp. 508518, 2000.
[6] D. L. Donoho, Compressed sensing, Information Theory, IEEE Transactions on, vol. 52, no. 4, pp. 12891306, 2006.
[7] R. Tibshirani, Regression shrinkage and selection via the lasso,
Journal of the Royal Statistical Society. Series B (Methodological), pp.
267288, 1996.
[8] Y. Chen, Y. Gu, and A. O. Hero, Sparse lms for system identification,
in Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE
International Conference on. IEEE, 2009, pp. 31253128.
[9] G. Gui and F. Adachi, Sparse least mean fourth algorithm for adaptive
channel estimation in low signal-to-noise ratio region, International
Journal of Communication Systems, vol. 27, no. 11, pp. 31473157,
2014.
[10] S. Narayan, A. M. Peterson, and M. J. Narasimha, Transform domain
lms algorithm, Acoustics, Speech and Signal Processing, IEEE Transactions on, vol. 31, no. 3, pp. 609615, 1983.
[11] K. Shi and X. Ma, Transform domain lms algorithms for sparse system
identification, in Acoustics Speech and Signal Processing (ICASSP),
2010 IEEE International Conference on. IEEE, 2010, pp. 37143717.
[12] A. H. Sayed, Fundamentals of adaptive filtering. John Wiley & Sons,
2003.
[13] S. Zhao, Z. Man, S. Khoo, and H. R. Wu, Stability and convergence
analysis of transform-domain lms adaptive filters with second-order autoregressive process, Signal Processing, IEEE Transactions on, vol. 57,
no. 1, pp. 119130, 2009.
[14] C. Samson and V. Reddy, Fixed point error analysis of the normalized
ladder algorithm, Acoustics, Speech and Signal Processing, IEEE
Transactions on, vol. 31, no. 5, pp. 11771191, 1983.