This article was downloaded by: [Birla Institute of Technology and Science]

On: 23 March 2015, At: 12:18

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office:
Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Systems Science

Publication details, including instructions for authors and subscription
information:
http://www.tandfonline.com/loi/tsys20

Nonlinear system identification using wavelet

networks
a

G. P. Liu , S A. Billings & V Kadirkamanathan

ABB ALSTOM Power Technology Centre , Cambridge Road, Leicester, LE8 6LH,
U.K.
b

Department of Automatic Control and Systems Engineering , University of

Sheffield , PO Box 600, Mappin Street, Sheffield, S1 3JD, U.K.
Published online: 26 Nov 2010.

To cite this article: G. P. Liu , S A. Billings & V Kadirkamanathan (2000) Nonlinear system identification
using wavelet networks, International Journal of Systems Science, 31:12, 1531-1541, DOI:
10.1080/00207720050217304
To link to this article: http://dx.doi.org/10.1080/00207720050217304

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content)
contained in the publications on our platform. However, Taylor & Francis, our agents, and our
licensors make no representations or warranties whatsoever as to the accuracy, completeness, or
suitability for any purpose of the Content. Any opinions and views expressed in this publication are
the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis.
The accuracy of the Content should not be relied upon and should be independently verified with
primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever
caused arising directly or indirectly in connection with, in relation to or arising out of the use of the
Content.
This article may be used for research, teaching, and private study purposes. Any substantial
or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or
distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can
be found at http://www.tandfonline.com/page/terms-and-conditions

Internationa l Journal of Systems Science, 2000, volume 31, number 12, pages 1531 1541

Nonlinear system identi cation using wavelet networks

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

G . P. LIU{* , S. A. BILLINGS{ and V. K ADIRKAMANATHAN{

A novel identi cation scheme using wavelet networks is presented for nonlinear dynamical systems. Based on xed wavelet networks, parameter adaptatio n laws are developed using a Lyapunov synthesis approach. This guarantees the stability of the overall
identi cation scheme and the convergence of both the parameters and the state errors,
even in the presence of modelling errors. Using the decomposition and reconstruction
techniques of multiresolution decompositions , variable wavelet networks are introduced
to achieve a desired estimation accuracy and a suitable sized network, and to adapt to
variations of the characteristics and operating points in nonlinear systems. B-spline
wavelets are used to form the wavelet networks and the identi cation scheme is illustrated using a simulated example.

1.

Introduction

Nonlinear system identi cation consists of model structure selection and parameter estimation. The rst problem is concerned with selecting a class of mathematical
operator as a model. The second is concerned with an
estimation algorithm based on input output data from
the process, a class of models to be identi ed and a
suitable identi cation criterion. A number of techniques
have been developed in recent years for model selection
and parameter estimation of nonlinear systems.
Forward and backward regression algorithms were analysed in Leontaritis and Billings (1987). Stepwise regression was used in Billings and Voon (1986), and a class of
orthogonal estimators was discussed in Korenberg et al.
(1988). Algorithms which save memory and allow fast
computations have also been proposed in Chen and
Wigger (1995) . Methods to determine a priori structural
identi ability of a model have also been studied (Ljung
and Glad 1994). A survey of existing techniques of nonlinear system identi cation prior to the 1980s is given in
Billings (1980), a survey of the structure detection of
input output nonlinear systems is given in Haber and
Unbehauen (1990), and a recent survey of nonlinear

Accepted 9 November 1999.

{ ABB ALSTOM Power Technology Centre, Cambridge Road,
Leicester LE8 6LH, U.K.
{ Department of Automatic Control and Systems Engineering,
University of She eld, PO Box 600, Mappin Street, She eld S1
3JD, U.K.

black-box modelling in system identi cation can be

found in Sjoberg et al. (1995).
The approximation of general continuous functions
by nonlinear networks has been widely applied to
system modelling and identi cation. Such approximation methods are particularly useful in the black-box
identi cation of nonlinear systems where very little a
priori knowledge is available. For example, neural networks have been established as a general approximation
tool for tting nonlinear models from input output data
on the basis of the universal approximation property of
such networks (see, e.g., Chen et al. 1990, Narendra and
Parthasarathy , 1990, Liu et al. 1996a). There has also
been considerable recent interest in identi cation of general nonlinear systems based on radial basis networks
(Poggio and Girosi 1990), fuzzy sets and rules (Zadeh
1994), neural-fuzzy networks (Brown and Harris 1994;
Wang et al. 1995) and hinging hyperplanes (Beriman
1993).
The recently introduced wavelet decomposition
(Grossmannand and Morlet 1984, Daubechies 1988,
Mallat 1989a, Chui 1992, Meyer 1993, IEEE 1996)
also emerges as a new powerful tool for approximation.
In recent years, wavelets have become a very active subject in many scienti c and engineering research areas.
Wavelet decompositions provide a useful basis for localized approximation of functions with any degree of
regularity at dierent scales and with a desired accuracy.
Recent advances have also shown the existence of orthonormal wavelet bases, from which follows the variability
of rates of convergence for approximation by waveletbased networks. Wavelets can therefore be viewed as a

International Journal of Systems Science ISSN 0020 7721 print/ISSN 1464 5319 online # 2000 Taylor & Francis Ltd
http://www.tandf.co.uk/journals

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

1532

G. P. Liu et al.

new basis for representing functions. Wavelet-based networks (or simply wavelet networks) are inspired by both
the feedforward neural networks and wavelet decompositions and have been introduced for the identi cation of
nonlinear static systems (Zhang and Benveniste 1992).
But little attention has been paid to identi cation of
nonlinear dynamical systems using wavelet networks
(Coca and Billings 1996, Liu et al. 1998, 1999a).
This paper presents a wavelet network-based identi cation scheme for nonlinear dynamical systems. Two
kinds of wavelet networks are studied: xed and variable
wavelet networks. The former is used for the case where
the estimation accuracy is assumed to be achieved by a
known resolution scale. But, in practice, this assumption
is not realistic because the nonlinear function to be identi ed is unknown and the system operating point may
change with time. Thus, variable wavelet networks are
introduced to deal with this problem. The basic principle
of the variable wavelet network is that the number of
wavelets in the network can either be increased or
decreased over time according to a design strategy in
an attempt to avoid over tting or under tting. In
order to model unknown nonlinearities, the variable
wavelet network starts with a lower resolution scale,
and then increases or reduces this according to the
novelty of the observation. Because the novelty of the
observation is tested, it is ideally suited for on-line identi cation problems. The objective behind the development is to gradually approach the appropriate network
complexity that is su cient to provide an approximation to the system nonlinearities and which is consistent
with the observation s received.
The parameters of the wavelet network are adjusted
by adaptation laws developed using a Lyapunov synthesis approach. The identi cation algorithm is performed over the network parameters by taking
advantage of the decomposition and reconstruction
algorithms of a multiresolution decomposition when
the resolution scale changes in the variable wavelet network. Combining the wavelet network and Lyapunov
synthesis techniques, the identi cation algorithm developed for continuous dynamical nonlinear systems guarantees the stability of the whole identi cation scheme
and the convergence of both the parameters and estimation errors. A simulated example shows the operation of
the proposed identi cation scheme.
2.

Wavelet networks

Wavelets are a class of functions which have some interesting and special properties. Some basic concepts about
orthonormal wavelet bases will be introduced initially.
Then the wavelet series representation of one-dimensional and multi-dimensional functions will be considered. Finally, wavelet networks are introduced.

Throughout this paper, the following notations will

be used. Z and R denote the set of integers and real
numbers, respectively. L2 R denotes the vector space
of measurable, square-integrabl e one-dimensional functions f x. For f ; g 2 L2 R , the inner product and
norm for the space L2 R are written as
1
h f ; gi :
f xgx dx
1
1

k f k2 : h f ; f i1=2

where g: is the conjugate of the function g:. L2 R n is

the vector space of measurable, square-integrabl e ndimensional
functions
f x1 ; x2 ; . . . ; xn .
For
f ; g 2 L2 R n , the inner product of f x1 ; x2 ; . . . ; xn
with gx1 ; x2 ; . . . ; xn is written as
h f x1 ; x2 ; . . . ; xn ; gx1 ; x2 ; . . . ; xn i
1 1
1

f x1 ; x2 ; . . . ; xn
1

gx1 ; x2 ; . . . ; xn dx1 dx2 . . . dxn

The original objective of the theory of wavelets is to

construct orthogonal bases in L2 R . These bases are
constituted by translations and dilations of the same
function . It is preferable to take as localized and
regular. The principle of wavelet construction is the following: (i) the function x k are mutually orthogonal for k ranging over Z ; (ii) is a scaling function
and for any j, f2j x kg constitutes an orthogonal
basis of its linear span subspace; (iii) the wavelet is
de ned as and the family 2j x k constitutes an
orthogonal basis of L2 R . It can also be proved that
the family f2j0 x k; 2j x k; for j j0 g also
forms an orthogonal basis of L2 R .
The wavelet subspaces Wj are de ned as
Wj f2j x k;

k2Z g

j 6 i

which satisfy
Wj \ Wi f1g;

Any wavelet generates a direct sum decomposition of

L2 R . For each j 2 Z , let us consider the closed subspaces:
Vj Wj2 Wj1

of L2 R , where denotes the direct sum. These subspaces have the following properties:
(1) . . . V1 V0 V1 . . .;

!
[
(2) closeL2
Vj L2 R ;
j2Z
\
(3)
Vj f1g;
j2Z

1533

Nonlinear system identi cation using wavelet networks

(4) Vj1 Vj W j , j 2 Z ; and

f x

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

(5) f x 2 Vj () f 2x 2 Vj1 ; j 2 Z .
Hence, the sequence of subspaces Vj is nested, as
described by property (1). Property (2) shows that
every function f in L2 R can be approximated as
closely as desirable by its projections, denoted by Pj f
in Vj . But, by decreasing j, the projections Pj f could
have arbitrarily small energy, as guaranteed by property
(3). The most important intrinsic property of these
spaces is that more and more variations of Pj f are
removed as j ! 1. In fact, these variations are
peeled o, level by level in decreasing order of the rate
of variations and stored in the complementary subspaces
Wj as in property (4). This process can be made very
e cient by an application of property (5).
If and are compactly supported, they give a local
description, at dierent scales j, of the considered function. The wavelet series representation of the one-dimensional function f x is given by
X
XX
f x
aj0 k j0 k x
bjk jk x
7
jj0 k2Z

k2Z

where

aj0 k h f x; j0 k xi

bjk h f x; jk xi

The wavelet series representation can easily be generalized to any dimension n. For the n-dimensional case
x x1 ; x2 ; . . . ; xn , we introduce the scaling function
U x x1 x2 . . . xn

10

and
the
2n 1
mother
wavelets
W i x,
n
i 1; 2; . . . ; 2 1, are obtained by substituting some
xj s by xj in equation (10). Then the following
family is an orthonormal basis in L2 R n :
2
jk x; . . . ; W

2n 1
xg
jk

for j j0 , j 2 Z , k k1 ; k2 ; . . . ; kn 2 Z
U

11

, and

jn=2
U 2 j x1 k1 ; 2j x 2 k2 ; . . . ; 2j xn kn
jk x 2

12

i
jk x

2jn=2 W

j
j
j
i 2 x1 k1 ; 2 x2 k2 ; . . . ; 2 xn kn

13

For f x 2 L2 R , the n-dimensional wavelet series

representation of the function f x is

n
1
X X 2X

jj0 k2Z

bi
jk W

i1

i
jk x

14

aj0 k h f x; U

j0 k xi

15

bi
jk h f x; W

i
jk xi

16

For system identi cation, f x is unknown. Then the

i
wavelet coe cients aj0 k and bjk can not be calculated
simply by equations (15) and (16). As equation (11)
shows, constructing and storing orthonormal wavelet
bases involves a prohibitive cost for large dimensions
n. In addition, it is not realistic to use an in nite
number of wavelets to represent the function f x. So,
we consider the following wavelet representation of the
function f x:
f^x

a j0 k U

j0 k x

k2Aj0

N X 2X
1
X

bi
jk W

jj0 k2B j i1

i
jk x

17

where Aj0 ; B j 2 Z n are the nite vector sets of integers

and N 2 R 1 is a nite integer. Because the convergence
of the series in equation (14) is in L2 R n ,
lim

and the wavelet coe cients aj0 k and bjk are

1
jk x; W

j0 k x

N!1;Aj0 ;B j0 ;...;B N !Z

jk x 2j=2 2j x k;

j0 k x; W

k2Z

a j0 k U

where the wavelet coe cients are

j0 k x 2j0 =2 2j0 x k;

fU

k f x f^xk2 0

18

Hence, given " > 0, there exists a number N* and vector

sets A*j0 ; B*j0 ; B*j0 1 ; . . . ; B*N such that for N N*,
Aj0 A*j0 and B j0 B*j0 , B j0 1 B*j0 1 ; . . . ; B N B*N ,
k f x f^xjj2 "

19

This shows that the required approximation accuracy of

the function f by f^ can be guaranteed by properly
choosing the number N and the vector sets
Aj0 ; Bj0 ; . . . ; BN . Following neural networks, the expression (17) is called a wavelet network. In this network,
i
the parameters aj0 k and bjk , and the number N and the
vector sets Aj0 ; B j0 ; . . . ; B N will jointly be determined
from the data, based on the scaling functions and
the wavelets .

3.

Identi cation using xed wavelet networks

Consider the multi-input multi-state (MIMS) continuous dynamical system described by,
r1

x_ fx; u;

x0 x0 ;

d1

20

where u 2 <
is the input vector, x 2 <
is the state
vector and f f1 ; f2 ; . . . ; fd T 2 Ld2 R n is an
unknown nonlinear function vector, where n r d. It
is also assumed that u; x are in compact sets.
Following the structure (17) of wavelet networks, at
the resolution 2N1 , the estimation ^fx; u of the func-

1534

G. P. Liu et al.
^
ex x x
~
Aj0 k A*j0 k Aj0 k

tion fx; u is the output vector of the wavelet network

which is expressed by
^fx; u

A j0 k U

k2Aj0

j0 k x; u

n
N X 2X
1
X

Bi
jk W

jj0 k2Bj i1

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

d
where Aj0 k ; Bi
are the wavelet coe cient vectors,
jk 2 R
the scaling function U j0 k x; u and the wavelet functions
i
W jk x; u are similarly de ned as U x and W j x, respectively, by replacing x with x; u.
Here, it is assumed that the number N and the vector
sets Aj0 ; B j are given. So, the wavelet network (21) for
the estimation of the nonlinear function fx; u is called
a xed wavelet network. Based on the estimation ^fx; u
by the xed wavelet network, the nonlinear function
fx; u can be expressed by
X
fx; u
A*j0 k U j0 k x; u

k2Aj0

n
N X 2X
1
X

i*
Bjk W

jj0 k2Bj i1

i
jk x; u

"N

22

where the optimal wavelet coe cient vectors A*j0 k and

*
Bi
jk are
2
3
h f1 x; u; U j0 k x; ui
6
7
6 h f2 x; u; U j0 k x; ui 7
6
7
7
A*j0 k 6
23
6
7
..
6
7
.
4
5
h fd x; u; U j0 k x; ui
2
3
i
h f1 x; u; W jk x; ui
6
7
6
7
6 h f2 x; u; W i
7
x;
ui
jk
6
7
i*
Bjk 6
24
7
6
7
.
..
6
7
4
5
h fd x; u; W

i
jk x; ui

"N "N1 ; "N2 ; . . . ; "Nd T is the modelling error vector

which is assumed to be bounded by
"N max sup fj"Ni tjg
i1;2;...;d t2R

25

Modelling the nonlinear function vector fx; u using

wavelet networks gives the following identi cation
model for the nonlinear dynamical system (20):
^_ Ax
^ x ^fx; u;
x

^x0 x0

26

^ denotes the state vector of the network model

where x
and A 2 R dd is a Hurwitz or stability matrix (i.e. all the
eigenvalues are in the open left-half complex plane).
De ne the state error vector and wavelet coe cient
error vectors as

~ i W
B
jk

i
jk x; u

jj0 k2B j i1

"N

30

Consider the Lyapunov function

1 X ~T ~
V eTx Pex
A A
j0 k2A j0 k j0 k
j0

N X 2X
1
X

29

so that the dynamical expression of the state error is

given by
X
~ j k U j k x; u
e_ x Aex
A
0
0

k2Aj0

28

~ i Bi* Bi
B
jk
jk
jk

i
jk x; u

21

27

N
1
X
2 X 2X
~ i T B
~ i
B
jk
jk

j k2B i1
jj
0

31

where P fpij g 2 R dd is chosen to be a positive de nite matrix so that the matrix Q PA AT P is also a
positive de nite matrix, and j0 and j are positive constants which will appear in the parameter adaptation
laws, also referred to as the adaptation rates.
The rst derivative of the Lyapunov function V with
respect to time t is
2 X ~_ T ~
V_ e_ Tx Pex eTx P_ex
A A
j0 k2A j0 k j0 k
j0

N
1
X
2 X 2X
T ~ i
~_ i
B
jk Bjk

j
jj
k2B i1
0

32

Substituting equation (30) into equation (32) gives

2 X ~_ T ~
~ j k U j k x; u
V_ eTx Qe x
Aj0 k Aj0 k j0 eTx PA
0
0
j0 k2A
j0

N
1
X
2 X 2X
T ~ i
T ~ i
~_ i
B
jk Bjk j e x PBjk W

j k2B i1
jj
0

2eTx P"N

i
jk x; u

33

*
Because A*j0 k and Bi
are constant vectors,
jk
i
_A
_
~ j k A_ j k and B
~ jk B_ i . If there is no modelling
jk
0
0
error, i.e., "N 0, Aj0 k and Bi
jk can simply be estimated
by the following adaptation laws:
A_ j0 k j0 eTx PU
i
B_ jk j eTx PW

j0 k x; u
i
jk x; u

34
35

In the presence of a modelling error "N , several algorithms can be applied to ensure the stability of the whole

1535

Nonlinear system identi cation using wavelet networks

identi cation scheme, e.g. the xed or switching -modi cation (Ioannou and Kokotovic 1983), "-modi cation
(Narendra and Annaswamy 1987) and the dead-zone
methods (Sastry and Bodson 1989).
De ne the following sets:
F ; M fz : kzk < M
kzk M

and

eTx Pz 0g

36

F ; M fz : kzk M

and

eTx Pz < 0g

37

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

or

where z 2 R d , is a function and M is a positive constant. Here, in order to avoid parameter drift in the
presence of modelling error, the application of the projection algorithm (Goodwin and Mayne, 1987) gives the
following adaptive laws for the parameter estimates Aj0 k
i
and Bjk :
8
j0 eTx PU j0 k
>
>
>
>

>
< if Aj0 k 2 F U j0 k ; Mj0 k
A_ j0 k
38
>
T
2 T
>
>
e
PU
k

M
e
PA
U
A

j x
j0
j0
j0 k x
j0 k j0 k j0 k
>
>
: 0
if Aj0 k 2 F U j0 k ; Mj0 k

B_ i
jk

8 T
i
>
j ex PW jk
>
>
>
>

>
>
if Bi
<
jk 2 F W

i
i
jk ; Mjk

>
>
i
i
i
>
eT PW jk j Mjk 2 eTx PBjk W
>
>
> j x
>
:
i
i
i
if Bjk 2 F W jk ; Mjk

i i
jk Bjk

39

where Mj0 k , Mjk are the allowed largest values ofkAj0 k k

i
and kBjk k, respectively. It is clear that if the initial
parameter vectors are chosen such that
and

Aj0 k 0 2 F U
i

j0 k ; M j0 k

[ F U

j0 k ; M j0 k

i
i
i
i

jk ; M jk [ F W jk ; Mjk ;
i
Aj0 k and Bjk are con ned to the

j0 k ; Mj0 k [ F U j0 k ; Mj0 k

Bjk 0 2 F W
then the vectors

and

F U
F W

i
i
jk ; M jk

[ F W

sets

i
i
jk ; Mjk ;

respectively. Using the adaptive laws (38) and (39),

equation (33) becomes
V_ eTx Qe x 2

d X
d
X

jpij jjexi jj"Nj j

eTx Qe x 2

d X
d
X

jpij jjexi j"N

i1 j1

i1 j1

where ex ex1 ; ex2 ; . . . ; exd T .

40

For the sake of simplicity, the positive de nite

matrix Q is assumed to be diagonal, i.e.
Q diagq1 ; q2 ; . . . ; qd . Also de ne
8

!2
<
d
d
X
X
jpij j
H ex :
qi jexi j

:
qi
i1
j1

d
d
X
X
i1

j1

!2 9
=
2
pij
qi ;

41

where is a positive variable, i.e., 0.

If there is no modelling error (i.e. "N 0), it is clear
from equation (40) that V_ is negative semide nite.
Hence the stability of the overall identi cation scheme
is guaranteed and ex ! 0, Aj0 k ! 0, Bi
jk ! 0. In the
presence of modelling errors, if ex 62 H "N , it is easy
to show from equation (40) that V_ is still negative and
the state error ex will converge to the set H "N . But, if
ex 2 H "N , it is possible that V_ > 0, which implies that
i
the weight vectors Aj0 k and Bjk may drift to in nity over
time. The adaptive laws (38) and (39) avoid this drift by
limiting the upper bounds of the parameters. Thus, the
state error ex always converges to the set H "N and the
whole identi cation scheme will remain stable in the
case of modelling errors.

4.

Identi cation using variable wavelet networks

For nonlinear systems, the system operation can change

with time. This will result in an estimation error for the
xed wavelet network that is beyond the required error.
In order to improve the identi cation performance, both
the structure and the parameters of the wavelet network
model need to be modi ed in response to variations of
the plant characteristics. This section takes into account
the modi cation of the wavelet network structure and
the adaptation of the parameters.
It is known that the modelling error "N can be
reduced arbitrarily by increasing the resolution of the
wavelet network. But, generally when the resolution is
increased beyond a certain value the modelling error "N
will improve very little by any further increasing the
resolution. This will also result in a large size network
even for simple nonlinear problems and in practice, this
is not realistic. In most cases, the required modelling
error can be given by considering the design requirements and speci cations of the system. Thus, the problem now is to nd a suitable sized network to achieve
the required modelling error. Following variable neural
networks (Liu et al. 1996b, 1999b), a variable wavelet
network is introduced below.
Like a neuron in neural networks, a wavelon is an
information-processin g unit that is fundamental to the
operation of a wavelet network. Generally speaking, a

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

1536

G. P. Liu et al.

variable wavelet network has the property that the

number of wavelons in the network can be either
increased or decreased over time according to a design
strategy. For the problem of nonlinear modelling, the
variable wavelet network is initialized with a small
number of wavelons. As observations are received, the
network grows by adding new wavelons or is pruned by
removing old ones.
According to the multiresolution approximation
theory, increasing the resolution of the network will
improve the approximation. To improve the approximation accuracy, the growing network technique
(Kadirkamanatha n and Niranjan 1993, Liu et al.
1996a) is applied. This means that the wavelets at a
higher resolution need to be added to the network.
Here it is assumed that at the resolution 2N the approximation of the function f by the wavelet network is
denoted as ^f N . Based on the growing network technique and the structure of the function ^f in equation
(21), the adding operation is de ned as
n

^fx; u ^f N x; u
_

1
X 2X

Bi
Nk W

k2B N i1

i
Nk x; u

n
1
X 2X

k2BN 1 i1

BN1k W

i
x; u
N1k

43

where _ denotes the removing operation. Equation (43)

implies that wavelets at the resolution 2N are removed
from the network. Similarly, to remove some old wavelons from the network, the following two conditions
must be satis ed. (i) The modelling error must be less
than the required accuracy. (ii) The period between the
two removing operations must be greater than the minimum response time of the removing operation.
In both the adding and removing operations, condition (i) means that the change of the modelling error
in the network must be signi cant. Condition (ii) says
the minimum response time of each operation must be
considered. The response time of the network re nement
operation varies with dierent systems. It is usually estimated by a trial and error process.
From the set H "N which gives a relationship
between the state error ex and the modelling errors "N ,

for

jexi j ei ;

i 1; 2; . . . ; n; t 2 R

44

where ei is the required accuracy. At the beginning, it is

very di cult to know how many wavelons are needed to
achieve the above identi cation requirements. In order
to nd a suitable sized network for this identi cation
problem, lower and upper bounds are set for the state
errors which are functions of time t. A variable network
such that
jexi j 2 iL t;

iU t ei ;

for

i 1; 2; . . . ; n 45

is then tried, where iL t; iU t are monodecreasing

functions of time t, respectively. For example,

42

_ denotes the adding operation. Equation (42)

where
means that wavelets at the resolution 2N1 are added
to the network. To add new wavelons to the network
the following two conditions must be satis ed. (i) The
modelling error must be greater than the required accuracy. (ii) The period between the two adding operations
must be greater than the minimum response time of the
adding operation.
The removing operation is de ned as
^fx; u ^f N x; u
_

it can be shown that the state error depends on the

modelling error. If the upper bound "N of the modelling
error is known, then the set H "N to which the state
error will converge is also known. However, in most
cases the upper bound "N is unknown.
In practice, systems are usually required to keep the
state errors within prescribed bounds, i.e.

iU t e U t iU 0

46

iL t e L t iL 0

47

where U ; L are positive constants, iU 0; iL 0 are the

initial values. It is clear that iU t; iL t decrease with
time t. As t ! 0, iU t; Li t approach 0. Thus, in this
way the state errors reach the required accuracy given in
equation (44).
From the relationship between the modelling error
and the state error, and given the lower and upper
bounds iU t; iL t ei of the state errors, the corresponding modelling error should be
"N t 2 "L t;

"U t

48

From equation (41), the area that the set H covers is a

hyperellipsoid with the centre

!
d
d
d
X
X
jp1j j X
jp2j j
jpdj j
49
;
; . . . ;

q1
q2
qn
j1
j1
j1
It can also be deduced from the set H "N t that the
upper bound "U t and the lower bound "L t are given
by
(
d
X
jpij j
"L t min
i1;2;...;d
qi
j1
0

d
X
k1

d
X
j1

pkj

!2

9
11=2 11
>
1 A C L =
A i t
>
qi qk
;

50

1537

Nonlinear system identi cation using wavelet networks

"U t max

i1;2;...;d

(
d
X
jpij j
j1

k2Aj1

qi

9
11=2 11

!2
>
=
d
d
X
X
1 A C
U
@
pkj

A
i
i
>
qi qk
;
k1 j1
0

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

51

Thus, given the upper and lower bounds of the state

error, the corresponding values for the modelling error
can be estimated by equations (50) and (51). But, if these
bounds are not known, the xed wavelet network with
an adequate number of wavelons may be used rather
than the variable wavelet network.
To smooth the identi cation performance when the
adding and removing operations are used, the decomposition and reconstruction algorithms of a multiresolution decomposition are applied to the initial calculation
of the wavelet coe cients. Here two important relations
are introduced. Firstly, because the family fU k g spans
V0 , then fU 2x kg spans the next lter scale
V1 V0 W0 . Both the scaling function and the
wavelet function can be expressed in terms of the scaling
function at the resolution 2j 21 , i.e.
p
X
U x 2
ck U 2x k
52

Aj1k U

j1k x; u

k2A j2

53

k2Z

where ck and dki are known as the two scale reconstruction sequences. Secondly, any scaling function U 2x in
V1 can alternatively be written using the scaling function
U x in V0 and wavelet function W x in W0 as
X
U 2x l
al2k U x k
k2Z

n
2X
1

bi
l2k W

i1

i x

54

k2B j2 i1

k2Aj1

where

k2Bj1 i1

Bi
W
j2k

i
x; u
j2k

56

Hence, if the state error ex 62 H "U t, the network

needs more wavelets. Add the wavelets at the resolution
2N1 into the network. Following the adding operation
(42) and the expression (55) of ^f j x; u at j N, the
structure of the approximated function ^fx; u is of the
form:
^fx; u

ANk U

k2AN

Nk x; u

n
1
X 2X

Bi
Nk W

k2BN i1

i
Nk x; u

Bi
Nk

57

The parameter vectors ANk and

are adapted by the
laws (38) and (39). Using the sequences ck and dki , the
initial values after the adding operation are then given
by the reconstruction algorithm (Mallat 1989b):

!
n
2X
1
X
i
i
ANk 0
cl2k AN1l
dl2k BN1l
58
i1

l2A N

BNk 0 0

59

where AN1k and BN1k are the estimated values

before the adding operation.
If the state error ex 2 H "L t, some wavelets need to
be removed because the network may be over tted. In
this case, remove the wavelets associated with the resolution 2N . In terms of the removing operation (43) and
the expression (55) of ^f j x; u at j N, the structure of
the approximated function ^fx; u is of the following
form:
X
^fx; u
AN2k U N2k x; u
k2AN2

where ak and bk are known as the decomposition

sequences, and l 2 Z n .
In addition, in terms of multiresolution decompositions, the approximation of the function fx; u at the
resolution 2j can be written as
X
^f j x; u
Aj1k U j1k x; u
1
X 2X

j2k x; u

1
X 2X

k2Z

p
X i
W i x 2
dk U 2x k

Aj2k U

Bj1k W

i
x; u
j1k

55

1
X 2X

k2BN2 i1

Bi
W
N2k

i
x; u
N2k

60

The adaptive laws for the parameters AN2k and

Bi
are still given by equations (38) and (39). But,
N2k
using the sequences ak and bi
k , the initial values after the
removing operation are then changed by the decomposition algorithm (Mallat, 1989b) as follows:
X
AN2k 0
al2k ANl
61
l2AN2

BN2k 0

l2BN2

bl2k ANl

62

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

1538

G. P. Liu et al.

where ANk and Bi

Nk are the estimated values before the
removing operation.
Clearly, in both the above cases, the adaptive laws of
the parameters are still given in the form of equations
(38) and (39), based on the above changed parameters.
It also follows that the convergence area of the state
error vector begins with H "U 0 H "L 0 and ends
with H ", where " "U 1.
The determination of the vector sets Aj and B j , for
j j0 ; . . . ; N is also important but simple. The basic rule
for choosing these sets is to make sure 2j x k, for
k 2 Aj or B j , is not out of the valid range of the variables of the scaling function U : or W i :, respectively.
The basis function of the variable wavelet network
can be constructed by tensor products of scalar wavelet
functions, but other constructions are also possible.
Because the wavelet network has multi-resolution capabilities, it can easily achieve the dierent modelling
accuracy. However, the burden in computation and storage of wavelet basis functions rapidly increases with the
dimension n, so the use of the wavelet network is in
practice limited to the case where the dimension n 3.
5.

Simulation

Consider a nonlinear system described by

2

x_ u xu xe0:5x

u2

63

where the input

u 0:5cos1:2t sin1:7t exp sint4 .

Figure 1.

Because n 2, we will need two-dimensional B-spline

wavelets for the wavelet network to identify this nonlinear dynamical system. The fourth order B-splines
were used as the scaling function. Thus, the two-dimensional scaling function is given by
U x; u B4 xB4 u

64

where B4 : is the fourth-order B-spline, which is a

piecewise cubic function (Chui 1992, Wang et al.
1995). For n 2, there are three two-dimensional
mother wavelets expressed by
W

1 x; u

B4 xu

65

2 x; u

xB4 u

66

3 x; u

xu

67

where the one-dimensional mother wavelet x is

10 X
4
X
1k 4
B8 k 1 lB4 2x k
x
8
l
k0 l0

68

For this identi cation, the wavelet network began with

16 wavelons at the resolution 20 and then with 81 wavelons at the resolution 21 using variable wavelet network
strategy. The state errors and the modelling errors for
the two dierent resolutions are shown in gures (1) (4).
The gures denoted as (b) are larger scale versions of the
gures denoted as (a).
As expected, at the beginning of the identi cation
larger state errors and modelling errors exist. After a

^ at the resolution 20 .
The state error x(t) - x(t)

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

Nonlinear system identi cation using wavelet networks

Figure 2.

1539

The modelling error f (x; u) - f^(x; u) at the resolution 20 .

Figure 3.

^ at the resolution 21 .
The state error x(t) - x(t)

while, these errors become smaller and smaller, and

nally they converge to certain ranges. It is clear from
the simulation results that the whole identi cation
scheme is stable from the beginning to the end. It has
also been shown that the state error and the modelling
error decrease with an increase in the resolution of the

wavelet networks by the use of the variable network

approach. Clearly, for nonlinear dynamical system identi cation using wavelet networks, a proper resolution
can be obtained through the proposed variable network
approach to achieve the desired practical identi cation
requirements.

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

1540

G. P. Liu et al.

Figure 4.

6.

The modelling error f (x; u) - f^(x; u) at the resolution 21 .

Conclusions

A wavelet network based identi cation scheme has been

proposed for nonlinear dynamical systems. Two kinds
of wavelet networks, xed and variable wavelet networks, were studied and parameter adaptation laws
were derived which achieve the required estimation
accuracy for a suitable sized network. The parameters
of the wavelet network were adjusted using laws developed by the Lyapunov synthesis approach. By combining wavelet networks with Lyapunov synthesis
techniques, adaptive parameter laws were developed
which guarantee the stability of the whole identi cation
scheme and the convergence of both the network parameters and the state errors. A simulated example was
used to demonstrate the operation of the identi cation
scheme which was realized using B-spline wavelets.
Acknowledgements
The authors gratefully acknowledge the support of the
Engineering and Physical Sciences Research Council
(EPSRC) of U.K. under the contract GR/J46661.
References
BILLINGS, S. A ., 1980, Identi cation of nonlinear systems a survey,
IEE Proceedings, Part D, 127, 272 285.
BILLINGS, S. A ., and VOON, W. S. F ., 1986, A prediction-error and
stepwise-regression estimation algorithm for non-linear systems.
Internationa l Journal of Control, 44, 803 822.
BERIMAN, L ., 1993, Hinging hyperplanes for regression, classi cation
and function approximation. IEEE Trans. Inf. Theory, 39, 999 1013.
BROWN , M ., and H ARRIS, C . J ., 1994, Neurofuzzy Adaptive Modelling
and Control (New York: Prentice-Hall).

C HEN, S., BILLINGS, S. A ., and G RANT, P . M ., 1990, Nonlinear system

identi cation using neural networks. Internationa l Journa l of
Control, 51, 1191 1214.
C HEN, S., and W IGGER, J ., 1995, A fast orthogonal least squares algorithm for e cient subset model selection. IEEE Trans. on Signal
Processing, 43, 1713 1715.
C HUI, C . K ., 1992, An Introduction to Wavelets (San Diego: Academic
Press).
C OCA, D ., and BILLINGS, S. A ., 1996, Continuous-time modelling of
Chuas circuit from noisy measurements using wavelet decompositions. In The Fourth Internationa l Workshop on Nonlinear Dynamics
of electronic Systems, Seville, Spain, pp. 271 276.
D AUBECHIES, I ., 1988, Orthonormal bases of compactly supported
wavelets. Commun. on Pure and Appl. Mathematics, 16, 909 996.
G OODWIN , G . C ., and M AYNE, D . Q ., 1987, A parameter estimation
perspective of continuous time model reference adaptive control.
Automatica, 23, 57 70.
G ROSSMANNAND, A ., and M ORLET, J ., 1984, Decomposition of Hardy
functions into square integrable wavelets of constant shape. SIAM
J. Math. Anal., 15, 723 736.
H ABER, H ., and U NBEHAUEN, H ., 1990, Structure identi cation of
nonlinear dynamic systems A survey on input/output approaches.
Automatica, 26, 651 677.
IEEE , 1996, Scanning the special issue on wavelets. In The Proceedings
of the IEEE, 84, 4.
I OANNOU, P . A ., and K OKOTOVIC, P . V., 1983, Adaptive Systems with
Reduced Models (New York, NY: Springer).
K ADIRKAMANATHAN , V., and N IRANJAN, M ., 1993, A function estimation approach to sequential learning with neural networks. Neural
Computation, 5, 954 957.
K ORENBERG, M ., BILLINGS, S. A ., L IU , Y . P ., and M CLLROY, P . J .,
1988, Orthogonal parameter estimation algorithm for non-linear
stochastic systems. International Journa l of Control, 48, 193 210.
L EONTARITIS, I . J ., and BILLINGS, S. A ., 1987, Model selection and
validation methods for non-linear systems. International Journa l of
Control, 45, 311 341.
L IU, G . P ., BILLINGS, S. A ., and K ADIRKAMANATHAN V., 1998,
Nonlinear system identi cation using wavelet networks. In
Proceedings of Control 98, Swansea, pp. 1248 1253.

Downloaded by [Birla Institute of Technology and Science] at 12:18 23 March 2015

Nonlinear system identi cation using wavelet networks

L IU , G . P ., BILLINGS, S. A ., and K ADIRKAMANATHAN V., 1999a,
Nonlinear system identi cation via variable wavelet networks. In
Proceedings of the 14th IFAC World Congress, Beijing, Vol. H, pp.
109 114.
L IU , G . P ., K ADIRKAMANATHAN V., and BILLINGS, S. A ., 1996a,
Stable sequential identi cation of continuous nonlinear dynamical
systems by growing RBF networks. Int. J. Control, 65, 53 69.
L IU , G . P ., K ADIRKAMANATHAN V., and BILLINGS, S. A ., 1996b,
Variable neural networks for nonlinear adaptive control. preprints
of the 13th IFAC Congress, USA, Vol. F, pp. 181 184.
L IU , G . P ., K ADIRKAMANATHAN V., and BILLINGS, S. A ., 1999b,
Variable Neural Networks for Adaptive Control of Nonlinear
Systems. IEEE Trans. on Systems, Man, and Cybernetics, 29, 34 43.
L JUNG, L ., and G LAD, T ., 1994, Modelling of Dynamic Systems,
Information and System Sciences Series (Englewood Clis, NJ:
Prentice Hall).
M ALLAT, S. G ., 1989a, A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. on Pattern Analysis
and Machine Intelligence, 11, 7, 674 693.
M ALLAT, S. G ., 1989b, Multifrequency channel decompositions of
images and wavelet models. IEEE Trans. Accoust., Speech, and
Signal Process., 37, 2091 2110.

1541

M EYER, Y ., 1993, Wavelets and Operators (Cambridge: Cambridge

University Press).
N ARENDRA, K . S., and A NNASWAMY, A . M ., 1987, A new adaptive law
for robust adaptation with persistent excitation. IEEE Trans. Aut.
Control, 32, 134 145.
N ARENDRA, K . S., and P ARTHASARATHY , K ., 1990, Identi cation and
control of dynamical systems using neural networks. IEEE Trans. on
Neural Networks, 1, 4 27.
P OGGIO, T ., and G IROSI, F ., 1990, Networks for approximation and
learning. In Proceedings of the IEEE, 78, 1481 1497.
SASTRY, S., and BODSON, M ., 1989, Adaptive Control: Stability,
Convergence and Robustness (Engelwood Clis, NJ: Prentice-Hall).
SJOBERG, J ., Z HANG, Q ., L JUNG, L ., BENVENISTE, A ., D ELYON, B.,
G LORENNEC, P ., H JALMARSSON, H ., and J UDITSKY, A ., 1995,
Nonlinear black-box modelling in system identi cation: a uni ed
overview. Automatica, 31, 1691 1724.
W ANG, H ., L IU , G . P ., H ARRIS, C . J ., and BROW N, M ., 1995,
Advanced Adaptive Control (Oxford: Pergamon Press).
Z ADEH, L ., 1994, Fuzzy logic, neural networks, and soft computing.
Commun. ACM, 37, 77 86.
Z HANG, Q ., and BENVENISTE, A ., 1992, Wavelet Networks. IEEE
Trans. on Neural Networks, 3, 889 898.