Reliability Engineering and System Safety 96 (2011) 15271534

Contents lists available at ScienceDirect

Reliability Engineering and System Safety

journal homepage: www.elsevier.com/locate/ress

Failure and reliability prediction by support vector machines regression

of time series data
Marcio das Chagas Moura a,n, Enrico Zio b,c,1, Isis Didier Lins a, Enrique Droguett a
a

Department of Production Engineering, Center for Risk Analysis and Environmental Modeling, Federal University of Pernambuco, Rua Academico Helio Ramos, S/N, Cidade
ria, 50740-530 Recife, PE, Brazil
Universita
b
Department of Energy, Polytechnic of Milan, Via Ponzio 34/3, 20133 Milan, Italy
c
Ecole Centrale Paris et Supelec, France

a r t i c l e i n f o

abstract

Article history:
Received 8 July 2010
Received in revised form
5 May 2011
Accepted 24 June 2011
Available online 2 July 2011

Support Vector Machines (SVMs) are kernel-based learning methods, which have been successfully
adopted for regression problems. However, their use in reliability applications has not been widely
explored. In this paper, a comparative analysis is presented in order to evaluate the SVM effectiveness
in forecasting time-to-failure and reliability of engineered components based on time series data. The
performance on literature case studies of SVM regression is measured against other advanced learning
methods such as the Radial Basis Function, the traditional MultiLayer Perceptron model, Box-Jenkins
autoregressive-integrated-moving average and the Innite Impulse Response Locally Recurrent Neural
Networks. The comparison shows that in the analyzed cases, SVM outperforms or is comparable to
other techniques.
& 2011 Elsevier Ltd. All rights reserved.

Keywords:
Time series regression
Learning methods
Support vector machines
Time-to-failure forecasting and reliability
prediction

1. Introduction
Reliability analysts are challenged by production plant owners
with the task of eliminating costly failures and long shutdowns to
minimize the reduction of prots from production losses and the
negative impact on reputation due to the external consequences
of accidents. The reliability of operating components and systems
varies during their lifetimes; the evolution of this quantitative
performance indicator must be tracked and controlled through
appropriate maintenance and renovation actions, so as to guarantee
the desired level of production and safety.
In the current competitive setting of production markets,
increasing attention has been given to the development of techniques for reliability forecasting as, for example, in [1]. In fact,
prediction modeling of the reliability of an item may be conducted
during various phases of its life-cycle, including the concept
validation and denition, the design, operation and maintenance
phases [2]. At any stage, the obtained reliability estimates serve the
purpose of anticipating performance evolution of the component

Corresponding author. Fax: 55 81 2126 8728.

E-mail addresses: marcio@ceerma.org (M.C. Moura),
enrico.zio@polimi.it (E. Zio), isis.lins@ceerma.org (I.D. Lins),
ealopez@ufpe.br (E. Droguett).
1
Tel.: 39 02 2399 6340; fax: 39 02 2399 6309.
0951-8320/$ - see front matter & 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ress.2011.06.006

so as to allow to take the proper actions for its maintenance and,

possibly, improvement.
Traditionally, stochastic methods have been employed for
evaluating the reliability behavior over time, often under simplifying assumptions to enable analytical or suitable numerical
treatment. For example, homogeneous Poisson processes and
constant failure intensity functions have been widely studied
[3,4], under the assumption that the system does not suffer from
the effects of aging processes. In accordance with this supposition,
the distribution of the number of failures depends only on the
length of the considered time interval; thus, this assumption may
be real only during short time periods [3]. According to perfect
repair, which is another typical simplifying premise [5], the
component is recovered to an as good as new condition as a
result of a repair action; it may serve the purpose for modeling
the reliability of some non-repairable systems through renewal
processes in which the time to failure is a random variable that is
independently and identically distributed with an arbitrary distribution. On the opposite side of perfect repair, stands the
minimal repair assumption. The aim of minimal repair is to
return the component to an as bad as old condition, which
may be described by non-homogeneous Poisson processes. Yet,
the concept of general or imperfect repair rises to tackle intermediary situations between perfect and minimal repairs (see, for
example [6,7]). For recent advances on stochastic point processes
in reliability engineering, see [8].

1528

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

In practice, the reliability measure is inuenced by a set of

time-dependent, external (operational and environmental) variables, which are dependent among them; this induces inhomogeneities in the real reliability performance. In this view, more
sophisticated probabilistic models are needed to capture the
complexities of the reliability behavior in a realistic manner,
giving rise to intricate mathematical formulations and burdensome numerical solutions, which at the end, however, may not
provide the required accuracy of the reliability estimates given
the scarce data available for tuning the model parameters.
In the last decade, new paradigms of reliability modeling have
emerged based on empirical techniques of failure data regression.
In particular, Articial Neural Networks (ANN) have been
proposed as a universal approach for regression of non-linear
functions varying continuously over a time or space domain [9].
Applications of this technique have included reliability estimation. For example, Karunanithi et al. [10], Dohi et al. [11], Cai et al.
[12], Ho et al. [13], Tian and Noore [14,15] and Hu et al. [16] used
both classical and recurrent multilayer perceptron (MLP) neural
networks for forecasting software reliability. Xu et al. [17] applied
feed-forward MLP and Radial Basis Function (RBF) neural
networks to predict engine reliability. A sensitivity analysis was
performed to determine the appropriate architectures of the
neural networks to be used; the experimental results illustrated
that the proposed neural model has superior predictive performance than the traditional MLP and Autoregressive Integrated
Moving Average (ARIMA) models. Su et al. [18] combined ARIMA
models with neural network techniques to predict engine reliability. The resulting model turned out to achieve better prediction
accuracy than the ARIMA, and back-propagation neural network
models were used separately. In [2], an Innite Impulse Response
Locally Recurrent Neural Network (IIR-LRNN) modeling approach
[19] is proposed for forecasting the reliability and the time to
failures of hardware components. To investigate the IIR-LRNN
capabilities in reliability prediction, the predictive performance
of various learning models including MLP, ARIMA and RBF
is evaluated in comparative studies regarding engine failure
processes. The results have shown that IIR-LRNN outperforms
the RBF and ARIMA models and has accuracy comparable to the
MLP model.
Another learning method, which is emerging as a powerful
modeling tool, is known as Support Vector Machines (SVMs).
SVMs are especially useful when the underlying process that
maps inputs into outputs is not known or when such a mathematical relation is very expensive to be obtained due to the
increased number of inuencing interdependent factors. Generally speaking, SVM has been successfully applied to regression
problems [2022], with noteworthy results even when a relatively small data set D {(x1, y1), (x2, y2), y, (xl, yl)} CRd  R is
available for training, where xt are the input vectors, yt the output,
t 1, 2, y, l, d is the dimension of xt and l is the number of
observed input/output pairs.
Experimental results have revealed the superior performance
of SVM with respect to other techniques [23]. The reasons behind
the successful performance of SVM when compared to other
powerful approaches like ANN are twofold. First, rather than
based on the Empirical Risk Minimization (ERM) as ANN, which
only minimizes the training errors, SVM makes use of the
Structural Risk Minimization (SRM) that seeks to minimize an
upper bound on the generalization error, which appears to be
computationally less demanding since the ERM is intended to
deal with large sample sizes [24]. The main idea of SRM consists
in choosing a model with adequate capacity to describe a training
dataset and to accurately predict unseen data, i.e. input/output
pairs not used for training also known as test data [25]. Thus, SVM
focuses on a trade-off between model accuracy during the

training phase and model ability in predicting future values,

whereas ANNs do not look at the generalization performance of
the machine, which may lead to either undertting or overtting
problems. This characteristic tends to improve the SVM effectiveness to forecast upcoming outputs. Second, nding SVM solution
corresponds to dealing with a convex quadratic optimization
problem. Thus, the KarushKuhnTucker (KKT) statements determine necessary and sufcient conditions for a global optimum
[26]. For ANNs, however, it is not guaranteed that even a well
selected optimization algorithm will achieve the global minimum
in nite computation time [27,28]. It is also worth noting that SVM
can be thought of a technique of data compression since it aims at
nding the subset of training data points, which present the whole
information contained in D [30]. Indeed, support vectors are those
points that summarize the information of the training data set and
allow regressing unseen data in an efcient form.
The present paper looks into SVM for reliability forecasting
applications. In spite of the promising results, the use of SVM for
solving reliability prediction problems is still limited: while
nancial market and electric utility forecasting are the two major
contexts treated by SVM, reliability prediction counts only a few
scientic works, as pointed out by Sapankevych and Sankar [29],
who surveyed the predominant research activities related to the
SVM approach. Hence, this paper also aims at reiterating the
effectiveness of SVM for reliability prediction applications; the
prediction results of engine reliability measures are compared
with those previously obtained by other advanced forecasting
techniques: RBF, MLP, ARIMA and IIR-LRNN. These outcomes
indicate the potential of the SVM-based approach, which is shown
capable of providing comparable or more accurate reliability
predictions. Moreover, using a universal technique such as SVM
there is no requirement regarding the specications of parametric
failure time distributions as well as the validation of certain
assumptions necessary for the use of stochastic processes. These
characteristics serve as an attempt to foster the applicability of
SVM to reliability prediction problems.
In this paper, the technique of SVM regression is adopted for
learning time-to-failure series data. After successful training, the
SVM regression model can be used to predict future failure times;
these predictions can be exploited for establishing adequate
maintenance strategies.
The rest of the paper is organized as follows. Table 1 presents
the notation used throughout the paper. Section 2 introduces the
theoretical background of SVM for regression problems. Section 3
Table 1
Notation.
b
C

Intercept of the regression function

Parameter representing the compromise between machine capacity
and training error
D
Training data set
d
Dimension of the input vectors
F
Feature space
f(x)
Regression function
K(xi,xj) Kernel function
l
Number of observations
p
Number of lags in a time series
t
Observation index
x
Input vector
y
Observed output
y^
Predicted output value
w
Weight vector
a, an Lagrange multipliers; dual variables; support vectors indicators
b, bn
Lagrange multipliers; dual variables
e
Width of the tube around the y values
U(x)
Mapping function
s
Parameter of the Gaussian kernel function
Slack variables
n, nn

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

develops the practical examples through which the SVM effectiveness will be challenged against the results obtained from
other techniques. Finally, Section 4 provides some concluding
remarks on the ndings of the research.

2. Support Vector Machines Regression (SVR)

1529

In order to nd the optimal regression hyperplane, besides the

empirical error, one must minimize the l2 norm of the linear
weights (:w:2), which is related to machine capacity (see
[24,25,30] for more details). The primal quadratic optimization
problem that represents the training phase is as follows:
X
2
n
n
Minimize 1=2:w: C
xt xt w,b, x, x
t

SVMs are learning methods, which build their approximation

function based on underlying concepts that rise from Statistical
Learning Theory [24] as well as on a set of input/output examples
that come up from the process under analysis. The regression
learning problem addressed by SVM can be formulated as follows:
there exists an unknown, real-valued and possibly non-linear
mapping y f(x) between a multidimensional input vector x and
output y; the only information available is the dataset D, which is
used over the learning process whose solution is the regression
function f(x) [25]. For Support Vector Regression (SVR), this leads
to solving a convex optimization problem. This section presents
the basic concepts underpinning linear SVR and the extension to
non-linear mappings by kernel functions.
Suppose a linear mapping f(x) is a suitable regression alternative for the problem at hand. This entails nding the linear
regression hyperplane, which best ts (describes) an available set
of input/output training data, so that it can be used to predict
future output values. This function may be written as
f x wT x b
where w and b are, respectively, the weight vector and intercept
of the model to be identied for optimal regression. In SVM, the
training (empirical) error is measured by means of a loss einsensitive function dened as follows [24]:
ex,y,f max0,9yf x9e

which disregards the error if the difference between the predicted

value f(x) and the observed value y is smaller than e; otherwise
Eq. (1) equals the absolute difference between (f(x)  y) and
e. Thus, the e-insensitive loss function denes an error-free tube
of radius e. One may then dene


yf xe x, for data points above the tube


yf xe xn ,

subject to

yt wT xt b r e xt

wT xt byt r e xt

xt Z0

xnt Z 0

where C modulates the trade-off between empirical and generalization errors. The primal Lagrangian function LP is
X
n
n
n
LP w,b, x, x , a, an , b, b 1=2wT w C
xt xt
t

X
X
n n
 bt xt bt xt  at wT xt b2yt e xt
t

X
n
 at n yt wT xt b e xt

where at, ant , bt and bnt are the Lagrange multipliers associated to
the restrictions (2), (3), (4) and (5), respectively. It may be
observed that at and ant both cannot be strictly positive since
there is not a point satisfying (2) and (3) simultaneously.
Eq. (6) needs to be minimized w.r.t. the primal variables w, b,
n, nn and maximized w.r.t. the dual variables a, an, b, bn. KKT
conditions state that the partial derivatives of LP w.r.t. primal
variables must vanish when evaluated at the optimal point. By
introducing some equalities provided by the KKT complementarity conditions back into LP, this later becomes function only of the
dual variables a, an, and the dual Lagrangian (LD) is obtained:
XX
Maximize LD a, an 1=2
a ant aj anj xTt xj
j t
t

X
X
 eyt at  e yt ant , t,j 1,2,. . .,l
t

a, an

for data points below the tube

where x and xn are the slack variables for the two possible and
mutually exclusive situations just presented. Fig. 1 illustrates this
loss function.

subject to

X
at ant 0
t

0 r at r C
0 r ant r C

Fig. 1. e-Insensitive loss function (adapted from [30]).

1530

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

The solution of the dual training problem provides the

estimated regression function:
X
f0 x wT0 x b0
at0 ant0 xTt x b0
8

input vector xt of the p-lagged last values of y itself, as indicated

in Eq. (10):
yt f xt f yt1 ,yt2 ,. . .,ytp

10

where the 0 index represents optimality.

The main idea underlying SVM non-linear regression is to
convey the input vectors xtARd into a higher dimensional feature
space F through a mapping function F(xi) and then solving a
linear regression in F. Note that in Eq. (7) the input data are
presented in the form of dot-products xTt xj, which may be
replaced by FT(xt)F(xj). However, in addition to the selection of
a proper mapping function, the explicit calculation of the dot
products with mapped input vectors can be computationally
expensive. In this way, dot products are replaced by kernel
functions K(xt, xj) dened in the input space that implicitly maps
x into F. One of these functions is the so-called Gaussian kernel:
2

Kxt ,xj exp20:5::xt 2xj : =s

in which s is its parameter. The main benet in using Gaussian
kernels is the possibility of operating in innite dimension spaces.
For further details about kernel functions see [31,32].
Thus, even if the regression function is non-linear in the original
input space, it is linear in the feature space. The SVR training phase
for non-linear regression functions is then analogous to the linear
case, but with xTi xj substituted by K(xi, x). After solving the
associated dual Lagrangian, the estimated regression function is
X
f0 x wT0 x b0
at0 2ant0 Kxt ,x b0
9
t

Note that if either at0 or ant0 is positive, the related xi indeed

contributes to the calculation of the regression functions (Eqs. (8)
and (9)). The observed input vectors with this characteristic are
the so-called support vectors.
There are three parameters to be tuned for setting an SVR with
Gaussian kernel, namely, C, e and s. The SVR performance is
strongly dependent on this triplet of parameters. Generally
speaking, the parameter selection problem consists in choosing
the values of the decision variables C, e and s so as to maximize
the SVM regression performance on test data, i.e., data not used
over the training stage. The quality of the SVR performance on
these data can be measured through the computation of the socalled Normalized Root Mean Square Error (NRMSE):

s
P
yt y^ t 2
NRMSE
P 2
yt
where y^ t is the SVR prediction for the t-th output observation.
Some works have applied heuristics to optimally select the
SVM parameter values: simulated annealing [33,34] and genetic
algorithms [9,23,35]. Here, the Particle Swarm Optimizationbased method described in [36] will be used.
In the next section, the application of SVR for predicting failure
times and reliability values of engine systems is illustrated. Two
literature case studies are considered with data taken from [17].
The results obtained by SVR are compared with other traditional
neural paradigms, such as IIR-LRNN.

3. SVR of time series data for failure and reliability prediction

In accordance with Fuller [37], a real-valued time series can be
considered as a collection of random variables indexed in time
(e.g. stock markets or demand) or a set of observations ordered in
time such as times-to-failure.
In the traditional setting of time series regression, the t-th
output value yt is represented as an (unknown) function of an

where p is the number of back time steps.

The SVR method described in Section 2 may be used for
learning time series data. Along these lines, the data presented
in the application examples discussed in the next two subsections
can be viewed as time series, and SVR can be applied. For
reliability prediction purposes, time series regression is an alternative to probability distribution models and even more sophisticated stochastic process models.

3.1. Forecasting the failure time of turbochargers in diesel engines

The rst example comes from a set of unscheduled maintenance actions observed in 40 turbochargers of a specic type
(Table 2). The turbochargers are a part of most modern diesel
engines; catching the possibly non-linear trend of their failure
times and reliability patterns plays an important role for reliability-based maintenance decision making. In accordance with
Fuller [37], the time-to-failures of Table 2 may be handled as a
time series process and the forecasting of future values may be
made on the basis of past lagged observations.
SVR aims at selecting an optimal function f in the sense of
minimum structural risk. For a given set of values of the SVR
parameters C, e and s, training is performed to catch the unknown
relationship between the time-lagged input and the future output. After this stage, the machine effectiveness is evaluated for
test data.
Two univariate (p 1) SVR settings have been developed. The
rst one builds a single-step ahead or short-term predictive
function, which uses the t-th time-to-failure as input to predict
the (t 1)-th time-to-failure as output. The second makes a twostep ahead prediction of the (t 2)-th time-to-failure, using the
same input as before. The SVR results are compared with the IIRLRNN-based results provided in [2]. As in [17], the data corresponding to the rst 35 failures are used to construct the training
data set. The values of the SVR parameters are shown in Table 3.
Fig. 2(a) and (b) shows the comparison between the actual
(Table 2) and predicted failure times for both single and two-step
ahead SVRs, respectively. The SVM predictions show close agreement
Table 2
Turbocharger failure data from Xu et al. [17].
Failure
order (t)

Time-to-failure
Tt/1000h

Failure
order (t)

Time-to-failure
Tt/1000h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

1.6
2.0
2.6
3.0
3.5
3.9
4.5
4.6
4.8
5.0
5.1
5.3
5.4
5.6
5.8
6.0
6.0
6.1
6.3
6.5

21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

6.5
6.7
7.0
7.1
7.3
7.3
7.3
7.7
7.7
7.8
7.9
8.0
8.1
8.3
8.4
8.4
8.5
8.7
8.8
9.0

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

Table 3
SVM training parameters for the turbocharger example.

1531

Table 4
Test NRMSE values and number of SVs.

SVR parameter

One-step ahead

Two-step ahead

One-step ahead

1486
0.1198
0.2741

1285
0.1715
0.0093

IIR-LRNN

SVR

IIR-LRNN

SVR

0.0149

0.0055
13

0.0199

0.0024
8

e
s

NRMSE
Number of SVs

Two-step ahead

Table 5
Failure data and reliability for the turbocharger.
Failure
order (t)

Time-to-failure/1000h
Real (Tt)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

1.6
2.0
2.6
3.0
3.5
3.9
4.5
4.6
4.8
5.0
5.1
5.3
5.4
5.6
5.8
6.0
6.0
6.1
6.3
6.5
6.5
6.7
7.0
7.1
7.3
7.3
7.3
7.7
7.7
7.8
7.9
8.0
8.1
8.3
8.4
8.4
8.5
8.7
8.8
9.0

Non-parametric
estimate R(Tt)

Predicted

Predicted

one-step (T^ t )

two-step (T^ t )

2.12
2.48
2.94
3.40
4.02
4.38
4.72
4.78
4.92
5.09
5.18
5.38
5.48
5.69
5.90
6.11
6.11
6.21
6.41
6.62
6.62
6.83
7.13
7.23
7.42
7.42
7.42
7.78
7.78
7.88
7.97
8.08
8.18
8.40
8.52
8.52
8.64
8.86
8.97

2.77
3.06
3.51
3.80
4.18
4.49
4.97
5.05
5.22
5.39
5.47
5.65
5.73
5.91
6.09
6.27
6.27
6.36
6.55
6.74
6.74
6.93
7.23
7.33
7.53
7.53
7.53
7.94
7.94
8.04
8.15
8.25
8.36
8.57
8.67
8.67
8.78
8.99

0.9930
0.9831
0.9731
0.9631
0.9532
0.9432
0.9333
0.9233
0.9133
0.9034
0.8934
0.8835
0.8735
0.8635
0.8536
0.8436
0.8337
0.8237
0.8137
0.8038
0.7938
0.7839
0.7739
0.7639
0.7540
0.7440
0.7341
0.7241
0.7141
0.7042
0.6942
0.6843
0.6743
0.6643
0.6544
0.6444
0.6345
0.6245
0.6145
0.6046

Fig. 2. Predictive performance of the SVR: (a) one-step ahead and (b) two-step ahead.

with the real failure times. In Fig. 2, the support vectors (SVs) are
depicted as circles, while the predicted values are square-marked.
Moreover, Table 4 conrms that both single-step and two-step
ahead SVRs outperform the results obtained from IIR-LRNNs in [2];
this empirically corroborates that SVR has good generalization
performance. Table 4 also provides the number of training points,
which are SVs for both one- and two-step ahead predictors.
The SVR predictions obtained can be used for forecasting the
component reliability R(Tt) at failure time Tt, computed as follows
[38] due to the Benard approximation:
RTt 1t0:3=n 0:4

20

where t is the failure index and n is the data sample size (in this
case, n 40).
SVR predictions for the time-to-failure and the non-parametric
reliability estimate of the turbocharger are summarized in
Table 5. Fig. 3(a) and (b) depicts the predicted one-step and
two-step reliability curves as a function of the time-to-failure in
this order. The solid lines represent the actual input pairs
(Tt, R(Tt)) whereas other points represent the SVM (training and
test) predictions (T^ t ,RT^ t ).
3.2. Forecasting miles-to-failure for car engines
Rather than using time-to-failure, the performance of car
engines is usually measured in terms of usage. In this example,

1532

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

Table 6
Miles-between-failures for a car engine.
Index
(t)

Miles between
failures

Index
(t)

Miles between
failures

Index
(t)

Miles between
failures

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

37.143
37.429
37.619
38.571
40.000
35.810
36.286
36.286
36.476
38.191
36.191
36.857
37.619
37.810
38.762
35.905
36.476
36.857
37.143
37.429
37.429
37.619
38.381
38.571
39.429
35.810
36.952
37.619
37.810
38.095
36.857
38.095
38.095
38.381

35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

39.048
37.238
37.333
37.524
37.810
38.571
37.143
37.238
37.619
38.191
38.571
36.095
37.238
37.429
37.524
39.048
37.143
37.810
38.095
38.667
40.062
36.191
36.381
37.048
37.238
38.000
35.714
36.476
37.333
37.619
38.476
36.857
37.143

68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

37.905
38.095
38.857
37.143
37.619
37.619
37.810
38.381
36.381
38.000
38.191
38.667
38.667
37.143
37.619
37.619
38.095
39.048
36.286
37.143
37.524
37.810
38.000
36.857
37.048
37.905
38.191
39.524
35.429
36.000
37.714
38.095
38.571

Fig. 3. Reliability of the turbocharger: (a) one-step and (b) two-step reliability
curves.

the application concerns the evaluation of a reliability indicator for a

specic engine brand from automotive industry; the reliability
indicator is distance (in miles) between two unscheduled and
consecutive corrective maintenance times. The data set is composed
of 100 car engine units and is reported in Table 6 [17].
As done for the previous example and according to Fuller [37],
the available data can also be treated as a time-series since milesto-failure in Table 6 are ordered in time. In order to forecast the
future miles between failures, given past single-lagged failure
observations, an SVR has been developed on 89 training data and
with the parameters C 1159, e 0.5720 and s 7.7970.
Fig. 4 shows the results of the training and test phases of SVM
for forecasting the car engine reliability. From the 89 training
data, 37 were identied as SVs. Table 7 summarizes the SVM
performance in predicting the car engine reliability and also
reports the comparison with the results obtained by IIR-LRNN
[2], MLP, RBF and ARIMA models [17].
Generally speaking, the higher NRSME obtained for this
example w.r.t. the previous one is expected since Fig. 4 does not
present a monotone reliability pattern as that of Fig. 3. In spite of
that, SVR has comparable accuracy to MLP (with Gaussian
activation function), outperforming all other considered techniques, i.e. IIR-LRNN, MLP (logistic activation), RBF and ARIMA.

Fig. 4. SVR predictive performance on the miles between failures of car engines.

4. Conclusions
Predicting reliability is a key issue for proper performance and
maintenance of manufacturing and production industries. The
difculty in catching the dynamics of any reliability indicator is
related to the considerable number of conditioning variables
inuencing its behavior over time. Many stochastic modeling
methods have been applied for predicting reliability, under

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

1533

Table 7
Prediction results of car engine miles-between-failures using various times series models.
Index (t)

Miles between
failures (actual)

SVR

IIR-LRNN

MLP (logistic
activation)

MLP (Gaussian
activation)

RBF (Gaussian
activation)

AR

1
2
3
4
5
6
7
8
9
10

36.857
37.048
37.905
38.191
39.524
35.429
36.000
37.714
38.095
38.571

37.618
37.524
37.810
37.823
38.815
35.621
36.202
37.054
38.255
38.096

36.494
37.305
37.66
38.402
38.977
36.569
36.800
37.093
37.331
38.729

37.026
7.818
37.825
38.545
38.821
36.516
36.475
36.849
37.943
38.719

36.992
37.499
37.755
38.424
39.433
36.115
36.304
36.953
37.291
38.382

37.11
37.753
38.002
37.931
38.206
36.938
36.327
36.879
37.287
38.178

37.355
36.321
37.06
38.301
38.708
39.306
34.177
36.316
38.37
40.18

NRSME

0.0125

0.0158

assumptions which may or may not be met in practice, thus

limiting the ability of modeling real situations. At the same time,
approaches that are capable to handle the complexity of systems
often run into intricate mathematical formulations that signicantly burden the tuning of their parameters and the achievement
of their numerical solution. Due to these factors, data-driven,
empirical techniques have emerged as a powerful alternative for
practical applications of non-linear reliability prediction.
In this paper, regression via SVM was used for forecasting
time-to-failure and reliability time series data of engineered
components. SVR outcomes were compared with the results of
other powerful data-driven empirical methods such as IIR-LRNN,
RBF, the traditional MLP model and the Box-Jenkins ARIMA. The
obtained SVR results outperformed or were comparable to the
ones provided by these other learning machines. Further, no
simplifying parametric assumption was required (as occurs for
lifetime distribution models and stochastic processes) to predict
future failure times. Finally, in general SVR shows noteworthy
performance in handling smooth and wavy time series data.

Acknowledgments
The authors are grateful to the three anonymous referees for
their valuable comments, which have signicantly helped to
enhance the quality of the paper.

References
[1] Moura MC, Droguett EL. Mathematical formulation and numerical treatment
based on transition frequency densities and quadrature methods for
non-homogeneous semi-Markov processes. Reliability Engineering & System
Safety 2009;94:3429.
[2] Zio E, Broggi M, Golea L, Pedroni N. Failure and reliability predictions by
innite impulse response locally recurrent neural networks. In: Proceedings
of the ECCOMAS 5th European Congress on computational methods in
applied science and engineering. Venice, Italy; 2008.
[3] Ross SM. Introduction to probability models. 10 ed. San Diego: Academic
Press; 2010.
[4] Ascher H, Feingold H. Repairable systems reliability: modeling, inference,
misconceptions and their causes. New York: Marcel Dekker, Inc.; 1984.
[5] Doyen L, Gaudoin O. Classes of imperfect repair models based on reduction of
failure intensity or virtual age. Reliability Engineering & System Safety
2004;84:4556.
[6] Yanez M, Joglar F, Modarres M. Generalized renewal process for analysis of
repairable systems with limited failure experience. Reliability Engineering &
System Safety 2002;77:16780.
[7] Gasmi S, Love CE, Kahle W. A general repair, proportional-hazards, framework to model complex repairable systems. IEEE Transactions on Reliability
2003;52:2632.
[8] Krivtsov VV. Recent advances in theory & applications of stochastic point
processes in reliability engineering. Editorial Review, Reliability Engineering
& System Safety 2007;92:54951.

0.0156

0.0122

0.0211

0.0422

[9] Chen K-Y. Forecasting systems reliability based on support vector regression
with genetic algorithms. Reliability Engineering & System Safety 2007;92:
42332.
[10] Karunanithi N, Whitley D, Malaiya YK. Prediction of software reliability using
connectionist models. IEEE Transactions on Software Engineering 1992;18:
56374.
[11] Dohi T, Nishio Y, Osaki S. Optimal software release scheduling based on
articial neural networks. Annals of Software Engineering 1999;8:16785.
[12] Cai KY, Cai L, Wang WD, Yu ZY, Zhang D. On the neural network approach in
software reliability modeling. Journal of Systems and Software 2001;58:
4762.
[13] Ho SL, Xie M, Goh TN. A study of the connectionist models for software reliability
prediction. Computers and mathematics with applications 2003;46:103745.
[14] Tian L, Noore A. Evolutionary neural network modeling for software cumulative failure time prediction. Reliability Engineering & System Safety
2005;87:4551.
[15] Tian L, Noore A. On-line prediction of software reliability using an
evolutionary connectionist model. J. Systems Software 2005;77:17380.
[16] Hu QP, Xie M, Ng SH, Levitin G. Robust recurrent neural network modeling
for software fault detection and correction prediction. Reliability Engineering
& System Safety 2007;92:33240.
[17] Xu K, Xie M, Tang LC, Ho SL. Application of neural networks in forecasting
engine systems reliability. Applied Soft Computing 2003;2:25568.
[18] Su CT, Tong LI, Leou CM. Combining time series and neural network
approaches. Journal of the Chinese Institute of Industrial Engineers 1997;4:
41930.
[19] Back AD, Tsoi AC. FIR and IIR synapses, a new neural network architecture for
time series modelling. Neural Computation 1991;3:37585.

[20] Muller
KR, Smola JA, Ratsch G, Scholkopf
B, Kohlmorgen J, Vapnik VN.
Predicting time series with support vector machines. In: Proceedings of the
7th international conference on articial neural networks (ICANN97).
Lausanne, Switzerland; 1997.

[21] Muller
KR, Smola AJ, Ratsch G, Scholkopf B, Kohlmorgen J. Using support
vector machines for time series prediction. In: Scholkopf B, Burges CJC, Smola
AJ, editors. Advances in Kernel methodssupport vector learning. Cambridge
(MA): The MIT Press; 1999.
[22] Mukherjee S, Osuna E, Girosi F. Nonlinear prediction of chaotic time series
using support vector machines. In: Proceedings of the 1997 IEEE signal
processing society workshop, Neural networks for signal processing VII.
Amelia Island (FL, USA); 1997.
[23] Pai P-F. System reliability forecasting by support vector machines with
genetic algorithms. Mathematical and Computer Modelling 2006;43:26274.
[24] Vapnik VN. The nature of statistical learning theory. 2 ed. New York:
Springer-Verlag; 2000.
[25] Kecman V. Support vector machinesan introduction. In: Wang L, editor.
Support vector machines: theory and applications. Studies in fuziness and
soft computing. Berlin Heidelberg: Springer-Verlag; 2005.

[26] Scholkopf
B, Smola AJ. Learning with kernelssupport vector machines,
regularization, optimization, and beyond. Cambridge (MA): The MIT Press;
2002.
[27] Secchi P, Zio E, Maio FD. Quantifying uncertainties in the estimation of safety
parameters by using bootstrapped articial neural networks. Annals of
Nuclear Energy 2008;35:233850.
[28] Zio EA. Study of the bootstrap method for estimating the accuracy of articial
neural networks in predicting nuclear transient processes. IEEE Transactions
on Nuclear Science 2006;53:146078.
[29] Sapankevych NI, Sankar R. Time series prediction using support vector
machines: a survey. IEEE Computational Intelligence Magazine 2009;4:
2438.
[30] Smola AJ, Scholkoff B. A tutorial on support vector regression. Statistics and
Computing 2004;14:199222.

[31] Scholkopf
B, Burges CJC, Smola AJ. Advances in kernel methods: support
vector learning. Cambridge (MA): MIT Press; 1999.

1534

M.C. Moura et al. / Reliability Engineering and System Safety 96 (2011) 15271534

[32] Cristianini N, Shawe-Taylor J. An introduction to support vector machines

and other Kernel-based learning methods. Cambridge (UK): Cambridge
University Press; 2000.
[33] Pai P-F, Hong W-C. Support vector machines with simulated annealing
algorithms in electricity load forecasting. Energy Conversion and Management 2005;46:266988.
[34] Pai P-F, Hong W-C. Software reliability forecasting by support vector
machines with simulated annealing algorithms. Journal of Systems and
Software 2006;79:74755.

[35] Chen K-Y, Wang C-H. Support vector regression with genetic algorithms in
forecasting tourism demand. Tourism Management 2007;28:21526.
[36] Bratton D, Kennedy J. Dening a standard for particle swarm optimization.
In: Proceedings of the IEEE Swarm intelligence symposium (SIS 2007).
Honolulu, Hawaii; 2007.
[37] Fuller W. Introduction to statistical time series. 2 ed. New York: John Wiley &
Sons; 1996.
[38] Modarres M, Kaminskiy M, Krivtsov V. Reliability engineering and risk
analysisa practical guide. New York, Basel: Marcel Dekker, Inc.; 1999.