Accred Qual Assur
DOI 10.1007/s00769-016-1245-5
GENERAL PAPER
An enhanced bootstrap method to detect possible fraudulent
behavior in testing facilities
Rosembergue P. de Souza1 Luiz F. R. C. Carmo1 Luci Pirmez1
Received: 12 March 2016 / Accepted: 12 November 2016
Springer-Verlag Berlin Heidelberg 2017
Abstract The testing process determines the specified
characteristics of an object. Testing is commonly used to
verify the quality, safety, reliability, efficiency, and envi-
ronmental sustainability of products and services. In Brazil,
the National Institute of Metrology, Quality and Technol-
ogy (Inmetro) is responsible for accrediting the facilities
that undertake testing and calibration activities. Typical
accredited testing services include: water quality, clinical
tests, emissions tests, and vehicle safety tests. Undertaking
fraudulent behavior when testing processes can result in
products that are dangerous to the ordinary citizen and to
the environment. Further, fraudulent behavior produces
unfair competition between concurrent organizations. This
paper presents a method to detect possible fraudulent
behavior in accredited testing facilities. The proposed
technique combines the bootstrap method and the Demp-
sterShafer theory of evidence, to investigate several
fraudulent aspects at once. To validate the proposed
method, we use a Brazilian real-world dataset with real
cases of fraud. The proposed technique successfully detects
the accredited testing facilities with fraudulent behavior.
Keywords Testing  Fraud  Bootstrap 
DempsterShafer theory
& Rosembergue P. de Souza
rosembergue.souza@ppgi.ufrj.br
1 involving comparisons between two or more groups. The
Av. Athos da Silveira Ramos, 274, Predio do CCMN -
Cidade Universitaria, Ilha do Fundao,
Rio de Janeiro 21941-916, Brazil
Introduction
The word fraud describes the actions that are surrepti-
tiously undertaken by one party to obtain an unjust
advantage over another [1]. Undertaking fraudulent
behavior when testing processes can result in products that
are dangerous to the ordinary citizen and to the environ-
ment. Further, fraudulent behavior produces unfair
competition between concurrent facilities. In Brazil, the
National Institute of Metrology, Quality and Technology
(Inmetro) is responsible for accrediting the facilities that
undertake testing and calibration activities.
Inmetro evaluates these accredited organizations annu-
ally to verify their compliance with technical requirements.
Inmetro also maintains a list of trained and qualified
experts. Additionally, Inmetro undertakes some surprise
visits to its accredited organizations. Those in-loco
inspections are to verify whether the accredited organiza-
tions are continually following the technical regulations.
However, the high inspection costs mean that it is not
financially viable to inspect all of the accredited testing
facilities. To enhance the efficiency of such surveillance
activities, we propose a statistical method that combines
the bootstrap method and DempsterShafer theory. The
objective is to identify the facilities with suspicious
behavior; that is, to identify the potential candidates for
Inmetro inspections. On identifying the suspicious facili-
ties, Inmetro could undertake detailed investigations on the
tests performed by these organizations, using the tech-
niques presented in [2, 3].
Bootstrapping is suitable to identify suspicious testing
facilities, because it can be used to test hypotheses
technique does not require any distribution assumptions,
and it is relatively easy to account for the specifics of each
123

situation. Further, bootstrapping allows for the use of
nonstandard test statistics [4]. Some studies use the
bootstrap method as a tool in the fraud detection process.
Taylor et al. examine the suitability of existing methods
for fraud detection. They propose a new technique to
identify fake questionnaire data, in both an informal
graphical mode and as a bootstrap method. The proposed
technique exploits the underlying correlation structure of
a questionnaire and the difficulty in fabricating such
details [5]. Subsequently, considering that fraudulent data
cases have certain clustering structures, Wu and Carlsson
discuss the potential for using statistical clustering
methods in fraud detection. They identify and explore
angular, neighborhood, and repeated measurements clus-
tering. They propose corresponding test statistics and
carry out bootstrap tests [6]. Finally, Kirkwood et al.
study the application of central statistical monitoring
methods, including bootstrapping, in data from real clin-
ical trials. In that case, bootstrapping is based on the sum
of the squared differences between the correlations in
each site and the correlations in all of the data sites [7].
While these proposed fraud detection methods are
somewhat satisfactory, there is room for improvements.
For example, such studies only consider one aspect of
fraud behavior when using bootstrapping.
We use the bootstrap method and DempsterShafer
theory to merge several aspects of fraudulent behavior in
one degree of anomaly. DempsterShafer theory allows us
to combine evidence from different sources and yields a
degree of belief that takes into account all of the available
evidence [8]. This method is particularly useful, because
when data are falsified to pass certain statistical tests,
these falsified data are likely to fail other tests [9].
Combining multiple tests could make it more difficult to
commit fraud.
The remainder of this paper is structured as follows: We
first present some basic concepts, followed by the general
model of the approach used in this work. We then check
the proposed model with real-world data, and the final
section concludes.
Background
Hypothesis testing with bootstrapping
A hypothesis test decides which of two contradictory
claims about a statistical distribution is correct. A sta-
tistical hypothesis is a claim or assertion about (1) the
value of a single parameter, or (2) the values of several
parameters, or (3) the form of an entire probability
distribution. The null hypothesis, denoted by H0, is the
claim that is initially assumed to be true. The
123
Accred Qual Assur
alternative hypothesis, denoted by Ha, is the assertion
that is contradictory to H0. The null hypothesis will be
rejected in favor of the alternative hypothesis only if
the sample evidence suggests that H0 is false. If the
sample evidence does not strongly contradict H0, then
we continue to believe in the plausibility of the null
hypothesis. The two possible conclusions from a
hypothesis testing analysis are (1) reject H0 and (2) fail
to reject H0 [10].
A one-sided hypothesis test is undertaken when the
critical area of the distribution falls under only one of the
distribution tails. The one-sided test decides whether a
parameter is less than, greater than, or equal to some
value. A two-sided hypothesis test is undertaken when the
critical area of the distribution falls under both distribu-
tion tails. The two-sided test decides whether a parameter
is either greater or less than some value. In testing a
hypothesis, the significance level a is the probability of
rejecting the null hypothesis H0 when it is true. The
p value is the smallest level of significance that would
lead to rejection of the null hypothesis H0 with the given
data. To make a proper decision about whether or not to
reject H0, one has to determine a cutoff probability for the
p value; this cutoff is the significance level a. If the
p value is greater than or equal to a, then we fail to reject
H0. Correspondingly, if the p value is less than a, then we
reject H0 [11].
In the bootstrap method, as in other hypothesis tests,
the first step is setting up the null hypothesis H0, the
alternative hypothesis Ha, and the significance level a.
Then, using the significance level, the number M of
resampling with replacement can be decided. M cannot be
less than 1000 for a test at the 5 % level of significance,
and M cannot be less than 5000 for a test at the 1 % level.
The data are then resampled through replacement, and the
statistic of interest S is calculated for the resampled data.
Finally, after repeating this procedure M times, the
empirical distribution is built for S. The value s0 of S for
the observed data is compared with the distribution of
S. For example, in a one-sided test to decide whether a
parameter is greater than some value, if s0 looks like a
typical value from the empirical distribution of S, then the
null hypotheses is accepted. Otherwise, if s0 is unusually
large, then the data are unlikely to have arisen if the null
model is true, and it can be concluded that the alternative
hypothesis is more plausible. The significance level of s0
is the proportion or percentage of values that are as
extreme, or more extreme, than this value in the empirical
distribution of S [4, 1215].
Traditionally, a bootstrap method is used to evaluate one
aspect at a time. To evaluate more than one aspect at the
same time, a fusion techniquesuch as the Dempster
Shafer theoryis more appropriate.
Accred Qual Assur
DempsterShafer theory
The DempsterShafer theory is a mathematical theory of
evidence that it is based on two ideas. The first is the notion
of obtaining degrees of belief for one question based on the
subjective probabilities for a related question. The second
idea is Dempsters rule for combining the degrees of belief
when they are based on independent items of evidence [8].
The set of all of the possible states of the system con-
sidered is called the frame of discernment, which is
represented by X. The set of all of the subsets of X is known
as the power set, which is denoted by the symbol 2X. A
subset of those 2X sets may consist of a single possible state,
or of a conjunction of possible states. Moreover, it is
required that all of the possible states are unique, are not
overlapping, and are mutually exclusive. Pieces of evidence
could be symptoms or events that occurred or may occur
within a system. One piece of evidence is related to a single
possible state, or to a set of possible states [16].
The function, called the basic belief assignment, is the
degree of belief that a possible state or a set of possible
states is warranted, which is denoted by m: 2X ? [0, 1].
This function assigns an evidential weight to a subset in X.
Thus, analogous to the case of probabilities, each possible
event in X is assigned a value, ensuring that the sum of all
of the possible outcomes sums to unity. Within Dempster
Shafer theory, these values denote the belief. A belief
measure is given by the function B: 2X ? [0, 1]. The belief
function can be interpreted as the total amount of justified
support given to a subset in X. Dempsters rule can be used
to combine two independent sources of evidence, expres-
sed by the following equation:
P defined according to the desired significance level. Sub-
C\DZ6; mC mD
mZ P ; 1 sequently, in the empirical distribution of S, we count the
1  C\D; mC mD
with C; D; Z  X and the basic belief assignment functions
m(C) and m(D). In others words, the numerator represents
the accumulated evidence for the sets C and D, which
supports the possible state Z, and the denominator sum
quantifies the amount of conflict between the two sets [16].
Proposed procedure
This study proposes a technique that combines the boot-
strap method with DempsterShafer theory. The objective
is to detect accredited testing facilities that have possible
fraudulent behavior. The first step is to select the sites. We
then perform the enhanced bootstrap method on the
selected sites. Then, when the sites are compared, we list
all of the suspicious sites. This list can be used to identify
the sites that should be inspected to check for fraudulent
behavior.
Enhanced bootstrap method
We combine bootstrapping with DempsterShafer theory
to propose an enhanced bootstrap method. This method
combines several aspects of fraudulent behavior in one
degree of anomaly. We want to test whether some sites are
anomalous compared to others. Our hypotheses for the
proposed enhanced bootstrap method are:
H0: The testing facility is normal
Ha: The testing facility is anomalous
The statistic to discriminate between the null hypothesis
and the alternative is the degree of anomaly S, calculated
using the DempsterShafer theory of evidence. In the case
of anomaly detection, the possible states for the testing
facility are anomalous A or normal (not anomalous)
N. Therefore, the frame of discernment for this context is
X = {A, N}. The power set is 2X = {;, A, N, V}, where
V represents the state of the subset {either A or N}, and ;
represents the empty set. To obtain the degree of anomaly
of a site, we need to model the basic belief assignment
function for all of the evidence available that assign an
anomaly degree to that site. Then, we can calculate degree
of anomaly S using B(A) = S.
The enhanced bootstrap procedure starts with calculat-
ing the degree of anomaly S for the selected site. Next, we
form a pseudo-site by sampling with replacement n sub-
jects from the entire study, where n is the number of
elements in the dataset of the selected site. Then, we cal-
culate the degree of anomaly S for the created pseudo-site.
We repeat this procedure of sampling with replacement and
anomaly degree calculation for M times, where M is
anomaly degree values that are as extreme, or more
extreme, than the anomaly degree of the selected site. If the
counter is larger than aM, then we accept the null
hypothesis H0 for the selected site. Otherwise, the alter-
native hypothesis Ha is more plausible for the selected site.
We perform the enhanced bootstrap procedure for all of the
sites being investigated. When the test rejects the null
hypothesis for a site, then this site is marked as suspicious.
Illustrative case
Complaints about fraud in testing facilities
In Brazil, Inmetro receives complaints and tip-offs about
fraudulent behavior in accredited testing facilities. Some of
this correspondence is from the customers and the
employees of the relevant accredited organization. For
example:
123

Complaint 1: The testing facility exchanges a substan-
dard object with one with no defects to obtain legitimate
results.
Complaint 2: The testing facility edits the test reports
with invented results.
Complaint 3: The testing facility tampered with its
equipment to falsify the test results.
When Inmetro receives complaints about fraudulent
behavior, it undertakes a surprise visit to that testing
facility. There, Inmetro undertakes an in loco evaluation to
verify the facilitys compliance with the law.
Dataset
The dataset used in this study is from six accredited vehicle
testing facilities in Rio de Janeiro, Brazil. The data were
extracted from the sideslip tester measures from each
organization. A sideslip tester is used to verify the safety
conditions in vehicles. Two of these testing facilities were
detected as fraudulent in unexpected visits performed by
Inmetro. Table 1 outlines the dataset used in this study,
including quantitative sideslip measurements, and whether
the testing facility was detected as fraudulent. All of the
data were collected in 2014.
Modeling basic belief assignment function
To use the DempsterShafer theory, we need to model the
basic belief assignment functions from the set of evidence.
First, we must identify the evidence that can indicate
fraudulent behavior in accredited inspection bodies. Some
of the patterns that could reveal fraud are: digit preference,
round number preference, too few or too many outliers, too
little or too much variance, strange peaks, and data that are
too skewed [9].
In the case of vehicle safety tests, we expect a high vari-
ability in the test results. Numerous factors affect the results
across different vehicles, leading to variable results. For
example, as vehicles age and accumulate mileage, they are
more likely to fail the safety tests. Moreover, the vehicle
model, the degree to which the owners maintain their vehicles,
Table 1 Sideslip measurements and detection of fraud in six
accredited vehicle testing facilities in Rio de Janeiro, Brazil, in 2014
Site Sideslip measurements Fraudulent?
1 137 No
2 142 No
3 237 Yes
4 117 No
5 130 No
6 174 Yes
123
Accred Qual Assur
and the misuse of a vehicle can also increase the likelihood of a
vehicle failing the safety tests. In an accredited testing facility,
when falsifying the test results, the fraudster aims for a
determined range of safety limit values. Therefore, the vari-
ability of the results is affected. The first symptom of potential
fraud is in the variability of the test results. The lower this
variability, the higher the suspicion of fraud.
Some of the complaints about fraud in accredited testing
facilities indicate that there may be some mechanism of
editing the test reports with invented results. Humans are
poor random number generators; hence, another way of
checking for fraudulent data is looking for digit preference
[9]. Therefore, the second evidence of potential fraud is digit
preferences in the test results, i.e., the tendency to use some
digits over others when falsifying test results. The higher the
digit preference, the higher the suspicion of fraud.
Using the variability and the digit preference as evi-
dence factors, we can now provide the basic belief
assignment functions. In this case, the weight of the basic
belief assignment function can be interpreted as the degree
of anomaly revealed by that evidence.
Evidence 1: Variability
To measure the variability of the data, we use the standard
deviation. For simplicity, we use a linear function to model
our basic belief assignment function. This measures the
anomaly degree based on the standard deviation of the
data. This basic belief assignment function has to map the
values of the standard deviation r in the range [0, 1]. To
complete our linear model, we need to determine the
domain range of the standard deviation. Where there are at
least three numbers in the dataset, the standard deviation
can never exceed 60 % of the range [17]. In this case, the
range of the sideslip tester is defined by -15 m/km to
?15 m/km. Considering the sideslip measures, a good
estimation for the standard deviation domain is [0, 18].
Equation 2 defines the basic belief assignment function
to measure the anomaly degree based on the standard
deviation of the data.
9
m1 A w1 >
=
m1 N 0 2
>
;
m1 V 1  m1 A
where
18  r
w1 :
18
Evidence 2: Digit preference
Digit preference patterns can reveal possible fraud. We use
the Euclidean distance to compare the distribution of the
Accred Qual Assur

leading (first) digits used from each site with the distribution m1 A m2 A m1 A m2 V m1 V m2 A
from all of the other sites together. First, we calculate the B A
K
proportions of the leading digits in the selected site. This m1 N m2 N m1 N m2 V m1 V m2 N
calculation uses the ratio between the leading digit fre- B N
K
quency and the number of observations on that site m1 V m2 V
p0 ; p1 ; p2 ; . . .; p9 . We do this for the digits 09. Next, we B V
K
calculate the proportions of the leading digits from all of the
where
other sites together. This calculation uses the ratio between
the leading digit frequency and the number of observations K m1 A m2 A m1 A m2 V m1 V m2 A
in all of the remaining sites P0 ; P1 ; P2 ; . . .; P9 . We do this m1 N m2 N m1 N m2 V m1 V m2 N
for the digits 09. Finally, using these two vectors of pro- m1 V m2 V
portions, Eq. 3 calculates the digit preference distance dDP.
q Finally, consider combining multiple pieces of evidence
dDP p0  P0 2    p9  P9 2 3 from different properties of A. We can compute B(A) by
combining any evidence pair and then combine that result
With an index to compare the digit preference between with the remaining bits of evidence [18].
sites, we can now provide a basic belief assignment
function for this evidence. Remember, the higher the digit
preference, the higher the suspicion of fraud. Therefore, the Results
higher the digit preference distance, the higher the anomaly
degree of the site. This section empirically investigates the proposed proce-
For simplicity, we also use a linear function to model our dure. We use a dataset of six real-world sites that contain
basic belief assignment function. This measures the anomaly two real cases of fraud (sites 3 and 6). We use MATLAB
degree based on the digit preference distance of the data. in the analysis. We use a = 1 % level of significance, and
This function has to map the values of the digit preference M = 5000 iterations. Table 2 gives the results.
distance in the range [0, 1]. To complete the linear model, Table 2 shows that each of the six pseudo-sites created
we need to determine the domain range of the digit prefer- has standard deviation and digit preference distance values
ence distance. In an extreme situation, one site could have a that lie in the intervals of our basic belief assignment func-
digit preference for only one digit. The remaining sites could tion models. Therefore, the proposed basic belief assignment
have digit preferences for only the other leading digits. This functions models are valid to compare the testing facilities.
implies that thepmaximum
digit preference distance, as Figure 1 shows the empirical cumulative distribution
defined here, is 2. Then, a good estimation
p for the domain function for the anomaly degree of the six accredited vehicle
of the digit preference distance is [0, 2]. safety inspection bodies in the study. The results shown are
Equation 4 defines the basic belief assignment function after M = 5000 sampling with data replacement using the
to measure the anomaly degree based on the digit prefer- enhanced bootstrap method. The cumulative distribution
ence distance of the data. function returns the probability of the values in the sample
9 being less than (or equal to) some value in the sample [19].
m2 A w1 >
= For site 1, the anomaly degree is S1 = 0.833. The per-
m2 N 0 4
>
; centage of values that are as extreme or more extreme than
m2 V 1  m2 A the anomaly degree of site 1 in the empirical distribution is
greater than the chosen level of significance a = 1 %. In
where
this case, the p value is 100 %. Therefore, we fail to reject
dDP
w2 p :
2 Table 2 Standard deviation range and digit preference distance after
M = 5000 iterations
In each of the previous basic belief assignments, the
Site r range dDP range
basic belief assignments of the zero value for all
m(N) mean that those evidence give a zero degree of not 1 [2.074, 3.333] [0.017, 0.181]
anomalous behavior about the inspection activities [18]. 2 [2.173, 3.280] [0.017, 0.187]
3 [1.913, 3.124] [0.013, 0.127]
DempsterShafer rule of combination 4 [2.051, 3.334] [0.015, 0.213]
5 [2.064, 3.285] [0.017, 0.263]
To combine two independent sources of evidence, we can 6 [2.065, 3.190] [0.018, 0.374]
use the following equations:

123
 Accred Qual Assur

Fig. 1 Empirical cumulative

distribution function and
anomaly degree for the six
accredited testing facilities after
M = 5000 iterations using the
enhanced bootstrap method,
where S1, S2, S3, S4, S5, and S6
are the observed anomaly
degrees for the sites 1, 2, 3, 4, 5,
and 6, respectively

the null hypothesis, and site 1 is not marked as suspicious. The results show that sites 3 and 6 are anomalous com-
Following, for site 2, the anomaly degree is S2 = 0.868. pared with the other sites. Although all of the sites have a
The percentage of values that are as extreme or more similar anomaly degree, the p values of sites 3 and 6
extreme than the anomaly degree of site 2 in the empirical indicate that those anomaly degrees rarely appear on their
distribution is greater than the chosen level of significance empirical distributions.
a = 1 %. In this case, the p value is 75.72 %. Therefore,
we fail to reject the null hypothesis, and site 2 is not
Conclusion
marked as suspicious. On the other hand, for site 3, the
anomaly degree is S3 = 0.926. The percentage of values
We present a technique to identify unusual test result pat-
that are as extreme or more extreme than the anomaly
terns in accredited testing facilities. This technique allows
degree of site 3 in the empirical distribution is less than the
us to map the accredited testing facilities with anomalous
chosen level of significance a = 1 %. In this case, the
behavior, considering several fraudulent aspects at once.
p value is 0.02 %. Therefore, we reject the null hypothesis,
To test the technique, we use a real dataset from
and site 3 is marked as suspicious. Next, for site 4, the
accredited testing facilities in Brazil. The proposed method
anomaly degree is S4 = 0.871. The percentage of values
successfully identifies the sites that presented fraudulent
that are as extreme or more extreme than the anomaly
data. Future studies could investigate methods to auto-
degree of site 4 in the empirical distribution is greater than
matically analyze the video records of the services
the chosen level of significance a = 1 %. In this case, the
performed by the testing facilities.
p value is 60.48 %. Therefore, we fail to reject the null
hypothesis, and site 4 is not marked as suspicious. Fol- Acknowledgements The authors would like to thank the National
lowing, for site 5, the anomaly degree is S5 = 0.879. The Institute of Metrology, Quality and Technology of Brazil for its
percentage of values that are as extreme or more extreme support.
than the anomaly degree of site 5 in the empirical distri-
bution is greater than the chosen level of significance
Compliance with ethical standards
a = 1 %. In this case, the p value is 30.88 %. Therefore,
we fail to reject the null hypothesis, and site 5 is not Conflict of interest The authors declare that they have no conflict of
marked as suspicious. Finally, for site 6, the anomaly interest.
degree is S6 = 0.917. Note that the percentage of values
that are as extreme or more extreme than the anomaly
degree of site 6 in the empirical distribution is less than the References
chosen level of significance a = 1 %. In this case, the
p value is 0.02 %. Therefore, we reject the null hypothesis, 1. Behdad M, Barone L, Bennamoun M, French T (2012) Nature-
inspired techniques in the context of fraud detection. IEEE Trans
and site 6 is marked as suspicious. Syst Man Cybern C 42:12731290
The proposed method successfully identifies the fraud- 2. Souza RP, Carmo LFRC, Pirmez L (2014) Deteccao de Dados
ulent sites, rejecting the null hypothesis for site 3 and site 6. Suspeitos de Fraude em Organismos de Inspecao Acreditados.

123
Accred Qual Assur
XIV Simposio Brasileiro em Seguranca da Informacao e de 11. Montgomery DC, Runger GC (2010) Applied statistics and
Sistemas Computacionais. http://www.sbseg2014.dcc.ufmg.br/
files/anais.pdf. Accessed 6 Nov 2014
3. de Souza RP, Carmo LFRC, Pirmez L (2016) A procedure to
detect suspected patterns of fraudulent behavior in vehicle
emissions tests performed by an accredited inspection body.
Accred Qual Assur 21:323. doi:10.1007/s00769-016-1231-y
4. Manly BFJ (2006) Randomization, bootstrap and Monte Carlo
methods in biology, Chapman and Hall, New Zealand, USA
ISBN 978-0-412-36710-6
5. Taylor RN, McEntegart DJ, Stillman EC (2002) Statistical
techniques to detect fraud and other data irregularities in clinical
questionnaire data. Drug Inf J 36:115125
6. Wu X, Carlsson M (2011) Detecting data fabrication in clinical
trials from cluster analysis perspective. Pharm Stat 10:257264
7. Kirkwood AA, Cox T, Hackshaw A (2013) Application of
methods for central statistical monitoring in clinical trials. Clin
Trials 10:783806
8. Ge Y, Xiong H, Liu C, Zhou Z (2011) A taxi driving fraud
detection system. In: IEEE 11th international conference on data
mining. doi:10.1109/ICDM.2011.18
9. Buyse M, George SL, Evans S, Geller NL, Ranstam J, Scherrer B,
Lesaffre E, Murray G, Edler L, Hutton J, Colton T, Lachenbruch
P, Verma BL (1999) The role of biostatistics in the prevention,
detection and treatment of fraud in clinical trials. Stat Med
18:34353451
10. Devore J (2012) Probability and statistics for engineering and the
sciences. Cengage Learning, Boston, USA. ISBN-10: 0-538-
73352-7
probability for engineers. Wiley, Massachusetts, USA. ISBN13:
978-0-470-05304-1
12. Efron B (1979) Computers and the theory of statistics: thinking
the unthinkable. SIAM Rev 21:460480
13. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap.
Monographs on statistics and applied probability, Vol. 57,
Chapman & Hall, London, UK. ISBN-10: 0412042312
14. Wehrens R, Putter H, Buydens LMC (2000) The bootstrap: a
tutorial. Chemometr Intell Lab 54:3552
15. Good P (1994) Permutation tests: a practical guide to resampling
methods for testing hypotheses. Springer, New York, USA. ISBN
978-1-4757-2348-9
16. Kay RU (2007) Fundamentals of the DempsterShafer theory and
its applications to system safety and reliability modelling. Reliab
Theory Appl 3:173185
17. Croucher JS (2004) An upper bound on the value of the standard
deviation. Teach Stat 26:5455
18. Dong F, Shatz SM, Xu H (2009) Inference of online auction shills
using DempsterShafer theory. In: Sixth international conference
on information technology, New Generations. doi:10.1109/ITNG.
2009.28
19. Downey AB (2014) Think stats: probability and statistics for
programmers. Green Tea Press, Massachusetts. ISBN-10:
1449307116
123