Sunteți pe pagina 1din 10

Measurement xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

journal homepage:

Interval data fusion with preference aggregation

Sergey V. Muravyov , Liudmila I. Khudonogova, Ekaterina Y. Emelyanova
Department of Control Systems and Mechatronics, National Research Tomsk Polytechnic University, Pr Lenina 30, 634050 Tomsk, Russia


Keywords: It is proposed in the paper the interval data fusion procedure intended for determination of an interval to be
Data fusion consistent with maximal number of given initial intervals (not necessary consistent among each other) and to be
Rank aggregation with maximal likelihood including a value x∗ that can serve as representative of all the given intervals. An
Interval data processing algorithm of the interval fusion with preference aggregation (IF & PA) is proposed and discussed that can be
Weak orders
carried out with help of representation of intervals on the real line by weak order relations (or rankings) over a
Kemeny ranking problem
Triangle numbers
set of discrete values belonging to these intervals. It is possible to determine a consensus ranking for collection of
Preferences discrete values rankings, corresponding to initial intervals. The highest ranked value, accepted as a result of the
Ties fusion, guarantees improved accuracy and robustness of the interval data fusion procedure outputs. It is con-
Robustness sidered a space of weak orders induced by the intervals, its properties and dimension. A reasonable number
Monte Carlo simulation choice problem of discrete values, representing the interval data, is investigated. Related to the problem,
Comparisons computing experiment results and recommendations are given. The interval data fusion procedures can be
widely applied in interlaboratory comparisons, prediction of fundamental constant values on the base of dif-
ferent measured values, conformity testing, enhancement of multisensor readings accuracy in sensor networks,

1. Introduction be absent in the measurement results, i.e. expectation E(εi) = 0; values

εi are assumed to be independent and to have the same variances, i.e. D
Let x be a measurand, xi be a result of i-th measurement, εi be a (εi) = σ2 for all i. Such the observations sometimes are deemed to be
uncertainty of i-th measurement, and m be a number of measurements. equidispersed, that is equally distributed random variables [5,8].
Description of the measurement results in form of intervals Legitimacy of the arithmetic mean use can be easily justified if, first
x i = x ± εi = x + |εi |,i = 1,…,m, (1) of all, to suggest that the intervals are equal to zero, i.e. εi = 0, and to
try to find x from the m equations
(the simplified form xi = x + εi is often used) whose bounds defined by
x = x i,i = 1,…,m. (2)
experimentally obtained or given beforehand uncertainty values is ra-
ther common both in theory and practice of measurement [1–3]. For each separate equation, x is uniquely defined, however, these so-
Usually, sets of such data are input of measurement results processing lutions are inconsistent among each other, and the whole system (2) is
procedure, aim of which, as a rule, is determination of a unique sum- inconsistent. As the system solution, one can use an approximate value
marized value x , by some justified way representing the input intervals. (estimate)
Notice, carrying out of multiple measurements instead of a single one x = f (x1,x2,…,x m) (3)
provides an opportunity to avoid blunders and enhance reliability of the
determination of x . that satisfies all m Eq. (2) with minimal error φ = (x −x i
∑i = 1 )2 .
The classic procedure of direct repeated measurements processing function φ has a minimum in a point where its derivative equals to 0,
dφ m
described in many textbooks and guides in the field of metrology [4–7] i.e. dx = 2 ∑i = 1 (x −x i ) = 0 , whence it appears the estimate x :
allows to find the summarized value x as the arithmetic mean and its 1

expanded uncertainty on the base of data of the same sample belonging x =

∑ x i.
i=1 (4)
to the same population where observations xi were obtained by the
same observer by means of the same methods and instruments under One can see from Eq. (4) that x is the arithmetic mean of measurement
the same ambient conditions. Additionally, the biases are supposed to results xi.

Corresponding author.
E-mail address: (S.V. Muravyov).
Received 5 April 2017; Received in revised form 5 July 2017; Accepted 28 August 2017
0263-2241/ © 2017 Elsevier Ltd. All rights reserved.

Please cite this article as: Muravyov, S.V., Measurement (2017),
S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

In order to check how the existence of uncertainty intervals affects the popular Procedure A [19] used for processing ICL data where the
the estimate (4) let us cancel the above supposition εi = 0 and sub- main task is to establish a reference value xref that characterizes a lar-
1 m
stitute Eq. (1) for xi into Eq. (4); we obtain x = m ∑i = 1 (x + εi) . Ac- gest consistent subset (LCS) of (reliable) measurement results provided
counting that E(εi) = 0, we have the expectation of estimate (4) by participating laboratories. When estimating the reference value xref
m m
the Procedure A uses a weighted mean value y and corresponding un-
⎡1 ⎤ mx
⎞ + E ⎛⎜
1 ⎞ certainty εy calculated by Eqs. (7) and (8) where m is the number of
μ = E (x ) = E ⎢
∑ (x + εi)⎥ = E ⎛
m⎠ m
∑ εi ⎟ = x .
⎣ i=1 ⎦ ⎝ ⎝ i=1 ⎠ (5) participating laboratories; xi is the nominal value estimate provided by
i-th laboratory; and εi is corresponding standard uncertainty. The cal-
It follows from Eq. (5) that the estimate (4) coincides with the mea- culated weighted average value y is accepted as the reference value xref
surand x, i.e. it is unbiased. Taking into account D(ε) = E[ε − E(ε)]2 if it is consistent with the data provided by the participating labora-
and D(aε) = a2D(ε), if a = const, we obtain the variance of estimate (4) tories in accordance with the criterion χ2. If the consistency test is not
m m m passed, it is proposed in [19] to use a scheme of successive exclusion of
⎡1 ⎤ mx
⎞ + D ⎛⎜
1 ⎞ 1
D (x ) = D ⎢ ∑ (x + εi)⎥ = D ⎛ ∑ εi ⎟ = 2 ∑ D (εi) outliers, i.e. measurement results which are not consistent with the
m ⎝m⎠ m m
⎣ i=1 ⎦ ⎝ i=1 ⎠ i=1
others. A result is considered as inconsistent if the following condition
mσ 2 σ2 is valid
= = ,
m 2 m (6)

whence it follows that the standard deviation of mean (4) is m times |x i−y|/ εi2 ± ε 2y > 2, i = 1,…,m. (9)
less than the sample standard deviation σ, that is, the estimate (4) is
consistent. It is well known that the properties of unbiasedness, con- The process of exclusion of one inconsistent result is repeated until the
sistency, and also efficiency [1] of the estimate (4) are valid under consistency of results by the criterion χ2 is confirmed. The reference
normal distribution N(μ, σ2) of measurement results. value for the obtained largest consistent subset is determined by Eq. (7),
There are many situations where it is necessary to find a most where the number of reliable laboratories m' is used instead of m. Clear
trustworthy quantity value and its uncertainty on the base of mea- that the Procedure A can be reasonably applied only if the measurement
surements carried out by different observers by means of various results provided by each participating laboratories are characterized by
methods and instruments in different laboratories and/or ambient a normal probability distribution.
conditions. Series of obtained in this way observations are considered to One can see from the above brief overview that purely statistical
be not equidispersed, if estimates of their variances are considerably methods for interval processing have a lot of limitations imposed on
differ from each other, i.e. D (εi) = σ i2 ≠ D (εi + 1) = σ i2+ 1, i = 1, …, permissible properties of the input intervals, such as normality of
m − 1, and arithmetic means are estimates of the same value μ [4,5,8]. (series of) the data probability distributions, independence of ob-
For example, it may be required to process the non-equidispersed servations, requirement of equidispersion, absence of outliers, etc. The
measurement data in the following situations: typical recommendation to overcome the difficulties would be to use
the non-parametric methods that, due to the reliance on fewer as-
• evaluation of key comparison data provided by national metrology sumptions, are more robust than the parametric ones [20,21]. However,
the non-parametric methods have their own shortcomings, in parti-
institutes (NMIs) for a single stable travelling standard [9,10];
• estimation of uncertainty of a fundamental physical constant that cular, even if they may work well on abnormal data, they have con-
siderably less efficiency in cases when the normal distribution well
can be merely carried out by means of several principally different
and independent methods [11]; enough approximates the measurement results. For example, in stan-
• revealing uncorrected systematic measurement errors, which re- dard [13], in order to check the consistency of different laboratories
measurement results, there are used estimates of laboratory bias, per-
quires use of multiple investigators to measure a quantity in ques-
tion [4,12]; centage differences, ranks and percentage ranks based on calculation of
• interlaboratory comparisons (ILC) to verify the technical compe- robust arithmetic mean and standard deviation. Authors of the docu-
ment [13] point out that many of these methods are unlikely to be
tence of calibration or measurement laboratories where similar
measurements are fulfilled by different laboratories by distinct applicable when the number of participating laboratories is small (e.g.
means, and outcomes are compatible or not [13–15]; and m < 10). Besides, the recommended procedures for outliers’ detection
• analysis of some measuring instrument regular calibrations data can lead to unnecessary data removal [14].
In this paper we propose the approach to interval data processing
accumulated for a long-term period where accuracy of observation
series is different due to change of instrument's metrological per- that fundamentally does not use any their statistical properties and does
formance in the course of time [16,17]. not employ any parametric (like t- and F-tests) and non-parametric
statistical significance tests. We call this approach interval data fusion
The list of the cases can be continued. where the latter two words “data fusion” designate the popular field of
For produced in some of similar situations set of input intervals, the investigations and the former word refers to its interval-oriented spe-
summarized value is defined as the weighted arithmetic mean [4,18] cificity.
Data fusion is a process of joint processing of data on some object
m m
x i ε−i 2/ ∑ ε−i 2, obtained from multiple sources aiming to acquire fuller, more objective
y= ∑
i=1 i=1 (7) and accurate knowledge of a characteristic under investigation than
knowledge derived from a single source. List of data fusion methods
whose uncertainty is usually includes mathematical statistics and probability theory (just in
m the context discussed above), fuzzy sets theory [22], possibility theory
ε 2y = 1/ ∑ ε−i 2, [3], Dempster-Shafer evidence theory [23], Bayesian inference [18],
i=1 (8)
different artificial intelligence methods [24], methods of voting and
where the weight of the measurement result xi is typically the reciprocal preference (or rank) aggregation [2,25].
square of the corresponding standard uncertainty 1/εi2 = 1/σ i2 [4]. It is Under interval data fusion we will understand a procedure of shaping
well known that the weighted mean is the maximum likelihood esti- an interval to be consistent with maximal number of given initial in-
mate of the mean of independent normal distributions N(μ, σ i2 ) with the tervals (not necessary consistent among each other) and to be with
same mean μ [18]. maximal likelihood including a value x∗ that can serve as representative
Consider, for example, how the weighted mean (7) is employed in of all the given intervals. Evidently, both the arithmetic mean x and the

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

weighted mean y mentioned above adequately illustrate the interval

data fusion results x∗.
In addition to the mentioned above cases of non-equidispersed
measurement data (key comparison data, different measured values to
predict fundamental constants, uncorrected systematic measurement
errors, interlaboratory comparison data, instrument regular calibrations
data), the interval data fusion procedures can be applied in conformity
testing [26], for enhancement of multisensor readings accuracy in
sensor networks [27], etc. Moreover, this approach will be definitely
fruitful in applications intended not only for measurement data proces-
sing but for interval data of more common origin, such as distributed
computations, data bases, systems and networks of data acquisition,
The distinctive feature of the proposed in this paper interval data
fusion procedure is the representation of intervals on the real line by
means of weak order relations (or rankings) over a set of discrete values
belonging to these intervals. One may determine a consensus ranking
for collection of discrete values rankings, corresponding to initial in-
tervals. The highest ranked value, accepted as a result of the fusion,
guarantees improved accuracy and robustness of the interval data fu-
sion procedure outputs.
The rest of the paper is organized as follows. Section 2 is devoted to
establishing key concepts and notations. In Section 3, a new notation of
inranking, i.e. ranking induced by interval, structure and properties of
inranking spaces are discussed. In Section 4, we present an algorithm of
interval data fusion using preference aggregation with explaining ex-
ample. In Section 5, we propose a method for partition of a range of
actual values belonging to the initial intervals on the base of Sheppard's
correction for variance of the discrete values. Section 6 is devoted to Fig. 2. Three stages of transition from continuous to discrete values space when forming
numerical experiments justifying the method of partition of the range of the RAV.
actual values and real life applicability and robustness of the proposed
in the paper interval fusion procedure. Conclusions derived from this
and following partition of the union result into equal subintervals.
work are given in Section 7.
However, existence of inconsistent intervals can produce a disjoint
output of the union. Then, we produce the discrete set A, starting from
2. Terms and notations the given continuous intervals {Ik }m
k = 1, in three steps shown in Fig. 2.
First, in order to exclude possible breaks, a least lower bound for all
2.1. Intervals the intervals is chosen to be a RAV's lower bound a1, i.e.

We will consider a collection of m closed intervals {Ik}, k = 1, …, m, a1 = min{lk |k = 1, ..., m}; (10)
in the real line. Each interval is characterized by lower bound lk, upper
bound uk and middle point xk, so that Ik = [lk,uk ]; lk < xk; and as the upper bound an a largest upper bound for these intervals is
xk = 0.5·(uk + lk ) ; lk, uk, xk ∈  (see Fig. 1). taken, i.e.
Frequently, the middle point xk is a result of some measurement,
an = max{uk |k = 1, ..., m}. (11)
uncertainty interval of which is [xk −0.5·(uk−lk ), xk + 0.5·(uk−lk )] =
[xk − εk, xk + εk]. In this case, k-th interval can be represented by Second, to generate elements a2, a3, …, an–1 let us partition the
pair < xk, εk > . obtained interval [a1,an] into n − 1 equal subintervals of length
Any two intervals do not necessary have an intersection, i.e. there
can be such two intervals Ij and Ik, j ≠ k; j, k = 1, …, m, that Ij ∩ Ik ≠ h = (an−a1)/(n−1). (12)
∅. Such the intervals we will call inconsistent.
The length h will be called norm (or mesh) of the partition. Clear
2.2. Range of actual values and its partition that, after partitioning, the norm is defined by the formula

h = |ai−ai − 1 |,i = 2,…,n, (13)

Notice that aim of interval fusion is a selection of a point x∗ on the
real line that belongs to maximal number of intervals from {Ik}. We are and i-th RAV's element is
going to choose the point from finite number of elements connected to
the intervals, though, generally speaking, each of closed intervals in the ai = ai − 1 + h,i = 2,…,n. (14)
real axis is infinite. For this purpose, let us introduce a range of actual
values (RAV) A = {a1, a2, …, an}, over that there exists (inherited from The problem with this second step is that the choice of an appropriate
the real line) binary relation of full order a1 < a2 < … < an, i.e. number n (which is a cardinality of the set A or what is the same of the
transitive, antisymmetric and linear binary relation. RAV partition) in formula (12) must guarantee a necessary and suffi-
The RAV could be shaped by means of union of the intervals {Ik} cient accuracy of values ai, i = 1, …, n, as just one of them will be taken
as the interval fusion result x∗. Selection of n can only be done if the
norm h is known (see Section 5 for details).
0 lk xk uk x Third, the set A = {a1 < a2 < … < an} of fully ordered discrete
values ai, i = 1, …, n, can now be used to shape rankings representing
Fig. 1. An interval Ik on the real line.
the intervals {Ik }m
k = 1.

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

2.3. Rankings Table 1

Spaces of inrankings and forbidden rankings for n = 1, …, 5 and their cardinalities.
Ranking of n elements of a set A = {a1, a2, …, an} in the form λ =
n |Ak| Inrankings Cardinality Tn Forbidden rankings Cardinality Fn
(a1 ≻ a2…∼ as ∼ at ≻ … ∼ an) specifies a binary relation of weak order
(or preference relation) on the set A. It means that the ranking λ is a 1 1 1 1 Ø 0
union of the two following relations: strict preference relation ρ, i.e. ai ≻ 2 1 12 3 Ø 0
aj, and equivalence relation τ (deemed as indifference, or tie), i.e. ai ∼ aj,
2 (12) Ø
that is λ = ρ∪τ. Sometimes, rankings without ties are termed full 3 1 1(23) 6 Ø 1
rankings and rankings including ties – partial rankings. Ties play a 2(13)
crucial role and are supposed to be an inherent part of any ranking in 3(12)
our consideration. 2 (12)3 (13)2
Set of m rankings Λ = {λ1, λ2, …, λm} of n elements is called a
3 (123) Ø
preference profile for m and n given. Each particular preference profile 4 1 1(234) 10 Ø 5
can include both different and multiple similar rankings. 2(134)
3. Inrankings: rankings induced by intervals 2 (12)(34) (14)(23)
(23)(14) (13)(24)
It appeared that far from all possible weak orders over the set A can (34)(12) (24)(13)
represent the intervals {Ik }m 3 (123)4 (124)3
k = 1. Then, one can say about rankings induced
(234)1 (134)2
by the intervals which we will call inrankings and the remaining rank- 4 (1234) Ø
ings that are fundamentally not fit to play this role. 5 1 1(2345) 15 Ø 16
3.1. Representation of intervals by rankings
Let A = {a1, a2, …, an} be the set of n discrete values obtained by 2 (12)(345) (15)(234)
partition of the RAV into equal n − 1 subintervals [28,29]. (23)(145) (14)(235)
For any single interval Ik, one can consider the set A as a union of (34)(125) (13)(245)
(45)(123) (24)(135)
two non-intersected subsets: the subset Ak, including all those elements (25)(134)
of A that belong to the interval Ik, and the complement Ak , including all (35)(234)
the rest elements of A, i.e. A = Ak ∪ Ak , Ak ∩ Ak = ∅, for any k = 1, …, 3 (123)(45) (124)(35)
m (Fig. 3). (234)(15) (125)(34)
(345)(12) (134)(25)
Then, for any k = 1, …, m, some k-th ranking λk induced by interval
Ik and composed of elements of the set A should satisfy the following (145)(23)
conditions for i, j = 1, …, n: (235)(14)
ai ∈ Ak ∧ aj ∉ Ak ⇒ ai ≻ aj; (15) 4 (1234)5 (1235)4
(2345)1 (1245)3
ai,aj ∈ Ak ∨ ai,aj ∉ Ak ⇒ ai ~aj; (16) (1345)2
5 (12345) Ø
ai ∉ Ak ∧ aj ∈ Ak ⇒ ai ≺ aj. (17)

We can treat the k-th ranking to consist of two equivalence classes Ranking that satisfies conditions (15)–(18) we will call a ranking in-
composed of elements of the sets Ak and Ak . What is more, elements of duced by interval, inranking, for short. Thus, the collection of intervals
the class Ak are strictly preferred over elements of the class Ak , i.e. {Ik}, k = 1, …, m, can be represented by the preference profile Λ =
always λk = Ak ≻ Ak . Hence, the ranking contains a single symbol of {λ1, λ2, …, λm}, where any λk is inranking.
strict order ≻ and n − 2 symbols of equivalence ∼. For example, one of
possible ranking for n = 5 can be λk = {a2 ∼ a3 ≻ a1 ∼ a4 ∼ a5} (see
Fig. 3), where Ak = {a2 ∼ a3}, Ak = {a1 ∼ a4 ∼ a5} and Ak ≻ Ak = 3.2. Space of inrankings
{a2 ∼ a3} ≻ {a1 ∼ a4 ∼ a5}.
Series of elements {ai} of the set A is strongly monotonic since A particular preference profile will consist of inrankings taken from
ai < ai+1 for all i ∈ . Clear that the class Ak ⊆ A can include only the space of all possible ones for the certain n. Therefore, the issue how
sequential subseries of elements of A without gaps, see Fig. 3, that is, many all different inrankings exist for different n is of interest.
indexes of these elements must constitute a segment of the natural series. Violation of condition (18) results in existence of forbidden rankings
It means that an index difference of any pair of neighbor elements ai that still satisfy the conditions (15)–(17). Composition of spaces of the
and aj from Ak cannot be larger than 1, i.e. the following condition is all inrankings and forbidden rankings for different n = 1, …, 5 is given
valid: in Table 1, where |Ak| = 1, …, n is the cardinality of set Ak. For the sake
of brevity, we denote ranking elements by their indexes and subsets Ak
ai,aj ∈ Ak are neighboring natural numbers ⇒ j ≡ i + 1. (18) and Ak – by parenthesizing appropriate sets of indexes; symbols ≻ and

Fig. 3. An example of partition of the set A into two subsets

Ak and Ak due to some interval Ik, n = 5, and corresponding
ranking λk.

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

T3 = 6 T2 = 3 T1 = 1
Fig. 4. All feasible intervals and corresponding triangular
numbers for n = 1, …, 5.
n=3 n=2 n=1

a1 a2
a1 a2 a3
n=5 T5 = 15
T4 = 10

a1 a2 a3 a4 a1 a2 a3 a4 a5

∼ are omitted; for example, {a3 ∼ a4 ≻ a1 ∼ a2 ∼ a5} ⇔ (34)(125). 4. Algorithm for fusion of intervals
It follows from consideration of Table 1 (see also Figs. 4 and 5) that
cardinalities of inranking spaces in dependence on n follow the series 1, By interval data fusion we will understand a procedure of determi-
3, 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91, 105, …, which is the integer nation of a point x∗ position in the real line that belongs to maximal
sequence having number A000217 in the OEIS [30]. Elements of this number of intervals from {Ik}, k = 1, …, m, and that can be considered
sequence are called triangle numbers Tn (Fig. 4) and they have quite a to represent all middle points xk at minimal uncertainty ± εk.
few interesting properties [31,32]. A triangle number is defined by We will call the procedure described below interval fusion with pre-
formula ference aggregation (IF & PA). This procedure works on the set A of
discrete values obtained in the way described in Section 2.2 that allows
Tn = n (n + 1)/2. (19)
to represent intervals by rankings. Main stages of the proposed proce-
In turn, cardinalities Fn of forbidden ranking spaces in dependence on n dure IF & PA are as follows. The stages outcomes are illustrated in Fig. 6
comply with the series 0, 0, 1, 5, 16, 42, 99, 219, 466, 968, 1981, 4017, for an example collection of seven initial intervals.
8100, 16278, …, integer sequence A002662 in the OEIS [33]. Cardin-
ality Fn is defined by the formula Stage 1. Formation of the range of actual values from initial intervals
{Ik}, k = 1, …, m, calculation of the norm h, and partitioning the
Fn = 2n−1−n (n + 1)/2. (20) RAV into subintervals to get the set A = {a1, a2, …, an} using Eqs.
It is possible to generate elements of Ak at fixed n using the formula
Stage 2. Composing inrankings in conformity with Eqs. (15)–(18)
n−1 n−j and forming preference profile Λ = {λ1, λ2, …, λm}.
Ak ∈ ⋃ ⋃ {ai + j}.
Stage 3. Determination of value x∗ as the best alternative in con-
j=0 i=1 (21)
sensus ranking for the profile Λ. For this aim, any algorithm can be
The validity of Eq. (21) at n = 5 is justified by considering the contents used that implements some of voting (preference aggregation) rules,
of Table 2. It is evident that Eq. (21) is valid for any n.Taking into for example, simple majority, Condorcet rule, Borda count, Kemeny
account Eq. (21), given n, we have the following common expression rule, approval voting [35], etc. Authors prefer to use the Kemeny
describing all inrankings: rule at this stage which is implemented in developed by them al-
gorithm for solving the Kemeny Ranking Problem [36]. The algo-
n−1 n−j n−j
λk ∈ ⋃ ⎧ ⎜⎛ ⋃ {ai + j} ⎟⎞ ≻ ⎜⎛Ak ⧹ ⋃ {ai + j}⎟⎞ ⎫. rithm produces multiple consensus rankings which should be
j=0 ⎨
⎩ ⎝ i=1 ⎠ ⎝ i=1 ⎠⎬⎭ (22) transformed into a single final consensus ranking. All the consensus
rankings are strict preference relations and the final consensus
Fig. 5 shows inranking spaces as belonging to permutohedra [34] of ranking β is a weak order [37].
order n = 1, …, 4. One can observe number of space dimensions is Stage 4. Deletion of inconsistent intervals in set {Ik}, i.e. those not
increasing with n, namely: the permutohedron of order 1 is a point including the value x∗. Then a cardinality of the output set becomes
(0D); order 2 gives a line segment (1D); order 3 leads to a regular to be equal to m′ .
hexagon (2D); order 4 forms a truncated octahedron (3D), etc. In Fig. 5 Stage 5. Determination of uncertainty ε∗ of the value x∗ as by the
forbidden rankings are not shown but strict orders (though they do not formula
belong to λk) are hold as they participate in forming inrankings. In
ε∗ = min ( max {lk ⩽ x ∗}, min {uk ⩾ x ∗}).
principle, all weak orders are originated from some combinations of the k = 1,...,m′ k = 1,...,m′ (23)
strict orders. Two weak orders in the spaces are shown as connected by
a line if they differ by two elements inversion. Investigation of prop-
erties of these spaces can help to improve quality of algorithms for
determination of consensus ranking for a given profile Λ.

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

Fig. 5. Inranking spaces as belonging to permutohedra of

123 order n = 1, …, 4.

1(23) (12)3
1 213
n=1 132

n=2 (123) 2(13)

12 (12) 21
312 231

3(12) (23)1



Table 2 a7 = 5.0 and h = 0.5.

Towards validation of Eq. (21) for n = 5. The continuous quantity x can be interpreted as a realization of
random variable X and be characterized by estimates of expectation μ
j = |Ak| Elements of Ak
and standard deviation σ. Let us make use of so called Sheppard's
1 {ai}, i = 1, …, n, correction applied to estimate of variance σ2 computed from binned
2 {ai} ∪ {ai+1}, i = 1, …, (n − 1), data [38,39]. In our case, the correction is described by the formula
3 {ai} ∪ {ai+1} ∪ {ai+2}, i = 1, …, (n − 2),
4 {ai} ∪ {ai+1} ∪ {ai+2} ∪ {ai+3}, i = 1, …, (n – 3), h2
5 {ai} ∪ {ai+1} ∪ {ai+2} ∪ {ai+3} ∪ {ai+4}, i = 1, …, (n − 4) σ 2d = σ 2 + ,
12 (24)

where σ2 and σ 2d are variances before (i.e. calculated for continuous

5. Method of the RAV partition values) and after (i.e. computed for discrete values) partition of the
RAV correspondingly. Following to suggestion in [39], from Eq. (24) we
Let us return to the problem of choice of an appropriate number n obtain the relative standard deviation
that must guarantee a necessary and sufficient accuracy of values ai,
i = 1, …, n. Selection of n can only be done if the norm h is known (see σd h2
= 1+ .
paragraph 2.2). In this section, we will try to develop an approach to σ 12σ 2 (25)
calculation of h.
Let w be some admissible error, i.e. difference of the standard de-
We will take into account that partitioning the RAV is similar to
viations σd and σ. The parameter w can be given or selected on the base
binning procedure which is allied to histogramming, rounding, classi-
of requirements to quality (accuracy) of the interval fusion procedure
fying, etc. and, in fact, is a form of quantization. Fig. 7 illustrates this
(see Section 5 and paragraph 6.1). Setting w in a relative form we have
fact showing an example of function that transforms values of con-
tinuous quantity x into discrete values ai for n = 7, where a1 = 2.0; σd ⩽ (1 + w )σ. (26)

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

Intervals Fig. 6. Example of initial collection of intervals {Ik}, k = 1, …, 7;

Preference profile Λ
range of actual values A = {a1, a2, …, a5}; its partition into four
I1 1: a2 ~ a3 a1 ~ a4 ~ a5 meshes; corresponding preference profile Λ = {λ1, λ2, …, λ7}; and
the interval fusion result x* = 4.91.
I2 2: a3 ~ a 4 ~ a 5 a 1 ~ a 2
I3 3: a2 ~ a3 a1 ~ a4 ~ a5
I4 4: a3 ~ a4 a1 ~ a2 ~ a5
I5 5: a 2 ~ a 3 a1 ~ a4 ~ a5
I6 6: a 2 ~ a 3 a1 ~ a4 ~ a5
I7 7: a1 ~ a2 a3 ~ a4 ~ a5

a1 a2 a3 a4 a5

4.17 4.54 4.91 5.28 5.65 : a3 a4 a2 a1 ~ a5

h = (a5 − a1 ) / 4
Consensus ranking
Range of actual values A x* = a3 = 4.91

ai 6. Numerical experimental investigations of IF & PA

To investigate experimentally the proposed method IF & PA (with
4.5 Kemeny rule at stage 3, see Section 3) authors developed a special
software in the environments Microsoft Visual C# and LabVIEW [40].
Input interval data can be both real and simulated by means of a pro-
3.5 gram pseudo-random numbers generator that provides an opportunity
to realize various modifications of the Monte-Carlo method when
3.0 conducting numerical computing experiments. There is a possibility to
2.5 choose uniform or normal distributions of the generated interval data.
h When preparing an experiment, one can preset a nominal value xnom
2.0 (that models a true quantity value), the number of intervals m, the
cardinality of RAV partition n, and the parameters w and σ. The input
intervals are represented as the pairs < xk, εk > with randomly gener-
x ated xk and εk under certain standard deviations σ of the samples.
0 2.0 2.5 3.0 3.5 4.0 4.5 5.0 In this particular experimentation, one experiment run means the
Fig. 7. Transformation function of values of continuous quantity x into discrete values ai generation of N = 1, 2, …, 100 uniformly and/or normally distributed
for n = 7. interval collections (individual problems) in the vicinity of a fixed
nominal value xnom = 3; number of intervals m = 15. An example of
generated data for one (number 33, the normal distribution) of the
Using Eqs. (25) and (26) we can write:
individual problems is shown in Table 3.It is appeared to be convenient
to judge various characteristics of the procedure's performance by the
σd h2
= 1+ ⩽ 1 + w, deviation
σ 12σ 2 (27)
ξ = |x nom−x ∗| (30)
wherefrom after simple transformations we obtain formula for calcu-
lation of the partition norm h: of the interval fusion procedure (IF & PA or other) outcome value x∗
from the nominal value xnom.
h ⩽ σ 24w + 12w 2 . (28)
6.1. Experimental justification of reasonable RAV partition
Accounting Eq. (12) cardinality n of the discrete values set A is defined
by the following expression: Aim of the experimentation was verification of usability of the in-
troduced in Section 5 Eq. (29) for calculation of the cardinality n; and
an−a1 selection of the reasonable value of parameter w.
n=⎡ ⎤ + 1.
⎢ σ 24w + 12w 2 ⎥ (29) There were done 8 experiment runs, each generated 100 individual
⎢ ⎥
problems at normal distribution. Each individual problem has its own
For estimation of the parameter σ one can use the well known re- standard deviation value σ assigned during generation and these values
commendations of GUM [7]. Experimental justification of reasonable were used in calculations by Eq. (29) where appropriate.
selecting parameters w and σ is discussed in paragraph 6.1. Results of processing of three arbitrarily selected out of the eight

Table 3
Generated data for individual problem #33.

k 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

xk 2.918 3.085 3.225 2.491 2.989 3.222 2.738 3.069 2.559 3.100 2.918 2.894 2.626 2.899 3.077
εk 0.190 0.244 0.289 0.053 0.213 0.288 0.133 0.239 0.075 0.249 0.191 0.183 0.097 0.184 0.242

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

Table 4
Values of ξb for probabilities P(ξ ≤ ξb) = 0.90, 0.95, and 1.00 at different values n.

n ξb

Experiment 1 (σ = 0.1–0.5) Experiment 2 (σ = 0.1–0.7) Experiment 3 (σ = 0.1–0.3)

P = 0.90 P = 0.95 P = 1.00 P = 0.90 P = 0.95 P = 1.00 P = 0.90 P = 0.95 P = 1.00

4 0.39 0.48 0.52 0.39 0.48 0.67 0.25 0.31 0.41

5 0.24 0.41 0.74 0.24 0.41 0.74 0.24 0.28 0.44
6 0.31 0.37 0.43 0.31 0.37 0.49 0.20 0.28 0.48
7 0.28 0.40 0.52 0.28 0.40 0.50 0.18 0.24 0.41
8 0.30 0.35 0.51 0.30 0.35 0.64 0.20 0.26 0.37
9 0.23 0.29 0.34 0.23 0.29 0.54 0.17 0.24 0.42
10 0.21a 0.28 0.52 0.28 0.38 0.51 0.17 0.24 0.35
11 0.26 0.35 0.47 0.26 0.35 0.47 0.19 0.23 0.39
12 0.25 0.40 0.45 0.25 0.40 0.59 0.16 0.22 0.33
13 0.22 0.32 0.54 0.22 0.32 0.51 0.17 0.24 0.36
14 0.25 0.29 0.44 0.25 0.28 0.56 0.16 0.19 0.34
15 0.23 0.29 0.43 0.23 0.29 0.37 0.18 0.26 0.33

Minimal values of ξb for the given probability are shown with bold font.

Table 5
Values of ξb for probabilities P(ξ ≤ ξb) = 0.90, 0.95, and 1.00 at different values w.

w ξb

Experiment 1 (σ = 0.1–0.5) Experiment 2 (σ = 0.1–0.7) Experiment 3 (σ = 0.1–0.3)

P = 0.90 P = 0.95 P = 1.00 P = 0.90 P = 0.95 P = 1.00 P = 0.90 P = 0.95 P = 1.00

0.004 0.24 0.28 0.40 0.25 0.28 0.32 0.16 0.20 0.34
0.005 0.23 0.34 0.46 0.22 0.34 0.45 0.17 0.20 0.35
0.006 0.23 0.32 0.41 0.28 0.34 0.47 0.16 0.20 0.35
0.008 0.21 0.34 0.52 0.24 0.36 0.54 0.20 0.24 0.42
0.10 0.23 0.28 0.36 0.27 0.38 0.48 0.17 0.24 0.42
0.13 0.24 0.32 0.46 0.25 0.33 0.64 0.18 0.26 0.37
0.15 0.22 0.35 0.51 0.29 0.33 0.64 0.20 0.26 0.34
0.2 0.26 0.33 0.52 0.29 0.38 0.51 0.17 0.20 0.48
0.3 0.23 0.29 0.45 0.28 0.35 0.49 0.19 0.30 0.48
0.4 0.29 0.36 0.64 0.29 0.37 0.69 0.22 0.28 0.45

collections of 100 individual problems each are shown in Tables 4 and 5 generated interval data were processed by the procedure IF & PA; the
under the names Experiment 1 (with σ varying from 0.1 to 0.5), Ex- probabilities P(ξ ≤ ξb) were estimated; corresponding results are re-
periment 2 (σ = 0.1–0.7), and Experiment 3 (σ = 0.1–0.3). duced to Table 5. Notice that concrete values of n cannot be explicitly
In order to estimate applicability of Eq. (29) for calculation of the shown in Table 5 as for a fixed w they are different in dependence on
cardinality n, we, first, implemented a simple sequential search of n. particular standard deviation σ of corresponding individual problem.
Values of n were taken from the set {4, 5, …, 15}. Low bound n = 4 of One can see from Table 5 that the deviation ξ values, in all the
the set is conditioned by idleness of the RAV partition into less than parameter w range 0.004–0.4, differ not more than (12–26)%. It means
three subintervals. Upper bound n = 15 is due to that the kernel of that the minimal value of w can be recommended to be used in calcu-
IF & PA, exact algorithm for the NP-hard Kemeny Ranking Problem [36] lation of n by Eq. (29), i.e. w = 0.004 = 0.4%. Then Eq. (28) and (29)
takes exponential time for solution. can be simplified to the following empirical equations:
For each value of n, the generated interval data were processed by
the procedure IF & PA; the fusion outcomes x∗ and deviations ξ were h ⩽ 0.31σ (31)
determined and they were used to compute estimates of the probabilities
P(ξ ≤ ξb) that the deviation ξ should not exceed some fixed boundary and
value ξb. For example, if ξ ≤ 0.21 is obtained in 90 individual problems
a −a
out of the 100, then the probability P(ξ ≤ 0.21) = 0.90. The values of n = ⎡ n 1 ⎤ + 1.
⎢ 0.31σ ⎥ (32)
ξb obtained in this way for probabilities of 0.90, 0.95, and 1.00 are
listed in Table 4.
Table 6 is composed of all minimal values (shown with bold font) of ξb
As can be seen from Table 4, various minimal values of ξb are ob-
taken from Table 4 and Table 5 in order to provide more convenient
served at very different cardinalities n. Evidently, in real life situations,
analysis of the data.
where the nominal value xnom is unknown and the sequential search of
It is clear from Table 6 that, at P = 0.95 and w = 0.004, the RAV
n is impossible, a voluntary choice of n can lead to an unpredictable and
partitioning by Eq. (29) allowed to obtain the smallest in our experi-
inaccurate fusion result x∗.
ments boundary value ξb. A discrepancy between minimal values of ξb
Now we are going to demonstrate that the use of Eq. (29) for cal-
computed for the two ways of assigning n, the sequential search and the
culation of the cardinality n provides the least values of ξb to be rather
use of Eq. (29), does not exceed 0.01.
stable for some reasonable values of parameter w and probability P.
Thus, on the base of the conducted numerical experiments, calcu-
The cardinality n was calculated by Eq. (29) at different values of
lation of the cardinality n by Eqs. (29) and (32) can be recommended
w = {0.004, 0.005, …, 0.4}. As before, for each value of n, the same
for partition of the RAV in procedure IF & PA.

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

Table 6
Minimal values of ξb at n obtained by sequential search and at w obtained by calculation by Eq. (29).

Experiment ξb

Sequential search Calculation by Eq. (29)

P = 0.90 P = 0.95 P = 1.00 P = 0.90 P = 0.95 P = 1.00

1 0.21 0.28 (n = 10) 0.34 0.24 0.28 (w = 0.004) 0.40

2 0.22 0.28 (n = 14) 0.37 0.25 0.28 (w = 0.004) 0.32
3 0.16 0.19 (n = 14) 0.33 0.16 0.20 (w = 0.004) 0.34

6.2. Robustness and accuracy of the IF & PA and the distances in case of IF & PA are ca. seven times fewer than those
obtained by the Procedure A for the uniform distribution. Hence, the
We will treat a robustness as an independence of an interval fusion experimental outcomes seem to support the Procedure IF & PA against
procedure outcomes on particular law of interval data distribution. The the procedure A in respect to not only robustness and also accuracy.
defined by Eq. (30) deviation ξ can serve as a measure of a data fusion
process robustness [28,29,40] since if, for some individual problem, values 7. Conclusion
ξ at normal and uniform data distributions are close enough to each other
then the corresponding procedure should be deemed as a robust one. It was proposed an interval data fusion procedure IF & PA on the
To examine the procedure IF & PA for robustness there were gen- base of representation of intervals on the real line by rankings over set
erated 100 uniformly and normally distributed interval collections of belonging to these intervals discrete values. Numerical experimental
(individual problems) at xnom = 3 and m = 15. Some fragment of the investigations have shown that using as a result of the fusion the best
obtained unsorted numerical experimental results is shown in Table 7. value in a consensus ranking found for rankings collection, corre-
For comparison, the data were processed not only by IF & PA but also by sponding to initial intervals, guarantees improved accuracy and ro-
the procedure A [19]. bustness of the interval data fusion procedure outputs.
In Fig. 8 dependencies ξ(N), N = 1, …, 100, are presented that were The proposed procedure starts to work on interval data obtained in
obtained as outcomes of both the procedure IF & PA and the procedure ratio scale, transforms the data into multiple ordinal scale entities, de-
A. The deviation ξ values are sorted in ascending order and shown for termines a best entity with aggregating multidimensional preferences, and
each of the 100 individual problems. The graphs in Fig. 8 in highly the best entity is again a point in ratio scale. We should conclude that the
tangible manner allows to show both robustness and accuracy of the transition from ratio to ordinal scale facilitates revealing essential in-
two compared procedures. formation in input interval data what, in turn, yields property of resilience
The procedure's robustness is demonstrated with a width of the “leaf” (robustness and accuracy) of the procedure IF & PA.
shaped by curves ξ(N) for normal and uniform distributions: the nar-
rower the leaf, the greater robustness of the corresponding procedure. Acknowledgment
One can see from Fig. 8 that in case of IF & PA (curves 3 and 4) the leaf
width does not mainly exceed 0.08. Thus, the procedure IF & PA is This work was supported in part by the Ministry of Education and
explicitly more robust than the procedure A that produced the con- Science of Russian Federation, basic part of the state task “Science”,
siderably wider leaf up to 0.46 (curves 1 and 2). project 2.5760.2017/8.9. The senior author was also partly supported
The procedure's accuracy, that is closeness of the value x∗ obtained by the Tomsk Polytechnic University Competitiveness Enhancement
by it to the nominal value xnom, is characterized with a (average) dis- Program grant, projects VIU-NRiI-23/2016 and VIU-IK-110/2017.
tance between the curve ξ(N) and the horizontal axis: the less the dis- Part of this work was performed while the senior author was at
tance, the higher accuracy of the corresponding procedure. One can see Vignan University, India, for three months in 2016. Vignan University's,
from Fig. 8 that the average distances obtained by IF & PA are ca. one particularly Prof. B. Ramamoorthy's, support is gratefully acknowl-
third of those obtained by the Procedure A for the normal distribution; edged.

Table 7
A fragment of numerical experimental results (unsorted).

Problem number Procedure IF & PA Procedure A

Normal distribution Uniform distribution Normal distribution Uniform distribution

x* ε* ξ x* ε* ξ x* ε* ξ x* ε* ξ

1 2.97 0.24 0.03 2.99 0.02 0.01 2.77 0.13 0.23 2.96 0.05 0.04
2 2.99 0.02 0.01 2.95 0.03 0.05 2.97 0.04 0.03 2.32 0.01 0.68
3 3.03 0.03 0.03 3.01 0.07 0.01 2.99 0.11 0.01 2.88 0.01 0.12
4 2.99 0.23 0.01 3.04 0.04 0.04 2.99 0.12 0.01 2.46 0.01 0.54
5 3.03 0.03 0.03 2.98 0.20 0.02 2.90 0.05 0.11 2.84 0.02 0.16
6 2.99 0.05 0.01 3.07 0.01 0.07 2.97 0.07 0.03 2.89 0.08 0.11
7 2.97 0.03 0.03 3.04 0.06 0.04 2.80 0.11 0.20 2.83 0.03 0.17
8 2.94 0.01 0.06 3.11 0.05 0.11 2.97 0.02 0.03 3.07 0.04 0.07
9 2.98 0.02 0.02 3.23 0.10 0.23 2.94 0.07 0.06 2.88 0.11 0.12

96 3.06 0.08 0.06 2.98 0.20 0.02 2.92 0.09 0.08 2.30 0.05 0.70
97 2.94 0.07 0.06 3.00 0.27 0.00 2.93 0.08 0.07 2.72 0.05 0.28
98 2.92 0.24 0.08 2.97 0.31 0.03 2.89 0.14 0.11 3.10 0.20 0.10
99 2.99 0.01 0.01 2.94 0.09 0.06 2.99 0.04 0.01 2.73 0.08 0.27
100 3.01 0.03 0.01 3.17 0.09 0.17 2.91 0.09 0.09 2.89 0.12 0.11

S.V. Muravyov et al. Measurement xxx (xxxx) xxx–xxx

Fig. 8. Deviations ξ (sorted) obtained with procedure A and procedure

IF & PA for uniform and normaldistributions of interval data.

References [21] V.P. Shulenin, Robust alternatives to the standard deviation in processing of physics
experimental data, Russ. Phys. J. 59 (6) (2016) 824–832.
[22] P. Prajitno, N. Mort, Fuzzy-model-based multisensor data fusion system, in: Proc.
[1] W.Q. Meeker, G.J. Hahn, L.A. Escobar, Statistical Intervals: A Guide for SPIE 4385, Sensor Fusion: Architectures, Algorithms, and Applications, 2001, pp.
Practitioners and Researchers, John Wiley & Sons, New York, 2017. 301–312.
[2] B. Parhami, Distributed interval voting with node failures of various types, in: Proc [23] D.L. Hall, S.A.H. McMullen, Mathematical Techniques in Multisensor Data Fusion,
12th IEEE Workshop on Dependable Parallel, Distributed and Network-Centric Artech House Publishers, Norwood, 2004.
Systems, DPDNS '07, 2007. [24] J. Halpern, Reasoning about Uncertainty, MIT Press, Cambridge, 2003.
[3] D. Dubois, H. Prade, Possibility theory in information fusion, in: G. Della Riccia, H.- [25] B. Parhami, Voting algorithms, IEEE Trans. Reliab. 43 (1994) 617–629.
J. Lenz, R. Kruse (Eds.), Data Fusion and Perception, Springer, Berlin, 2001, pp. [26] L.R. Pendrill, Using measurement uncertainty in decision-making and conformity
53–76. assessment, Metrologia 51 (2014) 206–218.
[4] J.R. Taylor, An Introduction to Error Analysis: The Study of Uncertainties in [27] S.V. Muravyov, L.I. Khudonogova, Multisensor accuracy enhancement on the base
Physical Measurements, University Science Books, 1997. of interval voting in form of preference aggregation in WSN for ecological mon-
[5] P.E. Eliasberg, Measurement Information. Now Much It Do We Need? How Should itoring, in: Proc. 7th Int. Cong. on Ultra Modern Telecommunications and Control
We Process It? Librokom, Moscow, 2011 (in Russian). Systems, ICUMT 2015, 2015, pp. 293–297.
[6] S.G. Rabinovich, Evaluating Measurement Accuracy: A Practical Approach, [28] S.V. Muravyov, I.A. Marinushkina, Processing of interlaboratory comparison data
Springer Science & Business Media, New York, 2013. by preference aggregation method, Meas. Techn. 58 (12) (2016) 1285–1291.
[7] JCGM 100:2008, Guide to the expression of uncertainty in measurement, BIPM. [29] S.V. Muravyov, L.I. Khudonogova, I.A. Marinushkina, Representation of interval
[8] V.V. Lyachnev, T.N. Siraya, L.I. Dovbeta, Fundamentals of Metrology, Elmor, Saint- data by weak orders yields robustness of the data fusion outcomes, J. Phys.: Conf.
Petersburg, 2007 (in Russian). Ser. 722 (2016) 012064.
[9] M.G. Cox, The evaluation of key comparison data, Metrologia 39 (2002) 589–595. [30] The On-Line Encyclopedia of Integer Sequences. The OEIS Foundation. < https://
[10] C. Elster, B. Toman, Analysis of key comparison data: critical assessment of ele-> (last accessed 25.03.17).
ments of current practice with suggested improvements, Metrologia 50 (2013) [31] N.J.A. Sloane, S. Plouffe, The Encyclopedia of Integer Sequences, Academic Press,
549–555. San Diego, 1995.
[11] G. Mana, E. Massa, M. Predescu, Model selection in the average of inconsistent data: [32] E.W. Weisstein, Triangular Number. MathWorld – A Wolfram Web Resource.
an analysis of the measured Planck-constant values, Metrologia 49 (2012) 492–500. <> (last accessed 25.
[12] J. Wiora, A. Kozyra, A. Wiora, A weighted method for reducing measurement un- 03.17).
certainty below that which results from maximum permissible error, Meas. Sci. [33] The On-Line Encyclopedia of Integer Sequences. The OEIS Foundation. < https://
Technol. 27 (2016) 035007.> (last accessed 25.03.17).
[13] ISO 13528:2015. Statistical methods for use in proficiency testing by inter- [34] R.R. Thomas, Lectures in Geometric Combinatorics, American Mathematical
laboratory comparisons, ISO. Society, 2006.
[14] ISO 5725-2:1994/Cor 1:2002. Accuracy (trueness and precision) of measurement [35] D.E. Berg, S. Norine, F.E. Su, R. Thomas, P. Wollan, Voting in agreeable societies,
methods and results – Part 2: Basic method for the determination of repeatability Amer. Math. Monthly 117 (2010) 27–39.
and reproducibility of a standard measurement method, ISO. [36] S.V. Muravyov, Ordinal measurement, preference aggregation and interlaboratory
[15] E. Bashkansky, V. Turetsky, Proficiency testing: binary data analysis, Accred. Qual. comparisons, Measurement 46 (2013) 2927–2935.
Assur. 21 (4) (2016) 265–270. [37] S.V. Muravyov, Dealing with chaotic results of Kemeny ranking determination,
[16] C.M. Wang, H.K. Iyer, Fiducial approach for assessing agreement between two in- Measurement 51 (2014) 328–334.
struments, Metrologia 45 (2008) 415–421. [38] H. Schneeweiss, J. Komlos, A.S. Ahmad, Symmetric and asymmetric rounding: a
[17] D.A. Skoog, F.J. Holler, S.R. Crouch, Principles of Instrumental Analysis, Brooks review and some new results, Adv. Stat. Anal. 94 (2010) 247–271.
Cole, Pacific Grove, 2007. [39] P.-Th. Wilrich, Rounding of measurement values or derived values, Measurement
[18] F. James, Statistical Methods in Experimental Physics, World Scientific, Singapore, 37 (2005) 21–30.
2006. [40] S.V. Muravyov, I.A. Marinushkina, D.D. Garif, Numerical experimental investiga-
[19] M.G. Cox, The evaluation of key comparison data: determining the largest con- tion of comparison data evaluation method using preference aggregation, Acta
sistent subset, Metrologia 44 (2007) 187–200. IMEKO 6 (1) (2017) 20–26.
[20] M. Hollander, D.A. Wolfe, E. Chicken, Nonparametric Statistical Methods, John
Wiley & Sons, New York, 2014.