Computers & Operations Research 33 (2006) 12891307

www.elsevier.com/locate/cor
Data envelopment analysis for weight derivation and aggregation
in the analytic hierarchy process
Ramakrishnan Ramanathan

College of Commerce and Economics, Sultan Qaboos University, Post Box 20, Postal Code 123, Oman
Available online 11 November 2004
Abstract
Data envelopment analysis (DEA) is proposed in this paper to generate local weights of alternatives frompair-wise
comparison judgment matrices used in the analytic hierarchy process (AHP). The underlying assumption behind
the approach is explained, and some salient features are explored. It is proved that DEA correctly estimates the
true weights when applied to a consistent matrix formed using a known set of weights. DEA is further proposed to
aggregate the local weights of alternatives in terms of different criteria to compute nal weights. It is proved further
that the proposed approach, called DEAHP in this paper, does not suffer from rank reversal when an irrelevant
alternative(s) is added or removed.
2004 Elsevier Ltd. All rights reserved.
Keywords: Analytic hierarchy process; Data envelopment analysis; Aggregation; Independence of irrelevant alternatives
1. Introduction
Data envelopment analysis (DEA) is one of the most popular tools in production management literature
for performance measurement, while the analytic hierarchy process (AHP) is a popular tool in the eld
of multiple-criteria decision-making (MCDM). MCDM has been succinctly dened as making decisions
in the face of multiple conicting objectives [1]. Many researchers have found similarities between
DEA and MCDM techniques. One of the earliest attempts to integrate DEA with multi-objective linear
programming (a MCDM technique) was provided by Golany [2]. Since then, there have been several
attempts to use the principles of DEA in the MCDM literature. The traditional goals of DEA and MCDM

Tel.: +968 513 333 X2849; fax: +968 514 043.

E-mail address: ramanathan@squ.edu.om (R. Ramanathan).
0305-0548/$ - see front matter 2004 Elsevier Ltd. All rights reserved.
doi:10.1016/j.cor.2004.09.020
1290 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
have been compared by Stewart [3]. The goal of DEAis to determine the productive efciency of a system
or decision-making-unit (DMU) by comparing how well the DMU converts inputs into outputs, while the
goal of MCDM is to rank and select from a set of alternatives that have conicting criteria. It has been
recognized for more than a decade that the MCDM and DEA formulations coincide if inputs and outputs
can be viewed as criteria for performance evaluation, with minimization of inputs and/or maximization
of outputs as associated objectives. [35]
It is not the intention in this paper to go into the details of similarities. Interested readers are referred to
recent reviews of multicriteria methods [6,7], books [8,9], and other related articles including those cited
above [1012]. This paper is concerned about using the concepts of DEAfor addressing specic problems
in the AHP [13], a popular technique of MCDM. The paper begins with brief discussions on DEA and
AHP and a literature survey on previous approaches linking the two methods. The proposed approach
synthesizing DEA concepts in AHP is discussed and illustrated in Section 3. Some of the salient features
of the proposed approach are discussed in more detail in Section 4. The paper ends with a summary and
conclusions of the paper.
2. The methodologies: DEA and AHP
2.1. Data envelopment analysis
DEA has been successfully employed for assessing the relative performance of a set of rms, usually
called the DMU, which use a variety of identical inputs to produce a variety of identical outputs. The
concept of Frontier Analysis suggested by Farrel [14] forms the basis of DEA, but the recent series of
discussions started with the article by Charnes et al. [15].
Assume that the there are N DMUs producing J outputs using I inputs. Let the mth DMU produce
outputs y
mj
using x
mi
inputs. The resulting outputinput structure of DMUs is shown in Table 1. The
objective of the DEA exercise is to identify the DMU that produces the largest amounts of outputs by
consuming the least amounts of inputs. This DMU (or DMUs) is considered to have an efciency score
equal to one. The efciencies of other inefcient DMUs are obtained relative to the efcient DMUs, and are
assigned efciency scores between zero and one. The efciency scores are computed using mathematical
programming.
Since DEA is now a widely recognized technique, it is not described in this paper. Interested readers
are referred to more detailed materials [8,1618]. Four basic DEA models (Models 14) are presented
below. These models can be used to calculate the DEA efciency score of mth DMU. Some of these
models will be used in the analyses later in this paper. The optimal objective function values of Models
1 and 3, when solved, represent the efciency score of the mth DMU. This DMU is relatively efcient if
and only if their optimal objective function value equals unity. As informed earlier, efciency scores for
inefcient units are between zero and one. For inefcient units, DEA also provides those efcient units
(namely peers), which the inefcient units can emulate to register performances that could improve their
efciency scores.
Because of the nature of formulations, the optimal objective function values of Models 2 and 4 represent
the reciprocal of efciency scores. Models 14 assume constant returns to scale (CRS) which is said to
prevail when an increase of all inputs by 1% leads to an increase of all outputs by 1% [19]. Models 1 and
3 are said to be input-oriented as the objective is to produce the observed outputs with a minimum input
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1291
Table 1
DMUs, outputs and inputs for a DEA model
Output 1 Output 2 Output J Input 1 Input 2 Input I
DMU 1 y
11
y
12
y
1J
x
11
x
12
x
1I
DMU 2 y
21
y
22
y
2J
x
21
x
22
x
2I
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
DMU N Y
N1
y
N2
y
NJ
x
N1
x
N2
x
NI
level [20]. Models 2 and 4 are said to be output-oriented. More complex DEA formulations, including the
variable returns to scale (VRS) and Archimedean innitesimals, are also available in literature [8,16,17].
In all these models, decision variables are the values of the multipliers (elements of matrices U and V
for Model 1, U

and V

for Model 2, for Model 3, and for Model 4). Scalars and are also decision
variables for Models 3 and 4. The subscript m denotes the DMU for which efciency computations are
made. X and Y are the matrices of inputs and outputs, respectively, for all the DMUs, while X
m
and Y
m
are the matrices of inputs and outputs, respectively, for the mth DMU.
Model 1
max
U,V
z = V
T
m
Y
m
such that U
T
m
X
m
= 1,
V
T
m
Y U
T
m
X0,
V
T
m
, U
T
m
0,
(1)
Model 2
min
U

,V

= U

m
T
X
m
such that V

m
T
Y
m
= 1,
V

m
T
Y U

m
T
X0,
V

m
T
, U

m
T
0,
(2)
Model 3
min
,

m
such that YY
m
,
X
m
X
m
,
0;
m
free,
(3)
Model 4
max
,

m
such that Y
m
Y
m
,
XX
m
,
0;
m
free.
(4)
1292 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
Goal
C
1
C
2
C
3
C
4
A
1
A
2
A
3
Fig. 1. A simple AHP model.
2.2. The analytic hierarchy process
AHP is one of the most popular MCDM tools for formulating and analyzing decisions. The technique
is employed for ranking a set of alternatives or for the selection of the best in a set of alternatives. The
ranking/selection is done with respect to an overall goal, which is broken down into a set of criteria. A
brief discussion of AHP is provided in this section. More detailed description of AHP and application
issues can be found in Saaty [13]. AHP has been applied to numerous practical problems in the last few
decades [21].
Application of AHP to a decision problem involves four steps [22].
Step 1: Structuring of the decision problem into a hierarchical model. It includes decomposition of
the decision problem into elements according to their common characteristics and the formation of a
hierarchical model having different levels. A simple AHP model (Fig. 1) has three levels (goal, criteria
and alternatives). Though the simple model with three levels shown in Fig. 1 is the most common AHP
model, more complex models containing more than three levels are also used in the literature. For example,
criteria can be divided further into sub-criteria and sub-sub-criteria. Additional levels containing different
actors relevant to the problem under consideration may also be included in AHP studies [13].
Step 2: Making pair-wise comparisons and obtaining the judgment matrix. In this step, the elements
of a particular level are compared with respect to a specic element in the immediate upper level. The
resulting weights of the elements may be called the local weights (to be contrasted with nal weights,
discussed in Step 4).
The opinion of a decision-maker (DM) is elicited for comparing the elements. Elements are compared
pair-wise and judgments on comparative attractiveness of elements are captured using a rating scale (19
scale in traditional AHP [13]). Usually, an element receiving higher rating is viewed as superior (or more
attractive) compared to another one that receives a lower rating. The comparisons are used to form a
matrix of pair-wise comparisons called the judgment matrix A. Each entry a
ij
of the judgment matrix are
governed by the three rules: a
ij
>0; a
ij
=1/a
ji
; and a
ii
=1 for all i. If the transitivity property holds, i.e.,
a
ij
=a
ik
a
kj
, for all the entries of the matrix, then the matrix is said to be consistent. If the property does
not hold for all the entries, the level of inconsistency can be captured by a measure called consistency
ratio [13] (see next step).
For the model shown in Fig. 1, ve judgment matrices need to be elicited fromDMone for estimating
the local weights of criteria with respect to the goal (see Table 2Afor an illustrative judgment matrix), and
one each for computing the local weights of alternatives with respect to each of the four criteria (Table
2BE). All the entries in Table 2AE, except the local weights, are hypothetical values.
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1293
Table 2
Illustration of AHP calculations for the model shown in Fig. 1
A: Comparison of criteria with respect to goal
C
1
C
2
C
3
C
4
Local weights
C
1
1 1 4 5 0.400
C
2
1 1 5 3 0.394
C
3
1/4 1/5 1 3 0.128
C
4
1/5 1/3 1/3 1 0.078
Consistency ratio = 0.088
B: Comparison of alternatives with respect to C
1
A
1
A
2
A
3
Local weights
A
1
1 1/3 5 0.279
A
2
3 1 7 0.649
A
3
1/5 1/7 1 0.072
Consistency ratio = 0.055
C: Comparison of alternatives with respect to C
2
A
1
A
2
A
3
Local weights
A
1
1 1/9 1/5 0.060
A
2
9 1 4 0.709
A
3
5 1/4 1 0.231
Consistency ratio = 0.061
D: Comparison of alternatives with respect to C
3
A
1
A
2
A
3
Local weights
A
1
1 2 5 0.582
A
2
1/2 1 3 0.309
A
3
1/5 1/3 1 0.109
Consistency ratio = 0.003
E: Comparison of alternatives with respect to C
4
A
1
A
2
A
3
Local weights
A
1
1 3 9 0.692
A
2
1/3 1 3 0.231
A
3
1/9 1/3 1 0.077
Consistency ratio = 0.000
F: Final weights of alternatives
A
1
0.261
A
2
0.590
A
3
0.148
1294 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
Step 3: Local weights and consistency of comparisons. In this step, local weights of the elements are
calculated fromthe judgment matrices using the eigenvector method (EVM). The normalized eigenvector
corresponding to the principal eigenvalue of the judgment matrix provides the weights of the correspond-
ing elements. Though EVM is followed widely in traditional AHP computations, other methods are also
suggested for calculating weights, including the logarithmic least-square technique (LLST) [23,24], goal
programming [25], and others [26]. When EVM is used, consistency ratio (CR) can be computed. For a
consistent matrix CR=0. Avalue of CRless than 0.1 is considered acceptable because human judgments
need not be always consistent, and there may be inconsistencies introduced because of the nature of scale
used. If CR for a matrix is more than 0.1, judgments should be elicited once again from the DM till he
gives more consistent judgments. Local weights computed for the illustrative judgment matrices using
EVM and the corresponding CR values are also shown in Table 2AE.
Step 4: Aggregation of weights across various levels to obtain the nal weights of alternatives. Once
the local weights of elements of different levels are obtained as outlined in Step 3, they are aggregated
to obtain nal weights of the decision alternatives (elements at the lowest level). For example, the nal
weight of alternative A
1
is computed using the following hierarchical (arithmetic) aggregation rule in
traditional AHP:
Final weight of A
1
=

j
__
Local weight of A
1
with
respect of criterion C
j
_

_
Local weight of
criterion C
j
__
. (5)
By denition, the weights of alternatives and importance of criteria are normalized so that they sum to
unity.
Note that a geometric variant of the above aggregation rule can also be used, especially when LLST
is used for computing local weights [24]. The nal weights represent the rating of the alternatives in
achieving the goal of the problem.
The nal weights computed using (5) for the illustration are shown in Table 2F. Thus, given the
hypothetical judgments of Table 2AE, alternative A
2
is the most preferred alternative, followed by
alternatives A
1
and then A
3
.
The popularity of AHP stems from its simplicity, exibility, intuitive appeal and its ability to mix
quantitative and qualitative criteria in the same decision framework. Despite its popularity, some short-
comings of AHP have been reported in the literature, which have limited its applicability. The number
of judgements to be elicited in AHP increases as the number of alternatives and criteria increase. This is
often a tiresome and exerting exercise for the DM. The issue of rank reversal [27] is one of the prominent
limitations of traditional AHP. The ranking of alternatives determined by the traditional AHP may be
altered by the addition or deletion of another alternative for consideration. For example, when a new
alternative A
4
is added to the list of alternatives discussed earlier, or when an existing alternative A
1
, A
2
orA
3
is removed, it is possible that their rankings change. Modications to the traditional AHP have been
suggested [24] to avoid the problem of rank reversal.
2.3. A literature survey on linking DEA and AHP
Both DEA and AHP are versatile tools in their own elds. While DEA has traditionally found ap-
plications for performance measurement of DMUs, AHP has been widely applied in MCDM problems
for estimating weights of alternatives when several criteria, both qualitative and quantitative, have to be
considered. In fact, many modern complex issues have been tackled using a combination of both the
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1295
methods. Some articles have used AHP for handling subjective factors and to generate a set of numerical
values, and then used DEA to identify efciency score based on the entire data, including those generated
by the AHP [28,29]. Sinuany-Stern et al. [30] have applied DEA on pairs of units, used the resulting DEA
scores to generate a pair-wise comparison matrix and then applied AHP to generate weights of units from
the matrix.
AHP has also been used to introduce preference information in DEA calculations. Preference informa-
tion is introduced in the form of subjective weights derived using AHP [31,32]. Both these studies have
incorporated AHP weights in DEA using the method of assurance region [33].
Some papers have applied both the methods for a given problem and compared their outputs [3436].
In addition, DEA and AHP have been linked with other techniques for specic applications (e.g., DEA
with discriminant analysis and goal programming [37], AHP with goal programming [38], and AHP
with compromise programming [39]). In this paper, the concepts of efciency measurement in DEA are
integrated with the concepts of weight measurement in AHP.
3. A synthesis of concepts of DEA in deriving weights in AHP
In this section, it is proposed that DEA concepts can be used in the last two steps of applying AHP to a
decision problemnamely, deriving local weights froma given judgment matrix (Step 3) and aggregating
local weights to get nal weights (Step 4).
3.1. Using DEA concepts for deriving local weights from a judgment matrix
In this section, we propose DEA for deriving local weights from a judgment matrix. Efciency calcu-
lations using DEA require outputs and inputs. Each row of the judgment matrix is viewed as a DMU and
each column of the judgement matrix is viewed as an output. Thus a judgement matrix of size n n will
have n DMUs and n outputs. Note that the entries of the matrix are viewed as outputs as they have the
characteristics of outputs, viz. an element having higher rating is viewed better than the one that has a
lower rating. Since DEA calculations cannot be made entirely with outputs and require at least one input,
a dummy input that has a value of 1 for all the DMUs is employed. A comparison of traditional AHP
view and the proposed DEA view of a judgement matrix is shown in Fig. 2. It is proposed in this paper
that the efciency scores calculated using DEA models such as Models 14 could be interpreted as the
local weights of the DMUs.
3.1.1. Local weights using DEA for consistent judgment matrices
When applied to a consistent matrix, for which weights are known, DEA correctly estimates the true
weights. Specically, suppose that a predetermined set of n weights, say w
1
, w
2
, . . . , w
n
, is used to create
a consistent judgment matrix of size n given by
W =
_
_
_
_
w
1
/w
1
w
1
/w
2
w
1
/w
n
w
2
/w
1
w
2
/w
2
w
2
/w
n
.
.
.
.
.
.
.
.
.
.
.
.
w
n
/w
1
w
n
/w
2
w
n
/w
n
_

_
.
1296 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
E
l
e
m
e
n
t

1

E
l
e
m
e
n
t

2

E
l
e
m
e
n
t

n
Element 1 1 a
12
a
1N
Element 2 1/ a
12
1 a
2N

Element N 1/ a
1 N
1/ a
2 N
1
O
u
t
p
u
t

1

O
u
t
p
u
t

2

O
u
t
p
u
t

n
D
u
m
m
y

i
n
p
u
t

DMU 1 1 a
12

a
1N
DMU 2 1/ a
12
1
a
2N
1
1

DMU N 1/ a
1N
1/ a
2 N
1 1
Traditional AHP View Proposed DEA View

Fig. 2. A comparison of the traditional AHP view and the proposed DEA view of a judgement matrix.
Any weight derivation method, when applied to the above matrix W, should generate the true weights
w
1
, w
2
, . . . , w
n
. DEA correctly calculates the true weights for a consistent judgment matrix as proved in
Theorem 1. Note that it is not possible to make a similar verication for inconsistent matrices, because
there is no unique correct answer.
Theorem 1. For a consistent judgment matrix, the local weights of alternatives computed using DEA
coincide with the correct weights used to form the matrix.
Proof. Let there be n elements for comparison, leading to a judgment matrix W of size n. Without loss
of generality, assume that w
1
w
2
w
n
where w
i
represents the true weight of element i.
Model 4, with one dummy constant input, for computing the weight of element m is the following:
Model 6
max
m
such that

i=1
w
i
w
m

mi
,
n

i=1

mi
1,
0;
m
free,
(6)
where are the multipliers. Subscript mrepresents the element for which the efciency is being computed.
Note that, because of the nature of the consistent judgment matrix, the set of constraints Y
m
Y
m
in
Model 4 reduces to a single constraint
m

n
i=1
(w
i
/w
m
)
mi
in Model 6. Again because of the presence
of single constant dummy input, the second set of constraints XX
m
in Model 4 reduces to a single
constraint

n
i=1

mi
1 in Model 6. Further, it has been proved that this constraint reduces to strict equality

n
i=1

mi
= 1 at the optimal solution [40].
Substitute m = n in Model 6 to compute the weight of the last element n. At the optimal solution,

nn
= 1

n1
i=1

ni
where asterisk indicates optimal solution. Substituting in the rst constraint of
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1297
Table 3
A look at the entries of Table 2A from the DEA perspective
DMU Output 1 Output 2 Output 3 Output 4 Input (dummy) DEA efciency
C
1
1 1 4 5 1 1
C
2
1 1 5 3 1 1
C
3
1/4 1/5 1 3 1 0.6
C
4
1/5 1/3 1/3 1 1 0.333
Model 6, we get

_
w
1
w
n

n1
+
w
2
w
n

n2
+ +
w
n1
w
n

n,n1
+
w
n
w
n

nn
,
w
1
w
n

n1
+
w
2
w
n

n2
+ +
w
n1
w
n

n,n1
+
_
1
n1

i=1

ni
_
,
1 +
_
w
1
w
n
1
_

n1
+
_
w
2
w
n
1
_

n2
+ +
_
w
n1
w
n
1
_

n,n1
.
(7)
If w
1
w
2
w
n
, all the entries insquare brackets inthe last equationof (7) will be negative. Hence,
the maximumvalue of

n
occurs when all the

are at their minimum, i.e., when

n1
=

n2
= =

n,n1
=0.
Hence the optimal solution is

n
= 1,

nn
= 1 and

ni
= 0, i = 1, 2, . . . , n 1.
Using a similar logic, it can be proved that

1
= w
n
/w
1
,

2
= w
n
/w
2
, . . . ,

n1
= w
n
/w
n1
. Due
to the nature of the output-oriented envelopment DEA formulation, weights are actually the recip-
rocals of . Thus, the weights of the elements are in the ratio [w
1
/w
n
, w
2
/w
n
, . . . , w
n1
/w
n
, 1] or
[w
1
, w
2
, . . . , w
n1
].
3.1.2. Numerical illustration
As an example, consider Table 2A, which is a judgment matrix for comparing criteria with respect to
the goal. In the proposed approach, the entries of Table 2A are viewed as the performance (row-wise) of
DMUs C
1
, C
2
, C
3
and C
4
in terms of four outputs. The outputinput structure of DEA is shown in Table
3. In this case, any of Models 14 can be applied to get the local weights of the criteria. For example, to
get the nal weight of criterion C
1
, the following model (based on Model 1) can be used:
Model 8
max z =v
11
+ v
12
+ 4v
13
+ 5v
14
s.t. u
11
= 1,
v
11
+ v
12
+ 4v
13
+ 5v
14
u
11
0,
v
11
+ v
12
+ 5v
13
+ 3v
14
u
11
0,
1/4v
11
+ 1/5v
12
+ v
13
+ 3v
14
u
11
0,
1/5v
11
+ 1/3v
12
+ 1/3v
13
+ v
14
u
11
0,
v
11
, v
12
, v
13
, v
14
, u
11
0.
(8)
1298 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
C
r
i
t
e
r
i
o
n

1

C
r
i
t
e
r
i
o
n

2

C
r
i
t
e
r
i
o
n

J
Alternative 1 y
11
y
12
y
1J
Alternative 2 y
21
y
22
y
2J

Alternative N y
N1
y
N2
y
NJ
O
u
t
p
u
t

1

O
u
t
p
u
t

2

O
u
t
p
u
t

J

D
u
m
m
y

i
n
p
u
t

DMU 1 y
11
y
12
y
1J
1
DMU 2 y
21
y
22
y
2J
1

DMU N y
N1
y
N2
y
NJ
1
Traditional AHP View Proposed DEA View
Fig. 3. A comparison of the traditional AHP view and the proposed DEA view of the matrix of local weights of alternatives with
respect to all criteria.
In the above model, the rst subscript (i.e., 1) refers to the reference DMU for which efciency is
being computed (criterion C
1
here). The optimal objective function value of Model 8, when solved, will
give the local weight of criterion C
1
. To get the nal weight of other criteria, models similar to Model 8
should be solved by changing the objective function. The DEA efciency scores representing the local
weights of DMUs (i.e., the criteria here) are presented in the last column of Table 3 (1.000, 1.000, 0.600,
and 0.333).
Weights computed using the DEA approach differs from those obtained using EVM (see Table 2A).
Note that, for comparing with the weights computed by DEA, the local weights obtained using EVM
should be adjusted by dividing all the weights by the largest one, resulting in the weights 1, 0.985, 0.319,
and 0.195. These weights are different from those calculated by DEA (last column of Table 3). The
difference is because of the assumptions underlying the methods.
Note that while DMU C
1
produces more Output 4, DMU C
2
produces more Output 3 and hence it
is not possible to say which one is better than the other. However, DMUs C
1
and C
2
have produced more
or equal amounts of outputs compared to DMU C
3
while all the DMUs produced more or equal amounts
of outputs compared to DMU C
4
. It is not possible to conclusively say that the DM prefers C
1
or C
2
, but
it is possible to conclusively say that he prefers C
1
and C
2
to C
3
, and, C
1
, C
2
and C
3
to C
4
. This leads
to a situation to believe that the DM would rank C
1
and C
2
at the same level, then rank C
3
and rank C
4
at the bottom. Thus DEA assigns in unit efciency scores for C
1
and C
2
, a lower score for C
3
(0.6) and
the lowest score for C
4
(0.333).
3.2. Aggregation of local weights to get nal weights
In this section, we propose DEA to aggregate local weights to get nal weights of elements (Step 4 of
AHP). Suppose the local weights of alternatives in terms of all the criteria are available as shown in Fig.
3. The entry y
mj
represents the local weight of alternative m with respect to criterion j. In traditional AHP,
the arithmetic aggregation rule (5) is employed for aggregation. DEA is proposed in this paper for the
aggregation. When DEA is used, alternatives are considered as DMUs and their local weights in terms of
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1299
the criteria will be viewed as outputs. With a dummy input, the DEA view of the matrix of local weights
is also shown in Fig. 3.
Normally, when local weights are aggregated to nal weights, the importance measures of criteria
(local weights of criteria in this case) are also used. For example, the aggregation rule (5) is weighted
arithmetic aggregation incorporating the local weights of weights. However, DEA does not normally
require the local weights of criteria for aggregation. Hence, aggregation using DEA could be discussed
under two cases: (a) without considering the local weights of criteria for aggregation and (b) considering
local weights of criteria.
Case (a): When DEA is used for aggregation, the importance measures of criteria are automatically
generated by DEA as the values of multipliers using linear programming. In this case, a simple DEA
model (any of models 14) of data in Fig. 3 could be used to get the nal weights of alternatives. Hence,
the local weights of criteria are not necessary in DEA.
Case (b): If needed, the importance measures of criteria can be imposed in DEA using the method of
assurance region [33]. This is done by appending additional constraints that specify relationships among
the multipliers in the original DEA model. For example, the importance of criteria are incorporated in
Model 1 in the form of multipliers v
m1
=d
j
v
mj
(for all j =1, 2, . . . , J and d
1
=1). If criterion 1 is half
and thrice as important as criteria 2 and 3, respectively, then d
2
=1/2 and d
3
=3. An interesting theorem
for this Case (b) is proved in the next section.
The above aggregation procedures assume a simple AHP model. The procedures could be extended
for more complex models having more levels, but we consider the simple model in the remainder of this
section.
3.2.1. Aggregation to nal weights using DEA when relationships among importance of criteria are
exogenously specied
When the importance of criteria are exogenously specied and incorporated in a DEA model by
providing additional constraints restricting the values of multipliers as in Case (b) above, the nal weights
of alternatives estimatedbyDEAis proportional tothe weightedsumof local weights as showninTheorem
2.
Theorem 2. Let the local weights of alternatives with respect to different criteria be given by
_
_
_
_
y
11
y
12
y
1J
y
21
y
22
y
2J
.
.
.
.
.
.
.
.
.
y
N1
y
N2
y
NJ
_

_
,
where y
mj
is the local weight of alternative m with respect to criterion j. There are N alternatives and
J criteria. If the importance of criteria are incorporated in the form of multipliers v
m1
= d
j
v
mj
(for all
j =1, 2, . . . , J and d
1
=1) then nal weights aggregated using DEA is proportional to the weighted sum

J
j=1
d
j
y
mj
for alternative m.
Proof. We use the formulation shown by Model 1 for the proof here. For computing the nal weight of
the alternative m Model 1 can be written as follows:
1300 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
Model 9
max
J

j=1
v
mj
y
mj
subject to
u
m1
= 1;
J

j=1
v
mj
y
nj
u
m1
0; n = 1, 2, . . . , N,
v
mj
, u
m1
0; j = 1, 2, . . . , J.
(9)
When the importance of criteria are incorporated by introducing additional constraints of the form
v
m1
= d
j
v
mj
(for all j = 1, 2, . . . , J and d
1
= 1) in Model 9, the model can be rewritten as follows:
Model 10
max v
m1
J

j=1
d
j
y
mj
s.t.
v
m1
J

j=1
d
j
y
nj
1; n = 1, 2, . . . , N,
v
n1
0.
(10)
Model 10 will choose the value of v
m1
such that the largest value of the constraints v
m1

J
j=1
d
j
y
nj
is
equal to unity for n = 1, 2, . . . , N. Since these constraints are the same for all the N DEA formulations
for nding the efciency of all the N DMUs, the value of v
m1
is the same for all the DEA programs. Thus,
the efciency score of the mth DMU is proportional to

J
j=1
d
j
y
mj
.
Corollary 1. If all the criteria have equal weights, then the nal weights are proportional simple sum

J
j=1
y
mj
.
Proof. If all the criteria have equal weights, the multipliers have equal value. In this case, d
j
= 1 for all
j =1, 2, . . . , J. Hence the efciency score of the mth DMU is proportional to

J
j=1
d
j
y
mj
or

J
j=1
y
mj
.

3.2.2. Numerical illustration

The calculations of the proposed approach can be illustrated using the entries of Table 2BE. The local
weights of alternatives with respect to each of the criteria can be computed using DEA as explained in
Section 3.1. The local weights are shown in Table 4 (columns 25). Aggregation using DEA is discussed
below for the two cases, (a) and (b).
Case (a): Without using the local weights of criteria: In this case, any of the models (1) to (4) are
applied for the local weights in Table 4. The local weights are considered as outputs of alternatives, and
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1301
Table 4
Local and nal weights of alternatives computed using the proposed approach
Local weights of alternatives with Final weights
respect to individual criteria
C
1
C
2
C
3
C
4
Case (a) Case (b)
no restrictions with additional restrictions
on criteria v
m1
= v
m2
;
weights v
m1
= (1/0.6)v
m3
; v
m1
= 3v
m4
A
1
0.714 0.111 1.000 1.000 1.000 0.712
A
2
1.000 1.000 0.600 0.333 1.000 1.000
A
3
0.143 0.556 0.200 0.111 0.556 0.346
a dummy input is introduced. For example, to get the nal weight of alternative A
1
, the following model
(based on Model 1) can be used:
Model 11
max z =0.714v
11
+ 0.111v
12
+ v
13
+ v
14
s.t. u
11
= 1,
0.714v
11
+ 0.111v
12
+ v
13
+ v
14
u
11
0,
v
11
+ v
12
+ 0.6v
13
+ 0.333v
14
u
11
0,
0.143v
11
+ 0.556v
12
+ 0.2v
13
+ 0.111v
14
u
11
0,
v
11
, v
12
, v
13
, v
14
, u
11
0.
(11)
The optimal objective function value of Model 11, when solved, will give the nal weight of alternative
A
1
. To get the nal weight of other alternatives, models similar to Model 11 should be solved by changing
the objective function. The resulting nal weights of all the three alternatives are shown in Table 4 (1.000,
1.000, and 0.556). These weights are logical because A
1
performs well in terms of Criteria C
3
and C
4
,
while A
2
performs well in terms of Criteria C
1
and C
2
, and hence it is not possible to decide which one
of them is better.
Case (b): using the local weights of criteria: In many AHP studies, the importance measures of criteria
are usually considered while calculating the nal weights. Using the values of local weights of criteria
(see Table 3), the following additional constraints can be introduced in the DEA model that calculates the
nal weight of DMU m: v
m1
=v
m2
; v
m1
=(1/0.6)v
m3
; v
m1
=3v
m4
. Specically, when calculating the
nal weight of alternative A
1
, the following constraints should be added to Model 11: v
11
= v
12
; v
11
=
(1/0.6)v
13
; v
11
= 3v
14
. The resulting nal weights of alternatives, shown in the last column of Table 4,
are 0.712, 1.000, and 0.346. As proved in Theorem 2, the nal weights are proportional to the weighted
sum of local weights. For example, for alternative A
1
, the weighted sum can be calculated as [(0.714
1) + (0.111 1) + (1 0.6) + (1 0.333)] = 1.7583. The weighted sum for the three alternatives are
1.7583, 2.471, and 0.856, which are proportional to 0.712, 1.000, and 0.346.
These weights again seem logical when compared with the weights obtained using DEA with no
restrictions on criteria weights. When criteria 3 and 4 are given lower importance, alternative A
1
which
performs well in terms of these two criteria, cannot be regarded as efcient as alternative A
2
. Hence, the
DEA scores now result in lower weight for alternative A
1
.
1302 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
It is suggested that Case (a) should be preferred when calculating nal weights of alternatives unless
there is a strong reason to introduce criteria weights (Case (b)). Even if criteria weights are very important,
the nal weights calculated with and without restrictions (i.e., Cases (a) and (b)) should be presented
together.
Because of the synthesis of concepts of DEA withAHP, the new approach proposed in this paper could
be called data envelopment analytic hierarchy process (DEAHP) (thanks to the common A in both
the acronyms).
4. Some important characteristics of DEAHP
In this section, some important characteristics of DEAHP are explored in more detail.
4.1. Independence of irrelevant alternatives
The rule of independence of irrelevant alternatives (IIR) can be stated as follows. If an alternative is
eliminated from consideration, then the new ordering for the remaining alternatives should be equivalent
(i.e., same ordering) to the original ordering for the same alternatives [41]. This issue of IIR is important
in the context of AHP because it has been reported that traditional AHP does not satisfy this rule [27].
DEAHP satises this rule as discussed below.
Note that in DEAHP, the weights of alternatives (i.e., the efciency scores) are calculated separately
for each alternative using a separate linear programming model. This can be contrasted from the EVM
where weights of all the alternatives are derived simultaneously. In addition, while traditional AHP uses
arithmetic normalization, no such normalization is done in the DEAHP. Further, the DEAHP weights are
calculated relative to the weight of the best rated alternative.
For the discussion below, efcient alternatives are interpreted as relevant alternatives because they play
an important role in the rank ordering of all the alternatives. In a general DEA formulation (which is
applicable for DEAHPalso), if the alternative being eliminated is not an efcient one (i.e., if the alternative
being eliminated is an irrelevant alternative), then the new ranking calculated will again be relative to
the highest ranked one, and the ordering of alternatives will not change. This is proved in Theorem 3
below. For the proof, we use the general DEA inputoutput matrix shown in Table 1. Since DEAHP is a
particular case of DEA, the results are applicable to DEAHP also.
Theorem3. Consider the inputoutput matrix showninTable 1. WhenDEAis appliedtondthe efciency
scores of the DMUs, the relative rankings of DMUs will not change if an inefcient DMU is removed
from the original set of DMUs or added to the original set of DMUs.
Proof. We use the DEA formulation given by Model 1 for the proof. The model maximizes the objective
function by choosing appropriate values of decision variables V
T
m
, U
T
m
. Note that the objective function
(i.e., the efciency score) is 1 for efcient DMUs and less than one for inefcient DMUs.
Consider the constraints V
T
m
Y U
T
m
X0. These represent N constraints, one each for the N DMUs.
First we show that these set of constraints will not be binding (i.e., will not be satised in equation form)
for inefcient DMUs. The proof is given by contradiction. If the constraint is binding for a DMU, say the
kth DMU, then the corresponding constraint for the kth DMU can be written as V
T
m
Y
k
U
T
m
X
k
=0 which
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1303
Table 5
Data used in Ref. [27]. Source: Ref. [24]
Criterion Alternatives A
1
A
2
A
3
A
4
A
1
1 1/9 1 1/9
C
1
A
2
9 1 9 1
Weight 0.333 A
3
1 1/9 1 1/9
A
4
9 1 9 1
A
1
1 9 9 9
C
2
A
2
1/9 1 1 1
Weight 0.333 A
3
1/9 1 1 1
A
4
1/9 1 1 1
A
1
1 8/9 8 8/9
C
3
A
2
9/8 1 9 1
weight 0.333 A
3
1/8 1/9 1 1/9
A
4
9/8 1 9 1
leads to V
T
m
Y
k
= 1 when U
T
m
X
k
= 1. Using the same set of multipliers (V
T
m
,U
T
m
) in the DEA formulation
for kth DMU, we get its efciency score as unity, which means that the kth DMU has to be efcient. Thus
for inefcient DMUs the constraint V
T
m
Y U
T
m
X0 is not binding.
From the theory of sensitivity analysis in Linear Programming [42], it is obvious that the opti-
mal solutions of linear program (Model 1 here) will not change if a non-binding constraint is re-
moved or added to the program. Thus the efciency scores and the relative rankings of DMUs will
not change if an inefcient DMU is removed from the original set of DMUs or added to the original set
of DMUs.
Note that, if the alternative being eliminated is the highest ranked alternative (i.e., if the alternative
being eliminated is a relevant one), the new ranking of remaining set of alternatives could change. Using
the logic of Theorem 3, it is clear that constraints given by V
T
m
Y U
T
m
X0 will be satised in equation
form for efcient DMUs and the theory of LP says that optimal solution of Model 1 may change when
a binding constraint is removed. The new DEA efciency scores are calculated relative to the weight
of the next highest ranked alternative. The efciency scores of those inefcient alternatives for which
the removed alternative is a peer will improve while the scores of other inefcient units for which the
removed alternative is not a peer will not change.
4.1.1. Illustration
It can be shown via illustration that DEAHP does not suffer from rank reversal when copies of alter-
natives are added. The famous rank-reversal problem pointed out by Belton and Gear [27] (as reported
in Ref. [24], see also Ref. [43]) in the context of traditional AHP is used as illustration here. The pair-
wise comparison data used by Belton and Gear [27] is shown in Table 5. Three alternatives, A
1
, A
2
,
and A
3
, and three criteria with equal weights were initially considered. Note that the entries in all the
matrices are consistent. The nal weights of the alternatives using the traditional AHP are 0.45, 0.47
1304 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
Table 6
Final weights of alternatives computed using DEAHP for the data in Table 5
No restrictions on criteria weights Equal criteria weights
No copy added Copy of A
2
is No copy added Copy of A
2
is
included as A
4
included as A
4
A
1
1.000000 1.000000 0.94737 0.94737
A
2
1.000000 1.000000 1.00000 1.00000
A
3
0.200000 0.200000 0.15789 0.15789
A
4
1.000000 1.00000
and 0.08, which show that A
2
is the alternative with the largest weight. When a copy A
4
of A
2
is
added to the set, the nal weights calculated using traditional AHP become 0.37, 0.29, 0.06, and 0.29,
which show A
1
as alternative with the largest weight, leading to a reversal in the rank ordering of the
alternatives.
The calculations using DEAHP for the same set of data are shown in Table 6. The table shows DEAHP
calculations when the local weights of alternatives under each criterion is aggregated with no restrictions
on the weights of criteria, as well as when equal criteria weights are forced. In both the cases the rankings
are not changed. Because of the absence of normalization, even the numerical values of weights are
preserved in DEAHP.
4.2. Some application issues
Normally, when DEA is applied for computing efciency scores, two issues are important.
1. The DMUs should be homogenous units [44]. They should perform the same tasks, and should have
similar objectives. The inputs and outputs characterizing the performance of DMUs should be iden-
tical except for differences in intensity or magnitude. AHP is also based on a similar homogeneity
axiom [45,46]. Hence, the homogeneity requirement is satised for the approach proposed in this
paper.
2. If the number of inputs and outputs are much larger than the number of DMUs, the discriminating
power of DEA will be affected. Usually, as the number of inputs and outputs increase, there will be
more number of DMUs that will get an efciency rating of 1, as they become too specialized to be
evaluated with respect to other units. Certain thumb rules are specied in DEA literature to avoid this
problem. For example, the number of DMUs is expected to be larger than the product of number of
inputs and outputs or the sample size should be at least 2 or 3 times larger than the sum of the number
of inputs and outputs [44]. However, there are many examples in the literature where DEA has been
used with small sample sizes.
This issue is likely to affect the performance of DEAHP. This may not be very serious when deriving
local weights using a judgment matrix of size n, as the total number of inputs (one dummy input) and
outputs (n) is not very large compared to the number of DMUs (n). But, this issue is likely to be serious
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1305
when computing the nal weights because the total of inputs (1) and outputs (number of criteria) is
normally smaller than the number of alternatives. This will affect the discriminating power of DEA.
At the extreme, when all the alternatives receive largest local weights in terms of some criteria, DEA
will assign unit nal weights to all the alternatives. However, note that this is a limitation of DEA in
the traditional production efciency applications, but need not be a limitation when applied for deriving
weights. Any method, including the traditional AHP, will suffer from the same limitation (approximately
equal nal weights for all alternatives) for a similar situation.
4.3. Computational complexity
For a single judgment matrix having n alternatives, DEA requires that n different linear programming
problems are solved to arrive at the weights. Hence, when compared to EVM, DEA requires more
computational effort. However, given the computational capacity of present-day computers, availability
of specialized DEA software and the tremendous progress in the computing speeds, this problem will be
of less concern.
5. Summary and conclusions
Data envelopment analysis (DEA) has been proposed in this paper for deriving weights from the
judgment matrices of the analytic hierarchy process (AHP). If the decision maker is not able, categorically,
to decide whether one alternative is better than another, he will not be in a position to think that one is
more important than the other, and this is the logic employed by DEA for calculating the weights. It has
been proved that DEA calculates true weights for consistent judgment matrices. DEA is further used to
aggregate local weights of alternatives in terms of different criteria in AHP to nal weights. Because of
the synthesis, the proposed approach is named data envelopment analytic hierarchy process (DEAHP).
It has been proved and illustrated using a well known example that DEAHP does not suffer from rank
reversal when an irrelevant alternative(s) is added or removed. Finally, some application issues of DEAHP,
namely homogeneity, discriminating power and computational efforts, are discussed. Thus, DEAHP has
been proposed in this paper as an alternative to the traditional methods of weight derivation in AHP, and
it has been proved that DEAHP can be superior as it can avoid rank-reversal problem in AHP.
As this paper proposes to combine the concepts in two popular decision making techniques, several
more issues have to be researched further. For example, it has been proved that DEAHP calculates correct
weights for a consistent judgment matrix, but its behavior with inconsistent matrices should be studied
further. This paper has outlined the underlying logic behind the approach and has provided a justication
for the use of DEAHP. However, the performance of DEAHP when used with matrices with different
levels of inconsistencies could be empirically studied, similar to the empirical research on comparing the
traditional eigenvector method of AHP and the logarithmic least square technique [23]. As DEA is based
on linear programming, DEAHP may be compared and contrasted with other linear programming based
techniques suggested for deriving weights from judgment matrices, such as goal programming [25,26].
Finally, DEAHP seems to be related to some research studies reported in the utility theory literature. Using
DEA (which aggregates inputs and outputs using a weighted additive function) to derive local weights
from AHP judgment matrices is closely related to the use of additive value functions in the presence of
1306 R. Ramanathan / Computers & Operations Research 33 (2006) 12891307
partial information about the weights [47,48]. The relationships could be explored further. They form
topics for further research in this direction.
References
[1] Zionts S. Some thoughts on research in multiple criteria decision making. Computers and Operations Research
1992;19(7):56770.
[2] Golany B. An interactive MOLP procedure for the extension of DEA to effectiveness analysis. Journal of the Operational
Research Society 1988;39:72534.
[3] Stewart TJ. Relationships between data envelopment analysis and multicriteria decision analysis. Journal of the Operational
Research Society 1996;47(5):65465.
[4] Belton V, Vickers SP. Demystifying DEAa visual interactive approach based on multi criteria analysis. Journal of the
Operational Research Society 1993;44:88396.
[5] Doyle J, Green R. Data envelopment analysis and multiple criteria decision-making. Omega 1993;21(6):7135.
[6] Malczewski J, Jackson M. Multicriteria spatial allocation of educational resources: an overview. Socio-Economic Planning
Sciences 2000;34:21935.
[7] Urli B, Nadeau R. Evolution of multi-criteria analysis: a scientometric analysis. Journal of Multi-Criteria DecisionAnalysis
1999;8:3143.
[8] Ramanathan R. An introduction to data envelopment analysis: a tool for performance measurement. New Delhi: Sage;
2003.
[9] Shi Y, Zeleny M, editors. Newfrontiers of decision-making for the information technology era. Singapore: World Scientic
Publishing, 2000.
[10] Agrell PJ, Tind J. Extensions of DEA-MCDM liaison. Working Paper 3/1998. Department of Operations Research,
University of Copenhagen, Denmark, 1998.
[11] Charnes A, Cooper WW, Wei QL, Huang ZM. Cone ratio data envelopment analysis and multi-objective programming.
International Journal of Systems Science 1989;20:1099118.
[12] Zhu J. Data envelopment analysis with preference structure. Journal of the Operational Research Society 1996;47:
13650.
[13] Saaty TL. The analytic hierarchy process: planning, priority setting and resource allocation. New York: McGraw-Hill;
1980.
[14] Farrel MJ. The measurement of productive efciency. Journal of Royal Statistical Society (A) 1957;120:25381.
[15] Charnes A, Cooper WW, Rhodes E. Measuring the efciency of decision making units. European Journal of Operations
Research 1978;2:42944.
[16] CharnesA, Cooper WW, LewinAY, Seiford LM. Data envelopment analysis: theory, methodology and applications. Boston:
Kluwer; 1994.
[17] Cooper WW, Seiford LM, Tone K. Data envelopment analysis: a comprehensive text with models, applications, references
and DEA-solver software. Boston: Kluwer Academic Publishers; 2000.
[18] Norman M, Stoker B. Data envelopment analysis: the assessment of performance. Chichester: Wiley; 1991.
[19] Golany B, Thore S. The economic and social performance of nations: efciency and returns to scale. Socio-Economic
Planning Sciences 1997;31(3):191204.
[20] Seiford LM, Thrall RM. Recent developments in DEA: the mathematical programming approach to frontier analysis.
Journal of Econometrics 1990;46:738.
[21] Shim JP. Bibliographical research on the analytic hierarchy process (AHP). Socio Economic Planning Sciences
1989;23(3):1617.
[22] Zahedi F. The analytic hierarchy processa survey of the method and its applications. Interfaces 1986;16(4):96108.
[23] Crawford G, Williams C. A note on the analysis of subjective judgment matrices. Journal of Mathematical Psychology
1985;29:387405.
[24] Lootsma FA. Multi-criteria decision analysis via ratio and difference judgement. Dordrecht: Kluwer; 1999.
[25] Bryson N, Joseph A. Generating consensus priority point vectors: a logarithmic goal programming approach. Computers
& Operations Research 1999;26(6):63743.
R. Ramanathan / Computers & Operations Research 33 (2006) 12891307 1307
[26] Choo EU, Wedley WC. A common framework for deriving preference values from pairwise comparison matrices.
Computers & Operations Research 2004;31(6):893908.
[27] Belton V, Gear AE. On a shortcoming of Saatys method of analytical hierarchies. Omega 1983;11:22730.
[28] Shang J, Sueyoshi T. A unied framework for the selection of a exible manufacturing system. European Journal of
Operational Research 1995;85(2):297315.
[29] Yang T, Chunwei K. A hierarchical AHP/DEA methodology for the facilities layout design problem. European Journal of
Operational Research 2003;147(1):12836.
[30] Sinuany-Stern Z, Abraham M, Yossi H. An AHP/DEA methodology for ranking decision making units. International
Transactions in Operational Research 2000;7(2):10924.
[31] Seifert LM, Zhu J. Identifying excesses and decits in Chinese industrial productivity (19531990): a weighted data
envelopment analysis approach. Omega 1998;26(2):279396.
[32] Takamura Y, Tone K. A comparative site evaluation study for relocating Japanese government agencies out of Tokyo.
Socio-Economic Planning Sciences 2003;37(2):85102.
[33] Thompson RG, Singleton Jr FD, Thrall RM, Smith BA. Comparative site evaluations for locating a high-energy physics
lab in Texas. Interfaces 1986;16:3549.
[34] Bowen WM. Subjective judgements and data envelopment analysis in site selection. Computers, Environment and Urban
Systems 1990;14(2):13344.
[35] Sarkis J. A comparative analysis of DEA as a discrete alternative multiple criteria decision tool. European Journal of
Operational Research 2000;123:54357.
[36] Tone K. A comparative study on AHP and DEA. International Journal on Policy and Information 1989;13:5763.
[37] Sueyoshi T. Extended DEA-discriminant analysis. European Journal of Operational Research 2001;131:32451.
[38] Ramanathan R, Ganesh LS. Energy resource allocation incorporating quantitative and qualitative criteria: an integrated
model using goal programming and AHP. Socio Economic Planning Sciences 1995;29(3):197218.
[39] Escobar MT, Moreno-Jimnez JM. A linkage between the analytic hierarchy process and the compromise programming
models. Omega 2002;30(5):35965.
[40] Lovell CAK, Pastor JT. Radial DEA models without inputs or without outputs. European Journal of Operational Research
1999;118:4651.
[41] Ramanathan R, Ganesh LS. Group preference aggregation methods employed in the AHP: an evaluation and an intrinsic
process for deriving members weightages. European Journal of Operational Research 1994;79:24965.
[42] Hadley G. Linear programming. Reading MA: Addison-Wesley; 1962.
[43] Dyer JS. Remarks on the analytic hierarchy process. Management Science 1990;36(3):24958.
[44] Dyson RG, Allen R, CamanhoAS, Podinovski VV, Sarrico CS, Shale EA. Pitfalls and protocols in DEA. European Journal
of Operational Research 2001;132:24559.
[45] Forman EH, Gass SI. The analytic hierarchy processan exposition. Operations Research 2001;49(4):46986.
[46] Saaty TL. Axiomatic foundation of the analytic hierarchy process. Management Science 1986;32:84155.
[47] Barron FH, Barrett BE. Decision quality using ranked attribute weights. Management Science 1996;42(11):151523.
[48] Kirkwood CW, Sarin RK. Ranking with partial information: a method and an application. Operations Research
1985;33(1):3848.