This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception
of pagination.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
Adaptive Learning-Based k-Nearest Neighbor
Classifiers With Resilience to Class Imbalance
Sankha Subhra Mullick, Shounak Datta, and Swagatam Das , Senior Member, IEEE
Abstract— The classification accuracy of a k-nearest neighbor
(kNN) classifier is largely dependent on the choice of the number
of nearest neighbors denoted by k. However, given a data set,
it is a tedious task to optimize the performance of kNN by tuning
k. Moreover, the performance of kNN degrades in the presence
of class imbalance, a situation characterized by disparate
representation from different classes. We aim to address both
the issues in this paper and propose a variant of kNN called
the Adaptive kNN (Ada-kNN). The Ada-kNN classifier uses the
density and distribution of the neighborhood of a test point and
learns a suitable point-specific k for it with the help of artificial
neural networks. We further improve our proposal by replacing
the neural network with a heuristic learning method guided by
an indicator of the local density of a test point and using
information about its neighboring training points. The proposed
heuristic learning algorithm preserves the simplicity of kNN
without incurring serious computational burden. We call this
method Ada-kNN2. Ada-kNN and Ada-kNN2 perform very
competitive when compared with kNN, five of kNN’s state-of-the-
art variants, and other popular classifiers. Furthermore, we
propose a class- based global weighting scheme (Global
Imbalance Handling Scheme or GIHS) to compensate for the
effect of class imbalance. We perform extensive experiments on a
wide variety of data sets to establish the improvement shown by
Ada-kNN and Ada- kNN2 using the proposed GIHS, when
compared with kNN, and its 12 variants specifically tailored for
imbalanced classification.
Index Terms— Heuristic learning, imbalanced classification, k-
nearest neighbor (kNN), parameter adaptation, supervised
learning.
I. I NTRODUCTION
A. Overview
C LASSIFICATION can be posed as the task of pre-
dicting a many-to-one mapping g(.) from a set X of
D-dimensional data points (thus X ⊂ R D , assuming that the
categorical features are replaced by suitable real values) to a
set of class labels C = {1, 2, . . . } , C . A classifier is
designed for the purpose of estimating the properties of the
mapping g : X → C. First, in the training phase, the
classifier is fed
with a training set P ( P ⊆ X and | P | = n) to learn about the
Manuscript received December 28, 2016; revised July 11, 2017 and
November 29, 2017; accepted February 27, 2018. (Corresponding author:
Swagatam Das.)
The authors are with the Electronics and Communication Sciences
Unit, Indian Statistical Institute, Kolkata 700108, India (e-mail:
sankha_r@isical.ac.in; shounak.jaduniv@gmail.com; swagatam.das@
isical.ac.in).
This paper has supplementary downloadable material for online publication
only, as provided by the authors. This includes the detailed proof of some
theorems stated in the main paper, details of the parameter tuning, experi-
mental results on individual datasets, and explanation of biases of some of
the indices used. This material is 260 KB in size.
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TNNLS.2018.2812279
2162-237X © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
characteristics of g(.). In this stage, for all data points ∈xi P,
where i 1,= 2, . . . n, the value of g(x i ) is available to the
classifier. Once trained, the classifier is expected to correctly
predict the value of g(yi ) for a new data point y∈ i Q (Q X,
| |m,
Q = and i 1, 2, . . . , m). This is called the testing
phase. The data point yi is known as a test or query point,
while the set Q of all such points is called a test set.
The k-Nearest Neighbor (kNN) classifier has always been
preferred for its methodical simplicity, nonparametric working
principle [1], and ease of implementation. The kNN classifier
involves tuning of a single parameter k (the number of nearest
neighbors to be considered). However, it is not easy to find
the value of k for which the algorithm performs optimally
on a wide range of data sets (or for all the points in the
same data set). Theoretical studies suggest that the number
of points in the training set (say n) and the value of k
(1 k≤n) both control the performance of the kNN
algorithm [2]. Furthermore, if k=1, then the probability of
misclassification will be bounded above by twice the risk of
the Bayes decision rule as n →∞ [2]. However, depending on
the data set, choices other than k=1 may be more suitable [3].
Therefore, the theory discussed in [2] and [4] does not help
with the choice of k in practical cases. Usually, a global k is
chosen, i.e., a single value of k for classifying all test points.
Conventional choices of such a global k value are 1, 3, 5, 7,
and 9 [5], [6], but may also be as large as k = √n [1], [7].
To optimize the performance, kNN is commonly run with a
number of different k values. Subsequently, several
techniques, such as cross validation and probabilistic
estimation [8], [9], may be used to choose the best k value
among the tested k val- ues. While probabilistic modeling-
based algorithms are hard to implement and usually depend on
prior assumptions about the data set, the technique of cross
validation is computationally rather expensive. Moreover, as
the distribution of the classes is not known a priori, any
choice of global k stands a risk of ignoring the local
distribution of the neighborhood of a test point, whereas the
consideration of the unique features of the locality of a test
point should decrease the chance of misclassification of that
point. In this paper, these facts encouraged us to choose a
data-point-specific k value using an indicator of the local
density and class distributions of its neighborhood.
Besides the difficulty with the selection of k, kNN classifi-
cation rule also faces a challenge over the data sets with class
imbalance, i.e., when all the classes do not have comparable
number of representatives [10]. Hence, we also introduce a
class-specific global weighting scheme to tackle the issue of
class imbalance.
2 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
B. Motivation
The conventional and widely accepted practice of using a
global k value completely ignores the local data density and
class distribution (also referred to as local information in the
subsequent text) of a test point’s neighborhood. Majority of
the practical data sets have a global density and distribution
of classes differing widely from those in the neighborhood
of a test point. In such data sets, the information of the
locality around a test point often proves to be vital for
achieving the right classification. In the case of the kNN
classifier, the locality information around a test point should
be considered while choosing a suitable k value for it. This
would result in data-point-specific choice of k. Since data
points belonging to the same locality are likely to be correctly
classified by similar values of k, the optimal k value can be
modeled as a suitable nonlinear function of the training points
(see Section III-D) called k-terrain. Hence, the proposed Ada-
kNN classifier, which learns the k-terrain using a supervised
learner, is likely to perform better than the traditional kNN
and its variants that use a global k value.
It is worth noting that Ada-kNN is reliant on the efficacy
of an additional learning algorithm which further adds to
the computational complexity. As the knowledge about the
entire nonlinear function of k is not an absolute necessity,
an approximation of the function in the vicinity of a test point
is likely to suffice. Thus, the number of neighbors, which
should influence the k value of a query point, can also be
estimated based on an indication of the local data density.
One may then use the correctly classifying k information of
these neighbors (reflective of the local information) to
estimate the optimal k value for the query point in question.
This idea motivated us to replace the supervised learner in
Ada-kNN with such a heuristic learning scheme, giving rise to
the Ada- kNN2 classifier which is also presented in this paper.
Achieving good classification for imbalanced data sets is a
challenge for the kNN classifier, as it relies on the expectation
that in the neighborhood of a test point, the number of training
points from its ideal class will be high. In reality, this may
not hold true for a test point belonging to the minority
class of an imbalanced data set, as most of its neighbors
may still be from one of the majority classes in spite of
an increase in the number of minority neighbors (due to the
global scarcity of the minority points). In such a scenario,
the basic kNN classifier, which does not consider the class
imbalance, will wrongly assign the test point to a majority
class. The class-specific weighted kNN classifier can be
proved to be efficient in such situations. However, while
choosing the weights, the local information should also be
considered. In Ada-kNN and Ada-kNN2, since the local
information is accounted for by the point-specific value of k, a
global class- specific weighting scheme may suffice.
Therefore, we present a scheme to combine class-specific
global weights with the proposed classifiers to handle class
imbalance.
C. Contribution
The main contributions of this paper are in order.
To facilitate the choice of data-point-specific k, the proposed
Ada-kNN method1 first finds for each of the training points,
a value of k for which kNN can accurately classify that point.
Assuming this k value to approximate the local information
of the neighborhood of a data point, our algorithm tries to
estimate a suitable k value for each query point. This gives
rise to a nonlinear regression problem, which can be solved by
a feedforward Multi-Layer Perceptron (MLP) using the Scaled
Conjugate Gradient (SCG) learning algorithm [11] to estimate
the k-terrain.
We further propose a new learning approach that directly
utilizes the correctly classifying k values of the neighbors
to estimate a suitable k value for each query point. We use
the nearest neighbor distance as an indicator of the local
density for a query point and to proportionally vary the
size of its neighborhood. The resultant algorithm is called
Ada-kNN21. Both the proposed methods are compared with
kNN, Linear Discriminant Analysis (LDA) [12], MLP using
Back-Propagation [13] (MLP-BP), MLP using SCG [11]
(MLP-SCG), Decision Tree (DTs) [14], AkNN [15], Extended
Nearest Neighbor (ENN) [16], dyn-kNN [17], Fuzzy Analogy-
Based Classification (FABC) [18], and k ∗ Nearest Neighbor
(k∗NN) [19] over 17 data sets (over six data sets for k∗NN
and eight data sets for FABC) of varying properties. Both the
proposed methods are observed to perform competitively,
with Ada-kNN performing equivalent to the best on most
small- and medium-scale data sets, while Ada-kNN2 generally
achieves the best result among all the contenders.
The proposed classifiers can tackle the local variations of a
data set with the varying value of k, thus dispensing the need
for local weighting schemes on imbalanced data sets. There-
fore, we define a simple Global Imbalance Handling Scheme
(GIHS) and combine it with kNN, Ada-kNN, and Ada-
kNN2 classifiers1 to observe the performance improvement
for imbalanced data sets. We perform extensive experiments
using 22 two-class data sets and 11 multi-class data sets
having varying degrees of imbalance and compare the
performance of the proposed algorithms against that of the
state-of-the-art variants of kNN for data imbalance [namely
kNN, k Rare class Nearest Neighbor (kRNN) [20], k
Exampler-based Nearest Neighbor (kENN) [21], k Positive-
biased Nearest Neighbor (kPNN) [22], Class Conditional
Nearest Neighbor Distri- bution (CCNND) [23], dyn-kNN
[17], Neighbor-Weighted kNN (NWkNN) [24], WkNN [25]],
Ada-kNN and Ada-kNN2
coupled with under-sampling [26] and over-sampling [27]
techniques. The proposed methods achieve competitive per-
formance compared with the competing algorithms, with Ada-
kNN2 coupled with GIHS being the best performer in most
cases.
D. Organization
We arrange the rest of this paper, subsequent to the
introduction in Section I in the following manner. In Section
II, we present a brief but comprehensive review of the notable
works done in the field. In Section III, we describe Ada-kNN
and Ada-kNN2, followed by an explanation of GIHS for
handling imbalance. In Section IV, we first discuss the choice
1
Codes are available from: https://github.com/SankhaSubhra/Ada-kNN.
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE
of the data sets and introduce the indices for quantifying
the performance of the classification methods. The rest of
Section IV contains experimental results and a comparative
discussion of the proposed methods along with their con-
tenders. We present our conclusion and the future directions
of research in Section V.
II. BACKGROUND
In this section, we briefly describe the notable past works
attempting to aid kNN classifier by incorporating information
about the neighborhood of a test point and making it suitable
for handling class imbalance.
A. Sensitivity to Locality Information of Nearest
Neighbor Classifiers
In the kNN classifier, the local density around a test point
should ideally affect the choice of k (see [19, Fig. 1]),
resulting in a better strategy for rightly classifying the test
point. This idea is employed by Wettschereck and Dietterich
[28] in four of their proposed methods to determine test-
point-specific values of k. The first two methods utilize the
information of the correctly classifying k values of the
neighbors of a test point. The third technique works by finding
the class-specific k values, which successfully classify a
maximum number of training members from their
corresponding classes. In the fourth method, the test points are
clustered, and each of the clusters uses a single k value for
classifying all test points belonging to that cluster.
Ougiaroglou et al. [29] proposed a dynamic kNN classi-
fier by modifying the incremental nearest neighbor searching
algorithm [30]. Their method introduced three heuristics to
reduce the computational complexity of the incremental kNN
search. Wang et al. [15] incorporated an adaptive distance
measure with kNN, forming an algorithm called AkNN. The
method finds for each training point x∈i P, the distance rxi
of the nearest neighbor belonging to any other class. While
finding the neighbors for a query point y∈i Q, the distance
between yi and xi is downscaled by corresponding rxi .
Bhattacharya et al. [31] devised an algorithm for assigning
a test-point-specific value of k. The method is based on
neighborhood density [see [31, eq. (1)]] and certainty factor
[see [31, eqs. (5) and (6)]]. Tang and He [16] proposed a
variant of the classical kNN decision rule known as the ENN
method. The key feature of their algorithm is the consideration
of not only the k neighboring training points of the test point
in question, but also those training points which include the
query point in their k neighborhoods. The authors iteratively
assign a test point to each of the possible classes and calculate
a class-specific statistic [see [16, eqs. (4) and (5)]]. That test
point is then assigned to the class having the highest value of
the statistic.
Another work by Garcia-Pedrajas et al. [17] (hereafter
called dyn-kNN) approached the task of a test-point-specific
k value by considering both the global and local implications
of each choice of k. The algorithm evaluates the global
performance for all k k∈ min,
[ kmax . A training-point-specific
] local
performance for each of the k values is also maintained.
3
The algorithm then assigns to a training point xi , the value
of k which performs the best (in terms of global and local
performance) over those training points of which xi is a
neighbor. For a test point, however, the k value corresponding
to its nearest neighbor is used. Thus, the method uses disparate
philosophies for assigning k values to the training and test
points. Furthermore, the algorithm suffers from a high time
complexity.
Anava and Levy [19] proposed k∗NN to estimate the class
label of a test point yi Q ∈as weighted average of the class
labels of training points. They showed that for yi , only a
test point dependent finite number of nearest neighbors will
contribute to the calculation of its class label. The problem of
finding the point-specific number of neighbors and their cor-
responding weights can be reduced to a convex optimization
problem, which can be solved exactly by the greedy method
proposed by the authors.
Ezghari et al. [18] introduced a new adaptive variant of
fuzzy kNN called FABC, which first transforms the data
points from their original numerical space to a fuzzy
membership space. A query point is classified by aggregating
its similarities with each of the training points and their
respective fuzzy membership values for the different classes.
B. kNN Classifiers for Handling Class Imbalance
The classification accuracy in the presence of class imbal-
ance can be improved in one of the two directions. In the first
direction, the original training set is kept unaltered, while the
classifier is modified to incorporate immunity against the
effect of imbalance. Along the second line of thinking, the
focus is shifted from the classifier to the training set itself,
which is properly pre-processed by either under-sampling (the
majority class) or over-sampling (the minority class) [10],
[32]–[34].
1) kNN Variant Designed Specifically for Class Imbalance
Problem: As a primary approach, Zhang and Mani [35] used
five different methods to under-sample the majority class such
that the effect of class imbalance can be alleviated.
Tan [24] proposed a weight selection scheme of the
weighted kNN method (called NWkNN) to handle imbalance.
In this technique, each class is assigned a weight depending on
its global probability and that of the smallest class. However,
the algorithm introduces an additional data-dependent parame-
ter, which needs to be optimized.
Wang et al. [36] modified the extended kNN proposed
by Wang and Bell [37] to make it suitable for handling
imbalance. The authors defined class-specific weights in the
form of differences between the local (calculated over the k
neighborhood) and global (calculated over P) probabilities,
respectively, for each of the classes.
Liu and Chawla [38] developed a weight selection
technique for handling class imbalance. They reduced the
problem of kNN to finding the maximum among the
probabilities of a test point yi belonging to each of the classes,
given the k neighborhood of yi . To achieve this, the authors
used the posterior probabilities of the classes as weights
(estimated by mixture models or Bayesian network). Li and
Zhang [21] presented an idea (called kENN) of extending
some of the
4 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
minority class members to Gaussian balls, to compensate
for the disadvantage of scarcity. Such a strategy effectively
reduces the distance between a test point and the minority
class members so that the minority class gets an edge.
Dubei and Pudi [25] utilized a weighting scheme to aid
the issue of rare class classification (hereafter called WkNN).
Their method incorporates a query-specific local weighting
alongside a set of global class-specific weights. However,
the method needs to employ additional kNN runs over the
neighborhood of each query point. Kriminger et al. [23] used
the nearest neighbor information to estimate the probability
of a class in their algorithm called CCNND. The algorithm
for each class finds the probability of a test point belonging
to that class given the distance of its nearest neighbor from
that class (using empirical cumulative distribution function of
the intraclass nearest neighbor distances). Zhang and Li [22]
proposed a variant of kNN called kPNN. The idea uses a
dynamic neighborhood (containing at least k minority class
members) and classifies based on the estimation of the locally
adjusted posterior probability of the minority class.
Recently, Zhang et al. [20] proposed the kRNN, which uses
a further constrained definition of the dynamic neighborhood
proposed in [22] for each test point. Furthermore, the clas-
sifier estimates both the global and local adjusted posterior
probabilities for the minority class.
However, the problems with all these methods are the
involvement of exhaustive search, the introduction of new
parameters, and significant computational overhead, which
hinder the scalability and easy implementation of kNN.
2) Removing Imbalance by Pre-Process the Data
Set: A data set can be pre-processed by under-sampling the
points of the majority classes or over-sampling the points
of the minority classes such that the number of representa-
tives from all the classes becomes comparable. A common
method of under-sampling is called Random Under-Sampling
(RUS) [26], which, as the name suggests, randomly discards
the representatives of the majority class from the training set,
until the effect of imbalance is sufficiently mitigated, whereas
a notable approach for over-sampling known as Synthetic
Minority Over-Sampling Technique (SMOTE) [27] randomly
creates new minority class points by interpolating between
the existing minority points and their neighbors. However,
the original training set characteristics are not entirely pre-
served through any of these techniques. In addition, both
methods introduce new parameters, which need to be properly
tuned.
III. P ROPOSED M ETHODS
Starting with some basic notations which we follow
throughout this paper, we first describe the canonical and
weighted kNN classification algorithms. Next, we propose the
Ada-kNN classifier and extend it to Ada-kNN2 using a heuris-
tic learning algorithm for point-specific k selection.
Thereafter, we present GIHS and use it to augment Ada-kNN
and Ada- kNN2 for imbalance classification problems.
Finally, we finish the section with the computational
complexity analysis of the proposed algorithms.
A. kNN and the Class-Specific Weighted kNN Classifiers
Let the class label be denoted as c(xi )∈ for a training
point xi P,∈ for i 1, 2, . . . , n. The set C of all training
points belonging to class c is denoted by |C |c = nc .
C c, and
The kNNs of a data point x ∈X , from a set S, form the
set S (x).
k Also, let us take I (.) to be an indicator function
defined as in (1), where cond is some condition to be satisfied
= 1, if cond is true
I (cond) (1)
0, if cond is false.
We can now describe the basic kNN classification rule, with
the help of these notations. For a test point yi , if the kNNs of
yi in the training set P form the set P k(yi ), then the predicted
class label of yi , denoted as cˆ(yi ) , will be
cˆ(yi ) = arg max I (c(xi ) = c). (2)
c∈1,2,...,C
xi ∈ kP (yi )
As indicated by (2), the kNN classifier actually labels a new
point yi , as belonging to the class having the maximum
number of members in the set P (yk i ). In the class-specific
weighted NN classifier [24], every class is associated with a
k
weight, and the number of points belonging to a certain class
is multiplied by the weight associated with that class. The
rest is similar to the conventional kNN classification, resulting
in the assignment of the new point to the class producing
maximum weighted number of members in P k(yi ). If the
weight for the c th class is denoted by wc, then the weighted
kNN classification rule can be expressed as
cˆ(yi ) = arg max wc I (c (xi ) = c). (3)
c∈1,2,...,C
xi ∈ P
k (yi )
The weights in (3) can have any real value greater than 0.
The weighted kNN classifier is actually a more general form
of the kNN classifier, which can be reduced to the basic
kNN classifier by making all the weights equal. In weighted
kNN classification, the user enjoys the flexibility to introduce
problem-specific class priorities, which can achieve better
results.
B. Description of Ada-kNN
The Ada-kNN classifier utilizes the information of the
neighborhood of a test point and mitigates the problem of
global choice of k, by introducing the concept of point-
specific k value. One way to do this is by finding the sets of k
values correctly classifying each of the training points∈xi P and
employing a supervised learner to model the k values as a
function of their corresponding training vectors. This function
can then be applied to a test point to identify a k value suitable
for it.
The Ada-kNN algorithm starts by finding the set of suc-
cessful k values, K x i for each of the training points x∈i P.
One may perform an exhaustive search over the set of all
possible k values, i.e., K 1, 2, . . . , kmax , to find the set Kx i
= { }
xi P.
∀ Here,
∈ kmax n, which √
= satisfies the following desirable
properties.
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE 5

1) kmax should neither be too large nor too small to preserve

the local structure of the neighborhood.
2) kmax should be data-dependent. a set {0, 1} defined as follows:
⎧
∈ 1, if x ∈ X is correctly classified by kNN
⎪ (4)
f k (x) = ⎨ ⎪ using k number of neighbors
However, exhaustive searching for all k K is a compu- ⎩
tationally expensive process, especially when the training set 0, otherwise.
contains a large number of instances. Furthermore, the set K x i We further define
for each xi ∈P generally consists of subsets of consecutive k
values for which the point xi is correctly classified. These zk = fk (xi ) (5)
observations lead us to perform the search over a random xi ∈ Pν(yi )
permutation of K , denoted by Krand, instead of K . Such a
randomized search will reduce the chance of testing for which is the number of training set neighbors of yi that
consecutive values of k. Moreover, such strategy will pick are correctly classified by kNN using k neighbors. Hence,
any k in the range of [1, kmax] with equal probability. This by construction, zk is proportionate to the probability with
which a value of k correctly classifies points in P (yi ).
ensures a better cover of K in less number of average trials, ν
increasing the chance of finding a correctly classifying k value Therefore, the problem of finding k yi can be expressed as
for a training point. follows:
After finding Kx i ∀xi P, we may feed that information to ky i = arg max z k . (6)
an MLP and proceed to estimate the nonlinear k-terrain for k∈{1,2,...,kmax
}
a given data set. The learning phase of MLP requires single Given
be the K x sets for all the x i ∈ (yi ), z k will
kxi for each training point xi ∈ P, and thus, finding the entire i
P
ν
set Kx i is not an absolute necessity. However, such a choice of maximized for the choice of k present in the maximum
kxi may be susceptible to the noise present in the data set. number of such sets. Hence, the optimum k value will be
the mode
of the multiset y i formed by concatenating all K , x∀x i ∈
Hence, we recommend trying α number of trials using the first P
i

α elements of Krand and record all the correctly classifying
k values in a set K x i (K x i K xi ). Only when all the α
trials are a failure, we look⊆for the first correctly classifying
value
of k among the subsequent members of Krand. At the time of
training the MLP, one can randomly choose a value of
k xi ∈ K x i .
In the test phase of Ada-kNN, the trained MLP can be fed
with a query point yi ∈ Q to obtain the value of kyi .
C. New Learning Method and Ada-kNN2
We use an MLP in the primary implementation of Ada-
kNN to model the nonlinear function of correctly clas- sifying
k values of the training points. However, a learning algorithm
like MLP involves a number of parameters (the number of
hidden layers, the number of hidden nodes in each layer,
learning rate, and a choice of the training algorithm).
Moreover, MLP incurs a high algorithmic complexity. Both
of these issues affect the simplicity of Ada-kNN. Furthermore,
MLP uses single k xi ∈ K x i , while the set may contain at most
α elements, reducing the utilization of available information.
This motivated us to design a simple parameter-free heuristic
learning algorithm.
The basic idea of the proposed heuristic scheme is based
on the assumption that a test point will have similar properties
as its training set neighbors. Let us consider a k value, which
correctly classifies some of the training set neighbors of a
test instance yi ∈Q. Such a k value can then be expected to
correctly predict the class label of yi with a certain probability.
The heuristic attempts to find the value of k for which such a
probability is maximized over the neighborhood of yi .
To define the problem more formally, let us assume that the
neighborhood of yi contains ν number of points. Furthermore,
(yi ). A random mode can be selected if yi is multimodal.
ν
Now, we need to find a suitable value for ν. It is reasonable
to assume that a test point lying in a dense region has
similar characteristics with a larger number of training set
neighbors, whereas a test point residing in a sparse location
has similarities with only a small number of neighbors.
Therefore, the number of neighbors should be varied
according to the local density in the vicinity of the test
point. Therefore, ν
(hereafter called νyi for a test point yi Q)∈ should be a test
point-dependent parameter instead of a global constant. A
direct way to find an optimal value of νyi is to use the density
of the locality surrounding yi . However, instead of calculating
or approximating the actual local density, one may use an
easily computable indicator of it, such as the nearest neighbor
distance. This claim is supported by the following
theorem.
Theorem 1: Let a D-dimensional hypersphere contain some
uniformly distributed points (including one lying at the
center). Let the expected distances of the nearest neighbor
from the
center be dN1 and dN2 , respectively, when the hypersphere
contains N1 and N2 number of points. If N1 > N2, then it can
be shown that dN1 < dN2 .
Proof: The proof follows from the derivation of Bhat-
tacharyya and Chakrabarti (see [39, Sec. III]), which we detail
in the Supplementary Material.
From Theorem 1, it can be seen that the local density
bears an approximately inverse relation to the expected
nearest neighbor distance. Therefore, if the nearest neighbor
distance is large, we can assume a sparse locality, whereas a
low distance will indicate a dense region. Let us denote the
S as d S
distance from a point x to its nearest neighbor in a set 1NN
∈
1NN
(x). Therefore, for a point yi Q, the nearest training set neighbor is at a distance d P (y i). Let us denote
consider a family of functions F = { f1, f2, ... fkmax }. Each dmin = min d P\{xi }(xi ) and dmax = max d P\{xi }(xi ).
of the fk ∈ F is a many-to-one mapping from a data set X to xi ∈
1NN
xi ∈
1NN
P P
6 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

We now describe three possible cases as follows.

P
1) d 1NN (yi ) < dmin: yi has a sufficiently dense locality, and
a suitably high number (denoted as νmax) of training set
neighbors should influence its k value.
P
2) d 1N (yi ) > dmax: yi has a scantly populated neighbor-
N
hood, and νyi is taken to be 1.
3) dmin ≤ d 1N P
(yi ) ≤ dmax : One should consider a νyi ∈
[1, νmax],Nfor finding the value of kyi .
We choose a data-dependent value of νmax 3 n, which √
=
satisfies the following desirable properties in the following.
1) νmax should be sufficiently small, as the local density
is likely to change with increase in the number of
neighbors.
2) νmax should not become too large even for data sets
containing very large number of data instances.
To estimate νyi when dmin ≤ d P (yi ) ≤ dmax, we use Fig. 1. Illustration of νyi estimation for a test point yi ∈ Q. Green curve:

expression (7) 1NN

exponential estimate decaying from νmax to 1. Blue line: linear estimate
decaying from νmax to 1. The respective estimates νexp(yi ) and νlin(yi )
νy = (νlin(yi )νexp(yi ))0.5 , (7) corresponding to d P (yi ) are also shown. νyi is taken to be the geometric
1NN
i mean of these two estimates.

where . denotes the ceiling function, TABLE I

, (8)
(1 − νmax) LIST OF Kxi AND RANDOMLY SELECTED kxi FROM Kxi , FOR EACH OF THE
ν (y ) = (y ) − + TRAINING POINTS xi ∈ P IN THE EXAMPLE
d ν
Pd
lin i 1NN i min max
(dmax −

and dmin)

ν (y ) = e 1NN (yi )−dmin β

, (9)
d
νexp i max P

having

loge νmax
β =− dmax − dmin . (10)

Here, νlin(yi ) is the linear estimate which models νyi to

be a linearly decreasing function of d P1NN(yi ) between νmax
and 1, which is useful for small data sets to maintain elements of K x i for all x i lying in the ν yi neighborhood of yi
a respectable neighborhood size. The exponential estimate to form the multiset yi . The value assigned to kyi will then be
νexp(yi ), on the other hand, is an exponentially decreasing the mode of yi .
P
function of d 1NN (yi ), between the same extremities as the
linear one, and is useful for larger data sets to restrict the
size of the neighborhood. Furthermore, the reason for using
geometric mean of the two estimates, instead of arithmetic
mean, is to allow the exponential estimate to have more
influence in the choice of νyi than the linear one (as geometric
mean is closer to the minimum, i.e., the exponential estimate).
This is done so that the neighborhood influencing the value of
k for a test point can be kept small even for large data sets.
We graphically show the situation in Fig. 1.
The Ada-kNN2 classifier differs from Ada-kNN in that the
MLP is replaced by the above-described heuristic learning
scheme. In Ada-kNN2, given a test point yi , the algorithm
starts by finding the nearest neighbor distance, which is then
compared with dmin and dmax to determine the value of νyi . The
algorithm then finds the set P (y ν i ) and concatenates the
D. Illustrative Example
We now illustrate Ada-kNN and Ada-kNN2 with the help
of a two-class classification data set. Let the training data set
be P containing 14 points from two classes 1 and 2. Thus, =
kmax 4,
νmax 3, and K 1, 2, 3, 4 . We present in Table I = = { }
training points xi along with their class labels c(xi ), sets Kx i
of correctly classifying values of k (ties broken arbitrarily),
and randomly selected values kxi K x i . ∈
The reader can make the following interesting observations
from Table I.
1) A single global k ∈ {1, 2, 3, 4} value may not be able
xi P xi to correctly classify all the data points, as K
φ. This establishes the importance of data-point-specific =∈
choice of k.
2) A data point can be usually classified correctly using
the same or close value of k as its neighbors, i.e., the
correctly classifying k values are influenced by the
locality information.
In Fig. 2, we plot each training point xi and the correspond-
ing kxi following Table I. This plot is discrete, and given a
new
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE 7

Fig. 2. Training points and their corresponding values of k from the example, illustrating the estimated k-terrain and predicted k values for the test points.

point, the optimal k value cannot be directly estimated from F. Complete Algorithm
it. However, this information can be used to train an MLP to The complete algorithms for Ada-kNN and Ada-kNN2,
approximate the nonlinear function of k (referred as k-terrain). with the addition of GIHS, are schematically shown,
As MLP uses random kxi ∈ Kx i for a training point xi , respectively, in Algorithms 1 and 2. If we do not wish to
the learned k-terrain is not unique. One of such possible k- consider data imbalance, we can skip employing GIHS by
terrain values is shown in Fig. 2. To illustrate the efficiency of initializing the weight vector W with all 1s, in which case the
our proposed classifiers, we take a test set Q 2, 5=, {where } two algorithms reduce to regular Ada-kNN and Ada-kNN2.
c(2) 1 and=c(5) 2. If we use Ada-kNN, then we take the Both of the algorithms use a subroutine called TrainRoutine(
ceiling of the k-terrain values corresponding to the test point.
P, W, K, xi ), which is described in Procedure 1, for
Following this process, one may obtain k2 = 1
finding the set K x i .
and k5 = 3. In the case of Ada-kNN2, we find dmin = 0.1,
Hereafter, we will denote the i th member of a set S as Si . The
function Concat(si |i = 1, 2, . . . n) takes values s1, s2, . . .
sn
= d (2) = d (5) 0.2. = ν2 1
P P
dmax 0.9, 1NN
0.1, and 1NN
Thus, and concatenates them to form a multiset. The subroutine
and ν5 =2, which gives = 2 ={ } 1 and 5 =
1, {2, 3, 3, 4}. Mode(S) takes a multiset S as input and produces a mode of
Taking the mode from 2 and 5 gives k2 1 and k5=3, respectively, S as output (resolving ties randomly in case S is multimodal).
similar to that of Ada-kNN. One can verify using P and (2) The function Rand(S) takes a set S as input and outputs a
that k2 and k5 can, indeed, correctly classify the corresponding random element of S, while RandPerm(S) returns a random
test points. permutation of the elements of set S.

E. Handling Imbalance in a Data Set
As discussed in Section I-B, the class-specific weighted
kNN classifier can be proved to be efficient in situations
plagued by class imbalance. The class-specific weights can
be used to magnify the increments in the number of members
from the minority classes in the neighborhood of a test point
while diminishing those of the majority classes to compensate
for their abundance.
One way to weigh the classes is to use a global class-
weighting scheme, which uses the same set of class weights
for all test points. Since there should ideally be equal number
of representatives from each class in P, the ideal probability
of a point belonging to class c∈ Cshould be r = (1/C). But
most data sets are not ideal, and thus, we have nc training
points belonging to the class c (where nc values are not all
comparable). Hence,
C
nc = n, and the prior probability
of class c is p c/ )
c=1 c
n . In order for = (n
each class to have a
fair chance in the neighborhood of a test point irrespective of
class imbalance, we assign the ratio of the ideal and current
probabilities for a class c as the global weight wc, associated
with that class, i.e.,
r a set of points by the Ada-kNN classifier is O(max{ n2.5, nψ }),
wc = c.
p
Procedure 1 TrainRoutine( P , W , K , xi )
Initialize: K r and ← RandPerm(K ), K x φ.
K x i ← all k among the first α elements of K r and that
successfully classifies xi using the training set P \ {x}i .
if K x =i φ then
K x i ← first k among the rest of the elements of K r and that
successfully classifies xi using the training set P \{ x}i .
end if
return K x i
G. Algorithmic Complexity of Ada-kNN and Its Variants
In this section, we analyze the computational complex-
ities of the proposed classifiers. In Theorem 2, we ana-
lyze the asymptotic complexity of the regular kNN classi-
fier, which we then utilize to examine the complexities of
Ada-kNN, Ada-kNN2, and their GIHS-coupled versions in
Theorems 3–5, respectively. Proofs of Theorem 2–5 can be
found in the Supplementary Material.
Theorem 2: The asymptotic time complexity for classifying
a set of points by the kNN classifier is O(n3).
Theorem 3: The asymptotic time complexity for classifying
(11)
where the supervised k learner takes O(nψ ) time.
8 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

Algorithm 1 Description of the Proposed Ada-kNN Algo-

rithm When Coupled With GIHS
Input: Training set P, test set Q, α.
Output: The set of predicted class labels (C) for Q.
Calculate a set of global weights W = {w1, w2, · · · wC } for
all the data po ints x ∈ P ∪ Q, using the set P and (11).
Define kmax ← √n and K ← {1, 2, ··· , kmax}.
Training Phase:
for all xi ∈ P do
K x i← TrainRoutine( P, W, K , x i ).
end for
Train MLP with x i as input and k xi ← Rand(K x i ) as
desired output.
Fig. 3. Category of data sets.
Testing Phase:
for all yi ∈ Q do
Feed yi to MLP, which will output kyi .
IV. E XPERIMENTS AND R ESULTS
Ci ← classify yi by using P, W and kyi .
end for In this section, we evaluate the performance of our proposed
techniques in terms of standard indices using a collection of
data sets having diverse nature.
Algorithm 2 Description of the Proposed Ada-kNN2 Algo-
rithm When Coupled With GIHS
A. Description of the Data Sets
Input: Training set P, test set Q, α.
Output: The set of predicted class labels (C) for Q. We use standard data sets (a detailed description of which
can be found in the Supplementary Material) from the Univer-
Calculate a set of global weights W = {w1, w2,· · ·wC }for sity of California at Irvine, machine learning repository [40],
all the data points x ∈ P Q, using the set P and (11). the IDA benchmark repository [41], the KEEL repository [42],
Define kmax ← √ n ,∪K ← { 1, 2,· · ·, kmax }, λmax ← Akbani et al. [43], the agnostic learning versus prior knowl-
√3
n , and initialize d1N Ni ← 0; ∀ i = 1, 2,· · · , n. edge challenge database [44], and LibSVM repository [45]
Training Phase: according to the two following criteria.
for all xi ∈ P do 1) Scale of Data Set: If a data set contains more than
K x i ← TrainRoutine( P, W, K , x i ). 4000 data points and/or has a data dimension greater
d 1NN ←d (xi ).
i
P\{xi
1NN
} than 45, then we call it large-scale in this paper. All the
end for other data sets are considered as small/medium-scale.
Find dmin and dmax among all d1N Ni s. 2) Degree of Imbalance: This can be quantified in the
Testing Phase: form of imbalance ratio (IR). IR is defined for a two-
for all yi P∈ Q do
Find←d1NN (yi ). and ν yi using (7),
class data set as the ratio of the number of points in
yi Concat(K x |∀x i ∈
P
(y(8),
i )).
and (9).
the majority class to that of the minority class. In the
i
νy case of multi-class data sets, IR is taken to be the
kyi ← Mode( yi ). i
maximum of IR values calculated between all of the
Ci ← classify yi by using P, W and kyi . pairs of classes. Based on its IR value, a data set
end for can be either balanced (IR ≤ 1.15), mildly imbalanced
(1.15 < IR ≤ 3.5), or highly imbalanced (IR > 3.5).
A data set can be placed into any one of the six possible cate-
Theorem 4: The asymptotic time complexity for classifying gories, based on the two above-mentioned criteria, as
a set of points by the Ada-kNN2 classifier is O(n2.5). described in Fig. 3.
Theorem 5: The asymptotic time complexity of Ada-kNN
and Ada-kNN2 does not change if GIHS is coupled with
B. Indices for Evaluation of Classification Performance
them.
From Theorems 2 and 3, we can conclude that the asymp- We use four different indices to evaluate the performance
totic computational complexity of Ada-kNN is better than that of the contending classifiers.
of kNN. Moreover, from Theorems 3 and 4, we can conclude 1) Accuracy: In a C-class classification problem, let the
that Ada-kNN2 can achieve equal or better asymptotic com- test set contain m number of points. Suppose that the clas-
plexity than Ada-kNN depending on the choice of supervised sifier correctly classifies m of the test set members. Then,
learner. Furthermore, Theorem 5 confirms that the addition the accuracy is given by acc=(m /m). Accuracy is used to
of GIHS does not increase the asymptotic complexity of Ada- evaluate the performance for our experiments, which are not
kNN and Ada-kNN2. concerned with class imbalance.
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE 9

2) Gmeans: Accuracy may still be rather high even if TABLE II

the classifier wrongly classifies most of the minority class COMPARISON OF DIFFERENT CLASSIFIERS IN TERMS OF ACCURACY
members while achieving high accuracy over the majority
classes. Gmeans [46] is a measure, which separately considers
the classifier’s performance on each of the classes. Here,
we extend it to the multi-class scenario in the following way.
Let us assume a C-class classification problem, where the
test set contains m points and m c points belong to class c
(c = 1, 2, . . . C). If we assume that the classifier correctly
classifies m c members of class c, then Gmeans is calculated
C (1/C)
c= = (
as GM (m
3) Area
/m )).cc1 Under Receiver Operating Characteristics Curve:
This index is defined as AUROC= ((1+ tpr fpr)/2) [47],
where tpr and f pr , respectively, stands −
for true positive rate TABLE III
and false positive rate [47], considering the minority class as ∗
COMPARISON OF FABC AND k NN WITH ADA-k NN AND ADAk NN2
positively labeled. In the case of a multi-class classification
problem, an extension of area under receiver operating char-
acteristics curve (AUROC) can be used [48].
4) Area Under Recall Precision Curve: Considering the
minority class members to be positively labeled, we calculate
the true positive rate (tpr ) and precision ( prec) of the
classifier [49]. The value of area under recall precision curve
(AURPC) is then obtained=as AURPC ((tpr prec)/2)
[49]. Similar to AUROC, one can extend the definition of
AURPC for the multi-class classification problem.
C. Experimental Procedure GIHS, NWkNN, kENN, kPNN (all with global k values
√ of
1, 3, 5, 7, 9, and n), CCNND (with nearest neighbor), dyn-
For all the experiments except the case involving k∗ NN , kNN (all parameters and fit ness evaluation set according
we perform a 10-fold cross validation (5-fold for fair compar- the literature), WkNN (with √n number of neighbors), kRNN
to
ison with k∗NN) to eliminate the training-test set dependence with
from our reported results. To further eliminate any bias gen-
erated from randomization involved in the algorithms, we run
them 10 times on each of the 10 training-test set pairs gener-
ated by cross validation [50]. The MLP used by Ada-kNN
has D nodes in the input layer, kmax nodes in the output
layer, and a single hidden layer containing 15 hidden nodes.
These settings for MLP are kept constant for all experiments,
as tuning them did not vary the results significantly. We have
used the default MATLAB implementation for MLP in our
experiments.
We perform two sets of experiments. In the first set,
we compare the performance of the regular kNN classifier,
the AkNN classifier, the ENN classifier (all with global k
√
values of 1, 3, 5, 7, 9, and n, where n is the number of
training points), MLP-BP, MLP-SCG (both using the default
MATLAB settings), DT (MATLAB implementation with
default settings), LDA, and dyn-kNN (implemented with set-
tings as advised in the literature) with those of Ada-kNN and
Ada-kNN2 classifiers, in terms of accuracy. We also compare
the performance of our proposed classifiers with those of
k∗NN and FABC reported, respectively, in [18] and [19].
These sets of experiments are performed to establish the
proposed classifiers as significant improvements over popular
classifiers and recent variants of kNN (in the absence of
severe class imbalance). For this reason, data sets belonging
to categories 1–4 are used in these experiments.
In the second set, we compare the performance of the regu-
lar kNN classifier and the regular kNN classifier equipped
(all parameters are set following the corresponding study),
Ada-kNN SMOTE and Ada-kNN2 SMOTE (the
number of neighbors is kept fixed at 5 as advised by the +
original article [27]; for two-class classification problem, the
minority class is over-sampled by 100%, 200%, 300%,
400%, and 500%, and for multiple classes, the choice of this
parameter is described in the Supplement Material), Ada-
kNN RUS and Ada-kNN2 RUS +
(for two-class data sets, the majority class is under-sampled +
by 10%, 30%, 50%, and 70%, and for multi- class problems,
the parameter is chosen according to the tech- nique
described in the Supplementary Material), Ada-kNN GIHS +
and Ada-kNN2 GIHS, in terms of GM, AUROC, +
and AURPC [51]. Since the kRNN, kPNN, and kENN
classifiers cannot be used on data sets containing multiple
classes, they are omitted from multi-class experiments. We
aim to evaluate the extent to which GIHS, SMOTE, or RUS
can improve the immunity of the proposed classifier against
imbalance, when compared with the state-of-the-art. For
this reason, we choose data sets belonging to categories 3–6
for these experiments.
We report the mean index values as well as Average Rank
(AR) over all experimented data sets for each of the
algorithms (ARSMS for Small and Medium-Scale data set
and ARLS for Large-Scale data sets are calculated
separately). We perform a nonparametric statistical test
known as the Wilcoxon rank sum test [52], to check if the
performance of Ada-kNN2 (Ada-kNN2 GIHS for +
experiments involving imbalanced classifiers, as we expect +
that the adaptiveness of Ada-kNN2 GIHS dispenses the need
for imbalance handling techniques
10 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

TABLE IV
COMPARISON OF CLASSIFIERS IN TERMS OF GM ON TWO-CLASS IMBALANCED DATA SETS

TABLE V
COMPARISON OF CLASSIFIERS IN TERMS OF AUROC ON TWO-CLASS IMBALANCED DATA SETS

with tunable parameters, such as SMOTE or RUS) for each
data set differs significantly from the competitors with atleast
5% level of significance. The result for this test is summarized
(detailed in the Supplementary Material) in terms of Wins
(W), Ties (T), and Losses (L). No statistical test was
undertaken while comparing the performance of Ada-kNN and
Ada-kNN2
with k∗NN and FABC, as the results for the latter contenders
are, respectively, quoted from [18] and [19].
D. Choice of Parameter α for Ada-kNN and Ada-kNN2
A suitable large choice of α will produce a good coverage
of K x i for a point x∈i P, but will also increase the computational
complexity. A small value of α, on the other hand, will reduce
the computational complexity, but may increase the chance
of overfitting by decreasing the noise immunity. Furthermore,
it is observed by us during experiments that tuning the value
of α does not significantly improve the results, suggesting
very little data dependence. These factors lead us to take a
constant value of α = 10 for all our experiments.
E. Performance of Ada-kNN and Ada-kNN2 Compared With
Other State-of-the-Art Classifiers in Terms of acc
We summarize the results of Ada-kNN, Ada-kNN2, and
the eight other competing algorithms over 17 (11 small/
medium-scale and 6 large-scale) data sets in terms of acc
in Table II. An inspection of the results on the small/medium-
scale data sets reveals that Ada-kNN2 achieved the best mean
acc and ARSMS. The Ada-kNN algorithm managed to attain
the fifth place both in terms of average acc and ARSMS.
However, its performance is statistically comparable to that of
Ada-kNN2 on 10 data sets.
In the case of the large-scale data sets, Ada-kNN2 achieved
the second best average acc slightly lower than that of MLP-
SCG. However, Ada-kNN2 achieved the best ARLS, which
confirms its better consistency and scalability. Ada-kNN,
on the other hand, exhibits lower competence on the large-
scale data sets, as is evident from the high ARLS and low
average acc. Ada-kNN failed to achieve good performance on
the large-scale data sets, possibly due to insufficient learning
of the k-terrain by the underlying MLP, leading to erroneous
prediction of the kyi values.
The comparison between Ada-kNN, Ada-kNN2, and FABC
(over eight data sets) is summarized in the first three rows
of Table III, and it shows that Ada-kNN2 achieved the best
AR. Ada-kNN and FABC both achieved the same AR with
Ada-kNN attaining the higher average acc. The comparison
between the two proposed classifiers and k∗NN (on six data
sets) is summarized in the last three rows of Table III (the
original study introducing k∗NN [19] used misclassification
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE 11

TABLE VI
COMPARISON OF CLASSIFIERS IN TERMS OF AURPC ON TWO-CLASS IMBALANCED DATA SETS

TABLE VII
COMPARISON OF CLASSIFIERS IN TERMS OF GM ON MULTI-CLASS IMBALANCED DATA SETS

TABLE VIII
COMPARISON OF CLASSIFIERS IN TERMS OF AUROC ON MULTI-CLASS IMBALANCED DATA SETS

error as index, which we convert to acc for the sake of
consistency). Ada-kNN achieved both the best average acc
and rank in this case, followed by Ada-kNN2.
These findings imply that the proposed classifiers outper-
form the state of the art on small/medium-scale data sets
due to the adaptive choice of k values. Ada-kNN2 is also
able to retain the good performance on the large-scale data
sets, unlike Ada-kNN which seems to suffer from scalability
issues.
F. Performance of Ada-kNN + GIHS and Ada-kNN2
GIHS on Two-Class Imbalanced Data Sets in Terms
of GM, AUROC, and AURPC
In Tables IV–VI, we summarize the results in terms of GM,
AUROC, and AURPC for 15 small/medium and 7 large-scale
two-class imbalanced data sets. An inspection of the results
for the small/medium-scale data sets reveals that Ada-kNN2 +
GIHS, Ada-kNN + GIHS, and kPNN outperformed the other
contenders. Ada-kNN2 + GIHS achieved the best average
index value as well as ARSMS for both GM and AUROC.
kPNN achieved slightly better index value and ARSMS com-
pared with Ada-kNN + GIHS for AURPC. However, the
W-T-L counts for AURPC point toward the greater
consistency of +Ada-kNN2 GIHS. Therefore, the Ada-kNN2
GIHS classifier seems to be the best choice
for this type of data sets.
On the seven high-dimensional data sets, Ada-kNN2 +
GIHS, kNN + GIHS, and kPNN achieved relatively bet-
ter performance than the other competitors. Though kPNN
achieved better average index values, the lower ARLS
+
attained by Ada-kNN2 GIHS on GM as well as
AUROC implies the greater consistency of the latter. The W-
T-L counts,
12
TABLE IX
COMPARISON OF CLASSIFIERS IN TERMS OF AURPC ON MULTI-CLASS IMBALANCED DATA SETS
which report statistically comparable AURPC performance for
Ada-kNN2 GIHS against kNN GIHS as well as kPNN,
+ We now summarize the various key findings of this paper
further attest to this fact. The slightly poorer performance of
Ada-kNN2 GIHS may be due to the increasing bias of the
+ choice of k values for the kNN classifier can prove to be
index toward the accuracy on the majority class with rising
IR, as shown by an example in the Supplementary Material.
Hence, we conclude that Ada-kNN2 GIHS remains com-
+ of data sets as well.
petitive to the state of the art on this type
Thus, Ada-kNN2 GIHS can outperform the state-of-the-
+
art imbalanced classifiers over two-class imbalanced data sets
of varying scale owing to the adaptive choice of k augmented
with GIHS.
G. Performance of Ada-kNN+GIHS and Ada-kNN2
GIHS on Multi-Class Imbalanced Data Sets in Terms
of GM, AUROC, and AURPC
We report the summary of results in terms of GM,
AUROC, and AURPC, for the 12 contending classifiers on
six small/medium and five large-scale multi-class imbalanced
data sets in Tables VII–IX. On small/medium-scale data sets,
Ada-kNN2 + GIHS and kNN GIHS emerge as the top per-
formers. Ada-kNN2 + GIHS achieved better index values as
well as ARSMS for all three performance evaluating indices.
Among the contenders, kNN+GIHS demonstrated the second
best performance in terms of both the ARMS and the average
index value for all the indices. Therefore, Ada-kNN2+ GIHS
is found to be a good choice for this type of data sets.
Ada-kNN2 + GIHS and kNN GIHS performed better
than the other contenders in terms of GM and AUROC for the
large-scale data sets. This indicates that both of these methods
were able to achieve equivalent classwise accuracies for all
the classes present in a data set. However, the performance of
Ada-kNN2+ GIHS in terms of AURPC lags behind those of
other methods, such as kNN, NWkNN, and dyn-kNN, due to
the bias of the index as explained for the two-class imbalanced
case. Therefore, Ada-kNN2+ GIHS should be the method
of choice for this type of data sets. Furthermore, the kNN
+ GIHS classifier is able to retain the good performance in
terms of AURPC as well. This points toward the efficacy of
the simple, parameter-free GIHS proposal.
Hence, one should employ Ada-kNN2+ GIHS for multi-
class imbalanced data sets of varying scale, which further
indicates the usefulness of adaptive choice of k and the
proposed weighting scheme.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
V. C ONCLUSION AND F UTURE WORK
and present our concluding remarks. The proposed adaptive
very useful for data classification over the conventional global
choices (such as k 1). = This is because such adaptive
choice of k can account for the properties of the neighborhood
of the test point, which may remain ignored by a single
global choice of k. The advantage of such an approach is
evident from the consistently commendable performance of
Ada-kNN and Ada-kNN2. GIHS is a simple weighting scheme
incurring constant time complexity, which does not involve
any additional parameters. Furthermore, in contrast to the pre-
process techniques used for imbalance handling, GIHS has
an additional advantage of preserving the class distributions
of the data set. GIHS is observed to further augment the
performance of Ada-kNN and Ada-kNN2 in the presence of
class imbalance.
The MLP-based proposal (Ada-kNN) for adaptively choos-
ing k values seems to suffer from scalability issues on large-
scale data sets. On the other hand, the proposed heuristic
counterpart (Ada-kNN2) is immune to such scalability issues
as evident from its admirable performance on small as well
as large-scale data sets. Moreover, the improved performance
of Ada-kNN2 GIHS + is observed to be resilient to the
increase in IR (see the Supplementary Material). An
interesting future extension of this paper may be to replace the
MLP in Ada-kNN with a more scalable supervised learning
algorithm. Another future direction should be to investigate
how different distance measures affect the performance of
Ada-kNN and Ada-kNN2. Specifically, it may be interesting
to study the effects (of the choice of distance measure) in the
case of high- dimensional data sets, where Euclidean distance
is known to perform poorly [53].
R EFERENCES
[1] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed.
Hoboken, NJ, USA: Wiley, 2000.
[2] T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE
Trans. Inf. Theory, vol. IT-13, no. 1, pp. 21–27, Jan. 1967.
[3] P. Hall, B. U. Park, and R. J. Samworth, “Choice of neighbor
order in nearest-neighbor classification,” Ann. Statist., vol. 36, no. 5,
pp. 2135–2152, 2008.
[4] D. O. Loftsgaarden and C. P. Quesenberry, “A nonparametric estimate
of a multivariate density function,” Ann. Math. Statist., vol. 36, no. 3,
pp. 1049–1051, 1965.
MULLICK et al.: ADAPTIVE LEARNING-BASED K NN CLASSIFIERS WITH RESILIENCE TO CLASS IMBALANCE
[5] C.-L. Liu and M. Nakagawa, “Evaluation of prototype learning algo-
rithms for nearest-neighbor classifier in application to handwritten
character recognition,” Pattern Recognit., vol. 34, no. 3, pp. 601–615,
2001.
[6] C. Domeniconi, J. Peng, and D. Gunopulos, “Locally adaptive metric
nearest-neighbor classification,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 24, no. 9, pp. 1281–1285, Sep. 2002.
[7] G. Bhattacharya, K. Ghosh, and A. S. Chowdhury, “An affinity-based
new local distance function and similarity measure for kNN algorithm,”
Pattern Recognit. Lett., vol. 33, no. 3, pp. 356–363, 2012.
[8] C. C. Holmes and N. M. Adams, “A probabilistic nearest neighbour
method for statistical pattern recognition,” J. Roy. Statist. Soc. Ser. B,
Statist. Methodol., vol. 64, no. 2, pp. 295–306, 2002.
[9] A. K. Ghosh, “On optimum choice of k in nearest neighbor classifi-
cation,” Comput. Statist. Data Anal., vol. 50, no. 11, pp. 3113–3123,
2006.
[10] Y. Sun, A. K. Wong, and M. S. Kamel, “Classification of imbalanced
data: A review,” Int. J. Pattern Recognit. Artif. Intell., vol. 23, no. 4,
pp. 687–719, 2009.
[11] M. F. Møller, “A scaled conjugate gradient algorithm for fast supervised
learning,” Neural Netw., vol. 6, no. 4, pp. 525–533, Nov. 1993.
[12] R. A. Fisher, “The use of multiple measurements in taxonomic prob-
lems,” Ann. Eugenics, vol. 7, no. 2, pp. 179–188, 1936.
[13] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning rep-
resentations by back-propagating errors,” Nature, vol. 323, no. 6088,
pp. 533–536, 1986.
[14] J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo, CA,
USA: Morgan Kaufmann, 1993.
[15] J. Wang, P. Neskovic, and L. N. Cooper, “Improving nearest neighbor
rule with a simple adaptive distance measure,” Pattern Recognit. Lett.,
vol. 28, no. 2, pp. 207–213, 2007.
[16] B. Tang and H. He, “ENN: Extended nearest neighbor method for
pattern recognition,” IEEE Comput. Intell. Mag., vol. 10, no. 3, pp. 52–
60, Aug. 2015.
[17] N. García-Pedrajas, J. A. D. Castillo, and G. Cerruela-Garcia, “A
proposal for local k values for k-nearest neighbor rule,” IEEE Trans.
Neural Netw. Learn. Syst., vol. 28, no. 2, pp. 470–475, Feb. 2015.
[18] S. Ezghari, A. Zahi, and K. Zenkouar, “A new nearest neighbor classi-
fication method based on fuzzy set theory and aggregation operators,”
Expert Syst. Appl., vol. 80, pp. 58–74, Sep. 2017.
∗
[19] O. Anava and K. Levy, “k -nearest neighbors: From global to local,” in
Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 4916–4924.
[20] X. Zhang, Y. Li, R. Kotagiri, L. Wu, Z. Tari, and M. Cheriet, “KRNN:
K rare-class nearest neighbour classification,” Pattern Recognit., vol. 62,
pp. 33–44, Feb. 2017.
[21] Y. Li and X. Zhang, “Improving k nearest neighbor with exemplar
generalization for imbalanced classification,” in Proc. 15th Pacific-Asia
Conf. Adv. Knowl. Discovery Data Mining, 2011, pp. 321–332.
[22] X. Zhang and Y. Li, “A positive-biased nearest neighbour algorithm for
imbalanced classification,” in Advances in Knowledge Discovery and
Data Mining. Berlin, Germany: Springer, 2013, pp. 293–304.
[23] E. Kriminger, J. C. Príncipe, and C. Lakshminarayan, “Nearest neighbor
distributions for imbalanced classification,” in Proc. Int. Joint Conf.
Neural Netw., 2012, pp. 1–5.
[24] S. Tan, “Neighbor-weighted k-nearest neighbor for unbalanced text
corpus,” Expert Syst. Appl., vol. 28, no. 4, pp. 667–671, 2005.
[25] H. Dubey and V. Pudi, “Class based weighted k-nearest neighbor over
imbalance dataset,” in Advances in Knowledge Discovery and Data
Mining. Berlin, Germany: Springer, 2013, pp. 305–316.
[26] N. Japkowicz, “The class imbalance problem: Significance and strate-
gies,” in Proc. Int. Conf. Artif. Intell. (ICAI), 2000, pp. 111–117.
[27] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer,
“SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell.
Res., vol. 16, no. 1, pp. 321–357, 2002.
[28] D. Wettschereck and T. G. Dietterich, “Locally adaptive nearest
neighbor algorithms,” in Proc. Adv. Neural Inf. Process. Syst., 1994, pp.
184–191.
[29] S. Ougiaroglou, A. Nanopoulos, A. N. Papadopoulos, Y. Manolopou-
los, and T. Welzer-Druzovec, “Adaptive k-nearest-neighbor classifica-
tion using a dynamic number of nearest neighbors,” in Advances in
Databases and Information Systems. Berlin, Germany: Springer, 2007,
pp. 66–82.
13
[30] G. Hjaltason and H. Samet, “Distance browsing in spatial databases,”
ACM Trans. Database Syst., vol. 24, no. 2, pp. 265–318, 1999.
[31] G. Bhattacharya, K. Ghosh, and A. S. Chowdhury, “A probabilistic
framework for dynamic k estimation in kNN classifiers with certainty
factor,” in Proc. 8th Int. Conf. Adv. Pattern Recognit., Jan. 2015, pp. 1–5.
[32] P. Branco, L. Torgo, and R. P. Ribeiro, “A survey of predictive mod-
eling on imbalanced domains,” ACM Comput. Surv., vol. 49, no. 2,
pp. 31:1–31:50, 2016.
[33] V. López, A. Fernández, S. García, V. Palade, and F. Herrera, “An
insight into classification with imbalanced data: Empirical results and
current trends on using data intrinsic characteristics,” Inf. Sci., vol. 250,
pp. 113–141, Nov. 2013.
[34] B. Krawczyk, “Learning from imbalanced data: Open challenges and
future directions,” Prog. Artif. Intell., vol. 5, no. 4, pp. 221–232, 2016.
[35] J. Zhang and I. Mani, “kNN approach to unbalanced data distributions:
A case study involving information extraction,” in Proc. ICML
Workshop Learn. Imbalanced Datasets, 2003, pp. 1–7.
[36] L. Wang, L. Khan, and B. Thuraisingham, “An effective evi-
dence theory based k-nearest neighbor (kNN) classification,” in Proc.
IEEE/WIC/ACM Int. Conf. Web Intell. Intell. Agent Technol., Dec. 2008,
pp. 797–801.
[37] H. Wang and D. Bell, “Extended k-nearest neighbours based on
evidence theory,” Comput. J., vol. 47, no. 6, pp. 662–672, 2003.
[38] W. Liu and S. Chawla, “Class confidence weighted kNN algorithms
for imbalanced data sets,” in Proc. 15th Pacific-Asia Conf. Adv. Knowl.
Discovery Data Mining, 2011, pp. 345–356.
[39] P. Bhattacharyya and B. K. Chakrabarti, “The mean distance to the
nth neighbour in a uniform distribution of random points: An appli-
cation of probability theory,” Eur. J. Phys., vol. 29, no. 3, p. 639,
2008.
[40] M. Lichman. (2013). UCI Machine Learning Repository. [Online].
Available: http://archive.ics.uci.edu/ml
[41] G. Rätsch. (2001). IDA Benchmark Repository. [Online]. Available:
http://ida.first.fhg.de/projects/bench/benchmarks.htm
[42] I. Triguero et al., “KEEL 3.0: An open source software for multi-stage
analysis in data mining,” Int. J. Comput. Intell. Syst., vol. 10, no. 1,
pp. 1238–1249, 2017.
[43] R. Akbani, S. Kwek, and N. Japkowicz, “Applying support vector
machines to imbalanced datasets,” in Proc. Eur. Conf. Mach. Learn.,
2004, pp. 39–50.
[44] I. Guyon. (2006). Datasets for the Agnostic Learning vs. Prior
Knowledge Competition. [Online]. Available: http://www.agnostic.
inf.ethz.ch/datasets.php
[45] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vec-
tor machines,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 3,
pp. 27:1–27:27, 2011.
[46] M. Kubat and S. Matwin, “Addressing the curse of imbalanced training
sets: One-sided selection,” in Proc. 14th Int. Conf. Mach. Learn., 1997,
pp. 179–186.
[47] M. A. Maloof, “Learning when data sets are imbalanced and when costs
are unequal and unknown,” in Proc. Int. Conf. Mach. Learn., 2003,
pp. 1–5.
[48] D. J. Hand and R. J. Till, “A simple generalisation of the area under the
ROC curve for multiple class classification problems,” Mach. Learn.,
vol. 45, no. 2, pp. 171–186, 2001.
[49] J. Davis and M. Goadrich, “The relationship between precision-recall
and ROC curves,” in Proc. 23rd Int. Conf. Mach. Learn., 2006,
pp. 233–240.
[50] T. Raeder, T. R. Hoens, and N. V. Chawla, “Consequences of variability
in classifier performance estimates,” in Proc. IEEE 10th Int. Conf. Data
Mining (ICDM), Dec. 2010, pp. 421–430.
[51] T. Raeder, G. Forman, and N. V. Chawla, “Learning from
imbalanced data: Evaluation matters,” in Data Mining: Founda-
tions and Intelligent Paradigms. Berlin, Germany: Springer, 2012,
pp. 315–331.
[52] M. Hollander and D. A. Wolfe, Nonparametric Statistical Methods.
Hoboken, NJ, USA: Wiley, 1999.
[53] C. C. Aggarwal, A. Hinneburg, and D. A. Keim, “On the sur-
prising behavior of distance metrics in high dimensional space,”
in Proc. 8th Int. Conf. Database Theory, London, U.K., 2001,
pp. 420–434.