Sunteți pe pagina 1din 9


4, JULY 2009

A Pattern Similarity Scheme for Medical

Image Retrieval
Dimitris K. Iakovidis, Member, IEEE, Nikos Pelekis, Evangelos E. Kotsifakos, Ioannis Kopanakis,
Haralampos Karanikas, and Yannis Theodoridis, Member, IEEE

AbstractIn this paper, we propose a novel scheme for efficient The benefits emanating from the application of content-based
content-based medical image retrieval, formalized according to the approaches to medical image retrieval range from clinical de-
PAtterns for Next generation DAtabase systems (PANDA) frame- cision support to medical education and research [5]. These
work for pattern representation and management. The proposed
scheme involves block-based low-level feature extraction from im- benefits have motivated researchers either to apply general-
ages followed by the clustering of the feature space to form higher- purpose CBIR systems to medical images [3] or to develop
level, semantically meaningful patterns. The clustering of the fea- dedicated ones explicitly oriented to specific medical domains.
ture space is realized by an expectationmaximization algorithm Specialized CBIR systems have been developed to support the
that uses an iterative approach to automatically determine the retrieval of various kinds of medical images, including high-
number of clusters. Then, the 2-component property of PANDA
is exploited: the similarity between two clusters is estimated as a resolution computed tomographic (HRCT) images [6], breast
function of the similarity of both their structures and the measure cancer biopsy slides [7], positron emission tomographic (PET)
components. Experiments were performed on a large set of refer- functional images [8], ultrasound images [9], pathology im-
ence radiographic images, using different kinds of features to en- ages [10], and radiographic images [11].
code the low-level image content. Through this experimentation, it Common ground for most of the systems cited earlier is that
is shown that the proposed scheme can be efficiently and effectively
applied for medical image retrieval from large databases, provid- image retrieval is based on similarity measures estimated di-
ing unsupervised semantic interpretation of the results, which can rectly from low-level image features. This approach is likely
be further extended by knowledge representation methodologies. to result in the retrieval of images with significant perceived
Index TermsContent-based image retrieval (CBIR), feature differences from the query image, since low-level features usu-
extraction, patterns, pattern similarity, semantics. ally lack semantic interpretation. This has motivated researchers
to focus on the utilization of higher-level semantic represen-
tations of image contents for content-based medical image re-
I. INTRODUCTION trieval. Recent approaches include semantic mapping via hybrid
NE of the primary tools used by physicians is the compar- Bayesian networks [12], semantic error-correcting output codes
O ison of previous and current medical images associated
with pathologic conditions. As the amount of pictorial informa-
(SECC) based on individual classifiers combination [13], and a
framework that uses machine learning and statistical similarity
tion stored in both local and public medical databases is growing, matching techniques with relevance feedback [14]. However,
efficient image indexing and retrieval becomes a necessity. these approaches involve supervised methodologies that require
During the last decade, the advances in information technol- prior knowledge about the dataset and introduce constraints to
ogy allowed the development of content-based image retrieval the semantics required for the image retrieval task.
(CBIR) systems, capable of retrieving images based on their A state-of-the-art CBIR approach has been presented in [15].
similarity with one or more query images. Indicative examples It utilizes a continuous and probabilistic image representation
of such systems are QBIC [1], SIMPLicity [2], and FIRE [3]. scheme that involves Gaussian mixture modeling (GMM) along
It is interesting that more than 50 CBIR systems are surveyed with information-theoretic image matching via the Kullback
in [4]. Leibler (KL) measure. The results reported in [15] show that
this approach is very effective for radiographic image retrieval;
however, its efficiency for large image retrieval tasks still re-
Manuscript received April 30, 2007; revised March 12, 2008. Current version mains a challenge.
published July 6, 2009. This work was supported in part by the Operational Pro-
gramme Information Society of the Greek Ministry of Development, General In this paper, we propose an unsupervised approach for ef-
Secretariat for Research and Technology, and in part by the European Union. ficient content-based medical image retrieval that utilizes sim-
D. K. Iakovidis is with the University of Athens, GR15784 Panepistimiopolis, ilarity measures, defined over higher-level patterns that are as-
Ilisia, Greece (e-mail:
N. Pelekis, E. Kotsifakos, and Y. Theodoridis are with the Department sociated with clusters of low-level image feature spaces. The
of Informatics, University of Piraeus, 185 34 Piraeus, Greece (e-mail: term pattern is considered in the context of a state-of-the-art;; framework called PAtterns for Next generation DAtabase sys-
I. Kopanakis is with Technological Educational Institute of Crete, 71 004
Crete, Greece (e-mail: tems (PANDA) developed for the representation and the man-
H. Karanikas is with the University of Manchester, Manchester M13 9PL, agement of data mining results, and it describes a compact,
U.K. (e-mail: rich-in-semantics result of a data mining process [16].
Color versions of one or more of the figures in this paper are available online
at The proposed approach combines the advantages of
Digital Object Identifier 10.1109/TITB.2008.923144 the clustering-based CBIR methodologies [17][19] with a

1089-7771/$25.00 2009 IEEE

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

semantically rich representation of medical images. Moreover, A pattern-type PT is called complex if its structure schema
unlike related CBIR approaches that exploit multidimensional SS includes another pattern type, otherwise PT is called simple.
indexing techniques, such as R-trees [1], [20], iconic index Thus, a EuclideanCluster is a simple pattern type, whereas a
trees [21], and meshes of trees [22], the efficiency of the pro- clustering extracted, e.g., by a partitioning clustering algorithm
posed approach is hardly affected by increasing the dimension- is considered a complex pattern type since it can be modeled as
ality of the low-level feature representation. a set of clusters with no measure component
The major contributions of this paper are the following.  
SS : {EuclideanCluster}
1) We define a novel representation of medical images treated PartitioningCluster = .
MS :
as rich-in-semantics complex patterns. Each complex pat-
tern comprises a set of simple patterns representing clus- In this notation, if PT is a pattern type, then p = s, m is
ters of image regions associated with anatomic specimens an instance of PT, where s, m are the corresponding structure
in an unsupervised way. The pattern representation of clus- and measure values of the pattern. With respect to the previous
ters involves both structural descriptors and quality mea- example, a possible instance of a 3-D EuclideanCluster could
sures. be
2) We propose a novel scheme for the assessment of the s : (center : [0.1, 0.3, 0.45], radius : 0.77)
similarity between complex patterns (i.e., medical images) Cluster1 = .
m : (sup p : 0.15)
for CBIR purposes.
3) We conduct a comprehensive set of experiments over a According to PANDA, the distance dis between two simple
publicly available set of radiographic images, in order patterns p1 , p2 of the same type is computed by combining
to thoroughly evaluate our approach and demonstrate its the distance between both the structure s and the measure m
effectiveness and efficiency in comparison to state-of-the- components with a gathering function fgath [16]
art techniques. dis(p1 , p2 ) = fgath (disstruct (p1 .s, p2 .s),
The rest of this paper is structured as follows. Section II
outlines the PANDA framework, which provides necessary dism eas (p1 .m, p2 .m)) (1)
background information to the reader. The proposed pattern where pi .s and pi .m denote the structure and the measure, re-
similarity scheme for medical image retrieval is presented in spectively, of the pattern pi . The dot in this notation denotes
Section III. The results obtained from the experimental evalu- that the variable on the right is a member of the pattern instance
ation of the proposed scheme are apposed in Section IV. The on the left, according to the notation used in object-oriented
conclusions along with the future perspectives are summarized modeling.
in Section V. On the other hand, the distance between two complex patterns
is defined as the aggregate distance between their constituent
II. PANDA FRAMEWORK patterns, according to a coupling that associates constituent pat-
The efficient management of patterns extracted from medical terns (this is a recursive definition since a complex pattern could
image databases is of vital importance due to the extremely large be composed of other complex patterns, and so on).
storage requirements as well as the complexity of such kind of The definition of p.s and p.m components of a pattern p ex-
raw data. Taking advantage of the PANDA framework [16], we tracted from medical images and the selection of appropriate
adopt the idea of a pattern-base (PB) keeping information about disstruct , dism eas , fgath functions are challenges adopted in this
extracted patterns in a compact and unified way. A PB consists paper, and will be discussed in depth in the following section.
of three basic layers: the pattern type, the pattern, and the class.
A pattern type is a description of the pattern structure. A pattern III. MEDICAL IMAGE RETRIEVAL USING PATTERNS
is an instance of the corresponding pattern type and class is a The proposed content-based medical image retrieval scheme
collection of semantically related patterns of the same pattern is outlined in Fig. 1. It involves four steps: 1) low-level fea-
type [16]. ture extraction from each of the registered and query images;
Formally, a pattern type PT is defined as a pair PT = SS, MS, 2) clustering of the extracted feature vectors per image; 3) pat-
where SS defines the pattern space by describing the structure tern instantiation of the resulted clusters; and 4) computation
schema of the pattern type, while the measure schema MS quan- of pattern similarities. The registration of a new image into
tifies the quality of the source data representation achieved by the database involves steps 1)3), whereas step 4) is processed
patterns of this pattern type. As an example, consider a pattern during the retrieval task.
type representing Euclidean-distance, spherical-like clusters in
a D-dimensional space. The structure of such a pattern type may A. Low-Level Image Feature Extraction
be modeled by specifying the cluster center (a D-dimensional
Each of the images registered in the database, as well as the
vector) and a radius (a real value). The measure for a cluster
query image are raster scanned with a sliding window of user-
might be, for instance, its support, that is, the fraction of the
defined size, sampling image blocks at a given sampling step.
data points represented by the cluster. As such
  The sampling step may allow consecutive blocks to overlap. For
SS : (center : [Real]D1 , radius : Real) each block, a set of N features fi , i = 1, . . ., N, is calculated to
EuclideanCluster= .
MS : (sup p : Real) form a single feature vector F. The number of feature vectors

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

from its approximation (lower frequency) coefficients [27]. A

compact representation of the distributions of the approxima-
tion and the detail coefficients can be obtained by first-order
statistical approximation.
However, it should be noted that this paper focuses on the
utility of the proposed pattern similarity scheme rather than on
the selection of an optimal feature set for a particular image
retrieval task.

B. Clustering
The low-level feature vectors are clustered using mixture
models that model the data by a number of Gaussian distri-
butions. A cluster corresponds to a set of distributions, one for
each dimension of the dataset. Each distribution is described in
terms of mean and standard deviation. A probabilistic approach
Fig. 1. Outline of the proposed content-based image retrieval methodology.
The black arrows indicate the data flow for image retrieval, whereas the grey
to assigning feature vectors to clusters is used.
arrows indicate the data flow for the registration of a new image. For 1-D datasets, a mixture is a set of c Gaussian probabil-
ity distributions, representing c clusters. The parameters of a
mixture model are determined by the expectation maximization
produced for each image depends on the size, the dimensions (EM) algorithm [31]. With c Gaussians, the probability density
of the sliding window, and the sampling step. Typically, the function of a variable X is
sampling parameters and the features characterizing the low- 
1 1
e 2 (X i ) (X i )
1 T
level image content are selected based on the details associated f (X|) = ppi  i (2)
with the image collection and the retrieval task [23]. Color, i=1 (2)d |i |
texture, and shape are the three major classes of image features 
commonly used in CBIR [1], [4]. Considering an image as a set where ppi > 0, ci=1 ppi = 1, and d is the dimension  of the
of block samples, the features used with the proposed pattern feature vector. The set of model parameters {ppi , i , i }, i =
similarity scheme should describe properly the local content 1, . . ., c, consists of the prior probabilities ppi of
the Gaussian i,
of the image. The appropriateness of different local descriptors the mean vector i , and the covariance matrix i for the Gaus-
depending on the kind of medical images is discussed in [5] sian i, respectively. The EM algorithm is used to estimate the
and [14]. maximum likelihood L of given a set of features {x1 , . . . , xN }
In the case of radiographic medical image retrieval, local grey

level intensity and texture features have proved to discriminate L( | X) = log f (xj | ). (3)
best the depicted specimens. Such features include raw pixel j =1
values used along with an image distortion similarity model,
local feature histograms, and local relational features [23], [24]. The model parameters are initialized with random values. The
Recently, in [15] it was shown that highest retrieval precision can algorithm starts by calculating the probabilities that a vector
be achieved by combining intensity and texture contrast along should belong to each distribution. These probabilities are used
with the corresponding spatial coordinates. However, the intro- to compute a new estimate for the parameters. The whole process
duction of spatial information into the feature vectors makes is repeated until the parameters converge to a constant or almost-
them dependent on the patients position. Although patients are constant estimate. The algorithm results in a set of distributions,
usually positioned in a standard way during the acquisition of a a vector of pairs of means and standard deviations , each of
radiograph, there are still many cases in which this is not practi- which corresponds to a feature, and outputs the size of the cluster
cally feasible. For example, this is the case with the acquisition (the number of vectors that belong to the cluster). The vector
of radiographs of critically ill patients using portable radio- of means of the distributions for every feature represents the
graphic devices [25] and with the acquisition of radiographs of centroid of the cluster.
upper or lower extremities [26]. The EM algorithm exhibits many advantages over other clus-
In this paper, we adopt a standard, multiscale statistical ap- tering algorithms that make it appealing for use with the CBIR
proach for the representation of the radiographic image regions methodology described in this paper. Combining EM with the
that preserves local features, and does not depend on spatial v-fold cross-validation algorithm [32], the number of clusters
coordinates. It is based on the 2-D discrete wavelet transform in the output of the algorithm can automatically be determined.
(2D-DWT), an efficient, yet effective transformation that has The v-fold cross-validation technique works by partitioning the
proved useful in a variety of medical image processing and anal- data into v equally sized segments. Starting with one cluster,
ysis applications, including the CBIR [5], [27][30]. It enables EM is performed v times holding out one segment at a time for
coding of image texture into detail (higher frequency) coeffi- test purposes and the likelihood is averaged over all the results.
cients, whereas image intensity information can be extracted Next, EM is performed over two clusters, and if the likelihood

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

increases, the number of clusters is set to two and the process is D. Computation of Pattern Similarities
repeated until the estimated likelihood begins to decrease [33]. Let us recall that the objective of a CBIR system is the estima-
Furthermore, the EM algorithm is more general than, e.g., tion and ranking of the similarity between query and registered
K-means [34], as it can find clusters of different sizes and ellip-
images. Aiming at the estimation of the similarity between two
soidal shapes. Most importantly, the distributions representing medical images [defined as complex patterns in (6), we first
the clusters at the output of the EM algorithm can be easily uti-
have to define the distance over the structures and the measures
lized for pattern instantiation by the PANDA framework, which
of two simple patterns P1 and P2 . Since complex patterns are
is discussed next. decomposed into a number of simple patterns, in comparing
two medical images, MI1 and MI2 , we need a way to associate
C. Pattern Instantiation component patterns of MI1 to component patterns of MI2 . To
this end, the coupling type constrains the way component pat-
The clusters resulting from the EM algorithm are considered
terns can be associated (i.e., matched). Next, we first propose
as patterns extracted from the image database, and are rep-
an effective way to measure the distance between two simple
resented and handled according to the PANDA formalization
patterns, and then we present (see 11), which is our choice for
presented in Section II. Hence, given a clustered image com-
coupling them.
prising of M simple patterns Pi , i = 1, . . ., M, and with respect
The distance between the measures of two patterns is pro-
to the output of the EM algorithm, a Specimeni is instantiated
posed to be defined as the absolute difference of the scatter
for each pattern Pi representing a physical anatomic specimen
values, each one weighted by the corresponding prior proba-
in a medical image
bility of the patterns, normalized by the sum of the two scatter
  values. Formally
SS : (D : [ : [Real], : [Real]]N1 )
Specimeni = . (4)
MS : (pp : [Real], SV : [Real])
|P1 .pp P1 .SV P2 .pp P2 .SV|
More specifically, the structure schema SS of a specimen is dism ean (P1 , P2 ) = . (7)
represented by the pair (, ) of the distribution Dj for each P1 .SV + P2 .SV
of the N features (j = 1, . . ., N) in pattern Pi , respectively.
Correspondingly, the measure schema MS of a specimen is Intuitively, (7) quantifies the interpattern divergence between
represented by two values, the prior probability (pp) and the the cohesiveness of two clusters. It should be noted that this
scatter value (SV) of Pi . Formally, the prior probability pp definition overrides the inefficiency of the relativeness of the
is defined as the fraction of the feature vectors of the image scatter value with respect to the number of items in the cluster,
that belong to pattern Pi . Intuitively, pp is equivalent with the as each scatter value is weighted by the fraction of the feature
support measure widely used in data mining models. In this vectors of the image that belong to pattern Pi .
case, it provides an indication of the size of the specimen. On Regarding the structural similarity between P1 and P2 , we
the other hand, SV is a measure of the cohesiveness of the data search for a measure that evaluates the closeness of two sets
items in a cluster with respect to the centroid of the cluster, of distributions, as P1 and P2 are. Further decomposing the
and it is a commonly used intrinsic measure of the quality of problem, we should first define a method of computing the
a cluster [35]. Formally, the scatter value SV of a specimen is similarity between two distributions D1 and D2 . To achieve this,
defined as we use the standardized difference d between two distributions,
 as defined by Cohen [36]. Cohens d is defined as the absolute
SV = (xk cP i )2 (5) difference between the means of the distributions, divided by
k P i the root- mean square of the two standard deviations
where xk are the feature vectors that belong to pattern Pi and

cP i is the corresponding centroid, which is also a vector having |D 1 .D 2 .|
, if D1 . = 0 or D2 . = 0
D . 2 D . 2
the same dimensionality as xk , and its value in each dimension d(D1 , D2 ) = 1
is computed as the average from the corresponding features |D1 . D2 .| , otherwise
values belonging to pattern Pi . A low scatter value indicates (8)
good scatter quality, but it should be noted that this is a relative Cohens distance d is a nonnegative real number interpreting
measure of quality, since it depends on the number of items in the overlap between two distributions. If d is zero, the distribu-
the cluster. tions are identical. Low d indicates quite similar distributions
In this context, a medical image MI is considered as a complex whereas high d indicates quite dissimilar distributions. If both
pattern standard deviations are zero, the absolute difference between
the means is used as the distance between the distributions.
SS : {Specimen} Cohens distance is the vehicle to automate and materialize
MI = (6)
MS : the intuitive overlap between two distributions. Having this, we
define that the structural distance between two sets of distri-
consisting of a set of simple patterns (i.e., specimens), which butions (i.e., two patterns P1 and P2 ) should be the result of
follow the definition in (4). an aggregate function gaggr (9), which interrelates the different

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

distance scores achieved by each pair of distributions patterns of each image as follows
M  K
d(Dj1 , Dj2 ) dis(MI1 , MI2 ) = dis(PiM I 1 , PjM I 2 ) (11)
disstruct (P1 , P2 ) = gaggr j = 1, 2, . . . , N M K i=1 j =1

(9) where M and K are the respective numbers of constituent simple
where d is Cohens distance and is a normalization factor patterns of each image with respect to the output of the EM
of the domain of values of the gaggr function (gaggr : [0, 1] algorithm. Though various coupling types can be applied in
[0, 1]), which intuitively corresponds to the Cohens d score the context of the PANDA framework [16], we adopt the all-
over which two distributions are considered totally dissimilar by-all matching expressed by (11) so as to avoid bias toward
(i.e., they do not overlap). In this connection, gaggr function specific patterns. The final outcome is the average of all possible
can be any mapping that initially performs a feature selection matchings.
process and subsequently applies the aggregation function upon
the selected features. Examples of such functions include: 1) IV. RESULTS
the minimum function gm in (i.e., selection of the most similar
A number of experiments were performed with radiographic
distributions); 2) the average function gavg (i.e., selection of
images from the image retrieval in medical applications (IRMA)
the average among the distances computed for each pair of the
dataset [37], which is often used as a reference for medical image
N features); and 3) the average of the k nearest distributions
retrieval tasks. It currently contains 10 000 arbitrarily selected
function gavg kND (i.e., selection of k N most similar pairs
anonymous radiographic images taken randomly from patients
of distributions). In the last case, the k parameter may not be
of different ages, genders, and pathologies during medical rou-
given explicitly, yet it can be defined implicitly by relaxing the
tine. The images are categorized into 116 classes according
parameter. Formally
to the IRMA code [38]. This code comprises four fields: 1) the
imaging modality; 2) direction of the imaging device and the pa-
N tient; 3) the anatomic body part that is examined; and 4) the sys-
gm in = min{d(Dj1 , Dj2 )}, gavg = d(Dj1 , Dj2 ), gavg k ND
j =1 N j =1 tem under investigation. The particular dataset comprises only
plain X-ray images of various directions (such as anteroposte-
k rior and mediolateral), anatomic body parts (such as cranium,
= kND(d(Dj1 , Dj2 )) spine, arm, elbow, and chest) and systems under investigation
k j =1
(such as musculoskeletal, gastrointestinal, and uropoietic). The
IRMA code information of each image is provided as the ground
where function kND returns the k most similar distributions. truth along with the dataset. Other patient data and pathology
To this point, we have defined dism eas and disstruct [(7) and information are unavailable. All radiographic images are in 8-bit
(9), respectively] between two patterns (i.e., clusters on fea- greyscale format and have been downscaled to fit into a 256
tures extracted by two medical images following the proposed 256-pixel bounding box maintaining the original aspect ratio.
CBIR methodology). In the sequel, we aggregate these distances From the available dataset, a subset of 90% of the images was
by using a wise weighted sum function. Formally, the distance registered in the database, whereas a nonoverlapping subset of
dis(P1 , P2 ) between two patterns P1 and P2 is defined as 10% of the images was used for querying the pattern-base.
Each image was sampled in blocks using overlapping sliding
dis(P1 , P2 ) = disstruct (P1 , P2 ) windows. The details of the feature extraction method used
+ (1 disstruct (P1 , P2 )) dism ean (P1 , P2 )2 . (Section III) include a 3-level biorthogonal spline wavelet de-
composition of each sampled block and the estimation of the
(10) first two wavelet moments from each band. This process results
in a 20-D feature vector per block.
The intuition behind our choice is that the more similar The determination of the sampling parameters was based on
the structures are, the more the measure distance should con- preliminary experiments seeking the maximum average dis-
tribute to the total distance score. This implies that if structures tance (11) between complex patterns MI of the different cat-
are totally different, the distance should be 1, irrespective of egories comprising the registered dataset. The sampling pa-
the measure. This choice further implies that we give emphasis rameters tested before each CBIR experiment include sliding
on the structural similarity. This is additionally strengthened by windows of 32 32, 64 64 and 128 128 pixels. In all
multiplying the factor 1 disstruct (i.e., the similarity between cases, the maximum average distance was obtained with win-
the structures of the patterns) with a smaller value than the actual dows of 64 64 pixels. Variation of the overlap (0%, 25%,
measure distance dism eas . Recall that dism eas takes values in the 50%, and 75%) between the sampled blocks did not affect
domain [0, 1], so by taking its square we denote the relaxing of this result. Increasing the overlap provides better localization
the dism eas contribution. of the patterns but produces many more sampled blocks, affect-
Having defined the distance between simple patterns, to com- ing the efficiency of both the feature extraction and the pattern
pare two medical images MI1 and MI2 (i.e., two complex pat- instantiation tasks. Thus, a 50% overlap, i.e. a 32 pixel step, was
terns), we adopt the coupling methodology between the different used as a compromise between localization and efficiency.

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

Fig. 2. (a) Original radiographic images, (b) clustering output, and (c) 3-D
visual representation of the feature spaces.

In the following, we present qualitative results of the pattern Fig. 3. Average precision versus recall using gav g k N D , gav g , and gm in ag-
instantiation realized via clustering, and measure the perfor- gregation functions for (a) all, (b) chest, and (c) cranium, categories.
mance of the proposed scheme, in terms of effectiveness and in
terms of efficiency.
defined as the ratio of the relevant images retrieved over the total
A. Pattern Instantiation via Clustering relevant images in the database, and precision is defined as the
ratio of the relevant images retrieved over the total number of
The feature vectors extracted from each image were clus-
images retrieved, relevant or not. To enable comparisons with
tered using an implementation of the EM algorithm available in
other medical image retrieval methodologies using a standard
the Waikato environment for knowledge analysis (WEKA) data
single-figure measure, the area under the interpolated precision-
mining tool [33] using the 10-fold cross-validation algorithm to
recall curve (AUC) is estimated [40].
determine the number of clusters. Each cluster was represented
The proposed scheme was tested using the three alternative
by a pattern Specimeni , i = 1, . . ., M (see (4)), and each image
aggregation functions gaggr discussed in Section III-D. The re-
was represented by a complex pattern MI [see (6)]. Fig. 2(a)
sults, in terms of average precision versus recall estimated for
illustrates three radiographic images from breast, abdomen, and
all 116 categories, are illustrated in Fig. 3(a). Indicatively, in
hand categories (from left to right). The respective clusterings
Fig. 3(b) and (c), we present the precision versus recall charts for
obtained are illustrated in Fig. 2(b). The different grey levels
two independent categories of chest and cranium radiographs. It
in Fig. 2(b) indicate the different specimen patterns found in
is evident that best retrievals are achieved by using the average
the images. Fig. 2(c) illustrates projections of the 20-D feature
of the k nearest distributions function avg kND .
vectors to a 3-D space constructed according to the centroid-
Fig. 3(a) shows that, for a recall of 90%, the average precision
preserving projection technique [39]. It can be observed that
achieved using avg kND is almost 45%, and the corresponding
the clustering produced is quite meaningful in terms of seman-
AUC estimated is 74%. It is worth noting that these results
tics, i.e., the breast and the perceived differences in its structure
could only marginally improve upon a denser sampling scheme.
are clearly depicted, the region of the abdomen is well defined
Compared with a simple method that uses global grey-level his-
and separated from the upper part of the body, and the palm
tograms as features and histogram intersection as an appropriate
is differentiated from the fingers. However, for the fingers, the
dissimilarity measure [41], the average precision for 90% re-
algorithm assigned two specimens instead of one, but this can
call is approximately 10%, and the corresponding AUC reaches
be attributed to the large size of the sampled blocks as compared
only 17%. The AUC obtained with the proposed scheme us-
with the gap between the fingers.
ing local grey-level histogram information reaches 34%. The
corresponding precision versus recall curves are illustrated in
B. Effectiveness
Fig. 4.
The patterns from the registered radiographic images were The precision reported in [15] for 90% recall seems to be
used to build a pattern base (see Fig. 1). In order to quantita- comparable with the one achieved with the proposed approach;
tively assess the effectiveness of the proposed pattern similarity however, the dataset from which that precision is estimated is
scheme, we evaluate its capability to retrieve images by adopt- significantly smaller comprising only 1500 radiographs from
ing the popular recall and precision measures, where recall is 17 categories. In order to derive comparable estimates between

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

Fig. 6. Speedup factor between the conventional and the proposed approach
as a function of the number of blocks per image.

Fig. 4. Comparative precision versus recall chart.

C. Efficiency
In this section, we measure the efficiency of the proposed
medical image similarity scheme that involves pattern com-
parisons, in comparison with the performance of the conven-
tional scheme that involves exhaustive comparisons of the fea-
ture vectors. A vector comparison in the conventional approach
is considered equivalent to a pattern comparison in the proposed
scheme. The experiments were performed on a workstation with
Intel Pentium M1.6 processor having 1 GB RAM and 60 GB
hard disk.
We have chosen the sequential, exhaustive scan as the yard-
stick for our method, as other common methods such as R-trees
are sensitive to the high dimensionality of the feature vectors,
which is usual in CBIR applications (e.g., a dimensionality of 64
in [42] and at least 2 N = 40 in our case, where N is the number
of features in a pattern). The performance of these approaches
degrades rapidly as dimensionality increases. For instance, it
has been shown that even for a dimensionality of as low as 5,
the R -tree behavior in similarity search is problematic [42].
Fig. 5. (a) Query requesting nine chest images similar to the upper-left image The main reason is that, with the growth of the dimensionality,
(1,1): All retrieved images belong to the same category; (b) A query requesting
nine abdomen-gastrointestinal system images similar to the upper-left image the overlap in the internal nodes of the tree increases and, as
(1,1): all retrieved images belong to the same category, except (1,4) and (2,5), such, its discrimination ability decreases.
which belong to the abdomen-uropoietic system. (Notation (i, j) indicates the The speedup factor between the conventional and the pro-
positioning of an image at the ith row, jth column in the figure.).
posed approach as a function of the number of blocks per image
is illustrated in Fig. 6. It can be observed that the advantage
of the proposed approach increases with the number of blocks
the two CBIR approaches, a retrieval experiment was run with per image (e.g., by increasing the sampling step), and for a few
the proposed scheme on a subset of the available data generated hundreds of blocks per image, it requires almost three orders of
according to the guidelines provided in [15]. The AUC estimated magnitude fewer comparisons than the conventional approach.
for the proposed approach on this subset reached 78%, whereas On the other hand, in [15], a speedup of two orders of magni-
the AUC estimated from [15] is approximately 66%. tude compared to the conventional approach is reported. More-
Two example retrievals using gavg kND are illustrated in over, in the same research, it is noted that the GMM-KL frame-
Fig. 5. The first image of each sequence is the query image, work is not yet capable of coping with large image retrieval tasks
and the rest are the nine retrieved images requested. Fig. 5(a) that extend more than 6000 images due to the computational load
shows that all the retrieved images belong to the same category. involved with the KL measure. We further estimated the aver-
Fig. 5(b) shows that two of the retrieved images belong to a age processing time (CPU plus I/O time) for the comparison
different class than that of the query image. However, the main of a pair of images. For the aforesaid experimental setting, the
difference between the two categories is hardly perceivable and proposed pattern similarity scheme requires always less than
located in the region of pelvis (lower part of the image at the 0.1 ms. The average time required for the mixture model pa-
center). Similar observations are valid for queries performed rameters to converge to a constant or almost-constant estimate is
using radiographic images from other categories. 0.22 0.04 s.

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

V. CONCLUSION [10] L. Zheng, A. W. Wetzel, J. Gilbertson, and M. J. Becich, Design and

analysis of a content-based pathology image retrieval system, IEEE
We presented a novel scheme for efficient content-based med- Trans. Inf. Technol. Biomed., vol. 7, no. 4, pp. 249255, Dec. 2003.
ical image retrieval. This scheme utilizes rich-in-semantics pat- [11] I. El-Naqa, Y. Yang, N. P. Galatsanos, R. M. Nishikawa, and M. N.
Wernick, A similarity learning approach to content-based image retrieval:
tern representations of medical images, defined in the context Application to digital mammography, IEEE Trans. Med. Imag., vol. 23,
of PANDA, a framework for representing and handling data no. 10, pp. 12331244, Oct. 2004.
mining results. The theoretical contributions of this paper are [12] C.-Y. Lin, J.-X. Yin, X. Gao, J.-Y. Chen, and P. Qin, A semantic modelling
approach for medical image semantic retrieval using hybrid Bayesian
validated by comprehensive experimentation on the IRMA ref- networks, in Proc. 6th Int. Conf. Intell. Syst. Des. Appl. (ISDA), 2006,
erence collection of radiographic images. The results advocate pp. 482487.
both its efficiency and effectiveness in comparison with state of [13] J. Yaoa, I, S. Antani, R. Longb, G. Thoma, and Z. Zhanga, Automatic
medical image annotation and retrieval using SECC, in Proc. 19th Int.
the art. Symp. Comput.-Based Med. Syst. (CBMS), Salt Lake City, UT, Jun. 2006.
Future perspectives of this paper include: 1) systematic eval- [14] M. M. Rahman, P. Bhattacharya, and B. C. Desai, A framework for
uation of the proposed scheme for the retrieval of various kinds medical image retrieval using machine learning and statistical similarity
matching techniques with relevance feedback, IEEE Trans. Inf. Technol.
of medical images, such as endoscopic [29] and ultrasound im- Biomed., vol. 11, no. 1, pp. 5867, Jan. 2007.
ages [43] according to their pathology; 2) the enhancement of the [15] H. Greenspan and A. T. Pinhas, Medical image categorization and re-
retrieval performance by using image indexing techniques based trieval for PACS using the GMM-KL framework, IEEE Trans. Inf. Techol.
Biomed., vol. 11, no. 2, pp. 190202, Mar. 2007.
on specialized data structures; and 3) the integration of the pro- [16] I. Bartolini, P. Ciaccia, I. Ntoutsi, M. Patella, and Y. Theodoridis, A
posed scheme with ontology-based information extraction and unified and flexible framework for comparing simple and complex pat-
data mining techniques for the retrieval of medical images us- terns, in Proc. 8th Eur. Conf. Principles Pract. Knowl. Discov. Database
(PKDD), 2004, pp. 496499.
ing heterogeneous data sources. By storing the semantically rich [17] R. O. Stehling, A. X. Falcao, and M. A. Nascimento, An adaptive and
patterns along with low-level features in a unified way, accord- efficient clustering-based approach for content-based image retrieval in
ing to the PANDA framework, will enable the extension of the image databases, in Proc. Int. Symp. Database Eng. Appl., pp. 356
CBIR methodologies with knowledge representation techniques [18] C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: Image
for semantic processing and analysis. segmentation using expectation-maximization and its application to im-
age querying, IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 8,
pp. 10261038, Aug. 2002.
ACKNOWLEDGMENT [19] C. Yixin, J. Z. Wang, and R. Krovetz, CLUE: Cluster-based retrieval of
images by unsupervised learning, IEEE Trans. Image Process., vol. 14,
We would like to gratefully acknowledge the contribution no. 8, pp. 11871201, Aug. 2005.
of the anonymous reviewers for their valuable comments and [20] E. G. M. Petrakis and C. Faloutsos, Similarity searching in medical image
suggestions, which allowed us to substantially improve the pre- databases, IEEE Trans. Knowl. Data Eng., vol. 9, no. 3, pp. 435447,
May/Jun. 1997.
sentation quality of our paper. The collection of images used in [21] J. K. Wu and A. D. Narasimhalu, Identifying faces using multiple re-
this paper is courtesy of Dr. T. M. Lehmann, Image Retrieval trievals, IEEE Multimedia, vol. 1, no. 2, pp. 2038, Aug. 1994.
in Medical Application (IRMA) Group, Department of Medical [22] W.-M. Jeng and J.-H. Hsiao, An efficient content based image retrieval
system using the mesh-of-trees architecture, J. Inf. Sci. Eng., vol. 21,
Informatics, RWTH Aachen, Germany ( pp. 797808, 2005.
[23] T. Deselaers, D. Keysers, and H. Ney, Features for image retrievala
quantitative comparison, in Proc. 26th DAGM Symp., Lect. Notes Com-
REFERENCES put. Sci., 2004, pp. 228236.
[24] L. Setia, A. Teynor, A. Halawani, and H. Burkhardt, Image classification
[1] C. Faloutsos, R. Barber, M. Flickner, J. Hafner, W. Niblack, D. Petkovic,
using cluster-cooccurrence matrices of local relational features, in Proc.
and W. Equitz, Efficient and effective querying by image content, J.
8th ACM Int. Workshop Mult. Inf. Retrieval, Santa Barbara, CA, 2006,
Intell. Inf. Syst., vol. 3, pp. 231262, 1994.
pp. 173182.
[2] J. Z. Wang, J. Li, and G. Wiederhold, SIMPLIcity: Semantics-sensitive
[25] F. S. Bongard and D. Y. Sue, Current Critical Care: Diagnosis and Treat-
integrated matching for picture libraries, IEEE Trans. Pattern Anal.
ment, 2nd ed. New York: McGraw-Hill/Appleton and Lange, 2002.
Mach. Intell., vol. 23, no. 9, pp. 947963, Sep. 2001.
[26] S. B. Dowd and B. G. Wilson, Encyclopedia of Radiographic Positioning.
[3] T. Deselaers, D. Keysers, and H. Ney, FIREflexible image retrieval
vol. 2, Philadelphia, PA: Saunders, 1995.
engine: ImageCLEF 2004 evaluation, in Lecture Notes in Computer Sci-
[27] S. Mallat and A. Wavelet, Tour of Signal Processing, 2nd ed.
ence, vol. 3491, 2004, pp. 688698.
Washington, DC: Academy, 1999.
[4] R. Veltcamp and M. Tanase, Content-based image retrieval systems: A
[28] J. Z. Wang, G. Wiederhold, O. Firschein, and S. X. Wei, Content-based
Survey, Dept. Comput. Sci., Utrecht Univ., Tech. Rep. UU-CS-2000-34,
image indexing and searching using Daubechies wavelets, Int. J. Digital
Libraries, vol. 1, no. 4, pp. 311328, 1998.
[5] H. Muller, N. Michoux, D. Bandon, and A. Geissbuhler, A review of
[29] S. A. Karkanis, D. K. Iakovidis, D. E. Maroulis, D. A. Karras, and
content-based image retrieval systems in medicineclinical benefits and
M. Tzivras, Computer aided tumor detection in endoscopic video us-
future directions, Int. J. Med. Inf., vol. 73, pp. 123, 2004.
ing color wavelet features, IEEE Trans. Inf. Technol. Biomed., vol. 7,
[6] C. R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. M. Aisen, and
no. 3, pp. 141152, Sep. 2003.
L. S. Broderick, ASSERT: A physician-in-the-loop content-based im-
[30] J. Z. Wang, Wavelets and imaging informatics: A review of the literature,
age retrieval system for HRCT image databases, Comput. Vis. Imag.
J. Biomed. Inf., vol. 34, pp. 129141, 2001.
Understand., vol. 75, pp. 111132, 1999.
[31] A. Dempster, N. Laird, and D. Rubin, Maximum likelihood from incom-
[7] F. Schnorrenberg, C. S. Pattichis, C. N. Schizas, and K. Kyriacou,
plete data via the EM algorithm, J. Roy. Stat. Soc., Ser. B, vol. 39, no. 1,
Content-based retrieval of breast cancer biopsy slides, Technol. Health
pp. 138, 1977.
Care, vol. 8, pp. 291297, 2000.
[32] M. Stone, Cross-validation choice and assessment of statistical predic-
[8] W. Cai, D. D. Feng, and R. Fulton, Content based retrieval of dynamic
tions, J. Roy. Stat. Soc., vol. B-36, pp. 111147, 1974.
PET functional images, IEEE Trans. Inf. Technol. Biomed., vol. 4, no. 2,
[33] I. H. Witten and E. Frank, Data Mining: Practical Machine Learning
pp. 152158, Jun. 2000.
Tools and Techniques, 2nd ed. San Francisco, CA: Morgan Kaufmann,
[9] D.-M. Kwak, B.-S. Kim, O.-K. Yoon, C.-H. Park, J.-U. Won, and
K.-H. Park, Content-based ultrasound image retrieval using a coarse
[34] J. A. Hartigan, Clustering Algorithms. New York: Wiley, 1975.
to fine approach, Ann. NY Acad. Sci., vol. 980, pp. 212224, 2002.

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.

[35] D. Littau, Using low-memory approximations to cluster very large data Evangelos E. Kotsifakos was born in Athens,
sets, in Proc. 3rd SIAM Int. Conf. Data Min. (SDM), 2003, pp. 341345. Greece, in 1978. He received the Bachelors and
[36] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Masters degrees in information systems from the De-
Hillsdale, NJ: Erlbaum, 1988. partment of Informatics, Athens University of Eco-
[37] T. M. Lehmann, M. O. Guld, C. Thies, B. Plodowski, D. Keysers, nomics and Business, Athens, in 2001 and 2003, re-
B. Ott, and H. Schubert, IRMAContent-based image retrieval in med- spectively. He is currently working toward the Ph.D.
ical applications, in Proc. 14th World Congr. Med. Inf. (Medinfo), IOS, degree in the Department of Informatics, University
Amsterdam, The Netherlands, vol. 2, pp. 842848, 2004. of Piraeus (UniPi), Praeus, Greece.
[38] T. M. Lehmann, H. Schubert, D. Keysers, M. Kohnen, and B. B. Wein, He has professional experience in software en-
The IRMA code for unique classification of medical images, in Proc gineering and information technology. His current
SPIE 2003, vol. 5033, pp. 109117. research interests include pattern management, data
[39] I. Kopanakis and B. Theodoulidis, Visual data mining modelling tech- mining, and scientific databases.
niques for the visualization of mining outcomes, J. Vis. Lang. Comput.,
Spec. Issue Vis. Data Min., vol. 14, no. 6, pp. 543589, 2003.
[40] J. Davis and M. Goadrich, The relationship between precision-recall and
ROC curves, in Proc. 23rd Int. Conf. Mach. Learn. (ICML), Pittsburgh,
PA, 2006, pp. 233240.
[41] M. J. Swain and D. H. Ballard, Color indexing, Int. J. Comput. Vis., Ioannis Kopanakis received the Diploma in com-
vol. 7, no. 1, pp. 1132, Nov. 1991. puter science from the University of Crete, Crete,
[42] R. Weber, H.-J. Schek, and S. Blott, A quantitative analysis and perfor- Greece, in 1998, and the M.Sc. degree in informa-
mance study for similarity-search methods in high-dimensional spaces, tion technology and the Ph.D. degree in computation
in Proc. Very Large Data Bases Conf. (VLDB), 1998, pp. 194205. from the University of Manchester Institute of Sci-
[43] D. E. Maroulis, M. Savelonas, D. K. Iakovidis, S. A. Karkanis, and N. ence and Technology (UMIST), Manchester, U.K., in
Dimitropoulos, Variable background active contour model for computer- 1999 and 2003, respectively.
aided delineation of nodules in thyroid ultrasound images, IEEE Trans. He is currently an Assistant Professor and Head
Inf. Technol. Biomed., vol. 11, no. 5, pp. 537543, Sep. 2007. of the Department of Commerce and Marketing at
the Technological Institution of Crete, Crete. He is
the Scientific Director of the e-Business Intelligence
Laboratory at the Center for Technological Research, Crete. His current research
interests include data mining, visual data mining, and business intelligence.

Dimitris K. Iakovidis (M05) was born in Athens,

Greece, in 1973. He received the B.Sc. degree in Haralampos Karanikas was born in 1972. He re-
physics from the University of Athens, Athens, and ceived the B.Sc. degree in physics from Aristotle
the M.Sc. degree (with honors) in cybernetics and the University of Thessalonica, Thessalonica, Greece, in
Ph.D. degree in informatics from the Department of 1996, and the M.Sc. degree in computation from the
Informatics and Telecommunications, University of University of Manchester Institute of Science and
Athens, in April 2001 and 2004, respectively. Technology (UMIST), Manchester, U.K., in 1998.
For over ten years, he has been involved in the Currently, he is working toward the Ph.D. degree at
field of medical informatics and has been collaborat- the University of Manchester, Manchester.
ing with many European hospitals and health centers He is actively involved in text mining in collabora-
as an expert on biomedical systems. Currently, among tion with leading European Universities, research in-
other academic positions, he holds a Senior Researcher position at the University stitutes, and industrial partners (i.e., PARMENIDES
of Athens. He has been actively involved in more than 10 European and National IST200139023 European Communities). His current research interests include
R&D projects. He is the author or coauthor of more than 60 research papers the field of temporal text mining, which, in turn, comprises document ware-
and book chapters. His current research interests include image processing and house, automatic ontology construction, data mining, and business intelligence.
analysis, data mining, and pattern recognition with applications on biomedical
systems and bioinformatics.
Dr. Iakovidis is a Reviewer of ten international journals, including the
Yannis Theodoridis (S93M95) was born in 1967.
He received the Diploma and Ph.D. degrees in elec-
trical and computer engineering from the National
Technical University of Athens, Athens, Greece, in
1990 and 1996, respectively.
He is currently an Assistant Professor in
the Department of Informatics, University of Pi-
raeus (UniPi), Piraeus, Greece. Currently, he is a
Nikos Pelekis was born in 1975. He received the Scientist in charge for UniPi in the EC-funded
B.Sc. degree from the Department of Computer Sci- GeoPKDD project (20052008) on geographic
ence, University of Crete, Crete, Greece, in 1998, and privacy-aware knowledge discovery and delivery, and
the M.Sc. degree in information sciences engineering is also involved in several national-level projects. He is the author or coauthor
and the Ph.D. degree in moving object databases in of three monographs and over 50 articles in scientific journals (including the
2002 from the Department of Computation, Univer- Algorithmica, ACM Multimedia, and the IEEE TRANSACTIONS ON KNOWLEDGE
sity of Manchester Institute of Science and Technol- AND DATA ENGINEERING) and conferences (including the ACM SIGMOD, the
ogy (UMIST), Manchester, U.K., in 1999 and 2002, PODS, the VLDB, and the ICDE) with over 400 citations in his work. His
respectively. current research interests include spatial and spatiotemporal databases, geo-
Since March 2004, he has been a Postdoctoral graphical information management, knowledge discovery, and data mining.
Researcher in the Information Systems Laboratory, Dr. Theodoridis is a member of the Association for Computing Machinery.
Department of Informatics, University of Piraeus, Piraeus, Greece. His current He serves on the Program Committee for several major conferences in databases
research interests include spatiotemporal databases, management of location- and data mining, and is on the Editorial Board for the International Journal on
based services, data mining, and geographical information systems. Data Warehousing and Mining.

Authorized licensed use limited to: Jawaharlal Nehru Technological University. Downloaded on July 20, 2009 at 22:29 from IEEE Xplore. Restrictions apply.