Pattern Recognition Letters 19 (1998) 1037±1043

Evaluating image processing algorithms that predict regions of

interest
Claudio M. Privitera *, Lawrence W. Stark
Neurology and Telerobotics Units, University of California, 486 Minor Hall, UC Berkeley, CA 94704-2020, USA
Received 8 August 1997; received in revised form 20 April 1998

Abstract

Several bottom-up, context-free, algorithms for the detection of regions of interest in pictures were analyzed,
evaluated and compared. Our aim is to develop new criteria related to human performance for these algorithms and
perhaps to be able to design more biologically plausible perceptive machines. We introduce the statistical and com-
putational platform we have been using to compare sequences of regions of interest, both biological (eye movements)
and arti®cial (algorithms). Ó 1998 Elsevier Science B.V. All rights reserved.

Keywords: Regions of visual interest; Algorithm comparisons; Eye movements

1. Introduction example (Reed and Hans Du Buf, 1993; Haralick,

Machine vision often requires a selection
mechanism whereby only a small subset of the
input visual stimulus is analyzed in detail. This
subset may be arranged into a number of loci,
classically called regions-of-interest (ROIs). These
ROIs can be determined using image processing
algorithms that analyze, for example, spatial fre-
quency, texture conformation or other informative
values of loci of the visual stimulus. Not only are
the location of the ROIs of value, but also their
sequences in terms of ordering of selection coe-
cients can be usefully evaluated and compared.
Image processing algorithms may be classi®ed into
three principal approaches (for a survey, see for
*
Corresponding author. Tel.: 510 642 5309; fax: 510 642
7196; e-mail: claudio@milo.berkeley.edu
1979)). Firstly, structural approaches based on an
assumption that images have detectable and rec-
ognizable primitives distributed according to some
placement rules; examples are matched ®lters.
Secondly, statistical approaches based on statistical
characteristics of the texture of the picture; ex-
amples are co-occurrence matrices and entropy
functions. Thirdly, model approaches that hy-
pothesize underlying processes for generation of
local regions. These are analyzed on the basis of
speci®c parameters governing these generators:
examples are fractal descriptors.
Human perception has been explained in part
by the scanpath theory that utilizes a top-down,
internal cognitive model of what we see, to control
not only our vision, but also to drive the sequences
of rapid eye movements and ®xations, or glances,
that so eciently travel over a scene or picture of
interest (Noton and Stark, 1969; Stark and Ellis,

0167-8655/98/$ ± see front matter Ó 1998 Elsevier Science B.V. All rights reserved.
PII: S 0 1 6 7 - 8 6 5 5 ( 9 8 ) 0 0 0 7 7 - 4
1038 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043
1981; Stark and Choi, 1996). The ROIs of our
algorithms are identifying bear analogy to these
multiple glances or ®xations in human vision.
Our overall purpose is to develop ecient image
analysis algorithms, to compare their properties
with one another and with human ®xations. We
have analyzed image processing algorithms
broadly selected from the above taxonomy (Stark
and Privitera, 1997). In this paper we compare
results from six of them and we introduce the
statistical and the algorithmic background we have
been exploiting.
2. ROI detecting algorithms
The useful information content of a generic
picture can be abstracted by di€erent image pa-
rameters which in turn can be identi®ed by rele-
vant image processing algorithms. In this sense,
applying algorithms to a picture means to map
that image into di€erent domains, where for each
domain a speci®c set of parameters is extracted.
After the image has been processed, only the loci
of the local maxima from each domain are re-
tained; these maxima are then clustered in order to
yield a limited number of ROIs.
Six di€erent algorithms were studied:
S, symmetry transform,
O, di€erence in the gray-level orientation,
E, edges per unit area,
F, center-surround, as in receptive ®elds of an-
imal vision,
N, entropy,
C, Michaelson contrast.
S, symmetry, a structural approach, appears to be
a very prominent spatial relation. For each pixel
x,y of the image, we de®ne a local symmetry
magnitude S(x,y) as follows:
X ues for big di€erences in orientation between the
S…x; y† ˆ s……i1 ; j1 †; …i2 ; j2 ††; …1†
…i1 ;j1 †;…i2 ;j2 †2C…x;y†
where C…x; y† is the neighborhood of radius 7 of
point x,y de®ned along the horizontal and vertical
axis (C…x; y† ˆ …x ÿ r; y†; . . . ; …x; y†; . . . ; …x ‡ r; y†;
…x; y ÿ r†; . . . ; …x; y ‡ r†) and s……i1 ; j1 †; …i2 ; j2 †† is
de®ned by the following equation:
s……i1 ; j1 †; …i2 ; j2 ††
ˆ Gr …d……i1 ; j1 †; …i2 ; j2 †††j cos…h1 ÿ h2 †j: …2†
The ®rst factor Gr is a gaussian of ®xed vari-
ance, r ˆ 3 pixels and d(á) represents the distance
function. The second factor represents a simpli®ed
notion of symmetry: h1 and h2 correspond to the
angles of the gray-level intensity gradient of the
two pixels (i1 , j1 ) and (i2 , j2 ). The factor achieves
the maximum value when the gradients of the two
points are oriented in the same direction. The
gaussian represents a distance weight function
which introduces localization in the symmetry
evaluation. Our de®nition of symmetry was con-
sequently based on the orientation correspon-
dences of gradients around the centered point
(Reisfeld et al., 1995). Alternatively, a normaliza-
tion of the axial quadratic moment could be used
instead to compute the symmetry transform (Di
Gesu and Valenti, 1995).
O, di€erence in the gray-level orientation, a
statistical-type kernel, is analyzed in early visual
cortices. Center-surround di€erence is determined
®rst convoluting the image with four Gabor masks
of angles 0°, 45°, 90° and 135°, respectively (see
also (Niebur and Koch, 1996)). For each pixels x,
y, the scalar result of the four convolutions are
then associated with four unit vectors corre-
sponding to the four di€erent orientations. The
orientation vector o…x; y† is represented by the
vectorial sum of these four weighted unit vectors.
We de®ne the center-surround di€erence trans-
form as follows:
O…x; y† ˆ …1 ÿ o…x; y†
m…x;
 y††k  y†k;
o…x; y†kkm…x;
…3†
 y† is the average orientation vector
where m…x;
evaluated within the neighborhood of 7 ´ 7 pixels.
The ®rst factor of the equation achieves high val-
center pixel and the surroundings. The second
factor acts as a low-pass ®lter for the orientation
feature.
E, edges per unit area is determined by detecting
edges in an image, using the canny extension of the
sobel operator (Canny, 1986) and then congregating
the edges detected with a gaussian of r ˆ 3 pixels.
 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043 1039

F, a center-surround on/o€ quasi-receptive ®eld local maxima (approximately 100) down to an

mask, positive in the center and negative in the
periphery, is convoluted with the image.
P N, entropy is locally calculated as
i2G fi log fi , where fi is the frequency of the i
gray level within the 7 ´ 7 surrounding of the
center pixel and G is the local set of gray levels.
Local maxima de®ned by this factor emphasize
texture variance. The block size of 7 ´ 7 (0.3 ´ 0.3
degree block) depends on the image scaling.
C, Michaelson contrast, is most useful in iden-
tifying high contrast elements, generally consid-
ered to be an important choice feature for human
vision. Michaelson contrast is calculated as
acceptance radius at each step. The termination
Lm ÿ LM
L ‡ L
; decision to end the clustering process was set when
m M
where Lm is the mean luminance within a 7 ´ 7
surrounding of the center pixel and LM is the
overall mean luminance of the image.
3. Clustering and sequencing
The algorithm procedures above (see, for ex-
ample, Fig. 1) resulted in de®ning local maxima
widely over the picture. Then a clustering algo-
rithm was utilized to reduce these initial set of
ordered set of about nine ROIs.
The initial set of local maxima was clustered
connecting local maxima by gradually increasing
the acceptance radius for joining them. During
each step of the clustering process, all local max-
ima less than a speci®c radius apart were clustered
together. Each cluster inherited the maximum
value of its component points (local maxima): the
locus of this highest valued maximum for each
cluster then also determined the locus of that
cluster. Only that maximum point was retained; all
the other composing local maxima were deleted.
The procedure was repeated while increasing the
only nine clusters remained. Thus, the image pro-
cessing ROIs were similar in number to human eye
®xation glances looking at similar pictures.
The nine ROI domains were assigned values
depending upon the value of the highest local
maximum incorporated into that domain; alter-
natively the number of local maxima included, or
other criteria could be used for this sequencing.
The values, ordered from highest to lowest, per-
mitted us to relate the sequences of image ROIs
(Fig. 2, left panel) to sequences of human ®xa-
tions. In this way no masking was necessitated.

Fig. 1. Picture transformation. The picture transformations are a result of and provide descriptions of the action of each algorithm
(upper and lower right, E and S, respectively). Note, the untransformed picture in the larger left panel. Local maxima (circa 100) from
each transformation were then retained.
1040 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043

Each of our six IP algorithms, of course, contrib-
uted the intensity of its selected parameter to
®nding the local maxima and the values of re-
sulting clustered ROI domains.
If we had used only image processing algo-
rithms and not the clustering procedure, we could
have selected nine local maxima directly and de-
®ned them to be the ROI domains. Those selected
ROIs, however, might be much more closely
spaced. Thus the clustering algorithm was actually
an eccentricity-weighting algorithm, where even
lower local maxima that were eccentrically located
could be selected to form a domain.
4. Similarity metrics
4.1. Similarity of ROI loci, Sp
Comparison of ®nal clusters of ROIs begun
with taking two sets of ROIs, generated by two
di€erent algorithms and combining these two sets.
Then this double set of ROIs was clustered using a
distance measure taken from a k-means pre-eval-
uation. This evaluation determined a region for
calling coincident any ROIs that were closer than
this distance and non-coincident for ROIs that were
further apart than this distance; the distance was
about two degrees, and similar in size to human
foveal spans. This ®nal selection of joined-ROIs
(Fig. 2) then enabled a distance-similarity metric,
Sp , to determine how close were the ROIs identi-
®ed by two algorithms. The individual source of
the elements, that is, the original ROIs, used in
these ®nal interactive steps were preserved to il-
lustrate the procedure (as circles and squares in
Fig. 2, right panel).
4.2. Similarity of sequences, Ss
As mentioned above, ROIs were ordered by the
value assigned by the image processing algorithm
conceptually similar in some ways to the temporal
ordering of human eye ®xations in a scanpath.
Two di€erent ROI sequences, squares and circles,
Fig. 2, left upper and lower panels, corresponded
to two sets of ROIs from two of the IP algorithms.
Then, joined-ROIs were ®nally ordered into strings
of ordered points. In Fig. 2, we have for example:
stringS ˆ abcfeffgdc and stringF ˆ afbffdcdf . The
string editing similarity index Ss was de®ned by an
optimization algorithm (Stark and Choi, 1996)
with unit cost assigned to the three di€erent op-
erations deletion, insertion and shifting.
Di€erent ROI sequences corresponding to two
sets of ROIs from two of the IP algorithms are

Fig. 2. Clustering algorithm. After the picture transformation, a clustering algorithm is used to reduce the high valued maxima (circa
100) to few (circa 10). These 10 were then ordered by value and the constructed ordered sequences (upper and lower left panel) were
then connected by vectors in analogy to human sequences of ®xations. The two set of ROIs are ®nally combined (right panel) into a
selected number of joined-ROIs which are used to de®ne distance measures between the two algorithms.
 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043
often similar in location, Sp , indicating that two IP
algorithms have chosen similar ROIs. However,
they are usually dissimilar in sequence, Ss , indi-
cating that the two IP algorithms have assigned
orders of values or sequences of values that were
dissimilar.
4.3. Markov matrices, St
We can also create, based on the above men-
tioned joined-ROIs, transition markov matrices,
[M1]s (Fig. 3, S left matrix and F right matrix).
The transition similarity index, St , is based upon
cross-correlation between the coecients of the
[M1]s.
Thus, all of our comparisons between pairs of
algorithms yielded three di€erent indices of simi-
larity. For the example illustrated above we have
Sp ˆ 1, Ss ˆ 0.34 and St ˆ 0.09.
4.4. Coherence of algorithms
The similarity measures, Sp , Ss and St de®ned
above, tell us how closely the ROIs identi®ed by
two algorithms resemble each other in locus, Sp , in
sequence, Ss , and in transition from one ROI to
the next, St . These similarities have been arranged
into three ordered matrices (Fig. 4).
For Sp , (Fig. 4, upper matrix) enclosed within
the dotted box are six similarity values docu-
menting the high Sp similarity for four selected
algorithms E, O, S and F. The average value of
similarity indices was L ˆ 0.62 for the selected
algorithms (right column, upper table). This can
be compared to the average value, L ˆ 0.4, of
similarity indices for the entire group of six algo-
rithms, including two disparate algorithms, C and
Fig. 3. Transition matrices. Transition matrices derived during the comparisons of the two algorithms shown in Fig. 2.
1041
N. Our collection of algorithms could thus be
sorted for similarities or for di€erences in gener-
ating ROIs. This is likely to be of value in selecting
algorithms for di€erent tasks.
The global Sp , G ˆ 0.23, represented the aver-
aged Sp for all algorithms and for the two di€erent
pictures used; some of these might be related, but
most clearly have no reason to be so related. This
G value could be considered a bottom anchor for
Sp ; a further bottom anchor was Ra, the random
Sp , calculated for coincidence among all loci of the
appropriate size.
The middle and lower left ordered matrices of
similarity coecients show little coherence even
among the four selected algorithms. This provides
quantitative support for our conclusion that we
can select independent algorithms that yet select
similar ROI loci. These communally identi®ed loci
have greater validity than those chosen by any one
algorithm. However, the sequences were dicult
to recapture from one algorithm to another and
therefore our selected group of algorithms yielded
no more precise estimate of sequential coherence.
5. Discussion
Two advantages of our methodology are task
de®nition and metrics for similarities. We have set
precised task for the image processing algorithms
studied ± to identify loci of ROIs and, if possible,
the sequences or strings of loci of ROIs. This is a
more de®ned task than just processing images for
human subjective judgement ± undeniably human
vision can compensate for many aspects of loss of
information. The quantitative measure of similar-
ity of loci of ROIs, Sp , enabled the selection of
1042 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043

Fig. 4. Similarity measures among algorithms. Cross-comparison values (left column) of the six algorithms, E,O,S,F,G,N, for the three
indices, Sp , Ss and St . Coecients range from 0 to 1. Averaged coecients are printed in adjoining table (right column). For Sp , in the
upper matrix (enclosed within the dotted box), are six similarity values documenting the high Sp similarity for four selected algorithms.
Note, that for the middle and lower matrices and tables, the selected group of algorithms and indeed, all the algorithms together, did
not show commonality of sequences, strings and transitions (see text).

algorithms that function for our task in a similar
fashion, even though they may have processed
pictures in quite di€erent manners. The fact that
we found only low similarities of sequences and
transitions, Ss and St , indicates that selecting for
commonality of algorithms in forming sequences is
not yet possible.
A wide selection of algorithms provided an
opportunity to study quantitatively their di€eren-
ces and similarities in terms of the precise task.
This is of great interest as it indicates the general
nature of an image and how it is processed either
by algorithms or by humans. We are concerned
that we might need to provide weighting coe-
cients for the di€erent algorithms in order to op-
timize the predicting capabilities of the ensemble
(Stark and Privitera, 1997).
We are in the process of comparing our algo-
rithms with eye movement ®xations (Privitera and
Stark, 1998, in preparation; for preliminary results
see Stark and Privitera, 1997). From this com-
parison we may learn something more about top-
down, context-dependent, visual processing and
understand the structure and the nature of internal
representations upon which humans perception is
based.
In other studies, beyond the scope of the pres-
ent paper, we have applied our methodology to a
much richer collection of pictures, scenes and
works of art. These range widely from natural and
constructed landscapes and cityscapes, to groups
of persons and animals and objects, to single
portraits then paintings (among them the two
paintings for the data in this Pattern Recognition
Letters) and still lifes (Privitera and Stark, 1998, in
preparation). In Fig. 4, the table coecients fall
into typical ranges that we have obtained with a
much larger collection of algorithms and a much
wider set of images from our extended corpus of
pictures. However, we wish to direct the readers'
 C.M. Privitera, L.W. Stark / Pattern Recognition Letters 19 (1998) 1037±1043
attention to the main thrust of this paper ± the
algorithms used and the methods employed to
compare them.
Many open problems remain: how to collect
and classify and relate even a wider variety of
image processing algorithms? What modi®cations
of a picture leave the picture information relatively
constant and what modi®cations strongly deform
the picture information?
Acknowledgements
We thank our sponsors for partial support: the
NASA-Ames Research Center (Drs. Stephen Ellis
and Robert Welch, technical monitors of di€erent
Cooperative Agreements) and the Fujita Research
and Neuroptics Corporations, whose presidents,
Ken Kawamura and Kamran Siminou, respec-
tively, allowed us to interpret our guidelines quite
loosely. Our colleagues in the laboratory, Michela
Azzariti, Ted Blackmon, Yeuk Fai Ho, Veit
Hagenmeyer, Yong Yu have always been helpful,
not the least in keeping a motley array of com-
puters functioning smoothly. Other scienti®c col-
leagues, past and present have been generous in
their advice ± Stephen Ellis and Charles Neveu at
NASA, Irwin Sobel at Hewlett-Packard, and at
UCB, Jerry Feldman, CS and David Brillinger,
Statistics; also the anonymous referees of Pattern
Recognition Letters.
1043
References
Canny, J., 1986. A computational approach to edge detection.
IEEE Transactions on Pattern Analysis and Machine
Intelligence 8 (6), 679±698.
Di Ges u, V., Valenti, C., 1995. The discrete symmetry trans-
form in computer vision. Technical Report 011-95, Labo-
ratory for Computer Science (DMA), University of
Palermo.
Haralick, R.M., 1979. Statistical and structural approaches to
texture. Proceedings of IEEE 67, 786±804.
Niebur, E., Koch, C., 1996. Control of selective visual
attention: Modeling the ``where'' pathway. In: Touretzky,
D.S., Mozer, M.C., Hasselmo, M.E. (Eds.), Advances in
Neural Information Processing Systems, Vol. 8. MIT Press,
Cambridge, MA, pp. 802±808.
Noton, D., Stark, L., 1969. Eye movements and visual
perception. IEEE Transactions on Man±Machine Systems
1, 1±9.
Reed, R.T., Hans Du Buf, J.M., 1993. A review of recent
texture segmentation and feature extraction techniques.
CVGIP: Image Processing 57 (3), 359±372.
Reisfeld, D., Wolfson, H., Yeshurun, Y., 1995. Context-free
attentional operators: The generalized symmetry transform.
International Journal of Computer Vision 14, 119±130.
Stark, L., Choi, Y., 1996. Experimental metaphysics: The
scanpath as an epistemological mechanism. In: Zangemeis-
ter, W.H., Stiehl, H.S., Freksa, C., Visual Attention and
Cognition. Elsevier, Amsterdam, pp. 3±69.
Stark, L., Ellis, S., 1981. Scanpaths revisited: Cognitive models
direct active looking. In: Fisher, M., Senders (Eds.), Eye
Movements, Cognitive and Visual Perception. Erlbaum, NJ,
pp. 193±226.
Stark, L., Privitera, C., 1997. Top-down and bottom-up image
processing. In: Proceedings of IEEE International Confer-
ence on Neural Networks, Vol. 4. Houston, TX, 9±12 June,
pp. 2294±2299.