416 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO.
4, DECEMBER 2005
A Vision-Based Approach to Collision
Prediction at Traffic Intersections
Stefan Atev, Hemanth Arumugam, Osama Masoud, Ravi Janardan, Senior Member, IEEE, and
Nikolaos P. Papanikolopoulos, Senior Member, IEEE
AbstractMonitoring traffic intersections in real time and pre-
dicting possible collisions is an important first step towards build-
ing an early collision-warning system. We present a vision-based
system addressing this problem and describe the practical adap-
tations necessary to achieve real-time performance. Innovative
low-overhead collision-prediction algorithms (such as the one
using the time-as-axis paradigm) are presented. The proposed
system was able to perform successfully in real time on videos
of quarter-video graphics array (VGA) (320 240) resolution
under various weather conditions. The errors in target position
and dimension estimates in a test video sequence are quantified
and several experimental results are presented.
Index TermsCollision prediction, machine vision, real-time
systems, tracking, traffic control (transportation).
I. I NTRODUCTION
I NTELLIGENT transportation systems (ITS) are built from
technologies such as sensing, control, engineering, and com-
puting to solve transportation-related problems. Such systems
have evolved from primarily addressing traffic-control prob-
lems to current applications that involve better lane design
for reduced congestion, developing systems that help reduce
traffic-related fatalities, and better vehicle- and pedestrian-
monitoring systems to study flow patterns in various traffic
scenarios. Such systems perform well in steady-state traffic
situations like those on a freeway, but perform badly when
applied to the highly unsteady flow of a busy traffic intersection.
Traffic-monitoring applications regularly make use of
computer-vision principles to model and analyze traffic scenar-
ios. For example, in [11], a contour tracker was used to model
each moving vehicle, while the problem of tracking vehicles
under challenging conditions in a freeway is addressed by [6]
using a feature-based tracking system. Kato et al. [10] have
used a hidden Markov model (HMM) for traffic monitoring.
Tracking technology has evolved to a point that techniques
such as particle filtering [1], [17], [19], combination of multiple
visual cues [13], [23], [25], data association [4], [22], and
occlusion handling [14] present new opportunities for research
in the area.
Manuscript received February 20, 2004; revised June 23, 2005. This work
was supported by the Minnesota Department of Transportation, the Intelligent
Transportation Systems (ITS) Institute at the University of Minnesota, and
the National Science Foundation through Grant CMS-0127893. The Associate
Editor for this paper was M. Kuwahara.
The authors are with the Department of Computer Science and Engineering,
University of MinnesotaTwin Cities, Minneapolis, MN 55455 USA
(e-mail: atev@cs.umn.edu; hemanth@cs.umn.edu; masoud@cs.umn.edu;
janardan@cs.umn.edu; npapas@cs.umn.edu).
Digital Object Identifier 10.1109/TITS.2005.858786
1524-9050/$20.00 2005 IEEE
Studies have shown that collisions between vehicles at traffic
intersections account for nearly a third of all reported crashes
in the United States [8], [18], [21]. This has led to considerable
interest at the federal level in developing an intelligent low-cost
system that can detect and prevent potential collisions in real
time. This paper presents the various components of a vision-
based system that monitors a traffic intersection and uses the
tracking results to predict collisions over a short time interval
extending into the future. Our goal is to establish the feasibility
of this approach; the specific way in which these predictions
can be used to prevent possible traffic accidents, however, is
not addressed at this time. As far as we know, this is the first
time that computer vision in conjunction with computational
geometry tries to supply a solution to this complex problem.
The rest of this paper is organized as follows: We initially
discuss the problem that our system addresses and explain
our particular design choices. The techniques used to predict
collisions and measure vehicle dimensions are then presented.
Finally, the paper presents and evaluates our results on test
video sequences.
II. P ROBLEM S PECIFICATION AND S YSTEM O VERVIEW
The purpose of our system is to monitor a traffic intersection
using a live video feed from one or more cameras over extended
periods of time. Possible collisions between tracked vehicles
must be detected before they occur so that a timely warning
may be issued. It is critical that the system runs in real time and
that it adapts to changes in the environment for the duration
of the monitoring. In addition, since the system must predict
collisions, we need to produce accurate estimates of both the
positions and dimensions of vehicles, preferably in real-world
units. Initially, we looked at deriving the bounding rectangles
(for the overall framework, please see Fig. 1), but we eventually
moved to more complex representations (bounding boxes) due
to vehicle-size-estimation inaccuracies associated with bound-
ing rectangles.
Appearance-based trackers, such as the kernel-based mean-
shift tracker presented in [7], are good at following moving ob-
jects, even in the presence of partial occlusions. Unfortunately,
such methods usually require a target model to be manually
specified and cannot delineate objects accurately. Since we
cannot provide target models for the various vehicles that can
enter a traffic scene in advance, and because we need the
outlines of vehicles to measure their dimensions, we chose to
base our target models on connected regions of foreground
pixels. Connected regions (blobs) extracted from a foreground
ATEV et al.: A VISION-BASED APPROACH TO COLLISION PREDICTION AT TRAFFIC INTERSECTIONS
Fig. 1. Initial approach used based on bounding rectangles.
mask provide us with good object outlines, allow for automatic
target identification, and are well-suited for real-time systems.
In order to classify foreground pixels, we need a background
model of the observed scene. Due to the gradual changes in
scene appearance over extended periods of time, we cannot use
a static background model. Instead, we use an adaptive back-
ground model based on the mixtures-of-Gaussians segmenta-
tion method in [24]. The resulting background/foreground clas-
sifier adapts well to gradual changes in the monitored outdoor
environment and allows for the detection of targets even if they
are not movinga common occurrence at traffic intersections.
Occasionally, the background extraction can fail, either because
of sudden illumination changes in the scene caused by passing
clouds or a cameras gain control circuitry, or due to minor
camera shakes resulting from road vibrations or wind load.
We have developed efficient methods for compensating for
sudden illumination changes and camera shakes. We also have
devised a fast implementation of the method presented in [5]
for cleaning up the foreground masks.
Vehicles that move throughout the scene will sometimes
occlude other moving objects, or be themselves occluded by
static objects such as road-sign poles, traffic lights, etc. Such
occlusions cause blobs to merge, split into smaller regions, or
to disappear completely. These interactions between connected
regions can either cause a single target to be visible as more
than one blob in the foreground, or cause several targets to
be represented as a single blob. The problem can be allevi-
ated by making use of multiple cameras to observe the same
intersection; proper camera placement can ensure that vehicles
417
are rarely occluded in the views of all cameras. However, even
when multiple cameras are used, it is advantageous to have
some means of handling occlusions in a single view. For that
reason, we represent targets as sets of regions and introduce
a second-level tracker that is capable of handling blob merges
and splits. The second-level tracker also makes use of camera-
calibration data in order to estimate the position and velocity of
vehicles in world coordinates. The calibration method we use is
presented in [16] and allows us to accurately map points from
the image planes of all cameras to positions on a single world-
coordinate ground plane.
It is common practice to use the centroids of connected
regions to represent a targets position. Such an approach is sub-
optimal, because the position of a centroid relative to the ground
plane depends on the size and orientation of vehicles and also
on the particular camera placement. The last fact complicates
multicamera tracking since the centroids tracked in different
camera views do not correspond to the same real-world point.
We introduce a method that can identify the centers of vehicular
bases on the ground plane given the outlines of the vehicles
and the camera-calibration data. The base centers identified by
our method correspond to the same real-world point, which
allows for the sequential incorporation of the measurements in
a targets state vector. The method also produces estimates of
the width, length, and height of vehicles that are critical for the
collision-prediction system.
The last component of our system is the collision-prediction
module. Given the visual measurements of all targets in the
scene, the module reports all target pairs that will collide within
418 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 4, DECEMBER 2005
Fig. 2. Low-level vision-system components and data flow for a single frame. Lighter lines indicate the use of data from the processing of the previous frame.
Fig. 3. Collision-prediction-system components and data flow. Size and position estimates are provided to the ground-plane tracking component of the
tracking module.
a time interval of length L in the immediate future, assuming
their velocities stay constant. Optionally, the actual time of
impact for each target pair that collides within the specified
L time units can be reported. We present several methods for
predicting pairwise collisions. The most effective method is
based on the idea of extruding the two-dimensional (2-D)
vehicular bases along a time axis to obtain three-dimensional
polytopes that can be tested for overlap efficiently by making
use of the Separating Axis Theorem [9].
III. L OW -L EVEL V ISION S YSTEM
In this section, we summarize the methods used. Initially, we
tried to base our methods on bounding rectangles. It worked
in an acceptable fashion, but we had to use the assumption
of known vehicle width, which really distorted the vehicle-
size estimation. We then developed more advanced methods for
background-model maintenance, illumination filtering, camera-
shake compensation, and noise removal in the foreground mask.
Fig. 2 shows the components of the low-level vision system and
the data interactions between them. More information about
these methods can be found in [2].
IV. C OLLISION P REDICTION
Initially, the collision-prediction module was responsible for
measuring the bounding rectangles. We eventually evolved the
module to measure the base center and dimensions of a target
given its outline, and to predict potential collisions between
vehicles (Fig. 3). The specific way in which we use object
outlines and calibration data for the first task is presented,
followed by a description of the collision-detection test.
A. Dimension and Position Measurement
To measure a targets position and dimensions, we fit a three-
dimensional box to its outline. The outline is the union of
all contour points of regions in the targets blob collection.
Assuming for the moment that we know the boxs position,
dimension, and orientation, we can extend its edges into lines.
Those lines intersect at three distinct vanishing points in the
image plane (we also handle lines parallel to the image plane
as a special case). The importance of this result is that we can
reverse the processstarting with the three vanishing points
and the objects outline, we can find some of the edges of
the box. The vanishing points in a given direction d R3 are
also determined. The relevant directions are found by making
two assumptions: that the targets orientation coincides with
its direction of motion and that the bases of the targets are
parallel to the ground plane. The first assumption determines
the vanishing points wx and wy in directions parallel to the
ground plane: the direction of motion dx and perpendicular
direction dy . The second assumption fixes the third vanishing
point wz in the direction dz = [0, 0, 1]T .
For each vanishing point, we find the two tangent lines to
the convex hull of the targets outline. If a vanishing point is
inside the outlines hull, it will be ignored in subsequent steps,
since all box edges that vanish to that point would be contained
in the outline and are thus, irrecoverable. There is no need to
compute the convex hull, since its vertices are a subset of the
ATEV et al.: A VISION-BASED APPROACH TO COLLISION PREDICTION AT TRAFFIC INTERSECTIONS
Fig. 4. Tangent lines to a targets outline from the wx and wy vanishing
points. The directions dx , dy , and dz are indicated as well.
outlines vertices. The tangent lines to the hull are the lines
through a vanishing point and an outline point that form the
least and greatest angles relative to a fixed axis. Fig. 4 shows a
sample target, the relevant vanishing points, and the directions
that determine them, as well as the tangent lines identified by
our algorithm.
The endpoints of the bounding box edges are found by
intersecting the tangent lines. The three cases that we need
to consider are shown in Fig. 5. In the last case, we cannot
determine the length of two of the edges. This case does not
occur if the camera is placed sufficiently high above the road
plane, however, even with proper camera placement, some of
the dimensions may be unavailable due to partial visibility.1
Partially visible edges will not be used to produce dimension
measurements, but will still be used for position measurements,
since the tracker requires them for every frame.
The center of a targets base is obtained by taking the
midpoint of a line segment connecting two edge endpoints
on opposite corners of the box. The real-world lengths of
edges in the ground plane can be identified by using endpoints.
The length of vertical edges can be determined by employing
the techniques described in [16]. The recovered heights are
necessary in order to determine the lengths of edges for which
both endpoints are vertically displaced from the ground plane.
B. Collision Prediction
Initially, we experimented with bounding rectangles. Since
the position, orientation and length of each vehicle are com-
puted at every time step (vehicle width is assumed known),
the input to the collision-detection module is essentially a set
of oriented rectangles in a plane. The general problem to be
solved by a collision-detection module can then be stated as
follows: Given the position, orientation, and size, at each time
step, of n oriented rectangles in 2-D, find all possible pairs of
rectangles that are within a distance in the current and future
time steps. Finding the minimum distance between n2 pairs
of rectangles is a harder problem to solve than to just check if
two rectangles are intersecting. Since the value , the maximum
allowed intervehicular distance, is fixed, each rectangle can be
increased in size by a value of /2 and the problem can be
1 We consider an edge partially visible if one of its endpoints is within a small
distance (3 or 4 pixels) of the image borders.
419
converted to a simpler collision test between pairs of rectangles.
It should be noted that the Minskowski Sum of the rectangle
with a circle of diameter /2 in 2-D will give a more accurate
representation of the resulting shape, but the resulting geometry
will no longer be a rectangle and would greatly affect the
choice of the collision-detection technique employed. To keep
the geometry simple, a rectangular approximation of the final
shape is initially used to represent the resulting shape.
The general problem to be solved by the collision-detection
module can then be restated as follows: Given the position,
orientation, and size, at each time step, of n oriented rectangles
in 2-D, find all possible pairs of rectangles that intersect in the
current and future time steps. We then tried various practical
computational techniques to detect collisions between rectan-
gles in real time. The problem of detecting collisions in the
current frame is solved first.
A direct approach to computing all possible pairs of inter-
secting rectangles is to test each pair of rectangles for such an
intersection. Such a brute-force approach will take O(n2 ) time
to compute all such pairs. When the number of vehicles in any
time instant is small (say, < 10), such a brute-force method
should be sufficient and there should be no need to use more
sophisticated algorithms. The actual test performed to detect
if two oriented rectangles are intersecting each other is to use
a general polygon-clipping algorithm. This algorithm proceeds
by using one of the two rectangles as a clipper and the other
as the target rectangle. The edges of the clipper rectangle are
assumed to be oriented in a clockwise direction. Each directed
edge of the clipper rectangle is used in turn to cut the target
rectangle into two pieces, retaining at each step the piece that
is to the right of the clipping edge until all the clipper edges
are exhausted. A collision exists if the target rectangle has a
nonzero area left at the end of the clipping process. The current
implementation of this brute-force algorithm in the collision-
detection module makes use of the polygon-clipping algorithm
found in the vision-something-libraries (VXL).
The brute-force method described above makes an implicit
assumption that each rectangle is equally likely to intersect with
every other rectangle. But in the problem domain under consid-
eration, each rectangle represents an actual vehicle in the real
world, and hence, moves in some predictable pattern. In order to
reduce the number of polygon-clipping computations, the entire
plane on which the vehicles move is divided into a grid of m
polygonal cells. The preprocessing step consists of assigning
each of the input rectangles (vehicles) to one or more of the m
polygonal cells, depending on whether they are fully inside or
across one or more of these cells. Since there are m cells and n
rectangles, the assignment will require a worst case overhead of
O(mn) polygonrectangle intersection computations. After the
preprocessing, the actual rectanglerectangle intersections are
computed using the brute-force approach described earlier, but
now doing the tests only among the rectangles within each cells.
The expected processing time to compute all possible vehicle
collisions is then O((n/m)2 m) = O(n2 /m). Thus, the total
processing time for the current time step is O(nm + n2 /m).
Hence, the design of the grid-cell size and shape becomes
important to achieve better running time than a brute-force
method. The optimization goal for such a design problem
420 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 4, DECEMBER 2005

Fig. 5. Three major cases for the tangent-line intersections. Thicker line segments indicate edges whose length can be retrieved. The location of the wy vanishing
point and the direction of motion dx are indicated. In (a) and (c), all three dimensions can be recovered, while in (b), only two are recoverable.

Fig. 6. Rectangles extruded in time. (a) A w l rectangle at position p0 , moving for L time units with velocity v (reaching point pL ); and (b) two overlapping
parallelepipeds with labeled edges; the vector c connects the centroids of the polytopes.

should be that the vehicles get distributed uniformly across the
cells. This depends on the knowledge of vehicle flow pattern,
density, and their variation with respect to time of the day. One
simple way out is to allow the end user who has knowledge
of traffic-flow patterns to draw these grids for a specific traffic
intersection.
A natural extension of the above fixed grid-based approach is
to make the grid adaptive to the conditions of vehicular traffic
flow. The goal is to maintain an invariant that the maximum
occupancy of any grid cell is given by a predetermined num-
ber p. The grid cell is recursively divided into four subcells
whenever the above invariant is violated. Vehicles can be added
to a grid cell, so long as its occupancy remains less than p.
When the (p + 1)st vehicle is added to a grid cell, that cell is
recursively divided into four subcells and all the p + 1 vehicles
are reassigned to the new child cells until the invariant is
satisfied.
The above design can be represented as a Quad Tree with
the root of the tree representing a bounding rectangular plane
that will contain any given rectangle (vehicle). Each leaf of
the tree will have a maximum of p vehicles. In addition to
the data required to represent a tree, each node v of the Quad
Tree will hold the rectangular quadrant R(v) that it represents.
Each leaf of the Quad Tree will additionally hold pointers to
the list of rectangles it contains (maximum of p). Finally at
this stage, we experimented with the popular interval-based
approach. Intersection of convex polygons is one of the well-
studied problems in computational geometry. The intersection
of oriented rectangles is a special case of the above problem. It
is shown that the intersection of simple polygons is linear-time
transformable to the line-segment intersection testing problem.
To find pairs of intersecting rectangles in a plane, the problem
is converted into a line-segment overlap problem.
All of the rectangle-intersection algorithms described above
are used to compute collisions at a particular time instance.
Collision prediction however requires the computation of all
vehicle pairs that could possibly collide within the next f time
steps, provided the predicted vehicle positions are available.
While these brute-force or space-based algorithms would suf-
fice for computing collisions in the current time frame, they do
not scale readily for solving the following problem: Given the
position, orientation, and size, at each time step, of n oriented
rectangles in 2-D, find all possible pairs of rectangles that could
intersect in f future time steps. In the existing implementation,
this is done by using the predicted positions through a Kalman
filter and repeating the collision-detection process once for each
of the f time steps, thus increasing the running time by a factor
of f . Algorithms like the interval-based approach can be used
instead to exploit the spatial proximity of a vehicles predicted
position across the time frame to update the data structure incre-
mentally, instead of recomputing the entire data structure again.
We then moved to more complex representations and again
tried to solve the collision problem. The number of vehicles
that can be present at a traffic intersection is fairly limited,
so the lower time complexity of advanced collision-detection
methods, such as those outlined in [12], does not translate to
better run-time performance because of the overhead imposed
by preprocessing steps and the use of advanced data structures.
Thus, we opted to test all vehicle pairs in a scene for possible
collision over a time interval and focused on maximizing the
performance of the individual tests.
Our method is based on the idea of extruding the bases
of vehicles along a time axis. The extrusion of a rectangle
moving from a point p0 to the point pL over L time units is
a parallelepiped like the one shown in Fig. 6(a). A collision oc-
curs if and only if the parallelepipeds representing two vehicles
overlap. Fig. 6(b) illustrates this and introduces our notation for
the following discussion.
Two convex polytopes are disjoint if there exists an axis
on which their projections are disjoint. The Separating Axis
ATEV et al.: A VISION-BASED APPROACH TO COLLISION PREDICTION AT TRAFFIC INTERSECTIONS
Fig. 7. Example illustrating (1). In this particular case, the objects are separated by the axis.
TABLE I
SIMILAR TERMS IN THE SEPARATING-AXIS TEST
All possible values for individual terms in (1). Identical letters
indicate identical values for the absolute value of the dot prod-
uct between the vectors indicated in the first column and the
vectors indicated in the first row. For example, from the table,
it can be seen that |(u1 u3 ) u2 | = |(u2 v3 ) u1 | (both
represented by an A in the appropriate table cells).
Theorem [9] establishes that it is sufficient to test for overlaps
on axes that are perpendicular to a face from either polytope
or perpendicular to edges from both polytopes. The 15 axes we
need to consider are defined by the pairwise cross products of
the vectors u1 , u2 , u3 , v1 , v2 , and v3 in Fig. 6(b). The six
pairwise cross products of the vectors u1 , u2 , v1 , and v2 are
parallel to the time axis, which eliminates them from further
consideration, since objects always overlap on the time axis. An
axis a chosen from one of the remaining nine axes separates the
two polytopes if the following condition holds
2|c a| > |u1 a| + |u2 a| + |u3 a|
+ |v1 a| + |v2 a| + |v3 a|. (1)
The vector c in (1) connects the centers of the two polytopes,
as shown in Fig. 6(b). The constant factor of 2 in front of the
left-hand side of (1) accounts for the fact that each polytopes
projection on the axis a is symmetric around the projection of
its center. Fig. 7 illustrates (1) for an equivalent 2-D example.
As soon as an axis is found to separate the two polytopes, we
can be sure that no collision between the respective vehicles is
possible.
It also turns out that terms in (1) that involve the same three
vectors produce the same values, as shown in Table I. This lets
us eliminate four more axes from consideration and allows us
to reuse some of the intermediate results (the terms indicated
by individual letters in Table I) of one axis-overlap test in
421
TABLE II
COMPUTING THE NUMBER OF NEAR-MISSES IN A SPARSE TRAFFIC
INTERSECTION MONITORING 60 VEHICLES OVER 4000 FRAMES
FOR D IFFERENT V ALUES OF I NTER V EHICULAR D ISTANCE d
subsequent tests. Our final implementation of the test requires
a total of 44 multiplications, 35 additions/subtractions, and
five comparisons. The resulting method for detecting possible
collisions over a time interval has an almost negligible impact
on our systems running time and requires no extra storage or
complicated data structures.
V. R ESULTS
Initially, we experimented with bounding rectangles. The
collision-detection module contains an implementation of all
the aforementioned algorithms, with the option of choosing one
at runtime. The choice of the algorithm primarily depends on
the number of vehicles n at any given instant. For n < 20, a
brute-force approach would suffice, and when the number of
vehicles is less, the module automatically chooses the brute-
force algorithm. When the number of vehicles becomes greater
than 20, the space-based approach is employed. The results for
such a sparse traffic intersection are shown in Table II.
We then implemented the more complex presentations in-
volving bounding boxes. Our system performs in real time
on current-generation hardware.2 The output of the system is
shown in Fig. 8target bounding boxes are overlaid on the
input video to allow for the visual inspection of the tracking
results. The foreground mask for each frame is also shown, in
order to highlight the behavior of the system with regard to
dynamic and static occlusions. No collision warning is issued
for the two occluding vehicles, since their bounding boxes are
separated.
We manually counted the number of cars in 86 regularly
spaced video frames. Of the 273 vehicles found, 231 (85%)
were correctly identified by our system, 19 (7%) were not
2 34.6 ft/s on a 2.66-GHz Pentium IV system.
422 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 4, DECEMBER 2005
Fig. 8. Tracking sequence. Two regions are merged in the second and
third frame (dynamic occlusion). Another region is split in the third frame
(static occlusion). Video of the output is available at http://www.cs.umn.edu/
research/airvl/its/.
TABLE III
POSITION - AND DIMENSION-MEASUREMENT ACCURACY
Root-mean-squared errors are reported in centimeters, while average
relative errors are reported as percentages. (a) Results from all 15
vehicles; (b) Excluding a turning vehicle whose long shadow caused
severe errors in the width estimate (18 of 292 samples removed).
identified at all, and 23 (8%) were identified, but merged with
other targets in the scene. We then manually fit bounding boxes
around 15 different vehicles for several consecutive frames,
which gave us 292 individual boxes. The root-mean-squared
error in the estimated quantities as well as the average relative
error for the three reported vehicle dimensions are tabulated in
Table III.
The indicated accuracy of our system is significant, given
the resolution at which the measurements were taken, and
considering that the reprojection error of our camera calibration
was as large as 40 cm in the central regions of the image.
The correctness of the collision-prediction algorithm was
evaluated using data from real intersections (at Union and
Washington in Minneapolis, MN, and at Rice and University
in St. Paul, MN). Since we had no real crashes to work on,
we focused on the number of near misses. The bounding-
boxes method performed significantly better than the methods
where the bounding rectangles were used due to the issues with
occlusions and the assumption of known width, which really
distorts the near misses due to underestimation of the actual
vehicle dimensions.
VI. C ONCLUSION
A. Summary
We presented a vision-based system for monitoring traffic
intersections that issues warnings about imminent collisions.
The sensors used comprise of one or more fixed video cameras,
calibrated in advance of the systems operation. Instabilities in
the input video due to camera shaking, sudden illumination
changes, and weather conditions (e.g., light rain) are handled
automatically by the low-level vision components of the sys-
tem. Accurate velocity, position, and size information in real-
world units were obtained for each vehicle in the scene by
the high-level vision components of the system. The collision-
prediction module makes use of this data to predict vehicle
trajectories and report possible collisions. The data generated
by the system is also suitable for vehicle-classification purposes
and for the categorization of the severity of collisions. The
system is robust to temporary static and dynamic occlusions,
and is capable of handling the stop-and-go situations that occur
in the setting of a traffic intersection. The proposed algorithms
were implemented on a general-purpose computer system with-
out specialized vision-processing hardware. We were able to
achieve real-time performance (greater than 30-Hz sampling
rate) on videos of sufficient resolution.
B. Future Work
The presence of shadows cast from moving objects poses
an interesting challenge for future research. Such shadows
exacerbate occlusion problems and degrade the quality of the
target outlines recovered by connected-region extraction. The
detection and elimination of shadows is an area of active re-
search, but it remains to be seen if a method capable of meeting
the stringent real-time performance and quality requirements of
our system will be found.
Another area of interest for further research is the exten-
sion of the camera-calibration techniques we use to allow for
tracking of objects on nonplanar road surfaces. This amounts
to developing supervised algorithms for the recovery of road
surfaces using common geometric primitives visible in a still
image of a traffic intersection.
Finally, we would like to investigate the benefit of using
more descriptive vehicle-motion models in order to improve the
position and velocity estimates of tracked vehicles. The noise
in image-space measurements precludes the use of high-order
filters, but a mixture of several high-bias filters combined using
a switching Kalman Filter may further improve the quality of
tracking results.
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers for
their valuable comments.
R EFERENCES
[1] M. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A tutorial on par-
ticle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE
Trans. Signal Process., vol. 50, no. 2, pp. 174188, Feb. 2002.
[2] S. Atev, O. Masoud, R. Janardan, and N. Papanikolopoulos, A colli-
sion prediction system for traffic intersections, in Proc. IEEE/RSJ Conf.
Intelligent Robots and Systems (IROS), Edmonton, AB, Canada, 2005.
[3] S. Atev, O. Masoud, and N. Papanikolopoulos, Practical mixtures of
Gaussians with brightness monitoring, in Proc. IEEE Conf. Intelligent
Transportation Systems (ITSC), Washington, DC, 2004, pp. 423428.
ATEV et al.: A VISION-BASED APPROACH TO COLLISION PREDICTION AT TRAFFIC INTERSECTIONS
[4] Y. Bar-Shalom and T. E. Fortmann, Tracking and Data Association. New
York: Academic, 1987.
[5] A. Bevilacqua, Effective object segmentation in a traffic monitoring
application, in Proc. 3rd Int. Association Pattern Recognition (IAPR)
Indian Conf. Computer Vision, Graphics and Image Processing, Ahmed-
abad, India, 2002, pp. 125130.
[6] B. Coifman, D. Beymer, P. McLauchlan, and J. Malik, A real-time
computer vision system for vehicle tracking and traffic surveillance,
J. Transp. Res., Part C, vol. 6, no. 4, pp. 271288, 1998.
[7] D. Comaniciu, V. Ramesh, and P. Meer, Kernel-based object tracking,
IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 5, pp. 564577,
May 2003.
[8] B. Ferlis, Analysis of infrastructure-based system conceptsIntersection
collision avoidance problem area, 1999. Unpublished FHWA document.
[9] S. Gottschalk, M. C. Lin, and D. Manocha, OBB-Tree: A hierarchical
structure for rapid interference detection, Comput. Graph., vol. 30, no. 3,
pp. 171180, 1996.
[10] J. Kato, T. Watanabe, S. Joga, J. Rittscher, and A. Blake, An HMM-based
segmentation method for traffic monitoring movies, IEEE Trans. Pattern
Anal. Mach. Intell., vol. 24, no. 9, pp. 12911296, Sep. 2002.
[11] D. Koller et al., Towards robust automatic traffic scene analysis in real-
time, in Proc. 12th Int. Conf. Pattern Recognition (ICPR), Jerusalem,
Israel, 1994, pp. 126131.
[12] M. Lin and S. Gottschalk, Collision detection between geometric mod-
els: A survey, in Proc. Institute Mathematics and Applications (IMA)
Conf. Mathematics Surfaces, Birmingham, U.K., 1998, pp. 3756.
[13] S. Lu, D. Metaxas, D. Samaras, and J. Oliensis, Using multiple cues
for hand tracking and model refinement, in Proc. Computer Vision and
Pattern Recognition Conf., Madison, WI, 2003, pp. 443450.
[14] S. Kamijo, Y. Matsushita, K. Ikeuchi, and M. Sakauchi, Occlusion robust
tracking utilizing spatio-temporal Markov random field model, in Int.
Conf. Pattern Recognition, Barcelona, Spain, 2000, vol. 1, pp. 140144.
[15] P. Mahalanobis, On the generalized distance in statistics, in Proc. Nat.
Institute Science India, Bhavnagar, India, 1936, vol. 12, pp. 4955.
[16] O. Masoud and N. Papanikolopoulos, Using geometric primitives to
calibrate traffic scenes, in Proc. IEEE/RSJ Int. Conf. Intelligent Robots
and Systems (IROS), Sendai, Japan, 2004, pp. 18781883.
[17] R. Van der Merwe, J. de Freitas, A. Doucet, and E. Wan, The unscented
particle filter, in Advances Neural Information Processing Systems,
Denver, CO, 2000, pp. 584590.
[18] T. Penney, Intersection Collision Warning System, 1999. Pub. No. FHWA-
RD-99-103.
[19] P. Prez, J. Vermaak, and A. Blake, Data fusion for visual tracking with
particles, Proc. IEEE, vol. 92, no. 3, pp. 495513, Mar. 2004.
[20] W. Power and J. Schoones, Understanding background mixture models
for foreground segmentation, in Proc. Imaging and Vision Computing
New Zealand (IVCNZ), Auckland, New Zealand, 2002, pp. 267271.
[21] H. Preston, R. Storm, M. Donath, and C. Shankwitz, Review of
Minnesotas rural intersection crashes: Methodology for identifying in-
tersections for Intersection Decision Support (IDS), Minnesota Dept.
Transp., St. Paul, MN, Tech. Rep. MN/RC-2004-31, 2004.
[22] C. Rasmussen and G. Hager, Probabilistic data association methods for
tracking complex visual objects, IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 23, no. 6, pp. 560576, Jun. 2001.
[23] J. Sherrah and S. Gong, Fusion of perceptual cues using covariance
estimation, in Proc. British Machine Vision Conf., Nottingham, U.K.,
1999, pp. 564573.
[24] C. Stauffer and W. Grimson, Adaptive background mixture models for
real time tracking, in Proc. Computer Vision and Pattern Recognition
(CVPR), Fort Collins, CO, 1999, p. 252.
[25] Y. Wu and T. Huang, Robust visual tracking by integrating multiple
cues based on co-inference learning, Int. J. Comput. Vis., vol. 58, no. 1,
pp. 5571, 2004.
Stefan Atev received the B.A. degree in com-
puter science and mathematics from Luther College,
Decorah, IA, in 2003. He is working towards the
Ph.D. degree in computer science at the University
of Minnesota, Minneapolis.
He has research interests in real-time computer-
vision systems, video surveillance, and image
processing.
423
Hemanth Arumugam received the M.S. degree in computer science from the
Department of Computer Science at the University of Minnesota, Minneapolis,
in 2004.
He has research interests in computational geometry and its applications to
computer vision and bioinformatics.
Osama Masoud received the B.S. and M.Sc. degrees
in computer science from King Fahd University of
Petroleum and Minerals, Dhahran, Saudi Arabia, in
1992 and 1994, respectively, and the Ph.D. degree in
computer science from the University of Minnesota,
Minneapolis, in 2000.
He is currently a Research Associate at the De-
partment of Computer Science and Engineering at
the University of Minnesota. In the past, he was a
Postdoctoral Associate at the same department and
served as the Director of Research and Development
at Point Cloud Inc., Plymouth, MN. His research interests include computer
vision, robotics, transportation applications, and computer graphics.
Mr. Masoud is the recipient of a Research Contribution Award from the
University of Minnesota, the Rosemount Instrumentation Award from
Rosemount Inc., and the Matt Huber Award for Excellence in Transportation
Research. One of his papers (coauthored by N. P. Papanikolopoulos) was
awarded the IEEE VTS 2001 Best Land Transportation Paper Award.
Ravi Janardan (M00SM01) received the Ph.D.
degree in computer science from Purdue University,
West Lafayette, IN, in 1987.
He is Professor of Computer Science and Engi-
neering at the University of MinnesotaTwin Cities.
His research interests are in the design and analysis
of geometric algorithms and data structures, and
their application to problems in a variety of areas,
including computer-aided design and manufacturing,
transportation, very-large-scale-integration (VLSI)
design, bioinformatics, and computer graphics. He
has published extensively in these areas.
Nikolaos P. Papanikolopoulos (S88M93
SM01) was born in Piraeus, Greece, in 1964.
He received the Dipl.Ing. degree in electrical and
computer engineering from the National Technical
University of Athens, Athens, Greece, in 1987,
the M.S.E.E. degree in electrical engineering from
Carnegie Mellon University (CMU), Pittsburgh, PA,
in 1988, and the Ph.D. in electrical and computer
engineering from CMU in 1992.
Currently, he is a Professor in the Department of
Computer Science at the University of Minnesota,
Minneapolis, and the Director of the Center for Distributed Robotics. He
was a McKnight Land-Grant Professor at the University of Minnesota for the
period 19951997. He was the recipient of the Kritski Fellowship in 1986
and 1987. His research interests include robotics, sensors for transportation
applications, control, and computer vision. He has authored or coauthored
more than 190 journal and conference papers in the abovementioned areas
(47 refereed journal papers).
Dr. Papanikolopoulos was a finalist for the Anton Philips Award for Best
Student Paper in the 1991 IEEE International Conference on Robotics and
Automation, the recipient of the NSF Research Initiation and Early Career
Development Awards and the Faculty Creativity Award at the University of
Minnesota in 19951997, and the recipient of the Best Video Award in the
2000 IEEE International Conference on Robotics and Automation. One of his
papers (coauthored by O. Masoud) was awarded the IEEE VTS 2001 Best Land
Transportation Paper Award. Finally, he received grants from DARPA, DHS,
Sandia National Laboratories, NSF, Microsoft, INEEL, U.S. Army, U.S. Air
Force, USDOT, MN/DOT, Honeywell, and 3M.