CAPC: A Road-Departure m

The C CCD

condii
CAPC

image

David J. LeBlanc, Gregory E. Johnson, Paul J.Th. Venhovens, Garth Gerber, Robert DeSonia, Robert D. Ervin, Chiu-Feng Lin, A. Galip Ulsoy, and Thomas E. Pilutti
he Crewmans Associate for Path Control (CAPC) is an automatic road-departure warning system for motor vehicle drivers. This article presents the design of, and preliminary field results from, the CAPC prototype vehicle, a modified 1995 Ford Taurus SHO. The CAPC system is intended for highway drivers who are drifting off the roadway at a shallow angle of departure due to inattention, drowsiness, intoxication, or other causes. A camera senses the roadway ahead, and a suite of transducers provide measurements of vehicle motion and driver steering commands. These are used to anticipate unintended road departures and wam the driver. Warning and intervention functions for CAPC have been developed during ongoing research at the University of Michigan; the algorithms are reviewed here in the context of the prototype. The prototype hardware and software are described in detail, and preliminary experimental data from the field is presented.

Introduction
Single-vehicle road departure accidents accounted for approximately one-third of U S. highway fatalities in 1991 [I].

This paper was originally presented at the Fifth IEEE International Conference on Control Applications, Dearborn, MI, Sept. IS, 1996. LeBlanc and Ulsoy are with the University of Michigan, Department of Mechanical Engineering & Applied Mechanics. Johnson and Ervin f Michigan TransportationResearch Institute, Ann Arbor: Venhovensis with BMWAG, Munich, Germany. Gerber are with the University o and DeSonia are with the Environmental Research Institute of Michigan, Ann Arbol; MI. Lin is with the National Pintung Institute of Technology, Taiwan. Pilutti is with Ford Research Laboratories, Dearborn, MI.

December 1996

0272-1708/96/$05.000 1996IEEE

61

t
Fig. 1. Time to lane-crossing (TLC) assuming constant steer angle.

and is among options considered for emerging yaw stability systems reaching the market [3, 41. See [5] for an estimate of controller authority and a sophisticated controller design. The limited control authority of differential braking guarantees that the driver always has primary steering control, even during intervention. The co-driver approach differs from autonomous road-following systems in both intent and design. Full-time autonomous steering systems are often motivated by congestion concerns 16, 71, and the resulting systems are often closed-loop control systems that keep the vehicle near the center of a lane while removing the driver from direct control [8]. These autonomous systems may switch between controllers-the driver, and the autonomous steering system-at specific moments, for instance, when entering a special commuting lane. Fig. 2(a) shows the lane-keeping controller structure with current conventional vehicles, and Fig. 2(b) shows a model of an autonomous lane-keeping approach. On the other hand, CAPC is a driver-assistance or active safety approach, which serves as a backup to the driver, monitoring the controlled error (lane-keeping errors), and intervening
Roadway Scene Driver

Nearly a third of these involved driver drowsiness, intoxication, or illness that led to the relinquishment of lane-keeping control. We present a prototype implementation of a driver assistance system called Crewmans Associate for Path Control (CAPC). CAPC is designed to anticipate road departures, to warn the driver of danger and, if necessary, to actively intervene to provide additional lateral path control. The system is intended to be unnoticeable by the driver, except for rare instances when the driver is unintentionally departing from the roadway. Automatic assistance is maintained only until the driver recovers control or until the vehicle has been brought to rest. Thus the system serves as a vigilant co-driver. The prototype currently issues warnings, but does not yet have a functioning intervention mode, although plans are to finish this function during the next year. To anticipate road departures, the system is equipped with computer vision to sense the roadway up to 100m in front of the vehicle. A computation, continually updated at 10 Hz, predicts the vehicles path and compares this prediction with the sensed roadway geometry to provide an estimate of the time to lanecrossing (TLC).The time to lane-crossing is defined as: the time until the vehicle CG will cross either edge of the roadway, assuming both the vehicle speed and front wheel steering angle remain unchanged. This is illustrated in Fig. 1. An audible warning is issued when the estimated TLC falls below a threshold. Another audible signal is issued when the estimated TLC falls below a smaller threshold. This second signal indicates when an intervention would begin, were the intervention system operational. Intervention, when implemented, will be achieved using differential braking-the unequal application of braking torques to the left- and right-side wheels. Differential braking provides a yawing moment of the vehicle that, properly controlled, has been shown in simulation to be suitable for laterally maneuvering to correct the vehicles path [Z]. Differential braking uses hardware that exists for antilock brake systems or traction control systems,

Vehicle/Road i y i v e Motion

(a) Control structure without lane-keeping system.

Roadway with infrastructure Vehicle/Road Relative Motion

(b) Control structure for autonomous lane-keeping system.

Driver Sensory

Roadway Scyne
Vehicle Vehicle/Road e Motion

(c) Control structure for CAPC, a lane-departure prevention system.

Fig. 2.

62

IEEE Control Systems

Table 1. Onboard Sensors and Their Role

Sensor

Used to Measure:

Used in the computation of Time to lane-crossing computation, estimation of vehicle state w.r.t. road edge. Time to lane-crossing,estimation of vehicle state w.r.t. road edge. Driver state assessment (not yet implemented). Time to lane-crossing,estimation of vehicle state w.r.t. road edge. Validating vehicle dynamics models used in algorithms design and tuning. Time to lane-crossing,estimationof vehicle state w.r.t. road edge.

Camera, B&W, direct digital readout CCD. Road edge geometry. LVDT on steering rack. Front wheel steer angle.

String potentiometer and grooved barrel on Steering wheel angle. steering wheel shaft. Optical gyroscope. Yaw rate.

Accelerometer (for testing only-not re- Lateral acceleration. quired for road departurewarning function). Wheel speed sensors (from existing ABS Vehicle forward speed. system).

1 masses, each wheel.

LVDTs between sprung and unsprung Low frequency sprung mass pitch and roll Correction to the projection from camera I coordinates to earth coordinates. estimates. I

(becoming a controller) rarely, when some set of error criteria are reached, as shown in Fig. 2(c). Such active safety systems represent a continuing move in automotive safety into providing not only crashworthy vehicles, but vehicles with crash-avoidance capabilities. To encourage near-term driver acceptance, CAPC seeks to always keep the driver in the loop; the driver always maintains primary control of the vehicle, even during intervention periods. With these differences, CAPC faces two challenges that autonomous lane-keeping systems do not: (1) to anticipate and detect dangerous situations, and (2) to interact with a driver, either in accommodating a variety of individual driving styles, or during intervention, by operating in a dual-controller mode with the driver. Potential advantages over autonomous lanekeeping systems include more rapid commercialization, since these systems may operate on many U.S. highways without any special infrastructure modifications. A system with some of the functionality of CAPC may be on the market within the next two to five years. As with other active safety technologies, CAPC might also serve as a bridge in the possible evolution of vehicles from completely driver-controlled machines to partially or fully automated transportation. The CAPC system prototype-a modified 1995 Ford Taurus SHO, shown in Fig. 3-is designed as a testbed to gain empirical evidence of favorable approaches for road departure prevention [9]. Accordingly, the sensing task is simplified to minimize the required investment in sensing technology. A simplified set of roadway conditions is assumed so that a relatively simple lanesensing system can deliver reliable road-edge detection, permitting the concentration of resources onto the study of the driver-assistance functionality: limited-access highways with white lane striping, away from entrance and exit ramps where road-edge striping is intentionally interrupted; daytime, non-shadowed, illumination of the pavement; pavement in good repair and free of water, snow, or other contaminants;

road shoulder clear of harmful objects for at least a carwidth distance; and other vehicles no closer than 50 meters in front of the CAPC prototype. This article provides an overview of the implementation on the prototype vehicle, as well as field experience to date. Previous papers have described algorithm design and evaluation of components of the system 12, 5, 10-161, including features not included on the prototype, or not yet included. The algorithm papers describe using TLC for road departure [12], a sophisticated computation of TLC and TLC uncertainty [ll-151, use of differential braking for road-departure control intervention 121, an advanced design for differential braking for path control [5], and an on-line approach to driver state assessment [16]. Planned additions to the CAPC vehicle system include algorithms of driver state assessment (to influence decisions to warn or intervene) [ 161 and differential braking control [2,5]. A recent paper [5] provided an overview of the algorithm designs, including a new differential braking control design. The paper presented here, however, focuses on the current prototype design and experience. Other research groups are studying related systems. Researchers at Carnegie Mellon University (CMU) report a TLCbased prototype for road-departure prevention 1171. The CAPC system is less focused on the sensing aspects than CMUs system (which looks largely to the side, and down); instead CAPC concentrates on other system aspects. The use of TLC, defined here as the predicted time until the vehicle CG crosses the road edge (assuming constant steering, and estimating the roadway geometry ahead), allows the system to anticipate departures and accommodate a wider range of driving styles. It is very significant that CAPC considers previewed road curvature, vehicle yaw rate, and driver steering in TLC computation, while the CMU system considers a time sequence of vehicle lateral positions relative to the lane marks immediately to the side of the vehicle. CAPC uses more information and can provide more anticipation; we believe this potentially enlarges the set of situations in which

December 1996

63

warning and intervention will be effective. Daimler-Benz has also demonstrated a prototype vision-based road departure prevention function on a truck [18]. Other systems have been developed to reduce driver fatigue by providing low-authority steering actuation in a feedback mode, leaving the driver to both monitor the system and provide supervision [ 19-21].This purpose is different from that of CAPC, although these systems may provide some road-departure safety, as a secondary effect. Finally, the vision-based estimation of vehicle and roadway state is similar to the single Kalman filter approach used by Dickmanns and colleagues at the Universitat der Bundeswehr Munchen [8]. Our novel use of two Kalman filters to estimate roadway geometry both locally and ahead of the vehicle is necessary because CAPC needs accurate sensing locally and far ahead-CAPC must anticipate departures and cannot rely on closed-loop feedback to provide robustness to roadway sensing. The next section describes the physical implementation of the CAPC prototype function on the Taurus SHO; following this, the computer vision-based lane-mark system (LMS) is presented. The fourth section describes algorithms used onboard the vehicle to effect the CAPC road-departure warning function. Experimental results-both quantitative and qualitative-are then presented.

sides of the lane are processed; the frame-rate is 10Hz. A blackand-white digital camera and a digital frame-grabber were chosen for the direct digital readout, controllable exposure time, and external triggering capabilities. The camera signal is digital right at the camera output; this produces images of such resolution that lane stripes are often resolved out to 150m. The image processing delivers the location of several discrete points on the lane marks (paint stripes) to the main computer, using simple projection from the image plane to an assumed flat earth. Simulation results using a 14 DOF model and actual road profiles show that pitch and roll motions of the sprung body introduce large errors in lane marker locations delivered by the image processing [9]. Therefore, low-frequency body pitch and roll motions are approximated using four linear variable displacement transducers (LVDTs) mounted between the sprung mass (car body) and each of the unsprung masses. These estimates are found to provide sufficient corrections for typical U.S. highway surfaces. The LMS system returns the lane mark locations to the Quadra at 10 Hz and receives pitch and roll estimates in return. The total latency of CAPC actions that are based on LMS information is 200 ms-100 ms to receive the information, and 100 ms to output a warning or intervention signal. This is a significant delay; prediction of states (as described below) is used by the closed-loop differential braking laws to account for the delay.
Transducers for Vehicle Motion and Steering Inputs The vehicle contains a suite of transducers for measunng or estimating vehicle motion and steering inputs. Table 1 lists the sensors and their role. Instrumentation at the four wheels is identical. LVDTs are installed to transduce the displacement between the sprung mass and the unsprung mass. Pitch and roll are estimated as described later in this section. Vehicle speed is transduced using the magnetic pickups provided with the ABS system, and a frequency-to-voltage conversion. An LVDT transduces the linear motion of the steering rack, providing data to deduce front-wheel steer angle. The string potentiometer to measure steering wheel angle is attached to an aluminum grooved barrel that is clamped around the shaft between the tilt mechanism and the universal joint at the firewall. The accelerometer and yaw rate transducer are commercial products and follow a typical installation. Anti-aliasing analog filters and digital lowpass filters are used for all analog transducer signals.

Physical Implementation of the CAPC System

Computers The primary computers onboard the CAPC vehicle include a Macintosh Quadra 800 executing the CAPC function computations and data acquisition, and a lOOMHz 80586 implementing the vision-based lane-mark sensor (LMS) system. Ancillary processors include a microcontroller in an analog signal conditioning rack that contains customized circuitry; a controller for the planned brake servo pressure loop; and processors in the imaging camera and on computer I/O cards. Communications between modules are primarily RS232 serial. Lane-Mark Sensor (LMS) System A single camera mounted behind the rearview mirror (near the top center of the windshield) provides image measurements used to locate lane marks from 6 to 100 m ahead of the vehicle (a detailed discussion of the LMS system follows later). Both

Fig. 3. The CAPCprototype vehicle-1995 Ford Taurus SHO.

Fig. 4. Sample imagefrom LMS camera.

64

IEEE Control Systems

Calibration of steering wheel angle (in terms of both front wheel angle and rack displacement) and suspension deflection was performed on UMTRIs Suspension Measurement Facility. In this facility, the vehicles suspension is exercised in a known manner and the outputs of the transducers calibrated to the motion. The front-wheel steer angle is calibrated to indicate the angle of the front tire plane (including static compliances at three different load conditions). The driver steering wheel input is calibrated with reference to a known rotary transducer. Yaw rate and lateral acceleration transducers were calibrated prior to installation in the vehicle. The following 3 0 noise values were typical while operating at highway speeds: steering wheel angle, 0.4 deg; front wheel steer angle, 0.03 deg; yaw rate sensor, 0.15 degls, vehicle speed, 0.5 mph.
Vehicle Speed, Roll, and Pitch The vehicle speed is calculated by averaging the two front wheelspin-derived velocities. The rear wheel-derived speeds are ignored in anticipation of the implementation of differential braking control on the rear axle. The roll and pitch angle of the sprung mass are estimated from measurements of the four suspension deflection LVDT sensors. The LVDTs are calibrated for the displacement in the wheel plane at the middle of the tire. The calibration test and computations take into account the deflection of the suspension as well as the tire. Anti-roll bar stiffness effects are also considered carefully when calibrating roll angle, since the bars do not respond to the calibration procedure of moving both wheels on an axle up and down. This procedure and the resulting equations for pitch and roll are described in [9]. Wamings and Intervention-Indication A simple speaker that outputs two different audible tones is installed in the prototype. This provides warnings and indicates when interventions would have occurred, if the differential braking system was fully implemented. One tone indicates a warning; a second tone indicates an intervention. Currently this simple feedback serves our purposes, but it is not a focus of our research and development efforts.

Parameter Front axle load Rear axle load Front tire cornering stiffness Rear tire cornering stiffness Steering elasticity Yaw moment of inertia Roll moment of inertia Front suspension damping Rear suspension damping

Estimates

I I

1,091k.g

723 kg 63,760 N/rad 66,430 N/rad 20,000 Nlrad 3,960 kgm2 464 kpm2

I I

2,590 Ns/m 2,063 Ns/m

Vision System Processing and Calibration

Lane-Mark Sensor Requirements and Design The computer vision lane-mark sensor (LMS) is engineered at the EnvironmentalResearch Institute of Michigan (ERIM) to support the CAPC prototype requirements: (1) sense lane-marks up to 100 m in front of a vehicle operating in conditions stated earlier; (2) update lane-mark data every 100 msec; (3) detect the position of lane-marksat 2-meter intervals in the near-range (6 to 20 m), and at 10-meter intervals in the far-range (30 to 100 m); (4) compensate reported lane-markpositions for vehiclepitch and roll; and ( 5 )report lane-mark positions in vehicle coordinates, assuming a flat-earth model of the road. The application environment given in the introduction is assumed. The system is designed to be low-cost, to support continuing development, and to represent a feasible approach toward an affordable automotive product. The requirement r t to resolve and report lane marks to lOOm pushes the state of the a in lane-mark sensing. This is necessary since CAPC is anticipating lane departures and needs to sense upcoming curves. Additionally,

since CAPC operates mostly in an open-loop mode (not intervening), the sensing accuracy needs to be relatively high, since the natural robustness to sensing errors that feedback supplies is not available. The LMS system consists of a Pulnix 9701 digital CCD camera (768 x 484 pixels), a MuTech MV-1000/MV-1100 PCIbus digital frame-grabber with interconnect cable, and a PCI-bus with an Intel 100 MHz Pentium computer. The Pentium computer performs all image processing operations. The camera is rigidly mounted inside the vehicle near the rear-view mirror and views the roadway at a slight horizontal depression angle. The lens-a 12.5 mm focal-length Cosmicar-provides a 40 deg horizontal field-of-view. Neighboring pixels subtend a horizontal angle of 0.96 mrad. At a range of 100 m, this corresponds to a single pixel covering approximately 9.6 cm (lane-mark widths are typically 10 cm). The CAPC system routinely locates highway-quality lane marks to 100 m and can function out to 150 m under good lighting and pavement conditions. The camera and frame-grabber possess unique capabilities that are essential to the detection of lane marks up to 100 m: 1. Full-frame snapshot exposure initiated via external TTL signal 2. Short freeze-frame exposure length controllable by software 3. Analog-to-digital conversion synchronized with CCD pixel readout Since the CAPC objectives require imaging from a moving vehicle, the camera needs to possess a full-frame snapshot exposure to capture a full frame without any blurring or interlace problems. Exposure control via software permits adjustment to enhance the detection of lane marks, in spite of bright objects in the scene, such as the sky. Conventional auto-gain or auto-iris responds to peak scene values. The camera also needed a highly accurate pixel registration since the lane-mark position was derived from a calibration of the image position. The best way to obtain this registration accuracy is by using a digital-readout camera. Digital readout also has the advantages of being quite immune to electrical interference in the vehicle, as well as producing a superior image quality over an analog camera.

December 1996

65

Roadway lmage Vehicle

Sensors

angle. Static calibration of the LMS system was performed to determine the exact camera orientation with respect to the vehicle coordinate system. This was done by manually measuring the image locations and the vehicle coordinate locations of a small reflective object.

Algorithms for the CAPC Function

Roadway Geometry Data Vehicle S t a t e s
~

Fig. 5. Computational modules of the CAPC algorithm

Together, these features make possible the capture of sharp roadway images. Fig. 4 shows a sample image from the camera.
Location and Tracking of Lane Marks The lane-mark detection algorithm exploits the relatively uncluttered near-range image to locate the lane marks over the range of 6 to 20 m. The near-range lane marks are then followed out to 100 m. Horizontal search regions are shown in Fig. 4; search windows are separated within the near-range by two meters in ground-sampled distance. Within each search region, the lane-mark location is determined as follows. First, within the search region, the locations of the maximum positive and negative changes (horizontal) in illumination are found. The pixel halfway between these changes is assumed to be the center of the mark, if two conditions are met: (1) if the mean of the absolute value of the maximum positive and negative transitions exceeds a threshold, and (2) if the spacing between the transitions suggests a lane mark width that is less than three times the expected width. The image position (row, column) is then converted into vehicle coordinates, using a flat-earth assumption. Before this conversion, however, the image coordinates of the lane mark locations are transformed to compensate for the estimated roll and pitch of the vehicle (provided as described earlier). If the two conditions above are not satisfied, the algorithm is not confident that the feature causing the illumination changes is a lane mark, and no data is reported to the CAPC computer. The placement of search windows is achieved in one of two modes: recursive and initialization. In the recursive mode, the search windows are determined during the processing of the previous image. The near-range algorithm assumes that the vehicle lateral deviations and heading angle undergo relatively small changes over a 100 msec interval. If four or more lane marks are found within the near-range region of the previous image, a least squares linear fit (in vehicle coordinates) is made to determine the lateral deviation of the vehicle CG from the right lane mark and the vehicle heading angle. Curvature in the lane mark (which is minimal over the near-range) is ignored. The heading angle and lateral deviation values are used to position the search regions for the successive frame. Far-field search windows are located by extrapolating the near-range line, and by approximating with each newfound lane mark location a curvature in the far-field. In initialization mode, the LMS process is permitted up to a few seconds to locate the lane marks and to set exposure times. Search windows are currently found by assuming that the vehicle is roughly in the center of the lane with a negligible heading

The CAPC computations onboard the vehicle consists of three modules: (1) estimation of the vehicles state with respect to the road edge; (2) computation of the time to lane-crossing (TLC), and (3)decision-making to initiate warnings or interventions. Continuing work is planned to implement other modules already developed in simulation or in driving simulator work, including (4) on-line assessment of the drivers alertness state, and ( 5 ) active intervention via differential braking. Fig. 5 shows all five modules.
Near-Range Kalman Filter The vision system returns from each image alist of coordinate pairs corresponding to locations of discrete points on the road edge markings (e.g., paint stripes). These locations are computed as described earlier, and are expressed in the vehicles coordinate system. The near-range Kalman filter uses the near-range vision data and vehicle motion sensor measurements-yaw rate, vehicle speed, front wheel steering angle-to estimate vehicle rates, vehicle orientation with respect to the roadway edge, and the local road curvature. The state estimates are used for computing time to lane crossing and for a state feedback design for the differential braking intervention. The state vector x is:

where: y = distance of vehicle CG from road edge, = heading error with respect to local road edge, K = local road curvature, v = side slip velocity in vehicle-fixed coordinates, and r = yaw rate. Reference [5] develops the state and measurement models used for the design of a linear time-invariant Kalman filter, which is updated at 10Hz. The state dynamics model used in the Kalman filter design consists of a two degree-of-freedom, linear lateral dynamics model (the bicycle model [22]), and an assumption that the time rate of change of road curvature is white noise. This model assumes a linear tire model - Le., that the lateral forces on the tire are proportional to the slip angle of the tire (the angle between the wheel plane and the velocity of the wheel center). Linear tire models can be accurate when the lateral manuevers are not very dramatic, for example, during typical lane-keeping activities. Because elements of the vehicle dynamics model depend on vehicle speed, the Kalman filter matrix elements are computed off-line at various speeds. Each element is expressed as a polynomial in vehicle speed. On-line implementation of the filter uses vehicle speed to compute the appropriate elements at any update time. The Kalman filter design requires noise models, as described in [5].Noise values for front wheel steer and yaw rate sensing are used; the errors in vision-based location of the lane-marks are modeled as zero-mean white noise with a covariance which

66

IEEE Control Systems

varies quadratically with down-range distance. Since the LMS reports image column positions for some subset of pre-specified rows, the covariance is constant. The road curvature model used for filter design assumes curvature changes only as white noise. This allows adaptation to slowly changing curvatures [8]. The variance of this noise value was determined using simulation on a range of road curvatures and vehicle speeds. Simulation and experience in the vehicle confirms that the current design performs with sufficient accuracy and convergence speed for the computation of TLC.
Time to Lane-Crossing The time to lane-crossing (TLC) value is updated continually, upon arrival of camera information. The TLC is the primary metric for making decisions to warn or intervene. Reference [ 121 shows that a 10 Hz measurement sample rate is sufficient for lane-keeping tasks, and the camera frame rate is set at this value. TLC is computed by comparing a predicted vehicle path with an estimate of the roadway geometry above. The intersection between the two is found by an interpolation technique [12] which provides improved TLC accuracy for a fixed amount of computations. Vehicle path projection.Vehicle path projection is computed anew at the CAPC l0Hz rate. The path is projected by numerically integrating the linear bicycle model for vehicle lateral motion using state estimates from the near-range Kalman filter and current vehicle motion measurements as initial conditions. It is assumed that the front wheel steer angle will remain constant. References [13] and [14] describe detailed aspects of vehicle path projection algorithms used on CAPC. One feature not yet implemented is the use of onboard estimates of disturbance forces acting on the vehicles lateral dynamics. Such disturbances can result from crosswind, road superelevation and crown, uneven tire inflation, and so on. The path of the vehicle CG is projected from the current moment until the predicted path either crosses the lane marker or until four seconds of projection has occurred. The projection is computed in the coordinates of the vehicle frame at the current time, therefore a transformation (linearized) is applied to use the vehicle-fixed dynamics. Simulation and vehicle testing has shown that the use, linearization, and the discretization of the two degree freedom bicycle model are all acceptable for the purposes of TLC used here [ 111. Road geometry estimation. The road geometry ahead of the vehicle is estimated using a second, far-range Kalman filter. This filter uses estimates from the near-range filter as well as far-range vision data (20 to 100 m). Two different Kalman filters are used-instead of combining them into one filter-because a single curve fit to the road edge over the entire 100 m range is not a suitably accurate fit to the actual road edge. True highway road edges involve either constant radii of curvature or, in transitions, Euler spirals (curvature that varies linearly with distance). Through extensive simulation studies, we have found that a single linear Kalman filter design model cannot satisfactorily represent both the vehicle motion and road geometry over 100 m. The far-range road edge is modeled at any instant as a thirdorder polynomial, written in the current vehicle-fixed coordinate system [ l l , 151. This model assumes a linear variation of road edge curvature K with distance. Reference [15] develops the

Kalman filter design. The Kalman filter gain is obtained from a process noise covariance matrix and a measurement error covariance matrix obtained by simulation runs with effects of pitch and roll, sensor resolution, and a 14 DOF vehicle dynamics model with a complex, nonlinear tire model known as the Magic Formula tire model [23]. Lin reports that a Kalman filter with a third-order polynomial road model gives better overall performance for far-range distances than a second-order model or a simple single-frame least squares fitting of the camera data [15]. Lin also shows that the far-range Kalman filter does not require speed-dependent gains.
Decisions to Warn the Driver Decision logic is applied upon each new update of TLC (every 100ms). The output of the decision logic in the vehicle is whether to warn, to indicate that intervention would occur, or to do neither. The TLC alone is used to make decisions with the current implementation, however, after field testing, a new tuning constant Y o f e t was added to shift the road edge laterally during TLC computation. For instance, if is 0.5m, then TLC(yofSset) is the time until the vehicle CG will be within 0.5m of the road edge, assuming (as before) a constant steer angle and vehicle speed. Thresholds are applied to TLC(yofset)to determine whether to issue a warning or indicate an intervention. Let T C w and denote thresholds on TLC. The basic rules in the vehicle are: Warn, if: TLC(yofset) 5 TLC, for three consecutive samples, Indicate intervention, if for three consecutive samples, TLCbofser) < Additional logic suppresses warnings or interventions if one or more of the following apply: the LMS reports less than four points in the near-range region of the right side

Yaw Rate (deg/s)

I I

-20

f 0
: .

-303::

Time (S) Lateral Acceleration (m/sec2)

! . . . .

10

15

20

-4-

CAPC Simulation, 14 DOF Non-linear

bCAPC

Simulation, 2 DOF Linear CAPC Prototype

5
0
-5 -10 f 0
5

10

15

20

Time(s)

Fig. 6. Simutation and vehicle data: Lane changes at 121 km/h (75 mph).

December 1996

67

Steering Wheel Angle-Deg

................ ...........,.................................................................... ..................,

Heading Angle-Deg

............................................................................................. .....................

-104 0

. . . . : . . . . : . . . . :
10
20 Time-Sec

. . . .
40
Time-Sec

30

-Near Field
Estimate

CG Distance From Right Edge-m 0.1 o,5 Estd Road Curvature-m 10-2 ..........................
10-8

[.... ....................

-0.5 x IO4

-0.1x10-2!
0

. . . . i . . . . 1 . . . . i
0

10

20 Time-Sec

30

40

10

20 Time-Sec

30
-0-U-

f 40

Near Field Est. Far Field Est.

& &

Estimated TLC-Sec 4

......
L l i l

h 1. ~ b...! r"'
....... ......
yaw

....................................................

i i

7 +... . . . . . . . . . . . . . . . . . . . . . . . . . .......... -

........:...........................
i

...........................

. I

.............................
i

; ;............................

2
0

0
-2

10

20
Time-Sec

30

40

-4

10

20 Time-Sec

30 -,-,-Gyro
-Near

40
Measurement Field KF Estimate

Fig. 7. Vehicle data: Sinusoidal steering to trigger warnings and intervention.

the warning or intervention has continued for 10 seconds or more less than one second has passed since action last took place vehicle speed is less than 30 kph or more than 150 kph. Values for the three main decision parameters, TLC,, , and are easily varied onboard. Nominal values for the parameters are: = 2.0sec; = 1.0sec; yoffset= 0.2m. We find through informal experience that drivers have slightly different preferences for these parameters. Furthermore, certain roadways may call for tighter bounds on allowed lane deviation, e.g., urban freeways which are closely contained by walls. The values above were chosen by acombination of simulation and experience in the field using the prototype vehicle. Simulations were used to choose initial values for the TLC thresholds: = 1.4 sec, = 0.8 sec. These were later adjusted during field testing the prototype to TLC, = 1.8 sec to allow the driver more time to react, and finally, = 2.0 and = 1.0 sec after additional drivers reported their preferences. The offset yoffSet = 0.2m was added later to provide tighter monitoring of lane-keeping. Simulation indicates that these rules will be effective for mild and moderate road departure scenarios. The moderate authority
~ ~

of differential braking means, of course, that there will be severe departure scenarios for which active intervention alone cannot prevent road departure. The thresholds chosen trade between two goals: (1) minimizing false warnings and unwanted interventions, and (2) taking action in time to prevent road departures for the largest possible set of departure conditions (vehicle speed, departure angle, steer angle error, road curvature).

Experimental Results
Vehicle Model Parameter Estimates and Model Validation

Experiments were conducted to estimate key vehicle dynamics parameters and to validate the models used in vehicle dynamics simulation. The resulting values of these parameters are shown in Table 2. Accurate knowledge of the parameters is useful for system design, for evaluation in simulation, and for use onboard to estimate the vehicle and roadway states and to compute TLC. The vehicle CG location and axle loads were found by simply placing the wheels on accurate shop scales. The methods used to estimate the remaining parameters are now described. The yaw inertia and cornering stiffnesses are estimated by comparing simulation responses and actual vehicle responses for handling tests. A series of lane-change manuevers with lateral accelerations less than 0.4g was performed with the vehicle. The

68

IEEE Control Systems

front wheel steer angle and vehicle speed data that was recorded were used as inputs into a vehicle dynamics model in simulation. The vehicle model used for this parameter identification was a simple 2 DOF vehicle with linear steady-state tire characteristics (based on a fixed cornering stiffness) and a transient tire model extension (based on a fixed relaxation length). The cornering stiffnesses and yaw moment of inertia were found by minimizing the sum (over the set of sampled data) of the squared errors between measured and simulated lateral acceleration and yaw rate:

where q1 = q 2 = 1 is used since the magnitudes of lateral acceleration (Ay, in d s 2 ) and yaw rate ( r ,in deg/sec) are similar. The front cornering stiffness value includes some effects of elasticity in the steering and front suspension. Roll inertia and suspension damping values were also estimated using handling tests. Random steering manuevers at highway speeds were performed, and frequency response functions (FRFs) were used to find the roll inertia and suspension damping values that provided the best fits between experimental data and simulation. For this experiment, two vehicle dynamics models were used-the simple 2 DOF model with a linear tire model, and the 14 DOF model with non-linear tire characteristics (using the Magic Formula model). The FRFs are computed by applying fast Fourier transform techniques to compute
Front Wheel Steering Angle (rack)-deg
..........................................
...................................................................................
,

power/cross spectral densities, transfer functions, and coherencies. The two models were linearized to allow analytical computations of the FRFs; the wheelspin degrees of freedom in the 14 DOF model were removed for simplicity, yielding a linearized 10 DOF model. The roll inertia and suspension damping were altered to provide a match between the FRFs of the measured and the 10 DOF model. Changes in damping were used to match the gain at the resonant frequency (about 2 Hz); the roll inertia was varied to match the roll natural frequency. Care was taken to keep suspension and anti-roll bar stiffnesses constant (values provided by the manufacturer). After trial and error, the FRFs matched well. The steering elasticity was derived later by comparing responses of the front wheel steering angle with the steering wheel angle itself. Fig. 6 shows the match between the vehicle and the 2 DOF (linear tire model) and 14 DOF (Magic Formula) models for a series of increasingly severe lane change manuevers. Good fits result, with the 14 DOF model providing better matching at the higher accelerations because it is able to model both the side-to-side load transfer and the nonlinear tire slip forces that become important as lateral manuevers increase in severity. The good matches between data and predictions provides confidence that simulation can predict these types of motion, and also that onboard algorithms that rely on parameter values employ sufficiently accurate values.
Onboard Estimation Results In the previous section, two Kalman filters were described which provide estimates of the vehicle state and both local and previewed road geometry information. Fig. 7 shows signals collected from the data acquisition system while driving on a straightaway at highway speed and while gently steering with a sinusoid-like motion, so that the vehicle travels back and forth within the lane. The figure shows that the estimates of lateral position are still consistent, even for relatively large steer motions. TLC is saturated in software at four seconds if no intersection is computed before that time. In Fig. 7, TLC drops below four seconds several times, and as steer amplitude is increased, warnings and intervention buzzers are triggered. Note the agreement between the two Kalman filters estimates. The yaw estimate coming from the near-range Kalman filter matches well with the direct output of the yaw rate sensor. The curvature estimates are nearly zero, as expected, and the far-range estimate for curvature is more accurate than the near-range, as expected. The smallest radius of curvatures computed by the far- and near-range filters on this straight section are 3300 and 1400 m, respectively. To date, our experiments do not include knowledge of true values-this requires independent and synchronized measurement of the actual vehicles position with respect to the road edge. Data is shown here to illustrate the characteristics of the estimates and to show qualitative success in sensing and estimation. Our experience is that when the LMS is tracking the road edges correctly-and tracking is almost always successful when running under the design conditions given earlier-then the state estimates and the TLC are sufficiently accurate. In some conditions, however, the vision system sometimes loses track; these conditions are not unusual on public roads. Examples include: heavy traffic, wet pavement, darkness, entrance and exit ramps, and changing ambient illumination. While the LMS has some

..........

1.............
:

i..............

i ......

-11

10

12

Estd CG Distance From Edge-m

Estd TLC and Warning Indicator

0
...........

2
.......

10
.....

12
........

Intervention Indicator

6
Time-Sec

10

12

Fig. 8. Vehicledata: Warning and intervention indicators as vehicle drifts out of lane.

December 1996

69

tolerance to conditions beyond the design conditions, a production system would need more robustness.
Road-Departure Examples Field tests were conducted to determine whether the warning and intervention signals feel appropriate. Fig. 8 shows data collected from the vehicles data acquisition system during a test in which the vehicle is steered into a path of road departure. In this test the decision parameters were = 2.0, = 1.0 sec, and yoffset= O.Om (the road edge is not shifted). At approximately 10.8sec, the warning was triggered, and a short time later, as the TLC dropped below one second, the intervention signal goes high. The estimate of CGposition shows that the CG crosses at 13 seconds, whereupon the TLC is also approximately zero. There is 200 msec latency from imaging to action, so the CG will cross ideally when TLC is computed as 200 msec. The testing experience is that when design conditions exist, the state estimates, the TLC computation, and the warning and intervention indications consistently feel appropriate. When the vehicle is driven over solid lane marks (road edges), the system issues warnings and indicates that interventions would occur. Likewise, when keeping the vehicle largely within the lane of travel, the system very rarely issues false warnings-dthough this observation depends strongly on driving style and decision parameters. The use of TLC helps to avoid false alarms for drivers who hug one side of the lane; the use of previewed road information and driver steering inputs in the TLC is noticeable when approaching curves without proper steering. The system is quite robust to not giving false alarms to drivers weaving moderately or drifting very slowly within the lane. This accomodates various driving styles, avoids annoying drivers who are not in danger, and results in a system which waits relatively long before taking action. When operating in more demanding conditions that violate the design specifications listed in the introduction section-e.g., urban freeways with poor pavement surfaces-the road edge can be lost by the LMS, or the LMS may be confused by a vehicle (or its shadow) when the vehicle is closer than specified in the design. False alarms are the most common problem in these cases; occasionally a missed warning of a real departure can result. These system errors occur because the road geometry estimation errors can become large, and the TLC is erroneously computed as too small or too large. Because the CAPC project seeks to study the entire road-departure prevention problem, and because so much other research has focused on vision-based lane tracking, these behaviors are considered troublesome only insofar as gathering data and experience is concerned.

be accomodated while minimizing false alarms and providing timely warnings for actual road departures. Future work may include: implementation of differential braking for active intervention; study of interactions between the active intervention controller and drivers; implementation of an on-line driver state assessment module to influence the timing of decisions to warn or intervene; and adding the capability to measure true values of vehicle state and TLC. CAPC and other active safety functions represent a step in the evolution between current motor vehicles and possible partially or fully automated transportation systems of the next century. Active safety systems pose significant challenges to the controls designer, including the design and development of functions to interact with drivers in a wide variety of conditions and scenarios. The future of such systems depends, too, on the acceptance by drivers of these new technologies-this will require the collaboration of control system and human factors professionals.

Acknowledgments
This work was supported primarily by U.S. Army Tank Automotive Command Contract No. DAAE07-93-C-R124. Ford Motor Company provided the Taurus SHO vehicle along with brake system modifications, expertise, and constant support-special thanks to M. Shulman and A. Malhotra. We are indebted to colleagues at the University of Michigan, including H. Peng, C. MacAdam, and C-S. Liu, as well as to the reviewers who provided valuable suggestions.

References
[l] J.3. Wang and R.R. Knipling, Single Vehicle Roadway Departure Crashes: Problem Size Assessment and Statistical Description, NHTSA technical report, DOT HS 808 113, March 1994. [2] T. Pilutti and A.G. Ulsoy, Vehicle Steering Intervention Through Differential Braking, Proc. of American Control Conference, Seattle WA, 1995. [3] Control of Vehicle Dynamics, Automotive Engineering, vol. 103, no. 5, pp. 87-93, May 1995. [4] J. Yamaguchi, Toyota Vehicle Stability Control System, Automotive Engineering, vol. 103, no. 8, p. 34, August 1995.

[SI D.J. LeBlanc, P. Venhovens, C.-E Lin, T. Pilutti, R. Ervin, A.G. Ulsoy, C. MacAdam, and G. Johnson, A Warning and Intervention System for Preventing Road Departure Accidents, The Dynamics of Vehicles on Roads and Tracks. Proc. 14th IAVSD Symposium, Ann Arbor, MI, August 1995.
[6] S.K. Kenue, LANELOK Detection of Lane Boundaries and Vehicle Tracking Using Image-Processing Techniques-Part 1: Hough Transform, Region-Tracking, and Correlation Algorithm, Proc. SPIE Conference on Mobile Robots IV, Philadelphia, PA, November 1989.
[7] S.E. Shladover, The California PATH Program of IVHS Research and its Approach to Vehicle-HighwayAutomation, Pvoc. Intelligent Vehicles 92 Symposium, pp. 347-352, Detroit, MI, June-July 1992.

Summary
This article describes hardware and software comprising the implementation of a road-departure prevention warning system, the Crewmans Associate for Path Control (CAPC). A single camera and a suite of vehicle motion and steering angle transducers are used by sophisticated model-based algorithms to anticipate imminent road departures through the computation of the time to lane-crossing. Image processing and the road-departure prevention function are computed easily at lOHz on two PC-class computers. Preliminary experimental data indicates that, for the set of design conditions, the system performs well. The definition used for TLC allows a variety of driving styles to

[8] E.D. Dickmanns and V. Graefe, Dynamic Monocular Machine Vision, Machine Vision andrlpplications, vol. 1,pp. 223-240, 1988. 191 R.D. Ervin, ed., The Crewmans Associate for Path Control (CAPC). Final report for TACOM Contract DAAE07-93-C-R124, Universily o f Michigan Transportation Research Institute Report 95-35, 1995. [lo] R. Ervin, A.G. Ulsoy, C.-E Lin, C. MacAdam, H. Peng, T. Pilutti, P. Venhovens, and D. Symanow, An Intervention Form of Path Warning and Control, Proc. of Assoc. for Unmanned Vehicle Systems, Arlington, VA, 1994.

70

IEEE Control Systems

[ll] C.-F. Lin, Lane Sensing and Path Prediction for Preventing Vehicle Road-Departure Accidents, Ph.D. dissertation, The University of Michigan, May 1995.

[I21 C-F. Lin and A.G. Ulsoy, Calculation Of The Time To Lane Crossing And Analysis Of Its Frequency Distribution, Proc. of American Control Conference, Seattle WA, June 1995.
[ 131 C-F. Lin and A.G. Ulsoy, Vehicle Dynamics And External Disturbance Estimation for Vehicle Path Projection, Proc. of the American Control Conference, Seattle, WA, 1995.

two-degree-of-freedom models using optimal control and state estimators He began doctoral studies, also at Delft, in 1990 and received his Ph D in 1993 for work on optimal control of vehicle suspensions He worked as a postdoctoral researcher at the University of Michigan Transportation Research Institute from 1993 untd 1995, when he joined the research staff at the Bavarian Motor Works (BMW) in Munich. His interests are in vehicle dynamics and controls

[I41 C-F. Lin, A.G. Ulsoy, and D.J. LeBlanc, Lane Geometry and the Characterization of its Associated Uncertainty, submitted to the ASME Journal of Dynamic Systems, Measurement, and Control, 1995. [15] C-F. Lin, A.G. Ulsoy, and D.J. LeBlanc, Lane Geometry Reconstruction: Least Square Curve Fit Versus Kalman Filter, Proc. ASME International Mechanical Engineering Conference and Exhibition, San Francisco, CA, 1995. [16] T. Pilutti and A.G. Ulsoy, Online Identification Of Driver State For Lane-Keeping Tasks, Proc. of American Control Conference, Seattle WA, 1995.
[ 171 M. Chen, T. Jochem, and D. Pomerleau, AURORA: A Vision-Based Roadway Departure Warning System, Proc. I995 Symposium on Intelligent Robots and Systems, Pittsburgh, PA, August 1995, pp. 243-248.

Garth Gerber received a masters degree in physics from the University of Michiganin 1975, and pursued aPh.D. programin observational astrophysics through 1983. In 1983, he joined Applied Intelligent Systems, where he developed image-processing algorithms and imaging techniques for industrial machine-vision applications. In 1987 Mr. Gerber joined KMS Fusion, where he developed infrared and low-light-level airborne sensors, and developed soft x-ray diagnostic instrumentation to support inertial confinement laser fusion research. In 1990, Mr. Gerber joined ERIM, where he applies remote sensing techniques in variety of areas, including vehicle trajectoly and orientation sensing.

In 1990, Robert DeSonia joined ERIM, where he develops remote sensing techniques for various applications, primarily transportation and automotive. He received his bachelor of science in electrical engineering at the University of Michigan in 1995.
Robert D. Ervin is head of the Engineering Research Division, University of Michigan Transportation Research Institute (UMTRI), and a research scientist in UMTRI and the Department of Mechanical Engineering and Applied Mechanics. He holds an M.S. in mechanical engineering from Cornell and has studied the dynamics and control of motor vehicles at Michigan since 1969. In 1988 he initiated intelligent transportation systems research at UMTRI, and currently leads active safety technology research. Mr. Ervin is a member of the Coordinating Council of the Intelligent Transportation Society of America and serves on committees of ITS America and the Transportation Research Board. Chiu-Feng Lin received a bachelors degree in 1986 in naval architecture and marine engineering from National Chen Kung University, Taiwan. He received a masters degree from Auburn University in 1990, and aPh.D. from the University of Michigan in 1995 for work in motor vehicle lane sensing for active safety systems. In 1995 he joined the Machine Tool Division, Fair Friend Company, Taiwan, where he served as director of the CNC machine tool laboratory. Currently, Dr. Lin is chairman of the Vehicle Engineering Department, National Pintung Institute of Technology. His research interests include vehicle control and vehicle dynamics.
A. Galip Ulsoy has been on the faculty at the University of Michigan since 1980, and is currently the William Clay Ford Professor of Manufacturing and the Director of the College of Engineering Program in Manufacturing. He is an associate editor of the IEEE/ASME Transactions on Mechatronics and a member of the editorial hoard of the international journal Mechanical Systems and Signal Processing. He is the author or co-author of more than 100 refereed technical articles on the dynamic modeling, analysis, and control of mechanical systems. Galip Ulsoy is a Fellow of ASME, a Senior Member of SME, and a member of AAAS, ASEE, IEEE, and Sigma Xi.
Thomas E. Pilutti has been with the Vehicle Electronic Systems Department of the Ford Research Laboratories, in Dearborn, Michigan since 1990. He is a Ph.D. candidate at the University of Michigan. His work involves system concepts for unintended road departure systems, as well as primary responsibility at Ford for the development of energy management strategies for coordinated vehicle system control. After his B.S.M.E. from Purdue University in 1983, he worked until 1987 at Owens-Illinois Company in St. Louis, where he earned a B.S.E.E. from Washington University. His M.S. in mechanical engineering came in 1989 from the University of Michigan.

[18] T. Zimmermann, A. Fuchs, U. Franke, and B. Klingenberg, VECTOR-A Vision EnhancedControlled Truck for Operational Research, Proc. International Truck & Bus Exposition, Seattle, WA, November 1994, SAE 942284.
[I91 K. Naab, and G. Reichart, Driver Assistance Systems for Lateral and Longitudinal Vehicle Guidance- Heading Control and Active Cruise Support, Proc. AVEC 94 Symposium, pp. 449-454, 1994. [20] S. Birch, New Technology Directions at Ford, Automotive Engineering, pp. 69-71, May 1995. [21] U. Franke, S. Mehring, A. Suissa, and S. Hahn, The Daimler-Benz Steering Assistant-A Spin-off from Autonomous Driving, Proc. IEEE Intelligent Vehicles 94 Symposium, pp. 120-124, Paris, October 1994. [22] T. Gillespie, Fundamentals of Vehicle Dynamics, Warrendale, PA: Society of Automotive Engineers, 1992. [23] E. Bakker, H.B. Pacejka, and L. Lidner, A New Tire Model with an Application in Vehicle Dynamics Studies, Proc. 4th Auto Technologies Conference, Monte Carlo, 1989, SAE 890087.

David J. LeBlanc has applied dynamics, control, and estimation techniques to a wide rangeof systems. HereceivedB.S.M.E. andM.S.M.E. degrees from Purdue University in 1983 and 1985, respectively. He joined Hughes Aircraft Company, performing controls and dynamics systems analysis for spacecraft and automotive programs. In 1994 he earned a Ph.D. in aerospace engineering from the University of Michigan for research on obstacle mapping using computer vision during low-altitude rotorcraft flight. He has since worked on vehicle control research at the University of Michigan. Dr. LeBlanc has authored or coauthored several papers and is an IEEE member. Gregory E. Johnson received B.S. and M.S. degrees in electrical engineering and computer science from the University of Michigan, Ann Arbor. He has designed and implemented image processing systems for medical imagery, and from 1992 to 1996 Mr. Johnson has been a research engineer with the University of Michigan Transportation Research Institute. There he has led work to implement measurement and control systems on several vehicles, in support of research in human factors, automotive safety, tire-modeling, vehicle dynamics, and vehicle controls. Mr. Johnson will continue graduate studies at the University of Colorado.

Paul J.Th. Venhovens studied mechanical engineering at the Delft University of Technology, from which he graduated in 1989 with work evaluating

December 1996

71