2ND WORKSHOP ON USER CENTRIC PERVASIVE ADAPTIVE SYSTEMS | 06.12.

2010

AN ADAPTIVE DRIVER ALERT SYSTEM MAKING USE OF IMPLICIT SENSING AND NOTIFICATION TECHNIQUES
Gilbert Beyer, Gian Mario Bertolotti, Andrea Cristiani, Shadi Al Dehni

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Agenda 1 2 3 4 5 Adaptation Strategy Sensor-Actuator Support Hardware and Software Performance Tests Discussion

Page 2 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Driver Alertness

Page 3 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Causes of Driver Distraction The driver himself (fatigue, drunkeness) The vehicle (type, condition) The environment (traffic, weather)

Page 4 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Long-Term Driver Distraction Psycho-physiological state (fatigue) Secondary tasks (operating car stereo) Social activities (chatting with co-driver)

Page 5 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Feedback Loops in Vehicles

adapted from Riener and Ferscha (2009)

Page 6 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Sensing Driver Distraction Hands-off-wheel Time Eyes-off-road Time Conversational Level Emotional expressions Signs of drowsiness

Page 7 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Notification Modalities 1 Vision, Hearing, Touch (each having its own timely behavior) 2 Multimodal information

Page 8 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Adaptation Strategy Actuator: Sensor 1: Sensor 2: Sensor 3: Visual, implicit Drowsiness Head-position Can Bus data

Page 9 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Combing Car and User Data Eyes-off-road while... 1 driving fast, is dangerous 2 standing, is safe 3 turning, is required

Page 10 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Head-up Displays

http://www.bmw.com/ (2010)

Page 11 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Adaptive Head-up Displays

Karvonen et al. (2006)

Dienelt, M. (2005)

Page 12 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Adaptive Head-up Displays

Blanco et al., US Patent No. 7764247

AWAKE Project, IST-2000-28062

Page 13 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Sensing Drowsiness 1 Intrusive techniques: Electromyogram, Respiration, GSR, Electro-oculography 2 Non-intrusive techniques: Vision-based

Page 14 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Automotive Demonstrator

Page 16 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Sensor-Actuator Support

Page 17 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Simulator Setup

Page 18 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Implicit Sensing Software 1 Drowsiness Detection 2 Head Position Detection

Page 19 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Drowsiness Detection

Johns, M.W. (2003)

Page 20 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Drowsiness Detection

Johns, M.W. (2003)

Page 21 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Main Constraints

Page 22 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Main Constraints 1 Reduce computational load 2 Reduce error rate

Page 23 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Eye Detection Algorithm 1 2 3 4 Search eye using Viola-Jones If eye is found, save copy as template If eye is not found: template matching Successive failures: re-initialization

Page 24 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Eye Detection Algorithm

Largest rectangle: Viola Jones, Template Matching

Adjusted search window

Page 25 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Blink Test Algorithm 1 2 3 4 Eye-image processed to binary picture Eliminating pupil reflections and noise Obtain vertical projection histogram Calculate eye openness level

Page 26 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Blink Test Algorithm

Greyscale image

Binary image

Page 27 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Blink Test Algorithm

Open eye

Closed eye

Page 28 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Head-Position Detection

Eye within rest area

Eye outside rest area

Page 29 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Experiments and User Analysis 1 Performance Tests 2 Driver Observation in the Lab 3 Road Tests

Page 30 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Passive Sensing Performance Tests Collected sets of data: 1 Recognition of slow/fast blinks 2 Tracking of slow/fast eye movements 3 Slow/fast head movements 4 Blink fail errors

Page 31 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Passive Sensing Performance Tests Kinds of error: 1 tracking miss (in successive frames) 2 detection miss (shadows or occlusions) 3 blink miss (e.g. due to rapid blinks) 4 blink fail (erroneously detected blink)

Page 32 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Passive Sensing Performance Tests

Page 33 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Passive Sensing Performance Tests

Page 34 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Night Vision Camera

Page 35 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Lab Study

Page 36 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Lab Study 1 2 3 4 Eyes-off-road and Car State Symbols 15 Participants (students, colleagues) Co-Driver was investigator Semi-structured Interviews and Camera Observations

Page 37 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Lab Study 1 80 % understood that displayed icons related to their driving behavior 2 66 % found notifications helpful to adapt their driving style

Page 38 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Onboard Testing Session 1 Displaying Car State Information 2 Thinking Aloud Method 3 Refinement of the hardware and software setup

Page 39 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Onboard Testing Session

Page 40 of 41

AN ADAPTIVE DRIVER ALERT SYSTEM | UCPA 2010 | 06.12.2010

Discussion Algorithm Improvements Sensing Techniques Future Tests and Studies HUD Development System Design

Page 41 of 41