A Vision-Based System for Monitoring the

Loss of
Attention in Automotive Drivers

UNDER THE GUIDANCE OF-: PROF. S. B. JAMGE

M.E. ELECTRONICS
W.I.T. SOLAPUR
2014-2015
Table of Content
 Introduction
 Problem Statement & Objective
 Required hardware and software
 Block Diagram & proposed method
 Opencv
 Haar classifier
 Template matching
 Flowchart and results.
 Future scope
 References.
Introduction
 Drowsiness is one of the major cause for the accidents. Every year 10 to
15% of the accidents are due to driver drowsiness.
 The drowsiness can be defined as an transition from awake state to sleep
state
 The aim of this project is to develop a drowsiness detection system. The
focus will be placed on designing a system that will monitor the open or
closed state of the eyes in real-time.
 By monitoring the eyes, it is believed that the symptoms of fatigue can be
detected early enough to avoid a accident.
PROBLEM STATEMENT
 The proposed work is to alert the driver when he is drowsy or distracted
away from normal alert mode to non-alert mode by means of buzzer or
vibration mechanism.
 In this project real time data is collected by video camera and other micro
electro mechanical system devices (MEMS). This data gives information
about driving condition of the driver which acts as input to controller. The
appropriate measures are taken by the controller to alert the driver.
OBJECTIVES
 Our proposed method is to design and develop a low cost system, which is
based on embedded platform for drowsiness detection.
 Most existing approaches for the drowsiness detection rely either on eye
closure or head nodding angles to determine the driver drowsiness or
distraction level.
 This method combines the both eye state and head position to detect the
drowsiness of the driver.
Used hardware and software

 Hardware-
 Raspberry Pi Board
 Camera
 Accelerometer
 Buzzer
 Software-
 Raspbian OS
 OpenCV Library.
 QT as editor.
 Wiring PI library.
Generalized block diagram
POWER
MEMS SUPPLY

USB PORT ARM

ARCHITECTURE
DISPLAY

CAMERA
BUZZER
Proposed Method
 The whole system is built upon Raspberry Pi board, Raspbian OS, OpenCV image
processing library and QT as editor.
 The primary focus is given to the faster drowsiness detection and processing of data.
 The system is used to detect the eyes whether they are closed or open in real-time
through the camera.
 Here accelerometer simulates the different head position.
 The buzzer is turned on when eyes are closed and head position are above the
threshold values of the accelerometer.
Raspbian Commands and Dependency Packages

 Sudo raspi-config
 Sudo apt-get update
 Sudo apt-get upgrade
 Sudo rpi-update
 Sudo apt-get install qt4-dev-tools
 Image I/O packages-: libjpeg8-dev, libtiff4-dev, libjasper-dev, libpng12-dev.
 Video I/O packages-: libavcodec-dev, libavformat-dev, libswscale-dev, libv4l-dev
 GTK development library for GUI support.
What is OpenCV?
 OpenCV is an open source computer vision library, designed for computational
efficiency and has a high focus on real-time image processing.
 It was developed by the Intel Research Initiative along with the Media Labs of MIT
University.
 It provides simple to use computer vision platform, which allows us to build the
sophisticated vision application quickly.
 Used by most of the companies such as SONY,INTEL,MICROSOFT,SIEMENS and
GOOGLE.
 One of the most popular project based on the OpenCV is GOOGLE STREET VIEW.
Why OpenCV?
 Execution Speed is fast than Matlab.
 Consumes less memory, which is important for the embedded platform.
 Interpretation time is less, as Opencv is programmed using optimized C language.
 Supports different processor family from INTEL,AMD,ARM,SPARC & POWERPC.
 Cross platform support for different operating system.
Structure of the OpenCV

CV MLL HIGHGUI
Image processing and Statistical Classifiers and
GUI, Image and video I/O
vision algorithm Clustering tools

CXCORE
Basic structure and algorithm, XML support, drawing functions.
Haar Cascade Classifier
 A Haar Classifier is a machine learning approach for visual object detection originally
developed by Viola & Jones. It was originally intended for facial recognition but can be
used for any object.
 Haar Cascade classifier has two stages-: training and detection of the object.
 The haar classifier will quickly reject regions that are highly unlikely to contain the
object, and the object detection can move on to the further region. It does this by
making use of the cascade of classifiers.
 The classifier outputs ‘1’ when it finds an object and ‘0’ otherwise.
 The necessary applications for implementing a Haar classifier are included in OpenCV
and these can be used to train a classifier for detecting objects in an image.
Cascade Classifier Training
 There are two types-: open_haartraining & open_transcascade. The open_transcascade
support both the haar and Local binary pattern (LBP).
 Steps in training and using a haar classifier-:
 Collect positive and negative training images.
 Markup positive images using objectmarker utility.
 Create a .vec file using createsamples utility.
 Train the classifier using haartraining utility.
 Run the classifier using cvhaarDetectObject().
Cont’d
 Collect the training images- generally about 1000 images are required to train the
classifier. It may include both positive and negative images.
 Mark positive images- This step creates a data file containing the file name and the
location of the object in the image. The data file is created using the object marker
utility.
 Create the vector file- It can be created using the createsamples utility of the opencv,
optionally we can set the –width and –height.
 Train the classifier- The classifier is trained with the utility opencv_haartraining.
 The final output will be .XML (extended markup language) file.
Cont’d
 The final output of the classifier after training is an .xml file which has vector values of
the objects to be detected. For example-:
<threshold>5.6097120977938175e-003</threshold>
<left_val>-0.7382487058639526</left_val>
<right_val>0.3885168135166168</right_val>
Examples of positive images

Examples of negative images

Template matching Technique
 There are two components-: source image(I) and template image (T).
 The template matching is a technique for finding the area’s of an image that match to a
template image (patch).
 To identify the matching area we have to compare the template image against the
source image by sliding it.
 For each location Of T over I, we store the metric in the result matrix (R). Each location
(x,y) in (R) contain match metric.
Detailed process of template matching.
 Loads an input image and a image patch (template)
 Perform a template matching procedure by using the OpenCV function
‘matchTemplate’
 Normalize the output of the matching procedure
 Localize the location with higher matching probability using minMaxLoc.
 Draw a rectangle around the area corresponding to the highest match

Method = CV_TM_SQDIFF

R(x,y) = 𝑥 ′ ,𝑦′(𝑇 𝑥 ′ , 𝑦 ′ − 𝐼(𝑥 + 𝑥 ′ , 𝑦 + 𝑦′))2

Method = CV_TM_SQDIFF_NORMED

𝑥′ ,𝑦′
(𝑇 𝑥 ′ ,𝑦 ′ − 𝐼(𝑥+𝑥 ′ ,𝑦+𝑦′))2
R(x,y) =
𝑥′ ,𝑦′
𝑇 𝑥 ′ ,𝑦 ′ 2 . 𝑥′ ,𝑦
𝐼(𝑥+𝑥 ′ ,𝑦+𝑦′)2
Project implementation
Cont’d
 The system comprises of 3 phases-
 Capturing- includes video capturing through camera.
 Detection- It includes the detection, whether eyes are closed or open and also
calculates the head position.
 Correction- If eyes are closed and head position is above threshold value the buzzer
turns on to alert the driver.
 Here eyes are located with the help of trained cascade classifier.
 Flowchart
Start

No
Capture video All feature
detected?

Divide video into Yes

frames
Save eye
Fetch frames template

Locate eye Haar cascade A

region samples of eyes

Eyes detection
B
 A B
Template
matching

Eye tracking

Drowsiness
Detection

Fatigue/Non
fatigue More yes
frames to
process?

No

End
Results and Discussion
Different head position’s

Figure shows x=19, y=3, z=17 head Figure Shows x=58, y=56, z=20
position= right Head position=left
Results of Eye Blink Detection

Eyes open Eyes Closed

Conclusion
 We have designed the drowsiness detection system, built upon raspberry pi as
development board.
 The system uses open source software called as OpenCV image processing libraries,
which is very much suitable for the embedded board.
 The Haar classifier are used for locating the eyes.
 Raspberry pi and opencv makes the overall system a low cost drowsiness detection
system.
Future Scope
 Use of DSP processor makes the system more suitable for faster processing for the
faster processing of data.
 Night vision or near infrared camera can be used instead standard camera that can
work even in low light condition and during night.
 An improvement in the algorithm can be made to detect the eyes much faster and
results would be more accurate.
References
 R. N. Khushaba, S. Kodagoda, S. Lal, and G. Dissanayake, “Driver drowsiness
classification using fuzzy wavelet-packetbased featureextraction algorithm,” IEEE Trans.
Biomed Eng., vol. 58, no. 1, pp. 121–131, Jan. 2011.
 [2] Y. Liang, M. L. Reyes, and J. D. Lee, “Real-time detection of driver cognitive
distraction using support vector machines,”IEEE Trans. Transp.Syst., vol. 8, no. 2, pp.
340–350, Jun.2007.
 [3] R. I. Hammoud, G. Witt, R. Dufour, A. Wilhelm, and T. Newman, “On driver eye
closure recognition for commercial vehicles,” SAE Int. J. Commercial Veh., vol. 1, no. 1,
pp.454–463, Apr. 2009.
 [4] Goel, P, et.al. "Hybrid Approach of Haar Cascade Classifiers and Geometrical
Properties of Facial Features Applied to Illumination Invariant Gender Classification
System," Computing Sciences (ICCS), 2012 International Conference on, vol., no.,
pp.132, 136, 14-15 Sept. 2012.
 [5] Eye Detection Using Morphological and Color Image Processing, Tanmay
Rajpathaka, Ratnesh Kumar and Eric Schwartz 2009 Florida Conference on Recent
Advances in Robotics, FCRAR 2009.
Thank You