A Drowsy Driver Detection and Security System

Rajat Garg, Vikrant Gupta, Vineet Agrawal

Department of Electronics and Communication VIT- University, Vellore- 600014, India {rajatgarg2006, vikrantgupta2006, vineetagrawal2006} @ vit.ac.in
Abstract This paper presents a new approach towards automobile safety and security. We propose three distinct but closely related concepts viz. an Iris Recognition system, a Drowsy Driver Detection system and a Distress Signalling system using non-intrusive machine vision based concepts. In recent times automobile theft and fatigue related crashes have really magnified. In order to minimize these issues, we have incorporated Biometric security through Iris recognition that will help in authentication and an improved sleep detection and driver alert system by monitoring both the drivers eyes as well as senseing the heat variation of the body via infrared thermal sensor. Distress Signalling system is incorporated for drivers to get assistance from the Police in need without revealing it to people present around him. This paper combines computer vision, pattern recognition and optics. All image processing was performed using NI Vision Assistant. Also NI LabVIEW was used to take the current body temperature from the temperature sensors attached to the DAQ (Data Acquisition) Signal Accessory. Keywords: - Image processing; pattern matching; iris recognition; biometrical control systems; distress call; drowsiness detection.

I.

INTRODUCTION

Automation offers lifestyle improvement through the combining of various systems into a unified, familiar, intuitive and non-threatening solution. Due to larceny and the urge to live a more comfortable life, it is anticipated that more and more people will prefer to choose the Smart Car solution for the range of benefits offered. Biometric identification technologies, including facial recognition, fingerprint recognition, speaker verification and so on, offer a new solution for personal identification due to natural manner and secure access which can be used in design of anti-theft car systems. Among the biometric technologies, iris recognition is one of the best measures for person identification and security because of the following advantages: (1) Human iris is not changeable and is stable. From one year of age until death, the patterns of the iris are relatively constant over a persons lifetime [1, 2]. (2) The forming of iris depends on the initial environment of embryo, so the iris texture patterns dont correlate with genetic determination [3]. Even the left and right irises for a given person are different from each other. (3) It is almost impossible that irises are modified by surgery without risk. 9781-4244-3941-6/09/$25.00 2009 IEEE

(4) The iris recognition is non-invasive and dont cause the damage to identifier. Driver fatigue is a significant factor in a large number of vehicle accidents. Recent statistics estimate that annually 1,200 deaths and 76,000 injuries can be attributed to fatigue related crashes [4]. The development of technologies for detecting or preventing drowsiness at the wheel is a major challenge in the field of accident avoidance systems. Because of the hazard that drowsiness presents on the road, methods need to be developed for counteracting its affects. Distress signaling system is another important feature requisite for smart cars. Some conditions can arise for the user for which he might need assistance of the police. It may also be required for the driver to give a distress call for assistance without revealing it to the people present around him. Hence there is a need for a system which can get distress signal from the user in a secretive way and pass it on to the concerned authorities. Any movement of the hands or other parts of the body to make such a signal can attract the attention of the surrounding people and hence are not appropriate for the task. This situation can be tackled by the usage of eyes. While driving a car it is hard for someone to look directly into the eyes of the driver and hence these are hidden even from the person sitting beside the driver. We propose to make such a system by giving a sequence of gestures from the eyes. This paper is organized into three main sections. Firstly the methodology involved in Biometric security using iris recognition has been described. This is followed by a detailed elucidation of the principles involved in Drowsy Driver Detection system and Distress signaling system. The final section presents the test results of our experimentations. II. SYSTEM CONFIGURATION

Fig. 1 System Configuration

The system consists of a CMOS web camera, image processing unit, decision making unit and a Data Acquisition (DAQ) unit. Firstly, image acquisition of the eye is done and subsequent image processing is performed using NI Vision Assistant and NI LabVIEW. Appropriate information in the gestures is extracted and issued as commands. Fig 1 demonstrates the general system setup. The Data Acquisition (DAQ) Unit is used to issue these commands to the Alarm/GPS module depending upon the gesture given. Also equipments such as infrared thermal sensors for monitring and evaluation of different body temperatures are retrofitted as shown in fig. 1 for enhanced security. We have used a PC with Pentium IV, 3GHz processor and 512MB RAM to perform our experiments. III. METHODOLOGY The camera is made to monitor the eyes constantly. The camera initially performs iris recognition to check for the authentication of the user and only if he is an authentic user, he will be allowed to operate the automobile. Upon verification of his authentication, image and temperature acquisition from camera and sensors takes place for processing. Depending on the results of processing, commands are issued as seen from fig.2.

reduced, and hence the signal is sent for triggering the alarm. 3) Distress signaling State- The driver needs the assistance of the police. His one eye is closed and other is open, and hence the signal is sent to the GPS module. The advantage of using such a system is that the safety and security features are implemented using a single camera and hence has a compact housing that can be placed around the rearview mirror or on the dashboard IV. IRIS RECOGNITION Biometric recognition offers a promising approach for security applications, with some advantages over the classical methods, which depend on something you have (key, card, etc.), or something you know (password, PIN etc). A nice property of biometric traits is that they are based on something you are or something you do, so you do not need to remember anything or to hold any token. Iris recognition security system combines computer vision, pattern recognition, statistical inference, and optics. Because the randomness of iris patterns has very high dimensionality, recognition decisions are made with confidence levels high enough to support rapid and reliable exhaustive searches through national-sized databases [5]. The three main stages of an iris recognition system are image preprocessing, template matching and authentication. The iris image needs to be preprocessed in order to obtain useful iris region. Image preprocessing is done by the process of Iris Segmentation which localizes the correct iris region in an eye image. It is critical to the success of subsequent feature extraction and template matching stages. Different templates of iris are extracted and stored in a database. These templates are then used to verify the persons identity. Fig. 3 shows the flowchart of our method.

Fig. 3 Flowchart illustrating the process of template matching. Fig. 2 Flowchart describing basic working of our system

There are three states of operation of the system: 1) Normal State- The driver is awake. His eyes are open and his body temperature is normal, and hence the process of image acquisition from the camera continues. 2) Drowsiness detection State- The driver is in a state of drowsiness. His eyes are closed and his body temperature is

A. Image Acquisition: It is a major challenge to obtain a high quality image of eye with sufficient details while retaining the system to be non-invasive. There are many constraints in implementing such a system. However such a system with the desired characteristics up to an extent can be implemented. The images used in this analysis are taken from the database. The image is then changed from RGB to gray level for

further processing. Fig. 4(a) and fig. 4 (b) illustrates the process.

Fig. 7(d) Fig. 7(e) Fig. 7{(a),(b),(c),(d),(e),(f)} shows the process of iris recognition using template matching Fig. 4(a) Original image Fig. 4(b) Gray level image

B. Preprocessing We segment the iris from the image boundaries by detecting the centers of the iris. Since its boundary is circular and darker then the surrounding portion, it can be detected using the circular edge detection function. After detecting the boundary, the iris portion is then extracted from the remaining image. This can be done using a circular mask of radius equal to circle which we found using circular edge detection function. Fig. 5(a) and fig. 5(b) illustrates the process.

While applying template matching to a very large set of data it is possible that templates of one portion of an image finds high correlation with any other portion on some different image. To remove this ambiguity a constraint of localization is applied to the templates. This is done by ensuring that all the templates which are taken from any person have their centres on a circle with a very little deviation allowed. Hence two or three incorrect template matching will not affect our system result. V. DROWSY DRIVER DETECTION AND DISTRESS SIGNALING

Fig. 5(a) Iris detection

Fig. 5(b) Iris extraction

C. Feature Extraction The pattern of pigment spots present in iris is different for every person (even uniovular twins doesn`t have same iris pattern [6]). These patterns are employed to create templates for iris recognition. More than three templates are created for each iris for better accuracy. The templates we create are such that template centers are cyclic i.e. lies on a circle. Fig. 6 shows the four templates of the iris.

Fig. 6 Templates

D. Iris Recognition On image acquisition, the system tries to match the stored templates with the given image. If more than three templates fits the given image and their centers lies on a circle, then person is recognized as an authentic user. Fig. 7{(a),(b),(c),(d)&(e)} shows the process of template matching and circle formation.

We are presenting an improved sleep detection and driver alert system that monitors both the drivers eyes as well as senses heat variation of the body via infrared thermal sensor. The variation in temperature of the body is an indicator for onset of sleep or drowsiness. The advantage of the system is that the safety and security features are implemented using a single camera and hence has a compact housing that can be placed around the rearview mirror or on the dashboard. The camera is calibrated to a short distance of one to two feet to monitor the open or closed state of the drivers eyes. An infrared thermal sensor detects heat loss/gain change in body temperature. The temperature change will be due to exhaled gas plume of normal breathing patterns, which will lower in volume as the driver begins to hypoventilate, thus increasing their blood level of carbon dioxide which is in most part the reason for early drowsiness associated with sleep. The combination of closer of eyes and a decrease in body temperature, which is a physiological response to hypoventilation thus initiating drowsiness, will trigger the alarm. Distress signal is passed on by the user whenever he feels a need to get some assistance from the Police. A peculiar gesture is assigned for such a purpose so that it does not occur normally. In the proposed method we choose a gesture with one eye closed and the other open. When this continues for two seconds a distress call is sent to the police. The recognition of such a gesture is done by template matching. Hence this signal can be replaced by another one just by changing the template. The position of the car can be tracked with the help of tracker attached to the car. A. Drowsiness and Distress signal detection from drivers eyes The focus is on designing a system that will accurately monitor the open or closed state of the drivers eyes in real-

Fig. 7(a)

Fig. 7(b)

Fig. 7(c)

time. By monitoring the eyes, it is believed that the symptoms of driver fatigue can be detected early enough to avoid a car accident and also distress signal given by the driver can be detected. Detection of fatigue involves a sequence of images of a face, and the observation of eye movements and blink patterns. The image acquired is processed, as explained in the preceding section, to find the drivers eye status [6, 7, 8, 9].

Fig 10(a) Face template matching

Fig. 10(b) Extracted face area

3) Eye Detection: The eyes are extracted from the face using the same template matching concept as explained above. This time template of eyebrows is used. Since eyebrows do not change much, so the minimum threshold level is set around 800. Fig 11 illustrates the process.

Fig. 11(a) Eyebrow template matching Fig. 11(b) extraced Eye area

4) Template Matching: Now a decision is made about the state of the system from the results of processing. This is done by comparing distress signal and closed eye templates with the extracted eye portion. Decision is taken accordingly. Following figures shows the distress call and closed eye template matching procedure.
Fig. 8 Flowchart describing the principle of driver fatigue and distress signaling

1) Image Acquisition: The Image acquired from the Camera is converted to grayscale by extracting the luminance plane of it. Fig 9 illustrates the process

Fig. 12(a) Distress call template matching with the extracted eye region

Fig. 12(b) Closed eye template matching with the extracted eye region.

Fig. 9 (a) Original image

Fig. 9 (b) Grayscale image

2) Face Detection: Initially the face of the user is extracted using template matching. In template matching, template from the database is compared with the given image using a matching metric. The matching metric will give a measure of similarity between the two face templates. This similarity is converted into a numerical value designated as score of the template. Original template has a score of 1000 i.e. 1000 means exact match. Since, we are taking images in different condition; hence we set the minimum threshold score to be around 600. Once the template is matched, the coordinates of its center will be known. Using the center coordinates and knowing the width and height of the template a mask is created and face portion is extracted. Fig 10 illustrates the process.

B. Drowsiness detection by variation of body temperature The focus is on designing a system that will accurately monitor the temperature of the driver in real-time. The temperature variation of the body is an indicator of the onset of sleep or drowsiness. An infrared thermal sensor detects heat loss/gain change in body temperature. The temperature change will be due to exhaled gas plume of normal breathing patterns, which will lower in volume as the driver begins to hypoventilate, thus increasing their blood level of carbon dioxide which is in most part the reason for early drowsiness associated with sleep. By monitoring the eyes, it is believed that the symptoms of driver fatigue can be detected early enough to avoid a car accident. The temperature acquired is processed, as explained in the preceding section, to find the drivers status. 1) Acquisition State: The current body temperature is taken from the temperature sensors attached to the DAQ (Data Acquisition) Signal Accessory every half second and

is plotted against time. Fig. 13 shows the Acquisition State block diagram.

Fig. 15(b) Time Check State True Case Fig. 13 Acquisition State

VI.

ILLUMINANCE

2) Data Log: The temperature acquired in the acquisition state is stored in the memory in one dimensional array format. Fig. 14 shows data log state.

A correct illumination is crucial in insuring that the image has the correct amount of contrast to allow the correct processing of the image. In case of the drowsy driver detection system, the light source is placed in such a way that the maximum light being reflected back is from the face. The drivers face is illuminated using a 60W light source. To prevent the light source from distracting the driver, an 850nm filter is placed over the source. Since 850nm falls in the infrared region, the illumination cannot be detected by the human eye, and hence does not agitate the driver. VII. EXPERIMENTAL RESULTS We developed an image database for real testing and to discover the efficiency of the implemented algorithm. Iris Recognition and Drowsy Driver Detection systems were tested separately. For Drowsy Driver detection system, five different drivers with different skin colors, eye sizes, facial positions and luminance conditions were taken. The camera was placed in the same position at which the actual camera of our system will be placed to guarantee reliability, reality and to decrease the probability of having unexpected bugs at the implementation. Our system takes a stream of continuous input images as frames that contain the face of the driver in different head positions to test the performance of our algorithm. Acceptable eye detection is performed with efficiency up to 90%. We used a computer with a 3GHz processor and 512MB RAM for our miniature prototype model. The image processing algorithm required about 379.50 ms and 67.24 ms for iris detection and drowsiness detection respectively on an average. Fig. 16 and fig. 17 show the performance meters obtained using NI Vision. The NI Vision scripts which were prepared for iris detection and drowsiness detection are shown in fig. 18 and fig. 19 respectively. Fig. 20 and fig. 21 shows front panel of iris detection and drossiness and distress call detection. Final LabVIEW VI for iris detection and drowsiness detection using image processing and temperature sensor input are shown in fig. 22 and fig. 23.

Fig. 14 Data Log State

3) Time Check: In this state, difference of the current temperature is taken from the preceding 10th stored value. Since the temperature is acquired every half second, so the difference will correspond to the difference in body temperature of the person in every 5 seconds. This difference can be used to check whether the body temperature of driver is decreased or not. If the calculated difference is greater than 0.5oC, it can be inferred that the driver is in a state of drowsiness. It also checks if half second has elapsed or not, in order for it to acquire the next reading. Fig. 15(a) and (b) shows two cases of time check state.

Fig. 15(a) Time Check State False Case

Fig. 16 Performance Meter for Iris identification.

Fig. 21 Front panel of VI for drowsiness and distress call detection

Fig. 17 Performance Meter for Drowsiness and distress signal detection

Fig. 18 NI vision script showing various processing stages of Iris identification.

Fig. 19 NI vision script showing various processing stages of drowsiness and distress call detection

Fig. 22 LabVIEW VI block diagram for iris identification

The Iris Recognition algorithm was tested for a set of images and the results obtained were found to be encouraging. The results can be described in terms of two parameters defined as follows CCR-Correct Classification Rate- Images of the people from within the database were presented to the system and was searched for a match. FAR-False Acceptance Rate- Images of the people outside the database were presented to the system and was analysed for number of matches. A database of templates for ten iris images was created and was matched with 150 different images from the database. The results were found to be very accurate for both CCR and FAR. Results obtained are tabulated below.

Fig. 20 Front Panel of VI for person authentication

Template Matching

CCR % 100

FAR % 0

Fig. 23 LabVIEW VI block diagram of drowsiness and distress call detection using image processing and temperature sensor inputs

VIII. COMPARISION WITH EXISTING METHOD The iris recognition technique is better than the fingerprint and face recognition techniques because of some of the inherent disadvantages of them. Some people may feel offended about placing their fingers on the same place where many other people have continuously touched. Some people have damaged or eliminated fingerprints. Fingerprint recognition performance is significantly influenced by fingertip surface condition, which may vary depending on environmental or personal causes [13]. In face recognition technique the main disadvantage is the variation of face with the age. Existing techniques for face detection in colour images are plagued by poor performance in the presence of scale variation, variation in skin colors, complex backgrounds etc [14]. IX. SUMMARY AND FUTURE WORK In this paper, we presented a new method for automobile safety and security. Experimental results confirm that it is a reliable and robust method. It addresses the concept of ambient intelligence (ami), which by combining computing and networking technologies, seeks to make the environment more sensitive, secure and intelligent. Iris

recognition method is more secure than other methods. In other words, false identification is less than others. We plan to use infrared camera to measures and determine changes in heat output from changes in respiration to make our system more robust. REFERENCES
[1] [2] [3] Biometrics: Personal Identification in a Networked Society. A. Jain, R. Bolle and S. Pankanti, eds. Kluwer, 1999. J. Daugman, Biometric Personal Identification System Based on Iris Analysis, United States Patent, no.5291560, 1994.. John Daugman, Neural Image Processing StrategiesApplied in RealTime Pattern Recognition, Real-TimeImaging, 1997, 3(3): pp 157171.. Weirwille, W.W. (1994). Overview of Research on Driver Drowsiness Definition and Driver Drowsiness Detection, 14th International Technical Conference on Enhanced Safety of Vehicles, pp 23-26.. J. Daugman. The importance of being random: Statistical principles of iris recognition. Pattern Recognition, vol. 36, no. 2, pp.279-291, 2003. Perez, C.A.; Palma, A.; Holzmann, C.A.; Pena, C.. Face and eyetracking algorithm based on digital image processing. Systems, Man, andCybernetics, 2001 IEEE International Conference on Volume 2, 7-10 Oct. 2001 Page(s):1178 - 1183 vol.2. Tianjian Liu, Shanan Zhu, Eyes detection and tracking based on entropy in particle filter. Control and Automation, 2005. ICCA '05.

[4]

[5]

[6]

[7]

International Conference on Volume 2, 26-29 June 2005 Page(s):1002 - 1007 Vol. 2 [8] Ito, T., Mita, S., Kozuka, K., Nakano, T., Yamamoto, S.. Driver blink measurement by the motion picture processing and its application to drowsiness detection. Intelligent Transportation Systems, 2002. Proceedings. The IEEE 5th International Conference on 2002 Page(s):168 173 [9] Perez, C.A.; Lazcano, V.A.; Estevez, P.A.; Estevez, C.M.. Real-time iris detection on faces with coronal axis rotation. Systems, Man and Cybernetics, 2004 IEEE International Conference on Volume 7, 1013 Oct. 2004 Page(s):6389 - 6394 vol.7. [10] Hayashi, K.; Ishihara, K.; Hashimoto, H.; Oguri, K.; Individualized drowsiness detection during driving by pulse wave analysis with neural network. Intelligent Transportation Systems, 2005. Proceedings. 2005 IEEE 13-15 Sept. 2005 Page(s):901 906

[11] Ilkwon Park; Jung-Ho Ahn; Hyeran Byun. Efficient Measurement of Eye Blinking under Various Illumination Conditions for Drowsiness Detection Systems. Pattern Recognition, 2006. ICPR 2006. 18th International Conference on Volume 1, 20-24 Aug. 2006 Page(s):383 386 [12] M.L. Berliner, Biomicroscopy of the Eye. Paul B Hoeber, Inc.1941. [13] Koichi Ito, Ayumi Morita, Takafumi Aoki, Tatsuo Higuchi, Hiroshi Nakajima, and Koji Kobayashi A Fingerprint Recognition Algorithm Using Phase-Based Image Matching for Low-Quality Fingerprints [14] An Efficient Skin Illumination Compensation Model for Efficient Face Detection C . N. Ravi Kumar , Bindu . A Department of Computer Science and Engineering, Sri Jayachamarajendra College of Engineering, Mysore 570006, Karnataka State, India