Sunteți pe pagina 1din 5

2012 Fourth International Conference on Digital Home


Atish Udayashankar, Amit R Kowshik, H S Prashanth
Chandramouli S Dept. of Telecommunication Engg.
Dept. of Telecommunication Engg. PES Institute of Technology
PES Institute of Technology Bangalore, India
Bangalore, India,,

Abstract - Paralysis is defined as the complete loss of muscle to call for help if required or switch on/off a light, a fan or
function in any part of the body. It occurs when there is a any other appliance. Hence, our application will help the
problem with the passage of messages between the muscles and patient to be self-sufficient to a certain extent.
the brain. Some paralyzed people cannot move even a single
There are innumerable applications which can be
part of the body other than their eyes. Hence, the main aim of
this paper is to design a real time interactive system that can derived from eye blink detection and these are not limited
assist the paralyzed to control appliances such as lights, fans for usage by only paralyzed people. An efficient, real time
etc. or by playing pre-recorded audio messages, through a blink detection algorithm can be used for almost any
predefined number of eye blinks. Image processing techniques purpose. It can be used for switching on/off appliances such
have been implemented in order to detect the eye blinks. In our as a television or a microwave oven. It can also be used to
system, the face tracking is accomplished by using a set of send an email or call someone on Skype. All this can be
trained Haar cascade classifier, and a template matching accomplished with just a few eye blinks.
technique is employed to track the eye. Initially, the Innumerable techniques have been devised for face
involuntary blinks of the paralyzed person are used to locate tracking. CamShift face tracking algorithm, Haar face
the patient’s eyes by finding the number of connected tracking algorithm and face tracking using Eigenfaces are
components in a frame. Once the eyes are detected, an online
some of these. There are some techniques devised for blink
template is created which is then used to track the patient’s eye.
detection as well. Some of them are software-oriented i.e.
Keywords - Paralyzed patients, template matching, Haar using image processing, and a few are hardware-oriented
cascade classifier, eye blinks, face tracking using sensors. Some of the image processing techniques
include blink detection using Gabor filters, contour
I. INTRODUCTION extraction, and eye blink detection using Median blur
filtering. The hardware-based approaches are primarily
Paralysis is the complete loss of muscle function for one using infrared and magnetic sensors [4, 7]. The advantage
or more muscle groups. Paralysis can cause loss of feeling of using sensors is that the entire system would be more
or mobility in the affected areas. Paralysis can be localized, compact. However, risks involving the safety of the eye are
or generalized, or it may follow a specific pattern. Most too high and outweigh the advantages of a sensor system. If
paralyses caused by nervous system damage (i.e. spinal cord the frequency of the infrared light emitted by the sensor is
injuries) are constant in nature; however, there are forms of outside the visible spectrum, it could cause permanent
periodic paralysis, including sleep paralysis, which are damage to the eyes.
caused by other factors. Paralysis is most often caused by The primary purpose of this paper is to propose a system
damage to the nervous system, especially the spinal cord. that can assist the paralyzed. It does so by tracking the
Major causes are stroke, trauma with nerve injury, person’s eye and counting the blinks, and employs this
poliomyelitis, amyotrophic lateral sclerosis (ALS), count to control various appliances and play pre-recorded
botulism, spina bifida, multiple sclerosis, and Guillain- audio messages.
Barré syndrome. According to a survey by the Christopher Although a number of techniques have been
and Dana Reeve foundation, nearly 1 in every 50 people are implemented for eye blink detection, there is no application
paralyzed [2]. that has been developed to actually put the blink detection
Fully paralyzed patients require 24 hour support. But in to practical use. The principal contribution of this paper is
the present day and age, it is not possible for anyone to be
the conceptualization of a system which will go a long way
available at all times. So in those situations where the
in helping the paralyzed and disabled to achieve some level
patient is alone in a room, he/she could use this application
of independence. Moreover, the algorithm provided by [1]

978-0-7695-4899-9/12 $26.00 © 2012 IEEE 104

DOI 10.1109/ICDH.2012.9
has been improved upon, by incorporating face tracking, so outline the face region, and the coordinates of the rectangle
as to reduce the effect caused by movements in the is provided to the next stage.
II. BLINK DETECTION B. Initialization
There are a number of image processing algorithms for In the first step, the difference between the current frame
eye blink detection. A brief overview of three of these and previous frame is found out. This difference image is
algorithms is provided. then thresholded. A threshold value of 5 was chosen. This
value was derived after some experimental trials.
A. Contour Extraction Next, a structuring element is passed over the binary
In this technique, a set of 16 landmarks are created at image in order to remove the noise. The structuring element
regular intervals to outline the contour of the eye. Eight used here is a 3x3 star shaped convolution kernel. This is
points are used to represent each eye. The distance between done in an open morphological operation. This operation
the highest and lowest landmark is denoted by d 1, and the first performs erosion of the image followed by dilation
distance between the centroids of the two eyes is denoted by using the same structuring element for both the operations.
d2. Now d1/d2 is computed, and assigned to a variable D. The primary effect of the opening operation is to preserve
Now the value of D is used to distinguish between open eye the foreground regions that have a shape similar to the
and closed eye. Generally, a value of D equal to 0.158 structuring element or which can completely contain the
implies open eye and a value equal to 0.016 implies closed element. The other regions of foreground pixels are
eye. These values have been experimentally derived by [3]. eliminated.
The next step is to find the number of connected
B. Gabor Filter components in the binary image. In order to do this, a
The Gabor filter is used to extract arcs of the eye. Here, recursive labeling procedure is used. The lesser the number
the eye region is first extracted and then the filter is applied of connected components detected indicates that the
to obtain the arcs of the eye. Then connected component algorithm is working as intended. In case, the number of
labeling method is used to detect the top-bottom arcs. The connected components detected is large in number, then the
distance between the arcs is measured to determine the binary image being used is discarded and the system
blinking [5]. reinitializes. This step helps in preserving the efficiency of
the system. For an eye pair, the number of connected
C. Median Blur Filtering components must be two.
In this method, the image of the eye is first thresholded If the image from the previous steps results in a small
and then a median blur filter is applied to it. The resultant number of connected components, the next step is to
image obtained after applying the filtering shows a clear determine which of these connected components constitute
difference between the open and the closed eye, and hence an eye pair. This filtering to determine possible eye pairs is
helps in identifying eye blinks [6]. done by applying experimentally derived heuristics. The
parameters used include: the vertical and horizontal
III. ALGORITHM USED IN THE PAPER distance between the centroids of the components and the
width and height of each of the components. If any of these
The algorithm used for eye blink detection is carried out parameters result in values which would normally not be
in five phases as shown in Figure 1. The first phase involves associated with an eye pair, then it can be concluded that the
face tracking in order to reduce the search window size. The connected components found do not constitute an eye pair.
second phase involves identifying the location of the eye by On the connected components having met the conditions as
detecting involuntary eye blinks. The third phase involves imposed by the heuristics, the system proceeds to open eye
creation of an online template of the open eye. The fourth template creation.
phase is eye tracking and the final phase is the detection of
eye blinks. The algorithm is initialized automatically i.e. if C. Open Eye Template Creation and Matching
the location of the eye is lost, then the algorithm reinitializes For the creation of an open eye template, the larger of
on its own to locate the eye. The system has a timer the two connected components is chosen. This is done
programmed to run for a period of ten seconds during which primarily because the larger connected component contains
the patient will need to blink a desired number of times. more brightness information and this helps in maintaining
Finally after the timer runs out, the number of eye blinks is the efficiency of the system. The open eye templates of
used to perform the desired action, if it corresponds to a pre- various sizes and orientations are shown in Figure 2.
set value.

A. Face Tracking
Initially, the Haar face detector is loaded into the
program. The input image is then converted from
multichannel to single channel for faster processing. The
face region is then detected. A rectangular box is drawn to

Figure1. Block diagram of the system- Block diagram representing the model of the entire system
0.4. Hence, if the value of R(x,y) exceeds 0.4, it is then
realized that it is not an eye and the re-initialization starts.

D. Blink Detection
The final step is the detection of the eye blink. In order
to accomplish this, the concept of motion analysis is used.
This step is very similar to the step where the user’s eyes
Figure 2. Open eye templates - Open Eye templates of various sizes used are located, except here the search window size is reduced.
for template matching In order to decide if there is an eye blink or not, two rules
The next step is to locate the eye using the created open are applied. The first one is that there must be only one
eye template. The method used to compare the current connected component. The second rule being that the
image of the eye and the previously created template is the component must be located at the centroid of the user’s eye.
normalized value of the square of the difference between the IV. RESULTS
brightness of the current frame and the template. The The system was primarily developed and tested on a
formula used is given below. Windows 7 PC with an Intel Core i7-2670QM CPU 2.20
GHz processor and an 8 GB RAM. The video was captured
with a Creative camera built in the laptop. The video was
captured at 30 frames per second and was processed as
grayscale images. These images were of a size of 320 x 240
pixels. Many functions from the Intel OpenCV library were
In the above formula [8], T(x,y) represents the utilized. The code was written in Microsoft Visual Studio
brightness of the video frame at the point x,y.I(x,y) the 2008 and the GUI was developed using the Visual Studio
brightness of the template at the point x,y. As the function MFC application.
computes the values of R(xy),i.e. the normalized square The user is required to assign the number of blinks
difference, for various images, the best matches are found which correspond to the different appliances and other
as global minimums. Here, the minimum threshold value, functions. This has to be set every time the eye blink
after performing experimental trials, has been chosen to be detection system is switched on. The number of blinks

corresponding to any application can be changed at any wearing glasses and another was with the user not wearing
given time. For this purpose, a simple user interface was glasses. Efficiency was also found based on the distance of
created where the user can enter values corresponding to the user from the camera. Yet another basis which was
different appliances or functions before starting the blink employed to obtain the efficiency was to remove the face
detection system. The following figure is a screenshot of the tracking algorithm from the main algorithm. As a result, the
graphical user interface developed. efficiency of the system improved considerably. Our
observations are tabulated as follows.

Table I. Without face tracking algorithm

Total number of blinks Number of blinks detected
20 19
20 14
20 14
20 17
20 20
Efficiency: 84%
Figure 3. The graphical user interface - The graphical user interface used to
set the number of blinks for a particular action Table II. Without face tracking algorithm and user is wearing glasses
Total number of blinks Number of blinks detected
20 13
20 14
20 15
20 14
20 17
Efficiency: 73%
Table III. Without face tracking algorithm and user is far from the
screen (90cm)
Total number of blinks Number of blinks detected
20 18
20 17
20 15
20 15
20 17
Figure 4. Blink detection without face tracking Efficiency: 82%

Table IV. With face tracking algorithm and people moving in the
Total number of blinks Number of blinks detected
20 8
20 7
20 12
20 8
20 10
Efficiency: 45%
Figure 5. Blink detection with face tracking and movement in the
From the above tabulations, it can be concluded that the
A number of trials were performed to find out the level algorithm works more efficiently when the user is not
of efficiency of the algorithm under various circumstances. wearing spectacles. Moreover, there is a very minute
The efficiency of the algorithm was computed, taking difference in the efficiency of the program with a change in
several cases into consideration. One case was with the user distance of the eyes from the screen. It is also observed that

there is a remarkable fall in the efficiency of the algorithm Science Technical Report No. 2005-12. Boston, USA
when face tracking is also implemented. However, if the [2] Christopher and Dana Reeve
face tracking is not included, the efficiency of the algorithm Foundation
with people moving in the background is nearly zero.
[3] Liting Wang , Xiaoqing Ding , Chi Fang , Changsong Liu , Kongqiao
Wang ,2009. Eye Blink Detection Based on Eye Contour Extraction.
V. CONCLUSION Proceedings of SPIE-IS&T Electronic Imaging. San Jose, CA, USA.
Although blink detection systems exist for other [4] M. Takagi, K. Mohri, M. Katoh and S.
purposes, an implementation of a blink detection system Yoshino,1994.MagnetDisplacement Sensor Using Magneto-Inductive
with the end use of controlling appliances has not been Elements for Sensing Eyelid Movement. IEEE Translation Journal
On Magnetics In Japan, Vol. 9,No.2, pp 78-83.
previously accomplished. While the system is intended to
[5] Kohei Aai and Ronny Mardiyanto, 2011. Comparative Study on
assist the paralyzed and physically challenged, it can Blink Detection and Gaze Estimation Methods for HCI, in Particular,
definitely be used by all types of individuals. The main Gabor Filter Utilized Blink Detection Method. Proceedings of Eighth
challenge involved in the implementation of the system is International Conference on Information Technology: New
the development of a real time robust blink detection Generations. Las Vegas, USA, pp. 441-446.
algorithm. Many algorithms have been developed to serve [6] Chinnawat Devahasdin Na Ayudhya, Thitiwan Srinark, A Method for
the purpose, with some being more accurate than the others. Real-Time Eye Blink Detection and Its Application
This paper presented a blink detection system based on [7] Cihan Topal, Ömer Nezih Gerek and Atakan Doan,2008. A
online template matching. HeadMounted Sensor-Based Eye Tracking Device: Eye Touch
System. Proeedings of the 2008 Symposium on Eye tracking research
The first phase involved the blink detection phase; the & applications. Savannah,GA,USA, pp. 87-90.
second phase involved the counting of blinks and [8] OpenCV documentation
subsequent control of appliances through a micro controller.
By enabling the paralyzed to gain control of albeit a small [9] Lae-Kyoung Lee, Su-Yong An , Se-Young Oh, Aug 2011. Efficient
part of their lives, the system can offer some level of Face Detection and Tracking with extended camshift and haar-like
independence to them. The helpers who are assigned the features. International Conference on Mechatronics and Automation
task of tending to paralyzed persons through the day can (ICMA), 2011. Pohang, South Korea, pp. 507-513.
then be afforded a break. The system needs moderate [10] Duan-Sheng Chen , Zheng-Kai Liu, Aug 2011. Generalized Haar-like
processing power, making it suitable for practical use. For features for Fast Face Detection. Conference on Machine Learning
and Cybernetics, 2007. Hong Kong, pp. 2131-2135.
continuous video input, laptops with built in webcams or
[11] Bhaskar, T.N., Foo Tun Keat , Ranganath, S., Venkatesh, Y.V., Oct
USB cameras will suffice. 2003. Blink Detection and Eye Tracking for Eye Localization.
The system is limited by the efficiency of the blink Conference on Convergent Technologies for Asia-Pacific Region,
detection algorithm and efficiency falls further under 2003,Singapore, pp. 821-824 Vol.2.
limited lighting conditions. Since the initialization phase of
the algorithm is based on differencing between consecutive
frames, background movement in the frame may lead to
inaccurate operation. Typically, background movement
causes non eye pairs to be detected as eye pairs. This is
overcome to some extent by limiting the search region to
the face of an individual, by implementing a face tracking
algorithm prior to blink detection. However, this in turn can
lead to reduced efficiency in blink detection. By giving an
option to the user to choose between the system with and
without face tracking, a level of flexibility can be reached.
The application of the blink detection system is not
limited to the control of appliances but can also be used for
a variety of other functions. Playback of audio distress
messages over an intercom system is one of the other
applications of the system. Future applications of the system
may include playback of video or audio files by eye blinks
and making a VOIP call to play a distress message. In the
future, the system can be implemented on a Digital Signal
Processor, making it a truly embedded system which could
be used as a standalone device without the need for a laptop
or desktop PC.

[1] Michael Chau and Margrit Betke,2005. Real Time Eye Tracking and
Blink Detection with USB Cameras. Boston University Computer