2012 Fourth International Conference on Digital Home
ASSISTANCE FOR THE PARALYZED USING EYE BLINK
DETECTION
Atish Udayashankar, Amit R Kowshik,
Chandramouli S
Dept. of Telecommunication Engg.
PES Institute of Technology
Bangalore, India
atishudayashankar@gmail.com,
amithamithu.kowshik@gmail.com,
chandru.m16@gmail.com
Abstract - Paralysis is defined as the complete loss of muscle
function in any part of the body. It occurs when there is a
problem with the passage of messages between the muscles and
the brain. Some paralyzed people cannot move even a single
part of the body other than their eyes. Hence, the main aim of
this paper is to design a real time interactive system that can
assist the paralyzed to control appliances such as lights, fans
etc. or by playing pre-recorded audio messages, through a
predefined number of eye blinks. Image processing techniques
have been implemented in order to detect the eye blinks. In our
system, the face tracking is accomplished by using a set of
trained Haar cascade classifier, and a template matching
technique is employed to track the eye. Initially, the
involuntary blinks of the paralyzed person are used to locate
the patient’s eyes by finding the number of connected
components in a frame. Once the eyes are detected, an online
template is created which is then used to track the patient’s eye.
Keywords - Paralyzed patients, template matching, Haar
cascade classifier, eye blinks, face tracking
I. INTRODUCTION
Paralysis is the complete loss of muscle function for one
or more muscle groups. Paralysis can cause loss of feeling
or mobility in the affected areas. Paralysis can be localized,
or generalized, or it may follow a specific pattern. Most
paralyses caused by nervous system damage (i.e. spinal cord
injuries) are constant in nature; however, there are forms of
periodic paralysis, including sleep paralysis, which are
caused by other factors. Paralysis is most often caused by
damage to the nervous system, especially the spinal cord.
Major causes are stroke, trauma with nerve injury,
poliomyelitis, amyotrophic lateral sclerosis (ALS),
botulism, spina bifida, multiple sclerosis, and Guillain-
Barré syndrome. According to a survey by the Christopher
and Dana Reeve foundation, nearly 1 in every 50 people are
paralyzed [2].
Fully paralyzed patients require 24 hour support. But in
the present day and age, it is not possible for anyone to be
available at all times. So in those situations where the
patient is alone in a room, he/she could use this application
978-0-7695-4899-9/12 $26.00 © 2012 IEEE
DOI 10.1109/ICDH.2012.9
H S Prashanth
Dept. of Telecommunication Engg.
PES Institute of Technology
Bangalore, India
prashanth@pes.edu
to call for help if required or switch on/off a light, a fan or
any other appliance. Hence, our application will help the
patient to be self-sufficient to a certain extent.
There are innumerable applications which can be
derived from eye blink detection and these are not limited
for usage by only paralyzed people. An efficient, real time
blink detection algorithm can be used for almost any
purpose. It can be used for switching on/off appliances such
as a television or a microwave oven. It can also be used to
send an email or call someone on Skype. All this can be
accomplished with just a few eye blinks.
Innumerable techniques have been devised for face
tracking. CamShift face tracking algorithm, Haar face
tracking algorithm and face tracking using Eigenfaces are
some of these. There are some techniques devised for blink
detection as well. Some of them are software-oriented i.e.
using image processing, and a few are hardware-oriented
using sensors. Some of the image processing techniques
include blink detection using Gabor filters, contour
extraction, and eye blink detection using Median blur
filtering. The hardware-based approaches are primarily
using infrared and magnetic sensors [4, 7]. The advantage
of using sensors is that the entire system would be more
compact. However, risks involving the safety of the eye are
too high and outweigh the advantages of a sensor system. If
the frequency of the infrared light emitted by the sensor is
outside the visible spectrum, it could cause permanent
damage to the eyes.
The primary purpose of this paper is to propose a system
that can assist the paralyzed. It does so by tracking the
person’s eye and counting the blinks, and employs this
count to control various appliances and play pre-recorded
audio messages.
Although a number of techniques have been
implemented for eye blink detection, there is no application
that has been developed to actually put the blink detection
to practical use. The principal contribution of this paper is
the conceptualization of a system which will go a long way
in helping the paralyzed and disabled to achieve some level
of independence. Moreover, the algorithm provided by [1]
104
has been improved upon, by incorporating face tracking, so
as to reduce the effect caused by movements in the
background.
II. BLINK DETECTION
There are a number of image processing algorithms for
eye blink detection. A brief overview of three of these
algorithms is provided.
A. Contour Extraction
In this technique, a set of 16 landmarks are created at
regular intervals to outline the contour of the eye. Eight
points are used to represent each eye. The distance between
the highest and lowest landmark is denoted by d 1, and the
distance between the centroids of the two eyes is denoted by
d2. Now d1/d2 is computed, and assigned to a variable D.
Now the value of D is used to distinguish between open eye
and closed eye. Generally, a value of D equal to 0.158
implies open eye and a value equal to 0.016 implies closed
eye. These values have been experimentally derived by [3].
B. Gabor Filter
The Gabor filter is used to extract arcs of the eye. Here,
the eye region is first extracted and then the filter is applied
to obtain the arcs of the eye. Then connected component
labeling method is used to detect the top-bottom arcs. The
distance between the arcs is measured to determine the
blinking [5].
C. Median Blur Filtering
In this method, the image of the eye is first thresholded
and then a median blur filter is applied to it. The resultant
image obtained after applying the filtering shows a clear
difference between the open and the closed eye, and hence
helps in identifying eye blinks [6].
III. ALGORITHM USED IN THE PAPER
The algorithm used for eye blink detection is carried out
in five phases as shown in Figure 1. The first phase involves
face tracking in order to reduce the search window size. The
second phase involves identifying the location of the eye by
detecting involuntary eye blinks. The third phase involves
creation of an online template of the open eye. The fourth
phase is eye tracking and the final phase is the detection of
eye blinks. The algorithm is initialized automatically i.e. if
the location of the eye is lost, then the algorithm reinitializes
on its own to locate the eye. The system has a timer
programmed to run for a period of ten seconds during which
the patient will need to blink a desired number of times.
Finally after the timer runs out, the number of eye blinks is
used to perform the desired action, if it corresponds to a pre-
set value.
A. Face Tracking
Initially, the Haar face detector is loaded into the
program. The input image is then converted from
multichannel to single channel for faster processing. The
face region is then detected. A rectangular box is drawn to
105
outline the face region, and the coordinates of the rectangle
is provided to the next stage.
B. Initialization
In the first step, the difference between the current frame
and previous frame is found out. This difference image is
then thresholded. A threshold value of 5 was chosen. This
value was derived after some experimental trials.
Next, a structuring element is passed over the binary
image in order to remove the noise. The structuring element
used here is a 3x3 star shaped convolution kernel. This is
done in an open morphological operation. This operation
first performs erosion of the image followed by dilation
using the same structuring element for both the operations.
The primary effect of the opening operation is to preserve
the foreground regions that have a shape similar to the
structuring element or which can completely contain the
element. The other regions of foreground pixels are
eliminated.
The next step is to find the number of connected
components in the binary image. In order to do this, a
recursive labeling procedure is used. The lesser the number
of connected components detected indicates that the
algorithm is working as intended. In case, the number of
connected components detected is large in number, then the
binary image being used is discarded and the system
reinitializes. This step helps in preserving the efficiency of
the system. For an eye pair, the number of connected
components must be two.
If the image from the previous steps results in a small
number of connected components, the next step is to
determine which of these connected components constitute
an eye pair. This filtering to determine possible eye pairs is
done by applying experimentally derived heuristics. The
parameters used include: the vertical and horizontal
distance between the centroids of the components and the
width and height of each of the components. If any of these
parameters result in values which would normally not be
associated with an eye pair, then it can be concluded that the
connected components found do not constitute an eye pair.
On the connected components having met the conditions as
imposed by the heuristics, the system proceeds to open eye
template creation.
C. Open Eye Template Creation and Matching
For the creation of an open eye template, the larger of
the two connected components is chosen. This is done
primarily because the larger connected component contains
more brightness information and this helps in maintaining
the efficiency of the system. The open eye templates of
various sizes and orientations are shown in Figure 2.
 Figure1. Block diagram of the system- Block diagram representing the model of the entire system
Figure 2. Open eye templates - Open Eye templates of various sizes used
for template matching
The next step is to locate the eye using the created open
eye template. The method used to compare the current
image of the eye and the previously created template is the
normalized value of the square of the difference between the
brightness of the current frame and the template. The
formula used is given below.
In the above formula [8], T(x,y) represents the
brightness of the video frame at the point x,y.I(x,y) the
brightness of the template at the point x,y. As the function
computes the values of R(xy),i.e. the normalized square
difference, for various images, the best matches are found
as global minimums. Here, the minimum threshold value,
after performing experimental trials, has been chosen to be
106
0.4. Hence, if the value of R(x,y) exceeds 0.4, it is then
realized that it is not an eye and the re-initialization starts.
D. Blink Detection
The final step is the detection of the eye blink. In order
to accomplish this, the concept of motion analysis is used.
This step is very similar to the step where the user’s eyes
are located, except here the search window size is reduced.
In order to decide if there is an eye blink or not, two rules
are applied. The first one is that there must be only one
connected component. The second rule being that the
component must be located at the centroid of the user’s eye.
IV. RESULTS
The system was primarily developed and tested on a
Windows 7 PC with an Intel Core i7-2670QM CPU 2.20
GHz processor and an 8 GB RAM. The video was captured
with a Creative camera built in the laptop. The video was
captured at 30 frames per second and was processed as
grayscale images. These images were of a size of 320 x 240
pixels. Many functions from the Intel OpenCV library were
utilized. The code was written in Microsoft Visual Studio
2008 and the GUI was developed using the Visual Studio
MFC application.
The user is required to assign the number of blinks
which correspond to the different appliances and other
functions. This has to be set every time the eye blink
detection system is switched on. The number of blinks
corresponding to any application can be changed at any
given time. For this purpose, a simple user interface was
created where the user can enter values corresponding to
different appliances or functions before starting the blink
detection system. The following figure is a screenshot of the
graphical user interface developed.
wearing glasses and another was with the user not wearing
glasses. Efficiency was also found based on the distance of
the user from the camera. Yet another basis which was
employed to obtain the efficiency was to remove the face
tracking algorithm from the main algorithm. As a result, the
efficiency of the system improved considerably. Our
observations are tabulated as follows.

Table I. Without face tracking algorithm

Total number of blinks Number of blinks detected
20 19
20 14
20 14
20 17
20 20
Efficiency: 84%
Figure 3. The graphical user interface - The graphical user interface used to
set the number of blinks for a particular action Table II. Without face tracking algorithm and user is wearing glasses
Total number of blinks Number of blinks detected
20 13
20 14
20 15
20 14
20 17
Efficiency: 73%
Table III. Without face tracking algorithm and user is far from the
screen (90cm)
Total number of blinks Number of blinks detected
20 18
20 17
20 15
20 15
20 17
Figure 4. Blink detection without face tracking Efficiency: 82%

Table IV. With face tracking algorithm and people moving in the
background
Total number of blinks Number of blinks detected
20 8
20 7
20 12
20 8
20 10
Efficiency: 45%
Figure 5. Blink detection with face tracking and movement in the
background
From the above tabulations, it can be concluded that the
A number of trials were performed to find out the level algorithm works more efficiently when the user is not
of efficiency of the algorithm under various circumstances. wearing spectacles. Moreover, there is a very minute
The efficiency of the algorithm was computed, taking difference in the efficiency of the program with a change in
several cases into consideration. One case was with the user distance of the eyes from the screen. It is also observed that

107
there is a remarkable fall in the efficiency of the algorithm Science Technical Report No. 2005-12. Boston, USA
when face tracking is also implemented. However, if the [2] Christopher and Dana Reeve
face tracking is not included, the efficiency of the algorithm Foundationhttp://www.christopherreeve.org/site/c.mtKZKgMWKwG
/b.4453145/k.564C/Paralysis_and_Its_Impact.htm
with people moving in the background is nearly zero.
[3] Liting Wang , Xiaoqing Ding , Chi Fang , Changsong Liu , Kongqiao
Wang ,2009. Eye Blink Detection Based on Eye Contour Extraction.
V. CONCLUSION Proceedings of SPIE-IS&T Electronic Imaging. San Jose, CA, USA.
Although blink detection systems exist for other [4] M. Takagi, K. Mohri, M. Katoh and S.
purposes, an implementation of a blink detection system Yoshino,1994.MagnetDisplacement Sensor Using Magneto-Inductive
with the end use of controlling appliances has not been Elements for Sensing Eyelid Movement. IEEE Translation Journal
On Magnetics In Japan, Vol. 9,No.2, pp 78-83.
previously accomplished. While the system is intended to
[5] Kohei Aai and Ronny Mardiyanto, 2011. Comparative Study on
assist the paralyzed and physically challenged, it can Blink Detection and Gaze Estimation Methods for HCI, in Particular,
definitely be used by all types of individuals. The main Gabor Filter Utilized Blink Detection Method. Proceedings of Eighth
challenge involved in the implementation of the system is International Conference on Information Technology: New
the development of a real time robust blink detection Generations. Las Vegas, USA, pp. 441-446.
algorithm. Many algorithms have been developed to serve [6] Chinnawat Devahasdin Na Ayudhya, Thitiwan Srinark, A Method for
the purpose, with some being more accurate than the others. Real-Time Eye Blink Detection and Its Application
This paper presented a blink detection system based on [7] Cihan Topal, Ömer Nezih Gerek and Atakan Doan,2008. A
online template matching. HeadMounted Sensor-Based Eye Tracking Device: Eye Touch
System. Proeedings of the 2008 Symposium on Eye tracking research
The first phase involved the blink detection phase; the & applications. Savannah,GA,USA, pp. 87-90.
second phase involved the counting of blinks and [8] OpenCV documentation
subsequent control of appliances through a micro controller. http://opencv.willowgarage.com/documentation/
By enabling the paralyzed to gain control of albeit a small [9] Lae-Kyoung Lee, Su-Yong An , Se-Young Oh, Aug 2011. Efficient
part of their lives, the system can offer some level of Face Detection and Tracking with extended camshift and haar-like
independence to them. The helpers who are assigned the features. International Conference on Mechatronics and Automation
task of tending to paralyzed persons through the day can (ICMA), 2011. Pohang, South Korea, pp. 507-513.
then be afforded a break. The system needs moderate [10] Duan-Sheng Chen , Zheng-Kai Liu, Aug 2011. Generalized Haar-like
processing power, making it suitable for practical use. For features for Fast Face Detection. Conference on Machine Learning
and Cybernetics, 2007. Hong Kong, pp. 2131-2135.
continuous video input, laptops with built in webcams or
[11] Bhaskar, T.N., Foo Tun Keat , Ranganath, S., Venkatesh, Y.V., Oct
USB cameras will suffice. 2003. Blink Detection and Eye Tracking for Eye Localization.
The system is limited by the efficiency of the blink Conference on Convergent Technologies for Asia-Pacific Region,
detection algorithm and efficiency falls further under 2003,Singapore, pp. 821-824 Vol.2.
limited lighting conditions. Since the initialization phase of
the algorithm is based on differencing between consecutive
frames, background movement in the frame may lead to
inaccurate operation. Typically, background movement
causes non eye pairs to be detected as eye pairs. This is
overcome to some extent by limiting the search region to
the face of an individual, by implementing a face tracking
algorithm prior to blink detection. However, this in turn can
lead to reduced efficiency in blink detection. By giving an
option to the user to choose between the system with and
without face tracking, a level of flexibility can be reached.
The application of the blink detection system is not
limited to the control of appliances but can also be used for
a variety of other functions. Playback of audio distress
messages over an intercom system is one of the other
applications of the system. Future applications of the system
may include playback of video or audio files by eye blinks
and making a VOIP call to play a distress message. In the
future, the system can be implemented on a Digital Signal
Processor, making it a truly embedded system which could
be used as a standalone device without the need for a laptop
or desktop PC.

VI. REFERENCES
[1] Michael Chau and Margrit Betke,2005. Real Time Eye Tracking and
Blink Detection with USB Cameras. Boston University Computer

108