PERSON IDENTIFICATION USING OPEN AND

SHUT EYE DETECTION

A MINOR PROJECT REPORT

SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF

BACHELOR OF TECHNOLOGY
(Information Technology)

SUBMITTED TO
DELHI TECHNOLOGICAL UNIVERSITY, DELHI

SUBMITTED BY

Ankur Agarwal 2K12/IT/013

Chetan Gulati 2K12/IT/022
Sunpreet Singh 2K12/IT/087
Vaibhav Chauhan 2K12/IT/094

SUPERVISED BY
Dr. SEBA SUSAN

May 2015

DELHI TECHNOLOGICAL UNIVERSITY, DELHI

 CERTIFICATE

This is to certify that the report entitled “Person Identification using Open and
Shut eye Detection” submitted by Ankur Agarwal, Chetan Gulati, Sunpreet Singh
and Vaibhav Chauhan in partial fulfillment of the requirements for the award of
Bachelor of Technology Degree in Information Technology from Delhi
Technological University (Formerly Delhi College of Engineering) for their Minor
Project in the Sixth Semester is an authentic work carried out by them under my
supervision & guidance.

Dr Seba Susan
Assistant Professor
Department of Computer Science & Engineering
Delhi Technological University
 DELHI TECHNOLOGICAL UNIVERSITY

DECLARATION
We hereby declare that the work which is being presented in the B.Tech Minor
Project Report entitled “Person Identification using Open and Shut eye
Detection”, in partial fulfillment of the requirements for the award of the
Bachelor of Technology in Information Technology and submitted to the
Department of Computer Science & Engineering of Delhi Technological
University, Delhi is an authentic record of our own work carried out during a
period of 6th semester under the supervision of Dr Seba Susan, Department of
Computer Science & Engineering

The matter presented in this Project Report has not been submitted by us for the
award of any other degree elsewhere.

Signature of students

(2K12/IT/13) (2K12/IT/22) (2K12/IT/87) (2K12/IT/94)

 DELHI TECHNOLOGICAL UNIVERSITY

ACKNOWLEDGEMENT

We would like to place on record our deep sense of gratitude to our mentor and
advisor Dr. Seba Susan, Department of Computer Science & Engineering
, Delhi Technological University, Delhi for her patience, generous guidance, help,
useful suggestions and continuous encouragement throughout the course of
present work and for enlightening us with her immense knowledge of the subject.
Her cooperation and support during the whole research process was one of the
main reasons we could complete this report with a satisfactory conclusion.

Ankur Agarwal (2K12/IT/013)

Chetan Gulati (2K12/IT/022)

Sunpreet Singh (2K12/IT/087)

Vaibhav Chauhan (2K12/IT/094)

 INDEX

Table of contents

1. Abstract……………………………………………………………………………………………………………..
2. Objective…………………………………………………………………………………………………………….
3. Introduction………………………………………………………………….……………………………………
4. Basic Flow…………………………………………………………………………………………………………..
5. Hardware and Software Requirements……………………………………............................
6. Haar Features……………………………………………………………...........................................
Cascade of Classifiers………………………………………………………………………………………….
7. Face detection……………………………………………………………………………………………………
Eye detection……………………………………………………………………………………………………..
8. Feature Extraction……………………………………………………………………………………………..
What are local binary patterns? ............................................................................
9. Support Vector Machines………………………………………………………………………………….
Person Identification………………………………………………………………………………………….
10. System Images………………………………………………………………………………………………….
11. System Testing…………………………………………………………….......................................
12 .Conclusion………………………………………………………………………………………………………..
13. Bibliography…………………………………………………………………………………………………….
 ABSTRACT

Due to the rising dependency of man on machines, artificial intelligence has

become very popular area of research. Under this category object detection
and machine learning are two of the most promising areas. They have many
areas of applications like face detection and recognition, etc. Our project
deals with eye detection and feature extraction. We used predefined Haar
classifiers for detecting face and eye using Opencv. It uses Viola Jones Object
Detection framework and for feature extraction we used LBP feature
extraction technique. Furthermore, we used support vector machine to train
the system to recognize eye as shut or open. Here we recognize whether the
eye is closed or open in an unknown image by comparing it with the known
training images stored in the database, which is used for person
Identification. This was implemented using MATLAB. These conditions were
not much robust, that is, shifting face or moving head caused error in
detection or no detection and the accuracy was low. However, for a fairly
aligned head with little or no movement, the program was successfully able
to run.

We also tried tracking pupil movement from a video capture and continuous
database of images. It helped us analyze where the person was looking. It
was done in order to check whether or not natural eye movement, without
any external stimulus, can be a measure of person recognition. This project
was implemented using OpenCV.
 OBJECTIVE

The objective is to identify person using open and shut eye method. The
development tool used is MATLAB and Opencv, and the emphasis is on
recognition and verification. MATLAB provides a good rapid application
development environment, with various tools for image processing.

The system, as expressed is composed of number of modules, which corresponds

to each stage of iris recognition. These stages are broadly classified as Face
detection – locating the face region in the given frame, eye detection –
identifying eye in the face region, feature vector – creating a vector containing
only the most selective features (Uniform LBP features)of the eye region and
SVM- applying machine learning to identify the person using number of open and
closed eye frames. The input to system will be several eye images of person, and
the system will identify the person.
 INTRODUCTION

Many approaches have been proposed for detecting and recognizing faces. For
many practical applications, some of the current systems are sufficiently mature
to be used.

However, in some other applications, such as video surveillance scenarios, there

is still work to be done in order to efficiently detect or recognize the faces. Among
the factors challenging the ongoing research are poor quality images, different
facial orientations and changes in illumination conditions.

To overcome the limitations of the color based approach, we also considered the
implementation of the well-known face detection algorithm proposed by Viola
and Jones. It uses Haar like features and AdaBoost learning algorithm. The Haar-
like features are extracted using the notion of integral image which allows very
fast feature extraction at different scales, while AdaBoost is used to select the
most prominent features among a large number of extracted features and
construct a strong classifier from boosting a set of weak classifiers. The use of a
cascade of classifiers made this approach one of the first real-time frontal-view
face detection methods

After that our aim is to detect whether the eye is closed or open.

There can be various measures to check whether the eye is shut or open like
physiological measures (using EEG), behavioral measures (uses body movements
and actions) and visual measures (using eye and face detection and analysis). Our
focus was using visual measures. Although the other two methods are highly
accurate, however they require much expensive equipment and technologies.

Finally, we want to identify a person using number of open and closed eye frames
in a video while both the persons under test are performing identical tests.
 BASIC FLOW

Read Frames from a video or a

folder

Find face using Haar Classifier

Find Eye rectangles within the

face region using Haar

Isolate the eye and build feature

vector using uniform LBP feature
extraction.

Create a SVM training matrix

using the feature vectors
obtained for open & shut eye
detection

Apply SVM to recognize

people using no of open and
closed eye frames.
 HARDWARE AND SOFTWARE REQUIREMENTS

HARDWARE REQUIREMENTS

CPU : Pentium Processor

Hard Disk : 40 GB or above
RAM : 512 MB or above
VDU : SVGA(800*600 pixels)
Operating System : Windows 7/8/8.1

SOFTWARE REQUIREMENTS

MATLAB : R2013a

DATABASE
VidTIMIT DATABASE
 HAAR FEATURES

We used OpenCV’s inbuilt Haar-like cascade classifiers for face and eye detection.
This is most common and reliable technique used in all kinds of applications.
 CASCADE OF CLASSIFIERS

Instead of applying all the features on a window, we group the feature into
different stages of classifiers and apply them one by one.

If the window fails at one stage, we discard it and don’t consider the remaining
features.

If it passes, we apply the next feature and continue with the process.

OpenCV's pre-trained classifiers

OpenCV already contains many pre-trained classifiers for face, eyes, smile etc.
Those XML files are stored in opencv/data/haarcascades/ folder.
 FACE AND EYE DETECTION

1. Load the required XML classifiers.

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')

2. Load input image:

img = cv2.imread('xfiles4.jpg')

3. Use cv.HaarDetectObjects () to find faces which has been defined like this:

cv.HaarDetectObjects(image, cascade, storage, scale_factor=1.1,

min_neighbors=3, flags=0, min_size=(0, 0))

Different parameters are:

image : Matrix of the type CV_8U containing an image where objects are

detected.

scaleFactor : Parameter specifying how much the image size is reduced at each

image scale.
minNeighbors : Parameter specifying how many neighbours each candidate
rectangle should have to retain it. This parameter will affect the quality of the
detected faces: higher value results in less detection but with higher quality.
We're using 5 in the code.

flags: Parameter with the same meaning for an old cascade.

minSize : Minimum possible object size. Objects smaller than that are ignored.

4. If faces are found, it returns the sequence of detected faces

CvSeq *faces = cvHaarDetectObjects( img, cascade_f, storage,2.0, 3, 0, cvSize(50,

50));

5. Draw the rectangle on the each face.

6. Estimate the region of interest for eye detection

7.Using cv.HaarDetectObjects () and eye cascade classifier, detect the sequence

of images.

CvSeq* eyes = cvHaarDetectObjects( img, cascade_e, storage, 1.2, 2, 0, cvSize(20,

20));

8. Draw rectangles on each eye found.

 FEATURE EXTRACTION

Feature extraction is the process that begins from an initial set of measured data
and builds derived values (features) intended to be informative, non-redundant,
facilitating the subsequent learning and generalization steps. Feature extraction
reduces the amount of resources required to represent current large set of data.

The technique that we used for feature extraction was Uniform LBP or Local
Binary Patterns. It is a particular case of Texture Spectrum Model.

WHAT ARE LOCAL BINARY PATTERNS?

The basic notion for creating the LBP operator was that two dimensional surface
textures can be portrayed by two complementary measures: local spatial patterns
and gray scale contrast. The LBP operator was extended to use neighborhoods of
different sizes. Using a circular neighborhood and bi-linearly interpolating values
at non-integer pixel coordinates allow any radius and number of pixels in the
neighborhood.

Image: Calculation of LBP Patterns

 UNIFORM LOCAL BINARY PATTERNS
Another method of use is that of uniform patterns, which can be used to reduce
the length of the feature vector and implement a simple rotation-invariant
descriptor. A local binary pattern is called uniform if the binary pattern contains
at most two bitwise transitions from 0 to 1 or vice versa when the bit pattern is
traversed circularly. For example, the patterns 00000000 (0 transitions),
01110000 (2 transitions) and 11001111 (2 transitions) are uniform whereas the
patterns 11001001 (4 transitions) and 01010010 (6 transitions) are not. In the
calculation of the LBP labels, uniform patterns are used so that there is a separate
label for each uniform pattern and all the non-uniform patterns are labeled with a
single label. For example, when using (8,R) neighborhood, there are a total of 256
patterns, 58 of which are uniform, which yields in 59 different labels. This result
was used in the following algorithm and formed a basic foundation of our
program.

LBP creates a feature matrix using following algorithm:

1. Break the window under consideration into cells

2. For each pixel in the cell, compare the given pixel with each of its 8
neighbors. Follow the pixels in a circular manner i.e. either clockwise or anti
clockwise.

3. If the center pixel’s value is greater than neighbor’s value write 1 else write
0. This gives an 8 bit binary number.

4. Compute the histogram, of the frequency of each 8 bit number occurring

5. Concatenate histograms of all cells. This will present feature vector for the
window.
After implementing this algorithm we presented the feature matrix to the SVM to
classify images.

Image: Thresholding and calcualtion of LBP value

 SUPPORT VECTOR MACHINES

WHAT ARE SVMs?

Support vector machines (SVMs, also called support vector networks) are
supervised learning models with associated learning algorithms that analyze data
and recognize patterns, used for classification and regression analysis. A SVM is a
discriminative classifier formally defined by a separating hyperplane. In other
words, given labeled training data (supervised learning), the algorithm outputs an
optimal hyperplane which categorizes new examples.

Steps to implement SVM:

1. Set up training data

2. Train the SVM

SVMStruct=svmtrain(training,label)

training – Training matrix

label- Corresponding Group Identifiers

3. Classify input into different groups

Group=svmclassify(SVMStruct,test)

test- testing matrix which is to be classified

Group- Corresponding group identifiers

After the implementation of this LBP, we have with us a feature matrix, a SVM
with trained data using above procedure implemented on database of images and
now we can check the unknown image for state of eye being open or shut.
 PERSON IDENTIFICATION

Knowing the state of the eye can be applied to many applications like driver
drowsiness detection, lazy eye detection. What our project aims at is to detect
whether the eye is closed or open. After checking this, we have to train SVM using
feature matrix and apply it to unknown images. Then using these features we
have to recognize a person based on the blink characteristics.

There can be various measures to check whether the eye is shut or open like
physiological measures (using EEG), behavioral measures (uses body movements
and actions) and visual measures (using eye and face detection and analysis). Our
focus was using visual measures. Although the other two methods are highly
accurate, however they require much expensive equipment and technologies.

People exhibit certain blink characteristics in particular situations like under

fatigue, watching television for continuous duration, reading etc. These
characteristics help us analyze their blinking trends and have made the detection
process simpler.

Normally, used as driver drowsiness detection program in vehicles, our project

can investigate the image and tell whether the eye is in open state or closed state.

This method exploits the blink characteristics for eye without any external
stimulus. Instead of traditional iris recognition, this process is far less invasive and
can easily be utilized even under normal circumstances. However, the accuracy
may vary due to fact that rate of blinking is influenced by external factors.
The following steps were performed:

Feature Extraction via LBP for

each frame

Training SVM using feature

matrix created

Training for two different

person

Taking input for recognition

 SYSTEM IMAGES

Face Detection:

Face and Eye Detection:

Isolated Image - Open Eye:

Isolated Image – Closed Eye:

 SYSTEM TESTING

1. We took 3 different open and closed eye frames, created feature vectors for
them using Uniform LBP feature extraction and used that as training data for
SVM.

2. We Used the SVM classifier to classify eyes as open or closed.

3. We took two videos for different persons, isolated eyes in frames and counted
number of open and closed eye frames using SVM classifier.

4. We Used number of open and closed eye frames as training data for SVM.

5. We isolated eyes from test video and counted number of open and closed eye
frames.

6. Finally, we used SVM classifier for identifying person in the test video based on
the number of open and closed eye frames in that video.

Training Data

Total number of frames taken for each person- 70

For person1

Number of open eye frames- 45

Number of closed eye frames- 25

For person2

Number of open eye frames- 24

Number of closed eye frames- 46

Test Data

For video1

Number of open eye frames- 51

Number of closed eye frames- 19

Output of SVM classifier: Person1

For video2

Number of open eye frames- 25

Number of closed eye frames- 45

Output of SVM classifier: Person2

Thus we were successfully able to identify the two persons.

 CONCLUSION

After reading several research papers, running numerous tests, changing several
segments of code, we were finally able to get our result.

Our approach started from eye detection using Viola Jones Method. Although the
training method was slow, the detection procedure was fast. Slowly but surely the
accuracy was targeted, so as to get better rate of eye detection.

Followed by feature extraction, much work had to be done to analyze the

features involved, in order to separate out the important information from
redundant and ineffectual data. Dry simulations were run to target the necessary
results. After creation of feature matrix using Linear Binary Patterns, our
approach became more defined and training part was focused.

In order for the SVM to be trained, the code was run on large data base of images
that should be in accordance with our functionality. A limitation encountered at
this stage was that, the for recognition purpose to be carried out later, the
blinking of individual in the database must be different for getting precise results.
Therefore, many databases were searched to find such images or video captures.

After completing all these steps, the final project is now functional and can
recognize two persons. The accuracy of the code is good, giving up to 95% correct
result in target images.

However, initially the basis of our project was different, that is, how behavioral
gaze patterns can help identify a person. Since much research was not done in
this field we were not able to establish a concrete proof whether they can or not.

We tried different methods such as finding variance of change in pupil position in

successive frames, finding entropy for the same, using the array of successive
changes in pupil positions as feature vector for applying SVM but nothing worked.
In nutshell, we would like to conclude that our project can now effectively run to
identify the person and the other machine learning algorithms can be used to
increase the number of people under identification.
 BIBLIOGRAPHY

 http://docs.opencv.org/modules/objdetect/doc/cascade_classification.html

 http://opencv-code.com/tutorials/eye-detection-and-tracking/

 http://in.mathworks.com/help/stats/support-vector-machines.html

 http://www.cse.oulu.fi/CMV/Downloads/LBPMatlab\

 http://www.cs.ru.ac.za/research/g09w0474/Thesis.pdf

 http://conradsanderson.id.au/vidtimit/

A Support Vector Machine (SVM) is a discriminative classifier formally defined by a

separating hyperplane. In other words, given labeled training data (supervised learning),
the algorithm outputs an optimal hyperplane which categorizes new examples.

In which sense is the hyperplane obtained optimal? Let’s consider the following simple
problem:

For a linearly separable set of 2D-points which belong to one of two classes, find a
separating straight line.
 In the above picture you can see that there exists multiple lines that offer a solution to the
problem. Is any of them better than the others? We can intuitively define a criterion to
estimate the worth of the lines:

A line is bad if it passes too close to the points because it will be noise sensitive and it
will not generalize correctly. Therefore, our goal should be to find the line passing as far
as possible from all points.

Then, the operation of the SVM algorithm is based on finding the hyperplane that gives
the largest minimum distance to the training examples. Twice, this distance receives the
important name of margin within SVM’s theory. Therefore, the optimal separating
hyperplane maximizes the margin of the training data.

4.2ow is the optimal hyperplane computed?

Let’s introduce the notation used to define formally a hyperplane:

where is known as the weight vector and as the bias.

See also

A more in depth description of this and hyperplanes you can find in the section 4.5
(Seperating Hyperplanes) of the book: Elements of Statistical Learning by T. Hastie, R.
Tibshirani and J. H. Friedman.
The optimal hyperplane can be represented in an infinite number of different ways by
scaling of and . As a matter of convention, among all the possible representations of
the hyperplane, the one chosen is

where symbolizes the training examples closest to the hyperplane. In general, the
training examples that are closest to the hyperplane are calledsupport vectors. This
representation is known as the canonical hyperplane.

Now, we use the result of geometry that gives the distance between a point and a
hyperplane :

In particular, for the canonical hyperplane, the numerator is equal to one and the distance
to the support vectors is

Recall that the margin introduced in the previous section, here denoted as , is twice
the distance to the closest examples:

Finally, the problem of maximizing is equivalent to the problem of minimizing a

function subject to some constraints. The constraints model the requirement for the
hyperplane to classify correctly all the training examples . Formally,

where represents each of the labels of the training examples.

This is a problem of Lagrangian optimization that can be solved using Lagrange

multipliers to obtain the weight vector and the bias of the optimal hyperplane.