Real-Time Face Detection and Recognition in Video Synopsis

Title: Real-Time Face Detection and Recognition in

Video

Problem Statement:

The aim of the project is to detect faces in a video and recognize them. This is a
challenging problem as there can be variations caused by several factors such as
appearance, lighting, background and expression.

Introduction:

Face detection and recognition have been regarded as a challenging problem in the field
of computer vision, due to the large intra-class variations caused by the changes in facial
appearance, lighting, and expression. Face detection and recognition have been object of
much interest in the last years. They have many applications in a variety of fields such as
identification for law enforcement, personal identification and security system access.
In our project, we will be using it for attendance through face recognition system.

There are two main steps. The Face Detection approach is based on modest AdaBoost
algorithm [3] which is used to select a small number of extended Haar-like features from
a very large set of potential features, which is the criteria for choosing the reduced set of
features. Faces are divided into several categories according to their poses, and for each
of these categories a form of weak classifiers in look-up-table type is designed. This face
detection approach can achieve fast, accurate face detection that is robust to changes in
illumination and background.

Viola-Jones method [5]: This approach to detecting objects in images combines four key
concepts:
• Simple rectangular features, called Haar features
• An Integral Image for rapid feature detection
• The AdaBoost machine-learning method
• A cascaded classifier to combine many features efficiently

The Face Recognition stage uses Eigenface face recognition algorithm [1]. This
algorithm is easy to implement. The steps used in eigenface are also used in many
advanced methods. Compared with conventional approaches, this method for face
recognition is shown to be more effective and capable of handling more complicated
variations.

Department of Information Science and Engineering

1
Real-Time Face Detection and Recognition in Video Synopsis

Literature Survey:

[Wee Lau Cheong et al, 2010] has proposed building a Computation Savings Real-Time
Face Detection and Recognition System. This paper presents two techniques in
developing a computation savings Real-Time Face Detection and Recognition System.
We focuses on our proposed methods to speed up and optimize the system components,
e.g. the image resolution resizing technique to perform faster face detection in high
resolution image, and face tracking to reduce redundant processes in face recognition.

[Jong-Min Kim et al, 2009] [2] has proposed Real Time Face Detection and Recognition
Using Rectangular Feature Based Classifier and Modified Matching Algorithm. This
paper proposes a classifier based on rectangular feature to detect face in real time. The
proposed algorithm consists of the following three stages: Feature creation, classifier
study and real time facial domain detection. Feature creation organizes a feature set with
the proposed five rectangular features and calculates the feature values efficiently by
using SAT(Summed-Area Tables). Classifier learning creates classifiers hierarchically by
using the AdaBoost algorithm. Real time facial domain detection finds facial domains
rapidly and efficiently through the classifier based on the rectangular feature that was
created.

[Zhen Lei1 et al, 2009] [3] has proposed Real-time Face Detection and Recognition for
Video Surveillance Applications. The face detection approach is based on modest
AdaBoost algorithm and can achieve fast, accurate face detection that is robust to
changes in illumination and background. The recognition stage is based on an improved
independent components Analysis approach which has been modified to cope with the
video surveillance application. The experimental results demonstrate the significant
performance improvement using the proposed approach over others. It can be seen that
the proposed method is very efficient and has significant value in application.

[Jong-Min Kim et al, 2010] [4] has proposed Study of Face Recognition using the PCA
and Error Back-Propagation. In this paper the real-time face region was detected by
suggesting the rectangular feature-based classifier and the robust detection algorithm that
satisfied the efficiency of computation and detection performance was suggested. in this
paper the face recognition method combined with PCA and the multi-layer network
which is one of the intelligent classification was suggested and its performance was
evaluated. As a preprocessing algorithm of input face image, this method computes the
eigenface through PCA and expresses the training images with it as a fundamental vector.
As a result of comparing with existing methods, Euclidean and Mahananobis method, the
suggested method showed the improved recognition performance with the incorrect
matching or matching failure.

Department of Information Science and Engineering

2
Real-Time Face Detection and Recognition in Video Synopsis

[Ramzi Abiantun et al, 2010] [5] has proposed Boosted Multi Image Features for
Improved Face Detection. This paper expands on the traditional Viola-Jones approach by
proposing to boost a plethora of mixed feature sets for face detection; we do this by
adding non-Haar-like elements to a large pool of mixed features in an Adaboost
framework. We show how to generate discriminative Support Vector Machine (SVM)
type features and Gabor-type features (in various orientations and frequencies and central
locations) and use this whole pool as possible discriminative candidate feature sets in
modeling the patterns of a frontal view human face. We show we can improve the
generalization performance of the AdaBoost approach, and as a result improving the
robustness of the face detector. We report performance on the MIT+CMU face database
and compare the result with other published face detection algorithms.

[Ziyou Xiong, et al, 2010] [6] has proposed Real Time Automatic Access Control
System based on Face and Eye Corners Detection, Face Recognition and Speaker
Identification. Our system also supports online training of the user models, i.e, a person
who has nor been registered in the database can be added on the spot in real time. This
system is composed of a face and eye corners detection module, a face recognition
module, a speaker identification module and a user interface module. We present the
algorithm of each module with more emphasis on our information maximization based
face and eye corners detection and the user interface design.

[Hanxi Li Peng Wang Chunhua Shen et al, 2010] [7] has proposed Robust Face
Recognition via Accurate Face Alignment and Sparse Representation. In this paper, we
present a robust real-time facial recognition system. The system comprises three
functional components, which are face detection, eye alignment and face recognition,
respectively. As for face detection, we have proposed a new approach termed Boosted
Greedy Sparse Linear Discriminant Analysis (BGSLDA) that produces better
performances than most reported face detectors. Since face misalignment significantly
deteriorates the recognition accuracy, we advocate a new cascade framework including
two different methods for eye detection and face alignment. We have adopted a recent
algorithm termed Sparse Representation-based Classification (SRC) for the face
recognition component.

[Mandalapu Sarada Devi et al, 2010] [8] has proposed Active Facial Tracking. This
paper presents a real-time human face tracker development using facial feature
extraction. The skin color method is adopted to obtain the face region because of its
efficiency in computing which is required in real time face tracker system. Facial feature
points are referred to in the literature as salient points, anchor points or facial landmarks.
The most frequently occurring facial features are the four eye corners, the tip of the nose
and the two mouth corners. Facial feature detection is a challenging computer vision
problem due to high interpersonal changes such as gender, race and the intra-personal
variability such as pose, expression and acquisition conditions like lighting, scale, facial
accessories.

Department of Information Science and Engineering

3
Real-Time Face Detection and Recognition in Video Synopsis

[Shan An et al, 2009] [9] has proposed Face Detection and Recognition with SURF for
Human-Robot Interaction. This paper describes a feature-based approach for face
detection and recognition for human-robot interaction (HRI), which is capable of
processing images rapidly and achieving high detection and recognition rates. We apply
an efficient method for detecting human face and extracting facial features for face
recognition to solve the above problems. We locate the face region by employing some
morphological steps after skin region detected. The face contour is obtained using an
ellipse fitting method. Experimental results with Caltech’s face database show that our
approach is able to detect and recognize different faces with different
illumination/expressions/backgrounds rapidly and accurately.

[Binglong Xie1 et al, 2006] [10] has proposed Multi-Camera Face Recognition by
Reliability-Based Selection. A face recognition system using multiple cameras
overcomes these limitations. In each channel, real-time component-based face detection
detects the face with moderate pose and illumination changes employing fusion of
individual component detectors for eyes and mouth, and the normalized face is
recognized using an LDA recognizer. A reliability measure is trained using the features
extracted from both face detection and recognition processes, to evaluate the inherent
quality of channel recognition. The recognition from the most reliable channel is selected
as the final recognition results. The recognition rate is far better than that of either single
channel, and consistently better than common classifier fusion rules.

Proposed Methods:
Department of Information Science and Engineering
4
Real-Time Face Detection and Recognition in Video Synopsis

Viola-Jones method: This approach to detecting objects in images combines four key
concepts:

• Simple rectangular features, called Haar features

• An Integral Image for rapid feature detection
• The AdaBoost machine-learning method
• A cascaded classifier to combine many features efficiently

The features that Viola and Jones used are based on Haar wavelets. Haar wavelets are
single wavelength square waves (one high interval and one low interval). The presence of
a Haar feature is determined by subtracting the average dark-region pixel value from the
average light-region pixel value. If the difference is above a threshold (set during
learning), that feature is said to be present.

To determine the presence or absence of hundreds of Haar features at every image

location and at several scales efficiently, Viola and Jones used a technique called an
Integral Image. In this case, the small units are pixel values. The integral value for each
pixel is the sum of all the pixels above it and to its left. Starting at the top left and
traversing to the right and down, the entire image can be integrated with a few integer
operations per pixel.

To select the specific Haar features to use, and to set threshold levels, Viola and Jones
use a machine-learning method called AdaBoost. AdaBoost combines many "weak"
classifiers to create one "strong" classifier. ". AdaBoost selects a set of weak classifiers to
combine and assigns a weight to each. This weighted combination is the strong classifier.

The acceptance threshold at each level is set low enough to pass all, or nearly all, face
examples in the training set. The filters at each level are trained to classify training
images that passed all previous stages. (The training set is a large database of faces,
maybe a thousand or so.) During use, if any one of these filters fails to pass an image
region, that region is immediately classified as "Not Face". When a filter passes an image
region, it goes to the next filter in the chain. Image regions that pass through all filters in
the chain are classified as "Face". Viola and Jones dubbed this filtering chain a cascade.

Eigenface Method: This method is used for Face Recognition. The basic principles
behind it are PCA and distance-based matching.

Department of Information Science and Engineering

5
Real-Time Face Detection and Recognition in Video Synopsis

Here's how recognition works: given example face images for each of several people,
plus an unknown face image to recognize,

1. Compute a "distance" between the new image and each of the example faces
2. Select the example image that's closest to the new one as the most likely known
person
3. If the distance to that face image is above a threshold, "recognize" the image as
that person, otherwise, classify the face as an "unknown" person

Distance, in the original eigenface paper, is measured as the point-to-point distance. This
is also called Euclidean distance.

Dimensionality Reduction By PCA

There are many methods for dimensionality reduction. The one that eigenface uses is
called Principal Components Analysis, PCA for short.

The PCA Subspace

The direction of maximum separation is called the first principal component of a dataset.
The direction with the next largest separation is the one perpendicular to this. That's the
second principal component. In a 2D dataset, we can have at most two principal
components. Since the dimensionality for images is much higher, we can have more
principal components in a dataset made up of images.

In eigenface, each 50x50 face image is treated as one data point (in a 2,500 dimensional
"space"). So the number of principal components we can find will never be more than the
number of face images minus one.

Projecting Data Onto a Subspace

When you project a point onto a subspace, you assign it the subspace location that's
closest to its location in the higher dimensional space.

Computing Distances Between Faces

In eigenface, the distance between two face images is the Euclidean distance between
their projected points in a PCA subspace, rather than the distance in the original 2,500
dimensional image space. Computing the distance between faces in this lower
dimensional subspace is the technique that eigenface uses to improve the signal-to-noise
ratio.

Picturing The Principal Components

For face recognition on 50x50 images, each eigenvector represents the slope of a line in a
2,500 dimensional space. As in the 2D case, we need all 2,500 dimensions to define the

Department of Information Science and Engineering

6
Real-Time Face Detection and Recognition in Video Synopsis

slope of each line. While it's impossible to visualize a line in that many dimensions, we
can view the eigenvectors in a different way. We can convert their 2,500 dimensional
"slope" to an image simply by placing each value in its corresponding pixel location.
When we do that, we get face-like images called eigenfaces.

Department of Information Science and Engineering

7
Real-Time Face Detection and Recognition in Video Synopsis

Flowchart:

Input From
Camera

Capture Frames

Preprocessing

Face Detection

Face Tracking

Face Realignment, Post Processing

Face Recognition

Output Result

Figure: Diagram of Face Detection and Recognition Process.

Department of Information Science and Engineering

8
Real-Time Face Detection and Recognition in Video Synopsis

Applications:

Face Detection:
 Used in Biometrics
 Video Surveillance
 Human Computer Interface
 Image Database Management
 Used for Autofocus in Digital Camera
 Intelligent Home Applications such as Personalized TV and Databases

Face Recognition:
 Identity verification for physical access control in buildings or security areas
 To allow secure transactions through the Internet
 Used in forensic applications for criminal identification (mug-shot matching) and
surveillance of public places to detect the presence of criminals or terrorists (for
example in airports or in border control)
 Used for government applications like national ID, driver’s license, passport and
border control, immigration, etc
 For interpreting human actions, human emotions, facial expressions, human
behavior and intentions, and is also an extremely natural and non-intrusive
technique, makes it an excellent choice for ambient intelligence applications

Hardware Requirements:

• High Resolution Web Cam

• Pentium IV Processor with 512 RAM or above.

Software Requirements:

• OpenCV 2.0
• Cmake 2.8.0
• Microsoft Visual Studio 2008 Express Edition

Department of Information Science and Engineering

9
Real-Time Face Detection and Recognition in Video Synopsis

References:
[1] Wee Lau Cheong, Cheng Mun Char, Yi Chwen Lim, Samuel Lim, Siak Wang Khor,
“Building a Computation Savings Real-Time Face Detection and Recognition System”,
2nd International Conference on Signal Processing Systems (ICSPS) 2010.

[2] Jong-Min Kim1 , Kyoung-Ho Kim and Maeng-Kyu Song, “Real Time Face Detection
and Recognition Using Rectangular Feature Based Classifier and Modified Matching
Algorithm”, Fifth International Conference on Natural Computation 2009.

[3] Zhen Lei1, Chao Wang1, Qinghai Wang, Yanyan Huang, “Real-time Face Detection
and Recognition for Video Surveillance Applications”, World Congress on Computer
Science and Information Engineering 2009.

[4] Jong-Min Kim, Myung-A Kang, “A Study of Face Recognition using the PCA and
Error Back-Propagation”, Second International Conference on Intelligent Human-
Machine Systems and Cybernetics 2010.

[5] Ramzi Abiantun and Marios Savvides, “Boosted Multi Image Features for Improved
Face Detection”, 2010.

[6] Ziyou Xiong, Yunqiang Chen, Roy Wang and Thomas S. Huang, “A Real Time
Automatic Access Control System based on Face and Eye Corners Detection, Face
Recognition and Speaker Identification.”, 2010.

[7] Hanxi Li Peng Wang Chunhua Shen, “Robust Face Recognition via Accurate Face
Alignment and Sparse Representation”, Digital Image Computing: Techniques and
Applications 2010.

[8] Mandalapu Sarada Devi, Dr Preeti R Bajaj, “Active Facial Tracking”, Third
International Conference on Emerging Trends in Engineering and Technology, 2010.

[9] Shan An, Xin Ma, Rui Song and Yibin Li, “Face Detection and Recognition with
SURF for Human-Robot Interaction”, International Conference on Automation and
Logistics Shenyang, China August 2009

[10] Binglong Xie1, Terry Boult2, Visvanathan Ramesh1, Ying Zhu, “Multi-Camera
Face Recognition by Reliability-Based Selection”, IEEE International Conference on
Computational Intelligence for Homeland Security and Personal Safety Alexandria, VA,
USA, 16-17 October 2006

Department of Information Science and Engineering

10