2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)

The Design of Face Recognition and Tracking for

Human-Robot Interaction
W.S. Mada Sanjaya1,2∗ , Dyah Anggraeni1,2 , Kiki Zakaria1,2 , Atip Juwardi1,2 ,
and Madinatul Munawwaroh1,2
1
dept. of Physics, Faculty of Science and Technology, UIN Sunan Gunung Djati, Bandung, Indonesia
2
Bolabot Techno Robotic Institute, CV Sanjaya Star Group, Bandung, Indonesia
∗
madasws@gmail.com

Abstract—This paper discusses the development of Social
Robot named SyPEHUL (System of Physic, Electronic, Humanoid
Robot and Machine Learning) which can recognize and tracking
human face. Face recognition and tracking process use Cascade
Classification and LBPH (Local Binary Pattern Histogram) Face
Recognizer method based on OpenCV library and Python 2.7.
The social robot hardware based on Arduino microcontroller
contains by 12 DoF (Degree of Freedom) motor servos to actuate
robotic head and its face. The face recognition system has been
implemented to Social Robot which can recognize and tracking
human face and then mentioned the person name. The face
recognition system of Social Robot result shows a good accuracy
for Human-Robot Interaction.
Index Terms—Face Recognition, Social Robot, Arduino, Python
2.7, Human-Robot Interaction, SyPEHUL.
I. I NTRODUCTION
Face recognition is an image processing method to locate
the human face which need camera to capture the image of
human face. The image processing will search the important
feature of a human face on the image, thus another object
will be ignored [1]. The image processing locates human
face used by various algorithm and method, for example:
AdaBoost [2], Viola-Jones method [3] [4] [5], Roberts Cross
method [6], and other [7] [6]. For the classifier to recognize
a human face can be use; Local Binary Pattern(LBP) [8],
Hidden Markov Model(HMM) [9], Bayesian [10], Support
Vector Machine(SVM) [11], and other [12] [13].
In this decade, face recognition can be implemented to many
projects are; social robot [14] [15], attendance automation [16],
home security [17] [18] [19], game [5], and other. Especially
for Social Robot project, face recognition can be increase
the Robot’s ability for Human-Robot Interaction. Because,
Social Robot is a future technology which can communicated,
entertaining or help human’s works. The Social Robot which
has build by researchers, for example; KISMET [14], Eddie
[20], Flobi [21], Muecas [22], Probo [23], and other [15].
In this paper will be described a development of Social
Robot named SyPEHUL (System of Physic, Electronic, Hu-
manoid Robot and Machine Learning) which can recognize
and track human face. The real-time face detection and
recognize human face uses Cascade Classification method
(Viola-Jones method) and Local Binary Pattern Histogram
(LBPH) Face Recognizer Method based on OpenCV Library
and Python 2.7. Finally, the face recognition system will be
implemented to 12 Degree of Freedom (DoF) Social Robot
named SyPEHUL to recognize and tracking human face based
on Arduino microcontroller for Human-Robot Interaction.
The paper is organized as follows. In section 2, described the
theoretical background of Face Detection and Face Recogni-
tion in details. In section 3, describe the experimental method
of this research. In section 4 described the results and the
implementation of Face Recognition and Tracking to Social
Robot in detail. Finally, in Section 5 the concluding remarks
are given.
II. T HEORETICAL BACKGROUND
A. Face Detection using Viola-Jones Method
Paul Viola and Michael Jones have published the face
detection method which usually referred as the Viola-Jones
method or only Viola-Jones in 2001 [2]. This method can
detect a human face in an Image by four key concepts:
1) Haar features: Haar feature uses to find out the existence
of human face on captured image [24]. This feature used
to detect the bright side and the dark side of the captured
image [25]. The existence of Haar features is determined by
subtracting the average of the dark-region pixels by the average
of the pixel light-region. The calculation forms a rectangle
shown at Fig. 1 around the detected face shown at Fig. 2. If
the difference is above the threshold, then the Haar features
are said to be ”exist”.
(a) (b) (c)
Fig. 1. Haar rectangular feature [2]: (a)Edge, (b)Line, (c)Four rectangle.
2) Integral Image: The integral image used to speed up
the feature detection in a way increase the pixel values from
of original image. The integrated value already represents the
sum of all pixels above the threshold and is to the left of the

978-1-5386-0658-2/17/$31.00 ©2017 IEEE 315

 2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)
Fig. 2. Haar feature result.
image. By starting the addition procedure, from the top left
of the image (original pixel) become the bottom right of the
image (addition pixel result) show at Fig. 3.
(a) (b)
Fig. 3. (a) Integration process, (b) Rectangle divided into multiple segments
[2].
3) AdaBoost Machine-Learning Algorithm: Viola and Jones
use an AdaBoost algorithm because of boosted the classifi-
cation performance of a simple learning, in other word are
fast and easy computation. AdaBoost learning process gives
a number of data, thus can be grouped into the classifier. A
classifier composed of small features on the face, therefore
commonly used as pattern detection. [26]
4) Cascade Classifier: The filter chain which was shown
in Fig. 4(a) is made by a series of AdaBoost classifier. The
sequence of filters in the cascade is based by the weight of
results from the AdaBoost. The cascade eliminates a candidate
if it was not passed the first stage. The larger the filter weights
in the first chain, then to eliminate the non-face image area will
also be done quickly. Thus, a Cascade Classifier that combines
many features efficiently.
Fig. 4. Cascade classifier illustration [2].
316
B. Face Recognizer using Local Binary Pattern (LBP) Method
Local Binary Pattern (LBP) is defined as texture descriptor
to the comparison of the pixel binary value at the center of the
image with 8-pixel values around it [13]. The LBP method can
be implemented on face images to extract the features which
can be used to get a measure of the similarity between images.
On Fig. 5 illustrated the LBP process using 3x3 pixel image.
The value of the image center is reduced by the surrounding
value. If the result more or equal ”0” then it is given ”1”, but
if less than ”0” then given ”0”. Then the 8 binary value sorted
clockwise or otherwise, and convert to decimal to replace the
pixel value image center.
Fig. 5. Illustration of Local Binary Pattern (LBP).
The decimal value can be expressed in (1) [27]:
LBPP,R (x0 , y0 ) = Σp−1 p
p=0 s(gp − g0 )2 , (1)
from the function s(x), define:
s(x) = {1,x≥0
0,x<0 , (2)
from (2) aims to eliminate the variability caused by the
contrast illumination, therefore the facial image with various
illumination will get the almost similar output. By extracting
the pixels value using LBP will get a new matrix value
which converts to a histogram for facial vector features. The
histogram of these image patterns called labels which forms
a feature vector and is represented the image texture. These
histograms can be used to measure the similarity between the
images. [27]
III. E XPERIMENTAL M ETHOD
A. Method and System Design
In general, the main tools and component to build Social
Robot in this research are Personal Computer, Arduino mi-
crocontroller, Social Robot (SyPEHUL), 8MP of WebCam,
speaker, connections, and others. The system program is writ-
ten in Arduino IDLE and Python 2.7 (with OpenCV Library).
Fig. 6 show the general process of Face Recognition and
Tracking of Social Robot SyPEHUL.
 2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)

Fig. 6. General research scheme.

From the Fig. 7 can be explained that the WebCam as a

Social Robot vision sensor. The face detection process by using
Cascade Classifier Method algorithm. The process divided into
2 sections: The first process makes a training data consist of
features extraction process using LBPH Feature Method and
classifying human face using LBPH Face Recognizer Method.
In this paper, the trained data consist by face image capture of
11 people, and each people has 11 images. The second process
is a testing of trained data by trained respondents which need
LBPH features extraction. Then new image which captured in
this section is matched with Trained Data. If the classification
to identifying the person has done, the system will mention
the person’s name. From the face detection, besides can detect
the human face is can get the human face coordinate. The
coordinate data can be convert become a Social Robot Head
control to follow the human face. The system is work in
real-time based on Python 2.7, OpenCV Library and Arduino
microcontroller.
B. Hardware Design
To developed Social Robot hardware, it has a motor servo to
Fig. 7. General system scheme of Social Robot algorithm.
actuate the robots neck and its face. In addition the WebCam
as a visual sensor, speaker as the sound and the Arduino
microcontroller as controller of the actuator. The concept
and the realization of Social Robot named SyPEHUL in this
research illustrated on Fig. 8 which the hardware has developed
on the previous research [28].
SyPEHUL mechatronics is consisted by 12 motor servos
which connected with Arduino microcontroller, divided to
control; eyebrow (by 2 servos), eye (by 2 servos), mouth (by 4
servos), ear (by 2 servos), and neck (by 2 servos). Each servo
supplied by the 5-volt battery which has 100 mA of current,
and each ground of servo connected to the ground of Arduino
(a) (b)
Board illustrated on Fig. 9.
Fig. 8. SyPEHUL design; (a)Concept, (b)Realization [28].

IV. R ESULTS AND D ISCUSSION

A. Face Recognition System
The first step to develop a Social Robot which can recognize
human faces is making a database which contains by human
face image. In this research, to capture a human face image

317
 2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)

TABLE I
T HE FACE RECOGNITION ACCURACY RATE OF TRAINED RESPONDENTS IN
PERCENT (%).

No Name Accuracy(%)
1 Dyah 80
2 Mada 100
3 Rizki 80
4 Fikri 100
5 Ikhsan 80
6 Jaka 100
7 Awin 100
8 Tiara 100
9 Indry 100
10 Lutfi 100
11 Ratih 80

Fig. 9. Robot social SyPEHUL schematic [28].

need a face detection algorithm which uses Cascade Classifica-

tion method based on OpenCV library and Python 2.7. Fig. 10
is a database of some people images to make a database.

Fig. 11. The examination of face recognition by respondents.

Fig. 10. Database example. B. The Face Tracking System

The database of the image made of 11 people (6 Male and 5
Female), which each person has 11 image data. Thus, the total
image of the database is 121 images. From the obtain database,
the classification process needs to distinguish between each
person. In this research, the classification uses Local Binary
Pattern Histogram (LBPH) Feature Extraction and LBPH Face
Recognizer based on OpenCV library and Python 2.7. The
database after the classification is now called the trained data.
After that, the classified data is trained by the respondents
for clarification. The test results are shown on TABLE I is
the average of accuracy rate of face recognition by trained
respondents (in the database).
TABLE I has been describing that the accuracy of Face
Recognition of the trained respondent is 92.73%. The result
shows the good accuracy to recognize the human face, so
the method can be implemented for Human-Robot Interaction.
Fig. 11 shows the examination of face recognition by respon-
dents.
The face detection algorithm can detect a face in all pixel
area of the image (because in the image not only appear human
face). Therefore, the detected human face on an image is has a
coordinate. The implementation of face coordinate for Social
Robot SyPEHUL is to control the robot neck to track human
face. In other words, the Social Robot head can following
human face.
To control Social Robot neck must set up the resolution
of the captured image. In this research, the video capture
resolution setup is 480 x 640 pixel. Then, to find a center
coordinate of a human face from the WebCam by using (3)
and (4).
xx = x + (w/2), (3)
yy = y + (h/2), (4)
x is the initial face coordinate on horizontal, w is width
coordinate of face, xx is a center value of face coordinate on
horizontal, y is initial face coordinate on vertical, h is height
of face coordinate, yy is a center value of face coordinate

318
 2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)
on vertical, and the units in pixel. From the equation can get
the value to control the neck movement which divided into
4 movements, are; Up, Down, Right, and Left, illustrated on
Fig. 12.
Fig. 12. WebCam pixel coordinate and angle of motor servo.
To make the Social Robot Head can following human
face, from the value result of center face coordinate must
be corresponding with the angle of the motor servo. As we
know, the Social Robot Head must be set the minimum and
maximum value of angle to control the Social Robot head
which can be looking up/down (vertically) or looking right/left
(horizontally). In this research, the motor servo setup is 90o to
150o for horizontal angle arrange, and 20o to 70o for vertical
angle arrange. The servos angle setup is written and upload
into Arduino Board to control Social Robot SyPEHUL’s head.
Fig. 12 describe the information of image pixel, direction,
and servos angle of Social Robot Head. The concept to make
the Social Robot head is moving; WebCam has captured the
human face. After that, the algorithm will get the center
coordinate value which will process on Python. From Python,
the information will be sent to Arduino IDLE to control the
Social Robot head to following human face.
C. The System Implementation to Social Robot
After the Face Recognition and Tracking system are success-
ful, the system can be applied to control the Social Robot Head
SyPEHUL. The result of the examination shown on Fig. 13.
The social robot head can following the human face properly,
and can recognize the human face. The implementation of
Face Recognition has been successful, in addition, the Social
Robot can be mentioned the human name. The result of
face recognition and tracking can be apply for Human-Robot
Interaction.
319
Fig. 13. The examination of face recognition and tracking for Human-Robot
interaction.
V. C ONCLUSION
In this research has been presented a development of Social
Robot named SyPEHUL which can recognize and track the
human face. The face recognition processed by the algorithm
based on Python 2.7 (with the OpenCV library) by using
Cascade Classification and LBPH Face Recognizer method has
a good accuracy rate (92.73%). Also, the implementation of
facial tracking to control 12 DoF of Social Robot SyPEHUL
based on Arduino microcontroller works effectively to control
Robot’s head. The future works will focus on the combination
of speech recognition and facial expression to enhance the
emotional expression of SyPEHUL for Human-Robot Interac-
tion.
R EFERENCES
[1] M. S. Kalas, “Real Time Face Detection and Tracking Using OpenCV,”
International Journal of Soft Computing and Artificial Intelligence,
vol. 2, no. 1, pp. 41–44, 2014.
[2] P. Viola and M. J. Jones, “Robust Real-time Object Detection,” Cam-
bridge Research Laboratory The, Cambridge, Massachusetts, Tech. Rep.
February, 2001.
[3] S. Tikoo and N. Malik, “Detection of Face using Viola Jones and
Recognition using Back Propagation Neural Network,” International
Journal of Computer Science and Mobile Computing, vol. 5, no. 5, pp.
288–295, 2016.
[4] R. Boda and M. J. P. Priyadarsini, “Face Detection and Tracking Using
KLT and Viola Jones,” ARPN Journal of Engineering and Applied
Sciences, vol. 11, no. 23, pp. 13 472–13 476, 2016.
[5] C. Zhan, W. Li, P. Ogunbona, and F. Safaei, “A Real-Time Facial Ex-
pression Recognition System for Online Games,” International Journal
of Computer Games Technology, vol. 2008, pp. 1–7, 2008.
[6] S. Das, “Comparison of Various Edge Detection Technique,” Inter-
national Journal of Signal Processing, Image Processing and Pattern
Recognition, vol. 9, no. 2, pp. 143–158, 2016.
[7] H. C. V. Lakshmi and S. PatilKulakarni, “Segmentation Algorithm for
Multiple Face Detection in Color Images with Skin Tone Regions using
Color Spaces and Edge Detection Techniques,” International Journal of
Computer Theory and Engineering,, vol. 2, no. 4, pp. 552–558, 2010.
 2017 2nd International Conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE)

[8] C. Shan, S. Gong, and P. W. Mcowan, “Facial expression recognition

based on Local Binary Patterns : A comprehensive study,” Image
and Vision Computing, vol. 27, no. 6, pp. 803–816, 2009. [Online].
Available: http://dx.doi.org/10.1016/j.imavis.2008.08.005
[9] A. Punitha and M. K. Geetha, “HMM Based Real Time Facial Expres-
sion Recognition,” International Journal of Emerging Technology and
Advanced Engineering, vol. 3, no. 1, pp. 180–185, 2013.
[10] S. V. Hedaoo, M. D. Katkar, and S. P. Khandait, “Feature Tracking and
Expression Recognition of Face Using Dynamic Bayesian Network,”
International Journal of Engineering Trends and Technology, vol. 8,
no. 10, pp. 517–521, 2014.
[11] N. O. Sadek, N. A. Hikal, and F. W. Zaki, “Intelligent Real-Time Facial
Expression Recognition from Video Sequences based on Hybrid Feature
Tracking Algorithms,” International Journal of Advanced Computer
Science and Applications, vol. 8, no. 1, pp. 303–311, 2017.
[12] M. I. Khan and M. A.-A. Bhuiyan, “Facial Expression Recognition for
Human-Robot Interface,” International Journal of Computer Science and
Network Security, vol. 9, no. 4, pp. 300–306, 2009.
[13] M. R. G and J. Rajesham, “Weighted Local Active Pixel Pattern
(WLAPP) for effective Face Recognition,” International Journal of
Scientific & Engineering Research, vol. 4, no. 8, pp. 1540–1546, 2013.
[14] C. Breazeal, “Toward Sociable Robot,” Robotics and Autonomous Sys-
tem, vol. 42, pp. 167–175, 2003.
[15] M. Shayganfar, C. Rich, and C. L. Sidner, “A Design Methodology for
Expressing Emotion on Robot Faces,” Int. Conf. of Intelligent Robots
and Systems, vol. 12, 2012.
[16] N. Thakare, M. Shrivastava, and N. Kumari, “Face Detection and
Recognition for Automatic Attendance System,” International Journal
of Computer Science and Mobile Computing, vol. 5, no. 4, pp. 74–78,
2016.
[17] N. Sharma and I. Thanaya, “Home Security System Based on Sensors
and IoT,” International Journal of Innovative Research in Science,
Engineering and Technology, vol. 5, no. 2, pp. 10 357–10 362, 2016.
[18] P. V. Hajari and A. G. Andurkar, “Review Paper on System for Voice
and Facial Recognition using Raspberry Pi,” International Journal
of Advanced Research in Computer and Communication Engineering,
vol. 4, no. 4, pp. 232–234, 2015.
[19] R. Manjunatha and R. Nagaraja, “Home Security System and Door
Access Control Based on Face Recognition,” International Research
Journal of Engineering and Technology, vol. 4, no. 3, pp. 437–442,
2017.
[20] M. Buss, S. Sosnowski, A. Bittermann, and K. Kolja, “Design and eval-
uation of emotion-display EDDIE Design and Evaluation of Emotion-
Display EDDIE,” International Conference on Intelligent Robots and
Systems, no. June 2017, pp. 3113–3118, 2006.
[21] F. Hegel, F. Eyssel, and B. Wrede, “The Social Robot Flobi ’: Key
Concepts of Industrial Design,” IEEE International Symposium on Robot
and Human Interactive Communication, vol. 19, pp. 107–112, 2010.
[22] F. Cid, J. Moreno, P. Bustos, and P. Nunez, “Muecas: A Multi-Sensor
Robotic Head for Affective Human Robot Interaction and Imitation ,”
Sensors, vol. 14, pp. 7711–7737, 2014.
[23] I. Doroftei, F. Adascalitei, D. Lefeber, B. Vanderborght, and I. A.
Doroftei, “Facial expressions recognition with an emotion expressive
robotic head,” 7th International Conference on Advanced Concepts in
Mechanical Engineering, vol. 147, 2016.
[24] M. K. Dabhi and B. K. Pancholi, “Face Detection System Based on
Viola - Jones Algorithm,” International Journal of Science and Research,
vol. 5, no. 4, pp. 2015–2017, 2016.
[25] M. Chaudhari, S. Sondur, and G. Vanjare, “A review on Face Detection
and study of Viola Jones method,” International Journal of Computer
Trends and Technology (IJCTT), vol. 25, no. 1, pp. 54–61, 2015.
[26] P. Viola, O. M. Way, and M. J. Jones, “Robust Real-Time Face
Detection,” International Journal of Computer Vision, vol. 57, no. 2,
pp. 137–154, 2004.
[27] M. A. Rahim, M. N. Hossain, T. Wahid, and M. S. A. Azam, “Face
Recognition using Local Binary Patterns (LBP),” Global Journal of
Computer Science and Technology Graphics & Vision, vol. 13, no. 4,
2013.
[28] W. S. M. Sanjaya, D. Anggraeni, A. Juwardi, and M. Munawwaroh,
“Design of Real Time Facial Tracking and Expression Recognition for
Human-Robot Interaction,” in ICCSE, 2018, pp. 1–10.

320