Face and Speech Recognition Based Smart Home
An offline Security and Automation System
Alina Munir
Dept. of Electrical and Computer Engineering
Air University
Islamabad, Pakistan
alinamuneer@gmail.com
S. M. Mohsin Raza
Dept. of Electrical and Computer Engineering
Air University
Islamabad, Pakistan
Mohsin4451@gmail.com
Abstract— Over the course of last decade, home security and
automation systems have been gaining extreme popularity
because of the enhancement in the field of science and the
concomitant risks of breaking in the system. This paper envisions
a smart home based on the two independent and emerging
technologies viz face recognition, for the security purposes, and
speech recognition, for the automation purposes. Moreover, the
latest work done is discussed briefly and why an offline system
was an urgent need to fill the gap in the recent technology
protocols introduced in the related field. Security system ensures
the working of the smart home by taking the image on the
ringing of the bell and comparing it with the stored database of
owners in a robust and reliable manner through Raspberry Pi
microcontroller and OpenCV. Voice control over the whole house
makes this system effective and efficient which could turn out to
be a great charity to handicapped, elder and paralyzed people in
the remote areas with no access to internet by a single voice
command in the mic connected to Arduino V3 module.
Keywords—Offline face detection and recognition,
Raspberry-pi, OpenCV, LBPH, Arduino V3 module.
I. INTRODUCTION
Security is important for everyone, everywhere, in the world
and due to the emerging technology, risks have been raised
higher and improvement in safety measurements has become a
necessity. Home automation is quite a hot debate and has a
high demand due to the helping nature of the technology for
handicapped, paralyzed and elder people. In this paper main
focus is to develop an offline smart home by making both face
recognition and speech recognition independent of the internet
connection for the use in remote areas. Many countries are
embracing advanced security systems. Accuracy in the
identification is the main concern and this can be achieved by
many different topologies as biometric finger print, key
passwords, sensors and facial recognition techniques [12].
Technology of Face Recognition is beneficial for a wide range
of applications. Image Detection and recognition of a low-
Kashif Ehsan S.
Dept. of Electrical and Computer Engineering
Air University
Islamabad, Pakistan
kashifehsen@rocketmail.com
Mumajjed Mudassir
Dept. of Electrical and Computer Engineering
Air University
Islamabad, Pakistan
mumajjed@mail.au.edu.pk
resolution up to 21x21 pixels is possible [6] and is the most
suitable solution for the controlled environment, which makes
it perfect for the guard system of our smart home.
In previous research, vast amount of work has been done to
make the face recognition technology more reliable and
accurate. Face recognition has already been done by various
different techniques; Principle Component Analysis, Linear
Discriminant Analysis, Local Binary Pattern, Haar-Cascade
Classifier, each having a competitive advantage over others
[8]. Priyanka and Namrata distinguished the spoof images
from genuine faces by analyzing factors i.e. reflection,
blurriness, chromatic effects and color-density [1]. Kim, Park,
Hong and Chung worked in adjusting the location of the
camera by the sound localization technique used on the door
to capture better CCTV footage [2]. To confirm the identity of
visitor, local communication with the owner’s phone has been
done by many means Zigbee [13], Bluetooth [15], GSM [14]
and UWB [10]. Mrutyunjaya, Chiranjiv, Abhijeet and
Biswajeet introduced a GSM and FPGA based system to
update the owner about the security of home and ask for the
permission of the entrance of visitors [3]. Nasimuzzaman,
Shiblee and Srijon used IOT (internet of things) to send the
image of the visitor to the owner and to enable the owner to
communicate back to the visitor by sending a message back on
the door screen [4].
In the proposed offline system, face is detected and then
recognized by matching the image taken in the real time with
the prestored database as also done in [7], without any
communication with the owner, for a faster, easy to use/install
and cheap system to help technologically uneducated people
in remote areas. Raspberry-Pi microcontroller is programmed
with Python language to work with OpenCV algorithms for
offline face recognition of smart home. Bienvenido Bartido
studied different algorithms of OpenCV, i.e. eigenface,
fisherface and LBPH, and has discussed that pre-processing
techniques, pose and illumination, affect the accuracy of
algorithms in hugely different manners and paper mainly
showed that pose and pose + illumination techniques highly different individuals in different lights and with different
improved the speed and threshold of LBPH compared to the angles to be more precise about each individual’s identity.
other algorithms of OpenCV [5]. LBP is used for the
improvement in the accuracy of results
.
Previously, speech recognition has been done by online
Bluetooth [18], Amazon Alexa [11] and Google Assistant [19]
for different purposes in the phones and automation systems.
Milton and Khan created an internet app and GSM network
for home automation [17] but this is quite costly and a
complex system to use/install. Personalized offline voice
recognition introduced by Google Inc happened to be 7 times
faster than the online counterpart [16]. For faster, cheap and
easy to use system; internet independency of smart home is
achieved by using Arduino V3 Module [20]. for speech-to-text
conversion and for the training of voice commands, through a
microphone, and comparison with real time voice orders to
control the working of appliances as commands are not
interpreted rather matched with the prestored commands [9].
Paper is arranged as a pattern of following section: 2)
‘Methodology’ to understand the working of smart home
through flowcharts and briefing of systems and technologies
used, section 3) ‘Results and Discussion’ to have a review of
progress and work done, section 4) ‘Conclusion’ and
‘References’.
II. METHODOLOGY
In Face recognition a real time image is given to the
system and through feature extraction process that image is Fig. 1. Block Diagram of Smart Home
checked against the database already stored in the system.
Features of test image will later be compared to the features of A. Face Recognition
database. Minimum difference in the features will help in
identification of the profile of person in the test image. In the
first phase of making a smart lock system using facial
recognition, raspberry pi 3B (the source hardware) having start
python & open-cv (open source computer vision) was used to
make this work precise and fast. Face detection was done by
using Haar cascade classifiers of open cv, that detected the camera
faces and then the detected faces were being stored by using
python commands in a database. Training was done so that the
images inside the data base were converted to yml file for Face
further use in recognition process. Finally, recognition process
detection
was done by using LBPH while keeping check on a yml file
created during training this process, if the image of the person
in front of the camera matched with the database records,
which meant that the difference of two images did not increase Face No
the given threshold, the door lock would open or else alarm detected
would activate. Face Yes Open
For automation phase, Appliances were controlled by Yes recognized door
offline speech recognition process through Arduino V3
module for speech to text and text to speech conversion and
transferring the packets by using serial communication while Face
Execute
accessing ip addresses of wifi module without internet. As recognition
described in Fig.1, in the project Face recognition technology profile
was installed on the entrance of the house to analyze if the
individual entering the house was an alien or a known entity.
Originally raspberry-pi was trained with the 100 images of 3 end
Fig. 2. Working of Face Recognition System
 As shown in Fig.4 in dataset creator and training process
raspberry-pi was given the command of ‘videocapture(0)’ to
take a video and through further commands in python and
OpenCV burnt in the raspberry-pi image was extracted from
the video in which the face was detected by harr cascade
classifier to detect the face like features i.e. eyes, mouth, nose
and ears. For a detected face the ID of the person was asked.
This image was grey scaled and rectangular features of the
face were extracted. 100 images of each ID were captured and
stored under the class of an entered identity in yml file. In
trained database each ID had 100 images with the name of the
individual.
In recognition stage whenever the doorbell rang and the
camera turned on, it detected the face through harr cascade
classifier and if a face is found camera captured the video and
taken image from the video is grey scaled and rectangular
components of the image were extracted and stored in xml
file. Now those yet unrecognized image’s rectangular
components were compared to the rectangular components of
the trained images pre-stored with known IDs in yml file. If
the difference of the features were lesser than the maximum
allowed distance, the known identity of the trained image
closer to the recognized image appeared on the screen and
door opened. On the other hand, if the image did not match
any of the known images an announcement places inside the
house and image was stored for future records and door was
kept closed.

Fig. 4. Grey Scaled Database of Person of specific ID

Fig. 3. Schematics of door lock system with camera and bell

Fig. 5. Execution of Person’s profile
B. Speech Recognition
Fig. 6. Working of Offline Speech Recognition System
In offline fast speech recognition innovation V3 module
was used for SST (speech to text) and TTS (text to speech)
which also offered a built-in port for microphone. This Voice3
module was connected to Arduino where coding was done for
the conditioning of text command as a signal to relay through
the GPI/O pins of Arduino. For another room i.e., Kitchen, the
high or low signal i.e. 129 or 140 respectively was sent to D2
pin of server esp-8266. With the help of the coding in the
server esp-8266 the static IP of the Wi-Fi router was accessed
without any internet connectivity and the signal was sent to
the client esp-8266. In client esp-8266 code was burnt to catch
the static IP address generated by the router, hence the offline
communication occurred. This client esp-8266 conveyed the
pin status as ‘high’ or ‘low’ to relay as a binary 1 or 0 signal.
Appliances were connected to relays and the corresponding
commands passed on either high or low voltage to the GPI/O
pins.
III. RESULTS ANS DISCUSSION
Face recognition performed by computers is different from
how we differentiate images. It compares pixels by pixels and
find the difference between the trained and real time captured
image and if that minimum difference is less than the threshold
value it considers it the same. Threshold is coded in the Python
language in OpenCV and it is changeable. Firstly, at threshold
65 almost 80% of the recognition failed due to same facial hair
of two persons but third person was recognized correctly due to
the lack of facial hair which distinguished widely. Secondly,
after reducing the threshold to 40 a drastic improvement of
90% in the results were obtained, making it a perfect minimum
difference limit for the system. Thirdly, reduction in threshold
level up to 30 was again increasing the fault rate in face
recognition. There are other factors than threshold to improve
the success rate of face recognition system which have to be
considered under the account for better performance. 20
images trained in the database for each class was not very
helpful and when each ID was trained with 100 different
images, system had a wide range of images to match from and
a huge improvement was discovered as well as delay was
highly reduced. To compensate differences in the lighting on
the door throughout the day, database was trained with images
with different poses and lightings which highly affected the
speed of our system.
Speech recognition has two main factors that affect the
performance i.e. voice recognition and command recognition.
When system was trained with the voice of person A, it was
hard for the system to compare the voice commands given by
person B to switch ON/OFF the appliances. Plus, V3 module
can only store up to 80 commands so it is not very favorable to
train for same command with different voices so an
improvement was made by training the system with standard
accent and an average tone. To avoid the failure due to high
noise and distortion in the surroundings, training is done in a
bit noisy ambiance to avoid failures while working with noisy
real time voice commands and a low-quality mic is used to
catch the loudest voice. There are other ICs that cancel out the
noise factors and provide a better scenario to work in but that is
a very costly solution. Another point to ponder, V3 Voice
Recognition Module only recognized and trained one word
after the ‘trigger’ command i.e. ‘kitchen’, ‘room’, ‘lounge’ or
‘parking’ etc. ‘Room One’ or ‘Room two’ were considered
same because the system did not pick the second words i.e.
‘one’ and ‘two’. Improvement has to be made in this regard.
IV. CONCLUSION
Smart home works as a robust and effective offline
security and automation system for handicapped and paralyzed
people as it is an easy to use/install, cheap and fast system.
Face recognition is highly accurate and efficient technology
and harr cascade classifier is flawless in detecting the face like
features. Open CV functions ensure the recognition of the
closest known faces in the minimal delay possible with a few
improvements i.e. using preprocessing techniques, diverse
options in database per person and optimal threshold
difference in python code. For speech recognition without
internet offered the least possible delay of 0.4second and
accuracy is tested out to be 96% due to the noisy
environments, sometimes the correct catch of the command is
lost. Internet independency opens a new path for working of
this modern technique in the remote areas for technology
illiterate population as well as for emergency situations.
 [19] Michaely AH, Zhang X, Simko G, Parada C, and Aleksic P, “Keyword
Spotting for Google Assistant Using Contextual Speech Recognition,”
REFERENCES Proceedings of ASRU, 2017.
[20] Mukesh Kumar and Shimi S.L, “Voice Recognition Based Home
Automation System for Paralyzed People,” in International Journal of
[1] Priyanka P. Raut and Namrata R. Borkar, “Techniques and Advanced Research in Electronics and Communication Engineering
Implementation of Face Spoof Recognition: Perspectives and (IJARECE), Vol. 4, Issue 10, October 2015.
Prospects,” IJESC , Vol. 8, Issue 1, 2018.
[2] T. Kim, H. Park, S. H. Hong, and Y. Chung, "System of face recognition
and sound localization for a smart door phone," IEEE Trans. on
Consumer Electronics, vol. 59, no. 3, pp. 598-603, Aug. 2013
[3] Mrutyunjaya Sahani, Chiranjiv Nanda, Abhijeet Kumar Sahu and
Biswajeet Pattnaik, “Web-Based Online Embedded Door Access
Control and Home Security System Based on Face Recognition,” Power
and Computing Technologies [ICCPCT], 2015 International Conference
on Circuit, 2015.
[4] Md. Nasimuzzaman Chowdhury, Md. Shiblee Nooman, and Srijon
Sarker, “Access Control of Door and Home Security by Raspberry Pi
Through Internet,” International Journal of Scientific & Engineering
Research, Vol. 4, Issue 11, November 2013.
[5] Bienvenido Bartido Abad Jr. St. Paul, “Proposed Image Pre-processing
Techniques for Face Recognition Using OpenCV,” 3rd SPUP
International Research Conference, St Paul University Philippines,
Tuguegarao City, February 2018.
[6] Tomasz Marciniak, Agata Chmielewska, Radoslaw Weychan, Marianna
Parzych, and Adam Dabrowski, “Influence of low resolution of images
on reliability of face detection and recognition,” Multimedia Tools and
Applications, June 2013.
[7] “A Review Paper on Face Recognition Techniques”, IOSR Journal of
Computer Engineering (IOSR-JCE), e-ISSN: 2278-0661, p-ISSN: 2278-
8727, PP 81-83.
[8] Varsha D. Patil, “A Review On Face Recognition Techniques,” in 8 th
Nation Conference on Emerging trends in Engineering and Technology
(NCETET-2018), 10TH March 2018.
[9] Dr. J. Selvakumar and K. Kannan, “Arduino Based Voice Controlled
Robot,” International Research Journal of Engineering and Technology
(IRJET), Vol. 02, Issue 01, Mar-2015.
[10] M Yan, T Jiang, and Y Liu et al., "QGA-based feature selection of target
recognition by UWB communication signal in foliage
environment[C]", Communication Workshop (ICCW) 2015 IEEE
International Conference, pp. 2524-2527, 2015.
[11] Xinyu Lei, Guan-Hua Tu, Alex X. Liu, Kamran Ali, Chi-Yu Li, and
Tian Xie, “The Insecurity of Home Digital Voice Assistants – Amazon
Alexa as a Case Study,” arXiv preprint arXiv:1712.03327, 2017.
[12] Vernon S. and Joshi S.S., "Brain–Muscle–Computer Interface: Mobile-
Phone Prototype Development and Testing," IEEE Transactions,
Information Technology in Biomedicine, vol.15, no.4, pp.531-538, July
2011.
[13] J. Han, C. S. Choi, W. K. Park, I. Lee, and S. H. Kim, "Smart home
energy management system including renewable energy based on
ZigBee and PLC," 2014 IEEE International Conference on Consumer
Electronics (ICCE), Las Vegas, NV, pp. 544-545, 2014.
[14] P. S. Chinchansure and C. V. Kulkarni, "Home automation system based
on FPGA and GSM, "Computer Communication and Informatics
(ICCCI), 2014 International Conference, Coimbatore, pp. 1-5, 2014.
[15] M. A. Ullah and A. R. Celik, “An Effective Approach to Build Smart
Building Based on Internet of Things (IoT)”, Journal of Basic and
Applied Scientific Research, issues 6, pp. 56-62, 2016.
[16] Ian McGraw, Rohit Prabhavalkar, Raziel Alvarez, Montse Gonzalez
Arenas, Kanishka Rao, David Rybach, Ouais Alsharif, Has¸im Sak,
Alexander Gruenstein, Franc¸oise Beaufay, and Carolina Parada,
“Personalized Speech Recognition on Mobile Devices,” Google Inc. 11
March 2016.
[17] M. A. A. Milton and A. A. S. Khan, "Web based remote exploration and
control system using android mobile phone," Informatics, Electronics &
Vision (ICIEV), 2012 International Conference, pp. 985-990, 2012.
[18] R. Piyare and M. Tazil, "Bluetooth based home automation system using
cell phone," Consumer Electronics (ISCE), 2011 IEEE 15th
International Symposium on, 2011.