Real-Time Hand Gestures

Recognition
By
Syed Akhlaq Hussain Shah
2013-NUST-MS-EE-61213F
Supervisor
Dr. FirstName LastName
Department of Electrical Engineering
A thesis submitted in partial fulllment of the requirements for the degree
of Masters of Science in Electrical Engineering (MS EE)
In
School of Electrical Engineering and Computer Science,
National University of Sciences and Technology (NUST),
Islamabad, Pakistan.
(May 12,2015)
Approval
It is certied that the contents and form of the thesis entitled Real-Time
Hand Gestures Recognition submitted by Syed Akhlaq Hussain Shah
have been found satisfactory for the requirement of the degree.
Advisor: Dr. FirstName LastName
Signature:
Date:
Committee Member 1: Dr. Name of First Committee Member
Signature:
Date:
Committee Member 2: Dr. Name of Second Committee Member
Signature:
Date:
Committee Member 3: Dr Name of Third Committee Member
Signature:
Date:
i
Dedication
I would like to dedicate this thesis to my mother.
ii
Certicate of Originality
I hereby declare that this submission is my own work and to the best of my
knowledge it contains no materials previously published or written by another
person, nor material which to a substantial extent has been accepted for the
award of any degree or diploma at NUST SEECS or at any other educational
institute, except where due acknowledgement has been made in the thesis.
Any contribution made to the research by others, with whom I have worked
at NUST SEECS or elsewhere, is explicitly acknowledged in the thesis.
I also declare that the intellectual content of this thesis is the product
of my own work, except for the assistance from others in the projects de-
sign and conception or in style, presentation and linguistics which has been
acknowledged.
Author Name: Syed Akhlaq Hussain Shah
Signature:
iii
Acknowledgment
I would like to thank my parents and my friends for supporting me, both
emotionally and economically.
iv
Table of Contents
1 Introduction 1
2 Literature Review 2
3 Proposed Approach 4
v
List of Figures
2.1 Visual TouchPad . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Hand Mouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 SixthSense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . 5
vi
Chapter 1
Introduction
With every passing day the computing machines are becoming more power-
ful, in terms of computations,than ever. Also the size of these machines is
reducing. The reduction is size is in response to the need of mobility. People
want to have their smart computing devices with them all the time, and as
a result computing machines are becoming smaller in size and are easier to
carry. But at the same time people want to have a very intuitive and natural
interaction with their smart devices. So The only bottleneck in this reduction
of size of smart devices are the interfaces that are used input data in and get
data out. So devices with very intuitive interaction do not provide mobility
and those who do not provide friendly interaction [6].Interfaces being used
these days only oer a narrow bridge between the circuitry and brain. The
limitations of the current tools of human computer interaction hold back the
ability of computers to serve humans in even more powerful way [3].
Human hand is the most versatile, natural, intuitive and the most eective
tool possible for human computer interaction.During the last few years a
lot of research has been done to replace the conventional tools of human
computer interaction with natural hands.
My proposed system uses hand gestures as a tool of interaction, providing
mobility and friendly interaction in one package. In my work, human hand
is detected in real time using colour and as well as feature information. Once
the hand is segmented, it is tracked used cam shift. Fingertips are extracted
using convex hull from the tracked hand. Then single stroke gestures are
recognized and translated into actions. The report is organized as follows.
First a literature review, entailing famous techniques used hand and nger
tip detection, the the proposed solution is described.
1
Chapter 2
Literature Review
Methods that man has been using to interact with computers have travelled
a long way. The emphasis has been developing such methods that make
human-computer interaction more friendly, intuitive and exible than ever.
The journey still continues and particularly for the few decades there has
been very a great deal of research work done in this eld. Hand gestures
have always been considered a very promising method for the development
of a natural and intuitive human-computer interaction. In gesture based
interfaces the location, orientation and posture of human hand is used as
the interactive method. Research in the eld of human of human computer
interaction can classied in three main eld: Sensor, Audio and visual based
HCI.In visual based human computer interaction,hand gestures have always
been considered a very promising method for the development of a natural
and friendly human-computer interaction. On the other hands it brings some
serious issues in detection:
The illumination changes with time. The skin tone varies dramatically
within and across individuals, goes from black to white. The quality of the
camera also plays a vital role, dierent cameras have dierent output for
a particular scene in a given environment.Blurring of colours makes hand
detection dicult as the hand is normally close to the camera and quick
movement makes the hand blurred.The ambient light and shadows change
the apparent colour of skin. Also the shape of the hand is very irregular
which makes hand detection based on features very dicult.
In hand detection, the research conducted can be classied in two ma-
jor areas: hand detection based on skin colour and feature based segmenta-
tion [2].Francke [1]used Boosted Classiers and Active Learning for real hand
detection and gesture recognition.Zhang elal [7] proposed a vision based ges-
tural interface by the name of Visual Panel, the panel in a very robust manner
2
CHAPTER 2. LITERATURE REVIEW 3
uses hands to use mouse, keyboard and joystick.
Laszlo proposed Visual TouchPad [7], a low cast device that allows inter-
action with desktop PCs, laptops or large wall displaysboth hands, as shown
in gure2.1.
Figure 2.1: Visual TouchPad
Kurata [4] proposed a system named Hand Mouse, used GMM hand-color
classication and mean shift tracking using a wearable camera,as shown in
gure2.2.
Figure 2.2: Hand Mouse
P. Mistry proposed SixthSense [5], hand gestures are used as the in-
teractive mechanism with projected information on any surface or wall, but
it required user to wear dierent colored markers interaction,as shown in
gure2.3
Figure 2.3: SixthSense
Chapter 3
Proposed Approach
The proposed solution uses natural hand as the tool of real time interac-
tion with computing device where as any plain surface using projected data
can be used as the interactive surface, thus providing mobility and friendly
interaction with computers in a single package.
The proposed solution comprises of a simple webcam, a pocket projector
and a real time mobile computing device. Projector, using a simple mirror,
will project the computer display on the wall or any other plain surface,
the camera will be used to detect and track the ngertips using various
digital image processing and computer vision techniques. Only Single stroke
gestures will be interpreted to act, thus providing an interaction mechanism
for the projected applications. The logical block diagram of the proposed
system is shown in gure 3.1.
Figure 3.1: Proposed Solution
On the captured image, rst of all, hand detection is applied using skin
colour information as well as features classication. Once hand is detected, it
will be tracked using cam shift tracking.On the tracked hand ngertip detec-
tion and gesture recognition will be performed.The output can be displayed
4
CHAPTER 3. PROPOSED APPROACH 5
on any plain surface using projector.
Hand detection in cluttered background is the most critical and dicult
part.My contribution in hand detection is described in gure 3.2 .
After capturing the frame, rst of all, hand will be detected using Haar
like features.Using the detected hand, histogram will be computed. Using
the histogram of the detected hand, probability of skin pixels in the next
frame are computed.Based on that probability hand area is estimated.So the
proposed solution should work in cluttered background and under varying
lighting conditions.
Figure 3.2: Proposed Solution
Bibliography
[1] Hardy Francke, Javier Ruiz-del Solar, and Rodrigo Verschae. Real-time
hand gesture detection and recognition using boosted classiers and ac-
tive learning. In Domingo Mery and Luis Rueda, editors, Advances in
Image and Video Technology, volume 4872 of Lecture Notes in Computer
Science, pages 533547. Springer Berlin Heidelberg, 2007.
[2] P. Kakumanu, S. Makrogiannis, and N. Bourbakis. A survey of skin-color
modeling and detection methods. Pattern Recognition, 40(3):1106 1122,
2007.
[3] Mathias Kolsch et al. Vision based hand gesture interfaces for wearable
computing and virtual environments. 2004.
[4] T. Kurata, T. Okuma, M. Kourogi, and K. Sakaue. The hand mouse:
Gmm hand-color classication and mean shift tracking. In Recognition,
Analysis, and Tracking of Faces and Gestures in Real-Time Systems,
2001. Proceedings. IEEE ICCV Workshop on, pages 119124, 2001.
[5] Pranav Mistry, Pattie Maes, and Liyan Chang. Wuw-wear ur world:
a wearable gestural interface. In CHI09 extended abstracts on Human
factors in computing systems, pages 41114116. ACM, 2009.
[6] S.A.H. Shah, A. Ahmed, I. Mahmood, and K. Khurshid. Hand gesture
based user interface for computer using a camera and projector. In Signal
and Image Processing Applications (ICSIPA), 2011 IEEE International
Conference on, pages 168173, Nov 2011.
[7] Zhengyou Zhang, Ying Wu, Ying Shan, and Steven Shafer. Visual panel:
Virtual mouse, keyboard and 3d controller with an ordinary piece of pa-
per. In Proceedings of the 2001 Workshop on Perceptive User Interfaces,
PUI 01, pages 18, New York, NY, USA, 2001. ACM.
6