Performing Mouse Events by Hand Gesture Recognition

T.K. Manglam (MS,C.Tech) Punit Tiwan (MS,C.Tech) Akhil Bansal (MS,C.Tech)

Performing Mouse Events by Hand Gesture Recognition

Project taken as part of course Multimedia Systems under the guidance of Prof. Santanu Chaudhary .

Introduction

So what has been Done ?

Various combinations of finger movements have been decided , where each combination corresponds to some particular mouse event . The presence of a particular combination is recognized to decide which mouse event to perform. To recognize finger movement , image segmentation has been used . For good segmentation results , color image segmentation has been used . For efficient segmentation , colored caps with different hues have been used .

So what has been Done ?

Perform various mouse events by moving fingers Various combinations of finger movements have been decided , where each combination corresponds to some particular mouse event . The presence of a particular combination is recognized to decide which mouse event to perform. To recognize finger movement , image segmentation has been used . For good segmentation results , color image segmentation has been used . For efficient segmentation , colored caps with different hues have been used .

Functionalities developed :
Move mouse pointer by hand gesture . Right Click mouse event by hand gesture Left Click mouse event by hand gesture . Drag/Drop an item by hand gesture .

Specifications
Platform : Windows7 Processor : Core2Duo RAM : 2GB

The application has not been tested on lower configurations , however it should work on system with lesser RAM and slower processor as well .

Hardware and Other Requirements

A camera , which may be a web-cam fixed on laptop or an external web-cam . 2 caps of colors with different hue .

Tools Used
MSDN library : The functions defined in <windows.h> have been used to move
perform various events on mouse .

OpenCV library : The funtions defined in OpenCV libraries have been used for
quick reference .

Flow of Algorithm Implemented

Learn colors of caps to be used for segmentation . Recognize the presence of each colored cap and segment the colored caps in each frame Determine the mouse event to be performed Find the displacement of colored cap in consecutive frames and move the cursor on screen .

Implementation of Algorithm

Step 1 : Learn colors of caps to be used for segmentation .

Algorithm Implemented

Initially start the camera using cvCameracapture( ) . Bring both caps in front of the camera and click an image using cvQueryFrame( ) . In the image , click at any point on the image of caps one by one . Determine the position of the point clicked on using GetCursorPos( ) . Find the hue component of the corresponding pixel in the image using setLimit( ).

Detecting Colors

Algorithm Implemented
Step 2 : Determine the presence of each colored cap and segment the colored caps in each frame

Capture the image continuosly using while( ) loop . Change the image in HSV plane using cvConvertScale( ) Find pixels which lies in the range of hue of a colored cap . Find the contour of the caps in the image using cvFindContour( ) .

Problems Faced
Step 1 : Learn colors of caps to be used for segmentation . Step 2 : Determine the presence of each colored cap and segment the colored caps in each frame

Segmentation using RGB space results in poor segmentation . Segmentation using connected components algorithm results in slower segmentation . Determining Hue of the colored caps using getPixel( ) of <windows.h> give different range of Hue .

Algorithm Implemented
Step 3 : Determine the mouse event to be performed Assumption : First color is green color and second color is red color .
Case 1 : Only green color present in image taken : Move mouse according to the displacement of green cap in present and previous image . Case 2 : Only red cap is present : Use left movement of red cap for Left Click , and right movement of red cap for Right Click Case 3.a : Both green and red cap present in image : No movement of mouse even if green cap is moved Case 3.b: Both green and red cap present in image and are nearby : Drag the mouse according to the displacement of caps .

Moving Mouse Pointer

Right Click and Left Click

Dragging and Object

Problems Faced
Step3 : Determine the mouse event to be performed

A lot of combinations were tried before deciding on final combination . A nice observation is that finger in the image moves in the opposite direction

Algorithm Implemented
Step 4 : Find the displacement of colored cap in consecutive frames and move the cursor on screen .

Stepwise velocity taken to make cursor movement look realistic . For very slow motion of finger , cursor doesnt move . For very fast motion of finger , cursor doesnt move .

Problems Faced
Step4 : Find the displacement of colored cap in consecutive frames and move the cursor on screen .

A lot of combinations were tried to make mouse pointer movement look realistic . Since finger doesnt have any support , so it keeps on displacing , and hence valid assumptions need to be tried to ignore small motion . If finger moves out of range of the camera, then bringing back finger cause motion of cursor in opposite direction , which might not be desirable .

Other Problems
Improper Illumination Problem Illumination Changes Color of background shouldnt contain any portion having same color as that of caps . The finger movement needs to be constrained in the range of camera which sometimes is impalpable experience for first time user .

Future Aspects
For our project we intend to use mobile camera for capturing images and using bluetooth to communicate with computer . Otherwise , Hand Gesture recognition to control events has a lot of scope e.g. streching an image by hand , playing games .

Is the code ready for commercial purpose ?

Not yet !!!! Since the most important component of a software cycle ,i.e, testing has not been done rigorously . However , we believe with little more hit and trial and testing , it can be made good enough for commercial purposes .

ThanK YoU