1

CHAPTER 1
INTRODUCTION












2

Introduction
A Touch screen is an electronic visual display that can detect the presence and location oI a
touch within the display area. The term generally reIers to touching the display oI the device
with a Iinger or hand. Touch screens can also sense other passive objects, such as a stylus.
Touch screen is common in devices such as all-in-one computers, tablet computers, and smart
phones.
Touch screen has two main attributes. First, it enables one to interact directly with whatever
is displayed, rather than indirectly with a cursor controlled by a mouse or touchpad. Secondly,
it lets one do so without requiring any intermediate device that would need to be held in the
hand. Such displays can be attached to computers, or to networks as terminals. They also play
a prominent role in the design oI digital appliances such as the personal digital
assistant (PDA), satellite navigation devices, mobile phones, and video games.
The Proposed design implements touch screen using Computer vision. Computer vision is
one out oI many areas that want to understand the process oI human Iunctionality and copy
that process with intention to complement human liIe with intelligent machines. For better
human -computer interaction it is necessary Ior the machine to see people. Computer vision is
the science and technology oI machines that see, where see in this case means that the
machine is able to extract inIormation Irom an image that is necessary to solve some task. As
a scientiIic discipline, computer vision is concerned with the theory behind artiIicial systems
that extract inIormation Irom images. The image data can take many Iorms, such as video
sequences, views Irom multiple cameras, or multi-dimensional data Irom a medical scanner.
As a technological discipline, computer vision seeks to apply its theories and models to the
construction oI computer vision systems. Examples oI applications oI computer vision include
systems Ior Controlling processes, Detecting events, Organizing inIormation, Modeling
objects or environments, Interaction.
Multitouch denotes a set oI interaction techniques that allow computer users to control
graphical applications with several Iingers. Multi-touch devices consist oI a touch screen
(e.g., computer display, table, wall) or touchpad, as well as soItware that recognizes multiple
simultaneous touch points, as opposed to the standard touchscreen (e.g. computer touchpad,
ATM), which recognizes only one touch point.


3

Problem Statement
Current wireless human-computer interaction devices such as wireless mice and touch
screens, by-and large, incorporate a sophisticated electronic architecture. The sophistication
achieves wireless capabilities but carries over a cost overhead. We lay the Ioundation Ior
developing a novel human-computer interaction device with reduced hardware sophistication.
We developed a inIrared based touch screen-type device Ior desktops using only single
webcam and an array oI seven inIrared LEDs, as opposed to, typically, 2 webcams in
conventional inIrared touch screens. Users can naturally use their Iingers or other tip pointers
to issue commands and type texts. Enabling it with multi-gesture Iunctionalities like rotate ,
zoom , scroll up and down etc have Iurther enhanced the degree oI interaction.
Hence we will obviate mouse as pointing device. The setup is easy: position a camera so it in
the image, which determines the projective mapping between the image plane and the screen.
The proposed approach uses inIrared technology Ior Iinding location oI touch. The region oI
touch is seen by computer as a glow due to inIrared emissions. InIrared webcam is used to
capture images oI the screen. AIter detecting touch on the screen, corresponding signals are
sent to perIorm mouse actions. The mouse actions that can be perIormed includes leIt click as
a tap and right click is simulated by keeping stylus Ior about 1 sec at point oI event. The
overall perIormance oI the system is Iast, accurate, and reliable.
The need to undergo with this project arises with the Iacts like:
O Existing touch screen methodologies uses capacitive or resistive approach, which are
costly. A touch screen based on capacitive and resistive approach suitable Ior desktop
costs around Rs.20,000 to Rs.50,000.
Key Features oI the proposed design are as Iollows:
O Use oI InIrared Ior detecting the touch, which is not interIered by the background.
O Works with desktop monitors/screens.
O Single Camera is placed inIront oI the screen.
O Design capable oI detecting multiple-gestures (gestures that requires multiple
touches).
O Requires minimal memory and computational resources.


4













CHAPTER 2
ACKGROUND MATERIAL











3


ackground
Jarieties of touchscreen technologies.
Resistive
A resistive touchscreen panel is composed oI several layers, the most important oI which are
two thin, electrically conductive layers separated by a narrow gap. When an object, such as a
Iinger, presses down on a point on the panel's outer surIace the two metallic layers become
connected at that point: the panel then behaves as a pair oI voltage dividers with connected
outputs. This causes a change in the electrical current, which is registered as a touch event and
sent to the controller Ior processing.
Surface acoustic wave
SurIace acoustic wave (SAW) technology uses ultrasonic waves that pass over the
touchscreen panel. When the panel is touched, a portion oI the wave is absorbed. This change
in the ultrasonic waves registers the position oI the touch event and sends this inIormation to
the controller Ior processing. SurIace wave touch screen panels can be damaged by outside
elements. Contaminants on the surIace can also interIere with the Iunctionality oI the
touchscreen.
Capacitive
A capacitive touchscreen panel is one which consists oI an insulator such as glass, coated with
a transparent conductor such as indium tin oxide (ITO). As the human body is also an
electrical conductor, touching the surIace oI the screen results in a distortion oI the
screen's electrostatic Iield, measurable as a change in capacitance. DiIIerent technologies may
be used to determine the location oI the touch. The location is then sent to the controller Ior
processing.

Figure Capacitive Touchscreen of a mobile phone
6

Surface capacitance
In this basic technology, only one side oI the insulator is coated with a conductive layer. A
small voltage is applied to the layer, resulting in a uniIorm electrostatic Iield. When
a conductor, such as a human Iinger, touches the uncoated surIace, a capacitor is dynamically
Iormed. The sensor's controller can determine the location oI the touch indirectly Irom the
change in the capacitance as measured Irom the Iour corners oI the panel. As it has no moving
parts, it is moderately durable but has limited resolution, is prone to Ialse signals Irom
parasitic capacitive coupling, and needs calibration during manuIacture. It is thereIore most
oIten used in simple applications such as industrial controls and kiosks.
Projected capacitance
Projected Capacitive Touch (PCT) technology is a capacitive technology which permits more
accurate and Ilexible operation, by etching the conductive layer. An X-Y grid is Iormed either
by etching a single layer to Iorm a grid pattern oI electrodes, or by etching two separate,
perpendicular layers oI conductive material with parallel lines or tracks to Iorm the grid
(comparable to the pixel grid Iound in many LCD displays).
The greater resolution oI PCT allows operation without direct contact, such that the
conducting layers can be coated with Iurther protective insulating layers, and operate even
under screen protectors, or behind weather and vandal-prooI glass. Due to the top layer oI a
PCT being glass, PCT is a more robust solution versus resistive touch technology. Depending
on the implementation, an active or passive stylus can be used instead oI or in addition to a
Iinger. This is common with point oI sale devices that require signature capture. Gloved
Iingers may or may not be sensed, depending on the implementation and gain settings.
Conductive smudges and similar interIerence on the panel surIace can interIere with the
perIormance. Such conductive smudges come mostly Irom sticky or sweaty Iinger tips,
especially in high humidity environments. Collected dust, which adheres to the screen due to
the moisture Irom Iingertips can also be a problem. There are two types oI PCT: SelI
Capacitance and Mutual Capacitance.
Mutual capacitance
In mutual capacitive sensors, there is a capacitor at every intersection oI each row and each
column. A 16-by-14 array, Ior example, would have 224 independent capacitors. A voltage is
applied to the rows or columns. Bringing a Iinger or conductive stylus close to the surIace oI
the sensor changes the local electrostatic Iield which reduces the mutual capacitance. The
capacitance change at every individual point on the grid can be measured to accurately
7

determine the touch location by measuring the voltage in the other axis. Mutual capacitance
allows multi-touch operation where multiple Iingers, palms or stylus can be accurately
tracked at the same time.
Self-capacitance
SelI-capacitance sensors can have the same X-Y grid as mutual capacitance sensors, but the
columns and rows operate independently. With selI-capacitance, the capacitive load oI a
Iinger is measured on each column or row electrode by a current meter. This method produces
a stronger signal than mutual capacitance, but it is unable to resolve accurately more than one
Iinger, which results in "ghosting", or misplaced location sensing.
Infrared
An inIrared touchscreen uses an array oI X-Y inIrared LED and photo-detector pairs around
the edges oI the screen. These LED beams cross each other in vertical and horizontal patterns.
This helps the sensors pick up the exact location oI the touch. A major beneIit oI such a
system is that it can detect essentially any input including a Iinger, gloved Iinger, stylus or
pen. It is generally used in outdoor applications and point oI sale systems which can't rely on
a conductor to activate the touchscreen.

Figure Infrared based Touchscreen
Optical imaging
This is a relatively modern development in touchscreen technology, in which two or more
image sensors are placed around the edges (mostly the corners) oI the screen. InIrared back
lights are placed in the camera's Iield oI view on the other side oI the screen. A touch shows
up as a shadow and each pair oI cameras can then be pinpointed to locate the touch or even
measure the size oI the touching object (see visual hull). This technology is growing in
popularity, due to its scalability, versatility, and aIIordability, especially Ior larger units.
8

Dispersive signal technology
Introduced in 2002 by 3M, this system uses sensors to detect the mechanical energy in the
glass that occurs due to a touch. Complex algorithms then interpret this inIormation and
provide the actual location oI the touch. The technology claims to be unaIIected by dust and
other outside elements, including scratches. Since there is no need Ior additional elements on
screen, it also claims to provide excellent optical clarity. Also, since mechanical vibrations are
used to detect a touch event, any object can be used to generate these events, including Iingers
and stylus. A downside is that aIter the initial touch the system cannot detect a motionless
Iinger.
Acoustic pulse recognition
This system, introduced by Tyco International's Elo division in 2006,
uses piezoelectric transducers located at various positions around the screen to turn the
mechanical energy oI a touch (vibration) into an electronic signal.|11| The screen hardware
then uses an algorithm to determine the location oI the touch based on the transducer signals.
The touchscreen itselI is made oI ordinary glass, giving it good durability and optical clarity.
It is usually able to Iunction with scratches and dust on the screen with good accuracy. The
technology is also well suited to displays that are physically larger. As with the Dispersive
Signal Technology system, aIter the initial touch, a motionless Iinger cannot be detected.
However, Ior the same reason, the touch recognition is not disrupted by any resting objects.
%5ical tasks of Com5uter Jision
Each oI the application areas described above employ a range oI computer vision tasks; more
or less well-deIined measurement problems or processing problems, which can be solved
using a variety oI methods. Some examples oI typical computer vision tasks are presented
below.
Recognition
The classical problem in computer vision is that oI determining whether or not the image data
contains some speciIic object, Ieature, or activity. This task can normally be
solved robustly and without eIIort by a human, but is still not satisIactorily solved in
computer vision Ior the general case: arbitrary objects in arbitrary situations. The existing
methods Ior dealing with this problem can at best solve it only Ior speciIic objects, such as
simple geometric objects (e.g., polyhedra), human Iaces, printed or hand-written characters, or
vehicles, and in speciIic situations, typically described in terms oI well-deIined illumination,
background, and pose oI the object relative to the camera.
9

DiIIerent varieties oI the recognition problem are described in the literature:
Object recognition: one or several pre-speciIied or learned objects or object classes can
be recognized, usually together with their 2D positions in the image or 3D poses in the
scene.
Identification: An individual instance oI an object is recognized. Examples: identiIication
oI a speciIic person's Iace or Iingerprint, or identiIication oI a speciIic vehicle.
Detection: the image data is scanned Ior a speciIic condition. Examples: detection oI
possible abnormal cells or tissues in medical images or detection oI a vehicle in an
automatic road toll system. Detection based on relatively simple and Iast computations is
sometimes used Ior Iinding smaller regions oI interesting image data which can be Iurther
analysed by more computationally demanding techniques to produce a correct
interpretation.
Several specialized tasks based on recognition exist, such as:
Content-based image retrieval: Iinding all images in a larger set oI images which have a
speciIic content. The content can be speciIied in diIIerent ways, Ior example in terms oI
similarity relative a target image (give me all images similar to image X), or in terms oI
high-level search criteria given as text input (give me all images which contains many
houses, are taken during winter, and have no cars in them).
Pose estimation: estimating the position or orientation oI a speciIic object relative to the
camera. An example application Ior this technique would be assisting a robot arm in
retrieving objects Irom a conveyor belt in an assembly line situation.
Optical character recognition (OCR): identiIying characters in images oI printed or
handwritten text, usually with a view to encoding the text in a Iormat more amenable to
editing or indexing (e.g.ASCII).
Motion analysis
Several tasks relate to motion estimation where an image sequence is processed to produce an
estimate oI the velocity either at each points in the image or in the 3D scene, or even oI the
camera that produces the images. Examples oI such tasks are:
Egomotion: determining the 3D rigid motion (rotation and translation) oI the camera
Irom an image sequence produced by the camera.
10

Tracking: Iollowing the movements oI a (usually) smaller set oI interest points or objects
(e.g., vehicles or humans) in the image sequence.
Optical flow: to determine, Ior each point in the image, how that point is moving relative
to the image plane, i.e., its apparent motion. This motion is a result both oI how the
corresponding 3D point is moving in the scene and how the camera is moving relative to
scene.
Scene reconstruction
Given one or (typically) more images oI a scene, or a video, scene reconstruction aims at
computing a 3D model oI the scene. In the simplest case the model can be a set oI 3D points.
More sophisticated methods produce a complete 3D surIace model.
Image restoration
The aim oI image restoration is the removal oI noise (sensor noise, motion blur, etc.) Irom
images. The simplest possible approach Ior noise removal is various types oI Iilters such as
low-pass Iilters or median Iilters. More sophisticated methods assume a model oI how the
local image structures look like, a model which distinguishes them Irom the noise. By Iirst
analysing the image data in terms oI the local image structures, such as lines or edges, and
then controlling the Iiltering based on local inIormation Irom the analysis step, a better level
oI noise removal is usually obtained compared to the simpler approaches. An example in this
Iield is the inpainting.
MULTI-TOUCH TECHNIQUES
At the moment there are Iive major techniques that allow Ior the creation oI a stable multi-
touch hardware systems include: JeII Han`s pioneering Frustrated Total Internal Reflection
(FTIR), Rear Diffused Illumination (Rear DI) such as MicrosoIt`s SurIace Table, Laser
Light Plan (LLP) pioneered in the community by Alex Popovich and also seen in
MicrosoIt`s LaserTouch prototype, LED-Light Plane (LED-LP) developed within the
community by Nima Motamedi, and Iinally Diffused Surface Illumination (DSI) developed
within the community by Tim Roth.
These five techniques being utilized work on the principal of Computer Vision and
optics (cameras). While optical sensing makes up the vast majority oI techniques , there are
several other sensing techniques that can be utilized in making natural user interIace and multi
touch devices. Some oI these sensing devices include proximity, acoustic, capacitive, re-
11

sistive, motion, orientation, and pressure. Often, various sensors are combined to form a
particular multi touch sensing technique.
Optical Multi-Touch Technologies :Optical or light sensing (camera) based solutions make
up a large percentage oI multi-touch devices. The scalability, low cost and ease oI setup are
suggestive reasoning Ior the popularity oI optical solutions. Stereo Vision , Overhead
cameras, Frustrated Total Interal ReIlection, Front and Rear DiIIused Illumination, Laser
Light Plane, and DiIIused SurIace Illumination are all examples oI camera based multi-touch
systems. Each of these techniques consist of an optical sensor (typically a camera),
infrared light source, and visual feedback in the form of projection or LCD IR LEDs are
used because they are eIIicient and eIIective at providing inIrared light. On the other hand,
DiIIused Illumination (DI) does not require IR LEDs, per se, but rather, some kind oI inIrared
light source like an inIrared illuminator (which may have LEDs inside). Laser light plane
(LLP) uses IR lasers as the IR light source. The resolution oI the camera is very important.
The higher the resolution the more pixels are available to detect Iinger or objects in the
camera image. This is very important Ior the precision oI the touch device. For small multi-
touch surIaces a low resolution webcam (320 x 240 pixels) can be suIIicient. Larger surIaces
require cameras with a resolution oI 640x480 or higher in order to maintain the precision. The
Irame rate is the number oI Irames a camera can take within one second. More snapshots
means that we have more data oI what happened in a speciIic time step. In order to cope with
Iast movements and responsiveness oI the system a camera with at least a Irame rate oI 30
Irames per second (FPS) is recommended. Higher Irame rates provide a smoother and more
responsive experience.
Tracking is very important to multi-touch technology. It is what enables multiple Iingers to
perIorm various actions without interrupting each other. We are also able to identiIy gestures
because the trajectory oI each Iinger can be traced over time, impossible without tracking. In
Iact much oI the perIormance bottleneck oI tracking systems tends to come Irom generating
and maintaining a model oI the background. Computational costs oI these systems heavily
constrict CPU usage unless clever tricks are employed. However, with the current inIrared
(IR) approaches in multi-touch hardware (e.g., FTIR or DI), an adaptive background model
turns out to be overkill.



12

Research Papers
. %ouchLight: An Imaging %ouch Screen and Dis5la for Cesture-Based Interaction
Andrew D. Wilson
MicrosoIt Research
INTRODUCTION:
A novel touch screen technology is presented. TouchLight uses simple image processing
techniques to combine the output oI two video cameras placed behind a semi-transparent
plane in Iront oI the user. The resulting image shows objects that are on the plane. This
technique is well suited Ior application with a commercially available projection screen
material (DNP HoloScreen) which permits projection onto a transparent sheet oI acrylic
plastic in normal indoor lighting conditions. The resulting touch screen display system
transIorms an otherwise normal sheet oI acrylic plastic into a high bandwidth input/output
surIace suitable Ior gesture-based interaction.

PROPOSED APPROACH:
The TouchLight system uses simple computer vision techniques to compute a touch image on
a plane situated between a pair oI cameras and the user. The author demonstrates these
techniques in combination with a projection display material which permits the projection oI
an image onto a transparent sheet oI acrylic plastic, and the simultaneous operation oI the
computer vision processes. TouchLight goes beyond the previous camera-based systems; by
not using a diIIusing projection surIace, it permits a high resolution touch image. For
example, a high resolution image oI a paper document may be captured using a high-
resolution still camera, or one oI the newer high resolution CMOS video cameras. The
absence oI a diIIuser also permits the cameras to see beyond the display surIace, just as they
would iI placed behind a sheet oI glass. This allows a variety oI interesting capabilities such
as using Iace recognition techniques to identiIy the current user, eye-to-eye video
conIerencing, and other processes which are typically the domain oI vision-based perceptual
user interIaces.


13


Figure 3 TouchLight physical configuration:
DNP graphic Holoscreen with IR cameras and
IR illuminant behind screen


Figure 4 TouchLight prototype displaying a
sample graphic

The goal oI TouchLight image processing is to compute an image oI the objects touching the
surIace oI the display, such as the user`s hand. Due to the transparency oI the display, each
camera view shows the objects on the display and objects beyond the surIace oI the display,
including the background and the rest oI the user. With two cameras, the system can
determine iI a given object is on the display surIace or above it. TouchLight image processing
acts as a Iilter to remove objects not on the display surIace, producing a touch image which
shows objects that are on the display surIace and is blank everywhere else.


Figure 5 Raw input from camera

Figure 6 Raw input from camera
14


Figure 7(above) (below) Input after lens
distortion correction


Figure 8 Input after perspective correction to
rectify both views to display


Figure Fused image obtained by multiplying perspective corrected images










I|gure 9 Input after perspect|ve correct|on
to rect|fy both v|ews to d|sp|ay
13

. ingermouse: A Wearable Hand %racking Sstem
P. de la Hamette|1|, P. Lukowicz|1|, G. Trster|1|, T. Svoboda|2|
|1|Wearable Computing Laboratory, ETH Zrich
|2|Computer Vision Laboratory, ETH Zrich
INTRODUCTION:
In this paper, we describe a system that implements limited hand tracking and gesture
recognition capabilities in a small low power module. A Iirst application is a mouse
replacement, worn on the body, that captures the hand movement and allows the user to
control a mouse pointer. The hardware is based on 2 image sensors, a distance sensor and a
low power DSP. The soItware consists oI diIIerent tracking algorithms supported by stereo
vision based background clutter removal. Our system proves that computer vision can be
useIul even Ior mobile systems with strictly limited computational and power resources.

PROPOSED DESIGN:
The proposed approach addresses the computational resources problem in three ways.
O By restricting the scope oI the recognition problem by requiring the hand to be in a
certain Iixed distance Irom the camera and to start tracking in a predeIined position.
O Secondly using a distance sensor together with stereoscopic vision to get rid oI the
cluttered background.
O Implementing a special purpose hardware module that allowed authors to execute the
necessary algorithms with a Iraction oI the power consumption oI a general-purpose
machine.
As a result a system that is small, low power but nevertheless able to acquire images and
compute them independently is designed, so that it acts as an autonomous peripheral to a
wearable computer.


Figure Fingermouse, top, cameras, dist
sensor

Figure Fingermouse, bottom, DSP, Flash
RAM
16

. HoloWall: Designing a inger; Hand, Bod, and Object Sensitive Wall
Nobuyuki Matsushita, Jun Rekimoto
INTRODUCTION AND PROPOSED APPROACH
The HoloWall is a wall-sized computer display that allows users to interact without special
pointing devices. The display part consists oI a glass wall with rear projection sheet behind it.
A video projector behind the wall displays images on the wall.

Figure 3 HoloWall setup
Inputs are recognized with inIrared lights and a video camera with an IR Iilter installed
behind the wall. Since the rear projection panel is semi-opaque and diIIusive, the user`s shape
or any other objects in Iront oI the screen are invisible to the camera. However, when a user
moves a Iinger close enough to the screen (between 0 cm to 30 cm, depending on the
threshold value oI the recognition soItware), it reIlects IR light and thus becomes visible to
the camera. With a simple, image processing technique such as Irame subtraction, the Iinger
shape can easily be separated Irom the background. By controlling the threshold value, it is
also possible to detect a human body when he/she approaches the screen. In the same way,
any physical objects that reIlect IR light are detectable. The authors tested VCR tapes, paper,
and document Iolders, and all were clearly separated Irom the background. The authors also
attached a 2D-barcode to the surIace oI an object to make it identiIiable. When a user puts an
object (such as a document, Iolder) with a SD-barcode on the wall, the system can detect its
ID as well as its position. This Ieature makes it possible to implement an object-sensitive
interIace.





17

. u%ouch: A cost effective touch screen solution for gesture based interaction
Usama GhuIran
INTRODUCTION
uTouch uses simple image processing techniques to combine the output oI two video cameras
placed in Iront any plane beIore the user. The resulting image shows objects that are on the
plane. This technique is well suited Ior all application that support single touch user
interactions. The resulting touch screen display system transIorms an otherwise normal plane
or computer screen into an input/output surIace suitable Ior gesture-based interaction.
PROPOSED APPROACH
A pair oI commonly available USB web cameras are mounted in Iront oI a plane which could
be a computer screen such that each camera can see all Iour corners oI the plane. The corners
oI the plane are manually marked Ior each camera view and this speciIies the region where
the touch is to be detected.


Figure 4 Configuration for uTouch
In the proposed approach the image is captured Irom two cameras. The two images are
rectiIied and then merged. The images are rectiIied by removing the skin color Irom the
background. The touch on the screen is detected Irom the merged image. II the merged image
has single Iinger point then the touch has occurred otherwise not. AIter detecting the touch the
point oI touch is determined and corresponding signals are Ior mouse operations.
18


Figure 5 Raw Input of camera A

Figure 6 Raw Input of camera

Figure 7 Perspective Correction for camera A

Figure 8 Perspective Correction for camera

Figure Rectified Image from Camera A

Figure Rectified Image from camera

Figure Fused Image
19

Due to the Iact that proposed approach uses two cameras and common point oI intersection
Irom two cameras determines the touch, the proposed approach lags in the Ieature oI multi-
gesture and multi-touch techniques.
. Jision-based Interaction with ingers and Pa5ers
Zhengyou Zhang
INTRODUCTION:
The paper presented two vision-based interIace systems. The Iirst, Visual Screen, uses an
inexpensive technique to transIorm an ordinary screen into a touch screen using an ordinary
camera. In runtime, our system locates the tip pointer (Iingertip in our current
implementation) in the image and converts the image position to the cursor position on the
screen. The second system, Visual Panel, extends the previous system Ior mobile applications.
It employs an arbitrary quadrangle-shaped panel (e.g., an ordinary piece oI paper) and a tip
pointer as an intuitive, wireless and mobile input device. The system can accurately and
reliably track the panel and the tip pointer. The panel tracking continuously determines the
projective mapping between the panel at the current position and the display, which in turn
maps the tip position to the corresponding position on the display. The system can IulIill
many tasks such as controlling a remote large display, and simulating a physical keyboard.
Users can naturally use their Iingers or other tip pointers to issue commands and type texts.
Furthermore, by tracking the 3D position and orientation oI the visual panel, the system can
also provide 3D inIormation, serving as a virtual joystick, to control 3D virtual objects.
PROPOSED APPROACH:
Figure 22 is the diagram oI the Visual Screen (VS in short) system. Four dash boxes represent
Iour major parts oI the system. From leIt to right, these boxes will be reIerred as Calibration
block, Model Extraction I block, Main block, and Model Extraction II block, respectively.
The Main block is the kernel oI the system. Its Iunctionality is to locate the tip point oI the
indicator and map its image coordinates to the screen coordinates. The task oI tip point
location contains two processes, i.e., to segment the indicator Irom the background, and to
Iind the tip point oI the indicator. The segmentation requires color models Ior both the
background and the indicator. The Model Extraction blocks I, and II in Figure 1 are used to
extract the background model and the Ioreground model, respectively. The Calibration block
is used to establish the mapping between the image coordinates and the screen coordinates.
This mapping is then used in the Main block to Iind the corresponding screen coordinates Ior
the tip point once its image coordinates are estimated.
20


Figure Diagram for Visual Screen system
It has been observed in experiments that the images oI screen pixels have some degrees oI
invariance in the color space. Authors have Iirstly computed a color model Ior the screen
without the indicator. A number oI pictures with rich color are displayed on the screen in
order to make the model as general as possible. To compute this background model all oI the
pixels in the image are histogrammed namely, Ior each pixel its color intensity is placed in the
proper bin oI a preIerred possible 256 intensity levels. This is preIerably done Ior each oI the
red, green and blue (RGB) channels thus generating three separate histograms. Alternately,
one histogram could be generated using some joint space representation oI the channels. Once
the histogram has been computed, a Gaussian distribution Ior each histogram is calculated to
provide the mean pixel intensity oI the background and the variance. Once the modeling oI
the background oI the screen has been completed, the model Ior the indicator or pointer is
computed in order to separate the indicator Irom the background. This is done by asking the
user to select a polygonal bounding area displayed on the screen Ior the indicator oI choice.
Only the pixels inside this polygonal area are used to compute the color model Ior the
indicator. Usually the color model Ior the indicator will be dominated by a diIIerent color in
color space than the background. Once both the screen background and indicator models are
determined, a standard Bayes classiIier is used to segment the indicator Irom the screen
background. II the extracted models oI the Ioreground and background are split into separate
RGB channels, the Bayes classiIier determines the probability a given pixel color is a
background pixel Ior each channel and these probabilities are multiplied together. The
classiIier also determines the probability a given pixel is a Ioreground pixel Ior each channel
and multiplies the probabilities together. Next, the background pixel probability product is
divided by the Ioreground pixel probability product. II this quotient is greater than one then
the pixel is determined to be a background pixel, otherwise it is determined to be a Ioreground
or indicator pixel.

21

. A Surface Acoustic Wave %ouchscreen-%5e Device Using %wo %ransducers
Mehrdad Ghaziasgar, James
INTRODUCTION:
We developed a surIace acoustic wave touchscreen-type device using only two transducers, as
opposed to, typically, three or more transducers in conventional surIace acoustic wave
touchscreens. The transducers are mounted on a glass surIace and connected into the line-in oI
a stereo sound card. User-initiated taps are detected, analysed and located on the surIace, and
the mouse cursor is moved to the computed screen location. Additionally, the surIace that we
used was not modiIied or specialised in any way, except in its dimensions. Where possible,
we shiIted the intelligence oI the device Irom the hardware to the soItware so that we relied
more on postprocessing intelligence than innovative hardware design. As an object taps on the
glass surIace, geometrical and physical identities as well as the sound inIormation collected
on each oI the transducers is used to determine the position oI the object. This position is then
mapped onto a corresponding location on the computer monitor. Subsequently, the mouse
cursor is moved to that location.


Figure 3 Illustration of Hardware Setup





22

PROPOSED APPROACH:

Figure 4 The generic algorithm of the software component

Figure 5 Graphcal representation of a typical tap on one of the transducers
LIMITATIONS OF PROPOSED APPROACH:
O Based on the principle used, the device is only able to detect taps, and is not able to
determine, aIter the tap, whether or not the object is still in contact with the surIace.
Thus, the user interacts only by means oI taps and not drags or contact-dependent
23

actions. Further, Ior the scope oI this project, only single - not double taps are
considered. Hence not suitable Ior multigesture recognition.
O High precision in placement oI hardware equipments in terms oI distances and angles
is required else errors creep in.
Existing computer-vision based approaches Ior touch screens have certain limitations:
O Incapablility oI handling multiple touches.
O Some cannot be employed Ior gesture recognition.
O Have problem with backgrounds, as they employ techniques oI skin detection.
Anything in background oI skin color aIIects perIormance oI the approach.
O Requires special display screen.
O Camera is placed behind the screen. So, can`t be used Ior desktop screens.
O Requires multiple cameras. Most oI the approaches require at least two cameras.
O Due to design constraints can`t be used Ior multiple touch or gesture recognition.
O Requires more oI the memory and computational requirements due to complex
algorithms Ior skin detection and identiIication oI touch.
Keeping in mind above drawbacks, we evolved our design oI computer-vision based touch
screen capable oI gesture recognition. The proposed approach is aimed to be cost eIIective
and reliable it has decent eIIiciency. We have designed the product Ior computer desktops to
increase its accessibility to consumers. No special screen will be required Ior touch detection.












24











CHAPTER 3
ANALYSIS, DESIGN AND MODELLING













23

3 Overall Description of the Project
a) Product Perspective: A Touchscreen is an electronic visual display that can detect
the presence and location oI a touch within the display area. The term generally reIers
to touching the display oI the device with a Iinger or hand. Touchscreens can also
sense other passive objects, such as a stylus. Touchscreen is common in devices such
as all-in-one computers, tablet computers, and smartphones.
Touchscreen has two main attributes. First, it enables one to interact directly with what
is displayed, rather than indirectly with a cursor controlled by a mouse or touchpad.
Secondly, it lets one do so without requiring any intermediate device that would need
to be held in the hand. Such displays can be attached to computers, or to networks as
terminals. They also play a prominent role in the design oI digital appliances such as
the personal digital assistant (PDA), satellite navigation devices, mobile phones, and
video games.
Multitouch denotes a set oI interaction techniques that allow computer users to
control graphical applications with several Iingers. Multi-touch devices consist oI a
touch screen (e.g., computer display, table, wall) or touchpad, as well as soItware that
recognizes multiple simultaneous touch points, as opposed to the standard touchscreen
(e.g. computer touchpad, ATM), which recognizes only one touch point.

b) Product Functions: New design Ior Computer vision based touch screen is proposed
which has Iollowing Ieatures
a. Cost EIIective
b. Use oI InIrared Ior detecting the touch, which is not interIered by the
background.
c. Works with desktop monitors/screens.
d. Single Camera is placed in Iront oI the screen.
e. Design capable oI detecting multiple-gestures (gestures that requires multiple
touches).
I. Requires minimal memory and computational resources.
g. No device driver need to be installed.
h. InIrared LEDs works using USB power.
The implemented touchscreen design has the Ieature oI single leIt click and right click.
LeIt clip is perIormed by tapping the wall while double click is perIormed by keeping
the stylus or Iinger on the selected object Ior about a second. Multi-gesture recognition
26

is used Ior Iunctions like zoom in and out, rotate clockwise and anti-clockwise , move
up and down using standard gestures . The technique is careIully tailored to reduce
computational complexity. Where possible, we shiIted the intelligence oI the device
Irom the hardware to the soItware so that we relied more on post processing
intelligence than innovative hardware design and even accuracy and precision is
achieved using robust programming.
c) User Characteristics: This product is Ior increasing intractability with computer. The
product is aimed Ior desktop users with TFT/LCD screen. Product aims at removing
use oI mouse. It is not the Iinal product that can be used by users. It is still under
design and testing phase. The person using it should have knowledge oI working on
computers. They can use their Iinger or stylus Ior touching the screen. Using stylus
improves eIIiciency because oI its narrow tip. It is to be noted that the user may not
block the camera by his/her hands.
d) Constraints: The product developed has the constraint oI slow calculations because oI
poor quality oI camera used in testing. It gives the Irame rate oI around 5-6 Ips. When
product will be tested using higher Irame rate cameras, it is sure that it will be
Iunctioning accurately.
e) Assumptions and Dependencies: It is assumed that:
O InIrared camera is available Ior capturing inIrared emissions.
O Desktop is in oIIice or in the room.
O USB power supply is available.
Windows version supports gesture messages.










27

3 Specific Requirements
3 External Interfaces
S.
No.
INPUTS OUTPUTS
1 Start the Touch screen soItware Window will appear Ior setting
screen area
2 Set the screen area by reducing
threshold using track-bar
Area oI the screen will be seen
enclosed in the red contour with
edges marked green.
3 Tap the window Ior single leIt click LeIt click action is perIormed on the
particular object.
4 Keep Iinger or stylus on the object
Ior about a second to perIorm single
right click.
Right click action is perIormed on
the selected object.
5 To select particular area on the
screen, drag stylus or Iinger Irom
initial to Iinal position.
Selection oI objects in the area will
be done.
6 For zoom in and out stretch the
image in /out using both hands
Image will be zoomed in/out
7 For rotating image clockwise and
anticlockwise rotate 3 Iingers oI
your hand accordingly
Image will rotate
8 For scrolling pdI up or down drag
your Iinger up and down in vertical
direction
Document is scrolled up/down
9 For moving next or previous while
viewing images move a single Iinger
horizontally towards leIt/right.
Image gets swapped with next or
previous image.
Table 1. External InterIaces



28

3 Functions
Input
Mark the screen area by adjusting threshold value on the trackbar. The screen area is marked
through contour in red color with edges marked with green circle.
Action:
We need to capture the image area. This is done by capturing live video Irom inIrared
webcam. The image captured consists oI background as well. We need to remove the
background image to keep region oI interest as screen only. This is done by changing image
to grayscale and then making contour depending on the threshold values. Threshold value is
changed using trackbar on the window. AIter capturing the screen area press 'q to start
working.
Input
Set threshold value Ior detecting touch.
Action:
This Iunction has been automated by increasing threshold value until contours are
encountered on the screen. Then, Iinally increase the threshold value by 13 Ior accuracy and
avoid errors due to reIlection.
Input 3
Tap the screen to simulate single leIt click.
Action:
Single leIt click is the most common action done day to day. This action is achieved by
tapping the screen. When the stylus or Iinger enters close to screen, the stylus or Iinger glows
by inIrared illumination. This is captured by inIrared webcam. When object enters close to
screen, event leIt mouse button down is Iired. While when object leaves the screen that area
glow is vanished. This is recognized by the system and leIt mouse button up event is Iired.
Thus, resulting in mouse leIt button click event.
Input 4
Hold the stylus/Iinger close to the object Ior about a second.
Action:
When stylus/Iinger enters close to the screen, time stamp is captured. When stylus/Iinger
leaves the screen, another time stamp is captured. II the stylus/Iinger has not moved then
validating Ilag Ior right click is validated. This validation is checked and time duration is
measured. II the diIIerence oI timestamps exceeds 1 second then right click event is Iired.

29

Input 5
Drag and drop operation.
Action:
When stylus/Iinger enters the screen, leIt mouse button down event is Iired and time stamp is
captured. Now iI one moves the stylus/Iinger, then this timestamp in invalidated. On moving
the stylus/Iinger the changes in location oI the glow Irom stylus/Iinger is captured by inIrared
webcam and corresponding change in location oI the mouse pointer is reIlected on the screen
by Iiring move mouse events. When the stylus/Iinger leaves the screen area then, leIt mouse
button up event is Iired. This whole situation leads to drag and drop simulation oI the mouse.
Input 6
Place both Iore Iingers on an image and either increase or decrease the distance between them
horizontally without changing angle.
Action:
The image will get zoom out or in depending upon motion oI Iingers. This perIorms the
standard zoom in/out Iunctionality using multi gesture recognition technique on a touch
screen thus making it more interactable.
Input 7
Place Iingers on an image and move them clockwise/anti-clockwise.
Action:
The image will get rotated depending upon motion oI Iingers. This perIorms the standard
rotate Iunctionality using multi gesture recognition technique on a touch screen thus making it
more interactable.
Input 8
Place Iinger on a pdI document or a html web page and move them down/up vertically.
Action:
The document will get scrolled depending upon motion oI Iingers. This perIorms the standard
move up/down Iunctionality using multi gesture recognition technique on a touch screen thus
making it more interactable.
Input
Place Iinger on a image viewer window and move it right or leIt horizontally.
Action:
The image will get swapped by next or previous image depending upon motion oI Iingers.
This perIorms the standard next/previous Iunctionality using multi gesture recognition
technique on a touch screen thus making it more interactable.
30

33 Performance Requirements
The static requirements include:
O InIrared webcam should be available.
O Availability oI array oI inIrared LEDs to illuminate stylus/Iinger when they are close
to it.
The dynamic requirements include:
O The user should not block path oI the camera and screen.
O The user should orient his hand not to block the camera.
O The screen area should be correctly speciIied by adjusting threshold value.
O PreIer stylus over Iinger.

34 Design Constraints
This Section discusses the design constraints:
O An array oI seven inIrared LEDs are mounted below screen to illuminate the
stylus/Iinger.
O InIrared camera is used to capture the inIrared illumination.
O The screen area should be careIully marked by adjusting threshold value.
O The user can switch on or oII the LEDs using control switch.

35 Software System Attributes
a) Reliability
In order to ensure reliability, this is being designed using soItware that is established to be
stable and easy to use. II the memory constraints are Iollowed, then this system should
present no reliability problems. The design being simple possess uses less memory and
computational resources.
b) Availability
Tool is available Ior use just by switching on the LEDs and running the program. Using stylus
is recommended.
c) Security
The tool is secured and does not causes any security threat to the system. For improving the
design a screen guard can be mounted on the screen to prevent it Irom scratches.
31

d) Maintainability
There will be no maintenance required Ior the soItware. For hardware side, the wires are
soldered. By craIting the design into metallic sheets will add to its maintenance.
e) Portability
The design is made portable by providing power to LEDs through USB. Array oI LEDs can
be detached or attached to screen easily without any discomIort. InIrared webcam is used
which is available in market or can be easily made.

33 Design
In this section we discuss about the design oI the proposed technique Ior computer vision
based touch screen Ior desktop PC. TFT/LCD screens provide the dull white background
when viewed through inIrared camera. This is the reason why, background is not point oI
concern. Array oI inIrared LEDs is mounted below the screen to illuminate the stylus/Iinger
when it reaches close to screen. Testing has been done with 6 LED array, 7 LED array, 12
LED array and 15 LED array. The optimized one turned out to be 7 LED array design, which
is implemented. A switch Ior turning on or oII the LEDs is provided close to the setup. The
LEDs are powered by the USB port. A simple webcam is transIormed to inIrared webcam.
This webcam captures the inIrared illuminations.













32


34 Use Case Diagram


Figure 26. Use Case Diagram









33



























34













CHAPTER 4
Implementation, Testing and Results











33


4 Implementation

Hardware: An array oI 7 inIrared LEDs are successIully mounted at bottom oI the display
screen. The LEDs are powered through USB slot. A switch is provided to switch on and oII
the LEDs. A normal webcam is converted to inIrared webcam by perIorming certain hardware
modiIications.
Software: A C program using OpenCV library, captures live video Irom camera, converts
the image to grayscale and Iind contours. Screen area is detected using contours. Contours are
adjusted using trackbars. Steps perIormed by soItware are as Iollows:
O Detect the camera.
O Capture live video Irom camera.
O Convert image to grayscale.
O Find contours in the image.
O Adjust the contour to cover screen area using trackbar on the window.
O As screen area is covered, Iind perspective transIorm to have region oI interest as the
screen.
O The touch screen is ready to Iunction.
O When user brings the stylus or Iinger close to the screen, it gets illuminated by the
inIrared rays. In case oI entry oI the stylus/Iinger, leIt mouse button down event is
Iired and timestamp is taken and mouse right click Ilag is set to valid or true.
O When user takes out the stylus/Iinger away Irom the screen, it screen returns to normal
mode. In this case leIt mouse button up event is Iired and checks Ior right click Ilag, iI
Iound valid and diIIerence in the timestamp is greater than 1 second then right click
event is Iired.
O When user moves the stylus/Iinger close to the screen, the right click Ilag is
invalidated and mouse pointer is moved accordingly.






36


4 Testing
4 Test Plan Identifier
O The webcam is well connected and working properly.
O Webcam is in position to view whole oI the screen area.
O All LEDs are working correctly.
O The switch is initially oII.
O LEDs array is connected to USB port.
O The screen area is careIully selected by adjusting threshold value.
O The camera position should remain Iixed throughout working oI the program..

4 Test Items
Items to be tested The intended outputs
1. Test the LEDs are working Iine by
hovering webcam over them by switching
them on.
All LEDs seen glowing
2. Test the connections oI webcam The webcam shows images.
3. Test the connections oI LEDs are made
through USB
LEDs glow

4. Check Ior the screen area captured
correctly, by putting stylus/Iinger at corners
oI the screen
Stylus/Iinger shows glow at corners
5. Check Ior the leIt click Iunctionality by
tapping the screen
LeIt click operation perIormed
6. Check Ior the right click Iunctionality by
holding stylus/Iinger close to the object.
Right Click operation perIormed
7. Check Ior drag and drop Iunctionality oI
the system by putting stylus/Iinger on the
Drag and Drop operation perIormed
37

object and then dragging it.
8. Check Ior next / previous Iunctionality by
moving a Iinger Iorm leIt to right or right to
leIt on an image viewer window
Image is swapped by next/previous image.
9. Check Ior zoom in/out Iunctionality by
moving both Iore Iingers away or towards
each other horizontally positioning them on
target image.
Image is zoomed in/out depending upon
motion oI Iingers.
10. Check Ior rotate clockwise and anti-
clockwise Iunctionalities by rotating three
Iingers on the image.
Image is rotated as per the motion oI Iingers.
11. Check Ior scrolling up and down oI a
document by moving a Iinger up/down
vertically.
Document is scrolled.
Table Test Items
43Features to be Tested
Features Risk
1. Test the LEDs Moderate
2. Test the connection oI webcam Moderate
3. Test the connection oI LEDs with USB socket Low
4.Test the screen area captured accurately Moderate
5. Check the right click Iunctionality High
6. Check the leIt click Iunctionality High
7. Check the drag and drop Iunctionality High
8. Check Ior Multi gesture zoom in/out
Iunctionality
Very High
38

9. Check Ior Multi gesture rotate
clockwise/anticlockwise Iunctionality
Very High
10. Check Ior Multi gesture based scroll up/down
Iunctionality
Very High
11. Check Ior Multi gesture based view
previous/next
Very High
Table 3 Features to be tested
44 Strategy to be used for writing Test Cases
Are any special tools to be used and what are they?
Special tools used
O Visual Studio 2008,2010
O Opencv 2.2
Will tool require special training?
Special training requirements
O The user need to have idea about using computer.
O The user should be able to select the screen area by changing threshold value.
O The user should keep stylus/Iinger stable while pointing on the screen.
How many diIIerent conIigurations will be tested?
SoItware
O Visual Studio 2008, 2010
O Windows vista/XP/windows7
O OpenCV 2.0, 2.2 library
Combinations oI HW, SW and other vendor packages
45 Test Cases
Table 4 Test cases
Sr.
No
Objective
Environm
ent
Pass/Fa
il
Data input
Expected
Output
Actual Results Remarks
39

1
Check iI
LEDs are
working
Windows
/ Webcam
soItware
Pass
Connection to
USB socket
made and
LEDs
switched on
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
output as
intended
2
Check iI
webcam is
working
Windows
/ Webcam
SoItware
Pass
Open the
webcam
viewing
soItware,
Images seen
Webcam
showed images
oI what it was
Iocusing
Web cam
showed images
oI what it was
Iocusing
output as
intended
3
Check iI
USB power
connection
are correctly
made
Windows
/ Webcam
SoItware
Pass
Connection to
USB socket
made and
LEDs
switched on
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
output as
intended
4
Check Ior
the screen
area covered
Execution
oI
program /
Windows
Pass
Place
stylus/Iinger
near the
corners oI the
screen
Corners
detected
Corners
detected
output as
intended
5
Check Ior
the leIt click
Iunctionality
Execution
oI
Program /
Windows
Pass
Tap the object
on screen
LeIt mouse click
operation
perIormed
LeIt mouse
click operation
perIormed
output as
intended
6
Check Ior
the right
click
Iunctionality
Execution
oI
program /
windows
Pass
Place the
stylus/Iinger
close to object
on screen
On removing
stylus/Iinger
right click
action
perIormed
On removing
stylus/Iinger
right click
action
perIormed
output as
intended
7
Check Ior
drag and
drop
Iunctionality
Execution
oI
program /
windows
Pass
Place the
stylus/Iinger
on the object
and drag it
Drag and drop
Iunctionality
perIormed
Drag and drop
Iunctionality
perIormed
output as
intended
8
Check Ior
rotate
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
image and
rotate them.
Rotate
clockwise/anti-
clockwise
Iunctionality
perIormed
Rotate
clockwise/anti-
clockwise
Iunctionality
perIormed
Output as
intended
40

9
Check Ior
scrolling
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
document and
move them
up/down.
Scroll
Iunctionality
perIormed
Scroll
Iunctionality
perIormed
Output as
intended
10
Check Ior
zoom in/out
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
image and
move them
apart or near.
Zoom in/out
Iunctionality
perIormed
Zoom in /out
Iunctionality
perIormed
Output as
intended

43 Results
The Snapshots depicting the results is shown in Iollowing Iigures:

Figure 27. Snapshot oI screen area selecting window
41


Figure 8 Screenshot after screen area is selected


Figure Snapshot showing LEFT AND RIGHT click action




Figure 3Screen shot showing drag and drop option
42







43


Fig: Multi gesture Manual
44 Risk Management
The Interrelationship Graph (IG)
The interrelationship graph is drawn between the Iollowing Iactors that we have identiIied Ior
our project:
1. Connection
2. PerIormance
3. Open Source
4. Testing Environment
44

5. Budget (Cost Time)
6. Project Scope
7. Hardware
8. External Inputs

The weights on basis oI importance between the Iactors considered are as Iollows:
1 ~ 'relationship is weak
3 ~ 'relationship is medium
9 ~ 'relationship is signiIicant
Budget ~ PerIormance (9)
Connectivity Problem ~ PerIormance (3)
PerIormance ~ Project Scope (3)
Hardware ~ PerIormance (9)
43

External Inputs ~ PerIormance (3)
External Inputs ~ Testing Environment (1)
Testing Environment ~ PerIormance (1)
Open Source ~ PerIormance (1)
Budget ~ Open Source (3)
Connection ~ Hardware (3)












SN Risk Area # of Risk
Statements
Weights (In +
Out)
Total
Weight
Priority
1 PerIormance 7 9393113 29 1
2 Connectivity 3 33 6 2
3 External Inputs 1 313 7 3
4 Open Source Code 5 133 7 4
5 Project Scope 2 3 3 5
6 Hardware 3 93 12 6
7 Budget 4 93 12 7
8 Testing Environment 5 11 2 8
46












CHAPTER 5
Findings, Conclusion And Further Work














47

5 Findings And Conclusion

The touch screen is Iunctioning accurately. The objects were selected to an accuracy oI about
10 square pixels area. The Hardware is working as expected. The USB power is successIully
delivered to LEDs. The webcam is successIully capturing inIrared rays.
The system successIully perIormed right and leIt clicks events. The program is consuming
just 8 Mbs oI system memory and the computational resources are used eIIiciently. Drag and
drop event can be used to drag and drop movies and songs to the media players.
The proposed approach used single webcam which is mounted in Iront oI screen. The
accuracy is Iound to be appropriate. There were no memory leaks in the execution oI the
program. Multi touch was successIully recognized and acted upon Ior the Iunctionalities like
rotate,zoom , scroll etc


5 Further Work
The product can be extended to include:
O Improving eIIiciency oI Multiple Gesture to increase usability.
O Improved webcam to increase the perIormance and accuracy oI the program.
O Making it robust by reIining hardware design.















48

REFERENCES
DIGITAL IMAGE PROCESSING by R C Gonzalez and R E Woods
OpenCV REFERENCE MANUAL
WE REFERENCES:
|1| "Computer Vision", http://en.wikipedia.org/wiki/Computervision
|2| 'Touch Screen, http://en.wikipedia.org/wiki/Touchscreen
|3| Andrew D. Wilson," TouchLight: An Imaging Touch Screen and Display Ior Gesture-
Based Interaction, ICMI`04, State College, Pennsylvania, USA, October 1315,
2004.
|4| P. de la Hamette, P. Lukowicz, G. Trster, T. Svoboda, 'Fingermouse: A Wearable
Hand Tracking System, UBICOMP conIerence (proceedings), 2002.
|5| Paul Dietz and Darren Leigh, 'DiamondTouch: A Multi-User Touch Technology,
UIST 2001, the 14th Annual ACM Symposium on User InterIace SoItware and
Technology, Orlando, Florida USA, pp. 219-226, November 11-14, 2001,.
|6| Nobuyuki Matsushita, Jun Rekimoto, HoloWall: Designing a Finger, Hand, Body,
and Object Sensitive Wall, UIST 97 Albert, Canada, pp 209-210.
|7| Usama GhuIran, 'uTouch: A cost eIIective touch screen solution Ior gesture based
interaction














49

(IX)
Appendix A
GANTT CHART
MODULARISATION OF PRO1ECT:
Increment 1:
O To carve out the problem statement and scope oI project.
O To study research papers available on touch screen and analyse their short comings to add
novelty and utility to our vision.
O Time expected: 4 weeks
Increment 2:
O To think oI hardware required ,procure components
O Time expected: 1 week
Increment 3:
O Design and veriIication oI hardware. Start with the soItware design Ior detection oI screen
area to be processed
O CareIully experiment with current and voltage values.
O Time Expected: 2 weeks
Increment 4:
O Study openCV, detect single touch, enable mouse Iunctions with touch
O We must keep in mind use oI computational resources are minimum and high eIIiciency is
reached .
O Time expected : 2 weeks
Increment 5:
O Implement Multi touch gestures with improvement oI eIIiciency
O Documentation and testing.
O Time expected: 3 week
30


(X)
Appendix
TEST CASES/REPORTS
Sr.
No
Objective
Environm
ent
Pass/Fa
il
Data input
Expected
Output
Actual Results Remarks
1
Check iI
LEDs are
working
Windows
/ Webcam
soItware
Pass
Connection to
USB socket
made and
LEDs
switched on
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
output as
intended
2
Check iI
webcam is
working
Windows
/ Webcam
SoItware
Pass
Open the
webcam
viewing
soItware,
Images seen
Webcam
showed images
oI what it was
Iocusing
Web cam
showed images
oI what it was
Iocusing
output as
intended
3
Check iI
USB power
connection
are correctly
made
Windows
/ Webcam
SoItware
Pass
Connection to
USB socket
made and
LEDs
switched on
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
On hovering
webcam over
the LEDs,
glowing LEDs
are seen
output as
intended
4
Check Ior
the screen
area covered
Execution
oI
program /
Windows
Pass
Place
stylus/Iinger
near the
corners oI the
screen
Corners
detected
Corners
detected
output as
intended
5
Check Ior
the leIt click
Iunctionality
Execution
oI
Program /
Windows
Pass
Tap the object
on screen
LeIt mouse click
operation
perIormed
LeIt mouse
click operation
perIormed
output as
intended
6
Check Ior
the right
click
Iunctionality
Execution
oI
program /
windows
Pass
Place the
stylus/Iinger
close to object
on screen
On removing
stylus/Iinger
right click
action
perIormed
On removing
stylus/Iinger
right click
action
perIormed
output as
intended
31

7
Check Ior
drag and
drop
Iunctionality
Execution
oI
program /
windows
Pass
Place the
stylus/Iinger
on the object
and drag it
Drag and drop
Iunctionality
perIormed
Drag and drop
Iunctionality
perIormed
output as
intended
8
Check Ior
rotate
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
image and
rotate them.
Rotate
clockwise/anti-
clockwise
Iunctionality
perIormed
Rotate
clockwise/anti-
clockwise
Iunctionality
perIormed
Output as
intended
9
Check Ior
scrolling
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
document and
move them
up/down.
Scroll
Iunctionality
perIormed
Scroll
Iunctionality
perIormed
Output as
intended
10
Check Ior
zoom in/out
Iunctionality
Execution
oI
program/
window
Pass
Place the
Iingers on the
image and
move them
apart or near.
Zoom in/out
Iunctionality
perIormed
Zoom in /out
Iunctionality
perIormed
Output as
intended













32



(XI)
Appendix C
Description of tools
OpenCV (Open Source Computer Vision Library) is a library oI programming Iunctions
mainly aimed at real time computer vision, developed by Intel and now supported by Willow
Garage. It is Iree Ior use under the open source BSD license. The library is cross-platIorm. It
Iocuses mainly on real-time image processing. II the library Iinds Intel's Integrated
PerIormance Primitives on the system, it will use these proprietary optimized routines to
accelerate itselI. Programming language
The library was originally written in C and this C interIace makes OpenCV portable to some
speciIic platIorms such as digital signal processors. Wrappers Ior languages such as C#,
Python, Ruby and Java (using JavaCV) have been developed to encourage adoption by a
wider audience.
However, since version 2.0, OpenCV includes both its traditional C interIace as well as a new
C interIace
|
, that seeks to reduce the number oI lines oI code necessary to code up vision
Iunctionality as well as reduce common programming errors such as memory leaks that can
arise when using OpenCV in C. Most oI the new developments and algorithms in OpenCV
are now developed in the C interIace. UnIortunately, it is much more diIIicult to provide
wrappers in other languages to C code as opposed to C code; thereIore the other language
wrappers are generally lacking some oI the newer OpenCV 2.0 Ieatures. A CUDA-based GPU
interIace has been in progress since Sept. 2010
$8:55479
OpenCV runs on Android, FreeBSD, iOS, Linux, Mac OS and Windows. The user can get
oIIicial releases Irom SourceForge, or take the current snapshot under SVN Irom there.
OpenCV uses CMake.


33


(XII)
Appendix D
Quality Assurance
Software System Attributes
a) Reliabilit
In order to ensure reliability, this is being designed using soItware that is established to be
stable and easy to use. II the memory constraints are Iollowed, then this system should
present no reliability problems. The design being simple possess uses less memory and
computational resources.
b) Availability
Tool is available Ior use just by switching on the LEDs and running the program. Using stylus
is recommended.
c) Security
The tool is secured and does not causes any security threat to the system. For improving the
design a screen guard can be mounted on the screen to prevent it Irom scratches.
d) Maintainability
There will be no maintenance required Ior the soItware. For hardware side, the wires are
soldered. By craIting the design into metallic sheets will add to its maintenance.
e) Portability
The design is made portable by providing power to LEDs through USB. Array oI LEDs can
be detached or attached to screen easily without any discomIort. InIrared webcam is used
which is available in market or can be easily made.





34



(XIII)
Appendix E
Reflections about the project
What is the 5roblem ou have tried to solve? Wh this work is im5ortant?

Current wireless human-computer interaction devices such as wireless mice and touch
screens, by-and large, incorporate a sophisticated electronic architecture. The sophistication
achieves wireless capabilities but carries over a cost overhead. We lay the Ioundation Ior
developing a novel human-computer interaction device with reduced hardware sophistication.
We developed a inIrared based touch screen-type device Ior desktops using only single
webcam and an array oI seven inIrared LEDs, as opposed to, typically, 2 webcams in
conventional inIrared touch screens. Users can naturally use their Iingers or other tip pointers
to issue commands and type texts. Enabling it with multi-gesture Iunctionalities like rotate ,
zoom , scroll up and down etc have Iurther enhanced the degree oI interaction.

What are our creative achievements in this 5roject? Does our 5roject o5en new was of
thinking? How? Wh is it better/different than other existing a55roaches or solutions?
The need to undergo with this project arises with the Iacts like:
O Existing touch screen methodologies uses capacitive or resistive approach, which are
costly. A touch screen based on capacitive and resistive approach suitable Ior desktop
costs around Rs.20,000 to Rs.50,000.
Key Features oI the proposed design are as Iollows:
O Use oI InIrared Ior detecting the touch, which is not interIered by the background.
O Works with desktop monitors/screens.
O Single Camera is placed inIront oI the screen.
O Design capable oI detecting multiple-gestures (gestures that requires multiple
touches).
O Requires minimal memory and computational resources.
33

What were the main challenges? How did ou address these challenges?
O Hardware design and procurement: Trial and error method was used to address this
challenge
O Software optimization: INCREMENTAL approach oI soItware development was
Iollowed
O Increasing software efficiency in constrained hardware to increase performance:
Built and debug was very helpIul along with revision oI idea again via group
discussion and mentor suggestions.
O Integrating normal touch screen functionalities with multi touch in existing
hardware without compromising system performance: Extensive survey and
designing was done.
What is our a55roach or solution?
The setup is easy: position a camera so it in the image, which determines the projective
mapping between the image plane and the screen. The proposed approach uses inIrared
technology Ior Iinding location oI touch. The region oI touch is seen by computer as a glow
due to inIrared emissions. InIrared webcam is used to capture images oI the screen. AIter
detecting touch on the screen, corresponding signals are sent to perIorm mouse actions. The
mouse actions that can be perIormed includes leIt click as a tap and right click is simulated by
keeping stylus Ior about 1 sec at point oI event. The overall perIormance oI the system is Iast,
accurate, and reliable.
What new things did ou learn? What 5ersonal technical and other 5rofessional
com5etencies have ou been able to strengthen b our engagement in this 5roject? How?
Entire project was new and quite challenging to us. This was the Iirst time we were working
in computer vision domain on such an idea. First we learned the optimum hardware design
that eveolved aIter rigrous trials . We also learned libraries like Open CV. Also not much oI
help in this domain both Ior hardware and soItware design was available as it is newer .
What mistakes did ou make with res5ect to our 5roject? (It is im5ossible not to make an
mistakes.)
We bought a webcam initially that was not conIigured to work as inIrared webcam and we
burnt several LEDs and resistors in experimentation this can be avoided .
If ou were to start again, how would ou a55roach the 5roject?
This is the best possible and most optimised approach till date as is evolved in phases
according to perIormance and cost so we will start with same approach though the time in
hardware design and soItware optimisation we gave initially will be saved this time.
36

Can our 5roject act like a seed for some future 5rojects? If es, how? List the 5roblem
statements for future 5rojects as a natural extension of our 5roject.
This is quite evolving domain with greater eIIorts being made to increase interactability oI
computer and humans. This will deIinitly be a milestone Ior work in inIrared based
technological solutions Ior the same.
Has our 5roject been used b 5ersons outside our grou5? Have ou done anthing to
have (5otential) user's feedback?
It was being demonstrated at project exhibition where it was used by Mr. Sanjay Goel, ProI
J.P. Gupta,Mr A.Farooqi and was well appreciated by them. Also it was being demonstrated
to Mr. Sandeep K Singh , Mr. Manish Thakur , Mr Vikas Saxena , Ms Deviya all admired the
utility and cost eIIectivenss oI the project and the eIIiciency it attained. It was also being
appreciated and used by peer groups without Ilaws and with appreciations.
Did ou collaborate with an other 5roject grou5? How?
No
















37