Emerging Technologies

Department of Computer Science & Engineering

R V College of Engineering, Bangalore

THARUN S 1RV12CS113

RVCE - Marching Ahead

4 Aug 2013

Project Title
Hand Gesture Recognition using
Opticl flow techniques

RVCE - Marching Ahead

4 Aug 2013

Literature survey
1) Gihan Shin,; Junchul Chun,. Vision-based Multimodal Human
Computer Interface based on Parallel Tracking of Eye and Hand Motion
2007 International Conference on Convergence Information Technology
This paper presented a vision-based multimodal human computer
interface system using eye and hand motion tracking. Conventional
vision-based human computer interface use eye or hand motion tracking
individually.
For motion tracking and recognition of eye and hand gesture optical
flow method and template matching is used.

RVCE - Marching Ahead

4 Aug 2013

 Conventionally, tracking the motion of hand gesture and eye

is considered a individual process for developing a different
virtual interface.
However, in this work, a vision-based multimodal human
computer interface system which combines eye and hand
motion tracking and their recognition was developed.
The proposed vision-based virtual interface integrated the
function of the motion tracking of eye blinking and hand
gesture with the function of their recognition.
The proposed interface could be easily used a substitute for
a traditional devices such as mouse.

RVCE - Marching Ahead

4 Aug 2013

2. Philippe Weinzaepfel,; Jerome Revaud., DeepFlow: Large

displacement optical flow with deep matching 2013 IEEE international
Conference on Computer Vision, (ICCV)

Optical flow computation is a key component in many computer vision systems

designed for tasks such as action detection or activity recognition.
However, despite several major advances over the last decade, handling large
displacement in optical flow remains an open problem.
Inspired by the large displacement optical flow of Brox & Malik, DeepFlow,
blends a matching algorithm with a variational approach for optical flow. it
proposes a descriptor matching algorithm, tailored to the optical flow problem,
that allows to boost performance on fast motions.
The matching algorithm builds upon a multi-stage architecture with 6 layers,
interleaving convolutions and max-pooling, a construction akin to deep
convolutional nets. Using dense sampling, it allows to effi- ciently retrieve
quasi-dense correspondences, and enjoys a built-in smoothing effect on
descriptors matches, a valuable asset for integration into an energy minimization
framework for optical flow estimation.

RVCE - Marching Ahead

4 Aug 2013

Guojian Wei, Zhiqiang Hou, Wu Li, Wangsheng Yu Color image optical

flow estimation algorithm with shadow suppression 2013 Seventh
Internation Conference on Image and Graphics (ICIG)
A new optical flow estimation algorithm for color image is proposed in
this paper overcome the influence of shadows and improve the accuracy
of optical flow estimation.
The idea is to compute optical flow of the color invariant space and then
fuse the result with optical flow of the RGB space.
Firstly, it computes edge strength from color invariants of two adjacent
frames to build three-channel color images, and then computes optical
flow of the images we have built before, finally, fuses the result of color
invariant optical flow with RGB image optical flow through L norm.
The experimental results using this method show that the target regions
detected by this method are more robust and accurate under shadow and
illumination conditions.

RVCE - Marching Ahead

4 Aug 2013

Research Gap
So far, most of the research done in this field
has been directed to computation of the flow
field between the frames of black-and-white
image sequences.
Colour by itself carries more information
about the objects than the grey value and
provides us more information that we can
use for computing the optical flow.
RVCE - Marching Ahead

4 Aug 2013

Motivation
Human computer interactions has always
been limited to few hardware devices.
Allowing gestures to be a part of interaction
with the help of image processing.
The benefits of using gestures for human
computer interaction.
Optical flow algorithms are efficient and
simpler to implement
RVCE - Marching Ahead

4 Aug 2013

Motivation
Motion is a rich source of information about the
world.
Tracking of a moving object over time is a
challenging issue on video processing.
Optical flow is very useful for detecting the moving
objects in a video.
Optical flow vector is calculated between adjacent
frames in a video and we can simply track the path
of moving object by simply following the arrows of
optical flow vectors.
RVCE - Marching Ahead

4 Aug 2013

Problem statement and objectives

Gesture hands
Optical flow
Increase efficienc
Explore new tecniques use papers
Use existing methods and change shit

RVCE - Marching Ahead

4 Aug 2013

Content and Ideas

Optical flow computation is a key component in many computer vision
systems designed for tasks such as action detection or activity
recognition.
Optical Flow estimation is used to estimate displacement vectors for
each pixels in two frames of a video. This displacement vector says how
quickly a pixel is moving across the image and direction of movement
of each pixel.
One of the key correspondence problems in Computer Vision is the
computation of the optic flow field between two consecutive frames of
an image sequence.

RVCE - Marching Ahead

4 Aug 2013

Optical Flow

A video is a collection of frames and we are trying to

estimate the direction and the velocity of the motion
introduced in such an image sequence.
In order to compute the motion we need to compare certain
features that stay invariant in the sequence and that can help
us to identify the objects and their movement in the scene.
The 2D displacement field that describes the motion of a
scene between two consecutive frames of an image
sequence is called optic flow.

RVCE - Marching Ahead

4 Aug 2013

 The image is represented as a pixel matrix, where every pixel has a

unique coordinate position.
In order to represent the problem mathematically, we consider a scalarvalued image sequence I(x, y, t)where (x, y) denotes the coordinates of
the pixel at time t.
Let the image frame at time t be denoted by I(x, y, t).Then the image
frame at time t + 1 will be denoted by I(x+u, y+v, t+1), where x + u and
y + v show the new position of the pixel which was previously at (x,y)
position.
Optical flow is a flexible representation of visual motion that is
particularly suitable for computers analyzing digital images. It
associates each image pixel (x, y) with a two component vector u =
[u(x, y), v(x, y)] indicating its apparent instantaneous 2D velocity.
The presented paper uses a combination of feature match algorithm and
super pixel segmentation
RVCE - Marching Ahead

4 Aug 2013

Sugitha, T.S.; Gladwin, J.K.T.,Optical Flow Based On Feature

Match and Super Pixel Segmentation Advances in Computing
and Commonications (ICACC), 2014
This paper describes optical flow estimation. The estimation
is carried out using feature match method followed super
pixel segmentation.
Feature match algorithm is based on the matching between
features of patches of adjacent frames in a video .
This displacement vectors obtained by feature match
algorithm are modified by using super pixel segmentation
based optical flow estimation.
The best optical flow associated with each superpixel is
computed by optimizing a cost function.

RVCE - Marching Ahead

4 Aug 2013

 Feature match algorithm computes the matching of patches between two

frames and find out displacement vector for each pixel in the image .
Then the super pixel segmentation is applied to the displacement vectors
obtained from the Feature match algorithm and it improves the the
result obtained by Feature Match algorithm
Superpixel based optical flow is computed by optimizing a cost
function.
Super pixel based optical flow estimation improves feature match
algorithm and increase the quality of optical flow estimation.
The resultant flow vectors is found to be more closer to the Ground
Truth.

RVCE - Marching Ahead

4 Aug 2013

 The core element of patch matching methods is a repeated

search of all patches in one image region for the most
similar patch in another image region. In other words, given
images A and B in figure 2, find for every patch in A the
nearest neighbor in B . We call this mapping as nearest
neighbor field mapping.
Computing Approximate Nearest-Neighbour
Field(ANNF)between a pair of images has become a major
problem in Optical flow estimation.

RVCE - Marching Ahead

4 Aug 2013

Feature match algorithm

The boxes (Red, Green, Blue and Yellow)in the figure denote
similar patches in the pair of images.
For each patch of size pp in the Target image, finding the closest
patch in the Source image is the Nearest Neighbour problem
between a pair of images.
Since dealing in higher dimensions has become a major problem ,
we propose that p p image patches can be approximated
intelligently to a low dimensional feature vector and then
conventional k-nearest neighbour algorithms can be used effectively
on this feature vector.

RVCE - Marching Ahead

4 Aug 2013

Super pixel segmentation based

optical flow
In computer vision, image segmentation is the process of partitioning a
digital image into multiple segments or sets of pixels called super
pixels.
The goal of super pixel segmentation is to simplify or change the
representation of an image into something that is more meaningful and
easier to analyze by using this super pixel segmentation , we can assign
a label to every pixel in an image such that pixels with the same label
share certain visual characteristics.
Optical flow result can be improved by using super pixel segmentation.
Displacement vector obtained by using Feature match algorithm can be
improved by using super pixel segmentation based on the assumption
that a super pixel should have same velocity vectors. Same velocity
vectors is assigned to a super pixel by using a cost function.
RVCE - Marching Ahead

4 Aug 2013

Testing

RVCE - Marching Ahead

4 Aug 2013

RVCE - Marching Ahead

4 Aug 2013

Performance
The most commonly used measure of performance for
optical flows are average angular error (AAE)and average
end point error(AEE). The angular error between two flows
(u0, v0)and (u1, v1) is the angle in 3D space between (u0,
v0, 1) and(u1, v1, 1). The AE is usually computed by
normalizing the vectors, taking the dot product, and then
and then taking the inverse cosine of their dot product .

RVCE - Marching Ahead

4 Aug 2013

Thank you

RVCE - Marching Ahead

4 Aug 2013