ComputingOpticalFlow

Lecture7

OpticalFlow
Motionvector(u,v)
ImageDisplacementinxandydirectionsbetween
twoconsecutiveframes

MotionFlow
DisplacementVector
Disparity(generalcase)

HamburgTaxiseq

OpticalFlowFieldExamples

OpticalFlow Examples

Encoding
Scheme

v
u

Videos

Color Coded Optical Flows

OpticalFlow Examples

Encoding
Scheme

v
u

Videos

Color Coded Optical Flows

OpticalFlow

Color: Orientation
Brightness: Magnitude

OpticalFlow
Applications
Motionbasedsegmentation
StructurefromMotion(3DshapeandMotion)
Alignment(Globalmotioncompensation)
Camcordervideostabilization
UAVVideoAnalysis

VideoCompression

Cited by 9557

Horn&SchunckOpticalFlow
Brightness constancy assumption
f ( x, y , t ) f ( x dx , y dy , t dt )

Horn&SchunckOpticalFlow
Brightness constancy assumption
f ( x, y , t ) f ( x dx , y dy , t dt )

TaylorSeries

f (x )

Horn&SchunckOpticalFlow
Brightness constancy assumption
f ( x, y , t ) f ( x dx , y dy , t dt )

Taylor Series
f ( x, y , t ) f ( x, y , t )

f
f
f
( x dx x ) ( y dy y ) (t dt t )
x
y
t

0 f x dx f y dy f t dt
dx
dy
dt
0 fx
fy
ft
dt
dt
dt
0 f xu f y v ft

Interpretationofopticalfloweq
f xu f y v ft 0
d=normal flow
p=parallel flow

f
f
v x u t
fy
fy

Equation of st.line

Horn&Schunck(contd)
{( f u f v f )
x

(u x2 u 2y v x2 v 2y )}dxdy

Brightness constancy

Smoothness constraint
Min (Variational Calculus

( f x u f y v f t ) f x ( 2u ) 0
( f x u f y v f t ) f y (( 2 v ) 0

DerivativeMasks(Roberts)
1
1

1
1

fx

1
first image
1
1
second image
1

1 1
1
first image
1

1 1
1
second image
1

fy

Apply first mask to 1st image

Second mask to 2nd image
Add the responses to get f_x, f_y, f_t

1 1
1 1 first image

1 1
1 1second image

ft

Laplacian
0

1
4

1
4

1
4

1
0
4
f xx f yy

f xx f yy f f av

Horn&Schunck(contd)
{( f u f v f )
x

(u x2 u 2y v x2 v 2y )}dxdy

min

( f x u f y v f t ) f x ( 2u ) 0
( f x u f y v f t ) f y (( 2 v ) 0

discrete version
( f x u f y v f t ) f x (u uav ) 0
( f x u f y v f t ) f y (( v vav ) 0

variational calculus
P
D
P
v vav f y
D

u uav f x

Algorithm1
k=0
u
v
Initialize
Repeat until some error measure is satisfied
(converges)
K

P
D
P
v vav f y
D

u uav f x

P f x uav f y vav f t
D f x2 f y2

SyntheticImages

Horn&SchunckResults

One iteration

10 iterations

Lucas&KanadeMethod

Cited by 8467

Lucas&Kanade(LeastSquares)
Opticalfloweq
f xu f y v ft

Consider 3 by 3 window

f x1

f x 9

f y1
f t1


u

f t 9
f y 9

f x1u f y1v f t1

f x 9u f y 9 v f t 9

Au ft

Lucas&Kanade
Au f t
A T Au A Tft
u ( A T A ) 1 A Tft

Pseudo Inverse

find (u, v) s.t. min ( f xi u f yi v f t ) 2

i

Least Squares Fit

Lucas&Kanade
min ( f xi u f yi v f t ) 2
i

( f xi u f yi v f t ) 2 0

u i

2
(
f
u

f
v

f
)
0

xi
yi
t
v i

( f

xi

u f yi v f ti ) f xi 0

( f

xi

u f yi v f ti ) f yi 0

Lucas&Kanade
( f

xi

u f yi v f ti ) f xi 0

( f

xi

u f yi v f ti ) f yi 0

f
f

2
xi

u f xi f yi v f xi f ti

2
f
u

f
yi v f yi f ti
xi yi

2
f xi

f xi f yi

f
f
f
f
xi yi u xi ti

v
f yi2 f yi fti

Lucas&Kanade

u
v

f xi2

2
f

yi

f yi2 ( f xi f yi ) 2 f xi f yi

f xi f yi

2
f
xi

f
f

xi ti

f
f

yi ti

Lucas&Kanade

Lucas-Kanade
without pyramids
Fails in areas of large
motion

Lucas-Kanade with Pyramids

Comments
HornSchunck andLucasKanade optical
methodsworkonlyforsmallmotion.
Ifobjectmovesfaster,thebrightnesschanges
rapidly,
2x2or3x3masksfailtoestimatespatiotemporal
derivatives.

Pyramidscanbeusedtocomputelargeoptical
flowvectors.

LucasKanadewithPyramids
ComputesimpleLKopticalflowathighestlevel

Atleveli
Takeflowui1,vi1 fromleveli1
bilinearinterpolateittocreateui*,vi*
matricesoftwiceresolutionforleveli
multiplyui*,vi* by2
computeft ,fx,fy usingmaskscenteredat
(x,y)and(x+ui*, y+vi*)
ApplyLKtogetui(x,y),vi(x,y)(the
correctioninflow)
Addcorrectionsui vi ,i.e.ui =ui*+ui,
vi =vi*+vi.

Pyramids

pyramid

pyramid

Interpolation

1DInterpolation

f(1)

f(2)

2DInterpolation
Bilinear

X (3,6) X (4,6)
X(3,5) X (4,5)

BilinearInterpolation
Four nearest points of (x,y) are:

3
5
4
6

(3.2,5.6)
X (3,6) X (4,6)
X(3,5) X (4,5)

BilinearInterpolation

Homework

Lucas-Kanade
without pyramids
Fails in areas of large
motion

Lucas-Kanade with Pyramids