Why extract features?

Motivation: panorama stitching

Lecture 12 We have two images how do we combine them?

Local Feature Detection

Guest lecturer: Alex Berg

Reading: We need to match (align) images

Harris and Stephens
David Lowe IJCV

Building a Panorama

Matching with Invariant Features

Darya Frolova, Denis Simakov

The Weizmann Institute of Science
March 2004
http://www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/InvariantFeatures.ppt
M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 2003

Why extract features? Why extract features?

Motivation: panorama stitching Motivation: panorama stitching
We have two images how do we combine them? We have two images how do we combine them?

Step 1: Detect feature points in both images Step 1: Detect feature points in both images
Step 2: Find corresponding pairs Step 2: Find corresponding pairs
Step 3: Use these pairs to align images

1
Matching with Features Matching with Features
Problem 1: Problem 2:
Detect the same point independently in both images For each point correctly recognize the corresponding
one

?
no chance to match!

We need a repeatable detector We need a reliable and distinctive descriptor

Selecting Good Features Applications

Whats a good feature? Feature points are used for:
Satisfies brightness constancy Motion tracking
Has sufficient texture variation Image alignment
Does not have too much texture variation
Corresponds to a real surface patch
3D reconstruction
Does not deform too much over time Object recognition
Indexing and database retrieval
Robot navigation

Contents Finding Corners

Harris Corner Detector
Description
Analysis
Detectors
Rotation invariant
Scale invariant Key property: in the region around a
Affine invariant corner, image gradient has two or more
Descriptors dominant directions
Rotation invariant
Corners are repeatable and distinctive
Scale invariant
Affine invariant C.Harris and M.Stephens. "A Combined Corner and Edge Detector.
Proceedings of the 4th Alvey Vision Conference: pages 147--151.

2
 The Basic Idea

An introductory example: We should easily recognize the point by looking

through a small window
Shifting a window in any direction should give a
Harris corner detector large change in intensity

C.Harris, M.Stephens. A Combined Corner and Edge Detector. 1988

Harris Detector: Basic Idea Contents

Harris Corner Detector
Description
Analysis
Detectors
Rotation invariant
Scale invariant
Affine invariant
flat region: edge: corner: Descriptors
no change in no change along significant change Rotation invariant
all directions the edge direction in all directions Scale invariant
Affine invariant

Harris Detector: Mathematics Harris Detector: Mathematics

Change of intensity for the shift [u,v]:
Window-averaged change of intensity for the shift [u,v]:
E (u , v) = w( x, y ) [ I ( x + u , y + v) I ( x, y ) ]
2

E (u, v) = w( x, y ) [ I ( x + u, y + v) I ( x, y ) ]
2
x, y
x, y

Window Shifted
function intensity
Intensity Second-order Taylor expansion of E(u,v) about (0,0)
(bilinear approximation for small shifts):
E (0,0) 1 Euu (0,0) Euv (0,0) u
Window function w(x,y) = or E (u , v) E (0,0) + [u v] u + [u v ] E (0,0) E (0,0) v
Ev (0,0) 2 uv vv
1 in window, 0 outside Gaussian

3
Harris Detector: Mathematics Interpreting the second moment
Expanding E(u,v) in a 2nd
order Taylor series expansion, we matrix
have,for small shifts [u,v], a bilinear approximation:
First, consider an axis-aligned corner:
u
E (u , v ) [ u , v ] M
v
where M is a 22 matrix computed from image derivatives:

I2 IxI y
M = w( x, y ) x
x, y I x I y I y2

Interpreting the second moment General Case

matrix 0
Since M is symmetric, we have M = R 1 1 R
First, consider an axis-aligned corner: 0 2
I x2 I I x y
1 0 We can visualize M as an ellipse with axis lengths
M = = determined by the eigenvalues and orientation determined
I x I y I 2
0 2
y by R
direction of the
fastest change
This means dominant gradient directions align Ellipse E(u,v) = const direction of the
with x or y axis slowest change

u (max)-1/2
If either is close to 0, then this is not a corner, [u v ] M = const (min)-1/2
so look for locations where both are large. v

Slide credit: David Jacobs

Harris Detector: Mathematics Harris Detector: Mathematics

Classification of 2 Edge Measure of corner response:

image points using 2 >> 1 Corner
R = det M k ( trace M )
2
eigenvalues of M: 1 and 2 are large,
1 ~ 2;
E increases in all
directions
det M = 12
trace M = 1 + 2
1 and 2 are small;
E is almost constant Flat Edge
in all directions region 1 >> 2 (k empirical constant, k = 0.04-0.06)
1

4
Harris Detector: Mathematics Harris Detector: Summary
R = det(M ) k trace(M ) 2 = 12 k (1 + 2 ) 2 , 0.04 k .06
Average intensity change in direction [u,v] can be expressed
2 as a bilinear form:
Edge Corner
u
R<0 E (u , v ) [ u , v ] M
R depends only on v
eigenvalues of M Describe a point in terms of eigenvalues of M:
R>0
measure of corner response
R is large for a corner
R = 12 k ( 1 + 2 )
2

R is negative with large

magnitude for an edge A good (corner) point should have a large intensity change in
all directions, i.e. R should be large positive
|R| is small for a flat Flat Edge
region |R| small R<0
1

Harris Detector Harris Detector: Workflow

Algorithm:
1. Compute Gaussian derivatives at each pixel
2. Compute second moment matrix M in a
Gaussian window around each pixel
3. Compute corner response function R
4. Threshold R
5. Find local maxima of response function
(nonmaximum suppression)

Harris Detector: Workflow Harris Detector: Workflow

Compute corner response R Find points with large corner response: R>threshold

5
 Harris Detector: Workflow Harris Detector: Workflow
Take only the points of local maxima of R

Contents Harris Detector: Some Properties

Harris Corner Detector
Rotation invariance
Description
Analysis
Detectors
Rotation invariant
Scale invariant
Affine invariant
Descriptors
Rotation invariant Ellipse rotates but its shape (i.e. eigenvalues)
remains the same
Scale invariant
Affine invariant
Corner response R is invariant to image rotation

Harris Detector: Some Properties Harris Detector: Some Properties

Invariance to image intensity change?
Partial invariance to additive and multiplicative
intensity changes
9 Only derivatives are used => invariance
to intensity shift I I + b

9 Intensity scale: I a I

R R
threshold

x (image coordinate) x (image coordinate)

6
Harris Detector: Some Properties Harris Detector: Some Properties

Invariant to image scale?

All points will be Corner !

classified as edges

Not invariant to scaling

Harris Detector: Some Properties Contents

Harris Corner Detector
Quality of Harris detector for different scale Description
changes Analysis
Detectors
Rotation invariant
Repeatability rate:
# correspondences Scale invariant
# possible correspondences Affine invariant
Descriptors
Rotation invariant
Scale invariant
Affine invariant

C.Schmid et.al. Evaluation of Interest Point Detectors. IJCV 2000

Invariance
We want features to be detected despite
geometric or photometric changes in the image:
if we have two transformed versions of the
We want to: same image, features should be detected in
corresponding locations
detect the same interest points
regardless of image changes

Darya Frolova, Denis Simakov

http://www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/InvariantFeatures.ppt

7
Models of Image Change Contents
Geometric Harris Corner Detector
Rotation Description
Analysis
Detectors
Scale Rotation invariant
Scale invariant
Affine invariant
Affine
Descriptors
valid for: orthographic camera, locally planar
Rotation invariant
object
Scale invariant
Affine invariant
Photometric
Affine intensity change (I a I + b)

Rotation Invariant Detection Contents

Harris Corner Detector Harris Corner Detector
Description
Analysis
Detectors
Rotation invariant
Scale invariant
Affine invariant
Descriptors
Rotation invariant
Scale invariant
Affine invariant

C.Schmid et.al. Evaluation of Interest Point Detectors. IJCV 2000

Scale Invariant Detection Scale Invariant Detection

Consider regions (e.g. circles) of different sizes around The problem: how do we choose corresponding circles
a point independently in each image?
Regions of corresponding sizes will look the same in
both images

8
 Scale Invariant Detection Scale Invariant Detection
Solution: Common approach:
Design a function on the region (circle), which is scale
invariant (the same for corresponding regions, even if
they are at different scales) Take a local maximum of this function

Observation: region size, for which the maximum is

Example: average intensity. For corresponding regions
(even of different sizes) it will be the same. achieved, should be invariant to image scale.

For a point in one image, we can consider it as a Important: this scale invariant region size is
function of region size (circle radius) found in each image independently!

f Image 1 f Image 2 f Image 1 f Image 2

scale = 1/2 scale = 1/2

region size region size s1 region size s2 region size

Scale Invariant Detection Scale Invariant Detection

A good function for scale detection: Functions for determining scale f = Kernel Image
has one stable sharp peak
Kernels:

f f f Good ! L = 2 ( Gxx ( x, y, ) + G yy ( x, y, ) )
bad bad (Laplacian)
region size region size region size
DoG = G ( x, y, k ) G ( x, y, )
(Difference of Gaussians)
For usual images: a good function would be a one
which responds to contrast (sharp local intensity where Gaussian
change) x2 + y 2
Note: both kernels are invariant to
G ( x, y , ) = 1
2
e 2 2
scale and rotation

Scale Invariant Detectors Scale Invariant Detectors

Laplacian

scale
Harris-Laplacian1 Experimental evaluation of detectors
Find local maximum of: w.r.t. scale change
Harris corner detector in
y
space (image coordinates)
Laplacian in scale
Harris x Repeatability rate:
# correspondences
SIFT (Lowe)2 scale # possible correspondences
DoG

Find local maximum of:

Difference of Gaussians in
space and scale y

DoG x
1 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 2001
2 D.Lowe. Distinctive Image Features from Scale-Invariant Keypoints. Accepted to IJCV 2004 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 2001

9
Scale Invariant Detection: Contents
Summary Harris Corner Detector
Description
Given: two images of the same scene with a large scale Analysis
difference between them
Detectors
Goal: find the same interest points independently in
each image Rotation invariant
Solution: search for maxima of suitable functions in Scale invariant
scale and in space (over the image) Affine invariant
Descriptors
Rotation invariant
Methods:
Scale invariant
1. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over
scale, Harris measure of corner response over the image Affine invariant
2. SIFT [Lowe]: maximize Difference of Gaussians over scale and space

Affine Invariant Detection Affine Invariant Detection

Take a local intensity extremum as initial point
Above we considered: Go along every ray starting from this point and stop when
Similarity transform (rotation + uniform scale) extremum of function f is reached

f I (t ) I 0
f (t ) = t

points along the ray

1
t I (t ) I
o
0 dt

Now we go on to:
Affine transform (rotation + non-uniform scale) We will obtain approximately
corresponding regions

Remark: we search for scale

in every direction

T.Tuytelaars, L.V.Gool. Wide Baseline Stereo Matching Based on Local,

Affinely Invariant Regions. BMVC 2000.

Affine Invariant Detection Affine Invariant Detection

Algorithm summary (detection of affine invariant region): The regions found may not exactly correspond, so we approximate
Start from a local intensity extremum point them with ellipses
Go in every direction until the point of extremum of some
function f Geometric Moments:
Curve connecting the points is the region boundary
m pq = x Fact: moments mpq uniquely
p
y q f ( x, y )dxdy
Compute geometric moments of orders up to 2 for this region determine the function f
2
Replace the region with ellipse
Taking f to be the characteristic function of a region
(1 inside, 0 outside), moments of orders up to 2 allow
to approximate the region by an ellipse

This ellipse will have the same moments of

orders up to 2 as the original region
T.Tuytelaars, L.V.Gool. Wide Baseline Stereo Matching Based on Local,
Affinely Invariant Regions. BMVC 2000.

10
Affine Invariant Detection Affine Invariant Detection :
Covariance matrix of region points defines an ellipse:
Summary
Under affine transformation, we do not know in advance shapes of
q = Ap the corresponding regions
Ellipse given by geometric covariance matrix of a region robustly
approximates this region
For corresponding regions ellipses also correspond.
pT 11 p = 1 qT 21q = 1
1 = pp T
2 = qqT
region 1
region 2
( p = [x, y]Tis relative Methods:
to the center of mass) 2 = A1 AT 1. Search for extremum along rays [Tuytelaars, Van Gool]:
2. Maximally Stable Extremal Regions [Matas et.al.]
Ellipses, computed for corresponding
regions, also correspond!

Contents Point Descriptors

Harris Corner Detector We know how to detect points
Description Next question:
How to match them?
Analysis
Detectors
Rotation invariant
Scale invariant
Affine invariant
?
Descriptors
Rotation invariant
Scale invariant
Point descriptor should be:
Affine invariant
1. Invariant
2. Distinctive

Contents Descriptors Invariant to Rotation

Harris Corner Detector Harris corner response measure:
Description depends only on the eigenvalues of the matrix M
Analysis
I2 IxI y
M = w( x, y ) x
Detectors
Rotation invariant
x, y I x I y I y2
Scale invariant
Affine invariant
Descriptors
Rotation invariant
Scale invariant
Affine invariant

C.Harris, M.Stephens. A Combined Corner and Edge Detector. 1988

11
 Descriptors Invariant to Rotation Descriptors Invariant to Rotation
Image moments in polar coordinates Find local orientation

mkl = r k e i l I (r , )drd Dominant direction of gradient

Rotation in polar coordinates is translation of the angle:

+0
This transformation changes only the phase of the moments, but
not its magnitude Compute image derivatives relative to this
orientation
Rotation invariant descriptor consists
mkl
of magnitudes of moments:

Matching is done by comparing vectors [|mkl|]k,l

1 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 2001
J.Matas et.al. Rotational Invariants for Wide-baseline Stereo. Research Report of CMP, 2003 2 D.Lowe. Distinctive Image Features from Scale-Invariant Keypoints. Accepted to IJCV 2004

Contents Descriptors Invariant to Scale

Harris Corner Detector Use the scale determined by detector to compute
Description descriptor in a normalized frame
Analysis
Detectors For example:
Rotation invariant moments integrated over an adapted window
Scale invariant derivatives adapted to scale: sIx
Affine invariant
Descriptors
Rotation invariant
Scale invariant
Affine invariant

Contents Affine Invariant Descriptors

Harris Corner Detector Affine invariant color moments
Description
Analysis pq =
mabc
region
x p y q R a ( x, y )G b ( x, y ) B c ( x, y )dxdy
Detectors
Rotation invariant
Scale invariant Different combinations of these moments
Affine invariant are fully affine invariant
Descriptors
Rotation invariant Also invariant to affine transformation of
Scale invariant intensity I a I + b
Affine invariant

F.Mindru et.al. Recognizing Color Patterns Irrespective of Viewpoint and Illumination. CVPR99

12
 Affine Invariant Descriptors SIFT Scale Invariant Feature Transform1
Find affine normalized frame Empirically found2 to show very good performance, invariant to
image rotation, scale, intensity change, and to moderate affine
A transformations
1 = ppT 2 = qqT

Scale = 2.5
11 = A1T A1 A1 A2 21 = A2T A2 Rotation = 450

rotation

Compute rotational invariant descriptor in this

normalized frame 1 D.Lowe. Distinctive Image Features from Scale-Invariant Keypoints. Accepted to IJCV 2004
J.Matas et.al. Rotational Invariants for Wide-baseline Stereo. Research Report of CMP, 2003 2 K.Mikolajczyk, C.Schmid. A Performance Evaluation of Local Descriptors. CVPR 2003

CVPR 2003 Tutorial Invariant Local Features

Image content is transformed into local feature
coordinates that are invariant to translation, rotation,
Recognition and Matching scale, and other imaging parameters

Based on Local Invariant

Features
David Lowe
Computer Science Department
University of British Columbia

SIFT Features

Advantages of invariant local features Scale invariance

Locality: features are local, so robust to occlusion and Requires a method to repeatably select points in location and scale:
clutter (no prior segmentation) The only reasonable scale-space kernel is a Gaussian
(Koenderink, 1984; Lindeberg, 1994)
Distinctiveness: individual features can be matched to An efficient choice is to detect peaks in the difference of
a large database of objects Gaussian pyramid (Burt & Adelson, 1983; Crowley & Parker,
1984 but examining more scales)
Quantity: many features can be generated for even lur
B

ubtrac
S

Difference-of-Gaussian with constant ratio of scales is a close

small objects approximation to Lindebergs scale-normalized Laplacian (can
Efficiency: close to real-time performance be shown from the heat diffusion equation)

Extensibility: can easily be extended to wide range of

differing feature types, with each adding robustness

13
Scale space processed one octave at Key point localization
a time Detect maxima and minima of
difference-of-Gaussian in scale space
Fit a quadratic to surrounding values
for sub-pixel and sub-scale
interpolation (Brown & Lowe, 2002)
Taylor expansion around point:
lur
B

ubtrac
S

Offset of extremum (use finite

differences for derivatives):

Select canonical orientation Example of keypoint detection

Create histogram of local gradient
Threshold on value at DOG peak and on ratio of principle
directions computed at selected scale
curvatures (Harris approach)
Assign canonical orientation at peak of
smoothed histogram (a) 233x189 image
Each key specifies stable 2D (b) 832 DOG extrema
coordinates (x, y, scale, orientation) (c) 729 left after peak
value threshold
(d) 536 left after testing
ratio of principle
curvatures

0 2

SIFT vector formation Feature stability to noise

Thresholded image gradients are sampled over 16x16 array Match features after random change in image scale &
of locations in scale space orientation, with differing levels of image noise
Create array of orientation histograms Find nearest neighbor in database of 30,000 features
8 orientations x 4x4 histogram array = 128 dimensions

14
 Feature stability to affine change Distinctiveness of features
Match features after random change in image scale & Vary size of database of features, with 30 degree affine
orientation, with 2% image noise, and affine distortion change, 2% image noise
Find nearest neighbor in database of 30,000 features Measure % correct for single nearest neighbor match

A good SIFT features tutorial

http://www.cs.toronto.edu/~jepson/csc2503/tutSIFT04.pdf
By Estrada, Jepson, and Fleet.

15