Computational Photography

Prof. Feng Liu

Spring 2011

http://www.cs.pdx.edu/~fliu/courses/cs510/

04/21/2011

With slides by Prof. C. Dyer and K. Grauman

LastTime
Re-lighting
HDR

2
Today
Panorama
Overview
Featuredetection
Mid-termprojectpresentation
Notrealmid-term
6minutes(5minutespresentation+1minuteQ&A)
May5and10
Schedule
https://spreadsheets.google.com/ccc?key=0AuQJeVAI0gnkdE9
KcktSWUc2LVQtZFhJejRMZnpIenc&hl=en#gid=0
3

PanoramaBuilding:History
4
Along the River During Ching Ming Festival
by Z.D Zhang (1085-1145 )
San Francisco from Rincon Hill, 1851,
by Martin Behrmanx
PanoramaBuilding:AConciseHistory
Thestateoftheartandpracticeisgoodatassembling
imagesintopanoramas

Mid90sCommercialPlayers(e.g.QuicktimeVR)
Late90sRobuststitchers(inresearch)
Early00sConsumerstitchingcommon
Mid00sAutomation

StitchingRecipe
Alignpairsofimages

Alignalltoacommonframe

Adjust(Global)&Blend

StitchingImagesTogether
7

Whendotwoimagesstitch?
8

Imagescanbetransformedtomatch
9
ImagesarerelatedbyHomographies
10
ComputeHomographies
11
AutomaticFeaturePointsMatching
Matchlocalneighborhoodsaroundpoints
UsedescriptorstoefficientlycompareSIFT
[Lowe04]mostcommonchoice

12

StitchingRecipe
Alignpairsofimages

Alignalltoacommonframe

Adjust(Global)&Blend

13
WideBaselineMatching
Images taken by cameras that are far apart make the
correspondence problem very difficult
Feature-based approach: Detect and match feature
points in pairs of images
Detect feature points
Find corresponding pairs
MatchingwithFeatures
MatchingwithFeatures
Problem1:
Detectthesamepoint
independentlyinbothimages
no chance to match!
We need a repeatable detector
MatchingwithFeatures
Problem2:
Foreachpointcorrectly
recognizethecorrespondingpoint
?
We need a reliable and distinctive descriptor
Local:featuresarelocal,sorobusttoocclusionandclutter
(nopriorsegmentation)
Invariant(orcovariant)tomanykindsofgeometricand
photometrictransformations
Robust:noise,blur,discretization,compression,etc.do
nothaveabigimpactonthefeature
Distinctive:individualfeaturescanbematchedtoalarge
databaseofobjects
Quantity:manyfeaturescanbegeneratedforevensmall
objects
Accurate:preciselocalization
Efficient:closetoreal-timeperformance
PropertiesofanIdealFeature
Problem1:DetectingGoodFeaturePoints
Hessian
Harris
Lowe:SIFT(DoG)
Mikolajczyk&Schmid:
Hessian/Harris-Laplacian/Affine
Tuytelaars&VanGool:EBRandIBR
Matas:MSER
Kadir&Brady:SalientRegions
Others
FeatureDetectors
C. Harris, M. Stephens, A Combined Corner and Edge Detector, 1988
HarrisCornerPointDetector
Weshouldrecognizethepointbylooking
throughasmallwindow
Shiftingawindowinanydirectionshouldgive
alargechangeinresponse
HarrisDetector:BasicIdea
flat region:
no change in
all directions
edge:
no change along
the edge direction
corner:
significant change
in all directions
HarrisDetector:BasicIdea
| |
2
,
( , ) ( , ) ( , ) ( , )
x y
E u v w x y I x u y v I x y = + +

Change of intensity for a (small) shift by [u,v] in image I:

Intensity
Shifted
intensity
Weighting
function
or
Weighting function w(x,y) =
Gaussian 1 in window, 0 outside
HarrisDetector:Derivation
Apply2
nd
orderTaylorseriesexpansion:

=
=
=
+ + =
y x
y x
y x
y
y x
x
y x I y x I y x w C
y x I y x w B
y x I y x w A
Bv Cuv Au v u E
,
,
2
,
2
2 2
) , ( ) , ( ) , (
) , ( ) , (
) , ( ) , (
2 ) , (
| |
( , )
A C u
E u v u v
C B v
( (
=
( (

x y x I I
x
c c = / ) , (
y y x I I
y
c c = / ) , (
HarrisDetector
| |
( , ) ,
u
E u v u v M
v
(
~
(

Expanding E(u,v) in a 2
nd
order Taylor series, we have, for small
shifts, [u,v], a bilinear approximation:
2
2
,
( , )
x x y
x y
x y y
I I I
M w x y
I I I
(
=
(
(

where M is a 2 2 matrix computed from image derivatives:

Note: Sum computed over small neighborhood around given pixel
x y x I I
x
c c = / ) , (
y y x I I
y
c c = / ) , (
HarrisCornerDetector
| |
( , ) ,
u
E u v u v M
v
(
~
(

Intensity change in shifting window: eigenvalue analysis

1
,
2
eigenvalues of M
direction of the
slowest change
direction of the
fastest change
(
max
)
-1/2
(
min
)
-1/2
Ellipse E(u,v) = const
HarrisCornerDetector

1
and
2
both large
Image patch
SSD surface
SelectingGoodFeatures
large
1
, small
2
SSD surface
SelectingGoodFeatures
small
1
, small
2
SSD surface
SelectingGoodFeatures

2
Corner

1
and
2
both large,

1
~
2
;
E increases in all
directions

1
and
2
are small;
E is almost constant
in all directions
Edge

1
>>
2

Edge

2
>>
1

Flat
region
Classification of
image points using
eigenvalues of M:
HarrisCornerDetector
HarrisCornerDetector
Measure of corner response:
( )
2
det trace R M k M =
1 2
1 2
det
trace
M
M


=
= +
k is an empirically-determined constant; e.g., k = 0.05
HarrisCornerDetector

2
Corner
Edge
Edge
Flat
R depends only on
eigenvalues of M
R is large for a corner
R is negative with large
magnitude for an edge
|R| is small for a flat
region
R > 0
R < 0
R < 0 |R| small
HarrisCornerDetector:Algorithm
Algorithm:
1. Findpointswithlargecorner
responsefunctionR
(i.e.,R>threshold)
2. Takethepointsoflocalmaxima
ofR(forlocalization)bynon-
maximumsuppression
HarrisDetector:Example
Slide credit: C. Dyer
Compute corner response R =
1

2
k(
1
+
2
)
2
HarrisDetector:Example
Slide credit: C. Dyer
HarrisDetector:Example
Find points with large corner response: R > threshold
Slide credit: C. Dyer
Take only the points of local maxima of R
HarrisDetector:Example
HarrisDetector:Example
Interest points extracted with Harris (~ 500 points)
HarrisDetector:Example
HarrisDetector:Example
HarrisDetector:Summary
Averageintensitychangeindirection[u,v]canbe
expressedinbilinearform:

DescribeapointintermsofeigenvaluesofM:
measureofcornerresponse:

Agood(corner)pointshouldhavealargeintensity
changeinalldirections,i.e.,Rshouldbealarge
positivevalue
| |
( , ) ,
u
E u v u v M
v
(
~
(

( )
2
1 2 1 2
R k = +
HarrisDetectorProperties
Rotationinvariance
Ellipse rotates but its shape (i.e., eigenvalues)
remains the same
Corner response R is invariant to image rotation
Butnotinvarianttoimagescale
Fine scale: All points will
be classified as edges
Coarse scale: Corner
HarrisDetectorProperties
HarrisDetector:Scale
R
min
= 0
R
min
= 1500
HarrisDetectorProperties
QualityofHarrisdetectorfordifferentscale
changes
Repeatability rate:
# correct correspondences
# possible correspondences
C. Schmid et al., Evaluation of Interest Point Detectors, IJCV 2000
47
Computetheimagegradientoverentireimage
Foreachpixelp:
formthematrixMover(2N+1)x(2N+1)
neighborhoodofp
computethesmallesteigenvalueofM
ifeigenvalueisaboveathreshold,savethe
coordinatesofpinalistL
SortLindecreasingorderofeigenvalues
Scanningthesortedlisttoptobottom:For
eachcurrentpoint,p,deleteallotherpointson
thelistthatbelongtotheneighborhoodofp
TomasiandKanadeCornerDetector
48
Results
49
Results
NextTime
Panorama
Featurematching

50