CS6350 COMPUTERVISION

LocalFeatureDetectors
and
Descriptors
 Overview
Localinvariantfeatures
Local invariant features
Keypoint localization
Hessiandetector
Harriscornerdetector
ScaleInvariantregiondetection
Laplacian
aplacian of
ofGaussian(LOG)detector
Gaussian ( OG) detector
DifferenceofGaussian(DOG)detector
Localfeaturedescriptor
Local feature descriptor
ScaleInvariantFeatureTransform(SIFT)
GradientLocalizationOrientedHistogram(GLOH)
Gradient Localization Oriented Histogram (GLOH)
Examplesofotherlocalfeaturedescriptors
 Motivation
Globalfeaturefromthewholeimageisoftennotdesirable

Insteadmatchlocalregionswhichareprominenttothe
object or scene in the image.
objectorsceneintheimage.
ApplicationArea
Objectdetection
Obj t d t ti
Imagematching
Imagestitching
 Requirementsofalocalfeature
Repetitive:Detectthesamepointsindependentlyineachimage.
Repetitive : Detect the same points independently in each image.
Invarianttotranslation,rotation,scale.
Invarianttoaffinetransformation.
Invarianttopresenceofnoise,bluretc.
Locality:Robusttoocclusion,clutterandilluminationchange.
Locality :Robust to occlusion clutter and illumination change
Distinctiveness:Theregionshouldcontaininteresting
structure.
Quantity:Thereshouldbeenoughpointstorepresenttheimage.
Quantity : There should be enough points to represent the image
Timeefficient.
 Generalapproach
pp
+ +
+
1. Findtheinterestpoints.
2. Considertheregion
around each keypoint.
aroundeachkeypoint.
+
+
+ ()
()
local descriptor

3. Computealocal
d
descriptorfromthe
i t f th
regionandnormalize
thefeature.
4. Matchlocaldescriptors
Match local descriptors

Slidecredit:BastianLeibe
 Somepopulardetectors
Hessian/Harriscornerdetection
Laplacian ofGaussian(LOG)detector
DifferenceofGaussian(DOG)detector
Hessian/HarrisLaplacian detector
Hessian/ Harris Affine detector
Hessian/HarrisAffinedetector
MaximallyStableExtremal Regions(MSER)
M
Manyothers.
th
Looksforchangeinimagegradientintwodirection CORNERS

Nochangein Changeinone Changeinboth

Slidecredit:
FeiFeiLi anydirection directiononly thedirections
 HessianCornerDetector
[Beaudet,1978]

Searchesforimagelocationswhichhavestrongchangein
Searches for image locations which have strong change in
gradientalongboththeorthogonaldirection.
I xx ( x, ) I xy ( x, )
H( x, )
I xy ( x, ) I yy ( x, )

d t( H ) I xxI yy I 2xy
det(
Performanon
Perform a nonmaximum
maximumsuppressionusinga3
suppression using a 3*3 3window.
window
Considerpointshavinghighervaluethanits8neighbors.

Select points where det( H )

 Hessian Detector Result

Effect: Responsesmainlyoncornersandstronglytexturedareas.
Responses mainly on corners and strongly textured areas.
 HarrisCorner
[Forstner andGulch,1987]
Searchforlocalneighborhoodswheretheimagecontenthas
Search for local neighborhoods where the image content has
twomaindirections(eigenvectors).
Consider2
Consider 2nd momentautocorrelationmatrix
moment autocorrelation matrix
~ ~ I x ( x, )
2
I x I y ( x , )
C( x, , ) G ( x, ) ~ 2
x y
I I ( x , ) I 2
y ( x , )

Gaussiansumsoverallthepixelsincircularlocal
G i ll th i l i i l l l
neighborhoodusingweightsaccordingly.
I 2x I I
1 1
0
C R
x y

I x I y I 2

2
R
y
0
If1 or2 isabout0,the
SSymmetric
ti
Matrix pointisnotacorner.
 Harriscorner
Eigendecomposition:visualization

Slidecredit:K.Grauman,B.Leibe
 HarrisCorner:Differentapproach
Insteadofexplicitlycomputingtheeigenvalues,the
stead o e p c t y co put g t e e ge a ues, t e
followingequivalenceareused
d t(C) 1 2
det(
trace(C) 1 2

1 trace 2
( C ) ( ) 2
( r ) 2
( r 1) 2
If, r ( 1),
If ) 1 2
2 2

2 det(C) 1 2 r 2
2
r
Consider
Consider
det(C ) .trace 2 (C ) threshold these
points
intherange0.04 0.25,experimentallyverified
 det(C) 1 2 trace(C) 1 2
1 trace 2 (C ) (1 2 ) 2 (r2 2 ) 2 (r 1) 2
r ( 1), r 2 (1 / r )
2 det((C ) 12 r2
2
r
Let, r = 2; trc dc * (4.5)
2

H c det(C ) .trace (C ) dc(1 4.5 * );

2

If H c 0; 1 / 4.5( 0.222);
say 0.1 (risky with 0.2, why ?)
For Edge,
g , r >>1,, say
y5 For Corners, r = 2

H c dc(1 7.2 * 0.1); H c dc

d (1 4.5 * 0.1);
)
0.3 * dc; dc * 0.55
 HarrisCorner:Example
Ix Iy
1.Image
derivatives

Ix2 Iy2 IxIy

2.Squareof
q
derivatives

3.Gaussian
filterG(I)
(Ix2)
g(I (Iy2)
g(I g(IxIy)

4.Cornerness
function both
eigenvaluesare
l
strong
Slidecredit:K.Grauman,B.Leibe
CORNERNESS HARRIS CORNER

= .04 = .08 = .1

= .14 = .17 = .2 = .25

HarrisCorner:Result

Effect: Averyprecisecornerdetector.
A very precise corner detector.
 ScaleInvariantregiondetection
HessianandHarriscornerdetectorsarenotscaleinvariant.

Solution:
Usethe
p
conceptof
ScaleSpace
 Laplacian
p ofGaussian(LOG)
detector[Lindeberg,1998]
Lindeberg,1998]

UsingtheconceptofScaleSpace.
Insteadoftakingzerocrossing(foredgedetection),consider
thepointwhichismaximumamongits26neighbors(9+9+8).

L( x, ) 2 ( I xx ( x, ) I yy ( x, ))

LOGcanbeusedforfindingthecharacteristic
scale foragivenimagelocation.
for a given image location
LOGcanbeusedforfindingscaleinvariant
regions bysearching3D(location+scale)extrema
by searching 3D (location + scale) extrema
oftheLOG.
LOGisalsousedforedgedetection.
LOG i l df d d t ti
LOGdetector:Flowchart

I xx ( ) I yy ( )

Listof
(x,y,s)

LOGdetector:Result
 DifferenceofGaussian(DOG)
Detector[Lowe,2004]
ApproximateLOGusingDOGforcomputationalefficiency

D(x, )
(G(x, k ) G(x, ))* I(x)
k=21/K
K=0,1,2,,constant
Considertheregionwherethe
DOG response is greater than
DOGresponseisgreaterthan
athresholdandthescalelies
in a predefined range s min , s max
inapredefinedrange
 DOGdetector:Flowchart

Xisselected
if it is larger
ifitislarger
orsmaller
thanall
neighbors.
DOGdetector:Result
 LocalDescriptors
p
Wehavedetectedtheinterestpointsinanimage.
We have detected the interest points in an image
Howtomatchthepointsacrossdifferentimagesof
thesameobject?

Use Local Descriptors

UseLocalDescriptors
Slidecredit:FeiFeiLi
 Listoflocalfeaturedescriptors
p
ScaleInvariantFeatureTransform(SIFT)
p p (
SpeedUpRobustFeature(SURF) )
HistogramofOrientedGradient(HOG)
Gradient Location Orientation Histogram (GLOH)
GradientLocationOrientationHistogram(GLOH)
PCASIFT
PyramidalHOG(PHOG)
Pyramidal Histogram Of visual Words (PHOW)
PyramidalHistogramOfvisualWords(PHOW)
Others.
Shouldberobusttoviewpointchangeor
g
illuminationchange
 SIFT[Lowe,2004]

Step1: DetectinterestingpointsusingDOG.

832DOGextrema
 SIFT:Step2
p
Step2: Accuratekeypoint localization
Aim:rejectthelowcontrastpointsandthepoints
thatlieontheedge.
g
Lowcontrastpointselimination:
x
Fitkeypoint attonearbydatausingquadraticapproximation.
D 1 T D
T 2 T
D(x) D x x x
x 2 x 2

Calculatethelocalmaximaofthefittedfunction.
l l h l l i f h fi d f i
1
D 2
D
x
x2 x
Discardlocalminima D( x ) 0.03
 SIFT:Step2
p
Eliminatingedgeresponse:
g g p
DOGgivesstrongresponsealongedges Eliminatethoseresponses
Solution: check cornerness
Solution:check cornerness ofeachkeypoint.
of each keypoint
Ontheedgeoneofprinciplecurvaturesismuchbiggerthan
another.
another
Highcornerness Nodominantprinciplecurvaturecomponent.
ConsidertheconceptofHessianandHarriscorner
Consider the concept of Hessian and Harris corner
Hessian I xx I xy
H
I yy
Matrix
I xy

Harris Tr( H ) 2 (r 1) 2 Discardpointswith

response below threshold
responsebelowthreshold
corner
corner Det( H ) r
criterion
 SIFT:Step2
p

729 out of 832 are left after contrast thresholding

536 out of 729 are left after cornerness thresholding

Slidecredit:DavidLowe
 SIFT:Step3
p
Step3: OrientationAssignment
Aim:Assignconstantorientationtoeachkeypoint based
onlocalimagepropertytoobtainrotationalinvariance.
g p p y

Totransform
To transform
relativedata
accordingl
accordingly

Themagnitudeandorientationofgradientofanimagepatch
Th i d d i i f di f i h
I(x,y)ataparticularscaleis:
m( x, y ) (I( x 1, y ) I( x 1, y )) (I( x, y 1) I( x, y 1))
2 2

I( x, y 1) I( x, y 1)
( x, y ) tan1

I( x 1, y ) I( x 1, y )
 SIFT:Step3
p
Step3: OrientationAssignment

Create
Createweighted(magnitude+
weighted (magnitude +
Gaussian)histogramoflocal
g ad e t d ect o s co puted at
gradientdirectionscomputedat
selectedscale

Assigndominantorientationof
the region as that of the peak of
theregionasthatofthepeakof
smoothedhistogram

Forlocationofmultiplepeaks
multiply key point
multiplykeypoint
Slidecredit:DavidLowe
 SIFT:Step4
p
Alreadyobtainedpreciselocation,scaleandorientationtoeachkeypoin
Step4: Localimagedescriptor
Aim Obtainlocaldescriptorthatishighlydistinctiveyet
Aim Obtain local descriptor that is highly distinctive yet
invarianttovariationlikeilluminationandaffinechange
Considerarectangulargrid16*16inthedirectionofthe
Consider a rectangular grid 16*16 in the direction of the
dominantorientationoftheregion.
Dividetheregioninto4*4subregions.
Di id th i i t 4*4 b i
ConsideraGaussianfilterabovetheregion
whichgiveshigherweightstopixelcloser
tothecenterofthedescriptor.
p
 SIFT:Step4
p
Step4: Localimagedescriptor
Create8bingradienthistogramsforeachsubregion
b d h f h b
WeightedbymagnitudeandGaussianwindow( ishalfthe
windowsize)
i d i )

Featurevector(128)
( )
8*4*4=128
Finally normalize 128 dim vector to make it illumination invariant
Finally,normalize128dimvectortomakeitilluminationinvariant
 SIFT:SomeResult
Objectdetection
j
 SIFT:SomeResult
Panorama
 GLOH
First3steps sameasSIFT
Step 4 Localimagedescriptor
Step4 Local image descriptor
Considerlogpolarlocationgridwith3differentradiiand8
angular direction for two of them in total 17 location bin
angulardirectionfortwoofthem,intotal17locationbin
Formhistogramofgradientshaving16bins
Formafeaturevectorof272dimension(17*16)
F f t t f 272 di i (17*16)
Performdimensionalityreductionandprojectthefeatures
toa128dimensionalspace.

192
192correct
matches
( ll ) d
(yellow)and
208false
matches
matches
(blue).
 Someotherexamples
p

SURF
PHOW

HOG
 Reference
1. KristenGraumanandBastianLeibe,VisualObjectRecognition,SynthesisLectures
onArtificialIntelligenceandMachineLearning,
g g, April2011,Vol.5,No.2,Pages1
p , , , g
181.
2. Beaudet,Rotationallyinvariantimageoperators,inInternationalJoint
ConferenceonPatternRecognition,pp.579583.,1978.
3. Frstner,W.andGlch,E.,Afastoperatorfordetectionandpreciselocationof
distinct points corners and centers of circular features,inISPRSIntercommission
distinctpoints,cornersandcentersofcircularfeatures in ISPRS Inter commission
Workshop,pp.281305,1987.
4. Harris, C. and Stephens, M., A
Harris,C.andStephens,M., Acombinedcornerandedgedetector
combined corner and edge detector,,in in Alvey
Alvey
VisionConference,pp.147151,1988.
5. Lindeberg,T.,Scalespacetheory:Abasictoolforanalyzingstructuresatdifferent
scales,JournalofAppliedStatistics21(2),pp.224270,1994.
6. Lowe,D.,Distinctiveimagefeaturesfromscaleinvariantkeypoints,International
J
JournalofComputerVision60(2),pp.91110,2004.
l fC t Vi i 60(2) 91 110 2004
7. Mikolajczyk,K.andSchmid,C.,Aperformanceevaluationoflocaldescriptors,
IEEE Transactions on Pattern Analysis & Machine Intelligence 27(10) 3137
IEEETransactionsonPatternAnalysis&MachineIntelligence27(10),31 37,2005.
2005
THANKYOU
THANK YOU