Sunteți pe pagina 1din 127

X

u img f x s p x r11 r12 r13 t1


Y
z v img = fx p y r21 r22 r23 t2
Z
1 1 r31 r32 r33 t 3
1
x K R 33 t X
http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
First person coordinate system World coordinate system

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
( R SO(3), t 3
)
First person coordinate system World coordinate system

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
( R SO(3), t 3
)
How to recover?

First person coordinate system World coordinate system

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
( R SO(3), t 3
)
How to recover?

First person coordinate system World coordinate system

Case 1: Using a single vanishing point

Towards vanishing point

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
Z direction in the world coordinate system
Z direction in the world coordinate system

u
v
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point

v
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point

v Columns of the rotation matrix represent


vanishing points of world axes.

z v z = K [r1 r2 r3 t ] z
z vanishing point z point at infinity

R = [r1 r2 r3 ] SO(3)
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point

v Columns of the rotation matrix represent


vanishing points of world axes.
0
0
z v z = K [r1 r2 r3 t]
1

0
R = [r1 r2 r3 ] SO(3)
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point

v Columns of the rotation matrix represent


vanishing points of world axes.

z v z = Kr3
r3 = K -1v z / K -1v z
R = [r1 r2 r3 ] SO(3)
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point
Geometric interpretation
v World z-axis
z-axis of world coordinate system


Camera axes
R = [r1 r2 r3 ] SO(3)
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point
Geometric interpretation
v World z-axis
z-axis of world coordinate system
K -1v z
r3 = -1
K vz

sin sin
Camera axes = cos
R = [r1 r2 r3 ] SO(3) cos sin
z = [0 0 1 0]
T

z point at infinity

Z direction in the world coordinate system

u v z : z vanishing point
Geometric interpretation
v World z-axis
z-axis of world coordinate system

= tan-1 ( r3 (1) / r3 (3) )
= cos-1r3 (2)
Camera axes Pan and tilt angles
R = [r1 r2 r3 ] SO(3)
( R SO(3), t 3
)
How to recover?

First person coordinate system World coordinate system

Case 2: Using two vanishing points

Towards vanishing point

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
z = [0 0 1 0]
T

z point at infinity

x = [1 0 0 0]
T
x point at infinity

X direction in the world coordinate system

Z direction in the world coordinate system

u v z : z vanishing point
Columns of the rotation matrix represent
v vanishing points of world axes.
r3 = K -1z v z
v x : x vanishing point r1 = K -1z v x
r2 = r3 r1
Orthogonal rotation matrix

R = [r1 r2 r3 ] SO(3)
Calibration via Vanishing Points
Calibration via Vanishing Points
1 v1
K v1

1
v1 v3
K v3 v3

r3 = K -1z v z
r1 = K -1z v x
r2 = r3 r1
R = [r1 r2 r3 ]

v2
Exercise I

vy

vx

r1 = K -1z v x r2 = K -1z v y

R = [r1 r2 r1 r2 ]
Exercise II
vx

vy
Exercise II
vx

r1 = K -1v x / K -1v x
-1 -1 vy
r2 = K v y / K v y
Scale normalization
Exercise II
vx

vy
Exercise II

Estimate pan/tilt from r3 .

= tan-1 ( r3 (1) / r3 (3) )


= sin-1r3 (2)
Exercise II

Pan angle
Tilt angle
( R SO(3), t 3
)
How to recover?

First person coordinate system World coordinate system

Case 3: Homography

http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-to-find-their-vanishing-
point.html
Planar world

X = [X Y 0 1]

u
v z m = K [r1 r2 r3 t] X
m = [u v 1]
T
X
Y X
= K [r1 r2 r3 t ] = K [r1 r2 t ] Y
0
1
1

P = [R t ]
2D homography
Planar world

X = [X Y 0 1]

u
v X
m = [u v 1] z m = H Y where H = K [r1 r2 t]
T

P = [R t ]
Exercise
Homography from four points:

H
H K=
-1
[r1 r2 t ] Note that r1 = r2 = 1
Exercise
Homography from four points:

: Normalization factor
Exercise
Homography from four points:

: Normalization factor
How to estimate the rotation and
translation of the robot from the
world point of view?
In the case of moving robot(rather than moving target), we
need to know the orientation/position of the robot in the world
==>
we need to how to pan/tilt the world oriented to the robot.

Note: pan/tilt of the camera is very different from the pan/tilt


of the world!
Third person (world) perspective
XW

{W }
World coordinate system

{W } XW

{C }
P = K [R t ]
Camera coordinate system
First person perspective
XC'

{C' }
Camera coordinate at
world origin

{W } {C' }
XW XC
R
{C }
P = K [R t ]
Camera coordinate system
Coordinate transform from {W } to {C }

XC' = RXW
First person perspective
XC

{C' }
Camera coordinate at
world origin

t {W } {C' }
XW XC
R
{C }
P = K [R t ]
Camera coordinate system
Camera translation seen from {C }
First person perspective
XC

{C' }
Camera coordinate at
world origin

t {W } {C' }
XW XC
R
{C }
P = K [R t ]
Camera coordinate system
Camera translation seen from {C }

=
XC RXW + t
Looking a point in world through
the camera view point
First person perspective
XW

{W }
World coordinate system

t {C } {W }

{C } t
P = K [R t ]
Camera coordinate system
Third person (world) perspective
XW

{W }
World coordinate system

-t {C } {W }

{C } t
P = K [R t ]
Camera coordinate system
Third person (world) perspective
XW

{W }
World coordinate system

{C } {W' }
{W' } -1
C = -R t
R-1 = R T
t
C = -R-1t
World coordinate at camera center
P = K [R t ] = -R T t
= K [R -RC ]
= KR [I33 -C ]
Camera center seen from world coordinate system
Robot Perception:
Compute Projective Transformations

Kostas Daniilidis
A perspective projection of a plane
(like a camera image) is always a
projective transformation
Using the projective transformation the pose
of a robot with respect to a planar pattern:
How can we compute the projective
transformation between

and

Floor tiles measured in [m] Points in pixel coordinates


The result of such a transformation would map any point in one plane to
the corresponding point in the other

and

Floor tiles measured in [m] Points in pixel coordinates


How can we compute the projective
transformation between

B is the image of the A is the image of


intersection of vertical parallel the intersection
lines (0,1,0) of horizontal
parallel lines
(1,0,0)

C
How can we compute the projective
transformation between

B is the image of the A is the image of


intersection of vertical parallel the intersection
lines (0,1,0) of horizontal
parallel lines
(1,0,0)

C
How can we compute the projective
transformation between

B is the image of the A is the image of


intersection of vertical parallel the intersection
lines (0,1,0) of horizontal
parallel lines
(1,0,0)

C
Let us introduce a 4th point D

B is the image of the A is the image of


intersection of vertical parallel the intersection
lines (0,1,0) of horizontal
parallel lines
(1,0,0)

C
Knowledge of this projective transformation
makes Virtual Billboards possible!
Microsoft Office Lens App
Office Lens
What happens when the original set of points is not a square?
d a
a
d
c
c
b
c
b
Perception: Projective Transformations
and Vanishing Points
Kostas Daniilidis
Projective Transformation
Aka Homography or Collineation
Represents the perspective projection from a ground plane to an
image plane !
It is an invertible 3x3 matrix but has 8 independent parameters
For example if (X,Y) measured in meters on the ground and (u,v) in
pixels

or
If we know H, we can make back and forth
measurements:
10.2,2.3 in meters

600,644 in pixels
Homography columns as vanishing points

B A

Y X
B A

Y X

So the first two columns are two Orthogonal vanishing points


If we connect two vanishing points we obtain the horizon!

horizon
B A

Y X
Equation of horizon:

horizon
B A

Y X
Horizon with projection center build a horizon plane with normal

horizon
B A

Y X

Projection center
The horizon plane is parallel to the ground plane and hence
is the normal to the ground plane expressed via pixels!

horizon
B A

Y X

Projection center
Horizon gives complete info about how ground plane is
oriented! If horizon is horizontal the center it means that
optical axis is parallel to groundplane!
Horizon gives complete info about how ground plane is
oriented! If horizon is horizontal the center it means that
optical axis is parallel to ground-plane!
Image Plane
Horizon Horizon Plane

Ground-plane
If horizon moves to the bottom it means we look upwards!
If horizon moves to the bottom it means we look upwards!

Image Plane
Horizon
Horizon Plane

Ground-plane
If horizon moves to the top it means we look downwards !
If horizon moves to the bottom it means we look upwards!

Image Plane
Horizon Horizon Plane

Ground-plane
Horizon tells us how camera is oriented.
Constrains the collineation !
horizon
B A
Perception: Projective Transformations
and Vanishing Points
Kostas Daniilidis
Projective Transformation
Aka Homography or Collineation
Represents the perspective projection from a ground plane to an
image plane !
It is an invertible 3x3 matrix but has 8 independent parameters
For example if (X,Y) measured in meters on the ground and (u,v) in
pixels

or
If we know H, we can make back and forth
measurements:
10.2,2.3 in meters

600,644 in pixels
Homography columns as vanishing points

B A

Y X
B A

Y X

So the first two columns are two Orthogonal vanishing points


If we connect two vanishing points we obtain the horizon!

horizon
B A

Y X
Equation of horizon:

horizon
B A

Y X
Horizon with projection center build a horizon plane with normal

horizon
B A

Y X

Projection center
The horizon plane is parallel to the ground plane and hence
is the normal to the ground plane expressed via pixels!

horizon
B A

Y X

Projection center
Horizon gives complete info about how ground plane is
oriented! If horizon is horizontal the center it means that
optical axis is parallel to groundplane!
Horizon gives complete info about how ground plane is
oriented! If horizon is horizontal the center it means that
optical axis is parallel to ground-plane!
Image Plane
Horizon Horizon Plane

Ground-plane
If horizon moves to the bottom it means we look upwards!
If horizon moves to the bottom it means we look upwards!

Image Plane
Horizon
Horizon Plane

Ground-plane
If horizon moves to the top it means we look downwards !
If horizon moves to the bottom it means we look upwards!

Image Plane
Horizon Horizon Plane

Ground-plane
Horizon tells us how camera is oriented.
Constrains the collineation !
horizon
B A
Two view metrology
Perception: Kostas Daniilidis
We have learnt that the transformation from a
world plane (like the road plane) to the image is a
projective transformation !
What about two views of the same plane (faade)?
Projective transformation between two
images of the same plane!
1966 World Cup: England-Germany
Was it a goal?
Can we infer from
two different
viewpoints
whether the ball
was inside the
goal ?
Today, this is done
with what is called
Goal Line
Technology!
Let B be the ball ! The question is whether the vertical
projection of the ball P is behind the goal-line or not!
Let us also assume that we can find the vertical
vanishing points v and v in both images !
Vertical vanishing points
Then we know in both images the projections of
vertical lines l and l.
If we project them back on the soccer field we will
find the shadows of the vertical through the ball.
If we project them back on the soccer field we will
find the shadows of the vertical through the ball.
Shadows intersect at the desired point P !
How can we find the projection of P in one of the
image planes?
By computing the projections of the shadows!
The red lines in the right image plane!
By computing the projections of the shadows l and ls,
the red lines in the right image plane!

ls
But we only know l. But ls and l (left) are the projections
from the same shadow line on the soccer field.

ls
Because the soccer field is planar, they are related by a
homography (collineation).

ls
Homography of soccer field can be computed
because we see the same features:
If H is the homography between points, then

ls
And the projection p is
intersection of two lines

ls
So, was it a goal ?
No, the ball did not cross the line in full.
A lot more engineering behind geometry:
Synchronizing frames cross cameras
Size of the ball
Motion blure
England Germany 2010 !
Today, multiple cameras with very accurate
relative pose tracked! The Hawk-Eye.
Thanks to Reid and Zisserman!
http://www.learnopencv.com/how-computer-vision-solved-the-greatest-soccer-mystery-of-all-times/