0 Voturi pozitive0 Voturi negative

7 (de) vizualizări8 paginiNov 27, 2010

© Attribution Non-Commercial (BY-NC)

PDF, TXT sau citiți online pe Scribd

Attribution Non-Commercial (BY-NC)

7 (de) vizualizări

Attribution Non-Commercial (BY-NC)

- The Law of Explosive Growth: Lesson 20 from The 21 Irrefutable Laws of Leadership
- Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race
- Hidden Figures Young Readers' Edition
- The E-Myth Revisited: Why Most Small Businesses Don't Work and
- Micro: A Novel
- The Wright Brothers
- The Other Einstein: A Novel
- State of Fear
- State of Fear
- The Power of Discipline: 7 Ways it Can Change Your Life
- The Kiss Quotient: A Novel
- The 10X Rule: The Only Difference Between Success and Failure
- Being Wrong: Adventures in the Margin of Error
- Algorithms to Live By: The Computer Science of Human Decisions
- The 6th Extinction
- The Black Swan
- The Art of Thinking Clearly
- The Last Battle
- Prince Caspian
- A Mind for Numbers: How to Excel at Math and Science Even If You Flunked Algebra

Sunteți pe pagina 1din 8

MIT Computer Science and Artificial Intelligence Laboratory

{sand,teller}@csail.mit.edu

of applications. Multiple observations of each scene point

This paper describes a new approach to motion estima- can be combined for super-resolution, noise removal, seg-

tion in video. We represent video motion using a set of par- mentation, and increased effective dynamic range. The cor-

ticles. Each particle is an image point sample with a long- respondences can also improve the temporal coherence of

duration trajectory and other properties. To optimize these image filters that operate independently on each frame. Ad-

particles, we measure point-based matching along the par- ditionally, long-range motion estimation can simplify inter-

ticle trajectories and distortion between the particles. The active video manipulation, including matting, rotoscoping,

resulting motion representation is useful for a variety of ap- and object removal.

plications and cannot be directly obtained using existing

methods such as optical flow or feature tracking. We demon- 1.1. Particle Video Representation

strate the algorithm on challenging real-world videos that

Our approach represents video motion using a set of par-

include complex scene geometry, multiple types of occlu-

ticles that move through time. Each particle denotes an in-

sion, regions with low texture, and non-rigid deformations.

terpolated image point sample, in contrast to a feature patch

that represents a neighborhood of pixels [12]. Particle den-

sity is adaptive, so that the algorithm can model detailed

motion with substantially fewer particles than pixels.

1. Introduction The algorithm optimizes particle trajectories using an

Video motion estimation is often performed using fea- objective function that incorporates point-based image

ture tracking [12] or optical flow [3]. Feature tracking fol- matching, inter-particle distortion, and frame-to-frame op-

lows a sparse set of salient image points over many frames, tical flow. The algorithm also extends and truncates particle

whereas optical flow estimates a dense motion field from trajectories to model motion near occlusion boundaries.

one frame to the next. Our goal is to combine these two Our contributions include posing the particle video prob-

approaches: to produce motion estimates that are both spa- lem, defining the particle video representation, and present-

tially dense and temporally long-range (Figure 1). For any ing an algorithm for building particle videos. We provide a

image point, we would like to know where the correspond- new motion optimization scheme that combines variational

ing scene point appears in all other video frames (until the techniques with an adaptive motion representation. The al-

point leaves the field of view or becomes occluded). gorithm uses weighted links between particles to implic-

itly represent grouping, providing an alternative to discrete

x x x layer-based representations.

Our primary goal is the ability to model complex mo-

tion and occlusion. We want the algorithm to handle gen-

t t t

Feature Tracking Optical Flow Particle Video eral video, which may include close-ups of people talking,

Figure 1. Each diagram represents point correspondences between hand-held camera motion, multiple independently moving

frames of a hypothetical sequence. Feature tracking is long-range objects, textureless regions, narrow fields of view, and com-

but sparse. Optical flow is dense but short-range. Our particle plicated geometry (e.g. trees or office clutter).

video representation is dense and long-range. A particle approach provides this kind of flexibility. Par-

1

ticles can represent complicated geometry and motion be- Other long-range optical flow algorithms do not assume

cause they are small; a particle’s appearance will not change temporal smoothness. Wills and Belongie [17] find dense

as rapidly as the appearance of a large feature patch, and it correspondences (between widely-separated image pairs)

is less likely to straddle an occlusion boundary. Particles using a layered representation initialized with sparse feature

represent motion in a non-parametric manner; they do not correspondences. Irani [8] describes linear subspace con-

assume that the scene consists of planar or rigid compo- straints for flow across multiple frames. Brand [5] applies

nents. a similar approach to non-rigid scenes. These rank-based

Any flexible system needs to be augmented with con- methods assume less temporal smoothness than other meth-

straints, which motivates another design decision: consis- ods, but are not designed to handle large-scale occlusions.

tency is more important than correctness. If the scene in- The primary difference between our algorithm and exist-

cludes arbitrary deforming objects with inadequate texture, ing optical flow algorithms is that we represent motion in

finding the true motion may be hopeless. Typically, this a scene-centric fashion, rather than as a vector field from

problem is addressed with geometric assumptions about each frame to the next. Simple concatenation of frame-to-

scene rigidity and camera motion. Instead, we simply frame flow fields will cause an accumulation of error. To

strive for consistency; for example, that red pixels from solve this, we refine motion estimates in order to enforce

one frame are matched with red pixels in another frame. long-range consistency.

For many applications, this kind of consistency is sufficient

(and certainly more useful than outright failure due to non- 2. Variational Optical Flow with Occlusion

uniqueness).

This flexibility in modelling complex motion can also be Our particle video algorithm uses frame-to-frame opti-

achieved by optical flow, but the optical flow representation cal flow to help guide the particles. The algorithm treats

is best suited to successive pairs of frames, not to long se- flow estimation as a black box that can be replaced with

quences. Frame-to-frame flow fields can be concatenated an alternate flow algorithm. Rather than assuming tempo-

to obtain longer-range correspondences, but the resulting ral smoothness, we estimate optical flow independently for

multi-frame flow must be refined at each step to avoid drift. each frame; this enables the algorithm to perform well on

hand-held video with moving objects.

1.3. Related Work We use a combination of the optical flow estimation

methods of Brox et al. [6] and Xiao et al. [18]. We estimate

Much of our work focuses on handling occlusion. Oc- flow over a sequence of resolutions obtained by recursively

clusion modelling is a difficult part of optical flow, stereo, reducing the original resolution by a factor η. A standard

tracking, and motion estimation in general. Thompson [15] image pyramid uses η = 0.5 whereas we (following Brox

describes issues caused by occlusion, including mixed pix- et al. [6]) use a larger factor (η = 0.9) to obtain better re-

els and systematic boundary localization errors. Zitnick sults at a cost of increased computation.

et al. [19] estimate optical flow using correspondences At each resolution level, we run a variational flow up-

between segmented image regions, explicitly modelling date (Section 2.1) then alternate 4 times between estimat-

mixed pixels at occlusion boundaries. Amiaz and Kiry- ing occluded regions (Section 2.2) and improving occlusion

ati [2] use level sets to refine variational motion boundaries. boundaries using a bilateral filter (Section 2.3).

Because occluded pixels violate a major assumption

of optical flow (that each pixel goes somewhere), several 2.1. Variational Flow Refinement

methods attempt to identify occluded pixels explicitly. Silva

and Victor [13] use a pixel dissimilarity measure to detect Our variational update scheme is based on the Brox et

brightness values that appear or disappear over time. Al- al. [6] algorithm. In our implementation, we replace the

varez et al. [1] simultaneously compute forward and reverse scalar-valued image I with a multi-channel image I [k] . We

flow fields, labelling pixels as occluded where the two dis- also modulate the data term by a visibility term r(x, y, t)

agree. Strecha et al. [14] treat occlusion labels as hidden (described in Section 2.2):

variables in an EM optimization. Xiao et al. [18] regular-

ize flow estimates using a bilateral filter that incorporates EF lowData (ū, v̄, t) =

flow from neighboring pixels that are similar in motion and

X

r(x, y, t)Ψ([I [k] (x+ ū, y+v̄, t+1)−I [k] (x, y, t)]2 ).

appearance and that lie outside occluded regions. x,y,k

Many algorithms propagate optical flow estimates from (1)

one frame to the next using a temporal smoothness assump-

tion [4, 7, 11]. Brox et al. [6, 10] estimate optical flow si- Here ū and v̄ denote the flow field (evaluated at (x, y, t))

multaneously over multiple frames using an objective func- and k is summed over image channels. We use the same

tion that provides robust spatiotemporal regularization. robust norm as Brox et al. [6]:

p 2.3. Bilateral Flow Filtering

Ψ(s2 ) = s2 + 2 ; = 0.001. (2)

To improve boundary sharpness, we use a bilateral filter

The original Brox et al. [6] formulation analytically en- based on the work of Xiao et al. [18]. The filter sets each

forces constancy of the image gradient (and optionally other flow vector to a weighted average of neighboring flow vec-

linear differential operators [10]), whereas we simply treat tors:

the gradient as another image channel. Specifically, we use P

x1 ,y1 ū(x1 , y1 )w(x, y, x1 , y1 )

image brightness I (range [0, 255]), the x and y derivatives 0

ū (x, y) = P . (8)

of brightness (Ix and Iy ), the green minus red color compo- x1 ,y1 w(x, y, x1 , y1 )

nent, and the green minus blue color component. We scale

The update for v̄ is analogous. The algorithm weights the

the color difference channels by 0.25 to reduce the impact

neighbors according to spatial proximity, image similarity,

of color sampling artifacts common in video.

motion similarity, and occlusion labelling:

We modify the Brox et al. [6] smoothness term using a

smoothness factor αl modulated by the image gradient:

p

w(x, y, x1 , y1 ) = N ( (x − x1 )2 + (y − y1 )2 ; σx )

q · N (I(x, y) − I(x1 , y1 ); σi )

b(x, y, t) = N ( Ix (x, y, t)2 + Iy (x, y, t)2 ; σb ), p

· N ( (ū − ū1 )2 + (v̄ − v̄1 )2 ; σm )

EF lowSmooth (ū, v̄, t) =

X · r(x1 , y1 )). (9)

(αg +αl · b(x, y, t)) · Ψ(ū2x + ū2y +v̄x2 +v̄y2 ). (3)

x,y Here ū denotes ū(x, y) and ū1 denotes ū(x1 , y1 ) (and v̄ sim-

ilarly). We set σx = 4, σi = 20, and σm = 1, and restrict

Here N denotes a zero-mean non-normalized Gaussian. We (x1 , y1 ) to lie within 10 pixels of (x, y).

set σb = 1.5, the local smoothness αl = 10, and the global For efficiency, we apply the filter only near flow bound-

smoothness αg = 5 (based on a variety of flow experi- aries, which we localize using the flow gradient magnitude:

ments). We optimize the functional as described in Brox

et al. [6], using 100 linear solver iterations inside 3 fixed-

q

g(x, y, t) = ū2x + ū2y + v̄x2 + v̄y2 . (10)

point iterations at a given resolution level.

The algorithm filters g(x, y, t) using a spatial Gaussian ker-

2.2. Occlusion Labelling nel (σg = 3), producing a smoothed gradient magnitude

Optical flow divergence distinguishes between different ĝ(x, y, t). We apply the bilateral filter (Equation 8) to pix-

types of motion boundaries: els with ĝ(x, y, t) > 0.05.

div(x, y, t) =

∂

ū(x, y, t) +

∂

v̄(x, y, t). (4) 3. Particle Video Overview

∂x ∂y

A particle video is a video and corresponding set of par-

The divergence of a flow field is positive for disocclud- ticles. Particle i has a time-varying position (xi (t), yi (t))

ing boundaries, negative for occluding boundaries, and near that is defined between the particle’s start and end frames.

zero for shear boundaries. To select occluding boundaries, We build a particle video by sweeping forward, then

but not disoccluding boundaries, we define a one-sided di- backward, across a video. For each frame, the following

vergence function d: steps are performed (Figure 2):

div(x, y, t) div(x, y, t) < 0

d(x, y, t) = (5)

0 otherwise. frame are extended into the current frame according

to the forward and reverse flow fields (Section 3.1).

Pixel projection difference provides another occlusion cue:

• Linking. Particle links are updated (Section 3.2).

e(x, y, t) = I(x, y, t)−I(x+ ū(x, y, t), y+v̄(x, y, t), t+1).

• Optimization. Particle positions are optimized (Sec-

(6)

tion 4).

We combine the modified divergence and pixel projection

using zero-mean Gaussian priors: • Pruning. Particles with high post-optimization error

are pruned (Section 4.3).

r(x, y, t) = N (d(x, y, t); σd ) · N (e(x, y, t); σe ). (7)

• Addition. New particles are added in gaps between

We set σd = 0.3 and σe = 20 based on experimental obser- existing particles (Section 5).

vation of occluded regions.

Links

(Section 3.1) (Section 3.2) (Section 4) (Section 4.3) (Section 5)

Figure 2. Each plot denotes a pair of adjacent frames. The al-

gorithm propagates particles from one frame to the next accord-

ing to the flow field, excluding particles (blue) that lie within the

flow field’s occluded region. The algorithm then adds links (red

curves), optimizes all particle positions, and prunes particles with

Optical Flow

high error after optimization. Finally, the algorithm inserts new

particles (yellow) in gaps between existing particles.

When propagating particles to a given frame, all parti-

cles defined in adjacent frames, but not defined in the given

frame, are placed in the frame according to the flow fields

Link Weights

between the frames. To propagate particle i from frame t−1

Figure 3. The algorithm uses an optical flow field (Section 2) to

to t, we use the flow field ū(x, y, t − 1), v̄(x, y, t − 1):

compute link weights lij (t) (darker values are lower). The left side

xi (t) = xi (t−1) + u(xi (t−1), yi (t−1), t−1), shows an entire frame. The right side shows a magnified portion

of the frame.

yi (t) = yi (t−1) + v(xi (t−1), yi (t−1), t−1). (11)

ogously. (When making the first forward pass through the 3.3. Particle Channels and Weights

video, there are no particles to propagate backward.)

By design, each particle is intended to be an image point

3.2. Particle Linking sample. A particle should have a limited spatial extent so

that it does not cross an occlusion boundary. However, if the

To quantify relative particle motion, our algorithm cre-

particle is not near an occlusion boundary, we may obtain

ates links between particles using a constrained Delaunay

better particle positioning by using a larger spatial extent.

triangulation [9] (Figure 3). For any given frame, we create

a particle link if the corresponding triangulation link exists We repeat the 5 channels used for flow estimation (Sec-

for the frame or an adjacent frame. Using links from adja- tion 2.1) at three different scales (via a Gaussian filter with

cent frames reduces temporal linking variability, while still σ = {1, 2, 4}), for a total of 15 channels. We assign par-

[k]

allowing links to appear and disappear as particles pass by ticle i a 15-dimensional appearance vector {ci } from the

one another. 15-channel frame {I } in which it is added.

[k]

For each link (i, j), we compute a squared motion differ- The algorithm weights each particle according to the fil-

ence according to the flow vectors: tered flow gradient magnitude (Section 2.3):

D(i, j, t) = (ū(xi (t), yi (t), t) − ū(xj (t), yj (t), t))2

+ (v̄(xi (t), yi (t), t) − v̄(xj (t), yj (t), t))2 . pi (t) = N (ĝ(xi (t), yi (t), t); σp2 ). (14)

(12)

We assign a weight to the link using a Gaussian prior (σl = When ĝ(xi (t), yi (t), t) is more than one standard deviation

0.5) on the motion difference: (σp = 0.05) from zero, the particle is probably near a flow

boundary, so we exclude all but the first channel, because

the other channels may be influenced by pixels on the other

p

lij (t) = N ( D(i, j, t); σl2 ). (13)

side of the boundary.

4. Particle Trajectory Optimization 4.2. Optimization Algorithm

The algorithm repositions particles to locally minimize Our algorithm optimizes Equation 19 in a manner simi-

an objective function that includes three parts: a data term, lar to the Brox et al. [6] algorithm, using a pair of nested

a distortion term, and a flow-matching term. loops around a sparse linear system solver. The outer

loop iteratively updates xi (t) and yi (t) using increments

4.1. Particle Objective Function dxi (t), dyi (t) computed by the inner loop.

The data term measures how well a particle’s initial ap- Within the objective function E, we substitute dxi (t) +

pearance vector matches the current frame. For particle i at xi (t) for xi (t) (and instances of y accordingly). Taking par-

time t the data term for channel k is: tial derivatives, we obtain a system of equations, which the

algorithm solves for dxi (t) and dyi (t):

[k] [k]

EData (i, t) = Ψ([I [k] (xi (t), yi (t), t) − ci ]2 ). (15)

∂E ∂E

= 0, = 0 | i ∈ P, t ∈ F . (20)

The distortion term measures the relative motion of linked ∂dxi (t) ∂dyi (t)

particles. Let ui (t) = xi (t) − xi (t − 1) and vi (t) = yi (t) −

yi (t − 1). The larger the difference between these motion For the data term, we use the image linearization from Brox

values, the larger the distortion term. The term is modulated et al. [6]:

by the link’s weight lij (t) (Section 3.2), so that a link across

[k]

an occlusion boundary (i.e. a low-weight link) is allowed Iz[k] = Ix[k] dxi (t) + Iy[k] dyi (t) + I [k] − ci , (21)

greater distortion for an equivalent penalty: [k]

∂EData (i, t)

EDistort (i, j, t) = lij (t)Ψ([ui (t) − uj (t)] 2 ≈ 2Ψ0 ([Iz[k] ]2 )Iz[k] Ix[k] . (22)

∂dxi (t)

+[vi (t) − vj (t)]2 ). (16)

We omit the (xi (t), yi (t), t) indexing of I, Ix , Iy , and Iz .

The flow-matching term measures how well each parti- Ψ0 is the derivative of Ψ with respect to its argument s2 .

cle trajectory matches the frame-to-frame flow estimates Note that this linearization occurs inside the inner fixed-

ū(x, y, t), v̄(x, y, t) (Section 2). It can be viewed as regular- point loop; the algorithm is still optimizing the original non-

ization (constraining the data term) or as a secondary data linearized objective function.

term (where the flow is provided as data). We modulate this For the distortion term, the derivative is more straightfor-

term according to the particle’s weight pi (t) (Equation 14); ward. We use dui (t) as shorthand for dxi (t) − dxi (t − 1)

if the particle is close to an occlusion boundary, matching and dvi (t) for dyi (t) − dyi (t − 1):

the flow is less important:

∂EDistort (i, j, t)

=

EF low (i, t) = ∂dxi (t)

pi (t)Ψ([ū(xi (t−1), yi (t−1), t−1)−ui (t)]2 2lij (t)Ψ0 ([ui (t) + dui (t) − uj (t) − duj (t)]2

+[v̄(xi (t−1), yi (t−1), t−1)−vi (t)]2 ). (17) + [vi (t) + dvi (t) − vj (t) − dvj (t)]2 )

· (ui (t) + dui (t) − uj (t) − duj (t)). (23)

The combined energy of a particle at a given time is:

X [k] The flow matching term uses the same linearization as the

E(i, t) = EData (i, t)

data term:

k∈Ki (t)

X

+ αd EDistort (i, j, t) U = ūx dxi (t−1) + ūy dyi (t−1) + ū − (ui (t)+dui (t)),

j∈Li (t) V = v̄x dxi (t−1) + v̄y dyi (t−1) + v̄ − (vi (t)+dvi (t)),

+ αf EF low (i, t). (18)

∂EF low (i, t)

Here Ki (t) denotes the set of active channels (Sec- ≈ 2pi (t)Ψ0 (U 2 +V 2 )U. (24)

∂dxi (t)

tion 3.3), and Li (t) denotes the set of particles linked to

particle i in frame t. We find that αd = 4 and αf = 0.5 pro- Here we omit the (xi (t−1), yi (t−1), t−1) indexing of ū, v̄,

vide a reasonable tradeoff between the three terms. Given and their spatial derivatives. Note that dxi (t) also appears

a set P of particle indices and a set F of frame indices, the in derivatives of E(i, t + 1).

complete objective function is: Each of these partial derivatives is linear in dxi (t) and

X dyi (t) except the Ψ0 factors. For each iteration of the inner

E= E(i, t). (19) loop, we compute the Ψ0 terms, then hold them constant

i∈P,t∈F within the linear system solver:

Loop (4 Times)

[k] [k]

Compute Ix (i, t), Iy (i, t)

dxi (t), dyi (t) ← 0

Loop (4 Times)

Compute Ψ0 terms

Solve system to update dxi (t), dyi (t)

(200 SOR iterations)

Video Frame

End Loop

xi (t) ← xi (t) + dxi (t)

yi (t) ← yi (t) + dyi (t)

End Loop

After optimizing the particles, we prune particles that

continue to have high energy E(i, t). To reduce the im- Scale Map

pact of a single bad frame, we filter each particle’s energy

values using a Gaussian (σt = 1 frames). Particle i obtains

a sequence of filtered objective values Ê(i, t). (Note: this

Gaussian is not strictly temporal; it filters the values for the

given particle, which is moving through image space.)

If Ê(i, t) > 10, the particle is marked as incorrect in

frame t. Given this marking, the algorithm prunes the par-

Particles

ticle to the longest contiguous interval of correct frames. If

the remaining duration of the particle is less than 3 frames Figure 4. For each video frame, the algorithm computes a scale

(and the particle’s nearest endpoint is more than 3 frames map that determines the placement of new particles (Section 5).

The left side shows an entire frame. The right side shows a mag-

from the current frame), the particle is deleted.

nified portion of the frame.

5. Particle Addition

Given the scale map, we iterate over the image adding

After optimization and pruning, the algorithm adds new particles. For each pixel, if the distance to the nearest parti-

particles in gaps between existing particles. (The same pro- cle is greater than s(x, y), we add a particle at that pixel.

cess is used to create all particles in the first video frame.) The algorithm does this efficiently in time (linear in the

The algorithm arranges for higher particle density in regions number of particles) by creating an occupancy map at each

of greater visual complexity, in order to model complex mo- scale.

tions.

To add new particles to a given frame, the algorithm de-

6. Results

termines a scale value s(x, y) for each pixel. The scale val-

ues are discrete, taken from the set {σ(j) = 1.9j | 0 ≤ j ≤ We evaluate the algorithm using a set of 20 videos, which

5}. To compute the scale map, we start by filtering the im- together include a variety of scenes, camera motions, and

age using a Gaussian kernel for each scale σ(j), producing object motions. Figure 6 shows particle correspondences

a set of images {Ij }. for several of the videos. Complete videos and results are

Then, for each pixel, we find the range of scales over available at http://rvsn.csail.mit.edu/pv/.

which the blurred pixel value does not change substantially. On a typical 720 by 480 video frame, the algorithm

If the pixel has the same color in a large scale image as in all spends about 7 seconds optimizing particles, 0.2 seconds

smaller scale images, it is a large scale pixel. Specifically, linking particles, 0.05 seconds pruning particles, and 2 sec-

the algorithm chooses the maximum scale index k(x, y) onds adding particles (each of these figures is given per

such that ||Ij (x, y) − I1 (x, y)||2 < 10 for all j ≤ k(x, y). sweep). The flow algorithm (Section 2) requires about 100

(Here we use (r, g, b) vector distance.) seconds per frame pair. In our experiments, we use one for-

These scale indices are filtered with a spatial Gaussian ward sweep followed by one backward sweep.

(σs = 2), producing a blurred scale index map k̂(x, y) Objectively evaluating the correctness of the algorithm is

(which we round to integer values). We then set the scale difficult given the lack of ground-truth data. The ideal eval-

values from the indices: s(x, y) = σ(k̂(x, y)). Figure 4 uation measurement should allow comparison with future

provides an example scale map. particle video algorithms and with non-particle approaches

Video 1 Video 2 Video 3

35 50 25

Mean Particle Position Difference

0 0 0

1 Frame Index 50 1 Frame Index 50 1 Frame Index 50

Figure 5. Each plot shows the mean distance of surviving particles from their positions in the start frame. As described in Section 6, the

videos are temporally mirrored, so we expect all unoccluded particles to return to their start positions. The dotted lines denote concatenated

flow vectors. In each case, particles return closer to their starting positions than concatenated flow vectors. Both algorithms have some

trouble capturing the face pose change in Video 2.

to long-range motion estimation. corporate constraints from multiple frames (often using a

For our evaluation, we construct videos that return to the temporal smoothness assumption), but they do not enforce

starting configuration by replacing the second half of each long-range correspondence consistency. Our algorithm im-

video with a temporally reversed copy of the first half. For proves frame-to-frame optical flow by enforcing long-range

such a video, we expect particles from the start frame to appearance consistency and motion coherence.

return to their starting positions in the end frame. For each Our algorithm does have several limitations. Like most

particle that survives from the start frame to the end frame, motion algorithms, the approach sometimes fails near oc-

the algorithm computes the spatial distance (error) between clusions. Also, the approach has difficulty with large

the particle’s start position and end position (Figure 5). changes in appearance due to non-Lambertian reflectance

As a simplistic point of comparison, for each particle po- and major scale changes. (Our use of color and gradient

sition in the start frame, we concatenate flow vectors, com- channels provides some robustness to reflectance, but not

puted as described in Section 2. These flow trajectories are enough to handle all cases correctly.)

terminated when they enter an occluded region, as deter- In the future, we will focus on these issues. We plan

mined by the flow algorithm. to explore more sophisticated occlusion localization and in-

This evaluation approach is flawed for several reasons. A terpretation, possibly using simultaneous particle and flow

particle video algorithm could easily obtain a lower spatial optimization. We will also experiment with allowing slow,

error by pruning more particles (at the cost of a lower parti- smooth changes in a particle’s appearance over time, to per-

cle survival rate). Furthermore, by allocating fewer particles mit larger reflectance and scale changes. We may obtain

near occlusions and more particles in other regions, an al- better performance by incorporating invariant feature de-

gorithm could both increase the survival rate and decrease scriptors (for particles away from occlusions), spatiotempo-

the spatial error. Thus, we provide the evaluation for de- ral segmentation [16], and geometric constraints (for rigid

scriptive purposes only. This evaluation should not be used portions of the scene).

to compare the results with future methods. Current limitations of the particle video algorithm arise

In the future, we envision that researchers will create from our methods for positioning particles, rather than a

photo-realistic rendered videos with known ground truth fundamental limitation of the particle representation. By

correspondences. These rendered videos should include making our data and results available online, we hope oth-

deforming objects, complex reflectance, detailed geometry, ers will explore the particle video problem.

motion blur, unstable camera motion, and optical artifacts.

References

7. Conclusion [1] L. Alvarez, R. Deriche, T. Papadopoulo, and J. Sanchez.

Symmetrical dense optical flow estimation with occlusion

The particle video algorithm provides a new approach detection. In ECCV, pages 721–735, 2002.

to motion estimation, a central problem in computer vi- [2] T. Amiaz and N. Kiryati. Dense discontinuous optical flow

sion. Dense long-range video correspondences could im- via contour-based segmentation. In ICIP, pages 1264–1267,

prove methods for many existing vision problems, in areas 2005.

ranging from robotics to filmmaking. [3] S. S. Beauchemin and J. L. Barron. The computation of op-

Our particle representation differs from standard motion tical flow. ACM Computing Surveys, 27(3):433–467, 1995.

representations, such as vector fields, layers, and tracked [4] M. Black and P. Anandan. Robust dynamic motion estima-

feature patches. Some existing optical flow algorithms in- tion over time. In CVPR, pages 296–302, 1991.

Video 1, Frame 0 Particle Correspondences Video 1, Frame 25

Figure 6. Each row shows a frame pair from one test video. Correspondences are shown for particles in common between the frames.

[5] M. Brand. Morphable 3D models from video. In CVPR, [13] C. Silva and J. Santos-Victor. Motion from occlusions.

pages 456–463, 2001. Robotics and Autonomous Systems, 35(3–4):153–162, 2001.

[6] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High ac- [14] C. Strecha, R. Fransens, and L. V. Gool. A probabilistic

curacy optical flow estimation based on a theory for warping. approach to large displacement optical flow and occlusion

In ECCV, pages 25–36, 2004. detection. In Statistical Methods in Video Processing, pages

[7] M. Elad and A. Feuer. Recursive optical flow estimation– 71–82, 2004.

adaptive filtering approach. Visual Communication and Im- [15] W. Thompson. Exploiting discontinuities in optical flow.

age Representation, 9(2):119–138, 1998. IJCV, 30(3):163–174, 1998.

[16] J. Wang, B. Thiesson, Y. Xu, and M. Cohen. Image and video

[8] M. Irani. Multi-frame optical flow estimation using subspace

segmentation by anisotropic kernel mean shift. In ECCV,

constraints. In ICCV, pages 626–633, 1999.

pages 238–249, 2004.

[9] D. Lischinski. Graphics Gems IV, chapter Incremental De-

[17] J. Wills and S. Belongie. A feature-based approach for deter-

launay Triangulation, pages 47–59. Academic Press, 1994.

mining dense long range correspondences. In ECCV, pages

[10] N. Papenberg, A. Bruhn, T. Brox, S. Didas, and J. Weick- 170–182, 2004.

ert. Highly accurate optic flow computation with theoreti- [18] J. Xiao, H. Cheng, H. Sawhney, C. Rao, and M. Isnardi. Bi-

cally justified warping. IJCV, 2006 (to appear). lateral filtering-based optical flow estimation with occlusion

[11] J. Shi and J. Malik. Motion segmentation and tracking using detection. In ECCV, 2006 (to appear).

normalized cuts. In ICCV, pages 1154–1160, 1998. [19] C. L. Zitnick, N. Jojic, and S. B. Kang. Consistent segmenta-

[12] J. Shi and C. Tomasi. Good features to track. In CVPR, pages tion for optical flow estimation. In ICCV, pages 1308–1315,

593–600, 1994. 2005.

- Chapter 1.1-Multimedia DataÎncărcat dehubillanth
- 1712.00080.pdfÎncărcat deDeepak Gangwar
- Image MotionÎncărcat deDavid Lichaa
- IJETTCS-2014-06-24-126Încărcat deAnonymous vQrJlEN
- TV BroadcastingÎncărcat deSaleem Shahzad
- nNovia - Common Manual v1.1Încărcat deinfo1986
- THECTFACE_p2Încărcat dechapi798815
- Best Ever Algorithm Found for Huge Streams of DataÎncărcat deSilvino González Morales
- Viewsonic VX2025wm Product SheetÎncărcat deJunks4Fun
- Optimal Reconfiguration and Distributed Generator Allocation in Distribution Network Using an Advanced Adaptive Differential EvolutionÎncărcat dewvargas926
- Video Point User GuideÎncărcat deAmy Brown
- RapidFIRE OSD CompatibilityÎncărcat deaeroseb1
- AVHÎncărcat deMaram Maram
- scan line algorithm.Încărcat deNitish Mhaske
- Quo-ugs-0030037 Sklb Lembah Beringin ProjectorÎncărcat deSamsudin Ahmad
- Multimedia SystemÎncărcat deJalms
- LiveRequirements EnÎncărcat de43AJF43
- LabVIEW Machine Vision -- Day 1Încărcat dearthicse
- Deconstructing IPv6 with Brest.pdfÎncărcat deGath
- 176 Estimating Bandwidth Tech Brief 10152008Încărcat deMohammed Adnan Khan
- Chapter 2Încărcat de박승환
- 10.1.1.38Încărcat dePrempal Tomar
- Fwcamsbandwwp.en USÎncărcat detheimagingsource
- Synthesizing Scheme and Reinforcement Learning with CADIÎncărcat dedrtechmester
- PowerDirector 16Încărcat deJPosadas
- Matrix SATATYA_ New Channel Presentation.pdfÎncărcat deŠhübhãm Khâńñå
- SyncMaster 214TÎncărcat desjachosz
- 150c5bs_00_ppd_eng.pdfÎncărcat deAnthony Gonzales
- METH302A - Major Elective III Thermal Energy SystemsÎncărcat deAmit Pandey
- QM1Încărcat deKV Rao

- 332 Indian Food Recipes Sanjeev KapoorÎncărcat depramod1955
- Infrared_Thermography.pptÎncărcat desssuganthi
- 06 Thermal Rs(3)Încărcat dekannanjbr
- Intro Image ProcessingÎncărcat desssuganthi
- 32-medical-infrared-thermologyÎncărcat desssuganthi
- Literature SurveyÎncărcat desssuganthi
- Intro_SSÎncărcat desssuganthi

- 22 Maller ScleralPanceaÎncărcat deScorpio
- Towards Star Tracker Only Attitude Estimation - John Enright Et Al. 2010Încărcat dealvaroroman92
- Physics EssayÎncărcat dekylesmulders
- A Survey On Image Matching MethodsÎncărcat deInternational Journal of Latest Research in Engineering and Technology
- Physics of relativistic laser-plasmas.pdfÎncărcat deAjit Upadhyay
- LIQUID LENSÎncărcat deKA Angappan
- Ldr InvestigatoryÎncărcat deSaransh Gaur
- Five Idiots and Company 2010 Comms NotesÎncărcat deRonald Bryan Lu Palo
- Make Your Amateur Photos More ProfessionalÎncărcat deRey-an A. Moreno
- Australian Led Lamps Trade Brochure Dec 2013Încărcat deAndreas B Kresnawan
- fslasers.ppsÎncărcat deMukul Suryawanshi
- 114923Încărcat deRina Rizzi
- optical communication project.pdfÎncărcat dege120120
- Información sobre el CO2Încărcat deppklista
- X-Ray Absorption SpectroscopyÎncărcat deLI Li
- article (1)Încărcat deKatherine Powell
- IP Final Exam SolutionsÎncărcat deamermer123
- Anne Rutherford. “Cinema and Embodied Affect.”Încărcat deGaston Carrasco Aguilar
- BD FACSAria II Users GuideÎncărcat dechinmayamaha
- visionÎncărcat deapi-430903239
- BaseÎncărcat denikkimaxinevaldez
- ICO_Inst_Bk_Clinical_Optics__Ref_2013.pdfÎncărcat deElBuen John
- ISO 3059 MT and PT View CondÎncărcat dezwfuis
- 6 Preventative MaintenanceÎncărcat deLê Duy Thăng
- PASCO WA9314C Exp Guide Microwave OpticsÎncărcat dePedro Vargas
- Cylindrical Coplanar WaveguideÎncărcat debrankoprivica
- Remote SensingÎncărcat deSrinivas Shetty
- The Engineering Language: A Consolidation of the Words and Their DefinitionsÎncărcat deMomentum Press
- One-An Integrated, Underwater Optical AcousticÎncărcat deJohn Mathew
- Microscope Maintenance Manual.pdfÎncărcat deJose Guillermo Muñoz Camacho

## Mult mai mult decât documente.

Descoperiți tot ce are Scribd de oferit, inclusiv cărți și cărți audio de la editori majori.

Anulați oricând.