Dense Motion Estimation based on Polynomial expansion

Pi19404
March 5, 2014

Contents

Contents
Dense Motion Estimation based on Polynomial expansion
0.1 Introduction 0.2 Introduction 0.3 Conclusion . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3
3 3 11 11

2 | 12

Dense Motion Estimation based on Polynomial expansion

Dense Motion Estimation based on Polynomial expansion

0.1 Introduction
In this article we will look at dense motion estimation based on polymonial repsentation of image.The polynomial basis representation of the image is obtained by approximating the local neighborhood of image using quadratic polynomial basis.The displacement between adjacent frames can be obtained by equating the coefficients of the basis.

0.2 Introduction
 This article describes a fast dense optical flow computation algorithm by Farnebck, 2002 . In the earlier articles it was seen that a local neighborhood of image can be represented using polynomial basis. Using this representation estimation of dense optical flow is obtained at each point in the image.

 Assuming how a polynomial transforms under translation using the polynomial expansion coefficients derived from current and previous frames and estimate of displacement vector is obtained.  The idea of polynomial expansion is to approximate a neighborhood of a point in a 2D function with a polynomial.Considering quadratic polynomial basis 1; x2 ; y 2 ; x; y; xy ,pixel values in a neighborhood of image is represented by
f x

( ) xT Ax + bT x + c

where A is a symmetric matrix,b is a vector and c is a scalar

 The coefficients can be estimated by weighted least square estimate of pixel values about neighborhood as seen in the earlier

3 | 12

Dense Motion Estimation based on Polynomial expansion

(a) 2D basis function

Figure 1: Polynomial basis

4 | 12

Dense Motion Estimation based on Polynomial expansion article. As with all optical flow algorithm,the brightness constancy assumption is made. The brighness of a path of image in adjacent frames is constant.
 Consider a translational motion d ecountered at point (x; y) in the image.

( ) = xT A1 x + bT 1 x + c1 T T f2 (x) = f1 (x d) = (x d) A1 (x d) + b1 (x d) + c1 T T f2 (x) = f1 (x d) = (x) A1 (x) + (b1 2A1 d) (x) + dT Ad bT 1 d + c1 T T f2 (x) = f1 (x d) = (x) A1 (x) + b2 (x) + c2
f1 x

equating coefficients in the two polynomials Due to brightness constancy assumption

A2 b2 c2

= A1 = (b1 2A1 d) = dT Ad bT 1 d + c1
A

Assuming
d

is non-singular

1 A 1(b2 b1 ) 2

Thus by equating the coefficients of the polynomial the displacement vector can be obtained at each point in the image assuming there is overlap between the region of interest ie image neighborhood in adjacent frames.

 We extract  Let us say we have the estimate of displacement d the ROI about the neighborhood at point P (x; y ) and point P (x + d:x; y + d:y ) The polynomial basis are extracted and the computation that is shown above is performed.
 The total displacement can be estimated as

 ) b2 = b1 2A1 (d ,A1 ,b1 we know d

d

= 0:5 A1 (b2 +  b2 b1 )

 Thus an iterative scheme can be used where in every successive iteration a better estimate of displacement vector is obtained.The iterations can be terminated when change is displacement vector is below is threshold in successive iterations or

5 | 12

Dense Motion Estimation based on Polynomial expansion specific number of iterations have been completed. The intial estimate of displacement vector is assumed to be (0; 0) Thus ROI or the image patch in the current and previous frames are at the same.

6 | 12

Dense Motion Estimation based on Polynomial expansion

/** * @brief updatePoly : The function computes the optical flow displacement * @param ptr1 : pointer to array containing the polynomial basis componen * @param ptr2 : pointer to array containing the polynomial basis componen * @param d : estimate of displacement vector at the current point * @param flag : the flag indicating if the point is border pixels or not * @param M : pointer to output array returning the computed coefficien * @param x : co-ordinate at the current point where the coefficients i * @param y : * @param s : windows size for averaging */ void updatePoly(const float *ptr1,const float *ptr2,Point2f d,bool flag, float *M,Point p,Size s) { int x=p.x; int y=p.y; const int BORDER = 5; static const float border[BORDER] = {0.14f, 0.14f, 0.4472f, 0.4472f, 0 float r[5]; float r2,r3,r4,r5,r6; if(flag==true) { //average A_1 and A_2 r4=(ptr1[2]+ptr2[2])*0.5; r5=(ptr1[3]+ptr2[3])*0.5; r6=(ptr1[4]+ptr2[4])*0.25; } else { r[0]=0.f; r[1]=0.f; r[2]=ptr1[2]; r[3]=ptr1[3]; r[4]=ptr1[4]*0.5; r2=r[0]; r3=r[1]; r4=r[2]; r5=r[3]; r6=ptr1[4]*0.5; }

7 | 12

Dense Motion Estimation based on Polynomial expansion

//computing -(b1-b2) r2=(ptr1[0]-ptr2[0])*0.5; r3=(ptr1[1]-ptr2[1])*0.5; //sum for iterative estimation b2=b2+\bar{b2} //r2 += r4*d.y + r6*d.x; ///r3 += r6*d.y + r5*d.x; r2 += r4*d.x + r6*d.y; r3 += r6*d.x + r5*d.y; if( (unsigned)(x - BORDER) >= (unsigned)(s.width - BORDER*2) || (unsigned)(y - BORDER) >= (unsigned)(s.height - BORDER*2)) { float scale = (x < BORDER ? border[x] : 1.f)* (x >= s.width - BORDER ? border[s.width - x - 1] : 1.f)* (y < BORDER ? border[y] : 1.f)* (y >= s.height - BORDER ? border[s.height - y - 1] : 1.f); r2 *= scale; r3 *= scale; r4 *= scale; r5 *= scale; r6 *= scale;

//computing final displacement d=A^-1(b2-b1)*0.5 M[0] = r4*r4 + r6*r6; // G(1,1) M[1] = (r4 + r5)*r6; // G(1,2)=G(2,1) M[2] = r5*r5 + r6*r6; // G(2,2) M[3] = r4*r2 + r6*r3; // h(1) M[4] = r6*r2 + r5*r3; // h(2)

 The method EstimateFlow computes the coefficients A; b2 ; b1 required for displacentt field computation.The EstimateFlow functions call the method UpdatePoly for each pixel in the image.  The displacement field obtained may be discontinuous and contain noise and other atrifacts.Since it is reasonable to assume that if motion is encounted at a point, the neighborhood pixels also encounter the same motion. The displacement vector can be averaged over a neighborhood to get a better estimate of the displacement field.

8 | 12

Dense Motion Estimation based on Polynomial expansion

 The method AverageFlow computes the average of coefficients A1 ; b2 ; b1 and then computes the displacement flow field.

/** * @brief AverageFlow * @param _R0 : Polynomial basis of prev frame * @param _R1 : Polynomial basis coefficients of current frame * @param _flow :estimate of current displacement field * @param matM :containing coefficients of polynomial basis for computing */ void AverageFlow( const Mat& _R0, const Mat& _R1,Mat& _flow, Mat& matM) { int x, y, i, width = _flow.cols, height = _flow.rows; //computing the average flow field cv::GaussianBlur(matM,matM,Size(15,15),sigma); for( y = 0; y < height; y++ ) { double g11, g12, g22, h1, h2; float* flow = (float*)(_flow.data + _flow.step*y); float *coeff=(float *)(matM.data+matM.step*y); // update borders for( x = 0; x < width; x++ ) { g11 = coeff[x*5]; g12 = coeff[x*5+1]; g22 = coeff[x*5+2]; h1 = coeff[x*5+3]; h2 = coeff[x*5+4]; //computing determinant for inverse double idet = 1./(g11*g22 - g12*g12 + 1e-3); //computing displacement flow field flow[x*2] = (float)((g11*h2-g12*h1)*idet); flow[x*2+1] = (float)((g22*h1-g12*h2)*idet); } } //calling EstimateFlow for updading coefficients //for computation of next iteration EstimateFlow(_R0, _R1, _flow, matM,0,flow.rows);

9 | 12

Dense Motion Estimation based on Polynomial expansion

}
 This approach may in case of large displaement.Hence a multi scale estimation is performed. The estimation of flow field is performed as the smallest resolution.The displacement computed at the lower resolution is used as estimate for peform displacement field computation at higher resolution.  Dense optial flow computed for two frames is shown in figure 2c

(a) Frame 1

(b) Frame 2

(c) Optical Flow

Figure 2: Displacement field

10 | 12

Dense Motion Estimation based on Polynomial expansion

0.3 Conclusion
This article describes the theory and implementation details of the dense optical flow algorithm based on paper by Farnebck, 2002.This code for the algorithm can be found at github repository https://github.com/pi19404/OpenVision in files DenseOf.cpp and DenseOf.hpp files. In the future article we will look at optimizing the code using SSE,NEOM and OpenCL optimizations to enable real time computation of the dense optical flow fields

11 | 12

Bibliography

Bibliography
[1] Kenneth Andersson and Hans Knutsson.  Continuous normalized convolution. In: ICME (1). IEEE, 2002, pp. 725728.

dblp.uni-trier.de/db/conf/icmcs/icme2002-1.html#AnderssonK02.
[2]

isbn:

0-7803-7304-9.

url: http://

Kenneth Andersson, Carl-Fredrik Westin, and Hans Knutsson.  Prediction from o-grid samples using continuous normalized convolution. In: Signal Processing 87.3 (Mar. 22, 2007), pp. 353365.

url: http://dblp.uni- trier.de/db/ journals/sigpro/sigpro87.html#AnderssonWK07.

EX-1596. MA thesis. SE-581 83 Linkping, Sweden: Linkping University, 1996.

[3]

Gunnar Farnebck.  Motion-based Segmentation of Image Sequences. LiTH-ISY-

[4]

Gunnar Farnebck.  Polynomial Expansion for Orientation and Motion Estimation. Dissertation No 790, ISBN 91-7373-475-6. PhD thesis. SE-581 83 Linkping, Sweden: Linkping University, Sweden, 2002.

[5]

Gunnar Farneback.  Two-Frame Motion Estimation Based on Polynomial Expansion. In: SCIA. LNCS 2749. Gothenburg, Sweden, 2003, pp. 363370.

12 | 12