Video Stabilization Using RANSAC

In this work, to stabilize a video, a salient feature in a frame of the video is used as an anchor to cancel all changes in the relationship with the salient feature. This procedure must be implemented on all video with the knowledge that a salient features in the video frames. With this way, the deformations which is caused by camera movements are corrected. In our algorithm; once, we determine the affine image transformations between all adjacent frames of a video sequence with a random sample consensus (RANSAC) method is used to show correspondence between two images. Then, we deform video frames to achieve a stabilized video.And we used Matlab vision toolbox to implement this algorithm. We handled this work under four step; 1. 2. 3. 4. Reading frames Finding salient points in frames and matching corresponded points Model fitting whit RANSAC Estimating affine transform warping frames

Reading Frames In this step we read the first and second frames of a video sequence as an intensity image. Because the color is not necessary for the stabilization algorithm, also grayscale values improve the algorithm speed.

dosyaIsmi='ornek.wmv'; hVideoSrc = vision.VideoFileReader(dosyaIsmi, ... 'ImageColorSpace', 'Intensity',... 'VideoOutputDataType', 'single'); kareA=step(hVideoSrc); kareB=step(hVideoSrc); cvexShowImagePair(kareA, kareB, 'A karesi', 'B karesi');

The first and the second frames of video

We can see the movie between frames so..

figure; imshow(cat(3,kareA,kareB,kareB));

Above we show two images side by side, and producing an image composed of red-cyan, pixel by pixel in order to demonstrate the difference between them. Clearly, there is a. Large vertical and horizontal movement between two frames

Finding Salient Points In Frames And Matching Corresponded Points In this step, to have the best chance we want the points around the image of great features such as corners. We used the CornerDetector object of which method is local intensity comparison (Rosen & Drummond).Also we can specify the method as one of Harris corner detection (Harris & Stephens) or Minimum eigenvalue (Shi & Tomasi), but we prefer the fastest of them, Local intensity comparison (Rosten & Drummond).

enFazlakosesayisi = 120; esikDeger = 1e-3; hCD = vision.CornerDetector( ... 'Method','Local intensity comparison (Rosen & Drummond)', ... 'MaximumCornerCount', enFazlakosesayisi, ... 'CornerThreshold', esikDeger, ... 'NeighborhoodSize', [11 11]);

Beyond, we restricted the maximum number corner to find with MaximumCornerCount Identified as 120.So we get rid of unnecessary workload.If we had used a video which has got lots of corners in frames, we must increase the maximum corner count to take a correct result from this stabilization algorithm. koselerA = step(hCD, kareA); koselerB = step(hCD, kareB); %burada yaptmz izleyelim cvexShowImagePair(kareA, kareB, 'A daki keler', 'B deki keler', ... 'SingleColor', koselerA, koselerB);

the corners

off-topic in parentheses : If the cornerdetector parametries are changed

the other video frames for maximumu corner count defined as 500

the video frames for maximumu corner count defined as 10 you can see the decreasing of corner counts (compare with previous frames

In addition we declared CornerThreshold which specifies the minimum metric value that indicates a corner as a positive scalar number as 1e-1 and NeighborhoodSize which is the size of suppressed region around detected corner as 11 to 11 matrix range.

the frames for NeigborhoodSize=[5 5]

We used in this step of the algorithm future based matching. Therefore before matching we extracted the features from the points so we get rid of some corners which are not suitable. For each point, extracted a 11 by 11 block centered around. The corresponding cost is used between the points is the sum of squared differences (SSD) between their respective areas of the image.

[featureA, noktalarA] = extractFeatures(kareA, koselerA); [featureB, noktalarB] = extractFeatures(kareB, koselerB); eslenenciftler = matchFeatures(featureA, featureB); noktalarA = noktalarA(eslenenciftler(:, 1), :); noktalarB = noktalarB(eslenenciftler(:, 2), :); cvexShowMatches(kareA, kareB, noktalarA, noktalarB, 'A', 'B');

Corresponded points frame A and frame B

Model Fitting With RANSAC And Estimating Geometric Transform

In the previous step we extracted the correspondence of the points but some of some the points is incorrect.So we must use RANSAC algorithm to find a robust estimate of the geometric transformation between two images. The Ransac Algorithm 1. Select randomly the minimum number of points(S) required to determine the model parameters. 2. Solve for the parameters of the model. 3. Determine how many points from the set of all points fit with a predefined threshold d. 4. If the fraction of the number of inliers over the total number points in the set exceeds a predefined threshold T , re-estimate the model parameters using all the identified inliers and terminate. 5. Otherwise, repeat steps 1 through 4 (maximum of N times).

hGTE = vision.GeometricTransformEstimator;%geometrik transformu tahmin etmek iin kullacaz hGT = vision.GeometricTransformer; hGTp = vision.GeometricTransformer; N = 5;% N Ransacta uygun modeli bulmak iin deneme saysn 5 olarak belirledik TFORM = cell(1,N); cost = zeros(1,N); INLIER_INDEX = cell(1,N); for j=1:N % affine transformunu tahmin et [TFORM{j},INLIER_INDEX{j}] = step(hGTE, noktalarB, noktalarA); % resmi dzeltelim kareBp = step(hGT, kareB, TFORM{j}); % hatay hesaplayalm cost(j) = sum(sum(imabsdiff(kareBp, kareB))); end % en iyi sonucu alalm. [~,ix] = min(cost); kareBp= step(hGT, kareB, TFORM{ix}); H = [TFORM{ix} [0 0 1]'];

To implement RANSAC, we defined N as 5 to take optimum result (if we increase N the stabilization improves but the process time increase.) Also we used to estimate geometric transform GeometricTransformEstimator object which estimates geometric transformation from matching point pairs and returns the transform in a matrix.

Inlier points

Estimating Affine Transform And Warping The Frame In this step, we implemented the estimating of affine transform with cvexTformToSRT function and warped the frames.

HsRt = cvexTformToSRT(H); Hcumulative = HsRt * Hcumulative; kareBp = step(hGTp, kareB, Hcumulative);

a wrapped frame

Here , the all function which stabilize a video, videoDengeleme(dosyaIsmi) and explanation in Turkish; function [] = videoDengeleme( dosyaIsmi ) %Bu fonksiyon video stabilizasyonu yapmaktadr % Mcahit ALTINTA 2012 %videoyu kullanacamz hale getirelim %Stabilizasyon ileminde renklerle iimiz olmad iin gray %deerlerle kullanyoruz. hVideoSrc = vision.VideoFileReader(dosyaIsmi, ... 'ImageColorSpace', 'Intensity',... 'VideoOutputDataType', 'single'); %Burada ke bulmak iin vision toolbox unda bulunan CornerDetector %fonksiyonunun Local intesity comparison (Rosen&Drummond) metodunu kullandk %MaximumCornerCount yani karemizde en fazla bulmak itediimiz ke saysn %120 olarak belirledik, CornerThreshold yani incelediimiz noktann ke %olarak kabl edilmesi iin (NeighborhoodSize kadar) etrafndaki intesisty %deerleriyle arasnda olamas gereken fark e-3 olarak belirledik ve %NeighborhoodSize yani her bir nokta iin baklacak matris aral 11 e 11 %http://www.mathworks.com/help/toolbox/vision/ref/vision.cornerdetectorclass.html enFazlakosesayisi = 500; esikDeger = 1e-3; hCD = vision.CornerDetector( ... 'Method','Local intensity comparison (Rosen & Drummond)', ... 'MaximumCornerCount', enFazlakosesayisi, ... 'CornerThreshold', esikDeger, ... 'NeighborhoodSize', [11 11]); %RANSAC hGTE = vision.GeometricTransformEstimator;%geometrik transformasyonu tahmin etmek iin kullacaz hGT = vision.GeometricTransformer; hGTp = vision.GeometricTransformer; N = 5;% N Ransacta uygun modeli bulmak iin deneme saysn 5 olarak belirledik TFORM = cell(1,N); cost = zeros(1,N); INLIER_INDEX = cell(1,N); hVPlayer = vision.VideoPlayer; kareB =step(hVideoSrc); kareBp = kareB; Hcumulative = eye(3); while ~isDone(hVideoSrc) % freamleri okuyalm kareA = kareB; kareAp = kareBp; kareB = step(hVideoSrc); koselerA = step(hCD, kareA); koselerB = step(hCD, kareB); %burada yaptmz izleyelim

%cvexShowImagePair(kareA, kareB, 'A daki keler', 'B deki keler', ... % 'SingleColor', koselerA, koselerB); %## %burada feature tabanl ilikili eleme (feature-based corresponding %matching) gerekletireceiz %## %biz yukarda keleri bulduk ama bu bulduumuz kelerden iimize yarayacak %olan feature (ayrt edici diyebiliriz) lar bizim iimize yarayacak.onun iin %kelerin iinden featurelarmz kartyoruz. bu ilem iin extractFeature %foksiyonunu kullanacaz.Bu fonksiyonu default halde blockSize deeri 11 dir. %http://www.mathworks.com/help/toolbox/vision/ref/extractfeatures.html [featureA, noktalarA] = extractFeatures(kareA, koselerA); [featureB, noktalarB] = extractFeatures(kareB, koselerB); % MatchFeature fonksiyonuyla bulduumuz ayrt edici feature lar sum of % square difference (SSD-kareleri farknn toplam) yntemiyle eliyoruz %http://www.mathworks.com/help/toolbox/vision/ref/matchfeatures.html %Computer Vision Study Notes sayfa 33 eslenenciftler = matchFeatures(featureA, featureB); noktalarA = noktalarA(eslenenciftler(:, 1), :); noktalarB = noktalarB(eslenenciftler(:, 2), :); %yaptmz izleyelim %cvexShowMatches(kareA, kareB, noktalarA, noktalarB, 'A', 'B'); %## %burada elenen iftlerin ou doru olmakla birlikte doru olmayan baz %elemelerde var. bunlardan kurtulmak iin RANSAC kullanacaz %## %Computer Vision Study Notes sayfa 44 %http://www.mathworks.com/help/toolbox/vision/ref/vision.geometrictransformestimatorclass.html %http://www.mathworks.com/help/toolbox/vision/ref/vision.geometrictransformerclass.html for j=1:N % affine transformunu tahmin et [TFORM{j},INLIER_INDEX{j}] = step(hGTE, noktalarB, noktalarA); % resmi dzeltelim kareBp = step(hGT, kareB, TFORM{j}); % hatay hesaplayalm cost(j) = sum(sum(imabsdiff(kareBp, kareB))); end % en iyi sonucu alalm. [~,ix] = min(cost); kareBp= step(hGT, kareB, TFORM{ix}); H = [TFORM{ix} [0 0 1]']; HsRt = cvexTformToSRT(H); Hcumulative = HsRt * Hcumulative; kareBp = step(hGTp, kareB, Hcumulative); % videonun dengenmi halini step(hVPlayer, cat(3,kareAp,kareBp,kareBp)); end

release(hVideoSrc); release(hVPlayer); end %NOT:"http://www.mathworks.com" kaynak olarak kullanlmtr

Print Scrn Image of this work

* the video 'bisikletli.avi' from http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/