RECURSIVE DISPARITY ESTIMATION ALGORITHM FOR REAL TIME
STEREOSCOPIC VIDEO APPLICATIONS
E.A. Hendriks
Delft University of Technology
Dept. of EE, Information Theory Group
Mekelweg 4
2628 CD Delft
The Netherlands
e-mail hendriks@it.et.tudelft.nl
ABSTRACT
Several implementations of a recursive disparity esti-
mator have been investigated in which bi-directional
spatial predictions as well as temporal predictions are
used. The recursive disparity estimator showed to be
very attractive for real time multi view interpolation
applications because of the use of the predictions and
small search ranges or update sets. The estimated dis-
Gy. Marosi
Budapest University of Technology
Dept. of Telecommunication and Telematics
H-1111 Sztoczek utca 2.
Budapest
Hungary
e-mail marosi@bme-tel.ttt.bme.hu
of view the most promising algorithms are those based
on block-wise processing. Block-based algorithms are
highly structured and can easily be pipelined or paral-
lelized.
The problem of disparity estimation is strongly re-
lated to motion estimation. Nowadays, block-based
motion estimators are being developed [6, 7, 8] or
even commercially available [9]. All implemented al-
parity elds are smooth, causing less visible artifacts gorithms are based on full search and/or hierarchical
in the interpolated intermediate viewpoints. Moreover, block matching. The main application for this class of
the recursive estimator is attractive for hardware im- motion estimators is MPEG coding. For coding pur-
plementation. Only a small number of updates have to poses the aim is to nd good matching values, where
be checked, thereby considerably reducing the number the quality of the estimator is usually judged upon the
of operations to be executed. compression ratio that can be achieved. For interpo-
lation purposes it is not enough to nd local optima,
1. INTRODUCTION but true vectors are necessary to prevent annoying ar-
tifacts.
One of the goals in the development of enhanced 3D-
telepresence systems is to provide the viewer an un-
limited number of viewpoints with a limited number of
cameras. This so-called multi viewpoint capability can
be achieved by interpolating intermediate viewpoints
using at least two recorded image sequences (left and
right) and their disparity information. Assuming par-
allel camera alignment the dierence between the left
and the right images is due to the objects' horizon-
tal translation depending on their depth in the scene.
This displacement is called disparity and its estimation
is required.
In recent years several disparity estimation algo-
rithms have been developed. One can distinguish be-
tween object-based [1], pixel-based [2] and block-based
Finding the true vector elds is not obvious. As-
suming ideal cameras and illumination, there are still
non-occluded areas where simple estimators fail, for ex-
ample with no texture or with repetitive pattern. Per-
ceptually the most irritating result of not having true
vector values is the lack of consistency in the output:
spatial inconsistencies in the interpolated still images
and additional temporal inconsistencies in interpolated
image sequences.
2. BLOCK-BASED RECURSIVE
DISPARITY ESTIMATION
There have been many simulations of various algo-
rithms for disparity estimation - all of them suer from
disparity estimators [3, 4, 5]. From a hardware point some errors in the result eld. Amongst the block-
This work was done in the framework of the European ACTS based algorithms, the simplest full search results in
AC092 'PANORAMA' Project and supported by a mobility annoying discontinuities that is even more irritating
grant of the TEMPUS program for moving sequences. The smoothness constraint of
 BBBBB CCCCC
BBBBB
CCCCC
BBBBB CCCCC
BBBBB BBBBB
CCCCC
BBBBB
BBBBB CCCCC
BBBBB
CCCCC
BBBBB BBBBB
CCCCC
BBBBB
BBBBB
BBBBB
BBBBB
Block in question
CCCCC
BBBBB
CCCCC
T (a) T (b) T (c) Spatial Predictor

CCCCC CCCCC CCCCC CCCCC CCCCC CCCCC &&&&&

CCCCC
&&&&&
Temporal Predictor
CCCCC CCCCC
BBBBB !!!!!
!!!!! !!!!!
BBBBB
CCCCC CCCCC
!!!!!
BBBBB !!!!!
!!!!! !!!!!
CCCCC CCCCC
!!!!!
BBBBB *****
&&&&&
CCCCC
CCCCC CCCCC
BBBBB
CCCCC CCCCC
&&&&&
!!!!! CCCCC
!!!!!
BBBBB
CCCCC CCCCC CCCCC
&&&&&
!!!!! CCCCC
!!!!!
BBBBB
CCCCC CCCCC *****
Temporal accelerator
*****
BBBBB &&&&&
!!!!! !!!!!
BBBBB
&&&&& &&&&&
!!!!!
!!!!! !!!!!
BBBBB
!!!!!
&&&&&
BBBBB &&&&& *****
!!!!! *****
!!!!!
&&&&&
*****
!!!!!
***** *****
!!!!!
*****
T (d)
T *****
T *****
T1 (e) T1 (f)

Figure 1: Dierent search strategies: a) full search, b)+c) one-directional spatial prediction, d) bi-directional spatial
prediction, e) spatial-temporal prediction, f) spatial-temporal prediction with temporal accelerators.

disparity paths suggests the usage of the spatial re- date values f? 2 , 0, + 2 g resulted in a further im-
L L

dundancies in real-life images, and also the temporal
redundancy in image sequences, hence a recursive al-
gorithm seems to be advantageous [10]. Let us consider
the steps that lead from the full search approach to the
spatial-temporal recursive algorithm with convergence
accelerators.
The disparity values for the images used over the
tests are known to be in the range of -16 and +40, hence
the block-based full search algorithm (Figure 1/a) was
applied in this interval only1; minimum absolute dier-
ence in the luminance domain was the search criterion
of the block matching. As a 0 order approach towards
th
spatial recursion, the estimated disparity value of a pre-
viously evaluated neighboring block was selected as the
relative zero value of the search range (Figure 1/b,c).
To utilize bi-directional convergence, the set of dispar-
ity candidates was extended by a second search ini-
tialized by the estimated disparity value of another
evaluated neighboring block (Figure 1/d). Obviously,
overlapping ranges of the search area have to be calcu-
lated only once, and due to the usually high spatial re-
dundancy of blocks, this bi-directional search requires
only a little extra computation. A smoother dispar-
provement in the smoothness of the disparity eld caus-
ing a consistent interpolated output and a much faster
estimation process. The optimal values of both 1 and
L
L 2 for the AQUA and the MANEGE test images with a
block-size of 8-by-8 turned out to be between 2 and 4.
To improve the consistency of the interpolated se-
quences temporal recursion was also utilized; a third
search is applied, initialized by the disparity value of
the same block in the previous frame (Figure 1/e). Al-
though this step increases the number of candidates,
the annoying temporal jumps of blocks are highly re-
duced. The spatially bi-directional search still has the
weakness of slow convergence at the upper edges of
objects. Using the estimated results of the previous
frame four-directional convergence can be introduced,
by adding the results of two more neighboring blocks
(referred to as the convergence accelerators) to the ini-
tializing set (Figure 1/f). To avoid the degradation of
spatial consistency, the update set of these convergence
accelerators is reduced to f0g, i.e., no search is done
around their disparity values, just the values them-
selves are taken into account as possible candidates.
ity eld can be achieved by strongly limiting the num- 3. RESULTS
ber of possible candidate disparity values. Keeping the
bi-directional convergence structure the search range All algorithms described above have been implemented
was radically reduced to the range of [? 1 + 1 ] L ::: L and tested on several real-life stereoscopic sequences;
around the disparity values of the initializing blocks for the evaluation with the AQUA sequence see Ta-
(Figure 1/d); decreasing the candidate set to the up- ble 1, for other sequences having the same camera set-
1 Except for the MAN sequence, where due to the larger base- up these algorithms show very similar characteristics
line there is an oset in the disparity values. as in case of AQUA.
 See Spatial/Temporal Update Number of Notes
Fig. Recursion Length Cost Function
1/  1 2 Operations [106]
L ;

a) No/No - 23.2 Full Search [-16,+40], annoying temporal

inconsistency; slow estimation process.
b) Range / No 2 3.0 Cannot follow edges, shadows on the in-
terpolated image.
c) Range / No 4 4.9 Good still pictures.
d) Range / No 8 8.5 Dizzy background.
e) Range / Range 2 3.4 Temporal consistency is introduced.
f) Range / Range 2 3.6 Faster convergence at the upper edges of
objects due to temporal accelerators.
f) Set / Set 2 2.5 Fast convergence due to temporal acceler-
ators and faster estimation process; best
results.

Table 1: Results of the recursive disparity estimator for the DISTIMA test sequence AQUA. Since there have been no
reliable and standard quality measurement and validation methods, Notes re ect the subjectively judged perceptual
quality achieved.

The usefulness of the predictors can clearly be seen mance and perceptual quality are demonstrated. These
both on the decreased computational requirement and kinds of disparity estimation algorithms with a sim-
on the more consistent disparity elds they provide. ple majority voting post-processing step provide a dis-
The \convergence accelerator" predictors not only im-
prove the estimation at the edges of objects, but they
also reduce the convergence delay introduced by the
recursions; for the test sequences a delay of 5{6 frames
(that is 0.2 sec supposing 25 frames/sec) was found.
The MAN sequence, due to the larger baseline, has
disparity values in the range of [+40 + 100], which
:::
fact was compensated by shifting both views 25 pixels,
thus transforming the disparity range to [?10 +50].
:::
Because of the large noisy background area of the MAN
sequence the initializers of the recursive disparity esti-
mators are not coherent, so they yield noisy output,
which results several times in multiple initial values
and non-overlapping search ranges for the recursions.
(Note that the noisy values do not cause more harm in
this sequence since the background has very few visible
patterns.) Hence, 30% more calculations were required
for MAN than for AQUA and MANEGE.
It has to be mentioned, that the stereo image se-
quences that were used during the tests suer from
luminance dierences between the left and the right
parity eld in the order of 1{2 seconds per frame on
a Sun-4m workstation, and the result of the spatial-
temporal recursive algorithm with convergence accel-
erators is comparable, for example, to the one yielded
by a hierarchical estimation method [12], while the lat-
ter is reported to take 6 hours of computing per frame
(i.e., 4 magnitudes more time) on the same platform.
It also has to be mentioned that these recursive algo-
rithms are hardware ecient, since they work on the
basis of very simple and regular rules.
From our evaluation it can be concluded that the
recursive disparity estimator requires operations about
one order of magnitude less than a comparable full
search estimator, while having less visible artifacts, es-
pecially in image sequences (see Figure 2). Therefore
it is a suitable candidate for hardware implementation
in a real time multi viewpoint stereoscopic video chain.
There are still some problems that have to be
solved, e.g. recursive disparity estimation without dis-
parity oset compensation failed on the sequences hav-
ing large baselines, which could be avoided by applying
images. In order to avoid it a stereo balancing prepro- a preprocessor to nd and realize the oset-shift. Some
cessing step was applied [11]. further improvement might be achieved by taking into
account some sort of reliability measure during the es-
4. CONCLUSIONS timation by weighting the possibilities for the recursion
initialization blocks, based, for example, on the error
Various block-based recursive disparity estimation al- eld or on the left-to-right versus right-to-left disparity
gorithms are discussed in this article and their perfor- path consistency; a cross checking technique would also
Figure 2: The 13 interpolated frame of the DISTIMA test sequence MAN: (a) with full search, and (b) with
th
spatial-temporal recursive disparity estimation algorithm.
improve results for one-side occluded parts of objects.
Simple frame correlators might be applied to skip tem-
poral recursion if successive frames dier more than a
certain threshold.
5. REFERENCES
[1] L. Falkenhagen and M. Strintzis. 3d object-
based depth estimation from stereoscopic image
sequences. Deliverable of the DISTIMA project,
March 1995.
[2] J.N. Driessen, L. Boroczki, and J. Biemond. Pel-
Transactions on Circuits and Systems for Video
Technology, October 1995.
[7] H. Yeo and Y.H. Hu. A novel modular systolic
array architecture for full-search block matching
motion estimation. IEEE Transactions on Circuits
and Systems for Video Technology, October 1995.
[8] A. Pirson et al. A programmable motion estima-
tion processor for full search block matching. Pro-
ceedings ICASSP, Detroit, May 1995.
[9] M. Querol. Sti3220, motion estimation processor
codec. Application Note, SGS Thompson, 1995.
recursive motion eld estimation from image se-
quences. Journal on Visual Comm. and Image [10] G. de Haan and H. Huijgen. Motion estimator for
Representation, September 1991. tv-picture enhancement. Proceedings Signal Pro-
cessing of HDTV, III, Turin, September 1991.
[3] S.S. Intille and A.F. Bobick. Disparity space im- [11] R.E.H. Franich. Disparity estimation in stereo-
ages and large occlusion stereo. MIT Media Lab scopic digital images. ph.D Thesis, Dept. of EE,
Vision and Modeling Group, Technical Report No
220, 1993. Delft University of Technology, September 1996.
[4] B. Chupeau. A multiscale approach to the joint [12] P. Doelam. Hierarchical disparity estimation using
computation of motion and disparity: applica- disparity space images. Msc. Thesis, Dept. of EE,
tion to the synthesis of intermediate views. Proc. Delft University of Technology, June 1995.
4th European Workshop on 3DTV, Rome, October
1993.
[5] R.E.H. Franich, R.L. Lagendijk, and J. Biemond.
A disparity space image path. Proceedings of the
International Workshop on Stereoscopic and Three
Dimensional Imaging, Santorini, September 1995.
[6] L. de Vos and M. Schobinger. VLSI architecture
for a
exible block matching processor. IEEE