3D VIDEO VISUALIZATION ON THE HOLOVIZIO™ SYSTEM
Zoltán Megyesi, Attila Barsi, Tibor Balogh
Holografika
Baross str. 3. Budapest
H-1192, Hungary
+36 1 282 4921
{z.megyesi,a.barsi,t.balogh}@holografika.com
ABSTRACT
The visualization of 3D content is an increasingly important
issue as computer graphics applications, computer vision
methods and display devices evolve rapidly. As a part of
this 3D evolution, fields related to 3D video have also
advanced, but displaying high quality large Field of View
(FOV) 3D videos in real time still pose a considerable
challenge. The HoloVizio™ system is a high quality
continuous view 3D display that is capable of displaying
large FOV 3D data. In this paper we present the 3D video
playback capabilities of the HoloVizio™ system, in
connection with recent applications. We focus on practical
3D representation formats, and their usability in the 3D
acquisition-transfer-display chain.
Index Terms— 3D display, 3D video, autostereoscopic,
multiview, free viewpoint video
1. INTRODUCTION
Realistic visualization is a goal that has been chased for
decades, inspiring development for several research fields,
including computer graphics, computer vision, and
acquisition and display technologies. Displaying 3D is a
major step towards that goal, as it fully exploits the
capabilities of the human vision system, giving more
realistic experience than conventional 2D displaying. This is
especially true for displaying 3D videos, which is often
considered the pinnacle of non-interactive visualization.
To address 3D video visualization, the principles of the
3D displaying must be understood, and the 3D information
must be identified. Alternative applications of 3D formats
exist. Considerable research effort is spent on Free
Viewpoint Video [2], which is a related application also
making use of 3D video formats. Since 3D displaying is
only one end of the fields related to 3D video, acquisition,
storage and transfer problems must also be taken into
consideration when selecting a solution.
In the following sections the principles of 3D
displaying is discussed followed by the discussion of the 3D
image, and the widespread 3D representations. Finally
solutions are summarized for fast playback of selected
representations on the HoloVizio™ system.
2. 3D DISPLAY HISTORY
The evolution of 3D related technologies has a long history,
in which 3D displaying advanced significantly only in
recent years. 3D display systems exploit the binocular
parallax to provide 3D information to the Human Vision
system. In the past this has been achieved by providing
separate (stereo) information for the two eyes, either by the
use of glasses or head mounted gears.
Autostereoscopic displays in contrast provide 3D
perception without the need for obtrusive tools. These
systems implement the separation for the left and right eye
using various optical or lens raster techniques directly above
the screen surface. All of these displays provide 3D views
through direction selective light emission, i.e. the visible
image on the screen depends on the viewer position.
The most widespread autostereoscopic solutions are
stereoscopic displays with tracking [5]. Stereoscopic
displays can emit only two distinguishable light beams from
each pixel, thus the viewer must be positioned at the sweet
point. Tracking techniques are used to locate the viewer
position to adjust the displayed content.
Multiview systems [4] produce 8-16 views, which is
still strongly compromised causing the jumpy character,
limited Field of View (FOV) and invalid zones.
Holographic systems generate holographic patterns to
reconstruct the wavefront. These systems suffer
fundamental limitations on realistically achievable image
sizes and viewing angle.
 3. HOLOVIZIO™ SYSTEM
The patented HoloVizio™ technology [1][3] uses a
specially arranged array of optical modules and a
holographic screen. Each point of the holographic screen
emits light beams of different color and intensity to various
directions. The light beams generated in the optical modules
hit the screen points in various angles and the holographic
screen makes the necessary optical transformation to
compose these beams into a perfectly continuous 3D view
(see Figure 1). With proper software control, light beams
leaving the pixels propagate in multiple directions, as if they
were emitted from the points of 3D objects at fixed spatial
locations. The direction selective light emission is a general
requirement for every 3D system and provides quantitative
data on the FOV, on the angular resolution, determining
field of depth of the displays, affecting the total number of
light beams necessary for high-end 3D displaying.
Figure 1. HoloVizio™ principle
The HoloVizio™ technology can be used to build both
small-scale and large-scale display systems. Different
prototypes have been developed in European projects, most
recently a medium sized desktop system and a wall
projected cinema system is being developed in the FP6
OSIRIS project.
4. 3D IMAGE COMPOSITION - LIGHT FIELD
RECONSTRUCTION
There are multiple ways to define a 3D image. From the
point of view of the human vision system, a static 3D image
can be considered as the collection of light rays entering the
two eyes (which the brain interprets exploiting the binocular
parallax). Similarly, from the displaying point of view a 3D
image can be considered as the light rays leaving the 3D
scene. See Figure 2. We will refer to this representation of
3D as the Light Field representation of the scene (after
Levoy at al. [6]).
Figure 2: Light Field reconstruction
5. 3D VIDEO FORMATS
A major question when addressing 3D visualization is how
to represent 3D information. When answering this question
it must also be considered how 3D content can be created or
acquired; how this information is to be transferred and
stored; and finally, how to use this information for fast
displaying.
There are several ways to represent 3D. It is convenient
to classify representations on the basis on how much image
information and how much geometry information is stored
within the data (as advised by Smolic et.al. [2]). Image
information based representations define the color of
specific light rays connected to the capturing cameras, while
geometry information based representations describe the
object of the scene using some modeling language. If a 3D
image is considered as the collection of rays emitted from
the scene, both extremities of these representations must be
converted to light rays. Properties of some popular
representations are discussed.
Polygon models and Surface models
These formats are the native representation of many 3D
computer applications, ranging from 3D games to CAD
software and 3D graphical and animation studios.
Although many formats fall into this category, it is
common that they represent 3D through defining geometry
information of object or object surfaces. Image information
appears implicitly through light source models, materials
and textures. Although these models hold the most
consistent 3D information, to create realistic looking scenes
detailed models are required, hence these format can be
large. 3D videos are created by animating scene objects. In
practice this is done through many hours of graphical work
or through using a motion capture system.
Volumetric data
This representation of 3D is classically used by medical
applications, but a large array of 3D measurement devices
also uses volumetric representation to store raw
measurements. This representation classifies units of 3D
space based on the object properties occupying that space.
Since this representation assigns color information to the
geometry defined by voxels, these formats can be used
similarly to the model based formats. Depending on the
space partitioning, this format tends to be large, and too
inflexible to be considered for 3D video applications.
Multiple images
This representation is purely image based, and can be
considered as a collection of light rays entering the cameras.
However, if we want the set of rays leaving the scene,
corresponding the two set of rays can be troublesome, and
may require computer vision algorithms. (See interpretation
of these formats in the next section.) On the other hand,
collecting image information (due to the availability of high
quality digital cameras) is much simpler than creating
realistic 3D models. The size of data represented in this
format depends on the number of images, but since image
information is highly redundant, the application of
compression techniques is beneficial.
Multiple images combined with depth data
Augmenting image information with geometry information
can mean the optimal solution for 3D video systems. If we
store depth information together with the set of rays (see
Figure 3.), the missing light rays can be calculated with
simple algorithms, while quality degradation is kept
minimal. This can mean that the number of image streams
can be reduced. Further benefit of this representation is the
compressibility inherited from the multiple image based
formats.
The price that must be paid is on the acquisition side.
The depth information must be calculated either from the
images themselves using computer vision applications or
using special hardware. The consistency of the 3D image
relies on the depth map quality.
Figure 3: N-View + N-Depth-map representation of the
Breakdancer [10] dataset
6. 3D VIDEO PLAYBACK
Understanding the concepts of 3D image composition on
large FOV Light Field reconstruction systems, it is plain that
the number of calculated light rays is a multiple of that of
2D systems. Also, the viewpoint directions of calculated
light rays must be on a very fine grid, to address all
necessary light rays. Extraction of this information from 3D
video formats requires significant graphical and in some
cases general processing power. This processing power is
provided in the HoloVizio™ systems by a computation
cluster, but the major properties (GPU, CPU, network,
storage) of the cluster must depend on the representation.
The considered representation cases are described below.
Polygon animations
Displaying polygon animations requires fast rendering of
the frames of the 3D scene from multiple viewpoints. In
general, ray-tracing would offer a solution to create the
LightField. However, since the computation requirements of
the algorithms involved is high, this could only yield an off-
line solution. In graphics applications however, to achieve
good images quality with adequate speed, polygon based
contents viewers exploit the capabilities of the graphics
hardware through the tools provided by graphics libraries
(OpenGL, DirectX). Consequently, real-time displaying of
3D model based contents may be possible through the
replication of the rendering process utilized by these
viewers.
The HoloVizio™ software pack provides an OpenGL
Wrapper [3] for viewers using this graphics library, which
intercepts function calls of the viewer, and attempts to
replicate the rendering process, from multiple viewpoints.
To achieve real-time displaying of such content, a strong
graphics render cluster is required. The HoloVizio™
software pack also provides a Modelviewer application that
is capable of displaying widespread model based formats.
Multiple image based representations
Depending on the number of image streams, multiple image
based representations can provide enough light rays for
highly realistic 3D image composition. If the number of
streams is high (70-120) and well distributed over the
displayed FOV, the conversion to Light Field is a simple
interpolation task. However, this data format is large and the
conversion may be slow, simply because a huge data set
must be handled. Compression techniques are the key to this
solution. A preprocessed image based data set, compressed
by standard 2D video compression (XviD) containing 50
million light rays has been shown to play with 24 frames per
seconds (see Figure 4).
Multiple image based representations using lower (7-11)
number of images do not contain enough light rays, instead
they carry implicit 3D information. CPU intensive
reconstruction [9] and image based rendering algorithms [8]
are required to calculate the missing viewpoints. Free
Viewpoint Video techniques [7] can yield fast solutions, but
since in this case multiple viewpoints must be calculated
simultaneously, real-time playback on a reasonable cluster
size is not yet feasible. Fast graphics based solutions are
currently under research by Holografika.
Combined multiple image plus depth based
representations
This 3D representation carries both color and geometry
information of the scene, and can be converted into Light
Field using a graphics processor intensive method. This
method is based on standard rendering techniques, but
integrates all information provided by the streams. A fast
GPU implementation has been developed by Holografika,
taking advantage of the graphics processing power of a
render cluster. The current implementation has reached 4
frames per seconds for the 8 stream Breakdancers [10]
dataset using a 16 node cluster using NV6600 GPUs. Using
NV8800 graphics cards we improved the rendering to 30
FPS. The small size of this representation makes it ideal for
3D data transmission.
7. APPLICATIONS
The applications for 3D video playback are emerging
together with the availability of 3D systems. 3D video
playing in connection with the HoloVizio™ technology is
being applied within the European Project OSIRIS, to create
a 3D desktop and a 3D Cinema system capable of
visualizing 3D videos. Within this project a 3D acquisition
system is also being built to provide natural 3D content for
playback. Both display systems feature high quality
continuous view 3D visualization. Further applications of
the developed technologies are foreseen.
Figure 4.: Natural 3D content displayed on HV640RC
8. CONCLUSIONS
In this paper the HoloVizio™ system is shown to be a
device capable of 3D video visualization. According to our
experiments, several 3D representations are appropriate for
visualization on high FOV 3D displays. It has been shown
that the HoloVizio™ system is capable of displaying 3D
video formats in different representations. Based on the
results, the most appropriate representation is the combined
image plus depth based representation, but other
representations must not be neglected for either quality or
accessibility reasons. The discussed methods are useful for
many applications including EU projects.
9. ACKNOWLEDGMENTS
This work has been supported by EU IST-FP6 Integrated
Project OSIRIS (IST-33799 IP).
10. REFERENCES
[1] T. Balogh, T. Forgacs, T. Agocs, E. Bouvier, F. Bettio, E.
Gobbetti and G. Zanetti: A Large Scale Interactive
Holographic Display. IEEE VR2006 Conference, March 25-
29, Alexandria, Virginia, USA
[2] Smolic, A.; Mueller, K.; Merkle, P.; Rein, T.; Kautzner, M.;
Eisert, P.; Wiegand, T., “Free viewpoint video extraction,
representation, coding, and rendering”, International
Conference on Image Processing, Volume 5, Issue , 24-27
Oct. 2004
[3] T. Balogh, T. Forgacs, T. Agocs, O. Balet, E. Bouvier, F.
Bettio, E. Gobbetti and G. Zanetti: A Scalable Hardware and
Software System for the Holographic Disp of Interactive
Graphic Applications. EuroGraphics 2005, Dublin.
[4] Cees van Berkel, David W Parker and Anthony R. Franklin:
Multiview 3D-LCD. In Stereoscopic Displays and Virtual
Reality Systems III (1996), Vol. 2653 of SPIE proceedings,
pp.32
[5] G. J. Woodgate, D. Ezra, J. Harrold, Nicolas S. Holliman, G.
R. Jones, R. R. Moseley: Autostereoscopic 3D display systems
with observer tracking. In Image Communication – Special
Issue on 3D Video Technology (EURASIP - 1998), pp.131
[6] M. Levoy, and P. Hanrahan, “Light Field Rendering”, Proc.
ACM SIGGRAPH, pp. 31-42, August 1996.
[7] C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R.
Szeliski, “High-quality video view interpolation using a
layered representation” ACM SIGGRAPH and ACM Trans.
on Graphics, Los Angeles, CA, Aug. 2004, pp. 600-608.
[8] A.W. Fitzgibbon, Y. Wexler, A. Zisserman, “Image-based
rendering using image-based priors”, Proc. International
Conference on Computer Vision, October 2003.
[9] Z. Megyesi, G.Kós and D.Chetverikov, "Dense 3D
Reconstruction from Images by Normal Aided Matching",
Machine Graphics & Vision, vol.15, pp.3-28, 2006.
[10] http://research.microsoft.com/vision/InteractiveVisualMediaG
roup/3DVideoDownload/