Sunteți pe pagina 1din 3


SPIE: Telemanipulator and Telepre-

Applying Mixed sence Technologies, vol. 2351, Society

of Photo-Optical Instrumentation
Engineers, Bellingham, Wash., 1994,
pp. 282-292). We expand this contin-

Reality to uum by incorporating the mixed-real-

ity participants imagination to the


To map our mixed-fantasy frame-
work, we constructed the three-sided
diagram shown in Figure 1. Using
Christopher Stapleton, Charles Hughes, Michael Moshell, Milgrams spectrum as the top edge,
Paulius Micikevicius, and Marty Altman we added two new facets: P.T.
University of Central Florida Barnums reality-imagination contin-
uum on the left and Aristotles media-
imagination continuum on the right.

he lack of compelling content
has relegated many promising Mixed-reality technologies
entertainment technologies to combine virtual objects with
laboratory curiosities. Al-
though mixed-reality tech- the real world to suspend
niques show great potential, the enter-
tainment business is not about
disbelief and engage audiences
technology. To penetrate these huge in a rich fantasy experience.
markets, MR technology must become
transparent for the content to have full
effect. The Past Is No Illusion, Digital We then made imagination the frame-
To achieve this goal, we have devised Illusion, Entertaining the Future with works third anchor point. We can
a framework that lets us integrate con- High Technology, Addison-Wesley, locate many media phenomena along
cepts from disparate areas such as Reading, Mass., 1997, pp. 49-57). the continuum edges, with the ultimate
theme parks, theater, and film into a When devising such effects, inventors sweet spot for mixed-reality entertain-
comprehensive research methodology. often undervalue artistic convention and ment applicationsmixed fantasy
We believe that our framework, which methodology. Their focus on technology lying in the center of these phenomena.
has already helped us create content sometimes gives them tunnel vision so A successful mixed fantasy should
for MR entertainment systems, can severe they see only a fraction of the pos- thus incorporate strong doses of phys-
provide these benefits to other devel- sibilities. Critical technical advance- ical reality with virtual elements that
opers as well. ments such as video see-through head- lead the imagination, and it should
sets can provide artists with the means underpin these elements with a central
MIXED-REALITY CONTINUUM to apply more than 100 years of media story built on the converging medias
Fantasy and reality meet effectively conventions to augmented-reality pro- artistic conventions. To develop a
in film-based thrill rides such as those jects. This technology lets them build framework for understanding how to
seen at Universal Studios and similar fantasies that go beyond the simple over- tell stories and entertain people in
attractions. The art of mixing reality lay of texts and graphics in space. mixed-reality environments, we draw
and fantasy through technology began Paul Milgram first introduced the upon continuums demonstrated by P.T.
long before Ivan Sutherlands Sketch- concept of mixed reality as a spectrum Barnum and Aristotle.
pad gave birth to virtual and augmented that extends from real to virtual expe-
reality in the 1960s. This was demon- riences, with augmented reality and Connecting reality
strated in the first hybrid ride-and-film augmented virtuality bridging the two and imagination
attraction, the Cinerama Air-Balloon (Paul Milgram et al., Augmented P.T. Barnum exemplifies modern
Panorama, which appeared more than Reality: A Class of Displays on the showmanship. He knew how to struc-
a century ago in Paris (Walt Brandsford, Reality-Virtuality Continuum, Proc. ture peoples expectations and excite

122 Computer
their imaginations so that they would
perceive ordinary objects in extraordi- Milgrams reality-virtuality continuum
nary waysand willingly pay for the Augmented virtuality (AV) Augmented reality (AR)
privilege. Peoples desire to believe is so
great, this type of magician can trans-
form the perception of reality to his will
Physical Virtual
by painting with the audiences gener- reality
ous imagination. The theme park P.T. Barnums (the world) (the medium) Aristotles
design industry is todays principal reality- media-
imagination Compelling imagination
reservoir of this skill for bending the mixed reality
continuum continuum
audiences perception of reality. Its (the play)
methods, while taught in very few aca- Film
demic curricula, are critical. Traditional
theme park
Connecting media reality Novel
Magic show (the story)
and imagination
After the spoken word, the stage
provided the first entertainment
medium, one populated by costumed Figure 1. Mixed-fantasy framework. Building on Milgrams continuum, which spans aug-
actors representing imaginary charac- mented and virtual reality, this framework adds one continuum for enhancing reality with
ters. Aristotles Poetics sought to struc- participants imagination and another for enhancing media with the audiences imagina-
ture these experiences by using the tion. Imaginary reality thus provides the frameworks anchor.
authors intent to excite the audiences
imagination, encouraging them to cre- science to blend the real and virtual matic matte shots, the orchestration of
atively supply the storys unseen parts. worlds. We can then use the story- middle ground, foreground, and back-
From this basis grew todays theatrical, telling process to heighten the audi- ground expands the illusion of immer-
film, and television industries. ences perception, trigger imagination, sion. The fidelity of the imagery can
As Aristotle pointed out when dif- and transcend augmented realitys cur- vary, but the tracking and registration
ferentiating the historian from the rent limitations. Projects such as must be precise or the illusion shatters.
poet, it is not the function of the poet Canons AquaGauntlet first applied Accurately placing composites of
to relate what has happened, but what this mixture to AR technology in the roaming virtual objects and avatars in
may happen. The storyteller uses this form of video games. both the foreground and background
what-if factor to spark the audiences Technology based on the companys in the physical set can be difficult when
imagination into providing a fantasti- Coastar head-mounted display system using only a head-tracking system. Any
cal realitya truth that led Ernest provides our mixed-fantasy systems errors in head tracking produce exag-
Hemingway to observe that the major- key component. Twin video cameras gerated parallax errors due to the win-
ity of any story exists beyond the mounted on the HMD and aligned dow frames screening effect. Using a
page, and depends on contributions with the users eyes feed real-world blue screen within a series of portals
from the readers imagination. imagery to an image-processing sys- windows or doorssupports real-time
tem. The image processora pair of matting that masks imprecise tracking.
Barnum versus Aristotle dual-processor PCsthen adds virtual The parallax then actually accentuates
Of these two approaches, Barnums imagery and feeds it back to the HMD. the dimensional illusion.
places users in the more active role, This video see-through technology For matting purposes, we have
while Aristotles stresses passive par- articulates the virtual aspects of an found that searching for a range of col-
ticipation. Yet both approaches can be entertainment productsuch as char- ors is insufficient, whether working in
used to construct fantasy worlds in acters, props, or effectsand blends RGB or YUV color space. While such
which the user validates fantasy with them in real time to match physical an approach works well in television
reality through imagination and sus- reality. For group viewing and interac- and film studios with high-quality cam-
pension of disbelief. tion, video-based technology also eras and precisely controlled lighting,
enables display options beyond the sin- working under entertainment condi-
BUILDING MIXED FANTASIES gle-user HMD viewpoint. tions with HMD cameras produces
Theatrical designs multimodal artis- poor results. For example, only part of
tic conventions can work with the Blue screen matting the blue screen might be interpreted as
algorithmic techniques of computer From proscenium portals to cine- blue, and that portion could vary from

December 2002 123

Entertainment Computing

frame to frame. We achieved stable and duce a complex series of real-world create a time-ordered record of all
consistent matting only by seeking a sights, sounds, and other sensations events that lets us monitor user actions
range of color ratios. that help validate the 360-degree illu- and reason about system behavior.
Depth-mapping in real-time Using macrostimulator events in a
Using video see-through HMDs, mixed-reality situation can jar the he successful adoption of new
graphics hardware can occlude virtual
objects with real-world image pixels
and replace real objects with virtual
viewers sensibilities in useful ways. A
virtual adversary shoots out real lights,
or its movements cause shutters to fly
T technologies for entertainment
applications depends on finding
creative models that spark the imagina-
ones if depth can be computed accu- open with a floor-shaking thump. tion and generate demand. Developers
rately and quickly for real entities. When all our senses validate a virtual must then apply these creative conven-
Fortunately, the images captured by the event, the experience moves us across tions to diverse business models, includ-
Coastar HMD can be used to compute a credibility threshold. ing theme parks, arcades, museums,
such depth from stereo coordinates Audio immersion also emphasizes and infotainment. MR Systems Labora-
(Akinari Takagi et al., Development events. The hybrid systems of surround tory, Canon, US Army Stricom, and the
of a Stereo Video See-Through HMD sound, point-source display, and audio- National Science Foundation all sup-
for AR Systems, Proc. IEEE/ACM haptic effectswhich produce sounds port work such as ours at several lead-
Intl Symp. Augmented Reality 2000, you can feellayer the audioscape and ing entertainment companies, providing
CS Press, Los Alamitos, Calif., 2000, stimulate the peripheral senses. an environment in which technical and
pp. 68-77). However, doing so for each creative innovation can act as equal
pixel requires a computationally inten- Homogeneous object picking partners.
sive process that yields unacceptable The video see-through HMD makes
frame rates. Rather than computing the it possible to use a simple laser pointer
depth for all pixels, our extensions to to select and interact with real or vir- Christopher Stapleton is a director of
Canons MRPlatform API compute the tual objects uniformly. The video sys- Entertainment Research at the Uni-
depth for selected object markers, tem determines the objects being versity of Central Floridas Media Con-
which are detected by color. By restrict- selected without the need to first build vergence Laboratory. Contact him at
ing the search space, we achieve the a comprehensive 3D model or prewire cstaplet@
speed required for real-time rendering. an entire space with sensors. The sys-
Additionally, our algorithms enhance tem models and tracks only those Charles Hughes is a professor of com-
the detected color regions, the bound- objects relevant to the scenario. Using puter science at the University of Cen-
aries of which may vary with time. macrostimulator technology lets play- tral Florida. Contact him at ceh@cs.
While the size of the color region ers cause changes to physical objects,
remains approximately the same, its thus diminishing the conceptual dis-
center tends to shift, resulting in varia- tance between the real and the virtual. Michael Moshell is a professor of com-
tions of the computed depth. Down- puter science and director of the Digi-
sampling the images dramatically Scripting a deep experience tal Media Program at the University
reduces this variation. Using the down- To incorporate artistic collabora- of Central Florida. Contact him at
sampled images for depth computation tions that make it easier for nontech-
only avoids adversely affecting the dis- nical developers to create mixed
played videos quality. realities, alter them dynamically, and Paulius Micikevicius is a postdoctoral
monitor them as they occur, we devel- research associate at the University
Special effects oped a scripting environment that con- of Central Florida. Contact him at
When creating a mixed-fantasy trols all aspects of the mixed-fantasy
experience, the physical world must be experience. This system communicates
controlled along with the virtual. with the graphics engine, audio clients, Marty Altman is a research associate
Theme parks apply audio, special and macrostimulator clients using at the University of Central Floridas
effects, action equipment, and scenery Gilderfluke show-control devices. Institute for Simulation and Training.
to manipulate physical reality. This cre- We use this system to develop a Contact him at
ates a tactile and visceral impact that structured approach to storytelling in
can expand the users scope of percep- mixed reality. In doing so, we seek to Editor: Michael Macedonia, Stricom,
tion beyond the limits of visually based blur the distinction between the real 12350 Research Parkway, Orlando, FL,
mixed-reality devices. Our system and virtual worlds still further. A cen- 32826-3276,
incorporates macrostimulators to pro- tral story-control system also lets us

124 Computer