1

TELE IMMERSSION
P VISHNU SUBRAMONIAN
Ist Semester M.Tech, Applied Electronics and Communication System, Nehru College of Engineering and Research Centre, Pampady, Thrissur.

ABSTRACT Tele presence and tele immersion systems are gro!ing to "e more realistic these years. #o!e$er, fe! sho! practical e%periences in Internet !ide tele immersion, especially in accepta"le end to end delay.A frame!or& for multi party tele immersion applications that allo! remote colla"orati$e '( interactions. It pro$ides an architecture that supports se$eral tele immersion nodes to communicate in groups. This paper addresses ho! to reconstruct, compress and esta"lish the system. The system has "een set up in China N)I. Preliminary e%periment results sho! that it*s possi"le to ha$e the delay under +,,ms !ith '-,.-/, depth map and 0/,./1, te%ture images. The frame!or& is also e%pected to "e further e%tended for interaction !ith full "ody reconstruction. Tele immersion ena"les users at geographically distri"uted sites to colla"orate in real time in a shared, simulated, hy"rid en$ironment as if they !ere in the same physical room. The considera"le re2uirements for tele immersion system, such as high "and!idth, lo! latency and lo! latency $ariation ma&e it one of the most challenging net applications. This application is therefore considered to "e an ideal dri$er for the research agendas of the Internet-. Tele immersion is that sense of shared presence !ith distant indi$iduals and their en$ironments that feels su"stantially as if they !ere in one3s o!n local space. Tele immersion differs significantly from con$entional $ideo teleconferencing.

INTRODUCTION
Tele immersion is referred to as adding some &ind of interactions to the simply tele presence so that one can 4touch4 the shared $irtual en$ironment "esides merely the '( reproduction of the human "ody. Modern tele immersion technology is mainly "ased on multi $ie! images captured from cameras in different directions to an o"5ect space. It conducts real time '( reconstruction from the images and then deli$ers the model !ith te%tures to the rendering or other distri"uted sites $ia net!or&. All the !or& need to "e done in one frame to o"tain real time '( perception. 6"$iously, this technology is a multi disciplinary topic related to computer $ision, graphics, net!or& and real time data processing etc. Tele immersion reconstructs the o"5ects in the real !orld in real time, as starts a ne! &ind of 7irtual Reality interaction that mi%es the real o"5ects !ith the $irtual en$ironment. Some !or& has "een done in the tele immersion field. Though, fe! sho! practical e%periences in Internet !ide tele immersion to no!, especially those of accepta"le end to end delay. There are t!o difficulties inside tele immersion. They all lie on the processing time that affects the end to end interaction delay. 6ne is the efficiency of '( content generation including reconstructing the mesh and e%tracting8compressing the high resolution te%ture. The other is the transportation efficiency since the si9e of the models refreshing se$eral times each second !ill "e $ery "ig. A real time stereo $ision "ased tele immersion frame!or& is presented in this paper. There are t!o main contri"utions: first, the frame!or& designs the communications among the users and the TI ;tele immersion< ser$er that pro$ides a platform for remote '( colla"oration in groups= second, !ith our techni2ues, !e de$eloped the system and pursued tele immersion e%periments o$er the Internet, !hich may "e useful for related reference. Some of our techni2ues in the system are also introduced including the stereo $ision "ased '( matching and There e%ist some !ell &no!n multi camera "ased tele immersion systems, such as )rImage and successi$e 7gate "y INRIA ,TI and TEE7E systems "y >.C.?er&eley, >.C.(a$is and >I>C, S7TE "y the #einrich #ert9 institute . These systems all reco$er the '( information from multi $ie! cameras, ma&e realtime '( reconstruction, and then apply the results to the mi%ed reality en$ironment. The real time reconstruction plays the &ey role in this process since the algorithms need hea$y computation to generate the '( mesh and te%ture in a short time no more than ,.@ second. Although the ne!ly coming time of flight ;T6A< cameras are a$aila"le in the mar&et from the BCam ;no! "elongs to Microsoft<, mesa, PM( or Canesta companies, their resolutions are al!ays $ery

lo! to "elo! -,,.-,, as is not sufficient for full"ody interaction. Cith the resolution impro$ement of T6A cameras, it !ill "e sure to change the traditional reconstruction algorithms. The real time reconstruction algorithms are mainly of t!o &inds. 6ne is the stereo $ision "ased reconstruction !hich calculates the depth of pi%els "ased on the '( matching of the pi%els in t!o images. The other is "ased on the structure from silhouette ;SAS< in !hich $isual hull algorithm is no! a hot topic. The stereo $ision "ased reconstruction adapts to $arious of o"5ect shapes, "ut it has to compromise in the matching precision and o$erhead !ith different "aseline. The ?er&eley TI, S7TE are "ased on the stereo $ision "ased reconstruction. The $isual hull algorithms can pro$ide good 2uality full "ody out loo&ing in some $ie!points "y calculating the intersection of $ision cones. It has higher computation comple%ity and difficulties in conca$e surfaces. The $isual hull algorithm is more costly "ecause of the re2uirements on the si9e of capture space and num"er of cameras.To no!, little !or& has "een done in the real remote sites e%cept the ?er&eley TI !hich sho!s high traffic and "ig delay in the e%periments. The '( reconstruction includes "oth the stereo$ision "ased and $isual hull algorithms, "ut for no! only the stereo $ision "ased reconstruction is integrated and tested. The $isual hull reconstruction has a lot of traffic in sending multiple te%ture images that "rings a "ig pro"lem to our real time remote communication. All the +, sites are e2uipped !ith ' camera "um"le"ee camera as in figure. The design and preliminary e%periment results are pro$ided in this paper. FRAMEWORK OVERVIEW Cor& of tele immersion is tested in the China N)I to $erify this ne! &ind of remote interaction. Considering the system scala"ility, the TI system is di$ided into the ser$er side and TI client as. The ser$er side includes the TI ser$er, system portal and the data"ase. The TI ser$er pro$ides a $irtual #all and room model to di$ide the users into groups. The system information such as user login8logout and their manipulations are logged in the TI ser$er, and some of them are also !ritten to the data"ase. The portal pro$ides user login and real time status display through !e" "ro!sers !ith the help of data"ase. The TI client gets the '( user data from reconstruction, stereo "ased or multi $ie! "ased one, and performs data compression, transmission and rendering in the system frame!or&. The frame!or& deals !ith the user interaction, some physical simulation, and the '( rendering.

Aig: frame!or& o$er$ie!

The TI ser$er pro$ides a $irtual #all and room model to di$ide the users into groups. The system information such as user login8logout and their manipulations are logged in the TI ser$er, and some of them are also !ritten to the data"ase. The portal pro$ides user login and real time status display through !e" "ro!sers !ith the help of data"ase. The TI client gets the '( user data from reconstruction, stereo "ased or multi $ie! "ased one, and performs data compression, transmission and rendering in the system frame!or&. The frame!or& deals !ith the user interaction, some physical simulation, and the '( rendering. The TI client has some "ig pro"lems inside such as the real time '( reconstruction and a scala"le communication model. T!o types of real time '( reconstruction are implemented in the system including the stereo "ased reconstruction and the silhouette "ased $isual hull reconstruction. Aor the stereo "ased reconstruction, the remote client only needs one image per frame to perform te%ture mapping "ecause actually the real time model is "ased on a depth map. #o!e$er, for the $isual hull modeling, the images of all the directions are re2uired for the full "ody mesh.Almost 1 +- cameras co$ering a space, so that images from all these cameras are all re2uired in each frame. It*s a hea$y "urden to the !hole system. Aor no!, only stereo "ased reconstruction is implemented in our system. More efficient multi $ie! "ased te%ture compression algorithms are still under in$estigation since current schemes are not efficient in real time situations. In a typical multi party tele immersi$e en$ironment li&e '( $ideo conference, each one sends and recei$es meshes and corresponding te%ture images to8from se$eral other ones. It !ill incur hea$y traffic and then affect the system scala"ility and performance. A communication model is designed as for the tele immersion, in !hich the TI ser$er

coordinates the client communication. The TI ser$er lists the user clients in groups each of !hich is associated !ith a $irtual room. Chen a ne! client logins, the TI ser$er !ill push the list including all the other clients in its group to it. The client !ill get connected !ith all the other partner clients $ia TCP or >(P. In this !ay, the clients !ill communicate their models, te%ture images and sound in real time. A test !as conducted of this model !ith $isual hull "ased model. The TI clients !ill e%change the full "ody meshes !ith the selected one or t!o te%ture images according to their $ie!points. Preliminary e%periment results sho! that the performance slo!s do!n hea$ily due to the traffic and delay. Aurther !or& on the reconstruction and compression algorithms need to "e conducted to meet the interaction re2uirements.

Aig: ser$er client model

(ifferent !ith the mesh8te%ture data, the user interactions including the login8logout, manipulation are for!arded "y the TI ser$er. The TI client !ill launch one specific thread to do!nload such initiali9ation data as '( scene resources and user group lists. The interactions are al!ays re2uired to "e relia"le deli$ered, so the connection "et!een the TI client and the ser$er is designed to "e TCP. 6n the other hand, the '( data including the meshes and te%ture images is updated in real time. Chen some '( frames are dropped, it*s accepta"le. #ence, the '( data is possi"le to "e transmitted in unrelia"le channel such as >(P.

Generating the 3 D I!age" +.An array of cameras $ie!s people and their surroundings from different angle . Each camera generates an image from its point of $ie! many times in a second. -. Each set of the images ta&en at a gi$en instant is sorted into su"sets of o$erlapping trios of images. '.Arom each trio of images, a Ddisparity mapE is calculated, reflecting the degree of $ariation among the images at all points in the (isparity map $isual field. The disparities are then analy9ed to yield depths that !ould account for the differences "et!een !hat each camera sees. These depth $alues are com"ined into a D"as reliefE depth map of the scene. /. All the depth maps are com"ined into a single $ie!point independent sculptural model of the scene at a gi$en moment. Process of com"ining the depth maps pro$idemaps opportunities Aor a "etter e%ploring of the en$ironment a stereoscopic $ie! is re2uired. Aor this, a mechanism for '( en$ironment scanning method is to "e used. It is "y using multiple cameras for producing t!o separate images for each of eyes. Polari9ed glasses are used to separate each of the $ie!s and get a '( $ie!. REAL TIME STEREO BASED RECONSTRUCTION Real time reconstruction is the critical algorithm in teleimmersion systems. It*s almost e2ually important to ha$e the !ell reconstructed models and to "e real time to meet the user interaction re2uirements. Aor the real time stereo "ased reconstruction, it captures t!o images !ith disparity in each frame, ma&es stereo matching to calculate the depth map and then gets the mesh through triangulation. The stereo matching algorithm is the main part that affects the accuracy and efficiency of the reconstruction. A real time stereo matching algorithm RealTimeFAC is de$eloped in the system that estimates scene depth information in real time. The stereo algorithm consists of t!o steps. Airst, the color similarities are used to compute a layered support !eight for each pi%el and to aggregate matching costs. )i$en a stereo image pair GI , I H , !here I and I are the reference and target images, the initial matching costs are computed using method .The e2uation is: the a"solute difference

!here IC and Ic are the color intensities in the left and right images. Sym"ol p denotes the pi%el ;x, y< in the left image, and d is the disparity hypothesis. The corresponding pi%el p in the right image is the one ;x+d, y<. Ce "uild a s2uare !indo! Wp of predefined si9e centered at each pi%el p. Aor each pi%el qWp, a !eighting function w(p,q) that computes a !eight representing the li&elihood to !hich pi%el q lies on the same surface !ith pi%el p. Second, !e use an amended scan line optimi9ation techni2ue !hich com"ines !inner ta&e all ;CTA< and dynamic programming ;(P< to compute disparities. ?y com"ining CTA and scan line (P, our approach can compute disparity of pi%els in depth discontinuity areas. The formula is:

!here p * I ;x +, y< , d ; p< is the disparity function of pi%el p. Sym"ol g is a constant used to penali9e depth discontinuities. The e%perimental results are e$aluated on the Middle"ury "enchmar& data sets, sho!ing that to no! RealTimeFAC achie$es the "est reconstruction accuracy among e%isting realtime stereo algorithms.

REAL TIME COMPRESSION To o"tain natural and prompt interaction, TI nodes need to get the '( models in time. The '( models include mesh triangles and te%tures. Te%tures are from the foreground pi%els, so the si9e is $ery "ig. Fi&e other image compression, real time compression is essential for real time deli$ery. A lossy compression scheme for the depth data and a lossless one for the color data in tele immersi$e en$ironments, "ut those are frame independent compression, so the compression ratio is limited. In fact, as the tele immersi$e en$ironments only care a"out theforeground o"5ects, the $alid te%tures only e%ist in the foreground pi%els. A proposed ?order descript Inter Arame compression ;?IAC< scheme, !hich applies inter frame motion stimation !ith the restriction of foreground o"5ect pi%els. The scheme has "een integrated to the ?er&eley TI system. Reconstruction is "ased on stereo modeling, so each $erte% of mesh is defined in a three dimension space ;%, y, disparity<. Its corresponding te%ture coordinates are set in t!o dimensions as a normali9ed form ;%8!idth, y8height<. Then, for each part of the stereo model, one frame is enough for the te%tures of the part.This means the te%ture mapping can "e conducted !ith )FJTRIAN)FES rendering in arrays. Aor the depth map, it3s easier to pursue fast compression. Aor e%ample, !e use the run length encoding ;RFE< algorithm for depth map compression. The compression ratio is usually a"out K:+ in our e%periments. Similar to all &inds of image compression, at the first stage the images are di$ided into macro "loc&s to increase the compression speed. The foreground mas& !ill also "e deri$ed after "ac&ground su"traction during reconstruction. Three types of "loc&s are defined in our scheme, A for total foreground, N for "ac&ground and B for ?order i.e. not all the pi%els inside foreground. #ere the "order "loc& type B is e%tra defined compared !ith other "itmap "ased methods.

Aig: compression techin2ue

According to the mas& in pi%el, all the "loc&s can ha$e their types. Fet MLw, hM "e the original mas& of !idth w , height h . Fet Macro ?loc& has the si9e d . d . (efine mas& as MLx, yM , ;i +< . d N x O i . d , , N i O w , ; j +< . d N y O j . d , , N j O h . Fet M *Li, jM "e the ne! mas& in "loc&s. Ce ha$e If all MLx, yM II , , then M *Li, jM I N .If all MLx, yM II +, then M *Li, jM I A . If exists any MLx, yM II , and any MLx, yM II + , then M *Li, jM I B . The inter frame compression uses reference frames for motion estimation. A group of frames "egins !ith the reference frame and then the P frames. The reference frame is compressed !ith PPE) compression, called intra frame compression. A modified PPE) compression is implemented !hich only compresses the "orderand inside "loc&s i.e. "ac&ground "loc&s are ignored for no usage in te%tures. The intra frame compression consists of color channeltransform, (CT ;discrete cosine transform<, 2uantification, RFE and #uffman coding. The reference frame3s encoding structure isa se2uence of macro "loc& type, mas& encoding and the image encoding data. The intra frame compression !ith mas& adopted is not simply searching line "y line. In stereo "ased modeling, the angle "et!een the $ie! directions of the t!o cameras is not "ig and the user !on3t act !ith large mo$ement in most of the time. So the "order set is used for a 2uic& search "ecause the "order "loc&s ha$e more distinct feature of ha$ing "oth foreground and "ac&ground pi%els. In this !ay some estimation !ith direction could "e applied. Count starts from the first "order "loc& that is found during scanning and then search "order "loc&s in its Q neigh"or "loc&s. Assume the $alid "order "loc&s is k. Fet the si9e of search !indo! "e ! !here is the edge length of the !indo!. Then search in the search !indo! of reference frame for the k+" "order "loc&s !ith the smallest de$iation. The "loc& de$iation is calculated "y the regular e2uation:

Then the mo$ing $ectors can "e o"tained from the paired "loc&s. The initial M# is set to "e the a$erage $alue of all the first se$eral a$aila"le "loc& $ectors. The TI client hosts are of / core Intel Reon C'@', -.1)#9, 0)? ((R' +'''M#9 ECC memory, -.@,,)? K-,,rpm SATA hard dis&, and N7I(IA )eforce )TR /K, display card. They are all running !ith !indo!s RP. The depth map is set to the resolution of '-,.-/, during the reconstruction, and the te%ture images ha$e the resolution of 0/,./1,. Normally

10

the time cost of '( reconstruction S encoding ran&s No.+ source of delay than the peer to peer net!or& deli$ery. The time &eeps less than 1,ms !hich $aries !ith the si9e of the foreground o"5ect. Chen the foreground o"5ect occupies a small portion, the time is smaller e$en to less than @,ms. The results of "and!idth occupation !hen ' teleimmersi$e users in a $irtual en$ironment. The client "and!idth maintains a"out 1M" and the model update speed is a"out +/ times per second. The results of the time cost of '( model decoding and rendering. The time cost also $aries !ith the si9e of foreground o"5ect. The time cost of this period is a"out 0 +- ms. Conse2uently, considering the time cost of '( reconstruction S encoding and that of decoding S rendering, the total system delay is possi"le to "e under +,,ms !hen the delay among the remote places are small. The delay under +,,ms is e%cellent for this &ind of Internet colla"oration. Tele presence and tele immersion systems are gro!ing recently. Tele immersi$e system o$er CN)I is presented in this paper including "rief introductions to the system frame!or&, real time '( reconstruction and compression algorithms. The frame!or& is designed considering the communication model and scala"ility to support future multi party communication in groups at the same time. The '( reconstruction has to deal !ith the accuracy and efficiency together. The idea of '( compression algorithm ispresented here !hich is also implemented in our pre$ious !or&. Preliminary e%periment results sho! that it*s possi"le to ha$ethe delay under +,,ms !ith '-,.-/, depth map and 0/,./1,te%ture images. Auture !or& !ill include the Internet !idee%periments, full "ody '( reconstruction, and mi%ed reality interaction etc. E#PERIMENT EVALUATION$ APPLICATIONS Teleimmersi$e holographic en$ironment ha$e a num"er of applications. Imagine a $ideo game free of 5oystic&s, in !hich you "ecome a participant in the game, fighting monsters or scoring touchdo!ns. Instead of tra$eling hundreds of miles to $isit your relati$es during the holidays, you can simply call them up and 5oin them in a shared holographic room. (octors and Soldiers could use teleimmersion to train in a simulated en$ironment. ?uilding inspectors could tour structures !ithout li$ing their des&s. Automo"ile designers from different continents could meet to de$elop the ne%t generation of $ehicles. Surgeons indifferent geographical space could e%periment !ith $irtual medical procedures "efore !or&ing on actual patients. Medical technologies that are physically inaccessi"le in some places could "e used to sa$e li$es "y manipulating $irtual models, instance, offshore oilrigs and ships. In the entertainment industry, "allroom dancers could train together from separate physical spaces.

11

The tele immersi$e system for research communication o$er CN)I has "een de$eloped. Some local and remote e%perimentsha$e "een carried out. The system )>I is as !hich pro$ides tens of $irtual rooms for each &ind of research scenarios!here the users can ma&e tele immersi$e communication. 6ne of the '( tele immersion en$ironment is as !here three users ma&ing discussions on the protein molecule structure. The users can manipulate and interact !ith the $irtual o"5ects !ith their "odies immersed in the $irtual en$ironment. A chess game in !hich people enter the chess en$ironment and play !ith their "odies. Since the real time '( construction produces the dynamic model se2uence, the collision detection algorithms "ased on the discrete time don3t fit !ell and !ill incur the phenomena of penetration. The successi$e collision detection algorithms !ill among one of the future !or& "esides the full "ody '( reconstruction. #ere the preliminary e%periment results are presented.

Aig: Interaction of t!o scientists at different places

12

Aig: Interaction in a chess game

The TI client hosts are of / core Intel Reon C'@', -.1)#9, 0)? ((R' +'''M#9 ECC memory, -.@,,)? K-,,rpm SATA hard dis&, and N7I(IA )eforce )TR /K, display card. They are all running !ith !indo!s RP. The depth map is set to the resolution of '-,.-/, during the reconstruction, and the te%ture images ha$e the resolution of 0/,./1,. CONCLUSION AND FUTURE WORK Tele presence and tele immersion systems are gro!ing recently. 6ur tele immersi$e system o$er CN)I is presented in this paper including "rief introductions to the system frame!or&, real time'( reconstruction and compression algorithms. The frame!or& is designed considering the communication model and scala"ility tosupport future multi party communication in groups at the sametime. The '( reconstruction has to deal !ith the accuracy and efficiency together. The idea of '( compression algorithm is presented here !hich is also implemented in our pre$ious !or&. Preliminary e%periment results sho! that it*s possi"le to ha$e the delay under +,,ms !ith '-,.-/, depth map and 0/,./1, te%ture images. Auture !or& !ill include the Internet !ide e%periments, full "ody '( reconstruction, and mi%ed reality interaction etc.

13

REFERENCES +. )regori5 Turillo , Ramanarayan 7asude$an , Edgar Fo"aton , Ru9ena ?a5csy, A Arame!or& for Colla"orati$e Real Time '( Teleimmersion in a )eographically (istri"uted En$ironmentLCM, Proceedings of the Tenth IEEE International Symposium on Multimedia ;ISM< -. Bhenyu Uang, Canmin Cu, Tlara Nahrstedt, Ena"ling Multi party '( Tele immersi$e En$ironments !ith 7ie!Cast, ACM Transactions on Multimedia Computing, Communications and Applications ;T6MCCAP<, $ol. 0, -,+, '. B#A6 Vinping. A sur$ey on $irtual reality. Series A: Information Sciences. -,,Q, @-;'<:'/1 /,, / ?en5amin Petit, Pean (enis Fesage, Edmond ?oyer, ?runo Raffin. 7irtuali9ation gate. ACM SI))RAP# -,,Q Emerging Technologies, Ne! 6rleans, Fouisiana, -,,Q @. Ramanarayan 7asude$an, Bhong Bhou, )regori5 Turillo, Edgar Fo"aton, Ru9ena ?a5csy, Tlara Nahrstedt. Real Time Stereo 7ision System for '( Teleimmersi$e Colla"oration. International Conference on Multimedia and E%po ;ICME<, Puly +Q -', -,+,, Singapore:+-,1 +-+'