Augmented Reality

1. I N T R O D U C T I O N
This paper describes the current state-of-the-art in Augmented Reality. It describes work performed at many different sites and explains the issues and problems encountered when building Augmented Reality systems. It summarizes the tradeoffs and approaches taken so far to overcome these problems and speculates on future directions that deserve exploration. Section 1 describes what Augmented Reality is and the motivations for developing this technology. Section 2 discusses the issues involved in building an Augmented Reality system. Section3 provides the comparison between the augmented reality and virtual reality. Section4 provides the details about how we can accomplish augmentation by combining the real and virtual worlds. Section5 deals on how augmented reality works. Section6 provides the use of augmented reality in a contact lens, the current progress and the future enhancements. The last section lists the different areas of augmented reality applications. Augmented reality will truly change the way we view the world. Picture yourself walking or driving down the street. With augmentedreality displays, which will eventually look much like a normal pair of glasses, informative graphics will appear in your field of view and audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. In this article, we will take a look at this future technology, its components and how it will be used. Tourists that visit historical sites, such as a Civil War battlefield do not see these locations as they were in the past, due to changes over time. It is often difficult for a modern visitor to imagine what these sites really looked like in the past. A tourist equipped with an outdoors AR system could see a computergenerated version of Living History. Tourists and students walking around the grounds with such AR displays would gain a much better understanding of these historical sites and the important events that took place there. After the basic problems with AR are solved, the ultimate goal will be photorealism has been demonstrated in feature films, but accomplishing this in an interactive application will be much harder.

2. Augmented Reality

E&C Dept.

VVCE, Mysore

Augmented Reality

Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space, Figure shows an example of what this might look like. All these things are optional also i.e. they can be ignored if the user wants some specific details rather than details for everything that comes in its way. AR can be thought of as the "middle ground" between VE (completely synthetic) and telepresence (completely real). That absorbing way of telling a story is called augmented reality, or AR. It promises to transform the way we perceive our world, much as hyperlinks and browsers have already begun to change the way we read. Today we can click on hyperlinks in text to open new vistas of print, audio, and video media. A decade from now--if the technical problems can be solved--we will be able to use marked objects in our physical environment to guide us through rich, vivid, and gripping worlds of historical information and experience. The technology is not yet able to show Dye in action. Even so, there is quite a lot we can do with the tools at our disposal. As with any new medium, there are ways not only of covering weaknesses but even of turning them into strengths--motion pictures can break free of linear narration with flashbacks; radio can use background noises, such as the sound of the whistling wind, to rivet the listener's attention. This survey defines AR as systems that have the following three characteristics: 1) Combines real and virtual 2) Interactive in real time 3) Registered in 3-D

Miligrams Reality-Virtuality Continuum

E&C Dept.

VVCE, Mysore

Augmented Reality

Miligram coined the term Augmented Virtuality to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects.

Figure2.1: Miligrams reality-virtuality continuum

REALITY

AUGMENTED REALITY

VIRTUAL REALITY

Figure2.2: Miligrams reality-virtuality continuum Think of digitally modified reality as a piece of a continuum that begins on one end with the naked perception of the world around us. From there it extends through two stages ofmixed reality (MR). In the first one, the physical world is like the main course and the virtual world the condiment. In the other stage of MR, the virtual imagery takes the spotlight. Finally, at the far end of the continuum lies nothing but digitally produced images and sounds, the world of virtual reality. Any AR system must meld physical reality with computer-modeled sights and sounds, a display system, and a method for determining the user's viewpoint.
E&C Dept. 3 VVCE, Mysore

Augmented Reality

3. DESIGN
A see-through HMD is one device used to combine real and virtual. Standard closed-view HMDs do not allow any direct view of the real world. In contrast, a seethrough HMD lets the user see the real world, with virtual objects superimposed by optical or video technologies.
E&C Dept. 4 VVCE, Mysore

Augmented Reality

3.1 Optical see-through HMD Optical see-through HMDs work by placing optical combiners in front of the user's eyes. These combiners are partially transmissive, so that the user can look directly through them to see the real world. The combiners are also partially reflective, so that the user sees virtual images bounced off the combiners from head mounted monitors. This approach is similar in nature to Head-Up Displays (HUDs) commonly used in military aircraft, except that the combiners are attached to the head. The optical combiners usually reduce the amount of light that the user sees from the real world. Since the combiners act like half-silvered mirrors, they only let in some of the light from the real world, so that they can reflect some of the light from the monitors into the user's eyes. Choosing the level of blending is a design problem. More sophisticated combiners might vary the level of contributions based upon the wavelength of light. For example, such a combiner might be set to reflect all light of a certain wavelength and none at any other wavelengths. This would be ideal with a monochrome monitor. Virtually all the light from the monitor would be reflected into the user's eyes, while almost all the light from the real world (except at the particular wavelength) would reach the user's eyes. However, most existing optical see-through HMDs do reduce the amount of light from the real world, so they act like a pair of sunglasses when the power is cut off.

Figure3.1.1: Optical see-through HMD conceptual diagram

E&C Dept.

VVCE, Mysore

Augmented Reality

Figure3.1.2: Typical HMD Device

3.2 Video see-through HMD Video see-through HMDs work by combining a closed-view HMD with one or two head-mounted video cameras. The video cameras provide the user's view of the real world. Video from these cameras is combined with the graphic images created by the scene generator, blending the real and virtual. The result is sent to the monitors in front of the user's eyes in the closed-view HMD. Figure shows a conceptual diagram of a video see-through HMD. Video composition can be done in more than one way. A simple way is to use chroma-keying, a technique used in many video special effects. The background of the computer graphic images is set to a specific color, say green, which none of the virtual objects use. Then the combining step replaces all green areas with the corresponding parts from the video of the real world. This has the effect of superimposing the virtual objects over the real world. A more sophisticated composition would use depth information. If the system had depth information at each pixel for the real world images, it could combine the real and virtual images by a 12 pixel-by-pixel depth comparison. This would allow real objects to cover virtual objects and vice-versa.

E&C Dept.

VVCE, Mysore

Augmented Reality

Figure3.2.1: Video see-through HMD conceptual diagram 3.3 Monitor based AR AR systems can also be built using monitor-based configurations, instead of seethrough HMDs. Figure shows how a monitor-based system might be built. In this case, one or two video cameras view the environment. The cameras may be static or mobile. In the mobile case, the cameras might move around by being attached to a robot, with their locations tracked. The video of the real world and the graphic images generated by a scene generator are combined, just as in the video see-through HMD case, and displayed in a monitor in front of the user. The user does not wear the display device.

Figure3.3.1: Monitor based AR conceptual diagram

E&C Dept.

VVCE, Mysore

Augmented Reality

4. Comparison against virtual environments

The overall requirements of AR can be summarized by comparing them against the requirements for Virtual Environments, for the three basic subsystems that they require.

4.1Scene generator
Rendering is not currently one of the major problems in AR. VE systems have much higher requirements for realistic images because they completely replace the real world with the virtual environment. In AR, the virtual images only supplement the real world. Therefore, fewer virtual objects need to be drawn, and they do not necessarily have to be realistically rendered in order to serve the purposes of the application. For example, in the annotation applications, text and 3-D wireframe drawings might suffice. Ideally, photorealistic graphic objects would be seamlessly merged with the real environment, but more basic problems have to be solved first.

4.2Display device

E&C Dept.

VVCE, Mysore

Augmented Reality

The display devices used in AR may have less stringent requirements than VE systems demand, again because AR does not replace the real world. For example, monochrome displays may be adequate for some AR applications, while virtually all VE systems today use full color. Optical see-through HMDs with a small field-of-view may be satisfactory because the user can still see the real world with his peripheral vision; the see-through HMD does not shut off the user's normal field-of-view. Furthermore, the resolution of the monitor in an optical see-through HMD might be lower than what a user would tolerate in a VE application, since the optical see-through HMD does not reduce the resolution of the real environment.

4.3 Tracking and sensing

While in the previous two cases AR had lower requirements than VE, that is not the case for tracking and sensing. In this area, the requirements for AR are much stricter than those for VE systems since it is done in real time.

5. Combining the Real and Virtual Worlds

Augmented reality requires that we combine both the real and virtual worlds. As said before augmented reality is a combination of real environment and virtual environment. This was shown in Milligrams reality-virtuality continuum. Accordingly we require combining these two environments in order to obtain the final working condition of the augmented reality.

E&C Dept.

VVCE, Mysore

Augmented Reality

Figure5.1: Components of AR system For combining real and the virtual worlds we need to look into four main aspects. First of all we require precise models for combining these two worlds. They may be the cameras we are using or the graphics generator i.e. the computer that generates the virtual stuff or for that matter the image arithmetics that is taking place in the system. Next aspect is the optical properties of the viewer or the camera and the display. Optical properties must be very good for the proper reception of the viewers scene that he/she is viewing at that point in time. This is required because, when we are using the GPS (Global Positioning System), along with all the other components of the system for knowing the present location of the viewer, a good quality of image is required to be captured from which we are able to detect or locate the user position using the GPS. Another aspect is to combine all local coordinate systems centered on the devices and the objects in the scene in a global coordinate system. It involves combining the coordinates of the viewers camera, the graphics camera and also the coordinates of the virtual objects and the real world objects. By doing so we acquire a final image where both the real world objects and the virtual world objects i.e. the objects generated by the graphics
E&C Dept. 10 VVCE, Mysore

Augmented Reality

system are having the same coordinates or in other words they are perfectly aligned and there is no overlapping of images or objects. The final and the most important aspect is the calibration of all devices.

6. How Augmented Reality Works

As we have already seen Augmented Reality is a combination of real and virtual environments. We have also seen the different devices used for generating the augmented video. No matter which device we use the working principle of all the devices are one and the same. In optical see-through HMD, the see-through device captures the real world scene. In case of video see-through HMD the camera captures the real world scene. Later to this captured real world scene we add virtual objects into the real world captured scene. These virtual objects are generated by a graphics system. If necessary we can delete a real world object. GPS can be used to provide the global position of the user. For instance if the viewer is in some foreign place then using the GPS, he/she will be able to
E&C Dept. 11 VVCE, Mysore

Augmented Reality

get the information about that place and the necessary directions if he/she requires. The most important of it all is the use of the 3D graphics card which does all the image arithmetics in order to obtain the final augmented scene.

Figure6.1: Augmented Reality System

E&C Dept.

12

VVCE, Mysore

Augmented Reality

Figure6.2: Image description on a working AR system

7. Augmented Reality in a Contact Lens

E&C Dept. 13 VVCE, Mysore

Augmented Reality

A new generation of contact lenses built with very small circuits and LEDs promises bionic eyesight. The human eye is a perceptual powerhouse. It can see millions of colors, adjust easily to shifting light conditions, and transmit information to the brain at a rate exceeding that of a high-speed Internet connection. Contact lens with simple built-in electronics is already within reach. Such devices are being produced in small numbers in my laboratory at the University of Washington, in Seattle. These lenses dont give us the vision of an eagle or the benefit of running subtitles on our surroundings yet. But they have built a lens with one LED, which they have powered wirelessly with RF. What they have done so far barely hints at what will soon be possible with this technology. Conventional contact lenses are polymers formed in specific shapes to correct faulty vision. To turn such a lens into a functional system, we integrate control circuits, communication circuits, and miniature antennas into the lens using custom-built optoelectronic components. Those components will eventually include hundreds of LEDs, which will form images in front of the eye, such as words, charts, and photographs. Much of the hardware is semitransparent so that wearers can navigate their surroundings without crashing into them or becoming disoriented. In all likelihood, a separate, portable device will relay displayable information to the lenss control circuit, which will operate the optoelectronics in the lens. These lenses dont need to be very complex to be useful. Even a lens with a single pixel could aid people with impaired hearing or be incorporated as an indicator into computer games. With more colors and resolution, the repertoire could be expanded to include displaying text, translating speech into captions in real time, or offering visual cues from a navigation system. With basic image processing and Internet access, a contact-lens display could unlock whole new worlds of visual information, unfettered by the constraints of a physical display. Besides visual enhancement, noninvasive monitoring of the wearers biomarkers and health indicators could be a huge future market. Weve built several simple sensors that can detect the concentration of a molecule, such as glucose. Sensors built onto lenses would let diabetic wearers keep tabs on blood-sugar levels without needing to prick a
E&C Dept. 14 VVCE, Mysore

Augmented Reality

finger. The glucose detectors were evaluating now are a mere glimmer of what will be possible in the next 5 to 10 years. 7.1 Challenges Three fundamental challenges stand in the way of building a multipurpose contact lens. First, the processes for making many of the lenss parts and subsystems are incompatible with one another and with the fragile polymer of the lens. To get around this problem, we make all the devices from scratch. To fabricate the components for silicon circuits and LEDs, we use high temperatures and corrosive chemicals, which means we cant manufacture them directly onto a lens. That leads to the second challenge, which is that all the key components of the lens need to be miniaturized and integrated onto about 1.5 square centimeters of a flexible, transparent polymer. We havent fully solved that problem yet, but we have so far developed our own specialized assembly process, which enables us to integrate several different kinds of components onto a lens. Last but not least, the whole contraption needs to be completely safe for the eye. Take an LED, for example. Most red LEDs are made of aluminum gallium arsenide, which is toxic. So before an LED can go into the eye, it must be enveloped in a biocompatible substance. 7.2 Current Progress Fabricated prototype lenses with 1 LED, a small radio chip, and an antenna, and transmitted energy to the lens wirelessly, lighting the LED. Device built from scratch starting with fabrication of components using high temperature and corrosive chemicals. It is done because it cannot be directly manufactured onto the lens. Components are shrunk to fit 1.5 sq.cm of flexible, transparent polymer. LEDs are made of aluminum gallium arsenide which is toxic. So before an LED can go into the eye, it must be enveloped in a biocompatible substance. Adding color and resolution would enhance uses to possibly offering visual cues from a navigation system.

E&C Dept.

15

VVCE, Mysore

Augmented Reality

With basic image processing and internet access, the possibilities grow even more.

Figure7.2.1: Solar powered augmented contact lens Hardware has been encapsulated in a biocompatible polymer. Has been tested on live rabbits for over 20 minutes at a time without causing any adverse effects. 7.3 Future Enhancements The future awaits a far more fascinating, exciting and a wonderful use of augmented reality in a contact lens. The research is going on in the University of Washington. The contact lens that is going to be designed is far more from imaginable; it consists of all those components that are required for accomplishment of augmentation into a single and a usable contact lens by each and every person.

E&C Dept.

16

VVCE, Mysore

Augmented Reality

Figure7.3.1: Future concept The future concept shown in the figure consists of a lot many things into a single lens that can be worn by humans. It consists of, Antenna collects incoming RF energy from a separate portable transmitter. Power-conversion circuitry provides DC power to other parts of the system and sends instructions to the display control circuit. The display might consist of LEDs, which would turn on and off, or LCD-like elements, whose transparency would be modulated by the control circuit.

E&C Dept.

17

VVCE, Mysore

Augmented Reality

An energy-storage module, perhaps a large capacitor, is connected to a solar cell, which could provide a boost to the lens.

A biosensor samples the surface of the cornea, performs an analysis, and provides data to the telecommunication module to transmit to an external computer.

Electrical interconnects provide the necessary connections between the modules.

8. Augmented Reality Applications

The technology could be used to stage dramatic experiences in historic sites and homes in cities throughout the world. Tourists could visit the beaches at Normandy and
E&C Dept. 18 VVCE, Mysore

Augmented Reality

watch the Allies invade France. One might even observe Alexander Graham Bell spilling battery acid and making the world's first telephone call: Mr. Watson, come here. The first, relatively rudimentary forms of AR technology are already being used in a few prosaic but important practical applications. Airline and auto mechanics have tested prototypes that give visual guidance as they assemble complex wiring or make engine repairs, and doctors have used it to perform surgery on patients in other cities. But those applications are just the beginning. AR will soon combine with various mobile devices to redefine how we approach the vast and growing repository of digital information now buzzing through the Internet. The shift is coming about in part because of the development of technologies that free us from our desks and allow us to interact with digital information without a keyboard. But it is also the result of a change in attitude, broadening the sense of what computers are and what they can do. We are already seeing how computers integrate artificially manipulated data into a variety of workaday activities, splicing the human sensory system into abstract representations of such specialized and time-critical tasks as air traffic control. We have also seen computers become a medium for art and entertainment. Now we will use them to knit together Web art, entertainment, work, and daily life. Ever more sophisticated games, historic tours, and AR social experiences will come as the technology advances. We represent the possibilities in the form of a pyramid, with the simplest mobile systems at its base and fully immersive AR on top. Each successive level of technology enables more ambitious designs, but with a smaller potential population of users. In the future, however, advanced mobile phones will become increasingly widespread, the pyramid will flatten out, and more users will have access to richer augmented experiences. Fully immersive AR, the goal with which we began, may one day be an expected feature of visits to historic sites, museums, and theme parks, just as human-guided tours are today. AR glasses and tracking devices will one day be rugged enough and inexpensive enough to be lent to visitors, as CD players are today. But it seems unlikely that the majority of visitors will buy AR glasses for general use as they buy cell phones today; fully immersive AR will long remain a niche technology.

E&C Dept.

19

VVCE, Mysore

Augmented Reality

On the other hand, increasingly ubiquitous mobile technology will usher in an era of mixed reality in which people look at an augmented version of the world through a handheld screen. You may well pull information off the Web while walking through the Oakland Cemetery or along Auburn Avenue, sharing your thoughts as well as the ambient sounds and views with friends anywhere in the world. Mobile mixed reality will call forth new media forms that skillfully combine the present and the past, historical fact and its interpretation, entertainment and learning. AR and mobile technology have the potential to make the world into a stage on which we can be the actors, participating in history as drama or simply playing a game in the space before us. Some areas where augmented reality can be implemented are: Cell phones/ Smart phones Robot path planning Web applications Entertainment Military aircraft Engineering design

9. Conclusion
E&C Dept. 20 VVCE, Mysore

Augmented Reality

Augmented Reality is far behind Virtual Environments in maturity. No commercial vendor currently sells an HMD-based Augmented Reality system. Today AR systems are primarily found in academic and industrial research laboratories. The first deployed HMD-based AR systems will probably be in the application of aircraft manufacturing. Both Boeing and McDonnell Douglas are exploring this technology. The former uses optical approaches, while the latter is pursuing video approaches. Annotation and visualization applications in restricted, limited-range environments are deployable today. Applications in medical visualization will take longer. Prototype visualization aids have been used on an experimental basis, but the stringent registration requirements and ramifications of mistakes will postpone common usage for many years. AR will probably be used for medical training before it is commonly used in surgery. The next generation of combat aircraft will have Helmet-Mounted Sights with graphics registered to targets in the environment. Augmented Reality is a relatively new field, where most of the research efforts have occurred in the past ten years. One area where a breakthrough is required is tracking an HMD outdoors at the accuracy required by AR. If this is accomplished, several interesting applications will become possible. Two examples are: navigation maps and visualization of past and future environments. The first application is a navigation aid to people walking outdoors. An AR system makes navigation easier by performing the association step automatically. If the user's position and orientation are known, and the AR system has access to a digital map of the area, then the AR system can draw the map in 3-D directly upon the user's view. The second application is visualization of locations and events as they were in the past or as they will be after future changes are performed.

References

E&C Dept.

21

VVCE, Mysore

Augmented Reality

[1] Simon Julier NRL Virtual Reality Lab/ITT Advanced Engineering, Blair MacIntyre Georgia Institute of Technology, Recent Advances in Augmented Reality [2] Augmented reality survey Available http://www.se.rit.edu/~jrv/research/ar/index.html [3] Augmented reality design Available: http://www.uxmatters.com/mt/archives/2009/08/inside-out-interactiondesign-for-augmented-reality.php [4] Augmented reality in contact lens Available:http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-acontact-lens/0 [5] Augmented reality applications Available http://spectrum.ieee.org/computing/software/is-it-live-or-is-it-ar/0

E&C Dept.

22

VVCE, Mysore