BRAINPORT VISION DEVICE

Seminar report on

In Partial Fulfillment of the requirement for the award of the degree of

Rajasthan Technical University, Kota

Electronics & Communication Engineering

Bachelor of Technology

2010-2011
Submitted by:Supervised by:-

(Dept. of Electronics & Communication Engg.)

Nayagaon, Rawatbhata Road, Kota 1

Chapter 1

INTRODUCTION
The BrainPort vision device provides information to blind individuals via a neuro-stimulating array placed on the tongue. This unique device provides immediate benefits to its blind users in areas of safety, mobility, and recreation and opens a new world of sensory experience and exploration. The BrainPort vision device operates by acquiring an image stream from a camera, similar to a camcorder. Like a camcorder, the moving images are sent to a display, which, in this case, is the electrode array on the tongue. The image stream is displayed on the tongue by converting light information to electrical stimulation, which feels like microscopic bubbles to the user. With minimal training, users learn to interpret the images on their tongue as information about the scene in front of them. The BrainPort vision device includes an imaging system capable of working both indoors and outdoors, with a field of view spanning 3-75 degrees (magnified versus wide angle views). The tongue array contains 400 electrodes and is connected to the controller via a flexible cable. The control system is approximately the size of a PDA and runs for about 3 hours per charge, with swappable batteries. An artists concept of the tongue array (or Intra-Oral Device, IOD) and camera mounting is shown below alongside the device. The IOD is attached to a flexible boom by a thin wire. The camera unit is mounted on a pair of eyeglasses frames. The user controls and the power supply are connected to a belt-worn, pager-style, controller. A blind woman sits in a chair holding a video camera focused on a scientist sitting in front of her. She has a device in her mouth, touching her tongue, and there are wires running from that device to the video camera. The woman has been blind since birth and doesn't really know what a rubber ball looks like, but the scientist is holding one and when he suddenly rolls it in her direction, she puts out a hand to stop it. The blind woman saw the ball through her tongue. Well, not exactly through her tongue, but the device in her mouth sent visual 2

input through her tongue in much the same way that seeing individuals receive visual input through the eyes. In both cases, the initial sensory input mechanism -- the tongue or the eyes - sends the visual data to the brain, where that data is processed and interpreted to form images. Braille is a typical example of sensory substitution -- in this case, you're using one sense, touch, to take in information normally intended for another sense, vision. Electro tactile stimulation is a higher-tech method of receiving somewhat similar (although more surprising) results, and it's based on the idea that the brain can interpret sensory information even if it's not provided via the natural channel.

2.1 Statistics on the Blind:

37 million: People in the world are blind India (9 million), Africa (7 million) and China (6 million) Every 5 seconds: One person in our world goes blind 75 million: People will be blind by 2020 (if trends continue)

2.2 Cybernetics:
Cybernetics is about having a goal and taking action to achieve that goal. "Cybernetics" comes from a Greek word meaning "the art of steering. Ironically but logically, AI and cybernetics have each gone in and out in the search for machine intelligence So I Can Read can be termed as a Cybernetics System For Disabled (blind)

Chapter 2

What is BrainPort Vision Device?

"BrainPort device" is a technology developed in US, which is making the world visible to the ones who lose their sight due to some accidental incidents. A Neuro scientist has developed the BrainPort Vision Device that allows the blinds to see using their tongue. Craig Lundberg, 24, is the first British soldier to test the BrainPort system, which is billed as the next best thing to sight The technology has made the dark-dependent world come alive and independent to Craig Lundberg who completely lost his sight after a grenade attack in Iraq, as he is now able to sense the visuals with his tongue. The soldier admits that his world has been transformed because of the technology. The device which sends visual input through tongue in much the same way that seeing individuals receive visual input through the eyes is called the BrainPort Vision Device. BrainPort could provide vision-impaired people with limited forms of sight. Technically, this device is underlying a principle called electro tactile stimulation for sensory substitution. To produce tactile vision, BrainPort uses a camera to capture visual data.

Chapter 3 Modes of stimulation and its various forms

From the very beginning of the electro tactile stimulation this journey has travelled a lot and the various forms may be described as follows: TVSS-Tactile Vision Substitution Systems. Vibrotactile (FINGERTIPS). Electro Tactile Stimulation for Tongue

3.1 Tactile vision substitution system

The TVSS may be characterized as a humanistic intelligence system. It represent a symbiosis between instrumentation-for example, an artificial sensor array (TV camera)-computational equipment, and the human user. Consistent with the terminology of this issue, this is made possible by "instrumental sensory plasticity," the capacity of the brain to reorganize when there is: (a) functional demand,(b) the sensor technology to fill that demand, and (c) the training and psychosocial factors that support the functional demand. To constitute such systems then, it is only necessary to present environmental information from an artificial sensor in a form of energy that can be mediated by the receptors at the human-machine interface, and for the brain, through a motor system (e.g., a head-mounted camera under the motor control of the neck muscles), to determine the origin of the information. This can now be extended into other domains with modern technology and the availability of artificial sensory receptors, such as: 1. A miniature TV camera for blind persons, 2. A MEMS technology accelerometer for providing substitute vestibular information for persons with bilateral vestibular loss, 3. Touch and shear-force sensors to provide information for spinal cord injured persons, 4. Instrumented condom for replacing lost sex sensation, or 5. A sensate robotic hand (Bach-y-Rita, 1999).

In first sensory substitution project, they developed tactile vision substitution Systems (TVSS) to deliver visual information to the brain via arrays of stimulators in contact with the skin of one of several parts of the body (abdomen, back thigh). Optical images picked up by a TV camera were transuded into a form of energy (vibratory or direct electrical stimulation) that could be mediated by the skin receptors. In these sensory substitute systems, the visual information reaches the perceptual levels for analysis and interpretation via somatosensory pathways and structures.

3.2 Vibrotactile
After sufficient training with the TVSS, our subjects reported experiencing the image in space, instead of on the skin. They learn to make perceptual judgments using visual means of analysis, such as perspective, parallax, looming and zooming, and depth judgments (Bach-yRita, Collins, Saunders, White, & Scadden, 1969; cf., Bach-y-Rita, 1972, 1989, 1995, 1996, 1999; Bach-y-Rita, Kaczmarek, & Meier, 1998; Bach-y-Rita, Kaczmarek, Tyler, & GarciaLara, 1998; Bach-y-Rita, Webster, Tompkins, & Crabb, 1987; Kaczmarek & Bach-y-Rita, 1995; White, Saunders, Scadden, Bach-y-Rita, & Collins, 1970). Although the TVSS systems have only had between 100 and 1032 point arrays, the low resolution has been sufficient to perform complex perception and "eye"-hand coordination tasks. These have included facial recognition, accurate judgment of speed and direction of a rolling ball with over 95% accuracy in batting a ball as it rolls over a table edge, and complex inspection-assembly tasks. The latter were performed on an electronics company assembly line with a 100 point vibrotactile array clipped to the work-bench against which the blind worker pressed the skin of his abdomen, and through which information from a TV camera (Substituting for the ocular piece of a dissection microscope) was delivered to the human-machine interface (Bach-y-Rita, 1995, pp. 187-193).

Child reproducing perceived image of a teachers hand as displayed on a modified Optacon. The tactile image is picked up with one finger statically placed on the 6 24 vibrotactile array. LED monitor in foreground is visual representation of active pattern on the tactile display, which is obtained by the child's head-mounted camera.

3.3 Electrotactile Stimulation for Tongue

In the TVSS studies cited above, the stimulus arrays presented only black-white information, without gray scale. However, the tongue electrotactile system does present gray-scaled pattern information, and multimodal and multidimensional stimulation is possible. Simultaneously, we have also modelled the electrotactile stimulation parameter space to determine how we might elicit tactile "colors." Aiello (1998a, 1998b) has identified six stimulus parameters: the current level, the pulse width, the interval between pulses, the number of pulses in a burst, the burst interval, and the frame rate. All six parameters in the waveforms can, in principle, be varied independently within certain ranges, and may elicit potentially distinct responses. For example, in a study of electrical stimulation of the skin of the abdomen, Aiello (1998a) suggested that the best way to encode intensity information independent of other percept qualities with a multidimensional stimulus waveform was through modulation of the energy delivered by the stimulus. In that case, the energy was varied in such a way that the displacement in the parameter space, corresponding to a given transition between energy levels, was minimal (gradient mode of stimulation). Although the gradient mode of stimulation requires a realtime fulfilment of mathematical constraints among all the parameters, its implementation could be included within a microelectronic package for signal treatment. The tongue interface overcomes many of these. The tongue is very sensitive and highly mobile. Since it is in the protected environment of the mouth, the sensory receptors are close to the surface. The presence of an electrolytic solution, saliva, assures good electrical contact. The results obtained with a small electrotactile array developed for a study of form perception with a finger tip demonstrated that perception with electrical stimulation of the tongue is somewhat better than with finger-tip electrotactile stimulation, and the tongue requires only about 3% of the voltage (5-15 V), and much less current (0.4-2.0 mA), than the finger-tip. The electronic system has been described elsewhere (Bach-y-Rita, Kaczmarek, Tyler, et al., 1998).

Electrotactile stimuli are delivered to the dorsum of the tongue via flexible electrode arrays (Figure 3) placed in the mouth, with connection to the stimulator apparatus (TDU) via a flat cable passing out of the mouth. The tongue electrode array and cable are made of a thin (100 m) strip of polyester material (Mylar) onto which a rectangular matrix of gold-plated copper circular electrodes has been deposited by a photolithographic process similar to that used to make printed circuit boards. The electrotactile stimulus consists of 40-s pulses delivered sequentially to each of the active electrodes in the pattern. Bursts of three pulses each are delivered at a rate of 50 Hz with a 200 Hz pulse rate within a burst. This structure was shown previously to yield strong, comfortable electrotactile precepts. Positive pulses are used because they yield lower thresholds and a superior stimulus quality on the fingertips.

Close-up of 144-point (12 x 12) "virtual ground" electrotactile tongue display

10

Action potentials (AP's) thus recorded had amplitudes from 0.1 to 1.0 mV and a 5 : 1 signalto-noise ratio (SNR).A circular electrode surrounding the recording site served as the ground reference. Following pre amplification and band pass filtering (200-10 000 Hz), a differential amplitude detector identified AP's, producing an output pulse whenever the recorded signal entered a predefined amplitude-time window. In the first experiment, electrotactile entrainment currents (iEN) were determined by adjusting the stimulation current from near zero to the minimal value resulting in one AP for each stimulation pulse. These currents exceeded the absolute thresholds (the currents causing occasional AP's) by approximately5%. The entrainment current was determined twice for positive- and negative-polarity stimulation pulses of ten different widths: 20, 30, 40, 50, 70, 100, 150, 200, 300, and 500 _s, delivered at a rate of 10 pulses/s. The width sequence was reversed during the second run on each of the three fibers.

Relative timing between simultaneous mechanical and electrotactile stimulation. The top trace represents the sinusoidal, 30-Hz, 50-100-_m (0-P) mechanical displacement.

11

Chapter 4

When a human looks at an object, the optical image entering the eyes does not go beyond the retina. Instead, it would turn into spatio-temporal nerve patterns of impulse along the optic nerve fibers. By analysing the impulse patterns, the brain recreates the images. Indeed, the channels such as eyes, ears and skin those carry sensory information to the brain are set up in a similar manner to perform similar activities. To substitute one sensory input channel for another, the big challenge to the scientists is how to correctly encode the nerve signals for the sensory event and send them to the brain through the alternate channel as the brain appears to the flexible when it comes to interpreting sensory input.

The concepts at work behind electro tactile stimulation for sensory substitution are complex. The idea is to communicate non-tactile via electrical stimulation of the sense of touch.

In practice, this typically means that "an array of electrodes receiving input from a nontactile information source (a camera, for instance) applies small, controlled, painless currents (some subjects report it feeling something like soda bubbles) to the skin at precise locations according to an encoded pattern."

For a blind person, it means the encoding of the electrical Pattern essentially attempts to mimic the input that would normally be received by the non-functioning sense vision. So patterns of light picked up by a camera to form an image are replacing the perception of the eyes and converted into electrical pulses that represent those patterns of light. In other words, when the encoded pulses are applied to the skin, the skin is actually receiving image data which would be then sent to the brain in the forms of impulse. Under normal circumstances, the parietal lobe in the brain receives touch information, while the occipital lobe receives vision information. When the nerve fibers forward the image-encoded touch signals to the

12

parietal lobe, "the electric field thus generated in subcutaneous tissue directly excites the afferent nerve fibres responsible for touch sensations".

Within the system, arrays of electrodes can be used to communicate non-touch information through pathways to the brain normally used for the touch related impulses. The breakthrough of the BrainPort technology is to use the tongue as the substitute sensory channel.

13

Chapter 5 THE STRUCTURE OF THE BRAINPORT DEVICE

The figure below shows the structure of the BrainPort Vision Device.

The optical information that would normally hit the retina is picked up by the digital camera in digital form. It uses radio signals to send the ones and zeros to the CPU for encoding. Each set of pixels in the camera's light sensor corresponds to an electrode in the array. After that, the CPU runs a program that turns the camera's electrical information into a spatially encoded signal. "The encoded signal represents differences in pixel data as differences in pulse characteristics such as frequency, amplitude and duration.

14

Multidimensional image information takes the form of variances in pulse current or voltage, pulse duration, intervals between pulses and the number of pulses in a burst, among other parameters." Then, the electrode array (shown in Fig) receives the resulting signal via the stimulation circuitry and applies it to the tongue. At last, the brain interprets and uses the information coming from the tongue as if it were coming from the eyes.

HOW DOES THE BRAINPORT VISION DEVICE WORK?

The BrainPort vision system consists of a postage-stamp-size electrode array for the top surface of the tongue (the tongue array), a base unit, a digital video camera, and a hand-held controller for zoom and contrast inversion. Visual information is collected from the useradjustable head-mounted camera (FOV range 390 degrees) and sent to the BrainPort base unit. The base unit translates the visual information into an stimulation pattern that is displayed on the tongue. The tactile image is created by presenting white pixels from the camera as strong stimulation, black pixels as no stimulation, and gray levels as medium levels of stimulation, with the ability to invert contrast when appropriate. Users often report the sensation as pictures that are painted on the tongue with Champagne bubbles.

15

16

With the current system (arrays containing 100 to 600+ electrodes), study participants have been able to recognize high-contrast objects, their location, movement, and some aspects of perspective and depth. Trained blind participants use information from the tongue display to augment understanding of the environment. Our ongoing research with the BrainPort vision device demonstrates the great potential of tactile vision augmentation and we believe that these findings warrant further exploration. As a result, we are currently working on improvements to the tongue display hardware, software, and usability, and on overall device miniaturization.

WHY TONGUE IS USED IN BRAINPORT TECHNOLOGY

Compare to all other skin areas, the tongue skin area is the most sensitive one because there are more nerve fibres and they are much closer to the surface. Moreover, there is no stratum conium (an outer layer of dead skin cells) on the tongue which act as an insulator. With these characteristics, it requires less voltage to stimulate nerve fibres in the tongue (5 to 15 volts) compared to areas like the fingertips or abdomen (40 to 500 volts). Also, since the tongue is surrounded by saliva which contains electrolytes, it would help to maintain the current flow between the electrode and skin tissues. Last but not least, the area of the cerebral cortex that interprets touch data from the tongue is larger than the areas serving other body parts. Therefore, the tongue is the best choice for conveying tactile-based data to the brain until this moment

17

Chapter 6

ADVANTAGES:
Blind users can use the Brain Port vision device independently - at home, at work, and in public spaces indoors and out as a tool for improved safety, mobility and object recognition. Secondary benefits include applying the technology toward specific hobbies and recreational situations. These benefits enable greater independence at home, school and in business, greatly improving quality of life.

SAFETY
Navigating difficult environments, such as parking lots, traffic circles, complex intersections Recognizing quiet moving objects like hybrid cars or bicycle

MOBILITY
Finding doorways, hall intersections, lobby or restaurant in an office or hotel. Finding continuous sidewalks, sidewalk intersections and curbs.

18

Chapter 7

OBJECT RECREATION
Locating people Locating known objects such as shoes, cane, coffee mug, keys

APPLICATIONS

Just a few of the current or foreseeable applications include providing elements of sight for the visually impaired in the medical field.

The BrainPort electrodes would receive input from a sonar device to provide not only directional cues but also a visual sense of obstacles and terrain.

BrainPort may also provide expanded information for military pilots, such as a pulse on the tongue to indicate approaching aircraft or to indicate that they must take immediate action.

BrainPort applications include robotic surgery. The surgeon could wear electro tactile gloves to receive tactile input from robotic probes inside someone's chest cavity.

19

20

Chapter 8

CURRENT AND POTENTIAL APPLICATIONS

While the full spectrum of BrainPort Vision Technology applications has yet to realized, the device has the potential to lessen an array of sensory limitations and to alleviate the symptoms of a variety of disorders. Just a few of the current or foreseeable applications include providing elements of sight for the visually impaired in the medical field. Beyond medical applications, scientists have been exploring potential military uses with a grant from the Defence Advanced Research Projects Agency (DARPA). They are looking into underwater applications that could provide the Navy Seals with navigation information and orientation signals in dark, murky water. The BrainPort electrodes would receive input from a sonar device to provide not only directional cues but also a visual sense of obstacles and terrain. BrainPort may also provide expanded information for military pilots, such as a pulse on the tongue to indicate approaching aircraft or to indicate that they must take immediate action. Other potential BrainPort applications include robotic surgery. The surgeon could wear electro tactile gloves to receive tactile input from robotic probes inside someone's chest cavity. In this way, the surgeon could feel what he's doing as he controls the robotic equipment. Race car drivers might use a version of BrainPort to train brains for faster reaction times, and gamers might use electro tactile feedback gloves or their controllers to feel what they're doing in a video game.

21

Chapter 9

CONCLUSION

Even though this is a field of scientific study that has been around for nearly a century, it has just picked up in this decade due to the miniaturization of electronics and dramatic improvement of the computer's speed. Already more streamlined than any previous setup using electro tactile stimulation for sensory substitution, BrainPort envisions itself even smaller and less obtrusive in the future. In the case of the BrainPort Vision Device the electronics might be completely embedded in a pair of glasses along with a tiny camera and radio transmitter, and the mouthpiece would house a radio receiver to receive encoded signals from the glasses. It's not exactly a system on a chip, but gives it 20 years -- we might be seeing a camera with the size of a grain of rice embedded in people's foreheads.

22

REFERENCES

Bach-y-Rita, Paul et al. "Form perception with a 49-point electro tactile stimulus array on the tongue: A technical note." Journal of Rehabilitation Research and Development. http://howstuffworks.com http://india.cchem.berkeley.edu/~reimer/courses/present/samples/sample2.pdf http://wistechnology.com/printarticle.php?id=2319 http://www.itechnews.net/2009/08/16/brainport-vision-device-lets-blinds-see-withtougues/ http://www.gizmowatch.com/entry/brainport-vision-device-for-the-visuallyimpaired-to-feel-images/ http://accessibletravel.suite101.com/article.cfm/brainport_vision_device http://www.scientificamerican.com/article.cfm?id=device-lets-blind-see-with-tongues http://topnews.net.nz/content/22766-brainport-device-transforms-world-blinds http://www.telegraph.co.uk/health/healthnews/7350218/British-soldier-blinded-inIraq-trials-new-technology-to-see-using-his-tongue.html

23