CHAPTER 1

INTRODUCTION

In virtual reality (VR) applications, actuating the sense of taste is

currently based on chemicals , which is not an effective long-term solution, as the
chemicals need to be refilled and maintained properly. Therefore, finding a digital
taste actuation technology which can produce various taste sensations, and does not
depend on chemicals, will be a useful future step for VR. Once we reach this goal,
people will be able to experience taste sensations through VR and share these
sensations across the internet, in the same way, as we experience audio, visual, and
haptic sensations. One way of producing taste sensations without using chemicals is
to apply thermal stimulation on the tongue and stimulate the TRPM5 (Transient
receptor potential cation channel subfamily M member 5) taste channel.

The TRPM5 taste channel is known to enhance the sensitivity for sweet, bitter,
and umami tastes . The deletion of TRPM5 in mice was reported to result in a specific
loss of sensitivity for the sweet, bitter, and umami tastes . Previous experiments from
the medical field investigated the sensitivity of TRPM5 for different stimuli. These
experiments found that raising the temperature on the tongue within a few seconds
produced sweet sensations for a specific group of people, and increasing temperature
to higher temperatures enhanced the sensitivity for sweetness . According to the
previous experiments, only about half or fewer participants felt taste sensations for
thermal stimulation. This subset of the population was named as ‘thermal tasters’
(TT).

However, these previous researchers have described thermal taste mainly

using sweetness or other basic tastes. Therefore, the other sensations produced by the
thermal stimulation which do not belong to five basic tastes have not been thoroughly
investigated. Regarding VR, no one has made a controlled interactive system to
induce different taste sensations using thermal stimulation. Having said that, some
researchers, including the authors, have proposed sweet taste actuation devices using
this technology.

1
 These previous works mainly focused on producing sweet taste (or other basic
tastes) sensations, but, by improving this technology, we have identified that thermal
taste is a complex taste sensation, and it can produce tasterelated sensations, as well as
non-taste-related sensations. To study these effects, we developed a new computer-
controlled thermal taste device called ‘Thermal Taste Machine’. By placing the silver
electrode of the device on top of the tongue, the device can produce a modification of
the temperature on the surface of the tongue from 25 ◦C to 40 ◦C while heating, and
from 25 ◦C to 10 ◦C while cooling. This device is a computer-controlled device and
can be easily integrated with software programs. After building the device, we
conducted a characterization study for thermal taste and described thermal taste using
taste related and nontaste related sensations.

We measured the induced sensations using a measurement index that can

record intensities of 20 different sensations. The sensations were divided into two
categories: The first category consisted of the five basic tastes (sour, sweet, salty,
bitter, and umami) and seven other sensations that people often used to describe
flavour experiences (carbonation, metallic, chemical, electric, fatty, minty, spicy).
Another reason was that the sensations such as carbonation, fatty, and chemical are
detected through their own receptors and we decided to investigate whether the
thermal stimulation is capable of exciting these secondary taste receptors as well. The
second category consisted of eight non-taste related sensations, which include
coldness, warmness, numbness, pain, pressure, lingering, pleasant, and
unpleasantness. These are general terms that are used by the people to describe
sensations related to temperature and touch. Further, we conducted two more
experiments. The second experiment studied how thermal taste can modify the
intensity of sweetness produced by sucrose solutions, and the third experiment studied
how different rates of temperature rise affect the intensity of thermal sweetness (T-
SW). Experiments conducted on our thermal taste device revealed the following
findings for the first time, for a virtual reality thermal taste technology:

Our device produced different taste sensations other than sweetness including
minty, fatty-oiliness, and electric taste, with some non-taste sensations including
pleasantness on human subjects. Also, we observed positive effects towards umami,
metallic, chemical, and spicy taste sensations. From the second experiment, we
showed that using our device before tasting chemical sweet solutions enhanced the

2
sweetness for both thermal and non-thermal tasters. Further, we found that different
rates of stimulation produced different intensity for sweetness in thermal tasters. In
addition, we described the thermal taste as a combination of different taste and non-
taste sensations.

3
 CHAPTER 2

LITERATRURE SURVEY

2.1 INTRODUCTION AND RELATED WORKS

Some recent investigations have already studied the effects of thermal

stimulation on the tongue. These investigations have been primarily from either the
medical field or from the HCI field. One experiment first showed that rapid heating
the tip of the tongue from 15°C to 35°C (temperature was varied at approximately ±
1.5°Cs−1) evokes sweetness, and cooling of the tongue from 35°C resulted in either
saltiness or sourness. Further, static cold temperatures (placing an ice cube on a side
of the tongue) could produce salty taste sensations . Another study showed that
increasing temperature on the tongue resulted in activation of the TRPM5 channel that
generates a depolarizing potential in the taste receptor cells. They claimed that this
effect caused the enhanced sweetness perception at high temperatures and ‘thermal
taste’, the phenomenon whereby heating or cooling of the tongue evokes sensations of
taste in the absence of chemical tastants. It was concluded that stimulating TRPM5
with temperature as an input results in different taste sensations as outputs. It has been
further suggested that other tastes, such as salty and sour, may be linked to the
temperature sensitivity associated with the channels involved in their chemical
transduction. Participants who perceived taste sensations while heating or cooling
were referred as the ‘thermal tasters’ (TT).

Even though the thermal effect with TRPM5 channel has been studied in
medicine, a proper controlled computer system that can reproduce thermal taste
sensations has yet to be developed. Having said that,has experimented with electrical
and thermal stimulation in combination on the tongue, and showed that it is possible
to generate four of the basic tastes: sour, sweet, bitter, and salty. During the
experiments conducted,stimulation of the tongue with heating and cooling the tongue,
participants reported sensations of sweetness and sourness. The first sweet-specific
device using thermal stimulation was proposed by the authors of this paper in 2015.
Meanwhile, the authors discussed an interface that influenced flavour observation,
without changing the nourishment itself, by applying thermal sensations to the skin
around the nose to cause skin temperature changes associated with charming or

4
 unpleasant feelings. However, findings of these studies were not extended towards the
other taste sensations, except sour taste. Therefore, finding the optimum stimulation
parameters to generate different taste sensations (like temperature levels, intensity,
rates of temperature rising and falling, and frequency) are yet to be discovered, and
integrating thermal taste interfaces with multisensory or VR applications remains
largely unexplored. Our approach is different from the works mentioned above in
many ways. Our main objective was to develop a controllable interactive thermal taste
technology to generate taste sensations for virtual reality. Our concept of stimulating
taste receptors using thermal stimulation is shown. This device is able to generate
different rates of temperature rise for heating and cooling. Further, we decided that
this technology should be a device that can be plugged into computers, and it should
be able to be programmed and controlled through the computer. Therefore, our work
is different from the previous works from the medical field. Also, the works from HCI
have mainly concentrated on inducing sweet taste sensations. In this study, we
considered thermal taste as a combination of taste and non-taste related sensations.
Therefore, one of our objectives was to describe thermal taste using known taste and
non-taste sensations. Further, we studied how thermal stimulation can change the
intensity of sweetness, and how different rates of temperature rise can affect
sweetness. Our future objective is to integrate this thermal taste technology for VR
and multisensory communication. According to our knowledge, no attempt has been
made so far to achieve this.

2.2 THERMAL TASTE

We are all familiar with the effect of temperature on taste - think about starting
to eat or drink something while it is warm and finishing when it has cooled, or vice
versa. The same food or beverage can taste different depending on its temperature. In
addition, in 20-30 percent of the population, heating or cooling small areas of the
tongue draws out a taste sensation without the presence of food or drink. These
individuals are known as 'thermal' tasters.

Can the temperature of the food we eat affect the intensity of its taste? It
depends on the taste, according to a new study. New research shows that changes in
the temperature of foods and drinks have an effect on the intensity of sour, bitter and
astringent (e.g. cranberry juice) tastes but not sweetness. In addition, research

5
suggests that in 20-30 percent of the population, heating or cooling small areas of the
tongue draws out a taste sensation without the presence of food or drink. These
individuals are known as 'thermal' tasters. They seem to taste foods more intensely
than others people do.

This device consisted of an Arduino microcontroller, a silver plateattached to a

Peltier module, a liquid cooler system, an H-bridge motor driver, a current sensor, a
temperature sensor, and a USB serial interface to communicate with the PC.

Therefore, finding a digital taste actuation technology which can produce

various taste sensations, and does not depend on chemicals, will be a useful future
step for VR. Once we reach this goal, people will be able to experience taste
sensations through VR and share these sensations across the internet, in the same way,
as we experience audio, visual, and haptic sensations.

6
 CHAPTER 3

SCHEME OF IMPLEMENTATION

This device consisted of an Arduino microcontroller, a silver plate attached to

a Peltier module, a liquid cooler system, an H-bridge motor driver, a current sensor,
a temperature sensor, and a USB serial interface to communicate with the PC. The
circuit diagram of the thermal taste device is shown.

Changing the temperature on the tongue was achieved by using a Peltier module that
was tightly attached with the silver plate as shown.When current flows through the Peltier
module, one side becomes cool and the other side becomes hot. By alternating the direction
of the current flow, wewere able to swap the heating and cooling sides. According to the
specification of the Peltier module we selected, the surface temperature could be changed
from -40°C to 80°C. This Peltier module was operated at 15.6 V and it consumed up to 8.5 A
for 100% duty cycle. Therefore, a high-power H-bridge motor driver was used to drive the
Peltier module. Wires carrying power to the Peltier module were shielded to improve the
safety. The motor driver received control signals from the Arduino microcontroller output
pins. To make the tongue stimulator plate, we used copper for our earlier prototypes, and

7
then used silver for the current prototype. The main reason for moving from copper to silver
was to achieve faster temperature change. Compared with copper, silver provides faster
temperature change, since silver has less specific heat. The specific heat of copper is 0.092
kCal(kg°C)−1 while silver is 0.057 kcal(kg°C)−1 [24]. Additionally, silver does not produce
metallic sensations in humans as in copper, which is a further advantage of silver over
copper. The silver plate chosen for this device had a thickness of 0.5 mm, and we reduced
the surface area of the metal plate to decrease the total weight, because a plate with less
weight can heat and cool faster [12]. Further, the surface area and the shape of the silver
plate affected heating and cooling. If the silver plate has more surface area than the surface
area of the Peltier, heating and cooling of the silver plate becomes slower. Regarding the
shape of the silver plate, the section that was placed on the tongue needed to be closer to
the Peltier module to ensure an efficient energy transmission. Therefore, we made the silver
plate not much larger than the surface area of the Peltier, and made sure the section that
touches the tip of the tongue was situated closer to the Peltier module (Please refer to the
Fig. 3). The silver plate was connected to the Peltier using a thermal epoxy (with 2.8 Wm−1K
thermal conductivity) that provided efficient heat transfer between the two surfaces.
Generally, Peltier modules have a cooling side and a heating side. By alternating the flow of
the current, it is possible to change the heating side to be used for cooling and the cooling
side to be used for heating, but it is not efficient. Therefore, during the development of this
device, we had to make a choice: whether we should mount the silver plate on the cooling
side or the heating side of the Peltier. Since most of the previous studies reported thermal
taste sensations for heating rather than cooling, we decided to mount the silver plate on the
heating side. This made our device optimized for heating. As a result, we had to use the
same heating side of the Peltier for cooling, which is not as efficient as using the original
cooling side of the Peltier. To increase the efficiency while cooling, the excess energy (heat)
produced on the cooling side needed to be drawn out continuously. Therefore, we used a
liquid cooler instead of only using a heat sink and mounted it on the cooling side of the
Peltier. For reaching the correct set temperature while heating and cooling, the device
controlled the current flow to the Peltier by modifying the PWM output, using a PID
controller [20]. Furthermore, this device was equipped with a current flow sensor and a
temperature sensor to measure the amount of current flow to the Peltier module and the
resulting temperature of the silver plate.

8
3.3 CHECKER MODULE

This block is used to provide two-level authentication. It receives ASCII key from
level differentiator unit if authorized pattern of brain waves is generated. After
satisfying brain wave condition checker module asks for a pass-key which is second
level of authentication. Checker module consists of an ARM processor interfaced with
a touch panel and a LCD. Touch panel is used as an input device and LCD as an
output device. User enters the pass-key on touch screen and then authenticated or
unauthenticated entry is indicated on LCD. Note here that Pass-key will be asked only
if the brain waves are matched.Fig.3.4 shows checker module.

Fig.3.4 Checker module

CHAPTER 4
EXPERIMENTAL SETUP

4.1 BRAIN SENSOR

9
Brainsense headband is used as brain sensor. It uses single electrode configuration.
One electrode is placed on forehead at frontal lobe of brain. Headband uses one
reference electrodewhich is connected to the left ear lobe as a reference node. Fig.4.1
shows headband used. This headband is battery powered and requires three 1.5V dry
batteries. Brainsenseheadband is a Bluetooth enabled device which transmits brain
signals in the form of packets through wireless communication.

Fig.4.1 Brainsense headband

4.2DATA EXTRACTION AND INTERPRETATION

Brain sensors records raw brain waves and transmits it to the LDU (Level
Differentiator Unit) in the form of packets through Bluetooth protocol. Level
differentiator unit is nothing but a computer using Matlab platform. A program
isdeveloped in Matlab whose objective is to extract the datafrom the packets. It also

interprets this brain data and checkswhether it matches the authorized pattern of brain
waves. Then it accordingly transmits data to the checker module through serial
communication. Authorized pattern is predefined in Matlab. This pattern is
determined using some fixed threshold values of brain signals. This program uses a

10
 library file, ‘thinkgear.dll’, supported by headband for data extraction and
interpretation.

4.3 CHECKER MODULE

Checker module consists of an ARM processor interfaced with LCD and touch panel.
Fig. 6 shows this. A program is developed, which runs on microcontroller. It receives
ASCIIkeys from LDU. These keys are transmitted by LDU only when user satisfies
brain wave condition. Checker module provides second level of authentication.

Fig.4.2 Checker module

ARM processor runs the program. It uses touch panel as an input device which provides co-
ordinates of touch made by user to give inputs. LCD is used as an output device to display
instructions for user and numeric keypad. Touch panel is placed and aligned on LCD. And
both are interfaced to aARM7 board based on LPC2148. Board is also incorporated with
level changer MAX232 and a serial DB9 connector through which it is connected to PC.

11
CHAPTER 5

RESULTS

12
 It is a two-level authentication system, in which brain wave condition needs to be
cracked first before traditional pass key system. Once brain sensor passes the brain
signals tothe LDU through Bluetooth, the LDU matches it with predefined pattern
defined in Matlab program. Two locks of first level authentication means brain waves
are used. One lock opens for 6 times eye blink signal. Second lock opens for attention
level of 90. Actually, the brain signal attention is calibrated to give value from 0 to
100. Also, eye blink will be counted only if the strength of eye blink signal is greater
than 35.

Brain signals are transmitted in packets to computer through Bluetooth. Matlab

program uses ThinkGear library to read those packets and performs the operations of
program. When it satisfies the conditions of eye-blink signal, it displays the window
shown in Fig.5.1 and transmits ASCII ’A’ to the checker module through serial
communication to open Lock 1.

Fig.5.1 Matlab window after Lock 1 opens

13
After this, Matlab starts reading and displaying the values of attention signals. When
attention signal condition satisfies, it displays Matlab window shown in Fig.5.2 and
transmits ASCII ’M’ to the checker module to open Lock 2.

Matlab starts drawing the waveform of eye-blink and attention signal after Lock 1
opens. A loop of 20 second is used to record signals for 20 second. Fig.5.3shows the
example waveform of 20 seconds recorded using this system.

Checker module uses ARM processor which works on a program. LCD along with
touch panel acts as touch screen which is interfaced with ARM. Touch screen first
displays title and then it draws Lock 1 and Lock 2 of brainwaves. It is shown in
Fig.5.4(a). Here, both locks are close. Touch screen indicates to touch lock 1. User
needs to generate 6 times eye blink signal after the touch for the lock1 to open it.
Fig.5.4(b) shows lock 1 open and lock 2 closed. Now, lock 2 need to be opened by
attention level of 90. Fig.5.4(c) shows both locks of brain waves opened. Then system
asks for pass key as a second level of authentication. Fig.5.4(d) shows this. Pass key
is defined in program written for ARM as ’2345’. If correct pass key is entered touch
screen displays ’Authenticated’. It is shown in Fig.5.4(e). If pass key entered is
wrong, screen displays ’Unauthenticated’.

Fig.5.2 Matlab window after Lock 2 opens

14
Fig.5.3 Sample waveform of eye-blink and attention signal

Fig.5.4Results on touch screen

15
 CHAPTER 6

ADVANTAGES
1) Keys, ATM cards, and other hardware means of authentication can be lost but not
the brain. It is present in every human (universality property).

2) Shoulder surfing or careful observation can help to determine the passwords typed.
But it is highly impossible to visualize one’s thought processes. Nevertheless, even if
the user shares or writes down his/her thought, it is highly impractical for the intruder
to reconstruct it in the same fashion as that of the user. This reduces the degree of
‘identity theft’, wherein the intruder steals the secret information of the user.

3) Physically handicap people cannot use retina scanning or fingerprinting, but brain
functions as long as a person is alive.

4) High degree of intricacy (uniqueness property) of brain makes it difficult for the
intruder to imitate other’s brain (circumvention property).

16
 CHAPTER 7

DISADVANTAGES
1) Maintaining the same thought process every time for the fixed entity is at an
increased level of complexity.

2) Brainwaves get basically altered with respect to increase in age. Therefore, the
system should adapt itself to this change effectively (stability property).

3) There is a wide range of EEG devices available in the market. Sophisticated, high
end devices are costly as well as non-portable. But they help in acquiring good quality
waveforms. On the other hand, cheap devices do not facilitate this (collectability
property).

4) The fear of leakage of mental processes in an individual is hampering its usage

(acceptability property).

17
 CHAPTER 8

COCLUSION
A portable, two-level authentication system is designed using embedded system
which records brain signals using wireless media. The design requires low-power and
low cost.

Advantages of using EEG as a biometric are described. Brain-computer interface is

developed that senses brain signals and communicates it to the computer using
Bluetooth communication protocol. The system uses battery operated, wireless,
sensing device. An algorithm is developed for extracting data from received brain
signals and interpret it to provide authentication. A sub-system is developed to
provide second level of authentication using pass key after successful brain-input
condition.

As a summary, the paper deals with the advantages of biometric based authentication
techniques and the challenges innate in them. Face recognition, though a more
challenging approach, is well established when compared to EEG based
authentication. The latter consumes considerable amount of time of the user in
wearing the device in conjunction to the processing time of the system; with respect to
the former. Brainwave authentication requires rigorous research so as to work in real
time systems and become widely acceptable.

18
 CHAPTER 9

FUTURE SCOPE
Biometric systems should be designed to handle all the vulnerabilities discussed in the
paper to a maximum extent. Neuro-signals based authentication should be fast, robust
and should gain the confidence of public by ensuring security against their mental
data disclosure. Face recognition systems should deploy all the parameters in a single
implementation, so as to do away with the existing shortcomings.

A perennial questionamong brain-machine-interface researchers is whether we should

focus on invasive or non-invasive approaches. In other words, should we design BMI
systemsthat rely on electrodes inserted inside the brain, sitting on its surface just
beneath the skull,or on those that simply rest on the scalp? This debate always
emerges at meetings and conferences.To me, however, it’s a pointless discussion.

Almost every disease can benefit from more than one type of treatment. You can treat
cancer with, among other things, surgery, chemotherapy, radiotherapy, or all of them
together. Also,there are other factors that affect the choice of therapy to follow, such
as the physical conditionof the patient. So why can’t BMI systems work in a similar
fashion? Some people may require electrodesinside the brain to get the most benefit
for controlling a robotic arm in a natural way, forexample; for others, a simple
electrode cap on the scalp may suffice say, for steering a wheelchair.

Some BMI systems record brain activity at a microscopic level, producing a signal
with two components. The high frequency part, related to the firing of individual
neurons, is known as single unitactivity, or SUA, and the low frequency part
represents local field potential, or LFP, which consistsof aggregate currents from
neurons surrounding the electrode. Another type of invasive implantrecords cortical
field potentials via electrocorticography. The fields can also be recorded from
thescalp via non-invasive electroencephalography, although the result is a noisier
signal.

In the long term, BMI systems may very well become a hybrid of different kinds of
neural signals, able to benefit from local, high-resolution information (for generating
motorcommands) and more global information (arousal, level of attention, and other
cognitivestates). Society can only benefit from the availability of both approaches, so

19
let them boththrive. And as many of my colleagues would say, at the end of the day
the decision of whichto adopt is entirely in the hands of two people: the patient and
the physician.

The initial applications of BMI, in helping patients suffering from paralysis due to
spinal cord injury or other neurologicaldisorders, including amyotrophic lateral
sclerosis and stroke,are still probably a decade or two away. But after this
technologybecomes mainstream in health care, other realms await inthe augmentation
of sensory, motor, and cognitive capabilities in healthy subjects a fascinating
possibility for sure, but onethat promises to unleash a big ethical debate. The world
wherewe’re able to do a Google search or drive a car just by thinkingwill be a very
different place.

Current user recognition systems rely on a subject’s physical attributes or behaviour.

Preliminary studies have demonstrated that using EEG signals as a biometric
identifier is potentially more secure and privacy compliant.

Researchers must overcome several challenges before a practical EEG-based person

recognition system can be deployed. These include identifying the stimuli that
produce the most discriminant mental signatures in EEG signals, optimizing the
electrode configuration to minimize subject inconvenience while guaranteeing
superior performance, and assessing signal stability over time for the same subject
and signal discriminability for different subjects.

20
 CHAPTER 10

REFERENCES

[1] G Patrizio Campisi and Daria La Rocca, “Brain Waves for AutomaticBiometric-
Based User Recognition”, IEEE Transactions on Information Forensics and Security,
Vol. 9, No. 5, pp 782-800, May 2014.
[2] S. Baca and Cubrilo, “Towards a general defination of Biometrics”, International
Journal of comouter Science Issues, 2009.
[3] E. Basar, ”Brain Function and Oscillations: Integrative Brain Function”,
Neurophysiology and Cognitive Processes (Springer series in synergetics). Berlin,
Germany: Springer-Verlag, 1999.
[4] G. Dornhege, J. d. R. Millan, T. Hinterberger, D. McFarland, and K.-R.
Moller,”Towards Brain-Computing Interfacing”, Cambridge, MA, USA: MIT Press,
2007.
[5] J. R. Wolpaw and E. W. Wolpaw, ”Event-Related Brain Potentials: Methods,
Theory, and Applications”, Hoboken, NJ, USA: Wiley, 2000.
[6] J. M. Carmena, ”Becoming bionic,” IEEE Spectr., vol. 49, no. 3, pp. 24- 29, Mar.
2012.
[7] H. H. Stassen, ”Computerized recognition of persons by EEG spectral patterns,”
Electroencephalogr. Clin.Neurophysiol., vol. 49, nos. 1-2, pp. 190-194, 1980.
[8] P. Campisi, D. La Rocca, and G. Scarano, ”EEG for automatic person
recognition,” IEEE Comput., vol. 45, no. 7, pp. 87-89, Jul. 2012.
[9] J. Berkhout and D. O. Walter, ”Temporal stability and individual diff erences in
the human EEG: An analysis of variance of spectral values,” IEEE Trans. Biomed.
Eng., vol. 15, no. 3, pp. 165-168, Jul. 1968.
[10] H. Van Dis, M. Corner, R. Dapper, G. Hanewald, and H. Kok, ”Individual
differences in the human electroencephalogram during quiet wakefulness,”
Electroencephalogr. Clin.Neurophysiol., vol. 47, no. 1, pp. 87-94, 1979.

21