Documente Academic
Documente Profesional
Documente Cultură
on
ARTIFICAL PASSENGER
Submitted in
Partial fulfillment of the requirements
for the award of
Bachelor of Technology
in
Information Technology
By
BHEEM REDDY SHIVANI
[B18IT062L]
2019-20
DEPARTMENT OF INFORMATION TECHNOLOGY
CERTIFICATE
I wish to take this opportunity to express my deep gratitude to all the people who have
extended their cooperation in various ways during my Seminar. It is my pleasure to
acknowledge the help of all those individuals.
In completing this Seminar successfully all our faculty members have given an
excellent cooperation by guiding us in every aspect. All your guidance helped me a lot and I
am very grateful to you.
BHEEMREDDY SHIVANI
(B18IT062L)
ABSTRACT
The system interacts verbally with a driver to reduce the likelihood of them falling
asleep at the controls of a vehicle while driving during a long trip or one that extends into
the late evening. It is equipped to engage a vehicle operator by carrying on conversations,
playing verbal games, controlling the vehicle's stereo system, and so on. It also monitors
the driver's speech patterns to detect fatigue, and in response can suggest that the driver
take a break or get some sleep. It may also be integrated with wireless services to provide
weather and road information, driving directions, and other such notifications systems.
Artificial Passenger
CONTENTS
CHAPTER 1: INTRODUCTION 6
1.1 Introduction
1.2 Aim of the Artificial Passenger
1.3 Organization of seminar report
CHAPTER 2: ARTIFICIAL PASSENGER 9
2.1 Introduction
2.2 Existing system of Artificial Passenger
2.3 Proposed system of Artificial passenger
2.4 Why Artificial Passenger
2.5 Need of Artificial Passenger
CHAPTER 3: COMPONENTS OF ARTIFICIAL PASSENGER 13
3.1 Introduction
3.2 Architecture
3.3 Working of eye tracking
3.4 Other components
3.5 Working of Artificial passenger
3.6 Driver
3.7Drowsiness prevention
CHAPTER 4: APPLICATIONS & CONCULUSION 24
4.1 Introduction
4.2 Applications
4.3 Advantages of using this system
4.4 Future Enhancements
4.5 Conclusion
REFERENCES 26
DEPARTMENT OF IT 5 KITSW
Artificial Passenger
Chapter:1
Introduction
An Artificial Passenger (AP) is a device that would be used in a motor vehicle to make
sure that the driver stays awake. IBM has developed a prototype that holds a conversation
with a driver, telling jokes and asking questions intended to determine whether the driver
can respond alertly enough. Assuming the IBM approach, an artificial passenger would
use a microphone for the driver and a speech generator and the vehicle's audio speakers
to converse with the driver. The conversation would be based on a personalized profile of
the driver.
A camera could be used to evaluate the driver's "facial state" and a voice analyzer
to evaluate whether the driver was becoming drowsy. If a driver seemed to display too
much fatigue, the artificial passenger might be programmed to open all the windows,
sound a buzzer, increase background music volume, or even spray the driver with ice
water .One of the ways to address driver safety concerns is to develop an efficient system
that relies on voice instead of hands to control Telematics devices. It has been shown in
various experiments that well designed voice control interfaces can reduce a driver’s
distraction compared with manual control situations. One of the ways to reduce a driver’s
cognitive workload is to allow the driver to speak naturally when interacting with a car
system (e.g. when playing voice games, issuing commands via voice).
1.1 Introduction:
DEPARTMENT OF IT 6 KITSW
Artificial Passenger
The natural dialog car system analyzes a driver’s answer and the contents of the
answer together with his voice patterns to determine if he is alert while driving. The
system warns the driver or changes the topic of conversation if the system determines that
the driver is about to fall asleep. The system may also detect whether a driver is affected
by alcohol or drugs.
During the night times the driver could get sleepier which may be porn to accidents. So in
order to overcome the sleepiness the driver could have taken one of the following or all
the below precautions.
As these methods are sometimes inefficient and it may affect the health conditions of the
driver. So in order to overcome the disadvantages of these methods IBM introduces a
new sleep prevention technology device called as “Artificial Passenger” which was
developed by Dimitry Kanevsky and Wlodek Zadrozny.
This software holds the conversation with driver to determine whether the driver can
respond alertly enough .The name artificial passenger was first suggested in new scientist
magazine which was designed to make solo journey safer and more bearable. Early
techniques for determining head-pose used devices that were fixed to the head of the
subject to be tracked. For example, reflective devices were attached to the subjects head
an during a light source to illuminate the reflectors, the reflector locations were
determined.
DEPARTMENT OF IT 7 KITSW
Artificial Passenger
As such reflective devices are more easily tracked than the head itself, the problem of
tracking head-pose was simplified greatly. Virtual-reality headsets are another example
of the subject wearing a device for the purpose of head-pose tracking. These devices
typically rely on a directional antenna and radio frequency sources, or directional
magnetic measurement to determine head-pose.
DEPARTMENT OF IT 8 KITSW
Artificial Passenger
Chapter:2
Artificial Passenger
2.1 Introduction:
The biggest problem for a driver while driving alone is that drowsiness can affect easily
leading to fatal accidents and subsequent injuries. The proposed system is designed in
DEPARTMENT OF IT 9 KITSW
Artificial Passenger
such a way that the driver of a car would be in constant awake stage through a virtual
passenger which chats with the driver.
The system tracks the face movement and various organ movements such as eyes,
lips etc. By analyzing the face of the driver the application can give out a warning signal
through the car stereo speakers and manage the driver to engage in a conversation. For
that firstly we have make our voice profile and save it in the system. During driving when
the application asks question, it will detect the variation in voice by comparing with the
existing sound profile and warns us if we are feeling sleepy or we have a lack of
concentration.
The time taken for response is also noted by the system and if the response is
swift and clear from the driver, a set of questionnaire will follow which the driver can
engage with. This application can make funny and meaningful conversations regarding
any topic such as music, news, weather, traffic etc. All these conversations are managed
by a dialogue manager which inter operates with other systems inside the car such as
stereo, GPS navigation, telephone directory etc. The software giant IBM developed a
speech system caalled conversational interactivity for telematics especially for alone
drivers, inside this system will be a natural language unit which manages the language of
interaction.
A NLU is installed as embedded chipset inside the car and will have various
grammar files to make it as authentic as possible. The response of the DU would be
spontaneous. There would be an option to record and playback conversations done
between the driver and the DU. If the system experiences driver fatigue it can alert or
warn the driver to stop the car or lessen the speed.
IBM received a patent in May for a sleep prevention system for use in
automobiles that is, according to the patent application, “capable of keeping a driver
awake while driving during a long trip or one that extends into the late evening. The
system carries on a conversation with the driver on various topics utilizing a natural
DEPARTMENT OF IT 10 KITSW
Artificial Passenger
dialog car system’'. Additionally, the application said, “The natural dialog car system
analyzes a driver’s answer and the contents of the answer together with his voice patterns
to determine if he is alert while driving.
The system warns the driver or changes the topic of conversation if the system
determines that the driver is about to fall asleep. The system may also detect whether a
driver is affected by alcohol or drugs.”If the system thinks your attention is flagging, it
might try to perk you up with a joke, though most of us probably think an IBM engineer’s
idea of a real thigh slapper is actually a signal to change the channel. Alternatively, the
system might abruptly change radio stations for you, sound a buzzer, or summarily roll
down the window. If those don’t do the trick, the Artificial Passenger (AP) is ready with
a more drastic measure: a spritz of icy water in your face.
DEPARTMENT OF IT 11 KITSW
Artificial Passenger
According to the national survey in UK and USA, it is observed that driver fatigue
annually causes
100,000 crashes
15000 deaths
71,000 injuries
Which cause annual cost of $12.5 billion.
A majority of the off-road accidents observed were preceded by eye closures of one-half
second to as long as 2 to 3 seconds. A normal human blink lasts 0.2 to 0.3 second.
DEPARTMENT OF IT 12 KITSW
Artificial Passenger
Chapter 3
Components in Artificial Passenger
3.1 Introduction:
The main devices that are used in this artificial passenger are:-
Alarm
Microphone
Camera
Eye tracker
Voice recognizer or speech recognizer.
Touch sensors.
3.2 Architecture:
3.2.1 Microphone:
For picking up the words and separate them by some internally used software for
Conversation.
DEPARTMENT OF IT 13 KITSW
Artificial Passenger
3.2.2 Camera:
This will track the lip movements of the driver and also used for the improvement for the
accuracy of the speech recognition.
It controls interruption of a dialog between the driver and the car dashboard (for
This component is used to measure the temperature inside the vehicle and it also
This sensor will periodically sprinkles the sweet air inside the vehicle.
3.2.7 Speaker:
DEPARTMENT OF IT 14 KITSW
Artificial Passenger
head or mounte dremotely. Both systems measure the corneal reflection of an infrared
light emitting diode (LED),which illuminates and generates a reflection off the surface of
the eye. This action causes the pupil to appear as a bright disk in contrast to the
surrounding iris and creates a small glint underneath the pupil . It is this glint that head-
mounted and remote systems use for calibration and tracking.
There are some of the components which supports for the working of the system:
Speaker independent: It will decode the driver voice and the decoded voice
signals will output to Natural Language Processor (NLP)
DEPARTMENT OF IT 15 KITSW
Artificial Passenger
Operates with a voice car media, decodes tapes, audio books, telephone mails.
Decoding outputs of the ASR module is analyzed by Intelligent text processor and it will
output data to conversational planner.
Processes the decoded signal of textual data from ASR module, identifies semantic and
syntactic content of the decoded message , produces variants of responses and outputs
this data to a text input of the driver analyzer.
Receives the textual data and voice data from NLP and measures the time of response
using a clock. This time responses, concludes about drivers alertness and it will output to
the conversational planner. This analysis is both objective & subjective.
This is generally referred as the heart of the system and it instructs the language generator
to produce the response. If the driver continues to be in a perfect condition, then
conversational planner instructs the language generator to continue the conversation
otherwise the language generator is instructed to change the conversation.
3.4.5 Alarm:
If the conversational planner receives information that the driver is about to fall asleep
DEPARTMENT OF IT 16 KITSW
Artificial Passenger
It controls interruption of a dialog between the driver and the car dashboard (for example,
interrupting a conversation to deliver an urgent message about traffic conditions on an
expected driver route).
3.4.7 Microphone:
3.4.8 Camera:
A camera built into the dashboard used to track the lip movement of the driver to improve
DEPARTMENT OF IT 17 KITSW
Artificial Passenger
A microphone picks up your answer and breaks it down into separate words with
speech-recognition software. A camera built into the dashboard also tracks your lip
movements to improve the accuracy of the speech recognition. A voice analyzer then
looks for signs of tiredness by checking to see if the answer matches your profile. Slow
responses and a lack of intonation are signs of fatigue. If you reply quickly and clearly,
the system judges you to be alert and tells the conversation planner to continue the line of
questioning. If your response is slow or doesn’t make sense, the voice analyzer assumes
you are dropping off and acts to get your attention.
The system, according to its inventors, does not go through a suite of rote
questions demanding rote answers. Rather, it knows your tastes and will even, if you
wish, make certain you never miss Paul Harvey again. This is from the patent application:
“An even further object of the present invention is to provide a natural dialog car system
DEPARTMENT OF IT 18 KITSW
Artificial Passenger
that understands content of tapes, books, and radio programs and extracts and reproduces
appropriate phrases from those materials while it is talking with a driver. For example, a
system can find out if someone is singing on a channel of a radio station. The system will
state, “And now you will hear a wonderful song!” or detect that there is news and state,
“Do you know what happened now—hear the following—and play some news.” The
system also includes a recognition system to detect who is speaking over the radio and
alert the driver if the person speaking is one the driver wishes to hear.”
Just because you can express the rules of grammar in software doesn’t mean a
driver is going to use them. The AP is ready for that possibility:“It provides for a natural
dialog car system directed to human factor engineering—for example, people using
different strategies to talk (for instance, short vs. elaborate responses). In this manner, the
individual is guided to talk in a certain way so as to make the system work—e.g., “Sorry,
I didn’t get it. Could you say it briefly?” Here, the system defines a narrow topic of the
user reply (answer or question) via an association of classes of relevant words via
decision trees. The system builds a reply sentence asking what are most probable word
sequences that could follow the user’s reply.”
Driver fatigue causes at least 100,000 crashes, 1,500 fatalities, and 71,000 injuries
annually, according to estimates prepared by the National Highway Traffic Safety
Administration, which estimated further that the annual cost to the economy due to
property damage and lost productivity is at least $12.5 billion. The Federal Highway
Administration, the American Trucking Association, and Liberty Mutual co-sponsored a
study in 1999 that subjected nine volunteer truck drivers to a protracted laboratory
simulation of over-the-road driving. Researchers filmed the drivers during the simulation,
and other instruments measured heart function, eye movements, and other physiological
responses. “A majority of the off-road accidents observed during the driving simulations
were preceded by eye closures of one-half second to as long as 2 to 3 seconds,” Stern
said. A normal human blink lasts 0.2 to 0.3 second. Stern said he believes that by the time
long eye closures are detected, it’s too late to prevent danger. “To be of much use,” he
said, “alert systems must detect early signs of fatigue, since the onset of sleep is too late
to take corrective action.” Stern and other researchers are attempting to pinpoint various
DEPARTMENT OF IT 19 KITSW
Artificial Passenger
Car computers are usually not very powerful due to cost considerations. The
growing necessity of the conversational interface demands significant advances in
processing power on the one hand, and speech and natural language technologies on the
other. In particular, there is significant need for a low-resource speech recognition system
that is robust, accurate, and efficient. An example of a low resource system that is
executed by a 50 DMIPS processor, augmented by 1 MB or less of DRAM can be found
in [2]. In what follows we give a brief description of the IBM embedded speech
recognition system .
Logically a speech system is divided into three primary modules: the front-end, the
labeler and the decoder. When processing speech, the computational workload is
DEPARTMENT OF IT 20 KITSW
Artificial Passenger
divided approximately equally among these modules. In this system the front-end
computes standard 13- dimensional Mel-Frequency Cepstral Coefficients (MFCC) from
16-bit PCM sampled at 11.025 KHz. Front-End Processing Speech samples are
partitioned into overlapping frames of 25 ms duration with a frame shift of 15ms. A 15
ms frame-shift instead of the standard 10 ms frame-shift was chosen since it reduces the
overall computational load significantly without affecting the recognition accuracy.
Fatigue causes more than 240,000 vehicular accidents every year. Currently,
drivers who are alone in a vehicle have access only to media such as music and radio new
switch they listen to passively. Often these do not provide sufficient stimulation to assure
wakefulness. Ideally, drivers should be presented with external stimuli that are interactive
to improve their alertness. Driving, however, occupies the driver’s eyes and hands, there
by limiting most current interactive options. Among the efforts presented in this general
direction, the invention suggests fighting drowsiness by detecting drowsiness via speech
biometrics and, if needed, by increasing arousal via speech interactivity.
DEPARTMENT OF IT 21 KITSW
Artificial Passenger
When the patent was granted in May 22, 2001, it received favorable
Worldwidemedia attention. It became clear from the numerous press articles and
interviews on TV, newspaper and radio that Artificial Passenger was perceived as having
the potential to dramatically increase the safety of drivers who are highly fatigued. It is a
common experience for drivers to talk to other people while they are driving to keep
themselves awake.
DEPARTMENT OF IT 22 KITSW
Artificial Passenger
Chapter:4
Applications and Conclusion
4.1 Introduction:
The AP is an artificial intelligence–based companion that will be resident in
software and chips embedded in the automobile dashboard. The heart of the system is a
conversation planner that holds a profile of you, including details of your interests and
profession.
4.2Applications:
DEPARTMENT OF IT 23 KITSW
Artificial Passenger
4.5 Conclusion:
DEPARTMENT OF IT 24 KITSW
Artificial Passenger
References:
[1]. L.R. Bahl et al., "Performance of the IBM Large Vocabulary Continuous Speech
Recognition System on Artificial Passenger", IEEE transaction based on Artificial
Intelligence,vol.1, pp 41-44. 6 December 2015.
[2] W.G.Lehnert - Boris –An Experiment In In-Depth Understanding Of Narratives,
Artificial Intelligence
[3]. Lawrence R. Rabiner, A Tutorial on Hidden Markov Models and Selected
Applications in Speech Recognition.
DEPARTMENT OF IT 25 KITSW