FACULTY OF AUTOMATION AND COMPUTER SCIENCE

COMPUTER SCIENCE DEPARTMENT

TECHNICAL SUPPORT CHATBOT

LICENSE THESIS

Graduate: Sorin Nicu MARIAN

Supervisor: S. l. Dr. Info. Camelia CHIRA

2017
 FACULTY OF AUTOMATION AND COMPUTER SCIENCE
COMPUTER SCIENCE DEPARTMENT

DEAN, HEAD OF DEPARTMENT,

Prof. dr. eng. Liviu MICLEA Prof. dr. eng. Rodica POTOLEA

Graduate: Sorin Nicu MARIAN

TECHNICAL SUPPORT CHATBOT

1. Project proposal: The present diploma thesis presents the development of an

application which aims to replace the first level technical support with a chatbot
that responds to simple technical issues and maintain its responses through
sentiment analysis reports.

2. Project contents: Presentation page, Introduction, Theoretical approach, Analysis

and design, Implementation, Testing and validation, Conclusions, References and
Appendix

3. Place of documentation: Technical University of Cluj-Napoca, Computer Science

Department

4. Consultants: S. l. Dr. Info. Camelia CHIRA

5. Date of issue of the proposal: November 1, 2016

6. Date of delivery: June 14th, 2017

Graduate: ____________________________

Supervisor: ____________________________
 FACULTY OF AUTOMATION AND COMPUTER SCIENCE
COMPUTER SCIENCE DEPARTMENT
Declaraie pe proprie rspundere privind
autenticitatea lucrrii de licen

Subsemnatul(a) Sorin Nicu MARIAN, legitimat cu CI seria SM nr. 506835

CNP 1940410303941, autorul lucrrii: Bot Conversaional pentru Suport Tehnic,
elaborat n vederea susinerii examenului de finalizare a studiilor de licen la Facultatea
de Automatic i Calculatoare, Specializarea Calculatoare din cadrul Universitii
Tehnice din Cluj-Napoca, sesiunea Iulie 2017 a anului universitar 2016 2017 declar pe
proprie rspundere, c aceast lucrare este rezultatul propriei activiti intelectuale, pe baza
cercetrilor mele i pe baza informaiilor obinute din surse care au fost citate, n textul
lucrrii, i n bibliografie.
Declar, c aceast lucrare nu conine poriuni plagiate, iar sursele bibliografice au fost
folosite cu respectarea legislaiei romne i a conveniilor internaionale privind drepturile
de autor.
Declar, de asemenea, c aceast lucrare nu a mai fost prezentat n faa unei alte
comisii de examen de licen.
In cazul constatrii ulterioare a unor declaraii false, voi suporta sanciunile
administrative, respectiv, anularea examenului de licen.

Data Sorin Nicu MARIAN

14.07.2017 _______________________________

Semntura
Table of Contents

Chapter 1. Introduction ............................................................................... 1

1.1. Project context ...................................................................................................... 1
1.1.1. Technical Support .......................................................................................... 1
1.1.2. Customer Service ........................................................................................... 1
1.1.3. Natural Language Processing ........................................................................ 2
1.1.4. Chatbots ......................................................................................................... 2
1.2. Project proposal .................................................................................................... 3
1.3. Project motivation ................................................................................................. 3
Chapter 2. Project Objectives ...................................................................... 7
2.1. Primary objectives ................................................................................................ 7
2.2. Detailed objectives................................................................................................ 8
Chapter 3. Bibliographic Research ............................................................. 9
3.1. Chatbots overview ................................................................................................ 9
3.1.1. Background .................................................................................................... 9
3.1.2. Current status ............................................................................................... 10
3.2. Bot structure........................................................................................................ 11
3.2.1. Microsoft Bot Framework ........................................................................... 11
3.3. Input text classification ....................................................................................... 13
3.3.1. Key concepts................................................................................................ 14
3.3.2. Intent detector models ................................................................................. 14
3.3.3. IBM Watson ................................................................................................ 15
3.3.4. Microsoft LUIS............................................................................................ 16
3.4. Sentiment Analysis ............................................................................................. 18
3.5. Related Work ...................................................................................................... 19
Chapter 4. Analysis and Theoretical Foundation .................................... 21
4.1. Functional Analysis ............................................................................................ 21
4.1.1. User Summary ............................................................................................. 21
4.1.2. Main use cases ............................................................................................. 23
4.1.3. Functional requirements .............................................................................. 26
4.2. Technical Analysis.............................................................................................. 26
4.2.1. Motivation ................................................................................................... 26
4.2.2. Bot Framework ............................................................................................ 27

1
 4.2.3. LUIS ............................................................................................................ 29
4.2.4. Sentiment Analysis ...................................................................................... 31
Chapter 5. Detailed Design and Implementation .................................... 35
5.1. Architecture overview ........................................................................................ 35
5.2. Database modelling ............................................................................................ 36
5.3. Layered Architecture .......................................................................................... 40
5.3.1. Common Data Access Layer ....................................................................... 40
5.3.2. Business Layer common implementation detail .......................................... 41
5.3.3. Bot Application Architecture....................................................................... 42
5.3.4. Reports Web Application Architecture ....................................................... 43
5.4. Main technologies: an implementation perspective ........................................... 44
5.4.1. Bot framework dialog management ............................................................ 44
5.4.2. LUIS application implementation ............................................................... 46
5.4.3. Sentiment analysis use of scores ................................................................. 47
Chapter 6. Testing and Validation ............................................................ 49
6.1. V-Model .............................................................................................................. 49
6.2. Low-level testing ................................................................................................ 49
6.3. High-level testing................................................................................................ 51
Chapter 7. Users manual .......................................................................... 53
7.1. Installation Manual ............................................................................................. 53
7.1.1. Deployment Diagram .................................................................................. 53
7.1.2. Publishing the application ........................................................................... 54
7.2. User Manual........................................................................................................ 56
Chapter 8. Conclusions............................................................................... 57
8.1. Contributions ...................................................................................................... 57
8.1.1. Contextualization ......................................................................................... 57
8.1.2. Real-time decision making .......................................................................... 57
8.1.3. Natural language understanding improvement ............................................ 58
8.2. Achieved results.................................................................................................. 58
8.3. Future work ......................................................................................................... 59
Bibliography ................................................................................................ 60

2
 Figure 1.1 Email trends of customers based on age group [6]................................ 4
Figure 1.2 24/7 People that responded to a survey concerning the 24/7 availability
of the technical support of companies [6] ........................................................................... 5
Figure 1.3 Unsatisfied customers regarding on-hold calls [7] ................................ 6
Figure 2.1 Example of communication of the user with the Technical Support
Chatbot ................................................................................................................................ 7
Figure 2.2 Overall sentiment score function of all conversations .......................... 8
Figure 3.1 A sample dialog with ELIZA [9]........................................................... 9
Figure 3.2 Relationships between classes of software-based dialog systems. [10]
........................................................................................................................................... 10
Figure 3.3 Bot Framework Architecture [11] ....................................................... 11
Figure 3.4 IBM Watson Workspace example [12] ............................................... 13
Figure 3.5 Mathematical representation of an Intent i [13] .................................. 14
Figure 3.6 Processing pipeline used for personal assistant dialog systems [13] .. 15
Figure 3.7 IBM Watsons process to derive a response from a question [14] ...... 16
Figure 3.8 ICE (Interactive Classification and Extraction): Tool at Microsoft
Research for interactive learning [13]............................................................................... 17
Figure 3.9 Actual conversation with Poncho the weather expert chatbot from
Facebook ........................................................................................................................... 19
Figure 4.1 Main Use Case Diagram ...................................................................... 23
Figure 4.2 Flow of events diagram of the Input requirement use case ............. 24
Figure 4.3 Example of an open incident created in ServiceNow ticketing system
[16] .................................................................................................................................... 25
Figure 4.4 Simple example of dialog hierarchy without IScorable [17]............... 27
Figure 4.5 Simple example of dialog hierarchy with IScorable [17] .................... 28
Figure 4.6 Utterances of a built-in intent Calendar.Add [18] ........................... 29
Figure 4.7 Prebuilt domains that can be added to a LUIS application [18] .......... 30
Figure 4.8 Basic example of results from the Microsoft Text Analytics Service [19]
........................................................................................................................................... 31
Figure 4.9 Manually plotted chart representing the user reply sentiment analysis of
conversation from Table 4.2 ............................................................................................. 32
Figure 4.10 Manually plotted chart representing the user reply sentiment analysis
of conversation from Table 4.3 ......................................................................................... 34
Figure 5.1 Overall architecture of the application, including external services ... 35
Figure 5.2 Initial Entities Database model ............................................................ 37
Figure 5.3 Intermediate form of the Entities Database model .............................. 38
Figure 5.4 Final form of the Entities Database model .......................................... 39
Figure 5.5 Data Access Layer Conceptual Architecture ....................................... 40
Figure 5.6 Communication between the Data Access Layer and Business Layer 40
Figure 5.7 BaseDomainService abstract class simple implementation ................ 41
Figure 5.8 Highlight of the UnitOfWork object inherited from the
BaseDomainService Class ................................................................................................ 41
Figure 5.9 Architecture of the Bot Application module ....................................... 42
Figure 5.10 Implementation detail of getting a Subscription Model in the form of a
TV Subscription Model..................................................................................................... 43

3
 Figure 5.11 Conceptual architecture of the Reports Web Application ................. 43
Figure 5.12 Class Diagram of the Dialogs folder within the Presentation Layer . 44
Figure 5.13 An implementation of the Task GotChannel() method that represents
the process of the ScorableTVIssueDialog ....................................................................... 45
Figure 5.14 LUIS application list intents with the number of utterances for each
intent ................................................................................................................................. 46
Figure 5.15 Example of Luis intent identification in the context of
ScorableMakePayment Class ............................................................................................ 47
Figure 5.16 Scaling conversations to 100 replies (left) and to 20 replies (right).. 48
Figure 6.1 V-Model Validation Testing Model .................................................... 49
Figure 6.2 Real-time request information given by the Bot Framework Channel
Emulator............................................................................................................................ 50
Figure 6.3 Chart representing the number of requests per day made to the LUIS
application, shown in the LUIS Web interface ................................................................. 51
Figure 6.4 Sentiment Score function of all conversations .................................... 51
Figure 6.5 Sentiment Score function of specific intent conversations.................. 52
Figure 7.1 Deployment Diagram .......................................................................... 53
Figure 7.2 All resources of the Azure Portal Server [22] ..................................... 54
Figure 7.3 Homepage of the Bot Application endpoint URL [21] ....................... 55
Figure 7.4 Overview of the present channels on which the Technical Support
Chatbot is published [21] .................................................................................................. 55

4
 Chapter 1

Chapter 1. Introduction
1.1. Project context

1.1.1. Technical Support

By definition, technical support constitutes a plethora of services which certain
companies offer as assistance to users of different technology products (software or
hardware) which are supplied by the respective companies.
As a general rule, the provided services may address specific problems rather than
the provision of training, customization or other. Also, in general, this support covers
products that the mentioned companies have sold, either for free or for a certain fee.
In the present, there are few limitations to the channels over which technical support
can be delivered. Usually, it can be delivered via e-mail, phone, different ticketing
platforms that may include live support or directly on a website.
Last but not least, a form of technical support which is not provided by companies
is the help of experienced users over the Internet.
The upcoming sub chapters will present the categories of technical support by
means of coverage [1].
Call in is the most common form of technical support in the industry. In this type
of support, the customer pays for the technical products (software or hardware), but he also
pays a technical person based on a pre-negotiated rate when he encounters a problem.
Block hours is another form of technical support which may constitute paying a
minimum fee for providing a service or purchase a number of hours upfront based on an
agreed price.
Managed services is a type of technical support that includes help desk and 24/7
monitoring of servers or different services. The managed services team receives a list of
well-defined requirements that they must fulfil on an ongoing basis for a fixed fee. The
companies that offer this type of technical support are known as managed service
providers.
Crowdsourced technical support is a low-cost solution, mostly represented by a
discussion board for users. An example of such a discussion board constitutes a forum
website on which the users of the companies products to interact.

1.1.2. Customer Service

Customer service is the supply of service to customers before, during and after an
acquisition. The perception of attainment of such actions is dependent on employees who
can fit themselves to the personality of the client.
Customer service holds the high status a company assigns to customer service
relative to units such as piece of work innovation and pricing.
Customer service plays a critical role in a system's capacity to produce profit and
revenue. From that view, customer service should be involved as part of a general approach
to regular improvement [2].

1
 Chapter 1

Customer Support is a particular type of customer service provides the user the
information to make cost effective and correct use of the product. It may include the
following: installation, training, maintenance.
Automated Customer Service constitutes a form of customer service that
functions without the help of humans. In the majority of the cases, customer service is
provided by humans. Because of the need of automation and the possibility to solve it,
nowadays different automated means are implemented.
A primary example would be an Internet site, which has the advantage of 24-hours
a day service and is much more cost efficient than to have human customer service stay in
24-hours a day. Other automated means constitute touch-tone phone with a main menu
using the keypads as options (i.e. For English press 1).

1.1.3. Natural Language Processing

Natural Language Processing (NLP) is an area of research and application that
explores how computers understand and manipulate natural language text or speech in
order to achieve meaningful results.
Through natural language processing, researchers aim to gather knowledge on how
people understand, process and make use of language. This is done in such a way that
appropriate tools and techniques can be developed to prepare computer systems understand
and manipulate natural languages to perform desired tasks.
The base of NLP relies on a number of disciplines, namely, computer and
information sciences, linguistics, mathematics, electrical and electronic engineering,
artificial intelligence and robotics, and psychology. The fields on which NLP can be
applied on, include a number of fields of study, such as machine translation, natural
language text processing and summarization, user interfaces, multilingual and cross-
language information retrieval (CLIR), speech recognition, artificial intelligence, and
expert systems.
NLP constitutes an alternative to the classic mouse and keyboard (or touchscreen
nowadays). Together with the high processing power and the high amount of available
data, an NLP algorithm can provide a much faster alternative than certain commands from
physical input devices. A huge advance in speed is obtained when combining with speech
to text recognition [3].

1.1.4. Chatbots
A chatbot is a machine conversationial system which interacts with humans via
natural conversational language. Transcribed dialogue corpus has been used in software to
machine-learn conversational patterns to create a range of chatbots speaking various
languages including varieties of English, as well as French, Chinese and many other.
Chatbot technology integrates a language model and computational algorithms to
emulate informal chat communication between a human user and a computer using natural
language. The idea of chatbot systems originated in the Massachusetts Institute of
Technology (Weizenbaum 1966, 1967), where Weizenbaum implemented the ELIZA
chatbot to emulate a psychotherapist.
The generalization concept behind it was simple and founded on keyword
matching. The input was expected from a keyword in the past, but in the present,
different input method exists with the use of speech to text conversion.
2
 Chapter 1

Based on the mentioned concept, if a keyword is matched, the sentence is mapped

according to a rule associated with the keyword. If there is no keyword found, a connected
free remark or an earlier transformation (under certain conditions) is retrieved.
For example, if the statement contains the keyword mother, ELIZA can reply
Tell me more about your family. This rule is based on the theory that a mother and
her family are central to psychological problems, so a therapist should encourage the
patient to open up about their family.
The difference between ELIZA and a therapist is that the program does not really
understand this psychological strategy. It just matches the keyword and answers with a
standard response [4].

1.2. Project proposal

This thesis focuses on a development of a Technical Support Chatbot that replaces
the first layer of technical support. Moreover, great contextual enhancements can be made
within the first stages of usage.
In order to place the Technical Support Chatbot in the contexts mentioned above,
placement in each sub category will be highlighted, detailing attributes based on their
criteria.
In the domain of Technical Support, the Technical Support Chatbot would take part
in the Managed Services. It is a 24/7 service available on different channels and it requires
a very well specified list of requirements that it needs to satisfy, in order to create a chatbot
that is as contextual as possible and has a great problem-solving rate.
In the domain of Customer Service, although this domain intersects with Technical
Support, it belongs to the Automated Customer Service. The reason behind is obvious.
Many of the Technical Support Chatbots responses are automated.
The Technical Support Chatbots stand-alone Natural Language Processing is at a
primary level. The great enhancement is brought by an external Language Understanding
Intelligent Service. By integrating it in an application, it can provide meaning extraction
from natural language input.
The last and most specific domain in which the Technical Support Chatbot takes
part in is obviously the domain of chatbots. It provides fast responses based on the matching
keyword generalization concept. This concept is enhanced by the external Language
Understanding Intelligent Service, but it is the same base principle.

1.3. Project motivation

In the century of speed, people are used to instant results over everything, no matter
what the value of that result is. Because of the amount of the existent data, the results have
to be extracted more rapidly than ever in order to increase customer satisfaction.
A single second delay in a website loading time can result in a 7% loss in
conversion. Furthermore 40% of web users will abandon a website if it takes longer than 3
seconds to load [5].

3
 Chapter 1

Figure 1.1 Email trends of customers based on age group [6]

Based on a study depicted in Figure 1.1, consumers under 24 use email 3.5 times
less frequently than other means of communication. For 25 44-year olds, its almost 2
times less.
These statistics prove that there is an ascendant trend of the messaging applications
which will further increase in the future. In order for companies to assure that they
understand the customer needs, they must try to satisfy this need on constant availability
over different instant messaging applications.
Another key argument sustaining this point is that smartphone functionality has
been greatly increased in the last years, together with greater internet coverage. This means
there are now 2.1 billion messaging application users.

4
 Chapter 1

Figure 1.2 24/7 People that responded to a survey concerning the 24/7 availability
of the technical support of companies [6]

Furthermore, the same study, analyzed the answer of customers when they were
asked if a business should respond to them 24 hours a day, 7 days a week. As shown in
Figure 1.2, 50,6% agreed (on different levels) that a business should ensure this type of
support. There is a customer expectation for the instantaneity and convenience the
consumer experiences when talking with peers to also happen with the support provided
by companies.

5
 Chapter 1

Figure 1.3 Unsatisfied customers regarding on-hold calls [7]

Other important statistics which is depicted in Figure 1.3, from another study show
that most customers get frustrated being connected to a phone or computer while waiting
for customer service help. This frustration is certainly understandable.
The findings in this report suggest that clients want to use texting for customer
service activities and are moving their business to companies that satisfy this need.
In conclusion, based on the statistics mentioned above, a comparison of the quality
of the service that can be obtained from a Technical Support Chatbot with other types of
human technical support can be drawn. This includes crowdsourced technical support or
even more precarious ways in which results can be extracted from data, like the user mainly
searching and filtering the information he needs.
An advantage over the crowdsourced technical support, by the use of natural
language processing, is that the user can state his information needs in a more accessible
way.
The biggest advantage over the other category, human technical support is cost
efficient support with ensured 24/7 availability. Another notable advantage is the
standardization of the messages. Throughout the responses of the Technical Support
Chatbot, all can be standardized by means required from the company. In order to achieve
this for human technical support, numerous costly trainings are needed.
Due to current resources and machine learning algorithms, chatbots require high
training time to develop a decent amount of accuracy on a relatively small domain. A
solution for this problem is to constrain that domain even further to a level that is still
relevant but minimizes training time and maximizes accuracy.
After the bot-human conversations, the results can be put to an emotional analysis
which classifies the given text, reply by reply, giving a polarity result (0% -negative
response, 100% - positive response). By using this, the efficiency can be verified and
adjustments of the current algorithm can be made in order to achieve a greater performance.
Although chat bots and emotional analysis are solutions to 2 different problems,
they compensate each other. The latter provides an end result that can prove the company
that may use this product that it satisfies the customer needs.

6
 Chapter 2

Chapter 2. Project Objectives

2.1. Primary objectives
The primary objective of this thesis is to design a Technical Support Chatbot that
can act as a first layer of support in a team of technical support and can be greatly enhanced
within the first months of use. It can respond to simple problems or requests and forward
to the second level support if it is not capable of solving them. Also, as mentioned before,
it has the ability to improve the solution in the context of the problem area, during use (in
production). This can be achieved by analyzing the results of the reports and charts that
represent the bot chat performance in some problem categories and improving the
classification done by the Language Understanding Intelligent Service.
Furthermore, by analyzing the Technical Support Chatbot by means of objectives,
we can split the application in two.

Figure 2.1 Example of communication of the user with the Technical Support
Chatbot
The first and principal is the Bot Application which has the main objective as
interaction with the user, with an example illustrated in Figure 2.1. This includes the actual
communication with the user, the emotional analysis done behind, persisting information
into the database and interaction with the first layer of the human technical support team.

7
 Chapter 2

Figure 2.2 Overall sentiment score function of all conversations

The second main objective is the ability to improve the Technical Support Chatbot.
This part is done with restricted web applications called "Reports". Here the administrator
can view the sentiment score charts. These charts contain the average sentiment scores
function of all the replies in all the conversations, as depicted in Figure2.2. Moreover, these
charts can be filtered by means of intent. In this way, the administrator can adjust the
Language Understanding Intelligent Service to better classify the replies of the users by
adding more utterances in the areas which the administrator identifies lack of performance.

2.2. Detailed objectives

In the upcoming paragraph, a detailed list of all the objectives of the Technical
Support Chatbot will be described. These objectives were chosen conforming to the
SMART criteria [8]. SMART is a mnemonic acronym that stands for the following criteria:
- Specific target a specific area for development
- Measurable quantify a progression indicator
- Assignable provide the responsible
- Realistic tailor to realistically achievement
- Time-related specify time bounds of the achievement
With respect to the outlined SMART criteria, below can be found the detailed list of
SMART objectives:
- Publish the Reports Web Application on a publicly addressable server
- Publish the Technical Support Chatbot in the Skype Directory
- Ensure the security of the Reports Web Application through authentication with an
administrator account
- Ensure security of the database behind the bot application
- Procure an accurate display of the overall sentiment scores chart
- Procure an accurate display of the specific sentiment scores chart based on intent
classification
- Prove the positive impact of the Technical Support Chatbot by the objective
analysis of the overall sentiment scores chart and identifying the ascendant slope
- Correctly identify the user
- Correctly identify the start and end of a conversation
- Prove Multi-channel support integration
- Prove correct ticket forwarding process to the human technical support team

8
 Chapter 3

Chapter 3. Bibliographic Research

The current chapter focuses on placing the Technical Support Chatbot in a projected
timeline of the Chatbots domain. In the first subchapters, brief background information is
given in order to get a feel of the starting points of chatbots. Moreover, the current status
of the Chatbots domain is described, highlighting evolution and impact. Afterwards, the
best findings in technologies are discussed, more explicitly: Microsoft Cognitive Services
and IBM Watson.

3.1. Chatbots overview

3.1.1. Background

Figure 3.1 A sample dialog with ELIZA [9]

Radziwill [10] highlights the history of chatbot technologies which is very vast and
dates back to 1960. Original systems like ELIZA [8], were built as a proof of concept to
pass the Turing Test (the computer system is put to a test by a human interrogator; in order
for the computer system to pass the Turing test, the human interrogator has to mark it as
intelligent as a human).
As depicted in Figure 3.1, ELIZA lacked such capabilities. It used keyword
matching and minimum context identification and it could not pass the Turing test. Even

9
 Chapter 3

though it was very primitive, ELIZA constituted the base of the following chatbot systems
and was a great step in this domain.
In 1980 ALICE was developed [10]. This chatbot marked another great step, not
for its capabilities, but for providing a foundation for the Artificial Intelligence Markup
Language (AIML) which is used to declare rules for pattern-matching. AIML is based on
eXtensible Markup Language (XML) and it supports the majority of chatbot platforms used
today.

Figure 3.2 Relationships between classes of software-based dialog systems. [10]

As depicted in Figure 3.2, chatbots are a sub-category of Dialog Systems and more
specifically of Conversational Agents. IVR systems constitute also Dialog Systems, but
they are not used as Conversational Agents in the majority of the use cases. An example of
IVR is typically used for Telephone Service Providers: For English Press 1. Furthermore,
Embodied Conversational Agents are virtual forms of entities that can exist in real life
(example: persons, animals, etc.) or imaginary (example: avatars, humanoid robots).

3.1.2. Current status

With the help of open-source code and Software as a Service (SaaS), chatbot
systems have greatly improved in the present time. A key argument sustaining this point
is, cited from [10], Between 2007 and 2015, chatbots were participating in a third to a half
of all online interactions.
This high percentage of the presence of chatbot systems in the online medium is
possible due to their increased performance. They mislead humans into thinking they are
real in many contexts and sustained conversations.
Furthermore, as Radziwill points out in [10], In the 2016 US Presidential election,
up to a fifth of the comments and responses on Twitter were driven by fully or partially

10
 Chapter 3

autonomous Twitter accounts. This was unfortunately intended to artificially increase the
number of followers, resulting in the increase of certain social statuses which had a great
impact on the entire globe, eventually.
In the following sub-chapters, alternative technologies for building a Technical
Support Chatbot will be discussed. The respective technologies are Microsoft Cognitive
Services and IBM Watson. Although they have different structures, the analysis will be
done with respect to a concrete structure. An example, contained in the Input text
classification part is that IBM Watson does not have a separate service for this need. All is
done within a single API.

3.2. Bot structure

Both technologies offer support for multiple channels, together with easy guides to
publish the chatbots. Also, the principal component of the bot structures in general, those
technologies included, is the Dialog. Further details are different, with Microsoft Bot
Framework having many advantages over the bot structure of IBM Watson.
The basic definition for Dialog is the following: Dialog is an entity that helps to
model a conversation and to manage a conversation flow. A dialog can be composed with
other dialogs to maximized reuse. Within a Dialog, alternative flows of the same use cases
should be handled in order to have an organized and correct structure which encourages
reusability.
Channels have the same meaning of concept in both technologies. Examples of
channels are: Facebook Messenger, Skype, etc. There is another advantage for the
Microsoft Bot framework regarding this point of view, namely Direct Line. Through an
HTML Embed code snippet, a direct line chat can be inserted in a page of a certain web
application. This feature is lacking in the IBM Watson.

3.2.1. Microsoft Bot Framework

Figure 3.3 Bot Framework Architecture [11]

11
 Chapter 3

In Figure 3.3, the Microsoft perspective of an architecture of a bot can be observed.

Integration with the Language Understanding Intelligent and multi-channel support are
highlighted. Microsoft Language Intelligent Understanding Service (short, LUIS) takes as
input data the messages received from the user through Microsoft Bot framework, it
classifies this information and sends output data back to the Bot API.
Furthermore, brief descriptions of the particular concepts used in the Microsoft Bot
Framework are mentioned.
Dialog flow is a concept for managing dialogs within a chatbot application. The
start of a dialog in a bot application is managed by the Message Controller. Within the
controller certain actions can be implemented if an activity occurs (example: Type ...
when the Typing activity from the context of the bot occurs).
Moreover, dialogs are kept in the Dialog Stack. Whenever a new dialog is created
and a message is processed within that Dialog, the current Dialog is added to the top of the
Dialog Stack.
However, managing Dialogs from a primary level perspective is done in the main
Dialog, namely Root Dialog. Here there are treated all the input text classification that may
lead to additionally creating new Dialogs.
A more complex approach to manage Dialogs dynamically is with IScorable. With
the IScorable Interface, a Scorable class can be implemented for each Dialog class which
grants the score. The score means the probability of a Dialog to be the right choice for a
given input from the user.
A medium complexity approach constitutes having scores that are integers (either
1 or 0). A high complexity approach has float scores varying from 0 to 1.
Activity represents the concept of an event, regardless of the actor that generated it
(user or bot) and has metadata which can be used to identify brief contextual information.
A very primary example would constitute the IsTyping activity, which can be
used to show graphical output so that the user is informed if the bot is thinking. Another
example that would be higher in the Activity hierarchy is the Reply Activity. By viewing
activity from this perspective, essential information can be extracted, like Text or
From.

12
 Chapter 3

3.2.1.1. IBM Watson Bot structure

Figure 3.4 IBM Watson Workspace example [12]

Although both technologies have a framework, the main difference is that IBM
Watson does not have very strict borders to separate the bot structure from the input
classification, while Microsoft goes with a more SaaS (Software as a Service) approach.
Workspace - as mentioned in the previous paragraph, all the information is placed
here. The brain of an IBM Watson chatbot is within the same place as the body. As
depicted in Figure 3.4, the Workspace can be accessed from a web interface with many
suggestions from IBM on how to model the present entities.
Dialog this concept is viewed as a service. Through it, a comprehensive platform
for conversation management can be built. What is done in the Microsoft Bot Framework
Dialog, IBM covers in the Dialog Flow.
Dialog Flow represents the management of a conversation with the help of decision
trees. Contextual conditions are implemented in nodes. Based on the user input,
classification is done by searching in the decision tree for a node that contains the condition
that is satisfied.
Further complexity can be built with the help of the decision tree algorithm, but
that will be later covered in the following sub-chapter, namely Input text classification.

3.3. Input text classification

The current subchapter focuses on finding the similarities and differences of the two
mentioned technologies with respect to input text classification. Both use Intent detector

13
 Chapter 3

models as classifiers, but for further understandings, the key concepts have to be
explained.

3.3.1. Key concepts

An Intent represents the purpose of an action a user wants to make expressed as in
a user input. A set of named intents may be that corresponds to actions that are specific to
the application may be defined. For example, in an Alarm application, the SetAlarm,
GetAllAlarms are possible intents.
An Utterance is the textual input from the user that the application has to interpret
and it represents an example of an Intent. It may be a sentence: Set an alarm at 8:00 AM
or a fragment: 8:00 AM alarm. Due to usual user input behavior, they are not always
well-formed and there can be vast variations for a particular intent.
An Entity can be viewed as the noun in a sentence but in a context of an Intent. It
represents an instance of an object relevant to the user intent. For example, in the Utterance
Set an alarm at 8:00 AM we can identify the Time Entity, more specifically
represented by 8:00 AM.

3.3.2. Intent detector models

In both mentioned technologies, Intent detector models classifiers are used as the
main input text classification algorithm. By using this type of classifiers, sequences of
words can be mapped to sets of pre-defined or contextualized intents.

Figure 3.5 Mathematical representation of an Intent i [13]

According to Williams [13] , a more mathematical description of an intent can be
found in Figure 3.5: Intent i is a model of Pi(y|x) structure, where x stands for the words in
the utterance, and y stands for the binary variable. If y = 1 the formula indicates that the
intent is present in the utterance and if y = 0 indicates it is not. If the input text is not present
in any domain utterances of all intents, the (i = ), the formula becomes: P(y = 1|x).
The model can be estimated in many ways, for instance by using machine learning
algorithms such as Support Vector Machine [13] and Deep Neural Networks [13], but this
implementation detail is not made publicly available for either of the mentioned
technologies.

14
 Chapter 3

Figure 3.6 Processing pipeline used for personal assistant dialog systems [13]
In Figure 3.6 the process of classification of a user input (text or speech) on a
concrete example can be found. It shows how Intent detection is separated from the Entity
extraction (parallel) and how a function call can be made after classification. In that latter
function call response management is handled.
Moreover, Figure 3.6 presents a case study for digital assistants scheduler
functionality. Built-in intents can be observed (example ReadFromCalendar) which infer
the user intent. In parallel, entity extraction from the utterance substring is done,
identifying, in this case, mostly entities such as the date today or noon. After
extraction, through entity resolution, the entities are mapped to canonical forms such as:
2017-03-22.

3.3.3. IBM Watson

An enhancement that IBM Watson brings, besides the algorithm mentioned in the
previous chapter, Identity detector models, this technology has a great capability of
understanding text of a higher size order.
When it comes to text Watson doesn't just look for key word matches or synonyms,
like a search engine. It actually reads and interprets text like a person does, by breaking
down a sentence grammatically, relationally and structurally, discerning meaning from the
semantics of the written material.
This is very different than simple speech recognition which is how a computer
translates human speech into a set of words. Watson tries to understand the real intent of
the users language and uses that understanding to possibly extract logical responses and
draw inferences to potential answers.

15
 Chapter 3

Figure 3.7 IBM Watsons process to derive a response from a question [14]

Knowledge corpus is very important for the operation of Watson mentioned in

Figure 3.7 by High [13]. This corpus corresponds to all categories of unstructured
knowledge. By going through the content to into a form that is easier to work with, Watson
assimilates the corpus.
The steps of the process mentioned in Figure 3.7 can be detailed as follows:
When a question is presented to Watson, it analyzes to extract the major
features of the question.
It develops a range of hypotheses by searching across the corpus for sections
that have a potentially valuable response.
It executes a deep examination of the language of the question and the
language of each potential response by using various reasoning algorithms,
which is a very challenging step.
Each reasoning algorithm yields one or more scores, indicating the depth to
which a likely response is inferred.
Each resulting score is then weighted against a statistical model which can
be used to summarize a degree of confidence that Watson has about the
candidate answer.
Watson repeats this process for each of the candidate answers until it can
discover responses that surface as being stronger candidates than the others.

3.3.4. Microsoft LUIS

Microsoft Language Understanding Intelligent Service (LUIS) provides increased
performance to understand human language and react accordingly to user requests. Any
application that initiates conversations with users, in this case, chatbots, can pass the user
input to the LUIS application and receive the results that provide natural language
understanding.
Williams [13] mentions a technique called Interactive Learning which is used in
Microsoft LUIS to solve the problem of rapidly adding intents, which is one of the main

16
 Chapter 3

bottlenecks to scaling applications like this. This technique is executed on the Intent
detector models previously mentioned.
Interactive Learning (IL) is a way of efficiently making classification models,
where classifier labeling, model building, and evaluation are done by a single developer.
Similar to active learning, IL is fit when unlabeled data is abundant but labeling is
expensive.
IL integrates active learning but extends it substantially. It requires a broad database
of unlabeled data instances, such as text of utterances. The database must contain positive
instances. This allows intent detectors to be built in advance of practical implementation,
at least for intents expressed at the first turn.

Figure 3.8 ICE (Interactive Classification and Extraction): Tool at Microsoft

Research for interactive learning [13]
In Figure 3.8 the ICE Tool that does the active learning behind the Microsoft
Language Understanding Intelligent Service is presented. In the top center, a search term
or a score from which to sample can be entered. Utterances are shown in the central panel,
with their scores under the current model shown to the left of each utterance. The red

17
 Chapter 3

(False) and green (True) bars indicate their suggested label by the model, which can be
changed by clicking.
The performance of the model is shown in the bottom panel, including confusion
matrices, precision-recall curves, and area under the curve (AUC) as a function of how
many labels have been entered. Features can be added and edited on the left.
The procedure starts by searching for data instances using textual search terms
based on their domain knowledge. The searches yield results are then labeled. Labels can
be positive, negatives, or a special dont know label.
After each label is entered, all the labels are randomly divided between a training
and test set. Afterwards, a model is built on the training set which is used in 3 ways.
First, all of the instances labeled so far can be displayed graphically, showing the
distribution of scores, giving an overview of performance.
Second, when searching for new instances, the model is used to propose labels.
This speeds up labeling and also gives an indication of the overall performance of the
model on unseen data.
Third, each time the model is rebuilt, it is applied to all of the unlabeled data, which
allows drawing samples of unlabeled instances at a particular score.

3.4. Sentiment Analysis

Sentiment Analysis identifies, extracts and quantifies the affective states of
subjective information, regardless of size. Its objective is to find the attitude of a human
actor which has provided the input text with respect to the overall contextual polarity or
emotional reaction to a document, interaction, or event.
The result of sentiment analysis is a percentage ranging from 0 to 100, that shows
how the input text is placed on a scale from negative (0) to positive (100). The accuracy of
a sentiment analysis is, how well it aligns with human assessments. This is usually
measured based on precision and recall over the two target categories of negative and
positive texts.
Emotional Analysis (widely known as Emotion recognition) is a higher form of
sentiment analysis that further categorizes the input text from a different perspective. The
final, practical result is not a number, but a label. These labels can be, for example:
Happiness, Anger Disgust, Fear, Sadness, Surprise, among many others.
These emotions are understood to be cross-culturally and universally communicated.
Microsoft Text Analytics has 3 key features: Sentiment analysis, Key phrase
extraction and Topic detection. In this subchapter research will be solely focused on
Sentiment analysis.
Sentiment score is produced by applying classification methods, and returns a score
between 0 and 1. The input features to the classifier include n-grams, features generated
from part-of-speech tags, and embedded words.
This type of analysis can be a method for evaluation of products, without having
the user to input feedback explicitly. In the domain of Chatbots, sentiment analysis can be
applied either on replies or conversation. For a more accurate, real-time analysis which can
be a source for decision making algorithms, the instant replies should be chosen.

18
 Chapter 3

3.5. Related Work

This subchapter focuses on a comparative analysis of relevant existing chatbots,
but not on the Technical Support domain only, because of the strict business environment
which limits to preview only features in order to protect their products.
Comparisons to chatbots that have publicly available algorithms would not be
objective nor relevant to the case, as most of them are outdated. The Technical Support
Chatbot would not be better only by means of natural language processing capabilities, but
also by the multi-channel support.
As comparison subject, the most popular chatbot from the most popular social
media channel was chosen, namely Poncho [15], available on Facebook Messenger. It is
designed to be a helpful weather expert. It can send alerts up to twice a day (if the user opts
in) with the weather conditions and can answer questions like Should I wear a jacket
today?.

Figure 3.9 Actual conversation with Poncho the weather expert chatbot from
Facebook

19
 Chapter 3

As previously mentioned, due to privacy policies, only capabilities of Poncho

and the Technical Support Chatbot will be compared.
A main feature Poncho provides is the daily recommendations, meaning that it is
a pro-active bot. It does not wait for user input in order to post messages and it can be seen
as a content generator. It is a different approach from Technical Support Chatbot which
always waits for user input to provide a message, like a response.
As depicted in Figure 3.9, Poncho provides rich user interface and a much faster
approach than writing text, through the help of Cards. Cards are substituent for buttons
in the environment of an instant messaging application. Through cards, Poncho
influences the behavior of the user in such a way that he will, most likely, pick one of the
options represented on the cards rather than writing text. A huge benefit of this usage is
that it can correct spelling mistakes, for example. As overview, this can have a great impact
on the overall performance of the chatbot.
The Technical Support Chatbot does not implement this type of input because of
contextualization. It does not make sense to use a default card, for example TV Program
is not working, because the user has to provide the exact TV Program that does not behave
correctly.
A feature that Poncho lacks is the contextualization. Although it has a scarcely
idea of context, its responses are modeled only by location. More explicitly, it will give
weather recommendations based on the user location, but it does not know the users
identity. All the responses to users from the same area are modelled exactly the same.
Different responses to users within the same location can be only acquired if the user sends
some particular context information.
The approach of the Technical Support Chatbot differs, meaning whenever the
chatbot is added to the channel of a user (note: the user has to be a customer of the company
that provides Technical Support through the respective chatbot), it gets all the information
of the user from the database. In this way, for example, all subscriptions that the user has
enrolled to, are available for the Technical Support Chatbot to work on and can constitute
a source of decision making.
In conclusion, the Weather Expert Chatbot provides rich user experience and may
have greater natural language processing power over the Technical Support Chatbot, but
the latter offers the great advantage of contextualization and may have increased
performance within the specific domain of technical support.

20
 Chapter 4

Chapter 4. Analysis and Theoretical Foundation

The current chapter focuses on presenting the requirements analysis, technologies
and algorithms used in the process of developing the Technical Support Chatbot. Following
that, the architecture of the application that integrates all of the above will be presented in
the next chapter, Detailed Design and Implementation.

4.1. Functional Analysis

4.1.1. User Summary

Table 4.1 User Summary
Name Description Responsibilities
Normal User The persons that will
have paid for a
service/subscription
or bought a product
and would complain
about their issues to
the Technical Support
Chatbot
Technical The persons that the
Support Team users will be
Member redirected to if the
problems are not
solved by the
Technical Support
Chatbot
Stakeholder
- Use the application in an Consumer
educated manner,
without spam or other
intrusive ways
- Check background Technical
information of the user Support
profile given by the Second Level
Technical Support
Chatbot
- Check conversation logs
- Identify the user problem
- Try to find a solution

Administrator The person that will - Ensure application Technical

have access to the maintenance Support Team
reports generated by Lead
- Analyze the charts
the Technical Support
Chatbot - Improve performance

Table 4.1 summarizes all the users of the Technical Support Chatbot application
and provides a brief overview of their responsibilities and stakeholder placement.
The Normal User constitutes the application user with the lowest level of access.
In order to have the ability to add the Technical Support Chatbot in a channel of their
choice, they must be enrolled as customers. This has to be done without their explicit

21
 Chapter 4

knowledge, in such a way that the chatbot is not publicly available in the specific channel
directory.
The responsibility of the Normal Users is to use the application with the specific
purpose of solving a problem they have with a product or service they are paying for.
The Technical Support Team Member is the application user which handles the
problem that cannot be solved by the Technical Support Chatbot.
When the chatbot does not have the necessary knowledge to find a solution to a
problem given by a Normal User, it forwards a ticket to an email address that represents
the whole Technical Support Team. The ticket is presented as an email. The subject of
the email contains the id of the conversation and the body would contain a short message,
together with the conversation logs.
The interaction of the Technical Support Team Member with the application is not
done via the instant messaging channel, nor via email. A ticketing platform should be
configured in such a way that whenever an email is sent to the Technical Support Team
email, a new ticket within the specific platform should be created.
The Administrator is the application user that has the responsibility of ensuring
that everything works in a correct manner. This type of user has the higher level of access,
including the server on which the application is deployed.
An important responsibility of the administrator within the first months of the
application usage is to improve performance. This can be done after the analysis of the
charts, by adding new utterances to the intents that lack performance.

22
 Chapter 4

4.1.2. Main use cases

Figure 4.1 Main Use Case Diagram

Figure 4.1 depicts the main use cases of the Technical Support Chatbot application,
highlighting the separation of the 3 actors within each different system.

23
 Chapter 4

Figure 4.2 Flow of events diagram of the Input requirement use case

24
 Chapter 4

Figure 4.2 provides a detailed overview of the single use case performed by the
Normal User. He is the only actor that interacts with the Chatbot application through the
specific channel interface. He has the option to write as text input a problem he has with a
paid service or product. He must provide objective information and try to respect the
requests provided as responses from the Technical Support Chatbot (example: I did not
understand that. Did the provided solution work? Please answer yes or no?).
Furthermore, after the user specifies the problem, the chatbot provides a possible
solution that has to be confirmed by the user. Based on this decision, the chatbot can
provide other similar solutions or end the communication. The action of forwarding the
problem to human technical support is included in the event A solution is provided in
order to set a clear separation between the Input problem use case and the Investigate
forwarded ticket use case.

Figure 4.3 Example of an open incident created in ServiceNow ticketing system

[16]

In Figure 4.2 an external Ticketing System environment is depicted. Within this

system, interaction between the Technical Support Team Member and the Technical
Support Chatbot is done via an email that opens a new ticket.
Depending on the policies and service level agreements of the specific company
that the Technical Support Team Member works for, he has to take the ticket forwarded
from the chatbot, which constitutes a replacement for the first layer of technical support in
a specified amount of time.
The administrator uses the application through the Reports Web interface. Within
this system, he can view the overall performance of the application or chose to view
25
 Chapter 4

specific performance of certain intents (example: view performance of the InternetIssue

intent).
Another use case which can be done by the administrator is improving performance.
After analyzing the performance charts. By accessing a specific intent, the administrator
can add new utterances. This results in increased chances of recognizing more forms of
user input text regarding the specified intent and, in the end, to an increased overall
performance.

4.1.3. Functional requirements

The current subchapter provides a list of the functional requirements of the
Technical Support Chatbot application based on the analysis provided in the previous
subchapters. They define the function of all the internal and external components included
in the Technical Support Chatbot application.
The functional requirements of the system are:

- The Normal User shall be able to add the Technical Support Chatbot in the
messaging channel only if he is a customer
- The Normal User shall be able to input a specific requirement via text in the
instant messaging channel and he shall receive a response from the
Technical Support Chatbot that contains a solution
- The Technical Support Team Member should be able to investigate a ticket
forwarded from the chatbot, given correct metadata and conversation logs
- The Technical Support Chatbot Team Member should be able to view all
the necessary information about a user profile from the external Ticketing
Platform System
- The administrator of the application should be able to authenticate given he
provides correct credentials
- The Technical Support Chatbot administrator should be able to view overall
performance charts
- The application administrator should be able to view specific intent
performance charts
- The administrator should be able to add utterances in order to improve
performance

4.2. Technical Analysis

4.2.1. Motivation
This subchapter focuses on the motivation behind the decision of choosing a set of
technologies for developing the application.
Based on the functional analysis illustrated in the previous subchapter, and the
analysis of the studies documented in chapter 3. Bibliographic Research, the following
conclusions can be drawn.
Microsoft Cognitive Services focuses on a more basic level of natural language
understanding that lets the developer formulate the answers, while IBM Watson goes a step
higher to extract possible answers directly from the processed input text.

26
 Chapter 4

The set of technologies included in Microsoft Cognitive Services is more

appropriate for chatbots with a medium natural language processing, but offers great
support for a complex and even dynamic structure of responses.
IBM Watson does not provide a great skeleton for a complex structure of responses.
It is appropriate for more complex problem-solving scenarios that can be found in the fields
of medicine, for example.

In the field of Technical Support, the approach taken by Microsoft is recommended,

because in this way, specific responses can be created and the full control over the response
management is granted.
In line with the conclusions drawn above, Microsoft Cognitive Services were
chosen as a base layer of technologies, which include the following:
- Bot Framework
- Language Understanding Intelligent Service
- Sentiment Analysis from the Text Analytics Service

In the following subchapters, all the chosen technologies from Microsoft Cognitive
Services will be described, highlighting the best approach to use them, considering the
most important problems that may be encountered.

4.2.2. Bot Framework

This technology provides great support for dynamically switching between dialogs
at run-time, based on the best score. The score is given by a matching pattern, which in this
case is handled by LUIS.
The ability to dynamically interrupt a dialog by a certain user input that gives the
best score from another dialog is possible with the help of the IScorable interface.

Figure 4.4 Simple example of dialog hierarchy without IScorable [17]

27
 Chapter 4

In Figure 4.3, Pretty [16] illustrates a simple dialog hierarchy in the domain of a
bank chatbot application. Within the BalanceDialog, the user can check the balance of
his current account or of a savings account. In the MakePayment dialog, the user can opt
to make a payment, provided he specifies the destination account and the amount.
Without the implementation of the IScorable interface, if the user asks to check
balance and changes his mind and wants to make a payment instead, he would have to
continue to the normal process which constitutes checking the current balance and then
returning to the Root Dialog. Only afterwards the user is able to issue an action of making
a payment.

Figure 4.5 Simple example of dialog hierarchy with IScorable [17]

In Figure 4.4, Pretty [16] the same simple dialog hierarchy is depicted, but with the
IScorable interface implemented. In comparison with the first approach, from Figure 4.3,
an obvious improvement in the loose coupling design principle can be identified.
With the IScorable approach, if the user issues a make payment action in the middle
of another action of checking balance, the current dialog is switched and the user can
execute the make payment action without going through the whole process.
When returning from the MakePayment dialog, the dialog is switched back to the
CheckBalance dialog. This is done with the help of the Dialog stack. When an interrupt
is initiated by the user input text, the current dialog CheckBalance is switched to
MakePayment dialog, meaning it goes to the top of the Dialog stack. At return, the
second dialog CheckBalance comes back to the top of the Dialog stack.

A key ability that Microsoft Bot Framework lacks is the poor support for easily
integrating a dependency injection container. The problem is that the dialog classes are
serializable and they cannot depend on instances of services that are not serializable.

28
 Chapter 4

The solution is to use a static dependency injection container that provides static
instances of the services in the serializable Bot Framework dialogs. In this way, even
though the referenced services are not serializable, it manages to successfully solve the
problem.

4.2.3. LUIS

Figure 4.6 Utterances of a built-in intent Calendar.Add [18]

Figure 4.5 depicts a simple example of the interface of the LUIS application. More
explicitly, it illustrates the Calendar.Add built-in intent which is provided by LUIS
among many others. Even though it is built-in, the intent can be specifically modelled for
the context of a particular application. For example, an utterance Meet Mike at 9:00 AM
tomorrow can be added.

29
 Chapter 4

Figure 4.7 Prebuilt domains that can be added to a LUIS application [18]
As depicted in Figure 4.6, LUIS offers the possibility to add an entire set of intents,
together with utterances in order to set up an application rapidly.
In the current domain of Technical Support Chatbots, the Utilities prebuilt
domain proved to be useful. It provides intents as Utilities.Confirm, Utilities.Cancel
which provide a greater performance rather than implementing these intents manually.
A very important aspect of the LUIS application constitutes the training. As
mentioned in chapter 3. Bibliographic research, LUIS makes use of interactive learning
which is a form of active learning. After adding new intents or utterances, the LUIS
application requires training. This results in a short down-time of the application and it
should not be done while the application is in production, but it should be scheduled, for
example once a month is a best practice recommendation.

30
 Chapter 4

4.2.4. Sentiment Analysis

Figure 4.8 Basic example of results from the Microsoft Text Analytics Service
[19]
In Figure 4.8 the results of the Text Analytics Service are depicted on a default
example. It is used to extract information from an input text of any size. The text analysis
includes language detection, key phrases extraction and sentiment detection.
Of the provided results, only the sentiment score is relevant to the Technical
Support Chatbot application.
In order to achieve a high accuracy sentiment detection of the user input,
communication with the user within the technical support domain was analyzed. The
analysis included frequency, size and total number of input requests forms from a user,
such as replies and conversations.

31
 Chapter 4

A clear advantage of extracting a sentiment score from each reply individually over
extracting the sentiment score from an entire conversation is the ability to monitor real-
time the emotional impact the chatbot responses has over the user.
Another advantage of this approach is the accuracy, but it comes with a cost.
Making a request to the Text Analytics Service each time the user sends a reply to the
Technical Support Chatbot may slow down the application.

Table 4.2 User reply sentiment analysis of a simple conversation in the technical
support domain
Sentiment score User reply text
98 Hello
31 I have a problem with the internet
2 My internet is not working
81 ok
58 I put it
48 now what?
81 ok
73 I'll wait
100 great
99 thank you

Table 4.2 depicts a simulation of a real-life scenario within the technical support
domain. A simple conversation was chosen in order to provide a base for constructing a
function of the sentiment score of a conversation.

Figure 4.9 Manually plotted chart representing the user reply sentiment analysis
of conversation from Table 4.2

32
 Chapter 4

In Figure 4.9, a manual plotting of the conversation sentiment score function is

depicted. In the first part of the conversation, up to point 3 we can identify a descendent
slope. This represents the transition of the users Hello greeting intent to the specific
problem he has. The users problem is represented by the minimum point of the function.
After the problem has been specified, the chatbot provides a solution which has a positive
impact on the user. This can be depicted from a scarcely ascendant slope from
approximately point 3 to point 9.
Problem-solving ratio - through the sentiment analysis of replies, a more concrete
performance metric of the application, namely problem-solving ratio of the Technical
Support Chatbot.
The problem-solving ratio should exclude the conversations which had problems
that were out of the Technical Support Chatbot knowledge base and forwarded to the
human technical support. A problem consists of wrongfully classifying this type of
conversations as successfully solved because of the ascendant slope given by the user
satisfaction if he, for example thanks the Technical Support Chatbot for forwarding to
human support. Further verifications have to be made in order to exclude them.
This can be calculated as following: from point 3 to point 9 the scarcely ascendant
slope can be identified. If the sentiment score function of a conversation has ends with an
ascendant slope, it can be concluded that the problem of the user has been solved, thus his
increase of sentiment polarity. If the conversation does not end in an ascendant slope, it
can be concluded that the users problem has not been solved.

Table 4.3 User reply sentiment analysis of a conversation in the technical support
domain with 2 problems
Sentiment score User reply text
97 Hi
26 I have an internet problem
8 INTERNET IS DOWN
80 ok
100 Great. It's solved
70 I have another problem
35 HBO channel shows a black screen since Monday
98 Ok. I forgot that, thank you.

In Table 4.3 a conversation with 2 different problems is depicted. More explicitly,

a problem of a possible intent InternetIssue and another one with TVIssue. This type
of conversations may provide inaccurate results for the construction of a sentiment score
function of a conversation.

33
 Chapter 4

Figure 4.10 Manually plotted chart representing the user reply sentiment analysis
of conversation from Table 4.3

In Figure 4.10 the sentiment score function of the conversation from Table 4.3 is
depicted. By analyzing points 3 and 7, the 2 problems can be clearly identified.
In order to provide a better accuracy of the sentiment analysis of replies, which
results eventually in a performance metric of the application, like the problem-solving
ratio, the decision of splitting conversations like the one depicted in Table 4.3 and Figure
4.10 in 2 has been made.
This is done with the help of LUIS and Bot Framework. If an intent from the
Issues sub-domain, such as InternetIssue or TVIssue, can be identified a second time
in a conversation, the Technical Support Chatbot creates a new conversation in the
background, without the user knowledge.

Identifying angry replies through the real-time sentiment analysis of replies,

finding out when the user gets angry can be made possible. For example, if 3 consecutive
replies have a sentiment score below 30, then it results the user is getting angry, and as a
result, it should be transferred to the second level support.

34
 Chapter 5

Chapter 5. Detailed Design and Implementation

The present chapter provides the detailed architecture of the system, main
components descriptions, references to external services, database modelling and critical
design decisions taken during the development process of the Technical Support Chatbot
application.

5.1. Architecture overview

Figure 5.1 Overall architecture of the application, including external services

In Figure 5.1 the Technical Support Chatbot architecture is depicted, highlighting

the main components and the relationships between them.
The key modules of the application are represented by the following:
- Bot Application
- Reports Web Application
- Data Access Layer
Both the Bot Application and the Reports Web Application are based on the
Layered Architecture, having a common Data Access Layer to the Entities Database.
The Bot Application module focuses primarily on the interaction with the user and
storing data that comes from this communication.
The Reports Web Applications focus is getting the conversations data from the
Entities Database, processing and showing the overall sentiment score functions of the
conversations through a Web interface.

35
 Chapter 5

The Data Access Layer focus is obviously to separate the logic that retrieves the
data from the Entities Database and maps it to the entity model from the business logic
through Entity Framework.
In the following subchapters, each principal module will be further detailed,
together with the common architecture of the Bot Application and the Reports Web
Application, but not before presenting the database model of the main entities.

5.2. Database modelling

The Entities Database has the following main models:
- User represents a unique instance of the Normal User that interacts with
the Bot Application
- Conversation is the model of an actual conversation between the Normal
User and the Technical Support Chatbot; it is mapped 1 to 1 to the Bot
Framework Conversation Class
- Reply models the activity of type reply and contains information such as
Sentiment Score that is accessed by the external services
- Payment models an invoice after the user makes a payment
- Subscription represents the subscription the user is enrolled to; can be TV
Subscription, Phone Subscription, Internet Subscription
- Equipment models the equipment that is given when a user enrolls for a
new subscription
- TV Channel represents the entity of all the TV channels available in the
TV Subscriptions (example: Digisport)
- Other tables that were solely used for mapping many to many relationships

In the process of development, the database suffered two major changes which are
detailed below, including the database diagrams.

36
 Chapter 5

Figure 5.2 Initial Entities Database model

In Figure 5.2 the initial form of the database is depicted, which had separate tables
for Subscription, TV Subscription, Internet Subscription and Phone Subscription, but did
not have the Payment table.
A key aspect in the initial database form was the decision of modelling a table for
each type of subclass subscription (internet, tv or phone).
When accessing the data in the Business Layer without a transitional mapping from
the Entity Framework models to Data Transfer Object (DTO) models and afterwards
Business Models, it is very difficult to use them in the Business Services. The actual
implementation of DTO models and separate Business Models and mapping between them
would have also been a difficult approach.

37
 Chapter 5

Figure 5.3 Intermediate form of the Entities Database model

In Figure 5.3 the addition of the Payment table is illustrated. This is the only
difference between the initial Entities Database Model and the intermediate form.
The Payment Table exists in the final form and makes an important subject in the
Input requirement use case, more specifically in the concrete cases of making a payment
or getting the latest invoice.
A second approach for solving the Subscription Table problem was found, that
constituted of eliminating the Subscription Table completely and adding the details
contained in this table (Name, Description and Price) to the Internet Subscription, TV
Subscription and Phone Subscription tables. The main disadvantage and reason for not
implementing this approach is that the many-to-many relationship between the
Subscription Table and the Equipment Table would have been very difficult to maintain.

38
 Chapter 5

Figure 5.4 Final form of the Entities Database model

with a single table for the Subscription entity

In Figure 5.4 the final form of the Entities Database model is depicted. The final
solution for the Subscription Table problem was solved by dropping the TV Subscription,
Internet Subscription and Phone Subscription Tables and moving all the fields from those
tables to a single Subscription Table. Furthermore, a Type field was added, which
represents the type of subscription (Internet, TV or Phone).
In this way, in the Business Layer, all the information from the Subscription Class
can be accessed, without having to manually map from two separate models in Entity
Framework to a single one in the Business Layer (example: number of minutes and
messages from the Phone Subscription Table and the price of this phone subscription taken
from the Subscription table).
By having all the information in one place, in the Business Layer, a Subscription
class can be accessed as a TV Subscription class with the help of an interface. This
implementation detail will be further discussed in the following subchapters.

39
 Chapter 5

5.3. Layered Architecture

The Layered Architecture or Multilayered Architecture provides a model for
creating applications that advocate flexibility and reusability. In the context of the
Technical Support Chatbot Application, the concrete model of 4-tier Architecture is
implemented.
The implemented 4-tier Architecture has the following layers:
- Presentation Layer which can provide a rich interface and heavy client side
in the case of the Reports Web Application and a minimal graphical
interface and heavy server side in the case of the Bot Application.
- Business Logic Layer is further modeled in Services which can be
categorized as Common Services and Core Services.
- Data Access Layer in this particular context, it made sense to externalize
it as a Class Library so it can be referenced from both main application
without any differences.
- Data Layer consists of the Entities Database and the Administrator
Database

As previously mentioned, the 4-tier Architecture is implemented for both the Bot
Application and the Reports Web Application, with slight differences and similarities,
which will be highlighted in the following subchapters.

5.3.1. Common Data Access Layer

Figure 5.5 Data Access Layer Conceptual Architecture

From the conceptual architecture illustrated in Figure 5.2, the use of Unit of Work
and Repository design patterns can be identified. Also, the modelling of the Entities
Database is done with the help of Entity Framework.

Figure 5.6 Communication between the Data Access Layer and Business Layer

40
 Chapter 5

The mapping from the Entity Framework models to Business Domain Models was
implemented with the help of Domain Mapper. This design pattern is practically
implemented as a service and located in the Business Layer, but conceptually, it stays
between the limit of the Data Access Layer and the Business Layer.

5.3.2. Business Layer common implementation detail

This subchapter focuses on concrete implementation detail (not structural) that are
common to the Bot Application and the Reports Web Application. More specifically, the
implementation of the BaseDomainService class.

Figure 5.7 BaseDomainService abstract class simple implementation

BaseDomainService illustrated in Figure 5.7, is an abstract class which has a

UnitOfWork object from the Data Access Layer. This UnitOfWork object is a protected
member so that all the Domain Services that inherit this abstract class are able to use it.

Figure 5.8 Highlight of the UnitOfWork object inherited from the

BaseDomainService Class

All the Domain Services extend the BaseDomainServices. By implementing the

domain services in this way, initializing a UnitOfWork object is avoided, each time a
different service is used. In Figure 5.8, a concrete example within the ConversationService
is depicted.

41
 Chapter 5

5.3.3. Bot Application Architecture

Figure 5.9 Architecture of the Bot Application module

Although the Bot Application is built with the help of Microsoft Bot Framework,
which already has a certain structure, further enhancement of structuring classes was
achieved by applying the 4-tier Architecture, as illustrated in Figure 5.7.

42
 Chapter 5

Figure 5.10 Implementation detail of getting a Subscription Model in the form of

a TV Subscription Model

An important implementation detail of referencing a Subscription Model object

from Domain Models in the Domain Services is through the implementation of interfaces
that represent the TV Subscription, Internet Subscription, Phone Subscription, which are
implemented by the Subscription Model Class.
For example, as illustrated in Figure 5.8: An ITVSubscriptionDomainModel
interface has the needed fields for the TV subscription only (id, type and channels). When
a TV Subscription Model object is needed in a Domain Service, an object of type
ITVSubscriptionDomainModel is declared but it references a SubscriptionDomainModel
object, which gets its reference from the Subscription Repository through the Unit Of
Work.
In this way, in the Domain Service, the referenced TV Subscription Model has only
the specific fields of the TV Subscription, even though all of the fields exist in the
Subscription Model and do not have values from the database, they are protected from
modification.

5.3.4. Reports Web Application Architecture

Figure 5.11 Conceptual architecture of the Reports Web Application

43
 Chapter 5

As illustrated in Figure 5.8, the Reports Web application architecture is very similar
to the Bot Application architecture, mostly following the same structure within the
Business Layer. In the Presentation Layer though, the Reports Web Application has
obviously a very different approach from the Chatbot, exposing the charts in a web rich
user interface. This is achieved with the use of the ASP.NET MVC 5.0 Framework.

5.4. Main technologies: an implementation perspective

5.4.1. Bot framework dialog management

Dialog management within the Bot Application is done with the concept of
Scorable Classes, mentioned in Chapter 4.

Figure 5.12 Class Diagram of the Dialogs folder within the Presentation Layer

As depicted in Figure 5.12, with the help of ScorableBase abstract class, it is

possible for the users within the process of a certain Dialog, to interrupt and dynamically
switch to another Dialog without having to iterate through the whole process.
Besides the RootDialog Class which only has the function to initiate the
conversation, 4 separate dialog classes are implemented: ScorableGetPaymentDialog,
ScorableMakePaymentDialog, ScorableTVIssueDialog, and ScorableInternetIssueDialog.
Each Scorable Dialog Class has a well-defined process that is implemented through
Task methods.

44
 Chapter 5

Figure 5.13 An implementation of the Task GotChannel() method that represents

the process of the ScorableTVIssueDialog

In Figure 5.13, the process of a TV Issue problem can be identified. The

StartAsync() method within this Dialog is called when the TVIssue intent is identified by
the LUIS service. After another exchange of replies between the user and the Technical
Support Chatbot in which the TV Channel name is provided, the method GotChannel() is
called.
In this method, a call is made to the TV Subscription Service in order to check if
the user specified TV Channel that is not working is included in his actual TV Subscription.
The case when the TV Channel is not included in the users subscription is, for
example, the following: the contract may have been expired and reverted to the standard
package, resulting in such a case that the user does not have the mentioned TV in his
subscription anymore).
If this case is met, the user is acknowledged through a message that his subscription
does not include the mentioned TV Channel and also the list of included TV Channels.
Also, the user is presented the possibility to make a call, in order to re-new his subscription.

45
 Chapter 5

5.4.2. LUIS application implementation

Figure 5.14 LUIS application list intents with the number of utterances for each
intent
In Figure 5.14 all the intents used in the Technical Support Chatbot application are
illustrated, within the Web interface of the LUIS application. The intents were selected
mostly with respect to the single use case possible for the Normal User, with the most
important of them being: InternetIssue, TVIssue, MakePayment and GetPayment.

46
 Chapter 5

Figure 5.15 Example of Luis intent identification in the context of

ScorableMakePayment Class

Figure 5.15 depicts an example of how the integration of the LUIS intent
identification and Bot framework dialog management is implemented. All the other
important intents mentioned above are integrated in the same manner.

5.4.3. Sentiment analysis use of scores

In order to be able to contain the functions of the sentiment score of all
conversations into an average function, the conversations need to be scaled to the same
number of replies. Methods for scaling a single conversation and multiple conversations
are implemented in the Math Service of the Reports Web Applications Business Logic
Layer.
For the graphical representation of the function, a Line Chart was implemented with
the help of Google Charts JavaScript library. Unfortunately, this library has a great
disadvantage in this particular context.
For example, given a conversation with 4 replies, when trying to scale the
conversation to 20 replies, the new position of each reply has to be calculated. This is
done by calculating a step with the following formula: step = (scale limit) / number of
replies 1. In this particular case, the step is 6.67. Given that, the position of the first reply
is the integer part of the newly calculated index, that is: index * step = 6.67 * 0 = 0. The
next positions of the replies are: second reply [6.67 * 1] = 7, third reply [6.67 * 2] = 13
and the last one [6.67 * 3] = 20. By scaling the conversations, a much more accurate
analysis can be done over them.
Another problem regarding scaling is filling up the empty spaces left between the
replies that have been scaled upwards. This first step of implementing this is by calculating
the number of empty spaces between a reply 1 and a reply 2. The next step is to calculate
the average values starting from reply 1 to reply 2 with the help of the mean function. For

47
 Chapter 5

example, given 3 empty spaces between 2 replies with sentiment scores of 20 and 60, the
5 empty spaces will be filled with: 30, 40 and 50.

Figure 5.16 Scaling conversations to 100 replies (left) and to 20 replies (right)
In Figure 5.16, a comparison of scaling to different number of replies is illustrated.
The readability of the right chart is clearly higher and that is due to 2 reasons. The first one
is regarding the limitations of the Google Charts library, which has obvious problems when
smoothing a curved function between 2 very close points. The second reason is that the
majority of the conversations within the Technical Support Chatbot context do not have
more than 20 replies. Therefore, scaling to 20 replies has been chosen. All the above-
mentioned steps are considered when scaling a single conversation.
The Sentiment Score function of all conversations uses the scaling of a single
conversation, for each conversation from the database, with the help of the weighted
average (weight = total number of conversations). The result is very similar to a function
representing a single conversation.
The first step of having the possibility to calculate the Sentiment Score function
of specific intent conversations, as presented in the Chapter 4. Analysis and Theoretical
Foundation, is to set a main intent from the most important ones (InternetIssue, TVIssue,
MakePayment and GetPayment) for each conversation and store it in the database. This
classification is done whenever a Scorable Class is reached. If the user opts, for example
to input a new requirement in the context of the same conversation and a new main intent
is identified, a new conversation is created in the back-end of the application, without the
user knowledge, in order to provide a better accuracy for the problem-solving ratio (1
problem solved per conversation).

48
 Chapter 6

Chapter 6. Testing and Validation

6.1. V-Model
The current chapter focuses on the process of evaluating the finite software product
of the Technical Support Chatbot in order to determine if the specified requirements are
satisfied.

Figure 6.1 V-Model Validation Testing Model

As depicted in Figure 6.1, the V-Model is a form of development process, more
specifically a particular extension of the Waterfall model and it includes a structured
process for the testing phase.
Based on the V-Model test process, a similar approach to test and validate the
Technical Support Application was made, starting with low-level tests and finalizing with
high-level tests.

6.2. Low-level testing

At a low level, different integrated technologies were tested, as it can be seen in the
following paragraphs.
For testing the SQL database interaction, an SQL script was used to test if a reply that
was given by the user would be correctly inserted into the database. Verification was done
by checking the id given by the Microsoft Bot Framework context to the id of the reply
inserted in the database, resulting in a success. The SQL script consisted of a simple
SELECT based on a known id of a reply.
Testing other technologies was mostly done manually during implementation. No
errors worth mentioning were encountered during testing.

49
 Chapter 6

Figure 6.2 Real-time request information given by the Bot Framework Channel
Emulator
As illustrated in Figure 6.2, a great help for testing purposes was the Bot Framework
Channel Emulator. The information provided by this emulator is given in real-time when
the bot application is running locally and it includes the status codes of the HTTP requests
to the external services from the application. Within this context, the connection to the
LUIS application and Text Analytics Service were tested successfully.

50
 Chapter 6

Figure 6.3 Chart representing the number of requests per day made to the LUIS
application, shown in the LUIS Web interface
LUIS request handling can be depicted in Figure 6.3. A test was made by sending
a high number of requests to the LUIS application, which resulted in a success rate of
100%, although an increase of the response time was observed.

6.3. High-level testing

Figure 6.4 Sentiment Score function of all conversations

A very high-level way of testing can be done by analyzing the chart illustrated in
Figure 6.4. The function can be split in 3 main areas, with the help of the two red lines.
In the first area, a transition from a high sentiment score to a low sentiment score
can be identified. The starting point of this area represents the greeting of the user to the
Technical Support Chatbot.
In the second area, depicted between the two red lines, it can be clearly concluded
that the user had a problem and may not be happy, thus the low consecutive sentiment
scores of his replies.

51
 Chapter 6

The third area is the most crucial for testing represents the possible problem
solution. As it can be clearly identified an ascendant slope, it results that the user overall
satisfaction is increased after the interaction with the Technical Support Chatbot, meaning
it is a successful test.

Figure 6.5 Sentiment Score function of specific intent conversations

The same principle can be used to analyze the function of specific intents. In Figure
6.5 the function for TVIssue intent is illustrated. As the above analysis proceeded, the
result is a successful test, meaning the user satisfaction is increased (through the ascendant
slope), but it has a lower slope than the overall function.

52
 Chapter 7

Chapter 7. Users manual

7.1. Installation Manual
The administrator that installs this solution has to satisfy the following
requirements:
Software requirements:
- Microsoft Visual Studio 2013 or newer for opening the application
locally, being able to publish it and to manage the SQL database
- Access to a LUIS app directory website [20] where the LUIS external
application will be imported
- Access to a Bot app directory website [21] where the final bot will be made
publicly available
Hardware requirements:
- Server Azure Portal (recommended) here will be deployed the whole
Technical Support Chatbot solution, excepting the external services

7.1.1. Deployment Diagram

Figure 7.1 Deployment Diagram

53
 Chapter 7

In Figure 7,1 the deployment diagram is illustrated. Mainly, the Technical Support
Chatbot Solution is installed on an Azure Portal Server, but further details will be given in
the following subchapter which will provide all the necessary steps to publish both the
Reports Web Application and the Bot Application (note: the publication of the Data Access
Layer is not necessary because it is referenced in both previously mentioned applications).

7.1.2. Publishing the application

Firstly, the administrator has to make sure that he has all the necessary files in order
to install the complete solution. The files are the following:
- The LUIS Application, exported as Call Center.json file
- The entire Technical Support Chatbot Solution (folder), together with
NuGet packages
- The Entities Database, exported as Entities.sql file
- The Admin Database, exported as Admin.sql file

The second step is to configure the Azure portal Server or any other server which
will provide a public endpoint for the Reports Web Application and the Bot Application.

Figure 7.2 All resources of the Azure Portal Server [22]

In Figure 7.2 all the necessary resources to be added to the Azure Server in order
to continue installation can be seen.

The next step is to open the Technical Chatbot Support solution with Microsoft
Visual Studio and change the data connection strings.
Afterwards, the following operations should be done:
- Deploy the Entities Database to the Database Server (in this case, the local
Entities Database was deployed to the tsc-db-server under the name of tsc-
db).
- Deploy the Admin Database to the Database Server in the same manner (in
this case, id-db)
- Publish the Reports Web Application to the Reports Web App Service (in
this case, to the admin-reports2)
- Publish the Bot Application to the TechnicalSupportChatbot Web App
Service
54
 Chapter 7

At this step, only the Reports Web Application should be working and should be
tested. For the Bot Application to work properly, publishing of the LUIS Application is
still needed and adding a new bot in the bot directory.
The following step is to import the LUIS Application from the LUIS Web Interface.

Figure 7.3 Homepage of the Bot Application endpoint URL [21]

As illustrated in Figure 7.3, at this point in the publishing process, the Bot
Application should work well if run locally (partially, because the LUIS application and
Text Analytics are called from the cloud and the database is accessed from the server).
The next step is to add a new bot in your bot directory, set the endpoint URL to one
similar to the one illustrated in Figure 7.3.
Finally, the Technical Support Chatbot should be added to the Skype channel or
any other channel the administrator values. For the endpoint URL, the value
customURL/API/messages should be entered, with customURL being similar to the
URL mentioned in Figure 7.3

Figure 7.4 Overview of the present channels on which the Technical Support
Chatbot is published [21]
If added to Skype, the result should look exactly as depicted in Figure 7.4.

55
 Chapter 7

7.2. User Manual

If the Technical Support Chatbot Application is in production use of a certain
company, then a special requirement that a Normal User has to satisfy in order to be able
to use the application is to be a customer of that company. Otherwise, no special
requirements need to be satisfied.
Software requirements:
- An operating system that supports Skype: Windows or Mac OS, IOS,
Android
- The Skype application
Hardware requirements:
- Desktop or mobile device that satisfies the software requirements
mentioned above

After having installed all the software from the mentioned requirements, the user
can just search for the Technical Support Chatbot in the Skype Directory and add it as a
contact.
As a general rule, the user has to the application in an educated manner, without
spam or other intrusive ways. This means that the user should contact the Technical
Support Chatbot only when he has a problem or requirement that is relevant to the
Technical Support Chatbot.

56
 Chapter 8

Chapter 8. Conclusions
This chapter contains a brief description of contributions, all the results obtained
during the development process of the Technical Support Chatbot relative to the objectives
that were presented initially in the second chapter of this document and finally, further
development points.

8.1. Contributions
This subchapter highlights the contributions added to an already existing plethora
of frameworks, algorithms and best practices within the domain of Technical Chatbot
Support or even to a higher domain of Chatbots.

8.1.1. Contextualization
The approach of the Technical Support Chatbot differs from the majority of chatbot
implementations with the advantage of storing much user information that can provide a
base for more contextual decision-making algorithms.
Whenever the chatbot is added to the channel of a user, it gets all the information
of the user from the database. In order for a user to be able to add the Technical Support
Chatbot to the Skype directory, he has to be a customer of the company that the chatbot
provides support for. In this way, the user information is added in the database when a new
user is registered and it is available right-away for the Technical Support Chatbot.
A possible reason for other chatbots not being able to implement such a
contextualized experience for users is the privacy policies from the majority of the instant
messaging channels. For chatbots that do not have such rich user information, a solution
may be to make use of a sign-in card and take profile information from a popular account
that most users have, like a Microsoft account.
Coming back to the Technical Support Chatbot easy contextualization, the user
information from the database includes, beside identification data, also other context
information such as the subscription for which the user is paying for.
In this way, the Technical Support Chatbot can provide uniquely modeled
responses to a user, not from a broad natural language perspective, but from the specificity
of the provided answers, meaning that the chatbot can satisfy the exact needs of the user.

8.1.2. Real-time decision making

The analysis of the function that identifies angry user replies, which was mentioned
in Chapter 4, provides a basic model that basically finds out when the user is angry, from
his last 3 replies.
The advantage of having such an accurate knowledge of the users feeling with
respect to the Technical Support Chatbot interaction in real time is enormous.
For example, in the case of the first release of the Technical Support Chatbot, the
above-mentioned function provides a back-up solution to all of the functionalities available
to the Normal User that may provide dissatisfaction. More explicitly, whenever the chatbot
is not capable of understanding the users requirement or is providing wrong solutions that
do not solve the users problem, the function that identifies if the user has got angry acts

57
 Chapter 8

as an interrupt, drops all the other possible response modelling, formulates a default
response and forwards the problem to the human technical support.

8.1.3. Natural language understanding improvement

The Reports Web Application has the main objective to improve the overall
performance, mostly through natural language understanding improvement.
This is achieved through reading and analyzing the charts of the sentiment score
function of a conversation with a specified main intent. More explicitly, if the TVIssue
Intent is chosen, the specific chart would provide the function of all conversations
regarding this intent.
The sentiment score function can be analyzed, with the main focus of identifying if
the function ends in an ascendant slope and overall range. If the function ends in an
ascendant slope, user problems that may appear correlated to the TVIssue intent are well
covered. If that is not the case, then the administrator of the Technical Support Chatbot
should access the LUIS application and add a higher number or better utterances in order
to improve the natural language understanding capability of the Technical Support Chatbot
regarding this intent.

8.2. Achieved results

Based on the detailed list of objectives presented in the Chapter 2, a thorough
analysis of the achieved results is listed below:
- The Reports Web Application is published on the Azure Portal
- The Technical Support Chatbot was published in the Skype Directory (in preview)
by the name Technical Support Chatbot
- The security of the Reports Web Application is ensured through ASP.NET MVC
Identity Authentication
- The Database security is enabled by the firewall from the Azure Portal (only a
provided list of IPs can access it; any new IP that tries to access it is documented)
- An accurate display of the overall sentiment scores chart is done with the help of
Google Charts and complex scaling of conversations to 20 replies in the Reports
Web Application
- An accurate display of the specific sentiment scores chart based on intent
classification is also provided in the Reports Web Application
- The positive impact of the Technical Support Chatbot by the objective analysis of
the overall sentiment scores chart and identifying the ascendant slope has been
successfully identified and is detailed in subchapter 6.3
- Correct user identification was achieved by accessing the Technical Support
Chatbot from a mobile Skype application
- Correct identification of the start and end of a conversation is achieved and further
detailed in subchapter 4.2.4
- Multi-channel integration is supported currently on Skype and Web Chat
- Correct ticket forwarding process to the human technical support team was
successfully implemented

58
 Chapter 8

8.3. Future work

Extending the intents and responses - Currently there are only 4 intents that may
be mapped to user input requirements. A great enhancement would be to add new intents
in the LUIS application and their corresponding responses in new Dialogs.
Provide a rich user interface through the use of cards by making use of cards,
the input text classification error drops significantly, because the user has choices to make.
The Technical Support Chatbot does not have to identify the intent in these cases.
Ability to add utterances directly from the Reports Web Application
enhancing the user interface of the Reports Web Application and implementing the
possibility to add utterances from the UI would centralize all the process of the Natural
Language Understanding Improvement. In this way, the LUIS application can be further
externalized and the Administrator would not need credentials to that application anymore.
Possibility to label unknown intents from the Reports Web Application with
the help of enhancing the user interface of the Reports Web Application, unidentified
utterances can be shown to the Administrator in a Table or List. For each utterance, a
dropdown menu should be made available on click, showing all the intents. The
Administrator has the possibility to label the unknown utterances very fast, resulting in a
faster improvement of the overall performance of the application.
Switch to Windows Authentication for the Reports Web Application
Currently the Reports Web Application is accessible from everywhere, provided
there is an internet connection. In the case of using the Technical Support Chatbot in
production by a certain company, switching to Windows Authentication would be
recommended.
With the current authentication implementation, there may be security risks and
providing the application would have access to a vast amount of important information,
those security risks should be addressed. By switching to Windows Authentication, the
application would not be publicly available anymore, but only within the intranet of the
company. It would provide a much more secured approach.

59
 Bibliography

Bibliography

[1] P. Phillip J. Windley, "Delivering High Availability Services Using a

Multi-Tiered Support Model," 2002.
[2] A. Kongthon, "Implementing an online help desk system based on
conversational agent," in Proceedings of the International Conference on
Management of Emergent Digital EcoSystems, 2009.
[3] G. G. Chowdhury, "Natural Language Processing," Annual Review of
Information Science and Technology, pp. 51-89, 2005.
[4] B. A. Shawar and E. S. Atwell, "Using corpora in machine-learning
chatbot systems," International Journal of Corpus Linguistics, vol. 10:4, p.
489516, 2005.
[5] J. Stevens, "E-commerce and Conversion Statistics 2016," 2016.
[Online]. Available: https://hostingfacts.com/internet-facts-stats-2016/.
[6] Ubisend, "2016 Mobile Messaging Report," 2016.
[7] A Harris Poll on behalf of OneReach, "The High Demand for Text
Messaging," 2014.
[8] L. A. Jung, "Writing SMART Objectives That Fit the ROUTINE,"
2007.
[9] J. Weizenbaum, " A Computer Program For the Study of Natural
Language Communication Between Man and Machine," 2005. [Online].
Available: http://www.masswerk.at/elizabot/eliza.html.
[10] N. M. a. B. M. C. Radziwill, "Evaluating Quality of Chatbots and
Intelligent Conversational Agents," 2017.
[11] D. Zientara, "The Bot Framework," Applied Information Sciences,
Inc. All Rights Reserved, 2017. [Online]. Available:
https://blog.appliedis.com/2016/08/16/the-bot-framework/.
[12] IBM, "IBM Watson Workspace," 2017. [Online]. Available:
https://idaas.iam.ibm.com.
[13] A. L. C. G. J. S. G. Z. Jason D. Williams, "Rapidly scaling dialog
systems with interactive," 2015.
[14] R. High, "The Era of Cognitive Systems: An Inside Look at IBM
Watson and How it Works," Redguides for Business Leaders, 2012.
[15] 2017 Poncho Inc., "Poncho," 2017. [Online]. Available:
https://poncho.is/.
[16] STA Consulting Kft., "Create ServiceNow incidents from SAP,"
2017. [Online]. Available: http://sta-technologies.com/create-servicenow-
incidents-from-sap/.

60
 Bibliography

[17] G. Pretty, "Using Scorables for global message handling and

interrupting dialogs in Bot Framework," 2017. [Online]. Available:
http://www.garypretty.co.uk/2017/04/13/using-scorables-for-global-
message-handling-and-interrupt-dialogs-in-bot-framework/.
[18] Microsoft, "Language Understanding Intelligent Service," 2017.
[Online]. Available: https://www.luis.ai.
[19] Microsoft, "Text Analytics," 2017. [Online]. Available:
https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics/.
[20] Microsoft, 2017. [Online]. Available:
https://www.luis.ai/applications .
[21] Microsoft, "Bot Framework," 2017. [Online]. Available:
https://dev.botframework.com.
[22] Microsoft , "Microsoft Azure," 2017. [Online]. Available:
https://portal.azure.com.

61