Interaction Analysis over Speech for Call Centre
Veerus Dmello
Computer Department
St. Francis Institute of Technology
Mumbai, India
Rhea Kolhapurkar Nicholas Patric
Computer Department Computer Department
St. Francis Institute of Technology St. Francis Institute of Technology
Mumbai, India Mumbai, India
Abstract - The aim is to apply well-known data mining
techniques to the problem of predicting the quality of
interactions like those done in call centre’s and the problem of
predicting the quality of service. The analysis of call centre
conversations will provide useful insights for enhancing Call
Centre Analytics to a level that will enable new metrics and key
performance indicators (KPIs) beyond the standard approach.
The main operations will be speaker diarization, speech to text,
agent analysis, emotion recognition and hot-topic analysis.
RAVDESS (Ryerson Audio-Visual Database of Emotional
Speech and Song) is the database used for emotion analysis,
CMU-Sphinx is used for speech to text transcription. Emotional
analysis uses Convolutional Neural Networks to recognize
emotions. The agent analysis and hot topic and root cause
analysis use Natural Language Processing (NLP) techniques. All
of this will happen in real-time as and when the call is taking
place. The data gathered will be stored in a database for easy
retrieval and for maintaining records.
Index Terms— Agent Analysis, Emotion recognition, Hot topic
analysis, Interaction mining, Speech to Text
I. INTRODUCTION
This paper presents a call monitoring system to assess calls
live and help call centres to collect data to analyse their
service and gain insights from them. In our increasingly
industrialized and globalized world, a large number of
companies include call centres in their structures and more
than $300 billion is spent annually on call centres around the
world. For a customer, addressing the call centre actually
means addressing the company itself, and any negative
experience on the part of the customer can lead to the rejection
of company products and services. Such data analysis can help
improve the quality of customer service and lower the costs.
Parth Kholkute
Computer Department
St. Francis Institute of Technology
Mumbai, India
Ms. Ankita Karia
Assistant Professor
Computer Department
St. Francis Institute of Technology
Mumbai, India
Call centre optimization is an important part of customer
relationship management which consists of people, processes,
technology and strategies. Service quality of a call centre is a
result of comparison of actual service performance and
customer expectations. Evaluating the service quality which is
offered by customer service agent to customer is more
difficult than evaluating the product quality.
Call centre’s provide services for many types of sectors
such as telecommunication, finance, transportation, health,
automotive etc. Several studies have proposed various
approaches and solutions for the problem of evaluating agent
performance. Performance evaluation in call centre’s is
generally performed through listening randomly selected calls
from recorded calls, and evaluating the words one by one in
the related conversation. obvious demand for automatic
performance evaluation systems to reduce employee costs and
to increase the time efficiency. Takeuchi has analysed the
recorded calls from a rental car reservation office with Trigger
Segment Detection to find whether a customer has the
intention of booking a car or not. This system is used to
analyse the content of call centre conversations and detect the
main issue addressed in the call. None of the previous existing
methods cover all the ranges of call centre analysis and give
worthwhile insights on this problem.
Call Centre Analytics is aimed at solving the above issue
by enabling tapping into the content of conversations. Just
text-based content analysis approaches are highly sensitive to
input quality and that conversational input is fundamentally
different than text. Therefore, conversations should be treated
differently. First, seek to diarize the speech as to separate the
components of the customer and the agent. Thereafter, use a
mixture of emotional analysis as well as speech to text
transcription while performing agent analysis through Natural
Language Processing (NLP) technology which needs be
adapted to and be robust enough to deal with the machine learning (Gilman et al. 2004; Zweig et al. 2006;
conversational domain in order to achieve acceptable Takeuchi et al. 2009). These methods failed in providing
performance. Moreover, the level of analysis of conversation satisfactory results even in very broad categories. The problem
cannot be set to semantics only. It must consider the purpose still lies on data sparseness and that huge amount of training
of language in its context, i.e., pragmatics. Our approach to data is necessary to achieve reasonable discriminatory power.
Call Centre Analytics is based on Interaction Mining. Getting huge training data is not an option also because
Interaction Mining is a new research field aimed at extracting training is highly influenced by domain specificity.
useful information from conversations. In contrast to Text Transferring trained models from a domain to another would
Mining, Interaction Mining is more robust, tailored for the be problematic [4].
conversational domain, and slanted towards pragmatic and
discourse analysis. All these details are stored in that will III. CHALLENGES IDENTIFIED
contain data about the current and previous calls to further aid The challenges identified are as follows:
the agents.
A. Lack of a concrete dataset of calls
Since the call centre data collected by various
organizations are confidential and contain privileged
II. LITERATURE SURVEY information, there is no dataset available that contain calls and
Call centres provide services for many types of sectors the interaction between the agent and the customer. This
such as telecommunication, finance, transportation, health, provides a hinderance in measurement of the system
automotive etc. Several studies have proposed various performance during evaluation.
approaches and solutions for the problem of evaluating agent
performance. Performance evaluation in call centres is B. Real time call analysis
generally performed through listening randomly selected calls The system proposed will perform real time analysis. Call
from recorded calls, and evaluating the words one by one in centre’s have a call module that is the platform used to take
the related conversation. Obvious demand for automatic and make calls. Since access to this call module is not
performance evaluation systems to reduce the employee costs available, the analysis will require a temporary software that
and to increase the time efficiency. Takeuchi has analysed the will act as the call module or replace it for the evaluation of
recorded calls from a rental car reservation office with Trigger the system proposed.
Segment Detection to find whether a customer has the
intention of booking a car or not. Mishne has proposed a call IV. PROBLEM DEFINITION
centre monitoring system that uses text analytics and
Call centre optimization is an important part of customer
information retrieval methods. The system is used to analyse
relationship management which consists of people, processes,
the content of call centre conversations and detect the main
technology and strategies. Service quality of a call centre is a
issue addressed in the call. The project has presented speech
result of comparison of actual service performance of the
analytics system adapted automatic speech recognition and
agent and customer satisfaction. Extracting data that will help
text mining technologies [1]. Minnucci (2004) reports that the
to reduce total call time and better the service provided by the
most required metrics by call centre managers are indeed the
call centre, i.e. get real time emotion of the speaker and the
qualitative ones topped by Call Quality (100%) and Customer
agent, mine data from the speech to text transcription and use
Satisfaction (78%) [2].
them to rate the agent’s performance.
However, these performance metrics are difficult to
implement with the adequate level of accuracy. For instance, V. PROPOSED SYSTEM METHODODLOGY
the Baird study (2004) points out that for Customer
Satisfaction, accuracy can be “negatively affected by A. System Architecture
insufficient number of administered surveys per agent
resulting in not enough samples of individual agent’s work to
constitute a representative sample. The result could be an
unfair judgment of the agent’s performance and allocations of
bonuses based more upon chance, good fortune than merit.
“Accuracy is defined in as true indication and it depends on
the actual level of performance attainment, especially with
regard to statistical validity [3].
Current approaches to Call Centre Analytics are mostly
based on Speech Analytics and Text Mining, which is
essentially Search and Sentiment Analysis. Recorded speech is
first indexed and searched against a set of negative terms and
relevant topics. There are currently two main approaches for
speech index: i) Phonetic Transcription and ii) Large
Vocabulary Conversational Speech Recognition (LVCSR).
Another common approach to the analysis of call centre data
is that of automatic call categorisation through supervised Fig. 1: Architecture block diagram for the proposed system
 As shown in Fig. 1, the system will process the live call.
Speaker diarization will be performed on the call to break
down the call into two speaker components. Customer Speech
audio and Agent Speech audio will be given for Emotional
Analysis, where the detected emotions are shown with the
help of a graph. Audios will then further be given for Speech
to Text Transcription, where the output is stored in a text file.
The output from the text file will be given as input for Agent
Analysis and Hot Topic Analysis, where performance results
of the agent will be generated and stored in the database.
B. Proposed technical solution
The solution is divided into modules, the output from all
the modules will aid in giving the final output which will then
be stored in a database.
Speaker Diarization – It is the process of partitioning an
input audio stream into homogeneous segments according to
the speaker identity. The partitions will be agent and customer
and will enable us to analyze each of them individually.
Emotional Recognition - Gauge the emotional state of
your customers by analyzing their voices for tell-tale
variations in pitch or tone. In essence, the software determines
the emotional tone behind a series of words, used to gain an
understanding of the attitudes, opinions and emotions
expressed. Emotion analysis is done by converting the audio
into spectrograms and predict the emotion from it.
Speech to Text Transcription - Speech to text then allows
us to process the speech to text which is stored as a .txt file.
This file is then used to analyze the agents.
Agent Analysis - The agent is analyzed using the speech to
text transcription on the following criterion (KPI factors) -
o Greeting score
o Number of slangs
o Number of repeated sentences
o Closing score
o Banned words
o Call length
o Whether the problem was solved or not
The above details will be stored in a .xml file for storage
purposes and easy retrieval. Each .xml file will be then used to
aid in further optimizing the agent performances.
Hot topics with root cause analysis - Uncover and track the
most frequent topics mentioned by customers, which aids in
identifying trending customer satisfaction issues.
VI. PERFORMANCE EVALUATION PARAMAETER
The individual modules will be unit tested according to
criterions set by us. The unit tests include and are not
restricted to test sets for all the modules.
The integrated system will evaluate scripted calls to
provide results. These results will be evaluated against
customer feedback. The customer feedback will help validate
against the performance of the system. Another evaluation
parameter will include an independent team to individually go
through each call recording and evaluate it by themselves.
This will be measured against the real time evaluation done by
the system to validate its performance.
VII. EXPERIMENTAL SET UP
Hardware requirements –
The hardware will require a high-performance CPU and
GPU to perform the complex operations in the software
model.
Minimum requirements -
CPU: Intel Core2 Quad Q6600 @ 2.4 GHz (or AMD
Phenom 7950 Quad-Core, AMD Athlon II X4 620 equivalent)
RAM: 2 GB
Software requirements –
Microsoft Windows 10/8.1/8/7
Python 3.6 or above
The set ups for individual modules are -
Emotional Analysis –
Database in use for testing and training - RAVDESS
(Ryerson Audio-Visual Database of Emotional Speech and
Song).
Method - Conversion to spectrogram, extraction of
features and MFCC’s, and use of convolution neural network.
Agent Analysis –
Database in use for testing and training - Raw database
received from STT.
Method - Using unique algorithms such as cosine
similarity, sequence matching, etc., to provide insights on the
KPI factors mentioned in the proposed solution.
Speech to text transcription –
Database in use for testing and training - CMU Sphinx,
which is a continuous-speech, speaker-independent
recognition system making use of Hidden Markov acoustic
models (HMMs).
Method - Glob for selection of files and use of CMU
Sphinx for speech recognition.
Speaker Diarization –
Method – Individually recognize the two different audio
frequencies and use RNN to automatically separate them into
two different audio streams.
Hot topic analysis –
Method – Using NLP algorithms like cosine similarity,
sequence matching, etc.
REFERENCES
[1] Betül Karakus,Galip Aydin, “Call Center Performance Evaluation
Using Big Data Analytic.”, International Symposium on Networks,
Computers and Communications (ISNCC), 2016, pp. 1-2.
[2] Vincenzo Pallotta, Rodolfo Delmonte, Lammert Vrieling, David
Walker, “Interaction Mining: the new frontier of Call Center
Analytics.”, In: Lai C., Semeraro G., Vargiu E. (eds) New Challenges in
 Distributed Information Filtering and Retrieval. Studies in
Computational Intelligence, vol 439. Springer, Berlin, Heidelberg,
2013, pp. 91-111.
[3] Baird H, “Ensuring Data Validity Maintaining Service Quality in the
Contact Center.”, Telecom Directions LLC, 2004, pp 3.
[4] Sathit Prasomphan, “Detecting Human Emotion via Speech Recognition
by using Speech Spectrogram.”, IEEE International Conference on
Data Science and Advanced Analytics (DSAA) Paris, France, IEEE
2015, pp 66-73.