Recommending resolutions of ITIL services tickets using
Deep Neural Network
Durga Prasad Muni1 , Suman Roy1 , Yeung Tack Yan John John Lew Chiang1 , Antoine Jean-Marie Viallet∗2 and Navin
Budhiraja1
1
Infosys Ltd., #44 Electronic City, Hosur Road, Bangalore 560100, India
2
SUPELEC engineering school, Gif-sur-Yvette, France
{Durgaprasad Muni,Suman Roy,Yeung Chiang,Navin.Budhiraja}@infosys.com,viallet.antoine@gmail.com
ABSTRACT
Application development and maintenance is a good exam-
ple of Information Technology Infrastructure Library (ITIL)
services in which a sizable volume of tickets are raised ev-
eryday for different issues to be resolved in order to deliver
uninterrupted service. An issue is captured as summary on
the ticket and once a ticket is resolved, the solution is also
noted down on the ticket as resolution. It will be beneficial
to automatically extract information from the description
of tickets to improve operations like identifying critical and
frequent issues, grouping of tickets based on textual con-
tent, suggesting remedial measures for them etc. In partic-
ular, the maintenance people can save a lot of effort and
time if they have access to past remedial actions for simi-
lar kind of tickets raised earlier based on history data. In
this work we propose an automated method based on deep
neural networks for recommending resolutions for incoming
tickets. We use ideas from deep structured semantic mod-
els (DSSM) for web search for such resolution recovery. We
project a small subset of existing tickets in pairs and an in-
coming ticket to a low dimensional feature space, following
which we compute the similarity of an existing ticket with
the new ticket. We select the pair of tickets which has the
maximum similarity with the incoming ticket and publish
both of its resolutions as the suggested resolutions for the
latter ticket. The experiment of our data sets shows that we
are able to achieve a promising similarity match of about
70% - 90% between the suggestions and the actual resolu-
tion.
Keywords
Deep Learning; Neural Network; Deep Neural Network; Ticket;
∗
This work was done when Antoine was an intern at Infosys
Ltd during July-Sept, 2016
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from Permissions@acm.org.
IKDD CODS 2017, March 9-11 2017, Chennai, India
Copyright 2017 ACM 978-1-4503-2776-3/14/02 . . . $15.00.
DOI: 10.1145/3041823.3041831
Resolution; Resolution Recovery
1. INTRODUCTION
Ticketing system is used as one of the inputs for Infor-
mation Technology Infrastructure Library (ITIL) services
such as problem management and configuration manage-
ment. Specifically, in production support related application
development and maintenance projects, the incident data in
the form of tickets are used for different purposes such as
SLA calculation, forecasting, optimum resource level check-
ing, quick metrics computation etc. A huge number of tick-
ets are raised by the user on the ticketing system for the pur-
pose of resolving their problems while using different support
systems. A ticketing system tries to minimize the business
impact of incidents by addressing the concerns of the raised
tickets. Any prior knowledge which can be obtained by min-
ing ticket data can help in quick redressal of the problem.
The incident tickets record symptom description of issues,
as well as details on the incident resolution using a range of
structured fields such as date, resolver, affected servers and
services and a couple of free-form entries outlining the de-
scription/summary of issues, note by users/administrators
etc.
The issues are captured as summaries on the tickets and
once a ticket is resolved, the solution is also noted down on
the ticket as resolution. The maintenance people can save a
lot of effort and time if they have access to previous remedial
actions for similar kind of tickets based on past history. In
this work we propose a method based on deep learning for
suggesting resolutions for new tickets.
Deep learning allows computational models that are com-
posed of multiple processing layers to learn representations
of data with multiple levels of abstraction [11], - these are
called deep neural networks. Deep neural networks (DNN)
are becoming popular these days for providing efficient so-
lutions for many problems related to language and informa-
tion retrieval [1, 3, 6, 22, 24, 21]. In this work we use a
framework of deep learning network for resolution recovery
by lifting ideas from similar deep structured semantic mod-
els (DSSM) for web search [9, 8, 7]. In those papers the
authors use DNN to rank a set of documents against a given
query in the following steps. First, the DNN carries out a
non-linear projection to map the query and the documents
to a common semantic space in which the relevance of each
 Figure 1: A schematic diagram of our Approach
document for the query is computed in that semantic space.
Then the neural networks are learned using the clickthrough
data by maximizing the conditional likelihood of the clicked
document given the query.
Motivated by this we propose a method for recommending
resolution for incoming tickets in ITIL services using similar
approach. We treat existing tickets (or historical tickets)
as a set of documents while an incoming ticket resembles a
query. The tickets are represented using features consisting
of words from their summaries after lemmatization and re-
moval of stop words. We map an incoming ticket and a small
subset of existing tickets to a low dimension feature space.
This is done by feeding feature vectors for existing tickets
and the incoming ticket to a deep neural network (DNN) and
thus obtaining the semantic representation of the feature
vectors in a very low dimensional space. Then we compute
the similarity between the low-dimensional feature vector of
the new ticket and the existing tickets to find the most sim-
ilar ticket with respect to the new ticket. Subsequently, we
publish the resolutions of the most similar (existing) ticket
pair as the recommended solutions for the fresh ticket. A
schematic diagram of our method is shown in Figure 1.
The paper is organized as follows. In the section 2 we
describe the schema of our data set and the steps for fea-
ture extraction from a ticket. We introduce our deep neural
network and its training and validation steps in Section 3.
Experiments of our company data set are discussed in Sec-
tion 4. We state related work in Section 5 before concluding
in Section 6.
2. TICKET DATA SET
We consider incident tickets with similar schema which
are frequent in ITIL. These tickets usually consist of two
fields [10, 14], fixed and free form. Fixed-fields are cus-
tomized and inserted in a menu-driven fashion. Example
of such items are the ticket’s identifier, the time the ticket
is raised or closed on the system or, if a ticket is of inci-
dent or request in nature. Various other information are
captured through these fixed fields such as category of a
ticket, employee number of the user raising the ticket etc,
and also maintenance team performance parameters like re-
sponse time, resolution time of the ticket etc. But, fixed
fields do not convey much information about the incident
itself. There is no standard value for free-form fields. The
concern/issue for raising a ticket is captured as “call descrip-
tion” or “summary” as free-formed texts, - it can be a just a
sentence that summarizes the problem reported in it, or it
may contain a detailed description of the incident. By using
freely generated part of tickets, administrators can get to
know about unforeseen network incidents and can also ob-
Figure 2: Snapshot of relevant parts of incident
ticket data
tain a much richer classification. A small note is recorded as
resolution taken for each ticket. A small part of ticket data
is shown in Figure 2.
2.1 Feature vector creation from Ticket Data
We assume a free field of ticket to contain a succinct prob-
lem description associated with a ticket in the form of a
summary (or call description). We consider the collection
of summary of tickets for extracting the feature vector cor-
responding to a ticket. We use light natural language pro-
cessing for feature vector generation. As a pre-processing
we remove the tickets which do not contain either a sum-
mary or a resolution or both. In the beginning we perform
lemmatization of the words in the the summary of tickets.
Then we use Stanford NLP tool [4] to parse the useful con-
tents in the summary of the tickets and tag them as tokens.
Next we set up some rules for removing tokens which are stop
words. We compute document frequency (DF)1 of each lem-
matized word. We discard the words whose DF is smaller
than 3. Ticket summary may contain some very rare words
like name of a person (user who raised the ticket) and some
noise words. By removing words with DF < 3, we can re-
move these very rare words which do not contribute to the
content of ticket summary. In this way, the feature vector
size could be reduced significantly.
We can model a ticket as a vector T (., . . . , .), where each
element T (xi ) represents the importance or the weight of a
word w with respect to the ticket. One needs to choose a
suitable weighing scheme to best describe these tickets. We
shall use TF*IDF [20] of a word as its weight2 . Finally, a
profile of a ticket is given as T = (x1 , . . . , xn ) = ~
x, where
x1 , . . . , xn are the appropriate weights for the chosen words
w1 , . . . , wn respectively from the summary of T . As ticket
summaries are short, most of the words may appear only
once. If we take TF (term frequency) to represent the weight
of an element in the feature vector then most of the entries
in the feature vector will be 1 which do not convey much
meaningful information. So, we have considered TF*IDF
values as weights for feature vector representation.
2.2 Relational schema on fixed elements
The fixed field entries of a ticket can be represented using
1
Document frequency of a word is the number of tickets
(ticket summaries) containing the word in the data set (cor-
pus) [12]
2
TF*IDF is a popular metric in the data mining litera-
ture [12]
a relational schema. For that we shall consider only a limited
number of fixed fields of a ticket for choosing attributes that
reflect its main characteristics (the domain experts’ com-
ments play an important role in choosing the fixed fields),
for example the attributes can be, - application name, cate-
gory and sub-category. They can be represented as a tuple:
Ticket(application name, category and sub-category). Each
of the tuples corresponding to entries in the fixed fields in the
ticket can be thought of an instantiation of the schema. Ex-
amples of rows of such schema can be, (AS400 - Legacy Man-
ufacturing, Software, Application Errors), (AS400 Legacy -
Retail, Software, Application Functionality Issue) etc. The
relation key can vary from 1 to number of distinct tuples
in the schema. One such key can hold several Incident IDs,
that is, it can contain several tickets with different IDs.
3. DEEP NEURAL NETWORK (DNN) FOR
RECOMMENDING RESOLUTION
Artificial Neural Networks (ANNs) or Neural Networks
(NNs) are networks inspired by biological neural networks.
These are made up of interconnected processing units. These
are used for tasks such as estimation, function approxima-
tion and pattern recognition.
We use feed forward Deep Neural Networks(DNNs) that
have multiple hidden layers. Each layer in a feed forward
Neural Network adds its own level of non-linearity that can
solve more complex problems. Our DNN model processes
input vectors in two stages. First, the DNN maps high-
dimensional sparse text features of a document layer by layer
into low-dimensional feature vector. In the next stage, the
low dimensional feature vectors are passed through cosine
similarity function computation gate.
The architecture of deep neural network DNN [9] that we
use to recommend resolution for a new ticket is given in
Fig 3. The objective of the DNN model is to find the ticket
from the existing tickets with the summary that is the most
similar wrt a new ticket. To achieve this goal, we can train
the DNN with a set of ticket summaries coupled with their
respective similar and dissimilar ticket summaries. These
similar and dissimilar tickets are obtained through multiple
stages of processing. At first, we assume that tickets with
the chosen combination of fixed fields or tuple (application
name, category and sub-category combination) are similar
to some extent. Alternatively, we assume that tickets from
different tuples are dissimilar to each other. To obtain sim-
ilar tickets for a given ticket, we randomly pick a subset of
tickets from same tuple as that of the given ticket.
However, we cannot definitely say that tickets from same
tuple are always similar. The combination of fixed fields,
application name, category and sub-category (or a sub-
combination) broadly bucket the tickets into same group,
but there may be difference among tickets within the same
tuple. Moreover, since a user chooses attributes of these
fixed fields (application name, category and sub-category)
for a ticket in a menu-driven fashion, there is always a chance
that the user may have chosen wrong entries for such menu-
driven options.
To avoid this issue, we follow two steps to obtain better
similar and dissimilar tickets which lead to better training of
the model. Firstly, for choosing similar (dissimilar) tickets,
we pick the tickets from same (different) tuple for a given
ticket, and then we use the approach based on semantic sim-
Figure 3: Structure of Deep Neural Network
ilarity to choose more similar (dissimilar) tickets out of those
and filter out those tickets which are not that much similar
(dissimilar) wrt the given ticket. The semantic similarity
approach is discussed in the Subsection 3.1. Secondly, we
have designed the DNN in such a way that it will consider
the tickets in pairs. By considering a pair of chosen similar
(dissimilar) tickets, we ensure there is higher chance that
they are more similar (dissimilar) as a combination. It is
like a committee of classifiers that improve the performance
over a single classifier model [15]. We may consider multiple
(≥ 2) chosen tickets for better similarity (dissimilarity) by
which the mismatch (match) will be balanced out. In this
paper, however we consider only pairs of similar (dissimilar)
tickets for training for ease of computation.
3.1 Finding semantic similar tickets for train-
ing
For training DNNs we need to find out similar tickets cor-
responding to a given ticket. Towards that we use a tech-
nique based on descriptive features for computing the sim-
ilarity of summaries of two tickets [13]. In this approach
(SS), a sentence is a sequence of words each of which carries
useful information. A joint word set is formed dynamically
using all the distinct words in a pair of sentences. A raw se-
mantic vector is carved out of each sentence with the help of
a lexical database which is augmented with information con-
tent for words. Finally we calculate the semantic similarity
for two tickets Ti and Tj using the cosine distance between
these two semantic vectors, denoted as sem-sim(Ti , Tj ). This
semantic similarity of two tickets is different from the simi-
larity of two tickets in probabilistic IR models in Section A
in Appendix.
Moreover, we say a ticket Ti is semantically similar (or
simply similar) to a ticket Tj if sem-sim(Ti , Tj ) > 0.5. Oth-
erwise, Ti is dissimilar with Tj . For training the DNN we
need to choose tickets which are semantically similar to the
ticket Ti belonging to the same tuple τ . Given a ticket Ti
having tuple τ one needs to pick tickets at random and check
if they are similar to Ti . Similarly we need to randomly pick
up tickets having tuples other than τ which are dissimilar
with Ti . However, we would like to restrict the number of
draws in such a way that the probability of choosing simi-
lar/dissimilar tickets is maximized.
It is possible to find out the probability of choosing tickets
which are similar to a ticket belonging to the same tuple
and also those which are dissimilar. Suppose there are Nτ
tickets which have the same tuple τ . Further assume that
there are Nτi number of tickets having tuple τ which are
similar to ticket Ti . Then the probability of choosing a ticket
 Ni
similar to ticket Ti is p = Nττ . Also the probability of failure,
that is, choosing a ticket which is dissimilar to ticket Ti is,
q = 1 − p. As we are required to pick a pair of tickets we
need to decide on the number of draws of tickets from the
collection in which there is a maximum chance of choosing
at least 2 tickets which are similar to the ticket Ti . Based
on the computation in Section B in Appendix we decide to
draw 5 tickets and choose two tickets which has got higher
semantic similarity values with the given ticket Ti .
Next we find an optimal number of draws in which exactly
2 number of dissimilar tickets are chosen. Recall that we pick
up dissimilar tickets from the set of tickets having a tuple
other than τ (to which the given ticket Ti belongs). We
argue that it is enough to draw 2 tickets from the collection
of tickets with non-τ tuples and select the tickets having less
similarity with Ti . To see this observe that in this case the
success corresponds to the event of selecting tickets which
are dissimilar with Ti . Drawing dissimilar tickets from a
collection of tickets having tuples other than τ can be seen
as independent events. Using the argument in Section C in
the Appendix we choose to draw 2 tickets randomly from
the collection of tickets having non-τ tuple.
3.2 Structure of the DNN
The structure of the DNN for recommending resolution for
a new ticket is given in Fig 3. The network finds similarity
between the new ticket summary and a subset of existing
ticket summaries. Prior to computing similarity, the DNN
reduces the high dimensional feature vectors representing
ticket summaries into low-dimensional vectors. For that, it
uses DNNR that reduces dimension. This is a multilayer
feed-forward deep neural network.
The structure of our DNNR is given in Fig 4. The input
layer of DNNR consists of n number of nodes where n is
the size of the feature vector of ticket summary. We assume
there are N − 1 number of hidden layers. Let ~ x be the input
feature vector, y as output vector. Let hi , i = 1, 2, . . . , N be
the intermediate hidden layers, Wi be the ith weight matrix
and bi be the ith bias term. We have
h1 = W1 ~x
hi = f (Wi hi−1 + bi ), i = 2, 3, . . . , N − 1. (1)
y = f (WN hN −1 + bN )
We use tanh as the activation function at the output layer
and at the hidden layers. Recall the tanh function is defined
as
1 − e−2z
f (z) = tanh(z) = (2)
1 + e2z
The output of DNNR is passed through a cosine similarity
function as shown in Fig 3. As we consider pair of tickets, the
output of two cosine similarity function nodes are combined.
3.3 Training of the DNN model
For the training of the DNN, we take a set of M ticket sum-
maries (Tm , m = 1, 2, . . . , M ). The summary of each ticket
m+
Tm is coupled with one pair of similar tickets T(i,j) and three
m−
pairs of dissimilar tickets T(i0 ,j 0 ) . We use the abridged nota-
m+
tion T(i,j) for the chosen similar tickets Ti , Tj corresponding
to the ticket Tm . Similarly, we use the notation T(im− 0 ,j 0 ) for
the chosen dissimilar tickets Ti0 , Tj 0 corresponding to the
ticket Tm . These similar (dissimilar) tickets are chosen us-
Figure 4: DNNR: Part of DNN that reduces the
dimension
ing the method as mentioned earlier. These four pairs of
similar and dissimilar tickets are represented by a set Tm .
The given ticket Tm and each of the pairs of similar and
dissimilar tickets are fed to the DNNR one by one. Let ym
be the output feature vector for Tm and yi and yj be the
output feature vectors for Ti and Tj respectively.
The cosine similarity between one output ym and another
output yi are computed using Eqn 3. Then this cosine simi-
larity of two tickets (Ti , Tj ) wrt the ticket Tm are combined
to generate R(Tm , T(i,j) ) in Eqn 4. Same equations are used
for computing the cosine metric and R-value for dissimilar
pairs.
ym T yi
cosine(ym , yi ) = (3)
||ym ||||yi ||
1
R(Tm , T(i,j) ) = (cosine(ym , yi ) + cosine(ym , yj )) (4)
2
These R-values of pairs of similar and dissimilar tickets
wrt Tm are fed to the Softmax function as shown in Fig 3.
The Softmax function computes posterior probabilities [9].
m
The posterior probability for R(Tm , T(i,j) )3 is given in Eqn 5
as below.
m
m exp(γR(Tm , T(i,j) ))
P (T(i,j) |Tm ) = P , (5)
T m0 ∈Tm exp(γR(T m , T(im0 ,j 0 ) ))
(i ,j 0 )
where γ is the smoothing parameter in the Softmax function.
As our objective is to find the most similar ticket (pairs) for
a given ticket Tm , we maximize the posterior probability for
the similar (or positive) pairs. Alternatively, we minimize
the following loss function
L(Ω) = −log
Y m+
P (T(i,j) |Tm ) (6)
m+
(Tm ,T(i,j) )
where Ω denotes the set of parameters {Wi , bi : i =
1, 2, . . . , N } of the neural networks. L(Ω) is differentiable
wrt Ω as it is continuous and its (partial) derivatives
are also continuous. So, the DNN can be trained using
gradient-based numerical optimization algorithm. The de-
tailed derivation is given in Section D of Appendix.
3
this posterior probability resembles the probabilistic IR
model similarity in Section A in Appendix for ranking tick-
ets
3.4 Validation of the DNN model
The resolution for a new ticket T is recommended as fol-
lows. At first we seek the tuple τ corresponding to T . Then
the existing tickets belonging to τ are fed to DNN. We as-
sume that a ticket having the tuple τ is relevant, at least
partially to the tickets belonging to the same τ in terms of
resolution. To find the most relevant resolution correspond-
ing to those tickets, we use the DNN to find the most similar
ticket summary wrt new ticket summary.
The feature vector of the new ticket T and those of the
existing tickets {Tk , 1 ≤ k ≤ K} from the tuple τ are fed
to DNNR one by one. Let y be the output of DNNR for
T and yk be the output of DNNR for Tk . We find the co-
sine similarity between y and yk for each ticket Tk having
tuple τ . Then we pair two cosine similarity outputs and ob-

tain the R-value (Eqn 4) for each pair. There will be K
2
possible pairs. However, we take only K/2 distinct pairs of
combinations. If K is odd then we consider the cosine sim-
ilarity of the remaining ticket summary Tk wrt T with one
of the randomly chosen tickets in τ -tuple and then compute
the R-value. Note we are computing the cosine similarity of
each ticket in tuple τ with the new ticket, we do not leave
out any ticket in the tuple.
The ticket pairs are ranked based on the R values. Higher
the value of R, more similar the ticket pair wrt the new
ticket T . Out of these K/2 (or dK/2e) pairs, suppose the
(Tu , Tv )-pair obtain the maximum R-value. Then this pair
collectively is the closest pair wrt T . Then the correspond-
ing resolutions of these two summaries are published as the
recommended resolutions for the new ticket if R-value of
this pair is greater than a threshold value θ. If more than
one pair have same maximum R-value (> θ) or almost same
maximum R-value (within 1% variation) then we publish all
the corresponding resolutions of these tickets pairs. In our
experiment, we have considered θ = 0.8.
4. EXPERIMENTAL RESULTS
In this section we report on our experiments. We imple-
ment our deep neural network using Java. We perform our
experiment using our internal data set. Our approach facili-
tates for various architectures of the DNN with many hidden
layers, and each layer in turn, contains different number of
nodes. However in this paper, we have considered one single
architecture with two hidden layers. So, in our experiments,
we have taken N = 2 + 1 = 3. The first and second hidden
layers consists of 200 nodes and 100 nodes respectively. The
output layer has only 50 nodes.
4.1 Internal ticket data
We have used data sets from different domains in ITIL
services in our company to validate our methodology, these
domain are AMD and Retail. The data from AMD domain
portrays information on application maintenance. It con-
sists of 10653 tickets. As mentioned earlier, the ticket sum-
maries are preprocessed and are represented by TF*IDF fea-
ture vectors. The dimension of a feature vector that repre-
sents a ticket summary is 1253. The Retail data includes
14379 tickets which contains data related to services to cus-
tomers. Each ticket summary in this domain is represented
by a 491-dimensional feature vector. The details of these
two data sets are given in Table 1.
4.2 Data Partition
Domain Total tickets No of Input feature
tuples vector dimension
AMD 10653 270 1253
Retail 14379 150 491
Table 1: Ticket data from different domains
Domain Total #Reco- Average Average
test mmended Similarity Similarity
tickets tickets (SS evaluator) (manual
evaluation)
AMD 816 369 0.71 0.92
Retail 1384 618 0.67 0.70
Table 2: The performance of DNN on recommend-
ing resolutions
We randomly pick 10% of each data set (tickets) as the
test set and 20% of the data set as the training set. These
20% of the data set as the training set constitute M number
of tickets for training. For each ticket of the training set
(Tm , m = 1, 2, . . . , M ), we take one pair of similar tickets
and three pairs of dissimilar tickets from the remaining 70%
of the data set (with repetition).
4.3 Recovering resolutions for new tickets
We train the DNN with 50 epochs or iterations. We at-
tempt training of the model with few values of learning rate
parameter t (Eqn 7). The learning rate t is uniform across
the iterations. Out of these trained models using different
values of learning rate, we consider the model for which the
learning curve (by minimizing loss function) was steadily
decreasing.
After obtaining the trained model, we validated the model
using test set. For each ticket of test set, our DNN model
recommends two or more resolutions if the R-value of win-
ning (best) pair(s) is greater than a threshold value θ. In
our experiment, we have considered θ = 0.8.
We compared the actual resolution of each test ticket with
the recommended resolution pair using semantic similarity
score ranging between 0 to 1. Then we computed the average
semantic similarity score over all recommended cases. The
total number of test tickets, the number of test tickets for
which the model recommended the resolutions with θ = 0.8,
and the average semantic similarity are given for both data
sets in Table 2.
Along with semantic similarity approach (SS)-based eval-
uation we have also manually evaluated the similarity be-
tween two tickets. Towards this we inspected the actual res-
olution of each ticket and the corresponding recommended
resolutions for evaluation purposes. We use three similarity
scores of 0, 0.5 and 1. If the meaning of a pair of actual res-
olution and recommended language appear to be the same
(using meta language oriented informal semantics) then we
assign a similarity score of 1 to this pair. If we find that the
meaning of the elements of this pair are not exactly same,
but there is some match then we provide a score of 0.5 to this
pair. Otherwise (in case the resolutions completely differ in
their meaning) we score this pair 0. As before we calculate
the average manual similarity score over all test tickets.
The effectiveness of our DNN approach to find similar
 Domain Average similarity Average Similarity Domain Total #Reco- Average
(SS evaluator) (manual evaluation) test mmended Similarity
AMD 0.93 0.98 tickets tickets (SS evaluator)
Retail 0.83 0.90 AMD 816 442 0.56
Retail 1384 988 0.60
Table 3: The performance of DNN on finding similar
ticket summaries Table 6: The performance of simple cosine similarity
approach on resolution recommendation
Tuple No. of test No. of % of
Name tickets matches matches
Report Masks Tuple No. of test No. of % of
Administration 99 99 100.00 Name tickets matches matches
Separation Request Report Masks
AS400 Legacy - Direct Response Administration 99 99 100.00
Software 19 12 63.16 Separation Request
Application Errors AS400 Legacy - Direct Response
AS400 Legacy - Manufacturing/Packaging Software 65 41 63.08
Administration 47 34 72.34 Application Errors
Configuration AS400 Legacy - Manufacturing/Packaging
AS400 Legacy - Purchasing Administration 41 28 68.29
Software 8 7 87.50 Configuration
Application Functionality Issue AS400 Legacy - Purchasing
AS400 Legacy - Manufacturing/Packaging Software 20 14 70.00
Software 2 1 50.00 Application Functionality Issue
Application Functionality Issue AS400 Legacy - Manufacturing/Packaging
Software 5 2 40.00
Application Functionality Issue
Table 4: The performance of DNN in top 5 tuples
Table 7: The performance of simple cosine similarity
in top 5 tuples
ticket summaries (comparison between actual test summary
and summaries of recommended tickets) is given in Table 3.
It shows that our DNN approach could successfully find sim- this approach we consider the TF*IDF vector representation
ilar ticket summaries for a given new ticket summary. Lower of ticket summaries (input to the DNN) and compute the
similarity values for recommending resolutions (Table 2) is similarity of a new ticket with other tickets in the same tuple
primarily due to the variations in resolutions as documented to find out the most similar ticket for resolution recovery. In
by different maintenance personnel for even the same ticket this approach the feature vector of a ticket T is represented
summary. as (x1 , . . . , xn ) using n terms/keyphrases, where xi denotes
The performance of the DNN for top (maximum tickets) the TF*IDF weight of the ith term in ticket T . Given two
5 tuples in test set is given in Table 4 for AMD domain. tickets T and T 0 with their feature vector representations
In this result, we count a match if the semantic similarity (x1 , . . . , xn ) and (y1 , . . . , yn ) respectively, their Cosine-based
score using SS approach between actual resolution and rec- Σn xi yi
similarity [12] is given by simC (T, T 0 ) = √ n i=1 2
√ n 2 .
ommended resolution is greater than 0.5. We have given the ( Σi=1 xi )( Σi=1 yi )
name of tuples in the order of application name, category Once a new ticket T 0 arrives, we find the tuple associated
and sub-category. with it. Then we compute the similarity of T 0 with each
We have also conducted experiment without employing ticket T in the same tuple. Then we pick out the ticket hav-
semantic similarity approach (SS) during training. In this ing the maximum similarity (tickets having similarity with
case we pick up 2 tickets randomly from the same tuple as a tolerance 0.05) and publish its resolution as the recom-
of the original ticket and compute their cosine similarity wrt mended resolution for the new ticket.
the latter. The results are shown in Table 5. It highlights The performance of the simple cosine similarity (approach
the fact that our DNN approach does not depend much on 2) is given in Table 6. Note that, the performance using
the SS approach to select similar/dissimilar tickets during simple cosine similarity approach is lower than the DNN
training. approach.
For this simple cosine similarity approach, we have also
4.4 Comparison with other methods provided performance in top 5 tuples in test set for AMD
We now compare our method with two approaches, one domain in Table 7.
based on cosine similarity (approach 2) of feature vector of
tickets and another related to clustering and kNN-search Approach 3: Clustering and kNN-search: In this ap-
(approach 3). Both approaches are described below. proach we consider tickets as bag of items for grouping them
into clusters and place the new ticket in the appropriate clus-
Approach 2: Cosine-based similarity of tickets: In ter to find its k nearest neighbors for recovering its resolu-
tion [18]. In this case we consider both the fixed field entries
and free field (summary) for finding the feature vector of
Domain Total #Reco- Average the tickets. That is, a profile of a ticket is given as T =
test mmended Similarity (x1 , . . . , xk , xk+1 , . . . , xn ), where x1 , . . . , xk are the proper
tickets tickets (manual evaluation) representation (categorical or bucket) of feature elements
AMD 822 363 0.89 corresponding to fixed field entries of T and xk+1 , . . . , xn
Retail 1384 583 0.68 are the appropriate weights (we use TF*IDF [20] in this
case also) for the chosen keywords/keyphrases from the free
Table 5: The performance of DNN on recommend- form entries of T . For more details see [17]. Also we con-
ing resolutions without using SS approach during sider a hybrid approach for computing the distance metric.
training We use Jacaard metric to compute the distance on fixed el-
 Domain Average
semantic similarity
AMD 0.38
Retail 0.50
Table 8: The performance of clustering and kNN-
search method
ement entries of feature space and Cosine distance metric
for keyphrases. Then we take a convex combination of these
distances to formulate our metric [18, 17].
Given the set of tickets and a hybrid distance metric on
it we group the tickets into different clusters. Once a new
ticket arrives we perform natural language parsing on its
summary to identify keywords, and thus generate the feature
vector for it based on these keywords (keyphrases). Then we
compute the distance of this ticket from all other clusters
and place the ticket in the particular cluster to which it is
the closest. Further we shortlist a group of tickets based on
the tuple of the new ticket. Using a KNN-based approach we
choose a ticket in this shortlist which are nearer to the new
ticket using the distant metric and publish the corresponding
resolution as the recovered resolutions for the new ticket.
The performance of the third approach (clustering and
kNN search) is given in Table 8. Our DNN approach per-
formed better than both cosine similarity approach and clus-
tering and kNN search method.
5. RELATED WORK
There are some pieces of works which deal with recom-
mendation of resolution for incoming tickets. Notable among
them is [26] where the authors proposed a kNN (k-nearest
neighbor)-based method to recommend resolution for event
tickets using a similarity metric between events and past
resolutions of other events. The method proposed therein
heavily depended on the underlying similarity measure in
kNN. In a similar work [23] an analysis of the historical
event tickets from a large service provider was carried out.
Two resolution-recommendation algorithms for event tickets
were proposed on historical tickets by considering false pos-
itive tickets which would often be generated by monitoring
systems. The authors built on the work of kNN-based algo-
rithms for recommending resolutions in [25] where they used
SCL (structural corresponding learning) based feature adap-
tation to uncover feature mapping in different time intervals
as ticket descriptions differ in case of change of servers’ en-
vironments over a period of time during which resolutions
remained unchanged. Further they applied this algorithm on
tickets grouped by different time interval granularities to ac-
count for the periodic regularities present in ticket datasets.
All the methods proposed above for recommending reso-
lution exploited the underlying similarity measure in kNN-
based search. In [18], Roy et al. proposed a method meant
to be applied to a more general setting. In this approach
it was possible to identify tickets similar to a new ticket by
suitably clustering the tickets using a Cosine metric. In par-
ticular, the authors put forth an automated method based
on unsupervised learning for recovering resolutions for in-
coming tickets using the traditional kNN search by which
they were able to achieve a promising similarity match of
about 48% between the suggestions and the actual resolu-
tion.
The idea of using deep learning in our work originates
from work on learning deep structured latent models for
web search [7, 8, 9]. Latent semantic models [5, 2] have
been used to bridge the gap between Web documents and
search queries by mapping query to its relevant documents
at the semantic level and grouping different terms appearing
in similar context into the same semantic cluster. There have
been further extensions of these approaches. New models
for clickthrough data like Bi-lingual Topic Models (BLTMs)
and linear Discriminative Projection Models (DPMs) [7, 8],
(which consist of queries and their clicked document) have
been used to plug the gap between search queries and web
documents. In another approach, the authors in [19] have
extracted a hierarchical semantic structure embedded in the
query and the document via deep learning. Combining both
the approaches, Huang et al. have proposed a series of deep
structured semantic models for ranking a set of documents
for a given query in the following manner [9]. They pro-
jected the query and the documents to a common semantic
space, following which the relevance of each document wrt
the given query was judged using the cosine similarity be-
tween their vectors in the common semantic space. By max-
imizing the conditional likelihood of the clicked document
given the query they trained their neural network models.
Motivated by these ideas we use deep structured models
to recommend resolutions for tickets in ITIL services. We
project existing tickets and a new ticket to a common low
dimensional space and then compute the similarity of the
new ticket with other tickets. We select the ticket which
has highest similarity with the new ticket and pick the reso-
lution of the former as the recommended resolution for the
new ticket. For training purposes we maximize the posterior
probability of the new ticket with similar tickets. However,
we do not need to hash the input ticket vectors because tick-
ets are short text data and hasing increases the dimension
of feature vectors instead of reducing them.
6. CONCLUSION
In this work we have proposed deep learning based recom-
mendation algorithm for recovering resolutions for incoming
tickets in ITIL services. Our learning algorithm provides
some advantages over the traditional resolution recommen-
dation techniques. The methods for recommending resolu-
tions in [26, 23, 25] use similarity measures to compute the
k nearest neighbors for the incoming ticket to suggest reso-
lutions for the latter. For improving the similarity measure
used in KNN the authors therein utilize both the event and
resolution information in historical tickets via a topic-level
feature extraction using the LDA (Latent Dirichlet Alloca-
tion) model. In our deep learning based technique we use a
high dimensional input vector for tickets which does not re-
quire much natural language processing. Further we do not
need to compare the similarity of each ticket in the reposi-
tory with the new ticket. Clustering-based recommendation
algorithm [18] alleviates the comparison of the new ticket
with each existing ticket, however proper clustering of tick-
ets is challenging as they have short text content. Also,
in our method we use deep structured models for ranking
the tickets against the new ticket in the same tuple. Thus
by mapping high-dimensional input vector of tickets to low-
dimensional vectors we avoid the “curse of dimensionality”
phenomenon.
7. REFERENCES
[1] Y. Bengio. Learning deep architectures for AI.
Foundations and Trends in Machine Learning,
2(1):1–127, 2009.
[2] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent
Dirichlet Allocation. Journal of Machine Learning
Research, 3:993–1022, 2003.
[3] R. Collobert, J. Weston, L. Bottou, M. Karlen,
K. Kavukcuoglu, and P. P. Kuksa. Natural language
processing (almost) from scratch. Journal of Machine
Learning Research, 12:2493–2537, 2011.
[4] M.-C. de Marneffe, B. MacCartney, and C. D.
Manning. Generating typed dependency parses from
phrase structure parses. In International Conference
on Language Resources and Evaluation (LREC’06),
pages 449–454, 2006.
[5] S. C. Deerwester, S. T. Dumais, T. K. Landauer,
G. W. Furnas, and R. A. Harshman. Indexing by
latent semantic analysis. JASIS, 41(6):391–407, 1990.
[6] L. Deng, X. He, and J. Gao. Deep stacking networks
for information retrieval. In IEEE International
Conference on Acoustics, Speech and Signal
Processing, ICASSP’13, Vancouver, BC, Canada,
pages 3153–3157, 2013.
[7] J. Gao, X. He, and J. Nie. Clickthrough-based
translation models for web search: from word models
to phrase models. In Proceedings of the 19th ACM
Conference on Information and Knowledge
Management, CIKM’10, pages 1139–1148, 2010.
[8] J. Gao, K. Toutanova, and W. Yih.
Clickthrough-based latent semantic models for web
search. In Proceeding of the 34th International ACM
SIGIR Conference on Research and Development in
Information Retrieval, SIGIR’11, pages 675–684, 2011.
[9] P. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. P.
Heck. Learning deep structured semantic models for
web search using clickthrough data. In 22nd ACM
International Conference on Information and
Knowledge Management, CIKM’13, pages 2333–2338,
2013.
[10] D. Johnson. NOC Internal Integrated Trouble Ticket
System Functional Specification Wishlist. In RFC
1297, 1992.
[11] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning.
Nature, 521:436–444, 2015.
[12] J. Leskovec, A. Rajaraman, and J. Ullman. Mining of
Massive Datasets. Cambridge University Press, 2nd
edition, 2014.
[13] Y. Li, D. McLean, Z. Bandar, J. O’Shea, and K. A.
Crockett. Sentence similarity based on semantic nets
and corpus statistics. IEEE Trans. Knowl. Data Eng.,
18(8):1138–1150, 2006.
[14] A. Medem, M.-I. Akodjenou, and R. Teixeira.
Troubleminer: Mining network trouble tickets. In
Integrated Network Management-Workshops, IM’09,
2009.
[15] L. Rokach. Ensemble-based classifiers. Artificial
Intelligence Review, 33(1-2):1–39, 2010.
[16] S. Ross. A First Course in Probability. Pearson
Education Limited, 9th edition, 2014.
[17] S. Roy, D. P. Muni, J. Y. T. Yan, N. Budhiraja, and
F. Ceiler. Clustering and labeling IT maintenance
tickets. In Service-Oriented Computing - 14th
International Conference, ICSOC 2016, Banff, AB,
Canada, Proceedings, pages 829–845, 2016.
[18] S. Roy, J. Y. T. Yan, N. Budhiraja, and A. Lim.
Recovering resolutions for application maintenance
incidents. In IEEE International Conference on
Services Computing, SCC’16, San Francisco, CA,
USA, pages 617–624, 2016.
[19] R. Salakhutdinov and G. E. Hinton. Semantic hashing.
Int. J. Approx. Reasoning, 50(7):969–978, 2009.
[20] G. Salton and C. Buckley. Term Weighing Approaches
in Automatic Text Retrieval. Information Processing
and Management, 1988.
[21] J. Schmidhuber. Deep learning in neural networks: An
overview. Neural Networks, 61:85–117, 2015.
[22] R. Socher, B. Huval, C. D. Manning, and A. Y. Ng.
Semantic compositionality through recursive
matrix-vector spaces. In Proceedings of the Joint
Conference on Empirical Methods in Natural Language
Processing and Computational Natural Language
Learning, EMNLP-CoNLL’12, pages 1201–1211, 2012.
[23] L. Tang, T. Li, L. Shwartz, and G. Grabarnik.
Recommending resolutions for problems identified by
monitoring. In IFIP/IEEE International Symposium
on Integrated Network Management (IM’13), pages
134–142, 2013.
[24] G. Tür, L. Deng, D. Hakkani-Tür, and X. He.
Towards deeper understanding: Deep convex networks
for semantic utterance classification. In IEEE
International Conference on Acoustics, Speech and
Signal Processing, ICASSP’12, pages 5045–5048, 2012.
[25] W. Zhou, T. Li, L. Shwartz, and G. Y. Grabarnik.
Recommending ticket resolution using feature
adaptation. In 11th International Conference on
Network and Service Management, CNSM’15, pages
15–21, 2015.
[26] W. Zhou, L. Tang, T. Li, L. Shwartz, and
G. Grabarnik. Resolution recommendation for event
tickets in service management. In IFIP/IEEE
International Symposium on Integrated Network
Management, IM’15, pages 287–295, 2015.
APPENDIX
A. PROBABILISTIC IR MODEL FOR CLAS-
SIFICATION
In this model the historical tickets are ranked according
to the probability of being relevant to a new ticket. By
this one can compute the semantic similarity of an incom-
ing ticket T 0 with an existing ticket T in the repository us-
ing a similarity function based on conditional probability,
sim(T 0 , T ) = P (T 0 |T ). Applying Bayes’ rule [16], we can
0
)P (T 0 )
rewrite the above as P (T 0 |T ) = P (T |T P (T )
.
For a given class of new tickets, we can take P (T ) to be
constant and can further assume that all tickets have the
same probability. Therefore in the presence of historical
tickets such as T , any incoming ticket T 0 can be ranked
0
)P (T 0 )
using, sim(T 0 , T ) = P (T 0 |T ) = P (T |T
P (T )
= P (T |T 0 ). We
shall use similar probabilistic IR model for ranking tickets.
B. OPTIMIZING THE NUMBER OF DRAWS above probability value always decreases. Thus we draw 2
FOR FINDING SIMILAR TICKETS tickets randomly from the set of tickets having tuples other
than τ .
We seek to find the probability of choosing tickets which
are similar to a ticket belonging to the same tuple. Suppose
there are Nτ tickets which are in the same tuple τ . Further D. GRADIENT DESCENT
assume that there are Nτi number of tickets having tuple τ
The DNN can be trained using gradient-based numerical
which are similar to ticket Ti with the same tuple τ . Then
optimization algorithms [9] because L(Ω) is differentiable
the probability of choosing a ticket similar to ticket Ti is
Ni
wrt Ω. The parameters in (Ω) are updated as
p = Nττ . Then the probability of failure, that is choosing a
ticket which is dissimilar to ticket Ti is q = 1 − p. As we are ∂L(Ω)
Ωt = Ωt−1 − t | , (7)
required to pick a pair of tickets we decide on the number ∂Ω Ω=Ωt−1
of draws of tickets from the collection in which there is a
maximum chance of choosing at least 2 tickets which are where t is the learning rate at the tth iteration, Ωt and Ωt−1
similar to the ticket Ti . are the model parameters at the tth and (t − 1)th iteration,
Suppose we are allowed m number of draws of tickets from respectively.
a collection of tickets in tuple τ . We need to determine the Let M be the number of the ticket summaries (Tm ), We
probability of picking at least 2 tickets similar to the given consider the combination of a pair of similar (positive) ticket
+
ticket Ti . Each such draw of ticket from the collection can summaries T(i,j) and three pairs of dissimilar (negative) ticket
be seen as a Bernoulli trial with the probability of success p summaries T(i+0 ,j 0 ) : 1 ≤ i0 , j 0 ≤ 3 for training the DNN. Then
and the probability of failure q. Let X be a random variable we can denote each m-th combination as (Tm , T(i,j) m+
). Then
which takes the value of the number of similar tickets to Ti we can write
drawn from the collection of tickets with tuple τ . Then X
follows a binomial distribution with the number of draws L(Ω) = L1 (Ω) + L2 (Ω) + · · · + Lm (Ω) + · · · + LM (Ω), (8)
being m and the probability of success p.
m+
We need to determine P (X ≥ 2). We find out the proba- where Lm (Ω) = − log P (T(i,j) |Tm ), 1 ≤ m ≤ M (9)
bility of the complementary event P (X < 2) and then com- M
pute 1 − P (X < 2). ∂L(Ω) X ∂Lm (Ω)
and, = (10)
Now P (X = k) = m

k m−k
k
·p q we have P (X < 2) = ∂Ω m=1
∂Ω
m 0 m m 1 m−1
2) = 1−q m +mpq m−1 .



0
·p q + 1
·p q and P (X >
Lm (Ω)
For different values of q we plot P (X > 2) against m. The
m+
resulting graph asymtotically approaches 1. For different = − log P (T(i,j) |Tm )
values of q being less than 0.5 we observe that the curve 
m+

begins to approach 1 when m is close to 5. In view of the exp(γR(Tm , T(i,j) ))
= − log  P 
above we choose the number of draw to be 5. That is to say, T m0 0 ∈Tm exp(γR(Tm , T(i0 ,j 0 ) ))
m
(i ,j )
we randomly pick 5 tickets from the collection of tickets with
m
P 
tuple τ and select two tickets which have higher semantic T m0 0 ∈Tm exp(γR(Tm , T(i0 ,j 0 ) ))
(i ,j )
similarity with the given ticket Ti . = log  m+

exp(γR(Tm , T(i,j) ))
 P m− 
C. OPTIMAL NUMBER OF DRAWS FOR T m−
(i0 ,j 0 )
m+ exp(γR(Tm , T 0 0 ))
∈Tm \T(i,j) (i ,j )
= log 1 +
FINDING OUT DISSIMILAR TICKETS

m+
exp(γR(Tm , T(i,j) ))
Now we need to find an optimal number of draws in which  
exactly 2 number of dissimilar tickets are chosen. Recall X  
that we pick up dissimilar tickets from the set of tickets = log 1 + exp γR(Tm , T(im− m+
0 ,j 0 ) ) − γR(Tm , T(i,j) ) 
 
having a tuple other than τ (the given ticket Ti belongs to T m−
0 0
(i ,j )
this tuple). We argue that it is enough to draw 3 tickets  
from the collection of tickets with non-τ tuples and select X  h i
m+
the tickets having less similarity with Ti because it will have = log 1 + exp −γ R(Tm , T(i,j) ) − R(Tm , T(im−
0 ,j 0 ) )
 

high probability of containing two dissimilar tickets. T m−
0 0
(i ,j )
To see this observe that the success will correspond to  
the event of selecting tickets which are dissimilar with T .
Drawing dissimilar tickets from a collection of tickets hav-
X
exp −γ∆m


= log 1 +

(i0 ,j 0 ) 
ing tuples other than τ can be seen as independent events.
T m−
0 0
Suppose we keep on picking up tickets from collection of (i ,j )

tickets having non-τ tuples until r dissimilar tickets are m+

where ∆m
(i0 ,j 0 ) = R(Tm , T(i,j) ) − R(Tm , T(im−
0 ,j 0 ) )
picked. If X denote the number of trials required (note X
follows

negative binomial distribution), then P (X = m) =  
m−1 r (m−r)
r−1
q (1−q) . In this case r = 2. Hence P (X = m) = X
exp −γ∆m


Therefore, Lm (Ω) = log 1 +

(i0 ,j 0 ) 
(m − 1)q 2 (1 − q)m−2 . For m = 2 this value becomes q 2 . For
picking dissimilar tickets from non-τ tuples we can take a T m−
0 0
(i ,j )
higher value of q = 0.7, then q 2 ≈ 0.5. As q 2 (1 − q)m−2 is (11)
monotonically decreasing function for m (having fixed q) the The gradient of the loss function w.r.t the N-th weight ma-
trix WN is The gradient of the loss function w.r.t the intermediate weight
∂∆m matrix, Wl , l = 2, 3, . . . , N − 1, can be computed as
∂Lm (Ω) X m (i0 ,j 0 )
= α(i0 ,j 0 ) (12) ∂∆m
∂WN ∂WN ∂Lm (Ω) X m (i0 ,j 0 )
m−
T 0 0 = α(i0 ,j 0 ) (17)
(i ,j ) ∂Wl m−
∂W l
T 0 0
(i ,j )
where
m+ where
∂∆m
(i0 ,j 0 ) ∂R(Tm , T(i,j) ) ∂R(Tm , T(im−
0 ,j 0 ) )
= − (13)
"
∂∆m

∂WN ∂WN ∂WN (i0 ,j 0 ) 1 (Tm ,Tim+ ) T (Tm ,Tim+ ) T
= δl,Tm hl−1,Tm +δ hl−1,T m+
∂Wl 2 l,T m+
i i
and  
(Tm ,Tjm+ ) (Tm ,T m+ )
m −γ exp(−γ∆m (i0 ,j 0 ) ) + δl,Tm hTl−1,Tm + δ m+j hTl−1,T m+
α(i 0 ,j 0 ) = (14) l,Tj j
exp(−γ∆m
P
1+ T m− (i00 ,j 00 ) )
00 00  m−

(i ,j ) (Tm ,T 0 ) T (Tm ,T m−
0 ) T
− δl,Tm i
hl−1,Tm + δ m− i
hl−1,T m−
Let ym , yi and yj be the outputs of DNNR with Tm , Ti l,T 0
i i0
and Tj ticket summaries. #
(Tm ,T m− (Tm ,T m−

0 ) 0 )
m
∂R(Tm , T(i,j) ) T T − δl,Tm j
hTl−1,Tm + δ j
hTl−1,T m−
l,T m−
  
∂ 1 ym yi ym yj j 0 j0
= +
∂WN ∂WN 2 ||ym ||||yi || ||ym ||||yj || (18)
1 (Tm ,Ti ) T (Tm ,Ti ) T
= (δym hN −1,Tm + δyi hN −1,Ti This derivation is based on the derivation given in [9]. We
2
(T ,T ) (T ,T )
modify the derivation in taking into account pairs of tickets
+δymm j hTN −1,Tm + δyj m j hTN −1,Tj ) (documents) instead of a single ticket (document) while con-
(15) sidering positive (similar) and negative (dissimilar) tickets
(documents).
where,

δy(Tmm ,Ti ) = (1 − ym ) ◦ (1 + ym ) ◦ (bc1 yi − a1 c1 b3 ym )

δy(Ti m ,Ti ) = (1 − yi ) ◦ (1 + yi ) ◦ (bc1 ym − a1 bc31 yi )

(T ,Tj )
δymm = (1 − ym ) ◦ (1 + ym ) ◦ (bc2 yj − a2 c2 b3 ym )

(T ,Tj )
δyj m = (1 − yj ) ◦ (1 + yj ) ◦ (bc2 ym − a2 bc32 yj )

T T
a1 = ym yi , a 2 = ym yj

1 1 1
b= , c1 = , c2 =
||ym || ||yi || ||yj ||
The operator ‘◦’ denotes the element-wise multiplication.

For hidden layers, we also need to calculate δ for each ∆m

(i0 ,j 0 ) .
We calculate each δ in the hidden layer l through back prop-
agation as
(T ,Ti ) T (T ,T )
δl,Tm
m
= (1 + hl,Tm ) ◦ (1 − hl,Tm ) ◦ Wl+1 m i
δl+1,T m
(T ,Tj ) T (T ,T )
δl,Tm
m
= (1 + hl,Tm ) ◦ (1 − hl,Tm ) ◦ Wl+1 m j
δl+1,T m
(T ,Ti ) T (T ,T )
δl,Tm = (1 + hl,Ti ) ◦ (1 − hl,Ti ) ◦ Wl+1 m i
δl+1,T (16)
i i
(T ,Tj ) T (Tm ,Tj )
δl,Tm
j
= (1 + hl,Tj ) ◦ (1 − hl,Tj ) ◦ Wl+1 δl+1,T j

with
(T ,T ) (T ,T ) (T ,Tj )
m i
δN,T m
= δy(Tmm ,Ti ) , δN,T
m j
m
= δymm
(T ,Ti ) (T ,Tj ) (T ,Tj )
m
δN,T i
= δy(Ti m ,Ti ) , δN,T
m
j
= δyj m