A data mining model of knowledge discovery

based on the deep learning

Yonglin Ma1, Yuanhua Tan2, Chaolin Zhang2, and Yici Mao3
1
Application Management office of SINOPEC IT management Department, Beijing, 100728, China
2
Karamay Hongyou Software Co., Xinjiang, 834000, China
3
Karamay Municipal People's Government Bureau of Information Industry, Xinjiang, 834000, China
E-mail: mayl@sinopec.com (Y. Ma), tanyh66@petrochina.com.cn (Y. Tan)
lin_2728@126.com (C. Zhang), 18909901617@126.com (Y. Mao)

AbstractWith the development of the database technology
and the spread of the internet, the amount of the data in
databases increases at an exponential speed, which yields the
difficult problems of excess data and information explosion,
etc. The traditional database technology is restricted in reading
and writing, querying and basic statics operations, but cant
acquire the deep data attributes or implicit information. Facing
with the huge database in all kinds of fields, it is more and more
difficult to cope with the big data only by using conventional
technology. New technique to deal with these data at a high level
is eagerly demanded. Therefore, the KDD (Knowledge Discovery
in Database) technology arises at the historic moment. KDD is an
integrated process, which includes data input, iterative solving,
user interface and many other custom requirements and design
decisions, where the data mining (DM) is a key and specific step
in KDD. This paper deeply analyzes state of the art technology of
DM, and points out the challenge and technological bottleneck of
DM. Moreover, a data mining model architecture of knowledge
discovery based on deep learning is proposed.
machine readable and machine interpretable format and must
represent knowledge in a manner that facilitates inference.
KDD can transform data to knowledge. DM plays a key role in
this transformation process. DM usually refers to extracting
effective information from a mass of data by using all kinds of
algorithms [2]. In most cases, DM has a close relationship with
computer science, by using statistics algorithm, on-line analytic
processing, information retrieval, machine learning, expert
system, pattern recognition and other latest techniques to
achieve the assignment [9]. When it comes to the relationship
of KDD and DM, many researchers hold different opinions. In
our opinion, KDD refers to the overall process of discovering
useful knowledge from data, while DM refers to a particular
step in this process [10],[11]. DM is the application of specific
algorithms for extracting patterns from mass of data. The
progress of KDD is shown in Fig. 1.

KeywordsKDD; data mining model; deep leaning;

I. INTRODUCTION
In recent years, with the unceasing development of the
computer and database technology, people have entered an
entirely new era where big data is everywhere [1]. The data
amount of different industries or businesses are accumulating
at a dramatic pace. According to statistics, the data transferred
from NASA satellites in orbit to ground monitoring stations is
over 50GB every minute. The number of transaction items of
Alibaba in China is over 200 million every day. In addition, the
data amount of meteorological record, medical treatment and
scientific research fields are also increasing at a nearly
exponential speed. How to find effective and useful
information from a mass of fuzzy database with noise is a
critical problem to be eagerly solved [2]-[7]. Traditional
database technology can only finish reading, writing, querying
and other basic statics operation, but does not have the ability
to explore the deep relationship of data, let alone extracting
high level knowledge. Fortunately, KDD provides a reasonable
idea to solve these problems [8].
KDD is the creation of knowledge from structured
(relational databases, XML) and unstructured (text, documents,
images) sources [4]. The resulting knowledge needs to be a
Fig. 1. The process of KDD. According to the information transfer theory, the
information amount cant increase during transform, so the information
amount decrease, but the information level becomes higher.
According to Fig. 1, it is easily found that in the process
from data source to knowledge, the information level increases
though the information amount decreases. The information
becomes more and more easy to be analyzed and processed and
meanwhile it is also easy to find the importance of DM in
KDD process. Thus, the DM model is the focus of this paper.
Facing with so enormous data, human analysts with no special
tools can no longer make sense. However, the data mining can
automate the process of finding relationships and patterns in
raw data and the results can be further utilized in an automated
decision support system.

978-1-4799-8389-6/15/$31.00 2015
c IEEE 1212
Fig. 2. The hierarchy structure of visual perception system in brain. The input information is pixel and after a process of iteration and abstraction, the input
information transforms to high level patterns.
Conventional data mining technology absorbs the ideas of
classical algorithms in AI field, and most of DM models have a
close relationship with AI algorithms [1]. Since the term Data
Mining has been introduced in the last century, it has achieved
great development up to now. Data mining is a natural
development with the increasing use of computerized databases
to store data and provides the answer to business analysts [12]-
[15]. In 1960s, people only managed to do data collection,
which only means recording the data due to the restriction of
computer technology and limited storage. In 1980s, with the
development of computer and database technology, the storage
device have been highly developed, and data analysis and
querying technology are also mature. Since 21 century, people
begin to pay more attention to the deep level information of
data to predict what will happen next [16], and the research of
data mining reaches a peak. So far, the data mining model can
be mainly classified into the following categories: 1) model
based on classical statics algorithm. 2) model based on cluster.
3) model based on SVM. 4) model based on artificial network.
However, after years of research, most of the conventional DM
models encounter a technological bottleneck which is hardly
broken. The core problem is that data analysis is not deep
enough, which cant transform raw data to knowledge
successfully. Moreover, many researches even stop at the level
of classification. In 2006, Hilton proposed greedy layer-wise
training of deep networks [1]. His work brings new hope for
deep structure analytical optimal problems. Later, multi-layer
automatic encoder for deep structure was proposed, which was
an outstanding breakthrough in deep analysis of raw data. Deep
learning research booms from that time and also provides a
new idea for DM model structure. In this paper, deep learning
is applied in DM model of knowledge discovery [16]-[17].
Moreover, a novel DM model architecture for KDD based on
deep learning is proposed.
In 1981, David Hubel and Torsten Wiesel found that
information processing of visual perception system in brain is
hierarchy structure, as shown in Fig. 2, and they carried out
many experiments on cat brain to prove their conjecture. This
result made people realize that the work pattern of nervous
system nerve-center-brain was probably an unceasing
iterative and abstract process. And deep learning is also based
on this theory. It tries to simulate the perception system in
2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA)
brain by using multi hierarchy structure. Unfortunately, most
of experiments before 2006 failed. Moreover, the Greedy
Layer-Wise Training provided a reliable method for the
implement of deep learning [6]. In view of latest work in deep
learning, deep belief network (DBN) is used in this paper [18],
and the supervised and unsupervised learning is also integrated
together in the proposed DM model based on deep learning.
II. CONVENTIONAL DM MODELS IN KDD
In this section, the convention DM models are presented
and compared. Moreover, the technological bottleneck of
conventional DM models is also pointed out.
A. Model based on classical statistic algorithm
At the preliminary development stage of DM, most of
models are based on classical statistic algorithms [19]-[21].
Among them, two representative models are Native Bayes
model and Decision Tree model [21]. Both of them are based
on the analysis of given data, and a prediction result will come
out when an unknown variable enters into the built model. The
representative application of these two models is to solve the
well-known problems, e.g., whether to play tennis or not
according to weather condition [8]. Eqs. (1)-(2) show that
Native Bayes model integrates the prior probability and
posterior probability by using probabilistic model as follows:
P( x | c) P(c)
P (c | x ) (1)
P( x)
P( x | X ) P( x1 | c) u P( x2 | c) u" u P( xn | c) u P(c) (2)
where P(x|c) is likelihood probability, P(c) is class prior
probability, P(x) is predictor prior probability, and P(c|x) is
posterior probability.
Decision tree model is based on ID3 algorithm which
divides the raw data into sub datasets, and the branches and
leaves of the decision tree gradually develop at the same time.
The decision tree is built from the top to the bottom. The raw
data can be classified into many sub datasets which contain
sample data according to a certain standard. ID3 algorithm uses
entropy to measure the amount of uncertainty in the data set S,
the Entropy H(S) is defined as follows:
1213
 H (S )  p ( x ) log 2 p ( x ) (3)
x X

where S is the current dataset whose entropy is being

calculated. X is the classes in S, p(x) is the proportion of the
number of elements in class x to the number of elements in set
S. Particularly, if H(S) = 0the subset S is perfectly classified.
Otherwise, the subset S can be further divided. In this case, ID3
algorithm uses information gain to measure the difference in
entropy from before to after when the subset S is split on an
attribute A [19]. In ID3, the information gain can be calculated
(instead of entropy) for each remaining attribute. The attribute
with the largest information gain is used to split the subset S on
this iteration.
Fig. 3. Schematic of SVM, where the green points and the red points are
B. Model based on Cluster separated into two regions by the support vectors.
Cluster algorithms usually merge the input data according
to center distance or hierarchical structure [23]-[25]. It tries to Subject to constraint in Eq. (8)
find the similar attributes of the raw data so as to classify the yi ( w xi  b) t 1  [i , i 1, 2" n (8)
data by comparing their similarity. Common cluster algorithms
includes K-means algorithm and expectation maximization In which C is penalty coefficient, [i is relaxant factor.
(EM) algorithm. Distance is usually used to measure the
similarity of data and there are three common distance D. Model based on artificial network
functions in cluster algorithm, which are defined as follows: Artificial network derives from the idea to simulate
k biological neural network and is an important branch of
Euclidean: d (x  y )i i
2
(4) machine learning [18]-[24]. Some well-known artificial
i 1 networks include biological neural network, back propagation
k network, and Hopfield network. In fact, all of these artificial
Manhattan : d | x  y |
i 1
i i (5) networks share two main ideas, namely, feedforward of raw
1/ q
data, and back propagation of error, as shown in Fig. 4.
k
Minkowski : d (| xi  yi |) q (6)
i1
x1 Ii w x ji j
x1
w j w
Cluster algorithm is a reasonable choice when we have to 1i
1i 'w K u d u x
classify the raw data to some kinds of categories. However,
w ji w ji
there is no transformation from data to knowledge, which xj f yi xj f H
means that the cluster algorithm can only finish preprocessing
for our data mining model.

ni
w
ni

C. Model based on SVM
SVM is a kind of regression algorithm, there are some
kinds of regression algorithms which are usually used in
mathematical analysis [9], such as ordinary least square,
logistic regression, stepwise regression, multivariate adaptive
regression, splines, locally estimated and scatterplot smoothing.
However, when dealing with high-dimensional data, these
methods easily encounter problems. Contrary, SVM has great
advantage in dealing with high dimensional data, and can also
divide data by hyperplanes [12], as shown in Fig. 3.
The original optimal hyperplane algorithm proposed by
Vapnik in 1963 was a linear classifier. After that, in 1992, B. E.
Boser suggested a way to create nonlinear classifiers by
applying the kernel trick to maximum-margin hyperplanes.
SVM can solve both the linear and nonlinear classification
problem. In most cases, searching the hyperplane is an
optimization problem, as shown in Eq. (7).
n the mentioned models. A novel DM model architecture based
1
arg min || w ||2 C [i
w,[ ,b 2
i 1
w
xn yi f (Ii ) xn
Fig. 4. Two main ideas in artificial network, where one is feedforward of raw
data, and the other is back propagation of error
By adjusting the input weight of perception and feedback
weight of error, the network can reach the optimum state.
E. Technological bottleneck in DM
Though many DM models have been proposed, its
development still encounters technological bottleneck after
years of research. The core problem is that most conventional
MD models do not have deep analysis ability. Thus, they
cannot simulate the thought of primate. These models mainly
work on mathematical analysis rather than intelligent analysis.
However, with the continuous increase of data amount, it is
more and more important to handle the raw data in a new way
which can consider the deep relationship of input data. Luckily,
the deep learning has some unique advantages compared with
(7) on deep leaning is proposed in this paper. The most important
component of the model is the deep learning unit. This unit is

1214 2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA)
composed of the deep belief networks (DBNs). Structure of
DBNs and the Train process is shown in the next section.
III. DM ARCHITECTURE BASED ON DEEP LEARNING
A. Basic unit for deep learning network
The main assignment of deep learning is to convert raw
data to high level knowledge [25]. It is a process of encoding in
essence. The raw data in dataset is different in term of length,
capacity and format, but after being encoded, these data
become formal and easy to be handled. In order to measure the
accuracy of coding, each encoder has a corresponding decoder
while the decoder can restore the code to raw data. If the
reductive result is identical with the raw data, or there is only a
little difference, it will mean that the encoding process is
effective. Otherwise, the encoding process is probably invalid
[26]. The decoder is not a necessary component of the final
DM model, but necessary in the construction of the model. In
this paper, the sparse auto encoder is used to construct the
encoder and decoder structure. Thats to say, we attach
regularity restriction of L1 on the base of auto encoder. As
shown in Fig. 5, the code and error are defined as Eq. (9).
Fig. 5.The scheme of encoder and decoder, sparse auto encoder applied in this
process.
input : X
T graphical model or an alternative deep neural network
code : h W X
error : E (W , X ) || Wh  X ||2  O | h j | (9)
j networks such as RBMs or auto-encoders, where each sub-
As shown in Fig. 5, the sparse auto encoder makes the code
result sparse, and the sparse expression usually more effective.
This mechanism is similar as the neural activity of primate,
since most of the nerve cells in nerve system are restrained and
only a few nerve cells are at excitatory state when the external
stimulation exists.
Fig. 6. Structure of Restricted Boltzmann Machine, where the bottom layer is
visible layer and the top layer is hidden layer.
2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA)
B. DBN structure
Supposing a two-dimensional diagram with two layers,
there is no connection in the same layer, the first layer is
visible layer or input layer, and the other layer is hidden layer.
If all the nodes are random binary variables, and the probability
distribution p(v, h) subjects to Boltzmann distribution, i.e., Eq.
(10), such structure is called Restricted Boltzmann Machine
(RBM) [25], as shown in Fig 6.
p(v, h1 , h 2 ," hl ) p (v | h1 ) p(h1 | h 2 )" p (hl 1 | hl ) (10)
DBN are made from multi RBMs, and a typical net
structure of DBN is shown in Fig. 7. This network can be
classified into a visible layer and some hidden layers. There is
connection between layers, but not connection in the layer
itself. Hidden layer unit are trained to capture the relationship
of high dimension data in visible layer. In our proposed model
based on deep learning, DBN is used to construct an auto
learning network.
Fig. 7. Structure of DBNs, which is composed of a visiable layer and some
hidden layers.
In machine learning, DBNs can be viewed as a generative
composed of multiple layers [19]. From Fig. 7, DBN can be
considered as a composition of simple and unsupervised
network's hidden layer serves as the visible layer for the next.
This also leads to a fast and layer-by-layer unsupervised
training procedure.
C. Deep learning Training
Supervised learning and unsupervised learning are
integrated together in the proposed DM model based on deep
learning. In supervised learning, the input data are called
training data and each group training data have an explicit
label or corresponding result. Moreover, the supervised
learning is a process of comparing prediction result and real
data, by correcting the error manually. The learning model is
adjusted until the prediction result reaches an expected
accuracy [26]. On the contrary, in unsupervised learning, the
input data are not labeled and the learning model tries to infer
the inner connection of the input data. In fact, the data needs to
be dealt with is enormous. It is impossible to label all data, so
our aim is to build an auto learning model, when coping with
the unlabeled data. The learning model should output the
accurate prediction result. However, it is only an ideal situation,
and, many new data is hardly to be recognized in unsupervised
1215
learning. To solve this problem, we set a recognition buffer in
proposed model. Once the data cant be recognized, it will be
added into recognition buffer, and enters the model again after
being attached a result manually. As mentioned above, our DM
model architecture based on deep learning is shown in Fig. 8.
Fig. 8. Architecture of proposed DM model based on deep learing. Supervised
learning and unsupervised learning are integrated together in this model and in
particular, a recognition buffer unit is used to take user input into consideration.
IV. CONCLUSION AND FUTURE WORK
Data mining is a hot research in recent years with the
urgent demand of handling Big Data. It is not an independent
branch in AI field but a subject that relates to statics, machine
learning, expert system and many other latest techniques. So
far, there are many DM models proposed, but in some extent,
these models are not successful because they cant simulate the
thought of primate. There is still a long way to achieve human
blueprint in robotic field.
According to the investigation on state of the art data
mining technology, a novel DM model architecture based on
deep learning is proposed in this paper. Compared with the
conventional DM models, the proposed model based on deep
learning has deep analytical ability and the raw data can be
converted to knowledge more explicitly. The proposed DM
model has a broad application prospect and can be applied in
weather forecast, data trend analysis, commercial prediction
and many other businesses or industries. The implement of
proposed model will be carried out in our future work to prove
the effectiveness of the proposed model.
V. ACKNOWLEDGEMENT
This research is funded by the National High Technology
Research and Development Program of China (863 Program)
under Grant No. 2013AA01A607 and the Twelfth Five-Year
Plan for Xinjiang Manufacturing Information Technology
Demonstration Project under Grant No. 201130110-3.
REFERENCES
[1] I. Nikolova, Deep learning architecture for data mining from surgical
data, Proceedings of the 35th International Convention. IEEE, 2012:
998-1002.
1216 2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA)
[2] G. Hinton, S. Osindero S, Y. W. The, A fast learning algorithm for deep
belief nets, Neural Computation, 2006, 18(7): 1527-1554.
[3] S. Shin, Y. Guo, Y. Choi, Y. Choi, M. Choi, and C. Kim, Development
of a robust data mining method using CBFS and RSM, Perspectives of
Systems Informatics, Springer Berlin Heidelberg, 2007, 377-388.
[4] C. Poultney, S. Chopra, Y. L. Cun, Efficient learning of sparse
representations with an energy-based model, Advances in Neural
Information Processing Systems, 2006, 1137-1144.
[5] U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, From data mining to
knowledge discovery in databases, AI Magazine, 1996, 17(3), 37.
[6] Y. Bengio, P. Lamblin, D. Popovici, et al, Greedy layer-wise training of
deep networks, Advances in Neural Information Processing Systems,
2007, 19, 153.
[7] G. E. Hinton, R. R. Salakhutdinov, Reducing the dimensionality of data
with neural networks, Science, 2006, 313(5786), 504-507.
[8] J. Han, Data mining techniques, ACM SIGMOD Record, ACM, 1996,
25(2), 545.
[9] J. Weston, C. Watkins, Support vector machines for multi-class pattern
recognition, ESANN, 1999, 99, 219-224.
[10] U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, et al, Advances in
knowledge discovery and data mining, 1996.
[11] S. Zhang, C. Zhang, and Q. Yang, Data preparation for data mining,
Applied Artificial Intelligence: An International Journal, 2003, 17(5-6),
375-381.
[12] C. Cortes and V. Vapnik, Support-vector networks, Machine Learning,
1995, 20(3), 273-297.
[13] J. Wu, and E. Chen, A novel nonparametric regression ensemble for
rainfall forecasting using particle swarm optimization technique coupled
with artificial neural network, Advances in Neural Networks, Springer
Berlin Heidelberg, 2009, 49-58.
[14] B. E. Boser, I. M. Guyon, and V. N. Vapnik, A training algorithm for
optimal margin classifiers, Proceedings of the fifth annual workshop on
Computational learning theory, 1992, 144-152.
[15] A. Aizerman, E. M. Braverman, and L. I. Rozoner, Theoretical
foundations of the potential function method in pattern recognition
learning, Automation and Remote Control, 1964, 25, 821-837.
[16] C. Christopher, Encyclopaedia Britannica: definition of data mining,
Retrieved, 2010-12-09, 2010.
[17] T. Hastie, R. Tibshirani, J. Friedman J, et al, The elements of statistical
learning: data mining, inference and prediction, The Mathematical
Intelligencer, 2005, 27(2), 83-85.
[18] J. Han, M. Kamber, Data Mining, Southeast Asia Edition: Concepts and
Techniques, Morgan kaufmann, 2006.ISBN 9781558604896
[19] I. H. Witten,E. Frank, Data Mining: Practical machine learning tools and
techniques, Morgan Kaufmann, 2005. ISBN 9780123748560
[20] M. Kantardzic, Data mining: concepts, models, methods, and algorithms,
John Wiley & Sons, 2011. ISBN ISBN 0471228524
[21] K. Zhao, B. Liu, T. M. Tirpak, et al, A visual data mining framework for
convenient identification of useful knowledge, Fifth IEEE International
Conference on Data Mining, 2005, 8 pp.
[22] H. P. Kriegel, P. Krger, J. Sander, et al, Density-based clustering,
Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery, 2011, 1(3), 231-240.
[23] R. Agrawal, J. Gehrke, D. Gunopulos, et al, Automatic subspace
clustering of high dimensional data, Data Mining and Knowledge
Discovery, 2005, 11(1), 5-33.
[24] E. Achtert, C. Bhm, H. P. Kriegel, et al, Detection and visualization of
subspace cluster hierarchies, Advances in Databases: Concepts, Systems
and Applications, 2007, 152-163.
[25] D. Ciresan, U. Meier, and J. Schmidhuber, Multi-column deep neural
networks for image classification, IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2012, 3642-3649.
[26] R. Salakhutdinov and H. Larochelle, Efficient learning of deep
Boltzmann machines, International Conference on Artificial Intelligence
and Statistics, 2010, 693-700.