1

A Data Mining & Knowledge

Discovery Process Model
scar Marbn1, Gonzalo Mariscal2 and Javier Segovia1
1

Facultad de Informtica (Universidad Politcnica de Madrid)

2 Universidad Europea de Madrid
Spain

Open Access Database www.intechweb.org

1. Introduction
The number of applied in the data mining and knowledge discovery (DM & KD) projects
has increased enormously over the past few years (Jaffarian et al., 2008) (Kdnuggets.com,
2007c). As DM & KD development projects became more complex, a number of problems
emerged: continuous project planning delays, low productivity and failure to meet user
expectations. Neither all the project results are useful (Kdnuggets.com, 2008) (Eisenfeld et
al., 2003a) (Eisenfeld et al., 2003b) (Zornes, 2003), nor do all projects end successfully
(McMurchy, 2008) (Kdnuggets.com, 2008) (Strand, 2000) (Edelstein & Edelstein, 1997).
Todays failure rate is over 50% (Kdnuggets.com, 2008) (Gartner, 2005) (Gondar, 2005).
This situation is in a sense comparable to the circumstances surrounding the software
industry in the late 1960s. This was what led to the software crisis (Naur & Randell, 1969).
Software development improved considerably as a result of the new methodologies. This
solved some of its earlier problems, and little by little software development grew to be a
branch of engineering. This shift has meant that project management and quality assurance
problems are being solved. Additionally, it is helping to increase productivity and improve
software maintenance.
The history of DM & KD is not much different. In the early 1990s, when the KDD
(Knowledge Discovery in Databases) processing term was first coined (Piatetsky-Shapiro &
Frawley, 1991), there was a rush to develop DM algorithms that were capable of solving all
the problems of searching for knowledge in data. Apart from developing algorithms, tools
were also developed to simplify the application of DM algorithms. From the viewpoint of
DM & KD process models, the year 2000 marked the most important milestone: CRISP-DM
(CRoss-Industry Standard Process for DM) was published (Chapman et al., 2003). CRISPDM is the most used methodology for developing DM & KD projects. It is actually a de
facto standard.
Looking at the KDD process and how it has progressed, we find that there is some
parallelism with the advancement of software. From this viewpoint, DM project
development entails defining development methodologies to be able to cope with the new
project types, domains and applications that organizations have to come to terms with.
Nowadays, SE (software engineering) pay special attention to organizational, management
or other parallel activities not directly related to development, such as project completeness
Source: Data Mining and Knowledge Discovery in Real Life Applications, Book edited by: Julio Ponce and Adem Karahoca,
ISBN 978-3-902613-53-0, pp. 438, February 2009, I-Tech, Vienna, Austria

www.intechopen.com

Data Mining and Knowledge Discovery in Real Life Applications

and quality assurance. The most used DM & KD process models at the moment, i.e. CRISPDM, SEMMA, has not yet been sized for these tasks, as it is very much focused on pure
development activities and tasks (Marbn et al., 2008). In (Yang & Wu, 2006) one of the 10
challenging problems to be solved in DM research is considered to be the need to build a
new methodology to help users avoid many data mining mistakes.
This chapter is moved by the idea that DM & KD problems are taking on the dimensions of
engineering problems. Hence, the processes to be applied should include all the activities
and tasks required in an engineering process, tasks that CRISP-DM might not cover. The
proposal is inspired by the work done in SE derived from other branches of engineering. It
borrows ideas to establish a comprehensive process model for DM that improves and adds
to CRISP-DM. Further research will be needed to define methodologies and life cycles, but
the basis of a well-defined process model will be there.
In section 2 we describe existing DM & KD process models and methodologies, focusing on
CRISP-DM. Then, section 3 shows the most used SE process models. In section 4, we
propose a new DM & KD process model. And, finally, we discuss the conclusions about the
new approach and future work in section 5.

2. DM & KD process models

Authors tend to use the terms process model, life cycle and methodology to refer to the
same thing. This has led to some confusion in the field.
A process model is the set of tasks to be performed to develop a particular element, as well
as the elements that are produced in each task (outputs) and the elements that are necessary
to do a task (inputs) (Pressman, 2005). The goal of a process model is to make the process
repeatable, manageable and measurable (to be able to get metrics).
Methodology can be defined as the instance of a process model that lists tasks, inputs and
outputs and specifies how to do the tasks (Pressman, 2005). Tasks are performed using
techniques that stipulate how they should be done. After selecting a technique to do the
specified tasks, tools can be used to improve task performance.
Finally, the life cycle determines the order in which each activity is to be done (Moore, 1998).
A life cycle model is the description of the different ways of developing a project.
From the viewpoint of the above definitions, what do we have in the DM & KD area? Does
DM & KD have process models and/or methodologies?
2.1 Review of DM & KD process models and methodologies
In the early 1990s, when the KDD process term was first coined (Piatetsky-Shapiro &
Frawley, 1991), there was a rush to develop DM algorithms that were capable of solving all
problems of searching for knowledge in data. The KDD process (Piatetsky-Shapiro, 1994)
(Fayyad et al., 1996) has a process model component because it establishes all the steps to be
taken to develop a DM project, but it is not a methodology because its definition does not set
out how to do each of the proposed tasks. It is also a life cycle.
The 5 As (Martnez de Pisn, 2003) is a process model that proposes the tasks that should be
performed to develop a DM project and was one of CRISP-DMs forerunners. Therefore,
they share the same philosophy: 5 As proposes the tasks but does not suggest how they
should be performed. Its life cycle is similar to the one proposed in CRISP-DM.
A people-focused DM proposal is presented in (Brachman & Anand, 1996): HumanCentered Approach to Data Mining. This proposal describes the processes to be enacted to

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

carry out a DM project, considering peoples involvement in each process and taking into
account that the target user is the data engineer.
SEMMA (SAS, 2008) is the methodology that SAS proposed for developing DM products.
Although it is a methodology, it is based on the technical part of the project only. Like the
above approaches, SEMMA also sets out a waterfall life cycle, as the project is developed
right through to the end.
The two models by (Cabena et al., 1998) and (Anand & Buchner, 1998) are based on KDD
with few changes and have similar features.
Like the KDD process, Two Crows (Two Crows, 1999) is a process model and waterfall life
cycle. At no point does it set out how to do the established DM project development tasks.
CRISP-DM (Chapman et al., 2003) states which tasks have to be carried out to successfully
complete a DM project. It is therefore a process model. It is also a waterfall life cycle. CRISPDM also has a methodological component, as it gives recommendations on how to do some
tasks. Even so these recommendations are confined to proposing other tasks and give no
guidance about how to do them. Therefore, we class CRISP-DM as a process model.

Fig. 1. Evolution of DM & KD process models and methodologies

Figure 1 shows a diagram of how the different DM & KD process models and
methodologies have evolved. It is clear from Figure 1 that CRISP-DM is the standard model.
It borrowed ideas from the most important pre-2000 models and is the groundwork for
many later proposals.
The CRISP-DM 2.0 Special Interest Group (SIG) was set up with the aim of upgrading the
CRISP-DM model to a new version better suited to the changes that have taken place in the
business arena since the current version was formulated. This group is working on the new
methodology (CRISP-DM, 2008). The firms that developed CRISP-DM 1.0 have been joined

www.intechopen.com

Data Mining and Knowledge Discovery in Real Life Applications

by other institutions that intend to input their expertise in the field to develop CRISP-DM
2.0. Changes such as adding new phases, renaming existing phases and/or eliminating the
odd phase are being considered for the new version of the methodology.
Cios et al.s model was first proposed in 2000 (Cios et al., 2000). This model adapted the
CRISP-DM model to the needs of the academic research community, providing a more
general, research-oriented description of the steps.
The KDD Roadmap (Howard et al., 2001) is a DM methodology used in the DM Witness
Miner tool (Lanner Group, 2008). This methodology describes the available processes and
algorithms and incorporates experience derived from successfully completed commercial
projects. The focus is on the decisions to be made and the options available at each stage to
achieve the best results for a given task.
The RAMSYS (RApid collaborative data Mining SYStem) methodology is described in
(Moyle & Jorge, 2001) as a methodology for developing DM & KD projects where several
geographically diverse groups work remotely and collaboratively to solve the same
problem. This methodology is based on CRISP-DM and maintains the same phases and
generic tasks.
Model

Fayyad et al.

Cabena et al.

Anand &
Buchner

CRISP-DM

Cios et al.

No of
steps

Human
Developing and
Resource
Business
Business
Understanding of
Identification
Objectives
Understanding
the Application
Determination
Problem
Domain
Specification
Creating a Target
Data
Data Set
Prospecting
Data
Domain
Understanding
Data Cleaning and
Knowledge
Pre-processing
Elicitation
Data
Data Reduction
Methodology
Preparation
and Projection
Identification
Steps
Data
Choosing the DM
Preparation
Task
Data Preprocessing
Choosing the DM
Algorithm
Pattern
Modeling
DM
DM
Discovery
Domain
Knowledge
Interpreting
Knowledge
PostEvaluation
Mined Patterns
Elicitation
processing
Consolidating
Assimilation of
Deployment
Discovered
Knowledge
Knowledge

Understanding
the Data

Understanding
the Data

Preparation of
the data

DM
Evaluation of
the Discovered
Knowledge
Using the
Discovered
Knowledge

Table 1. Comparison of DM & KD process models and methodologies (Kurgan & Musilek,
2006)

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

DMIE or Data Mining for Industrial Engineering (Solarte, 2002) is a methodology because it
specifies how to do the tasks to develop a DM project in the field of industrial engineering.
It is an instance of CRISP-DM, which makes it a methodology, and it shares CRISP-DMs
associated life cycle.
Table 1 compares the phases into which the DM & KD process is decomposed according
some of the above proposals. As Table 1 shows, most of the proposals cover all the tasks in
CRISP-DM, although they do not all decompose the KD process into the same phases or
attach the same importance to the same tasks.
However, some proposals described in this section and omitted the study by (Kurgan &
Musilek, 2006), like 5 As and DMIE, propose additional phases not covered by CRISP-DM
that are potentially very useful in KD & DM projects. 5 As proposes the Automate phase.
This phase entails more than just using the model. It focuses on generating a tool to help
non-experts in the area to perform DM & KD tasks. On the other hand, DMIE proposes the
On-going support phase. It is very important to take this phase into account, as DM & KD
projects require a support and maintenance phase. This maintenance ranges from creating
and maintaining backups of the data used in the project to the regular reconstruction of DM
models. The reason is that the behavior of the DM models may change as new data emerge,
and they may not work properly. Similarly, if other tools have been used to implement the
DM models, the created programs may need maintenance, e.g. to upgrade the behavior of
the user application models.
2.2 CRISP-DM
We focus on CRISP-DM as a process model because it is the de facto standard for developing
DM & KD projects. In addition, CRISP-DM is the most used methodology for developing DM
projects (KdNuggets.com, 2002; KdNuggets.com, 2004; KdNuggets.com, 2007a).
Analyzing the problems of DM & KD projects, a group of prominent enterprises (Teradata,
SPSS ISL, Daimler-Chrysler and OHRA) developing DM projects, proposed a reference
guide to develop DM & KD projects. This guide is called CRISP-DM (CRoss Industry Standard
Process for Data Mining) (Chapman et al., 2000). CRISP-DM is vendor-independent so it can
be used with any DM tool and it can be applied to solve any DM problem.
CRISP-DM defines the phases to be carried out in a DM project. CRISP-DM also defines for
each phase the tasks and the deliverables for each task. CRISP-DM is divided into six phases
(see Figure 2). The phases are described in the following.

Business
understanding

Data
understanding
Data
preparation

Deployment

Data
Modeling

Evaluation

Fig. 2. CRISP-DM process model (Chapman et al., 2000)

www.intechopen.com

Data Mining and Knowledge Discovery in Real Life Applications

Business
Data
Data
Modeling
understanding understanding preparation
Select
Determine
Collect
initial
Select data
modeling
business
data
objectives
techniques

Evaluation
Evaluate
results

Generate test Review

design
process

Assess situation Describe data

Clean data

Determine DM
Explore data
objectives
Produce project Verify
data
plan
quality

Construct
Determine
Build model
data
next steps
Integrate
Assess model
data
Format data

Deployment
Plan
deployment
Plan
monitoring &
maintenance
Produce final
report
Review
project

Table 2. CRISP-DM phases and tasks.

Business understanding: This phase focuses on understanding the project objectives

and requirements from a business perspective, then converting this knowledge into a
DM problem definition and a preliminary plan designed to achieve the objectives.

Data understanding: The data understanding phase starts with an initial data collection
and proceeds with activities in order to get familiar with the data, to identify data
quality problems, to discover first insights into the data or to detect interesting subsets
to form hypotheses for hidden information.

Data preparation: The data preparation phase covers all the activities required to
construct the final dataset from the initial raw data. Data preparation tasks are likely to
be performed repeatedly and not in any prescribed order.

Modeling: In this phase, various modeling techniques are selected and applied and
their parameters are calibrated to optimal values. Typically, there are several techniques
for the same DM problem type. Some techniques have specific requirements on the
form of data. Therefore, it is often necessary to step back to the data preparation phase

Evaluation: What are, from a data analysis perspective, seemingly high quality models
will have been built by this stage of the project. Before proceeding to final model
deployment, it is important to evaluate the model more thoroughly and review the
steps taken to build it to be certain that it properly achieves the business objectives. At
the end of this phase, a decision should be reached on how to use of the DM results.

Deployment: Model construction is generally not the end of the project. Even if the
purpose of the model is to increase knowledge of the data, the knowledge gained will
need to be organized and presented in a way that the customer can use it.
Table 2 outlines the phases and generic tasks that CRISP-DM proposes to develop a DM
project.

3. Software engineering process models

The two most used process models in SE are the IEEE 1074 (IEEE, 1997) and ISO 12207 (ISO,
1995) standards. These models have been successfully deployed to develop software. For
this reason, our work is founded on these standards. We intend to exploit the benefits of this
experience for application to the field of DM & KD.

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

Figure 3 compares the two models. As Figure 3 shows, most of the processes proposed in
IEEE 1074 are equivalent to ISO 12207 processes and vice versa. To select processes as
optimally as possible, the IEEE 1074 and ISO 12207 processes should be mixed. The selection
criterion was to choose the processes that either IEEE 1074 or ISO 12207 defined in more
detail. We tried to not to mix processes from different groups in either process model. In
compliance with the above criteria, we selected IEEE 1074 processes as the groundwork,
because they are more detailed. As IEEE 1074 states that it is necessary to acquire or supply
software but not how to do this, we added the ISO 12207 (ISO, 1995) acquisition and supply
processes.

Fig. 3. Mapping ISO 12207 to IEEE 1074 (Marbn et al., 2008)

Figure 4 shows the joint process model developed after examining IEEE 1074 and ISO 12207
according to the above criteria. Figure 4 also shows the activities in each major process
group according to the selected standard for the group in question.
We describe the key processes selected for this joint process model below:

The acquisition and supply processes are taken from the Primary Life Cycle Processes
set out in (ISO, 1995). These processes are part of the software development process
initiation and determine the procedures and resources required to develop the project.

The software life cycle selection process (IEEE, 1997) identifies and selects a life cycle for
the software under construction.

The project management processes (IEEE, 1997) are the set of processes that establish
the project structure, and coordinate and manage project resources throughout the
software life cycle.

Development-oriented processes (IEEE, 1997) start with the identification of a need for
automation. It may take a new application or a change of all or part of an existing
application to satisfy this need. With the support of the integral process activities and
under the project management plan, the development processes produce software (code
and documentation) from the statement of the need. Finally, the activities for installing,

www.intechopen.com

Data Mining and Knowledge Discovery in Real Life Applications

operating, supporting, maintaining and retiring the software product should be

performed.

Integral processes (IEEE, 1997) are necessary to successfully complete the software
project activities. They are enacted at the same time as the software developmentoriented activities and include activities that are not related to development. They are
used to assure the completeness and quality of the project functions.
In the following section we are going to analyze which of the above activities are in CRISPDM and which are not. The aim is to build a process model for DM projects that is as
comprehensive as possible and organizes the activities systematically.
PROCESS
Acquisition
Sypply
Software life cycle selection

ACTIVITY

Identify available software life cycles

Select software life cycle
Project management activities
Initiation
Create software life cycle process
Allocate project resources
Perform estimations
Define metrics
Project monitoring and control Manage risks
Manage the project
Retaib records
Identify software life cycle process improvement needs
Collect and analyze metric data
Project planning
Plan evaluations
Plan configuration management
Plan system transition
Plan installation
Plan documentation
Plan training
Plan project management
Plan integration
Deployment oriented activities
P re-d ev elo p m en t

Concept exploration

System allocation
Software importation

Identify ideas or needs

Formulate potencial approaches
Conduct feasibility studies
Refine and finalize the idea or need
Analyze functions
Decompose system requirements
Develop system architecture
Identify imported software requirements
Evaluate software import sources
Define software import method
Import software

D evelo p m en t

Requirements

Define and develop software requirements

Define interface requirements
Priorizate and integrate software requirements

PROCESS
Design

Implementation

ACTIVITY
Perform architectural design
Design data baase
Design interface
Perform detailed design
Create executable code
Create operating documentation
Perform integration

P o st-D evelo p m en t

Installation

Operation and support

Maintenance

Retirement

Distribuye software
Install software
Accept software in operational environment
Operate the system
Provide technical asstanse and consulting
Maintain support request log
Identify software improvement needs
Implement problem reporting method
Maintenance support request log
Notify user
Conduct parallel operations
Retire system

Integral activities
Evaluation

Conduct reviews
Create traceability matrix
Conduct audits
Develop test procedures
Create test data
Execute test
Report evaluation results
Software configuration management Develop configuration identification
Perform configuration control
Perform status accounting
Documentation development
Implement documentation
Produce and distribuye documentation
Training
Develop training materials
Validate the training program
Implement the training program

Fig. 4. Joint process model

4. A data mining engineering process model

A detailed comparison of CRISP-DM with the SE process model described in section 3 is
presented in (Marbn et al, 2008). From this comparison, we found that many of the
processes defined in SE that are very important for developing any type of DM engineering
project are missing from CRISP-DM. This could be the reason why CRISP-DM is not as
effective as it should be. What we proposed there was to take CRISP-DM tasks and
processes and organize them by processes as SE researchers did. The activities missing from
CRISP-DM are primarily project management processes, integral processes (that assure
project function completeness and quality) and organizational processes (that help to
achieve a more effective organization).
Note that the correspondence between CRISP-DM and SE process model elements is not
exact. In some cases, the elements are equivalent, but the techniques are different. In other
cases, the elements have the same goal but are implemented completely differently. This
obviously depends on the project type. In SE the project aim is to develop software and in
DM & KD it is to gather knowledge from data.

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

Figure 5 shows an overview of the proposed process model, including the key processes.
The KDD process is the project development core. In the following we describe the
processes shown in Figure 5. We also explain why we think they are necessary in a DM
project.
4.1 Organizational processes
This set of processes helps to achieve a more effective organization. They also set the
organizations business goals and improve the organizations process, product and
resources. Neither the IEEE 1074 nor the ISO 12207 SE process models include these
processes. They were introduced in ISO 15504 or SPICE ISO (ISO, 2004). These processes
affect the entire organization, not just one project.
This group includes the following processes (see Figure 5):

Improvement. This activity broadcasts the best practices, methods and tools that are
available in one part of the organization to the rest of the organization.

Infrastructure. This task builds the best environment in the organization for developing
DM projects.

Training. This activity is related to training the staff participating in current or ongoing
DM projects.
No DM methodology considers any of these activities. We think that they could be adapted
from the SPICE standard because they are all general-purpose tasks common to any kind of
project.
ORGANIZATIONAL PROCESSES
Improvement

Infrastructure

PROJECT MANAGEMENT
PROCESSES

DEVELOPMENT
PROCESSES
Pre-Development processes

Life cycle selection

Concept
exploration

System
allocation

Business
modelling

Knowlege
importation

Training

INTEGRAL
PROCESS
Evaluation

Acquisition
Development processes
Requirements processes

Supply

Initiation

KDD Process
Data selection

Preprocess

Data
transformation

Data Mining
Result analysis

Configuration
management

Documentation

Post-Development processes

Project planning
Project monitoring
and control

Installation

Operation and
support
processes

Maintenance

Retirement

User training

Fig. 5. DM engineering process model (Marban et al., 2008)

4.2 Project management processes
This set of processes establishes the project structure and also how to coordinate and
manage project resources throughout the project life cycle. We define six main processes in
the project management area. Existing DM methodologies or process models (such as
CRISP-DM) take into account only a small part of project management, i.e. the project plan.

www.intechopen.com

10

Data Mining and Knowledge Discovery in Real Life Applications

The project plan is confined to defining project deadlines and milestones. All projects need
other management activities to control time, budget and resources. The project management
processes are concerned with controlling these matters.

Life cycle selection. This process defines the life cycle to be used in the DM project
(Pressman, 2005). Until now, all DM methodologies had a similar life cycle to CRISPDM: a waterfall life cycle with backtracking. However, there is a fair chance of new life
cycles being developed to meet the needs of the different projects. This is what happens
in SE from where they could be adapted.

Acquisition. The acquisition process is related to the activities and tasks of the acquirer
(who outsources work). Model building is one possible example of outsourcing. If there
is outsourcing, the acquirer must define the acquisition management, starting from the
tender and ending with the acceptance of the outsourced product. This process must be
included in DM processes because DM projects now developed at non-specialized
companies are often outsourced (KdNuggets.Com, 2007d). The acquisition process
could be an adaptation of the process proposed in the ISO 12207 standard. It defines
software development outsourcing management from requirements to software (this
depends on which part is outsourced).

Supply. The supply process concerns the activities and tasks that the supplier has to
carry out if the company acts as the developer of an outsourcing project. This process
defines the tasks the supplier has to perform to interact with the outsourcing company.
It also defines the interaction management tasks. As above, this process can be adapted
for DM & KD projects from ISO 12207.

Initiation. The initiation process establishes project structure and how to coordinate
and manage project resources throughout the project life cycle. This process could be
divided into the following activities: create DM life cycle, allocate project resources,
perform estimations, and define metrics. CRISP-DMs Business Understanding phase
partly includes these activities, but fails to suggest how to estimate costs and benefits
(Fernandez-Baizan et al., 2007). Although some DM metrics have been defined (ROI,
accuracy, space/time, usefulness) (Pai, 2004) (Shearer, 2000) (Biebler et al., 2005)
(Smith & Khotanzad, 2007), they are not being used in ongoing DM developments
(Marbn et al., 2008).

Project planning. The project planning process covers all the tasks related to planning
project management, including the contingencies plan. DM has not shown much
interest in this process. DM methodologies focus primarily on technical tasks, and they
overlook most of the project management activities. The set of activities considered in
this process are: Plan evaluation (Biffl & Winkler, 2007), Plan configuration
management (NASA, 2005), Plan system transition (JFMIP, 2001), Plan installation, Plan
documentation (Hackos, 1994), Plan training (McNamara, 2007), Plan project
management (Westland, 2006), and Plan integration. CRISP-DM does consider the Plan
installation and Plan integration activities through its Deployment phase. Although
documentation is developed in DM projects, no documentation plan is developed.
CRISP-DM states that the user should be trained, but this training is not planned. The
other plans are not considered in DM & KD methodologies.

Project monitoring and control. The project monitoring and control process covers all
tasks related to project risk and project metric management. The activities it considers
within this process are: Manage risks, Manage the Project, Retain records, Identify life

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

11

cycle process improvement needs, and Collect and analyze metrics (Fenton & Pfleeger,
1998) (Shepperd, 1996). CRISP-DM considers Manage risks in the Business
Understanding phase and Identify life cycle process improvement needs in the
Development phase. The Manage the project activity is new to CRISP-DM, which
considers project planning but not plan control. The other two activities (Retain records
and Collect and analyze metrics) are new to DM & KD methodologies.
4.3 Development processes
These are the most highly developed processes in DM. All DM methodologies focus on
these processes. This is due to the fact that development processes are more related to
technical matters. Consequently, they were developed at the same time as the techniques
were created and started to be applied. These processes are divided into three groups: predevelopment, development, and post-development processes.
The development process is the original KDD process defined in (Piatetsky-Shapiro, 1994).
The pre- and post-development processes are the ones that require a greater effort.
4.3.1 Pre-development processes
These processes are related to everything to be done before the project kicks off. From the
process model in Figure 5, we can identify the following Pre-development processes:

Concept exploration. In CRISP-DM these activities are considered in the Business

Understanding phase. However, these tasks do not completely cover the activities
because they focus primarily on the terminology and background of the problem to be
solved. In (Marbn et al., 2008) we conclude that some tasks adapted from SE standards
need to be added to optimize project development.

System allocation. CRISP-DM considers these tasks through its Business

Understanding phase

Business modeling. This is a completely new activity. The CRISP-DM business model
is described in the Business Understanding phase, but there are no business modeling
procedures or formal tools and methods as there are in SE (Gordijn et al., 2000).

Knowledge importation. This process is related to the reuse of existing knowledge or

DM models from other or previous projects, something which is very common in DM.
CRISP-DM does not consider this process at all, and its SE counterpart is related to
software and cannot be easily adapted. Consequently, the process must be created from
scratch.
4.3.2 Development processes
This is the most developed phase in DM methodologies, because it has been researched
since late 1980s. CRISP-DM phases include all these processes in one way or another. In SE
the process is divided into requirements, analysis, design, and implementation phases. We
can easily map the requirements, design and implementation phases to DM projects. The
design and implementation phases match the KDD process, and we will stick with this
process and its name.

Requirements processes. CRISP-DM covers this set of processes, but they are
incomplete. The requirements are developed in CRISP-DMs Business Understanding
phase. In CRISP-DM this process produces a list of requirements, but the CRISP-DM
user guide does not specify or describe any procedure or any formal notation, tool or

www.intechopen.com

12

Data Mining and Knowledge Discovery in Real Life Applications

technique to obtain the requirements from the business models. Neither does it specify
or describe how to translate requirements into DM goals and models for proper use in
the subsequent design and implementation phases: the KDD process. We believe that
requirements can be described formally like they are in SE (Kotonya, G. & Sommerville,
1998). For example, something like use-case models could be adapted to specify the
project requirements. Further work and research, possibly inspired by SE best practices,
should be put into developing a core of formal methods and tools adapted to this
process in the DM area.
KDD process. The KDD process matches the design and implementation phases of a
software development project. This set of processes is responsible for acquiring the
knowledge for the DM project. KDD includes the following activities: data selection,
pre-processing, data transformation, DM, and result analysis. CRISP-DM covers the
KDD process (Marbn et al., 2008).

4.3.3 Post-development processes

Post-development processes are the processes that are carried out after the knowledge is
gathered. They are applicable during the later life cycle stages.

Installation. This process is commended with transferring the knowledge extracted

from the DM results to the users. The knowledge can be used as it is, i.e. to help
managers to make decisions about a future marketing campaign, or could involve some
software development, i.e. to improve an existing web-based recommender system.
CRISP-DM considers planning for deploying the knowledge at the client site, but it
does not regard the development of software installed and accepted in an operational
environment as part of this deployment (Reifer, 2006).

Operation and support process. This process is necessary to validate the results and how
they are interpreted by the client, and, if software is developed, to provide the client with
technical assistance. CRISP-DM only includes results monitoring. In addition, we propose
tasks to validate the results (this is a new task) and to provide technical assistance if
necessary (this task could be directly incorporated from IEEE 1074).

Maintenance. The maintenance process has two different paths. On the one hand, if
knowledge is embedded in software, this process will provide feedback information to
the software life cycle and lead to changes in the software. For this path, the task can be
adapted from the IEEE 1074 maintenance process. On the other hand, CRISP-DM does
not include a task for knowledge used as it is, and this needs to be developed from
scratch.

Retirement. The knowledge gathered from data is not valid forever, and this task is in
charge of retiring obsolete knowledge from the system. CRISP-DM does not cover this
process, but it can be adapted from IEEE 1074.
4.4 Integral processes
Integral processes are necessary to successfully complete the project activities. These
processes assure project function completeness and quality. They are carried out together
with development processes to assure the quality of development deliverables. The integral
processes group the four processes described below.

Evaluation. This process is used to discover defects in the product or in the process
used to develop the DM project. CRISP-DM covers the evaluation process through

www.intechopen.com

A Data Mining & Knowledge Discovery Process Model

13

evaluation activities spread across different phases: Evaluation, Deployment, Business

Understanding and Modeling. But we think the organization of the SE process is more
appropriate and covers more aspects.
Configuration management. This process is designed to control system changes and
maintain system coherence and traceability. The ultimate aim is to be able to audit the
evolution of configurations (Buckley, 1992). We consider this to be a key process in a
DM project because of the amount of information and models generated throughout the
project. Surprisingly, DM methodologies do not account for this process at all.
Documentation. This process is related to designing, implementing, editing, producing,
distributing and maintaining the project documentation. CRISP-DM considers this
process across different phases: Deployment, Deployment, Modeling, and Evaluation.
User training. Current DM methodologies do not consider user training at all. This
process is related to training inexperienced users to use and interpret the results of the
DM project.

5. Conclusions and future development

After analyzing the SE process models, we have developed a joint model based on two
standards to compare, process by process and activity by activity, the modus operandi in SE
and DM & KD. This comparison revealed that CRISP-DM does not cover many project
management-, organization- and quality-related tasks at all or at least thoroughly enough.
This is now a must due to the complexity of the projects being developed in DM & KD these
days. These projects not only involve examining huge volumes of data but also managing
and organizing big interdisciplinary human teams.
Consequently, we proposed a DM engineering process model that covers the above points.
To do this, we made a distinction between process model, and methodology and life cycle.
The proposed process model includes all the activities covered by CRISP-DM, but
distributed across process groups that conform to engineering standards established by a
field with over 40 years experience, i.e. software engineering.
The model is not complete, as the need for the processes, tasks and/or activities set out in
IEEE 1074 or ISO 12207 and not covered by CRISP-DM has been stated but they have yet to
be adapted and specified in detail.
Additionally, this general outline needs to be further researched. First, the elements that
CRISP-DM has been found not to cover at all or only in part would have to be specified and
adapted from their SE counterpart. Second, the possible life cycle for DM would have to be
examined and specified. Third, the process model specifies that what to do but not how to
do it. A methodology is what specifies the how to part. Therefore, the different
methodologies that are being used for each process would need to be examined and adapted
to the model. Finally, a methodology is associated with a series of tools and techniques. DM
has already developed many such tools (like Clementine or the neural network techniques),
but tools that are well-established in SE (e.g. configuration management techniques) are
missing. It remains to be seen how they can be adapted to DM and KD processes.

6. References
Anand, S.; Patrick, A.; Hughes, J. & Bell, D. (1998). A data mining methodology for crosssales. Knowledge Based Systems Journal. 10, 449461.

www.intechopen.com

KDD, SEMMA AND CRISP-DM: A PARALLEL OVERVIEW

Author Name *
Affiliation *
Address *
E-mail *

Author Name *
Affiliation *
Address *
E-mail *

ABSTRACT
In the last years there has been a huge growth and consolidation of the Data Mining field. Some efforts are being done
that seek the establishment of standards in the area. Included on these efforts there can be enumerated SEMMA and
CRISP-DM. Both grow as industrial standards and define a set of sequential steps that pretends to guide the
implementation of data mining applications. The question of the existence of substantial differences between them and
the traditional KDD process arose. In this paper, is pretended to establish a parallel between these and the KDD process
as well as an understanding of the similarities between them.
KEYWORDS
Data Mining Standards, Knowledge Discovery in Databases, Data Mining.

1. INTRODUCTION
Fayyad considers Data Mining (DM) as one of the phases of the KDD1 process (Fayyad et al., 1996). The
data mining phase concerns, mainly, to the means by which the patterns are extracted and enumerated from
data. The literature is a source of some confusion because de two terms are indistinctively used, making it
difficult to determine exactly each of the concepts (Benot, 2002). In this paper, data mining is seen as one of
the phases of the KDD process, as presented in (Fayyad et al., 1996) and in (Brachman & Anand, 1996).
The growth of the attention paid to the area emerged from the rising of big databases in an increasing and
differentiate number of organizations. There is the risk of wasting all the value and wealthy of information
contained on these databases unless there are used the adequate techniques to extract useful knowledge (Chen
et al, 1996) (Simoudis, 1996) (Fayyad, 1996).
In the latest years, it has been occurring the growth and consolidation of the data mining area. Some
efforts are being done that seek the establishment of standards in the area, both by academics and by people
in the industry field. The academics efforts are centered in the attempt to formulate a general framework for
DM (Dzeroski, 2006). The bulk of these efforts are centered in the definition of a language for DM that can
be accepted as a standard, in the same way that SQL was accepted as a standard for relational databases (Han
et al, 1996) (Meo et al, 1998) (Imielinski et al, 1999) (Sarawagi, 2000) (Botta et al, 2004). The efforts in the
industrial field concern mainly the definition of processes/methodologies that can guide the implementation
of DM applications. In this paper, SEMMA and CRISP-DM have been chosen, because they are considered
to be the most popular. Although it is not scientific this perception exists, because SEMMA and CRISP-DM
are presented in many of the publications of the area and are really used in practice.
During the analysis of the documentation on SEMMA and on CRISP-DM, the question of the existence
of substantial differences between them and the traditional KDD process arose. In this paper, it is pretended

1 Knowledge Discovery in Databases

to establish a parallel between these and the KDD process as well as an understanding of the similarities
between them.
The paper begins, on section 2, by presenting KDD, SEMMA and CRISP-DM. Next, on section 3, a
comparative study is done, presenting the analogies and the differences between the three processes. Finally,
on section 4, conclusions and future work are presented.

2. KDD, SEMMA AND CRISP-DM DESCRIPTION

The term knowledge discovery in databases or KDD, for short, was coined in 1989 to refer to the broad
process of finding knowledge in data, and to emphasize the high-level application of particular data mining
methods (Fayyad et al, 1996). Fayyad considers DM as one of the phases of the KDD process and considers
that the data mining phase concerns, mainly, to the means by which the patterns are extracted and
enumerated from data. In this paper there is a concern with the overall KDD process, which will be described
in section 2.1.
SEMMA was developed by the SAS Institute. CRISP-DM was developed by the means of the efforts of a
consortium initially composed with DaimlerChryrler, SPSS and NCR. They will be described in sections 2.2
and 2.3, respectively. Despite that SEMMA and CRISP-DM are usually referred as methodologies, in this
paper they are referred as processes, in the sense that they consist of a particular course of action intended to
achieve a result.

2.1 The KDD process

The KDD process, as presented in (Fayyad et al, 1996) is the process of using DM methods to extract
what is deemed knowledge according to the specification of measures and thresholds, using a database along
with any required preprocessing, sub sampling, and transformation of the database. There are considered five
stages, presented in figure 1:
1. Selection This stage consists on creating a target data set, or focusing on a subset of variables
or data samples, on which discovery is to be performed.
2. Pre processing This stage consists on the target data cleaning and pre processing in order to
obtain consistent data.
3. Transformation This stage consists on the transformation of the data using dimensionality
reduction or transformation methods.
4. Data Mining This stage consists on the searching for patterns of interest in a particular
representational form, depending on the data mining objective (usually, prediction)
5. Interpretation/Evaluation This stage consists on the interpretation and evaluation of the mined
patterns.
Figure 1. The five stages of KDD

The KDD process is interactive and iterative, involving numerous steps with many decisions being made
by the user. (Brachman, Anand, 1996).
Additionally, the KDD process must be preceded by the development of an understanding of the
application domain, the relevant prior knowledge and the goals of the end-user. It also must be continued by
the knowledge consolidation by incorporating this knowledge into the system (Fayyad et al, 1996).

2.2 The SEMMA process

The SEMMA process was developed by the SAS Institute. The acronym SEMMA stands for Sample,
Explore, Modify, Model, Assess, and refers to the process of conducting a data mining project. The SAS
Institute considers a cycle with 5 stages for the process:
1. Sample This stage consists on sampling the data by extracting a portion of a large data set big
enough to contain the significant information, yet small enough to manipulate quickly. This
stage is pointed out as being optional.
2. Explore This stage consists on the exploration of the data by searching for unanticipated trends
and anomalies in order to gain understanding and ideas.
3. Modify This stage consists on the modification of the data by creating, selecting, and
transforming the variables to focus the model selection process.
4. Model This stage consists on modeling the data by allowing the software to search
automatically for a combination of data that reliably predicts a desired outcome.
5. Assess This stage consists on assessing the data by evaluating the usefulness and reliability of
the findings from the data mining process and estimate how well it performs.
Although the SEMMA process is independent from de DM chosen tool, it is linked to the SAS Enterprise
Miner software and pretends to guide the user on the implementations of DM applications.
SEMMA offers an easy to understand process, allowing an organized and adequate development and
maintenance of DM projects. It thus confers a structure for his conception, creation and evolution, helping to
present solutions to business problems as well as to find de DM business goals. (Santos & Azevedo, 2005)

2.3 The CRISP-DM process

The CRISP-DM process was developed by the means of the effort of a consortium initially composed
with DaimlerChryrler, SPSS and NCR. CRISP-DM stands for CRoss-Industry Standard Process for Data
Mining. It consists on a cycle that comprises six stages (figure 2):
1. Business understanding This initial phase focuses on understanding the project objectives and
requirements from a business perspective, then converting this knowledge into a data mining
problem definition and a preliminary plan designed to achieve the objectives.
2. Data understanding The data understanding phase starts with an initial data collection and
proceeds with activities in order to get familiar with the data, to identify data quality problems,
to discover first insights into the data or to detect interesting subsets to form hypotheses for
hidden information.
3. Data preparation The data preparation phase covers all activities to construct the final dataset
from the initial raw data.
4. Modeling In this phase, various modeling techniques are selected and applied and their
parameters are calibrated to optimal values.
5. Evaluation At this stage the model (or models) obtained are more thoroughly evaluated and the
steps executed to construct the model are reviewed to be certain it properly achieves the business
objectives.
6. Deployment Creation of the model is generally not the end of the project. Even if the purpose
of the model is to increase knowledge of the data, the knowledge gained will need to be
organized and presented in a way that the customer can use it.
(Chapman et al, 2000)

Figure 2 The CRISP-DM life cycle

(Chapman et al, 2000)

The sequence of the six stages is not rigid, as is schematize in figure 2. CRISP-DM is extremely complete
and documented. All his stages are duly organized, structured and defined, allowing that a project could be
easily understood or revised (Santos & Azevedo, 2005). Although the CRISP-DM process is independent
from de DM chosen tool, it is linked to the SPSS Clementine software

3. A COMPARATIVE STUDY
By doing a comparison of the KDD and SEMMA stages we would, on a first approach, affirm that they
are equivalent:
Sample can be identified with Selection,
Explore can be identified with Pre processing
Modify can be identified with Transformation
Model can be identified with Data Mining
Assess can be identified with Interpretation/Evaluation.
Examining it thoroughly, we may affirm that the five stages of the SEMMA process can be seen as a
practical implementation of the five stages of the KDD process, since it is directly linked to the SAS
Enterprise Miner software.
Comparing the KDD stages with the CRISP-DM stages is not as straightforward as in the SEMMA
situation. Nevertheless, we can first of all observe that the CRISP-DM methodology incorporates the steps
that, as referred above, must precede and follow the KDD process that is to say:
The Business Understanding phase can be identified with the development of an understanding
of the application domain, the relevant prior knowledge and the goals of the end-user
The Deployment phase can be indentified with the consolidation by incorporating this
knowledge into the system.
Concerning the remaining stages, we can say that:
The Data Understanding phase can be identified as the combination of Selection and Pre
processing
The Data Preparation phase can be identified with Transformation
The Modeling phase can be identified with Data Mining
The Evaluation phase can be identified with Interpretation/Evaluation.
In table 1, we present a summary of the presented correspondences.

Table 1. Summary of the correspondences between KDD, SEMMA and CRISP-DM

KDD
Pre KDD
Selection
Pre processing
Transformation
Data mining
Interpretation/Evaluation
Post KDD

SEMMA
------------Sample
Explore
Modify
Model
Assessment
-------------

CRISP-DM
Business understanding
Data Understanding
Data preparation
Modeling
Evaluation
Deployment

4. CONCLUSIONS AND FUTURE WORK

Considering the presented analysis we conclude that SEMMA and CRISP-DM can be viewed as an
implementation of the KDD process described by (Fayyad et al, 1996). At first sight, we can get to the
conclusion that CRISP-DM is more complete than SEMMA. However, analyzing it deeper, we can integrate
the development of an understanding of the application domain, the relevant prior knowledge and the goals
of the end-user, on the Sample stage of SEMMA, because the data can not be sampled unless there exists a
truly understanding of all the presented aspects. With respect to the consolidation by incorporating this
knowledge into the system, we can assume that it is present, because it is truly the reason for doing it. This
leads to the fact that standards have been achieved, concerning the overall process: SEMMA and CRISP-DM
do guide people to know how DM can be applied in practice in real systems.
In the future we pretend to analyze other aspects related to DM standards, namely SQL-based languages
for DM, as well as XML-based languages for DM. As a complement, we pretend to investigate the existence
of other standards for DM.

REFERENCES
Fayyad, U. M. et al. 1996. From data mining to knowledge discovery: an overview. In Fayyad, U. M.et al (Eds.),
Advances in knowledge discovery and data mining. AAAI Press / The MIT Press.
Benot, G., 2002. Data Mining. Annual Review of Information Science and Technology, Vol. 36, No. 1, pp 265-310.
Brachman, R. J. & Anand, T., 1996. The process of knowledge discovery in databases. In Fayyad, U. M. et al. (Eds.),
Advances in knowledge discovery and data mining. AAAI Press / The MIT Press.
Chen, M. et al, 1996. Data Mining: An Overview from a Database Perspective. IEEE Transactions on Knowledge and
Data Engineering, Vol. 8, No. 6, pp 866-883.
Simoudis, E., 1996. Reality check for data mining. IEEE Expert, Vol. 11, No. 5, pp 26-33.
Fayyad, U. M., 1996. Data mining and knowledge discovery: making sense out of data. IEEE Expert, Vol. 11 No. 5, pp
20-25.
Dzeroski, S., 2006. Towards a General Framework for Data Mining.. In Dzeroski, S and Struyf, J (Eds.), Knowledge
Discovery in Inductive Databases. LNCS 47474. Springer-Verlag.
Han, J. et al, 1996. DMQL: A Data Mining Query Language for Relational Databases. In proceedings of DMKD-96
(SIGMOD-96 Workshop on KDD). Montreal. Canada.

ISBN: 978-972-8924-63-8 2008 IADIS

KDD, SEMMA AND CRISP-DM: A PARALLEL OVERVIEW

Ana Azevedo

CEISE ISCAP IPP

Rua Jaime Lopes de Amorim, s/n 4465 S. M. de Infesta - Portugal

Manuel Filipe Santos

DSI - UM
Campus de Azurm 4800-058 Guimares

ABSTRACT
In the last years there has been a huge growth and consolidation of the Data Mining field. Some efforts are being done
that seek the establishment of standards in the area. Included on these efforts there can be enumerated SEMMA and
CRISP-DM. Both grow as industrial standards and define a set of sequential steps that pretends to guide the
implementation of data mining applications. The question of the existence of substantial differences between them and
the traditional KDD process arose. In this paper, is pretended to establish a parallel between these and the KDD process
as well as an understanding of the similarities between them.
KEYWORDS
Data Mining Standards, Knowledge Discovery in Databases, Data Mining.

1. INTRODUCTION
Fayyad considers Data Mining (DM) as one of the phases of the KDD process (Fayyad et al., 1996). The DM
phase concerns, mainly, to the means by which the patterns are extracted and enumerated from data. The
literature is a source of some confusion because de two terms are indistinctively used, making it difficult to
determine exactly each of the concepts (Benot, 2002). The growth of the attention paid to the area emerged
from the rising of big databases in an increasing and differentiate number of organizations. There is the risk
of wasting all the value and wealthy of information contained on these databases, unless there are used the
adequate techniques to extract useful knowledge (Chen et al, 1996) (Simoudis, 1996) (Fayyad, 1996). Some
efforts are being done that seek the establishment of standards in the area, both by academics and by people
in the industry field. The academics efforts are centered in the attempt to formulate a general framework for
DM (Dzeroski, 2006). The bulk of these efforts are centered in the definition of a language for DM that can
be accepted as a standard, in the same way that SQL was accepted as a standard for relational databases (Han
et al, 1996) (Meo et al, 1998) (Imielinski et al, 1999) (Sarawagi, 2000) (Botta et al, 2004). The efforts in the
industrial field concern mainly the definition of processes/methodologies that can guide the implementation
of DM applications. In this paper, SEMMA and CRISP-DM have been chosen, because they are considered
to be the most popular. Although it is not scientific this perception exists, because SEMMA and CRISP-DM
are presented in many of the publications of the area and are really used in practice. During the analysis of
the documentation on SEMMA and on CRISP-DM, the question of the existence of substantial differences
between them and the traditional KDD process arose. In this paper, it is intended to establish a parallel
between these and the KDD process as well as an understanding of the similarities between them. The paper
begins, on section 2, by presenting KDD, SEMMA and CRISP-DM. Next, on section 3, a comparative study
is done, presenting the analogies and the differences between the three processes. Finally, on section 4,
conclusions and future work are presented.

182

IADIS European Conference Data Mining 2008

2. KDD, SEMMA AND CRISP-DM DESCRIPTION

The term knowledge discovery in databases or KDD, for short, was coined in 1989 to refer to the broad
process of finding knowledge in data, and to emphasize the high-level application of particular DM
methods (Fayyad et al, 1996). In this paper there is a concern with the overall KDD process. SEMMA was
developed by the SAS Institute. CRISP-DM was developed by the means of the efforts of a consortium
initially composed with Daimler Chrysler, SPSS and NCR. Despite SEMMA and CRISP-DM are usually
referred as methodologies, in this paper they are referred as processes, in the sense that they consist of a
particular course of action intended to achieve a result.

2.1 The KDD Process

KDD process, as presented in (Fayyad et al, 1996), is the process of using DM methods to extract what is
deemed knowledge according to the specification of measures and thresholds, using a database along with
any required preprocessing, sub sampling, and transformation of the database. There are considered five
stages, presented in figure 1: Selection - this stage consists on creating a target data set, or focusing on a
subset of variables or data samples, on which discovery is to be performed; Pre-processing - this stage
consists on the target data cleaning and pre processing in order to obtain consistent data; Transformation this stage consists on the transformation of the data using dimensionality reduction or transformation
methods; Data Mining - this stage consists on the searching for patterns of interest in a particular
representational form, depending on the DM objective (usually, prediction); Interpretation/Evaluation this stage consists on the interpretation and evaluation of the mined patterns.

Figure 1. The five stages of KDD

The KDD process is interactive and iterative, involving numerous steps with many decisions being made
by the user (Brachman, Anand, 1996). The KDD process is preceded by the development of an understanding
of the application domain, the relevant prior knowledge and the goals of the end-user. It must be continued
by the knowledge consolidation, incorporating this knowledge into the system (Fayyad et al, 1996).

2.2 The SEMMA Process

The acronym SEMMA stands for Sample, Explore, Modify, Model, Assess, and refers to the process of
conducting a DM project. The SAS Institute considers a cycle with 5 stages for the process: Sample - this
stage consists on sampling the data by extracting a portion of a large data set big enough to contain the
significant information, yet small enough to manipulate quickly; Explore - this stage consists on the
exploration of the data by searching for unanticipated trends and anomalies in order to gain understanding
and ideas; Modify - this stage consists on the modification of the data by creating, selecting, and
transforming the variables to focus the model selection process; Model - this stage consists on modeling the
data by allowing the software to search automatically for a combination of data that reliably predicts a
desired outcome; Assess - this stage consists on assessing the data by evaluating the usefulness and reliability
of the findings from the DM process and estimate how well it performs. The SEMMA process offers an easy
to understand process, allowing an organized and adequate development and maintenance of DM projects. It
thus confers a structure for his conception, creation and evolution, helping to present solutions to business
problems as well as to find de DM business goals. (Santos & Azevedo, 2005)

183

ISBN: 978-972-8924-63-8 2008 IADIS

2.3 The CRISP-DM Process

CRISP-DM stands for CRoss-Industry Standard Process for Data Mining. It consists on a cycle that
comprises six stages (figure 2): Business understanding-this initial phase focuses on understanding the
project objectives and requirements from a business perspective, then converting this knowledge into a DM
problem definition and a preliminary plan designed to achieve the objectives; Data understanding-the data
understanding phase starts with an initial data collection and proceeds with activities in order to get familiar
with the data, to identify data quality problems, to discover first insights into the data or to detect interesting
subsets to form hypotheses for hidden information; Data preparation-the data preparation phase covers all
activities to construct the final dataset from the initial raw data; Modeling-in this phase, various modeling
techniques are selected and applied and their parameters are calibrated to optimal values; Evaluation-at this
stage the model (or models) obtained are more thoroughly evaluated and the steps executed to construct the
model are reviewed to be certain it properly achieves the business objectives; Deployment-creation of the
model is generally not the end of the project. Even if the purpose of the model is to increase knowledge of the
data, the knowledge gained will need to be organized and presented in a way that the customer can use it.
(Chapman et al, 2000)

Figure 2. The CRISP-DM life cycle

CRISP-DM is extremely complete and documented. All his stages are duly organized, structured and
defined, allowing that a project could be easily understood or revised (Santos & Azevedo, 2005).

3. A COMPARATIVE STUDY
By doing a comparison of the KDD and SEMMA stages we would, on a first approach, affirm that they are
equivalent: Sample can be identified with Selection; Explore can be identified with Pre processing; Modify
can be identified with Transformation; Model can be identified with DM; Assess can be identified with
Interpretation/Evaluation. Examining it thoroughly, we may affirm that the five stages of the SEMMA
process can be seen as a practical implementation of the five stages of the KDD process, since it is directly
linked to the SAS Enterprise Miner software. Comparing the KDD stages with the CRISP-DM stages is not
as straightforward as in the SEMMA situation. Nevertheless, we can first of all observe that the CRISP-DM
methodology incorporates the steps that, as referred above, must precede and follow the KDD process that is
to say: The Business Understanding phase can be identified with the development of an understanding of the
application domain, the relevant prior knowledge and the goals of the end-user; The Deployment phase can
be indentified with the consolidation by incorporating this knowledge into the system. Concerning the
remaining stages, we can say that: The Data Understanding phase can be identified as the combination of
Selection and Pre processing; The Data Preparation phase can be identified with Transformation; The
Modeling phase can be identified with DM; The Evaluation phase can be identified with
Interpretation/Evaluation. In table 1, we present a summary of the correspondences.

184

IADIS European Conference Data Mining 2008

Table 1. Summary of the correspondences between KDD, SEMMA and CRISP-DM

KDD
Pre KDD
Selection
Pre processing
Transformation
Data mining
Interpretation/Evaluation
Post KDD

SEMMA
------------Sample
Explore
Modify
Model
Assessment
-------------

CRISP-DM
Business understanding
Data Understanding
Data preparation
Modeling
Evaluation
Deployment

4. CONCLUSIONS AND FUTURE WORK

Considering the presented analysis we conclude that SEMMA and CRISP-DM can be viewed as an
implementation of the KDD process described by (Fayyad et al, 1996). At first sight, we can get to the
conclusion that CRISP-DM is more complete than SEMMA. However, analyzing it deeper, we can integrate
the development of an understanding of the application domain, the relevant prior knowledge and the goals
of the end-user, on the Sample stage of SEMMA, because the data can not be sampled unless there exists a
truly understanding of all the presented aspects. With respect to the consolidation by incorporating this
knowledge into the system, we can assume that it is present, because it is truly the reason for doing it. This
leads to the fact that standards have been achieved, concerning the overall process: SEMMA and CRISP-DM
do guide people to know how DM can be applied in practice in real systems. In the future we pretend to
analyze other aspects related to DM standards, namely SQL-based languages for DM, as well as XML-based
languages for DM. As a complement, we pretend to investigate the existence of other standards for DM.

REFERENCES
Fayyad, U. M. et al. 1996. From data mining to knowledge discovery: an overview. In Fayyad, U. M.et al (Eds.),
Advances in knowledge discovery and data mining. AAAI Press / The MIT Press.
Benot, G., 2002. Data Mining. Annual Review of Information Science and Technology, Vol. 36, No. 1, pp 265-310.
Brachman, R. J. & Anand, T., 1996. The process of knowledge discovery in databases. In Fayyad, U. M. et al. (Eds.),
Advances in knowledge discovery and data mining. AAAI Press / The MIT Press.
Chen, M. et al, 1996. Data Mining: An Overview from a Database Perspective. IEEE Transactions on Knowledge and
Data Engineering, Vol. 8, No. 6, pp 866-883.
Simoudis, E., 1996. Reality check for data mining. IEEE Expert, Vol. 11, No. 5, pp 26-33.
Fayyad, U. M., 1996. Data mining and knowledge discovery: making sense out of data. IEEE Expert, Vol. 11 No. 5, pp
20-25.
Dzeroski, S., 2006. Towards a General Framework for Data Mining.. In Dzeroski, S and Struyf, J (Eds.), Knowledge
Discovery in Inductive Databases. LNCS 47474. Springer-Verlag.
Han, J. et al, 1996. DMQL: A Data Mining Query Language for Relational Databases. In proceedings of DMKD-96
(SIGMOD-96 Workshop on KDD). Montreal. Canada.
Meo, R. e tal, 1998. An Extension to SQL for Mining Association Rules. Data Mining and Knowledge Discovery Vol. 2,
pp 195-224. Kluwer Academic Publishers.
Imielinski, T.; Virmani, A., 1999. MSQL: A Query Language for Database Mining. Data Mining and Knowledge
Discovery Vol. 3, pp 373-408. Kluwer Academic Publishers.
Sarawagi, S. et al, 2000. Integrating Association Rule Mining with Relational Database Systems: Alternatives and
Implications. Data Mining and Knowledge Discovery, Vol. 4, pp 89125.
Botta, Marco, et al, 2004. Query Languages Supporting Descriptive Rule Mining: A Comparative Study. Database
Support for Data Mining Applications. LNAI 2682, pp 24-51.
SAS Enterprise Miner SEMMA. SAS Institute.
Accessed from http://www.sas.com/technologies/analytics/datamining/miner/semma.html, on May 2008
Santos, M & Azevedo, C (2005). Data Mining Descoberta de Conhecimento em Bases de Dados. FCA Publisher.
Chapman, P. et al, 2000. CRISP-DM 1.0 - Step-by-step data mining guide.
Accessed from http://www.crisp-dm.org/CRISPWP-0800.pdf on May 2008

185

International Journal of Innovation and Scientific Research

ISSN 2351-8014 Vol. 12 No. 1 Nov. 2014, pp. 217-222
2014 Innovative Space of Scientific Research Journals
http://www.ijisr.issr-journals.org/

A Comparative Study of Data Mining Process Models (KDD, CRISP-DM and SEMMA)
Umair Shafique and Haseeb Qaiser
Department of Information Technology,
University of Gujrat,
Gujrat, Pakistan

Copyright 2014 ISSR Journals. This is an open access article distributed under the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

ABSTRACT: Data Mining is about analyzing the huge amount data and extracting of information from it for different purposes.
From the last few years the field of Data Mining becomes prominent and makes huge growth. There are different standard
models for data mining. All these models are defined in sequential steps. These steps help in implementing the data mining
tasks. In this paper we will compare these models and give brief understanding about them.
KEYWORDS: Knowledge, Applications, Patterns, Modeling, Information.
1

INTRODUCTION

Over the last few years the field of data mining [1] becomes very important for different industries, co-corporations and
businesses etc because of its ability to use huge amount of data that had previously no use and makes analysis and predicting
trend and patterns. Basically the risk of wasting the wealthy and valuable information contained by the big databases was
arise and this requires the use of adequate techniques to get useful knowledge [2] so that the field of data mining had been
emerged in 1980s and is still making progress. With the emergence of this field different process models were introduced.
These process models guides and carry the data mining tasks and its applications.
Efforts were made to use data mining process models that may guide the implementation of data mining on big or huge
amount of data. In our paper we mainly focuses on three most popular data mining process models and these models are
Knowledge Discovery Databases (KDD) process model, CRISP-DM and SEMMA. These three models are mostly practiced by
the data mining experts and researchers. In this paper we will elaborate these models. We will do comparative study of these
models and try to explain that what insight of them is.

OVERVIEW OF KDD, CRISP-DM AND SEMMA

The Knowledge Discovery Databases (KDD) model is an iterative and interactive model [3]. It has total nine steps. It refers
to finding knowledge in data and emphasizes the high level of specific data mining method.
Cross-Industry Standard Process for Data Mining (CRISP-DM) [4] was launched in late 1996 by Daimler Chrysler (then
Daimler-Benz), SPSS (then ISL) and NCR. This models the refines over the years. It has six steps or phases.
Sample, Explore, Modify, Model, Assess (SEMMA) [5] model was developed by SAS institute. It has five different phases.
2.1

THE KDD PROCESS MODEL

The KDD or Knowledge Discovery Databases [6] is the process of extracting the hidden knowledge according from
databases. KDD requires relevant prior knowledge and brief understanding of application domain and goals. KDD process
model is iterative and interactive in nature. There are nine different steps or stages of this model and these are given below.

Corresponding Author: Umair Shafique

217

A Comparative Study of Data Mining Process Models (KDD, CRISP-DM and SEMMA)

2.1.1

DEVELOPING AND UNDERSTANDING OF THE APPLICATION DOMAIN

This is the first stage of KDD process in which goals are defined from customers view point and used to develop and
understanding about application domain and its prior knowledge.
2.1.2

CREATING A TARGET DATA SET

This is the second stage of KDD process which focuses creating on target data set and subset of data samples or variables.
It is an important stage because knowledge discovery is performed on all these.
2.1.3

DATA CLEANING AND PRE-PROCESSING

This is the third stage of KDD process which focuses on target data cleaning and pre-processing to complete and
consistent data without any noise and inconsistencies. In this stage strategies are develop for handling such type of noisy and
inconsistent data.
2.1.4

DATA TRANSFORMATION

This is the fourth stage of KDD process which focuses on transformation of data from one form to another so that data
mining algorithms can be implemented easily. For this purpose different data reduction and transformation methods are
implemented on target data.

Fig. 1.

2.1.5

Knowledge Discovery Databases (KDD) Process Model

CHOOSING THE SUITABLE DATA MINING TASK

This is the fifth stage of KDD process in which appropriate data mining task is chosen based on particular goals that are
defined in first stage. The examples of data mining method or tasks are classification, clustering, regression and
summarization etc.
2.1.6

CHOOSING THE SUITABLE DATA MINING ALGORITHM

This is the sixth step of KDD process in which in which one or more appropriate data mining algorithms are selected for
searching different patterns from data. There are number of algorithms present today for data mining but appropriate
algorithms are selected based on matching the overall criteria for data mining.
2.1.7

EMPLOYING DATA MINING ALGORITHM

This is the seventh step of KDD process in which selected algorithms are implemented.

ISSN : 2351-8014

Vol. 12 No. 1, Nov. 2014

218

Umair Shafique and Haseeb Qaiser

2.1.8

INTERPRETING MINED PATTERNS

This is the eighth step of KDD process that focuses on interpretation and evaluate of mining patterns. This step may
involve in extracted patterns visualization.
2.1.9

USING DISCOVERED KNOWLEDGE

This is the last and final step of KDD process in which the discovered knowledge is used for different purposes. The
discovered knowledge can also be used interested parties or can be integrate with another system for further action.
2.2

THE CRISP-DM PROCESS MODEL

CRoss-Industry Standard Process for Data Mining (CRISP-DM) was developed by Daimler Chrysler (then Daimler-Benz),
SPSS (then ISL) and NCR in 1999, CRISP-DM 1.0 version was published and is complete and documented. It provides a uniform
framework and guidelines for data miners. It consists of six phases or stages which are well structured and defined [7]. These
phases are described below.
2.2.1

BUSINESS UNDERSTANDING

This is the first phase of CRISP-DM process which focuses on and uncovers important factors including success criteria,
business and data mining objectives and requirements as well as business terminologies and technical terms.
2.2.2

DATA UNDERSTANDING

This is the second phase of CRISP-DM process which focuses on data collection, checking quality and exploring of data to
get insight of data to form hypotheses for hidden information.
2.2.3

DATA PREPARATION

This is the third phase of CRISP-DM process which focuses on selection and preparation of final data set. This phase may
include many tasks records, table and attributes selection as well as cleaning and transformation of data.

Fig. 2.

ISSN : 2351-8014

CRISP-DM Process Model

Vol. 12 No. 1, Nov. 2014

219

A Comparative Study of Data Mining Process Models (KDD, CRISP-DM and SEMMA)

2.2.4

MODELING

This is the fourth phase of CRISP-DM process selection and application of various modeling techniques. Different
parameters are set and different models are built for same data mining problem.
2.2.5

EVALUATION

This is the fifth stage of CRISP-DM process which focuses on evaluation of obtained models and deciding of how to use
the results. Interpretation of the model depends upon the algorithm and models can be evaluated to review whether
achieves the objectives properly or not.
2.2.6

DEPLOYMENT

This is the sixth and final phase of CRISP-DM process focuses on determining the use of obtain knowledge and results.
This phase also focuses on organizing, reporting and presenting the gained knowledge when needed.
2.3

THE SEMMA PROCESS MODEL

The SEMMA stand for (Sample, Explore, Modify, Model, and Access) is data mining method developed by SAS institute. It
offers and allows understanding, organization, development and maintenance of data mining projects. It helps in providing
the solutions for business problems and goals. SEMMA is linked to SAS enterprise miner and basically a logical organization of
the functional tools for them. It has a cycle of five stages or steps.
2.3.1

SAMPLE

This is the first and optional stage of SEMMA process which focuses on sampling of data. A portion from a large data set is
taken that big enough to extract significant information and small enough to manipulate quickly.
2.3.2

EXPLORE

This is the second stage of SEMMA process which focuses on exploration of data. This can helps in gaining the
understanding and ideas as well as refining the discovery process by searching for trends and anomalies.
2.3.3

MODIFY

This is the third stage of SEMMA process which focuses on modification of data by creating, selecting and transformation
of variables to focus model selection process. This stage may also looks for outliers and reducing the number of variables.
2.3.4

MODEL

This is the fourth stage of SEMMA process which focuses on modeling of data. The software for this automatically
searches for combination of data. There are different modeling techniques are present and each type of model has its own
strength and is appropriate for specific situation on the data for data mining.
2.3.5

ACCESS

This is the fifth and final stage for SEMMA process focuses on the evaluation of the reliability and usefulness of findings
and estimates the performance.

COMPARATIVE ANALYSIS OF KDD, CRISP-DM AND SEMMA

There are nine, six and five stages for KDD, CRISP-DM and SEMMA process model respectively. By examining all the three
data mining process models they clearly shows that they are somehow equivalent to each other even SEMMA is directly
linked to the SAS enterpriser miner software and CRISP-DM Daimler Chrysler (then Daimler-Benz), SPSS (then ISL) and NCR.
The comparison between them shows that

ISSN : 2351-8014

Vol. 12 No. 1, Nov. 2014

220

Umair Shafique and Haseeb Qaiser

The KDD process step Developing and Understanding of the Application Domain can be identified with Business
Understanding phase of CRISP-DM process.
The KDD process steps Creating a Target Data Set and Data Cleaning and Pre-processing can be identified with
Sample and Explore stages of SEMMA respectively, and/or these can be identified with Data Understanding phase
of CRISP-DM.
The KDD process stage Data Transformation can be identified with Data Preparation stage of CRISP-DM and Modify
stage of SEMMA process respectively.
The three stages of KDD Choosing the suitable Data Mining Task, Choosing the suitable Data Mining Algorithm and/or
Employing Data Mining Algorithm can be identified with Modeling phase of CRISP-DM and/or Model stage of
SEMMA process respectively.
The KDD process step Interpreting Mined Patterns can be identified with Evaluation phase of CRISP-DM process
and/or Assessment stage of SEMMA process respectively.
The KDD step Using Discovered Knowledge can be identified with Deployment phase of CRISP-DM process.
The comparison of these models is given in below table.
Table 1: Summary of KDD, CRISP-DM and SEMMA Processes

Data Mining Process Models

No. of Steps

KDD
9
Developing and
Understanding of
the Application
Creating a Target
Data Set
Data Cleaning and
Pre-processing
Data Transformation

CRISP-DM
6
Business
Understanding

---------Sample

Data
Understanding

Name of Steps
Choosing the suitable Data
Mining Task
Choosing the suitable Data
Mining Algorithm
Employing Data Mining
Algorithm
Interpreting
Mined Patterns
Using Discovered Knowledge

SEMMA
5

Explore

Data
Preparation

Modify

Modeling

Model

Evaluation

Assessment

Deployment

----------

CONCLUSION

Our study was on comparison between KDD, CRISP-DM and SEMMA data mining processes. As we know most of the
researchers and data mining experts follow the KDD process model because it is more complete and accurate. In contrast
CRISP-DM and SEMMA or mostly company oriented especially SEMMA that is used by SAS enterprise miner and integrate
with their software. However, study shows that CRISP-DM is more complete as compare to SEMMA. All and all these process
models guides and helps the people and experts to know that how they can apply data mining into practical scenarios.

ACKNOWLEDGMENT
We will like to thank our Parents and Family members. It is due to their support and guidance that encourage us to write
this paper.

ISSN : 2351-8014

Vol. 12 No. 1, Nov. 2014

221