2009

CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

Practical Report: CMMI Measurements and Analysis practices

based on Agile (Scrum) Method
Ahmed Mahdy
Senior Software Quality Engineer, Agile Coach
Raya Corporation, Egypt
aamahdys@gmail.com
ahmed_mahdy@rayacorp.com

Sree Yellayi
SCAMPI Lead Appraiser, SEI Authorized CMMI Instructor, Certified Scrum Master
Siemens, Greater New York City Area
sree.yellayi@siemens.com
sreeramamurthy@yahoo.com

Reviewed by Hillel Glazer

Certified High Maturity SCAMPI Lead Appraiser, SCAMPI B&C Team Leader, and an Introduction to
CMMI® instructor for both CMMI for Development and CMMI for Services
hillel@entinex.com

Abstract
Software industry is one of the most empirical industries due to its high dependence on technology and people. Each
software company adopts a specific development methodology, and furthermore seeks to build a system for its process
improvement by adopting one or more Software Process Improvement (SPI) models. CMMI is a process framework
which is widely adopted by software and systems development companies, while Scrum is one of the more
recent project management agile method whose adoption is growing rapidly. CMMI is basically a process
improvement framework which provides a set of processes for software and system development management, Scrum
can be thought of as an iterative project management framework for development activities, CMMI has a wider scope
and different aims to those of Scrum and covers production support, maintenance, product implementation and
application transition type projects as well.
Scrum and other agile methods have clearly appeared in 2001, this paper shows how to implement the Measurements
and Analysis (M&A) process area of CMMI model and clarify how M&A can be achieved in the agile (Scrum)
organization.
Most of the promising objectives of any software development method or process are delivering working software on
time, quality and budget. Software Engineering Institute (SEI) has paid a lot of efforts to mostly satisfy these
objectives in its process improvement models, and lately CMMI version 1.2 has been released.
Nevertheless, the world becomes convinced with adding another objective, which is delivering a business value to the
customer. Along the years, software engineers have proposed several methodologies. Agile is the most known and
latest proposed development method that can achieve this objective beside the other mentioned objectives.
Our objective in an agile environment is not to do software measurement. We must learn to build reliable software
measurement process based on valid software measurement tools. If we try to do too much too soon, we will likely
fail.

Introduction
Software industry is one of the most empirical industries due to its high dependence on technology and people. Each
software company adopts a specific development methodology, and furthermore seeks to build a system for its process
improvement by adopting one or more Software Process Improvement (SPI) models. CMMI is a process framework
which is widely adopted by software and systems development companies, while Scrum is one of the more recent
project management agile method whose adoption is growing rapidly. CMMI is basically a process improvement
framework which provides a set of processes for software and system development management, Scrum can be
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

thought of as an iterative project management framework for development activities, CMMI has a wider scope and
different aims to those of Scrum and covers production support, maintenance, product implementation and application
transition type projects as well [1]. In 1996, Kent Beck joined the Chrysler C3 payroll project. It was in this context
that the full set of XP practices matured, with some collaboration by Ron Jeffries and inspiration from earlier 1980s
work at Tektronix with Ward Cunningham. XP went on to garner significant public attention because of its emphasis
on communication, simplicity, and testing, its sustainable developer-oriented practices, and its interesting name. [2]
Scrum and other agile methods have clearly appeared in 2001, Alan MacCormack reported a study of key success
factors in recent projects; first among these was adopting an IID life cycle: Now there is proof that the evolutionary
approach to software development results in a speedier process and higher-quality products. […] The iterative process
is best captured in the evolutionary delivery model proposed by Tom Gilb [3]. Scrum appeared when a group of 17
process experts—representing DSDM (Dynamic Systems Development Method), XP (extreme Programming) ,
Scrum, FDD (Feature Driven Development), and others— who are interested in promoting modern and simple IID
(Iterative and Incremental Development) methods and principles, they met in Utah to discuss common ground. From
this meeting came the Agile Alliance (www.agilealliance.org) and the now popular catch phrase “agile methods,” all
of which apply IID. And in 2002, Alistair Cockburn, one of the participants, published the first book under the new
appellation [4]. Moreover, prominent software-engineering thought leaders from each succeeding decade supported
IID practices, and many large projects used them successfully [5]. Agile is all about value individuals and interactions
over processes and tools, working software over comprehensive documentation, customer collaboration over contract
negotiation, and responding to change over following a plan [15]. This paper shows how to implement the
Measurements and Analysis (M&A) process area which is an important Process Area (PA) in CMMI model; the
organization cannot be appraised without satisfying this PA, and clarifying how M&A can be achieved in the agile
(Scrum) organization.
Basically, software engineering measurement is not a resource issue; it is a commitment issue. The bottom line is that
all the usable measures from practicing Scrum on a project could be used to address the practices of M&A process
area. In fact, the alignment of these measures to the information needs is very visible due to the very nature of “value-
based” focus of the scrum method. In our research, we found that there need not be any additional measures that are
required to be “invented” to fulfill the M&A goals from CMMI model. Usage of existing ones is enough and we will
share those mappings and arguments with you for your application and subsequent extension to other process areas in
the model.

Agile CMMI
Why Agile CMMI
Most of the promising objectives of any software development method or process are delivering working
software on time, quality and budget. Software Engineering Institute (SEI) has paid a lot of efforts to mostly
satisfy these objectives in its process improvement models, and lately CMMI version 1.2 has been released.
Nevertheless, the world becomes convinced with adding another objective, which was somehow neglected
more than highly considered, which is delivering a business value to the customer. Along the years, software
engineers have proposed several methodologies (see figure 1),
 1965

Years

Yes! [6]
Waterfall (Royce): Requirements,
1970 Design, Implementation, Verification
and maintentance

Feasible together?

Model Components [7]

other fore mentioned objectives.
V-Model (Anon): Align Testing to
1980 waterfall development.

Spiral Model (Barry Boehm)

1985 Iterative

RAD (James Martin)

1991 Prototyping, iterative, time-boxed, and

Requirement of CMMI Measurements and Analysis (M&A)

user driven

RUP (Rational)
(Figure-1: History of Software Development Methodologies)

Object Oriented, Iterative, time-boxed, and user

1998
driven
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

1999
Agile (Kent Beck)
Incremental, user driven, customer interaction
2009

Agile is the most known and latest proposed development method that can achieve this objective beside the
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

(Figure-2: CMMI Model Components)

Measurements and Analysis Required Objectives and Expected Practices

o SG 1: Align Measurement and Analysis Activities

SP 1.1: Establish Measurement Objectives

SP 1.2: Specify Measures

SP 1.3: Specify Data Collection and Storage Procedures

SP 1.4: Specify Analysis Procedures
o SG 2: Provide Measurement Results

SP 2.1: Collect Measurement Data

SP 2.2: Analyze Measurement Data

SP 2.3: Store Data and Results

SP 2.4: Communicate Results

Implementation of M&A
M&A Procedure Flow
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

(Figure-3: Measurements & Analysis Procedure Flow)

Identify Measurement Objectives

A child must learn to crawl before he walks. He must learn to walk before he runs. So it will be for an
incipient software measurement program. We must learn to crawl first. At the outset, the complexity of the
measurement problem space appears astonishingly large. There are many products, such as requirements,
design, and code. Each of these products was produced by a process, in a development environment, by
people [8]. It is very difficult to measure a software process. This is not a good place to start. It is very
dangerous to measure people. This measurement data is so easily misused. Until we have gained some real
sophistication in measuring people and process, we will have little success in trying to measure aspects of
the software development environment. [9]

Our objective in an agile environment is not to do software measurement. We must learn to build reliable
software measurement process based on valid software measurement tools. If we try to do too much too
soon, we will likely fail. Basically, software engineering measurement is not a resource issue; it is a
commitment issue. We are going to build a measurement process that will generate great volumes of data.
These data must be converted to information that can be used in the software development decision-making
process. The principle is a simple one. Even a small amount of measurement data that can be converted to
useful information is better than large volumes of measurement data that have little or no information value.
We must begin with simple tools and focus most of our attention on the measurement and management
processes. We must remember that we learned how to measure distances in the first grade with a very crude
ruler. Our teachers did not give us micrometers to learn measurement. Ultimately, we learned that our rulers
could be used to quantify size attributes of the objects in our environment. These rulers, in fact, had utility
beyond their obvious use as bludgeons that we could use on our classmates. [10] In order to monitor and evaluate
process performance we must consider views of different stakeholders that take part in the process. The best performance is
achieved when the objectives of all stakeholders are satisfied. Kueng [11] defines process performance as “the degree of
stakeholder satisfaction”.
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

The objectives are subjective and should be defined from the organization; however, here are some
examples of what objectives vs. stakeholders can be established (table 1):

Objectives Stakeholder
• Timely information on project information performance
and progress
• Product Quality Improvement Progress IT Management
• Process Quality Improvement Progress

• Time Utilization and Management Project Team Members

• Roles and Responsibilities Commitment

• Efficient and effective resolutions for the impediments Project Scrum Master and
• Process Compliance Quality Assurance

• Customer Satisfaction Customer

Table 1: Practical Example of Measurement Objectives and Stakeholders

To illustrate the meaning of who could most likely be the fore mentioned stakeholders:

IT Management: General Manager, Project Manager who are concerned with the traditional aspects of
software development from the perspective of time, cost, and quality.

Project Team Members: Developers, Testers, Team Leader, and Technical Writers

Project Quality Assurance & Coach: Concerned with facilitating the use of SCRUM and CMMI, creating
conditions for smooth running of the development process, hence the main goal is to provide an efficient
impediments resolution.

Customer: the customer, customer representative, and/or other third parties.

Specification of Measures
The measures are obtained based on the defined objectives, by thinking of each objective and all related
links that negatively or positively affect it.
According to CMMI, measures may be either “base” or “derived”. While data for base measures are
obtained by direct measurement, data for derived measures come from other data, typically by combining
two or more base measures. Derived measures serve as performance indicators showing the achievement of
particular goals. Originally, Scrum had only one base measure: the estimate of the amount of work
remaining that needs to be done in order to complete a Product Backlog item or a task in the Sprint Backlog
(SB). At the task level, this measure is collected every day for each task in the Sprint Backlog separately.
[12]
Fortunately, the daily Scrum meetings give an outstanding help to achieve Measurements & Analysis,
especially the data collection.
Here are some proposed measures as a practical example:

Objective Measures
• Timely information • Remaining work from Sprint S
on project • Remaining work on day d for each task in the Sprint Backlog
information (SB)
performance and • Each task progress for day d in the SB
progress
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

• Time Utilization • Number of working hours per day d

and Management
• Efficient and • Number of impediments
effective • Number of resolved impediments
resolutions for the
impediments
• Customer • Customer Acceptance
Satisfaction • Customer Survey Result

• Product Quality • The number of bugs found during the Sprint S.

Improvement • The number of bugs reported by the user in a fixed period
Progress after Release R.
• Total number of Product Backlog Items (PBIs) or done stories
committed in the Sprint S.
• The number of PBIs completed in the Sprint S.
• Total number of tasks in the Sprint
• The number of tasks completed during the Sprint S.
• Actual Vs Planned in any Process Improvement Change
• Number of Process Improvement Requests

• Roles and • The number of completed tasks for each team member (for
Responsibilities each Sprint Backlog)
Commitment
• Process • Number of Non-Compliances (NCs) reported by Quality
Compliance Assurance per Sprint S.
• Number of resolved NCs per Sprint S.

Table 2: Practical Example of Measurement Objectives and its Measures

Data Collection Points

All base measures are based on data

(Figure-3: Data Collection Points for the specified measures)

It’s not recommended to propose a mapping between the measures and its collection points in a general
proposal, to allow more flexibility to different environments and practices.
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

The collection points should be defined in the process and can be customized per project (if needed).
Moreover, the use of automated tools is much recommended after gaining the main concept of Agile and
deeply knowing the goals of CMMI.

Measurements Analysis

Derived measures are derived from the base measures, such derived measures support the decision
makers in different management levels that serve for analyzing software process performance in
comparison to target values set by software development organization.
As practical examples:
The objective “Timely Information on Project Progress” can be analyzed using the following indicators:
- Work Effectiveness
It’s the ratio between the decrement of remaining work and done work.
The decrement of remaining work between days d1 and d2 of Sprint should be equal or greater than the done
work in the same interval, meaning that the best value for this indicator is 1 or more; however, the values
significantly greater than 1 may be a sign of poor planning.

WS d
WE d = (Eq. 1)
WP d

Where WEd denotes the Work Effectiveness of a specific day d in a Sprint, WP d denotes the planned
work that should be accomplished in the beginning of the day d, WS d denotes the work spent in the
day d. The calculation of Works can be in terms of tasks’ number or the average percentages of
tasks’ completion.

However, burn-up or burn-down charts can do this job in an easier visualized way, and it can be done
weekly or every 2 days.

- Schedule Performance Index (SPI)

It’s the ratio between the Earned Value (EV: the value of all completed tasks) and Planned Value (PV: the
estimated planned value of all tasks to be completed in a certain time during the project) [13]. The best value
for SPI is 1 or more; however, if it’s greater than 1, then the project is ahead of schedule.

And to calculate the Schedule Performance Index using the work remaining and work spent measures we
assume that the amount of tasks that must be accomplished at a certain point in the Sprint is proportional to
the time elapsed from the beginning of the Sprint. The work remaining and work spent measures allow a
precise definition of the Earning Value equation EVd , j for each task j in the Sprint Backlog on the day d of a
Sprint. It can be computed as a ratio between the amount of work already spent and all the work required
(spent and remaining) to accomplish the task:

d −1

∑WS
i =1
i, j
EVd , j = d −1 (Eq. 2)
∑WS
i =1
i, j + WRd , j

WSi , j denotes the amount of work spent for task j on day i, i=1,2,…,d-1, and WRd , j denotes the amount of
work remaining for task j on day d.

By using the earning value, we can get the SPI by the following formula:
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

∑ EV
j =1
d, j WRinitial , j
SL
SPI d = n
• (Eq. 3)
DE
∑WR
j =1
initial , j

Where WRinitial , j denotes the initial estimate of the work remaining for task j, SL the Sprint Length in the unit
of days, and DE the number of days elapsed.

- Cost Performance Index of employee costs (CPI)

It’s the ratio between the Earned Value (currency unit) and actual costs.
The best value for CPI is 1 or more, and this means that the cost of completing the work is planned rightly or
less than planned respectively [14].

And for the other objectives:

- The achievement degree of work per Sprint that shows what percentage of the completed tasks
compared with the planned tasks, the ratio between the number of tasks completed in the Sprint and
total number of tasks in the Sprint Backlog or between the number of PBIs completed in the release
and total number of PBIs committed (per Release).
- The average amount of overtime at Sprint/release/project level considering the expected hours, the
amount of work spent and administrative days.
- The average number of projects that each employee works (or participates) in parallel.
- Qualitative evaluation of working conditions or working values like communication and teamwork,
physical discomfort, psychological well-being, workload, supervision, opportunities for growth,
transparency etc. and easily the team can get these measures by practicing Scrum retrospectives per
Sprint or/and Release, in more detail:
1. At the beginning of project or Release, specify the working values (or conditions)
2. At each retrospective, add a part to get a self-assessment for each value from the team
members in one time, and ask for justification for each low and far different assessments,
finally get the recommendations and suggestions from the team members on how to improve
these assessments next Sprint.

Summary
In fact, there is no practice in scrum like the expected practices of Measurements and Analysis (MA).
Nevertheless, in scrum you have the opportunities to get the measures that implement MA easily.
And to summarize the specific practices (table3) and generic practices (table4) of MA process area [16]:

SP 1.1 Establish and maintain • Team Kickoff or Chartering of the project

measurement objectives that • The more important thing in any measurement system is
are derived from identified the settlement of objectives and values/purposes for each
information needs and measure
objectives. • The measures can be tailored by project also to add the
best value
SP 1.2 Specify measures to address • Sprint Burndown chart that tracks effort remaining.
the measurement objectives. • Release Burndown or Burnup chart that tracks story
points that have been completed. This shows how much
of the product functionality is left to complete.

SP 1.3 Specify how measurement • This can be declared/defined in Team Kickoff or

data will be obtained and Chartering of the project
stored.

CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110
SP 1.4 Specify how measurement
data will be analyzed and
reported.
SP 2.1 Obtain specified
measurement data.
SP 2.2 Analyze and interpret
measurement data.
SP 2.3 Manage and store
measurement data,
measurement specifications,
and analysis results.
SP 2.4 Report results of
measurement and analysis
activities to all relevant
stakeholders.
Table 3: Summary of MA specific practices in Scrum
GP 1.1 Perform the specific
practices of the measurement
and analysis process to
develop work products and
provide services to achieve
the specific goals of the
process area.
GP 2.1 Establish and maintain an
organizational policy for
planning and performing the
measurement and analysis
process.
GP 2.2 Establish and maintain the
plan for performing the
measurement and analysis
process.
GP 2.3 Provide adequate resources
for performing the
measurements and analysis
process
GP 2.4 Assign responsibility and
authority for performing the
process, developing the work
products, and providing the
services of the MA
GP 2.5 Train the people performing
or supporting the MA
process as needed
GP 2.6 Place designated work
products of the measurement
and analysis process under
appropriate levels of control.
2009
• The Scrum process does describe the purpose and use
the Sprint and Release burndown charts.
• Define the analysis procedure clearly and the mechanism
of reporting.
• Daily Scrum meeting where Sprint and Release
burndown/burnup data are collected.
• Daily Scrum meeting where Sprint and Release
burndown data are analyzed.
• This can be declared/defined in Team Kickoff or
Chartering of the project
• Daily Scrum meeting where Sprint and Release
burndown charts are reviewed.
• Table 3 satisfies this practice
• Specify the goals and objectives in the organizational
policy to be aligned with the plan of measurements and
the intended measures along the project.
• The Scrum lifecycle definition and the releases (or
milestones) to perform Scrum.
• The defined measures and data collection points
• The resources and schedule time allocated to perform
Scrum planning, monitoring and requirements activities
(for example: the internal and external chartering that
shows the environment, roles and resources)
• The resource roles allocated to perform the activities in
process documents and in the chartering as well (i.e.
define who is responsible for measurement planning,
collection… and all activities)
• Scrum team training for the aspects of Scrum that relate
to measurements & analysis. (i.e. training materials,
…etc)
• Here is CM role.
• To make it simpler, you can take pictures of measures,
make an excel sheet for the measures and measurement
files, and define their location in CM based on project
repository structure
• Using any version control will help you to achieve this
practice (like SharePoint, VersionOne, …etc)
 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

• Include in the repository of measurement & analysis: the

used tools for measurement analysis, selected measures
for this project, data collection points…etc

GP 2.7 Identify and involve the
relevant stakeholders of the
measurement and analysis
process as planned.
GP 2.8 Monitor and control the
measurement and analysis
process against the plan for
performing the process and
take appropriate corrective
action.
• The list of Scrum team members, their roles and
allocations in the chartering (the beginning of the
project)
• Report to the relevant stakeholders with the results of
measurement analysis.
• Scrum master follows the plan & schedule of
measurements, and data collection points.
• Scrum master assures that the measurements are
collected and analyzed according to the plan & schedule.

GP 2.9 Objectively evaluate • Provide measurement results.

adherence of the • Align measurement & analysis activities
measurement and analysis • Specifications of base and derived measures
process against its • Data collection and storage procedures
process description,
standards, and
procedures, and address
noncompliance.

GP2.10 Review the activities,
status, and results of the
measurement and
analysis process with
higher level management
and resolve issues.
(other generic goals for
level 4 and 5)
Table 4: Summary of MA generic practices in Scrum
• The senior manager has a visibility into scrum teams’
work and progress by burnup or burndown charts.
• The senior manager can be reported with the reports of
measurement & analysis results.
• The actions that are taken by the senior manager
according to the reported measurement & analysis
results.

Conclusion
The bottom line is that all the usable measures from practicing Scrum on a project could be used to address
the practices of M&A process area. In fact, the alignment of these measures to the information needs is very
visible due to the very nature of “value-based” focus of the scrum method. In our research, we found that
there need not be any additional measures that are required to be “invented” to fulfill the M&A goals from
CMMI model. Usage of existing ones is enough.

References
[1] Scrum and CMMI: A High level assessment of compatibility, Srinivas Chillara and Pete Deemer

[2] K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, 1999

[3] A. MacCormack, “Product-Development Practices That Work,” MIT Sloan Management Rev., vol. 42,
no. 2, 2001

[4] A. Cockburn, Agile Software Development, Addison-Wesley, 2002

 2009
CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110

[5] Craig Larman, Victor R. Basili , Iterative and Incremental Development: A brief History

[6] Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, Sandy Shrum, CMMI or Agile: Why Not
Embrace Both, Nov 2008

[7] CMMI Product Team, CMMI for Acquisition Ver 1.2, Nov 2007

[8] Juran, J.M., Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services,
Free Press, New York, 1992

[9] Deming, WE., Out of the Crisis, MIT Press, Cambridge, MA, 2000

[10] Munson, J.C. and Coulter, N.S., "An Investigation of Program Specification Complexity," Proceedings
of the ACM Southeastern Regional Conference, April 1988

[11] P. Kueng, “Process performance measurement system: a tool to support process-based organizations,”
Total Quality Management, 2000

[12] J. Sutherland et al., “Scrum and CMMI Level 5: The Magic Potion for Code Warriors,” in Proc.
AGILE 2007

[13] Project Management Knowledge, Schedule Performance Index [online], http://www.project-

management-knowledge.com/definitions/s/schedule-performance-index , 2008
[14] Project Management Knowledge, Cost Performance Index [online], http://www.project-management-
knowledge.com/definitions/c/cost-performance-index, 2008

[15] Manifesto for Agile Software Development [online], http://www.agilemanifesto.org , 2008

[16] Neil Potter and Mary Sakry, Process Group Newsletter Volume 16 No 2, March 2009