Software Reliability

Organization of this Lecture:

• Introduction.
• Reliability metrics
• Reliability growth modelling
• Statistical testing
• Summary
Software Reliability – Alternate
Definitions
• Informally denotes a product’s
trustworthiness or dependability.
• Probability of the product working
“correctly” over a given period of
time.
Software Reliability
• a software product having a large
number of defects is unreliable.
• reliability of a system improves if
the number of defects is reduced.
Reliability
• User’s concern
– An important attribute determining
the quality of the product.
• Demand for quantitative
estimation of reliability before
making buying decision.
Software Reliability Estimation
• Tricky problem
– Several factors contribute to making
measurement of software reliability
difficult.
Problems in Reliability
Measurements
• Errors do not cause failures at the
same frequency and severity.
– measuring latent errors alone not
enough
• The failure rate is observer-
dependent
Difficulties in Software
Reliability Measurement (1)
• No simple relationship observed
between system reliability and the
number of latent software defects.
• Removing errors from parts of
software which are rarely used
makes little difference to the
perceived reliability.
The 90-10 Rule
• Experiments from analysis of
behavior of a large number of
programs:
– 90% of the total execution time is
spent in executing only 10% of the
instructions in the program.
– This 10% part is the core of the
program.
Effect of 90-10 Rule on
Reliability
• removing 60% defects from least
used parts would lead to only
about 3% improvement to product
reliability.
• Reliability improvements from
correction of a single error
depends on whether the error
belongs to the core or the non-core
part of the program.
Difficulty in Software Reliability
Measurement (2)
• The perceived reliability depends
to a large extent upon how the
product is used
• In technical terms on its operation
profile.
Effect of Operational Profile on
Software Reliability
Measurement
• If we select input data so that only
“correctly” implemented functions
are executed none of the errors
will be exposed perceived
reliability of the product will be
high.
• On the other hand, if we select the
input data such that only functions
containing errors are invoked
perceived reliability of the system
will be low.
Software Reliability
• Different users use a software
product in different ways.
– defects which show up for one user
may not show up for another.
• Reliability of a software product
– clearly observer-dependent
– cannot be determined absolutely
Difficulty in Software Reliability
Measurement (3)
• Software reliability keeps changing
through out the life of the product
each time an error is detected and
corrected
Hardware vs. Software
Reliability
• Hardware failures inherently
different from software failures.
• Most hardware failures are due to
component wear and tear:
– some component no longer functions
as specified.
Hardware vs. Software
Reliability
• A logic gate can be stuck at 1 or 0,
– or a resistor might short circuit.
• To fix hardware faults:
– replace or repair the failed part.
Hardware vs. Software
Reliability
• Software faults are latent:
– system will continue to fail unless
changes are made to the software
design and code.
Hardware vs. Software
Reliability
• Because of the difference in effect
of faults, metrics appropriate for
hardware reliability measurements
are not good software reliability
metrics
Hardware vs. Software
Reliability
• When a hardware is repaired:
– its reliability is maintained
• When software is repaired:
– its reliability may increase or
decrease.
Hardware vs. Software
Reliability
• Goal of hardware reliability study :
– stability (i.e. interfailure times
remains constant)
• Goal of software reliability study
– reliability growth (i.e. interfailure
times increases)
Reliability Metrics
• Different categories of software
products have different reliability
requirements:
– level of reliability required for a
software product should be specified
in the SRS document.
Reliability Metrics
• A good reliability measure should
be observer independent,
– so that different people can agree on
the reliability.
Rate Of Occurrence of Failure
• ROCOF measures:
– frequency of occurrence failures.
– observe the behavior of a software
product in operation over a specified
time interval and calculate the total
number of failures during the interval.
Mean Time To Failure
• MTTF is average time between two
successive failures:
– Determine by observing over a large
number of failures.
Mean Time To Failure
• MTTF is not as appropriate for
software as for hardware:
– Hardware fails due to a component’s
wear and tear thus indicates how
frequently the component fails
– When a software error is detected
and repaired the same error never
appears.
Mean Time To Failure
• We can record failure data for n
failures:
– let these be t1, t2, …, tn
– calculate (ti+1-ti)
– the average value is MTTF
(ti+1-ti)/(n-1)
Mean Time to Repair
• Once failure occurs additional time
is lost to fix faults
• MTTR measures average time it
takes to fix faults.
Mean Time Between Failures
• We can combine MTTF and MTTR:
– to get an availability metric:
MTBF = MTTF + MTTR

• MTBF of 100 hours would indicate

– Once a failure occurs, the next failure
is expected after 100 hours of clock
time (not running time).
Availability
• Measures how likely the system
shall be available for use over a
period of time:
– considers the number of failures
occurring during a time interval,
– also takes into account the repair
time (down time) of a system.
Availability
• Important for Systems having real
time requirement:
– telecommunication systems,
– operating systems, etc. which are
supposed to be never down
– where repair and restart time are
significant and loss of service during
that time is important.
Failure Classes/ Category
• Transient:
– Transient failures occur only for certain
inputs.
• Permanent:
– Permanent failures occur for all input
values.
• Recoverable:
– When recoverable failures occur the system
recovers with or without operator
intervention.
Failure Classes
• Unrecoverable:
– the system may have to be restarted.
• Cosmetic:
– May cause minor irritations. Do not
lead to incorrect results.
• Eg. mouse button has to be clicked twice
instead of once to invoke a GUI function.
Reliability Growth Modelling
• A reliability growth model:
– a model of how software reliability
grows as errors are detected and
repaired.
• A reliability growth model can be
used to predict when (or if at all) a
particular level of reliability is likely
to be attained.
– i.e. how long to test the system?
Reliability Growth Modelling
• There are two main types of
uncertainty which render any
reliability measurement inaccurate:
• Type 1 uncertainty:
– our lack of knowledge about how the
system will be used, i.e.
• its operational profile
Reliability Growth Modelling
• Type 2 uncertainty:
– reflects our lack of knowledge about
the effect of fault removal.
• When we fix a fault we are not sure if the
corrections are complete and successful
and no other faults are introduced
• Even if the faults are fixed properly we do
not know how much will be the
improvement to interfailure time.
Step Function Model
• The simplest reliability growth
model:
– a step function model
• The basic assumption:
– reliability increases by a constant
amount each time an error is
detected and repaired.
Step Function Model

ROCOF

Time
Step Function Model
• Assumes:
– all errors contribute equally to
reliability growth
– highly unrealistic:
• we already know that different errors
contribute differently to reliability growth.
Jelinski and Moranda Model
• Realizes each time an error is
repaired reliability does not
increase by a constant amount.
• Reliability improvement due to
fixing of an error is assumed to be
proportional to the number of
errors present in the system at
that time.
Jelinski and Moranda Model
• Realistic for many applications,
• Shortcomings
– Most probable failures (failure types
which occur frequently) discovered
early during the testing process.
– Repairing these faults contribute
maximum to the reliability growth
initially.
– This implies rate of reliability growth
should be large initially slow down
later on
Littlewood and Verall’s Model
• Assumes different fault have
different sizes, thereby
contributing unequally to failures.
• Allows for negative reliability
growth
Littlewood and Verall’s Model
• Large sized faults tends to be
detected and fixed earlier
• As number of errors is driven down
with the progress in test, so is the
average error size, causing a law
of diminishing return in debugging
Other Reliability growth models
• Variations exists
– LNHPP (Littlewood non homogeneous
Poisson process) model
• Goel – Okumoto (G-O) Imperfect
debugging model
– GONHPP
• Musa – Okumoto (M-O) Logarithmic
Poisson Execution Time model
Applicability of Reliability Growth
Models
• There is no universally applicable
reliability growth model.
• Reliability growth is not
independent of application.
Applicability of Reliability Growth
Models
• Fit observed data to several
growth models.
– Take the one that best fits the data.
Statistical Testing
• A testing process:
– the objective is to determine
reliability rather than discover errors.
– uses data different from defect
testing.
Statistical Testing
• Different users have different
operational profile:
– i.e. they use the system in different
ways
– formally, operational profile:
• probability distribution of input
Operational profile: Example
• An expert user might give
advanced commands:
– use command language interface,
compose commands
• A novice user might issue simple
commands:
– using iconic or menu-based interface.
How to define operational
profile?
• Divide the input data into a
number of input classes:
– e.g. create, edit, print, file operations,
etc.
• Assign a probability value to each
input class:
– a probability for an input value from
that class to be selected.
Steps involved in Statistical
testing (Step-I)
• Determine the operational profile
of the software:
– This can be determined by analyzing
the usage pattern.
Step 2 in Statistical testing
• Manually select or automatically
generate a set of test data:
– corresponding to the operational
profile.
Step 3 in Statistical testing
• Apply test cases to the program:
– record execution time between each
failure
– it may not be appropriate to use raw
execution time
Step 4 in Statistical testing
• After a statistically significant
number of failures have been
observed:
– reliability can be computed.
Statistical Testing
• Relies on using large test data set.
• Assumes that only a small
percentage of test inputs:
– likely to cause system failure.
Statistical Testing
• It is straight forward to generate
tests corresponding to the most
common inputs:
– but a statistically significant
percentage of unlikely inputs should
also be included.
• Creating these may be difficult:
– especially if test generators are used.
Advantages of Statistical Testing
• Concentrate on testing parts of the
system most likely to be used:
– results in a system that the users find
more reliable (than actually it is!).
Advantages of Statistical Testing
• Reliability predictions based on
test results:
– gives an accurate estimation of
reliability (as perceived by the
average user) compared to other
types of measurements.
Disadvantages of Statistical
Testing
• It is not easy to do statistical
testing properly:
– there is no simple or repeatable way
to accurately define operational
profiles.
• Statistical uncertainty.
Software
Quality
 Quality comparative-
dictionary
a degree or level
fitness-for-purpose- quantitative -
of innate excellence manufacturing
Juran

a measure of extent of departure

usefulness and from the standard,
practicality degree of conformity
subjective -
artistic...
a measure of the skill of the creator
and value to the person
for whom it was created
Introduction
• Consider a software product:
– functionally correct
– but unusable user interface.
• Functional correctness alone does not
determine quality
Modern view of quality
• Associates several quality factors with a
software product :
– Correctness
– Reliability
– Efficiency (includes efficiency of resource
utilization)
– Portability
– Usability
– Reusability
– Maintainability
Correctness
• A software product is correct,
– if different requirements as specified
in the SRS document have been
correctly implemented.
– Accuracy of results.
Portability
• A software product is said to be
portable,
– Work on different operating systems
– Or on different machines
– Or with other software products
Reusability
• Different modules of the product
can easily be reused to develop
new products.
Usability
• A software product has good
usability, if different categories of
users (i.e. both expert and novice
users) can easily invoke the
functions of the product.
• HCI aspect
Maintainability
• A software product is
maintainable,
– If faults can be corrected
– Functionalities can be added or
modified (customized)
Software Quality
• the totality of features and
characteristics of a product or
service that bear on its ability to
satisfy stated or implied needs .
– ISO 8402
Difficulties in achieving software
quality
• its ethereal nature
• its discreteness (no tolerances)
• informal/semiformal discipline
• volatility of requirements
• inherent complexity of large
software systems
Cont…
• intellectual (read, semi-disciplined)
input
• complex intra-team
communication
• deadline pressures
• rapid advances in technology
• high (often unreasonable)
expectations of clients
McCall’s Model of Software Quality
• Operation
• Revision
• Transition
Operation Attributes

correctness
reliability integrity

usability efficiency
 Revision Attributes

maintainability flexibility

testability
Transition Attributes

portability reusability

interoperability
Crosby’s Maturity Level

Level 5: Certainty

Level 4: Wisdom

Level 3: Enlightenment
based on
Level 2: Awakening management
attitudes
Level 1: Uncertainty
SQA Environment
• Quality Policy
• Quality Management
• Quality Assurance
• Quality Control
• Quality Management System
• Total Quality Management
Evolution of Quality Systems
• Initial product inspection method :
– gave way to quality control (QC).
• Quality control:
– not only detect the defective products
and eliminate them
– but also determine the causes behind
the defects.
Quality control (QC)
• Quality control aims at correcting
the causes of errors:
– not just rejecting defective products.
• Statistical quality control
– quality of the output of the process is
inferred using statistical methods
– in stead of inspection or testing of all
products
Quality control (QC)
• The next breakthrough,
– development of quality assurance
principles
Quality assurance
• Basic premise of modern quality
assurance:
– if an organization's processes are
good and are followed rigorously,
• the products are bound to be of good
quality.
Quality assurance
• All modern quality paradigms
include:
– guidance for recognizing, defining,
analyzing, and improving the
production process.
Total quality management
(TQM)
• Advocates:
– continuous process improvements
through process measurements.
Quality Policy

the overall quality intentions

and direction of an
organization as regards
quality, as formally expressed
by top management.
- ISO 8402
Quality Management

that aspect of the overall

management function that
determines and implements the
quality policy.
- ISO 8402
Quality Assurance

a planned and systematic

pattern of all actions
necessary to provide
adequate confidence that the
item or product conforms to
established technical
requirement.
- IEEE 729
Quality Control

the operational techniques

and activities that are used
to satisfy the quality
requirements.
- ISO 8402
Quality Management System

the organizational structure,

responsibilities, procedures,
processes and resources for
implementing quality
management.

- ISO 8402
Total Quality Management

Quality is to satisfy customer's

requirements continually
Total quality is to achieve quality
at low cost
Total quality management is to
obtain total quality by involving
everyone's daily commitment
- Kanji
SQA Techniques - Error
detection & removal

• inspections
• walkthroughs
• reviews
• testing
• demonstrations
SQA Techniques - Error
avoidance
• systematic and error-proof
determination of user needs
• systematic and error-proof translation of
user needs into software products
• methods and tools
• rigorous specifications
• configuration management
• adherence to standards
• root cause analysis
Role of SQA

• evolve and approve a Software

Quality Assurance Plan (SQAP) for
every project
• Audit the development process for
proper execution in conformance
with SQAP and Software
Development Plan (SDP)
• Participate in the assessment of
internal deliverables and of overall
software quality
Role of SQA

• Perform, witness or audit tests; and

attend selected life cycle reviews to
assure appropriate Verification &
Validation
• Participate in configuration
management and QIP
• Provide appropriate visibility in situation
of non-conformance
Quality Guidelines

• start with management

commitment to ensure proper
attention
• specify quality goals to promote
understanding
• establish strategy to achieve
quality to ensure proper resources
for quality
• apply predictive measures to
evaluate quality and increase
Quality Guidelines

• assess quality achieved in final

product to determine how well
project satisfied goals
• analyze results to identify areas to
improve
• provide feedback to exploit lessons
learnt
Quality System Activities:
• Auditing of projects
• Development of:
– standards, procedures, and
guidelines, etc.
• Production of reports for the top
management
– summarizing the effectiveness of the
quality system in the organization.
• Review of the quality system
itself.
Business Process reengineering
• A term related to TQM.
• Process reengineering goes a step
further than quality assurance:
– aims at continuous process
improvement.
Business Process reengineering
• BPR aims at reengineering the way
business is carried out in any
organization
– not just software development
organizations.
Total quality management
(TQM)
• TQM goes beyond documenting
processes
– optimizes them through redesign.
• Over the years the quality
paradigm has shifted:
– from product assurance to process
assurance.
ISO 9000
• ISO (international Standards
Organization):
– Consortium established to formulate
and foster standardization.
• ISO published its 9000 series of
standards in 1987.
What is ISO 9000 Certification?
• ISO 9000 certification:
– serves as a reference for contract
between independent parties.
• The ISO 9000 standard:
– specifies guidelines for maintaining a
quality system.
What is ISO 9000 Certification?
• ISO 9000 specifies:
– guidelines for repeatable and high
quality product development.
– Also addresses organizational aspects
• responsibilities, reporting, procedures,
processes, and resources for
implementing quality management.
ISO 9000
• A set of guidelines for the
production process.
– not directly concerned about the
product it self.
– a series of three standards:
• ISO 9001, ISO 9002, and ISO 9003.
ISO 9000
• Based on the premise:
– if a proper process is followed for
production:
• good quality products are bound to
follow.
ISO 9001:
• Applies to:
– organizations engaged in design,
development, production, and
servicing of goods.
– applicable to most software
development organizations.
ISO 9002:
• ISO 9002 applies to:
– organizations who do not design products:
• but are only involved in production.
• Examples of this category of industries:
– steel or car manufacturing industries
– buy the product and plant designs from
external sources:
• only manufacture products.
– not applicable to software development
organizations.
ISO 9003
• Applies to organizations involved
only in installation and testing of
the products.
ISO 9000 for Software Industry
• ISO 9000 is a generic standard:
– applicable to any industry
• Many clauses of ISO 9000
documents:
– use generic terminologies
– very difficult to interpret them in the
context of software organizations.
Software vs. other industries
• No direct interpretation exist in
context of software industry
Software vs. other industries
• Software is intangible. Therefore
difficult to control.
– In contrast, in a car manufacturing
unit:
• we can see a product being developed
through stages such as fitting engine,
fitting doors, etc.
• one can accurately tell about the status
of the product at any time.
– Software project management is an
altogether different ball game.
Software vs. other industries
• During software development:
– the only raw material consumed is
data.
• For any other product
development:
– Lot of raw materials consumed
• e.g. Steel industry consumes large
volumes of iron ore, coal, limestone, etc.
• ISO 9000 standards have many
clauses corresponding to raw
material control .
Software vs. other industries
• Radical differences exist between
software and other product
development,
– difficult to interpret various clauses of
the original ISO standard in the
context of software industry.
ISO 9000 Part-3
• ISO released a separate document
called ISO 9000 part-3 in 1991
– to help interpret the ISO standard for
software industry.
• At present official guidance is
inadequate
Benefits of ISO 9000
Certification?
• Gain confidence of customers on
organization
• Government contracts. This is
especially true in the international
market.
How to Get ISO 9000
Certification?
• An organization intending to obtain
ISO 9000 certification:
– applies to a ISO 9000 registrar for
registration.
• ISO 9000 registration process
consists of several stages.
How to Get ISO 9000
Certification?
• Application stage:
– Applies to a registrar for registration.
• Pre-assessment:
– the registrar makes a rough
assessment of the organization.
How to Get ISO 9000
Certification?
• Document review and adequacy
audit:
– process and quality-related
documents.
– the registrar reviews the documents
– makes suggestions for improvements.
How to Get ISO 9000
Certification?
• Compliance audit: the registrar
checks
– whether the suggestions made by it
during review have been complied.
How to Get ISO 9000
Certification?
• Registration:
– The registrar awards ISO 9000
certificate after successful
completions of all previous phases.
• Continued surveillance:
– The registrar continues monitoring
the organization periodically.
ISO 9000 Certification
• An ISO certified organization
– can use the certificate for corporate
advertizements
– cannot use the certificate to advertize
products.
• ISO 9000 certifies organization's process
• not any product of the organization.
– An organization using ISO certificate for
product advertizements:
• risks withdrawal of the certificate.
Summary of ISO 9001
Requirements
• Management responsibility(4.1):
– Management must have an effective
quality policy.
– The responsibility and authority of all
those whose work affects quality:
• must be defined and documented.
Management responsibility(4.1)
• Responsibility of the quality
system.
– independent of the development
process,
– can work in an unbiased manner.
• The effectiveness of the quality
system:
– must be periodically by audited.
Quality system (4.2) and
contract reviews (4.3):
• A quality system must be
maintained and documented.
• Contract reviews (4.3):
– Before entering into a contract, an
organization must review the contract
• ensure that it is understood,
• organization has the capability for
carrying out its obligations.
Design control (4.4):
• The design process must be
properly controlled,
– this includes controlling coding also.
• A good configuration control
system must be in place.
Design control (4.4):
• Design inputs must be verified as
adequate.
• Design must be verified.
• Design output must be of required
quality.
• Design changes must be
controlled.
Document control (4.5):
• Proper procedures for
– document approval, issue and
removal.
• Document changes must be
controlled.
– use of some configuration
management tools is necessary.
Purchasing (4.6):
• Purchased material, including
bought-in software:
– must be checked for conforming to
requirements.
Purchaser Supplied Products
(4.7):
• Material supplied by a purchaser,
– for example,
• client-provided software must be properly
managed and checked.
Product Identification (4.8):
• The product must be identifiable at
all stages of the process.
– In software development context this
means configuration management.
Process Control (4.9) :
• The development must be properly
managed.
• Quality requirements must be
identified in a quality plan.
Inspection and Testing (4.10) :
• In software terms this requires
effective testing i.e.,
– unit testing, integration testing and
system testing.
• Test records must be maintained.
Inspection, measuring and test
equipment(4.11):
• If integration, measuring, and test
equipments are used,
– must be properly maintained and
calibrated.
Control of nonconforming
product (4.13) :
• In software terms,
– keeping untested or faulty software
out of released product,
• or other places whether it might cause
damage.
Corrective Action (4.14) :
• This is both about correcting
errors when found,
– investigating why they occurred
– improving the process to prevent
further occurrences.
• If an error reoccurs despite the
quality system,
– the system needs improvement.
Handling (4.15) and Quality
audits (4.17):
• Handling (4.15) Deals with:
– storage, packing, and delivery of the
software product.
• Quality Audits (4.17) :
– quality system audit must be carried
out to ensure its effectiveness.
Training (4.18) :
• Training needs must be identified
and met.
• Most items of ISO standard
– are largely common sense.
Salient features of ISO 9001
requirements:
• All documents concerned with the
development of a software product
– should be properly managed,
authorized, and controlled.
• Proper plans should be prepared
– progress against these plans should
be monitored.
Salient features of ISO 9001
requirements:
• Important documents
independently checked and
reviewed:
– for effectiveness and correctness.
• The product should be tested :
– against specification.
• Several organizational aspects:
– e.g., management reporting of the
quality team.
Shortcomings of ISO 9001
Certification (1)
• ISO 9000 requires a production
process to be adhered to:
– but does not guarantee the process to
be of high quality.
– Does not give any guideline for
defining an appropriate process.
Shortcomings of ISO 9001
Certification (2)
• ISO 9000 certification process
– not fool-proof
– no international accredition agency
exists.
– likely variations in the norms of
awarding certificates:
• among different accredition agencies and
among the registrars.
Shortcomings of ISO 9001
Certification (3)
• Organizations qualifying for ISO
9001 certification:
– tend to downplay domain expertise.
– tend to believe that since a good
process is in place,
• any engineer is as effective as any other
engineer in doing any particular activity
relating to software development.
Shortcomings of ISO 9001
Certification (4)
• In manufacturing industry
– clear link between process quality and
product quality
– once a process is calibrated:
• can be run again and again producing quality
goods
• Software development is a creative
process:
– individual skills and experience is significant
Shortcomings of ISO 9001
Certification (5)
• Many areas of software
development are very specialized:
– special expertize and experience
(domain expertize) required.
• ISO 9001
– does not automatically lead to
continuous process improvement,
– does not automatically lead to TQM.
Shortcomings of ISO 9001
Certification (6)
• ISO 9001 addresses mostly
management aspects.
• Techniques specific to software
development have been ignored
– Configuration management
– Reviews
– Release builds
– Problem Notification system
– Intranets
SEI Capability Maturity Model
• Developed by Software
Engineering Institute (SEI) of the
Carnegie Mellon University, USA:
– to assist the U.S. Department of
Defense (DoD) in software
acquisition.
– The rationale was to include:
• likely contractor performance as a factor
in contract awards.
SEI Capability Maturity Model
• Major DoD contractors began CMM-
based process improvement initiatives:
– as they vied for DoD contracts.
• SEI CMM helped organizations:
– Improve quality of software they developed
– Realize adoption of SEI CMM model had
significant business benefits.
• Other organizations adopted CMM.
SEI Capability Maturity Model
• In simple words,
– CMM is a model for apprising the
software process maturity of a
contractor into different levels.
– Can be used to predict the most likely
outcome to be expected from the
next project that the organization
undertakes.
SEI Capability Maturity Model
• Can be used in two ways:
– Capability evaluation
– Software process assessment.
Capability Evaluation
• Provides a way to assess the
software process capability of an
organization
– Helps in selecting a contractor
– Indicates the likely contractor
performance
Software Process Assessment
• Used by an organization to assess
its current process:
– Suggests ways to improve the
process capability.
– This type of assessment is for purely
internal use.
SEI Capability Maturity Model
• The SEI CMM classifies software
development industries into:
– Five maturity levels.
– Stages are ordered so that
improvements at one stage provide
foundations for the next
– Based on the pioneering work of
Philip Crosby
SEI Capability Maturity Model
Optimizing (5)

Managed (4)

Defined (3)

Repeatable (2)

Initial (1)
Level 1: (Initial)
• Organization operates
– without any formalized process or
project plans
• An organization at this level is
characterized by
– ad hoc and often chaotic activities.
Level 1: (Initial)
• Software production processes are
not defined,
– different engineers follow their own
process
– development efforts become chaotic.
– The success of projects depend on
individual efforts and heroics.
Level 2: (Repeatable)
• Basic project management practices
– tracking cost, schedule, and functionality
are followed.
• Size and cost estimation techniques
– function point analysis, COCOMO, etc. used.
• Production process is ad hoc
– not formally defined
– also not documented.
Level 2: (Repeatable)
• Process used for different projects
might vary between projects:
– earlier success on projects with
similar applications can be repeated.
– Opportunity to repeat process exist
when a company produces a family of
products.
Level 3: (Defined)
• Management and development
activities:
– defined and documented.
– Common organization-wide
understanding of activities, roles, and
responsibilities.
Level 3: (Defined)
• The process though defined,
– process and product qualities are not
measured.
• ISO 9001 aims at achieving this
level.
Level 4: (Managed)
• Quantitative quality goals for
products are set.
• Software process and product
quality are measured:
– The measured values are used to
control the product quality.
– Results of measurement used to
evaluate project performance
• rather than improve process.
Level 4: (Managed)
• Organization sets quantitative
quality goals
• World-wide about 100
organizations assessed at this
level.
Level 5: (Optimizing)
• Statistics collected from process
and product measurements are
analyzed:
– continuous process improvement
based on the measurements.
• Known types of defects are prevented
from recurring by tuning the process
• lessons learned from specific projects
incorporated into the process
Level 5: (Optimizing)
• Identify best software engineering
practices and innovations:
– tools, methods, or process are
identified
– transferred throughout the
organization
• World-wide about 50 organizations
have been assessed at this level.
Key Process Areas
• Each level is associated with a key
process area (KPA) identifies
– where an organization at the previous
level must focus to reach this level
Level 2 KPAs
• Software project planning
– Size, cost, schedule.
– project monitoring
• Configuration management
• Subcontract management
Level 3 KPAs
• Process definition and
documentation
• Reviews
• Training program
Level 4 KPAs
• Quantitative measurements
• Process management
Level 5 KPAs
• Defect prevention
• Technology change management
• Process change management
Comparison between ISO 9001
and SEI CMM
• ISO 9001 awarded by an
international standards body
– can be quoted in official documents
and communications
• SEI CMM assessment is purely for
internal use.
Comparison between ISO 9001
and SEI CMM
• SEI CMM was developed
specifically for software industry:
– addresses many issues specific to
software industry.
• SEI goes beyond quality assurance
– aims for TQM
– ISO 9001 correspond to SEI level 3.
Comparison between ISO 9001
and SEI CMM
• SEI CMM provides a list of key areas
– on which to focus to take an organization
from one level to the other
• Provides a way for gradual quality
improvements over several stages.
– e.g trying to implement a defined process
before a repeatable process:
• counterproductive as managers are
overwhelmed by schedule and budget pressure.
Remarks on Quality Model
Usage
• Highly systematic and measured
approach to software development
process suits certain circumstances
– negotiated software, safety-critical
software, etc
• What about small organizations?
• Typically handle applications such as internet, e-
comm.
• without an established product range,
• without revenue base, experience on past
projects, etc.
• CMM may be incompatible
Small Organizations
• Small organizations tend to
believe:
– We are all competent people hired to
do a job, we can’t afford training
– We all communicate with one another
• Osmosis works because we are so close
– We are all heroes
• We do what needs to be done
• Therefore rules do not apply to us
Small Organizations
• Often have problems:
– Undocumented requirements
– Inexperienced managers
– Documenting the product
– Resource allocation
– Training
– Peer reviews
Small Organizations
• A two week CMM-based appraisal
is probably excessive:
• Small organizations need to
operate more efficiently at lower
levels of maturity
– Must first fluorish if eventually they
are to mature
Personal Software Process (PSP)
• Based on the work of Humphrey
• PSP is a scaled down version of
industrial software process
– suitable for individual use
• Even CMM assumes that engineers
use effective personal practices
Personal Software Process (PSP)
• A process is the set of steps for
doing a job
• The quality and productivity of an
engineer
– largely determined by his process
• PSP is framework that
– helps software engineers to measure
and improve the way they work.
Personal Software Process (PSP)
• Helps developing personal skills and
methods
– Estimating and planning method
– Shows how to track performance against
plans
– Provides a defined process
• can be fine tuned by individuals
• Recognizes that a process for individual use is
different from that necessary for a team project.
Time Management
• Track the way you spend time
– Boring activities seem longer then
actual
– Interesting activities seem short
• Record time for
– Designing
– Writing code
– Compiling
– Testing
Personal Software Process (PSP)

Planning
Design
Code Logs
Compile
Test Project plan 
Postmortem
summary
PSP-Planning
• Problem definition
• Estimate max, min, and total LOC
• Determine minutes/LOC
• Calculate max,min, and total
development times
• Enter the plan data in project plan
summary form
• record the planned time in Log
PSP-Design
• Design the program
• Record the design in specified
format
• Record the Design time in time
recording log
PSP-Code
• Implement the design
• Use a standard format for code
text
• Record the coding time in time
recording log
PSP-Compile
• Compile the program
• Fix all the defects
• Record compile time in time
recording log
PSP-Test/Postmortem
• Test
– Test the program
– Fix all the defects found
– Record testing time in time recording
log
• Postmortem
– Complete project plan summary form
with actual time and size data
– Record postmortem time in time
record
Personal Software Process (PSP)

PSP 3 • Personal process

evolution

PSP 2 • Personal quality management

• Design and code reviews
PSP 1 •Personal planning
• Time and schedule
PSP 0 • Personal measurement
• Basic size measures
Fitting pieces
Building the CMM Structure
Capability Maturity Models
• CMMI SM – CMM IntegrationSM
• SW-CMM – Capability Maturity Model®
for Software
• P-CMM – People Capability Maturity
Model
• SA-CMM – Software Acquisition
Capability Maturity Model
• SE-CMM – Systems Engineering
Capability Maturity Model
• IPD-CMM – Integrated Product
Development Capability Maturity Model
SW - CMM
• Describes principles and practices
underlying software process
maturity
• Intended to help software
organizations improve the maturity
of their software processes
SE - CMM
• Describes essential elements of an
organization's systems engineering
process
• Provides a reference for comparing
actual systems engineering
practices against these essential
elements.
IPD - CMM
• Guide organizations in IPD design,
development, appraisal, and
improvement.
• Involves a teaming of the
functional disciplines to integrate
and concurrently apply all
necessary processes to produce an
effective and efficient product that
satisfies the customer's needs
CMMI
• The content and characteristics of
SW-CMM [draft version 2(c)], SE-
CMM [EIA/IS 731], and IPD-CMM
[Version 0.98] models provide the
basis for the CMMI product suite
• CMMI models eventually replace
SW-CMM Version 1.1
Motivation for Integrations
• Models created within an
environment of evolving national
and international standards and
frameworks
• Maintaining harmonization
between them and improvement
models become a continuing
challenge, particularly across
discipline
Frameworks Quagmire
CMM Integration
• Objective is to develop a product with a
set of integrated products to support
process and product improvement.
• Preserve investment in process
improvement and enhance the use of
multiple models.
• Output will be integrated model(s),
assessment method(s), and training
material.
CMMI Objectives
• Eliminating inconsistencies
• Reducing duplications
• Increasing clarity and
understanding
• Assuring consistency with ISO
15504

Longer-term objective is to lay

foundation for later addition
(acquisition, security)
CMM Based Appraisals
• CMM-Based Appraisal for Internal
Process Improvement (CBA IPI): To
determine the state of one's own
processes
• Software Capability Evaluation (SCE):To
determine the state of someone else's
processes.
• To ensure that these methods are both
rigorous and consistent, the Process
Program developed and maintains the
CMM Appraisal Framework (CAF), which
defines the standards that an appraisal
When to apply CBA IPI
• For investigating your
organization, using mostly your
own team members, and making
principal use of the results
yourself, then a CBA IPI
assessment is the recommended
tool. This method assumes a
collaborative environment and a
collegial approach to gathering
data.
When to apply SCE
• If, on the other hand, you are
investigating somebody else's
organization, using your own team
members (not theirs), and you will use
the results for your purposes (even if
you plan to share the results with the
organization), then an SCE is the
recommended tool. This method
assumes a potentially audit-like
environment and a more intrusive
approach to gathering data
SEI’s Role In CMM-Based
Appraisals
• Authorized Lead Assessors for CBA
IPI are required to return a report
to the SEI for each completed
assessment. The data is entered
into the Process Appraisal
Information System. All
assessment data that is returned
to the SEI is kept confidential.
• The SEI trains and authorizes
qualified persons to lead appraisals
Cont…
• The SEI does not validate or certify
appraisal results.
• An assessment's findings and
maturity level are owned by the
assessment sponsor. The sponsor
may publicize this information at
their discretion.
• The Maturity Profile Report is
published by the SEI twice a year
That’s all?
• Not exactly
• SQA Activities
– Cleanroom methodology
– Six Sigma
• Assessment models
– SPR Assessment
– Malcolm Baldrige National Quality
Award
Cleanroom Software
Engineering
• Metaphor coming from integrated
circuit manufacturing
– Free from dust, flecks of skin etc
• Whenever defect occurs, they are
considered as defect in process
than product
– Invoke process inspection, review and
improvement cycle
Six Sigma
• Motorola’s baby that won them
MBNQA award for quality in 1988
• Six sigma is a quantitative
approach to eliminate defects
• Applicable to any industry -
manufacturing, product
development, service
Six Sigma
• To achieve six sigma
– a process must not produce more than 3.4
defects per million opportunities.
– 5 Sigma -> 230 defects per million
– 4 Sigma -> 6210 defects per million
• Six sigma methodologies
– DMAIC (Define, Measure, Analyze, Improve,
Control)
– DMADV: (Define, Measure, Analyze, Design,
Verify)
Six Sigma Methodologies
• The methodologies are
implemented by Green belt and
Black belt workers
– Supervised by Master black belt
worker
• Pareto Chart:
– Simple bar chart to represent defect
data
– Identify the problems that occurs with
greatest frequency
Software Productivity Research
Inc.
• Methodology developed parallel to SEI
Process maturity model
• SPR question covers both Strategic
corporate issues and tactical project
issues that affect quality, productivity
and user satisfaction
• 400 questionnaire as against SEI’s 85
questions
– Lickert Scale for response (Excellent, Good,
Average, Marginal, Poor)
Malcolm Baldrige National Quality
Award
• Established in 1988 by US
Department of Commerce
• Examination criteria against seven
categories
– Leadership
– Information and analysis
– Strategic Quality Planning
– Human Resource Utilization
– Quality assurance of products and
services
– Quality Results
– Customer satisfaction
References
• CMMI Distilled
– Ahern, Clouse, Turner. Addison
Wesley
• CMM in Practice
– Pankaj jalote. Pearson Education
• Introduction to Personal SW
Process, Introduction to Team SW
Process
– Watts Humphrey. Addison Wesley
 Web References
• General Info
– http://www.sei.cmu.edu
– http://www.sei.cmu.edu/sema/profile.html
• PSP, TSP
– http://interactive.sei.cmu.edu
/Features/1999/June/Background/Background.jun99.
pdf [Both]
– http://www.sei.cmu.edu/pub/documents/articles/pdf/
pdf [PSP]
– http://wuarchive.wustl.edu/languages/ada/sei/docum
pdf [TSP]
Alternatives/Additions
• ISO Standards
– Generic Specifications in areas of Quality
Requirements (9000), Environmental
Management (14000)
– Latest on Information Technology (ISO/IEC
TR 15504-#:1998)
, where 0<#<10
– Some client demand for ISO Certified
institutes

http://www.iso.ch/iso/en/iso9000-14000/tour/m
Other References
• http://www.software.org/quagmire/
• Presentation by Rajib Mall