1 Overview 5 Management

2 Life cycle 6 Tools

3 Static testing 7 Web testing

4 Dynamic test techniques 8 Software quality

From "Software quality assurance", Daniel Galin

Contents
Software quality and software quality management

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Software quality
(IEEE definition) Software quality is: the degree to which a system, component, or process meets specified requirements the degree to which a system, component, or process meets customer or user needs or expectations

McCalls triangle of quality

Maintainability Flexibility Testability Portability Reusability Interoperability

Product Revision

Product Transition

Product Operation
Correctness Reliability Efficiency Integrity Usability

Software quality factors

Correctness accuracy, completeness of required output up-to-dateness, availability of the information Reliability maximum failure rate Efficiency resources needed to perform software function Integrity software system security, access rights Usability ability to learn, perform required task

Software quality factors

Maintainability effort to identify and fix software failures (modularity, documentation, etc.) Flexibility degree of adaptability (to new customers, tasks, etc) Testability support for testing (e.g. log files, automatic diagnostics, etc.)

Software quality factors

Portability adaptation to other environments (hardware, software) Reusability use of software components for other projects Interoperability ability to interface with other components/systems

Software quality management

Ensuring that required level of product quality is achieved defining procedures and standards applying procedures and standards to the product and process checking that procedures are followed collecting and analyzing various quality data The aim is to manage the quality of software and of its development process

Quality management activities

Software Quality Planning (SQP)

Software Quality Assurance (SQA)

Software Quality Control (SQC)

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Software quality planning

A quality plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes

The quality plan should define the quality assessment process

It should set out which organisational standards should be applied and, where necessary, define new standards to be used

Software quality planning (contd)

Quality plan structure product introduction product plans process descriptions quality goals risks and risk management Quality plans should be short, succinct documents

Software quality planning (contd)

Software quality attributes
Safety Security Reliability Resilience Robustness Understandability Testability Adaptability Modularity Complexity Portability Usability Reusability Efficiency Learnability

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Software quality assurance

(IEEE definition) SQA is:

1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements 2. A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with: quality control

Objectives of SQA in development

Assuring an acceptable level of confidence that the software will conform to functional technical requirements

Assuring an acceptable level of confidence that the software will conform to managerial scheduling and budgetary requirements Initiation and management of activities for the improvement and greater efficiency of software development and SQA activities

Typical activities of a SQA process

Requirements validation

Design verification
Static code checking Dynamic testing

Process engineering and standard

Metrics and continuous improvement

SQA Group
Who involves quality assurance activities?
software engineers, project managers, customers, sale

people, SQA group,...

The SQA groups role

serves as the customers in-house representative
assist the software engineering team in achieving high-

quality

The SQA groups responsibility

quality assurance planning oversight, record keeping,

analysis and reporting

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Software quality control (SQC)

A set of activities designed to evaluate the quality of a developed or manufactured product (IEEE, 1991) involves the series of inspections, reviews, and tests used throughout the software process to ensure each work product meets the requirements placed upon it Contrast with quality assurance

There are two approaches to quality control quality reviews automated software assessment and software measurement

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Software quality metrics

Definition (IEEE, 1993) a quantitative measure of the degree to which a system, component, or process possesses a given attribute Main objectives to facilitate management control as well as planning and execution of the appropriate managerial interventions to identify situations that require or enable development or maintenance

Classification of software quality metrics

Classification by phases of software system process metrics metrics related to the software development process product metrics metrics related to software maintenance Classification by subjects of measurements quality timetable effectiveness (of error removal and maintenance services) productivity

23

Process metrics categories

Software process quality metrics error density metrics error severity metrics Software process timetable metrics

Software process error removal effectiveness metrics

Software process productivity metrics

24

Error Density metrics

Code
CED DED

Name
Code Error Density Development Error Density

Calculation formula
NCE CED = ----------KLOC NDE DED = ----------KLOC WCE WCDE = --------KLOC WDE WDED = --------KLOC WCE WCEF = ---------NFP WDE WDEF = ---------NFP

WCED
WDED WCEF WDEF

Weighted Code Error Density

Weighted Development Error Density Weighted Code Errors per Function Point Weighted Development Errors per Function Point

NCE = The number of code errors detected by code inspections and testing. NDE = total number of development (design and code) errors) detected in the development process. WCE = weighted total code errors detected by code inspections and testing. WDE = total weighted development (design and code) errors detected in development process.

The concept of indicator

A software development department may apply two alternative metrics for calculation of code error density: CED and WCED

The unit has to determine indicators for unacceptable software quality:

CED > 2 and WCED > 4

26

Error Severity metrics

Code Name Calculation formula
WCE ASCE = ----------NCE WDE ASDE = ----------NDE

ASCE Average Severity of Code

Errors ASDE Average Severity of Development Errors

NCE = The number of code errors detected by code inspections and testing. NDE = total number of development (design and code) errors detected in the development process. WCE = weighted total code errors detected by code inspections and testing. WDE = total weighted development (design and code) errors detected in development process.

Software process timetable metrics

Code Name Time Table Observance Calculation formula
MSOT TTO = ----------MS TCDAM ADMC = ----------MS

TTO

ADMC Average Delay of Milestone

Completion

MSOT = Milestones completed on time. MS = Total number of milestones. TCDAM = Total Completion Delays (days, weeks, etc.) for all milestones.

Error removal effectiveness metrics

Code Name Calculation formula NDE Development Errors Removal DERE DERE = ---------------Effectiveness NDE + NYF WDE DWERE Development Weighted DWERE = -----------------Errors Removal Effectiveness WDE+WYF
NDE = total number of development (design and code) errors) detected in the development process. WCE = weighted total code errors detected by code inspections and testing. WDE = total weighted development (design and code) errors detected in development process. NYF = number software failures detected during a year of maintenance service. WYF = weighted number of software failures detected during a year of maintenance service.

Process productivity metrics

Code Name Calculation formula
DevH DevP = ---------KLOC DevH FDevP = ---------NFP ReKLOC Cre = -------------KLOC ReDoc DocRe = ----------NDoc

DevP

Development Productivity

point Development FDevP Function Productivity

CRe

Code Reuse

DocRe Documentation Reuse

DevH = Total working hours invested in the development of the software system. ReKLOC = Number of thousands of reused lines of code. ReDoc = Number of reused pages of documentation. NDoc = Number of pages of documentation.
30

Product metrics categories

HD quality metrics: HD calls density metrics - measured by the number of calls HD calls severity metrics - the severity of the HD issues raised HD success metrics the level of success in responding to HD calls HD productivity metrics HD effectiveness metrics Corrective maintenance quality metrics software system failures density metrics software system failures severity metrics failures of maintenance services metrics software system availability metrics Corrective maintenance productivity and effectiveness metrics

HD calls density metrics

Code HDD Name HD calls density Calculation Formula
NHYC HDD = -------------KLMC WHYC WHYC = -----------KLMC WHYC WHDF = -----------NMFP

WHDD Weighted HD calls density

WHDF Weighted HD calls per function point

NHYC = the number of HD calls during a year of service. KLMC = Thousands of lines of maintained software code. WHYC = weighted HD calls received during one year of service. NMFP = number of function points to be maintained.

Severity of HD calls metrics

Code Name Calculation Formula
WHYC ASHC = -------------NHYC

ASHC

Average severity of HD calls

NHYC = the number of HD calls during a year of service.

WHYC = weighted HD calls received during one year of service.

HD success metrics
Code HDS Name HD service success Calculation Formula
NHYOT HDS = -------------NHYC

NHYNOT = Number of yearly HD calls completed on time during one year of service. NHYC = the number of HD calls during a year of service.

HD productivity and effectiveness metrics

Code Name
HD Productivity

Calculation Formula
HDYH HDP= -------------KLNC HDYH FHDP = ---------NMFP HDYH HDE = -------------NHYC

HDP

FHDP Function Point HD Productivity HDE

HD effectiveness

HDYH = Total yearly working hours invested in HD servicing of the software system. KLMC = Thousands of lines of maintained software code. NMFP = number of function points to be maintained. NHYC = the number of HD calls during a year of service.

Software system failures density metrics

Code Name
Software System Failure Density Weighted Software System Failure Density
Weighted Software System Failures per Function point

Calculation Formula
NYF SSFD = -------------KLMC WYF WFFFD = --------KLMC
WYF WSSFF = ---------NMFP

SSFD WSSFD
WSSFF

NYF = number of software failures detected during a year of maintenance service. WYF = weighted number of yearly software failures detected during one year of maintenance service. NMFP = number of function points designated for the maintained software. KLMC = Thousands of lines of maintained software code.

Software system failure severity metrics

Code

Name ASSSF Average Severity of Software System Failures

Calculation Formula
WYF ASSSF = -------------NYF

NYF = number of software failures detected during a year of maintenance service. WYF = weighted number of yearly software failures detected during one year.

Failures of maintenance services metrics

Code

Name MRepF Maintenance Repeated repair Failure metric -

Calculation Formula
RepYF MRepF = -------------NYF

NYF = number of software failures detected during a year of maintenance service. RepYF = Number of repeated software failure calls (service failures).

Software system availability metrics

Code Name Calculation Formula
NYSerH - NYFH FA = ----------------------NYSerH NYSerH - NYVitFH VitA = ----------------------------NYSerH NYTFH TUA = -----------NYSerH

FA VitA

Full Availability Vital Availability

TUA

Total Unavailability

NYSerH = Number of hours software system is in service during one year. NYFH = Number of hours where at least one function is unavailable (failed) during one year, including total failure of the software system. NYVitFH = Number of hours when at least one vital function is unavailable (failed) during one year, including total failure of the software system. NYTFH = Number of hours of total failure (all system functions failed) during one year. NYFH NYVitFH NYTFH. 1 TUA VitA FA

Software corrective maintenance productivity and effectiveness metrics

Code Name Corrective Maintenance Productivity Function point Corrective Maintenance Productivity Corrective Maintenance Effectiveness Calculation Formula
CMaiYH CMaiP = --------------KLMC CMaiYH FCMP = -------------NMFP CMaiYH CMaiE = -----------NYF

CMaiP FCMP

CMaiE

CMaiYH = Total yearly working hours invested in the corrective maintenance of the software system. NYF = number of software failures detected during a year of maintenance service. NMFP = number of function points designated for the maintained software. KLMC = Thousands of lines of maintained software code.

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Quality management standards

Quality assurance and standards standards are the key to effective quality management may be international, national, organizational or project standards Importance of standards encapsulation of best practice they are a framework for quality assurance processes they provide continuity ISO 9001 and ISO 9000-3 standards

SEI CMM and CMMI assessment standard

ISO
ISO: International Organization for Standardization

ISO 9000: an international set of standards for quality management, applicable to a range of organisations from manufacturing to service industries ISO 9000 : Fundamentals and vocabulary ISO 9001 : Requirements ISO 9004 : Guidelines for performance improvements ISO 9000-3: represents implementation of the general methodology of quality management ISO 9000 Standards to the software development and maintenance

ISO/IEC 9000-3: 2004

Provides guidance for organizations in the application of ISO 9001: 2000 to the acquisition supply development operations, and maintenance of computer software and related software services Does not add or change the requirements in ISO 9001 realignment to ISO 9001: 2000

Capability Maturity Models CMM

Developed by the Software Engineering Institute (SEI), sponsored by the U.S. Department of Defense (DoD) A methodology used to develop and refine an organization's software development process

CMM components maturity levels key process areas (KPA) goals and common features key practices process capabilities

Maturity levels - Key Process Areas

Maturity levels initial repeatable defined managed optimized Key Process Areas (KPA) related activities for achieving goals issues to address to reach each maturity level

Level 1 - Initial
Process is ad hoc, occasionally chaotic

Few and informally defined processes

No mechanism to ensure they are used consistently Ineffective planning

Unpredictable
Success due to heroic effort

Level 2 - Repeatable
Basic management processes, quality assurance and configuration control procedures in place Can repeat earlier successes Realistic project commitments based on results of previous project Still has frequent quality problems Stable planning and tracking

Key Process Areas Level 2

Requirement management

Software project planning

Software project tracking Software subcontract managent

Software quality assurance Software configuration management

Level 3 - Defined
The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software

Key Process Areas Level 3

Organization process focus

Organization process definition

Training program Integrated software management

Software product engineering

Intergroup coordination Peer reviews

Level 4 - Managed
Quality and performance is measured using metrics and can be predicted Quality and productivity quantitative goals are established Exceptional cases are identified and addressed

Process is measured and operates within limits

Detailed measures of the software process and product quality are collected

Both the software process and products are quantitatively understood and controlled

Key Process Areas Level 4

Quantitative process management

Software quality management

Level 5 - Optimizing
Organization focuses on continuous process improvement Goal is to address and prevent problems by analyzing their cause in the process Process improvement is budgeted, planned, and part of the organizations process Identify and quickly transfer best practices

Key Process Areas Level 5

Defect prevention

Technology change management

Process change management

Comparisons, contrasts and applicability

Similar the CMM and ISO 9000 address similar issues and have the common concern of quality and process management Difference
ISO CMM focus on the customer-supplier the attention on the software relationship supplier to improve its internal processes wide range of industries specific to the software industry has less depth has more depth

Comparisons, contrasts and applicability

ISO follow a set of standards to make success repeatable the next step is only to maintain that level of certification CMM emphasizes a process of continuous improvement an on-going process of evaluation and improvement, moving from one level of achievement to the next. Even at the highest level of maturity in CMM, the focus is on continuous improvement

CMMI
CMMI models provide guidance for developing or improving processes that meet the business goals of an organization

CMMI is the successor of the CMM In 2002, CMMI Version 1.1 was released, Version 1.2 followed in August 2006, and CMMI Version 1.3 in November 2010

CMMI models
CMMI-SE/SW: integrates the system engineering and software engineering CMMI-SE/SW/IPPD: integrates system engineering, software, integrated product/process and supplier sourcing aspects CMMI-SE/SW/IPPD/SS: integrates system engineering, software engineering and integrated product/process and supplier sourcing engineering aspects

Contents
Software quality and management of software quality

Software quality planning

Software quality assurance Software quality control

Software quality metrics

Quality management standards Cost of software quality

Cost of software quality

Objectives control organization-initiated costs to prevent and detect software errors evaluation of the economic damages of software failures as a basis for revising the SQA budget evaluation of plans to increase or decrease SQA activities or to invest in a new or updated SQA infrastructure on the basis of past economic performance

Classic model of cost of software quality

Prevention costs Cost of software quality Costs of control
Appraisal costs Internal failure costs

Costs of failure of control costs

External failure costs

Prevention costs
Investments in development of new or improved SQA infrastructure Regular implementation of SQA preventive activities: instruction of new employees in SQA subjects certification of employees Control of the SQA system through performance of: internal quality reviews external quality audits by customers and SQA system certification organizations management quality reviews

Appraisal costs
Costs of reviews: formal design reviews (DRs) peer reviews (inspections and walkthroughs) expert reviews Costs of software testing: unit, integration and software system tests acceptance tests (carried out by customers) Costs of assuring quality of external participants

Internal failure costs

Represent the costs of error correction subsequent to formal examinations of the software during its development (design reviews, software tests and acceptance tests), prior to the systems installation at the customers site costs of redesign or design corrections subsequent to design review and test findings costs of re-programming or correcting programs in response to test findings costs of repeated design review and re-testing (regression tests)

External failure costs

Entail the costs of correcting failures detected by customers or maintenance teams after the software system has been installed at customer sites

Typical external failure costs (overt cost)

resolution of customer complaints
correction of software bugs

correction of software failures after the warranty period

damages paid to customers reimbursement of customer's purchase costs

insurance against customer's claims

External failure costs (contd)

Typical examples of hidden external failure costs reduction of sales to customers that suffered from software failures severe reduction of sales motivated by the firm's damaged reputation increased investment in sales promotion to counter the effects of past software failures