Results-Oriented Software Process Improvement

Jessee Ring Software Quality First 40119 San Carlos Place Fremont, CA 94539 510-915-2353 Fax: 510-573-7464 Email: sqa1st@attbi.com Web: www.sqa1st.com

Results-Oriented SPI: Overview

Software Process Improvement

(SPI): An effective means for improving performance in software development organizations. Results-Oriented SPI: Faster, cheaper appraisals that facilitate more timely and effective improvement action. An overall focus on results-oriented, measurable software process improvement action.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 2

Certification vs. Improvement

Use the process model as a

guide to making improvements . li mprovement projects Al shou l d be l inked to business resu l ts . Don t do anything just because the process model says so . iter ia Using the model as a cr for cert i f ication often leads to irrat iona l behav i or . lts or iented . M etrics . Be resu
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 3

Performance

Time

Getting Started on SPI

Usually Begins With An

Appraisal Appraisal have tended to be:

* * * * *

SOFTWARE DEVELOPMENT AND QA ACTIVITIES

Very formal; Done by a team; Results presented in verbiage only; Identify deficiencies, not what to do about them; Approached from the perspective of certification vs. improvement.
sqa1st@attbi.com www.sqa1st.com 4

Questionnaire Interviews Doc. Review

PROCESS MODEL

2003 Software Quality First

Results-Oriented SPI: The Appraisal

) Done by one person ) Must have extensive

experience in multiple areas: t ware Systems and sof deve l opment i ng Test SQA SPI ividua l As an ind contr i butor & manager
sqa1st@attbi.com www.sqa1st.com 5

2003 Software Quality First

Results-Oriented SPI: The Appraisal Process

Reference: CMM Appraisal Framework SEI publication dated Feb., 1995

Plan

Process Data

Write Report

Kick-off

Evaluate Data

Present Results

Collect Data
2003 Software Quality First
sqa1st@attbi.com

Make Judgments
www.sqa1st.com 6

Take Action

Results-Oriented SPI Time Requirements For The Appraisal

One Calendar Week 40 50 hours by the appraiser Kick-off Meeting: 1 hour Participants: 2 3 hours during the week Results presentation: 1 Hour

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

Results-Oriented SPI: Appraisal Data Collection

Questionnaire Based upon the process model being used. Small group discussions Groups can be by job function or cross-functional. Document review
Selected documents, such as requirements specs, design docs, defect reports, CM records, project plan & schedule, etc.

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

Results-Oriented SPI Appraisal Questionnaire Data Analysis

Name Person 1 Person 2 Person 3 Person 4 Person 5 Person 6 Person 7 Number of Yes (1) % Yes Number of No (2) % of No Number of N.A. (3) % of N.A. Number of Don't Know (4) % of Don't Know RM-1 1 1 1 1 1 1 2 6 85.7 1 14.3 0 0.0 0 0.0 sqa1st@attbi.com RM-2 1 1 1 1 1 1 1 7 100.0 0 0.0 0 0.0 0 0.0 RM-3 1 1 1 1 2 2 2 4 57.1 3 42.9 0 0.0 0 0.0 9 RM-4 1 1 1 1 1 2 1 6 85.7 1 14.3 0 0.0 0 0.0 RM-5 1 1 2 2 2 2 3 2 28.6 4 57.1 1 14.3 0 0.0 RM-6 1 1 2 2 2 2 4 2 28.6 4 57.1 0 0.0 1 14.3 2.4 2.4 31.0 64.3 RM-all

2003 Software Quality First

www.sqa1st.com

Achievement Ratings
1 2 -3 4-5 67 8 9 - 10
2003 Software Quality First
sqa1st@attbi.com

Nothing in place. Just starting Making progress, getting some results. Some experience and better results. Fully implemented with a history of success. A model for others.
www.sqa1st.com 10

Results-Oriented SPI Process Model Based Ratings

Requirements Management
Goal 1: System requirements allocated to software are controlled to establish a baseline for software engineering and management use. Goal 2: Software plans, products and activities are kept consistent with the system requirements allocated to software. Rating Average Per KPA

10

10.0

10

Software Project Planning

Goal 1: Software estimates are documented for use in planning and tracking the software project. Goal 2: Software project activities and commitments are planned and documented. Goal 3: Affected groups and individuals agree to their commitments related to the software project.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 11

3 5 7

5.0

Results-Oriented SPI: Process Model Based Ratings Graphical Presentation

KPA Ratings
Scale of 1 10 with 10 Being Highest

10 9 8 7 6 5 4 3 2 1 0
P E F IS M RM PR TO M OP SP SQ

Software XYZ - Year 1

2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 12

SP

SC

OP

Software XYZ - Year 2

SP

TP

IC

Results-Oriented SPI: Appraisal Results

Summary of Results
Overall Strengths Major Issues

Consequences Recommendations

Detailed Results
Per KPA Strengths Issues Recommendations

2003 Software Quality First

sqa1st@attbi.com www.sqa1st.com 13

Results-Oriented SPI: Appraisal Results - Consequences

Consequences" of the

issues that were found are described. Makes the issues of more immediate interest. Answers the question: So what?. Justifies the allocation of resources.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 14

Results-Oriented SPI: Appraisal Results - Recommendations

Recommendations are a critical element of the Results-Oriented SPI approach. By including them in the Appraisal Report, people become focused on Improvement action. It also speeds up the whole process since the organization doesnt have to wait to find out what to do.

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

15

Example Of Results Reporting

Appraisal Results

Issue: Peer reviews are unstructured; no data or records are kept. Consequences: Peer review effectiveness is unknown but is probably low. Recommendation: Train participants on real inspections vs. walkthroughs (with roles, required preparation, record keeping). Track all items found in inspections to closure. Use data to evaluate inspection effectiveness and make improvements.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 16

Results-Oriented SPI: Metrics Plan

Two types of metrics: product & process
Product Metrics Number of open defects by severity Defect density Defect arrival rate Total size of the software Number of lines of code added, deleted, modified Number of new features and/or fixes contained in a new version Reliability growth Complexity of the software
2003 Software Quality First
sqa1st@attbi.com

Process Metrics Test effectiveness (number of escaped bugs) Percent of bugs found by test and inspections. Amount of time to find a bug by test and by inspections. Cycle time of regression testing. Test coverage Inspection coverage Process maturity ratings over time
17

www.sqa1st.com

Results-Oriented SPI Sample Metrics

160 140
Number of Defects

Open Defects at Time of Release

120 100 80 60 40 20 0
Version Version Version Version Version Version Version 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Crash, lock-up, etc. Functionality - minor Functionality - major Cosmetic Functionality - medium

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

18

Results-Oriented SPI: Sample Metrics

200 180 160 140 120 100 80 60 40 20 0

Defect Aging

Number of Defects

Over 30 Days
Crash, lock-up, etc. Functionality - minor

Over 60 Days

Over 90 Days

Functionality - major Cosmetic

Functionality - medium

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

19

Results-Oriented SPI: Sample Metrics

Software Reliability Growth
1000 900 800 700 600 500 400 300 200 100 0

Hours

Jan. Feb. Mar. Apr. May June July Aug. Sep. Oct. Nov. Dec. Mean time between failure
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 20

The Need For Software Process Improvement

Mars Climate Orbiter
Lost because of failure to convert from English to metric units in the navigational (orbiting) program.

Mars Polar Lander

Lost because of a missing line of code that caused the braking engines to shut down prematurely.

Result
$360 million out-of -pocket cost Indefinitely postpone all future launches to Mars Address shortcomings at JPL in areas ranging from systems analysis and testing to staffing and communications.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 21

Results-Oriented SPI: Summary

Use one very experienced person to perform a

software process appraisal. Give numeric ratings to achievement levels of process model requirements. Use the process model as a guide; not a mandate. Link all process improvements to business results. Use process maturity ratings to guide improvement actions, not as a final goal. Dont publish them. In the process appraisal report, identify consequences of the issues that were found, and recommendations for addressing them. Use metrics to manage the improvement actions. KISS
sqa1st@attbi.com www.sqa1st.com 22

2003 Software Quality First

Results-Oriented SPI: Benefits

Faster, less expensive appraisal. A sense of urgency. Can start on improvement immediately after appraisal (dont

have to wait on another team to formulate a plan). Can track process maturity levels over time graphically. Improvement actions are focused on the issues that will impact the bottom line. Participant see it as being worthwhile. Greater likelihood of upper management support for SPI. All of which leads to a successful improvement program.
2003 Software Quality First
sqa1st@attbi.com www.sqa1st.com 23

Results-Oriented SPI

2003 Software Quality First

sqa1st@attbi.com

www.sqa1st.com

24