Software Reviews,

Walkthroughs, Inspections, and

Audits
Better to find an error twice than
never to find it at all.

[Freedman/Weinberg 90] Freedman,
D.P., G.M. Weinberg, "Handbook of
Walkthroughs, Inspections, and
Technical Reviews", Dorset House
Publishing Co., Inc., 1990, pp. 89-
161.

Update terminology
according

Give references
to ONeil
Why Do We Have Formal
Technical Reviews?
1. To err is human.
2. Lots of errors escape the originator more
easily than anyone else.
3. Reviews are educational.
Purpose: Provide
visibility into state of project
opportunities for personnel (project and
non-project) to discuss topics related to
project
intermediate milestones and sense of
progress
assessment of technical adequacy of project
Use the power of a team to
Point out needed improvements of a
product.
Confirm the parts in which improvement is
not needed or desired.
Make technical work more uniform and
predictable in quality, which makes it more
manageable.
Technical review is different
from other reviews
For example, budget and management
reviews
Technical review answers question: will this
product do the job its supposed to do
If answer is no, then
no schedule is on time, and
no cost is cheap enough.
When?
Single review that occurs on a particular date
Single review that occurs in response to a
particular condition
Multiple reviews that occur periodically: e.g.,
monthly
Multiple reviews that occur in response to a
particular condition, e.g., code review for
subsystem
Benefits
10X reduction in errors in products
50-80% cost reduction
Why not just pass it around and
have reviewers sign off

Why not just pass it around and
have reviewers sign off
Comments are too large
Comments are unread
Reviewers dont have to take responsibility
for the content.
Requirements
Team participation
Documented procedures
Documented results

Types of Reviews
Management Review
Technical Review
Walkthrough
Inspection
Audit
Types of Reviews
Management Review
Evaluation of project-level plan or project progress
Monitors status of schedules and compliance to
requirements
Technical Review
Walkthrough
Inspection
Audit
Management Review
Objectives:
Inform management of project status
Resolve higher-level issues (management decisions)
Agree on risk mitigation strategies
Decisions
Identify corrective actions
Change resource allocation
Change project scope
Change schedule
Types of Reviews
Management Review
Technical Review
Focus on specification and design
Determine whether products meet specs
Walkthrough
Inspection
Audit
Scenario for a typical review
Leader: Welcome. We are here to review X for the
purpose of Y.
Each person is introduced.
The recorder is introduced.
An agenda is presented and the review process
explained.
The list of material in the review packet is stated.
The review begins
Review
Agenda set by leader, agreed to by
committee
Review proceeds line-by-line.
Reviewers bring up issues as they encounter
them.
Issues are added to an Issues List.
Agenda changes as issues are encountered.
Round Robin Review
Force every member to participate.
Each person takes the lead for a section:
paragraph,
line of code,
function,
test case
Good way to get involvement from team.
Types of Reviews
Management Review
Technical Review
Walkthrough
Used to find anomalies and evaluate conformance
Typically used for source code
Effective form of education
Inspection
Audit
Walkthroughs
Producer guides review
Step by step presentation of product:
Code, design, report, test cases
Can cover lots of material
Can have more people attend, less prepared
More work for presenter
May be difficult to control interactions
Too many interruptions
Always hard to have a producer present
Walkthroughs
Participant should spend 1 hour preparing
for each hour in meeting
Should be scheduled to be brief
Should only review completed code or
document
Should require participants to sign report
Types of Reviews
Management Review
Technical Review
Walkthrough
Inspection
More formal type of walkthrough
Used to identify specific errors or problems
Audit
Inspections
Rapid evaluation of material with specific
aspect in mind
Confine attention to one aspect only
Eg assume details are correct, but look for
missing items at a global level
Eg look for occurrences of a particular kind of
bug such as buffer overflow.
Selection of aspects is key point.
Types of Reviews
Management Review
Technical Review
Walkthrough
Inspection
Audit
Independent evaluation of process or product
Ensures that the process is being followed
Review Teams
Permanent or semi-permanent teams
IV&V or QA teams
Get to be very good at reviews
Need to have some way of reviewing the
reviewers.
The team
Leader
Job: obtain a good review or report why it was
not possible
Reflects the quality of the review process,
not the quality of the product
Must be accurate
Recorder
Provide info for accurate report of review
Short, public notes
Capture essence of each issue
Must ensure group has reached conclusion
Dont video tape

Tasks of leader
Monitor preparedness of team members
Set pace
Keep meeting on track
Poll members to reach consensus
Ensure participation
Has right to terminate review if it is unproductive
Disagreements
Fatal flaws
Consensus
Decision of the group is equal to the most
severe opinion of the group members.
Be conservative.
Accept the doubts of the most doubting
member.
No I told you so, but I got outvoted.
Reviewers
Must be qualified to contribute
Must be unbiased
Dont invite management if it causes
conflict
The point is to review the project, not the
producers.
Rules for Reviewers
I. Be prepared
II. Evaluate product, not people.
III. Watch your language.
IV. One positive, one negative.
V. Raise issues, dont resolve them.
VI. Record all issues in public.
VII. Stick to technical issues.
Rules for Reviewers
If you find something, assume its a mistake.
These are your peers, not your enemy. Remember
people are involved
Avoid why did you why didnt you say
instead I dont understand
Why did you set the upper bound to 10 here?
I dont understand why the upper bound is 10 here.
seems trivial, but its not.
Do not get bogged down in style issues. For
example, if efficiency isnt an issue, then dont
make it one.
If it makes things less maintainable, thats an issue.
If there are standards, then either stick to the
standard, or dispose of the standard.



Number of Reviewers
Ensure material is covered
Have enough
Dont have too many
3-7 is good
Count participants only
3: ensures material is understood
Outsiders can be good. Must be unbiased
Time
At most 2 hours at a time
Come back again if necessary
Depends on
Complexity
Size of project
Closeness to completion


Everyone must prepare!
Review packet
Everything relevant to making correct
judgment: code, specs, test data, results,
standards, designs,
80% of failures stem from unprepared teams


Tactics for participation
Devils Advocate
One person makes strongest case against a
product
Job is to find fault
Needs to be an actor
Debugging
Producers leave known bugs in
Quality of review depends on how many found
Alarms
Time each persons contribution, cut off
Stand-up Reviews
Talk as long as you stand on one leg
Report
Goal: Does the product do the job its supposed to
do?
Complete.
Correct.
Dependable as basis for other work
Measurable for purposes of tracking.
What was reviewed?
Signatures of reviewers.
Leader and recorder.

Notes
Requires learning: to review and to write
Need to allocate time
Not a form of project management
But can provide information for tracking
Review early and often
But not too often
Need for Reviews
Reviews help organization deliver quality
product
Examine individual parts early in process
Identify problems spanning multiple items or
phases
Reviews help identify problems early
Reviews help build image of the product
and the vision of the product in the minds of
the team members
Cost of Software Reviews
Up-front cost for
Training
Staff preparation
Review Conduct
Code review may be 8-15% of total cost of project
Offset by savings: reduced rework later in project
(25-35% of total project cost saved)
Raytheon: pre-1988: 43% of software cost in defect
correction
1994 (after software reviews instituted): 5% of project
cost in defect correction

Writing Issues
Groups of 3
Ill give you an issue from an issue list,
You tell me whats wrong with it.
Issues (and responses) from Handbook of
Walkthroughs, Inspections, and Technical
Reviews, Daniel Freedman and Gerald
Weinberg, 3
rd
Ed, Dorset House.
Writing Issues #1
Some of the explanations of user commands
were misinterpreted by members of the
review committee.
Comments on Issues #1
Not specific enough. Which explanations were
misinterpreted? Dont make the producers
guess, or they may change the wrong thing.
Writing Issues #2
A maximum of 10 values may be specified
even though none of the standard system
ABEND codes has been made not eligible.
Comments on Issues #2
In writing, clarity isnt the most important
thing: its the only thing. If you dont want
to be not understood, dont never use no
double negatives.
Another thing what is the issue anyway? Is
the maximum too low? Too high? Or is it
that some of the codes should have been
made not eligible? Or eligible? Be direct.
Writing Issues #3
The referenced table of constant values was
not part of the review packet.
Comments on Issues #3
Always give the most direct reference
available. There could be more than one
table, now or in the future. Why make the
producers search?
Writing Issues #4
The method used for maintaining the message
queue seems to solve a severe performance
problem weve been having with the
production version of the RKY system.
Comments on Issues #4
This is an interesting observation, clearly and
directly stated. But what does it have to do
with the product under review? It might be
worth millions, but it doesnt belong on the
Issues List for this product. It should go on
the Related issues list.
More issues #5
The price/performance table should be sorted
using either Quicksort or Shell sort.
Comments #5
Raise issues, but avoid all temptation to give
advice in the Issues List. They wont be
welcome if they come in that form, so if
you really must give advice, find some
unofficial vehicle. Take the producers to
lunch, or out for a beer, before you share
your vast experience on matters of sorting.
If your idea isnt worth the price of a beer,
why not forget it?
More issues #6
The three diagrams drawn by Harold Mitter
are not in the standard format (DS-109)
required for such diagrams in our
installation.
Comments #6
This point is nicely specific, but why do we
have to mention poor Harold? We are
reviewing the product, not the people. Find
another way to identify the diagrams and
leave peoples names out of the Issues List.
More issues #7
The committee was unable to understand the
significance of paragraph 3.1.
Comments #7
Nothings wrong with this one. Its specific about
which paragraph is under discussion, and it says
the committee doesnt understand it. Who can
argue with that? Yet, surprisingly, people seem to
be afraid to express issues this way We dont
understand Dont worry about being thought
stupid.
If you dont understand it, its at least a potential
issue in documentation. And besides, lack of
understanding may mean there is something
dreadfully wrong.
Yet more issues #8
The bubble sort used for sorting the table of
price/performance figures is a stupid
approach if the table should grow any
bigger than the present 20 entries.
Comments #8
The word stupid doesnt add anything at all and
might antagonize the producers. Take it out. Then
try to express factually and quantitatively why the
method is inappropriate for larger tables.
And if the approach is stupid, what of it? As Arthur
C. Clarke expressed it, It has yet to be proved
that intelligence has any survival value.
Yet more issues #9
If the BY NAME option is not used, the
structuring of the structure operands must
be equivalent to the structuring of the
structures in the arrays of structures.
Comments #9
This wasnt really taken from an Issues List,
but from an old PL/I manual. Still, Ive read
real issues that were almost as obscure. Id
advise you to follow the KISS axiom
(Keep It Simple, Stupid), except that I
would be contradicting my own advice from
the previous question.
Yet more issues #10
Frieda Sonntag has no computer science
background, and her experience with PL/I
is nil. She should never have been
assigned the coding of this module.
Comments #10
Its non of the business of the review
committee who management has assigned
to particular jobs. The committees business
is to review the product and tell what state
its in. How it got to that state is another
issueand not for the Issues List. Who is
responsible is even less of an issue, and
raising it is sure to be ineffective.