Documente Academic
Documente Profesional
Documente Cultură
Improvement Program
Rick Holcomb
AIR-4.2 Cost Department
(301) 342-2450
holcombra@navair.navy.mil
30 January, 2003
Dan Galorath
Galorath Incorporated
(310) 414-3222
dgalorath@galorath.com
36th ADoDCAS
Agenda
30 January, 2003
36th ADoDCAS
Whats Necessary
for Accurate SW
Estimation?
30 January, 2003
36th ADoDCAS
What is Accuracy?
How close is the initial estimate to the
final estimate?
How has functionality changed from
the initial technical baseline to the
final product? What impact has this
had on the estimate?
30 January, 2003
36th ADoDCAS
30 January, 2003
36th ADoDCAS
30 January, 2003
36th ADoDCAS
SW-CMM Summary
Levels of Process Maturity
5
RISK
LOW
HIGH 1
30 January, 2003
4
3
2
MANAGED
DEFINED
REPEATABLE
INITIAL
Continuously
improving process
OPTIMIZING
Predictable process
Standard, consistent process
Disciplined
Ad hoc; occasionally chaotic
36th ADoDCAS
Probability of Success
120%
Product Succe ss
100%
80%
60%
SEI 1 & 2 Success
40%
20%
0%
0
10
100
1,000
10,000
100,000
30 January, 2003
36th ADoDCAS
Over/Under Percentage
140%
0%
-140%
. .
.. .
..... .... ..... ..... ................. ....
.. ...... ......... .. . .
.. . .
.
.
.
.
.
.
.
.
. . ... ..
. . . . . .. .. .. .. .. . . . .. .. . .
.
.. .
..
. .. . . . . . .. .
. .. .. . .. . .. . ... . . .. .. .
. . .. ... . .. . .. .
.. . . . .. . .. . . ....
. . .. . . . . .. . . . .
. .. .
. ..
.
.
.
30 January, 2003
36th ADoDCAS
NAVAIR
Initiatives
30 January, 2003
36th ADoDCAS
10
Process Improvement
Developers of ACAT I, II, III, & IV Software
Intensive Systems will achieve a SEI SW-CMM
Level 3 or equivalent.
NAVAIRINST 5234.1 Policy on Software Evaluations for Naval Air Systems Command Programs
30 January, 2003
36th ADoDCAS
11
30 January, 2003
36th ADoDCAS
12
Communications
The best results are achieved
when all levels of the organization
agree with the need for change
rather then having it forced upon
them.
A major aspect of NAVAIRs
initiatives is the training and
communications with the
workforce.
30 January, 2003
36th ADoDCAS
13
Progress
F/A-18 Weapons Software Support Activity (WSSA), SWCMM Level V.
Longest history of process improvement in NAVAIR. Prime
contractor, Boeing, is also a Level IV. Recent developments have
been very successful at meeting cost and schedule objectives.
One of Top 5 DoD Software projects in 2001,
http://www.stsc.hill.af.mil/top5projects/
30 January, 2003
36th ADoDCAS
14
Improved Tools
SBIR N01-020, Software Cost & Schedule
Estimation:
Demonstrate processes and tools that will estimate software
cost and schedule with a standard deviation of 10% or less.
Make best possible use of limited data sample sets for
developing software cost and schedule estimates.
Enhance update of estimates based on actual project data.
Enhance collection and archival of project software metrics
and measurements in order to support calibration and improve
accuracy for both current and future projects
30 January, 2003
36th ADoDCAS
15
Galoraths Phase II
Implementation of
SBIR N01-020
30 January, 2003
36th ADoDCAS
16
Project Plan
ID
1
2
106
160
161
246
273
300
Task Name
ProjectMiner
ProjectMiner Release 1
ProjectMiner Release 1B
SEER-SEM Integration
ProjectMiner Release 2
ProjectMiner Release 3
ProjectMiner Release 4
Transition - non-iterated tasks
30 January, 2003
Finish
anuary 21
June 1
2/10
4/14
6/16
8/18
October 11
February 21
10/20 12/22
2/23
4/27
July 1
6/29
8/31
Nov ember 11
11/2
1/4
March
3/7
Mon 4/19/04
Thu 7/3/03
Tue 9/23/03
Tue 10/21/03
Mon 1/26/04
Wed 2/18/04
Tue 3/23/04
Mon 4/19/04
36th ADoDCAS
17
Data Rights
Government has standard SBIR data rights
for application.
Can view all requirements, design and source code
of product.
After 5 years, Government may go to another
source to modify application, IF:
9Galorath Incorporated has not copyrighted the application;
however, it probably will.
9Government has not contracted with Galorath for further
modifications of the application.
30 January, 2003
36th ADoDCAS
18
Sizing
Sizing made easier, and more possible earlier in the estimation process
Estimating
Decision makers will better understand the origins of an estimate
Estimate will be more accurate, more reliable, and better integrated into the
project life cycle
Impact of new technologies and specific circumstances will be captured
Planning
Estimate will be integrated into earned value and other project management
methods
An integrated estimating methodology will be supported
Data Warehousing
Past project acutals will be sorted and accessed for more accurate, more
defensible estimates
30 January, 2003
36th ADoDCAS
19
Analogies
Size new jobs against
standardized benchmarks.
30 January, 2003
+ Characteristic
Rating
- Rigor of requirements
Very High
- Number of interfaces
70-100
- Complexity
Very High
COTS Estimating
COTS Characteristic
By Characteristic
- Application Domain Specify
Aero
- Application Type
Signal
processing
Characteristic
Rating
- << Next characteristic >>
Rigor of requirements
High
COTS entry
Help
Next >>
OK
Cancel
20
Metric
Quantity
Legacy
SLOC
SLOC
ELOC
Classes
SLOC
Use Cases
COTS Sizing
SLOC
SLOC
Function Based
Detailed Objects
SLOC
1700
120
450
12...
60
2...
select for info
15000
300
select for info
45...
1200
Pre-existing
Pre-existing
Pre-existing
New
New
Pre-existing
COTS
Pre-existing
Pre-existing
Pre-existing
Pre-existing
Pre-existing
36th ADoDCAS
21
Using extremely
limited, relevant
past data.
Results obtained using 5
aviation data points and
Least Absolute Error
estimation.
Using sufficient
relevant past data.
Within 20%
Within 12%
Within
10%
Using extremely
limited, relevant
past data, with good
assumptions about
predicted project.
Results obtained using
maximum information pair-wise
comparisons, with accurate
conjecture on probable range.
results on data that would be used in a production system. All data and tests are available for independent review.
* These are actual
Please note that final results should be better, as the ProjectMiner methodology is still being prototyped.
30 January, 2003
36th ADoDCAS
22
Best
Models,
Methods
& Data
Settings
For
Current
Estimate
Estimating Methods
D
at
a
NAVAIR,
contractors,
industry,
etc
Se
t
ProjectMiner Concept
Many Approaches Make a Better
Estimate
Multiple
statistical
methods,
etc
30 January, 2003
Combined
Combined
Results
Resultsof
of
Various
Various
Statistical
Statistical
Methods
Methods
Functional Forms
y=a*xb, etc
36th ADoDCAS
23
Determine
Relevant
Data
Unlimited number
of core models
30 January, 2003
Develop
Potentially
Relevant
Equations
Adjust Best
Estimate With
Empirically Derived
Add-On Models
Intelligently
Combine
Best
Approaches
Unlimited
number of addon models
Assess
Performance
Of Each
Approach
36th ADoDCAS
24
n
ion riso r
s
res mpa othe
g
re co
NAVAIR data
Contractor data
Conformal
Conformalestimating
estimatingkeeps
keeps
unique
uniqueinfo
infoabout
aboutpast
pastprojects
projects
and
andthen
thenuses
usesthat
thatto
toimprove
improve
future
futureestimates.
estimates.
Standard
Info:
Unique
Info:
Size, Platform,
Application, etc.
Specific Tools,
Circumstances, etc.
Other data
Invaluable additional
info
RESULT:
A much more accurate estimate.
36th ADoDCAS
25
30 January, 2003
36th ADoDCAS
26
Midcourse Calibration
Calibration Dialog
Dialog lets
estimator input
earned value
information for
auto updating
Quick Estimate
MS
Project
Phase duration
estimates are
updated on the
chart, with risk
ranges (in red)
Prototype
Spec
Code
Unit Test
Midcourse Calibration
______________________________________________________________________________________
Month
3
6
9
12
15
18
21
24
30 January, 2003
36th ADoDCAS
27
30 January, 2003
36th ADoDCAS
28
Custom Lifecycles:
Data-Driven or User-Defined
Lifecycle Designer
30 January, 2003
36th ADoDCAS
29
A set of estimated
WBS elements
A set of properly
scheduled elements
Determine
Determinewhich
which
tasks
tasksshould
shouldfollow
follow
which,
which,how
howcritical
critical
the
therelations
relationsare
are
Automatic optimal
scheduling
30 January, 2003
36th ADoDCAS
30
30 January, 2003
36th ADoDCAS
31
30 January, 2003
36th ADoDCAS
32
..\..\..\business\01-016
NAVAIR SBIR N01-020 Software Estimating\Deliverables and Briefings\August 28th Briefing\scatterplot_run.avi
30 January, 2003
36th ADoDCAS
33
30 January, 2003
36th ADoDCAS
34