Thankyoufortheopportunitytospeaktodayaboutanimportanttopic.

ThisconferenceisprimarilyaboutapplyingEarnedValuetoprogramsinanattemptto
managetheirperformance.Tokeepthemonschedule,controltheircosts,andassurethat
thedeliverablesarecompliantwithtechnicalperformancerequirements.
ButhavingEVcorrectlyperformanceitsroleisanecessarybutnotsufficientconditionfro
success.
St ti
StartingwithacrediblePerformanceMeasurementBaseline(PMB)movesustowardthe
ith dibl P f
M
t B li (PMB)
t
d th
sufficientcondition.

Illpresenttodayeight(8)corestepsindevelopingacrediblePerformanceMeasurement
Baseline(PMB).
Theseappeartobeeasystepsinprinciple.Inpracticeofcourseallthingsarehardwhen
itcomestosuccessfullymanagingprograms.
Butletsstartwiththeprinciples.AndletsalsorememberYogiBerrasquote
Intheorythereisnodifferencebetweentheoryandpractice.Inpracticethereis.
Soletstalkalittlebitaboutthetheory,butfocusourtimehereonthepracticeofbuilding
acrediblePerformanceMeasurementBaseline.

Glen B. Alleman
Lewis & Fowler, 8310 South Valley
Highway, Englewood CO80112
www.lewisandfowler.com

Thismaybeanewconcept,thisisthephraseProbabilityofProjectSuccess(PoPS),willbe
ourguidinglighthere.IftheprocessesdescribedheredontincreaseourProbabilityof
Success,then
Youmayassumethatallprojectsaresuccessful,butweallknowthatsnotthecase.
Therearemanyreportsabouttroubledprojects.Infactsomefirmsmaketheirlivingby
reportinghowbadprojectsare.Itisassumedthenthatyourprojectwillbetroubledas
well.
Butletsinvertthatnotionandlookatprojectmanagementfromanotherpointofview.
Howdoweincreasetheprobabilityofsuccessofourproject?Wevebeentoldourproject
isnotdoingwell,buthowcanweimprove?
Thereareseveralimportantwordshere:
Increase
Increase makebetter.
make better
Probability thestatisticalchancethatsomethingwilloccur.
Success thebeneficialoutcomeofsomekindofeffort.
Soifwewanttoincreasetheprobabilityofourprojectssuccesswhatdowehavetodo?
Buttheresonewordthatkeepsrecurring DONE.
ItsreallyDONEthatwereafter.Thatswhatthecustomerbought,thatwhatthecustomer
wantstohappenfromyourproject.

TheexistenceofthePerformanceMeasurementBaseline(PMB)isnecessarybutnot
sufficientforprogramsuccess.IntheEVMIGthereismentionofthebudgetplanfor
accomplishingthework.
Thefirstquestionis whatwork?
IfwestartwithaSystemsEngineeringparadigm,thecoremeasurementsofMeasuresof
EffectivenessandMeasuresofPerformance
MOE
MOE Operationalmeasuresofsuccessthatarecloselyrelatedtotheachievementof
O
ti
l
f
th t
l l
l t d t th
hi
t f
themissionoroperationalobjectivebeingevaluated,intheintendedoperational
environmentunderaspecifiedsetofconditions:(1)statedfromthecustomerpointof
view;(2)focusedonthemostcriticalmissionperformanceneeds;(3)independentofany
particularsolution;(4)actualmeasuresattheendofdevelopment.
MOP Acriterionusedtoassessfriendlyactionsthatistiedtomeasuringtask
accomplishment Measures that characterize physical or functional attribute relating to
accomplishment.Measuresthatcharacterizephysicalorfunctionalattributerelatingto
thesystemoperation:(1)supplierspointofview;(2)Measuredunderspecifiedtesting
oroperationalconditions;(3)assessesdeliveredsolutionperformanceagainstcritical
systemlevelspecifiedrequirements;(4)riskindicatorsthataremonitoredprogressively.

Building a credible PMB starts with identifying the architecture of the IMP and the
supporting tasks in the IMS.
Although this is restating the obvious the process to do this is actually quite hard.
Adding schedule and cost risk identification and mitigation to the process is the
minimal result for a winning proposal.
It cannot be emphasized enough the architecture of the IMS is critical to
id tif i a risk
identifying
i k tolerant
t l
t schedule.
h d l The
Th rats
t nest
t approach
h iis simply
i l unacceptable
t bl
to the success of any program.

Thetermcredibleisimportant. Credibleisnotthesameasaccurate,precise,orwords
likethat.Crediblemeansitisatrustworthymeasure believable.
Thereareotherdefinitions,buteachshouldsupportthenotionthattheinformation
providedinthePMBisatrustedbasisofdecisionmaking.
ThismeansifImakeamanagementdecisiononthisdata,Iwontbedisappointedlateto
findoutthedatawasnotcorrect.

Thislistcomesfromguidanceinthedefenseprogrammanagementbusiness.Whilesome
ofthestatementmaynotbeappropriateforthecommercialworld,everystatementhas
someapplicabilitytoeveryproject notmatterwhatthedomain.Whetherformalor
informaltheprogrammaticriskassessmentofaprojectneedstoaskoranswerthese
statements.
Theactionsneededtocloseanygapsfromthesestatementsareoutsidethescopeofthis
presentation.
Butthenextstepistohavetheprojectmanagementteamstarttoanswerthesequestions.

HeresthetoplevelelementsofthePerformanceMeasurementBaseline.
Thecustomerboughtaproduct.Whatisthetopologicalarrangementofthatproductin
theformofaProductTree.MILSTD881AintroducesthenotionofaProductTreeasthe
basisofacredibleWorkBreakdownStructure
Theteamsthataregoingtopproducetheproductmustbedefined.Theartifactofthis
definitionistheResponsibilityAssignmentMatrix
TheStatementofWorkfootandtiestheproductswiththedeliverables
Th St t
t f W k f t d ti th
d t ith th d li
bl
ThecostbasisofestimateisderiveddirectlyfromtheworkdefinitionattheWork
Packagelevel
TheIntegratedMasterPlandefineshowtheprogramstrategywillbeexecuted.
TheIntegratedMasterScheduledefinesthesequenceofworkthatmustbeperformedto
implement the strategy
implementthestrategy.

Heresour8stepstobuildingthePerformanceMeasurementBaseline.
Eachstepshouldbeperformedintheorderdefinedhere.
1. Definetheproductorservicetobedelivered
2. Definethematurityassessmentpoints theplaceswherethepredefinedtechnical
performancemeasureswillbeassessed
3. Definetheaccomplishmentsneededtomovetheproducttoapredefinedlevelof
maturity
4. Definethemeasurementcriterianeededtoperformancethematurityassessment
measurements.Thesearetheexitcriteriafortheindividualworkpackages.
5. Definethoseworkpackages.Keeptheminsideadurationthatanswersthequestion
howlongareyouwillingtowaitbeforeyoufindoutyourelate?Mostfirmslimitthe
work packages to 60 calendar days
workpackagesto60calendardays
6. Connecttheworkpackagesinasequencetoprovidesincreasingtechnicalmaturityand
minimalexposuretoprogrammaticandtechnicalrisk.
7. AssembleallthisinformationintoanIntegratedMasterSchedule
8. Dothisintegrationinanincrementalmanner oneprogrameventatatime.

HerearetheprimarypartsofthePerformanceMeasurementBaseline(PMB)andthe
orderinwhichtheyaredevelopedandusedbysubsequentparts.
ThepartsshownherearethemacrolevelelementsofthePMB.Thedetailsoftheir
individualconstruction,content,semantics,andsyntaxareoutsidethispresentation.
ThekeyconcepthereisthatthePMBflowsdownstartingwiththeWBS/CWBS.This
conceptisrepeatedoverandoverinthispresentation.Itcannotbesaidenough no
credibleWBS,nocrediblePMB.
OnecriticalelementinthisprocessistheWBSDictionary.Thisisadescriptionofthework
productsoftheWBSelements.TherearemanyexamplesofgoodWBSdictionaryonthe
web.Googlewillfindthemforyou,evenanavionicssystemWBSdictionary.Usethese
exampletodeveloptheOH58dictionary,checkitagainstknowworkingexamples,and
improvethecurrentWBSdictionaryfromtheseexamples.
ThedictionaryshowsthehierarchicalrelationshipoftheelementsanddescribeseachWBS
The
dictionary shows the hierarchical relationship of the elements and describes each WBS
elementandtheresourcesandprocessesrequiredtoproduceit.Italsoprovidesalinkto
thedetailedtechnicaldefinitiondocuments.TheWBSdictionaryshouldberoutinely
revisedtoincorporatechangesandshouldreflectthecurrentstatusoftheprogram
throughouttheprogram'slife. from2.2.3.1ofMILSTD881A

10

Thereareactuallythreeperformancemeasurementbaselinesthatneedtobeestablished.
1. TheTechnicalBaseline thisiswhatisbeingbuiltbytheprogram.
2. TheScheduleBaseline thisishowweregoingtobuildit
3. TheCostBaseline thisishowmuchitwillcosttobuildwhatweneedtobuild

11

Soletstryouttheseideasonasemirealenviornment

12

HeresapictureofactualoutputacapabilitiesdevelopmentsessionforamajorERPsystem
integrationprogram.
Startingintheupperleft,theneededcapabilitiesarecapturedthroughaProduct
DevelopmentKaizenwiththestakeholdersinanoffsitesession.TheseKaizensarefull
contactmeetingswheretheparticipantsworkonthewalltorevealthecapabilitiesand
theattributesofthesecapabilities.
Thesestickynotesarethencapturedinsomeorganizingtool.Myfavoriteforthislevelof
detailisMindJetsMindManager.Itisahierarchicalorganizingtoolthatcanexportits
structuretoMSFTProject.
Nomatterwhattoolyouuse,youvegottohavesomewaytodiscoverthebusiness
capabilitiesbeforemovingthenextstageoftheproject.
Withoutaclearandconcisedescriptionoftheneededcapabilitiesyoullhaveahardtime
recognizing Done
recognizing
Done whenitarrives
when it arrives ifiteverarrives.
if it ever arrives
Weveallheardof,orpossiblyusedasystemthatmetalltherequirementsbutfailedto
providethebusinessvaluethatwaspromised.
ThedevelopmentoftheCapabilities preferablyinaConceptofOperationsdocumentis
thefoundationfortheremainingeffortsinincreasingtheprobabilityofsuccessforyour
project.
WiththeCapabilitiesinhand,thetechnicalandoperationalrequirementsbecomeclear
oratleastclearer.

13

TheelementsoftheIMPandIMSalongwiththecostandtechnicalperformancemeasures
aredrivenbythefollowingsourcesofdata.
TheStatementofWorkisthestartingpoint
TheWBS/CWBSdescribestheproductstructure
Taskinworkpackagesshowtheworkactivities
TheCDRLsguidethedeliverablesatleastthroughPDR
AccomplishmentCriteriaaretheexistcriteriafortheWorkPackages
SignificantAccomplishmentscollecttheworkproductsneededtomovetheprogramto
thenextlevelofmaturity
TheProgramEventsaretheassessmentofthismaturity
y
g
theanswerishere
Whensomeoneaskswhyarewedoingthis

14

Whenwespeakaboutincreasingmaturity,evolutionarydevelopmentandmeasuresof
physicalpercentcomplete onegoodwaytovisualizethisisthroughaproductmaturity
flowdiagram.
ThisdiagramisarealonefromaclaimsprocessingERPsystemrollout,completewith
COTS(CommercialOffTheShelf)integrationwithlegacysystemsandcustomsoftware
development.Theworseofbothworlds.
Thecriticalsuccessfactorhereisthedefineupfrontwhatdonelookslikeforallthe
Thecriticalsuccessfactorhereisthedefineupfrontwhat
done lookslikeforallthe
incrementalbusinessvalueelementsoftheprogram.
Eachofthecirclesisapointofmaturity,wherethedeliveredproductorservicecanbeput
touseinsomeway.
Aswelltheprojectcanbecanceledateachoftheseandtheinvestmentputtowork.Many
peoplewouldbeunhappy,butthefinancialaspectsoftheprogramwouldremainintact.
Thistypeofdiagramalsoshowsthedependenciesbetweenthedownstream(right)andup
stream(left)elementsoftheproject.Whathastocomefirst,beforebusinessvaluecanbe
delivered.
Inotherdomains(defenseandgovernment)thesediagramsarebuiltduringtheProduct
DevelopmentKaizenprocesstoshowwhatmustbedonetogettodone.

15

Nowletsputallthistogether.
StartingwiththeWBS,theterminalnodesarerepresentedinprojectwithWorkPackages.
YoucannothaveaprojectwithoutsomeformofaWorkBreakdownStructure(WBS).Buildinga
goodWBSisadaylongworkshopallinitself.
SohowmanyherehaveaWorkBreakdownStructurefortheirproject?
Thismaynotbeatermyouveheardofbefore.AWorkPackageisalumpofworkthatproducesa
singleoutcome.Apackageofworkthatproducessomething.
ForthescheduleputtingtheWorkPackagesintheproperorderisthestartingpointforacredible
For
the schedule putting the Work Packages in the proper order is the starting point for a credible
schedule.IfyoucangettheWorkPackagesintherightorder,withtheirdurationsdefined,then
youhavethestartofthecredibleschedule.
ThenextstepistonotdonaymoreschedulinginMSFTproject.InsteadlettheWorkPackage
managerlookhavetheactivitiesinsidetheWorkPackageandbedonewithit.
ThisishowlargeDefenseandSpaceprogramsschedule.Therearethreelevelsofschedules
mandatedbyagovernmentrule,DID81650.
Forthecommercialsidethereisnomandatedregulation.But881AandthePMIWorkBreakdown
g
StructureGuidearethebestplacestostart.
TheMasterschedule atoplevelpicturewhatishappeninginwhatorder.
TheIntermediateSchedule thesequenceofWorkPackages.
TheDetailedSchedule thedaytodayactivitiesoftheproject.
ForITprojects,theIntermediatescheduleisasweetspot.Onethatconnectscostwithworkand
deliverables.Onethatminimallyimposeseffortontheteamandfitswellwiththeagileworldof
Corporate IT software development
CorporateITsoftwaredevelopment.
Sinceagileisfocusedondefiningdeliverablesandmeasuringprogressthroughworkingsoftware,it
isanaturalfitforaWBS.

16

Nowletsputallthistogether.
StartingwiththeWBS,theterminalnodesarerepresentedinprojectwithWorkPackages.
YoucannothaveaprojectwithoutsomeformofaWorkBreakdownStructure(WBS).Buildinga
goodWBSisadaylongworkshopallinitself.
SohowmanyherehaveaWorkBreakdownStructurefortheirproject?
Thismaynotbeatermyouveheardofbefore.AWorkPackageisalumpofworkthatproducesa
singleoutcome.Apackageofworkthatproducessomething.
ForthescheduleputtingtheWorkPackagesintheproperorderisthestartingpointforacredible
For
the schedule putting the Work Packages in the proper order is the starting point for a credible
schedule.IfyoucangettheWorkPackagesintherightorder,withtheirdurationsdefined,then
youhavethestartofthecredibleschedule.
ThenextstepistonotdonaymoreschedulinginMSFTproject.InsteadlettheWorkPackage
managerlookhavetheactivitiesinsidetheWorkPackageandbedonewithit.
ThisishowlargeDefenseandSpaceprogramsschedule.Therearethreelevelsofschedules
mandatedbyagovernmentrule,DID81650.
Forthecommercialsidethereisnomandatedregulation.But881AandthePMIWorkBreakdown
g
StructureGuidearethebestplacestostart.
TheMasterschedule atoplevelpicturewhatishappeninginwhatorder.
TheIntermediateSchedule thesequenceofWorkPackages.
TheDetailedSchedule thedaytodayactivitiesoftheproject.
ForITprojects,theIntermediatescheduleisasweetspot.Onethatconnectscostwithworkand
deliverables.Onethatminimallyimposeseffortontheteamandfitswellwiththeagileworldof
Corporate IT software development
CorporateITsoftwaredevelopment.
Sinceagileisfocusedondefiningdeliverablesandmeasuringprogressthroughworkingsoftware,it
isanaturalfitforaWBS.

17

Thecollectionofworkpackages,theirrelationshiptotheexitcriteria,andthentothe
SignificantAccomplishmentslandingontheProgramEventisthetopologyofthe
PerformanceMeasurementbaseline oncethebasisofestimateandtheTechnical
PerformanceMeasuresareadded.

18

Acriticalsuccessfactor foranyprogramistomakevisibletheflowofincreasingmaturity
foreachdeliverable.Itisnecessarytoshowhowtheworkeffortsaresequencedto
producethedeliverables.Intheabsenceofthemeasureofincreasingmaturitythese
workeffortshavenotunitsofmeasure.
Theresultisprogressismeasuredasthepassageoftimeandconsumptionofresources.
WhatisneededistomeasureprogressinunitsofplannedcompliancewiththeTechnical
PerformanceMeasures.NotafewTPMsfortheprogramasawhole.Butaplanned
measureofcomplianceforeachWorkPackageoutcome.
Thisapproachassuresthatadescriptionofwhatdonelookslikeisavailableforeach
workeffort.

19

HeresasampleofaMicrosoftProjectfilethatmodelsthestructureshownafewslides
back.

20

One of the new ideas about building the PMB is to only connect the Work Packages
together instead of the Tasks. Tasks within the Work Package can be
interconnected.
But between the Work Packages, if there is interconnections the work on the right
will be using partially completed work on the left.
If this is done, then the work package on the right will absorb this partially
completed work, and when the work on the left is completed, re-work
re work may be
needed of anything changed.
Treat the work packages as complete lumps of work, with 100% fidelity to the
planned deliverable.
This way the work package on the right can start with fully formed raw materials.

21

ThesameapproachshouldbetakenfortheSignificantAccomplishmentsandtheProgram
Events.
PartiallycompletedworkshouldnotcrossaProgramEventboundaryunlessitislonglead.
Otherwiseyouaremovingtothefuturereworkfromthepast.

22

Heresanexampleofhowthislooksinarealproject,usingthePERTChartExperttoolfrom
www.criticaltools.con

23

ThemanagementofriskishowAdultsmanageprojects TimLister
TheIntegratedMasterSchedulemustshowhowriskisbeingretiredifthatscheduleisto
beconsideredcredible.
Withoutthisdescription,whentheriskturnsintoanissue,therewillbelittletimetomake
thecorrectionandmistlikelythereserveassignedtotheriskwillhavebeenconsumed.
RiskRetirementisbetterthanRiskMitigation.

24

Thecoreelements ofanyprogram cost,schedule,andtechnicalperformancemeasures

arerandomvariables.Theyhavevarianceandstandarddeviations.Thesestatistical
behaviorsaredrawnfromanunderlyingprobabilitydistribution.
Withouttheunderstandofthestatisticalnatureofthecost,schedule,andtechnical
performancemeasures,theinformationproducedduringtheprogramsexecutionisnot
onlyuntrustworthy,itisnotcredible.

25

ToproduceaCrediblePerformanceMeasurementbaselinewemustdomorethan
announcewehaveacredibleperformancemeasurementbaseline,wemustactually
demonstratethiscredibilityinunitsofmeasuremeaningfultothecustomer.

26

Whenwesaycrediblewhatdowereallymean?
What aretheunitsofmeasureofcredible?

27

Ifwedontunderstandtheunderlyingstatisticsandmoreimportantly thatthesestatistics
drivetheprobabilisticbehavioroftheplan,therelatedcostsandthetechnicalperformance
aswell,thenweareworkinginthedark.
Rememberthepreviousdiscussionofriskandcredibility.
Knowingtheunderlyingvariancesfromtheunderlyingprobabilitydistributionfunctionsis
thebasisofcredibility.
With t thi i f
Withoutthisinformationonlya50/50guessispossible.
ti
l 50/50
i
ibl

28

Analysis of Probabilistic Schedule and

Cost Models

Last Updated: 11/2/2009

When we speak of probabilistic risk analysis, we also need to speak of the statistical
nature of the activity network.
When we speak of a probabilistic activity network (a Bayesian network) we also
need to speak in terms of probability.
A question that can be asked of the network is what is the probability of
completing this task by a certain date?
A second
d question
ti th
thatt can be
b asked
k d iis what
h t are the
th underlying
d l i statistics
t ti ti off th
the
activities of the network?
A final question that needs to be asked is what is the inherent uncertainty in these
estimates?
In other words how good is our ability to guess in the presence of a statistical
process?

Prepared by Glen B. Alleman

December 2005

29/176

Sonowwerebacktotheproblemofmeasuringcredibilityinunitsmeaningfultothe
customer.
Heresthebeginningofthismeasurement.
1. Doweknowtheunderlyingstatisticalbehaviorofthecost,schedule,andtechnical
performancemeasures?
2. Dowehaveacrediblearchitectureforproducingandassessingtheincreasingmaturity
oftheproductsorservices?Notjustassessingbutalsoproducing.
f th
d t
i ?N tj t
i b t l
d i
3. Canwedefinewhatdonelookslikeinmeasuresofeffectivenessandmeasuresof
performance?

30

SowhathappenedtoalltheEarnedValuestuff.ThatswhatwearehereforEarnedValue.
Itshere,butnowithasnewmeaning.
Actuallyithasadifferentmeaning.
EarnedValuereportsperformancefromthepast.
Whatwewantofforecastsoftheperformanceinthefuture.

31

Andweneedtoconnectthatforecastoffutureperformancetothetechnicalperformance
ofproduct.
f
d
Beingonscheduleandonbudgetbutbeingoutofcompliancewiththetechnical
performancemeasuresisnotgood.Moreworkisnowneededtogettheproducton
spec.
Somemightsaythatoffspecisrecognizable.Butiftheperformancespecification(MoP
orMoE)arestatisticalcompliance,itsnotthatstraightforward.
Wemustunderstandwhatitmeanstobestatisticallycompliant.Thisisastatisticalprocess
controlissue.InwhatStandardDeviationarewecompliant.Itsdifferenttobeinthe1st
standarddeviationthaninthe2nd standarddeviation.
Thispresentationistooshorttodiveintotheissuesaroundprobabilisticcompliance
models,butrememberissimplestory
Whatisthemostlikely(themode)oftemperaturesinTrinidadTobagoandCody
Wyoming?
Wyoming?
Welltheyarethesame78
Whatisthevariance?
Wellthatsdifferent
InTrinidaditsabout 5
InCodyits
In Cody its 50
50 to+20
to +20

32

Sonowwecanunderstandthatweneedallthreemeasures
Cost,schedule,andtechnicalperformancetoestablishthebasisofcredibility.
Eachmeasuremustbestatisticallyadjustedforitsparticularprobabilitydistribution.
Eachmeasuremusthaveamodelofhowitinteractswithalltheothermeasures.
Eachmeasuremustdescribetheworkprocessesthatareusedtoincreasethematurityand
thereforetheprobabilityofprogramsuccess.
Asimplestaticmodeltheprogramisnotcredible.Adynamicmodelisthebeginningofthis
credibility.

33

34