Hope is NOT a Valid Management Strategy Mega Projects Estimates - A History of Denial Glenn Butts February, 2010 The source of knowledge is experience. Experience is of two kinds-our own, and that of others. Knowledge gained from our own experience is most expensive. Mystery and ignorance are always coexistent Learn the truth and the mystery will disappear. ‘Special Thanks to Kent Linton for help in preparing this presentation. _Used with Permission ¢ Independence, a new 418-foot warship * Top speed in excess of 45 knots, or about 52 mph ¢ Eventually, Navy wants to build 55 of them. + Early estimates for Littoral Combat Ships were about $220 million each * Costs escalated because of new requirements and desire to expedite construction. + Cost of ships now capped at $460 million each, starting in new fiscal year. Excusable? a A— Initial delivery date May 2008 — Test flight delayed over 2 years Dreamilifter built to bring large parts to Seattle * Stil faces 12 months of testing {from around the world for final assembly * 60 Order cancelations lions in costs increases, including toes: EXxcusable? — Purchasing suppliers = Additional management oversight costs "The initial plan outran our ability to execute it,” ] ______ Bowing Chief Executive Jim McNemey Don Pm’ Boei tus ¢ Boeing’s conventional plan to build new 747-8 freighter variant * Project started in 2005 — 2years after 787 m © Variant of existing 747 Socing Nas — New wings — New Engines Major Cx = New cockpit technologies © Uses traditional materials instead of composites © Orthodox manufacturing process Contract Conventional Plan didn’t work either... * Besieged by design and engineering problems from onset * Boeing announced $1 Billion dollar charge as a result of problems in Oct 09 * Initial delivery delayed at least a year ~ Test Flight Mid January 2010 # No intention to pick on Boeing * Airbus's new A380 experiencing similar problems __| We believe that building the 747-8 should have been a tre ightforward development process on a derivative program.” Aerospace Analyst Douglas Hamed(uel JO sexy 8uQ 3sNr uoljjiw .Z79¢ Buiddoum e - }sOO Jeuly SUL — 800z Jequisseg u! aynpayos pulyeq sueah arly pajyajduoa Ayjeuly e : uoI|IW GOZ$ Jo yeBpng — ® JO} 7OOT JO EYep uoHE|dwWoD — pesiwoid e ym 00z eunr UeBeq UOI}OMU}sUOD &Bow pur payuasaidaisiw Ajayeseqijap ae seyewyse Auey — /pepuny 9 aigeiA wesBosd Buidesy s! Useou0s Aiewiid OYL — ‘sayewSe 97e1N99e JULM 3,UOP Ajjead SAA | pue ‘a|qeyiuenbun ‘xa|dwod 00} ee se|qeven Sy] — is Ajsee ye Ajjeroadso ‘ajqissodui! ase SazeLWUSa aye4NIOVY e !SUOISN|DUOD Bj|qISSOdg sjoefoid Bulyewl}se-JEpUN JO} aSNOXy — ys09 Aau} eum jsoo sBuly} Aum Jo} SuOsesi pI|eA eJe GO] — aaisuedxe 00} si onb snjeys Buluee| — ,onb snje}s 8u} 306 |j/M OM ‘pewnsse si onb snjejs jl, & eujuewW JUoWaBeUuRW + juewBeuew Aq peyioddns ae pue ‘puesnouyy UO AjUO Jo oUSUINDI0 Jo pooysja>!) & YW sysu soyeBNILW AjBAHoe WRENS — sisAjeue JNO WOJJ SYS B|GIPasD SpNjoxy e , awolaq UaYo JEU} SE}EWI}SS JNO WO. adoos ajqeqoid WO Sa}EWISO |el}IU! ONsIWUNdo Burjiwugns ul }sIsled ©Accuracy is gained at each successive stage of progress. e Promoters routinely ignore, hide, or otherwise leave out * Cost underestimation occurred in almost 9 out of 10 projects. ee This study looked at comparatively simple projects. s = Overruns above 100% are not uncommon. important project costs and risks in order to make total costs appear low. = The likelihood of actual costs being larger than estimated is 86% = The likelihood of actual costs being lower than estimated is 14% statistics for more complex projects would likely be worse! ee ae Cd t 8. OMIT PROBABLE SCOPE from estimate OMIT POSSIBLE RISKS from analysis RAND Study — Reason ¢ Intemal & External ? for 74% of Cost Growth UNREALISTIC, OPTIMISTIC assumptions J Use historically LOW ESCALATION projections * RAND Study — Reason for 11.2% of Cost Growth Issue cost estimates in BASE YEAR dollars * Estimates should be in then year dollars (escalated to year in which it is spent) Many estimates NOT PREPARED BY A BONA FIDE ESTIMATOR Everyone's a estimator Being certified no guarantee of having necessary experience REWARD failure, PUNISH honesty NOT ENOUGH TIME to prepare CREDIBLE estimates * Time often spent doing “what if" exercises, or splitting dollars into arbitrary buckets person from its consequences” - Barack Obama -io Forgotten SPACE SHUTTLE Facts ¢ Initially approved by 1 Vote é ‘* Congress WAS concemed about estimates « FY 1972 Estimate ‘+ DDT&E $5.5 Billion + 20% Contingency = $6.6 Billion — Contingency Reportedly Removed by Congress + Facilities $0.3 Billion = 18 Manned Flight projected by June 1976 (4 yrs.) ¢ Reality - 1982 “The chance of a Shuttle launch on ekcaviegste: 917.769 Billion November 20 1980 is probably 10 to 20%” — DDT&E $13.138 Billion Dr. Frosch NASA Administrator — Production _$ 3.900 Billion ‘Congressional Testimony 1980 ~ Facilities $ 0.703 Billion ‘* 1% Manned flight occurred in April 1984 (9 yrs.) ¢ Increase WAS Foreseen... ‘* “Considering all of the technological and operational unknowns involved in the shuttle development and the fact that no vehicles of similar function have ever been designed before or have ever operated over the range of fight regimes required for the shuttle, prudent ‘extrapolation of prior experience would indicate that estimated development costs may be 30 to 50 percent on the low side. Thus, the estimates of $6.5 billion in RDT&E for the Mk I/Mk I! shuttle program may range between $8.5 to $10 billion, reflecting increased program costs of $2.5 to $3.5 billion. Similar uncertainties must be considered to apply to other non-recurring Costs such as production and facilities (amounting to about $4 bilion). Thus a possible cost uncertainty of about $5 billion for total program costs might be envisioned giving a high estimate of total non-recurring cost of about $15 billion.” - October 1974 Report - Re Page 14 Realistic Assumptions ARE Critical Space Shuttle Operations Concept - 1974 Orbiter Processing Facility - Today This is what was estimated. This is what was built.Pa Jone 1962 PERT ae + Ignored possibility of unlikely events ¢ Sold as greatest thing since sliced bread + Credited for the unmitigated success of Polaris program + Actually a “gimmick” to wow Congress * Only applied to a few program activities « Was subsequently expanded to include cost ¢ Was then used to manage the Trident Program * Trident became the “Single worst managed project in US Military History” ¢ Because it did not work, it fell out of favor in the mid to late 70’s Source - The Polaris System Development, 1972 Page 17 The Real Reason for Pola is' Success ¢ The Polaris Program: Attracted top talent Created a new office with special privileges Authorized special uniforms Provided first class travel Initiated hassle free expense accounts Gave liberal promotions and awards for success ¢ The Staff was told to “Think big or get out” ¢ Required minimal paperwork and reports + Didn't burden talent with bureaucratic paperwork ¢ Got personal pledges from contractors & personnel that schedules would be met ¢ Issued swift & sure ramifications for failure + Branch heads were dismissed within 24 hours, * Contractor personnel were prohibited from returning to previous assignments ¢ Was very receptive to innovative ideas ¢ Candid honesty was actively sought ‘+ Lying cardinal crime and treated accordingly! Source ~ The Polaris System Development, 1972,‘Management CAN'T Figure Out Why * S-Curves Flatten Over Time - Can You? . > 100% 2 om © SDT Design 2 80% = Final Design| 3 4 $299M Estimate ° a with 20% Reserve Sz 60% Final Cost $551M BS 50% 2 40% = 30% i Ss Does this help = 20% Credibility? é 10% + 0% - = a $200 $250 $300 $350 $400 $450 $500 $550 $600 $650 $700 $750 $800 Total Mission Cost (FYO7SM) _ “Graphic by David Bearden ~ Aerospace Corp trom cosmace vemos st tom bit configuration _| Answer: Red S-Curve was initial project estimate from cost model based on numerous optimistic assumptions Blue S-Curve is model estimate based on as built configuration. Page 19 Possible Solutions ¢ Hold project managers accountable for the original estimate « Project Managers will THEN begin demanding credible estimates * Recognize that historically many original assumptions are OPTIMISTIC or simply NOT valid * Resist the legacy practice of labeling COST growth as "SCOPE" growth « Rather than scope "growth" historically, scope “reduction” is often what has actually occurred. Real requirements are often trimmed - NOT expanded — Space Station was originally touted with 8 functions - but cut down to 1 — Shuttle’s original requirement was for a payload of 65,000 Ibs. - Maximum current payload capability is 49,000 Ibs. — Mercury Program deleted two monkey passenger test flights - Etc., etc... OR ¢ Include all possible risks in cost projections (internal and external) * Projects tendency to spend everything available must be constrained ~ Congress budget contraints aggrivate issue — they demand that all money is spent. * Make it fashionable to be austere and frugal and encourage “under run" * It's always possible to spend more time and moneyExample of Concept Growth: Substantial Differences Exist between STEREO Initial Concept and Final Implemented Configuration spt Final ‘Schedule (months) 40 70 Launch Vehicle Taurus Delta Il Technical Mass (kg) Satellite (wet) 2n1 630 Spacecraft (dry) 134 424 Payload 69 149 Power (W) Satellite (Orbit Average) 152 503 Payload (Orbit Average) 58 116 Other Transponder Power (W) 20 60 Downlink Data Rate (kbps) 150 720 Data Storage (Gb) i 8 * Science Definition Team (SOT) Example Project With JCL PC Applied ¢ Mars Science Lab - MSL © Initial Estimate was $650 Million » PDR Estimate was $1.4 Billion (2009 Launch) © 2009 Estimate $1.9 Billion (2011 Launch) GAO — “Our work has identified a variety of reasons for cost growth, most notably that programs are incentivized to produce and use optimistic cost and schedule estimates in and 0% Complete Based on Initial Estimate order to successfully compete for funding.”[ How English Government Deals With Bad Estimates ¢ European Solution - apply an “optimism bias adjustment” factor tT cee « Factor derived from empirical data Cptinim Bia? * How to Determine: (Official Project Type Works | Capital Guidance) ee i ilable data” Upper | Lower | Upper | Lower = Use the “best available da Racine Sears ~ Develop ranges from business Se data vensandoauings | 99 | 2 | st | 4 | © “Initial estimates may require Sandal CuaEnghong | 20 | 1 | | 3 higher adjustments” t- — - - Nonstandard Ci | ~ Initial estimates should start in | Engneerng aia|el|s Mocs EquipmentiDevelopment sa | 10 | 20 | 0 Reduce adjustment factor with ~ ~ —— project progress [Ovmouog | NA | NA | at “There is a demonstrated, systematic, tendency for project appraisers to be overly optimistic. Many project parameters are affected by optimism. Appraisers tend to overstate benefits and SEACADS URE understate timings and costs - both capital and operational. ‘Source - HM Treasury - The Green Book Appraisal and Evaluation in Central Government Treasury and Supplementary Green Book Guidance - Optimism Bias Paget Conclusions + Most large projects « Projects everywhere suffer the same fate. Early estimates are optimistic - much more than commonly believed * Any early estimate for a development project that fails to consider possibility of triple digit cost growth is NOT being realistic. + No other experience is more valuable than the experience of failure, it must be shared honestly and completely — Coverups & revisionist history must be stopped in attempt to bury the truth ¢ Two requirements to prevent cost overruns. 1. Create BETTER initial estimates by real estimators Stop mandating Include all potential risks (internal and external) processes and * =~ 50% of project cost growth due to external factors reports - start Add probabilistic allowance for true unknowns rewarding success and punishing failure. IMPROVEMENTS WILL OCCUR! 2. Hold Project Managers accountable to original estimate They will stop “spinning numbers” and will demand * Qualified estimators & credible numbers + Failure analysis to determine issues ¢ Supporting research is covered in depth within the white paper * “Mega Project Estimates - A History of Denial." Please read it! * Email for a copy Glenn.c.Butts@nasa.gov Pagesit sounds impossible - consider this! The dictator of small country: @ Appears on national TV and says: @ “There will be no more drugs in this country.” @ Everyone laughs at his naiveté and says such a result is impossible. The next night 2 dozen people were lined up and shot @ The dictator appears on TV and says these people were caught with drugs. This happened every night for the next 2 weeks & The number of people shot every night diminished until there were none! © In-country rehabilitation centers filled up. Point Before things change - the approach and consequences must change! ope 32 Backup; 4 daooe ue Bunjedo| swes8q “seiesal joeloid Jo asn pIjeA SEM YOIYM ‘BAREUJSHE e1Ae} D isqad payoadxeun ey] Sel} Ajd seiq jo esiwep 84} YM ynNO peseud oe uoKes yng - jeueyew erneiqe ue se uoKel sn O} peyedxe asem $,gUs emnus ‘ejdwexg -wajqoud B awooag * ** Waqoid e aq 0} peyedxe JON ee yeu) sBuiy} Aljeuojseo9Q, bAd $$ UOIeEdO|/e AJOSAJ JUIOd = |2}0} @AJESEJ X % UONJEOOI|E OAJBSO JUIOd © % ,UOHedO||e /uesei, JUIod = ayewijse julod + yeHpnq peuueld A4 SOeMAaSOY = SJEWI}SE JUIO — S}ELUI}SE [SAB] SOUSPIJUOD %OZ pouj}ew peysebing « yoloid ajqeiaun ue Ul }/NSe ||IM JeeA jse| Ul SoAJesel {je Buidwng e poloid ay) Jo ay!) ay} JnoUBNOJU; peyeso)je A|WJOy!UN eq jsnlU SeAJasEYy SNOIAGO ay} Buneys + 32afo1g 84} JO 9417 84} jNOYBnoaYL s1qeleAy ape og SN SeAJBSay 4Cost Growth"© Average: 219% or 3.19 times the original atinnats * Median: 207% ¢ Schedule Growth * Average: 103% or 2.03 Times the original estimate * Median: 108% 400% . Growth cae 23 Mercury Program Space Shutle-Wih Space Shute poole Space Staton Reserver If we keep doing the same things the same wa} we will keep getting the same results. Specific Recommendations Ina Policy Statement: 1. Specify that all risks be included in the JCL Assessment. 2. Mandate precise criteria upon which we are being asked to provide JCL assessments. 3. Require all estimates to be created by a bona fide estimating department with direct reporting to senior management. 4, Recognize that cost control is as important as the cost estimate. Require valid cost impacts of all changes to be estimated, and not submitted as “no cost” changes. 5. Require managers to identify and publicize all valid elements/components identified by qualified personnel even if those items are not admitted or baselined by the program. 6. Require cost estimates to be submitted in future year dollars reflecting historical long term escalation rates - never in base year dollars. 7. Require that the current development stage of the program be identified on the imate. 8. Disenfranchise the risk reward system that provides strong incentives for underestimation. (Important details on this point are listed in the paper: "The Joint Confidence Level Paradox - A History of Denia 9. Remove the prevailing stigma that under-runs are unacceptabl required to make portfolio effect work. 10,Begin to use the JCL-PC estimating equation introduced in this paper. Under runs are# The Spitzer Space Telescope's was launched in 2003 * The Baseline Estimate was $472 Million ¢ It was re-baselined 4 times * The final cost was $712 Million (1.51 times more than the original estimate) ~ Another reference puts the final cost at $825 Million. ¢ On June 22, 2005 - 411 people received a Group Achievement Award * The team award was given to a group comprised of both government employees and non-government personnel for outstanding accomplishment through the coordination of many individual efforts contributing substantially to the NASA mission. Explicit consideration was given for: ¢ 1) Quality of results and level of impact on NASA programs or operations * 2) effective management of cost and schedule © 3) customer satisfaction What 22? * 4) team growth and capacity for future contribution * 5) development of innovative approaches, use of and contributions to lessons learned data banks, and/or success in responding to unforeseen crises. Sources - GAO-04-642 Lack of Disciplined Cost-Estimating Processes Hinders Effective Program Management - hitp:/wwwspitzer caltech.edu/Media/nappenings/20050707/ Page 40 ¢@ “NASA has commissioned several studies to determine primary @ There are thousands of valid studies on this phenomenon Root Causes contributing factors to cost and schedule growth. These studies, as well as others in the field, identified several common themes: e There is significant optimism in initial designs. » There are changes in scope associated with design evolution over time. © There is inherent technical difficulty developing world class technologies. e There are external influences on the project such as funding instability.” Gary P. Pullman, Vice President Aerospace Corporation 2009 « The same recurring themes always surface. Despite countless attempts to rectify what has proven to be a formidable challenge, cost overruns & schedule slips continue.ia Random Walk ¢ Projects can be compared to a random walk ¢ There are many possible destinations « Every decision made along the way affects the outcome * The final destination is not known until the walk is completed @ The estimate is usually prepared for the “Low Cost” route i he onal Excerpts of Christopher Scolese Testimony a provid ith an approach that fully integrates technical, cost, and schedule plans and risks to develop both an understanding of the sensitivity of parameters to each other and the most likely estimate « Using this approach allows NASA to understand and document how the mitigation of technical risks would enable an increase in project confidence level. + Conversely, introduction of a budget reduction would increase technical and schedule risks and thus lower confidence level for the project. ¢ Use of probabilistic estimates also generates baseline values that include funding to address impacts associated with contingencies and uncertainties, such as industrial base, partner performance and technology optimism. ¢ Introduction of probabilistic joint cost and schedule estimating puts NASA on the leading edge of applying these techniques in both the federal and space sectors. ¢ This estimating approach requires employment of new tools and techniques, full implementation will take some time; + Given the deployment and typical project development cycle of 3-5 years, it is unlikely NASA will be able to evaluate impact of changes for a few more years. ¢ Further, as we implement this joint confidence level policy, we are looking back at existing projects in development to ascertain risks and make adjustments where prudent to improve our cost and schedule posture.MILESTONES CONFIGURATION STUDIES COMPLETE PLANNED START VEHICLE DESIGN & DEVELOPMENT START HORIZONTAL TEST PROGRAM FIRST MANNED ORBITAL FLIGHT Pages ¢ People Don't Understand Risk «Recent study % people don't understand what 20% chance of rain means * Odds of making mistake lower if chance of rain and no chance of rain given ¢ Underestimates Nothing New « US 1* aircraft development project 1898 ‘+ 400% over schedule ‘+ 40% over budget * Failed & Canceled! ¢ How We Underestimate * We actively mitigate safety risks with probability of occurrence of 2000 to 1 (0.0005%) = 0r 99.95% probability will NOT happen ‘+ But routinely ignore costs risks ¢ Escalation * Civil Servant Raises past 61 Years © 10% or more 4 times a ions ~90% 2.6% rate will d (50 years of History) * Space Shuttle » Two week turnaround requirement. * 6 months historical time (Apollo, mercury) + Human Space Flight If the last 5 times | crossed the street | got hit by a car What should | expect when | cross the street again? Apollo Program Initial estimate 4-25-1958 1.5 Billion — Air Force Ballistic Missile Division NASA Created 6-29-58 Apollo 1 Fire Jan 27-1967 - 21 month delay Apollo program final costs $20-$25 BillionDate [Cost Rakee ba July 10m- [The fist cost projection provided by the Seotish Office is or housing 1997 Je4om MSPs in a new Scottish Parliament, The estimate takes no account of the location or design of any new building. |The design of Miralles is chosen and the revised estimate updates the '¢50- _ {Preliminary figure recognising that the initial projection was based on a 6 Fuly 1998) sso feleared site of 16,000 m? (172,222 sq ft) on brownfield land in Leith, [Haymarket or Holyrood. The figure does not include VAT or site I [acquisition costs aes First Minister, Donald Dewar provisionally estimates the costs at £109m, boss |f109m |The increased figure takes account of consultancy fees, site costs, \demolition, VAT, archaeology work, risk and contingencies. ce I£195m__|Cost projections increase by £86m. “| [The new figure is officially announced and takes into account increases in Ispace and major design changes resulting from a changed brief over the November 1-41; [previous year. Rising costs are also blamed on construction problems ahead| 2001 Jof an attempt to try to complete the building project by May 2003. The then| Presiding Officer Sir David Stee! informs the Finance Committee of the (Scottish Parliament that rescheduling work is increasing costs, [A cost increase to £295m in October 2002, is reported to be due to increased security needs, requiring that bombproof cladding be incorporated into designs for the external fabric of the building. Rising December b-s200m {costs are also put down to “hidden extras" in the construction process and 2002 lby December 2002 “ongoing delays” raise costs above the £300m barrier ‘The completion date for the building slips again, and plans for a grand "Opening Ceremony" are shelved indefinitely. [In July, the new Presiding Officer, George Reid produces the first of his "monthly reports" on the cost and schedule of the building, and provides a September | soo, _|figure of £373.9m. The new figure comes in the ight of reports that 2003 consultancy fees for the project top £50m. By September costs break the \£400m barrier and are blamed on construction problems in the interior of L Ihe building, [Costs are revealed to have increased again due to further problems with february 16430m + |onstruction. The official ‘opening of the building is tentatively put back id lagain to some time in 2005, however the building finally opens in October. [February [The final cost is announced by the Scottish Parliament Corporate Body, a 007, [414-4m Jreduction of £16.1m on the previous estimate.It is intended to replace both Schénefeld and Berlin Tegel Airport, and to become the single ~ commercial airport serving Berlin and the surrounding state of Brandenbur it ‘combined 6 million inhabitants. eee eae With a projected annual passenger number of around 27 million, Berlin Brandenburg Airport would become the third busiest airport in Germany, surpassing Diisseldorf Airport, and one of the fifteen busiest in Europe. Originally planned to open in 2010, the airport has encountered a series of delays due to poor construction planning, management, execution and corruption. Unfinished construction and corrective work means an opening prior to late 2016 is unlikely according to the WSP CBP time frame. Remarks made in August 2014, by airport CEO Hartmut Mehdorn, point toward 2017 or 2018. Mehdorn announced no opening date as expected 14 October 2014, so a special commission established by the Brandenburg Parliament will retain oversight of the project. Current estimates suggest that the airport will open in 2018 or 2019, at the latest. Citing dissatisfaction with construction progress, members of the airport board indicated during their 8 December 2014 meeting that they were beginning the search for a new CEO, although Mehdorn's contract runs through 2016. Current total costs amount to €5.4 billion. Additional plans suggest additional costs amounting to an extra €2.19 billion. ‘As of 3 June 2015, Germany applied for an additional €2.5 billion spending approval at the EU, in addition to the previous total of €4.3 billion. Total costs thus mount to €6.8 billion. The EU will only permit an additional €2.2 billion. ¢ to open, officials are planning a possible third runway for approximately €1 billion and other new projects such as an additional terminal, expanded baggage system and another freight facility. The total additional spending would amount €3.2 billion. The teard wamed of a further rise in costs because the airport will not open before 2017. The current time-cost frame is limited to 2016. Although the airport has yet