Septemb ber26,2013 Ms.

Rebe eccaPearce Maryland dHealthBen nefitExchange 4201Pat ttersonAven nue,4thFloor Baltimore,MD21215 5 Rebecca: :
a IV&VontheMarylandHealth H BenefitExchange(M MDHBE)proj jectistoinde ependentlyan nd Ourroleas objectivel lyassesstheDesign,Development,and dImplementa ation(DDI)of fthesystemsandsystems infrastruc cturethatsup pporttheMDHBE.Inthiscapacity, c ourteamhasauniqueunders standingofth he challengesthatthedev velopmentte eam(s)areexperiencingon nthisproject, ,theprogress smadethusfar f withdesig gn,developm ment,andtest tingactivities, ,andtherisks/issuesassociatedwithmaking m thecurrent systemsenvironment e operational. o With W thegolivedateimm minent,wefee elitisimporta antforustoshare s ourpersp pectivewithyoupriortoth hefinalgonogodecisionbeingmade e. First,Berr ryDunnrecog gnizesthetrem mendouseffo ortsboththeStateanditssolutionprov vidershave expended dsincewebecameinvolve edwiththispr rojectinOcto ober2012,an ndparticularly yoverthepastsix monthsas stheOctober1duedatehas h grownclo oser.Welearnedearlyonduringourin nvolvementwith w theprojec ctthatourad dviceandguid danceshouldnotmerelyadhere a toindustrystandar rdsandbest practices,butratherfo ocusonpract ticaladviceth hatrecognizes stheamount toftimeavail lable,resourc ce constraint tsfacedbyth heproject,lac ckofclearguidancefromthe t federalgo overnment,and a themyria adof otherchallengesthatare a beyondth hecontroloftheprojectte eam. Wealsoknow, k thatwh hendocumen ntingrisksand dissues,theMD M HBEhaschallenged c us stofocusour r energiesto t provideeff fectivemitiga ation(risks)andresolution n(issues)reco ommendation ns.Wehave workedwith w youclose elytodevelop paprocessby ywhichtheMD M HBEcouldidentifywhic chrisksandis ssues wehavedocumented d that t theMDHBEaccepts,becauseforonereasonor o another,theMDHBEdo oes notplanto t workonmitigationorre esolutionacti ivities.Theintentofthisle etterisnotto ofocusonthe e detailedrisks r andissue esthathavebeen b docume entedinourmonthly m report,butrather rtoprovideyou y a highlevel lsummaryofthingswerecommendyo ouconsideras syoumakeyo ourfinalgo/ /nogodecision. vethedecisio onyouneedtomakeisfoc cusedondete ermininghow wfartoreduce ethescopeof o the Webeliev functionalityofthesys stemfortheOctober O 1,20 013golive.On O September r16,IV&Vlea arnedthatthe eMD HBEwasplanning p togo olivewithth heminimumamount a offun nctionalitypo ossible(Accou untCreation).On Thursday, ,September19 1 wecalledJustin J tocom mmunicatetha atIV&Vfullysupported s thedecisionto golivewiththeminim malfunctionali itypossible.We W mentione edtoJustinon nthiscallthatwehadbee en pingaletterdescribing d ourconcernwit ththeimport tanceofgoing glivewiththe e contemplatingdevelop minimumamountoffu unctionalitypossible, p butthat t basedon nthedecision nmadebyMD DHBE,weno o longerfelttheletterwas w necessary y.OnSeptemb ber24welea arnedthatthe eMDHBEisplanning p togo olive withwhat tMDHBEbel lievesistheminimum m amo ountoffunctionalitypossibleandthiswill w include AccountCreation, C Enro ollment,andPlanSelection.

Ms.RebeccaPearce September26,2013 Page2

WerecommendthatyoucontinuetoconsiderwaystoreducethescopeoftheOctober1golivetothe minimumamountoffunctionalitypossible.Ourmonthlyreportincludesmanyrisksandissuesthat supportthisrecommendation;however,wedidnotfeelitwouldmakesensetosimplyrepeatthe concernswehavebeenraisinginourmonthlyreport.Instead,belowareeightkeyreasonsforour recommendationbasedonourcurrentunderstanding,presentedinnoparticularorder: 1. Overthecourseofthelastfewmonths,problemsandchallengeswiththeDDIhave continued,andalthoughprogresshascertainlybeenmadewithdevelopmentandtesting activities,veryfewactivitieswentexactlyasplannedorwithoutidentificationofunexpected problems. Goinglivewiththeminimalfunctionalitypossiblehelpstoreducetheriskthat unexpectedproblemswillarise.Thiswillprovideadditionaltimetoensuredevelopmentthat hasoccurredinthelastfewweekswillnotintroducesignificantand/orunplannedproblems priortothisfunctionalitybeingmadeavailabletothepublic. 2. Wearenotawarethatsecurityhasbeenfullyimplementedand/orappropriatelytested. Althoughathirdpartycompanyreviewedthesystemsenvironmentfromahardwareand intrusionsecurityperspective,wedonothaveevidencethatapplicationanddatalevelsecurity featureshavebeenadequatelytestedandbeenproperlyconfiguredforliveoperation. Minimizingtheamountoffunctionalityavailableonthefirstdayofoperationallowsformore timetotestandconfigurecriticalsecurityfunctions.Securityconcerns,particularlyrelatedto data,shouldbeparamounttoeverydecisionmakingprocess. 3. WeareconcernedthatCMStestingactivitiesmayhavesetthebarverylowandinfact,could bepotentiallymisleadingwhensettingexpectationsforspecifictestingactivitiessuchas regressionandendtoendtesting. AlthoughIV&Vhasbeenverycarefultowitnessand respondtoallCMStestingactivities,andthemajorityofthesehavepassedbasedonthe definitionsofthetestsprovidedbyCMS,wedonotbelievethatthesetestsreplacetheneedfor conductingcompleteregressionandendtoendtestingactivitiesonallaspectsofthesystem Marylandplanstomakeoperational.Wealsobelievethatthesetestsmightbeprovidingthe MDHBEwithfalseperceptionsthatthesystemhasbeenfullytestedendtoendorthat regressiontestshavebeenfullycarriedoutwithallnewcodechangesbeingintroduced, whichIV&Vhasnotseen.Goinglivewiththeminimumamountofsystemfunctionalitywill provideadditionaltimetotestcriticalfeaturespriortotheseadditionalfeaturesbeingmade operationalanditwillalsohelptominimizethepotentialimpactofunexpectedproblems. 4. IV&Vhasnotseentestsortestresultsthatsimulatetheproductionenvironmentandthereis insufficienttimeremainingtotestthefrozencodefromasystemperformance,stress,and volumeperspective. Systemperformance,stress,andvolumetestingisparticularlycriticalon projectswheretransactionsrelyonexchangingdatabetweenmultiplesystems.Weare concernedthatdevelopmenthascontinuedtotheverylastweekpriortogoliveandtherehas notbeentimefortestingtoprovethatthesystemwillbecapableofsupportingthenumberof

expectedusersorthatperformancewillbeacceptableintheliveenvironment. Minimizingtheamountoffunctionalityavailableonthefirstdayofliveoperationwill helptoreducetheimpactofunanticipatedproblemsrelatedtoapplicationorhardware performance.

Ms.RebeccaPearce September26,2013 Page3 5. UserAcceptanceTesting(UAT)concludedpriortodevelopmentbeingcomplete,did nottestallsystemfunctionalityplannedforgolivewithinUAT,andwasnever conductedinanenvironment(neithersoftwarenorhardwareenvironments)thatwill bemadeoperational.Goinglivewiththeminimumamountofsystemfunctionality providesadditionaltimeforUATtestingofapplicationfunctionalitypriortobeingmade operational. 6. Theapplicationfunctionalitythatwillbemadeoperational,toourknowledge,hasnot beenfullytestedinaproductionlikeenvironment.Goinglivewiththeminimum amountofsystemfunctionalityprovidesadditionaltimetotestcriticalfeaturesina productionlikeenvironment,priortothembeingmadeoperationalandhelpsto minimizetheimpactofunexpectedproblemsshouldtheyoccur. 7. Toourknowledge,anumberofcriticalsystemareashavehadminimalornotesting, forexample,userinterfacesingeneral,caseworkerportal,security,interfaces,and notices(thisisarepresentativelist).Goinglivewiththeminimumamountofsystem functionalityprovidesadditionaltimetotestcriticalfeaturesinaproductionlike environment,priortotheseadditionalfeaturesbeingmadeoperational. 8. Eachtimenewcodehasbeendeployed,thedatabasehastoberecreatedanddata reloaded.Shouldthissystembemadeoperational,itsimperativethataprocessfor softwarechangemanagementexistthatdoesnotimpactthereliabilityortheintegrity ofthedatacollectedbytheExchange.Goinglivewiththeminimumamountofsystem functionalityminimizesthepotentialimpactofthisrisk. BerryDunnisavailabletoyoushouldyoulikeustoparticipateinadditionaloperational readinessdiscussions.Wearepreparedtocontinuetorollupoursleeves,andworkwithyou diligentlyinanefforttosupportthebestpossibleoutcomeforthisprojectonOctober1. Regards, CharlesK.Leadbetter,PMP Principal Cc:JustinStokes,KevinYang