Application Note

LTE and EPC Test An Overview of Test Concepts and Tools for Trials
Table of Contents
Specific LTE Test Areas 3 Overview3 Gainingdeeperinsights4 Performancetesting5 Single-userthroughputtesting5 Cellperformanceverificationwithmulti-userthroughputtesting 6 Realisticmulti-userthroughputtesting8 Idle-to-activetransitiontimes9 Latencytesting10 KPIverificationandcalculation10 ValidatingLTEvoice12 Signalingvalidation 13 VoiceQoSandQoE(MOS) 13 TestingQoSandQoEofLTEstreamingvideo14 EvaluatingLTEMIMOandfrequency-selectivescheduling 15 Testingnetworkcoverage16 TestingLTEhandover16 ValidatingLTEbackhaul18 VerifyingLTEhandsetIOT18 Validatingdeviceconfiguration18

WEBSITE: www.jdsu.com/test

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Appendix: Outline for a Basic Phase 1 LTE Test Plan 20 Overview20 Achievabledataratesandlatency:single-userthroughputforUL/DLandTCP/UDP20 Achievabledataratesandlatency:cellthroughputandMUthroughputforUL/DLandTCP/UDP21 Achievabledataratesandlatency:latency21 Intra-LTEmobility:mobilityandhandoverperformance 21 Achievabledataratesandlatency:applicationperformance 22 Coverageandcapacityradiofeaturesefficiencyandgainassessment:linkbudget 22 Coverageandcapacityradiofeaturesefficiencyandgainassessment:scheduler22 Evaluationofantennaconfigurationoptions23 Self-configurationandself-organizingnetworkfeatures 23 Evaluationoffrequencyreuse:onedeploymentscenario23 BasicQoS:userdifferentiationbetweennon-GBRuserswithdifferentQCI 24 BasicQoS:user-differentiationbetweenGBRandnon-GBRusers24 Basicapplicationperformance:webbrowsing,streaming,voicecalls,e-mail,VPN,on-linegaming25 References 26 EPSspecificationreferences26 3GPPreferences26 NGMNreference28 ETSIreference28 Glossary 29

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Specific LTE Test Areas

Overview Creatinganoverallframeworkforthetesting,evaluation,andoptimizationofLTEandSAEisalargeand complextopicThisdocumentprovidesastartingpointthatcoversthetopicatarelativelyhighlevel LookingacrossthelifecycleofatechnologysuchasLTE/SAE,thetools,processes,andmeasuresmustbe tailoredtosuitorganizationalprioritiesatspecifictimeswithinthelifecycle(Figure1)JDSUprovides acost-effectivesetofsolutionsthatenablereusingassetsacrossthelifecycleThisensuresnotonly completecoveragebutalsothereuseofresultsandassets,leadingtoasolidreturnoninvestment(ROI)

Technology eld trials

Lab trials

Field trials and vendor evaluation

Friendly customer trials

Commercial launch

Optimzation and wider deployment

Figure 1. A simple model of the LTE deployment lifecycle

TheexampleslistedbelowaddressdifferentaspectsoftestingatvariousstagesintherolloutofLTE/ SAETheseareofferedasanoverviewofthemajorelementsoftheLTEdeploymentlifecycle  Technologyfieldtrials  EvaluatethetechnologyagainstNGMNorotherindustryrequirements  Testandvalidatetechnologyimplementations  Adoptaprocessthatismoreopenthanfullyclosedbilateralfieldtrials  Labtesting  EvaluateeNodeBschedulingperformance  Evaluatesecurityandbillingpolicies  Evaluateserviceperformanceincontrolledenvironments  EvaluateUEperformance  EvaluateMIMOperformancegains  Fieldtrialsandvendorevaluation  Evaluateend-to-end(E2E)performance  Evaluatenetworkcoverage  Evaluatecellandnodeperformanceunderloadedconditions  Evaluateself-optimizingnetworkcapabilities  EvaluateIRATcapabilitiesandperformance  PerformKPImonitoringandbenchmarking

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

 Friendlycustomertrials  Understandend-userQoE/QoS  Confirmhistorictroubleshootingcapabilities  TesthandsetIOT,conformanceandpre-conformance  Monitorhandsetperformance  Includetrending,statisticsandVIPreporting  Commerciallaunch  Performservice-andaccesstechnology-awaremonitoring  VerifyE2Evisibilityandtroubleshootingcapabilities  Validateandunderstandsubscriberbehavior  UnderstandoverallnetworkandIRAThandover(HO)performance  Includeintegrationwithnode-loggingcapabilities  Optimizationandwiderdeploymentphase  ValidateandensureSONcapabilitiesareworking  Locateareasforexpansion  Monitorbackhaulperformanceimpactonend-userQoE  Benchmarkserviceperformancebetweenmacroandfemtoroll-out TheremainderofthisdocumentfocusesonthefieldtrialsportionoftheprocessEachsection describesthetestcasesmostoperatorswillwanttoundertakeandalsooutlinesthetoolsthatwillensure effectiveperformanceofsuchtestsItisassumedthatnotalloperatorswillperformallstepssowillneed tomatchthetestcases(andpossiblesolutions)touniqueneeds Gaining deeper insights Foryears,personnelfromacrossAgilentTechnologieshavebeen involvedwithtechnologystandardizationandtestdevelopmentfor LTEandSAESince2008,Agilenthasactivelyprovidedtoolstohelp developerstakeLTE/SAEforwardtothemarketplaceInApril2009,the companypublishedacomprehensivebookcalledLTEandtheEvolution to4GWireless:DesignandMeasurementChallenges(ISBN:978-0-47068261-6)Thisresourcehasreceivedsignificantandwidespreadpositive feedbackfromthewirelessindustryInMay2010,JDSUacquiredthe NetworkSolutionsDivisionfromAgilentWiththisacquisition,several contributingauthorstothisLTEbookincludingtheauthorofthis applicationnotetransitionedfromAgilenttoJDSUThebookis justoneexampleofhowAgilentandnowJDSUcontributetothe overalllandscapeoftheLTEandSAEindustryAsacompaniontothebook,thisapplicationnote containsrelevantreferencestotheLTEbook,whichcanbestudiedseparatelyinpursuitofadeeper understandingofthevarioustopicsandconceptsinvolvedinLTEtesting

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Performance testing WithLTE,muchofthefocushasbeenonincreasingsystemperformanceinareassuchasend-user throughput,latency,andidle-to-activetransitionsAlthoughtheseimprovementsalongwith significantlyimpressivesingle-userdataratesareimportantformarketingreasons,inthemselves theyplayalimitedroleinhowrealuserswillutilizeadeployedLTEnetworkThus,measurementsare neededforbothsingle-userpeakratesandmulti-userthroughputInanalyzingmulti-userthroughput andoverallcellcapacity,itisimportanttounderstandtheimpactofacellsrealdistributionofusersas wellasthemobilityandusagepatternsofthoseusers Single-user throughput testing Intheory,single-userthroughputtestingisquitesimpleHowever,inrealityitcanprovetobequitetricky Asanexample,thefirstaspectstounderstandarewhat to measure and how to benchmark the result LookingatthepublicresultsoftheLTESAETrialInitiative(LSTI)ProofofConcept(PoC)group,it canbeseenthatpeakrateswillvaryfromafewhundredkilobitspersecondatthecelledgetoover 150Mbpsinverygoodradioconditions(forexample,ina20MHz2x2MIMOsystem)Inpractice, measuringthiswiderangeofratescouldbeperformedwithadedicatedhardwaretrafficgenerator thatwillhaveguaranteedperformanceandwillproducetrafficpreciselyaccordingtoananticipated trafficprofileSuchaprofilecouldincludevaryingdatarates,differenttypesoftraffic,anddelay characteristicssuchasjitterOtheralternativesincludesoftware-basedtrafficgenerators(for example,iperf)orsimplygeneratingthetrafficfromexistingstandardapplicationserversusedfor FTPorvideostreaming Ifasoftware-basedtrafficgeneratorisused,itisimportanttounderstandtheconditionsunderwhich itwilldeliverreliableresultsIngeneral,currentlyavailablesoftware-basedtrafficgeneratorsproduce acorrectaveragethroughput;however,theinstantaneousvariationscouldbesignificantTogenerate highrates,dedicatedCPUresourcesmustbeconsistentlyavailabletothegeneratorShouldother processesstartonthetrafficPC,andthisisoftenthecasewithcertainoperatingsystems,theremay becomplicatedsideeffectsintheEPCortheeUTRANbecausebufferingmightoccurunexpectedly Consequently,theremaybeagapinthegeneratedtrafficandthusnothingtotransferovertheair interfaceforagivenTTI Ifnotmanagedproperly,theuseofanexistingapplicationserver(forexample,anFTPserver)to generatethetrafficcouldalsoproduceunexpectedsideeffectsForexample,anFTPservernormally accessesafilefromatraditionalharddiskIfmultipleuserstrytodownloadthesamefileatthesame time,therecouldbeabottleneckcausedbytheFTPserverratherthantheLTEairinterface(ingood radioconditions)ItispossibletosetupanFTPservertomanagethisifpropercareistakenandthe appropriatedisksystemisapplied;however,oversightsassimpleasanimproperlyconfigureddiskcan skewtheoverallresults Whenitcomestomeasuringtheperformance,oneessentialruleofthumbistounderstandthebasic aspectsthatwillcontroltheresultsForexample,anunderstandingoftheunderlyingradiolayer transportsettingsiscrucialHARQprovidesanexample:ifHARQisnotenabled,thelinkwillactually deliverahigherthroughputingoodradioconditionscomparedtotimeswhenHARQisnotswitched onOntheotherhand,reliabletransportdependsonHARQbeingswitchedonIfnot,problemssuch asasignificantamountofTCPretransmissionwillcropup,leadingtoaverypooreffectivethroughput EvenifHARQisswitchedon,theamountofHARQretransmissioncanbeconfiguredSettingthisto averylowvaluewillincreasetheinstantaneousthroughputbutwillleadtoapooreffectivethroughput undernon-idealradioconditions

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Foradetailedunderstandingofbearerthroughput,measurementsshouldbemadeatdifferentlayers intheprotocolstackMAClayer,IPlayerandUDPorTCPlayersafterapplicableretransmissions Duringthesemeasurements,itisimportanttounderstandandrecordtheactualsettingsthatwere configuredfortheMAClayerandtheTCPlayer,andifanyspecificserviceorapplicationlayersettings havebeenapplied(forexample,iftheapplicationusingTCPhadoneormultipleTCPflows) InsomeveryspecificcasesitmaybeusefultomeasurethehighestpossiblebitratethattheLTEair interfacecandeliverForthismeasurement,therecommendedapproachistomeasurethelossless (forexample,zeropacket-lossthroughput)ofasustainedUDPstreamInsuchcases,thespecific MAClayersettingsshouldberecordedaswellastheUDPpacketsizeItcouldaddsubstantialvalueto performanRFC2544testovertheLTEconnection;thiswillstepthroughseveraldifferentpacketsizes andthroughputratestolocatethelosslessthroughputforeachrelevantpacketsize Tounderstandwhythroughputischangingindifferentenvironmentsatdifferenttimes,thebest approachistorecordasetofLTE-relevantparameterswhileperformingthethroughputtestThese parametersshouldincludetheinputstotheeNodeBschedulingdecisions(forexample,CQIforall oftheranks(widebandandsub-band),thePMI,andtherankindicator)andinformationaboutthe resultingeNodeBschedulingdecisions(forexample,selectedmodulationandcoding,MIMOmode, etc)Whentestinguplinkperformance,therelevantpower-controlinformationshouldalsobelogged JDSUprovidesallofthetoolsandprocessesneededtoplan,perform,evaluateresults,andprovide reportsforsingle-andmulti-userthroughputtestingThisincludesbothtoolstogenerateE2Etraffic aswellastoolstologdatafromhandsetsandrelevantnetworkinterfacesFortestinginthefield,signal sourcesavailablefromothervendorsemulateuplinkinterferenceandloadingcausedbyusersinother cellsBecauseloadinghasasignificantimpactonthroughputratesitshouldbeincludedaspartofany realisticevaluationThegeneratedinterferenceshouldideallyberepresentativeofanLTEairinterface inbothULandDLAlthoughwhitenoisewasacceptableforUMTS,whichusesanoise-likeCDMAair interface,itdoesnotrepresentthepowervariationsacrossfrequencyandtimethatcharacterizeLTEs OFDMandSC-FDMAmodulationschemesRapidvariationsmaytrickschedulersintomakingeither optimisticorpessimisticpredictionsofthechannelconditionsandsubsequentmodulationandcoding schemethatcanbereliablytransmittedJDSUtoolsprovidethemeanstoevaluatethisimpactbefore networksbecomeheavilyloaded Cell performance verification with multi-user throughput testing MeasuringtheperformanceofanLTEsysteminasingle-usercaseprovidesabasicunderstanding uponwhichamorerealisticanalysisofthemulti-usercasecanbeperformedItshouldbe understoodthat,inmanycases,single-userpeakthroughputperformancewillbemuchhigherthan eventheaggregatecellthroughputwithseveralusersactivelydownloadingFromindustrystudies performedbyLSTIandothers,DLcellthroughputisexpectedtobearound35-40Mbps(assuming 20MHzwith2x2MIMO)with10usersspreadoverthecell,allwithfullbufferdownloadsanda proportionalfairscheduler1,2

1Theseexpectationsarerelativelyconsistentwithprevious3GPPsimulations 2ThetermfullbufferreferstotheideathattheeNodeBisconstantlyschedulingdatafromtheS1interfaceThebuffermustalwayscontain enoughdatatofullysaturatethelinkeveninthecaseofaperfectRFenvironmentandthehighestthroughoutrate

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Whenbasicmulti-usertestingisperformed,theresultsmayvarygreatly,spanningfromvalues belowtheanticipatednumberstoratesnearthesingle-userpeakperformanceThereasonforthis widevariationisduetotwokeyfactors:theoverallbehavioroftheeNodeBsschedulingandthe environmentforeachindividualuserItisthereforeimportanttosetupthetestinarepeatablewayto ensurethatresultsareusefulontheirownandcanbecomparedagainstsimilartestsineitheradjacent cellsinthesamenetworkorwithcellsthatusenetworkequipmentfromanothervendor Theprocessofsettingupatesttoverifycellperformancewithmultipleusershastoomanystepsto presenthereindetailInoutlineform,herearethekeysteps: 1 Evaluatethedistributionofsignalqualityinthenetworkthroughexhaustivedriveorwalktesting tobuildaCDF a IftheCDFsfordifferentcellsaresignificantlydifferent,theendresultwillbesignificantlydifferent b Thedistributionwilldependonthetypeofterraincoveredandwherethewalkordrivetestis performedForexample,includingindoorlocationswillproducelowersignalqualitiesthan outdoor-onlydrivetesting c Itisexpectedthatroughly80percentoftheLTEdatatrafficwillbegeneratedindoors 2DistributetheUEsaccordingtothesignalqualityCDFtoobtainapropertrafficprofile a ThechoiceofdistributionshouldmatchnotonlythemeasuredCDFbutalsotheanticipated distributionofrealusersintherealnetworkForexample,ifitisanticipatedthatseveralusersare locatedinagroupinapubliclocation(forexample,anairportorcaf)thenitisadvisabletoplace someoftheusersinsimilarpatterns b DifferentMIMOconditionsmightbestressedduetothistypeofdistribution(forexample,multiusergainsfromMIMO) 3Intherealnetwork,locatetheusersaccordingtotheplanneddistribution a Notethatitwillbenearlyimpossibletogetanexactmatchsoitisimportanttoinsteadlocatethe usersinanareasimilartotheoneidentifiedMakesurethattheactualradioconditionsofthe chosenlocationsareloggedandstoredforthecompletedurationofthetest b Forcasesinwhichsomeoftheuserswillbeinamovingenvironment,makesurethatthiscanbe managedinacontrolledway 4 EnsureOCNGisenabledintheDLfortheadjacentcellstocreateDLloadinginthecell 5EnsurerealisticULloadingisgeneratedfromUEsinadjacentcells 6GeneratetraffictothetestUEsandensurethateachUEissetuptoreceiveortransmittrafficwith fullbuffers 7Logalloftherelevantparametersasidentifiedearlierincludingthelocation(latitudeand longitude)ofeachUE a Logallrelevanttrafficfromthenetworkinterfacestoensurethatfullbufferswerepresentforall UEsinallconditions b VerifythatallUEswereactiveandthattheindividualUEthroughputratesarerealisticcompared toexpectations 8CorrelatetheUEsentireindividualthroughputforMAC,IP,andUDPorTCPlayers a IfoneorseveraloftheUEsweremoving,theaggregatedthroughputmaychangeovertime dependingontheschedulingandspecificradioperformanceofeachdevicesenvironment

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

Itshouldbenotedthatthisisnotarealisticwaytotestacellsactualperformanceinarealnetwork scenariowhenbuffersarenotalwaysfullInstead,itisatestofhowtheeNodeBwouldbehavein averyspecificscenarioThiscouldnormallybeconsideredaworst-casescenarioforaspecific distributionofusers Realistic multi-user throughput testing Despitethecomplexity,therearestilllimitationsinmulti-usertestingthatusesfullbufferdownloads anduploadsInarealistictrafficscenario,usersareexpectedtohaveburstytrafficprofiles,possibly comprisingHTTP,voice,FTP,IM,e-mail,etcInthefull-buffercase,userswithgoodconnections downloadmoredatathanuserswithlowdatarates,whichbiasesresultsAlthoughthisispartlytrue, userstendtodomorewithafasterconnectionthefull-buffercaseisperhapstooextremeThe question,then,iswhatwouldbeamorerealistictrafficmodel?Woulditbevaluabletopursuethis testingand,ifso,whatistheaddedvalue?Theanswerdependsonwhichphaseofthetechnologythe specificoperatorisinatthetimeIftheoperatorisabouttoperformvendorselection,thiscouldprove tobeanessentialtesttoensurethattheequipmentbehavesappropriatelyintermsofscheduling,ability todeliverthedesiredQoS,andoverallfairnesstothedifferentusers ItisalsoimportanttounderstandthatdeliveringtheexpectedQoSorbeingabletoprovidefairness inthesystemisnotnecessarilyadifficulttaskforaneNodeBInpractice,thedifficultyistheabilityto delivertheexpectedQoSwithaminimumofoverheadfactorsthatimpacttheoverallcellcapacity Thefollowingisaproposedtesttohighlightcertainpossibledeficienciesinasystem 1 QoSoverheadprovisioninganalysis a Setupastaticmulti-usercelldownloadwith6to10usersperformingfull-bufferdownloads b Measuretheoverallcellcapacity c Reassignoneuserasamobileuserwithafixed-rateUDPstreamwiththesamemegabit-persecondratethatwasachievedinthefull-bufferdownload i SettheQoSparametersforthisusertomatchthefixedrateoftheUDPstream ii Verifythatthethroughputrateisintherangeof2to5Mbps d Measuretheoverallcellcapacity e Movetheuserfrommediumtopoorradioconditioninfivestepsandrepeatthecell-capacity measurementsforeachstep f Movetheusertothecelledgeforthespecificdataratethatwasprovisioned g Measuretheoverallcellcapacity  ComparethecellcapacityfirstwithandthenwithouttheQoSprovisionedforthebandwidththat canbedeliveredwithoutanyactualimpactontheoverallcellcapacityThencompare,stepbystep, theoverallimpactofusermobilityoncellcapacityandguaranteedQoS 2Testtheimpactofrealistictrafficbehaviorontheoverallscheduling a Setupagroupofusers(6to10)witharealistictrafficprofileforthetypeofusersenvisionedfor theLTEnetworkThefollowingareexamplesofusescenarios i Downloadfivewebpageswitha30-seconddelaybetweeneachdownloadEachpage contains20objectsforatotalof14MBperpage ii Receiveane-mailmessagewithalarge(forexample,10MB)attachment

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

iii Send10e-mailmessages,fourwith2-MBattachmentsandsixsmalloneswithonly50KBof dataineach iv ListentoInternetradio(256Kbps) v PerformabackgroundFTPdownloadofsevenlargefiles(totalof100MB,similartoa monthlyoperatingsystemupdate) vi MakeoneVoIPcallusing122KbpsonaguaranteedQoSbearer  Notethatthetestconditionsmustbesetupsuchthattheradiointerfacewillbecongestedin certainconditionsduringtestexecutionUsethepreviouslymeasuredanticipatedcellcapacity numberstoselectthecorrecttest-caseparameters

b Thesequenceofeventsoccursinparallelforeachuser;however,theusersareseparatedintime by10secondseach c EachuserwouldbelocatedinaspecificradioenvironmentdefinedbytheCDFforthespecificcell d LetthetestruntocompletionwhileloggingalldatafromboththetraceUEsandthenetwork basedprobesIfpossible,alsologthedatafromtheLTEUuinterface  ThedatashouldthenbeanalyzedtorevealactualbehaviorduringcongestedconditionsOne shouldspecificallystudytheimpactofthescheduleronboththeguaranteedQoStraffic(VoIP) andthestaticRTPstreamfortheInternetradiobecausethesetwoservicesarethemostlikely todegrade Asimpleandbasicmeasureistobenchmarkthetotaltimeofcompletionforthetestsequence (excludingtheInternetradiostreamandtheVoIPsession,whichcouldrunindefinitely)The minimumcompletiontimeshouldberelativelyeasytocalculatebasedonthepreviously measuredperformanceforthemulti-userandfull-bufferdownloadscenariosThedifference fromthisminimumtimeshouldbeanalyzedtounderstandtheoverallefficiencyofthesystem AddingmobilitytothistestcasewouldallowforfurtheranalysisHowever,itwouldprobablyadd somuchcomplexitythatitcouldnotbejustifiedasabasiccase Allaspectsoftheschedulingandperformanceoftheradioenvironmentshouldbeanalyzedto understandtheefficiencyineachlayerandduringeachprocessThiswillalsohelprevealwhich conditionsleadtosuccessfulschedulingresultsandwhichcauseschedulingissues

 

JDSUcanprovidethetoolsandtheprocessestoexecuteandanalyzetheresultsfromthesetypesof testsJDSUtoolscanhelpcharacterizethebehaviorandallowforbothcompetitivebenchmarkingand regressiontestingwhenalaundrylistmustbemaintainedforaspecificsupplier Idle-to-active transition times OneofthemainexpectationsofLTEistoprovideanalwaysconnectedexperienceforendusersThis isachievedinpartbyensuringaswifttransitionbetweentheidleandactivemodesTheoverallRRC statemachinehasbeenoptimizedandthenumberofpossiblestateshasbeenminimizedtoensure reducedcomplexity,lowerpowerconsumption,andfastertransitiontimes Tomeasuretheidle-to-activetransitiontimesonemustbeabletoeitherfullycontrolUEbehavior orbeabletologalloftheassociatedsignalingtoensurethedataisavailabletobemeasuredfromthe overalltraffic JDSUtoolscanmeasureidle-to-activetimeaswellasotherrelevanttransactionandproceduraltimes Thesemeasurementscanbeperformedusingeitherdatafromatracemobilealoneordatacombined fromtheLTEandEPCnetworklinkstoenabletruecorrelatedE2Emeasurements

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

10

Latency testing Theoverallexperienceismorethanthespecificbandwidththatanendusercanreceiveandhow quicklythenetworkwillenableswitchingbetweentheidleandactivemodesTheexperiencealso dependsontheE2ElatencyofapackettransitioningthroughthecompleteEPCandeUTRAN KPI verification and calculation TheJDSUbookLTE and the Evolution to 4G Wireless: Design and Measurement Challengesincludes anextensivesectionaboutKPIs,thecalculationmethods,andtheoverallmethodology(pleasereferto Chapter6)Asaresult,theKPImaterialcoveredhereiskepttoaminimum ItisunfortunatethatthephraseskeyperformanceindicatorandKPIhavebecomecommonly misunderstoodandmisusedAtthemostbasiclevel,aKPIisnothingmorethanastatisticora measurementHowever,itisthetestobjectiveormarketrequirementforagivenservicethatallowsa particularstatisticormeasurementtobeconsideredasakeyindicatorofperformance EventhetermperformancecanmeanverydifferentthingsdependingonthetestingcontextFor example,serviceperformanceforVoIPmaybemeasuredintermsofjitter,latency,anddropped packetsNetworkperformancemaybemeasuredbythenumberconcurrentVoIPusersthatcanbe servedwithanacceptablelevelofjitter,latency,andpacketlossThus,whenmeasuringqualityor performance,oneofthekeychallengesisagreeingondefinitionsthatenableconsistentinterpretations ofresults Accordingto3GPP,KPIsgenerallyfitintooneoffivecategories:accessibility,retainability, integrity,availability,andmobilityThelistissometimesexpandedtoincludethecategoriesof utilizationandusability 3GPPKPIstandardizationeffortsarelargelyfocusedonmeasurementsrelatedtotheendusers perceivedQoSThesemetricstendtobemoreoperator-centricastheyrelatespecificallytomeasuring theabilityofcustomerstoobtainandmaintainaconnectiontothenetworkandtherebymakeuseof oneormoreavailableservices KPIsarebestunderstoodinthecontextoftheactualobjectiveofthemeasurementEachpartofthe networkhasdifferentresponsibilitiesassociatedwithdeliveringasingleserviceTherefore,LTEspecificKPIsfocusontheeUTRANitselfand,inmanycases,relyontheeNodeBtoactuallymeasure itsownperformanceOnechallengeforanNEMisdefiningawaytoverifythatKPIscalculatedbythe eNodeBarecorrect,especiallywhenrunningathighloadorfullcapacityAnotherchallengeforboth NEMsandWSPsismakingtheshiftfromsimplylookingatKPIstotroubleshootingandidentifying therootcausesofproblems ForeachKPIcategory,eachservicemayhaveadifferentQoSprofileorQCIlabelToidentifythe performanceofeachtypeofservicebeingaccessed,themeasurementsshouldbemadeonaper-QCI basisAdditionalKPIsineachofthesecategoriesshouldbeconsideredinordertoevaluatetheend-toendusabilityandmanageabilityofaservice Keyaspectsofawell-designedend-to-endtestsystemarethedatasourcesandthepossiblemonitoring pointsthatexistinanLTEandSAEnetworkSomeofthetopicsdiscussedherearealreadypartofthe industrystandardsOtherareasmayormaynotbepartoffuturestandardizationeffortsby3GPP,ETSI, orotherindustrybodies

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

11

Fundamentaltothetopicofdatasourcesandmonitoringpointsisanunderstandingofmeasurement theory,basicphysicallaws,andhowtheLTE/SAEtechnologyworksandisdeployedConsiderthe followingexamplecommontobothLTE/SAEandUMTSinwhichtwoengineersneedtomeasurethe RRCconnectionsuccessratiointhenetwork   efirstengineer,whoisexperiencedindrivetesting,commissionsatargeteddrive-testcampaign, Th measuringtheRRCconnectionsetupsuccessratioforawideareaClearly,thenumberof measurementpointsisdirectlyrelatedtothedurationofthetestingandthenumberofactual attemptsperunitoftimeForthisexample,theengineerconcludedthattheRRCconnection successratiowas985percent   esecondengineerisaccustomedtonetworkcountersandlink-monitoringtoolsAsaresult, Th heextractslogsfromthesystemThisprovideshimwithareportofalloftheRRCconnection attemptsforthefullnetwork;hismeasuredRRCconnectionsuccessratiois995percent Whyisthereadifferenceofnearlyonefullpercentagepoint?Theanswerisfundamentaltotherestof thisdiscussion Thedifferenceintheresultsisnotduetoflawsinthedatasource,beitthedrivetest,thenetwork counters,orthelink-monitoringtoolsThedisparityiscausedbythetwoengineersmeasuringdifferent networkproceduresfromdifferentangles  Thedrive-testmethodanalyzesnetworkperformanceasseenfromasinglehandsetatspecific physicalpointsinthenetworkatdistinctpointsintime  Thenetworkcountersandlink-monitoringtoolsrecordallofthetrafficandalloftheoccurrences ofsignalingandusertraffictheyaredesignedtomonitorThisisadifferentframeofreference:it analyzesnetworkperformanceasseenonthenetworkandatthenetworkmonitoringpoints Differencesbegintoaccumulateifanydrive-testlocationsarewelloutsidetheactual,andpotentially, intendednetworkcoverageareaAsaresult,RRCconnectionrequestsmadeoutsideofthenetwork coverageareawillberecordedbythedrive-testsystembutnotbythenetworkcountersorlinkmonitoringsolutionsThishighlightsakeypoint:anextensivedrive-testcampaignprovidesadditional informationbeyondwhatnetworkcountersorlink-monitoringtoolscanprovide Intheoptimizationcommunity,itisgenerallyagreedthatKPIsshouldbecomparedtoeachotheronly whentheyarederivedfromthesamedatasourceorwhentheyarenormalizedtoremoveanybiasdue tomethodorsourceThisisespeciallytrueifcomparisonsshowunexpectedresultsToday,thelackof propercomparisonsisoneofthelargestcontributorstounsoundoptimizationdecisionsinthemobile industryValidcomparisonsandmeaningfuloptimizationcanbeensuredonlyifastringentand coherentagreementondatasourcesandmonitoringpointshasbeensettledinadvance3 WhetherengagedintheR&Dprocessortheoptimizationphase,onemustoftenchoosebetween severaldifferentstrategieswhendevelopinganLTEtestplanSelectingthemostcost-effectiveand results-effectivestrategyisoneofthemostimportantdecisionstobemadeearlyineachphaseofthe workOnceastrategyisselected,theboundaryconditionsofitsapplicabilitymustbeestablishedItis importanttonotethatastrategythatisappropriateforonephaseoftheworkprobablyhassignificant shortcomingsinanyotherphaseInotherwords,itisseldomagoodideatousethesamefundamental KPIsbecausethedatasourceswillprovidedifferentresultsindifferentphasesofanetworks deploymentandmaintenance

3PleasenotethatthisissimplyamatteroffundamentalmeasurementtheoryItisnotintendedtobeadiscussionaboutthepotentialriskof resultsbeingincorrectduetoacertainmeasurementtoolnotworkingasdefined

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

12

Toillustratethisidea,consideracaseinwhichQoSiscontrastedwithQoEMonitoringtheend-user IPtrafficonamobilenetworkprovidesafullanddetailedunderstandingofthetrafficflows(TCPor UDP),theapplications(voice,video,HTTP,e-mail,etc),andpotentiallyperformanceSomebelieve thatthemonitoringofonlytheUDPorTCPflowswillprovideenoughinformationabouttheend-user QoStobeabletodeduceagoodapproximationoftheend-userQoEAsthefollowingscenarioshows, thisisnotthecaseforapplicationssuchasstreamingvideo   useriswatchingstreamingvideoonhishandset,buttheradioqualityisnotsufficienttodeliver A thefullbandwidthovertheairinterfaceWhentheUDPstreamismeasuredinthecorenetwork, nodegradationoftheRTP/UDPstreamisobservedThreeotherlocationsprovideabetterplaceto observethedegradation:ontheairinterface,ontheusershandset,orbetweenthetwoendpoints oftheRLCentity(intheUTRANbetweentheUEandtheRNCorintheLTEeUTRANbetween theUEandtheeNodeB)Thenetworkmonitoringtoolinthecorewouldreportahighandstable bitrate(goodQoS);however,theenduserwouldreportapoorQoE   hiftingthescenarios,assumethattheendusersapplicationisquality-awareand,duetotheradio S conditions,signalsthatthevideostreamingservershouldchangethebitrateofitscodecWhen thisnewRTP/UDPbitstreamappearsonthenetwork,thenetworkmonitoringtoolwillassociate thechangewithalowerQoSstreambecauseithasalowerbitrateOntheotherhand,theradio conditionsaregoodenoughtodeliverthisadaptedbitstream,andtheenduserQoEhasnowincreased ThisscenarioshowsthatthecrucialelementintheQoEisnotthebitratemeasuredinthecorenetwork, butrathertheend-to-endabilitytodeliveraspecificservicetotheenduserTheapplicationdomain will,inthiscase,ensurethatthebestpossibleQoEisachievedTherefore,themonitoringtoolmustbe application-awaretodelivercorrectQoSmeasurementsthatleadtothecorrectestimationofend-userQoE Validating LTE voice Oneoftheto-be-defineditemsfortheEPSishowcircuit-switchedservicessuchasvoice,CSUDIvideo, SMS,LCS,andUSSDwillbemanagedFouralternativesarecommonlyconsidered  Circuit-switchedfallback(CSFB)  VoiceoverLTEgenericaccess(VoLGA)  VoiceoverIMS(VoIMS)  Proprietaryoptions CSFBandVoLGAarebothstandardizedandcouldbereadilyimplementedVoIMSislikelytofollow andisanticipatedtobeawidespread,long-termsolutionAmongthese,CSfallbackinEPSisdescribed indetailbelow;theotheroptionsarecoveredinbrief Anotheraspectthatisnotyetspecifiedin3GPPR8istheuseofavoicecodecSeveraldifferentoptions areathand;however,duetoalackofagreement,thispartofthestandardizationmightbedelayeduntil R10Onekeyreasonforthedelayisthelackofclarityontheobjective:shouldthequalitybeimproved orshouldthecapacitybeimprovedbythechoiceofcodec?Itislikelythatachoiceofcodecforinitial EPSdeploymentswillbebasedonamutualbilateralagreementbetweentheUEvendors,theoperators, andtheEPSproviderItwouldnotbeasurpriseifAMRandAMR-WBwereusedinitially ItiscrucialtounderstandthattheITUhasalreadymovedaheadwiththedefinitionoftheG718codec G718isbuiltontheAMR-WBfoundationand,forthemostpart,providesthequalityofAMR-WB at1265KbpsonthesamecapacityasAMR795KbpsOverall,thismeansthatG718providesa57 percentincreaseincapacitywithaverylimitedimpactonspeechdelayAdetaileddescriptionis outsidethescopeofthispapersothereaderisencouragedtostudyITU-TRecG718infulltofurther understandthistopic

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

13

Signaling validation Figure2isatypicalexampleofanIMScallflowforLTEinteractingwiththeSS7networkAswillbe shown,thisisfarfromonlySIPandsomeofthecriticalaspectscannoteasilybeseeninthissimplified viewOneexampleisthelackofvisibilityintothechoiceofthebearersthathavebeensetupandhow theymaptoacertainQCI,etc Anyshort-termanalysisofavoice-over-LTEimplementationwillmostprobablybeimpactedbyone orseveralshortcutseveniftheintentionistobestandardscompliantThisshouldbeconsideredwhen performinganalysisanddrawingconclusionsKeyaspectstoconsideraretheusageofROHCforthe airinterface;theusageofacodecforE2EspeechandtheintegrationofapagingprocedurewhentheUE isinidlemode;and,howthemobilitybetweenaccesstechnologieswouldbemanaged

Originating UE Delay for RACH scheduling period Rach preamble

eNB

MME

Core IMS

PSTN

Terminating UE

TA + Scheduling RRC Connection Request

IDLE TO ACTIVE

RRC Connection Setup RRC Connection Setup complete + NAS service request

Connection Request Connection Setup

Security Mode Command + RRC Connection Recon guration RRC Connection Recon guration complete

SIP INVITE SIP 183 Session Progress

IAM

CALL SETUP

SIP PRACK SIP 200 OK SIP UPDATE SIP 200 OK

IAM

COT ACM/CPG

COT ACM/CPG

SIP 180 Ringing

Figure 2. Typical IMS call flow for LTE interacting with SS7

Voice QoS and QoE (MOS) VoiceQoSandtheresultingQoEisatopicofmanygoodpublicationsandthusthissectionwillnotgo intotoomanydetails Overall,onemustbecarefulaboutwhatistestedunderwhichconditions,andonemustbeawareofthe governingfactorsthatcontroltheoutcomeIfacertainvoiceserviceistestedwithabearerdeliveringa specificQoS,thentheresultingvoicequalitywillbelimitedbythequalitydeliveredbythebearer

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

14

Whilethissoundssimpleintheory,itcausessignificantpracticalconcernsinalmostallnew technologiesbeforetheE2EstructureiswellunderstoodInUMTS,voicewasnormallydelivered usingaradiolinkwithaone-percentBLERtargetCertainimplementationsusedonepercentasa minimumqualitytargetbutover-deliveredonqualityifandwhenresourceswereavailableThis meantthatcomparingasituationinwhichtheBLERwasalwaysheldattheconfiguredBLERtarget (regardlessofavailableresources)andBLERwasadjustedtodeliverthebestpossibleservicewithout causingdegradationforothers,theresultsinanunloadednetworkwouldalwaysfavortheflexible implementationThisisnotalwaystheintendedtestobjectandthusthevalueofvoice-qualitytesting canbedegradedorworse,beconsidereduselessTherefore,itiscriticaltounderstandtheunderlying conditionsthatwillimpactvoiceserviceandtoeithercontrolthesefullyorrecordonlythose parametersusedThiswillensurefaircomparisonswhenbenchmarkingaresulteitherovertimeor betweenimplementations JDSUsuppliestoolsthatcanperformE2Evoice-qualitytestingandalsobenchmarkvoicequality passivelyinsidethenetwork(Figure3)
JDSU J7830A Signaling Analyzer and J6900A Triple Play Analyzer JDSU VoIP Office End

EPC

Serving/PDN GW

JDSU J6804A DNA HD eNB eNB

Element Management System / Network Management System

UE JDSU E6474A NiXt

UE JDSU E6474A NiXt

Figure 3. Network architecture for E2E voice-quality testing

Testing QoS and QoE of LTE streaming video QoSandQoEmonitoringofavideoserviceisdirectlyanalogoustotheearlierdiscussionofKPIsIn short,allofthesamemethodsandissuesidentifiedintheKPIsectionapplytothesemeasurements Ratherthanrepeatingthatmaterial,pleasereferbacktotheearlierdiscussion Thereisonekeypointtoadd:understandingvideoQoSandtheresultingQoEshouldbedoneacrossall relevantaspectsoftheLTEandEPCThismeansthatonecannotonlyidentifytheactualQoSatapoint butcanalsoensuretheserviceupstreamfromthispointifthetrafficflowisgoodatthemonitored locationOntheotherhand,ifthetrafficflowispooratthemonitoringlocation,thenafurther investigationupstreamshouldbeperformedNotethatinmanycasestheclientcanusejitterbuffersto compensateforacertainamountandtypeofQoSimpairmentsThus,adegradedQoSdoesnotalways leadtothedegradationofQoE

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

15

JDSUprovidesalloftherelevanttoolstomonitorvideoQoSandQoEfromeitherapassiveperspective orfromanactiveperspective(Figure4)SupportisprovidedforbothstandardIPTVandforMSTV

Evolved UTRAN (E-UTRAN)

Evolved Packet Core (EPC)

S-GW P-GW

Core, Services, IMS

Internet

Uu

UEs

Evolved Node B

MME

HSS

CSCF

MRF

PCRF

Figure 4. Tools for monitoring end-to-end LTE QoS and QoE (VoIP, IPTV, data)

Evaluating LTE MIMO and frequency-selective scheduling ComparedtoHSPAsystems,muchofthevalueinLTEcomesfromtheeffectiveusageofbothMIMO andfrequency-selectivescheduling(FSS)Inmanyaspectsthesearetechnologiesthathaveyettobe provedintermsofaddingrealtangiblevaluetothecustomerinafieldenvironmentTheirrealvalue dependsonthreethings:theactualdeploymentscenario;theoveralltrafficmodeling;and,thespecific implementationandconfigurationofthesystem AfeaturesuchasFSSmakesittheoreticallypossibletoaddsignificantperformancetothesystemIn practice,however,performanceislimitedbythealgorithmusedtocontrolschedulingThisisdue tooperationaltradeoffssuchasreducingsignificantuplinktrafficforsub-bandCQIreporting,or technicalfactorssuchasthepracticalavailablecomputationalperformanceintheeNodeB AsfarasMIMOisconcerned,itshouldbeunderstoodthatMIMOusageorthemeasurements leadingtoaspecificMIMOconfigurationforspecifictransmissionstoauserduringaspecificTTIis typicallynotastaticbehaviorWhatsmore,anyexpectationthatMIMOcanbecontrolled,modeled, orunderstoodfromoneorafewmeasurementsistypicallyunrealisticAsaresult,itisimportantto identifytwokeyelements:thoseaspectsofthetechnologythatshouldbeevaluatedfromastatistical behaviorpointofview;and,thoseparametersthatcanbeanalyzedfromasinglediscretemeasurement withoutlookingatalargersample ItisvitaltounderstandandcharacterizetheactualbehaviorofbothMIMOandFSStoensurethe properdimensioningofthecells,thebackhaul,andtheQoSprovisioningThestatisticalnatureof MIMOandFSScanbewellunderstoodonadetailedTTIlevelonlyiftherighttoolsandprocessare appliedThesecapabilitiesareallavailableusingJDSUtoolsandmethodsThetoolsallowadetailed analysisandcharacterization,enablingaclearunderstandingofactualbehavior

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

16

Testing network coverage Duringnetworkplanning,modelingisperformedandassumptionsaremadeIftheseareatall inaccurate,theundesirableconsequencesincludeeitherunplannedcoverageholesorunwantedsignal leakageintoadjacentcellsEarlyintheplanningandtuningprocess,aclearunderstandingofactual networkcoverageversustheplannednetworklayoutenablestheplanningandengineeringteamto developastrongmethodologythatallowsforverycost-effectivenetworkdeploymentandtuningEarly verificationofthecoveragewillallowcreationofaCDFofthenetworkperformanceandallowfor propertestingofcellperformance JDSUprovidesdetailedRFcoveragemeasurementsincludingRSSI,RSRP,RSRQ,andRS_CINR Thesecapabilitiescaneffectivelycoveralargenumberofbandsinverycomplexnetworktopologies Theresultsfromthemeasurementscanbepost-processedwitheithercustomsoftwareoranyofthe industry-standardpost-processingapplications Testing LTE handover CoveragetestingiscloselylinkedtothecharacterizationofHOvalidationandperformanceTofully graspHOperformance,onehastofirststudyandunderstandtheoverallproceduresandprocessesthat underliethehandoverinLTEAsiswidelyunderstood,nosofthandoverispresentinLTEAsaresult,the handoverisalwaysahardbreakbeforemaketransitionTheresultingimpactonend-userQoEmust beunderstoodandoptimizedItisalsoimportanttounderstandthehandoversoverheadimpacton theeNodeBbecauseLTEhandoverscanbeusedtoproactivelymanageserviceloadinginaspecificarea Severaldifferenttypesofhandoversarepresent:thosebetweencellsinthesameeNodeB;thosebetween cellsindifferenteNodeBswithoutanyX2dataforwarding;and,thoseinwhichX2dataforwarding isensuringaminimuminterruptionofdatatrafficItiscommonlyunderstoodthattheX2data forwardingfeaturemightbeimplementedinlaterreleasesoftheeUTRANsoftware;however,some vendorsmightmakethiscapabilityavailableearlyinthelifecycle AUEcannormallydiscriminatebetweenthetypesofhandoversbymonitoringtheSFNaroundtheHO timeandcheckiftheSFNisjumpingoriscontinuous Figure5isagoodexampleofahandoffbetweentwocellsinasingleeNodeBTheplotshowsthatthe BLERisincreasingthemomentsbeforetheHOandthatthisresultsindegradedthroughputonthe MAClayerAftertheHOisperformed(forexample,anRRCconnectionreconfiguration),MAClayer throughputisincreasedandnoBLERisreported

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

17

Figure 5. MAC throughput (green line) sags before an RRC event but recovers immediately after.

Followingthisfromasignalingperspective,thecalltrace(Figure6)revealstheresultingHO interruptiontime,whichisontheorderof12to35msNotethatthisanalysisrequiresaccesstoboth theprotocollogsfromtheUE(orfromapassiveUuprobeorfromaeNodeBfeed)withthosefromthe networklinks(forexample,S1,X2,etc)

Figure 6. A call trace can reveal the resulting HO interruption time

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

18

Asinallmobiletechnologies,itisrelativelystraightforwardtoachievesuccessfulhandoversHowever, achievinganoptimizednetworkwithoptimizedhandoversrequiresgreatinsightandtremendousskill intheartofoptimization Validating LTE backhaul LTEoffersgreatlyincreasedbandwidthtotheenduserInturn,itisveryimportantthatthisbandwidth capabilityisavailablethrougheverypartofthenetworkOneofthebigareasoffocusforthisisthe mobilebackhaulnetwork,typicallyEthernet,betweenthecellsite(eNodeB)andthecorenetwork InLTEfieldtrialswehavewitnessedtodate,thewirelessbackhaulnetworkisbroughton-lineprior toimplementationoftheLTE-specifictestcasesCommissioningtheEthernetbackhaulquicklyand easilyisvitaltokeepinganLTEfieldtrialonscheduleTheJDSUNetCompleteServiceAssurance SolutionforWirelessBackhaulverifiesEthernetserviceoperationtotrialcellsitesinadvanceofthe LTEtrialexecutionAdherencetoRFC-2544standardsismaintainedwithcost-savingderivedthrough automationandefficientdeploymentoftechniciansevenacrossmultiplecellsitesHowever,since suchtestingisnotuniquetoanLTEfieldtrial,itisbeyondthescopeofthisapplicationnoteGeneral informationisavailablesuchastheIETFsRFC2544(1999),theIEEE,andtheITUwhichdefinetest andperformancemonitoringmethodologiesforEthernetnetworkinterfacedevices Verifying LTE handset IOT ForeachoperatorintheearlyphasesofbringingLTEtomarket,itisessentialtoensureproper interoperabilityofLTEhandsetsordatacardsOfcourse,theseareearlydevicesthatwillcontinue toevolverapidlyovertimeConsequently,itisimportanttounderstandthecapabilitiesandensure interoperabilitybetweenthehandsetandtherelevantnetworkelementsItwillnotbepossibleto simplyrelyonpre-conformanceorconformancetestresultsbecausethesewillseldomreflectactual userbehaviororissuesfoundintheearlyphasesofnetworkdeployment IndustryforumssuchasLSTIhavedefinedaminimumfeaturesetandacorrespondingIoDTand IOTtestplanTheseareavailableonlytoLSTImembersandmaybeusedonlyforLSTIpurposesAsa result,non-membersmustrelyonothermeanstosecurehandsetinteroperabilityAsanLSTImember, AgilenthasmadesignificantcontributionstotheIoDTandIOTphasesThroughthisexperience personnelinbothAgilentandJDSUarebetterabletocontributeeffectivelytothewiderindustry JDSUprovidesmonitoringtoolssuchastheSignalingAnalyzerthatprovidethecapabilitytoanalyze andcorrelateinformationfromeachinterfaceandhighlightanydiscrepanciesversustheanticipated behavioronasignalinglevelTheJDSUNiXTapplicationcanactivelystimulateahandsettoexecute differentapplicationsandtasksThiscorrelatestocontroloftherelevantRFparameters(forexample, fading,MIMOprofiles,etc)andaneffectiveLTEhandsetinteroperabilityenvironmentcanbe providedIfappropriatelyandeffectivelyautomated,thiswillensurethatcost-effectiveandreliable testingisperformed,therebymaximizingassetutilization Validating device configuration OnepartofLTEisyettobefullystandardizedandwidelyagreedupon:theeffectiveperformanceof deviceconfigurationEvenifsignificantinteroperabilitytestinghasbeenperformed,anyoperator launchinganLTEserviceearlywillberequiredtoremotelyupdatehandsetsandparameterssuchas networksettings,software,orotherrelevantparametersduringoperationofthenetwork4
4ManyofthetraditionalOTAsystemsarebasedonshortmessageservices(SMS)thatarenoteasilysuppliedinanLTEenvironmentwithout theuseofeitherCSFBIMSorVoLGA,bothofwhicharerelativelycomplexanderror-pronetechnologies

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

19

Itisessentialtoprovidefullvisibilityintodevicebehaviorduringpre-configuration,configuration, andpost-configurationphasesandtocomprehendthefullE2Esignalingneededtodeliverthe configurationenvironmentWithJDSUtools,thepre-configurationstatecanbeanalyzedboth qualitatively(forexample,theperformanceofalloperationalservices)andinsimplego/no-gotesting (whichservicesworkordontwork)Specificnetworkbehaviorsthatidentifyincorrectlyconfigured devicescanbeanalyzedanddocumentedtobeusedinsubsequentnetwork-widemonitoringsystems Duringtheconfigurationprocedure,differenttypesofpositiveandnegativetestingcanbeperformed tostimulateerrorconditionsandtoprovethattheconfigurationissuccessfulifcertainconditionsare metThenegativeconditionsareespeciallycriticalbecauseitisoftendifficulttoevaluatethestateofan end-userdeviceinarealnetwork

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

20

Appendix: Outline for a Basic Phase 1 LTE Test Plan

Overview Thissectioncontainsabasicsetoftestcasesthatcanbeusedintwoways:toevaluateLTEasa technology;and,toassessthebasicperformanceofLTEinfrastructurevendors Thebasictestplandescribedhereiswell-alignedwiththethinkingofmostinfrastructurevendorsIt isalsoconsistentwiththeirdetailedtestplansAsaresult,thissectioncanbeusefultooperatorstoo Theycanusethisplanwhenaskingvendorstocreateadetailedoutlinethatspellsouthowtestingwill beconductedTheoperatorcouldthenvalidatethatthedetailedtestplanmeetstheintentionand basicexpectationsoftheplanTheoveralltestexecution,datacollection,analysis,andreportingcanbe managedwithintegrity,ensuringavalidevaluation Subsequentphasesofatestplancanbeprovidedthatwillallowtheoperatortomovetowardvendor selectionandnetworkroll-out Thesuggestedtestplanhaseightmajorsections  Achievabledataratesandlatency:single-userthroughputforUL/DLandTCP/UDP  Intra-LTEmobility  Coverageandcapacity  Evaluationofantennaconfigurationoptions  Self-configurationandself-organizingnetworkfeatures  Evaluationoffrequencyreuse  BasicQoS  Basicapplicationperformance Below,AchievabledataratesandlatencyhasfoursubsectionsandCoverageandcapacityhastwo Eachofthe13totalentriespresentsabasictestoverviewthatcanbeeasilyleveragedintoatestplan Achievable data rates and latency: single-user throughput for UL/DL and TCP/UDP Basic test overview  Evaluatesingle-userthroughputacrossarangeofradioconditionsGenerateULandDLloadingof 70percentfromadjacentcells  PerformseparateULandDLthroughputtestsindividuallywithTCPandUDPMeasureaverage throughputfor30secondsduringstationaryconditionsTheTCPandUDPtestsshouldmeasure losslessthroughputratherthanrawthroughput  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsides  Providerawandnormalizedresultsaccordingtothecommontestdescription

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

21

Achievable data rates and latency: cell throughput and MU throughput for UL/DL and TCP/UDP Basic test overview  GenerateaCDFofthecellaccordingtothecommontestdescriptionPlace10UEsaccordingto theCDFdistribution  Verifyplacementofthe10UEsinappropriateandrepresentativelocations  GenerateULandDLloadingof70percentfromadjacentcells  PerformseparateULandDLthroughputtestsindividuallywithTCPandUDPMeasureaverage throughputfor30secondsduringstationaryconditionsTheTCPandUDPtestsshouldmeasure losslessthroughputratherthanrawthroughput  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsides  Providerawresultsandnormalizedresultsaccordingtothecommontestdescription Achievable data rates and latency: latency Basic test overview  Evaluatetheend-to-endlatencyinarangeofradioconditionsandloadingscenarios  No-loadscenario  Performend-to-pinginarangeofradioconditionsforaminimumof100samplesperlocation andpacketsize(32,1000,1500bytes)  With-loadscenario  GenerateULandDLloadingof70percentfromadjacentcells  Generatesame-cellDLloadfromoneUEingoodradioconditionswithfullUDPdownload  Generatesame-cellULloadfrom1UEinpoorradioconditionswithfullUDPupload  Performend-to-endping(32,1000,1500bytes)inarangeofradioconditionsforaminimumof 100samplesperlocationandpacketsize  Logallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanefrom thenetworkandUEsides  ProvideRANandEPClatencyand,ifrelevant,thebackhaultransmissionlatency  Providerawresultsplusminimum,maximumandaveragevaluesforeachradioconditionand packetsize Intra-LTE mobility: mobility and handover performance Basic test overview  TestandcompareresultsforhandoverbasedonbothS1aloneandonX2forwarding;testsamecell loadedandunloaded  Samemethodologyforboth(S1andX2)testcases  Repeattestforatleast20HOofeachtype  PerformthetestwithDLandULTCPandUDPtrafficsubsequently  MeasureandreportHOsuccessrate  MeasureandreportcontrolplaneHOtime  MeasureandreportuserplaneHOinterruptiontimeandpacketloss

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

22

 Logallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanefrom thenetworkandUEsides  ForloadedHOperformance,testaddtwoUEsingoodradioconditionsinbothcellsperforming full-bufferDLandUL  Othercellloadshouldbe70percentinDLInthemobilitycase,notethatULloadingisnot requiredduetopracticalissues Achievable data rates and latency: application performance Basic test overview  Fromidlemode,connectanddownloadtheCopernicuswebsite(seecommontestdescription)and measurethedownloadtimeinarangeofradioconditionsAminimumoffiverepetitionsateach locationisrequiredAminimumoffive(10isrecommended)differentradioconditionsshouldbe coveredfromgoodradioconditionstocell-edgeconditions  Theothercellloadshouldbe70percentinULandDL  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsides Coverage and capacity radio features efficiency and gain assessment: link budget Basic test overview  Testfirstinstaticlocations:  TheUEislocatedclosetothecellcenterAUDPDLtransferwithfullbufferisinitiated  Theusermovesawayfromthebasestation,sopathlossincreasesRepeatthemeasurementsat thenewlocation  Thisisrepeatedforatleast10differentlocationsacrossthecellAmajorityofthelocations shouldbeinpoorradioconditions;continueuntilcoverageislost  Repeatthesametestwhileinmobileconditions:drivefromcellcentertocelledgeandrepeat severaltimes  Logallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanesfrom thenetworkandUEsides Coverage and capacity radio features efficiency and gain assessment: scheduler Basic test overview  LocatefourUEsingoodradioconditionsascloseaspossibletoequalradioconditionsEnsurethat allUEshavethesamepriorityandQoSsettings  SetupDLloadingofUDPwiththefollowingtrafficcharacteristics:UE1=X;UE2=2*X;UE3=3*X; UE4=4*XEnsureXischosensothattheaggregatedthroughputoftheusersexceedsthecellcapacity  Measuretheresultingbehavior  LocatetheUEsatfourdifferentlocationsandgeneratethesameUDPstreamthroughputtoeach UEEnsureaggregatedthroughputoftheusersexceedsthecellcapacity  Measuretheresultingbehavior  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduser planesfromthenetworkandUEsides

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

23

Evaluation of antenna configuration options Basic test overview  Performthesingle-userthroughputtestcaseandthemultiusercellcapacitytestcasewiththe followingantennaconfigurations:  SIMO  2x2MIMO  4x4MIMO(Optional:Performifpossibleforthevendor) Self-configuration and self-organizing network features Basic test overview  EnsureatleasttwoeNodeBsareoperationalandnoneighborsexistinthetargetcell  VerifyintheO&MsystemthatnoneighborsorX2interfacesareconfigured  ActivateANRineNodeB  TurnontheUEinacellwherenoneighborsaredefinedAfterUEreportingofsignalstrengthsof surroundingsectors,theANRfunctionshalladdneighborstotheneighborlist  Performahandover  VerifyintheO&MsystemthatallcorrectneighborsandX2interfaceshavebeenadded  PerformanHOtoandfromthenewneighbor  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsides Evaluation of frequency reuse: one deployment scenario Basic test overview  LocateaUEatthecelledge(in100percentothercellload)andpreparefull-bufferdownloads  Withoutothercellload,measureaveragethroughputfor30secondsforbothTCPandUDP  With50percentothercellload,measuretheaveragethroughputfor30secondsforbothTCP andUDP  With70percentothercellload,measuretheaveragethroughputfor30secondsforbothTCP andUDP  With100percentothercellload,measuretheaveragethroughputfor30secondsforbothTCP andUDP  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsides  Comparethecell-edgeperformanceunderthevariousloadingconditionsandcomparethe measuredSINRforeachloadingconditionatthesamephysicallocation

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

24

Basic QoS: user differentiation between non-GBR users with different QCI Basic test overview  OneUE(A)shouldhaveafull-bufferdownloadofUDPtrafficingoodradioconditionsandwith theQCIoflowestpriority  OneUE(B)shouldhaveadownloadofaUDPstreamof10Mbpsandbelocatedinmediumradio conditionswithaQCIofsecond-lowestpriority  OneUE(C)shouldhaveadownloadofaUDPstreamof10Mbpsandbelocatedinmediumradio conditionswithaQCIofhighestpriority  Measuretheresultingthroughputandpacketloss(onUEBandC)andreporttheactualreceived throughputandtheS1andSGithroughput  Othercellloadshouldbe70percentinULandDL  Logallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanefrom thenetworkandUEsides Note:Toensurethatcongestionoccurs,itmaybenecessarytonormalizethethroughputofUEBandC Basic QoS: user-differentiation between GBR and non-GBR users Basic test overview
User GBR DL (Mbps) GBR UL (Mbps) MBR DL (Mbps) MBR UL (Mbps) ARP (Allocation and Retention Priority is NOT equal for all flows) UE1 X1 Y1 A1 B1 N1 UE2 X2 Y2 A2 B2 N2 UE3 X3 Y3 A3 B3 N3 UE4 X4 Y4 A4 B4 N4

Assume:N1>N2>N3>N4(User1hasthehighestpriorityandUser4hastheleastpriority)

 LocatefourUEingoodradioconditionsConfigureDLbearerswithGBRvaluesforUE1,UE2, UE3,andUE4EnsurethatX1,X2,X3,andX4DLthroughputsareaggregatedabovethecell capacityatallUElocationsbutthatX1,X2,andX3aggregatedarebelowthecellcapacity  GenerateUDPDLtraffictoclientsUE1,UE2,UE3,andUE4  StopDLtransferafteratleast30secondsofdatatransfertotheUEclients  MeasureDLthroughput  VerifythattheDLratesforUE1,UE2,andUE3arenolessthantheirpredefinedGBRvalues(X1, X2,andX3,respectively)  VerifythatUE4(lowest-prioritytraffic)wasthemostnegativelyaffected(theDLbitratefailedto achieveX4,whichisthedesiredDL-GBRforUE4)  ChangetheARPsettingofUE4tobethehighestpriority(N1)  RepeatwithsimultaneousDLtransfertoallfourUEclients  VerifythattheDLdataratetoUE3(lowestpriorityamongallUEs)ismostnegativelyaffected(its dataratewillbelowerthanitsdesiredDL-GBR)

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

25

 PerformthesametestbutlimittheS1downlinkperformancetocausecongestionatalevelbelow theRFcellcapacitybutabovetheaggregatedX1,X2,andX3VerifythroughputisbelowtheS1 peakthroughputlimitation  ThelimitationcanbeachievedviaVLANtuning Basic application performance: web browsing, streaming, voice calls, e-mail, VPN, on-line gaming Basic test overview  Identifytherelevantservicesforeachlocalenvironment  Selectaheavilytraffickedlocalwebsite:  1-5MBforthemainpage(includingimages)  SelectaVoIPapplicationsupportedbytheoperatorandthelocalenvironment(forexample, Skype,GoogleChat,MSN,etc)  SelectalocalcorporatecustomerusingaVPNinamobileenvironment;theVPNconnection shouldhaveadatarateontheorderof2-10Mbpsend-to-end  Identifythreelocal,well-recognizedservices:  Ahigh-requirement(low-latency)onlinegame  Ahighperformancefixed-lineInternetservice(forexample,xDSL,packetcable,etc)  Astreamingvideo(RTP/UDP)service  Executeandcomparetheapplicationperformanceonthefollowingaccesstechnologies:  HSPA(bestlocalcommercialavailableservice)  Fixed-lineInternetservice  LTEingood,medium,andpoorradioconditions  Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes fromthenetworkandUEsidesforLTEand,ifpossible,forHSPAandthefixed-lineInternet service  Forthevoiceservice,measureandcompareMOS  Forthestreamingvideoservice,measureandcomparevideoMOSorasubjectivequality comparisonwithfriendlyusers  Fortheonlinegame,getasubjectivestatementfromanexperiencedgamerandanE2Elatency measurement  Reporttheabsoluteandrelativeperformancebetweenthedifferentaccesstechnologies

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

26

References
EPS specification references
3GPP TS 24.301 36.413 36.423 29.118
29.168 29.272 29.274 29.275 29.276 29.277 29.280 29.281

Title Non-Access-Stratum (NAS) Protocol for Evolved Packet System (EPS); Stage 3 E-UTRAN: S1 Application Protocol (S1AP) E-UTRAN: X2 Application Protocol (X2AP) Mobility Management Entity (MME) -Visitor Location Register (VLR) SGs Interface Specs. Cell Broadcast Center Interfaces with the Evolved Packet Core; Stage 3 MME Related Interfaces Based on Diameter Protocol Evolved General Packet Radio Service (GPRS) Tunneling Protocol for Control plane (GTPv2-C); Proxy Mobile IPv6 (PMIPv6) based Mobility and Tunneling Protocols; Stage 3 Optimized Handover Procedures and Protocols between EUTRAN Access and cdma2000 HRPD Optimized Handover Procedures and Protocols between EUTRAN Access and 1xRTT Access 3GPP EPS Sv Interface (MME to MSC) for SRVCC GPRS Tunneling Protocol User Plane (GTPv1-U)

Protocol EPS NAS S1-AP X2-AP SGsAP

SBc-AP Diameter+ GTPv2-C PMIPv6 S101-AP S102-AP Sv GTPv1-U

Interface(s) S1-C S1-C X2-C SGs

SBc S6a, S6d, S13 S3-C, S4-C, S5/8-C, S10, S11-C S5, S8 (PMIP) S101 S102 Sv S1-U, X2-U, S4-U, S5/8-U, S12-U

3GPP references 3GPP TS 23272:CircuitSwitchedFallbackinEvolvedPacketSystem;Stage2 3GPP TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3 3GPP TS 29118:MobilityManagementEntity(MME)VisitorLocationRegister(VLR)SGs interfacespecification TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN)access 3GPP TS 21905:Vocabularyfor3GPPspecifications 3GPP TS 22278:ServicerequirementsfortheEvolvedPacketSystem(EPS) 3GPP TS 43318:GenericAccessNetwork(GAN);Stage2 TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN)access TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3 TS 29272:EvolvedPacketSystem;MMEandSGSNrelatedinterfacesbasedonDiameterprotocol TS 33102:3GSecurity;Securityarchitecture TS 33203:AccesssecurityforIP-basedservices TS 33210:3GSecurity;NetworkDomainSecurity;IPnetworklayersecurity

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

27

TS 33401:3GPPSystemArchitectureEvolution(SAE):SecurityArchitecture; TS 36300:EvolvedUniversalTerrestrialRadioAccess(E-UTRA)andEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN);Overalldescription;Stage2 TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN)access TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3 TS 29272:EvolvedPacketSystem;MMEandSGSNrelatedinterfacesbasedonDiameterprotocol TS 33102:3GSecurity;Securityarchitecture TS 33203:AccesssecurityforIP-basedservices TS 33210:3GSecurity;NetworkDomainSecurity;IPnetworklayersecurity TS 33401:3GPPSystemArchitectureEvolution(SAE):SecurityArchitecture; TS 36300:EvolvedUniversalTerrestrialRadioAccess(E-UTRA)andEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN);Overalldescription;Stage2 TS 36420:X2layer1generalaspectsandprinciples TS 36421:X2layer1 TS 36422:X2signallingtransport TS 36423:X2ApplicationProtocol(X2AP) TS 36424:S2datatransport TS 29281:GPRSTunnelingprotocolforuserplane(GTPv1-U) TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN)access TS 36300:EvolvedUniversalTerrestrialRadioAccess(E-UTRA)andEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN);Overalldescription;Stage2 TS 29305:InterworkingFunction(IWF)betweenMAP-basedandDiameter-basedinterfaces TS 29274:TunnelingprotocolforControlplane(GTPv2-C);Stage3 TS 23060:GeneralPacketRadioService(GPRS);servicedescription;Stage2 TS 36410:S1layer1generalaspectsandprinciples TS 36412:S1signallingtransport TS 36413:S1Applicationprotocol(S1AP) TS 36414:S1datatransport TS 29281:GPRSTunnelingprotocolforuserplane(GTPv1-U)

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

28

TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3 3GPP TR 22968:StudyforrequirementsforaPublicWarningSystem(PWS) 3GPP TS 22168:EarthquakeandTsunamiWarningSystem(ETWS)requirements;Stage1 3GPP TR 23828:EarthquakeandTsunamiWarningSystem(ETWS),requirementsandsolutions TS 23060:GeneralPacketRadioService(GPRS);servicedescription;Stage2 TS 23107:QualityofService(QoS)conceptandarchitecture TS 23203:PolicyandChargingControlArchitecture TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN)access TS 36300:EvolvedUniversalTerrestrialRadioAccess(E-UTRA)andEvolvedUniversalTerrestrial RadioAccessNetwork(E-UTRAN);Overalldescription;Stage2 TS 23002:Networkarchitecture TS 23003:Numbering,addressingandidentification NGMN reference wwwngmnorg/uploads/media/White_Paper_NGMN_Beyond_HSPA_and_EVDOpdf ETSI reference ETSI TS 102 250-1:Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects for popular services in GSM and 3G networks: Part 1: Identification of Quality of Service aspects

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

29

Glossary
AAA AMR AMR-WB APN ASP BICC BLER BSC BTS CAPEX CDF CDMA CLID CLIP CSUDI CSFB DL DMT E2E eNodeB EPC EPS ESN ETSI eUTRAN FSS GGSN GPRS GUI HA HARQ HLR HO HTTP IM IMS IMSI authentication,authorization,andaccounting adaptivemulti-rate adaptivemulti-ratewideband accesspointname applicationserviceprovider bearer-independentcallcontrol blockerrorrate basestationcontroller basetransceiverstation capitalexpenditures cumulativedistributionfunction codedivisionmultipleaccess callinglineidentification callinglineidentificationpresentation circuit-switchedunrestricteddigitalinformation circuit-switchedfallback downlink dataminingtoolkit end-to-end evolvedNodeB;NodeBisaUMTSbasetransceiverstation(BTS) evolvedpacketcore evolvedpacketsystem electronicserialnumber EuropeanTelecommunicationsStandardsInstitute evolvedUMTSterrestrialradioaccessnetwork;alsoabbreviatedasE-UTRANorEUTRAN frequency-selectivescheduling gatewayGPRSsupportnode generalpacketradioservice graphicaluserinterface homeagent hybridautomaticrepeatrequest homelocationregister handover hypertexttransferprotocol instantmessaging instantmessagingservice internationalmobilesubscriberidentity

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

30

IP-TV IOT IP IRAT KPI LCS LSTI LTE MGW MIMO MSC MSC-S MSS MSTV NAI NE NEM NGMN OCNG OFDM OPEX OLAP PCF PDSN-FA PDSN-HA PoC PSTN QCI QoE QoS QoSM RNC ROHC RRC SAE SC-FDMA SCTP SFN SGSN

Internetprotocoltelevision inter-officetrunk Internetprotocol inter-radioaccesstechnology keyperformanceindicator locationservices LTESAEtrialinitiative Longtermevolution mediagateway multiple-input/multiple-output mobileswitchingcenter mobileswitchingcenterserver mobilesoftswitch MaximumServiceTelevision networkaddressindicator networkelement networkequipmentmanufacturer next-generationmobilenetworks OFDMAchannel-noisegeneration orthogonalfrequency-divisionmultiplexing operatingexpenses onlineanalyticalprocessing packetcontrolfunction packetdataservingnode,foreignagent packetdataservingnode,homeagent proofofconcept publicswitchedtelephonenetwork QoSclassidentifier qualityofexperience qualityofservice QualityofServiceManager radionetworkcontroller robustheadercompression radioresourcecontrol SystemArchitectureEvolution single-carrierfrequency-divisionmultipleaccess StandardControlTransmissionProtocol single-frequencynetwork servingGPRSsupportnode

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

31

SIP SMS SON STP TDR TTI UE UMTS UL URI USSD VIP VoIMS VoLGA WAP WSP

sessioninitiationprotocol shortmessagingservice self-optimizingnetwork signalingtransferpoint transactiondetailrecord transmissiontimeinterval userequipment UniversalMobileTelecommunicationsSystem uplink uniformresourceindicator unstructuredsupplementaryservicedata veryimportantperson voiceoverinstantmessagingservice voiceoverLTEgenericaccess WirelessApplicationProtocol wirelessserviceprovider

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials

32

Test & Measurement Regional Sales

NORTH AMERICA LATIN AMERICA ASIA PACIFIC EMEA WEBSITE: www.jdsu.com/test

TEL: 1 866 228 3762 FAX: +1 301 353 9216

TEL: +1 954 688 5660 FAX: +1 954 345 4668

TEL: +852 2892 0990 FAX: +852 2892 0770

TEL: +49 7121 86 2222 FAX: +49 7121 86 1222

Productspecificationsanddescriptionsinthisdocumentsubjecttochangewithoutnotice2010JDSUniphaseCorporation301682005000810LTEEPCTESTANNSDTMAEAugust2010