NSO Service Delivery

UMTS Drive Testing & Benchmarking -release 4Applies to TEMS 2.3.1 20 July 2004

Razvan Georgescu

Motorola Confidential Page 1

NSO Service Delivery

Table of Content
Table of Content ......................................................................................................................................... 2 1. Introduction ...................................................................................................................................... 3 2. Executive Summary ....................................................................................................................... 3 2.1 Summary Report Format - Example ........................................................................................ 3 2.2 Deliverables ................................................................................................................................. 3 2.3 Tools ............................................................................................................................................. 3 3. Drive Test Setup ............................................................................................................................. 4 3.1 Call scenarios .............................................................................................................................. 4 3.2 Drive test routes .......................................................................................................................... 5 3.3 Duration ........................................................................................................................................ 5 3.4 Test Case Description................................................................................................................ 5 4. Drive Test Analysis ......................................................................................................................... 7 4.1 TEMS Outputs ............................................................................................................................. 7 4.2 Formulas, Counters, Measurements ...................................................................................... 7 4.2.1 Formulas - Voice calls MO .................................................................................................... 7 4.2.2 Formulas - Data calls MO..................................................................................................... 7 4.2.3 Voice calls MO TEMS Raw Output ................................................................................... 8 4.2.4 Data Calls TEMS Raw Output ........................................................................................... 9 4.2.5 Radio Health Indicators ...................................................................................................... 12 4.3 SMAP Outputs .......................................................................................................................... 14 4.4 NRG Outputs ............................................................................................................................. 14 4.5 Pareto Analyis Outputs ............................................................................................................ 14 5. Equipment List .............................................................................................................................. 15 6. Appendix ........................................................................................................................................ 16

Motorola Confidential Page 2

NSO Service Delivery

1. Introduction
This document intends to give a general guideline for collecting drive test data with the primary scope of collecting KPI ( benchmarking) and secondary to collect basic call log data

2. Executive Summary
Summary Report Format - Example

CSSR (average)
LOAD RAB
AMR MOC AMR MTC AMR MTM AMR Total CS 64 MTM PS64 PS128 PS384 (static) 0% 98,25 98,58 97,78 98.18 97,76 100,00 97.16 100,00 Call Attempts 343 212 270 825 268 44 423 154 Requirement 97 97 97 97 95 97 97 97

DCR (average)
LOAD RAB
AMR CS 64 PS64 PS128 PS384 (static) 0% 1,2219 0,57416 0 0.97 0 Successful Call Setups 1064 406 44 411 154 GSD Req. 3 5 3 3 3

Deliverables
KPI counters Logs for offline analyisis and post processing stored at a Motorola location and available on the intranet Pareto for call failure causes

Tools
Note: At this date TEMS 2.4 is available, however no time for tests yet, release notes do not suggest changes in the event definition which could impact the KPI calculation described further in thei document.

Motorola Confidential Page 3

NSO Service Delivery

Tools to be used: TEMS Investigation WCDMA 2.3.1 , Motorola A835 phones with NETMON - UE logging and call counting SMAP RNC logging NRG post processing

System Retainability & System Accessibility statistics will be generated from calls which are 2 minutes long for all types of calls, voice, video, data in order to align to the test calls done in Porto by GSD. This is to have KPIs deducted from the same type of calls in each location subjected to GSD aligned benchmarking. If this alignment is not needed, then the TEMS Practices document describes an alternative procedure which is also used by other major UMTS test equipment manufacturers.

3. Drive Test Setup

Call scenarios
All tests done in car and in motion. Voice calls & Video calls: 2 phones simultaneously MS1 calls MS2 both in the same car. This is an atipycal subscriber behaviour. This scenario is adopted though for testing purposes ,because we need to log with 1 TEMS kit both the call originating mobile and the paged mobile with the scope of collecting logs for debugging. KPIs only for paging can be deduced much easily only using UTRAN stats.

120 sec calls + 30 sec breaks

Data calls: - 1 phone FTP, the data quantity adapted for each bearer speed. 2 minutes long + 30 seconds break UL FTP tests also to be included UDP as required

Motorola Confidential Page 4

NSO Service Delivery

Drive test routes

To ensure proper functionality validation of the UTRAN, the drive test routes for KPI benchmarking should be done in areas with proper coverage. For orientation we define the following values to be measured outdoor. o Good RF coverage, UTRAN Carrier RSSI > -70 dBm, CPICH Ec/Io > 10 dB, 1/2/3 way SHO o Degraded RF coverage, UTRAN Carrier RSSI < -90 dBm, CPICH Ec/Io = -10 dB to 14 dB

Duration
TBD. Depending on call volume.

Can be from minimum 3 days to more than 1 week for the complete set of tests. Remark: Reports must be delivered weekly. This will be a live table where KPIs for different calls must be updated daily as the revelant test was performed.

Test Case Description

All tests done in car and in motion. with the possible exception of 384K A description of the initial test cases is in the follwing table. Please treat this as a guideline to be adapted locally according to the needs and resources.

Motorola Confidential Page 5

NSO Service Delivery

Which mobile

Scope of Calls

Call Sequenc e Descripti on

Call Sequence Handling

Log Ue

Log UTRA N

Post Proce ssing

Minimum time needed for execution 5 hrs absolute minimum effective drive time

Comments

Day 1, Voice Calls MS1 Motion to MS2 Motion CSSR, DCR, Pareto 120 sec call + 30 sec break TEMS logging both mobiles

Practically takes whole day.

100 MTM voice call

automatic

SMAP

TEMS NRG 5 hrs absolute minimum effective drive time

Day 2, Video Calls MS1 Motion to MS2 Motion 120 sec call + 30 sec break TEMS logging both mobiles

100 MTM video call Day 3, Voice Calls MO to fixed + Downloads

CSSR, DCR, Pareto

manual

SMAP

TEMS ,NRG

5 hrs absolute minimum effective drive time MS1 Motion to fixed line (answerin g machine, or TEST number ) MS2 Motion ftp ownload 128 K (or preffered service)

100 MO voice call

CSSR, DCR, Pareto

120 sec call + 30 sec break

automatic

TEMS

SMAP

TEMS NRG

Need fixed line and answering machine or TEST number

100 Download sessions Day 4, Uploads (+ optional MO to fixed) 100 Upload sessions TOTAL= 6 working days

CSSR, DCR, Pareto

120 sec download + 30 sec break

automatic

TEMS

5 hrs absolute minimum effective drive time MS2 Motion ftp upload 64 K CSSR, DCR, Pareto 120 sec upload + 30 sec break TEMS NRG

automatic

SMAP

Motorola Confidential Page 6

NSO Service Delivery

Notes:
1. Day 5 & 6, Additional tests to be performed as needed on non preffeded data bearers - 64 K 384 K, UDP 2. TOTAL= 6 working days

3. The TEMS logging and UE control is done according to the general TEMS practices, for details please review Drive Testing and Benchmarking TEMS practices. 4. SMAP, NRG usage are subject to usage guidelines from GSD who are the tool developers. Any issues related to these tools should be directed to GSD. 5 .FULL BENCHMARKING of all services requires more than one working week with 1 team or alternative 2 teams or reduce call volume as per TEMS Practices if the performance is good and no alignment is required.

4. Drive Test Analysis

TEMS Outputs
Note: All these reports are generated using the Report Generator function in TEMS. Reports can be generated by anyone having TEMS instabbeld on his PC, without the dongle.

Formulas, Counters, Measurements Formulas - Voice calls MO

CSSR = [Call Established / Call Initialization ]* 100 rounded at 2 decimals

DCR = [Dropped Call / Call Established ] *100 rounded at 2 decimals

Formulas - Data calls MO

CSSR = [Session Start / RAS Dial ]* 100 rounded at 2 decimals

DCR = [Session Error / Session Start] *100 rounded at 2 decimals

Motorola Confidential Page 7

NSO Service Delivery

Voice calls MO TEMS Raw Output

Event Blocked Call Call Established Call Initialization Call End Call Setup Dropped Call Missing Neighbor Detection Call Attempt RRC Established

#[no.of] 6 75 82 66 76 9 0 85 76

#Log 2 1 ,2 1 ,2 1 ,2 1 ,2 2 1 ,2 1 ,2

CSSR = [Call Established / Call Initialization]* 100 = [75 / 82 ]*100 = 91.5 % rounded at 2 decimals

DCR = [Dropped Call / Call Established ] *100 = [9 / 75 ]*100 =12 % , rounded at 2 decimals Setup Times

Motorola Confidential Page 8

NSO Service Delivery

Data Calls TEMS Raw Output

Event

#[no.of]

#Log 1 1 1 1 1 1 1 1 1 1 1 1

Packet Service Attach 1 Packet Service Attach 0 Failure Packet Service Detach PDP Context Activation PDP Context Activation Failure PDP Context Deactivation RACH Error RRC Connection Abnormal Release RRC Connection Reject RRC Established RAS Error RAS Dial RAS Hangup IP Address Assigned Session Start Session End Session Error 0 7 0 7 0 5 1 6 0 7 7 7 7 6 1

A normal Call Sequence for Data Calls is:

DATA CALL SETUP PART

Motorola Confidential Page 9

NSO Service Delivery

RAS Dial .. . based on the fact that the phone responds properly to the RAS Access from the PC PDP Context Activate IP Address Assigned Session Start .. .. FTP Get starts here .. I have successfully set up a Data Call, this is including also FTP server authentication which prooves that my data pipe is usable and I do not just have a blind PDP Context. If the call fails during server authentication before Session Start, this is technically a dropped call. The user may interpret it as a setup failure, that is why if the counters for IP Address Assigned and Session Start are not equal, these cases are recommended to be analysed by replaying the logs. DATA CALL END PART

Session End . The Data Transfer was successfull / or Session Error- Call is Dropped RAS Hangup release RAS from PC PDP Context Deactivated . end PS connection

If Session Start counter = IP Address Assigned counter CSSR = [Session Start / RAS Dial ]* 100 =xy.z % rounded at 2 decimals If Session Start < IP Address Assigned , analyse and adjust the Call Setup figure and the DRC figure accordingly to compensate. Note: Low variance is to be expected here. Generally if IP Address Assigned but Session cannot Start then there is a functionality issue in the PS core or FTP server. DCR = [Session Error / Session Start] *100 = xy.z % , rounded at 2 decimals

Link Reestablishments : these are meant to save the PS call if the RL fails, due to the specifics of the user data, such a procedure is not realistic for voice / video CS calls. Reestablishments are not to be considered into the DRC statistics. If the Reestablishments are too often, causes must be analysed.

Visual investigation of the data transfer behaviour:

Motorola Confidential Page 10

NSO Service Delivery

The 64 K bearer. On the graph it can be seen at 58 kbps app throughput (ftp) The 128 K bearer is reaching above 110 k the horizontal blue line is showing that. The 384K bearer is not really stable like the 128k and 64k. It suffers either from reconfiguration back to 128 triggered every few seconds ( without dropping, which is good) or when it is stable assigned the throughput stays usually above 250 kbps, mostly above 300.

Setup Times Attach

Motorola Confidential Page 11

NSO Service Delivery

PDP Context Activation

Radio Health Indicators

Below only some examples which can help you grossly evaluate the quality of the radio environment in a network.

Motorola Confidential Page 12

NSO Service Delivery

Motorola Confidential Page 13

NSO Service Delivery

SMAP Outputs
Documented by GSD on request.

NRG Outputs
Documented by GSD on request.

Pareto Analyis Outputs Attached the original Pareto template click to open.

Motorola Confidential Page 14

NSO Service Delivery

5. Equipment List
Item

Equipment Description 1. Car 2. DC/ AC 12V >> 100240 V inverter

Qty 1 1

Comments Renault Espace or similar MPV For use with the equipment not running on accus or requiring a usage time longer than possible with own accus For Flashing and Data Transfer

3. USB Cables for A835 for Flash and Data 2 4. Portable USB Hub 5. Serial GPS for TEMS 6. TEMS kits + 2 x A835 TEMS phones 7. A835 phones 8. Digital Maps for TEMS 9. Laptops for drive testing 10. Iu Analyser 11. SIM cards 12. Additional accus for A835 13. Dongles for RTA/PST/Radicom 14. PSTN line 15. NRG 16. SMAP 17. Laptop Holder 18. 1 1 3 2 4 1 1 1 1 1 1 1

For Drive Testing 1 per engineer

Min PIII with accus lasting more than 2 hrs. For IuPS, CS, IuB Agilent ( CS& 64 + CS&128 + CS&384) 1 extra per phone 1 per Motorola location For Mobile Terminated Calls

For incar

Motorola Confidential Page 15

NSO Service Delivery

6. Appendix
Tools package available in compass at http://compass.mot.com/go/140807456

Motorola Confidential Page 16