KPI measurement methodology. Orange Optimisation.

Document reference: aa.bb.ccc.xxxx Document issue: 01.01 / EN Document status: Draft Date: 1 July 2004

External Document

Copyright 2001 Nortel Networks, All Rights Reserved Printed in France NORTEL NETWORKS CONFIDENTIAL: The information contained in this document is the property of Nortel Networks. Except as specifically authorized in writing by Nortel Networks, the holder of this document shall keep the information contained herein confidential and shall protect same in whole or in part from disclosure and dissemination to third parties and use same for evaluation, operation and maintenance purposes only. The content of this document is provided for information purposes only and is subject to modification. It does not constitute any representation or warranty from Nortel Networks as to the content or accuracy of the information contained herein, including but not limited to the suitability and performances of the product or its intended application. The following are trademarks of Nortel Networks: *NORTEL NETWORKS, the NORTEL NETWORKS corporate logo, the NORTEL Globemark, UNIFIED NETWORKS. The information in this document is subject to change without notice. Nortel Networks assumes no responsibility for errors that might appear in this document. All other brand and product names are trademarks or registered trademarks of their respective holders.

KPI measurement methodology. Orange Optimisation.

PUBLICATION HISTORY
1 July 2004 Issue 01.01 / EN, Draft Creation

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 2/32

KPI measurement methodology. Orange Optimisation.

CONTENTS
1. CS VOICE LONG CALLS...............................................................................................5 1.1. XCAL CONFIGURATION.......................................................................................................5 1.2. METRICS......................................................................................................................5 1.2.1 NCOUP.................................................................................................................5 1.2.2 DL BLER...............................................................................................................6 1.2.3 UL BLER...............................................................................................................7 2. CS64 VIDEO 2 MIN CALLS............................................................................................9 2.1. XCAL CONFIGURATION.......................................................................................................9 2.2. METRICS......................................................................................................................9 2.2.1 CR ........................................................................................................................9 2.2.2 DL BLER.............................................................................................................10 2.2.3 UL BLER.............................................................................................................10 3. PS 128 DL FTP 1M FILE DOWNLOAD........................................................................12 3.1. XCAL CONFIGURATION.....................................................................................................12 3.2. METRICS....................................................................................................................13 3.2.1 DL Throughput....................................................................................................13 3.2.2 DL BLER.............................................................................................................16 3.2.3 CR.......................................................................................................................18 4. PS 64 UL FTP 0.5M FILE UPLOAD.............................................................................20 4.1. XCAL CONFIGURATION.....................................................................................................20 4.2. METRICS....................................................................................................................20 4.2.1 UL throughput.....................................................................................................20 4.2.2 UL BLER.............................................................................................................20 4.2.3 CR.......................................................................................................................21 5. CS / PS SHORT CALLS .................................................................................................22 5.1. XCAL CONFIGURATION.....................................................................................................22 5.2. METRICS....................................................................................................................22 5.2.1 CR.......................................................................................................................22 6. CS 3G/2G HHO................................................................................................................26 6.1. XCAL CONFIGURATION.....................................................................................................26 6.2. METRICS....................................................................................................................26 6.2.1 HHO Success......................................................................................................26 7. ANNEX - EXCLUSION ZONE..........................................................................................27 8. ANNEX DROP AND FAIL CLASSIFICATION .............................................................28

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 3/32

KPI measurement methodology. Orange Optimisation. 9. ANNEX - RF CONFIGURATAION...................................................................................30 9.1. MOBILE DANS VALISE D'ATTENUATION.....................................................................................30 9.2. MOBILE SEUL (TESTS OUTDOOR SANS CHARGE OU OUTDOOR AVEC CHARGE DL) :............................30 9.3. MOBILE SEUL (TESTS INCAR) :..............................................................................................30 10. ANNEX - OCNS.............................................................................................................31

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 4/32

KPI measurement methodology. Orange Optimisation.

1.
1.1.

CS VOICE LONG CALLS

XCAL CONFIGURATION

XCAL Version 1.44 Phone Idle Time: SetUpTime: Traffic Time: Call Count: Voice Call Call Type : Dialed digit : AMR Rate : Call Flow : 10 20 99999 99999 Origination 04 37 50 15 98 (or 08 00 12 34 56 when not available) 12.2 kbps Using Alert Msg

1.2.

METRICS

1.2.1 NCOUP
NCOUP = number of call drops per hour 1. Use XCAP to generate and export the VOICE CALL EVENT table. 2. Calculate Total Call Duration (T). There are two methods that can be applied to do so: a. Sum up the time difference between the Call Drop and Call Start events (i.e. do not consider the Call Fail events). i. If the call ends normally (as in the case of a Stop by User) use the normal end event timestamp. b. Count the number of TX Power samples (binning is done every 1s). 16:28:00 16:28:01 16:28:01 16:28:06 16:28:06 16:28:11 16:42:51 16:42:51 16:43:02 16:43:03 16:43:03 16:43:08 16:43:08 16:43:13 16:48:49 16:48:50 Call Start Connection Request Connection Setup Complete Call Connected Call Establishment Call Connected Call Drop Call End Call Start Connection Request Connection Setup Complete Call Connected Call Establishment Call Connected Call Drop Call End

00:14:51

00:05:47

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 5/32

KPI measurement methodology. Orange Optimisation.

3. Select only the Call Drop events. 4. Analyse each one of the drops identifying RF, UE and System issues. 5. Report all the drops and their causes after investigation. Example : 16:42:51 16:48:49 Call Drop Call Drop 00:14:51 00:05:47 Pas de mesures sur le 144 (com sur 113) No ASU Remove 82 mesures ASU

6. Count the number of DROPs EXCEPT (cf. 8 ANNEX Drop and Fail Classification for details on the Fail/Drop classification): a. Drops occurred in exclusion areas b. Drops due to hardware failures. Document the failure reporting the city, area, drive (out/indoor), measurement protocol, date, site and type of issue. c. Drops/Fails due to calls setup on another frequency d. Drops due to UE issues approved/validated by Orange teams e. Failures on the tools Call Scenario setup f. Events that are wrongly identified as drops by XCAP/XCAL (Normal Call Clearing) NCOUP = DROPs / T

1.2.2 DL BLER
1. Use XCAP and the filtering option using the adequate threshold (cf. 7 ANNEX EXCLUSION ZONE) to generate a table containing : a. scanner Top 1 RSCP data b. BLER info / Channel Record Count / Channel 1 c. BLER info / Channel Record Error / Channel 1 2. Use XCAP to generate and export to Excel the table showing Qualcomm signalling data (layer 3 messages)

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 6/32

KPI measurement methodology. Orange Optimisation. 3. For each call filter out the BLER data generated before the CC DTAP CONNECT message and after the CC DTAP DISCONNECT message, as well as the last point before a drop. For instance, in the example below, exclude the yellow samples : 20:38:58.517 20:39:01.505 20:39:04.000 20:39:06.000 20:39:08.000 20:39:08.479 20:39:08.489 20:39:10.000 20:39:12.000 20:40:56.000 20:40:58.000 20:41:00.000 20:41:01.517 20:41:02.000 20:41:02.128 20:41:03.213 20:41:04.000 20:41:13.165 Call Call 101 100 100 DL UL 101 100 100 100 100 Call 100 Call UL 54 Call Start Connected 5 2 1 DCCH DCCH 2 1 6 0 0 Success 0 End DCCH 29 Start

-58.6 -57.6 -59.1 downlinkDirectTransfer uplinkDirectTransfer -57.9 -51.7 -57.7 -62.1 -59.4 -54.8 uplinkDirectTransfer -55.7

CONNECT CONNECT ACK

DISCONNECT

4. Sum up all transmitted blocks on the traffic channel (Channel Record Count / Channel 1) = BLOCKS 5. Sum up all erroneous blocks on the traffic channel (Channel Record Error / Channel 1) = ERROR DL BLER = 100 * ERROR / BLOCKS Additional analysis (optional): 1. with XCAP generate a map showing the DL BLER geographical distribution 2. generate the cumulative distribution of DL BLER values

1.2.3 UL BLER
1. 2. 3. 4. Collect the RNC call trace data corresponding to the drive test being analysed Import and post-process the call trace data with Nortel RFO tool Generate the average UL BLER value with the RFO UL BLER Analyser. Report Average UL BLER on DCH 1

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 7/32

KPI measurement methodology. Orange Optimisation.

Figure 1 - Example of UL calculation in a PS call

Important Note: It is not possible to exclude the UL BLER samples measured in the exclusion zones with this method. A complex and time consuming method is needed to synchronise UL and DL traces. A tool enhancement has been requested in order to be able to produce BLER distributions.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 8/32

KPI measurement methodology. Orange Optimisation.

2.
2.1.

CS64 VIDEO 2 MIN CALLS

XCAL CONFIGURATION

XCAL Version 1.44 Phone Idle Time: SetUpTime: Traffic Time: Call Count: CS Call Type: Option: DialNum: 10 20 120 99999 Origination at+cbst=116,1,0 atd 06 31 31 11 69 (number of B-Party phone)

2.2.

METRICS

2.2.1 CR
CR = number of calls successfully setup and maintained for 2 minutes / number of call attempts 1. Use XCAP to generate and export the CS CALL EVENT table. 2. Select the Call Drop and Call Fail events. Do not take into account non voice calls: location/routing area updates yyyy-06-dd 20:01:36.357 yyyy-06-dd 20:01:39.479 yyyy-06-dd 20:03:39.358 yyyy-06-dd 20:03:39.968 yyyy-06-dd 20:03:50.999 yyyy-06-dd 20:04:20.999 yyyy-06-dd 20:04:21.609 yyyy-06-dd 20:04:32.650 yyyy-06-dd 20:04:37.374 yyyy-06-dd 20:04:49.256 yyyy-06-dd 20:04:49.866 yyyy-06-dd 20:05:00.910 yyyy-06-dd 20:05:04.965 yyyy-06-dd 20:07:04.910 yyyy-06-dd 20:07:05.519 Call Start Call Connected Call Success Call End Call Start Call Fail Call End Call Start Call Connected Call Drop Call End Call Start Call Connected Call Success Call End

3. Analyse each one of the drops and fails identifying RF, UE and System issues.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 9/32

KPI measurement methodology. Orange Optimisation. 4. Report all the drops and fails and their causes after investigation. Example : 16:42:51 16:48:49 Call Drop Call Fail pas de mesures sur le 144 (com sur 113) Exclusion Zone mesures EZ

5. Count the number of DROPs and FAILs EXCEPT (cf. 8 ANNEX Drop and Fail Classification for details on the Fail/Drop classification): a. Drops/Fails occurred in exclusion areas b. Drops/Fails due to hardware failures. Document the failure reporting the city, area, drive (out/indoor), measurement protocol, date, site and type of issue. c. Drops/Fails due to calls setup on another frequency d. Drops/Fails due to UE issues approved/validated by Orange teams e. Failures on the tools Call Scenario setup f. Events that are wrongly identified as Drops/Fails by XCAP/XCAL (Normal Call Clearing) 6. Count all the Call Start events = EVENTs CR = 100 * ( 1 (DROPs + FAILs) / EVENTs)

2.2.2 DL BLER
The methodology to calculate the DL BLER is the same as the one described in section 1.2.2 except that the transport channel to be used for a CSD RAB is Channel 4. BLER information is given in groups of 200 blocks.

2.2.3 UL BLER
1. 2. 3. 4. Collect the RNC call trace data corresponding to the drive test being analysed Import and post-process the call trace data with Nortel RFO tool Generate the average UL BLER value with the RFO UL BLER Analyser. Report Average UL BLER on DCH 4

Figure 2 - Example of UL calculation in a PS call

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 10/32

KPI measurement methodology. Orange Optimisation.

Important Note: It is not possible to exclude the UL BLER samples measured in the exclusion zones with this method. A complex and time consuming method is needed to synchronise UL and DL traces. A tool enhancement has been requested in order to be able to produce BLER distributions.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 11/32

KPI measurement methodology. Orange Optimisation.

3.
3.1.

PS 128 DL FTP 1M FILE DOWNLOAD

XCAL CONFIGURATION

XCAL Version 1.44 Phone Idle Time: SetUpTime: T.SetUpTime: Traffic Time: Call Count: FTP Call Host: Login ID: Password: Change Dir: Get File: Retry Count: Dormant: Wait PPP Release: Logging FTP data: Pending: 10 60 30 99999 99999 10.66.216.193 urm (urce URCE) xxxxxx downloads 1Mo 1 unticked ticked ticked unticked

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 12/32

KPI measurement methodology. Orange Optimisation.

3.2.

METRICS

3.2.1 DL THROUGHPUT
1. Use XCAP and the filtering option using the adequate threshold (cf. ANNEX EXCLUSION ZONE) to generate a table containing : a. scanner Top 1 RSCP data b. Throughput info / FTP / Total Rx Bytes c. Throughput info / FTP / FWD Throughput The resulting table will show:

2. Report the Average throughput (115.796 kbps in the example above). DL Throughput (kbps) = Average Throughput

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 13/32

KPI measurement methodology. Orange Optimisation.

1.1.1.1

ZERO THROUGHPUT SAMPLES AFTER A DROP

In the calculation of the average Throughput, we count all the samples except those under Exclusion Zones including those that correspond to the calls that dropped (the transfer of the file has not completely finished). We observe that before each drop, the tool will report throughput values equal to 0 during some seconds. This will make that, taking into account our methodology; the average throughput will be biased by those null samples. Thus, the final KPI is degraded. Indeed so many zero samples before a drop are not representative of the average throughput of successful FTP sessions. See below an example taken from a UL FTP (Upload PS): Zero throughput samples are reported from 20:22:30.000 BCCH channel is listened by the mobile at 20:23:20.892 (around 50 seconds later) The Drop is detected by the tool at 20:23:20.892 (when BCCH is received)
UL CCCH DL CCCH UL DCCH UL DCCH DL DCCH UL DCCH DL DCCH UL DCCH UL DCCH DL DCCH UL DCCH DL DCCH rrcConnectionRequest rrcConnectionSetup rrcConnectionSetupComplete initialDirectTransfer downlinkDirectTransfer uplinkDirectTransfer securityModeCommand securityModeComplete uplinkDirectTransfer radioBearerSetup radioBearerSetupComplete downlinkDirectTransfer originatingHighPrioritySignalling cell_DCH SERVICE REQUEST AUTHENTICATION AND CIPHERING REQ. AUTHENTICATION AND CIPHERING RESP.

20:22:19.872 20:22:20.215 20:22:20.334 20:22:20.340 20:22:21.335 20:22:21.513 20:22:23.029 20:22:23.031 20:22:23.306 20:22:25.115 20:22:26.315 20:22:27.265 20:22:27.544 20:22:27.544 20:22:27.984 20:22:27.992 20:22:27.992 20:22:28.909 20:22:29.000 20:22:29.393 20:22:29.393 20:22:29.733 20:22:30.000 20:22:31.000 20:22:32.000 20:23:08.000 20:23:09.000 20:23:10.000 20:23:11.000 20:23:12.000 20:23:13.000 20:23:14.000 20:23:15.000 20:23:16.000
aa.bb.ccc.xxxx

ACTIVATE PDP CONTEXT REQUEST sf16 / sf16 ACTIVATE PDP CONTEXT ACCEPT

IPCP Complete PPP Logon Start PPP Logon End Traffic Setup Start FTP Connecting FTP Connected 0 FTP Load CMD Traffic Start FTP Work Upload 41.632 63.488 58.4 58.4 58.4 46.72 70.08 23.36 0 0 0 0
01.01 / EN Draft

0 Upload

5204 7936 7300 7300 7300 5840 8760 2920 0 0 0 0

1 July 2004 Page 14/32

KPI measurement methodology. Orange Optimisation.

20:23:17.000 20:23:18.000 20:23:19.000 20:23:20.000 20:23:20.892 20:23:20.892 20:23:20.892 20:23:20.936 20:23:20.940 20:23:20.951 0 0 0 0 DL BCCH Drop DUN Release DL BCCH DL BCCH DL BCCH systemInformationBlockType11 Start systemInformationBlockType-m systemInformationBlockType11 systemInformationBlockType11 0 0 0 0

1.1.1.2

SAMPLES OUTSIDE THE ACTUAL DOWNLOAD

The Call Events given by XCAP of a successful Data session are the following ones: TIME_STAMP yyyy-06-dd 13:22:57.230 yyyy-06-dd 13:23:07.231 yyyy-06-dd 13:23:07.274 yyyy-06-dd 13:23:07.550 yyyy-06-dd 13:23:07.550 yyyy-06-dd 13:23:07.589 yyyy-06-dd 13:23:07.617 yyyy-06-dd 13:23:07.626 yyyy-06-dd 13:23:07.633 yyyy-06-dd 13:23:07.635 yyyy-06-dd 13:23:07.644 yyyy-06-dd 13:23:07.645 yyyy-06-dd 13:23:16.162 yyyy-06-dd 13:23:16.162 yyyy-06-dd 13:23:16.488 yyyy-06-dd 13:23:16.493 yyyy-06-dd 13:23:16.493 yyyy-06-dd 13:23:17.445 yyyy-06-dd 13:23:18.214 yyyy-06-dd 13:25:44.493 yyyy-06-dd 13:25:44.494 yyyy-06-dd 13:25:49.125 Data Call Event Idle Setup AT Call Start ATDT AT Connect LCP Start LCP Start LCP Complete Auth Start Auth Complete IPCP Start IPCP Complete PPP LogOn PPP LogOn Traffic FTP Connecting FTP Connected FTP download /upload Start FTP Success Release Call End

Actual FTP Download occurs between the FTP Download/upload Start and FTP Success events. It has been observed that Coueis average FTP when a filter is applied is sometimes pessimistic with regards to a manual calculation using Microsoft Excel. If the average throughput is not within the targets do a detailed throughput analysis between the above two call states taking into account the Exclusion Zones. The Throughput calculation would consist in accounting the bytes downloaded between those two events once the samples recorded in the exclusion zones are removed, and dividing the result by the number of samples (with the binning used, there is one throughput sample per second).
aa.bb.ccc.xxxx 01.01 / EN Draft 1 July 2004 Page 15/32

KPI measurement methodology. Orange Optimisation.

3.2.2 DL BLER
1. Use XCAP and the filtering option using the adequate threshold (cf. 6 ANNEX EXCLUSION ZONE) to generate a table containing the following fields: a. scanner Top 1 RSCP data b. BLER info / Channel Record Count / Channel 4 c. BLER info / Channel Record Error / Channel 4 BLER information is given in groups of 800 blocks. 2. Use XCAP to generate and export to Excel the table showing the Data Call Events. Actual FTP Download occurs between the FTP Download/upload Start and FTP Success events. TIME_STAMP yyyy-06-dd 13:22:57.230 yyyy-06-dd 13:23:07.231 yyyy-06-dd 13:23:07.274 yyyy-06-dd 13:23:07.550 yyyy-06-dd 13:23:07.550 yyyy-06-dd 13:23:07.589 yyyy-06-dd 13:23:07.617 yyyy-06-dd 13:23:07.626 yyyy-06-dd 13:23:07.633 yyyy-06-dd 13:23:07.635 yyyy-06-dd 13:23:07.644 yyyy-06-dd 13:23:07.645 yyyy-06-dd 13:23:16.162 yyyy-06-dd 13:23:16.162 yyyy-06-dd 13:23:16.488 yyyy-06-dd 13:23:16.493 yyyy-06-dd 13:23:16.493 yyyy-06-dd 13:23:17.445 yyyy-06-dd 13:23:18.214 yyyy-06-dd 13:25:44.493 yyyy-06-dd 13:25:44.494 yyyy-06-dd 13:25:49.125 Data Call Event Idle Setup AT Call Start ATDT AT Connect LCP Start LCP Start LCP Complete Auth Start Auth Complete IPCP Start IPCP Complete PPP LogOn PPP LogOn Traffic FTP Connecting FTP Connected FTP download /upload Start FTP Success Release Call End

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 16/32

KPI measurement methodology. Orange Optimisation. 3. For each call, filter out the BLER data generated before the FTP Download/upload Start event and after the FTP Success event as well as the last point before a drop. For instance, in the example below, exclude the yellow samples : TIME_STAMP Record Count Channel 4 45 670 790 800 800 741 714 673 139 36 Record Error Channel 4 0 0 0 0 0 0 0 0 10 36 Top 1 RSCP (dBm) -72.6 -74.9 -72.7 -73.7 -73.3 -72 -66.5 -66.5 -67 -66.3

yyyy-06-dd 13:23:18.000 yyyy-06-dd 13:23:20.000 yyyy-06-dd 13:23:22.000 yyyy-06-dd 13:23:24.000 yyyy-06-dd 13:23:26.000 yyyy-06-dd 13:23:28.000 yyyy-06-dd 13:25:42.000 yyyy-06-dd 13:25:44.000 yyyy-06-dd 13:25:46.000 yyyy-06-dd 13:25:47.000

4. Sum up all transmitted blocks on the traffic channel (Channel Record Count / Channel 4) = BLOCKS 5. Sum up all erroneous blocks on the traffic channel (Channel Record Error / Channel 4) = ERROR DL BLER = 100 * ERROR / BLOCKS Additional analysis (optional): 1. with XCAP generate a map showing the DL BLER geographical distribution 2. generate the cumulative distribution of DL BLER values

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 17/32

KPI measurement methodology. Orange Optimisation.

3.2.3 CR
CR = number of calls successfully setup and maintained until download is completed / number of call attempts 1. Use XCAP to generate and export the DATA CALL EVENT table. 2. Select the Call Drop and Call Fail events. Do not take into account non data calls: location/routing area updates yyyy-06-dd 13:54:09.407 yyyy-06-dd 13:54:19.408 yyyy-06-dd 13:54:19.452 yyyy-06-dd 13:54:19.752 yyyy-06-dd 13:54:19.752 yyyy-06-dd 13:54:19.765 yyyy-06-dd 13:54:19.792 yyyy-06-dd 13:54:19.797 yyyy-06-dd 13:54:19.804 yyyy-06-dd 13:54:19.806 yyyy-06-dd 13:54:19.816 yyyy-06-dd 13:54:19.818 yyyy-06-dd 13:55:19.752 yyyy-06-dd 13:55:19.752 yyyy-06-dd 13:55:21.356 yyyy-06-dd 13:55:22.353 yyyy-06-dd 13:55:32.353 yyyy-06-dd 13:55:32.405 yyyy-06-dd 13:55:32.711 yyyy-06-dd 13:55:32.711 yyyy-06-dd 13:55:32.725 yyyy-06-dd 13:55:32.753 yyyy-06-dd 13:55:32.757 yyyy-06-dd 13:55:32.764 yyyy-06-dd 13:55:32.766 yyyy-06-dd 13:55:32.775 yyyy-06-dd 13:55:32.777 yyyy-06-dd 13:55:40.619 yyyy-06-dd 13:55:40.619 yyyy-06-dd 13:55:40.941 yyyy-06-dd 13:55:40.947 yyyy-06-dd 13:55:40.947 yyyy-06-dd 13:55:42.388 yyyy-06-dd 13:55:43.191 yyyy-06-dd 13:56:49.429 yyyy-06-dd 13:56:49.429 yyyy-06-dd 13:57:52.569 yyyy-06-dd 13:57:53.570 Idle Setup AT Call Start ATDT AT Connect LCP Start LCP Start LCP Complete Auth Start Auth Complete IPCP Start Setup Fail Release Call End Idle Setup AT Call Start ATDT AT Connect LCP Start LCP Start LCP Complete Auth Start Auth Complete IPCP Start IPCP Complete PPP LogOn PPP LogOn Traffic FTP Connecting FTP Connected FTP download /upload Start Drop Release Call End Idle

3. Analyse each one of the drops and fails identifying RF, UE and System issues.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 18/32

KPI measurement methodology. Orange Optimisation. 4. Report all the drops and fails and their causes after investigation. Example : 16:42:51 16:48:49 Call Drop Call Fail pas de mesures sur le 144 (com sur 113) Exclusion Zone mesures EZ

5. Count the number of DROPs and FAILs EXCEPT (cf. 8 ANNEX Drop and Fail Classification for details on the Fail/Drop classification): a. Drops/Fails occurred in exclusion areas a. Drops/Failures due to hardware failures. Document the failure reporting the city, area, drive (out/indoor), measurement protocol, date, site and type of issue. b. Drops/Fails due to calls setup on another frequency c. Drops/Fails due to UE issues approved/validated by Orange teams d. Failures on the tools Call Scenario setup e. Events that are wrongly identified as Drops/Fails by XCAP/XCAL (Normal Call Clearing) 6. Count all the Call Start events = EVENTs CR = 100 * ( 1 (DROPs + FAILs) / EVENTs)

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 19/32

KPI measurement methodology. Orange Optimisation.

4.
4.1.

PS 64 UL FTP 0.5M FILE UPLOAD

XCAL CONFIGURATION

XCAL Version 1.44 Phone Idle Time: SetUpTime: T.SetUpTime: Traffic Time: Call Count: FTP Call Host: Login ID: Pasword: Change Dir: Put File: Retry Count: Dormant: Wait PPP Release: Logging FTP data: Pending: 10 60 30 99999 99999 10.66.216.193 urm (urce URCE) xxxxxx uploads 500k 1 unticked ticked ticked unticked

4.2.

METRICS

4.2.1 UL THROUGHPUT
Use the same methodology as described in section 3.2.1. Just export the Total Tx Bytes instead of the Total Rx Bytes. Therefore, the first step would be: 1. Use XCAP and the filtering option using the adequate threshold (cf. ANNEX EXCLUSION ZONE) to generate a table containing : a. scanner Top 1 RSCP data b. Throughput info / FTP / Total Tx Bytes c. Throughput info / FTP / RVS Throughput

4.2.2 UL BLER
1. 2. 3. 4. Collect the RNC call trace data corresponding to the drive test being analysed Import and post-process the call trace data with Nortel RFO tool Generate the average UL BLER value with the RFO UL BLER Analyser. Report Average UL BLER on DCH 4

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 20/32

KPI measurement methodology. Orange Optimisation.

Figure 3 - Example of UL calculation in a PS call

Important Note: It is not possible to exclude the UL BLER samples measured in the exclusion zones with this method. A complex and time consuming method is needed to synchronise UL and DL traces. A tool enhancement has been requested in order to be able to produce BLER distributions.

4.2.3 CR
CR = number of calls successfully setup and maintained until upload is completed / number of call attempts Refer to section 3.2.3.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 21/32

KPI measurement methodology. Orange Optimisation.

5.
5.1.

CS / PS SHORT CALLS
XCAL CONFIGURATION

XCAL Version 1.44 CS Phone Idle Time: SetUp Time: Traffic Time: Call Count: Voice Call Call Type: Dialed digit: AMR Rate: Call Flow: PS PPP scenario

10 20 10 1 Origination 04 37 50 15 98 (or 08 00 12 34 56 when not available) 12.2 kbps Using Alert Msg

5.2.

METRICS

5.2.1 CR
CR = number of calls successfully setup and maintained following the call scenario / number of call attempts To compute the CR for the CS calls, it is possible to follow the same methodology as described in 2.2.1, i.e. using XCAPs Voice Call Events. However, XCAP does not generate any Data Call Events when the PPP scenario has been used for the PS calls. This means that the process described in 3.2.3 cannot be fully applied here. In order to calculate the CR for this test scenario, it is needed to look at the Layer 3 messages in order to identify the Drops and the Fails of the PS call (calculation of a PS CR). We can use the same methodology to detect the Drops and the Fails for the CS call (CS CR). In order to simplify the procedure it is proposed here to calculate the CS CR and the PS CR using the same table generated once with XCAP. CS CR can also be calculated following the method described in 2.2.1.

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 22/32

KPI measurement methodology. Orange Optimisation. 1. Use XCAP to generate a table with the Logging Message information and filter the Qualcomm Signalling Msg a. In order to save time during the export, you can unselect messages that are not being used during the first steps of the analysis like: i. DL BCCH:BCH ii. DL DCCH activeSetUpdate iii. UL DCCH measurementReport 2. Export to Excel all the messages of the resulting Filtered Message Window (do not select any Export Option). The resulting table will contain the following sequence of messages: b. CS Call is initiated at 23:00:51.980 c. PS Call is initiated at 23:01:20.121
23:00:51.980 23:00:52.301 23:00:52.421 23:00:52.429 23:00:53.075 23:00:53.268 23:00:53.524 23:00:53.524 23:00:53.719 23:00:53.768 23:00:53.770 23:00:54.126 23:00:54.644 23:00:55.705 23:00:57.278 23:00:57.643 23:00:57.653 23:01:07.568 23:01:07.828 23:01:07.871 23:01:08.075 23:01:08.115 23:01:08.202 23:01:08.239 23:01:08.435 23:01:08.572 23:01:20.121 23:01:20.453 23:01:20.557 23:01:20.562 23:01:22.387 23:01:22.556 23:01:24.312 23:01:24.313 23:01:24.512 23:01:27.647 23:01:28.804 23:01:30.433 23:01:31.563 23:01:33.267 23:01:33.471 23:01:33.475 23:01:33.476
aa.bb.ccc.xxxx

rrcConnectionRequest rrcConnectionSetup rrcConnectionSetupComplete initialDirectTransfer downlinkDirectTransfer uplinkDirectTransfer securityModeCommand securityModeComplete uplinkDirectTransfer downlinkDirectTransfer uplinkDirectTransfer downlinkDirectTransfer radioBearerSetup radioBearerSetupComplete downlinkDirectTransfer downlinkDirectTransfer uplinkDirectTransfer uplinkDirectTransfer downlinkDirectTransfer uplinkDirectTransfer rrcConnectionRelease-DCCH rrcConnectionReleaseComplete rrcConnectionRelease-DCCH rrcConnectionReleaseComplete rrcConnectionReleaseComplete rrcConnectionReleaseComplete rrcConnectionRequest rrcConnectionSetup rrcConnectionSetupComplete initialDirectTransfer downlinkDirectTransfer uplinkDirectTransfer securityModeCommand securityModeComplete uplinkDirectTransfer radioBearerSetup radioBearerSetupComplete downlinkDirectTransfer uplinkDirectTransfer downlinkDirectTransfer rrcConnectionRelease-DCCH rrcConnectionReleaseComplete rrcConnectionRelease-DCCH
01.01 / EN

originatingConversationalCall cell_DCH CM SERVICE REQUEST AUTHENTICATION REQUEST AUTHENTICATION RESPONSE

SETUP IDENTITY REQUEST IDENTITY RESPONSE CALL PROCEEDING sf128 / sf64 ALERTING CONNECT CONNECT ACKNOWLEDGE DISCONNECT RELEASE RELEASE COMPLETE normalEvent normalEvent

originatingHighPrioritySignalling cell_DCH SERVICE REQUEST AUTHENTICATION AND CIPHERING REQ. AUTHENTICATION AND CIPHERING RESP.

ACTIVATE PDP CONTEXT REQUEST sf16 / sf16 ACTIVATE PDP CONTEXT ACCEPT DEACTIVATE PDP CONTEXT REQUEST DEACTIVATE PDP CONTEXT ACCEPT normalEvent normalEvent
Draft 1 July 2004 Page 23/32

KPI measurement methodology. Orange Optimisation.

23:01:33.535 23:01:33.631 23:01:33.788 23:01:33.948 rrcConnectionRelease-DCCH rrcConnectionReleaseComplete rrcConnectionReleaseComplete rrcConnectionReleaseComplete normalEvent

3. Use Excels AutoFilter feature to display only lines that containing, in their third column, the messages CM SERVICE REQUEST and DISCONNECT.
initialDirectTransfer uplinkDirectTransfer initialDirectTransfer initialDirectTransfer uplinkDirectTransfer initialDirectTransfer initialDirectTransfer uplinkDirectTransfer initialDirectTransfer uplinkDirectTransfer CM SERVICE REQUEST DISCONNECT CM SERVICE REQUEST CM SERVICE REQUEST DISCONNECT CM SERVICE REQUEST CM SERVICE REQUEST DISCONNECT CM SERVICE REQUEST DISCONNECT

fail/drop

fail/drop

4. If a CM SERVICE REQUEST is not followed by a DISCONNECT, the call is classified as having suffered either a CS Call Fail or a CS Call Drop 5. Use Excels AutoFilter feature to display only lines that containing, in their third column, the messages SERVICE REQUEST and DEACTIVATE PDP CONTEXT ACCEPT.
initialDirectTransfer downlinkDirectTransfer initialDirectTransfer downlinkDirectTransfer initialDirectTransfer initialDirectTransfer initialDirectTransfer downlinkDirectTransfer initialDirectTransfer downlinkDirectTransfer SERVICE REQUEST DEACTIVATE PDP CONTEXT ACCEPT SERVICE REQUEST DEACTIVATE PDP CONTEXT ACCEPT SERVICE REQUEST SERVICE REQUEST SERVICE REQUEST DEACTIVATE PDP CONTEXT ACCEPT SERVICE REQUEST DEACTIVATE PDP CONTEXT ACCEPT

fail/drop fail/drop

6. If a SERVICE REQUEST is not followed by a DEACTIVATE PDP CONTEXT ACCEPT, the call is classified as having suffered either a PS Call Fail or a PS Call Drop 7. Analyse each one of the CS and PS DROPs and FAILs identifying RF, UE and System issues. 8. Report all the CS and PS DROPs and FAILs and their causes after investigation. Example : 16:42:51 16:48:49 Call Drop Call Fail pas de mesures sur le 144 (com sur 113) Exclusion Zone mesures EZ

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 24/32

KPI measurement methodology. Orange Optimisation. 9. Count the number of DROPs and FAILs EXCEPT (cf. 8 ANNEX Drop and Fail Classification for details on the Fail/Drop classification): f. Drops/Fails occurred in exclusion areas g. Drops/Fails due to hardware failures. Document the failure reporting the city, area, drive (out/indoor), measurement protocol, date, site and type of issue. h. Drops/Fails due to calls setup on another frequency i. Drops/Fails due to UE issues approved/validated by Orange teams j. Failures on the tools Call Scenario setup k. Events that are wrongly identified as Drops/Fails by XCAP/XCAL (Normal Call Clearing) 10. Count all the SERVICE REQUEST and CM SERVICE REQUEST messages = EVENTs CR = 100 * ( 1 (DROPs + FAILs) / EVENTs)

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 25/32

KPI measurement methodology. Orange Optimisation.

6.
6.1.

CS 3G/2G HHO
XCAL CONFIGURATION

XCAL Version 1.44 QCT mobile is to be used WITHOUT the external antenna in a INCAR configuration (no attenuation box). Put the test mobile always in the same position inside the car during all the tests. Use a 2G mobile with network monitoring capabilities in order to check the availability of GSM coverage.

Setup a voice call under 3G coverage and drive outside it towards a 2G covered area. Use the long call setup scenario for XCAL. Phone Idle Time: SetUpTime: Traffic Time: Call Count: Voice Call Call Type : Dialed digit : AMR Rate : Call Flow : 10 20 99999 99999 Origination 04 37 50 15 98 (or 08 00 12 34 56 when not available) 12.2 kbps Using Alert Msg

6.2.

METRICS

6.2.1 HHO SUCCESS

HHO Success = percentage of calls initiated under 3G coverage that have been successfully handed over and maintained at least 20 seconds in 2G coverage. TBC

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 26/32

KPI measurement methodology. Orange Optimisation.

7.

ANNEX - EXCLUSION ZONE

Outdoor: Indoor: scanner Top1 RSCP < -98dBm scanner Top1 RSCP < -81dBm ( = -98 -(- 17) )

All acceptance tests are done with simulated DL and UL load. Exclusion zones are defined as:

Samples within exclusion zones can be filtered out using exports to Excel (UE data is synchronised with Scanner Anritsu Top 1 RSCP values) or thanks to the FILTERING option in XCAP. Filters can be defined as follows:

Figure 4 - Filtering option Outdoor Exclusion Zone To apply a filter in a table, we select the field to drag on the right hand side of XCAP window, right click and tick the filter we want to apply. The table will contain only samples that follow the filter rules.

Figure 5 - Example of applying Filter EZ in XCAP

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 27/32

KPI measurement methodology. Orange Optimisation.

8.

ANNEX DROP AND FAIL CLASSIFICATION

If the column COUNT has a N, the Drops/Fails classified in that category are not to be accounted in the KPI formula after agreement with Orange teams.

Class

Sub Class

Count (Y/N)

Failure Name Exclusion Zone

Failure Description Top 1 RSCP from Scanner Threshold is below -98 dBm for Outdoor and -81 dBm for Indoor. Moving the antenna location or changing the antenna type may improve coverage and provide better dominance in pilot polluted area. Change has been approved by Orange but not implemented during the audit. An Area defined by high number (> 3) of SC with equal RSCP and Ec/Io Values with no dominant SC SC not included in the reference SC's neighbor list Overshoot is defined as a cell who's coverage extends beyound the second tier of the cell There is no adequate overlap between the sectors. The efect is strong SC seen by UE very abruptly causing fast interference that causes a drop Other Radio causes not listed before. Includes All fails (R1-R4) that would not otherwise happen if the PA site was present. Document properly this type of fail. Loss of communication from RNC to Node-B. The Cell is still radiating causing interference Cell is not radiating Cell is radiating but UE can not hand-in or Hand-out TMA is off or alarmed. Usually indicated by a high UL_Interference value in the SIB 7 Message on the BCH channel Poor Ec/Io and or RSCP in line of sight to cell Neighbor List has 16 neighbors

RSCP < -98 Outdoor RSCP < -81 Indoor RF Design Antenna Location / Type

Antenna choice

Pilot Pollution Missing Neighbour Overshoot Radio Fails

Pilot Pollution Missing Neighbor Y Overshoot

No Overlap

No Overlap

Other Missing PA site RSCP>-98

Other Drops due to Missing PA site LL Down / RF ON Site Down / RF OFF

Networ k Fails

Site Down/Hardware Fails

Island Cell Bad TMA Bad antenna/RF Cable

Syste m Fails

NL Full

Neighbor List Full

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 28/32

KPI measurement methodology. Orange Optimisation.

Missing Active Set Update Incorrect ASU decision Wrong Cell LAC Measurement report indicates a SC that meets the add criteria but no active set update is received by the UE Wrong ASU message received taking into account what the UE is sending in the Measurement Reports. The location areas code of the cell is not correct. The result is a call drop or access failure. The cell may behave like an island cell Describe and document every DROP/FAIL not classified The mobile goes to the SFR frequency. The mobile goes to the integration frequency The UE does not find a strong SC yet the scanner finds it. Found with software release 5.1.48 UE has only 1 radio link, CPICH RSCP and Ec/Io are good, UE Tx power is OK, but DL BLER increases. Most likely unexplained QCT defect. PDP Context Activation fails when done just after a PS Attach or RAU Request with FOR bit set to 0. XCAP reports dial fail During a UE mobile origination there are no rrcConnect messages. Data collection has to be stopped and computer re-booted

ASU

Datafill Other

Y Y

Wrong Frequency

SFR Frequency Integration Frequency UE SC Detection Fail High DL BLER in good RF not in SHO FOR bit to 0 Dial Fail

Mobile Fails & Tools

Mobile Issues Pending

Mobile Issues Approved

XCAL/XCAP

No rrcConnect Request Need further investigatio n RRC Release Cause unspecified End by User Distant clearing

Not Classe d RRC Releas e Unspe cified Normal Call Clearin g

Investigation in progress

All Failures that can not be classified and require further study

rrcReleaseUnsp

RRC Release Cause unspecified probably due to UL SYNCH lost.

End by User N Distant clearing

Normal end by user call clearing Normal call Termination: DTAP UL/DL DISCONNECT sent/received

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 29/32

KPI measurement methodology. Orange Optimisation.

9.
9.1.

ANNEX - RF CONFIGURATAION
MOBILE DANS VALISE D'ATTENUATION

Scanner configuration: Scanner User Correction Value : -3dB (*) (pour enlever 3dB a la mesure du scanner pour compenser les pertes de la boite) Scanner Antenna Adjust : DISABLED (pour ne pas corriger le gain d'antenna car le mobile ne corrige pas cette valeur)

(*) If the attenuation box has around 3dB losses. Put here the losses of the box. Chaine att. UE : +5 (antenne) -2 (cable) -3 (valise) = 0 Chaine att. Scanner : +5 (antenne) -2 (cable) -3 (correction) = 0 Tests Outdoor sans charge : Tous les attenuateurs de la valise a 0dB Tests Outdoor avec charge : Attenuateur path UL a 3dB Tests Indoor sans charge : Attenuateur path UL a 17dB + Attenuateur path DL a 17dB Tests Indoor avec charge : Attenuateur path UL a 20dB + Attenuateur path DL a 17dB

9.2.

MOBILE SEUL (TESTS OUTDOOR SANS CHARGE OU OUTDOOR AVEC CHARGE DL) :

Scanner configuration: Scanner User Correction Value : 0dB Scanner Antenna Adjust : DISABLED (pour ne pas corriger le gain d'antenna car le mobile ne corrige pas cette valeur)

Chaine att. UE : +5 (antenne) -2 (cable) = +3 Chaine att. Scanner : +5 (antenne) -2 (cable) = +3 Avec cette configuration nous mettons le mobile et le scanner sous les memes conditions. Cette configuration n'est pas actuellement utilise en UR CE. Inconvenient : les mesures du scanner du parcours fait avec la valise seront differentes de celles faites sans la valise. Les signaux sont "theoriquement" amplifies de 3dB. L'information est a traiter avec precaution.

9.3.

MOBILE SEUL (TESTS INCAR) :

Ce n'est pas une configuration "standard" donc a traiter cas par cas. L'attenuation incar est difficile a caracteriser donc nous ne pourrons pas compenser correctement la mesure scanner. Il est donc recommende d'utiliser : Scanner User Correction Value : -3dB

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 30/32

KPI measurement methodology. Orange Optimisation.

10. ANNEX - OCNS

The OCNS is activated during the test under loaded conditions and set at 50% of the maximum Node B transmitted carrier power. Following the agreed recommendations we use the activity factor hypothesis indicated below when calculating the power used in average by the Common Channels. Example (50% load):
dBm 42.1 dBm 33 31 28 28 26 28 31 W 16.218 W Activity Factor 1.995 1 0.9 1.259 0.1 0.631 0.1 0.631 0.1 0.398 0.96 0.631 0.1 1.259 CommonChannelsPower Watt 1.995 1.133 0.063 0.063 0.040 0.606 0.126 4.026

maxTxPower Common Channels Primary CPICH Power P-CCPCH Power / BCH P-SCH Power S-SCH Power AICH Power PICH Power SCCPCH Power

= Pcc

CommonChannelsPower to be used Traffic Power % LOAD vs maxTxPower LOAD Power Watt PowerOCNS Watt 40.9

4.026 12.192 50% 8.109 4.083

36.1

= LOAD Pwr - Pcc

%OCNS Power

33%

= PowerOCNS vs Traffic Power

The power actually reserved (but not necessarily radiated) for the common channels at the BTS is: Pcc = Pcpich power + max {Pccpch power ; (Psch power + Ssch power)} + aichPower + pichPower + sccpchPower

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 31/32

KPI measurement methodology. Orange Optimisation.

END OF DOCUMENT

aa.bb.ccc.xxxx

01.01 / EN

Draft

1 July 2004

Page 32/32