Documente Academic
Documente Profesional
Documente Cultură
..
..
..
..
Core RF Engineering
Dept. 5395
Product Line:
Version:
Issue Date:
Author:
Department:
Authorizing Manager:
Table of Contents
TABLE OF CONTENTS ............................................................................................................................. 2
1
INTRODUCTION ................................................................................................................................ 8
1.1
1.2
1.3
1.4
1.5
SHAKEDOWNS ................................................................................................................................. 65
6.1
6.2
OBJECTIVES ................................................................................................................................... 65
EVDO SITE SHAKEDOWN PROCESS .............................................................................................. 65
6.3
EVDO INTER SUBNET ACTIVE HANDOFF (ISSHO) SHAKEDOWN PROCESS .................................. 69
6.4
(A13) DORMANT SESSION HANDOFF SHAKEDOWN PROCESS ........................................................ 71
6.5
DATA PROCESSING AND ANALYSIS ............................................................................................... 73
6.5.1
EVDO Site Shakedown Analysis ........................................................................................... 73
6.5.2
Inter RNC Active Handoff Shakedown Analysis ................................................................... 77
6.5.3
A13 Dormant Session Handoff Analysis ............................................................................... 78
7
INTRODUCTION .............................................................................................................................. 79
FTP SERVER CONFIGURATION ...................................................................................................... 79
GOLDEN VALUE TESTING PROCESS ............................................................................................... 79
DATA PROCESSING AND ANALYSIS ............................................................................................... 82
10
TROUBLESHOOTING............................................................................................................... 103
10.1 SESSION SETUP AND A13 HANDOFF SUCCESS RATE ................................................................... 103
10.1.1 Data Collection Methods.................................................................................................... 103
10.1.1.1
10.1.1.2
10.1.1.3
10.1.2
10.1.2.1
10.1.2.2
10.1.2.3
10.1.2.4
AT Logs..................................................................................................................................... 103
RNC Logs .................................................................................................................................. 103
OMs ........................................................................................................................................... 109
10.2.2
10.2.2.1
10.2.2.2
10.2.2.3
10.2.2.4
AT Logs..................................................................................................................................... 119
RNC Logs .................................................................................................................................. 119
OMs ........................................................................................................................................... 119
10.3.2
10.3.2.1
10.3.2.2
10.3.2.3
10.3.2.4
AT Logs..................................................................................................................................... 120
RNC Logs .................................................................................................................................. 120
OMs ........................................................................................................................................... 120
10.4.2
10.4.2.1
10.4.2.2
10.4.2.3
10.4.2.4
AT Logs..................................................................................................................................... 122
RNC Logs .................................................................................................................................. 124
OMs ........................................................................................................................................... 125
10.5.2
10.5.2.1
10.5.2.2
10.5.2.3
10.5.2.4
AT Logs..................................................................................................................................... 133
RNC Logs .................................................................................................................................. 133
Blocking OMs............................................................................................................................ 134
10.6.2
10.6.2.1
10.6.2.2
10.6.2.3
10.6.2.4
AT Logs..................................................................................................................................... 138
RNC Logs .................................................................................................................................. 139
Dropped Connection OMs ......................................................................................................... 140
10.7.2
10.7.2.1
10.7.2.2
10.7.2.3
10.7.2.4
AT Logs..................................................................................................................................... 154
RNC Logs .................................................................................................................................. 154
OMs ........................................................................................................................................... 154
AT Logs..................................................................................................................................... 155
RNC Logs .................................................................................................................................. 155
10.8.1.3
10.8.2
10.8.2.1
10.8.2.2
10.8.2.3
10.8.2.4
10.9.2
AT Logs..................................................................................................................................... 157
RNC Logs .................................................................................................................................. 157
OMs ........................................................................................................................................... 158
10.9.2.1
10.9.2.2
10.9.2.3
10.9.2.4
10.10.2
10.10.2.1
10.10.2.2
10.10.2.3
10.10.2.4
AT Logs..................................................................................................................................... 160
TCP Trace Logs......................................................................................................................... 160
RNC Logs .................................................................................................................................. 162
OMs ........................................................................................................................................... 162
10.10.3
Troubleshooting Guide ................................................................................................... 163
10.11
FL PER-SECTOR THROUGHPUT................................................................................................ 180
10.11.1
Data Collection Methods................................................................................................ 180
10.11.1.1
10.11.1.2
10.11.1.3
10.11.2
10.11.2.1
10.11.2.2
10.11.2.3
10.11.2.4
AT Logs..................................................................................................................................... 180
RNC Logs .................................................................................................................................. 180
OMs ........................................................................................................................................... 180
10.11.3
Troubleshooting Guide ................................................................................................... 181
10.12
RL PER-USER THROUGHPUT ................................................................................................... 181
10.12.1
Data Collection Methods................................................................................................ 181
10.12.1.1
10.12.1.2
10.12.1.3
10.12.2
10.12.2.1
10.12.2.2
10.12.2.3
10.12.2.4
AT Logs..................................................................................................................................... 181
RNC Logs .................................................................................................................................. 182
OMs ........................................................................................................................................... 182
10.12.3
Troubleshooting Guide ................................................................................................... 183
10.13
RL PER-SECTOR THROUGHPUT ............................................................................................... 185
10.13.1
Data Collection Methods................................................................................................ 185
10.13.1.1
10.13.1.2
10.13.1.3
10.13.2
10.13.2.1
10.13.2.2
AT Logs..................................................................................................................................... 185
RNC Logs .................................................................................................................................. 185
OMs ........................................................................................................................................... 185
10.13.2.3
10.13.2.4
10.13.3
Troubleshooting Guide ................................................................................................... 186
10.14
FL ACHIEVABLE DATA RATE .................................................................................................. 186
10.14.1
Data Collection Methods................................................................................................ 186
10.14.1.1
10.14.1.2
10.14.1.3
10.14.2
10.14.2.1
10.14.2.2
10.14.2.3
10.14.2.4
AT Logs..................................................................................................................................... 186
RNC Logs .................................................................................................................................. 186
OMs ........................................................................................................................................... 186
10.14.3
Troubleshooting Guide ................................................................................................... 187
10.15
RL ACHIEVABLE DATA RATE ................................................................................................. 191
10.15.1
Data Collection Methods................................................................................................ 191
10.15.1.1
10.15.1.2
10.15.1.3
10.15.2
10.15.2.1
10.15.2.2
10.15.2.3
10.15.2.4
AT Logs..................................................................................................................................... 191
RNC Logs .................................................................................................................................. 191
OMs ........................................................................................................................................... 191
10.15.3
Troubleshooting Guide ................................................................................................... 192
10.16
PACKET LATENCY ................................................................................................................... 193
10.16.1
Data Collection Methods................................................................................................ 193
10.16.1.1
10.16.1.2
10.16.1.3
10.16.2
10.16.2.1
10.16.2.2
10.16.2.3
10.16.2.4
AT Logs..................................................................................................................................... 193
RNC Logs .................................................................................................................................. 193
OMs ........................................................................................................................................... 193
10.16.3
Troubleshooting Guide ................................................................................................... 194
10.17
DO-TO-1X HANDOFF DELAY .................................................................................................. 194
10.17.1
Data Collection Methods................................................................................................ 194
10.17.1.1
10.17.1.2
10.17.1.3
10.17.2
10.17.2.1
10.17.2.2
10.17.2.3
10.17.2.4
AT Logs..................................................................................................................................... 194
RNC Logs .................................................................................................................................. 195
OMs ........................................................................................................................................... 195
10.17.3
Troubleshooting Guide ................................................................................................... 195
10.18
DO-TO-1X SUCCESS RATE ...................................................................................................... 195
10.18.1
Data Collection Methods................................................................................................ 195
10.18.1.1
10.18.1.2
10.18.1.3
10.18.2
10.18.2.1
10.18.2.2
10.18.2.3
10.18.2.4
10.18.3
AT Logs..................................................................................................................................... 195
RNC Logs .................................................................................................................................. 195
OMs ........................................................................................................................................... 196
Introduction
This document provides a guideline for RF engineers to do the RF optimization of a 1xEV-DO system.
1xEV-DO system is optimized to carry non real-time packet data services and hence does not support voice
services. In the forward link, 1xEV-DO air interface is designed to provide up to 2.4 Mbps data rate with
only one 1.25 MHz carrier using TDM (Time Division Multiplexing) system. In the reverse link, 1xEV-DO
provides up to 153.6 kbps data rate similar to 1xRTT. Since the forward link physical layer implementation
of 1xEV-DO and CDMA/1xRTT are different, they cannot exist in the same frequency. 1xEV-DO has to
be deployed in a separate carrier.
1.1
Related Documents
[1] 1xEV-DO Coverage and Capacity Performance, v2.0, Muhieddin Najib, Core RF
Engineering, February 2004 (http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/rf/lb)
[2] 1xEV-DO Provisioning Guideline, v2.1.03, Kenneth Ho, Core Network Engineering, August
2004 (http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/dim/gl)
[3] 1xEV-DO RF Parameters and Datafill Guide, v3.01, Mike Garramone, Core RF
Engineering, October 2005.
(http://livelink.us.nortel.com/livelink/livelink.exe?func=ll&objId=9655009&objAction=browse&s
ort=name)
[5] QTP-5500 Access Terminal Users Guide, 80-B1311-2, Rev A, Qualcomm, March 2004.
[6] 1xRTT and 1xEV-DO RF Link Budgets, Muhieddin Najib, Core RF Engineering, October
2004 (presentation to Orange).
[7] 1xEV-DO RF System Performance, Miro Budic, Core RF Engineering, October 2004
(presentation to Orange).
[8] 1xEV-DO RF Parameters, Brian Troup, Core RF Engineering, September 2004
(presentation in Verizon Users Group).
[9] Inter-RNC Active Handoff RF Engineering Guideline, Version 0.02, Brian Troup, October
2005.
(http://livelink.us.nortel.com/livelink/livelink.exe?func=ll&objId=9655009&objAction=browse&s
ort=name)
[10] Optis-M Quick Start Guide, v1.05, WirelessLogix, 2004 [11] OPTis-M User Guide,
v1.05, WirelessLogix, 2004
[11] Maximum 1xEV-DO Application Layer Throughput, Dept. 2N51, October 2002.
[12] Generic Network Acceptance Process 1xEVDO Network, v4.1, Mike Woodley, January
2005.
[13] Microsoft Windows 2000 TCP/IP Implementations Details, Dave MacDonald and Warren
Barkley, 2005 Microsoft Corp.
[14] RFC 3481 TCP over Second (2.5G) and Third (3G) Generation Wireless Networks,
www.faqs.org/rfcs/rfc3481.html
[15] TAP Users Guide, Rev. A, Airvana, April 22, 2001
[16] 1xEVDO Troubleshooting Guide, Std 2.05, NTP 411-2133-949, April 2005
[17] 1xEVDO Performance Measurements, Draft 4.07, NTP 411-2133-924, September 2005.
[18] Invex3G Quick Start Guide, version 4.3, Andrew Corporation, 2005.
[19] 1xEV-DO CELLDM Logging Feature Application Guide, Wei Lou, Version 0.1, October
20, 2005.
[20] A13 Dormant Handoff and UA10 Minimization Application Guide, Miroslav Budic, Draft,
September 23, 2005.
1.2
Scope
1.3
Revision History
Issue
1.0
1.1
Date
November 4, 2005
March 3, 2006
Reason
Initial release for EVDO rel. 3.0.
Updated to reflect EVDO 3.0 ChR.
1.4
Contributors
The following are the key contributors to the contents of the document:
1.5
Rubianto Satrio
Wei Lou
Brian Troup
Miro Budic
Martin Kendall
Mike Woodley
Alexander Contreras
Saied Kazeminejad
Gordon Edwards
Remo Agostino
Audience
This document is intended for Nortel Networks and customer RF engineers that are involved in the 1xEVDO deployment and want to understand how to optimize a 1xEV-DO network. A basic understanding of
IS95 and IS2000, as well as general RF principles is expected of the reader.
THIS VERSION IS CURRENTLY FOR INTERNAL AUDIENCES ONLY
Y
Review Customer CIQ responses and
current network provisioning
(Section 3.1.5)
Y
RF Datafill Parameters Review and
Verification (Section 4.1)
Exit Market
Exit Criteria
Satisfied?
Perform Troubleshooting
(Section 10)
10
3.1
Entrance Criteria
In general there are four important steps that need to be done prior to the 1xEV-DO RF optimization:
Review the 1xEV-DO RF design (in iPlanner) to make sure that it meets the coverage and
capacity/throughput objectives.
Check that the 1xEV-DO network design matches the requirements listed in the CIQ (Customer Input
Questionnaire) response.
Ensure that the recommended datafill is used for the initial RF parameters.
1xEV-DO basic system readiness checkup is conducted.
The 1xEV-DO drive test routes are determined and agreed upon.
Exclusion criteria and warranty boundaries agreed upon by Nortel and the customer
Exit Criteria
The detailed exit criteria need to be negotiated between Nortel Networks and the customer. They have to be
agreed upon prior to the 1xEV-DO RF optimization effort. Metrics that can be used for this purpose
include:
Data-rate-to-DRC matching (i.e., showing that the decoded data rate received matches the DRC
requested).
Single-user application throughput under a certain RF and mobility condition.
Access failure rate.
Connection drop rate.
Connection setup time.
3.1.1
RF Design Review
Just like in CDMA, a good 1xEV-DO network starts with a good 1xEV-DO RF design. Therefore, the first
step in 1xEV-DO RF optimization should be to review the final RF design. Some important items to check
are discussed below.
3.1.2
Coverage Control
One of the most important principles of CDMA RF design, i.e. coverage control, still applies to 1xEV-DO.
In 1xEV-DO, the AT (Access Terminal) continuously estimates the SINR (signal-to-interference-plus-noise
ratio) in the forward link and requests a data rate based on the SINR. In general, the better the SINR, the
higher the data rate requested (the exact algorithm depends on the AT vendor implementation). Hence we
should check the RF design to ensure that each sector covers only the area it is intended to cover. Any
potential pilot pollution area should be cleaned up as much as possible, e.g., by using different antenna
configuration. The final RF design result should show that:
3.1.3
RF Coverage Availability
As in CDMA, the 1xEV-DO RF design must satisfy the path loss requirement for the designed data rate
from the link budget. This can be shown using the pilot composite coverage plot or the required AT
Transmit power plot.
Nortel Networks Confidential and Proprietary
11
The 1xEV-DO cell radius depends, among other things, on the designed data rate. If the 1xEV-DO network
is overlaid 1-to-1 on an IS-95 network, then it is good for the forward link data rate of 307 kbps (at 1900
MHz and with AT receive diversity) and reverse link data rate of 19.2 kbps as the chart below shows [1]. If
higher data rate is needed, then additional cell sites are needed.
1900
MHz
850
MHz
450
MHz
38.4
kbps
76.8
kbps
153.6
kbps
307.2
kbps
614.4
kbps
921.6
kbps
1228.8
kbps
1843.2
kbps
2457.6
kbps
w/o
Diversity
136%
111%
99%
88%
73%
61%
55%
45%
40%
w/
Diversity
w/o
Diversity
w/
Diversity
161%
131%
116%
104%
87%
72%
65%
53%
47%
158%
129%
114%
102%
85%
71%
64%
52%
46%
185%
151%
134%
120%
100%
83%
75%
62%
54%
w/o
Diversity
w/
Diversity
145%
118%
105%
93%
78%
65%
59%
48%
42%
169%
137%
122%
109%
91%
76%
68%
56%
49%
All
Frequencies
9.6 kbps
19.2 kbps
38.4 kbps
76.8 kbps
153.6 kbps
111%
101%
89%
75%
57%
38.4
76.8
12
3.1.4
In Nortels EVDO release 3.0 there is a new feature called Inter RNC Active Handoff (IRAHO) that will
allow ATs to add pilots to its Active set from DOMs primarily homed on adjacent RNCs. The intent of the
feature is to:
This feature allows the AT to add pilots from a DOM in another subnet through secondary homing.
Secondary homing allows an operator to home DOMs to up to 8 secondary RNCs in addition to its Primary
RNC (i.e. data fill the routing table in the DOM with the primary RNC and up to 8 secondary RNCs).
Through secondary homing an AT can add pilots from DOMs in an adjacent subnet without having any
knowledge of that subnets Color Code, or Subnet ID. The figure below shows an example of the physical
connectivity of DOMs and RNCs in a multi homed scenario.
13
users. Care should be taken in border planning such that there will be a dominant PN in the coverage
overlap region between border sectors. The absence of a dominant server in these areas may lead to Ping
Pong. Excess ping pong can lead to excessive session setups, access failures and dropped connections. The
following figure illustrates secondary sector deployment in an area of high mobility. The secondary border
can be defined as Subnet 1+ in the figure below. The shape of the secondary border for Subnet 1 will allow
ATs originating calls in Subnet- 1 to move to sectors deep within Subnet-3 and Subnet -7 without
dropping the active call and would be ideal for extending coverage in Subnet-1 along the highway or any
other high speed road [9].
(Subnet-2)
RNC-2
(Subnet-3)
RNC-3
(Subnet-7)
RNC-7
Highway or other
high speed road
RNC-1
(Subnet-1)
Subnet-1+
RNC-4
(Subnet-4)
RNC-6
(Subnet-6)
RNC-5
(Subnet-5)
Figure: IRAHO Example Showing the Ability to Maintain an Active Connection from RNC-7 -> RNC-1
-> RNC-3
The level of mobility can be seen as the percentage of calls entering or leaving a cell at any particular
period of time. It could be expressed by the following ratio:
Cell
Coverage
A
ATs entering cell
C
ATs leaving cell
14
Level of mobility = (A + C) / (A + B + C)
One way to get an idea of low mobility areas would be to look at the 3G data mobility behavior of the
underlying 1xRTT network. There is no formula that can provide an exact mobility ratio, but the BTS OMs
FCHOriginationNonBlocking3GData and the FCHHandoffNonBlocking3GData can help to provide an
educated guess. They can be used in the following ratio which is similar to the mobility ratio above:
Level of mobility (FCHHandoffNonBlocking3GData) /
(FCHHandoffNonBlocking3GData + FCHOriginationNonBlocking3GData)
Similarly, one way to get an idea of the level of data activity in a 1xEV-DO network is to analyze the data
activity in the underlying 1xRTT network. This can be done by comparing, on a per sector basis, the data
Minutes of Use (MOU) within the 1xRTT network. On a Nortel Networks 1xRTT data network, the MOU
can be determined using the 3G Data MOU, i.e. PrimaryFrameCntFCH[3,5,7] from the BTS performance
OMs. Lower amount of 3G Data MOU would indicate a lower data activity area.
After initial deployment, RNC logs should be collected and analyzed using the NEWTUN tool. The
NEWTUN tool will provide a list of DOMs that should be homed as secondary DOMs to the RNC under
consideration. The use of NEWTUN to refine the DOM homing configuration will be iterative but should
be highly refined after about the 3rd round of analysis.
Setting and managing the number of primary and secondary DOMs per RNSM is done at the DO RNCs
topology manager. In addition to managing the number of primary and secondary DOMs per RNSM, each
DOM has a primary selection table used for data filling the IP address of the primary RNC. Likewise there
is a secondary RNC homing table which contains fields for adding the IP addresses of secondary RNCs
and a field the indicates whether secondary homing has been enabled or not [9].
3.1.5
Prior to the 1xEV-DO RF optimization, it is a good practice to review the 1xEV-DO CIQ that the customer
filled out and check that the 1xEV-DO network design matches the requirements listed in the CIQ
response. Several important lists/information to check are as follows:
15
Reference [3] provides the complete provisioning rules and procedures for the DOM, DO-RNC, DO-EMS,
and the backhaul networks. Of particular interest to RF engineers is probably the DOM provisioning.
Currently each Metrocell can house three DOMs. However, there can be only one DOM per carrier (for 3
sectors). Therefore, if 1xEV-DO is deployed in only one carrier, the maximum number of DOMs is the
same as the number of Metrocells. If the required number of DOMs is bigger than the number of
Metrocells, then cell splitting may be considered.
The required number of DOMs depends on the DOM capacity limitation, reverse link Erlang requirement,
forward link throughput requirement, and the Metrocell limitation mentioned above [2]. A DOM has 96
channel elements and can serve about 90 connections in the reverse link (across 3 sectors). Assuming an
average CE/user of 1.5, the number of primary users that can be served by a sector in a tri-sector site is 20
[1].
16
Since 1xEV-DO introduces many new RF parameters, it is important to verify that the correct RF
parameters have been configured. The default RF datafill parameters are the recommended starting point
for optimizing a network. Please refer to [3] and [8] to get detailed descriptions these parameters and their
recommended values.
EVDO Editor is a PC based application for viewing and editing specific RF parameters at the DOM level
(i.e. RF Gain, Cell Radius, RAB Offsets, and Neighborlists). It also will allow users to view parameters at
the RNC level. Currently, EVDO Editor has business rules checking capabilities for DOM level
parameters. Future releases will have business rules checking capabilities for RNC level parameters.
EVDO Editor can be found at http://navigate.us.nortel.com/imds?pg=/eng/wne/core/tool/adm/cdma/doed.
EVDO Editor
4.2
The basic system readiness checkup is done to verify that the 1xEV-DO system is ready to carry traffic. In
summary, the following attributes should be checked:
Verify status of all Nodes (DOM and RNC) using NORTEL-XX# show node have an administrative
and operational status of Up
Verify status of all Modules (BIO Cards, RNSM, SC, DOM, etc) using NORTEL-XX# show module
has a status of Active.
Verify status of all the interfaces (i.e. Ethernet, T1/E1, PPP, node 1/0/1) using NORTEL-07# show
interface command. All interfaces must have an administrative and operational status of UP.
17
Ethernet interfaces:
DOM Backhual If a T1 is connected to a DOM along with 100/10 base T ethernt
connection, the DOM will prompt traffic from the RNC over T1.
2 Ethernet ports per BIO card for support of A10/A11 interface communication,
Abis, A12, and OA&M paclets. On CLI, BIO Ethernet interfaces are listed as
Ethernet 1/x/1, 1/x/2 where x denotes the slot of the card.
1 Ethernet port per SC control module - these are optional interfaces for carrying
OAM traffic. On CLI, the active SC port is listed as Ethernet 1/0/1.
Management IP Interface an optional interface that is used in conjunction with the
SC aux. Ethernet if operators want to forward management IP traffic to a virtual
management interface. On CLI the management IP interface is listed as mgmt1/0/1.
o All Used T1/E1:
DOM Backhaul used to backhaul DOM physical and link layer traffic to the
aggregation router.
o PPP Interface for each T1/E1 interface
DOM Backhaul - a PPP link is associated with each T1 between the DOM and the
aggregation router to provide duplexed, simultaneous, bidirectional, sequential,
packet transfer of encapsulated Abis and OAM IP packets between two dedicated
network peers. On CLI, the DOM T1 / PPP interface is denoted as: ppp1/0/1,
ppp1/0/2, ppp1/0/3, and ppp1/0/4 interfaces.
o Node 1/0/1 interface
Abis and OAM data - provides a loop back or virtual interface for the transfer of
Abis data from the DOM to the primary RNC and OAM data to the EMS. On CLI it
is shown as : node 1/0/1 interface.
Status of the Abis links between the DOM and the RNC using NORTEL-07# show abis peer. Every
RNC should have an Abis status of Connected to every DOM that is homed to it.
Status of the traffic and control channels established between the DOM and the RNC.
Ping the Node IP address of DO RNC, IP Addresses on the aggregation router associated with DOM
backhaul links, and the DO EMS IP address from the DOM. NORTEL-07> ping 10.0.0.0
Ping the IP address of all DOMs, IP addresses of DOM PPP Links, all PDSNs, and the DO EMS from
the RNC.
Status of GPS for all DOMs. NORTEL-07> show GPS health
o
The basic system readiness checkup should be done from the EMS. An example of how to do it from the
command line interface can be found in Appendix B. In the event any of the system readiness checks fail
refer to the troubleshooting or recovery sections of reference [16].
18
4.3
FTAP and RTAP tests should be performed to network end to end performance capabilities inclusive of the
EVDO airlink. FTAP and RTAP are built in test EVDO testing utilities that send test frames to determine
the available bandwidth of the network.
Prior to testing, a location where RF is known to be good should be selected. In addition, AT logs should be
captured during TAP testing to measure the PER% seen at the AT.
Data rates given by FTAP tests should be equivalent to the average DRC request rate and RTAP tests
should show equivalence to the average Reverse link Rate limit of the sector.
The following is an example of how to configure TAP tests from CLI: For more information on TAP tests
see, reference [15].
19
4.4
TCP / IP Parameters
Prior to starting optimization a review of configured TCP / IP parameters should be performed. Due to the
fact the FTP protocol, used extensively in optimization activities, is based on TCP, a thorough review of
configurable TCP and PPP layer parameters should be performed to insure accurate performance reporting.
See Reference [13] for more information on TCP/IP parameter descriptions.
Recommended TCP/IP Configuration:
Data Collection Laptop configuration Insure that the following TCP / IP settings are
optimized. TCP send and receive buffers should be optimized according to the bandwidth delay
product (i.e. Buffer Size = Bandwidth x delay). For EVDO, ideal Buffer Size =
(2450kbps/8bits/byte * 150msecs) = 45.9kBytes.
o
Note: Windows 2000 TCP/IP uses PMTU detection by default and queries the NIC
driver to find out what local MTU is supported. Altering the MTU parameter is
generally not necessary and may result in reduced performance.
20
When a connection is established, the two hosts involved exchange their TCP maximum segment
size (MSS) values. The smaller of the two MSS values is used for the connection. Historically, the
MSS for a host has been the MTU at the link layer minus 40 bytes for the IP and TCP headers.
However, support for additional TCP options, such as time stamps, has increased the typical
TCP+IP header to 52 or more bytes.
EnablePMTUDiscovery
Key: Tcpip\Parameters
Value Type: REG_DWORDBoolean
Valid Range: 0, 1 (false, true)
Default: 1 (true)
21
Data compression enables information to be transmitted beyond the actual connection speed. Data,
particularly text and graphics, usually contain repeated sequences of identical information. Data
compression works by replacing many characters of repeated information with a few characters
and transmitting only one copy of repeated sequences of data.
Communication software, such as Network and Dial-up Connections, may support data
compression. For example, using a 14.4 Kbps V.32bis modem, you can enable software
compression, and experience an average throughput of 28.8 Kbps. Tests show that software
compression can result in higher data transfer rates than hardware compression.
To insure accurate reporting of application or physical layer throughput over the access network
the Enable Software Compression setting in the Windows Dialup Networking Properties ->>
PPP Settings should be disabled.
Figure Disable Software Compression
22
Identification [12] allows the application to identify itself to its peer (Link Maintenance
Packet)
Time Remaining [13] notifies the peer of the time remaining in the session
Frame Check Sequence (FCS) Alternatives [9] provides a method to specify another FCS
format to be sent by the peer or to negotiate it away altogether.
Self Describing Padding [10] Indicates to the peer that the use of padding is understood.
Used when some protocols, such as network layer or compression protocols, are configured
which require detection and removal of any trailing padding.
Callback [13] provides a method for the implementation to request a dial up peer to call
back.
Compound Frames [15] allows the implementation to send multiple PPP encapsulated
packets within the same frame. (self describing padding must be used in conjunction with this
option)
Default: No value; the default behavior is as follows: do not initiate options but if requested
provide them.
Description: This parameter controls RFC 1323 time stamps and window-scaling options. Time
stamps and window scaling are enabled by default, but can be manipulated with flag bits. Bit 0
controls window scaling, and bit 1 controls time stamps.
23
SACK is controlled by the SackOpts registry parameter. The Network Monitor capture below
illustrates a host acknowledging all data up to sequence number 54857341, plus the data from
sequence number 54858789-54861685. SACK is enabled by default in Windows
+ FRAME: Base frame properties
+ ETHERNET: ETYPE = 0x0800 : Protocol = IP: DOD Internet Protocol
+ IP: ID = 0x1A0D; Proto = TCP; Len: 64
TCP: .A...., len:0, seq:925104-925104, ack:54857341, win:32722,
src:1242 dst:139
TCP: Source Port = 0x04DA
TCP: Destination Port = NETBIOS Session Service
TCP: Sequence Number = 925104 (0xE1DB0)
TCP: Acknowledgement Number = 54857341 (0x3450E7D)
TCP: Data Offset = 44 (0x2C)
TCP: Reserved = 0 (0x0000)
+ TCP: Flags = 0x10 : .A....
TCP: Window = 32722 (0x7FD2)
TCP: Checksum = 0x4A72
TCP: Urgent Pointer = 0 (0x0)
TCP: Options
TCP: Option Nop = 1 (0x1)
TCP: Option Nop = 1 (0x1)
+ TCP: Timestamps Option
TCP: Option Nop = 1 (0x1)
TCP: Option Nop = 1 (0x1)
TCP: SACK Option
TCP: Option Type = 0x05
TCP: Option Length = 10 (0xA)
TCP: Left Edge of Block = 54858789 (0x3451425)
TCP: Right Edge of Block = 54861685 (0x3451F75)
The tcp_xmit_hiwat parameter is effectively the size of the send buffer. The send
buffer high watermark tunes the transport layer socket buffer size on a kernel wide
basis. The socket buffer can also be changed on a per-socket basis by using the
SO_SNDBUF socket option within an application. Mind that for UDP the size of the
output buffer represents the maximum datagram size.
4.5
Shakedowns
The first step in 1xEV-DO RF optimization is to perform a shakedown of each cell site and all inter subnet
borders. Like in CDMA, the objective of the shakedown is to verify that we can get a connection on each
sector and switch to the adjacent sector in the forward link (and softer-handoff to it in the reverse link), and
to ensure that the sector RF parameters (like PN offset and neighbor list) are correct. Chapter 6 presents the
detailed procedure to do the shakedown.
4.6
The second step in 1xEV-DO optimization is to perform a stationary data testing to measure the maximum
single-user application throughput in the forward and reverse link. During this exercise, the TCP, RLP,
and/or other RF parameter setting can be optimized to improve the throughput. Once those settings are
optimized and finalized, the measured single user application throughput becomes the golden value, i.e.,
Nortel Networks Confidential and Proprietary
24
the upper bound of what we can achieve in the network. Chapter 7 presents the detailed processes involved
in performing stationary (Golden Value) data testing.
4.7
Once the stationary data testing is done, we can do drive tests along the important roads to measure various
RF performance in a cluster. We can also do more stationary data testing in different locations (that are
deemed important) throughout the cluster. Based on these test results, the datafill (i.e. Power and resources
related parameters, etc) can be adjusted to improve the 1xEV-DO performance. Chapter 9 details the
process involved in performing typical 1xEV-DO cluster drive testing.
4.8
Troubleshooting
If RF performance problems are encountered during the stationary and drive testing (or anytime during the
deployment period), engineers will have to analyze various performance attributes in efforts to diagnose
where the root cause lies. Depending on the problems, troubleshooting may require more than the regular
mobile logs obtained from the cluster drive testing. Chapter 10 presents various tips and examples to help
engineers troubleshoot various 1xEV-DO RF performance problems.
25
26
FTP Client application such as WS-FTP Pro. Chariot may also be used.
Optional tools:
TCP performance logging tools: Windump, Snoop, Ethereal and TCPTrace.
OM analysis tool: OMAX (Nortel OM analysis tool available through Mike Anderson)
There are several TCP parameters that need to be set correctly. See Section 4.4 above.
If either Optis-M or Invex3G are used as the data collection tool, then the TCP parameters need to be set
inside the tool. However, if either XCAL-DO or Invex3G - PC are to be used, then we need to change these
parameters in the laptop. One easy way to do it is by using DrTCP utility software (available in the internet,
for example from http://www.dslreports.com/drtcp).
Section Appendix 4.4 contains a detailed description of settable TCP / IP paremeters on how to optimize
them for the data collection tools (i.e. laptop, etc) and the FTP server.
5.2 AT Setup
The ATs (mobiles) that are recommended to be used for 1xEV-DO RF optimization along with data
collection platform support are given below:
If XCAL-DO or Invex3G - PC is used as the data collection tool then the right driver for the AT needs to
be installed in the laptop.
27
Note: The timestamps given in RNC logs are based on GMT time. In order to correlate the RNC logs with
AT logs it will be necessary to adjust the RNC log timestamps according to the difference in local time and
GMT time.
For a fully loaded DO-RNC, there is one active System Controller on slot 7, one redundant SC on slot 9,
one hot swap SC on slot 10 and one hot swap redundant SC on slot 8. A single DO-RNC can have up to
four RNC-BIO modules and up to eight RNC-RNSM modules.
For a DOM, there is one BIO/SC module and one 1xEVDO modem. The 1xEVDO-Modem contains two
modules, Forward Link Processor Module (FLM) and Reverse Link Processor Module (RLM). Given that
call processing resources could be assigned to any cards in the RNC, logging has to be enabled on all the
cards (RNSM or SC) in the system.
Logging can be configured through the DO EMS or CLI to configure the logging system of each individual
card.
Logging Trap Severity
Within each hardware module that has a CPU, various software components generate all kinds of events.
The verbal description of these events is considered as event messages. Those event messages can be
selectively captured as a log by logging system of the module. The selective capturing functionality is
implemented by setting up a set of input traps which is also called Severity.
So in the end, the captured logging messages by the logging system are the subset of all event messages
generated by those software components within the hardware module. The EVDO system supports severity
levels from 1-32. For most events a severity level of 8 is all that is required. Other events like Forward
Sector Switching may require logging severity levels of 22.
The following is shows the components of an RNC log message:
Time Stamp
of the Date
Message
Severity Level
Component ID
Message
Body
Time Stamp of
hh:mm:ss.sss
CPU ID
Message ID
28
To configure the log files using the DO-EMS go to Network Database DO-RNC RNC
Menu Show Cards as shown below.
Select the SC Card from the Cards Menu
From the SC Card Menu select Show logfacilityMgr Modify Logfacility 5 (Call Control)
The following boxes need to be configured in the Call Control Log facility Manager on the SC Card
o Logging Enabled = True
o Maximum Severity = 5, 8, 16, 22, 32, etc
o Output to File = On
o Maximum Severity on File = Maximum Severity entered above
29
SC Card Menu
30
31
NORTEL-07>en
NORTEL-07#config
NORTEL-07(config)#module-logging bio1/11/1
NORTEL-07(module-logging-bio1/11/1)#trap severity 16 resource-control
NORTEL-07>en
NORTEL-07#config
NORTEL-07(config)#module-logging rnsm1/13/1
NORTEL-07(module-logging-rnsm1/13/1)#trap severity 16 call-control
To Enable or disable logging from the CLI prompt do the following:
On RNC SC card for call-control component:
NORTEL-07>en
NORTEL-07#config
NORTEL-07(config)#logging start call-control
(To enable log capture)
NORTEL-07(config)#no logging start call-control (To disable log capture)
Parsing and Retrieving Log Files
To parse the logs under RNC: (This step is not necessary if using NEWTUN)
NORTEL-07(config)#logging convert file <rnc_binary_log.bin> <rnc_output_file.txt>
No wild card supported under command line, so log need to be parsed one by one.
To check the log files under RNC:
NORTRL-07>shell
NORTRL-07(shell)(disk0)>cd logs
NORTRL-07(shell)(disk0)>ls
To get the log the log files, users can use various FTP programs to access the log directory and then extract
those logs. Once those logs are download to users local hard drive, they can be viewed with any kind of
text file editor program.
For more detailed information on RNC logging see reference [4].
5.3.2
Optis-M Setup
Once the Optis-M GUI is installed, the user needs to do the following:
Connect AC/DC Converters input connector to power source.
Connect AC/DC Converters output connector to the Power supply connector on the front panel of
Optis-M.
Connect Ethernet cable to Ethernet connector on the front panel of Optis-M and laptop.
Turn on the Optis-M unit using the power switch on the front panel.
Connect Mobile data cable to the USB port of each slot on the front of Optis-M.
Connect GPS Antenna to the GPS Antenna connector on the front panel of Optis-M.
Set the laptop IP address to the appropriate one (usually 1.1.1.1) so that it can communicate with
Optis-M box.
Start the Optis-M software. Ensure that the client hardware status light (at the bottom right corner of
Optis-M window) is green.
Configure the Optis-M ports. Below is an example for two Chesters connected to Optis-M.
32
33
34
5.3.3
The log mask settings for the general 1xEV-DO data collection (e.g. cluster drive testing) are shown in the
pictures below. The 1xEV-DO log masks are always needed. The CDMA/1xRTT log mask is needed for
measurement with hybrid AT (e.g. for 1xEV-DO to 1xRTT handoff analysis).
For a specific testing or troubleshooting, different log masks may be needed.
35
36
OPTis-M
CDMA/1xRTT
log mask setting.
37
5.3.4
Note: Invex3G Chassis, Release 4.3 / 4.4 should be provisioned with at least 2 DI (digital
communications interface) boards for EVDO optimization testing. The legacy CI boards are
processor limited which will have negative consequences when measuring data throughput
performance.
DI Card Slot
Note: If Invex3G DI boards are not available at the commencement of optimization testing,
Invex3GPC or Wirelesslogix tools should be used to insure success.
Connect the RJ45 Ethernet crossover cable from the System Controller Module (GWSC-0100) of the
Invex3G mainframe to the Network Interface Card of the Personal Computer (laptop) that will be used
to run the Invex3G software.
Connect the GPS antenna to the GPS ANT connector on the mainframe.
Insert the AT (PCMCIA) into the Digital Card Interface (DCI) Module or attach the AT to the
Communications Interface Card (CI) using the appropriate connector cable to the CI ports (STS-A and
STS-B).
Invex3G mainframe also charges all ATs that use the proprietary cable interface to the CI board.
The upper port (A) of the CI module supports Data and Voice application, and the lower port (B) is
reserved for Voice ONLY. Therefore if doing a data call, use ONLY the upper ports in the
Communication Interface (CI) Card.
Each CI port and DI card slot has dedicated LED Indicators (STS-A and STS-B) for status check:
Red: Fault
Amber: Waiting
38
Open the start menu on the laptop and go to My Network Places or Network and Dial up
Connections and select the Local Area Connections icon and then select properties.
On the General tab go the Internet Protocol (TCP / IP) component and select Properties
On the General tab select Use the Following IP Address and enter the following:
o IP Address 192.168.3.100
o Subnet mask 255.255.255.0
Click OK and Close the Networking tab
Power the laptop and the Invex3G unit. The unit can be powered either by connecting directly to the
cigarette lighter or using an inverter in the van.
Once the software has been launched it is necessary to create new device connections if they were not
created beforehand.
o Click on the Connections tab in the workspace
o Next, the add new connection dialog appears. There are three options available for
connection configuration:
o Invex3G Chassis specifies connection to the hardware mainframe
o Invex3G Serial Device specifies connection to a device attached to
the PC (Invex3G-PC option)
o After selecting connect all the Open Connections dialog begins. Click the Andrew
chassis name to highlight it and then click OK.
39
o The Invex connection status dialog box will open up to display the connection progress.
At the initial start up it may take several minutes for the chassis flash memory to update
so, be patient.
Once the connection process is complete a list of devices attached to the chassis will be displayed in
the devices tree
5.3.5
Configure Invex3G logs masks as follows for EVDO and CDMA log collection.
Go to the Devices tab in the Invex3G workspace and right click on the collection device and
select properties. The following window will appear.
40
Select the CDMA L3 and QCP 1xEVDO L3 tabs and click Collect 1xEVDO Layer 3 Messages
and Collect CDMA Layer 3 Messages boxes.
Select the QCP 1xEVDO tab and then select all messages.
Select the QCP1X tab and select all messages.
For more information on Invex3G configuration see reference [18].
RF Optimizer EV-DO
41
In the "File Locations" tab, be sure to select a location for "Extraction Logs" that you can easily locate
later. The extraction log accumulates many informative messages during raw processing. Since new
messages always get appended to the existing file, the file can become quite large. It's a good idea to
periodically delete the extraction log once you have reviewed its contents.
When selecting locations for "Database Data Files", "Extraction Logs" or "Temporary Files", be sure
to select only folders to which you have full access rights, and on the drive that has plenty of available
space. These files, created by RF Optimizer EVDO during processing, can consume a significant
amount of disk space.
In the "Binning" tab, set temporal and spatial bin sizes (the default setting of 1s and 100m is a good
starting point). Keep in mind that the smaller the bin sizes, the longer it will take RF Optimizer EVDO
to process the data, and the larger the resulting databases will be.
In the "Processing Options" tab, it is strongly recommended to keep the "Delay..." option checked.
This will delay the Data Synchronization analysis for any log file, which can be very time consuming
due to the amount of data processed, until the first time you have a need to open the DRC Information
Viewer. Keeping this option checked will significantly save upfront processing time, compared to not
checking this option.
You are ready to process data.
Please keep in mind that by Microsoft's design, each of your database (MSDE) is limited to 2GB. Due
to the varying nature of raw data, it is not possible to predict how much database space each raw file
42
will take up after it is processed, so exercise caution and avoid processing too many files into one
single database. You are free to create as many databases as your hard drive can hold.
5.4.2
XCAP-DO
XCAP DO software also needs a dongle key to run. Once launched, click on Tools Options to configure
the tool. There are six tabs in the Options window, and most of the default values are good for typical use.
General: contain settings for distance unit, message viewer option, sync option, etc.
Path: select the directories for the log data (raw data), model (processed data), and exported data.
To process a log file (XCAP Do calls it creating a new model), click on File New:
43
Selective Parsing: If you want to process only certain messages, you can select them from here. For
example, if you are measuring RF coverage and not concerned about throughput, then you can uncheck
PPP Data and Throughput Info box.
44
5.5.1
The binary to text file parser resides in the RNC SC and DOM card. To parse a DOM or RNC log, you
need to telnet into the DOM or the RNC. The following steps should be followed once logged into the
DOM or RNC to convert the binary data to text:
Nortel-07> en
Nortel-07# shell
Nortel-07#(shell)(disk0:/)# cd logs
Copy the filename of the log
Nortel-07(config)# logging convert file <bin filename> <new text file name>
After converting the file ftp it down to a PC for analysis.
5.5.2
NEWTUN (Nortel Engineering Wireless Tuning Tool) is a Windows based tool developed by Nortels
Wireless Tools group for processing and analyzing EVDO call statistics according to events captured in
RNC logs. NEWTUN is currently only a Beta release and is only available to select groups. To obtain a
license key please contact Tag Support. The processing of RNC logs with NEWTUN requires the input
files from the RNC to be in the raw *.bin or *.gz format (format prior to text conversion). During post
processing of the logs NEWTUN will convert the logs to a viewable format. NEWTUN will not read RNC
logs that are in *.txt format.
NEWTUN supports the following functionality in the current release:
Call Performance statistics such as dropped connection and access failure rates,
Call flow analysis in various states
Call radius configuration tuner
In the future it will support Neighborlist tuning and integration with EVDO editor output files
File Locations
45
It is recommended that the User save the Database files in a location where there is a lot of free
disk space available when processing large amounts of data.
NEWTUN creates a number of temporary files during processing that may be large depending
upon the amount of data processed. These temporary files are cleaned up upon completion of
processing activities.
To save browsing time, the user can setup default location of where the program should search for
raw binary logs.
Process Filtering
46
Filter Messages
When checked the tool will only process Call Control messages
Certain useless CC messages will also be excluded
Skip Files
Currently does nothing but will allow automatic removal of files with Severity levels lower than
required by call model analyzer.
Max Severity
When calculating fail / drop statistics it is important that the log files are collected at the
appropriate severity level. Should the level be too low the tool will warn the user accordingly.
Database Creation
Raw data must be processed and stored into a database for post-analysis. To create a new database:
1.
2.
47
3.
File Processing
NEWTUN processes raw binary Airvana RNC log files. For space reasons, the tool will accept gzipped
binary log files. To process Files
1.
2.
3.
4.
5.
6.
48
Once all files have been successfully processed, select the database of interest and click the Select
button.
By default, the statistics view will automatically be populated for post-analysis.
Notes
When selecting data, all files within a database are automatically selected
Files cannot be removed from a database.
Clicking the delete button will remove the highlighted database
Tool does NOT process RNC converted text files
Statistics Display
The statistics view is the main apparatus for displaying performance results and navigating throughout the
call messaging for debugging purposes.
49
The statistics displayed are based upon all the calls identified within the database selected from the data
manager.
Connection types
Two types of connections are defined within NEWTUN:
Session Connections assisting with the configuration of a session startup
Data Connections associated with the transfer of user data
How these connection types are determined is beyond the scope of this document. As mentioned in the
Known Issues section, break-up of connection types is not accurate and is undergoing improvements.
Description
Total number of Connection Attempts
Connections that failed to acquire traffic channel
Connections that dropped once on traffic channel
Dropped calls that may be close to a border RNC region
Calls that closed normally
Calls that had no more messaging to indicate outcome of Access Attempt
Call that had no more messaging to indicate whether call would terminate
successfully or not
Calls that failed but had insufficient information to determine whether traffic was
50
Unknown
Access Failure Rate
(%)
Drop Call Rate (%)
acquired or not
Calls that were found (End of calls) but no information to classify was available
100 *
Failed Access
Attempts Access Incomplete Unknown Failures Unknown
100 *
Determination of how the calls are classified is beyond the scope of this document.
Notes
All the classification types mentioned above apply to both Session and Data Connection types.
Where possible, call drops / failures are pegged against a DOM / PN.
If no DOM / PN information is available the call will be pegged under Unknown DOM entry.
Debugging Calls
To review the cause of a drop or failure the user must Double-Click on the calls of interest within the
Statistics View
Tip
Double-click a cell containing connections of interest to investigate will bring up message view
51
Message View
The message view will display all decoded messages for the UATI relating to the call selected by the user.
The green color coded messages highlight the connection of interest (Not exact but provides an
approximation)
Note
The Route Update Detail message has been re-formatted to accommodate later NEWTUN tuning
module development.
This message resolves all the rows of a Route Update providing one line per PN
52
Active List
When the user selects calls to analyze from the Statistics view, this pane will show the list of calls for
examination.
Double-clicking on any of the calls will update the message view with the messaging relating to that
connection.
Note
It is recommended to not select a Statistics View Cell with too many calls (100 or more) to
minimize time delays during subsequent analysis.
53
Peg Selection
To choose which metrics to display within the statistics, click the edit tool button.
Saving Results
Later versions of NEWTUN will generate reports of results. Until that time users can save results to Excel
for custom report generation.
5.6
The Nortel 1xEVDO system is very flexible for collecting operational measurements. The system includes
many pre configured data collection templates that allow users to collection measurements with various
degrees of granularity. In addition the system allows users to build customized data collection templates.
Creating a data collection configuration allows the user to tailor the metrics to meet specific needs. The
following instructions explain how to create a data collection configuration on the DO-EMS.
54
Click Add Data Collection Configuration from the menu. The DO-EMS will open the Data
Collection Configuration Form
55
Sample Rate - The period at which the network elements in the configuration read and
store the performance statistics. Select a sampling rate within the range 1 - 3600 seconds.
It is not recommended to select rates below 900 s because it may degrade network
performance, result in data collection errors, or impact OM synch time.
o Push Interval - The collected data is pushed from the NE to EMS at this time frequency.
Select a push interval within the range 60 - 7200 seconds. It is not recommended to use
values less than 1800s because it may result in data collection errors.
o Template - Select a data collection template from the list. The template determines which
object IDs are sampled at the specified sample rate. Click the link to browse all existing
templates.
o Instance IDs to collect data on a specific instance list the instance ID here. If it is
desired to collect data on all instances leave this field blank. For table based data
collection templates it is not recommended to collect data on all instances as this may
degrade NE performance.
Select the node group to collect date from and submit the new template.
Data collection will start and end at the times specified.
Data collection results can be viewed on a per node basis in DO-EMS by selecting Data Collection
Files from the statistics view.
Alternatively, engineers can FTP the results down from the EMS using an FTP client.
o
5.7
CELLDM or Cellular Diagnostics and monitoring logs provide a means to retrieve airlink performance
information for each sector from the baseband module. CELLDM has the following components:
The airlink performance statistics comprise per sector and per user statistics. The per sector statistics are
part of the OMs but the per users stats can be determined from CELLDM logging
CELLDM provides the user two interfaces, NE CLI and GUI DO-EMS to access the airlink performance
statistics and perform diagnostic logging of the baseband module. There are three different configuration
options for CELLDM. These options are given in the table below.
56
Statistics managed by CELLDM are sector based, user based, and broadcast channel based. These statistics
are always collected by different components. The sector and user based statistics are of two types: traffic
statistics and signal quality statistics. Traffic statistics can be divided in to cumulative statistics and
throughput statistics.
Cumulative statistics
These statistics are the sum total of various traffic-related variables associated with a sector or connection.
These statistics are monotonically increasing as a function of time. Examples are:
The total number of bytes transmitted for a connection.
The total number of bytes received from the AT for a connection.
The number of slots for which a specific DRC value was requested by the AT.
Throughput statistics
These statistics are the time-average of the cumulative statistics. Examples are:
Forward link sector throughput (the rate at which bytes are transmitted out of a specific sector).
Average requested DRC (the sum total of all the DRC values requested by an AT averaged over time).
Collection of statistics
An operator can choose to collect any of the statistics using the method of data collection. Data collection is
a feature of the software release and provides the ability to sample any MIB OID at a configurable rate. The
CELLDM component also provides a mechanism that allows the operator to sample the statistics associated
with a sector, connection or broadcast channel at a configurable interval and log these statistics to disk.
Sector-based traffic statistics
Sector-based signal-quality statistics
User-based traffic statistics
Broadcast channel statistics
User-based signal-quality statistics
CELLDM Loggable Attributes
The following operator provisionable attributes can be configured through the DO-EMS GUI or CLI. All
attributes are configured on a per modem card basis
Nortel Networks Confidential and Proprietary
57
sectorStatsLoggingStatus
sectorThruputStatsLogMask
sectorOverheadStatsLogMask
From the DO-EMS GUI, expand the Network Database DOM folders.
Select the DOM that where CELLDM logs are to be collected.
From the DOM Menu, select Config CELLDM.
58
4.
59
60
5.
After the logging attributes have been configured select Submit to begin logging.
Accessing Statistics
In release 3.0, there is a new (Connections) pull down menu for accessing per user CELLDM statistics data
6.
7.
From the DOM Menu select Show Connections to access the per user statistics.
The available user Based statistics are as follows:
FL Rate Statistics Information
RL Rate Statistics Information
Connection Sector Statistics Information
Flow Statistics Information
Flow Queue Statistics Information
Rx Signal Quality Information
8.
9.
In order to access per sector Statistics, from the DOM Menu select ConfigSectorElements.
Once on the Sector Elements page, go to the IS856 Sector Element Menu and select SectorPerfStats
10.
The following Sector Performance stats can be viewed from this page.
Access Channel Statistics Information
Broadcast Sector Statistics Information
Control Channel Statistics Information
FTC Information
RTC Information
Rx Signal Quality Information
General Sector Statistics Information
5.8
OM Analysis Tools
The core RF engineering team has developed OMAX (OM Analysis for 1xEVDO and 1xRTT) for
retrieving and calculating OM performance metrics for 1xEVDO and 1xRTT. The process of creating the
OM statistics comprises; retrieving data from EMS, reading data into MySQL Control Center, and
extracting performance data via SQL Queries.
RAW OM data from EMS is organized into standard data collection templates (dc files). The data collected
in the templates should be transferred to a local server where they can be FTP ed to a local PC with
MySQL control center installed.
Perl scripts are used on the PC analysis machine to parse the RAW OM data into a database (mySQL)
MySQL can be used to run default performance queries or to build specialized queries for custom
performance reports.
The following is needed to install and configure MySQL:
Files Needed
o mysql-4.0.18-win.zipsly
o mysql-4.1.1a-alpha-win.zip
o mysqlcc-0.9.4-win32.zip
The files can be found at the following locations:
o http://dev.mysql.com/downloads/
o \\mikej-2\MySQLDownloads\rirs
Unzip mysql-4.0.18-win.zip in temporary directory:
o Run the Setup.exe
o Follow setup instructions
o Remove temporary directory contents when installation is complete
Nortel Networks Confidential and Proprietary
61
>make install
o This installs the compiled client files under /usr/local/mysql.
o Remove unpacked dir
To compile DBI for Cygwin:
o unpack the DBI-1.42.tar file
>cd into the unpacked directory
>make
>make test
>make install
o Remove unpacked dir
To compile DBD-mysql for Cygwin:
o Unpack the DBD mysql-2.9003.tar file
o cd into the unpacked dir
>perl Makefile.PL -libs=-L/usr/local/mysql/lib/mysql lmysqlclient lz
-cflags=-I/usr/local/mysql/include/mysql -testhost=127.0.0.1
>make
62
5.9
Windump and Snoop logs are generated to analyze the TCP performance between client (data collection
computer receiver) and server (FTP server sender). Typically Windump would be installed on the data
collection laptop connected to the AT and the Snoop logs would be installed on the FTP server such that
the end to end TCP performance of the network can be analyzed during testing.
Windump
Windump is a Windows portable version of TCPdump that runs on top of a low level packet capture
utility called winpcap. Windump is freeware and can be downloaded from the following location:
http://windump.polito.it/install/default.htm
Snoop
Snoop is a Unix command that is used to capture network packets in a readable format Snoop captures
packets from the network and displays their contents. snoop uses both the network packet filter and
streams buffer modules to provide efficient capture of packets from the network. Captured packets can be
displayed as they are received, or saved to a file (which is RFC 1761- compliant) for later inspection.
Command Options:
-C List the code generated from the filter expression for either the kernel packet filter, or
snoop's own filter.
-D Display number of packets dropped during capture on the summary line.
-N Create an IP address-to-name file from a capture file. This must be set together with the -i
option that names a capture file. The address-to-name file has the same name as the capture file
with .names appended. This file records the IP address to hostname mapping at the capture site
and increases the portability of the capture file. Generate a .names file if the capture file is to be
analyzed elsewhere. Packets are not displayed when this flag is used.
-P Capture packets in non-promiscuous mode. Only broadcast, multicast, or packets
addressed to the host machine will be seen.
-S Display size of the entire ethernet frame in bytes on the summary line.
Procedure:
1.
2.
3.
4.
5.
63
TCPTrace
TCPTrace can take as input the files produced by several popular packet-capture programs, including
tcpdump, snoop, etherpeek, HP Net Metrix, and WinDump. tcptrace can produce several different types of
output containing information on each connection seen, such as elapsed time, bytes and segments sent and
recieved, retransmissions, round trip times, window advertisements, throughput, and more. It can also
produce a number of graphs for further analysis. Follow the link to download a copy of TCPTRACE:
http://www.tcptrace.org/download.html
ETHEREAL
Ethereal can read capture files from tcpdump (libpcap), NAI's Sniffer (compressed and uncompressed),
Sniffer Pro, NetXray, Sun snoop and atmsnoop, Shomiti/Finisar Surveyor, AIX's iptrace, Microsoft's
Network Monitor, Novell's LANalyzer, RADCOM's WAN/LAN Analyzer, HP-UX nettl, i4btrace from the
ISDN4BSD project, Cisco Secure IDS iplog, the pppd log (pppdump-format), the AG
Group's/WildPacket's EtherPeek/TokenPeek/AiroPeek, or Visual Networks' Visual UpTime. It can also
read traces made from Lucent/Ascend WAN routers and Toshiba ISDN routers, as well as the text output
from VMS's TCPIPtrace utility and the DBS Etherwatch utility for VMS. Any of these files can be
compressed with gzip and Ethereal will decompress them on the fly. Follow the link to download a copy of
Ethereal: http://www.ethereal.com/download.html
64
Shakedowns
The first step of the 1xEV-DO RF optimization once RF datafill, FTAP / RTAP, and TCP/IP parameter
verifications have all been performed is the shakedown of each BTS and RNC Border areas to verify call
processing functionality and RF parameters/datafill.
6.1 Objectives
The objectives of the shakedown are as follows:
To verify that we can get a connection on each sector.
To verify that we can switch to the adjacent sector in the forward link and softer-handoff to the
adjacent sector in the reverse link.
To verify that the following parameters have been datafilled correctly for each sector:
o PN offset
o Neighbor list
To verify that PER (Packet Error Rate) is 1%0.2%.
To verify that the forward frame rates match the requested DRC.
To Verify that Inter Subnet Active handoffs (ISSHO) are successful
To verify that A13 dormant handoffs are successful
Invex3G
Optis-M
Invex3G and Optis-M FTP Get call scenario for shakedowns.
Nortel Networks Confidential and Proprietary
65
In Optis-M select the call scenario you just created for the autocall. Select All scenario for the
AutoCall Logging Option.
Optis-M
Scenario Select
Invex3G Connect
All Devices
Optis-M - Verification
Check the OPTis-M EVDO AT Status window to make sure that:
o The AT state changes from idle to connected.
o The session state is open.
Invex3G - Verification
In Invex3G verify that:
o The Device color changes from Red to Green
o
66
Optis-M - Verifications
Check the 1xEV-DO Pilot Set Graph window in Optis-M and verify that:
o Alpha sector is the serving sector PN.
o The neighbor list of alpha sector is correct.
o The channel number is correct.
Check the Network Status window in Optis-M and verify that:
o The AT receive diversity is on.
Optis-M
Network Settings
67
From the EVDO Rate Statistics window, verify that the reverse link is in variable rate.
68
Start driving towards beta sector. Monitor the Pilot Pn[ ] and verify the AT does the softer handoff
with alpha and beta sector (both PNs are in the active set). Ensure that sector switching to beta sector
occurs without a connection drop. Stop the van at a location that is covered only by beta sector. Verify
that now only beta sector PN is in the active set.
From the EVDO Rate Statistics window, verify that PER = Bad CRC/Total CRC = 1%0.2%.
Stop the log.
Repeat the steps above for:
o Connection origination at beta sector and softer handoff to gamma sector.
o Connection origination at gamma sector and softer handoff to alpha sector.
Fill out the shakedown form (an example is given in Appendix B).
Setup the GPS, AT, Laptop, and Optis-M or Invez3G inside the test van
Drive to the subnet boundary on the source side
Start the laptop, AT and the data collection tool. Select all the EVDO attributes for the log mask.
Optis-M
Click on the Auto Call button in Optis-M and then, click on the Create New Scenario button. Create
an FTP Get scenario (please see the example below). Then click OK.
Select the call scenario you just created for the autocall. Select All scenario for the AutoCall
Logging Option.
Start the log by clicking OK.
Invex3G
Right Click on the device in Invex3G and select properties followed by the data connection tab.
Select Add from the Data Connection tab and configure an FTP Get task then select OK.
Select the Connections menu object then Connect All. Logging will start.
Optis-M
Check the OPTis-M EVDO AT Status window to make sure that:
o The AT state changes from idle to connected.
o The session state is open.
Check the 1xEV-DO Pilot Set Graph window and verify that:
o The serving PN is on the source side of the subnet boundary
Begin driving across the subnet boundary toward a site on the target subnet
o Verify that a PN from a site on the target subnet is added to the Active set by looking at the
Best ASP SINR, EVDO Finger Information window, and EVDO Airlink Summary window.
EVDO Finger Information Optis-M
69
EVDO AT Status
Invex3G
Verify from the Invex3G state window that:
o Connected State = Open
o Session State = Open
Open a bar Chart or History window and verify that:
o Active set pilot PN is from a DOM on the source RNC
Begin driving across the subnet or RNC border and verify:
o Active set has a PN from the target subnet or RNC is visible in the bar chart or History
window.
70
Invex3G
After completing the Inter RNC active handoff (IRAHO) test stop AT logs and proceed to one side of
the subnet / RNC boundary.
RNC logs will need to be collected to verify dormant session handoffs
Process to setup logs:
o RNC logs can be configured using CLI or DO-EMS GUI. The GUI method is given
below.
o Open the EMS GUI and from the EMS elements tree, select Network Databas DORNC
o Click on the RNC(s) involved in the A13 shakedown process
o From the RNC Menu, select Show Cards
o Select each RNSM and SC Card followed by RNSM or SC Card Menu Show Log
FacilityMgr
o Select LogFacility Modify
71
Insure that :
Logging is enabled for the Call Control Component
Logging Enabled = True
Maximum Severity = 25
Output to File = True
Maximum Severity on File = 25
Select Submit
Check logging status
o When Logging is complete ftp the log files in bin or GZ format down to a PC for
analysis.
NORTEL -07>shell
NORTEL-07(shell)(disk0)>cd logs
NORTEL-07(shell)(disk0)>ls
Configure the data collection tool to have the AT perform an FTP GET which will insure an Active
session. After one download allow the AT to transition to a dormant state (i.e. No active packet data
session).
Before crossing the border, verify from the EVDO AT Status window that the AT State goes from
Idle Accessing Connected Idle and the Session State is Open.
Start driving towards the subnet border and verify that the Session stays Open but the Subnet ID
changes
o
72
Invex3G State
Verify that UATI, Subnet Mask, and Color Code change when the subnet border is crossed and the
session stays open as the subnet border is crossed.
o Note: There are two ways in which the AT can trigger an A13 dormant session handoff:
1. UATI Request using its present UATI - occurs when the AT has continuous
reception of the network broadcast messages as it moves from one subnet to the
other.
2. UATI Request using RATI (also referred to as Prior Session-initiated dormant
handoff), this typically occurs when the AT loses CCH supervision as it crosses
from one subnet to the other (e.g. AT powers down in one subnet and powers up in
another).
Repeat the test coming from the opposite side of the subnet boundary.
6.5.1
After the shakedowns have been completed, process the shakedown files with RF Optimizer. Do the
following:
Verify that there is only one connection (i.e., there is no connection drop) by looking at the Statistics
Tool Calls Statistics.
73
Example of shakedown call statistics showing access success and no connection drop.
Verify that softer handoff occurs. Look at the PN1 and PN2 columns in Grid View and check that the
PNs change from alpha sector PN to alpha and beta sector PNs to beta sector PN only (similarly for
beta to gamma and gamma to alpha).
Verify that average PER = 1%0.2%. Right click on red bar below the PER column in Grid View and
click on Average.
Verify that the Rx Power is appropriate for the shakedown locations.
See the illustration below for an example of softer handoff occurrence, average PER, and Rx Power
display.
Example shakedown file analysis showing softer handoff and average PER.
Definition
Access Parameters
ACC_CYCLE_DURATION
ACCESS_SIG
OPEN_LOOP_ADJ
Recomm.
Value
-
32
-86
74
PROBE_INITIAL_ADJ
PROBE_NUM_STEP
POWER_STEP
PREAMBLE_LENGTH
CAPSULE_LENGTH_MAX
PERSIST_COUNT
Sector Parameters
COUNTRY_CODE
SUBNET_MASK
LATITUDE
LONGITUDE
ROUTE_UPDATE_RADIUS
REV_LINK_SILENCE_DURATION
REV_LINK_SILENCE_PERIOD
NEIGHBOR_COUNT
NEIGHBOR PNs
NEIGHBOR_SRCH_WIN_SIZE_INC
NEIGHBOR_SRCH_WIN_SIZE
NEIGHBOR_SRCH_WIN_OFFSET_INC
NEIGHBOR_SRCH_WIN_OFFSET
Pilot Sets v2
ASET_WINDOW
RSET_WINDOW
(Neighbor) WindowSize
State Information
AT State
Session State
Idle State
Connected State
Route Update State
HDR_Hybrid_Mode
0
8
6
2
4
0
104
0
(disabled)
0
(disabled)
0
(disabled)
0
0
-
4
58 (-6)
6
32
-
60 chips
100 chips
100 chips
On =
(hybrid)
75
76
SYSTEM_TYPE : 0
BAND_CLASS : 1
CHANNEL_NUMBER : 1125
[ 7 ]
CHANNEL_INC : 1
SYSTEM_TYPE : 0
BAND_CLASS : 1
CHANNEL_NUMBER : 1125
[ 8 ]
CHANNEL_INC : 1
SYSTEM_TYPE : 0
BAND_CLASS : 1
CHANNEL_NUMBER : 1125
[ 9 ]
CHANNEL_INC : 1
SYSTEM_TYPE : 0
BAND_CLASS : 1
CHANNEL_NUMBER : 1125
NEIGHBOR_SRCH_WIN_SIZE_INC : 0
[ 1 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 2 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 3 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 4 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 5 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 6 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 7 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 8 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
[ 9 ]
NEIGHBOR_SRCH_WIN_SIZE : Not Found
NEIGHBOR_SRCH_WIN_OFFSET_INC : 0
[ 1 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 2 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 3 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 4 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 5 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 6 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 7 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 8 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
[ 9 ]
NEIGHBOR_SRCH_WIN_OFFSET : Not Found
6.5.2
After completing the Inter subnet handoff runs an analysis of the collected logs should be performed to
determine rate of success.
Verify from the statistics view there was only one connection for each run.
If there were any dropped connections, a determination of the cause should be made.
Open the Grid view and verify that the AT was in soft handoff with PN(1) from sector with the source
RNC as a primary and PN(2) from sector with the source RNC as a secondary.
77
PN 116 is Primary
and PN 164 was
homed as
secondary
6.5.3
Process RNC logs from A13 session handoff shakedowns with NEWTUN.
Verify from the logs that a UATI Request initiated handoff triggers a A13 Request and finally that an
A13 Confirm is sent indicating that the session is successfully handed off from the source suibnet to
the target subnet. (See Figure Below).
78
7.1 Introduction
The objective of the Golden Value Testing (i.e. stationary throughput testing) is to measure the maximum
user application throughput in the forward link using the network components (PDSN, DO-RNC, DOM,
FTP server, AT, and laptop) that will be used in the cluster optimization stage. During this exercise, the
TCP and other settings can be optimized to improve the application performance. Once those settings are
optimized and finalized, the measured user application throughput becomes the golden (or baseline)
value, i.e., the upper bound of what we can achieve in the network.
This measurement should be done in a stationary location with a good RF link (close to the cell site, SINR
6 dB or Ec/Io -1 dB) and covered by a single sector. The user application throughput is measured with
FTP test using a single AT. Since we want to measure the maximum user application throughput, we need
to ensure that the network is unloaded (no other users in the network).
For the stationary golden value testing it is recommended to perform 10 downloads of a 5MB test file. The
test file should be a zip file or MP3 to insure that any compression algorithms do not skew the results..
79
o Slot utilization is high (in 80% - 90% range) due to single user and full-queue condition.
Check the reverse traffic statistics in EVDO Rate Statistics window and verify that the average reverse
data rate is sufficient to support the forward link data rate [2].
Verify that the Application Throughput Rx is consistent with the data rate distribution shown in the
Forward Traffic Statistics in EVDO Rate Statistics window, and also with the average DRC and Slot
Utilization. Since the SINR in this location is good, RLP retransmission should be minimal, and an
Application Throughput Rx should not be much less than 80% of the average physical layer
throughput (the overhead being about 20%; please see [11]). Lower throughput may indicate TCP
problem. See Section 4.4 (TCP/IP Performance)
80
81
82
Verify that the average user application throughput obtained in this test is appropriate for the SINR
(C2I) condition. The following plot can be used as a guide. Please refer to [1] to more information on
the throughput performance number.
2400
Throughput (kb/s
2000
3 km/h - 1 Ant.
3 km/h - 2 Ant.
1600
1200
800
400
0
-8
-6
-4
-2
10
12
14
83
This chapter discusses methods for collecting and analyzing operational metrics for purposes of
performance benchmarking. 1xEVDO system operation measurements have the following granularity:
Per RNC
Per Slot (slot corresponds to the physical RNSM slot in the RNC chassis)
Per DOM (RN)
Per Sector Carrier
Per Connection
Per Interface (physical or logical)
Per Channel
Operational measurements should be collected during the baseline drive test as well as during subsequent
optimization cluster drive rounds of the network to get a system level view of the performance attributes
given below.
DO-RNC has a connection open occurs when a connection request is received from the AT while a
current connection is already open. In this case the Connection request is cached until the current
connection can be closed. The following OMs are pegged in the case:
numConnectionReqsWhileOpen and numConnectionRequestsfromAT
DO RNC is in the process of tearing down a connection in this case the connection request is cached
and the previous connection is allowed to close normally before the new request is processed. The
following OMs are pegged in this case: numConnectionReqsWhiletearingdown and
numConnectionRequestsfrom AT
DO RNC is in the process of setting up a connection a connection request is received by the RNC
while the network is waiting for a TCC from the AT. In this case the previous TCA is retransmitted to
the AT. The following OMs are pegged in this event: numConnectionReqsWhileSettingUp and
numConnectionRequestsfrom AT
Total Connection Setup Performance OMs can be found in section 7-4 of [17]
84
UATI Request
UATI Assignment
A13 dormant session Handoff
Session configuration
Terminal authentication
Hardware ID retrieval
AT ID validation
Total 1xEV-DO Session Setup Success Rate OMs are defined in section 2-8 of [17]
8.4.1
Regular A13 handoff is triggered when an AT crosses the subnet border without losing supervision of the
DO carrier.
UATIRequest Initiated A13 Dormant Handoff OMs are defined in section 2-11 in [17]
8.4.2
A prior session initiated dormant handoff occurs when the AT, during the session configuration phase
requests to retrieve a prior session from the RNC or when the AT looses supervision of the source RNC and
regains it on the target RNC. In this situation the target RNC will attempt to retrieve the prior session
information from the source RNC over the A13 interface. For session transfer to proceed, A13 dormant
handoff must be enabled on the target RNC.
Prior Session Initiated Dormant Handoff OMs can be found in section 2-13 of [17].
8.4.3
In the previous releases, an A10 connection was setup every time the AT completed a location notification
or after every session configuration regardless of the need to send data or not. In release 3.0 we now have a
UA10 minimization feature that prevents the setting up needless A10 connections. The minimization
feature looks for the following events to trigger the opening of an A10 connection. Otherwise A10 remains
closed.
ULN Mode
New Session
Inter RNC Dormant Handoff (Regular)
Inter RNC Dormant Handoff (Prior Session)
Inter Tehnology handoff intra RNC
Inter Tehnology handoff inter RNC
Open A10 if
ULN received
Non-0 PDSN IP in A13 Response OR ULN Received
ULN Received
ULN Received
ULN Received
MIP Mode
New Session
Inter RNC Dormant Handoff (Regular)
Inter RNC Dormant Handoff (Prior Session)
Inter Tehnology handoff intra RNC
Inter Tehnology handoff inter RNC
Open A10 if
First RLP Packet received from AT (a.k.a PPP resync)
Non-0 PDSN IP in A13 Response OR PPP Resync
PPP resync
PPP resync
PPP resync
Refer to section 6-1 of [18] for A10 Connection Setup Performance OMs
Nortel Networks Confidential and Proprietary
85
8.4.4
There are two phase of session configuration; the AT initiated portion where the AT initiates negotiation of
protocol attributes, and the AN initiated portion where the AN initiates negotiation of protocol attributes.
The session configuration process is not complete until both the AT and the AN have committed to the
recommended protocol attributes. Session configuration is unsuccessful if: the protocol negotiation fails
between the AT and the AN, the session configuration timer expires (60secs), or the DO Airlink connection
drops during the process.
Session Configuration Failure metrics can be found in section 5.1 of [17].
8.6 Inter RNC Active Handoff Traffic Statistics and Secondary Border Measurement
Prior to release 3.0 there was no mechanism for preventing call drops at subnet (RNC) boundaries. In prior
releases, when the AT would send a route update containing a Pilot from another subnet, the Pilot Lookup
would fail (RNC has no knowledge of pilots in other subnets) and the connection could drop as a
consequence.
With the introduction of release 3.0, the Inter RNC Active Handoff feature will be added. This feature will
allow DOMs from adjacent RNC to be secondarily homed to the anchor or source RNC in the immediate
subnet. This capability will allow active calls to continue on DOMs in adjacent subnets. New OMs are
available to help answer the following questions:
Should the DOM be homed to other secondary RNCs to extend the border region?
Should the DOM not home with any secondary RNCs it is currently configured with?
Is the DOM homed to the optimum Primary RNC?
Refer to section 13-8 of [17] for description of Inter RNC Active Handoff Traffic Statistics and Secondary
Border Measurement OMs.
86
P3 but a traffic channel can not be assigned for P2, a block due to Reason X will be pegged against Pilot P2
only and success counts will be pegged against pilots P1 and P3. In the end the connection setup will fail.
These OMs measure Airlink resources allocation for primary and secondary DOMs during connection
setup and soft / softer handoff.
Refer to Section 13.1 of [17] for more details on Per Sector Airlink Resources Allocation OMs.
87
Select the call scenario you just created for the autocall. Select All scenario for the AutoCall
Logging Option. This will allow data from both scenarios (ATs) to be captured in one log file.
Start the log by clicking OK.
Invex3G Scenario Setup
For EVDO data collection it is imperative that the Invex3G Chassis be configured with at least 2 DI
boards. The legacy CI boards are processor limited and will result in a reduction in throughput
performance.
Right Click on the device (AT#1) and select Properties. Select the Data Connection tab to configure
the data connection properties.
Follow the same process to configure (AT#2). See figures below for Reference.
o Disconnect Timer = 15 seconds For Disconnect tasks, enter a Hang up Timeout value as
required. If Invex3G does not see data from the network for a certain length of time, it
assumes that the data task is dead and tries to hang up the call (session). Invex3G waits for an
acknowledgement from the network (LCP TERM ACK); however, at some point the software
gives up waiting for the acknowledgement and starts another task loop. If the
acknowledgement is not received, it is possible that the network will keep the original data
session open. When a new task loop is started by Invex3G, the network may not allow the
new session since it is keeping the previous session open. The Hang up Timeout determines
the time that Invex3G will keep trying to get a hang up acknowledgment from the network to
confirm that the previous data session was cleared. If you are experiencing call backs from the
network when a session is ended, or the network will not allow new sessions, try increasing
the Hang up Timeout value to a larger number.
o Wait Timer = 5 seconds time between calls where phone will be inactive.
o Initial Connect - Parameters required for connection to the EVDO network. See the Data
Connection Task example below.
Connection Type = EVDO
Authentication = Chap / PAP
Compression = None
TCP Input (RX) window = default
88
Invex Initial
Connection
Config.
89
Invex Data
Connection
Configuration
After all data connection tasks have been defined click ok. Go to file and save workspace as Cluster
Drive.IWF
Select Connect All from the connections menu then Record logfile from the file menu to start logs.
Drive Testing Process
Start driving along the cluster drive test route.
Check the following display in Optis-M throughout the drive test:
o GPS Status window: to ensure that GPS is working.
o 1xEV-DO Pilot Set Graph: to monitor that the AT does soft handoff as necessary with other
sites.
o EVDO Signal Graph and Call Statistics window: to monitor the data call progress and the call
statistics.
Check the following displays in Invex3G throughout the drive test:
o Go to displays and select New and add a State display and a 1D bar Chart.
o In the state display drag the Current Task Status from the data connection Codec of each
device listed in the device tree.
o The Current Task Status will list the following attributes:
Connection State - Verify Connected
Connection Type Verify 1xEV-DO
Task State Verify Running
Task Name Verify FTP GET 5MB or 100kbyte
Task RX Throughput (kbits/s) (Cumulative) Verify > 0 kbits/s
o In the Bar Chart drag the Satellites icon from the GPS Codec. Verify
GPS lock to at least 7 satellites
Verify from the Reports icon in the device tree that Alerts configured to show:
o Dropped Calls
o Call Set up Failures
o Data Connection Failures
o Data Task Faliures
If any of these events occur there will be an audible alarm as well as a recoding of the event in the
Alerts tab of the Invex3G workspace. (See example Below)
90
Note down the areas where access failures, connection drops, and/or handoff failures occur.
At the end of the drive test, stop the logs. Try to keep each log to be no more than 30-minute long.
91
9.2.1
An access failure is defined as the number of times (during the cluster drive test) the AT failed to arrive at
the traffic channel. It should measure the RF-related failures excluding the ones caused by resource
blocking and other non-RF related failures.
A successful access attempt is marked by Connection Request message. AT success on arriving on the
traffic channel is marked by Traffic Channel Complete message.
Connection
Request (Access)
AF
AC Ack (Control)
Traffic Channel
Assignment
(Control)
Traffic Channel
Complete
(Rev Traffic)
RTC Ack
(Fwd Traffic)
After processing the RNC log files in NEWTUN select the files from the Data Manager window.
The call statistics from the RNC logs will be displayed in Statistics view.
The statistics shown in these views will be for all users on the network. There are utilities in
NEWTUN that will allow engineers to calculate failure statistics for the AT under test.
Select the Access Terminal tab then click on the data icon.
92
To determine the access failure rate, look for the IMSI of AT#1 (AT used for Access Failure
Testing.
The Access Failure Rate for the AT inclusive of only Data or Session Configuration related
connections is defined as:
X 100
AF % = failures
Attempts (Blocks + IncompleteAccesses )
Note: Incomplete Access connections that reached the access phase, but had no more messaging to
indicate the outcome.
Click on the Sessions icon and sort by IMSI to determine the Session related Access Failures.
The get the total access failure rate click on the All: icon.
X 100
AF % = TotalFailedCalls
TotalCallAttempts (TotalBlocks + TotalIncompleteAccess )
For more information on which calls failed, click on the Total Failed Calls in the statistics view.
The actual calls will be listed in the Active List.
93
Active Call
List
By clicking on the calls in the Active list the engineer will open the message view where the actual
RNC level messages will be highlighted in green in the message view.
Connection was closed because there was an unknown UATI (Local Connection Close, Reason
Code 2).
Reason Description
Session Close
Session Configuration
Close
Connection Request
Error Close
10
DLSM No FTC
11
Comments
Called by SSM to close
connection.
94
Post Process the AT logs from the Cluster Drive using EVDO RFO
Open the Statistics View and select Call Statistics.
9.2.2
A normal call is ended by a Connection Close message from the AT with the reason = Normal Close or a
Connection Close from the AN with the reason = Normal Close. Any other call termination is considered a
Nortel Networks Confidential and Proprietary
95
connection drop (e.g. forward Connection Close message with reason = Connection Error or QCP
Connection Release packet with reason = System Lost).
Dropped Connection Rate from RNC logs
After processing the logs with NEWTUN, open the Data Manager window and select the database
the files where processed to.
After opening the data base the statistics view will open. As stated above, call statistics are divided
into three categories:
o Network Statistics statistics for calls displayed by RNC, IP, and PN.
o Connection Details Call Summary for every connection identified in the logs
o Access Terminal call statistics that are displayed according to ESN, IMSI, or Hardware
type.
Each of the above categories are further grouped by call type for more granularity:
o All includes all calls
o Data connections setup for an active data session
o Session connections setup for EVDO session configuration
Select the Access Terminal tab from the call statistics view and sort the statistics by IMSI.
The data presented in the file will be representative of all ATs in the network at the time of testing.
Find the IMSI of AT#2. The statistics on this line will be used to calculate the dropped connection
rate.
Dropped connection statistics should be calculated for individually for Data, Session, and All
connection types according to the following equation.
X 100
DroppedConnection% = Drops
SuccessfulAccesses IncompleteTraffic
Note: Incomplete Traffic is defined as a connection that made it to the traffic channel but there
were no additional messages to indicate the outcome (i.e. if there was a normal or abnormal
connection close).
For more information on why connections were dropped double click on Total Dropped Calls in
the statistics view. This will bring up all of the calls in the Active list.
As above, by clicking on each call in the Active List, RNC logging messages for that call will be
displayed in the message view.
96
Call dropped due to loss of the Reverse Traffic Channel (Local Connection Close, Reason Code
9).
Another useful method for diagnosing reasons for a call failure is by clicking on the Connection
97
Once the particular failure is isolated, find the IMSI of the AT used for testing.
Expand the view for that IMSI. All the failures meeting the criteria used to sort the list are listed
under the selected IMSI.
Various Attributes are available in this view that may give clues as to why the failure occurred.
The available attributes are selectable by using the attribute selector.
By double clicking on each connection under the IMSI the RNC message details will be shown
message view.
In RFO, open the Statistics Tool Calls to find the number of connection drops. Example:
98
The connection drop rate should meet the optimization criteria. Please see troubleshooting section
for guidance on the typical connection drop rate value and ways to troubleshoot if it is not
achieved.
The illustration below shows a connection drop that happened due to poor C2I.
9.2.3
Only files/data from AT#2 are used to calculate throughput. There are no throughput statistics available
from the RNC logs therefore, all per user mobility statistics will come from AT logs.
After post processing the AT log files with EVDO RFO, click on the Statistics Tool Data
Statistics Application.
Look for Rx Active Throughput in the statistics list. This is the average user application
throughput.
EVDO RFO displays the application throughput for each FTP download as well as the average
throughput for each drive test file.
99
100
Example RFO Grid View of Traffic C21, DRC Request Rate, Forward Physical RX Rate, etc
The user application throughput under mobility condition here has to meet the optimization
criteria.
The throughput number for cluster drive testing will generally be lower than the one obtained in the
stationary data testing due to the mobility condition (as shown in the picture in section 7.4). The areas with
access failures, connection drops, handoff problems, and coverage problems (low SINR) will also
contribute to the decrease in throughput number. Therefore, fixing all of those problems and improving the
SINR condition will improve the throughput number. Please see section 10 for guidance on the typical peruser mobility throughput value and ways to troubleshoot if it is not achieved.
101
9.2.4
Only files/data from AT#1 are used to calculate the connection setup time.
Connection setup time is calculated from the time AT sends a Connection Request message to the time it
sends a Traffic Channel Complete message.
In RFO, click on the Statistics Tool Connection Statistics Connections. Export the table to Excel and
calculate the average Time Delta to get the average Connection Setup Time throughout the drive test.
102
10 Troubleshooting
10.1 Session Setup and A13 Handoff Success Rate
EVDO Session setup is simply defined as the initial interaction between the AT and the AN. An EVDO
session setup is analogous to a registration in CDMA. Session setup is always an AT initiated action. Only
a UATI Request message from the AT with either RATI or foreign UATI can initiate a valid EVDO
session setup attempt. In release 3.0 A13 dormant session transfer protocol is introduced and will impact
the session setup success. EVDO session setup can be classified as follows:
Regular 1xEV-DO session setup initiated with an RATI (also known as regular session setup)
UATIRequest initiated dormant handoff (also known as Regular A13-dormant handoff)
Prior session-initiated dormant handoff
103
AT
AN
PDSN
UATI Request
UATI Assignment
UATI Complete
Open Traffic Channel
Session Configuration
Close Traffic Channel
Page
Open Traffic Channel
Term Auth, HID, LUP
Open A10 (If data to send)
PPP Traffic
Regular Session Setup without A13 Handoff
104
AT
Target
AN
UATI Request
(foreign)
Source
AN
PDSN
A13-Request
A13-Response
UATI Assignment
UATI Complete
Skipping Hdw Id
exchange
Source
closes DO
session
Location Assignment
Location complete
Open A10
A13-Confirm
Close A10
PPP Traffic
Component
Debug
DO-RNC CPUs
Call Control
011
RNSM and SC
Message
ID
81
16
Call Control
011
RNSM and SC
151
19
Call Control
014
RNSM and SC
156
13
Call Control
008
RNSM and SC
150
Description
Session State Transitions
(UATI Req - HwId Resp)
Session Configuration
Event Succeeded
Terminal Authentication
Success event
Session Configuration
Failed Event
105
5-info
Call Control
N/A
RNSM and SC
164
13
Call Control
008
RNSM and SC
157
5-info
Call Control
N/A
RNSM and SC
174
5-info
Call Control
N/A
RNSM and SC
167
5-info
Call Control
N/A
RNSM and SC
414
Session Configuration
Failed
Terminal Authentication
Failure
Terminal Authentication
Failure
Closing Session
Unknown UATI
Session did not Receive
HW ID Resp. from the AT
106
Note: SSM = Session State Machine, CSM = Connection State Machine, and SCSM = Session
Configuration State Machine
A13 Dormant Handoff Sessions
With release 3.0, the A13 dormant session handoff feature is added. This feature forces a session handoff
for dormant ATs based on the following:
UATI request with foreign UATI (UATI from session on source RNC)
UATI request with RATI (prior session handoff, usually occurs when the AT has lost supervision
prior to A13 handoff request
In most cases A13 handoff should be successful. The following are some typical failures that may be a
result of the process not completing:
Possible A13 Session Failures
Target RNC fails to locate the source. This typically means that the source is not configured in the
targets A13 configuration table
Session does not exist at the source RNC and consequently the source sends an A13 reject.
Configuration parameters are not acceptable to the target. This may happen if there is some delta
in session configuration between the RNCs
Failure can also occur in some fast inter-RNC ping pong cases.
107
The following RNC log sample shows an A13 (foreign UATI) handoff as seen from the target RNC:
108
Upon receiving A13-Reject, (a) will close session and proceed with new session establishment.
Therefore the consequence of this scenario is that session is closed and needs to be re-established. This is
slow and it does require set-up of traffic channel. Also, in this scenario numDormantHandoffAttempts is
pegged, but not numDormantHandoffSuccesses. Therefore this constitutes session setup failure.
Such fast ping-pong scenarios can occur in challenging RF situations, with no clear dominant server, where
UATI procedures take long to complete.
CASE 4: Target (b) receives A13 Response from the source (a), and UATI complete from the AT;
completes the re-configuration of session protocols, but AT ping-pongs back to (a) before Hardware Id
exchange is completed.
Consequence would be the same as in cases 2 and 3, but this should not occur frequently, because A-13
should be configured to pass hardware id in A-13 Response message and therefore hardware id exchange is
skipped at the target (b).
CASE 5: Target (b) receives A13 Response from the source (a), and UATI complete from the AT;
completes the re-configuration of session protocols and hardware Id is retrieved, but AT ping-pongs back
to (a) before (b) receives Location Notification.
In this case A10 between (b) and PDSN is not opened. Also A13 Confirm is not sent to (a), so old session
remains opened at (a). When AT bounces back to (a), that RNC will request session info from (b). This will
complete and (a) will open new session. However at hardware id stage (a) will realize that it already ahs
[old] session for that AT and will proceed closing it. This closes Associated A10 as well, which in turn
clears PPP at the PDSN. Note that AT still keeps the PPP session so there is a state mismatch.
10.1.1.3 OMs
The following OMs should be observed for Session Setup Performance:
109
110
UATIAssignment Stage =
100% *
(numDormantHandoffFailureNoUatiReqRNC +
numDormantHandoffFailureNoUatiCmpltRNC +
numDormantHandoffFailureInvalidUatiCmpltRNC +
numDormantHandoffFailureNoRncResourceRNC) /
(numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC)
One possible cause of the following failure is the different setting for terminal
authentication between the source and target RNCs.
Percentage of Regular A13-Dormant Handoff Failures due to Retrieved
Session Configuration Unacceptable =
100% *
numDormantHandoffFailureRetrievedConfigUnacceptableRNC /
(numA13TotalRejectRNC +
numTotalDormantHandoffFailureRNC)
The session configuration failure is usually caused by the connection setup
failure or connection drop during the session configuration phase. Such
events are usually due to the poor RF condition along the DO-RNC subnet
border.
Percentage of Regular A13-Dormant Handoff Failures during the
Session Configuration Stage =
100% *
numDormantHandoffFailureSessionConfigDuringReconfiguration
RNC /
(numA13TotalRejectRNC +
numTotalDormantHandoffFailureRNC)
Percentage of Regular A13-Dormant Handoff Failures during the
Terminal Authentication Stage =
100% *
numDormantHandoffFailureTAAfterA13RspRNC/
(numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC)
The event pegged by numDormantHandoffFailureHdwIdTimeoutRNC is
frequently caused by poor RF condition. The airlink messages associated with
hardware ID retrieval could be lost in many cases.
Percentage of Regular A13-Dormant Handoff Failure during the
Hardware ID Acquisition Stage =
100% *
(numDormantHandoffFailureHdwIdTimeoutRNC+
numDormantHandoffFailureInvalidHdwIdValueRNC+
numDormantHandoffFailureInvalidHdwIdTypeRNC) /
(numA13TotalRejectRNC +
numTotalDormantHandoffFailureRNC)
The AT ID validation happens within the DO-RNC. Such failure is caused by
the DO-RNC and not related to any external environment.
Percentage of Regular A13-Dormant Handoff Failures during the AT ID
Validation Stage =
100% *
Nortel Networks Confidential and Proprietary
111
numDormantHandoffFailureAtIdTimeoutRNC/
(numA13TotalRejectRNC +
numTotalDormantHandoffFailureRNC)
Prior Session Initiated Handoff OMs
Similar OMs as those given for regular A13 initiated dormant handoff can be developed for prior
session initiated dormant handoffs. For more information on A13 handoff see reference [21].
Using OMs to diagnose common problems (See [21])
OM
Attempts
Successes
Rejection Messages
Failure Hardware ID
112
operational problem.
This is a very rare error that can occur during a prior session
handoff. This requires the AT to ping-pong between subnets
rapidly. If the AT attempts a prior session on the target and if the
prior session attribute matches the local subnet, then this error is
pegged.
A socket problem occurred on the target RNC. If this counter
constantly increases, it could indicate a problem with the
system. Determine which slot has the problem (use the per-slot
OMs) and reboot that slot.
UATI Request was not received after an initial foreign UATI
message was received at the target. This may be due to the AT
moving away after the initial message. This does not indicate an
operation problem.
This may indicate that the RNC is either temporarily overloaded,
or has encountered internal resource problems. If this OM
continues to increment over a long period of time, it may be prudent
to reboot the RNC.
Indicates that the source and target RNC has different operator
settings. Does not necessarily indicate any operational
problems.
If this counter does not increment during dormant handoffs, then
A13 handoffs might be disabled.
If this counter does not increment during dormant handoffs then
A13 might be disabled or there may be a connectivity problem.
113
# A13Confirms
114
The UATIRequest Initiated Dormant Handoff (also known as Regular A13-dormant handoff) is
triggered when an AT crosses the subnet border without losing supervision of the 1xEV-DO carrier.
However in order for the A13-dormant handoff to proceed, the A13-dormant handoff must be
enabled on the target RNC.
Regular A13-Dormant Handoff Success Rate =
Regular A13-Dormant Handoff Successes /
Valid Regular A13-Dormant Handoff Attempts
Where:
Valid Regular A13-Dormant Handoff Attempts =
(numDormantHandoffAttemptsRNC
numDormantHandoffFailureATInitiatedCloseRNC
numDormantHandoffFailureRNCInitiatedCloseRNC)
Prior Session Initiated Dormant Handoff Performance
A prior session initiated dormant handoff is triggered when the AT, during the session configuration
phase, requests to retrieve a prior session from the source RNC. When such request is received, the
target RNC will attempt to retrieve the session information from the source RNC via the A13
interface. For the prior session retrieval to proceed, the A13-dormant handoff must be enabled on the
target RNC.
Prior Session Initiated Dormant Handoff Success Rate =
numDormantHandoffSuccessesPriorSessionRNC /
(numDormantHandoffAttemptsPriorSessionRNC
numDormantHandoffFailureATInitiatedClosePriorSessionRNC
numDormantHandoffFailureRNCInitiatedClosePriorSessionRNC)
Nortel- XX# show 1xEVDO counters all
The following equation should be used to determine the Session Setup Success Rate from OMs:
Formula: numSessionSetupSuccessful/(numSessionSetupAttempts-numSessionsTerminated
ToReceivingUatiReq )
115
session setup events occur between RNCs, the failures that result skew the overall network performance
trend although the impact as a whole is only to a handful of users.
If the AT does not complete Terminal Authentication or reach the Hardware ID retrieval phase of session
setup IMSI information is unavailable to identify the user.
A method for troubleshooting Repeat offenders will be presented in a later release of this document.
2. Session Configuration Failure: Look for Session Configuration Failed Events in the RNC logs. From
the OMs look for numSessionSetupFailedtoSessionConfig
3. Terminal Authentication Failure: Look for TerminalAuthenticationFailureEvents in the RNC logs. In
the OMs look for numSessionSetupsFailedtoTermAuth.
4. Follow the following steps to troubleshoot Terminal Authentication or Session Configuration problems:
a. For terminal authentication to work the AT, DO-RNC, and AN-AAA servers all should be
configured properly.
b. Required AT Configuration:
i. NAI must be equal to the username configured for the AT in the AAA server(s)
ii. Password - must equal the password setup for the AT in the AAA server(s).
iii. Compare the ATs Network Access Identifier (NAI) and Password to the Username and
Password configured for the AT in the AAA server, as follows:
a. On the AT, use QPST (or another provisioning tool) to display the NAI and Password for
the AT, typically found on the 1xEV tab.
b. On the access network AAA server, check the user profile for this AT and display the
username and the password for the particular user.
If the AAAs User name for the AT does not match the NAI setting configured in the AT, or if
the AAAs Password for the AT does not match the Password configured in the AT, there is a
misconfiguration of the authentication parameters for this AT.
To recover, use QPST to configure the ATs:
NAI to equal the Username configured for the AT in the AAA server
Password to equal the Password configured for the AT in the AAA server
c.
116
iv. Examine and verify the Authport for each AAA server listed.
The Authport is the IP port number on which the AAA server is configured to listen for
terminal authentication requests. This is typically 1812.
If the AAA is configured to listen on a different IP port number, modify the DO-RNC
configuration of the AAA to match the correct port number with the Configure Termauth
Server mode server <name> <secret> <A.B.C.D> [authorization port number] command, as
follows, assuming a name of AAA1, a secret of xyz, and a AAA server IP address of
10.11.12.13, and an authorization port number of 1812:
NORTEL-XX(config-termauth-server)#server AAA1 xyz 10.11.12.13 1812
vi. Ping each listed AAA server IP address from the DO-RNC with the Enable mode ping
<Ip address> command, as follows:
NORTEL-XX#ping <AAA server ip address>
If the ping fails, check the AAAs actual server IP address. If it is not the same as configured
in the DO-RNC, change the DO-RNCs configuration with the Configure Termauth Server
mode server <name> <secret> <A.B.C.D> command, as follows, assuming a name
of AAA1, a secret of xyz, and an AAA server IP address of 10.11.12.13:
NORTEL-XX(config-termauth-server)#server AAA1 xyz 10.11.12.13
vii. To check current authentication activity use the following command while in Enable mode at the
RNC: Nortel-XX# show termauth statistics scalar
Useful OMs used to diagnose A13 HO problems
117
118
10.2.1.3 OMs
There are no system OMs currently available that track session setup time. However, the engineer can
go through CLI to view the Max, Min, and Ave Session setup time.
Nortel-XX# >show 1xEVDO counters all
Session Setup Time :Minimum :
00:00:02.320
Maximum : 00:00:02.320
Average :
00:00:02.320
Because this should be a rare event, we can afford more time for a setup without end-user performance
degradation. But since Session Setup is a normal system event using normal system messages and
signaling, the time for setup should be defined. Session Setup is affected by the amount of
configuration data that must be negotiated.
Session Setup Time: 11-12 seconds for unloaded network and 14-15 seconds for loaded network (with
TA). {We need to define start/stop points and test method. This will have to be measured with drive
testing}
119
10.3.1.3 OMs
Method of Collection:
1. There are currently no specific OMs for measuring connection setup time per UATI or overall.
2. Per UATI connection setup statistics can be collected from the CLI prompt but the statistics are only
for the last or current AT connection.
NORTEL-XX# show 1xEVDO connection <UATI> counters all
Connection Setup Time : 00:00:00.220
Nortel Networks Confidential and Proprietary
120
3. The average connection setup time for the entire network can be determined from CLI using the
following command:
NORTEL-XX# show 1xEVDO counters all
Connection Setup Time:Maximum: 00:00:00.280
Minimum: 00:00:00.150
Average: 00:00:00.220
4. At the end of the cluster drive test log into the CLI prompt
121
Y
From the Access Attempt Message is
the Probe Sequence Count greater
than 1?
N
Identify top ten sectors with longest
connection setup times. Is ROT
excessive?
NORTEL-XX#show rate-control
rev-link-stats <sector> e1=alpha
END
122
123
1. In Optis-M Insure that the Allow Dormant State checkbox is not checked unless it is desirable to
have the AT wait for the data inactivity timer to expire such that the system closes the RF connection
between downloads.
2. For data collection select all EVDO attributes for the log mask
3. Drive the golden route to collect AT logs.
124
CSM CSMAwaitTCC
10.4.1.3 OMs
1. In order to setup data collection, follow the guidelines in section 5.4.4 to setup the data collection
configuration with the RNCPerfBySectorCarrier_R3.0 template selected.
2. The following OMs are available from the RNCIS856 Sector Carrier Performance Table:
numATConnectionSetupsFailedTccTimeoutSC
numANConnectionSetupsFailedTccTimeoutSC
numFCConnectionSetupsFailedTccTimeoutSC
o The overall per RNC FCA rate is given by the following level 1 formula:
Percentage of Overall Connection Setup Failure rate =
((numConnectionSetupsBlockedByRncResources +
umConnectionSetupsFailedByRncResources + numConnectionSetupsBlockedByRn +
numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn +
numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout +
numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError + numConnSetupsAborted) /
(numConnectionRequestsFromAT + numFastConnectsAttempted
numConnReqsWhileSettingUp numConnReqsWhileTearingDown
numConnReqsWhileOpen)) * 100
3. The following level 2 and 3 OMs show the percentage of connection setup failures broken out
according to the following categories:
a. Unsuccessful DO-RNC resource allocations numConnectionSetupsBlocked +
FailedBy_[RNCResources / RN]
o Includes events such related to RNC and DOM resources
b. RF Failures numConnSetupsFailedRU + TCCTimeout
o RF failure metrics include events related to TCC timeouts, and RU timeouts etc, that
contribute to the overall connection setup failure rate.
c. Misc Failures cnumConnSetupsFailedSWError
o Usually includes events like software errors
o FCA OMs are available for Session related FCA and are denoted by (sNum***) whereas regular data
oriented connection setup failures are denoted by (cNum***).
125
126
numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted)
4.Unsuccessful DOM Resources allocations
Percentage of Connection Setup Failures due to Unsuccessful DOM
Resources Allocation =
100% *
(numConnectionSetupsBlockedByRn +
numConnectionSetupsFailedByRn) /
(numConnectionSetupsBlockedByRn +
numConnectionSetupsBlockedByRncResources +
numConnectionSetupsFailedByRn +
numConnectionSetupsFailedByRncResources +
numConnSetupsFailedRncTimeout +
numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted)
5. Connection Setup Failures due to RF Failures
Percentage of Connection Setup Failures due to RF Failures =
100% *
(numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout) /
(numConnectionSetupsBlockedByRn +
numConnectionSetupsBlockedByRncResources +
numConnectionSetupsFailedByRn +
numConnectionSetupsFailedByRncResources +
numConnSetupsFailedRncTimeout +
numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted)
6. Connection Setup failures due to Misc. Reasons
Percentage of Connection Setup Failures due to Misc Failures =
100% *
(numConnSetupsFailedRncTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted) /
(numConnectionSetupsBlockedByRn +
numConnectionSetupsBlockedByRncResources +
numConnectionSetupsFailedByRn +
numConnectionSetupsFailedByRncResources +
numConnSetupsFailedRncTimeout +
numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted
127
1.
2.
3.
4.
More detailed information regarding each access failure can be determined by selecting the
Connection Details tab from the statistics view and selecting the Connections List.
Form the Connections List sort by Failure Classification Access Failure. This will pull up all the
access failures and the connection / RF attributes seen just prior to the failure.
If a failure reason cannot be readily inferred from the data in the connections list, click in the access
failures in the Network Statistics which will add the connection attempts that ended in a setup failure
to the Active List.
Select the call in question from the Active list to view the call sequence in the Message View.
128
Multiple TCAs
without an ACK
from the AT.
New CR
NEWTUN Message View
5.
Information about the distance the AT was from the site can be obtained from the Cell Radius Tool in
6.
NEWTUN.
By matching the DOM IP and sector PN where the call setup failed from the Network Statistics view
in the Cell Radius Tool the distance and relative Ec/Io of the originating PN can be determined.
129
130
9.
Percentage Contribution of Route Update Timeouts to Connection setup failures due to RF Failures
(Level 3, per RNC)
o If this metric is the highest contributor among level 3 metrics, it indicates the issues related to RU
timeouts.
o RU timeout happens when Route Update message is not received by the RNC from AT within a
predefined time interval. The timer used for this event is not configurable.
o Higher RU timeouts indicates that the RF conditions were not good enough for the RU message
to reach the DOM. Other contributing factors to this event exists but could not be fixed by us
(needs to have design take care of the software). Link budget need to be verified for the forward
and reverse link coverage balance
131
For DO-RNCs the timing source must be the IP address of a DOM serving as an SNTP server.
For DOMs the timing source must be GPS
o Use the following command to show the SNTP time NORTEL-XX>show sntp time
o If the NE is the DO-RNC and the SNTP time is incorrect see TroubleshootDO-RNC Time
Reference on page 4-31 in NTP 411-2133-949
Check if there are traffic channels between the DOM and RNC
o Check for Abis Peer NORTEL-XX#show abis peer
Check if there are sessions and connections in this DOM
o Check for Sector Carrier connections NORTEL-XX#show 1xevdo rn 10.10.10.10
counters
o Verify that Airlink Resources Currently Allocated is not equal to zero.
o
Access Fails following Connection Drops Troubleshoot connection drop symptoms first.
Access Fails due to insufficient cell coverage radius Data connection access failures due to ATs
being in the fringe of coverage.
Coverage radius statistics can be determined using the Cell Radius Tool in NEWTUN.
In the event that per sector carrier access failures are occurring due to poor RF coverage at an average
distance, steps should be taken to deny access to the site beyond the pre determined coverage radius.
o Collect RNC logs with severity set to 16.
o The distance from the serving cell can be determined by reviewing the last Route Update Detail
prior to an Access Failure.
o Within the Route Update Detail there is a parameter called EPNO that gives the RTD value
which can be used to determine the distance from the site prior to failure.
132
o
o
o
o
Calculate the average RTD for all Access Failures on a per DOM basis. This value will be used to
set the Cell Radii for Cells with a high AF rate.
To modify a carrier associated with a particular DOM open the DOEMS and go to the DOM to
be modified by opening the Network Database in the left hand pane under DOEMS.
Expand the Network Database and select Carriers.
In the Right-hand pane select the appropriate DOM according to IP address.
o
o
Component
ID
DO-RNC CPUs
Call Control
009
RNSM and SC
Message
ID
231
Description
No RC Resources to
Allocate
133
2. RNC logs should be collected simultaneously with AT logs. The RNC logs can be post processed
with NEWTUN on a per RNC basis to determine the number of RC (Resource Control) blocks that
occurred.
3. Drive the cluster route and stop the RNC logs at the conclusion of drive testing.
X 100
Blocking % = DataBlocks + SessionBlocks
TotalCallAttempts
134
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted)
3.Connection Setup failures due to unsuccessful DOM resource allocations can be determined from the
following OMs:
Percentage of Connection Setup Failures due to Unsuccessful DOM
Resources Allocation =
100% *
(numConnectionSetupsBlockedByRn +
numConnectionSetupsFailedByRn) /
(numConnectionSetupsBlockedByRn +
numConnectionSetupsBlockedByRncResources +
numConnectionSetupsFailedByRn +
numConnectionSetupsFailedByRncResources +
numConnSetupsFailedRncTimeout +
numConnSetupsFailedRuTimeout +
numConnSetupsFailedTccTimeout +
numConnSetupsFailedSWError +
numConnSetupsAborted)
4.Airlink Resources allocation blocks during connection setup and soft / softer handoff on Primary
DOMs can be determined from the following OMs:
Airlink Resource Blocks (Connection Setup +
Soft/Softer Handoff) on Primary DOM =
numAllocationBlockRnDriverResourceSC +
numAllocationBlockRnConnectionLimitSC
5.Airlink Resource allocation blocks during connection setup and soft / softer handoff on Secondary
DOMs can be determined from the following OMs:
Airlink Resource Blocks (Connection Setup +
Soft/Softer Handoff) on Secondary DOM =
numAllocationBlockRnDriverResourceSSC +
numAllocationBlockRnConnectionLimitSSC
135
3.To determine the top contributors to blocking from the OMs, evaluate sector performance and determine
Sectors which contribute highly to DOM related resource allocation failures. Following Origination
Sector (ACH of which used for sending/receiving messages) OMs need to be utilized to isolate high
contributors.
sum of OMs:
numATConnectionBlockedByRnSC, numANConnectionBlockedByRnSC and
numFCConnectionBlockedByRnSC
need to be evaluated to identify sectors which experience higher blockings. As OMs are pegged against
these sectors for connection blocks even-if other sectors may be blocking when additional links were
setup during the connection setup, determine the neighboring sectors of these high running sectors.
4.Once neighboring sectors are identified, numAllocationBlockDriverResourceSC OR
numAllocationBlockDriverResourceSSC (for lack of Mac Index and CE),
numAllocationRnConnectionLimitSC OR numAllocationRnConnectionLimitSSC (for Max. allowed
airlinks configuration), numallocationBlockRnSectorCarrierDownSC OR
numallocationBlockRnSectorCarrierDownSSC, numAllocationBlockRnModemTimeoutSC OR
numAllocationBlockRnModemTimeoutSSC ( for internal timeout between SC and Modem), and
numAllocationBlockRnNoConnectionSC OR numAllocationBlockRnNoConnectionSSC (for not being
able to find existing connection during softer handoff) OMs need to be evaluated and appropriate actions
need to be taken to solve the performance issues.
5.Percentage Contribution of TCC Timeouts to Connection setup failures due to RF Failures (Level 3, per
RNC). If this metric is the highest contributor among level 3 metrics, it indicates the issues related to TCC
timeouts.
a.
TCC timeout happens when Call Control on RNC times out waiting for Traffic Channel
Completes message from the AT.
b.
Higher TCC timeouts indicates that the RF conditions were not good enough for the TCC
message to reach the DOM or TCA message reach the AT or TCC Wait Timer parameter value
is not good enough. Link budget need to be verified for the forward and reverse link coverage
balance. Power used (configurable parameter) for sending TCA need to be verified
6.Percentage Contribution of Route Update Timeouts to Connection setup failures due to RF Failures
(Level 3, per RNC). If this metric is the highest contributor among level 3 metrics, it indicates the issues
related to RU timeouts.
136
a.
RU timeout happens when Route Update message is not received by the RNC from AT within a
predefined time interval. The timer used for this event is not configurable.
b.
Higher RU timeouts indicates that the RF conditions were not good enough for the RU message
to reach the DOM. Other contributing factors to this event exists but could not be fixed by us
(needs to have design take care of the software). Link budget need to be verified for the forward
and reverse link coverage balance
7.Percentage Contribution of Software errors to Connection setup failures due to Miscellaneous Failures
(Level 3, per RNC)
a.
If this metric is the highest contributor among level 3 metrics, it indicates the issues related to
software errors. Software errors occur when the memory pool usage reaches the maximum
1500 connections and failure to create HDR instance occur
8.Percentage Contribution of Software errors to Connection setup failures due to Miscellaneous Failures
(Level 3, per RNC)
a.
b.
c.
d.
e.
If this metric is the highest contributor among level 3 metrics, it indicates the issues related to
aborted connection setups due to following
Route update message with Keep = No for all pilots received and Enable Access when last RU
has all keep = No is set to false
Failure to allocate memory from Heap
Local close request is generated by operator
Page timer expires and AT initiated connection setup was not in progress.
9.From CLI determine if the blocking is related to the maximum number of connections being exceeded:
NORTEL-XX>enable NORTEL-XX# show 1xevdo sector carrier <IP> < channel number> <pn
offset> counters all
RN Resource Allocation Counters
Number of Requests Sent: 2
Successful Allocations: 2
Blocks:
Driver: 0
Connection Limit: 0
Sector Down: 0
Modem Timeout: 0
No Connection: 0
Malformed message: 0
10. The connection limit in the DOM is determined from two parameters: Maximum Forward Distribution
Delay and Connection Limit per sector element. The DOM logic sets the connection limit to the
minimum of the two of these parameters. In addition, the RNC Maximum Airlink Connnections per
Sector Carrier overrides these two parameters [3].
11.The following is an overview of how the DOM connection limit algorithm works.
12.While setting up a connection, the configured values for Connection Limit per sector at the DOM and
Maximum Airlinks per SectorCarrier are checked separately, in the following order:
Step 1: Check against RNC Maximum Airlinks Per SectorCarrier setting. If there is a block,
totalBlockedAirlinkRsrcAllocationsSectorCarrier is pegged and setup is aborted.
Step 2: Check against DOM per-sector Connection Limit setting. If there is a block,
numAllocationBlockRnConnectionLimitSC is pegged and setup is aborted.
137
Step 3: Send allocation request to modems (driver). It will check for number of channel elements
available. If is a block, numAllocationBlockRnDriverResourceSC is pegged and setup aborted
(setup aborted meaning the next step is not executed).
Here are some examples to illustrate how this algorithm works:
With RNC Maximum Airlinks per SectorCarrier = 40, the DOM Connection Limit per sector =
48, and Number of Channel Elements = 30, a block at 3rd step of the algorithm will occur for 31st
connection
With RNC Maximum Airlinks per SectorCarrier = 30, the DOM Connection Limit per sector =
48, and Number of Channel Elements = 30, a block at 1st step of the algorithm will occur for 31st
connection
With RNC Maximum Airlinks per SectorCarrier = 48, the DOM Connection Limit per sector =
30, and Number of Channel Elements = 30, a block at 2nd step of the algorithm will occur for 31st
connection
13. Verify from the RNC RF datafill that the Maximum Airlinks per sector carrier is according to the Nortel
recommended value of 48.
NORTEL-XX>enable
NORTEL-XX#show 1xevdo config
Maximum Airlinks per Sector-Carrier 48
14. If the value is below recommended it can be reset through CLI as follows:
NORTEL-XX(config)#1xevdo max-airlinks-per-sector-carrier 48
15. Verify from the DOM RF datafill that the connection limit is at the recommended value of 48 for the
sector carrier being considered. If the value is less than the recommended value use the following CLI
command to adjust the value:
NORTEL-XX>enable
NORTEL-XX# configure
NORTEL-XX(config)# sector-element element1
NORTEL-XX(config-element)# connection limit 48
16. If Blocking persists there may be provisioning issues. Contact network engineering for a verification
provisioning of the network elements.
Setup the log masks to collect all EVDO attributes. If the hybrid mode testing is to be performed
also collect all CDMA log attributes.
Following the process given in the Cluster drive testing section , configure the AT to download a
500 MB test file wait for a period greater than the data inactivity timer (datafilled in the system
to allow the connection to transition to a dormant state between tasks. Repeat the download over
and over until at least 30 minutes worth of AT logs have been collected. Please see the example
below.
138
Drive the cluster where drops are occurring to collect AT logs. Keep the drive test route within one
RNC boundary.
Follow the RNC logging setup procedure in section 5.3.1 to configure logging.
The most important log to collect is:
Severity
Level
5
3.
Component
ID
DO-RNC CPUs
Call Control
009
RNSM and SC
Message
ID
423
Description
Connection Drop
This message contains useful information such as: drop reason (FTC lost/RTC Lost), frame error
rate, and the last 2 sets of TCA and Route Update messages. An example of Message 423 is shown
below:
Component
ID
5
9
9
16
Call Control
Call Control
Call Control
Call Control
Call Control
9
9
9
9
9
DO-RNC
CPUs
RNSM and SC
RNSM and SC
RNSM and SC
RNSM and SC
RNSM and SC
9
Nortel Networks Confidential and Proprietary
Message
ID
Description
213
215
216
220
Connection Closed
Connection Request
Traffic Channel Complete Message
Local Connection Close Event : Reason #
Rx from AT : Connection Close Message :
Reason #
221
139
Call Control
9
9
9
Call Control
Call Control
Call Control
9
9
9
9
RNSM and SC
RNSM and SC
RNSM and SC
RNSM and SC
266
269
326
429
140
NumConnectionsOpened: The number of connections opened successfully on this DORNC (the AT arrives on the Traffic Channel).
Effective Calls = Total Elapsed Active Time / Average Data Call Holding Time
Example: Method #1 from EVDO RFO Statistics
141
o
o
o
x100
DroppedConnection% = Drops
Successful
Accesses
Incomplete
Traffic
5. The Dropped Connection Rate should be within the exit criteria specifications
142
(sNumSoftHandoffReleatedDrops +
cNumSoftHandoffRelatedDrops) =
numRevLinkSHOBlockedByRncResources +
numRevLinkSHOBlockedByRn +
numRevLinkSHOFailedByRncResources +
numRevLinkSHOFailedByRn +
numRevLinkSHOFailedTccTimeout +
numRevLinkSHOFailRncTimeout
(sNumMiscDrops +
cNumMiscDrops) =
numConnectionCloseInternalError +
numConnectionCloseRlp +
numConnectionCloseStateMismatch +
numConnectionCloseSectorDown +
numConnectionCloseSsmDisable
Total Dropped Connection Rate (DC) =
(cNumRFRelatedDrops +
cNumSoftHandoffRelatedDrops +
cNumMiscDrops) /
(cNumConnectionSetupSuccess +
numSToCCrossovers)
Where: numSToCCrossovers number of connections opened before the demarcation point, and were
closed normally after the demarcation point.
Poor RF coverage (probably want to sub-divide into fwd/rvs link, edge of network vs.
internal)
AT LogsMobile Rx Power < -85 dBm
Mobile Tx Power > -10dBm
Traffic C/I <= -2dB
Drop
RNC LogsUse EVDO remote optimization scripts to process and analyze RNC logs. The scripts will
be available as a module in the Nortel version of EMS parser due for release in June 2005.
Currently, the scripts are stand alone and are available on a case by case basis for
Nortel Networks Confidential and Proprietary
143
optimization. The following output from the CROS scripts shows a drop due to coverage
related issues:
RNC :
Drop
File
Drop
Num
HMS
23
49
####
#
Fail
Trigger
Connection
Drop Log
23
50
####
#
Connection
Drop Log
Drop
or AF
FCA
Drop
Estimated
Root Cause
Whole call not logged so
unclassified
Fwd and Rvs coverage both
bad, genuine coverage issue
Connection Suicide
Poor Forward link coverage typically will result in a dropped connection when the Ec/Io
(Strength) of all pilots is less than -9dB.
Example from RNC Logs
MOB CSMOpen] : Rx from AT : Route Update Message : Seq# 7
MOB CSMOpen] : Route Update Detail : Processing Seq# 7 : Connected :
RefPN 228 : RefStrength 37 : RefKeep Y RefStrength x -0.5 = -18.5dB
MOB CSMOpen] : Route Update Detail : RefPN IP Address 10.245.36.39
MOB CSMOpen] : Route Update Detail : Other Pilot# 1 : PNPhase 18942 :
Strength 63 : Keep Y : ChanIncluded N : Channel 0|00 Strength x -0.5 = 31.5dB
MOB CSMOpen] : Route Update Detail : PNPhase 18942 : PNOffset 296 : RN
IP 10.245.37.2 : EPNOin4Chips 0
FlowControl 2 Open] : Rx : Route Update Event event
MOB CSMOpen] : Counts after formMobility: Add (0) : Keep (2) : Delete (0) :
Keep(2) : Delete (0)
SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection
ID:0x00103965] : Rx from RN : FTC Stopped Indication Message : Success
SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection
ID:0x00103965] : Rx from RN : FTC Desired Indication Message : Success
SDU CSMOpen] : Setting Active soft handoff leg [IP:10.245.36.39,
CR:0x00000aee, RNC Connection ID:0x00103965] : DRC Mask 0x00000004
SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection
ID:0x00103965] : Rx from RN : FTC Stopped Indication Message : Success
SHO SHOSM_Open] : [IP:10.245.37.2, CR:0x00000aee, RNC Connection
ID:0x000f2c54] : Rx from RN : RTC Lost Status Indication Message :
DRCMask:0x00000002
SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection
ID:0x00103965] : Rx from RN : RTC Lost Status Indication Message :
DRCMask:0x00000004
CSM CSMOpen] : Rx : Local Connection Close Event : Reason 9
CSM CSMOpen] : Tx to AT : Connection Close Message : Reason Code 2
Solution
For areas where dropped connections result from poor RF coverage changes to the
underlying 1xRTT network may be required. Any changes to the underlying network will
have to be coordinated with the customer.
Comments
144
Description
Symptoms
in Data
(AT
Logs/RNC
Logs/OMs)
Slow handoff
AT Logs
N/A
RNC Logs
Observe the timestamps between last TCA and Traffic Channel Complete Message from AT not received
in time. If the delta between the two is greater than the TCC wait timer it may be necessary to increase this
value especially if the TCC is received after the timeout.
OMs
Percentage of Reverse Link Soft Handoff due to RF Failure =
100% *
numRevLinkSHOFailedTccTimeout /
(numRevLinkSHOBlockedByRn +
numRevLinkSHOBlockedByRncResources +
numRevLinkSHOFailedByRn +
numRevLinkSHOFailedByRncResources +
numRevLinkSHOFailRncTimeout +
numRevLinkSHOFailedTccTimeout +
numRevLinkSHOAborted)
Solution
Comments
Description
Symptoms in Data
(AT Logs/RNC
Logs/OMs)
Full active set - no mechanism to select best 6 if 7 or more requested ( Fixed in 3.0)
AT Logs
RNC Logs
Look for Route update messages in the logs where the number of other pilots listed
is six and the SHO SHOSM Failed.
145
: Route Update Detail : Processing Seq# 142 : Connected : RefPN 476 : RefStrength 7 : RefKeep Y
: Route Update Detail : RefPN IP Address 10.245.37.59
: Route Update Detail : Other Pilot# 1 : PNPhase 28433 : Strength 42 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : Other Pilot# 2 : PNPhase 23523 : Strength 38 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : Other Pilot# 3 : PNPhase 20468 : Strength 47 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : Other Pilot# 4 : PNPhase 26139 : Strength 63 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : Other Pilot# 5 : PNPhase 27183 : Strength 63 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : Other Pilot# 6 : PNPhase 27904 : Strength 63 : Keep Y : ChanIncluded N : Channel 0
: Route Update Detail : PNOffset 444 : PNPhase 28433 : RN IP 10.245.37.61 : EPNOin4Chips 7
: Route Update Detail : PNOffset 368 : PNPhase 23523 : RN IP 10.245.37.72 : EPNOin4Chips 0
: Route Update Detail : PNOffset 320 : PNPhase 20468 : RN IP 10.245.37.30 : EPNOin4Chips 0
: Route Update Detail : PNOffset 408 : PNPhase 26139 : RN IP 10.245.37.67 : EPNOin4Chips 9
: Route Update Detail : PNOffset 424 : PNPhase 27183 : RN IP 10.245.37.71 : EPNOin4Chips 14
: Route Update Detail : PNOffset 436 : PNPhase 27904 : RN IP 10.245.37.61 : EPNOin4Chips 3
: Rx : Route Update Event event
: Created PBTSCM node [IP:10.245.37.72, CR:0x00000aee
: Created PBTSCM node [IP:10.245.37.61, CR:0x00000aee
: Created PBTSCM node [IP:10.245.37.30, CR:0x00000aee
: Created PBTSCM node [IP:10.245.37.67, CR:0x00000aee
: Created PBTSCM node [IP:10.245.37.71, CR:0x00000aee
: Counts after formMobility: Add (6) : Keep (1) : Delete (0)
Solution
OMs
numRevLinkSHOFailedByRn
This problem is not readily recognized in the OMs as it will be aggregated with
other SHO failure reasons.
Patch 2.1.0.25 (Performance Patch) will fix the 7th Pilot issue by limiting the number
of pilots to be added to the active set such that it never exceeds 6. At most (6
(Current Active Set Size)) pilots may be added to the Active Set. Pilots will be
chosen according to signal strength.
Comments
Description
Symptoms in Data
(AT Logs/RNC
Logs/OMs)
In the logs above the Route update is for PNs 93, 45, and 426. The source has no
knowledge of the PNs 93 and 426. It is likely that these PNs live on another RNC.
146
OMs
Number of Times an Unknown Pilot in RU when a Primary Pilot Is
Strongest =
numPilotLookupFailuresUnknownPilotSC
Number of Times an Unknown Pilot in RU when a Primary Pilot Is
Strongest =
numPilotLookupFailuresUnknownPilotSSC
Descriptio
n
Symptoms
in Data
(AT
Logs/RNC
Logs/OMs)
Add source RNC as secondary RNC in interfering DOMs secondary Homing table.
Add adjacent (interfering) DOM to the neighborlist of source sector carrier.
In 3.0, the existence of alien pilots in the route update message will not cause the
call to fail but will contribute to interference which can cause drops and impede
maximum sustained data rates.
Hybrid AT non-returning tune away
AT Logs
In a long tune away the AT will release the EVDO connection after being tuned away for
longer than 5 seconds. To recognize long tuneaways from the message view in EVDO set the
message filter for the following:
1. CDMA 2000 f-csch (Paging Messages) - All
2. CDMA 2000 forward Traffic Channel Neighborlist Update
3. IS856 Access (Connection Request, Route Update)
4. IS856 Control Channel Directed (Traffic Channel Assignment, ACAck,
5. IS856 Control Channel Broadcast All overhead messages
6. IS856 Forward Traffic (Connection Close, Traffic Channel Assignment,
7. IS856 Reverse Traffic (Connection Close, Route Update, Traffic Channel Complete
8. QCP EVDO Connection Release, ARQ Effective Receive Rate, Connection Attempt
From RFO Map View:
147
148
149
RNC LogsIn the RNC logs long tuneaways can typically be identified by an FTC Stopped Indication
Message (Null DRC from AT) followed by at least 5.12 seconds of inactivity in the RNC
messaging. In the example below it can seen that the stopped indication is sent and roughly 5
seconds later the AT communication is re-established with the DO network but the RTC is
lost as a result.
Example from RNC logs:
16 03-10-05
03:45:10.972
51 03-10-05
03:45:11.002
ca 03-10-05
03:45:11.002
(No activity)
57 03-10-05
03:45:15.882
50 03-10-05
03:45:15.882
08 03-10-05
03:45:15.882
Solution
OMs
numConnectionClosenoFTC Pegged when an FTC Stopped is received and an FTC
Desired message is not received within 5 seconds
Currently there is no complete fix for a drop due to a long tune away. Some theories on
insuring that the 1x Channel lists are consistent from BTS to BTS.
Comments
Description
Symptoms
in Data
(AT
Logs/RNC
Logs/OMs)
SHO block
RNC LogsIn the RNC logs a drop due to a SHO block may be identified by a RC lookup fail.
Example from RNC Logs:
150
Description
Symptoms in Data
(AT Logs/RNC Logs/OMs)
Solution
Comments
Description
Symptoms in
Data
(AT Logs/RNC
Logs/OMs)
OMs
numConnectionCloseRtcLost
numConnectionCloseRtcLostSlot
Solution
Increase the Reverse Link Fade Timer. Care must be taken when doing this as it may
waste resources when adequate RF does not exist to overcome a fading.
Comments
151
Description
Symptoms
in Data
(AT
Logs/RNC
Logs/OMs)
Missing neighbor
AT LogsN/A
RNC Logs
In the logs it can be seen from the Route Update Detail that the IP lookup failed for PN 348
and PN 351.
Solution
If the drop is in fact due to a missing neighbor the neighborlists for the reference site and its
neighboring sectors should be evaluated to insure all the required PNs are included and
that the proper IPs are also included.
Alternatively, if the requested PNs are from sites in an adjacent RNC insure that those
DOMs have the source RNC datafilled as a secondary RNC and that this DOM is in the
neighborlist of the source DOM.
Comments
152
Description
Symptoms in Data
(AT Logs/RNC
Logs/OMs)
PDSN Release
PDSN release A10 de registration
AT Logs
RNC Logs
When a connection to the AT is opened LCP requests are sent over the open
channel, but are not responded to since there is no PPP end point at the AT. The
connection will stay up until the PDSN times out (typically about 30 seconds +).
This condition will lead to an increase in dropped calls.
OMs
Percentage of A10 Connection Close due to PDSN Initiated Close =
100% *
numA10ClosedPDSNInitiatedRelease /
totalA10Closed
Solution
Comments
Insure that the UA10 minimization feature is enabled. The UA10 feature will
prevent unnecessary setup of A10 and thus PDSN timeouts. With UA10 enabled
the AT will only establish a PPP session with the PDSN when the following
events occur:
Dormant RNC handoff when IP of PDSN that has the PPP session is passed
in the A13 Response message
153
Description
Symptoms
in Data
(AT
Logs/RNC
Logs/OMs)
Solution
Comments
10.7.1.3 OMs
Method of Collection
1.Setup the OM data collection as outlined in section 5.4.4.1 using the RNCPerfByRNC_R3.0 template.
2.To get a statistically significant trend of performance, particularly in a market that is already
commercial it will be useful to collect OMs over an entire week.
3.If the market is not commercial perform drive test of each cluster on the golden route.
1.Forward Sector Switching Success Rate should be in >=95% for the 90th percentile of all attempts.
154
Solution
RNC Logs
Look for FTC Desired Not Received in Time.
Increase the FTC Desired Wait Timer. Insure neighborlist integrity and
reciprocity between BTS.
Comments
155
N/A
10.8.1.3 OMs
Forward Sector Switching performance metrics are only available for DOM to DOM (inter-DOM) sector
switching. Intra DOM sector switching is transparent to the DO-RNC. The following OMs are available
to calculate the switching success or failures between BTSs.
CLI can be used to determine the DRC switching successes between DOMs on a network or Session
specific basis as follows:
NORTEL-XX#show 1xevdo counters all (Network wide)
NORTEL-XX#show 1xevdo session <UATI> counters all (UATI / Session specific)
156
Unit
Recommended
1000 30000
1/1000
seconds
5000
NORTEL-07>enable
NORTEL-07#configure
Then the TCC Wait Timer can be changed through the following command
NORTEL-07(config)#1xevdo timer tcc-wait <TCC Wait Timer setting>
To verify the timer was changed, go back to the enable state and enter the following,
NORTEL-07#show 1xevdo config
The parameter is listed as CSM Traffic Channel Complete Wait Timer.
157
Method of Collection:
1. Follow RNC logging setup given in section 5.3.1 of this document.
2. Collect logs at Severity level : 8
10.9.1.3 OMs
Method of Collection:
1. Follow the procedure in section 5.4.4.1 for configuring OM collection
2. The following OMs are available to measure the RL SHO attempts, successes and Success Rate:
Rev Link SHO Attempts = numRevLinkSHOAttempts
Rev Link SHO Successes = numRevLinkSHOSuccess
Successful Reverse Link SHO Rate= numRevLinkSHOSuccess / numRevLinkSHOAttempts
3. Reverse Link Soft handoff OMs can be determined from the CLI prompt using the following
commands:
NORTEL-XX# enable
NORTEL-XX# show 1xevdo counters all
NORTEL-XX# show 1xevdo session <UATI> counter all (per UATI based for specific ATs)
158
0
0
0
0
0
159
Failed by RNC :
0
Failed because of no TCC :
0
Failed because of RNC Resource Timeout :
0
4. If any of the RL SHO failures are due to blocking refer to the troubleshooting section on call blocking.
5. The unsuccessful DOM resources allocation could be the result of improper setting of the maximum
airlinks per sector attribute in the DOM. See Per Sector Airlink Resources Allocation on page 13-1 and
DOM Resources Usage on page 16-1 of reference [18].
6. Any RL SHO failures due to Failed RNC Timeouts can only be due to the DOM being on a software
load prior to Release 3.0.
7. TCC timeouts may also be enhanced by increasing the TCC Wait Timer using CLI. The timer can be
configured in increments of 1 millisecond between 1000 to 30000 milliseconds. (default 5000)
NORTEL-XX> enable
NORTEL-XX#(configure)
NORTEL-XX(config) #1xevdo timer tcc-wait <time>
160
5.
6. At the end of the download stop the TCP trace capture. Ethereal will process the trace logs and
display the protocol messages in the workspace.
7. Insure that within the PPP IPCP protocol configuration request message options field, that IP
Header compression is not a requested option or that it is being rejected. (see below)
Example: No Request for Header Compression Option
161
10.10.1.4 OMs
There are no OMs currently available for measuring per user throughput
100
80
20
80
10
10
10%
10%
10%
10%
10%
10%
2.457
2.457
1.2288
2.457
1.843
1.2288
Application
Throughput (Mbps)
1.83
1.464
0.183
1.464
0.1372
0.0915
Delta from
Requested Rate
23.75%
25.5%
21.7%
162
4. The Graph view in RFO can also be used to show Application, and Physical layer throughput versus
time.
Example: Application Throughput Statistics
Formula:
Average Physical Layer Throughput => Avg Actual Data Rate (kbps)
163
2. It should be expected that user throughput will suffer in areas at the fringe of coverage, inter RNC
borders, or where user traffic is high (scheduling contention).
3. Verify that he DRC requested rate and the Application Rate track within about 15% of one another. In
addition insure that the C/I and PER are valid for the data rates being requested. Also verify from the
EVDO Rate Statistics in XCAL, that the majority of DRC requests are between 1.2288 and
2.457Mbps.
Example: Avg. Effective DRC, Application RX Bytes, C/I1, and TC_PER
164
Example: Grid View App. RX Rate, Avg. Effective DRC, C/I, and TC_PER
Formula % difference DRC and App rate: ((Avg. Effective DRC App Rx Rate)/Avg. Effective DRC)
4. If the C/I and PER values are ideal (i.e. between 8 12dB) and the DRC rate is low, verify the port
settings in the data collection tool are correct. If using XCAL-DO go to My Computer Properties
Hardware Device Manager Modems and Ports. Verify the COM ports for the modem are those
configured in XCAL COM Port and Data Port Settings screens. For Optis-M, port configuration is
automatically detected.
165
Data Port
Control Port /
Diagnostics Port
5. If DRC and C/I are within ideal limits, verify Application throughput is between 1.5 and 2.1Mbps.
166
6. If the application throughput seen with the data collection tool is lower than expected. Perform a DOS
FTP download to confirm the achievable throughput on the FL. If the DOS throughput agrees with the
tool, verify that IP header compression is disabled. Go to: Network and Dialup Connections PPP
Adapter Advanced TCP/IP properties.
Example: IP Header Compression
7. To determine if there are per user throughput problems where coverage is generally expected to be
good, post process collected AT logs with RFO and verify that the C2I and DRC request rates track and
that the PER does not exceed 1%. Remember that the mobile will decrease the DRC request rate to
maintain PER at 1%. (see below)
8. Verify that the application throughput is approximately 15% less than that of the Forward physical rate.
An inspection of the graph view in RFO should indicate that application throughput trends less than that
of the physical layer. In addition to the graph view, an estimate of the difference in throughput can be
determined from the grid view by filtering out all the blank values in the Application RX Rate column
(see below).
167
Example: Graph View App RX Rate vs. Fwd Physical Data Rate
9. If the physical layer throughput is less than that of the application layer throughput and slot utilization is
low, software compression is probably enabled in the network and dial up settings on the collection tool
/ laptop. Software compression will also exaggerate the actual application layer throughput. Software
compression should be disabled prior to performing further tests.
168
Expected
Example Calculation: ((Fwd Physical Data RX Rate Application RX Rate) / Forward Physical Data
Rate) = 13.5%
10. An initial review of the application statistics in RFO may NOT indicate an immediate problem with
achievable data rate. However, a review of the physical layer statistics may show disparity in
throughput between the physical and application layer. Application layer throughput should always be
less than the Physical Layer rate.
Example: Data Statistics RX Active Throughput (App. Data Rate)
169
11. Slot utilization indicates if the application is making efficient use of the physical resources (time slots).
In ideal RF conditions slot utilization should be between 65 90%. (See below)
Example: Graph View of Application, Physical layer Throughput, and Avg. Effective DRC
Application RX Rate
is higher than the
Physical layer
throughput rate.
Compression could be
enabled.
Example: Graph View of Poor Fwd Slot Utilization %
170
App RX Rate (kbps) = Average Application Layer throughput per reporting period
Forward Physical Data RX Rate (kbps) = Instantaneous Physical Layer throughput from Forward Rate
Statistics V2 message. Indicative of how many packets are sent at a given rate every second. Packets per
rate are accumulative on a per second basis.
Average Effective DRC Rate = Data rate requested by the AT based on measured C2I (RF conditions).
Fwd Slot Utilization % = Slot utilization is low due to inefficient use of available slots. In ideal RF
conditions where 1 user has the full queue of the scheduler, slot utilization should be in the 80 -90% range.
Example: Fwd Rate Statistics V2 Message
Poor Slot Utilization ~ 2%
171
12. Low slot utilization with relatively good application layer throughput may be indicative of software
compression. Otherwise poor slot utilization is likely due to poor RF conditions (i.e. PER, C/I, TX poor,
etc), AT problem, or many users accessing the sector.
13. If RF problems (i.e. Dropped connections and Access Failures) are excessive, refer to: Section 7.4 and
7.6 for tips on how to recognize and address these problems.
14. It is important to remember that the proportional fairness scheduler will impact throughput as more
users access the network (scheduling contention). The scheduler divides time on the system amongst
users according to current RF conditions, data rate requests, and past data rate history. The following
equation describes how the scheduler determines the throughput per slot per user. .
1
1
Tk (n + 1) = 1 Tk (n ) + Rk (n), n = 0,1,2,...
tc
tc
Where:
TK(n) = mobile #ks throughput (T) history for time slot n
TC is a time constant
RK(n) = the rate at which the user is being served in slot n
15. To determine the number of users / links in use at a particular time in the AT log file, open the
Message View in RFO and look for the Quick Config message. This message gives an indication of
how many links are attached to a sector while data throughput testing is being performed.
16. The RPC count indicates the number of users attached to the sector at the time. According to the
example below, there are four links in use.
Example: Quick Config Message
172
17. Given that the maximum forward data rate is contingent upon the reverse link data rate capability, a
verification of Forward Physical Rate to reverse data rate ratio should be made. A ratio of no more than
63:1 (optimally 40:1)is sufficient to support data rate requests in excess of 2.0Mbps.
Example: Fwd Physical Rate to reverse data rate ratio
Recommended
Num-AT-To
59
Rate
153.6 Kbps
Recommended
128
64
32
8
255
32
16
16
19. RL SHO failures can impact FL throughput by limiting the ACK rate on the RL. To determine the
per UATI RL SHO failure rate, from CLI go to enable mode and view the per UATI connection
counters:
1xEVDO session <UATI> counters <All>:
Reverse Link Soft Handoffs :Attempts : 0
Success : 0
Nortel Networks Confidential and Proprietary
173
Blocked by RN : 0
Blocked by RNC : 0
Failed by RN : 0
Failed by RNC : 0
Failed because of no TCC : 0
In EVDO soft or softer handoffs are initiated by the AT sending a Route update message to request
the addition or removal of a pilot to its Active Set. Each Route update is counted as a handoff attempt.
Soft handoff failures generally occur as a result of resource blocking or failures at the RNC or RN.
Currently there are no reason codes for specific resource blocks.
Other failures may have occurred as a result of configuration errors. In the event of failures verify the
following RF data fill parameters:
TCC Wait Timer
The amount of time the RNC waits for a traffic channel complete (TCC) message from the AT. A
Traffic Channel Assignment message is typically sent by the RNC to the AT as part of a reverse link
soft handoff. If the TCC is not received in time the connection will be closed.
Range
Unit
Recommended
1000 30000
1/1000
seconds
5000
IS856Neighbor
Defines the neighbors surrounding the sector an AT may be connected to. Inconsistencies in the
neighborlist may result in missed handoffs. Insure that reciprocal neighbors are not missing in
adjacent sites neighborlists.
19. If the Fwd Physical Rate to reverse data rate ratio is acceptable, a verification of the RLP Timeout rate
should be performed. Based on an access layer PER of 1% the system will inherently encounter NAK
timeouts (i.e. RLP packet retransmission is not received in .5ms or received in error) but the percentage
should be small. If the percentage is large then there will be failures or retransmissions at the TCP layer.
It should be noted that the relationship between PER of the initial packet and the PER of a retransmitted
packet in effect makes the worst case PER .01% (i.e. PER initial =1% x PER for retransmitted =1% =
.01%)
By example, assume that a 5Mbyte file (40Mbits) is being downloaded. If this is application layer we
need to assume roughly 15% overhead (40Mbits *1.15 = 46Mbits). Each FTC MAC packet is 1024 bits
so the system will send ~ 45,922 packets. Then the number of NAK timeouts that should be expected =
45,922x.01% = ~5NAK timeouts (i.e. packets not received or received in error on retransmission).
174
DOM_Logging.txt
(1 KB)
21. If the celldm logs show packets are being received out of sequence at the DOM and there are excessive
RLP NAKS as a result, the SDU Resequencing Wait Timer may be too short. The default wait period is
25milliseconds and is adjustable in increments of 5 milliseconds. The time may be updated with
following CLI command:
NORTEL-XX(config)#1xevdo sdu-resequencing-wait <period>
22. To verify of further impact on throughput, TCP trace data should be collected using Windump or
Ethereal in parallel with AT logging. TCP traces will show whether or not there were delays in
application throughput due to TCP segment losses / retransmissions.
23. TCP segment retransmissions are common when IP header compression is utilized in wireless
networks. It is known that TCP has a slow recovery time when a TCP segment is lost or corrupted as
the decompressor becomes unsynchronized. The decompressor only knows the next packet sequence it
is supposed to receive by validating the checksum value calculated based on difference values from one
header to the next. In the case of high speed wireless links, the contents of the TX buffer window (64k)
will be exhausted before TCP recovers. All TCP packets will be discarded without Acknowledgment by
the receiver before TCP sends a packet with an uncompressed header. The packet with the
uncompressed header will allow TCP to recover by sending retransmissions of the discarded packets.
175
Compressor
E D C B A
RF Link
E D B A
Decompressor
D C B A
This segment is really D not C
176
Packets are
discarded.
24. If many instances of stalled throughput or packet discards are seen in TCP traces, process the logs with
TCPTrace http://tcptrace.org/. Use the (tcptrace-lr) command to determine what node is most affecting
packet loss and / or latency.
Example: TCPTrace Output
================================
TCP connection 6:
host k:
h226.13.39.162.ip.alltel.net:20
host l:
216.96.112.225:1112
may24_16_eth
k->l:
l->k:
total packets:
38734
total packets:
25446
38733
25446
25444
236
0
1
177
38731
293
416556
outoforder pkts:
843
0
0
0
0
outoforder pkts:
0
0
1/2
req sack:
req sack:
sacks sent:
sacks sent:
236
1/1
0 pkts
0 bytes
mss requested:
1428 bytes
mss requested:
1428 bytes
452 bytes
0 bytes
1427 bytes
0 bytes
17136 bytes
65535 bytes
16384 bytes
2703 bytes
0 times
17135 bytes
initial window:
2856 bytes
initial window:
2 pkts
1460 bytes
0 bytes
0 times
65472 bytes
initial window:
0 bytes
initial window:
0 pkts
missed data:
0 bytes
missed data:
truncated data:
0 bytes
truncated data:
0 pkts
0 bytes
truncated packets:
0 pkts
0 bytes
0 bytes
0 bytes
truncated packets:
0 pkts
503.634 secs
0.000 secs
idletime max:
18076.0 ms
idletime max:
17915.8 ms
throughput:
108944 Bps
throughput:
24962
RTT samples:
RTT samples:
RTT min:
10.0 ms
RTT max:
200.3 ms
RTT avg:
5.8 ms
RTT stdev:
12.1 ms
0.0 ms
RTT min:
RTT max:
RTT avg:
RTT stdev:
0 Bps
2
130.2 ms
140.2 ms
135.2 ms
0.0 ms
130.2 ms
178
24959
10.0 ms
200.3 ms
5.8 ms
12.1 ms
post-loss acks: 9
1
140.2 ms
140.2 ms
140.2 ms
0.0 ms
post-loss acks: 0
48
ambiguous acks:
0.0 ms
190.3 ms
11.5 ms
0.0 ms
35.0 ms
0.0 ms
13420
duplicate acks:
408
triple dupacks:
18
triple dupacks:
max # retrans:
max # retrans:
0.0 ms
0.0 ms
duplicate acks:
190.3 ms
3014.3 ms
1258.5 ms
856.2 ms
0.0 ms
0.0 ms
0.0 ms
0.0 ms
25. Throughput problems related to packet loss issues are often related to T1 connectivity of timing issues
in the backhaul that result in path or line code violations. To see whether path or line code violations are
consistently occurring, monitor the link status alarms in the DO EMS
27. Verify if the T1 timing source is incorrect or faulty T1 Timing Source. Set the interface to the correct
timing source. Insure interfaces at both ends are the same.
NORTEL-XX>enable
NORTEL-XX#configure
NORTEL-XX(config)#controller t1 1/0/1
NORTEL-XX(config-controller)#clock source line-loop
Confirm timing configuration change:
NORTEL-XX#clear counters
NORTEL-XX#show controller t1 1/0/1
26. If packet loss is found from the TCP trace logs to be greater than 1%, a system health check should be
performed as given in [17] page 30.
179
10.11.1.3 OMs
The aggregated sector throughput at the physical layer can be estimated using the OM PerfFLThruput.
This OM only gives a time averaged estimate of the sector throughput based on a sliding 10 second
window.
Method of Collection:
1. Telnet into the DOM and enable the FTC Scheduler. It is necessary to enter configuration mode to
enable FTC scheduler NORTEL-03> config NORTEL-03 celldm scheduler ftc on
2. Log out of the DOM NORTEL-03> exit.
3. Repeat this process for all DOMs
4. The Average FTC Physical Layer throughput loading = perfFLThruput / 1000 (Unit: kbps)
5. Note: The throughput value given will only be for the last reported sampling window
6. A histogram of slot utilization and throughput can be generated using CLI show celldm flthruput
<sector>
Sample: flthruput histogram from celldm
NORTEL-XX#show celldm flthruput alpha
RATE: 38k4 76k8 153k6 307k2 614k4 921k6 1228k8
1843k2 2457k6
SLOTS: 0 0 0 0 0 0 0
0 159235
FL data throughput for sector alpha: 534749 bps
180
181
6. At the end of the test stop AT logs and post process using EVDO RFO.
10.12.1.3 OMs
The Reverse link physical layer throughput is estimated over a 3 second sliding window and is
available from the per sector RL throughput OM perfRLThruput. The per user throughput can be
estimated as follows:
Method of Collection:
1. Collect the Reverse Link histogram from the celldm logs show celldm rlthruput <sector>.
NORTEL-XX#show celldm rlthruput alpha
RATE: 9k6 19k2 38k4 76k8 153k6
BYTES: 139648 533632 428928 38656
512
RL data throughput for sector alpha: 26112 bps
1. After post processing the AT logs with RFO, open Statistics View Data Statistics Application
TX Active Throughput.
Example: RFO RL throughput statistics
( Bytes * 8)
TotatTransmissionTime = (( Bytes * 8) / Rate)
TotalBytes Re ceived =
AllRates
AllRates
182
3.Verify that the laptop / collection tool TCP/IP settings are in accordance with Appendix E TCP/IP
Configuration.
4.Verify if the problem is isolated to a specific DOM or DOMs. If the problem is over several DOMs
troubleshoot Ethernet Performance or GRE / A10 resequencing. See page 4-12 and 4-75 in [18].
5.If the problem is isolated to 1 DOM then troubleshoot the site for Radio malfunction, antenna problems,
feeder problems, DOM chassis malfunctions, or interference. See page 5-126 [18].
6.If the problem is random throughout the network troubleshoot the backhaul timing source or connectivity.
See page 4.34 36 [18]
7.Verify the following RF datafill parameters are within recommended values established by Nortel and the
customer:
Reverse Link Max Rate Table
This table provides a per sector datafill method to control the maximum data rate granted to the AT. CLI
must be used to make changes to this table. Changes made to this table are not persistent (i.e. any restart
of the DOM will cause the operator to have to go in and reset the values.
Num-AT-From
1
Recommended
Num-AT-To
59
Rate
153.6 Kbps
RAB Threshold
Range
Unit
Recommended
0 16384
1/16384 %
of Load
9830 (60%)
183
Recommended
128
64
32
8
255
32
16
16
8.RL SHO failures can impact the RL throughput. From CLI go to enable mode and view the per UATI
connection counters to determine the number RL SHO failures:
1xEVDO session <UATI> counters <All>:
Reverse Link Soft Handoffs :Attempts : 0
Success : 0
Blocked by RN : 0
Blocked by RNC : 0
Failed by RN : 0
Failed by RNC : 0
Failed because of no TCC : 0
In EVDO soft or softer handoffs are initiated by the AT sending a Route update message to request the
addition or removal of a pilot to its Active Set. Each Route update is counted as a handoff attempt. Soft
handoff failures generally occur as a result of resource blocking or failures at the RNC or RN. Currently
there are no reason codes for resource blocking. Other failures may have occurred as a result of
configuration errors. In the event of failures verify the following RF data fill parameters:
TCC Wait Timer
The amount of time the RNC waits for a traffic channel complete (TCC) message from the AT. A Traffic
Channel Assignment message is typically sent by the RNC to the AT as part of a reverse link soft
handoff. If the TCC is not received in time the connection will be closed.
Range
Unit
Recommended
1000 30000
1/1000
seconds
5000
IS856Neighbor
Defines the neighbors surrounding the sector an AT may be connected to. Inconsistencies in the
neighborlist may result in missed handoffs. Insure that reciprocal neighbors are not missing in adjacent
sites neighborlists.
9.If problems persist perform a FTAP and RTAP tests.
184
10.13.1.3 OMs
The Reverse link physical layer throughput is estimated over a 3 second sliding window and is
available from the per sector RL throughput OM perfRLThruput. The per sector reverse link physical
layer throughput is available as a per sector OM.
Method of Collection:
1. The reverse link physical layer throughput can also be collected using the celldm logs.
2. Collect the Reverse Link histogram from the celldm logs show celldm rlthruput <sector>.
NORTEL-XX#show celldm rlthruput alpha
RATE: 9k6 19k2 38k4 76k8 153k6
BYTES: 139648 533632 428928 38656
512
RL data throughput for sector alpha: 26112 bps
(1
# ofAT
AggregatRLSectorThroughput =
TXActiveThroughput )
TotalNumberofATs
185
2. Collect the Reverse Link histogram from the celldm logs show celldm rlthruput <sector>.
NORTEL-XX#show celldm rlthruput alpha
RATE: 9k6 19k2 38k4 76k8 153k6
BYTES: 139648 533632 428928 38656
512
RL data throughput for sector alpha: 26112 bps
10.14.1.3 OMs
There are currently no per user throughput OMs that can be used for determining the achievable data
rate on the forward link. Achievable FL throughput can be estimated from the celldm logs show celldm
flthruput <sector>
Method of Collection:
1. Telnet into the DOM and enable the FTC Scheduler. It is necessary to enter configuration mode to
enable FTC scheduler NORTEL-03> config NORTEL-03 celldm scheduler ftc on
2. Log out of the DOM NORTEL-03> exit.
3. Repeat this process for all DOMs
4. The Average FTC Physical Layer throughput loading = perfFLThruput / 1000 (Unit: kbps)
5. Note: The throughput value given will only be for the last reported sampling window
186
6. A histogram of slot utilization and throughput can be generated using CLI show celldm flthruput
<sector>
Sample: flthruput histogram from celldm
NORTEL-XX#show celldm flthruput alpha
RATE: 38k4 76k8 153k6 307k2 614k4 921k6 1228k8
1843k2 2457k6
SLOTS: 0 0 0 0 0 0 0
0 159235
FL data throughput for sector alpha: 534749 bps
WeightedAverageForwardRate =
( RatexSlots)
AllRates
Slots
AllRates
187
2.If the average application layer throughput is low (e.g. < 1.5Mbps) where RF conditions are ideal (e.g. C/I
is in the 10-12dB range) perform a sanity check by performing a DOS FTP of the file. If the throughput is
good (i.e. 1.5Mbps or greater) the check to insure that all compression algorithms that may be selectable
in the collection tool are disabled.
3.Verify from the Sector Parameters Message in the AT logs that the correct Pilot PN, Band Class,
Channel Number and Subnet Mask for the sector are correct.
4.Us DrTCP to insure that the TCP RX Window size is = 64240 (TCP Window Size in Windows Registry)
and the MTU size = 1500Bytes (ProtocolMTU in Windows registry). Refer to Appendix E for more tips
on TCP IP configuration.
5.If the throughput from the collection tool and DOS FTP are low verify that there are no other users on the
site. In Optis or XCAL view the User Size in the EVDO Signal Graph or from the Quick Config
message in the Layer 3 messaging available in RFO.
Example: User Size in Optis-M EVDO Signal Graph
6. If the User Size is equal to 1 verify that the DRC request rate is between 1.2288 and 2.457Mbps.
7. If the DRC request rate is low verify from the AT logs that the Fwd Physical rate to Reverse data rate
ratio is approximately 40:1.
188
8. In the event that the Fwd Physical to Rvs Data rate is much greater than ~65:1 perform the following
checks:
o
Is the RL SHO factor > 2? Show 1xEVDO connection <UATI> counters <all>
Verify that there are no failures due to resource allocation failures? show 1xevdo connection
<UATI> counters <all>
Verify the RF datafill for RL Max Rate Table, and the Reverse Transition probabilities table See [8]
for Nortel recommended RF datafill values and methods for updating datafill tables.
Perform FTAP and RTAP tests to verify the physical connection between the AT and RNC.
Example: Fwd Physical to Reverse Data rate ratio
189
Dropped TCP
Packets
TCP Packet
Retransmission
12. An inspection of the RLP statistics in the AT logs will indicate NAK timeout (NAK timeout occurs
when an RLP packet is retransmitted in error or not received by the AT within the .5msec wait time)
rate. If the NAK timeout rate exceeds 5 %, the SDU Resequencing Wait Timer may be too short.
13. To adjust the SDU Resequence Wait Timer use the following CLI command: NORTEL-XX#
(config)#1xevdo sdu-resequencing-wait <period>
Example: RLP NAK Timeout Rate
190
Formula: ((RX Total Bytes x 8) x 1.15)/1024 = RX Total packets (SQRT (NAK Timeouts / RX
Total Packets)) x100 = NAK Timeout Rate
14. Perform FTAP and RTAP tests to validate the integrity of the physical link between the RNC and
the AT [17].
10.15.1.3 OMs
The Reverse link physical layer throughput is estimated over a 3 second sliding window and is
available from the per sector RL throughput OM perfRLThruput. The per user throughput can be
estimated as follows:
Method of Collection:
1. Collect the Reverse Link histogram from the celldm logs show celldm rlthruput <sector>.
NORTEL-XX#show celldm rlthruput alpha
RATE: 9k6 19k2 38k4 76k8 153k6
BYTES: 139648 533632 428928 38656
512
RL data throughput for sector alpha: 26112 bps
191
2. To determine the maximum RL throughput at the physical layer in RFO, Statistics View Data
Statistics Reverse Rate Avg. Data Rate.
Example: RFO RL Physical Layer Statistics
TotalBytes Re ceived =
( Bytes * 8)
AllRates
TotatTransmissionTime =
(( Bytes * 8) / Rate)
AllRates
192
6. An alternative to using Ping to measure packet latency is to collect TCP trace logs with Ethereal or
Windump.
7. Since XCAL and Invex3G-PC uses the PCs protocol stack for data collection, Ethereal or Windump
can be used concurrently to capture TCP trace data. In the event XCAL or Invex3G-PC are not
available for use, the engineer can open an Ethereal or Windump log and subsequently perform a
DOS FTP download or upload to the test server.
8. After the Ping test and trace logs are collected stop logs and use TCPTrace or the playback function
in XCAL to determine the average packet latency.
10.16.1.3 OMs
There are no packet latency OMs available
Nortel Networks Confidential and Proprietary
193
samples:
min:
max:
avg:
stdev:
24962
10.0 ms
200.3 ms
5.8 ms
12.1 ms
RTT
RTT
RTT
RTT
RTT
samples:
min:
max:
avg:
stdev:
2
130.2
140.2
135.2
0.0
ms
ms
ms
ms
194
7. Make note that the EVDO to 1xRTT handoff is actually an EVDO connection drop followed by an
origination on 1xRTT. The FTP download application started on EVDO should continue on 1xRTT
from where it stopped on the EVDO network.
10.17.1.3 OMs
There are no current OMs to show EVDO to 1xRTT handoff delay.
195
10.18.1.3 OMs
N/A
196
Is the network in a
commercial or launched
state?
Perform Stationary
troubleshooting tests
Page 3 (#2)
Perform mobility
troubleshooting tests
Page 4 (#3)
Y
N
197
RF Datafill
-Verify RF Datafill is in accordance
with Nortel recommended values or
what has been mutually agreed to with
Y
NE operational status is Active?
NORTEL-XX# show node
Y
All modules have Active status?
NORTEL-XX# show modules
System Integrity
Verification Checklist
N
Go to troubleshooting sections of
NTP 411-2133-949
Y
Administrative & operational status of all
interfaces is Active? (PPP, LCP,
Ethernet) NORTEL-XX# show interfaces
Y
Can Ping the following from DOM /
AT?
- Node IP of DO-RNC
- IP of Agg Router interfaces for
backhaul
- IP of DO -EMS
Y
DO-RNC must be able to Ping:
- Node IP of all connected DOMs
- IP address of DOM PPP interfaces
- IP address of DO-EMS
- All PDSNs
Y
Is ABIS Peer status = Connected for all
DOMs? NORTEL-XX# show abis peer
Y
Is GPS present and locked for DOMs?
NORTEL-XX# show GPS health
Y
Is SNTP or system time correct / present?
NORTEL-XX# show SNTP Time
Y
Is GRE A10 Resequencing enabled for all
BIO modules?
NORTEL-XX (config) # fastpath a10 reseq 1
N
TAP Tests (Tap Users Guide)
(FTAP / RTAP) Passed
198
Stationary RF Troubleshooting
Stationary
Test Start #1
Go to site or location where
problems are and perform FTP
DL using data collection tool.
Verify that there are no
antenna feeder issues
Y
Go to Page
5 (#3)
Are Results as
expected?
-Try a different AT
- Verify AT COM ports
are configured correctly?
N
Post Process AT logs
with RFO
Go to Page
5 (#3)
N
Is Fwd. Physical Rate to Rvs. Data
rate ratio ~ 40:1 but not > 63:1?
N
- Are there excessive RL
NAKS/Timeouts in AT logs?
Determine resource
issue and resolve
199
Mobile RF Troubleshooting
Start
Mobility
Tests #2
END
Collect AT & RNC logs
Refer to the EVDO Optimization
Guide, Chapter 10 for tips on
recognizing and addressing failures.
Y
- From NEWTUN identify DOMs
with high fail rates.
- From RFO and NEWTUN output,
categorize failures (missing neighbor,
border handoff, long tune away,
Session setup fail, poor coverage, etc).
N
From AT and RNC logs, are
there excessive drops and access
failures (i.e. >10%)?
N
From AT and RNC logs are there
areas with poor coverage (C/I < 2dB, PER >1%, > 3 ASet Pilots)?
N
Verify data fill for RL Max Rate and
RL trans Probabilities table
1xEVDO RF Dataill Guide
Identify localized
areas with poor FL
tput
Go to Page 3 #1
N
RL SHO failure due to resource allocation fails?
show 1xevdo connection<uati> counters<all>
Optimize coverage to
introduce dominant server
N
Go to areas where ratio is compromised
and verify max uplink rate via FTP PUT.
Are results as
expected?
Go to Page
5 (#3)
200
TCP Trace
analysis - #3
Y
From CellDM (DOM) logs for RTC
(rtclogmask 0xFFFFFFFF) are
GRE / Abis packets arriving out of
order?
N
Use Ethereal or Win Dump to
collect TCP Trace logs
Y
Is RTT excessive between any
nodes (e.g. >>300ms)?
Y
Verify TCP/IP Settings See
EVDO Opto Guide Appendix E
Y
Verify which node or interface is
problematic and troubleshoot.
See NTP411-2133-949
201
Telnet to <DOM node> (i.e., specific DOM IP address) via either Console Port (Dial up connection) or
Backhaul if available.
2.
NORTEL-03>en
NORTEL-03#show interface
Name
Connection IP Address
In Octets
Out Octets Operational
----------------------------------------------------------------------------------------------------ethernet1/3/1
10.10.32.156/20
0
0
Down
node1/3/1
47.135.203.191/32
0
0
Up
ppp1/3/1
0.0.0.0/0
0
0
Down
ppp1/3/2
10.11.2.2/30
452184005 340
Up
ppp1/3/3
10.11.3.2/30
440974939 51272523 Up
ppp1/3/4
0.0.0.0/0
0
0
Down
t11/3/1
0.0.0.0/0
0
0
Down
t11/3/2
0.0.0.0/0
452184005 330
Up
t11/3/3
0.0.0.0/0
440974939 58087164
Up
t11/3/4
0.0.0.0/0
0
0
Down
-----------------------------------------------------------------------------------------------------
This displays all interface status from DOM. For instance, the DOM Ethernet port is not used, the DOM
node is up, 2 physical links (ppp1/3/2 and ppp1/1/3/3) between DOM and router are up, 2 T1 ports are up,
and 2 T1 ports are not used. This information indicates that the interfaces are up and running.
This only indicates that the DOM side interfaces are up but not necessarily all interfaces running
properly. For example, the show interface command may indicate that the TWO T1s are up, but in fact one
of the T1s may not be carrying any traffic at all. We have to ensure that IP route table is properly filled,
and also check if the displayed values of In Octets Out Octets are incrementing for all the T1s. (In Octets
and Out Octets information is displayed using the show interface command in the enable mode).
202
NORTEL-03>show ip route
IP ROUTING TABLE
---------------------------------------------------------------------------Destination
Flags
Gateway
Owner Interface
---------------------------------------------------------------------------0.0.0.0/0
10.23.1.1
S
ppp1/0/1
10.23.2.1
S
ppp1/0/2
10.25.3.1
S
ppp1/0/3
10.25.4.1
S
ppp1/0/4
10.23.1.0/30
L
10.23.1.2
L
ppp1/0/1
10.23.2.0/30
L
10.23.2.2
L
ppp1/0/2
10.25.3.0/30
L
10.25.3.2
L
ppp1/0/3
10.25.4.0/30
L
10.25.4.2
L
ppp1/0/4
47.135.201.245/32 L
47.135.201.245 L
node1/0/1
3.
Under DOM prompt, type show abis peer in the enable mode as the following example:
NORTEL-03>en
NORTEL-03#show abis peer
--------------------------------------------------------------------PeerAddress Port State RxMsgs Rxhellos
47.135.203.161 5604 UP 18189 8163
---------------------------------------------------------------------This displays the link status between the DOM and the ANC.
4.
Under DOM prompt, type show rrc state in the enable mode as the following example:
NORTEL-03>en
NORTEL-03# show rrc state
----------------------------------------------------------------------------MODEM 0: ADMIN STATE: OPERATING OPERATION STATE: UP
=======================================
SECTOR 16:
ADMIN STATE: UP
OPERATION STATE: UP
=======================================
SECTOR 17:
ADMIN STATE: UP
OPERATION STATE: UP
=======================================
SECTOR 18:
ADMIN STATE: UP
OPERATION STATE: UP
------------------------------------------------------------------------------This displays sector status of the DOM. For instance, each sector under this DOM is up and running.
SECTOR 16 = Sector Alpha, SECTOR 17 = Sector Beta, and SECTOR 18 = Sector Gamma.
We can also use show sector-element command in the enable mode to display similar information:
NORTEL-03#show sector-element
Name
Carrier Sector CAI Channel PN Power Admin
------------------------------------------------------------------------------element1 carrier1 sector1 IS-856 1125 116 25 dBm UP
element2 carrier1 sector2 IS-856 1125 120 25 dBm UP
element3 carrier1 sector3 IS-856 1125 124 25 dBm UP
Oper
UP
UP
UP
203
5.
NORTEL-03>en
NORTEL-03#show traffic-channel
BTS BSC
Flow Pri Dn Pkts Up Pkts
------------------------------------------------0x0010 0x000034d6 CCH 128 124
26
0x0011 0x000034d7 CCH 128 183
76
0x0012 0x000034d8 CCH 128 105
2
This information provides how many traffic channels are established between DOM and ANC. The
provided Example displays only CCH (Control Channel) since there is no active traffic channel at the time
the command show traffic-channel is used.
6. From the DOM prompt, ping ANC, PDSN (i.e. ISP), and FTP servers as the following example:
-----------------------------------------------------------------------------------------NORTEL-03>ping 47.135.203.161
Sending 5, 100-byte ICMP Echos to 47.135.203.161, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/6/10 ms
NORTEL-03>ping 47.135.203.234
Sending 5, 100-byte ICMP Echos to 47.135.203.234, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/6/10 ms
NORTEL-03>ping 47.135.203.18
Sending 5, 100-byte ICMP Echos to 47.135.203.18, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/8/10
------------------------------------------------------------------------------------------Repeat these steps for each DOM in DO network.
FTAP and RTAP Tests:
To insure the integrity of the data link between the AT and the RNC, FTAP and RTAP tests should be
performed. These tests are similar to Markov calls in CDMA voice as it sends simulated data packets from
the RNC to the AT to test the link between the two.
The following document should be used as a guideline for FTAP and RTAP tests:
D:\Profiles\mwoodley
My Documents\CDMA
204
Status:
ALPHA (X)
Datafill
Pilot PN
Height
Orientation
Mech. DT
Antenna
Neighbor List
Datafill
BETA (Y)
Observed
Observed
Datafill
Pilot PN
Height
Orientation
Mech. DT
Antenna
Neighbor List
Datafill
GAMMA (Z)
Observed
Observed
Datafill
Pilot PN
Height
Orientation
Mech. DT
Antenna
Neighbor List
Datafill
Call Initiation/Term.
Call Initiation/Term.
Call Initiation/Term.
Coverage/HO
Coverage/HO
Coverage/HO
Observed
Observed
205
206
207
On a prior tune away, the mobile saw PN 324 with Ec/Io just 2dB below PN 320. As a result,
on the next tune away the AT tuned to PN 324 where, at that time, the Ec/Io was very poor.
Resolutions / Workarounds:
Insure neighbor reciprocity to reduce chances that mobile will idle handoff to a new PN
but cannot get back to the original if it needs to.
Avoid having neighbor lists with sites 2 tiers away from the reference site.
If a long tune away is the result of RF optimization needs on the 1x network, permission
from the customer will be required before going forward with any changes to the
underlying 1x network.
208
Required Tools:
EVDO RFO
CDMA RFO
XCAP-DO
Problem Description:
Connection Deny received from RNC if AT sends Route Update while RNC is in an Await
TCC state. This issue is more commonly referred to as an RNC State mismatch.
Symptom:
Typically occurs after the first access Attempt (Connection Request) following an AF in the
AT logs. This occurs when the mobile does not receive the TCA due to tune away or because
it was in poor coverage. The state mismatch will occur when a Route Update (Connection
Request) is received when the RNC is in a CSMAwaitTCCState rather than the
CSMDormantState. When the mismatch occurs the system will send a Connection Close with
Reason Code 2 which equates to a Connection Deny. The Connection Deny means that
system resources are currently unavailable.
Resolution:
In prior releases there was a 5 sec timer associated with the tear down of the MAC channel. In
a recent patch to release 2.1, this timer has been removed to insure that Connection Closes
happen more quickly.
In addition to removing the MAC channel timer to improve connection close performance, the
2.1 release also has been improved such that if a new Connection Request is received while a
previous Connection is not closed out, the system will release the prior connection and set up
the new one without pegging an Access Failure.
Problem Description:
Cannot exceed 1.5MBps on FL
Symptoms:
In good RF conditions where C/I >=10dB forward link throughput does not exceed 1.5MBps.
Resolution or Workaround:
Insure that backhaul T1s are bundled and not separate. If T1s are not bundled,
throughput will be limited to 1.544MBps.
Log into the backhaul aggregate router and insure that there are no Path line-code
errors on configured T1s.
Insure that the TCP window size and MTU settings for data collection tool are set to
the optimum configuration as discussed in the throughput troubleshooting section
below:
209
Problem Definition:
Dropped connection during connection close This condition is more commonly referred
to as All or Nothing.(i.e. the connection is granted all or the connection is denied
nothing.
Symptoms:
In some cases after an FTP session has completed and the system is in the process of
closing the connection, the AT may send a handoff Route Update prior to receiving the
Connection Close (Race Condition). In these situations the system may send the TCA for
the Route Update prior to the Connection Close. If the AT receives the Connection Close
prior to receiving the TCA it may close out the connection without sending a TCC. As
such, the system closes the connection with an error instead of a normal connection. This
pegs as a drop, with reason code 2.
Resolution:
The system should ignore Route Updates received after a Connection Close has been sent
thus avoiding this problem. The patch (.21) to 2.1 that speeds up the Connection Close
will also help prevent occurrences of this symptom.
210