MOP for Airspan SC

eNB alarm Troubleshooting from Field Actionable

September, 2015
 1. eNB Alarm trouble shooting Introduction.

Following document will provide all Netspan eNB alarm information & Trouble shooting guide lines from field point of view.
 Scenarios in Which the Issues should be raised to Airspan L2 Team.

1. If any issue Occurs Check if there is TT generated against that alarm & if not then kindly get the TT generated
from the surveillance team.
2. Follow the below MOP against the respective alarm & do L1 troubleshooting.
3. After L1 troubleshooting & your analysis assign the issue to respective teams.
a. If issue related to IP escalate the same to Circle / NOC IP teams.
b. If issue related to SMC then escalate the same to Airspan NOC team with TT Number
4. For checking Site & Alarm Status kindly use the Logins provided to Circle teams.

2. Alarms SC eNB

Node Lost Comms (SNMP Down, Services Affected )

Description

The Netspan server is unable to communicate with the SNMP agent on the node.
This alarm makes the EMS management impractical (for examplethe configuration is suspended, events and alarms are not logged, or the statistics
is not logged). However, if configured, the node operates appropriately.

Impact

Services will be suspended, No Ping possible, Alarm is common for ISC & OSC.
Recommended field Actions
a. Step1: Check CSS working status and power supply on site.
b. Step2: Check the network Physical connections between CSS to L2 Switch and L2 Switch to SMC e-NB. (Loose
connection check )
c. Step3: Replace optical fibre patch cord between CSS to L2 Switch and L2 Switch to SMC e-NB one by one and
check alarm status.(Cable fault check )
d. Step4: Clean all 4 SFPs used at CSS, L2 Switch and SMC e-NB. (Dust /Moisture check )
e. Step5: Loop L2 switch port connected to SMC e-NB with single patch cord and restore Connectivity.( Port
malfunction check )
f. Step6: If there is high temperature / power fluctuation, need to check such cases as SMC may be rebooting due
to these issue.
g. Step7: Loss of communication is typically due to a temporary network condition, can be identified if multiple
nodes experience alarm from same time. (Check from NOC if all the nodes are down at same time).
h. Step8: If yes then escalate the issue to IP Team (Circle/NOC) along with Proper TT.
i. Step9: If connections, SFP, Patch chord, and SFP ports are fine Give local login of e-NB to NOC team to run ping
check, Packet loss, and Trace-route and IP configuration.
j. Step3: Ensure no external agents like water or dust inside the e-NB causing malfunction.
Where SNMP response is getting failed & Ping is OK in such cases also need to Involve IP Team. Kindly Escalate Such Cases to IP Circle/NOC along with TT & Snaps
Scenario1:
Few Nodes from Bunch of SMC are Down: (Perform Steps1 to 5 & then escalate the issue to IP CIRCLE /NOC Team)

Scenario2:
If all the Nodes are in Comms Fail (After Checking Power & Backhaul Fibre Connectivity Escalate the issue to IP Team)
 Channel Out of Service (OOS) (Services Affected)

Description

Channel OOS can be a result of the management configuration, or an underlying hardware or software issue and other service impacting alarms.

Impact

Services will be suspended.

Recommended Field Actions

a. Step1: Select the state & Control TAB, & verify that cell is not manually set to the Out-of-Service state.

b. Step2: If this alarm is accompanied by another alarm like MME Connectivity Lost, RF Transmitter Off, LOW SNR, Clock Issue (Node Holdover,
Node Loss of Sync, Node PTP Degraded Stream) follow the recommended actions to clear those alarms.

GPS Signal Unacceptable

Description

Impact

Services will be suspended.

Recommended Field Actions

a. If this Condition Persist, Check GPS Antenna & Cabling.

b. Check the Voltage at the Port where the GPS is connected it should be +-5V

GPS SNR LOW (Only in OSC SC Cases)

Description

Impact

Minor.

Recommended Field Actions

a. Check GPS Antenna SKY View & GPS Cabling.

b. Check the Voltage at the Port where the GPS is connected it should be +-5V
 Node in Holdover

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: This Alarm Occurs with GPS SNR Low & Node PTP Grand Master Lost, Rectify both the alarms to clear this alarm.

Wi-Fi Board Connectivity

Description

Impact

Critical for WIFI Operation & Minor for LTE Operation.

Recommended Field Actions

a. Step1: Check WIFI Status from WIFI NMS and make sure it is active & has correct SW version.
b. Step2: Reset the eNB
c. Step3: If required, After Troubleshooting Confirmation from NOC then only replace the HW
 Wi-Fi Board Initialization

Description

Impact

Critical for WIFI Operation & Minor for LTE Operation

Recommended Field Actions

a. Step1: Check WIFI Status from WIFI NMS and make sure it is active & has correct SW version.
b. Step2: Reset the eNB
c. Step3: If required, After Troubleshooting Confirmation from NOC then only replace the HW

MME Connectivity Lost

Description
Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Verify the IP Connectivity to the Specified MME (Ping & Trace route from eNB to MME)
b. Step2: Check whether the Specified MME is Operational.
c. Step3: Check for eNB Configurations in MME & Vice versa.
d. Step4: Check for eNB ID from ODD
 Node Internal clock

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Reboot the Node, If Still Issue Persist After Confirmation from NOC then only replace the HW

Node Loss Sync

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: This Alarm Occurs with GPS SNR Low & Node PTP Grand Master Lost, Rectify both the alarms to clear this alarm.
 Node PTP Grand master Lost

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Check the Connectivity with PTP Grand Master from the Status bar in Netspan.
b. Step2: If not connected Kindly Escalate the Issue to IP Team/NOC Team.
How the PTP Alarms Occur?
1. First the Node goes in Holdover (Not SA)
2. It waits for 2.5min for Clock i.e. it is the Holdover time.
3. If the Holdover Time exceeds the Node goes in Holdover Expired Mode.
4. After which the Node goes in loss of Sync.
5. After Loss of Sync Your services will be affected & Channel Out of service Alarm will occur on Node.

To Rectify & Troubleshoot the PTP Related Alarms refer the below Mention Graphical presentation & Scenarios.

Note: To understand why Jitter Occurs in Network & How they Impact to our SC & occurrence of PTP
Alarms refer the Word Doc Attached below.
 Microsoft Word
Document

Scenario1. If all the SMC are having the Node PTP Grand Master Lost issue (Kindly take UP the issues with Circle / NOC IP Team)

Scenario2. If only Few SMC out of bunch of SMC having the Node PTP Grand Master Lost issue (Kindly Inform Airspan NOC Team)

Scenario3. If all the SMC are having the Node PTP Degraded Stream But Clock in acquiring mode. (Kindly take UP the issues with Circle / NOC IP Team)

Scenario4. If only Few SMC out of bunch of SMC having the Node PTP Degraded Stream but Clock in Acquiring mode. (Kindly Inform Airspan NOC Team)
Note: Node PTP Degraded Stream Alarm Causes is Mentioned in below snaps.
 Node Self-test failed

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Reboot the Node, If Still Issue Persist After Confirmation from NOC Airspan Team Replace the HW.
 SGW Connectivity loss

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Verify the IP Connectivity to the Specified SGW (Ping & Trace route from eNB to SGW)
b. Step2: Check whether the Specified SGW is Operational
c. Step3: Check for eNB Configurations in SGW & Vice versa

MCE Connectivity Lost

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Check MCE Status & IP Connectivity to MCE.

eSON Server Connection lost

Description

Impact

Major Service Affecting.

Recommended Field Actions

a. Step1: Check IP Connectivity to Specified ESON Server.

b. Step2: Check whether the specified ESON server is operational.
 Emergency Call reset delay

Auto PCI Allocation Failed

 Auto RSI Change

Command Failed
 MBSFN SF configuration conflict

Node Temperature unacceptable

 Node Voltage unacceptable

PnP Configuration mismatch

 3. NOC Escalation Matrix (After L1 Troubleshooting From Circle as per MOP)
ESCALATION MATRIX
Zone 2nd Escalation Leads 3rd Escalation Leads 4th Escalation Leads
AIRSPAN SC L2 Support FM Desk & troubleshooting for existing TT for RAN, CSS & MW
North Zone North, East & West : 022-413 88114 / 022-413 88680 LTE RAN Access : : RF Access Head
East Zone JIO.L2SmallcellAS@ril.com Raghavendra Parigi -7718877473 Nusrat Mohammad
West Zone 8454040463/
South Zone 7021246741
IP L2 / SWITCHES Support FM Desk & troubleshooting for existing TT for RAN, CSS & MW
North, East & West : 022 413 88138 / 139 IP MPLS ACCESS : : RF Access Head
North Zone
Jio.IPAccessL2Support@RIL.COM Harsharan Singh (8433708315) Nusrat Mohammad
East Zone Saurabh Goyal (9987962659) 8454040463/
West Zone Sanjay Ahuja (8828031025) 7021246741
044 42430037
South Zone RJILTN.DRNOIPL2@ril.com / RJILTN.DRNOCIPFM@ril.com / Rajesh Appasamy - 9840760544
RJILTN.DRNOCShiftHeadTeam@ril.com
UBR Support FM Desk & troubleshooting for existing TT for RAN, CSS & MW
North, East & West : 022 4475003 / 41388538 UBR / Microwave Access : : RF Access Head
JIO.NOCFMMW@RIL.COM Nitin K Jadhav-8452876175 Nusrat Mohammad
PAN India Support
8454040463/
7021246741

Contact Details of the Airspan NOC Team Contact Details of the RJIO FM NOC Team
S.No. Contact S.No. Contact
1 41388114 1 41388022
2 41388719 2 41388023
3 41388680 3 41388024
 Airspan Login ID & Password for Circle Teams.

Sr
Circle Server Host Name IPv4 IPV6northbound Username Password
No
1 Delhi
Delhi DL-NOID-NS-01 10.70.32.229 2405:0200:330:769::13
2 Rajasthan
3 Uttar Pradesh (East)
4 Uttar Pradesh (West) Lucknow UP-LCKN-NS-01 10.70.48.198 2405:0200:080a:769::11
5 Uttarakhand
6 Punjab
7 Haryana Ambala HR-AMBL-NS-01 10.70.104.134 2405:0200:0815:769::11
8 Himachal Pradesh
9 Jammu
Srinagar JK-SRGR-NS-01 10.70.90.196 2405:0200:0811:769::11
10 Kashmir
11 Kolkata
12 West Bengal
13 Assam
14 North East Kolkata KO-KLKT-NS-01 10.70.66.101 2405:200:806:769::11
rjilview jio@12345
15 Bihar
16 Jharkhand
17 Orissa
18 Mumbai Mumbai NS-MU-HV1 10.192.23.100 2405:0200:0802:769::11
19 Maharashtra
20 Goa
Nagpur MH-NGPR-NS-01 10.70.56.196 2405:0200:080B:769:0:0:0:11
21 Madhya Pradesh
22 Chhattisgarh
23 Gujarat Gujarat GJ-AMBD-NS-01 10.70.18.4 2405:0200:0809:769:0:0:0:11
24 Andhra Pradesh
Hyderabad AP-HDBD-NS-01 10.70.24.196 2405:0200:0807:769:0:0:0:11
25 Telangana
26 Karnataka Bangalore KA-BGLR-NS-01 10.70.98.228 2405:0200:0808:769:0:0:0:11
27 Kerala
Chennai TN-CHNN-NS-01 10.70.80.100 2405:0200:080c:769::11
28 Tamil Nadu
 Thank you