Documente Academic
Documente Profesional
Documente Cultură
Keep in mind images in this presentation are illustrative images - it just show how we are monitoring that interfaces. It doesnt show, what is between of point A and Z of that links (switches, routers, backbone network etc..)
Links troubleshoting
#>/ x perf
qmpa> perf Sample interval=30 Cur load = 1804 bps Max load = 83083 bps at UCT 05.12.2005 00:13:00 PDUs/ExPDUs FISUs Ticks InputConnAlarm SigInputAlarm SigInputError ProbeAlarm LinkStatistics Suppressed Unknown qmpa> = = = = = = = = = = 31650728 0 69153361 2 0 0 0 41492022 0 0
ATM/STM-1
DX-VAM
probe
NodeB
ATM/STM-1
DX-VAM
probe
ATM/STM-1
DX-VAM
probe
Usually four linksets VPI/VCI depend on sequence of RNC on site: 1. 2. 3. 4. RNC1 RNC2 RNC3 R1_LAPRN1-LAPSW1_01A fwd direction first RX pair; VPI/VCI 1,5000 1,5032 1,5064 R1_LAPRN1-LAPSW1_01A bwd direction first RX pair; VPI/VCI 1,5000 1,5032 1,5064 R1_LAPRN1-LAPSW1_01B fwd direction first RX pair; VPI/VCI 1,5001 1,5033 1,5065 R1_LAPRN1-LAPSW1_01B bwd direction first RX pair; VPI/VCI 1,5001 1,5033 1,5065 R1_LAPRN1-LAPSW1_02A fwd direction second RX pair; VPI/VCI 1,5000 1,5032 1,5064 R1_LAPRN1-LAPSW1_02A bwd direction second RX pair; VPI/VCI 1,5000 1,5032 1,5064 R1_LAPRN1-LAPSW1_02B fwd direction second RX pair; VPI/VCI 1,5001 1,5033 1,5065 R1_LAPRN1-LAPSW1_02B bwd direction second RX pair; VPI/VCI 1,5001 1,5033 1,5065
10
R5_PVALLAR_ICS02:~# x chnst Channel 268447058/0: Prot=130 Load= 76.263 kbps State=No Alarm SIM=1 LIM="0" Port="0" R5_PVRR2-PVRMG1_01A Channel 268447058/1: Prot=130 Load= 86.015 kbps State=No Alarm SIM=1 LIM="0" Port="0" R5_PVRR2-PVRMG1_01A Channel 268447060/0: Prot=130 Load= 0.000 bps State=No Alarm SIM=1 LIM="0" Port="0" R5_PVRR2-PVRMG1_01B Channel 268447060/1: Prot=130 Load= 0.000 bps State=No Alarm SIM=1 LIM="0" Port="0" R5_PVRR2-PVRMG1_01B Channel 268447062/0: Prot=130 Load= 0.000 bps State=No Alarm SIM=1 LIM="0" Port="1" R5_PVRR2-PVRMG1_02A Channel 268447062/1: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="1" R5_PVRR2-PVRMG1_02A Channel 268447064/0: Prot=130 Load= 77.055 kbps State=No Alarm SIM=1 LIM="0" Port="1" R5_PVRR2-PVRMG1_02B Channel 268447064/1: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="1" R5_PVRR2-PVRMG1_02B
eINPUT_CONNECTOR_ALARM....NoSignal in 99 % of cases no signal on input connector in this case, 2nd pair port 1, connector 4th (the last one on the bottom of STM-1 card) Possible reasons: 1. wrong Telcel position check with core people whether they change position 2. wrong cable, connectors, optical splitter in DX-VAM check on-site 3. weak signal check on-site 4. wrong receiver on HDLU STM-1 card check on-site (try to swap that cable with working cable from another connector)
11
[root@R5_BANDERA_ICS01 c110606xjvr006]# x chnst Channel 268438162/0: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="0" R5_BANR1-BANSW3_01A Channel 268438162/1: Prot=130 Load=242.330 kbps State=No Alarm SIM=1 LIM="0" Port="0" R5_BANR1-BANSW3_01A Channel 268438164/0: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="0" R5_BANR1-BANSW3_01B Channel 268438164/1: Prot=130 Load= 0.000 bps State=No Alarm SIM=1 LIM="0" Port="0" R5_BANR1-BANSW3_01B Channel 268438166/0: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="1" R5_BANR1-BANSW3_02A Channel 268438166/1: Prot=130 Load= 0.000 bps State=No Alarm SIM=1 LIM="0" Port="1" R5_BANR1-BANSW3_02A Channel 268438168/0: Prot=130 Load= 0.000 bps State= eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) SIM=1 LIM="0" Port="1" R5_BANR1-BANSW3_02B Channel 268438168/1: Prot=130 Load=240.323 kbps State=No Alarm SIM=1 LIM="0" Port="1" R5_BANR1-BANSW3_02B
no signal on input connector in this case: 1st pair port 0, connector 1st 2nd pair port 1, connector 3rd
Possible reasons: 1. wrong Telcel position check with core people whether they change position 2. wrong cable, connectors, optical splitter in DX-VAM check on-site 3. weak signal check on-site 4. wrong receiver on HDLU STM-1 card check on-site (try to swap that cable with working cable from another connector)
12
no ALARM on all input connectors but also no traffic on all conectors Check: check whether VPI/VCI are correctly configured according to available information check if there is a traffic (Telcel people)
13
IuPS
14
Fast Ethernet
2x TAP
probe
15
16
KONTRON BOARD:
Usually two linksets: 1. R1_LAPRN1-R1LMSGSN_01 fwd direction first link; R1_LAPRN1-R1LMSGSN_01 bwd direction first link; 2. R1_LAPRN1-R1LMSGSN_02 fwd direction second link; R1_LAPRN1-R1LMSGSN_02 bwd direction second link;
eth3 193.88.224.2 eth4 193.88.224.3
17
no signal on input connector in this case: 2nd pair port 1, connector 3rd (ETH4) Usually its mean, that the link is down on that ETH interface ETH4 in this case Check: use mii-tool for information about ETH interfaces (test whether interface is up/down) try to restart /etc/init.d/qapm stop; /etc/init.d/qapm start check TAP, cables, connectors, pozitions on-site (whether monitored links are connected and are up; check the connection from TAP -> CT9)
18
19
20
STP
Fast Ethernet
TAPs
Extreme SW
1x ETH probe
21
BSC
Fast Ethernet
MSC
2x TAPs
Extreme SW
1x ETH probe
22
Note: it seems, that when you can see that there is exactly 100.000 kbps traffic mean, that there is a traffic according to IP filter (on that port), but there is no traffic with filtered OPC/DPC.
23
there is a traffic on other 2315 port check whether configuration of the probe is according to Telcel information (OPC/DPC sim filter settings ). When its OK, check the settings/links with Telcel. there is no traffic on other ports (for example 3103 port):
find in cabling table current position for example: 3103-> FE TAP1, port 15+16-> Extreme1 port 29,30,31,32
24
If there ports are UP and there is a traffic, we need to do more investigation aggregating only this ports on probe, run tcpdump to capture data. Analyze the traffic for IP/OPC/DPC. Check this IP/OPC/DPC with Telcel core people to know, whether that is a correct traffic on this link or not if the traffic is OK, update configuration of the probe according to obtained information. If ports are DOWN check with Telcel, whether the links should have a traffic. Check TAP/cables/connection on-site.
25
26
Gn,Gp
26
27
SGSN
Gigabit ETH
GGSN
TAPs
Extreme SW
4x ETH probe
Gn typical problems
Example 1: Gn connected to LU5 card (special card for capturing big amount of traffic)
R9_MORALES_GGN02:~# x chnst Channel 268438720/0: Prot=233 Load=665.524 Channel 268438720/1: Prot=233 Load=472.537 Channel 268438750/0: Prot=233 Load=677.230 Channel 268438750/1: Prot=233 Load=507.436 kbps State=No Alarm kbps State=No Alarm kbps State=No Alarm kbps State=No Alarm SIM=1 LIM="0" Port="hpe0.0,hpe0.1" SIM=1 LIM="0" Port="hpe0.0,hpe0.1" SIM=1 LIM="0" Port="hpe0.2,hpe0.3" SIM=1 LIM="0" Port="hpe0.2,hpe0.3" R9_R9MRSGS_Gn_01 R9_R9MRSGS_Gn_01 R9_R9MRSGS_Gn_02 R9_R9MRSGS_Gn_02
28
Port=hpe0.X, hpe0.X - this indicate using HPE (High Performance card). on this card TCPDUMP command cant be used necessary to use special commands/programs to capture data, move data to local disk, analyze the traffic using Wireshark If there is no traffic, ask Telcel core people, whether there should be the traffic check Extreme SW port for traffic (use cabling table to find that ports); check TAPs, cables we are not using any filter on Gn links - all traffic should be processed We are processing GTP-U (user plain) & GTP-C (signaling) GTP-U TDRs are generated (every 5 minutes) in /var/nettest/li-doip-gn-kpi/output/ directory GTP-C signaling traffic is processed and stored in probes database (x csdrlog)
Gn typical problems
Example 2: Gn connected via Pass-Thru
R5_BANDERA_GGN01:~# x chnst Channel 268435478/0: Prot=233 Load=398.485 Channel 268435478/1: Prot=233 Load=648.865 Channel 268435482/0: Prot=233 Load=415.531 Channel 268435482/1: Prot=233 Load=112.190 kbps State=No Alarm kbps State=No Alarm kbps State=No Alarm kbps State=No Alarm SIM=1 LIM="0" Port="eth1,eth2" SIM=1 LIM="0" Port="eth1,eth2" SIM=1 LIM="0" Port="eth3,eth4" SIM=1 LIM="0" Port="eth3,eth4" R5_R5BASGS_Gn_Erricson_01 R5_R5BASGS_Gn_Erricson_01 R5_R5BASGS_Gn_Erricson_02 R5_R5BASGS_Gn_Erricson_02
29
Port=eth1, eth2 - this indicate using standard ETH cards + pass-trhu module. TCPDUMP command can be used for analyze the traffic in case of problems If there is no traffic, ask Telcel core people, whether there should be the traffic use TCPDUMP to check the traffic (both GTP-U & GTP-C should be there) check Extreme SW port for traffic (use cabling table to find that ports); check TAPs, cables we are not using any filter on Gn links - all traffic should be processed We are processing GTP-U (user plain) & GTP-C (control - signaling) GTP-U TDRs are generated (every 5 minutes) in /var/nettest/li-doip-gn-kpi/output/ directory GTP-C signaling traffic is processed and stored in probes database (x csdrlog)
Gp typical problems
Example 3: Gp connected via Pass-Thru (always)
R9_PORTALE_GGP01:~# x chnst Channel 268439998/0: Prot=233 Load=187.590 kbps State=No Alarm Channel 268439998/1: Prot=233 Load=232.400 kbps State=No Alarm SIM=1 LIM="0" Port="eth1" R9_R9PTBGN_GP_01 SIM=1 LIM="0" Port="eth1" R9_R9PTBGN_GP_01
30
Gp links (Portalers & Morales) are connected only via pass-thru TCPDUMP command can be used for analyze the traffic (GTP) in case of problems If there is no traffic, ask Telcel core people, whether there should be the traffic use TCPDUMP to check the traffic (both GTP-U & GTP-C should be there) check Extreme SW port for traffic (use cabling table to find that ports); check TAPs, cables we are using IP filter on Gp links - check whether IP filter settings is correct We are processing GTP-U (user plain) & GTP-C (control - signaling) GTP-U TDRs are generated (every 5 minutes) in /var/nettest/li-doip-gn-kpi/output/ directory GTP-C signaling traffic is processed and stored in probes database (x csdrlog)
31
Gi/Gy links
31
32
GGSN
Fiber optic
MAC switch
Gy (VLAN x)
OCS
(online charging system)
SGSN
probe
1x ETH
33
Gi/Gy overview
33
34
TCPDUMP command can be used for analyze the traffic in case of problems If there is no traffic, use TCPDUMP to check the traffic (internet traffic /www, snmp, etc./ should be there) when there is no traffic, ask Telcel core people, whether there should be the traffic check VSS24 ports for traffic (use cabling table to find that ports) - we shoud have traffic on all FO ports (1,2,3,4 in this example):
35
no signal on all or some ports: check links with Telcel core whether there should be the traffic check cables, FO connectors, signal levels on DX VAM output and VSS24 input (signal level should be at least -15 dBm, recommended -12 dBm) BAD PACKETS on some ports: check cables, FO connectors, signal levels on DX VAM output and VSS24 input (signal level should be at least -15 dBm, recommended -12 dBm)
36
If there is no traffic on probe, check also that filters (ports) on VSS24 the same filter are on probe to divide traffic on logical links (check it in Configuration Editor)
37
37
38
BSC
E1
SGSN
coaxial splitter
39
SIM=1 LIM="0" Port="0" R6_LSL_STMR1311S00T01_R2AZTM01_A_01 SIM=1 LIM="0" Port="0" R6_LSL_STMR1311S00T01_R2AZTM01_A_01 SIM=1 LIM="0" Port="1" R6_LSL_STMR1312S01T01_R2AZTM01_A_01 SIM=1 LIM="0" Port="1" R6_LSL_STMR1312S01T01_R2AZTM01_A_01
eINPUT_CONNECTOR_ALARM(4),MB2_NoSignal(4) - this means, no signal on input connector check this link with Telcel whether we should have traffic on this link (find this link in cabling table file), check configuration of links (timeslots) if they are according to Telcel information & cabling table check balun, cables & connectors on-site
There is no possibility to check traffic like using TCPDUMP - you need to correctly configure links and check the traffic in Protocol Analysis.