603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13

Problem Statement: 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13
Description:
 A program block has not been able to contact the program block acting as its partner
program block. Either the partner program block has not responded within the set time or
no connection has been established to the unit in question.
 Call set-up time is the summation of: (a) call request time—the time from initiation of a
calling signal to the delivery to the caller of a proceed-to-select signal; (b) selection time—
the time from the delivery of the proceed-to-select signal until all the selection signals have
been transmitted; and (c) post selection time—the time from the end of the transmission of
the selection signals until the delivery of the call-connected signal to the originating
terminal.
Preliminary Analysis:

 Volte Launch Date : 04-10-2017 – MP/CG..

 CC 603 & High CST issue increased after Volte soft launch in MP/CG circle.
 high latency between hops when we are taking trace from AMSS13(MGCF) IPDUs towards
CFX(SCSCF) as per preliminary analysis towards Mumbai MPLS end IP.
 And high call setup issue observed in breakout calls in AMSS13 (SIP BGCF TG 4000,4001,4002)
 Call Setup Time (CST) with MSS 13  >5 secs and more..
 CST with Bypass MSS 13  <2 secs
 Breakout Calls is failing under CLEAR INFO : IC3_SC 00ACH 0000H & corresponding to CC A03
observed in TAS
 00ACH means:- A timer expired while waiting for response for message
cc_bearer_establishment_req_s.
 High complains increasing due to high CC603 and CST issue
 Care case MP - CAS-72363-P8Z8 also raised for resolving  Volte High Call setup time for Call
through MGCF (AMSS13) due to that CC-603 triggered.
 Node SCR also decreased as per mentioned trend.
 As per Wireshark traces, SIP INVITE is properly reaching to MGCF/MSS13 but there is issue in
bearer establishment between MGCF and CS-Core terminating network. That’s why such timer is
getting expired (cc_bearer_establishment_req_s) and getting 500 sip error over ZS7 prb.

CC-603-NO
RESPONSE
FROM
PARTNER
PROGRAM
BLOCK
26- 27- 28- 1-Oct- 2-Oct- 3-Oct- 4-Oct- 6-Oct- 7-Oct- 8-Oct- 9-Oct- 10- 11- 12- 22- 23- 24-
NODE Sep-17 Sep-17 Sep-17 17 17 17 17 5-Oct-17 17 17 17 17 Oct-17 Oct-17 Oct-17 Oct-17 Oct-17 Oct-17

MSS07 6217 6085 5569 4609 4990 5827 7640 7568 7875 7074 6030 6772 6437 6450 6176 6358 6933 6895

MSS13 15990 16729 15721 13480 14672 15952 31358 35364 37802 36246 31306 37804 37996 36914 33594 31961 37195 37163
26-Sep- 27-Sep- 28-Sep- 1-Oct- 2-Oct- 3-Oct- 4-Oct- 6-Oct- 7-Oct- 8-Oct- 9-Oct- 10-Oct- 11-Oct- 12-Oct- 22-Oct- 23-Oct- 24-Oct-
Node ▼ 5-Oct-17
17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17
AMSS13IND-
SCR 98.08 98.03 98.08 98.10 98.12 98.09 97.58 97.57 97.37 97.11 96.95 98.00 97.99 97.87 98.03 98.06 98.14 98.07
 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13

CC 603
40000 98.4
35000 98.2
98
30000 97.8
25000 97.6
97.4
20000
97.2
15000 97
10000 96.8
96.6
5000 96.4
0 96.2

MSS07 MSS13 AMSS13IND-SCR

Root cause:

 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13 due to 6 sec delay was causing
in each breakout call due to wrong configuration in DNS parameter set/LOOkUP order.

Reasons for CC603:

 CC 603 continuously in AMSS13 (MGCF) under CLEAR INFO : IC3_SC 00ACH 0000H and
corresponding to CC A03 observed in TAS.
 wrong configuration in DNS parameter set/LOOkUP order in AMSS13 as per care.
 And high call setup issue observed in breakout calls in AMSS13 (SIP BGCF TG 4000,4001,4002)
 Call is originated from IMS towards LRN-3099 (Multiple LRN but this is Sample Case), As per
below log, INVITE Request is forwarded from S-CSCF (10.5.83.198) towards MGCF
(172.19.32.36). Here, MGCF is unable to terminate the call in its domain and give clear code-603
with Ext Cause Code-1F4 (500) to S-CSCF. Now when S-CSCF forwards this Error to TAS, TAS in
return generates CC-A03.
Calling Number:- +917389259594
Called Number:- +9130999664014515
"172.19.32.36","10.5.83.198","SIP","Status: 500 Server Internal Error | "
Internet Protocol Version 4, Src: 172.19.32.36, Dst: 10.5.83.198
Source: 172.19.32.36 ------------------------------------------- (IPDU IP of MGCF)
Destination: 10.5.83.198 ----------------------------------------(IP of S-CSCF)
Session Initiation Protocol (500)
Status-Line: SIP/2.0 500 Server Internal Error
Status-Code: 500
[Resent Packet: False]
Message Header
Content-Length:0
From:<sip:+917389259594@ims.mnc093.mcc404.3gppnetwork.org;user=phone>;tag=0Z71geC00D2j3gAi
 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13

To:<sip:+9130999664014515@ims.mnc093.mcc404.3gppnetwork.org;user=phone>;tag=70BC3157b47U.225
[truncated]Via:SIP/2.0/UDP
10.5.83.198:5050;branch=z9hG4bK_IMSCL1703182.000_ced2a27748a83c55e596ba98a61d45e3;lskpmc=BXS,S
IP/2.0/TCP
10.5.83.198:5090;branch=z9hG4bK_IMSCL1703182.000_8f0719f3a49705e773698d100b4d109e;lskpmc=SXS,S
IP/2.0/TCP 1
Call-ID:89D3B69C31D1ABDA4B5A2A17@0570ffffffff
CSeq:1 INVITE
Reason:X.int ;reasoncode=0x00000603;add-info=0135.00AC.0000
Reason protocols: X.int
Cause: 0
Resolution Step:

 As per preliminary analysis Breakout Calls is failing under CLEAR INFO : IC3_SC 00ACH 0000H &
corresponding to CC A03 observed in TAS
 00ACH means:- A timer expired while waiting for response for message
cc_bearer_establishment_req_s.
 Continuously taking logs and MM capturing root cause of such type of critical issue.
 Care Case raised for early resolution and as per care suggestion so many changes done in MSS13
for resolution.
 Finally care/R&D suggested for correction in DNS parameter. As per below mentioned detail.

MAIN LEVEL COMMAND <___>

< QRJ:;

LOADING PROGRAM VERSION 27.72-0

MSCi AMSS13IND 2017-11-06 23:21:42
INTERROGATED DNS PARAMETERS

PARAMETER VALUE
--------------------- --------------------------------------------------
PRIMARY DNS SERVER: 10.73.8.27
SECONDARY DNS SERVER: 10.73.8.28
TERTIARY DNS SERVER: 0.0.0.0

DOMAIN NAME:

RESOLVER CACHE: IN USE

ROUND ROBIN: IN USE
LOOKUP ORDER: CACHE, NETWORK AND HOSTS. ------------------------- Pre changes

COMMAND EXECUTED

TCP/IP STACK DATA HANDLING COMMAND <QR_>

 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13

QRJ:;

MSCi AMSS13IND 2017-11-08 15:44:04

INTERROGATED DNS PARAMETERS

PARAMETER VALUE
--------------------- --------------------------------------------------
PRIMARY DNS SERVER: 10.73.8.27
SECONDARY DNS SERVER: 10.73.8.28
TERTIARY DNS SERVER: 0.0.0.0

DOMAIN NAME:

RESOLVER CACHE: IN USE

ROUND ROBIN: IN USE
LOOKUP ORDER: CACHE, HOSTS AND NETWORK. ------------------ Post Changes in MSS13

COMMAND EXECUTED

 After Suggest changes CC603 & High CST come on normal trend
 Same is under close observation.

Preventive Steps:

 Continuously monitoring and health check performed for avoiding such type of critical issue.
 Avoid latency issue in n/w which may be one of the reason of call failure.
 Current case give us good learning to understand to solve CST issue.

Result as mentioned: shows improvement in CC603 & High CST.

 After taking corrective action CC603 and high CST come on normal trend.
CC-603-NO
RESPONSE
FROM
PARTNER 26- 27- 28- 1- 2- 3- 4- 5- 6- 7- 8- 9- 10- 11- 12- 22- 23- 24- 8-
PROGRAM Sep- Sep- Sep- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Oct- Nov-
BLOCK 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17

NODE

MSS07 6217 6085 5569 4609 4990 5827 7640 7568 7875 7074 6030 6772 6437 6450 6176 6358 6933 6895 6907

MSS13 15990 16729 15721 13480 14672 15952 31358 35364 37802 36246 31306 37804 37996 36914 33594 31961 37195 37163 16683
 603H NO RESPONSE FROM CO-PROC & High CST issue in AMSS13

CC603 trend
50000
45000
40000
35000
30000
25000
20000
15000
10000
5000
0

NODE MSS07 MSS13

30- 31- 2- 3- 4- 5- 6- 7- 8- 9-
Oct- Oct- Nov- Nov- Nov- Nov- Nov- Nov- Nov- Nov-
17 17 17 17 17 17 17 17 17 17
SL.
No. KPI Name Unit Parameter Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed
Average
successful
call setup
2.5 time in TAS. TAS Performance 4.05 3.94 4.04 4 3.89 3.93 3.9 3.87 2.06 2.04

Observation:

 The call setup success rate is one of the key performance indicators (KPI) used by the network
operators to assess the performance of the networks. It is assumed to have direct influence on
the customer satisfaction with the service provided by the network and its operator.
 improving radio coverage, expanding the capacity of the network, and optimizing the
performance of its elements.
 CC603 & CST issue is under close observed by us and leaning noted for future reference.
 Thanks for the support…

------------------------------------------end of Document----------------------------------------------------

Prepared By :-

Rahul Puloria…