Troubleshooting Unified communication Manager and Cisco Telepresence Video Communication Server Integration

BRKEVT -3304

BRKEVT-3304 BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public Cisco Public

Troubleshooting Unified communication Manager and Cisco Telepresence Video Communication Server Integration

Mahesh Anjan
Sr.Software Engineer
Cisco Systems

BRKEVT-3304

Cisco Public

BRKEVT-3304 Session Logistics

A brief overview of the deployment will be provided, however, to maximize the material learned, the attendee should be familiar with Cisco telepresence Video communication server and Unified Communication Manager Infrastructure. Attendees should also have a good understanding of SIP and H.323 call flows.
Appendix slides ( Interop call flows, additional case studies) Session time: 90 minutes Please ask questions as we go 3 types of questions: Questions Ill answer

Questions Ill defer to later in the session Question I dont know the answer to, outside the scope of our session, or those that consume too much time.
Come see me after session to chat and/or leave your name, email, and question(s) so I can get back to you.
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Complete Your Online Session Evaluation BRKEVT-3304

Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and ondemand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Objective
With UC Manager and Cisco Telepresence VCS bridged in enterprise network, the session aims at troubleshooting various interoperability issues that arises between these call controls and take appropriate steps to resolve them

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Agenda
Deployment Overview

UC Manager Integration with Cisco Telepresence VCS Cisco Telepresence VCS Integration with Unity Connection voicemail
UC Manager Session Layer Bandwidth UC Manager BFCP Interoperability Cisco Telepresence Native Interoperability UC Manager Integration with Cisco Telepresence MCU

UC Manager Integration with ISR G2 Conference Bridge Cisco Telepresence endpoints Native registration with UC Manager Troubleshooting Summary Appendix Extra case Studies

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Baseline Software version

UC Manager
Cisco Telepresence Video Communication Server

8.6

X6.1
4.2

Cisco Telepresence MCU Cisco Unity Connection

8.6
IOS : 15.1.4M

ISR G2 Video Conference Bridge

EX/C Series

TC 5
TE 4.0 1.7.4 F 9.0.2

E Series
Cisco Telepresence Codec 500/1000/3000

Cisco Telepresence MXP Series Polycom HDX 4000

3.0.0.1
9.0.6.1 3.4.2

Polycom VSX 7000

Sony PCS-1

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Abbreviations
UCM
VCS

Unified Communication Manager

Video Communication Server Media Conference Unit Delayed Offer in UCM ( Invite without SDP)

MCU DO

EO
PT

Early offer in UCM ( Invite with SDP)

Payload Type

Uconn
BFCP

Unity Connection Binary Floor Control Protocol

Integrated Service Router Transport Independent Application Specific

ISR TIAS

AS
CT

Application Specific
Conference Total

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Deployment Overview

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Deployment Overview
9971 CIUS-SIP MCU 4205 SONY-PC1-H323 E20-SIP

Ex90 H323/SIP

Polycom VSX 7000

VCS

DO:SIP Trunk
CUCIlync

Ex90-SIP
MXP 1700 H323/SIP E20-SIP C20-SIP

MOVi-SIP

MCU

SP
SONY-PC1-H323
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

SIP Trunk SIP Registration

Cisco Public

H225 Trunk

10

Related Sessions

For more information about UCM VCS Deployment consider attending:

BRKEVT-2315 Understanding the Cisco Tandberg Infrastructure

Thursday, July 14th @ 8:00AM

For more information about Deploying IP Video Telephony Networks Consider attending :

BRKEVT-2300 Designing and Deploying IP Video Telephony Networks

Tuesday, July 12th @ 12:30 PM

For more information about Video Quality troubleshooting Refer to:

BRKEVT- 3934 Troubleshooting Voice and Video Quality in Unified Communication Network
Monday, July 11th @ 8:00AM

For more Information about UC Manager Serviceability Troubleshooting Refer to: TECUCC-3000 Cisco Unified Communications Manager Serviceability and Troubleshooting Monday, July 11th @ 8:00AM

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

11

Log collection from VCS , VCS endpoints & UC Manager

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

12

Diagnosing VCS Issues How do I access VCS and what are the Common Logs to Collect

Serial Console connection SSH , Telnet Access Protocol level decode Protocol level decode,hex dump
SIP H323
Interworking logs
BRKEVT-3304

Connect the serial cable from the DATA port on the front of the VCS to the serial port on a PC. baud rate : 115200 bits per second data bits : 8 Parity : none stop bits : 1 flow control (hardware and software): none

Root Login (default) : root / TANDBERG Telnet (default) : admin / TANDBERG

SSH to the endpoint and enter

netlog 2

SSH to the endpoint and enter

netlog 3

netlog off xcom log network debug xcom log developer.iwf debug console on xcom log network warn xcom log developer.iwf warn

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

13

Diagnosing VCS Endpoint Issues What are the Common Logs to Collect
Enabling Root Access SSH , Telnet Access Logs from VCS MXP 1700 h323 endpoint SIP logs from Ex/C Series Logs from VCS Ex/C series h323 endpoint
Collecting Logs from E20
SSH using root username and TANDBERG as password , If root access is disabled , SSH using GUI username and password and then configure system tools rootsettings on [password] , this will set root password on the unit.
root Login (default) : root / TANDBERG Telnet (default) : admin / TANDBERG
set-a qwertsys syslog off sys-f s-t 0x1300 upr aciptel acipadm h323m uis uic com h323m 0 log call rtp Quit

Login as "admin" via SSH and then enable "log ctx SipPacket debug 9"

Login as "admin" via SSH and then enable log ctx H323Packet debug 9
Login as "admin" via SSH and then enable log ctx SipPacket debug 3 log ctx MC debug 3 log ctx APPL debug 3

Collecting tcpdump from endpoints

BRKEVT-3304

Login as "admin" via SSH and then run the command tcpdump -n -s 1500 -w /tmp/log.pcap ip and not port 22

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

14

Diagnosing VCS Endpoint Issues Collecting logs from E20

Taking Screenshots on E20
Taking screenshots on E20 Login using SSH and enter tshell mkdir snapshots CD /tmp/snapshots sys-f com cuilfsm 0 event MenuMagic CreateScreenShot Current /tmp/snapshots/test.png Copy the file from the endpoint using scp root@e20:/tmp/snapshots/test.png

Power cycle via CLI

can be done by logging into the endpoint with ssh root@ip and doing an /etc/init.d/S90osd restart

Cannot access web page

Verify IP connectivity (e.g. ping). Get this working first. Ensure E20 does not have both HTTP and HTTPS enabled

Retrieving Logs via SSH/Telnet

The E20 saves logs in the /var/log/eventlog folder, where we have an application, main, video and audio logs. In the case of a reboot or a crash, logs are stored in the /config/logs folder so it may be useful to gather logs from here in the case of bug reporting. Log into the system via SSH/Telnet (default username admin, no password) The log from the previous session (ie. the session before the system has rebooted) will be /config/logs/log.tar.gz In the case of a crash, it could be useful to not only tar up the eventlog folder, but the whole log folder The log from the current session is in /var/log, to get it and then scp it do the following: $tar zcvf /tmp/current.tar.gz /var/log/eventlog $scp /tmp/current.tar.gz user@remotebox:current.tar.gz
Cisco Public
15

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

UC Manager Log Collection

1

From UCM Serviceability Go to Trace Configuration Ensure SIP Call Processing trace and SIP Stack Traces are checked

Collect UCM Logs from RTMT , Go to System Trace and Log Central Collect Files

For more Info on UC Manager log Collection and Troubleshooting , Refer TECUCC-3000 - Cisco Unified Communications Manager Serviceability and Troubleshooting
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

16

UC Manager Log Collection

2
Cli to Capture pcap trace via UCM CLI

3
Wireshark Capture

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

17

UC Manager Integration with Cisco Telepresence VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

18

Case Study 1 : 9971 gets fast busy and the call does not get connected

SIP Trunk( DO / EO)

E20
503 Service Unavailable

UCM
9971

VCS

408 Request time out

Topology
9971 UCM SIP Trunk E20

Logs to Collect
UCM SDI / SDL Logs
Sniffer trace from UCM

Cause Codes from UCM trace

503 Service unavailable or might encounter

BRKEVT-3304

408 Request timeout

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

19

Whats the Root Cause ?

SIP Trunk Failure

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

20

Debugging SIP Trunk Failure

If OPTIONS ping is enabled under SIP Profile of the SIP Trunk that connects to VCS. UCM traces will log :SIPTrunkOOS alarm in sdi traces Snapshot of sdi trace :
%UC_CALLMANAGER-3-SIPTrunkOOS: %[DeviceName=options_ping_to_tbd3][UnavailableRemotePeersWithReasonCode=10.4.21.75, 5060, remote=503][AppID=Cisco CallManager][ClusterID=tbd7-pub2-7825I][NodeID=tbd7-pub27825I]: All remote peers are out of service and unable to handle calls for this SIP trunk
SIP Profile Config

Ensure Zone configuration in VCS shows UCM peer status to be Active and Zone Profile is set to Cisco Unified Communication Manager

Link to UCM shows green

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

21

Debugging SIP Trunk Failure

Trunk reset/restarted or if there a network issue while the call is active
UCM IP

Failure Code : UCM sends BYE with cause code 41

Cause Code

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

22

Debugging SIP Trunk Failure

OPTIONS PING

SIP OPTIONS PING from VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

23

Case Study 2 : Call from H323 VC endpoints results in One way Video & Two way Audio

Audio No Video D0: SIP Trunk

UCM VCS

E20 Display : No Incoming Video

SONY-PCS1

Topology : Sony PCS GK UCM VCS E20 Logs to Collect

UCM SDI/SDL Logs Sniffer Trace from UCM switch port
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

24

Debugging One way Video

INVITE from UCM to VCS
200ok from VCS to UCM: Payload 97 for H264 MIME type

PT:97

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

25

Debugging One way Video

UCM ACKs back to VCS with 105 Payload for H264 MIME Type

UCM IP

PT:105

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

26

Whats the Root Cause ?

Dynamic Payload Negotiation Issue

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

27

Solution Proposed
9971 Topology : Sony PCS VCS SIP Trunk UCM 9971

D0: SIP Trunk

UCM

VCS

SONY-PCS-1

SONY-PCS-1

E20

Transition H323 devices behind GK to VCS VCS do H323 SIP Interworking

Registered to VCS

Eliminate RAS Aggregator trunk , UCM and IOS gatekeeper config

VCS will handle H323 registration and CAC

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

28

Successful Call : Two Way video Established

SONY PCS Calls 9971 behind UCM --- >INVITE from VCS to UCM
200OK from UCM to VCS carries PT 97

UCM IP

PT: 97

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

29

Successful Call : Two Way video Established

VCS ACKs back to UCM with payload 97 and two way video is established

VCS IP

PT: 97

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

30

Cisco Telepresence VCS Integration with Unity Connection voicemail

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

31

Case Study 3 : Unable to leave voicemail to E20 registered to VCS

VCS UCM

DO: SIP Trunk

9971 UConn 28100: Pilot Number
Call Fwd All

E20

Topology

9971UCMSIP Trunk VCS E20 (Cfwdall) E20SIP TrunkUCM SIP Trunk Unity Connection
Symptom Caller gets redirected to Unity connection welcome prompt

Logs to Collect UCMSDI/SDL and Sniffer traces

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

32

SIP Exchange between UCM and VCS

INVITE from VCS to UCM
Request-Line: INVITE sip:28100@10.4.21.11SIP/2.0
Uconn Pilot Number Invite from VCS without Diversion header

Call Forward Scenario in UCM case

Diversion Header

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

33

SIP Exchange between UCM and Unity connection

Invite from UCM towards unity connection
Request-Line: INVITE sip:28100@10.4.21.49:5060 SIP/2.0
VCS Case : Invite from UCM to Unity Connection

UCM Case : Invite from UCM to Unity connection

Subscriber DN

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

34

Whats the Root Cause ?

Cisco Telepresence VCS does not support diversion header

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

35

Workaround : Direct SIP Trunk to UC from VCS

Unity Connection
Pilot Number : 59000

VCS

DO: SIP Trunk

UCM

9971 E20
Create Zone, Search Rule from VCS to Unity connection
51991

Ensure SIP Connection to Unity connection shows active on VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

UConnIP

36

Working logs
Request-Line: INVITE sip:59000@10.4.21.49 SIP/2.0 from VCS to Unity connection
Request URI and TO header are different and Unity Connection Considers To header of user portion to direct the caller to 51991 Subscriber mail box
VCS IP

Pilot DN

Fwd DN

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

37

What really happened in UCM failure case ? Request URI and TO Header across SIP Trunk
INVITE from VCS to UCM
Request-Line: INVITE sip:28100@10.4.21.11 SIP/2.0
VCS IP

Fwd DN
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

38

To Header and Request URI Across SIP Trunk

Request sent out from UCM towards Unity connection
Request-Line: INVITE sip:28100@10.4.21.49:5060 SIP/2.0
UCM to UC

UC Pilot

UC Pilot not Fwd DN UCM does not preserve To header and Request URI across SIP Trunk
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

39

Solution for UCM Failure Case

Normalization Script can be Implemented at the UC Manager Trunk level to modify To Header and Request URI Diversion header support in VCS is in the roadmap

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

40

UC Manager Session Layer Bandwidth

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

41

What are the Common Bandwidth Attributes ?

The Transport Independent Application Specific Maximum (TIAS) bandwidth modifier has an integer bit-rate value in bits per second. The bandwidth value is the maximum needed by the application (SDP session level) or media stream (SDP media level) without counting IP or other transport layers like TCP or UDP

b=TIAS:<bandwidth-value>

b=AS <application Specific>

The Application Specific maximum bandwidth is indicated by the modifier "AS. The interpretation of this attribute is dependent on the application's notion of maximum bandwidth.

b=CT <conference total>

Conference Total gives a maximum bandwidth that a conference session will use. Its purpose is to decide if this session can co-exist with any other sessions.

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

42

A Quick Glance
BW Attributes Pre 8.6
Processed at session level and pass through at media level

8.6
Supported

Impact
Video Quality dropped to as low as 300k and 13 fps

b= TIAS (b/s)
Used as session and media level . Does not include transport overhead

b= AS

(Kb/s)

Ignored processing b=AS

Supported

Some endpoints recognized only AS as BW modifier that resulted in over or under subscription of UC Manager region bandwidth Some endpoints recognized only CT as BW modifier modifier that resulted in over or under subscription of UC Manager region bandwidth
43

b= CT

(Kb/s)

Ignored processing b=CT

Started processing b=CT

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Case Study 4 : Session layer Bandwidth Behavior with UCM 8.5 and 8.6

1
E20

DO: SIP Trunk

UCM 8.5

UCM 8.5
9971 9971

2
E20

DO: SIP Trunk

9971

UCM 8.6

UCM 8.6

9971

Topology
E20 UCM SIP Trunk UCM 9971 E20 calls 9971

Logs to Collect UCM SDI / SDL and Sniffer trace

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

44

UCM 8.5 : Region Bandwidth set to 384 Kbps

Initial INVITE from E20 to UCM
Initial Offer from E20 to UCM set to 1152

Offer goes in form of AS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

45

UCM 8.5 : Region Bandwidth set to 384 Kbps

200ok from UCM to E20

No AS parameter in the answer

No TIAS for Audio m-line

384000 64000 = 320000

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

46

UCM 8.5 : Region Bandwidth set to 384 Kbps

E20 Screenshot

Will experience choppy or black video on E20

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

E20 channel rate falls to very low bit rate degrading video quality
Cisco Public
47

Comparing with UCM 8.6 traces : Region Bandwidth set to 384 kbps
Initial INVITE from E20 to UCM

Default call set to 1152

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

48

Comparing with UCM 8.6 traces : Region Bandwidth set to 384 kbps
200ok from UCM to E20

UCM Indicates Region Bandwidth

TIAS: 64000 for G711

384000 64000 = 320000 for Video

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

49

Comparing with UCM 8.6 traces : Region Bandwidth set to 384 kbps
E20 is offered back with AS and E20 adheres to UCM bandwidth management
E20 Adheres to UCM bandwidth

E20 Screenshot

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

Channel rate does not to a very low BW

50

Session Layer Bandwidth config Snapshot

In the outbound offer sent by UCM, which includes the Early Offer or re-invite Offer, UCM will send session level bandwidth modifier based on SIP Profile configuration, and this configuration can be applied to both SIP line side device and SIP trunk device.
TIAS and AS default

SIP Profile config

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

51

UC Manager Offer / Answer Behavior

Callers offer: b=AS:384 Callees offer: b=CT:512 Region bw = 512 Answer to caller: b=AS:512 Answer to callee: b=CT:384

Callers offer:b=AS:384,b=CT:384 Callees offer: b=CT:512 Region bw = 512 Answer to caller: b=AS:512 Answer to callee: b=CT:384 Note: If AS and CT were present, CT will be ignored.

Callers offer: b=AS:384, b=CT: 384, b=TIAS:384 Callees offer: b=TIAS:512 Region bw = 512 Answer to caller: b=TIAS:512 Answer to callee: b=TIAS:384 Note: If TIAS, AS and CT were present, AS and CT are ignored

Callers offer: b=AS:740 Callees offer: b=CT:384 Region bw = 512 Answer to caller: b=AS:384 Answer to callee: b=CT:512

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

52

UC Manager BFCP Interoperability

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

53

Understanding BFCP Interop

UCM Cluster
BFCP is supported on Early Offer (EO) and Delayed Offer (DO) SIP trunks

UCM Cluster

SIP Trunk EO / DO

SIP

UCM never terminates the BFCP protocol CUCM never terminates the BFCP protocol UDP / BFCP Supported : Preferred Industry Standard TCP /BFCP : Unsupported : Legacy Deployments
Audio Video (main) Video (Presentation) BFCP

SIP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

54

Case Study 5 : Unable to View Presentation on C20

DO SIP Trunk VCS

UCM

C-20
Ex90

Topology
Ex90 UCM SIP Trunk VCS SIP C20 Ex90 starts presentation

Logs to Collect
netlog 2 , netlog 3 from VCS UCM Sdi and SDK Trace Sniffer traces from UCM
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

55

Debugging Sniffer trace

Initial Invite from EX90 to UCM

Ex90 Negotiates UDP/BFCP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

56

Debugging Sniffer trace

Invite from UCM to VCS

No UDP/BFCP app line in SDP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

57

Whats the Root Cause ?

Allow Presentation Sharing using BFCP is disabled

Other reason Could be : Flow Control Not negotiating Server and Control exhibiting same behavior

BFCP capabilities on a SIP Trunk or SIP Line can be turned on or off via SIP Profile Configuration

BFCP Disabled on SIP Profile

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

58

Whats the Root Cause ?

Allow Presentation Sharing using BFCP is disabled Other reason Could be : Flow Control Not negotiating Server and Control exhibiting same behavior

BFCP Disabled on SIP Profile

BFCP capabilities on a SIP Trunk or SIP Line can be turned on or off via SIP Profile Configuration

SIP Profile with BFCP disabled is configured

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

59

Is there any Config on VCS Side to enable BFCP ?

VCS Default Communication Manger Profile sets SIP UDP/BFCP filter to ON

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

60

Is there any Config on VCS Side to enable BFCP ?

when it is ON , VCS does TCP/BFCP Which is not supported with UC Manager and not the industry preferred choice

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

61

Is there any Config on VCS Side to enable BFCP ?

Create a CUSTOM Profile and set the SIP UDP/BFCP filter mode to OFF

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

62

Working Config
After enabling Allow Presentation Sharing using BFCP in SIP profile

SIP Profile under SIP Trunk is enabled with BFCP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

63

Working Config
After enabling Allow Presentation Sharing using BFCP in SIP profile

UCM Negotiates UDP/BFCP App line

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

64

BFCP and Security

Enabling / Disabling BFCP Security in UCM

When this parameter is set to True (default), calls utilizing a nonsecured BFCP application stream will be treated as secure, provided that the remaining media streams in the call are secured

Negotiated SDP
Audio: Secure Main Video: Secure Presentation Video: Secure BFCP App: Non-Secure
Audio: Secure Main Video: Secure Presentation Video: Non Secure BFCP App: Non-Secure

Service Parameter
True

Security Status of Call

Secure

True

Non-Secure

Audio: Secure Main Video: Secure Presentation Video: Secure BFCP App: Non-Secure

False

Non-Secure

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

65

BFCP Device and endpoint Support

BFCP Transmit and Receive SIP Line Devices SIP trunk (DO & EO ) EXSeries -SIP ( Registered to UCM /VCS ) E20-SIP ( Registered to UCM / VCS ) MOVi ( Registered to VCS ) MXP 1700 SIP ( Registered to VCS ) CSeries- SIP ( Registered to VCS / UCM ) UC Endpoints (99xx , CUPC) Gateways ( SIP , MGCP , H323, SCCP ) CiscoTelepresence MCU Cisco Telepresence Server Cisco Telepresence Codecs 1000/3000/500 MXE BFCP Receive only BFCP Transmit Only

UCM
H.239 to BFCP BFCP to BFCP
BRKEVT-3304

VCS

Use VCS as Proxy

Cisco Public
66

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Case Study 6 : Getting 481 Call transaction does not exist from VCS
481 SIP Message D0: SIP Trunk
UCM

VCS

E20 Polycom-VSX-7000

Topology: E20 (DN: 51991) VCS SIP Trunk UCM GK Polycom(H323)

Logs to Collect :
UCM SDI/SDL Logs Sniffer trace Net log 2 from VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

67

Debugging 481 Call Transaction

UCM sends UPDATE without SDP to provide the called/connected party name and number information during the call setup as well as after call is answered. UCM uses PAI & RPID headers to update this information.
SIP UPDATE :
Remote-Party-ID: "Polycom-VSX7000-H323-26002" <sip:26002@10.4.21.11>;party=calling;screen=yes;privacy=off
180 Ringing had (no display name)
Remote party

P-Asserted-Identity: "Polycom-VSX7000-H323-26002" <sip:26002@10.4.21.11>

P-Asserted-Identity: <sip:26002@10.4.21.11> Remote-Party-ID: <sip:26002@10.4.21.11>;party=called;screen=yes;privacy=off

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

68

Debugging 481 message

2010-09-03 11:19:31,855 Module="network.sip" Level="INFO": Dst-ip="10.4.21.11" Dst-port="5060" Detail="Sending Response Code=481, Method=UPDATE, To=sip:51991@abi.com,

Call-ID=f75faf53903b3170@10.4.22.107"
2010-09-03 11:19:31,855 Module="network.sip" Level="DEBUG": Dst-ip="10.4.21.11" Dst-port="5060" SIPMSG:

Response from VCS to UCM

|SIP/2.0 481 Call/Transaction Does Not Exist

Via: SIP/2.0/TCP
10.4.21.11:5060;branch=z9hG4bK36b215d882ef;received=10.4.21.11 Call-ID: f75faf53903b3170@10.4.22.107 CSeq: 101 UPDATE From:

<sip:26002@abi.com>;tag=109550~78e066fc-a56b-45ac-83ce-0a62eb5c06b2-8124
2816 To: <sip:51991@abi.com>;tag=bae9771a1d8e8596 Server: TANDBERG/4099 (X5.2)

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

69

Debugging 481 Call transaction does not exist

In response to 481 , UCM responds back with 487 with cause code 127
2010-09-03 11:19:31,875 Module="network.sip" Level="INFO": Src-ip="10.4.21.11" Src-port="5060" Detail="Receive Response Code=487, Method=INVITE, To=sip:26002@abi.com, Call-ID=f75faf53903b3170@10.4.22.107" 2010-09-03 11:19:31,875 Module="network.sip" Level="DEBUG": Src-ip="10.4.21.11" Src-port="5060" SIPMSG:

Response from UCM to VCS

|SIP/2.0 487 Request Cancelled

Via: SIP/2.0/TCP 10.4.21.75:5060;egress-zone=ABICUCMNeighbor;branch=z9hG4bKbb7ef2e619cb05 3c53d5819b5a80736e78322.a5b35826b378cf4a7d515b197978bf9d;proxy-call-id=a 6d049c4-b76e-11df-ab04-0010f31abd40;rport,SIP/2.0/TCP 10.4.22.107:5060;branch=z9hG4bK41018499480896c339bfc702fc974502.1;receiv ed=10.4.22.107;rport=49665;ingress-zone=localregistration Call-ID: f75faf53903b3170@10.4.22.107 CSeq: 100 INVITE From: <sip:51991@abi.com>;tag=bae9771a1d8e8596 To: <sip:26002@abi.com>;tag=109550~78e066fc-a56b-45ac-83ce-0a62eb5c06b2-8124 2816 Reason: Q.850 ;cause=127 Content-Length: 0
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

70

Whats the Root Cause ?

VCS does not support the UPDATE before answer, or PRACK (Provisional response Acknowledgment)

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

71

Solutions
Configure Custom Profile in VCS version < 6.0 version to strip SIP UPDATE message

SIP UPDATE being stripped off

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

72

Solutions
VCS 6.1 Version ignores or suppresses SIP UPDATE messages VCS will start supporting SIP UPDATE header (in roadmap)

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

73

Cisco Telepresence Native Interop

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

74

What is Native Interop ?

CIUS
CTS now supports unrestricted media and handles variable frame rate and timestamps.

9971

E20

EX90

EX90

UCM

CTS - 1000

CTS 1.7.4

CUCM 8.5 Yes

CUCM 8.0 No for interop calls

MXE 5600

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

SIP Trunk

Cisco Public

75

Sysop Logs : Is the call interop or non-interop ?

2011-04-25 03:50:27: INFO Connected to Senthil-12006 at 12006 2011-04-25 03:50:28: INFO Negotiated Media Configuration: LTRP: Enabled GDR: Enabled High Profile: Disabled Cabac: Enabled 2011-04-25 03:50:28: INFO Endpoint used protocol MUX version 7 for this call 2011-04-25 03:50:28: INFO The presentation has been negotiated to 30 FPS

Non-Native Interop

2011-04-25 03:52:16: INFO Local user dialing 81123136 2011-04-25 03:52:19: INFO Connected to 81123136 2011-04-25 03:50:28: INFO CTS is in an interop call, and is not using TIP or MUX. Native Interop

Web interface changes for statistics

Legacy added for indicating the interop statistics

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

76

How to identify resolution from CTS Logs ?

SSH to CTS codec login root/ c1sc0cts3 To view resolution of active calls: cmastat | grep state
CTS:>cmastat | grep state
TBA source pid 2529 channel 0 state off inputs 0x200002 format 0x01000007 dspCmds 206 TBV source pid 2984 device 0 state off imageSize 0x0

TBV sink pid 2526 device 1 state on imageSize 352x288 videoClock 26999768.000000
TBV source pid 2985 device 2 state off imageSize 0x0 TBV source pid 2984 device 4 state on imageSize 352x240

Sink means receivers Source means encoders

state on is the image resolution that CTS is sending

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

77

Native Interop Summary ( Call from CTS to)

Endpoint Ex-series C-series E-20 Media 1080p / 30 720p / 30 w360p sent* w448p received
(*E20 can recv up to448p)

Signaling SIP (pt 2 pt) TIP (MultiPt) SIP (pt 2 pt) TIP (MultiPt) SIP

Presentation BFCP (pt2pt) TIP (MultiPt) BFCP (pt2pt) TIP (MultiPt) BFCP receive

Registration UCM 8.6 VCS UCM 8.6 VCS CUCM 8.5 VCS

MOVi
MXP

720p / 30
w360p sent* w448p received*
(MXP can send recv720p)

SIP
SIP
(H.323 on VCS)

BFCP
Switched

VCS
VCS

Cius
CSF clients

720p / 30
720p / 30 or w360p

SIP
SIP

NA
NA

UCM
UCM

9951, 9971

w360p sent* w360p received

(*99xx can recv up to 448p)

SIP

NA

UCM

8941, 8945

CIF, initially*
(*w360p send/recv post-8.6, 89xx can be sent up to w448p)

SIP
(SCCP or SIP on UCM)

NA

UCM

3rd party (Polycom)

BRKEVT-3304

720p / 30

SIP
(H.323 on VCS)

BFCP
(H.239 on VCS)
Cisco Public

VCS
78

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Resolution , Video Quality and max-br mapping

How does CTS uses the max-fs to choose the encoding resolution
Max-fs Resolution

96 919
920 3599

352x288
640x360

3600 8159
> 8159

1280x720
1920x1080

How bitrates affect the resolution

Resolution 1920x1080
1280x720 640x360 352x288

Bitrate > 3M
> 936Kbps > 500Kbps >100Kbps

Max-fs > 8159

3600 8159 920 - 3599 96 - 919

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

79

Cisco telepresence Video Quality & max-br mapping

Codec Config in UCM

Quality (per Display)

Highest detail, Best Motion : 1080p
Maxres:720 p

max-br (kbps)
4000
3500 3000 2250 1500 1000 936

Highest Detail, Better Motion:1080p Highest Detail, Good Motion:1080p High Detail, Best Motion : 720p High Detail, Better Motion: 720p High Detail, Good Motion : 720p High Detail, Limited Motion : 720p (Lite)

Limited Bandwidth Mode : 360p (For TIP Call) : 30fps

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

475
Cisco Public
80

UC Manager Integration with Cisco Telepresence MCU

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

81

MCU Conference Overview

MXP 1700 SIP
CIUS-SIP MCU

CUPC

CUCILync

Intercluster SIP Routing

UCM UCM
C20

Ex90-SIP E20
E20-SIP C20-SIP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

82

How do we add MCU to UCM ?

MCU is added to UCM via Media Resources Conference Bridge as Cisco Telepresence MCU under

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

83

Registration Status

MCU Registered

Http credentials of MCU

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

84

How do we add MCU to UCM ?

MCU resource is added to MRG and then MRGL

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

85

How do we add MCU to UCM ?

MCU contained MRGL is configured as part of phone MRGL

Phone MRGL

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

86

MCU Side Config

Media Port Reservation was changed to "Enabled

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

87

MCU Side Config

Incoming calls to unknown conferences or auto attendants was changed to "Disconnect Caller
Time to wait when setting up ad hoc conference PIN was changed to "never configure PIN"

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

88

How to Debug MCU Registration Issues

Ensure MCU shows registered in RTMT under UCM Cluster Media resources

MCU Stattus

Verify that the MCU address is correct

Verify that the SIP port incoming, and Outgoing match what is configured on the MCU Verify that the MCU is on line.
Verify that the MCU is configured for SIP. Check the UCM traces for the OPTIONS ping to the MCU

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

89

Case Study 7 : MCU Adhoc bridge in UCM allocates only Video ports not Audio
9971

9971

Adhoc conference
MCU

Non Video phone

7945

Topology

9971 UCM 9971

|__Conference :conference Button 7945

Logs to Collect
MCU Logs
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

90

Debugging Port Usage on Codian

Non Video phone

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

91

Debugging MCU Logs

Initial DO Invite from UCM to MCU

MCU offers both Audio and Video Caps for Initial late media

Video port Allocated

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

92

Whats the Root Cause ?

UCM will always include video in the ACK to the Cisco Telepresence MCU because the UCM treats the MCU as pure video bridge

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

93

Workaround to use only audio ports for audio-only capable endpoint

While the conference is Active, modify conference audio port limit on configuration tab
Initial Call to MCU

video port Allocated

Audio-only capable endpoint

Audio port getting used

reserve only audio port

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

94

Alternate Solution
UC Manager will request Video or Audio specific port based on the phone capability ( is in the roadmap)

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

95

UC Manager and VCS Multiway Conference

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

96

Case Study 8 : Multiway URI Conference Fails when UC endpoints are Conferenced

SIP : Multiway Conference 9971 UCM

VCS
Codian MCU

9971

C20 - Multiway Conference

Topology
C20 UCM 9971
C20 Multiway Conference 9971 C20 Multiway Conference VCS Codian Logs to Collect

UCM SDI /SDL , Sniffer Trace

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

97

VCS and MCU Side config

Conference alias

MCU Conference alias config

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

98

UCM Side Routing config

RHS matches TLD and SIP Route pattern

REFER from VCS to UCM when multiway is invoked

UCM & VCS Domain : abi.com

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

99

How UCM interprets SIP URIs

Is RHS the IP address of a cluster member?
yes
no

Does RHS match CFQDN?

no

Does RHS match OTLD?

yes

no

Match RHS against SIP Route Pattern

Route or block

yes

Analyze LHS

Does LHS find a match?

yes

no

Route or block

Offer call

user @ mydomain.org
left-hand-side (LHS)
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

right-hand-side (RHS)
Cisco Public
100

What happens when C20 selects join Calls

VCS to UCM : 302

VCS sends 302 with multiway factory number to join the bridge

REFER TO Header

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

101

Debugging multiway
UCM does DNS Query of host part in the sip contact header i.e abi.com
UCM Fails to do digit analysis to the redirected contacts and the call does not get routed correctly
Search in DNS

DNS IP configured under UCM platform

DNS resolution fails abi.com does not get resolved to IP address

Redirect by Stack
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

102

One of the Root Cause

Redirect by Application under SIP Profile of the SIP trunk is Unchecked

Ensure Redirect by Application is checked

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

103

Solution via redirect by application

With Redirect by Application checked

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

104

Solution via redirect by application

With Redirect by Application checked
VCS sends 302 with multiway factory number to join the bridge
VCS to UCM : 302

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

105

Working logs
With Redirect by Application checked

NO DNS Query

UCM Does digit analysis to the redirected contacts and makes sure that the call get routed correctly

UCM matches SIP route pattern and sends INVITE to VCS with URI set to Refer to contact header

VCS IP

UCM sends INVITE to LHS of REFER header

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

106

All 3 participants are in Multiway conference

MCU Screenshot

Active Multiway Conference

Active Registration
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

107

UC Manager Integration with ISR Conference Bridge

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

108

Configuring Conference bridge in ISR G2

HQ/Campus Branch
Video conferencing WAN

UCM

UCM call control

Supports H264 AVC: Baseline Profile, level 1.0 / 1.3 /2.0/ 2.1/ 3.0 / 3.1

Resolution: QCIF; CIF; w360p; VGA; w448p; 4CIF; 720p(Switchingonly)

Audio Codec : G.711alaw, G.711ulaw, G.729ab, G.729a, G.729b, G.729, G.722r64, and ILBC

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

109

UCM Config : Heterogeneous & Homogenous Mode

Added to UCM via Media Resource Conference Bridge

Adding New conference Bridge

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

110

Registration Status with UCM

Conference Bridge registered

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

111

Mode 1 : Homogenous mode

router(config)#

dspfarm profile 1 conference video homogeneous

router(config-dspfarm-profile)#

codec h264 w360p maximum conference-participants 16 maximum sessions 10 associate application SCCP no shut

Same Phone type Being Used Ex: two 9971

router(config)#

sccpccm1.2.3.4 identifier 1 version 7.0 sccp sccpccm group 1 debug dsp-resource-manager flex video debug vxcmsp error
Cisco Public
112

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Mode 2: Heterogeneous mode

router(config)#

dspfarm profile 2 conference video heterogeneous

router(config-dspfarm-profile)#

codec h264 w448p codec h264 vga Different Phone models used codec h264 cif maximum conference-participants 16 video-cap-class 3 maximum sessions 2 associate application SCCP no shut
router(config)#

sccp ccm 1.2.3.4 identifier 2 version 7.0 sccp sccp ccm group 2
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

113

Cisco Telepresence endpoints Native registration with UC Manager

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

114

Cisco Telepresence EX/C-Series Native Registration with UC Manager 8.6

SIP

UCM
EX90

1
C20

SIP UCM

Topology
Ex90 SIP UCM C20 SIP UCM

Logs to Collect :
UCM SDI / SDL Logs UCM Sniffer Trace

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

115

Debugging EX/C-Series 90 Registered Native to UCM

From EX/C-Series Web Interface : http://<ex/c-series_ip>

SIP
TFTP

URI

UCM

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

116

Debugging EX/C - Series

EX/C-Series shows Not Registered after UCM upgrade
On the touchpad, go to More > Settings > Admin Settings > SIP Settings and change the Proxy address from the hostname to the IP address of the UCM.

EX/C-Series Factory Reset

xcommand SystemUnit FactoryReset Confirm: Yes

Troubleshooting Restart/ApplyConfig
If restart or apply config doesn't work from the CCMAdmin pages check the EX90 config and make sure it is in UCM mode: Advanced Configuration->Provisioning->Mode: UCM. Also verify the Provisioning->ExternalManager is set to the TFTP server of your UCM.

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

117

Troubleshooting Summary

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

118

So far we leaned how to troubleshoot:

UCM VCS SIP Trunk failure
VCS Interop with Unity connection voicemail Session Layer Bandwidth Attributes BFCP interop with UCM and VCS

Cisco Telepresence Native Interop with UC and VCS endpoints

MCU Video port allocation and Multiway with UC endpoints ISR G2 Conference bridge setup

Cisco telepresence Native registration issues

BRKEVT-3304

Cisco Public

Additional Resources
Check out VCS UCM Deployment Document

For Your Reference

http://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Cisco_Unified_Communications_M anager_Deployment_Guide_CUCM_6-1_7_8_and_X6.pdf

Multiway Configuration
http://www.cisco.com/en/US/docs/telepresence/infrastructure/articles/cisco_telepresence_mcu_configure_multiway _kb_166.shtml

UCM and MCU configuration

http://www.cisco.com/en/US/docs/telepresence/infrastructure/articles/mcu_send_receive_sip_calls_cucm_kb_346.s html

Cisco unity connection administration

http://www.cisco.com/en/US/docs/voice_ip_comm/connection/8x/administration/guide/8xcucsagx.html

ISR G2 Conference configuration

http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/gatecont/ps4952/qa_c67-649850.html

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

120

APPENDIX

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

121

Endpoint feature and Interop Summary

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

122

Endpoint features and Interop Summary

Endpoints Call Control Sharing Secure Video Max TX:RX Resolution TIP

Phone + CUVA
9951/71

UCM
UCM

future

future

CIF:wCIF
VGA:w448p

not in 8.6

8941/45 Cius CUCILync, CUPC, CUCIConnect

CTS E20 (VCS/UCM)

UCM UCM
UCM

future webex
webex TIP BFCP* BFCP receive

future future
future

VGA:w448p (w360p future) 720p

720p

not in 8.6 not in 8.6

not in 8.6

UCM Either

/ future /future

1080p w448p

not in 8.6

EX/C Series (VCS/UCM)

MXP

Either
VCS

BFCP, TIP in multi-pt

H.263 switched

1080p EX 720p C
720p / w448p

not in 8.6

Movi

VCS

BFCP

720p

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

123

UCM and Cisco Telepresence VCS Video / Audio Security

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

124

Configuring Security between UCM and VCS

D0: SIP Trunk : sRTP/TLS

VCS UCM

CTS - 1000
EX90

Topology
CTS(Encrypted) UCM SIP Trunk (sRTP/TLS) VCS Ex90 (Encrypted)

Logs to collect
UCM SDI / SDL & netlog2 from VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

125

Configuring Security on VCS Side

SIP Mode Settings

Active on 5061 Port

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

126

Configuring Security on VCS Side

Upload Private and Server Certificates

Register Secure endpoint with VCS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

127

UCM Side Config

1
2

Should match VCS Cert x.509

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

128

UCM Side Config

Make sure LUA Script that supports best encryption is uploaded to UC Manager LUA script can be downloaded from :
http://www.cisco.com/cisco/software/release.html?mdfid=283423434&flowid=21301&softwareid=282074295&release=8.5&reli nd=AVAILABLE&rellifecycle=&reltype=latest

LUA Script

Port has to be 5061

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

129

Determining Security from UCM Traces

UCM Invite

VCS Response

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

130

Additional Case Studies

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

131

Case Study 9 : Audio Codec Negotiation Failure on MXP 1700

MXP 1700

D0: SIP Trunk- G729

UCM

VCS

MXP 1700
Topology:

E20

1700 ( 3rd Party Advanced ) UCM SIP Trunk VCS SIP E20 UCM Phone & Trunk region is set to G729
Logs to collect : UCM SDI / SDL

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

132

Debugging Payload type from SDI logs

UC Manager Trace : 000060889 |2010/08/10 13:37:10.395 |100 |SdlSig-I |MXCapabilitiesIncoming |waitInterfacesCapabilities |MediaExchange(4,100,130,18) |H245Interface(8,100,173,28) |8,100,17,28.2^10.4.71.11^Port 43211 |[R:N-H:0,N:1,L:0,V:0,Z:0,D:0] Caps[2] = 11(20),12(20)VideoCapCount[3] = 101-0,100-0,107-0extendVideoCapCount[0] = h245ICTVersion20useOldGWBytesForGSMConversionFcryptoCapCount=0 DTMF Profile (1,2,119,1,F)

UCM Region se G729

No Codec type 11

000060828 |2010/08/10 13:37:10.378 |100 |SdlSig |SDPOfferInd |waitSDPResponse |SIPInterface(4,100,62,15) |SIPCdpc(4,100,68,26) |4,100,62,15.1^*^* |[R:NH:0,N:1,L:0,V:0,Z:0,D:0] nAudio=1 stackIdx=1 CapCount=4,6(20),4(20),2(20),10(10) port=46784 IP= ipAddrType=0 ipv4=10.4.22.182 SDPMode=0 SP=F RTP=T SRTP=F idle=F QoS=F nVideo=1 stackIdx=2 CapCount=4,103,101,101,100 port=46786 ipAddrType=0 ipv4=10.4.22.182 SDPMode=0 RTP=T SRTP=F idle=F nApp=0 numT38Fax=0 MTPAllocated=0 keepAudiomLineForT38=F transID=0 DTMFMethod=2 DTMFConfig=1 RFC2833PT=103 wantDTMF=0 provideOOB=F FCOffer=0 mSipCallState = 1 InactiveSDPReq=F RSVPLastCollab=F mSIPAccessDevice=F mConfiguredAddrMode=0 000060829

MXP Negotiating codec with UCM

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

133

Determining payload type from UCM traces

Media Payload
G711Alaw64k 2

UCMPayload mapping G.729 Capability Type

G711Alaw56k
G711Ulaw64k G711Ulaw56k G722_64k G722_56k _G722_48k G7231 G729 G729AnnexA

3
4 5 6 7 8 9 11 12

Other codec payload mapping can be found under UCM Documentation

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/service/8_0_2/cdrdef/cdrcodes.html

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

134

UCM Response
Cause code : SDI Snapshot
Outgoing SIP TCP message to 10.4.22.182 on port 49254 index 4
[2122,NET] Cause Code : 503 Service Unavailable

SIP/2.0 503 Service Unavailable^M

Via: SIP/2.0/TCP 10.4.22.182:5060;branch=z9hG4bKfb084a45bf5e6347f57d735b4b1c5f21.1;rport^M From: "21989" <sip:21989@10.4.21.11>;tag=a3c7d054a3fba9d0^M

To: <sip:8333780014@10.4.21.11>;tag=697~78e066fc-a56b-45ac-83ce-0a62eb5c06b2-75907411^M
Date: Tue, 10 Aug 2010 18:37:10 GMT^M Call-ID: 768c980444587620@10.4.22.182^M CSeq: 101 INVITE^M Allow-Events: presence^M Response from UCM to Endpoint

Reason: Q.850;cause=47^M
Content-Length: 0^M

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

135

Whats the Root Cause ?

1700 MXP and other MXP products do not support G729, call fails if the region between UCM and the phone or the outgoing trunk is configured to do only 729

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

136

Solution
Configure Region between MXP (or If you have C-Series) and Outbound trunk to be G711 or G722 (preferred) or any other codec that is listed below
G.711 G.722 G.722.1 G.728 64 Bit MPEG4 AACLD

128 Bit MPEG4 AACLD

Supported Audio Codec

H.261

H.263

H.263+

H.263++ Natural Video

H.264

Supported Video Codec

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

137

Case Study 10: Call fails to connect between 9971 and E20 Via GK Controlled Trunk

UCM

VCS

9971

E20

SP

Topology
E20 SIP VCS H323 GK UCM RT Phone

Logs to Collect
UCM SDI/SDL

Sniffer trace from UCM

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

138

Debugging H225 and H245

In UCM , look for H225 and H245 Negotiation

Typical H.245 Message flow

H245 TCS not coming from VCS

VCS

UCM

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

UCM Session timer times out

Cisco Public

139

Whats the Root Cause

H.245 channel has been established already ,UCM has been waiting for the H245 TCS from the VCS side, which never comes back.

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

140

Solution
For GK controlled calls from VCS to UCM : Wait for Far End H.245 Terminal Capability Set should be unchecked on H225 trunk. (UCM defaults to checked state) With checked state VCS will not send H245 TCS to UCM and the call will not be established. H.225 Trunk to VCS

Uncheck this box

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

141

Case Study 11 :Video Region Bit Rate (384 kb) < Offer Bandwidth
SIP TB E20
INVITE with SDP
SDP 1
o=tandberg 6 1 IN IP4 10.10.10.33 s=c=IN IP4 10.10.10.33 b=AS:1152 m=audio 2326 RTP/AVP 100 101 102 9 18 11 8 0 103 b=TIAS:64000 m=video 2328 RTP/AVP 97 98 99 34 31 SDP 2 b=TIAS:1152000 CiscoSystemsCCM-SIP 2000 1 IN IP4 10.81.55.245 s=c=IN IP4 10.10.10.33 b=CT:384 m=audio 2326 RTP/AVP 100 101 102 9 18 11 8 0 103 b=TIAS:64000 m=video 2328 RTP/AVP 97 98 99 34 31 b=TIAS:320000
BRKEVT-3304 Cisco Public

UCM 1

UCM 2
EO SIP Trunk and CT Only

SIP 9971

UCM 1 will take min (b=AS:1152, video region bit rate) to reserve BW from Location video pool, in this case, video region is less than Offer bandwidth value, so 384 kb will be reserved. UCM 1 sends EO Invite to UCM 2, SIP Profile configured to send CT INVITE with SDP

UCM 2 will take min (b=CT:384, video region bit rate) to reserve BW from Location video pool, in this case, video region = Offer bandwidth value, so 384 kb will be reserved. UCM 2 sends delay offer to RT phone

INVITE w/o SDP

Session layer Bandwidth modifier Case Study 5 : Video Region Bit Rate (384 kb) < Offer Bandwidth
SIP TB E20
SDP 4 o=Cisco-SIPUA 27291 0 IN IP4 172.18.203.94 s=SIP Call b=CT:384 t=0 0 m=audio 31490 RTP/AVP 0 8 18 102 9 116 124 101 b=TIAS:64000

UCM 1

UCM 2

SIP 9971

RT phone sends Answer with SDP, no Session Level Bandwidth, only video line TIAS 200 OK (INVITE)
SDP 3

200 OK (INVITE)

m=video 31604 RTP/AVP 126 97 c=IN IP4 172.18.203.94 b=TIAS:320000

o=Cisco-SIPUA 27291 0 IN IP4 172.18.203.94 s=SIP Call t=0 0 m=audio 31490 RTP/AVP 0 8 18 102 9 116 124 101 m=video 31604 RTP/AVP 126 97 c=IN IP4 172.18.203.94 b=TIAS:1000000

UCM 1 connect media and negotiate codec. E20 RT direction will take min (b=CT:384, video region bit rate) to adjust Location BW. RT E20 direction will take min (b=CT:1152, video region bit rate) to adjust Location BW.
BRKEVT-3304

UCM 2 connect media and negotiate codec. E20 RT direction will take min (b=TIAS:1000000+negotAudio, video region bit rate) to adjust Location BW. RT E20 direction will take min (b=CT:384, video region bit rate) to adjust Location BW. UCM 2 will echo back CT in Answer (200 ok invite), value = video line TIAS + negotAudio
Cisco Public
143

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Session layer Bandwidth modifier Case Study 5 : Video Region Bit Rate (384 kb) < Offer Bandwidth
SIP TB E20
UCM 1 UCM 2
ACK with SDP (INVITE)
SDP 5

SIP 9971

200 OK w/ SDP (INVITE)

SDP 4 O=CiscoSystemsCCM-SIP 2000 1 IN IP4 10.81.55.245 s=SIP Call b=AS:384 t=0 0 m=audio 31490 RTP/AVP 0 8 18 102 9 116 124 101 b=TIAS:64000 m=video 31604 RTP/AVP 126 97 c=IN IP4 172.18.203.94 b=TIAS:320000

o=CiscoSystemsCCM-SIP 2000 1 IN IP4 10.81.55.241 s=SIP Call c=IN IP4 172.18.203.75 t=0 0 m=audio 16616 RTP/AVP 9 103 b=TIAS:64000 m=video 16618 RTP/AVP 98 b=TIAS:320000

UCM 1 send Answer (200 ok invite) SDP which echo back Session level Bandwidth AS.

UCM 2 send Answer (ack invite) SDP with no Session level Bandwidth. Video line TIAS = 384000 64000 = 320000

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

144

Mid call Negotiation Issues

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

145

Case Study 12 : One way Video between 8941 and CTS

UCM

8941
CTS 1000

Topology
8941 UCM CTS

Logs to collect
UCM via RTMT
CTS logs from endpoint and look for sipXXXXX.log under CTS log files

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

146

Debugging CTS logs

TX:
SIP/2.0 200 OK

No common Video Codec is negotiated

Via: SIP/2.0/TLS 10.4.21.11:5061;branch=z9hG4bK9ed40f8dee6

a=rtpmap:97 H264/90000 a=fmtp:97 profile-level-id=42001F;packetization-mode=0;max-mbps=108000;max-fs=3600;max-br=4000

Level 3.1 and VGA

RX:
ACK sip:20065@10.4.22.146:47464;transport=tls SIP/2.0

Level 2.0 from 8941 and VGA

a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=420014;packetization-mode=0;max-mbps=36000;max-fs=1280;level-asymmetry-allowed=1

TX: SIP REINVITE

a=rtpmap:97 H264/90000

Level 1.3 from CTS

INVITE sip:20048@10.4.21.11:5061;transport=tls SIP/2.0

a=fmtp:97 profile-level-id=42000D;packetization-mode=0;max-mbps=11880;max-fs=396;max-br=4000

RX : Post Re-Invite
SIP/2.0 200 OK

Level 2.0 from 8941 and VGA

Via: SIP/2.0/TLS 10.4.22.146:47464;branch=z9hG4bK15d3420a a=rtpmap:97 H264/90000 a=fmtp:97 profile-level-id=420014;packetization-mode=0;max-mbps=36000;max-fs=1280;level-asymmetry-allowed=1

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

147

Whats the Root Cause ?

CTS does not support VGA and it re-INVITES to lower the resolution to w360p 8941 does not support w360p

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

148

Solution
CTS will support VGA ( in roadmap)
8941 will support w360p ( in roadmap)

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

149

Case Study 13 : Mid Call resolution issue with 8941 while joining conference

D0: SIP Trunk

UCM 8941

VCS MCU

Topology
8945 SCCP UCM SIP Trunk VCS SIP Codian MCU ,

User Experience
Video freezes when the endpoint joins the conference

Logs to collect
UCM SDI /SDL logs MCU logs
BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

150

Whats the Root Cause ?

Codian first sends CIF (Welcome screen), then briefly SIF (8941 video downscaled), then finally VGA (8941 video full-res) 8941 has trouble switching decode resolutions mid-stream to VGA

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

151

Workaround

Maintains VGA resolutuion throughout the conference

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

152

Workaround

8941 connected to MCU in VGA mode

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

153

Solution
Upgrade 8941 firmware to 9.1 (2)SR1 , Firmware adapt to dynamic resolution during mid call

Available on cisco.com

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

154

Identifying Profile Level from H.264 SPS while the call is active

level 2.0 in the H.264 SPS

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

155

Resolution and Profile Level mapping

Resolution Expected Profile Level

SIF : 352 x 240

CIF : 352 x 288 WCIF : 512 x 288

1.1
1.1 2.2

VGA : 640 x 480 : w360 720p : 1280 x 720 1080p : 1920 x 1080

3.0 3.1 4.0

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

156

Debugging Third party Payload Negotiation Issues

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

157

Case Study 14 : Polycom is not able to retrieve voicemail from Unity connection

Unity connection

Polycom 4000

Topology
Polycom HDX 4000 Third Party SIPUCM SIP Trunk Unity Connection

Logs to Collect : UCM SDI, SDL &Sniffer Trace

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

158

Debuging SIP Invite from Polycom

Comparison between Polycom and 9971 INVITE to Unity connection
Polycom Trace

9971 Phone

PT: 101 not negotiated

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

PT: 101 negotiated

Cisco Public
159

Root Cause
Polycom Video Endpoint does not negotiate Telephony-event in its initial INVITE

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

160

Solution
Upgrade Polycom Software version to to 3.0 available online

3.0 software advertises telephony-events

Telephonyevent negotiated

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

161

Case Study 15 : One way Video between CTS and Polycom

UCM

CTS 1000

Polycom 4000

Topology CTS SIP UCM Third Party SIP Polycom HDX 4000
Logs to Collect UCM SDI / SDL and Sniffer Traces

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

162

Debugging one way video:200ok from polycom negotiates H.264 in 109 PT

Polycom IP

Polycom negotiated PT for H.264 :109

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

163

Debugging one way video : UCM ACKs back with different payload

UCM IP

UCM negotiated PT : 97

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

164

Whats the Root Cause ?

Different payload negotiated leads to Video codec negotiation failure

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

165

Solution
Polycom is working on addressing this issue

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

166

Debugging Third Party registration issues with UC Manager < 8.5

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

167

Unable to register E20 /EX90/C20 ?

To register any Third party Advanced SIP endpoint to UCM
Add mac address of the endpoint to UCM Associate the endpoint with digest user and password which is created under end user Go to Advanced configuration on the endpoint and configure the following. Phone Configuration EnsureDefaultTransport is set to UDP or TCP towards UCM and Type is set to :Standard

Configure loginName and Password that is set as digest user and pin under device page in UCM
Under conference 1 , ensuredegault call is set to SIP

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

168

SIP , SCCP , H323 , BFCP Interop Call flows

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

169

H323- SIP Call Flow

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

170

H323- SCCP Call Flow

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

171

SCCP- SIP Call Flow

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

172

Typical BFCP Call Flow Scenario : Presentation negotiated in Initial Setup

3333 , 4444 , 1

Pointer to media stream

Role of Media stream Main Video Channel

Presentation Channel

Presentation using mstream 12

BRKEVT-3304
2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Presentation established

Cisco Public

173

BRKEVT-3304

Cisco Public

Visit the Cisco Store for Related Titles http://theciscostores.com

BRKEVT-3304

2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Public

175

Thank you.

BRKEVT-3304

Cisco Public