Documente Academic
Documente Profesional
Documente Cultură
- Poland
The topology includes two Datacenters and two Remote Branches. The topology has three
different VPN/VRF Segments.
• Corporate VPN (VPN 10) Requires full mesh connectivity across ALL sites.
• IOT/PCI Segment (VPN 20) Requires Hub-n-Spoke between the DC and the Branches.
• GuestWifi (VPN 40) Not needed in the DCs. From the branches require DIA. No Site -
to-Site communications.
Note: OSPF is running in the DCs and Branch 2 in VPN 10. All other segments are using static
routing/VRRP.
Table 1. Device Addresses
For best performance, connect to the workstation with Cisco AnyConnect VPN and the local
RDP client on your laptop.
Note: You can also connect to the workstation using the Cisco dCloud Remote Desktop
HTML5 client. The dCloud Remote Desktop client works best for accessing an active session
with minimal interaction. However, many users experience connection and performance
issues with this method.
1. Control connections faults
Please use “Troubleshooting Control Connections” article as a general help for this section.
This is example of the complex misconfiguration consisting of multiple failures and will be
guided step by step.
1. Check control connections history and observe that cEdge is even not trying to
establish control connections:
BR3-CEDGE1#show sdwan control connections
BR3-CEDGE1#
Even “debug platform software sdwan vdaemon misc high” won’t give any debugging
output. Usually it indicates that some mandatory parameters/configuration are/is missing
from the cEdge device.
2. Check local-properties:
BR3-CEDGE1#show sdwan control local-properties
personality vedge
sp-organization-name Cisco Sy1 - 19968
organization-name
root-ca-chain-status Installed
certificate-status Not-Installed
certificate-validity Not Applicable
certificate-not-valid-before Not Applicable
certificate-not-valid-after Not Applicable
dns-name vbond.cisco.com
site-id 500
domain-id 1
protocol dtls
tls-port 0
system-ip 10.5.0.1
chassis-num/unique-id CSR-3299b46e-90cd-4c5b-bebb-b6babfd8e0b1
serial-num No certificate installed
keygen-interval 1:00:00:00
retry-interval 0:00:00:18
no-activity-exp-interval 0:00:00:12
dns-cache-ttl 0:00:02:00
port-hopped FALSE
time-since-last-port-hop 0:00:00:00
number-vbond-peers 0
number-active-wan-interfaces 3
Please note that “organization-name” is missing from the configuration, set it and check
control connections again:
BR3-CEDGE1#config-transaction
certificate-status Not-Installed
certificate-validity Not Applicable
certificate-not-valid-before Not Applicable
certificate-not-valid-after Not Applicable
dns-name vbond.cisco.com
site-id 500
domain-id 1
protocol dtls
tls-port 0
system-ip 10.5.0.1
chassis-num/unique-id CSR-3299b46e-90cd-4c5b-bebb-b6babfd8e0b1
serial-num No certificate installed
keygen-interval 1:00:00:00
retry-interval 0:00:00:18
no-activity-exp-interval 0:00:00:12
dns-cache-ttl 0:00:02:00
port-hopped FALSE
time-since-last-port-hop 0:00:00:00
number-vbond-peers 0
number-active-wan-interfaces 3
4. If you check control connections again now, you’ll notice that situation has not
improved, and we are still seen same as on STEP 1 – cEdge is not trying to initiate any
control connections.
Let’s make sure that we have root-cert-chain is installed and certificate of the device
is installed, based on the below output from the cEdge (BR3-CEDGE1) we can see that
device certificate is not installed. We need to install the device certificate in order to
make control connection established.
BR3-CEDGE1#show sdwan control local-properties
personality vedge
sp-organization-name Cisco Sy1 - 19968
organization-name
root-ca-chain-status Installed
certificate-status Not-Installed
certificate-validity Not Applicable
certificate-not-valid-before Not Applicable
certificate-not-valid-after Not Applicable
dns-name vbond.cisco.com
site-id 500
domain-id 1
protocol dtls
tls-port 0
system-ip 10.5.0.1
chassis-num/unique-id CSR-3299b46e-90cd-4c5b-bebb-b6babfd8e0b1
serial-num No certificate installed
keygen-interval 1:00:00:00
retry-interval 0:00:00:18
no-activity-exp-interval 0:00:00:12
dns-cache-ttl 0:00:02:00
port-hopped FALSE
time-since-last-port-hop 0:00:00:00
number-vbond-peers 0
number-active-wan-interfaces 3
Note: Since we don’t have spare license on vManage for vEdge -cloud device, we need to
decommission existing device. By doing so, we will lose all template variables values, so it’s
good idea to backup them first.
Go to the vManage https://198.18.1.10 admin/admin
To save template variables values, follow vManage Path: Configuration -> Devices
For corresponding chassis number click on “…” and choose “Change Device Values”
BR3-CEDGE1#show sdwan control local-properties
...
chassis-num/unique-id CSR-3299b46e-90cd-4c5b-bebb-b6babfd8e0b1
...
Click arrow down button to export device template variables, also remember the template
name while saving values.
If you forget to save the template, there is a copy located in Downloads folder you can use to
return device into vManage mode from CLI mode:
Once you exported device template values, navigate to “Devices” page of vManage, find
corresponding chassis number and click “Decommission WAN Edge”:
Once the device is decommissioned vManage will prompt the message like below:
5. We need to generate the Bootstrap configuration again after the cEdge device is
decommissioned. We can click “…” and there is an option “Generate Bootstrap
Configuration” and after that select “Cloud-Init” format this will generate One Time
Password (OTP) for that chassis number like below:
6. Now we need to reactivate cEdge using chassis number and token received on
previous step using command “request platform software sdwan vedge_cloud
activate chassis-number <uuid> token <otp>”. cEdge will finally establish control
connections:
PEER PEER
CONTROLLER
PEER PEER PEER SITE DOMAIN PEER
PRIV PEER PUB
GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP
PORT PUBLIC IP PORT LOCAL C OLOR PROXY
STATE UPTIME ID
---------------------------------------------------------------------------
---------------------------------------------- ----------------------------
-----------------------------------
vsmart dtls 12.12.12.12 10 1 198.18.1.12
12446 198.18.1.12 12446 biz -internet
up 0:00:01:15 0
vsmart dtls 22.22.22.22 20 1 198.18.1.22
12446 198.18.1.22 12446 biz -internet
up 0:00:01:15 0
vmanage dtls 10.10.10.10 10 0 198.18.1.10
12446 198.18.1.10 12446 biz -internet
up 0:00:01:15 0
7. Reattach the template “BranchType3Template-CSR” to the device and import device
variables from the CSV file saved in Step 2.
Navigate to “Templates” page find “BranchType3Template” click “…” and choose
“Attach Devices”:
“Choose File”:
When you choose the file click “Open” and then “Upload”:
“Config Diff” button to see difference from current config. Now you can click
“Configure Device”:
Now you will go through Config Validation->Updating Config and when it is finished
status will change to “Success”:
1.2 Control connections issue on DC2-VEDGE2
DC2-VEDGE2 can’t establish control place connectivity with controllers due to some issues
with certificate.
This task is not guided as contrast to previous one, try to find fault yourself and fix the
problem. If you experience difficulties and spent more than 15 minutes on this task, please
refer to solutions.
Try to fix the issue using same techniques as described in 1.1. There are two problems. If
you spot difficulties, refer to the solution.
LOCAL
CFG V
PEER SITE DOMAIN LOCAL PRIVATE PUBLIC
REMOTE PRIVATE CONTROLLER SYSTEM ORG
BEHIND
INSTANCE TYPE ID ID PRIVATE IP PORT PUBLIC IP PORT
SYSTEM IP PROTOCOL LOCAL COLOR COLOR PRIVATE IP PORT STATE
UPTIME GROUP ID IP NAME PROXY
---------------------------------------------------------------------------
---------------------------------------------------------------------------
-------------------------------------------------
0 vsmart 10 1 10.2.254.2 12366 198.18.1.12 12446
12.12.12.12 dtls biz-internet default 198.18.1.12 12446 up
1:01:38:48 0 - - No
0 vsmart 20 1 10.2.254.2 12366 198.18.1.22 12446
22.22.22.22 dtls biz-internet default 198.18.1.22 12446 up
1:01:38:48 0 - - No
0 vbond 0 0 100.64.0.10 12426 198.18.1.11 12346
0.0.0.0 dtls mpls mpls 198.18.1.11 12346 connect
- 0 - - -
0 vmanage 10 0 10.2.254.2 12366 198.18.1.10 12446
10.10.10.10 dtls biz-internet default 198.18.1.10 12446 up
1:01:38:48 0 - - No
DC2-VEDGE1#
Try to figure out the reason for that behavior yourself, but this lab task is guided.
1. To be able to establish data place tunnel, Edge router should comply some
conditions. One of the most important is to have appropriate TLOCs announced and
received from the vSmart controller(s).
2. From the output below we can see that we have OMP sessions established with both
the vSmarts:
DC2-VEDGE1# show omp peers
R -> routes received
I -> routes installed
S -> routes sent
DC2-VEDGE1#
3. If we check the TLOCs information which we received from the vSmart, we can see
that we only have local TLOC information available and we did not install remote
TLOCs and service routes in SD-WAN router.
DC2-VEDGE1# show omp tlocs
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Stg -> staged
Inv -> invalid
PUBLIC PRIVATE
PSEUDO
PUBLIC PRIVATE PUBLIC IPV6 PRIVATE IPV6 BFD
TLOC IP COLOR ENCAP FROM PEER STATUS KEY
PUBLIC IP PORT PRIVATE IP PORT IPV6 PORT IPV6
PORT STATUS
---------------------------------------------------------------------------
---------------------------------------------------------------------------
-------------------
10.2.0.1 biz-internet ipsec 0.0.0.0 C,Red,R 1
100.64.2.14 12366 10.2.254.2 12366 :: 0 ::
0 up
As we can see above, only locally originated TLOC is seen and we don’t see any TLOCs from
remote sites.
One of the possible reasons is the device certificate in the “Staging” state. As we know that
when a SD-WAN router is in staging state, the vSmart controller learns about the router and
learns routes from the router, but it does not advertise these routes to any other WAN Edge
router in the network. During the staging state, OMP does not send any routes, data policies,
or TLOCs to the WAN Edge router, hence no BFD tunnels are established.
Now, we can see TLOCs received and data plane tunnels established:
PUBLIC PRIVATE
ADDRESS PSEUDO
PUBLIC PRIVATE PUBLIC IPV6 PRIVATE IPV6 BFD
FAMILY TLOC IP COLOR ENCAP FROM PEER STATUS KEY PUBLIC
IP PORT PRIVATE IP PORT IPV6 PORT IPV6 PORT STATUS
---------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
ipv4 10.1.0.1 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.26 12346 100.64.2.26 12346 :: 0 :: 0 up
22.22.22.22 C,R 1
100.64.2.26 12346 100.64.2.26 12346 :: 0 :: 0 up
10.1.0.2 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.30 12406 100.64.2.30 12406 :: 0 :: 0 up
22.22.22.22 C,R 1
100.64.2.30 12406 100.64.2.30 12406 :: 0 :: 0 up
10.2.0.1 biz-internet ipsec 0.0.0.0 C,Red,R 1
100.64.2.14 12366 10.2.254.2 12366 :: 0 :: 0 up
10.2.0.2 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.14 18628 10.2.254.6 12366 :: 0 :: 0 down
22.22.22.22 C,R 1
100.64.2.14 18628 10.2.254.6 12366 :: 0 :: 0 down
10.3.0.1 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.2 5062 100.64.2.38 12386 :: 0 :: 0 down
22.22.22.22 C,R 1
100.64.2.2 5062 100.64.2.38 12386 :: 0 :: 0 down
10.3.0.2 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.2 12346 100.64.2.2 12346 :: 0 :: 0 down
22.22.22.22 C,R 1
100.64.2.2 12346 100.64.2.2 12346 :: 0 :: 0 down
10.4.0.1 biz-internet ipsec 12.12.12.12 C,I,R 1
100.64.2.6 12366 100.64.2.6 Aborted: by user
DC2-VEDGE1#
N.B. From here it might be good idea to jump to Section 3.1 and fix it, so all the data tunnels
are established.
Try to figure out what’s the problem and make sure BR2-VEDGE1 can establish data plane
tunnels with other routers having TLOC “biz-internet”.
Note: On MPLS-PE router you can find BGP neighbor is shut downed. You don’t need to fix
this!
Try to figure out what’s the reason yourself. Refer to solutions workbook if needed.
3. Centralized control policy faults
Try to figure out what is the reason and refer to solutions workbook if needed.
Hint: The problem is caused by some mistakes in addressing. If issue is fixed, you should be
able to run traceroute from vManage:
Note: During attempt to get access to the Troubleshooting dashboard, you might see error
message regarding Data Stream activation requirement. Go to Administration -> Settings ->
Data Stream -> Enable. As hostname use System IP of vManage: 10.10.10.10, as VPN – 0.
3.3 Redundancy problems.
All Branches have requirement to redirect PCI VPN traffic to DC and have MPLS circuit as a
preferred path. It was reported that problem with connectivity from PCI VPN exists when
branches can't reach to DC1-VEDGE1 via MPLS color as was discovered during last
maintenance window when DC1-VEDGE1 was down for upgrade.
Hint: This problem is caused by control policy issues with “set tloc-list” action.
4. Data policy issues
Hint: from the section title we can guess that the problem is caused by some data policy
issues. Try to figure out what’s the problem.
5 Bonus. Other troubles.