Router Guide

AT&T VPN Service
Customer Router Configuration

Guide
Release 2.4
January, 2015
ESC Release 2.4
AT&T Proprietary For Use by AT&T Customers

2015 AT&T Intellectual Property. All rights reserved.
Technical Assistance
This is an AT&T proprietary document developed for use by AT&T customers. For additional
technical assistance contact your AT&T sales team. (This document was prepared by the AT&T
Enterprise Solution Consulting Division.)
Legal Disclaimer
This document does not constitute a contract between AT&T and a customer and may be withdrawn or
changed by AT&T at any time without notice. Any contractual relationship between AT&T and a
customer is contingent upon AT&T and a customer entering into a written agreement signed by
authorized representatives of both parties and which sets forth the applicable prices, terms and
conditions relating to specified AT&T products and services, and/or, to the extent required by law,
AT&T filing a tariff with federal and/or state regulatory agencies and such tariff becoming effective.
Such contract and/or tariff, as applicable, will be the sole agreement between the parties and will
supersede all prior agreements, proposals, representations, statements or understandings, whether
written or oral, between the parties relating to the subject matter of such contract and/or tariff.
ESC Release 2.4

Table of Contents
CHANGES FROM PREVIOUS RELEASES ...............................................................................4
Changes from Release 2.3 to 2.4 .................................................................................................... 4
Changes from Release 2.2 to 2.3 .................................................................................................... 4
1
INTRODUCTION ...............................................................................................................5
1.1 Access Types ...................................................................................................................... 5
1.2 Features............................................................................................................................... 6
1.3 Reference Network and Conventions ................................................................................... 6
2
IP ADDRESSING REQUIREMENTS .................................................................................8
3
CONFIGURING INTERFACES ..........................................................................................9
3.1 PPP ................................................................................................................................... 11
3.2 BFD, Bidirection Forwarding Detection ............................................................................ 14
3.3 MLPPP for Multiple T1 or E1s .......................................................................................... 15
3.4 Frame Relay Encapsulation over Dedicated Access (T1, T3, OC-x) ................................... 18
3.5 MLFRMulti-Link Frame Relay with Unilink (Multiple VPNs) ....................................... 20
3.6 Frame Relay Access .......................................................................................................... 22
3.7 ATM Access ..................................................................................................................... 23
4
STATIC AND BGP ROUTING BASICS ...........................................................................28
4.1 Routing Basics, BGP and Static ......................................................................................... 28
4.2 Routing Configurations ..................................................................................................... 34
4.3 IGP Considerations............................................................................................................ 40
5
LOAD SHARING WITH BGP ..........................................................................................44
5.1 Outbound Load Sharing..................................................................................................... 45
5.2 Inbound Load sharing ........................................................................................................ 45
5.3 Case Study: Load Sharing the Default Route ..................................................................... 50
5.4 Case Study: Load Sharing The Default Route and No Backdoor ........................................ 53
6
ROUTE SELECTION .......................................................................................................55
7
REDUNDANCY AND FAILURE RECOVERY..................................................................56
7.1 AVPN Diversity Options, Domestic U.S. ........................................................................... 57
7.2 AT&T VPN Diversity Options in Most of World (MOW) ................................................... 61
7.3 Site Disaster Recovery....................................................................................................... 61
7.4 Dual Hub Site Designs ...................................................................................................... 64
8
MIGRATION AND HYBRID NETWORKS .......................................................................65
8.1 Example Topology ............................................................................................................ 65
8.2 Policy Based Routing ........................................................................................................ 69
8.3 AT&T Managed Sites Interoperating with Customer Managed Sites .................................. 71
9
INTERNATIONAL CONSIDERATIONS...........................................................................72
9.1 NNIs with Partners to Extend Reach .................................................................................. 73
9.2 Using BGP Community Values to Have AT&T Set Local Preference ................................ 73
9.3 Inter-Region Route Filtering .............................................................................................. 74
9.4 Backup Paths..................................................................................................................... 76
9.5 Load Sharing ..................................................................................................................... 78
10 AT&T NETBOND, IAAS (INFRASTRUCTURE AS A SERVICE) ....................................80
10.1 IP Routing ......................................................................................................................... 81
10.2 Mobiilty CCS to NetBond ................................................................................................. 86
10.3 COS .................................................................................................................................. 86
11 AT&T NETBOND WITH SAAS PROVIDERS ..................................................................88
12 MOBILITY CCS (COMMERCIAL CONNECTIVITY SERVICE) ........................................89
12.1 New CCS to AVPN Service Architecture Overview .......................................................... 89
ESC Release 2.4

12.2 Routing between AVPN and CCS endpoints ...................................................................... 90

12.3 Use of the Default Route or Summary Route ..................................................................... 90
12.4 CCS with AVPN type-converted VPNs ............................................................................. 91
13 ALTERNATIVE ACCESS ................................................................................................92
13.1 ANIRA, Broadband Internet .............................................................................................. 92
13.2 DSL .................................................................................................................................. 93
14 OTHER CONSIDERATIONS WITH AVPN SERVICE ......................................................94
14.1 Testing (Pings, ICMP messages, and Route Convergence) ................................................. 94
14.2 Primary-Backup via BGP Attributes (MED, Local Preference and Community
Values) 95
14.3 BGP Timers ...................................................................................................................... 97
14.4 VPN Route Limit .............................................................................................................. 97
14.5 Route Dampening.............................................................................................................. 98
14.6 Broadcast, Directed Broadcast, Bridging, Auto-Install and CDP ........................................ 98
14.7 GRE Tunnels ..................................................................................................................... 98
14.8 Remote Site Reach-Ability Status .................................................................................... 100
14.9 Restrictive Route Filtering (Default Route Only) ............................................................. 101
14.10 Outbound Route Filtering (ORF) ..................................................................................... 101
14.11 Network Based Firewall (NBFW) .................................................................................... 102
14.12 MTU ............................................................................................................................... 103
14.13 Trace-Route .................................................................................................................... 104
APPENDIX A: BGP PATH SELECTION CRITERIA ..............................................................105
APPENDIX B: ATM IMA CONFIGURATIONS FOR 26XM, 3660, 37XX ROUTERS .............106
APPENDIX C: DS3/E3 FRAME RELAY CONFIGURATIONS ...............................................109
APPENDIX D: DIAL BACKUP ..............................................................................................111
APPENDIX E: ACRONYMS & DEFINITIONS........................................................................121
APPENDIX F: ADDITIONAL RESOURCES ..........................................................................123
Changes From Previous Releases

Changes from Release 2.3 to 2.4
AT&T NetBond added (Section 11)
Mobility CCS LTE added (Section 12)
Orderable MTU (Section 14.13)
MLFR, multi-link frame relay for NxT1 access for multiple logical channels to multiple VPNs
(Unilink), (Section 3.5)
ORF limitation update (Section 14.10)
Changes from Release 2.2 to 2.3
Updated the introduction and the interface recommendations

BFD for dedicated ports (IP ports)
Limiting control traffic with PERs (i.e. ping, ICMP)
Miscellaneous updates and corrections
ESC Release 2.4

1 Introduction
This document has been created to assist AT&T customers configuring their routers connecting to the
AT&T VPN (AVPN) service. This service uses MPLS technology in the AT&T network to provide
private IP routed transport for customers. The customer can have AT&T manage the routers for any
customer site in the VPN or manage the router themselves. This document focuses on the customer
managing the router. For information on AT&T managed router capabilities see your sales representative.
AVPN is, at its most basic, a private, IP routed network running on an AT&T MPLS backbone. The
customers virtual private network (VPN) provides full mesh, any-to-any connectivityevery IP route
that the customer advertises is forwarded on to every other customer site according to industry standard
BGP rules. AVPN also has a hub-spoke VPN restricted topology option.
This guide is written for AVPN service but much of it also applies to IPFR/IPATM service. Both services
use the same network infrastructure with some new additions for AVPN that are not available in IPFR
such as PPP, MLPPP and ethernet ports, and inter-region MPLS Option C (using MPeBGP), replacing
Option A IPVCs (see the International Section 10).
These guides are not a product or feature availability guidethey are intended to assist customers in
configuring routers. For product or feature availability, please see the AVPN and IPFR service guides.
There are other customer guides, listed in APPENDIX F: Additional Resources (on topics such as COS,
multicast, ethernet access, route groups, PNT-AVPN service interoperability, VoIP, mobility access,
ADSL access, troubleshooting, telepresence and migration).
The concepts contained in this document need to be tailored to each customer's specific environment.
While several basic routing environments and configurations are addressed, this document is not intended
to be exhaustive. If assistance is needed with more complex scenarios, contact AT&T technical experts
through the AT&T sales team.
AT&T consulting services are also available for a fee to assist with on-site detailed design, configuration
and implementation.
1.1
Access Types
AVPN offers several choices for access and several choices for the layer 2 protocol:
Ethernet. This is addressed in a separate guide: AT&T VPN Service, Ethernet Access Customer
Router Configuration Guide.
Dedicated access circuit with one of the following layer 2 protocols:
PPP
MLPPP (for bonding multiple T1s)
Frame relay encapsulation (for supporting multiple logical channels to multiple VPNs)
MLFR (for bonding multiple T1s and multiple logical channels for multiple VPNs)
Sub-rate DS3, OC3 and OC12 ports
Mobility (See Section 12)
Broadband (See Section 13)
Frame Relay Service (CIR configured to the maximum value and no PVC pricing element). Even
though frame relay and ATM are not referred to as dedicated access, the access network is
engineered to support full port bandwidth on access all the way to the PER.
ATM Service, including IMA (inverse multiplexing with ATM, multipleT1s)
ESC Release 2.4

1.2
Features
AVPN offers many features and capabilities, a few of which are listed below:
Unilink. This feature allows multiple layer 2 logical channels on the same physical port; each
logical channel terminating into a separate VPN on the PER. Unilink is available with ethernet
access and dedicated access using frame encapsulation.
IP multicast (as well as unicast). Please reference AT&T MPLS Service Multicast Customer
Router Configuration Guide.
Route Groups, which is the ability to have a different routing policy (preference for routes) for
different groups of sites. Please reference AT&T VPN Route Groups Customer Guide For
Internet Gateway Selection
Hub-Spoke VPN, which has a restricted topology: spoke sites can only route to hub sites; hub
sites can route to all sites. For a complete description and case study of hub-spoke VPNs, see:
ATT MPLS VPN Routing Design Case Studies 16: Hub-Spoke VPNs.
COS, class of service. See AT&T Network-Based Class of Service Features which a guide
created to assist AT&T customers with configuring their routers for class-of-service to match
AT&T CoS capabilities.
Network Based FireWall (NBFW). See Section 14.12.
See APPENDIX F: Additional Resources for a list of other configuration guides.
1.3
Reference Network and Conventions
Figure 1 illustrates the basic network topology that will be used throughout this document. Note that
AT&T refers to routers RTA, RTB and RTC., as "customer edge routers" (CERs) in AT&T's VPN
provisioning form. In a few select cases, the corresponding AT&T "provider edge router" (PER)
configuration will be illustrated as well. Note that the configurations contained in this document use
Cisco syntax 1. For router vendors other than Cisco, the concepts will be similar although the command
syntax will differ.
The command syntax has been provided to clarify the example configurations. Command syntax may vary by IOS
and current Cisco documentation should be referenced for specific features. The configuration examples should also
provide a conceptual basis for other vendor implementations.
ESC Release 2.4

Figure 1 Reference Network Topology
The IP numbering scheme used throughout the document (which has been arbitrarily selected) assumes
that the VPN interface addresses (the CER-PER links) share the same network range as the rest of the
addresses used at a site. For example, RTA's CER-PER subnet (192.168.10.0/30) is taken out of a block
used for Site A, (e.g. 192.168.0.0/16). This approach is only a suggestion and is not required. Another
approach is to take all CER-PER subnets out of the same block.
The service will automatically advertise all CER-PER link networks into the VPN, so the customer need
not advertise these addresses into the VPN.
The customer supplies the addressing (i.e., /30 subnets) for the CER-PER links for customer managed
routers. 2 See Section 3 on IP addressing requirements. (When AT&T manages the CER, AT&T provides
the CER-PER IP addresses.)
The configurations sometimes have a parameter as, for example, <ip address> to mean that the customer
should replace <ip address> with the real IP address that applies to their situation.
Note that these addresses can now come from the 10.0.0.0/16 range. They were once restricted.
ESC Release 2.4

2 IP Addressing Requirements
AT&T VPN service allows the customer to use any IP addressing scheme they wantincluding
RFC1918 private addressing, as long as all addresses are unique to the customer VPN 3, with a few
important exceptions:
1. The customer supplies the addressing for the CER-PER links (i.e., /30 subnets).
2. Customers should not use IP addresses from AT&T registered address space in their VPN (unless
AT&T specifically provided them for customer use). Some subnets from this space are used for
network management so they are advertised into VPNs but they are blocked from being advertised to
non-AT&T managed customer premises routers. For example, the following IP network blocks are
registered with AT&T and should not be used:
12.0.0.0/8
32.0.0.0/8
135.75.0.0/16 to 135.83.0.0/16 = 135.75.0.0/16, 135.76.0.0/14 and 135.80.0.0/14
135.89.0.0/16
165.87.0.0/16
192.231.9.0/24
199.37.0.0/16
If AT&T gives you an AT&T registered address to use, for example for VDNA or IP Flex service, then
you can use those in your VPN because those would not be the network management addresses
mentioned above.
The term "VPN" is used throughout this document to mean all sites and networks which are reachable via the VPN
network for a specific customer.
ESC Release 2.4

3 Configuring Interfaces
The first step in configuring a CER is to configure the interface for the link to AVPN. There are several
choices for the access type and the layer 2 protocol:
Ethernet Access (see the separate guide AT&T VPN Service, Ethernet Access Customer Router
Configuration Guide)
Dedicated access circuit to the PER (also called an IP port), with one of the following layer 2
protocols:
PPP for access to a single VPN
MLPPP for bonding multiple T1s for access to a single VPN
Frame relay encapsulation for access to multiple VPNs 4
MLFR for bonding multiple T1s for access to multiple VPNs
Frame Relay Access (phasing out)
ATM Access (phasing out)
Recommendations
Several factors will influence the choice of access type and layer-2 protocol. The primary factors to
consider are:
1. Availability
2. Reliability
3. Price
4. Speeds available (e.g., T1, NxT1, T3, OC-3, OC-12)
5. Access to a single or multiple VPNs (Unilink)
6. Feature support (e.g., CoS, IP multicast)
7. Access and node diversity requirements for each site
In general, AT&T makes the following recommendations:
A. Ethernet is AT&Ts direction
B. For lower speeds, dedicated T1 access with PPP and frame encap for unilink to multiple VPNs
may be preferred until there is a price competitive ethernet solution. To order unilink via frame
encap on lower speed dedicate access one must order the port as dual stack (IPv4 and IPv6) even
if not using IPv6.
C. Where ethernet is not available, use dedicated access with PPP or frame encap for unilink to
multiple VPNs. For unilink, a dedicated access circuit with frame encapsulation may be the best
choice if you want the COS policy applied on the main port and not on each logical channel. See
the Network Services Class-of-Service Customer Router Configuration Guide for more details.
D. For high speed (DS3, OC-x) access to a single VPN, dedicated access circuit with PPP or frame
encap is also available. For unilink, a dedicated access circuit with frame encapsulation may be
the best choice if you want the COS policy applied on the main port and not on each logical
4
When using frame relay encapsulation we mean that the layer 2 protocol is frame relay however the physical
access still terminates directly on the PER. The purpose of frame relay encapsulation is to allow multiple logical
channels, each associated with a different VPN. (AT&T refers to this feature as Unilink.) Unilink does not allow
the use of a logical channel to AT&T Managed Internet Service.
ESC Release 2.4

channel. See the Network Services Class-of-Service Customer Router Configuration Guide for
more details.
E. For NxT1 speeds, ethernet is preferred but if not available or price competitive, MLPPP and
MLFR are options. MLFR is for NxT1 with frame encap for unilink to multiple VPNs.
Ethernet access is described in detail in a separate document. If you use ethernet we recommend you
order bidirectional forwarding detection (BFD) to get a fast keep-alive function for detecting access
failures. Note the following with ethernet:
There is no layer 1 alarming to the CER and PER if there is a physical layer failure (hence the
recommendation to order BFD)
There is no layer 2 keepalives to inform the CER and PER of a failure

The access is a usually shared infrastructure and sometimes provided by a third party. The access
network is engineered to deliver the full VLAN speed (not necessarily the ethernet physical
speed).
There is more overhead with Ethernet access (42 bytes versus 6 with frame relay and PPP; even
ATM has less overhead for small packet sizes, like VoIP)
Some ethernet interfaces on routers are not capable of doing the necessary CoS (including traffic
shaping). While many platforms may have built in GE Interfaces, they may not be suitable for
WAN access at full GE (or even FE) rates especially when CBWFQ configurations or traffic
shaping is used for CoS support. Customers are encouraged to work with their equipment vendor
to determine the most appropriate edge platform when using Gigabit or Fast Ethernet WAN
access.
When using Frame Relay or ATM Access to AVPN, the virtual circuit terminates on the PER. Usually
the PER and layer 2 switch co-reside in the same AT&T central office. Even though frame relay and
ATM access are not referred to as dedicated access, the access network is engineered to support the full
port bandwidth on access all the way to the PER. Using Frame Relay or ATM access allows easy
migration from traditional frame relay or ATM service to AVPN, in addition to allowing a connection to
multiple VPNs. And, each virtual circuit to a VPN has its own COS profile. Frame relay and ATM access
generally does not decrease or increase performance compared to PPP. Although ATM has more packet
(cell) overhead (depending on packet sizes), ATM offers some features not offered on dedicated access.
IP routing concepts and configurations which are at the IP layer are the same regardless of the access
service or link layer deployed. Some sites in a VPN may use Frame Relay Service while other sites use
ATM Service or dedicated access with PPP at layer 2.
The term link is used to mean any of the access methods regardless of type or layer 2 protocol. On the
CER this represents the interface or sub-interface configured.
IP addresses for interfaces to AVPN are provided by the customer unless AT&T is managing the router.
When changing the encapsulation type, we have occasionally had to reboot the router to completely
clear the router of the old configuration state. This may only be prevalent if the interface is not first
shutdown.
Timing/Clocking
Physical interface timing (clocking) will be provided by AT&T for the network.
For customer routers using dual or multi-port interface cards, each interface should have the ability to
take loop-timing independently. Some interface cards supply only one oscillator for all interfaces (e.g.
Cisco VWIC-2MFT version 1). Single oscillator hardware is not recommended and cannot be supported
by AT&T for installations where more than one port is used on the card. Questions as to whether the
ESC Release 2.4

10
router WAN interfaces have the ability to support independent clocking should be directed to your router
provider.
Example configurations for each type of interface are provided next.
3.1
PPP
Below are configuration examples for PPP on various physical interface types on a Cisco router.
DS1 PPP Interface
interface Serial0/1
ip address 192.168.10.1 255.255.255.252
service-module t1 timeslots 1-24
encapsulation ppp
no peer neighbor-route 5
no cdp enable
! Change timeslot range for fractional T1 access
! Default Cisco settings provide clock source line, DS1 ESF framing, and B8ZS line coding. 6
Note that E1 interfaces in Europe are typically delivered as 2048kps unframed G703/RJ48.
DS3 PPP Interface, U.S.
interface Serial3/0
ip address 192.168.10.1 255.255.255.252
encapsulation ppp
no peer neighbor-route
no cdp enable
framing c-bit
scramble
dsu bandwidth 44210
! note that E3 PPP ports should use no

scrambling
! 44210kbps is used for full DS3 and is the
default
CRC16 is the default on DS3. The customer can ask for CRC32 during test and turn-up.
Scrambling is on by default for all dedicated access DS3 ports (whether PPP or frame encap, including
sub-rate speeds).
5
Cisco routers automatically create a host route (/32) to the peer address on a point-to-point interface configured
with PPP. The no peer-neighbor route statement removes the host route. A no peer-neighbor route statement is
configured on PE router customer interfaces to avoid the redistribution of PPP host routes into the VPN. Using this
statement does not impact the connected /30 route for the CE-PE interface which is automatically redistributed in the
VPN.
6
The following are Cisco default settings for a T1: framing= extended super frame (ESF), Line build-out (LBO) =
short 133, Linecode = B8ZS, Facilities Data Link (FDL) = ansi, Yellow = enabled, though the defaults may vary by
IOS. These are acceptable in most cases; however, the best setting for FDL is both, if available, or next best is
ATT but ANSI will also work. The ATT standard allows AT&T maintenance to retrieve from the CSU function
the most data for troubleshooting.
ESC Release 2.4

11
Note that we have had some customers with problems bringing up the PPP interface when running AAA
authentication/authorization, for example:
aaa authorization network default group tacacs+ if-authenticated .
This will run authorization on network links, including the PPP link between PE and CE. As such, PPP
will not come up. If you have this problem, configure:
no aaa authorization network default
E3 and DS3 PPP Interfaces Outside the U.S.
There are a few differences for E3 and DS3 PPP interfaces outside the U.S. compared to DS3 PPP in the
U.S.
Scrambling should be disabled. There are some PER Juniper cards that support scrambling but
some do not, so the default is no scrambling.
DSU compatibility mode should be set to kentrox. Note, there is the possibility that your access
circuit will go on a PER card type that does not work with dsu mode 1 (kentrox). In this case you
will need to set it to dsu mode 0. If you ordered 6COS you should use dsu mode 1 (kentrox).
Here are some example configurations.

NM-1T3/E3 module on an ISR or a PA-E3 on a 7200 with PPP:
card type e3 2
interface Serial2/0
ip address 10.20.30.2 255.255.255.252
encapsulation ppp
dsu bandwidth 34010
dsu mode 1
! 1 is Kentrox
SPA-2xT3/E3 in a 7304, 7600, or ASR with frame encapsulation:

card type e3 5 1
interface Serial5/1/1
no ip address
encapsulation frame-relay IETF
framing g751
dsu mode kentrox
frame-relay lmi-type ansi
!
interface Serial5/1/1.200 point-to-point
ip address 10.20.30.26 255.255.255.252
no cdp enable
frame-relay interface-dlci 200
Note that scrambling is not enabled (it is disabled by default).
ESC Release 2.4

12
OC3, OC12 and OC48 PPP SONET Interfaces

interface pos0/0/0
encapsulation ppp
no cdp enable
ip address 192.168.10.1 255.255.255.252
crc 16
crc 32
pos scramble-atm
! Default for OC3 VPN ports;

! Default for OC12 and OC48 AVPN ports
! Required, to match the PER. Not related to
ATM, despite the syntax.
One should also add the following command under OCx interfaces, to match the PER:
pos delay triggers path 2500
! delays declaring the interface down until after

receiving an alarm for 2.5 seconds. This helps
prevent flapping interfaces from causing
problems and matches the PER.
This command causes the interface to go down when it receives a path alarm indication signal (PAIS)
or a path remote defect indication (PRDI) on the receive fiber. These alarms indicate that the transmit
side of the circuit has failed. Without this command, if only the transmit side of the circuit fails, the CER
will not declare the interface down immediately (as the PER will; the CER will wait for layer 2 or layer 3
keepalives to fail).
Sub-Rate DS3, OC3 and OC12 Service with Dedicated Access Circuit
Sub-rate port refers to a port whose bandwidth is limited to something less than the physical speed of
DS3, OC3 or OC12. PPP (access to a single VPN) or frame relay encapsulation (access to multiple VPNs)
is used at layer 2.
The CER must shape the outgoing rate on the port to the purchased sub-rate speed to prevent packet loss
if the rate were to exceed the purchased rate. Here is an example configuration for shaping:
policy-map SHAPE
class class-default
shape average <subrate>
! shape all traffic on the port

! where <subrate> is the subrate speed
purchased in bps such as 5000000 bps,
10000000 bps, etc.
interface serial0
service-policy output SHAPE
See the COS guide: AT&T Network-Based Class of Service Features. for a complete example that
includes CBWFQ.
ESC Release 2.4

13
3.2
BFD, Bidirection Forwarding Detection
BFD is an IP level keep-alive mechanism that is primarily needed on ethernet because ethernet has no
layer 1 alarming. However, it can be useful on dedicated access with PPP or frame encap to get consistent
route convergence behavior.
The availability of BFD on IPv6 is limited so contact your sales team. On Cisco enterprise routers (for
CERs) it may only be in 15S IOS versions. Support for this feature came out in 15S release only, on
7200,7300, 7600 & ASR1000s starting 15.1(2)S release.
AT&T supports the following Tx and Rx values selected at the time BFD is ordered (one value selected
and that value is used for both Tx and Rx): 1, 2, and 3 (default) seconds (300 milliseconds may be
possible via a custom request). Although Tx and Rx values can be set independently of each other, AT&T
PE routers will be configured to support the identical Tx and Rx value. Note that current Cisco CE routers
only support values up to a maximum of 999 milliseconds. Therefore in almost all cases, the value
selected when BFD is ordered will be the value negotiated between peers and the CE configuration values
will have no effect (unless it is higher).
See AT&T VPN Service Ethernet Access Customer Router Configuration Guide for details on BFD.
Below is an example configuration on a dedicated access interface, on a Cisco 2801 Software Version
15.1(3)T4.
interface Serial0/3/0
ip address 216.100.233.133 255.255.255.252
encapsulation ppp
ipv6 address 2001:506:2004:67C::11F6:C0C1/64
bfd interval 999 min_rx 999 multiplier 3
!
router bgp 65075
bgp log-neighbor-changes
neighbor 216.100.233.134 remote-as 65236
neighbor 216.100.233.134 fall-over bfd
no auto-summary
ESC Release 2.4
!999 is in milliseconds.

14
show bfd neigh details

NeighAddr
LD/RD RH/RS State Int
216.100.233.134
1/-2145124351 Up
Up
Se0/3/0
Session state is UP and not using echo function.
OurAddr: 216.100.233.133
Local Diag: 0, Demand mode: 0, Poll bit: 0
MinTxInt: 999000, MinRxInt: 999000, Multiplier: 3
Received MinRxInt: 9990000, Received Multiplier: 3
Holddown (hits): 7276(0), Hello (hits): 3000(151)
Rx Count: 145, Rx Interval (ms) min/max/avg: 1/2996/2741 last: 1724 ms ago
Tx Count: 151, Tx Interval (ms) min/max/avg: 2260/2996/2643 last: 48 ms ago
Elapsed time watermarks: 0 0 (last: 0)
Registered protocols: BGP
Uptime: 00:06:36
Last packet: Version: 1
- Diagnostic: 0
State bit: Up
- Demand bit: 0
Poll bit: 0
- Final bit: 0
Multiplier: 3
- Length: 24
My Discr.: -2145124351
- Your Discr.: 1
Min tx interval: 9990000 - Min rx interval: 9990000
Min Echo interval: 0
Note that testing this is a challenge because removing the cable from the router or removing the IP
address from the interface causes the interface to drop and BGP to drop as fast or faster than BFD. One
possible solution is to set carrier delay higher, such as 5 seconds and set BFD timers to 1:3. Beware that
carrier delay can behave in unexpected ways, depending on the value and the IOS and the port type.
3.3
MLPPP for Multiple T1 or E1s
Multilink Point-to-Point (MLPPP) provides link aggregation for multiple T1s (between two and eight
T1s). MLPPP sends packets between the CER and PER across the individual links in a round robin
fashion. Only one network address needs to be configured for the entire NxT1 bundle since MLPPP
works at the link layer creating one logical link to the upper layer protocols in the router. MLPPP will
automatically detect the failure of an individual link and will remove that link from the remaining T1s in
the bundle.
In countries where E1 is used, MLPPP uses 2048kbps E1s (unframed). Customers should ensure that
their hardware supports unframed E1.
MLPPP keeps track of packet sequencing and buffers packets that arrive early to preserve packet order
across the entire bundle. For MLPPP to work properly, the maximum delay differential allowed between
T1s is 30 to 100 milliseconds depending on the CER and PER platforms. Please be aware that MLPPP
requires greater CPE CPU processing than other access solutions as packet reordering and the MLPPP
protocol itself increases the CPU load. Fragmentation is not supported with the MLPPP access option.
MPLS VPN service assigns an interface to a VPN. Since the MLPPP protocol has no concept of a layer 2
logical channel or sub-interface within the physical port, the MLPPP bundle is assigned to a single VPN.
Multiple VPNs on one interface (i.e., Unilink) are not possible.
ESC Release 2.4

15
Note that there is another AT&T feature called MLPPP LFI (link fragmentation and interleaving) which
is used with a single fractional T1 interface (768 kbps and less) to break each packet into smaller
fragments to improve latency for real-time (e.g., voice over IP) applications. This capability is
documented in a separate guide on configuring class-of-service.
ip cef
interface multilink <group-number>
ip address <address> <mask>
encapsulation ppp
no cdp enable
no bandwidth <BW in kbps>
ppp multilink
ppp chap hostname <hostname>
ppp multilink fragment disable
ppp multilink group <group-number>
ppp multilink links minimum <#T1s> mandatory
Interface serial x/0 8

encapsulation ppp
ppp multilink-group <group-number>
ppp multilink
no ip address
keepalive
Interface serial x/1
encapsulation ppp
ppp multilink
no ip address
keepalive
Interface serial x/2
encapsulation ppp
ppp multilink
no ip address
keepalive
! cef distributed for some hardware

! No keepalives are configured on the multilink
interface, only on the physical interfaces below
! do not use the bandwidth statement 7

! Required. See following paragraphs.
! or no ppp multilink fragmentation
! Must match multilink group number above
! Brings down MLPPP bundle if number of active
links drops below the minimum. Mandatory
keyword required. See paragraph below.
! Same group number used on multilink interface 9
! Enable keepalives
! Same group number used on multilink interface
! Same group number used on multilink interface
Bandwidth command on the MLPPP bundle should not be used because it may adversely affect QOS policies,
when a link fails, e.g. it cannot be used to guarantee 50% of original link bandwidth for the RT class if on T1 goes
down.
8
Clocking has on rare occasions caused problems when using VWIC-2MFT-T1 version one cards (not version 2).
This might happen only when the T1s have different clock sources (e.g. two different access providers).
9
The multilink-group command was replaced by the ppp multilink group command.
ESC Release 2.4

16
Do not change the MTU from the default of 1500. The PER will only accept an MTU of 1500 (not lower
or higher).
The CE router must uniquely name the packets it sends on the links in the MLPPP bundle to the PE so
that the PE can combine packets from the correct links. Cisco routers send two names to identify the
MLPPP bundle: the PPP username and an MLPPP endpoint discriminator. The router can uniquely
identify the links in a MLPPP bundle if either or both are unique. The ppp chap hostname command
sets the PPP username equal to the hostname. The endpoint discriminator defaults to the PPP username,
unless separately configured. This used to be required under each serial interface but that is no longer
needed, only needed on the multilink bundle interface.
By default, without the ppp chap hostname command, the PPP username is set to the global router
hostname and the endpoint discriminator is set to the PPP username. Because the customer router
hostnames may not be unique to the PE across multiple customers, this is not an acceptable approach.
AT&T recommends the following convention to provide a unique MLPPP name:
< Interface IP Address><Customer Name>
For example, the following command is used to set the PPP username for a customer named ABC
Plumbing Supply with a multilink interface IP address of 192.168.213.9:
ppp chap hostname 192.168.213.9ABCPlumbing
Warning: If you use the router hostname as the MLPPP bundle name (i.e. you dont directly specify the
bundle name) and you change the hostname, you have to bounce all the member interfaces and the bundle
(or a clean reload of the CER). If this procedure is not followed, any T1s that flap subsequently will not
return to the bundle. Cisco has installed a warning message to this affect, in some IOSs.
Note that characters beyond 20 are truncated (which is ok as long as the remaining name is unique on
the PER). Letters, spaces, numbers, periods and dashes are all valid characters. While individually the
customer name and IP address may not be unique, the combination of the two creates a high probability
that the resulting MLPPP full name will be unique to the PER.
There are no ordering/provisioning requirements associated with the use of the name or PPP CHAP
hostname. The customer just needs to configure the PPP CHAP hostname on the CER when the NxT1
MLPPP bundle is brought into service 10. The customer does not need to tell AT&T what this name is. The
PER also configures a name, which it will use when sending packets to the CER. The CER and PER full
names do not have to match nor does either router have to be statically configured to know the others full
name. They will be learned dynamically.
The ppp multilink links minimum <#T1s> mandatory command is optional. If not specified the
multilink interface stays up until all T1s fail. If a minimum is specified on the CER, the CER will bring
down the multilink interface when the number of active links falls below the configured minimum. This
may be desired when there is a high bandwidth alternate link at the site that should be used rather than
running with reduced bandwidth on the MLPPP connection. Note, however, that when the CER drops the
MLPPP bundle, the PER does not know to also drop the bundle. Once the BGP holddown timer expires,
the PER will tear down BGP and withdraw all routes. The CER can set the BGP holddown timer down to
45 seconds, which the PER will honor, to reduce the delay from the default of 180 seconds (reference
Section 14.4 on BGP timers).
10
Cisco has stated that the PPP multilink endpoint command should also work to create a unique name. It
provides the endpoint discriminator as opposed to the username, but makes the full name unique. AT&T has not
tested it.
ESC Release 2.4

17
A note on troubleshooting: to get a bundle working we had to shut down the multilink interface and all
serial interfaces followed by no shut on the serial interfaces and then no shut on the multilink
interface. This may have been related to the name change issue mentioned abovei.e., required whenever
the bundle name is changed.
3.4
Frame Relay Encapsulation over Dedicated Access (T1, T3, OC-x)
This method is very similar to PPP interfaces but uses the frame relay protocol at layer 2 to allow multiple
logical channels, each to a separate VPN. This is a feature called Unilink. It allows the customer to
connect to multiple VPNs with one physical port. Each virtual circuit is bound to a different VPN. Up to
twelve logical channels are allowed as standard.
Unilink may not be available on dedicated access at T1 and NxT1 and E3 speeds (it is available on
dedicated DS3 ports in the U.S.). Use frame relay or ATM access for unilink at these speeds. Please refer
to your sales team for the latest feature availability information.
AT&T recommends using LMI to verify L2 connectivity and the DLCIs configured on the port. These
LMI options are orderable:
None. No LMI keep-alive is exchanged between CE and PE. This is the default.
ANSI T1.617 Annex D, This is the AT&T default for Juniper PERs.
ITU Q933 Annex A
Cisco LMI is an option on a Cisco router to use the specification published by Cisco, Digital,
Stratacom and Northern Telecom called the gang of four. Not available on frame encapsulation
ports outside the U.S. (because the PER is a Juniper).
The customer chooses the DLCIs when ordering service, for example start numbering at 101 then 102,
103, etc. Sometimes AT&T references a port DLCI. This is not really a DLCI but just a port identifier
for AT&T administrative systems, not used for any router configurations.
Having multiple virtual circuits on a single port may introduce both security and performance issues.
Security issues are not covered in this document but AT&T does have case studies on separating VPN
traffic. For guidance on configuring class of service, reference the document: AT&T Network-Based
Class of Service Features. The customer chooses a COS profile which applies to the main port (not a
profile for each logical channel). COS does not care which logical channel a packet is on. Packets from
any LC could use the full port speedthere is no restriction or CIR for a given LC.
ESC Release 2.4

18
Frame Encapsulation Example on DS3 Dedicated Access

interface serial x/y:z
no ip address
no cdp enable
no ip directed-broadcast
no ip redirects
scramble
framing c-bit
dsu mode 0 [1]
! The default is no scrambling, which may not

show up in the config
! 0 is the default value and good in the U.S. 1
or kentrox needed outside the U.S.
! Default value. Should not need to configure
! CRC16 is configured by default by AT&T.
Customers can ask for CRC32 at turn-up
dsu bandwidth 44210

crc 16
clock source line
frame-relay lmi-type cisco
! Or, ANSI or Q933 or no keepalive for no

LMI. LMI is recommended.
interface serial x/y:z.101 point-to-point

ip address <ip address>
no cdp enable
no ip redirects
no ip proxy-arp
frame-relay interface-dlci 101 IETF
! assumes DLCI of 101
E3 and DS3 Dedicated Access Interfaces Outside U.S.

There are a few differences for E3 and DS3 dedicated access interfaces outside the U.S. compared to in
the U.S.
Scrambling should be disabled. There are some PER Juniper cards that support scrambling but
some do not, so the default is no scrambling.
DSU compatibility mode should be set to kentrox. Note, there is the possibility that your access
circuit will go on a PER card type that does not work with dsu mode 1 (kentrox). In this case you
will need to set it to dsu mode 0. If you ordered 6COS you should definitely use dsu mode 1
(kentrox).
Frame-relay LMI-type on dedicated frame encap ports does not have an option for Cisco
outside the U.S. It can be either ANSI Annex D or ITU Annex A. Th former is the AT&T default
unless ordered as ITU Annex A.
ESC Release 2.4

19
Frame Encapsulation Example on OCx and STM

interface posx/y:z
no ip address
no cdp enable
no ip redirects
pos framing sonet
clock source line
crc16
pos scramble-atm
bandwidth 155000
pos flag c2 22
interface serial x/y:z.101 point-to-point

ip address <ip address>
no cdp enable
no ip redirects
no ip proxy-arp
frame-relay interface-dlci 101 IETF
3.5
! Or, ANSI or Q933 or no keepalive for no

LMI. LMI is recommended.
! use sdh instead of sonet for STM circuits
! for OC3, 32 is possible (changed at turn-up);
crc32 for OC12 and higher
! Required, to match the PER. Not related to
ATM, despite the syntax.
! For STM1 for example
! May be required for some STM1 connections
to set the SONET overhead bytes in the frame
header to meet the specific standard. C2
identifies the payload content type (22=hex
0x16 is for scrambling enabled; the default is
0xCF)
! Sub-interface, assumes DCLI 101
MLFRMulti-Link Frame Relay with Unilink (Multiple VPNs)
For NxT1 speeds, ethernet is preferred but if not available or price competitive, MLPPP and MLFR are
options. MLFR is for NxT1 running the frame relay protocol at layer 2 instead of PPP. The purpose is to
get multiple logical channels to multiple VPNs on a single physical port. This feature in AVPN is called
unilink.
Having multiple logical channels to multiple VPNs on a single port may introduce both security and
performance issues. Security issues are not covered in this document, but AT&T does have case studies
on separating VPN traffic. For guidance on configuring class of service, reference the document: AT&T
Network-Based Class of Service Features. The customer chooses a COS profile which applies to the
main port (not a profile for each logical channel). COS does not care which logical channel a packet is on.
Packets from any LC could use the full port speedthere is no restriction or speed or CIR configured
for a logical channel.
ESC Release 2.4

20
Below is an example configuration for MLFR.

!
interface MFR1
no ip address
no keepalive
frame-relay multilink bid router1
!
interface MFR1.160 point-to-point
ip address <ipv4 address>
no ip redirects
no ip proxy-arp
ipv6 address <ipv6 address>
ipv6 enable
no cdp enable
!
interface MFR1.180 point-to-point
ip address <ipv4 address>
no ip redirects
no ip proxy-arp
ipv6 address <ipv6 address>
ipv6 enable
no cdp enable
!
interface Serial0/0/0:0
description ***MLFR bundled T-1***
no ip address
encapsulation frame-relay MFR1
no arp frame-relay
frame-relay multilink lid first-link
!
interface Serial0/0/1:0
description *** MLFR bundled T-1***
no ip address
encapsulation frame-relay MFR1
no arp frame-relay
frame-relay multilink lid second-link
!
! Configure a MLFR bundle interface
! Optional - assign a bundle id name to bundle

! Configure MLFR subinterface to VPN A
! IPv6 portion of dual stack config
! Configure MLFR subinterface to VPN B
! Associate MLFR links with bundle interface

! Optional - assign a bundle link id name
! Associate MLFR links with bundle interface

and must match the name of other bundled
interfaces.
! Optional - assign a bundle link id name
Cisco routers by default send MLFR keepalive packets every 10 seconds with a max retry of 2 before
declaring a connection down. These values can be adjusted on the MLFR interfaces using the following
commands: frame-relay multilink hello <sec> and frame-relay multilink retry <##>. If using the
default values, then these commands are hidden in the configuration as the case in the above example.
Please note: AT&T Provider Edge routers are configured for 10sec timers with max retry of 2.
Mismatch in the keepalive parameters will not impair the service but may cause the CE or PE to initiate
route reconvergence first.
ESC Release 2.4

21
3.6
Frame Relay Access
When using Frame Relay access, an ePVC is connected through an AT&T frame relay switch and then on
to the PER.
Unilink is available with frame relay access. Unilink allows the customer to connect to multiple VPNs
with one physical port. Each virtual circuit is bound to a different VPN. Reference the COS guide for
unilinks affect on COS.
An example configuration is shown below. Note the use of IETF frame-relay encapsulation, which is
required by the service to match the PER and is defined in RFC1490.
interface Serial 1
! AT&T Frame Relay physical port
description T1 Frame-Relay Port DHEC.xxxxxx
no ip address
! Must use IETF (RFC 1490) for VPN ePVCs;
! or ANSI or Q933 or autosense (the default) 11
logging event subif-link-status
! log failure events
logging event dlci-status-change
!
interface Serial 1.1 point-to-point
! Sub-interface for PVC (not multi-point)
description ePVC to AVPN
ip address 192.168.10.1 255.255.255.252
! /30 address
no cdp enable
! CDP should be disabled on CE-PE connections
! DLCI chosen to reach the VPN PER (referred to
as the local DLCI on the provisioning form)
One can also configure IETF at the sub-interface level only i.e., substituting the command shown above
with "frame-relay interface-dlci 777 ietf." This allows PVCs defined on different sub-interfaces to use
different encapsulation types if necessary. If this type of function is required, "encapsulation frame-relay"
must be used on the main serial interface. Note that changing the encapsulation type does not require one
to shut down and restart the sub-interface.
The sub-interface should be point-to-point.
E1 Circuit, Framed. If the access is framed E1 (1984kbps) rather than T1, with a built-in CSU, the
controller configuration is as shown below (the interface configurations are the same as with T1):
controller E1 0/0/0
channel-group 0 timeslots 1-31
! Change timeslot range for fractional E1 access
! Default Cisco settings provide clock source line, CRC4 framing, and HDB3 line coding.
11
cisco is the default used by the frame relay service. One can also use the ANSI or Q933 specifications (which
must also be specified in the AT&T ordered). If not set, Cisco will autosense with the frame relay switch. To assure
no difficulties with autosensing, hard code a type and match it to the frame relay order.
ESC Release 2.4

22
E1 Circuit, Unframed. (2048kbps):

controller E1 0/0/0
channel-group 0 unframed
! all timeslots are used for data
! Default Cisco settings provide clock source line, unframed, and HDB3 line coding.
Sub-Rate DS3 Frame Relay Service
DS3.
if the rate were to exceed the purchased rate. Shaping is accomplished with frame relay traffic shaping
feature in Cisco.
interface Serial0/0
frame-relay traffic-shaping
! This turns on traffic shaping for all

subinterfaces
interface Serial0/0.100 point-to-point

frame-relay class shapeDS3
map-class frame-relay shapeDS3
no frame-relay adaptive-shaping
frame-relay cir <subrate>
frame-relay bc <subrate/100>
frame-relay be 0
3.7
! This configures the specific parameters for this

PVC.
! For CIR set <subrate> to the subrate bps

purchased, e.g. 5000000, 10000000
! BC is set to about 1% of CIR to achieve a Tc
value of about 10ms; Bc/CIR=Tc
! no bursting allowed above the subrate port
speed purchased
ATM Access
When using ATM access, an ePVC is connected through an AT&T ATM switch and then on to the PER.
Unilink is available with ATM access. Unilink allows the customer to connect to multiple VPNs with
one physical port. Each virtual circuit is bound to a different VPN. Reference the COS guide for unilinks
affect on COS.
Both of the sample configurations shown below use UBR (unspecified bit rate) VCs, which is the Cisco
default VC type. This is the easiest approach since the customers ATM port is conceptually a lower
speed than AT&Ts VPN PER port, thus the customer does not need traffic shaping. However, if you
configure class-based weighted fair queuing, you may need to include a VBR-nrt statement with shaping,
which is described in AT&T Network-Based Class of Service Features, a guide by AT&T. Even though
the ATM VCs accessing AVPN are provisioned internally as VBR-nrt (Variable Bit Rate - non-real
time), there is no requirement that the customer router be configured the same as the network. In most
cases, either UBR or VBR-nrt can be configured in the CER without impacting the actual cell emission
rate from the CER to PER. In addition, using UBR simplifies the CER configuration by avoiding having
to get into the details of VBR-nrt parameter settings and the associated impact on CPU and buffer
overhead on the CER.
ESC Release 2.4

23
Sub-Rate DS3 ATM Service

DS3.
if the rate were to exceed the purchased rate. For ATM, traffic shaping is built into the configuration of
VBR-nrt VCs. The sub-rate speed purchased is like a physical port speed so it must carry all bits
transmitted, including ATM overhead. On Cisco routers the VBR-nrt parameters PCR and SCR count the
bits including the ATM headers, AAL5, etc, so the PCR and SCR should both be set to the sub-rate
purchased. IP throughput will be a little less than 48/53rds of this rate since an ATM cell payload is 48
bytes out of the 53 bytes cell and AAL5SNAP header and trailer are carried in the 48 byte ATM payload
of the first and last cell for the packet.
interface ATM0/0.800 point-to-point
pvc 0/800
vbr-nrt <subrate> <subrate> 32
! Use the sub-rate port speed for PCR and SCR

in kbps, such as 5000, 10000, etc. (not cell
rate), MBS is in cells but has no effect when
PCR=SCR
Recommendations Using OAM

AT&T and Cisco have seen occasional bugs with OAM operation in switches and routers that has led to
confusion over what works most effectively and what customers should configure on their routers. Based
on these issues, AT&T recommends the following.
"oam-pvc manage 0 which means react to AIS but the 0 means no F5 keepalives
"oam ais-rdi 10 3" which means that if the router receives ten AIS or RDI, it will take the PVC
down. If the router doesn't receive an AIS/RDI for 3 seconds, it declares the PVC back up.
Note that PERs use "oam-pvc manage 0" thus no keepalives are sent on the ePVC. AT&T PERs typically
uses the equivalent of "oam ais-rdi 10 3", so the PER will not declare an ePVC down until 10 seconds of
AIS alarms are received. This avoids issues where a circuit could take a short (1-2 second) hit and declare
the ePVC down, which would could cause the PER to reset the BGP session with the CER.
Sample Cisco 3640 Router Configuration
The configuration shown below is for a DS3 ATM interface on a Cisco 3640 router running IOS 12.2.10.
Note: For this IOS, the DS3 framing method defaults to ATM Direct Mode (cbitadm). If the port was
ordered as PLCP, this option should be changed using the command: "atm framing cbitplcp."
ESC Release 2.4

24
interface ATM1/0
description ** AT&T DS3 ATM Port: DNECxxxxxx, Global DLCI-40 **
no ip address
atm scrambling cell-payload
! Scrambling is enabled by default in recent IOS.
Must match AT&T circuit options
no atm ilmi-keepalive
!
interface ATM1/0.1 point-to-point
! Syntax for creating sub-interface .1 on ATM1
description ** To AT&T PER - VPI/VCI 1/777 ** ! Documentation line
ip address 192.168.10.1 255.255.255.252
! IP address
pvc att-per 1/777
! Establish the pvc named att-per on VPI = 1,
VCI = 777 for this sub-interface
oam-pvc manage 0
! Causes the router to monitor for OAM AIS
cells but not generate OAM keepalives
encapsulation aal5snap
! All ATM ePVCs must use aal5snap
!
Sample Cisco 3640 Router Configuration

Sample configuration for a 4-port NM-IMA card with 2 active T1 ports is illustrated below. Appendix-C
contains sample configurations for AIM-ATM and VWIC-MFT card based IMA implementations.
ESC Release 2.4

25
interface ATM2/0
description AT&T T1, DHEC.xxxxxx
no ip address
ima-group 3
interface ATM2/1
description AT&T T1, DHEC.yyyyyy
no ip address
ima-group 3
!
interface ATM2/ima3
description AT&T IMA Port
no ip address
atm uni-version 3.1
no atm auto-configuration
no atm ilmi-enable
!
interface ATM2/ima3.155 point-to-point
description ePVC to AT&T PER VPI/VCI 1/155
ip address 192.168.10.1 255.255.255.252
pvc att-per 1/155
oam-pvc manage 0
! Creates IMA group 3 and assigns this port to

IMA group 3 (note 1)
! Assigns this port to IMA group 3

! Configure IMA group 3
! Create sub-interface on IMA group 3

! IP address
! Establish the pvc named att-per with
VPI = 1, VCI = 155 for this sub-interface 12
!
Note 1: The choice of IMA-group numbers must be done carefully to avoid any potential issues if/when any
individual T1 failures occur. If supported by the IOS, choose a group number between 17 & 255. If only values
between 0-3 are allowed, initially try 3. You will need to pay extra attention to the following issue. Problems can
arise if a device's TxGpID (a function of the IMA-group # in the CPE) is the same value as the RxGpID (a function
of the IMA group number in the AT&T switch). If there are failures in the underlying T1s comprising the IMA port,
and the group IDs are equal, it is possible for the AT&T IMA port to lock up, preventing any further transmission of
traffic. The following command can be used to determine both the Tx and Rx Group IDs: show controllers
atm1/ima0 detailed. Transmit (Tx) and receive (Rx) IMA group IDs are displayed in hexidecimalan example
from the output of this command follows: Detailed group Information: Tx/Rx Ima_id 0x10/0x0.
Note 2: The following are Cisco default settings (varies among IOSs) for the T1 & IMA Group Configuration
(acceptable in most cases; FDLcan be set to both, if available, or ATT or ANSI):
T1 Settings
IMA Settings
Framing = extended super frame (ESF)
Clock mode = common (ctc)
Line build-out (LBO) = short 133
Differential delay = 25 milliseconds
Linecode = B8ZS
Frame length = 128 cells
Facilities Data Link (FDL) = ansi
Test link = first link in the group
Yellow = enabled
Test pattern = value of test
Active-links-minimum 1
12
To set VPI/VCI to 1/777, for example, you many need to configure: "atm vc-per-vp 1024 (the default maximum
number of VCIs is 256 in some IOSs). Once you do this, you can add a VCI of 777. If you set vc-per-vp to 1024,
then you can use VCI values up to 1024 but are constrained on the values available for VPI. Since AT&T uses the
low order VPI values for VCC service, this is normally Ok.
ESC Release 2.4

26
Sample E3 ATM Interface Configuration on Cisco NM-1A-E3 card

interface ATM1/0
mtu 1500
bandwidth 34368
no ip address
no ip proxy-arp
atm scrambling cell-payload
no atm auto-configuration
no shutdown
!
interface ATM1/0.40 Point-to-point
mtu 1500
ip address 32.1.1.2 255.255.255.252
no ip proxy-arp
pvc toper 1/40
!
ESC Release 2.4

27
4 Static and BGP Routing Basics

The first sub-section provides a conceptual overview of using static and BGP routing to interface with
AVPN. This information is followed by two sub-sections that provide detailed examples for each method,
a discussion on IGP routing and a description of the AS override feature.
Most configuration examples in this document use frame relay as the link layer, however ATM or PPP
can replace frame relay as the link layer in any of the examples. All routing concepts and configurations
presented in this document are identical regardless of the link layer deployed. Therefore, any site can
interface with the service using either PPP, ATM or frame relay and have connectivity with each other
over the VPN MPLS service backbone.
4.1
Routing Basics, BGP and Static
The two primary methods for exchanging routes with the AT&T VPN network are either ordering static
routes or using dynamic BGP4 (Border Gateway Protocol version 4) routes. Different sites within the
same network may use either method; for example, a very simplistic implementation might use static
routes at remote sites with BGP at the data center. However, in all but the simplest cases, AT&T
recommends implementing BGP at all sites. The reasons for this are discussed further in Section 5.1.3.
In its simplest form, interfacing to AVPN only requires a few additional changes to existing router
configurations 13. These changes can be summarized as follows:
If using static routes, simply define a default route to the VPN network (PER) and order
(provision) the static routes for AT&T to put in the PER.
If using BGP for dynamic routing, simply configure BGP and define the PER as the eBGP
neighbor with AS 13979 and advertise the local network(s) associated with the site.
Then, if all devices at a site can follow a default route to the CER, all packets not destined for a local
network will be forwarded into the VPN network. The AT&T PER will route the packet to the appropriate
destination. Figure 2 illustrates this basic approach.
13
Many factors may impact this simplistic approach, such as whether or not a sites default route is required for
Internet access. This and other routing scenarios are addressed in Sections 5.1.3 thru 9.
ESC Release 2.4

28
Figure 2 Conceptual Routing Overview
From the customer's perspective, the AVPN network (including all PERs needed to support the
customers VPN) can conceptually be thought of as one virtual router with all interfaces dedicated to the
customer's VPN. All PERs assigned to the customer's VPN contain the routes from all CERs within the
VPN whether learned from BGP or static routes. AVPN by default does not summarize any routes or
filter any routes incoming or outgoing with the CERs. AVPN also does not generate any routes, such as
the default route, except that it redistributes the directly connected CER-to-PER networks into the VPN.
If a CER does not use BGP, the customer must tell AT&T via the provisioning process to configure static
routes in the PER for networks associated with (i.e., behind) the CER. The static routes provisioned in the
PER for a specific site must include all subnets existing at the site that need to be reachable by other sites
within the customer's VPN. This static route is then redistributed via iBGP to all other PERs in the VPN.
For any CER running BGP, the PER will pass all routes associated with the VPN (either statically or
dynamically learned) to the CER using eBGP. Note that AVPN by default does not summarize any
routes or filter any routes incoming or outgoing with the CERs. Two features are available to limit the
networks advertised from the VPN network via eBGP to a CER. The first feature is known as "Restrictive
Route Filtering" where only a single default (0.0.0.0/0) route is passed to the CER. The second feature is
"Outbound Route Filtering (ORF) which uses BGP prefix lists sent from the CER to the PER to inform
the PER of the networks the CER needs to learn. Both of these features are described further in Sections
14.10 & 14.11.
ESC Release 2.4

29
The following examples illustrate how the static and dynamic routing concepts can be translated into
router configurations based on the network topology shown in Figure 2. The required configuration
statements are listed on the left, with comments explaining the statement listed to the right. Note that any
parameter enclosed in "< >" i.e., <Network A>, <WAN subnet A>, and <PER A> refer to IP addresses
shown in Figure 1. Both of the examples shown assume that a standard frame relay (or ATM) permanent
virtual circuit (PVC) has already been configured to the VPN network as defined in the Section 4 on
configuring interfaces.
4.1.1
Approach to Static Routing
For static routing, the following command is used to establish a default route from router A (RTA) to the
VPN network:
ip route 0.0.0.0 0.0.0.0 <AVPN-PER A>
! Default route all non-local traffic to AVPN
Section 5.2.1 will discuss in further detail the use of static routing with AVPN.
4.1.2
Approach to BGP Routing
For BGP routing, the following three simple steps are used:
1. Assign an autonomous system number (ASN) to the site (see ASN section below) to be used when
configuring the BGP process on the CER.
2. Define the PER as an eBGP neighbor,
3. Define the networks to be advertised from the site into AVPN.
An example of the additional statements on RTA that illustrate the previous steps is as follows:
router bgp 65000
neighbor <AVPN-PER A> remote-as 13979
! Pick a private unique ASN for the site.

! Identifies the PER (IP address and ASN)
as the eBGP peer
network <network A> mask <network A subnet mask> ! Advertise site A's local network(s)
Section 5.2.2 will discuss in further detail the use of BGP routing with AVPN.
While not all implementations will be this straightforward, the same concepts will apply, particularly for
most remote sites. The remainder of this document provides detailed examples of specific configurations
for reference. These examples are intended to provide insight into configuring sites with Internet
gateways and hybrid AVPN/frame relay networks. Some material has also been provided in Sections 5.3
IGP Considerations and Section 9 Hybrid Networks that may be required in more complex scenarios.
4.1.3
BGP AS Numbering, AS Override and Sharing AS Numbers
In general, AT&T recommends each site have a unique "private" autonomous system number (ASN) as
referenced in RFC 1930 (AS numbers range between 64512 and 65535). Technically, any number can be
used except AT&T ASNs (see next paragraph) assuming no connection between the VPN BGP processes
and BGP used for Internet access. If necessary, sites can share an ASN and request AT&T to override the
site ASN with AT&Ts ASN using AT&T's ASN Override feature (see the ASN Override section
below).
The diagram below is a good representation of the major ASNs but is not the latest or complete:
ESC Release 2.4

30
Figure 3 AT&T BGP Autonomous Systems
(Note that the black lines just represent the BGP routing exchange between autonomous systems, not
trunks carrying user packets.)
An AVPN customer can use any ASN on their CERs that they wish however if you use one used someone
where else in the network, these two routers will throw away eachothers routes (the BGP AS path rule
for loop detection). AT&T or partners use the ASNs shown in the diagram above and the ones below,
which can potentially conflict if your network uses any of these services:
All AT&T registered ASNs (including BellSouth and SBC). Many of these will not conflict with
AVPN service, so listed here are only the significant ASNs to avoid: 8030 (cloud services),
12641, 32328, 9335 and 4295.
Private ASNs used by AT&T for connected networks. These only apply if the following service is
used with the VPN:
NBFW: 65534, 65533 and 65532 (same ASNs in all regions) and 65531, 65532 and 65533
(for NBFW customers using IPv6 or FIPS).
ANIRA (VIGvirtual gateway): 65000 to 65004. If there is an existing conflict, there is a
workaround using ASN override feature. ANIRA may also accept a custom ASN, on
request.
Managed AVPN service (AT&T manages the CER): 65315, 65525, 65526, 65527, 64700 to
64899.
Inter-operating with other AT&T MPLS services: PNT 7108, NetVPN 6389, NVPN 7132.
Other MPLS VPN service providers may impact the AS design if used somewhere in the
customers IP network. 14
14
Note that one new AT&T partner, Saudi Telecom, uses 65000 in part of their network; however, they use
commands to remove that ASN from the AS path of routes going to AT&T and to the customer routers.
ESC Release 2.4

31
AS 23456 used for 4-Byte ASN transition. This ASN is reserved by ARIN to be used by routers
when transitioning between an AS that supports 4-byte ASNs and an AS that only supports 2-byte
ASNs. To avoid confusion, do not configure a CER with a BGP process as 23456 to avoid
confusion with the use of 23456 by routers that support 4-byte ASN and substitute 23456 for 4byte ASNs (see following sub-section).
4.1.3.1 4-Byte (32-bit) ASNs

4-byte (32-bit) ASNs are becoming more common on the internet because ARIN is running out of 2-byte
numbers to hand out. This is not a problem in VPNs since people dont use the same BGP process on both
the public, untrusted side of the firewall and on the private side. However, if a customer is using 4-byte
ASNs on their private network, AVPN accepts them, except on some older PER platforms.
The format for 4-byte ASNs is 65000.1111.
There are two primary places where ASNs are used, thus where 4-byte ASN support is a factor:
BGP Process as a 4-byte ASN. The routers BGP process ASN (e.g. router BGP 65000) which is
normally the ASN used to peer with a BGP neighbor like the PER.
When the CERs BGP process is configured as a 4-byte ASN, it needs to peer with AVPN with a
2-byte ASN, because there is no way to order a 4-byte ASN on an AVPN port. This is not to say
the PERs IOS doesnt support 4-byte ASN, only that it cannot be ordered as the customers
ASN. CERs that support 4-byte ASNs should have the logic to recognize when a BGP neighbor
does not support 4-byte ASNs and replace its 4-byte with ASN 23456 when establishing the
BGP peering session. So, with AVPN, the customer with a CER configured with a 4-byte ASN
should order the AVPN port with ASN 23456.
AS Path attribute with a 4-byte ASN.
Once the CER and PER establish the BGP session, they exchange routes and the AS Path
attribute in the routes could contain 4-byte ASNs. If a router sees those and needs to send the
routes on to a BGP peer that does not support 4-byte, then it should replace the 4-byte ASNs in
the path with 23456, and add a transitive attribute that carries the 4-byte ASNs for the other router
to pass on in case a later, downstream router, does support 4-byte.
AVPN Support of 4-Byte ASNs

4-Byte ASNs are supported in some PER IOSs (Juniper and GSR XR; but not GSR IOS or RPM IOS). If
a 4-byte ASN is not orderable, either the CER needs to be a 2-byte ASN or if it is a 4-byte ASN and
cannot be changed, the customer can order the AVPN port with ASN 23456 if their CER supports
converting its 4-byte ASNs to 23456 (if it only does this automatically, when it detects the PER does not
support 4-byte, then there may be an issue when the PER does support 4-byte (the CER may need to hard
code the 4-byte to 2-byte transition with the PER neighbor to assure the BGP session is established with
23456, which is what the PER is expecting).
BGP Loop Detection
CERs configured as a 4-Byte ASN, even if converting to 23456 to peer with a PER, will not drop routes it
receives from the PER with 23456 in the AS Path, because the CER itself is not ASN 23456.
However, if a CER in the VPN is configured with 23456 (i.e. router BGP 23456), if would drop routes
that contain 23456 in the AS path. So, AT&T does not recommend configuring CERs as ASN 23456
(e.g. router BGP 23456).
ESC Release 2.4

32
Note the difference between configuring the CERs BGP process as 23456 versus configuring it as a 4byte ASN and ordering its AVPN port as 23456.
4.1.3.2 ASN Override
AVPN has implemented the feature ASN override which allows multiple sites to use the same ASN,
thereby allowing for ASN conservation in extremely large networks (i.e., > 1000 sites). ASN override
replaces all occurrences of the CERs ASN with the AVPN ASN, preventing the CER from seeing it as a
local route and discarding it. The AS override feature works when the PER sends a route to the
customer site. If a route at the PER has an ASN in the AS path that matches the associated CER's ASN,
then that ASN is replaced in the AS path with AVPNs ASN (i.e., 13979). This prevents the CER from
discarding the route because it looked as if it came from its own ASN. No other ASNs are changed and
path length is preserved. The AS override feature is enabled per interface or subinterface on the PER and
is activated by selecting AS override on the provisioning form.
As an example, if a customer configures the BGP process on their CER with an AS of 65001 and has
AT&T provision the interface with AS override, the PER will replace all occurrences of 65001 with
13979 in the AS path when sending BGP updates to this site. This way, this CER will not discard routes
from other sites also using AS 65001. The CER will see the AS Path field as "13979 13979". Refer to
Figure 4.
All sites sharing the same ASN should order this AS override feature. Generally the AS override feature
should not be enabled for sites using their own, unique ASN. Some exceptions to this apply during
specific types of load sharing across multiple sites; refer to Section 6.3, Case Study: Load Sharing the
Default Route.
Figure 4 AS Override Example
A note on sight of origin (SOO): Some MPLS services use this BGP extended community to prevent
routing loops. This is not normally needed because standard BGP algorithms have built in loop
detectioni.e., if a router receives an eBGP route with its own ASN in the path, it sees this as a loop and
discards the route. If you use unique ASNs at every site, you have no need for SOO. If you do use the
same ASN at multiple sites and also order the AS override feature, and a site has multiple AVPN
connections, you could have the potential for routing loops. The better solution is to use unique ASNs for
ESC Release 2.4

33
all sites with multiple AVPN connections. Route filters (i.e., inbound or outbound distribute lists, prefix
lists or route-maps) are another solution to avoid routing loops.
Juniper PERs and newer Cisco IOS PERs do not follow the split horizon concept when AS override is
enabled. This means that for a port ordered with AS override, the routes that the CER sends to the PER,
the PER will return (send) back to the same CER (assuming they are the best routes). See Figure 5 below
to illustrate.
Figure 5 Split Horizon
Normally, CER-A will not prefer the returned routesit will prefer its own local routes. However, if it
(CER-A) has an inbound route map that changes the BGP attributes to prefer inbound routes over local
routes (e.g. changing the weight), and since AS override on the PER changes the ASN, this could cause a
looping situation.
4.2
Routing Configurations
Static routes represent the simplest approach to routing with the VPN network. However, there are
functional limitations when using static routes, the most significant being the inability to detect a failed
remote connection. This is of particular concern for those sites that need to failover to an alternate path in
the event of a failure. The second limitation is that static route changes impacting the network require a
provisioning order. Provisioning orders require configuration changes in the AT&T PERs and the
customer does not have complete control over when the static routes become effective.
With BGP the customer has complete control over which routes are added or deleted and when this will
occur. This makes network migrations and IP address changes easier for the customer. Since static routing
is much less flexible, AT&T recommends that customers use BGP routing between the CER and PER
whenever possible.
4.2.1
Static Configuration
The example shown in Figure 6 uses static routes in both the CER and the AT&T PER. To get static
routes into the AT&T PER, the customer submits an order to provisioning (or the sales team). The
customer can order up to 20 static routes per interface or logical channel. Once provisioned in the
ESC Release 2.4

34
appropriate PER for the customers interface, the static routes are redistributed into BGP for the entire
VPN (all PERs in the VPN will have a routing entry associated with the static route).
Figure 6 Static Route Reference Diagram
The sample (partial) router configurations for RTA and RTB are provided below. The relevant routing
commands are highlighted. Note that the frame relay interface definition may be replaced with ATM, PPP
or ethernet at the link layer; refer to Section 4 for other interface configuration examples.
Partial RTA Configuration
interface Serial 1
description AT&T T1 VPN Service Port DHEC.xxxxxx
no ip address
encapsulation frame-relay IETF15
! Must use IETF (RFC 1490) for AVPN
!
! Sub-interface for ePVC
ip address 192.168.10.1 255.255.255.252
! /30 (2 point network) address
no cdp enable
! DLCI chosen to reach the PER;
!
ip route 192.169.0.0 255.255.0.0 192.168.10.2
! Static route to site B networks through PER
ip route 192.170.0.0 255.255.0.0 192.168.10.2
! Static route to site C networks through PER
ip route 0.0.0.0 0.0.0.0 192.168.1.1
! Default route to the Internet router (RTD)
15
If IETF encapsulation is configured on the main interface, as is shown, IETF encapsulation will also be active on
all sub-interfaces.
ESC Release 2.4

35
The AT&T PER must have static routes as well, specifically for this site. Again, the customer would tell
AT&T via an AVPN provisioning form the static routes to be used, as highlighted below. All networks
that are attached to or behind RTA that are to be accessed by other sites within the customer's VPN need
to be provisioned in AT&Ts PER associated with RTA.
Provisioned Routes in AT&T (PER) for RTA
ip route 192.168.0.0 255.255.0.0 192.168.10.1
ip route 0.0.0.0 0.0.0.0 192.168.10.1
! Static route to site A's networks

! Default route for any other networks not
explicitly known by AVPN is site A 16
If a default route is required, it must also be specified to AT&T in the provisioning form. When supplying
a static default route for the AT&T PER, be sure not to also use a static default route in the CER (RTA)
with a next hop of the PER, otherwise a routing loop will occur.
All other sites not currently using a locally significant default route (e.g., sites without their own Internet
access) can use a similar static route configuration as shown below or use BGP routing as defined in
Section 5.2.2.
Partial RTB Configuration
interface Serial 1
AT&T VPN Service Port DHEC.yyyyyy
no ip address
!
description ePVC to AVPN Service
ip address 192.169.10.1 255.255.255.252
no cdp enable
!
ip route 0.0.0.0 0.0.0.0 192.169.10.2
! Must use IETF (RFC 1490) for AVPN

! Sub-interface for PVC

! DLCI chosen to reach the PER;
! Default route to AVPN
Provisioned Routes in AT&T (PER) for RTB

ip route 192.169.0.0 255.255.0.0 192.169.10.1
! Static route to site B
AVPN will automatically advertise all WAN links connecting the CERs to the PERs, in this case
192.168.10.0/30, 192.169.10.0/30 and 192.170.10.0/30. This ensures that all VPN WAN interfaces are
reachable (e.g., via ping or telnet) Therefore a static route in AT&Ts PER is NOT required to specifically
advertise the WAN link addresses.
16
The preferred way to generate default routes is to use BGP for the site that has the Internet router and let BGP
advertise the default route to AVPN. AVPN then distributes it to the AT&T edge routers for all the other sites. If the
other sites are using static routes, then the customer router at those sites needs a default route pointing to AVPN.
ESC Release 2.4

36
4.2.2
BGP Routing
This section addresses using BGP routing between the customers autonomous system and the AVPN
autonomous system. The examples use EIGRP or OSPF running within the customers autonomous
system. Later sections address more complex scenarios.
The example illustrated in Figure 7 assumes that AVPN provides the only connectivity between sites.
This example uses a different ASN per site. In general, AT&T recommends each site have a unique
"private" autonomous system number (ASN) as referenced in RFC 1930 (AS numbers range between
64512 and 65535). However, if desired, sites can share an ASN and request AT&T to override the site
ASN with AT&Ts ASN using AT&T's ASN Override feature. With ASN Override, multiple sites may
use the same ASN. ASN override replaces all occurrences of the CERs ASN with the AVPN ASN,
preventing the CER from seeing it as a local route and discarding it. ASN override is discussed further in
Section 5.1.3.
Figure 7 BGP Reference Network Topology
4.2.2.1 Hub Site Configuration

There are two main approaches to advertising networks at a site using BGP. The first is to specifically
advertise each network using specific route information at the site. The second method, and most
preferred when possible, is to summarize all networks at a site. Both options will be presented in these
examples (see Figure 7, above). As with all configurations presented in this document, the relevant
routing commands are highlighted and frame relay link layer configurations can be replaced by the
equivalent ATM commands. Refer to Section 0 for ATM link layer configuration examples.
ESC Release 2.4

37
Partial RTA Configuration (specific routes)

interface serial 1.1 point-to-point
description ePVC to AVPN Service
ip address 192.168.10.1 255.255.255.252
no cdp enable
frame-relay interface-dlci 777 ietf

! DLCI chosen to reach the PER; "ietf" must be used
if not specified under main serial interface.
!
router bgp 65000
network 0.0.0.0
! Pick a private ASN for site A

! Use this to advertise the default route into AVPN
(vs. the default originate command)
! Advertise this network to your VPN
! Advertise each specific X/24 network at this site
! PER IP address and ASN
! Disable automatic network summarization
network 192.168.1.0 mask 255.255.255.0

network 192.168.X.0 mask 255.255.255.0
no auto-summary
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
! Default route to the Internet router - RTD - at site A
Note that BGP will advertise routes specified by the network statement only if they are also in the global
routing table. If a network such as 192.168.1.0/24 is included in a BGP network statement, then
192.168.1.0/24 must specifically be in the routing table via an IGP, static route, or a directly connected
interface. Additionally, if a default static route is not configured on RTA, then an IGP (OSPF/EIGRP)
must populate RTA's IP routing table with a valid default route coming from another source.
AVPN will automatically advertise CER-PER (e.g. /30) subnets, in this case 192.168.10.0/30. This
ensures that all WAN AVPN interfaces are reachable (e.g., via ping or telnet). Therefore, the CER is NOT
required to specifically include a network statement to advertise the WAN subnet address.
Partial RTA Configuration (summary routes)
ip address 192.168.10.1 255.255.255.252
no cdp enable
router bgp 65000
network 192.168.0.0 mask 255.255.0.0
network 0.0.0.0
no auto-summary
!
ip route 192.168.0.0 255.255.0.0 null 0
ip route 0.0.0.0 0.0.0.0 192.168.1.1
ESC Release 2.4

! DLCI chosen to reach the PER; "ietf" must be used if
not specified under main serial interface.
! Pick a private ASN for site A
! Advertise one summary route that includes all local
networks
! Use this to advertise the default route into the VPN
(vs. the default originate command
! BGP needs the exact network in the routing table
before it will advertise it - see Note 1 below
! Default route to the Internet router at Site - see note2

38
Note 1: BGP will advertise routes specified by the network statement only if the route is in the IP routing
table. If a summary route such as 192.168.0.0/16 is in one of the network statements, then 192.168.0.0/16
must specifically be in the routing table.
Note 2: It is usually best to learn the default route dynamically through an IGP from other routers at the
site with knowledge of the actual state of the Internet connection (assuming this is the ultimate destination
for default routed traffic). This way, an alternative route may be used - if available - to reach the Internet
router. The static route shown in this example is only provided for completeness since BGP will not
advertise the default route unless it is known by some other means. Using the command neighbor
192.168.10.2 default-originate will always advertise the default regardless of that route existing in the IP
routing table. This is generally not recommended.
4.2.2.2 Remote Site Configuration

As with the hub site configurations, both a specific route and summary route example will be provided.
Partial RTB Configuration (specific routes)
ip address 192.169.10.1 255.255.255.252
no cdp enable
!
router bgp 65001
network 192.169.1.0 mask 255.255.255.0
network 192.169.X.0 mask 255.255.255.0
no auto-summary

! DLCI chosen to reach the PER; "ietf" must be used if
not specified under main serial interface.
! Private ASN for site B, each site should use own
ASN
! Advertise this network to the PER
! Advertise each specific X/24 network at this site
Site B will receive its default route from the PER which learned it from the hub site's (RTA)
advertisement of network 0.0.0.0/0.
ESC Release 2.4

39
Partial RTB Configuration (Summary Route)

ip address 192.169.10.1 255.255.255.252
no cdp enable
!
router bgp 65001
network 192.169.0.0 mask 255.255.0.0
no auto-summary
!
ip route 192.169.0.0 255.255.0.0 null 0

! DLCI chosen to reach the PER; "ietf" must be used
if not specified under main serial interface.
! Private ASN for site B, each site has their own
ASN
! Advertise one summary route that includes all nets
! BGP needs the exact network in the routing table
before it will advertise it.
RTC would have a similar configuration as RTB, except advertising network 192.170.1.0.
RTD is the Internet router whose configuration should not change as a result of AVPN. RTD is involved
because it provides the default route for site A. RTA uses RTD as the destination for its default route. It is
assumed that RTD will be configured with static routes or will learn routes via an IGP with RTA to reach
remote VPN sites.
4.3
IGP Considerations
This section provides guidelines for merging BGP with IGPs, such as OSPF and EIGRP, running at
individual sites. These examples are basic configurations. If assistance is needed in more complex
scenarios, engage AT&Ts technical resources by contacting your AT&T sales team.
4.3.1
Hub Sites (RTA)
Since BGP provides the network statement to propagate routes into AVPN in a controlled manner, it is
typically not recommended to redistribute the site IGP into BGP without carefully reviewing the potential
impact. Caution must be exercised if there are multiple routes to VPN sites as routing loops can occur.
It is expected that in most cases BGP will need to be redistributed into the IGP at complex sites
(especially hub sites) and examples of this are shown later in this section. At simple sites, it may be
possible to make use of default routes (or default gateway settings on client LANs) to direct traffic to the
AT&T CER. In this case, no redistribution (and probably no separate IGP) needs to run at the site.
When an AVPN site also has its own Internet access or contains another default route, then other
mechanisms must be used to distribute AVPN routes into that site (e.g., supernets that summarize all
AVPN remote sites and networks - 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, etc.). Using the example
depicted in Figure 7, some approaches for handling routing at site A (with and without BGP
redistribution) might include the following:
ESC Release 2.4

40
Use RTA to supply site A's default route to other routers at the site. Additionally, any devices
directly attached to LAN segment 192.168.1.0/24 would use RTA as their default gateway 17.
Since RTA contains a static default route to the Internet router (RTD) and also knows all remote
site networks via AVPN, all traffic for site A would traverse RTA.
The Internet router (RTD) could remain site A's default route destination which would result in
both all Internet and outbound remote site AVPN traffic being routed first through RTD. In this
case, RTD will need to know about the remote site networks reachable via AVPN. This may be
accomplished by configuring individual static routes on RTD for each network reachable via
AVPN (in general a supernetted static route is preferred when possible).
In all but the simplest environments, the preferred approach would be redistribute BGP into site
A's IGP at RTA. This approach will not only keep outbound AVPN traffic from being routed
through RTD but would also allow all site A Internet traffic to route directly to RTD.
This last option is illustrated below. The exact configurations used in your environment will vary
depending upon your specific routing architecture.
Whenever redistribution needs to be done from one routing protocol to another (i.e., from BGP into the
IGP at a site), AT&T recommends that route-maps be used. This is especially critical if two-way
redistribution is ever implemented (i.e., from BGP into the IGP and from the IGP into BGP) to avoid
potential routing loops. In most cases, a route-map, configured to match a specific list of networks, is
sufficient. Route-maps, and their corresponding rule sets, document the desired redistribution behavior
and define the networks that should be redistributed from one protocol to another.
Additional RTA Statements for EIGRP
A simple example of redistributing BGP into EIGRP is shown below:
router eigrp 10
redistribute bgp 65000 metric 1536 2000 255 1 1500 route-map bgp-in (see note below)
passive-interface serial1.1
! Interface connected to AT&T PER
network 192.168.10.0
! Used to make 192.168.10.0 appears as an
EIGRP Internal network at site A
network 192.168.1.0
! Run EIGRP with other site A routers on
the192.168.1.0/24 network.
eigrp log-neighbor changes
no auto-summary
!
access-list 1 permit 192.169.0.0 0.0.255.255
! Allows all site B LAN & WAN addresses
! Allows all site C LAN & WAN addresses
!
route-map bgp-in permit 10
! The 10 after the permit is a sequence number
for the route-map
match ip address 1
! 1 refers to the ACL listing the networks which
are allowed to be redistributed into EIGRP
17
In all but the simplest topologies, multiple routers would probably exist at the hub sites and HSRP, MHSRP,
GLBP or VRRP would run on all LAN segments containing clients & servers.
ESC Release 2.4

41
Note on redistribution metric: The metric shown in the redistribution statement above was chosen based
on default EIGRP characteristics of a serial interface running at 1536 kbps. In general, it is recommended
that the metric match the actual characteristics (especially bandwidth and delay) of the interface
connecting to the AT&T PER. The metric chosen for redistribution can be included in the redistribution
statement - as shown in the above example - or listed separately - see below:
The values shown after metric are defined as follows:
a) 1536 - bandwidth in kbps
b) 2000 - delay (in tens of microseconds - the default for a serial interface is 20000
microseconds)
c) 255 - reliability (value between 0 & 255 where 255 means 100% reliability)
d) 1
- loading (value between 0 & 255 where 255 is 100% loaded)
e) 1500 - MTU (in bytes)
Additional RTA Statements for OSPF
router ospf 10
network 192.168.1.0 0.0.0.255 area 0
network 192.168.10.0 0.0.0.3 area 0
! Assumes the site A local LAN network(s) are

in area 0. Runs OSPF with other routers on
this segment.
! Ensures this subnet is also advertised via OSPF
to other routers at site A
! OSPF is not run over this interface to AVPN
log-adjacency-changes
redistribute bgp 65000 metric-type 1 subnets route-map bgp-in
! See note below
default-metric 2
! See note below
Note: Route map "bgp-in" and the corresponding reference to ACL 1 would be identical to the previous
EIGRP example. In this example, the redistributed BGP routes will go into OSPF as Type-1 Externals 18.
This means that the routes will carry both their external cost (as specified by the default-metric command)
and internal-cost (computed at each subsequent OSPF router hop). OSPF metrics (costs) are computed by
taking a reference bandwidth (default 100 Mbps) and dividing it by the interface bandwidth. In the
example above, the value of 2 would be the metric (or cost) of a 50 Mbps link, assuming the default cost
for 100 mpbs reference bandwidth, equal to 100,000,000/BW calculation.
18
In this example, OSPF Type 1 Externals are used. OSPF Type 2 externals may be preferred for some customer
environments.
ESC Release 2.4

42
4.3.2
Remote Sites (RTB)
The primary concern for the remote sites is receiving or creating a default route in the CER and
advertising it to other routers at the site. Some sample statements to accomplish this are provided below
for EIGRP and OSPF.
Additional RTB Statements for EIGRP, Example 1
router eigrp 10
redistribute static
network 192.169.1.0
default-metric 1536 2000 255 1 1500
no auto-summary
!
ip route 0.0.0.0 0.0.0.0 192.169.10.2
! Independent of other EIGRP 10 processes run at any

other sites
! Propagate static default route to any local routers
! Run EIGRP with other site B routers on the
192.169.1.0/24 network
! Metric for injected BGP routes for a T1
! Don't run EIGRP on the BGP interface to AVPN
! Define PER as the next hop for the default route
Additional RTB Statements for OSPF, Example 2

router ospf 10
network 192.169.1.0 0.0.0.255 area 1
network 192.169.10.0 0.0.0.3 area 0
default-information originate
! Independent of other OSPF 10 processes run at any

other sites
! Assumes the site B local LAN network(s) are in area-1
Runs OSPF with other routers on this segment.
! Ensures this subnet is also advertised via OSPF to other
routers at site B
! Don't run OSPF on the BGP interface to AVPN
! OSPF gets default route for site B via BGP
Note that the previous example configurations establish a default route at the remote sites using a static
route in the EIGRP case, and the default-information originate command in the OSPF case. Either
approach is appropriate when an alternate path for the default route is not required. When an alternate
default route path is required, redistribution of BGP into EIGRP or OSPF should be done to pass the
default route from AVPN to the IGP running at the site. One example of the need for alternate default
routes is dial backup. Dial Backup is discussed in APPENDIX D: DIAL BACKUP.
ESC Release 2.4

43
5 Load Sharing with BGP

Most customers will probably need multiple connections from their data center(s) to the VPN network,
either for added capacity or redundancy. Most of these customers will want some form of load sharing
across all possible AVPN links to gain the best performance. This section describes common load sharing
scenarios. Load sharing does not imply a 50:50 traffic split (though sometimes the term load
balancing implies a 50:50 split). Many configurations are probably straightforward, however the
routing implications can become complex in certain topologies. Some examples are provided here for
reference. Please contact AT&T technical resources when required. Figure 8 below illustrates an example
network.
Figure 8 Example Customer Network with Multiple Links
With traditional frame relay and ATM Service, load sharing of traffic across ports at a hub location was
accomplished by having remote site PVCs mapped to specific ports at a hub site 19. With this model, any
changes to the distribution of PVCs on hub site ports often involved re-provisioning PVCs. With AVPN,
however, traffic can be balanced at the IP layer and is simply based on which IP networks are advertised
over which links. Redistribution of the load on individual ports can be as simple as moving the
advertisement of a network from one CER-PER link to another.
Note that if the only objective is to achieve capacity greater than T1, the best solution is to implement
MLPPP, IMA or subrate DS3, all of which are supported by AVPN.
However, if increased reliability is an objective, a single IMA or T3 port may not meet the customers
requirements. To improve reliability, the solution may require multiple, diverse links out of multiple
routers (e.g. RTA 1 and RTA 2 in Figure 8). This solution requires load sharing using routing information
at layer 3.
19
This is not really load balancing since all traffic for a particular remote site network would still only follow one
PVC to the site regardless of the port it was provisioned on. It is more of a capacity management function in terms
of deciding how many and which PVCs to assign to a particular port.
ESC Release 2.4

44
To accomplish load sharing, the inbound and outbound directions must be examined independently.
Outbound route advertisements from CER to PER affect inbound traffic towards the site while inbound
route advertisements from PER to CER affect outbound traffic from the site. The following sections
discuss outbound load sharing, inbound load sharing and a complete network case study.
5.1
Outbound Load Sharing
Routes that RTA 1 and RTA 2 receive from the PER impact site A's outbound traffic. By default, AVPN
will advertise all of the VPNs routes (via BGP when enabled) across all links connected to RTA 1 and
RTA 2. Site A's routers could then use various methods to direct outgoing traffic across the multiple
CER-PER interfaces. These include:
With only one CER terminating multiple links to AVPN, the customer can use BGP multi-path to
do outbound load sharing. With this architecture, the BGP command max-paths n (2 <= n <=
6) can be used to instruct the router to accept two or more paths rather than only one best path.
Then, with CEF on, CEF will load share across up to six identical paths to the VPN network.
AT&T recommends using CEF per-destination and not per-packet.
5.2
If most traffic comes through core routers before reaching the CERs, then the site IGP could be
used to load balance traffic between the two CERs. This solution assumes that the BGP routes are
redistributed into the IGP with the same metric.
Another way to split outbound traffic is to control the default gateway settings on the data center
hosts (clients and servers). In Figure 8, for example, half of the servers could use a default
gateway pointing to RTA 1 and half the servers could point to RTA 2. This solution assumes that
there are no additional routers between servers/hosts and the routers that connect to AVPN.
MHSRP, GLBP or VRRP could be used between RTA 1 and RTA 2 to provide failover.
If there is no other router at the site and many computers (not routing devices) sourcing packets,
one could use GLBP (global load balancing protocol) between the two CERs. GLBP is like a
smart, dynamic HSRP ARP function. Reference Cisco documentation for configuration. This
technique does not work if there is only one device sending the ARP requests, such as a firewall.
The data center (site A) routers can use the BGP attribute local preference in combination with a
route map to assign different local preference values to particular routes received from AVPN to
impact the load sharing of the outbound traffic. For example, half of the learned routes could be
set to prefer RTA 1 path and the other half to prefer RTA 2. Higher local preference values are
preferred and the default local preference value is 100. Note: This solution requires that the
CERs, RTA 1 and RTA 2 run iBGP between them.
Inbound Load sharing
This section covers the AVPN load sharing feature first, then discusses other inbound load sharing
methods (PER to CER)
5.2.1
AVPN Load Sharing Feature
AVPN has an inbound (PER to CER) load sharing feature simply called load-balancing
The load balancing feature works automatically (customer does not have to order the feature) across
any set of links that advertise the exact same route with equal attributes to AVPN. Even the AS numbers
in the AS path must be the same (not just the AS hop count). Note that Juniper PERs (serving customer
ports in the U.S. and outside the U.S.) have the same BGP load sharing and closest exit behavior as the
Cisco PERs as of 2011.
ESC Release 2.4

45
This feature works by alternating IP flows across equal cost routes, not necessarily load balancing
physical links. AVPN load balancing will distribute traffic across up to six routes. The routes balanced
may include the default route (0.0.0.0/0) or any other customer routes. Important note: AVPN does not
take into account the bandwidth of the links.
The AVPN load sharing feature for PERs, uses a feature called eiBGP multi-path 20 which puts multiple
routes into the forwarding table using per source-destination IP address pair (in Cisco, this is CEF PERdestination). This is commonly referred to as flow-based load sharing. With this approach, all packets
sharing the same source-destination IP address pair will take the same route. This also means that perfect
50:50 load balancing is unlikely.
Note that the load sharing feature might cause packets to take a longer path to achieve load sharing. This
is because the originating PER 21 does the load sharing. Figure 9 demonstrates this concept for traffic
originating from RTC for a network x (e.g., 0.0.0.0/0). In this case, some packets will go to network A via
PER1 and some will take the longer route through PER2 to achieve the load sharing. Load sharing will
not consider geography or distance in the routing decision.
Figure 9 Load Sharing and Path Length
The load sharing feature operates differently, in certain scenarios, depending on whether BGP or static
routing is used on connections associated with the same routes. The next two sections describe the
differences in behavior.
The load sharing feature also operates differently outside the U.S. for MLPPP or T1 and lower speed PPP
ports if both the remote site and one of the load-shared links are on the same PER. All the traffic from that
remote site will go to the link on the same PER. This is similar to the static route case discussed below.
20
21
Not to be confused with eBGP multi-hop. AVPN does not support Cisco eBGP multi-hop.
Meaning the PER which first receives the packet from its associated CER.
ESC Release 2.4

46
5.2.1.1 AVPN Load Sharing Feature with BGP Routing

AVPN will load share across up to six customer paths if the routes are advertised with the same BGP
attributes. For load sharing to occur, the links advertising the routes must be in the same AS and the
routes must be advertised the same way from customer routers because these BGP attributes must be the
same: AS-path, local preference, origin and multi-exit discriminator (MED) 22.
Note that the router automatically sets the MED based on the IGP metric. If you want to assure load
sharing you should explicitly set the MED the same on all neighbor advertisements.
Figure 10 and Figure 11 show two different load sharing scenarios when using BGP. Figure 10 shows a
customer site (site B) sending traffic to Network A. PER 3 receives the traffic destined to Network A and
looks in its VPN routing table for Network A. In this example, PER3 will have two equal-cost IBGP
routes to Network A, one via PER1 and one via PER2. PER3 is the originating or ingress PER for traffic
from site B destined to Network A. As such, PER3 load balances using CEF per source-destination across
the two routes resulting in approximately half of the flows going to PER1 and the other half going to
PER2.
Figure 10 Load Sharing with BGPFrom site B
Traffic reaching PER 2 from iBGP links (PER3) is forwarded to Site A using the single link to RTA2.
Traffic reaching PER1 from iBGP links is further load balanced across the two links to RTA1. As such,
the distribution of flows from site B will be approximately 50% to the link on RTA2, and 25% to each of
the links on RTA1. Notice that the term flows was used and not number of packets or bandwidth
consumed.
To achieve an even distribution of flows across all links (33:33:33 rather than 50:25:25), provision
multihoming to move one RTA1 link to a third PER.
If a second link were added between RTA2 and PER2, the distribution of flows would be approximately
25% per link. The same distribution would occur if there were four links from Site A and each connected
to a separate PE router. The amount of bandwidth consumed on each link will depend on the amount
traffic associated with the specific flows. Therefore, the bandwidth distribution may not match the
distribution of flows.
22
The use of BGP network statements results in BGP routes with an IGP (i) origin code while routes brought into
BGP via the redistribution command result in BGP routes with an Incomplete (?) origin code. The MED value is
generally the IGP metric for the route.
ESC Release 2.4

47
Figure 11 shows a different customer site (site C) sending traffic to Network A.

Figure 11 Load Sharing with BGPFrom Site C
The important difference between site C and site B is that site C connects to one of the PERs used for
connections from Site A. When PER 1 receives the traffic destined to Network A and looks in its VPN
routing table for Network A, it will have two EBGP routes from the direct connections to RTA1 and one
IBGP route from PER2. Normally, the BGP process would select the EBGP routes over IBGP. However,
Cisco PERs have the eiBGP multipath feature enabled, which overrides this normal behavior and allows
load sharing between EBGP and IBGP routes. Therefore, PER1 will load balance flows across its local
links and the remote link. The resulting distribution of flows from site C should be approximately 33%
per link. The same distribution would occur if the three links from site A connected to three different
PERs.
5.2.1.2 AVPN Load Sharing Feature with Static Routing
When multiple links are desired at a site or group of sites, AT&T recommends using BGP on the links
that advertise equal-cost routes. However, if static routing is necessary, some load sharing can be
achieved. This section uses the same scenarios as the previous section to explain the similarities and
differences in behavior when static routes are used. Figure 12, below, shows the scenario where site C
sends traffic to Network A. The load sharing behavior for traffic originated by site B is the same as the
BGP case. The routing type (static or BGP) at site B and C is not important for this load sharing
discussion.
ESC Release 2.4

48
Figure 12 Load Sharing with Static RoutesFrom Site C
As before, the important difference between site C and site B is that site C connects to one of the PERs
used for connections from Site A. However, the load sharing behavior on PER1 differs from the BGP
case. When PER1 receives the traffic destined to Network A, it looks in its VPN routing table for
Network A and will only have the two routes associated with the static routes to RTA1. PER1 received an
IBGP route for Network A from PER2, but did not select that route for load sharing. This is because
PER1 does not see the local static routes and the IBGP routes as equal routes the way it would if the
customer were using BGP 23. The EIBGP multi-path feature does not apply when comparing static routes
and other routes (EBGP or IBGP). Instead, the static routes look better from a routing perspective 24.
Therefore, PER1 will load balance flows across only its local links, not remote links. The link between
RTA2 and PER2 will not carry Network A traffic from site C. Similarly, if there were a site D connected
to PER2, all of its traffic to Network A would only use the local static route on PER2 and would used the
link between RTA2 and PER2.
This behavior may or may not be a problem. If site C is only one of a few sites connected to either PER1
or PER2 and there are many other sites not connected to either PER1 or PER2, the effect described above
may be negligible when looking at aggregate traffic on the links to Site A.
If it is a problem, the preferred solution is to change to BGP routing on the links for Site A. Alternatively,
site C and other like sites could be moved to a different PER (not PER1 or PER2) and will load balance to
Network A like site B. This may not be practical for regional networks where a large number of sites are
concentrated in a small geographic area.
5.2.2
Other Inbound Solutions
Other inbound load sharing solutions (besides the AVPN load sharing feature) that a customer could
implement are as follows:
23
Based on EIBGP Multipath feature treating EBGP and IBGP routes the same.
Static routes have a default Administrative Distance of 1, while IBGP routes have a default Administrative
Distance of 200. Routing protocols with lower Administrative Distances are preferred.
24
ESC Release 2.4

49
1. Advertise half of the networks (or individual host routes) out one interface (or CER) and half out a
second interface (or CER). Both interfaces would also advertise a more general route (that includes
all networks) to allow fail over. For example, advertise 10.1.0.0/24 and 10.1.1.0/24 from interface1 or
CER1, and 10.1.2.0/24 and 10.1.3.0/24 from interface2 or CER2. In addition, advertise the supernet
10.1.0.0/22 out both interfaces or CERs. Note that each CER would need to have the supernet route in
its local route table (e.g., via ip route 10.1.0.0 255.255.252.0 null0). Alternatively the "aggregateaddress" command could be used.
2. On one interface (or CER), advertise half the networks to AVPN with their default BGP metrics and
the other half out the same link but with a longer AS path (using the BGP AS path pre-pending).
Do the reverse out the other interface (or CER). An example of this type of configuration is shown in
Section 6.3.
3. Somewhat similar to option 2, one could change the BGP multi-exit discriminator (MED) attribute on
outgoing route advertisements. Lower MEDs are preferred over higher MEDs.
4. Lastly, AVPN supports BGP community strings requesting a specific local preference setting in
AVPN.
Options 3 & 4 are explained further in Section 14.2. In general, AT&T recommends using AVPN load
sharing feature, or options 1 or 2 as the simplest and most commonly implemented solutions.
Load sharing by manipulating routing advertisements and preferences is not likely to achieve 50:50
balancing. If the customer has few server addresses or networks (routes) to manipulate, 50:50 balancing is
even more difficult to achieve. This is because all of the previously described routing options address how
the VPN network will route a packet with a common destination IP address (or network). However, there
are several methods that end-user applications can use to choose a destination IP address (i.e., static,
dynamic). While the different methods are well beyond the scope of this document, a few options are
discussed below:
If there are multiple services (e.g. web, SMTP, FTP) running on multiple servers on the same network
segment, then some servers could potentially be left on one network and while others are moved to a
different network. Alternatively a combination of server load balancers (using virtual IP addresses to
access an application or service) along with DNS to resolve names to these virtual IP addresses is
common for many larger server farms. Ideally, the servers would have multiple addresses from different
networks (whether physical or logical networks). In this type of environment, assuming applications can
use DNS names, then one can configure DNS to supply different addresses for the same service. The
DNS server will alternate which address it supplies first ("smart DNS" appliances can be used to take into
account more sophisticated rules). Different clients could then access the same service using different
destination IP addresses.
For further explanation of some of the routing concepts discussed in this section, see the next two sections
which explore several different topologies and their impact on traffic flows. If additional assistance is
required, please contact AT&T technical resources.
5.3
Case Study: Load Sharing the Default Route
Figure 13, below, shows an example of load sharing to multiple CERs. This could be across CERs in one
hub site or across multiple hub sites connected by a reliable backdoor. Backdoor in this context refers
to non-AVPN connectivity linking networks between two or more sites. The critical assumption here is
that there is a reliable network between the two CERs and that the network has sufficient bandwidth to
support the traffic loads, even during failure conditions. While such a reliable high-speed connection is
usually assumed between routers at the same site (e.g., via one or more LAN connections), it can also be
accomplished between two diverse sites by using some type of SONET infrastructure or diverse WAN
circuits.
ESC Release 2.4

50
It is important in this topology that the CERs at the hub site(s) must share the same ASN to allow the
AVPN automatic load sharing feature to work. Sharing the same ASN also means that when one CER
receives the other CERs routes from AVPN, it will discard the routes. This is the case since the ASN in
the route path will be the same as the CERs own ASN. This a basic BGP rule to prevent routing loops.
Note that in this situation, with reliable backdoor connectivity, the customer should not provision the AS
override feature, because one CER does not need to reach the other CER via AVPN.
In the example shown below, both CERs will advertise the default route (0.0.0.0/0) to AVPN, so that
Internet traffic from remote sites will load balance across both CERs.
In this example, we are also advertising the local LANs at site A and D using primary/backup routing
via AS path prepending. In the diagram below, the sites A and D have a backdoor connection (labeled
internal path) that can backup each others AVPN connection. So, we can advertise site As LAN
network out of site Ds link to AVPN as a backup route, and vice versa. The route-map is used to make
the site D 192.167.0.0/16 network advertisement less preferred from site A (RTA) by using AS path prepending. This will add an additional 65000 to the AS-path so that the route for 192.167.0.0/16 when
received by the PER will show 65000 twice. Since the same route from RTD will only have 65000 in the
AS-path once, this will be the preferred path unless a failure occurs affecting RTD's advertisement to the
VPN network.
Figure 13 Load Sharing Internet Connections
ESC Release 2.4

51
Site A, RTA Partial Configuration

(LAN & WAN interface configurations are not shown but are functionally equivalent to earlier
examples)
router bgp 65000
! In this case, ASN must be the same for site's A & D
! See Note 1
network 0.0.0.0
! Default - assumes route is learned reliably from an IGP
network 192.168.1.0 mask 255.255.255.0
! Site A core LAN segment
network 192.168.2.0 mask 255.255.255.0
..
! More Site A LAN segment network statements
network 192.167.1.0 mask 255.255.255.0
! Site D core LAN segment
network 192.167.2.0 mask 255.255.255.0
..
! More site D LAN segment network statements
aggregate-address 192.168.0.0 255.255.0.0 summary-only
! See Note 2
aggregate-address 192.167.0.0 255.255.128.0 summary-only ! See Note 2
neighbor 192.168.10.2 soft-reconfiguration inbound
! See Note 3
neighbor 192.168.10.2 distribute-list 1 in
! See note 4
neighbor 192.168.10.2 route-map PREPEND out ! See Note 5
no auto-summary
!
ip classless
! Allows use of supernetting -e.g. default route
.
access-list 2 permit 192.167.0 0.0.255.255
!
route-map PREPEND permit 10
match ip address 2
set as-path prepend 65000
!
! Allows all remote site B LAN networks

! Allows all remote site C LAN networks
! Allow any other remote site networks in ACL 1
! Site D address space
! Needed to allow all other networks to be advertised

without the longer AS Path
Note 1: Allows for better troubleshooting - logs an entry anytime a BGP neighbor goes up or down.
Note 2: Will only send a summary route as long as at least one more specific route in the 192.168.0.0/16 or
192.167.0.0/16 address space exists in the BGP table
Note 3: Allows for better troubleshooting - can view all BGP routes advertised from PER, not just routes that make
it through the ACL.
Note 4: Ensures that only networks defined in ACL 1 are allowed inbound from the AT&T network. This provides
an explicit way to ensure that only the networks that this site needs from the VPN network are allowed into the BGP
route table.
Site D, RTD Partial Configuration

LAN & WAN interface configurations are not shown but are functionally equivalent to earlier examples.
ESC Release 2.4

52
router bgp 65000

! In this case, ASN must be the same for Site's A & D
! See note 1
network 0.0.0.0
! Default. Assumes route is learned reliably from an IGP
network 192.168.1.0 mask 255.255.255.0
network 192.168.2.0 mask 255.255.255.0
..
! More Site A LAN segment network statements
network 192.167.1.0 mask 255.255.255.0
network 192.167.2.0 mask 255.255.255.0
..
! More site D LAN segment network statements
aggregate-address 192.168.0.0 255.255.0.0 summary-only
! See note 2
aggregate-address 192.167.0.0 255.255.128.0 summary-only ! See note 2
neighbor 192.167.10.2 soft-reconfiguration inbound
! See note 3
neighbor 192.167.10.2 distribute-list 1 in
! See note 4
neighbor 192.167.10.2 route-map PREPEND out
! See note 5
no auto-summary
!
ip classless
! Allows use of supernetting -e.g. default route
access-list 1 permit 192.169.0.0 0.0.255.255 ! Allows all remote site B LAN networks
access-list 1 permit 192.170.0.0 0.0.255.255 ! Allows all remote site C LAN networks
.
! Allow any other remote site networks in ACL 1
access-list 2 permit 192.168.0 0.0.255.255
! Site A address space
!
match ip address 2
set as-path prepend 65000
! See note 5
!
! Needed to allow all other networks to be advertised
without the longer AS Path
Note 1: Allows for better troubleshooting - logs an entry anytime a BGP neighbor goes up or down.
Note 2: Will only send a summary route as long as at least one more specific route in the 192.168.0.0/16 or
192.167.0.0/16 address space exists in the BGP table
Note 3: Allows for better troubleshooting - can view all BGP routes advertised from PER, not just routes that make
it through the ACL.
Note 4: Ensures that only networks defined in ACL 1 are allowed inbound from the AT&T network. This provides
an explicit way to ensure that only the networks that this site needs from the VPN network are allowed into the BGP
route table.
Note 5: The route-map is used to make Site A 192.168.0.0/16 advertisement less preferred from site D (RTD) by
using AS pre-pending. This will add an additional 65000 to the AS-Path so that the route for 192.168.0.0/16 when
received by the PER will show 65000 twice. Since the same route from RTA will only have 65000 in the AS-Path
once, this will be the preferred path unless a failure occurs affecting RTA's advertisement to the VPN.
Remote Site configurations are the same as in earlier sections.

5.4
Case Study: Load Sharing The Default Route and No Backdoor
This is similar to the previous case; however, the hub sites must use AVPN to communicate because there
is no direct link between the two Internet hub sites (only AVPN VPN). Figure 14, below, shows this
example. Both Internet hub sites will advertise the default route to VPN (for remote sites to use) and the
customer requires AVPN to load share the Internet traffic across the two hub sites. To load share traffic,
AVPN needs both default routes to have the same ASNs in the AS path attribute.
ESC Release 2.4

53
When load sharing, AVPN will send all packets for a given source-destination IP address pair to the same
path. This allows stateful firewalls to work. However, a given users web browser session may have some
TCP connections going through one hub site and some through the other. If the firewalls are requiring
authentication, this may cause a different behavior. Note that for browsers using a proxy server, the
destination IP address in the packets is that of the proxy server which might not use the default route, but
rather a specific route for the proxy servers network.
Note that each Internet hub site CER will learn the other hub sites routes, including the default route. If
the two hub sites are using the same ASN, they will discard the routes advertised from the other hub site
because the CER will see its own ASN in the path (standard BGP rule). However, since there is no
backdoor connection between the two hub sites and they must communicate with each other over AVPN,
RTA and RTDs links should be provisioned with AS override. This way each Internet hub site will
accept the routes, including the default, from AVPN from the other hub site. If the default route from the
other site is to be used as a backup, it must be configured with a lower administrative distance (e.g. 200)
to assure that the CER will only use this route when its own local default route fails. One can accomplish
this by using the distance command with an access list that permits only the default route, as shown
below:
router bgp 65000
distance 200 192.167.10.2 0.0.0.0 10
access-list 10 permit 0.0.0.0
! 192.167.10.2 is the BGP neighbors address
Figure 14 Load Sharing with no Internal Path Between Hub Sites
ESC Release 2.4

54
6 Route Selection
Sometimes the same IP network is advertised from two different sites. The most common example of this
is the default route advertised by multiple internet gateways into the corporate private network. Another
example is advertising the same server subnets from both the primary data center and a backup or disaster
recovery center. There are several options available to the customer when advertising a route from
multiple places. The customer has several options for dictating which route the PER will choose, when it
receives the same IP network number from two routes:
Primary/Backup. The customer can dictate the which route is primary and which is backup by
sending the route with BGP AS prepending or with a BGP community value that AT&T
recognizes for setting BGP local preference within the AVPN AS, or with a BGP MED (see
section 14.2.4 for more information on BGP attributes). Note that this primary/backup selection
applies the same to all customer sites in the VPN in the region. Packets from all sites will follow
the same primary route.
Closest Exit. The customer advertises both routes into the same AT&T AS (same region) with
the same BGP attributes except from two different ASNs. AT&T PERs will see all attributes
equal so will use the BGP closest exit rule which picks the route with the lowest metric to the
BGP next hop. In AT&Ts backbone this is based on OSPF metrics on the trunks between PERs.
These metrics are roughly proportional to latency or distance. So, for example, remote sites on the
west coast will follow the default route coming from the customers internet gateway on the west
coast, while remote sites on the east coast will follow the default route coming from the
customers internet gateway on the east coast.
Note that Juniper PERs will also do closest exit routing (as of 2011).
Load Sharing. The customer advertises both routes into the same AT&T AS with the same BGP
attributes including the same ASNs in the AS path. This triggers the eiBGP multipath feature
that causes the PER to put both routes into the forwarding table and load share IP flows among
the multiple routes. See the previous on AVPN load sharing, Section 6.2.1.1.
Route Groups. Route groups is a feature in AVPN that allows the customer to have a different
routing policy within the AVPN VPN for different groups of sites. For example, all sites within
one business unit could have their PERs prefer a different default route than another group of
sites. AT&T has written two case studies on the use of route groups. Please contact your AT&T
sales team for technical support.
The PERs, which are essentially standard routers like CERs, follow standard BGP path selection rules and
AT&T makes every effort to have Cisco PERs behave the same as Juniper PERs. For both CER and PER
behavior, see APPENDIX A: BGP Path Selection Criteria for a summary of Ciscos use of BGP
attributes and other criteria in picking the best route to use.
ESC Release 2.4

55
7 Redundancy and Failure Recovery

This section describes some of the solutions available to customers for failure recovery. Section 8.1 and
8.2 address diversity options, Section 8.3 addresses site disaster recovery, and APPENDIX D: DIAL
BACKUP addresses dial backup scenarios.
When deciding on the level of failure protection to include in a network design, it is important to identify
the failure types that may occur and also which of the failure types warrant a recovery solution. The
decision about whether or not to protect against a given failure type should be based on the level of
exposure to the company, financial or otherwise, imposed by potential failures. For example, the exposure
associated with the loss of a site for customers in the financial sector, where time-sensitive transactions
are critical, may be significantly greater than for an insurance provider whose agents can enter policy
information offline and not lose business. In this example, the additional expense and complexity
associated with failure recovery may be justified for the financial services company but possibly not for
the insurance provider.
Generally, network failures fall into the following broad categories; Customer Premises Equipment
(CPE), access, long-haul carrier service, and site disaster recovery. The following diagram, Figure 15,
will be used to discuss AVPN failure recovery.
Figure 15 AVPN Failure Recovery Reference
In Figure 15, locations 1, 2 and 3 are customer locations. Location 1 is a network data center location.
Location 2 is a disaster recovery site for Location 1. Location 3 is a branch or remote location. AVPN
access is represented by the links labeled A, B, C, and D and their associated PERs. The cloud labeled
with the letter E represents the AT&T VPN network servicing the customer VPN.
The following failure types are possible, although some are much less likely than others:
CPE Failure For the purposes of this document, CPE failures will be limited to failures of the
CER.
ESC Release 2.4

56
Access Failures Included within the portion labeled access in the figure are the LEC access link
and the connection from the AT&T ATM/FR central office to the AVPN central office, and the
PER. A failure of any one or more of these components would constitute an AVPN access failure.
VPN Failure This improbable failure type, includes any failures of the customer's VPN or the
AVPN Service as a whole.
Site Disaster This refers to the loss of a site due to a disaster (e.g., fire, earthquake, flood, etc).
A given failure recovery solution may restore connectivity for one or all of the above components.
Section 8.1 and 8.2 addresses diversity options; Section 8.3 addresses site disaster recovery; and
APPENDIX D: DIAL BACKUP addresses dial backup scenarios.
7.1
AVPN Diversity Options, Domestic U.S.
AVPN offers several capabilities for achieving diversity. The capabilities differ with the different access
types. For the most up-to-date and official description, refer to the AVPN Service Guide.
AT&T VPN offers multiple access methods: dedicated access ("IP ports"), Frame Relay and ATM access,
and Ethernet access. The diversity options for these are different so they are discussed in their own
subsections below.
7.1.1
Frame Relay and ATM Access
For Frame Relay and ATM access, it is important to note that there is a layer 2 switch separate from the
layer 3 provider edge router (PER) and they may be in separate central offices. The layer 2 switch
terminates the customers access circuit. The "AVPN Diversity options" for ATM and Frame Relay ports
apply to this layer 2 switch. Diversity for the PER (the layer 3 port) is achieved with the "Multihoming"
option detailed below. To achieve diversity at both layer 2 and layer 3, the customer must order both
"Multihoming" and the "AT&T VPN Diversity Option".
AVPN Diversity Options (Layer 2 Switch)
When the customer has more than one Frame Relay or ATM port, they can order those ports to be diverse
from each other. There are two options available: MPLS Port Service Diversity Option (SDO) and
MPLS Port PoP Diversity ("Point of Presence" or "Central Office"). SDO assigns the multiple ports
onto different switches in the same central office. The PoP Diversity option puts the ports into different
central offices (usually different cities).
Note: These diversity options provide customers with the same capabilities as legacy AT&T High Speed
Packet Services Frame Relay and ATM High Speed Packet Switch Diversity Options (HSPS-SDO)
available when traditional point-to-point PVCs are used.
"AVPN Diversity Options" work as follows:
A collection of ports which require diversity are assigned to an arrangement. Each arrangement is
numbered to maintain separation from other arrangements which the customer might have.
Within an arrangement, the specific ports which are to be made diverse are placed into separate
groups (which are named). So, all ports in a group are made diverse from all the ports in the other
groups. Ports within a group are not made diverse from each other.
The customer can order up to three mutually exclusive groups in each arrangement.
With Switch Diversity, the groups are put on diverse switches in the same central office (or
"PoP").
ESC Release 2.4

57
With PoP Diversity, the groups are put into diverse central offices (or "PoPs"), which are
usually in different cities.
A maximum of 1,000 ports are allowed in one arrangement. (No limit on the number of
arrangements.)
A port may be assigned to only one group and to only one switch diversity arrangement.
However, the port may also be in a POP Diversity Arrangement.
Note that if Layer 2 Frame Relay or ATM ports do not have diversity and are provisioned onto the same
switch, AT&T will automatically attempt to put the ports onto different cards. This feature, call "Standard
Card Diversity," is provided automatically, if available, whenever a customer has more than one Port in a
given Frame Relay or ATM Switch POP.
After the Layer 2 ports are provisioned, an enterprise Permanent Virtual Circuit (ePVC) then goes from
the Layer 2 switch to a Provider Edge Router (PER). That router might be in the same central office or it
might be in another central office, so the customer may want to order Layer 3 diversity, called
Multihoming, to get multiple ePVCs onto different PERs.
Multihoming (Diversity for the Layer 3, PER Port)
"Multihoming" provides Layer 3 diversity. When the customer has more than one ePVC to Provider Edge
Router (PER) ports, they can order those ports to be diverse from each other. There are two options
available: Switch Diversity and Central Office (CO) Diversity.
With multihoming, one arrangement can have up to six ports and all the ports in the arrangement are
made diverse from each other. (This is equivalent to saying there is one port per group in the terminology
used for AVPN Diversity Options, above.)
Multihoming Switch Diversity. The virtual circuits (ePVCs) in this type of multihoming group are
provisioned on up to six (6) different PERs25. Note that the PERs may be in the same central office. If
there are more than six (6) links in a multihoming group, the links seven (7) and above are assigned in
round-robin fashion on the six (6) selected PERs. As a result, the links above the first six (6) will not be
diverse from all other links in the group. PER diversity does not guarantee that the PERs will be in
different AT&T Central Offices.
Multihoming Central Office (CO) Diversity. The virtual circuits (ePVCs) in this type of multihoming
group are provisioned to PER ports in different Central Offices (usually different cities). A single PER or
CO failure will not bring down all connectivity to a site. The advantage of this option is higher reliability
but the disadvantage is potentially longer packet latency to reach a different central office. The links in a
multihoming group with CO diversity will have a PER assigned from at most six (6) different COs. If
more than six links are assigned, all links will be provisioned to the six COs. Links in the same CO are
not guaranteed PER diversity within the CO. For example, if seven links are included in a multihoming
group with CO diversity, a possible scenario could be that two links are provisioned on the same router in
one CO. When designing diversity arrangements with three or more ports, the potential for substantial
latency differences between ports exists. It may be advantageous to create multiple arrangements rather
than group all ports in one arrangement.
To illustrate, refer the following three examples. In each example the customer has a single data center
with two connections. The various illustrations show how the AT&T VPN Diversity feature for ATM and
Frame Relay MPLS Ports and Multihoming might function together to provide various levels of diversity
for this site. Other possibilities exist, but these are the most common.
25
Note that the term switch is used in the name of this feature but that is a misnomer since the device is a router
(the provider edge router or PER)
ESC Release 2.4

58
Figure 16 Diversity Options for Frame Relay and ATM Access
7.1.2
Dedicated Access (IP Port)
Dedicated access circuit (IP port) diversity options work the same as the "multihoming" PER diversity
descriptions outlined above for Frame Relay access. The "MPLS IP Port Switch Diversity Option
functions the same as Multihoming Switch Diversity Option and the MPLS Port PoP Diversity
Option functions the same as the Multihoming CO Diversity Option.
There is no separate Layer 2 switch used with dedicated access ("IP ports").
ESC Release 2.4

59
Figure 17 Diversity Options for IP
7.1.3
Ethernet Access Diversity
Standard ethernet access in the U.S. looks like the following:
Outside the U.S. the access is similar but there is no AT&T ethernet gateway switch (EGS).
There are several diversity options in the U.S. for ethernet access, depending on what is available in your
area, that you can order (these are some representative examples):
Diverse PER PoPs
Diverse ethernet PoPs if they are available in the LATA

Diverse PER PoPs with diverse ethernet PoPs if an alternate PoP is available in the LATA
Alternate ethernet service providers, if available
Alternate ESPs with ethernet PoP diversity
Alternate ESPs with ethernet PoP diversity and PER PoP diversity
Access route diversity (within the ESP, sometimes called route otherthan normal) if available
from the ESP
Access route diversity (within the ESP) with ethernet PoP diversity
Access route diversity (within the ESP) with ethernet PoP diversity and PER PoP diversity
Duplicate NPEs, if available from the ESP
ESC Release 2.4

60
7.1.4
Caveats
CO/POP diversity can only be provided for like ports. For example, diversity can be provided for two
IP MPLS Ports but not for an IP MPLS port and an ATM MPLS Port.
All of the AVPN diversity options described above do not provide diversity for access or backhaul
layer 1 facilities used for connectivity between customer locations and AT&T VPN Service Elements.
Diversity assignments are automatic. The customer is not provided the option of choosing the
alternate port assignment locations.
7.2
AT&T VPN Diversity Options in Most of World (MOW)
Diversity outside the U.S. (MOW) can be accomplished by purchasing access to specific POPs (e.g.
one link to one POP and an alternate link to a different PoP). There is no specific feature called
diversity for MOW.
7.3
Site Disaster Recovery
With a single AVPN link at each location, a customer has full mesh connectivity to every site on the
network. With this type of design, it can be expected that many customers will want to provide
connectivity to a backup data center using AVPN. There are two primary ways to accomplish this. The
first option is a manual recovery that requires intervention by an individual to make configuration
changes at the backup site when the primary data center fails. This would be a good option for customers
who only want to manually control the time when the backup is initiated when a site disaster occurs. The
second option is an automatic recovery that involves setting up the routers so that all traffic destined for
the primary data center would dynamically be routed to the backup site during a failure.
Figure 18, below, will be used to illustrate both recovery methods. Site A is the primary data center and
site C is the backup site. Site C could be another customer location or a third party disaster recovery
vendor location. Site B is a typical remote site that needs access to the 192.168.1.0 network. In the
following examples, it should be noted that Internet access would not be available when Site A fails. This
is because the only route to the Internet is through Site A (in this example network).
7.3.1
Manual Recovery with AVPN Service
When a failure occurs at Site A, RTC now needs to advertise a replicated Site A LAN subnet to all remote
sites. In this manual recovery mode, a network 192.168.1.0 mask 255.255.255.0 statement simply needs
to be added under the BGP process on RTC. This assumes that RTC already had the LAN interface
192.168.1.0/24 configured and active.
With this solution, no changes need to be made on the remote sites (site B). The VPN network will learn
the new route from RTC and transmit it to all other sites. In this example RTB's traffic destined for
192.168.1.0/24 will now be routed to RTC after the manual configuration change at site C.
ESC Release 2.4

61
Figure 18 Site Disaster Recovery
The following example defines the backup site (site C) configurations before and after the manual change.
Note that before the outage connectivity to the AVPN can be verified by observing the state of BGP on
RTC. BGP should be active; it just will not be advertising the 192.168.1.0/24 network.
Partial RTC Configuration (before outage)
interface Ethernet0
ip address 192.170.1.1 255.255.255.0
!
interface Ethernet1
ip address 192.168.1.1 255.255.255.0
shutdown
!
router bgp 65002
network 192.170.1.0 mask 255.255.255.0
no auto-summary
ESC Release 2.4
! IP address of Ethernet subnet

! Ethernet interface is shut until needed
! Private AS number for site C
! Advertise this network to the AVPN VPN

62
Partial RTC Configuration (during outage)

interface Ethernet0
ip address 192.170.1.1 255.255.255.0
!
interface Ethernet1
ip address 192.168.1.1 255.255.255.0
!
router bgp 65002
network 192.170.1.0 mask 255.255.255.0
network 192.168.1.0 mask 255.255.255.0
no auto-summary
7.3.2
! IP address of Ethernet subnet (no shut)

Dynamic Recovery
For a dynamic site recovery, RTA and RTC are both configured to advertise the same IP subnet,
192.168.1.0/24. However, using pre-pended AS numbers, RTC is configured to have a longer path than
RTA. During normal operation, the routers within the VPN network (AS 13979) hear both advertisements
but choose RTA as the better route because it has a shorter path. If RTA becomes unreachable, then BGP
will dynamically choose the path to RTC. Additionally, when RTA is restored, BGP will establish RTA
as the shorter path. Other BGP mechanisms besides AS path pre-pending, such as MEDs or sending
community values to set local preference in AVPN, can also be used to make the primary route more
attractive than the backup.
Partial RTA Configuration
router bgp 65000
network 192.168.1.0 mask 255.255.255.0
network 0.0.0.0
no auto-summary
ESC Release 2.4
! Private AS number for Site A

! Advertise Site As default route to other
sites
! AVPN edge PER IP address and ASN

63
Partial RTC Configuration

router bgp 65002
network 192.168.1.0 mask 255.255.255.0
network 192.170.1.0 mask 255.255.255.0
network 192.168.1.0 mask 255.255.255.0
neighbor 192.170.10.2 route-map PREPEND out
no auto-summary
!
!
match ip address 1
set as-path prepend 65002 650002

! Advertise this network to your VPN
! Apply PREPEND route map to outgoing route
for 192.168.1.0/24 advertised to the PER
! Access list defined for all packets
! Define route map name and sequence number
! Route map to act on any packets that match
access-list 1
! All packets matching access-list 1 will be
prepended with AS 65002 twice
! Ensures all other routes are advertised without
the prepended AS information
In this example, network 192.168.1.0/24 advertised from RTC will be pre-pended with additional AS
numbers so that the route is only used if the same network advertisement from RTA disappears for any
reason.
7.4
Dual Hub Site Designs
AT&T has written several routing case studies to aid customers with designing the routing between dual
hub sites. Ask your Technical Sales Consultant if case studies one through five apply to you. If they do
they can review the appropriate cases with you. For example, case study two covers two hub sites with a
backdoor connection (non AVPN connection like a private line or metro ethernet link) with no load
balancing between the hub sites. Case study four is similar but the two hub sites communicate between
each other via the backdoor, with AVPN as a backup.
ESC Release 2.4

64
8 Migration and Hybrid Networks

There may be instances where AVPN is deployed with other transport services like private line, metro
ethernet or traditional frame relay or ATM point-to-point PVCs. This situation might be temporary, e.g.,
during a migration from an existing frame relay network to AVPN. Or, a customer might use point-topoint connections indefinitely, as a compliment to AVPN, for example, to direct specific remote sites to a
specific hub site for Internet traffic.
(Note that when a frame relay or ATM port has multiple PVCs on the same port, AVPN CoS may not
function as intended. This is because congestion can occur on the frame relay or ATM port due to nonAVPN PVC's, for which the PER has no control.)
The following sub-sections provide sample configurations for hybrid scenarios. Section 9.1 provides an
example hybrid topology including potential IGP (EIGRP/OSPF) site configurations. Section 9.2 will
illustrate an example of how traffic can be segregated onto different links using policy routing.
Section 9.3 describes another type of hybrid network where AT&T manages some sites and the customer
manages the other sites. Although all sites are in one VPN, AT&T could use different routing techniques
than the customer but they must be compatible.
8.1
Example Topology
The following Figure 19, is one example of a hybrid architecture. An IGP (such as OSPF or EIGRP)
would be used across the direct frame relay PVCs. BGP can be used for AVPN. Routing a packet out of a
remote site will favor the BGP route by default if the routes are of equal length 26. This is because an
eBGP learned route has a lower administrative distance (20) than all IGPs. For packets destined for the
data center, one may want to use the direct frame relay PVC for better interactive application performance
and traffic segregation. In BGP terminology this is called a backdoor. The backdoor approach gives the
BGP route (when received from a BGP peer) an administrative distance of 200, which is higher than the
IGP, ensuring that the IGP route through the traditional frame relay PVC is used.
Below is a backdoor configuration that results in traffic using AVPN for all the remote-to-remote
connections and using frame relay PVCs for the traffic destined for the data center. This prevents classic
tandem routing through the data center when remote sites communicate with each other. This also
provides some level of redundancy with dual paths to each remote site.
Another application of the hybrid topology can be found in Section 9.2, Policy Routing, which
describes how packets can be routed on AVPN vs. traditional FR based on traffic type, e.g. Web traffic
over traditional frame relay and voice over AVPN.
26
Equal length refers routes with the same prefix. E.g. 192.168.1.0 will be preferred over 192.168.0.0 because it is
more specific. However, if the IGP and BGP routes are both advertising 192.168.1.0, then the BGP route will be
favored.
ESC Release 2.4

65
Figure 19 Hybrid AVPN and Classic Frame Relay Topology
ESC Release 2.4

66
8.1.1
Hub Site Configuration
RTA Partial Configuration

interface Serial 1
description T1 Frame-Relay Port DHEC.xxxxxx
no ip address
encapsulation frame-relay
!
ip address 192.168.10.1 255.255.255.252
no cdp enable
!
description FR PVC to site B
ip address 192.172.10.1 255.255.255.252
!
description FR PVC to site C
ip address 192.172.10.5 255.255.255.252
!
router bgp 65000
network 192.169.1.0 mask 255.255.255.0 backdoor
network 0.0.0.0
network 192.168.1.0 mask 255.255.255.0
no auto-summary
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1

! DLCI chosen to reach the PER- - ietf must be
specified for this PVC!
! DLCI chosen to reach site B using FR
! DLCI chosen to reach site C using FR

! ASN for Site A
! Treat this route from RTB with high admin dist
! Advertise Site As default route to PER
! Treat this route from RTC with high admin dist
! Advertise site As networks
! PER
! Site As default route to Internet router (RTD)
Additional RTA (Hub Site) Statements for EIGRP

router eigrp 10
redistribute bgp 65000 metric 1536 2000 255 1 1500 route-map bgp-in
!Tell EIGRP about all
routes learned over BGP assign metrics equivalent to a
T1. Only redistribute routes specified in route-map
network 192.168.1.0
! Run EIGRP on LAN segment 192.168.1.0
network 192.172.10.0
! Run EIGRP on FR WAN Links to Sites B & C
no auto-summary
!
ESC Release 2.4

67
route-map bgp-in......
! Create a route map to control redistribution to add

stability and prevent accidental routing loops by only
allowing known networks to be redistributed
Additional RTA (Hub Site) Statements for OSPF

router ospf 10
passive-interface ser1.1
network 192.168.1.0 0.0.0.255 area 0
! Don't run OSPF over AVPN link

! Run OSPF on LAN segment 192.168.1.0 area0
network 192.172.10.0 0.0.0.3 area 1
! Run OSPF for site B FR WAN link, area 1
network 192.172.10.4 0.0.0.3 area 2
! Run OSPF for site C FR WAN link, area 2
network 192.168.10.0 0.0.0.3 area 0
! To locally advertise network 192.168.10.0/30
redistribute bgp 65000 metric 2 metric-type 1 subnets route-map bgp-in
In this example, the hub site is OSPF area 0 and each traditional frame relay PVC connects to a remote
site with their own OSPF area.
8.1.2
Remote Site Configuration
RTB (Remote Site) Partial Configuration

router bgp 65001
network 192.169.1.0 mask 255.255.255.0
no auto-summary
! ASN can be private or public

! Treat this route from RTA with high admin
dist.
! Advertise site Bs LAN network
! PER
Note that the redistribution of routes between IGP and BGP applies here, as discussed in Section 5.3.1.
Using this topology, traffic can be segregated and routed over parallel paths using policy routing, which is
addressed in Section 9.2.
Additional RTB (Remote Site) Statements for EIGRP
router eigrp 10
redistribute static
network 192.169.1.0
network 192.172.10.0
no auto-summary
ESC Release 2.4
! Propagate default route to local routers

! Run EIGRP on LAN segment 192.169.1.0
! Run EIGRP across FR WAN Link to Site A

68
Additional RTB (Remote Site) Statements for OSPF

Note that networks in different OSPF areas can communicate directly without going through Area 0 by
traversing AVPN using the BGP learned routes.
router ospf 10
passive-interface serial 1.1
network 192.169.1.0 0.0.255.255 area 1
network 192.172.10.0 0.0.0.3 area 1
network 192.169.10.0 0.0.0.3 area 1
8.2
! Don't run OSPF over AVPN link

! Run OSPF on LAN segment 192.168.1.0 area0
! Run OSPF over Site A FR WAN link
! To locally advertise network 192.169.10.0/30
Policy Based Routing
Because AVPN has different characteristics than other transport technologies, there may be a desire to
place only certain applications on the VPN network. For example, one may want to add AVPN ePVCs to
an existing frame relay network and keep some data traffic on frame relay but move some applications to
AVPN. By implementing Ciscos policy-based routing (PBR) solution, this is possible. For the purposes
of this discussion, we'll assume that the network traffic is split so that the interactive telnet traffic
traverses the VPN network and the bulk FTP traffic traverses the legacy frame relay PVC.
Figure 20 Policy Based Routing Example
In the configuration below, the first line of access list 100 is looking for telnet traffic from RTA to RTB
with a destination port of 23 (telnet). The second line is looking for telnet traffic traveling in the same
direction but with a source port of 23 (telnet). By specifying port 23 as both a source and a destination, all
ESC Release 2.4

69
telnet traffic traveling between these two Ethernet segments will match the access list. Access list 101 is
set up similarly except that ports 21 (ftp) and 20 (ftp-data) are defined.
The redirect route maps are defined to send the matching access-list traffic out the specified serial
interfaces. Lastly, Ethernet0 is enabled so all traffic on the interface will be screened through the defined
route maps.
For the traffic coming from site B to Site A, RTB would have a very similar configuration. It would be
necessary to set up access lists and route maps so RTB would know which interface to send packets out.
RTA (Hub Site) Partial Configuration
interface Ethernet0
ip address 192.168.1.2 255.255.255.0
ip policy route-map redirect
! Enable policy route-map redirect
!
ip address 192.168.10.1 255.255.255.252
no cdp enable
! DLCI to reach PER
!
description FR PVC to site B
ip address 192.172.10.1 255.255.255.252
! DLCI to reach RTB using frame relay
!
router bgp 65000
! ASN for Site A
network 0.0.0.0
! Advertise As default route to AVPN
network 192.168.1.0 mask 255.255.255.0
! Advertise site As networks
! PER peer
no auto-summary
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
! Site As default route to RTD
!
access-list 100 permit tcp 192.168.1.0 0.0.0.255 192.169.1.0.0.0.255 eq telnet
access-list 100 permit tcp 192.168.1.0 0.0.0.255 eq telnet 192.169.1.0 0.0.0.255
access-list 101 permit tcp 192.168.1.0 0.0.0.255 192.169.1.0 0.0.0.255 eq ftp
access-list 101 permit tcp 192.168.1.0 0.0.0.255 192.169.1.0 0.0.0.255 eq ftp-data
access-list 101 permit tcp 192.168.1.0 0.0.0.255 eq ftp 192.169.1.0 0.0.0.255
access-list 101 permit tcp 192.168.1.0 0.0.0.255 eq ftp-data 192.169.1.0 0.0.0.255
!
route-map redirect permit 10
! Route-map name and sequence number
match ip address 100
! Look for packets matching access-list 100
set ip next-hop 192.168.10.2
! Send matching packets (telnet) to RTB via s1.1
route-map redirect permit 20
! Route-map name and sequence number
match ip address 101
! Look for packets matching access-list 101
set ip next-hop 192.172.10.6
! Send matching packets (ftp) to RTB via s1.2
ESC Release 2.4

70
8.3
AT&T Managed Sites Interoperating with Customer Managed Sites
A customer may manage some sites in the VPN while AT&T manages other sites (via the AT&T EVPN
or Managed AVPN offers). In these situations the customer must assure their design interoperates well
with the AT&T managed CERs. Two areas are impacted: COS and routing. For COS, the customer
should work with the sales team and their Data Network Analyst to assure a compatible design. For
routing, the customer should work the sales team and the managed service engineer to understand what
ASNs the AT&T managed CERs will use and any AS prepending or community values AT&T may use
to influence routing decisions. For example, the EVPN service generally uses AS 65000 with AS override
for all managed sites in one region (on the same AT&T ASN) and AS path prepending for backup links.
ESC Release 2.4

71
9 International Considerations
AVPN is a global service with a BGP autonomous system for each region of the world, as show in Figure
21. Although there are separate BGP autonomous systems, there is one, global MPLS network with a full
mesh of label switched paths globally. AVPN uses a variation of the MPLS inter-AS Option C with
multi-protocol eBGP (MPeBGP). MPeBGP runs in a full mesh between inter-region route reflectors in
every region. Physical location of AT&T routers and physical circuit routing is not shown. Some AT&T
routers in AS 21326 and 21302 may be physically in the U.S.
Figure 21 International AVPN BGP Autonomous Systems
Notes on the ASNs shown in the diagram:
Connectivity between ASs is shown by either overlapping the ovals or connecting with the
double black line. This is just representing routing connectivity between the ASs.
NBFW is the Network Based Fire-Wall service, which uses those same three ASNs in each of the
five AT&T regions.
China Telcoms legacy MPLS service (pre-CN2) uses metropolitan MPLS services which each
use a private ASN. These ASNs have the potential to conflict with customer private ASNs. Check
with AT&T provisioning to pick ASNs that will not conflict with those used by the China
metropolitan MPLS services.
Although much of the diagram and some of this section also apply to IPFR, IPFR uses IP-VCs to connect
customer VPNs across regions, not Option C. These virtual circuits are dedicated to the customer and
engineered similar to an ePVC with bandwidth and class of service while residing on high speed ports
engineered by AT&T. See the IPFR Customer Router Configuration Guide for details.
ESC Release 2.4

72
MLPPP and lower speed dedicated (IP) ports outside the U.S. may have varying feature support.
Discussions about backup routes across regions (e.g. for the default route) and inter-region route filtering,
are in the following subsections.
9.1
NNIs with Partners to Extend Reach
To extend AVPNs reach in countries that AT&T cannot own a network or to just to provide further
points of presence in a country, AT&T partners with providers via network-to-network interfaces (NNIs)
either at layer 2 (essentially long haul access) or layer 3 (at the IP level, when the provider has their own
MPLS VPN service). Partner MPLS VPN networks may or may not support the same features as AT&T
PERs such as routing protocol support, BGP attributes, load balancing and COS.
The NNIs are redundant, deployed in pairs. Since the PERs in AVPN have eiBGP multipath enabled,
those AT&T PERs will load share packets going toward the NNI. The partner may or may not have
multipath enabled, so whether packets from partner PERs load share toward the NNI is up to the
particular partner. When this document was written, most NNI partners were doing closest exit routing.
They could also use multipath or primary-backup. AMX just started doing primary-backup (circa Jan
2015).
Some features that are not available through any layer 3 NNIs are: multicast, route groups, and hub-spoke
VPNs.
9.2
Using BGP Community Values to Have AT&T Set Local Preference
As mentioned in section 14.2.4, the CER can advertise routes with BGP community values to cause the
PER to set a corresponding BGP local preference within its own region (say EMEA). This only affects the
routing of packets originating within the same region. If a site in another region, say AP, sends a packet,
the AP PER will make the routing decision and it does not have the local preference change, and no other
IP routing lookup is performed in the backbone as the packet goes into the destination region (EMEA)
and into the destination PER.
The next few diagrams give example cases using CVs to set local preference.
The first example below is for sites using ANIRA as backup. The ANIRA gateway (VIG) advertises the
routes into the VPN with a local preference of 90 as a backup to the normal routes which have the default
LP=100. Note that this works, even with a global AVPN VPN.
ESC Release 2.4

73
DC
10.1.1.0 LP=100 21326 64555 i

10.1.1.0 LP=100 21326 64555 i
PER at VIG:
10.1.1.0 LP=100 21326 64555 I *
10.1.1.0 LP= 90 65000 i
US
AS 13979
EMEA
AS 21302
* Best route, so withdraws its own

route (the one from the VIG).
VIG
Asia Pac
AS 21326
Remote
Site
BGP AS
65000
Internet
Primary path
Backup path
Remote
Site
Remote Site
10.1.1.0
BGP AS
64555
Second case: Note that with this case, the third region (AP) will pick the best path based on which
destination PER is closest to the originating PER, based on the lowest OSPF metric to the BGP next hop
(the destination PER) in the AT&T backbone. The OSPF metrics are generally proportional to latency.
BGP AS 65000
CV=13979:100
BGP AS 65001
CV=13979:110
DC U.S.
DC EMEA
> 0.0.0.0 LP=110 65001 i

0.0.0.0 LP=100 13979 65000 i
> 0.0.0.0 LP=100 65000 i*

0.0.0.0 LP=100 21302 65001 i
EMEA
AS 21302
US
AS 13979
Asia Pac
AS 21326
Remote
Site
RRs do not pick the best path because

they see the Route Distinguishers
which make the prefixes unique.
> 0.0.0.0 LP=100 65000 i*

0.0.0.0 LP=100 21302 65001 i
LP=100 because not passed
across regional ASs
Remote
Site
0.0.0.0 LP=100 65001 i OSPF= 2000

>0.0.0.0 LP=100 65000 i* OSPF=1000
Remote
Site
All attributes are equal so go to closest exit

rule: lowest IGP metric to BGP next hop
(destination PER)
Note that AT&T partner networks, which we connect to with NNIs using the MPLS Inter-AS Option A
standard, may or may not support the setting of local preference in their network, based on BGP CVs that
customers send.
9.3
Inter-Region Route Filtering
Inter-region BGP route reflectors exchange BGP routes between AT&T regions. To add routing
flexibility for customers, they have route maps that match on BGP community values on routes received
ESC Release 2.4

74
from another region to deny routes or set local preference. The inter-region route reflectors look for these
BGP community values (CVs), set by the customer, and take the following action on routes received from
another region:
Community
Value
Region
Receiving
the Route
RR Action on the Route
13979:6300
Any-other
Deny
13979:6301
US
Deny
13979:6302
AP
Deny
13979:6303
EMEA
Deny
13979:6304
LA
Deny
13979:6305
Canada
Deny
13979:6311
US
Set BGP Local Preference=110
13979:6312
AP
13979:6313
EMEA
13979:6314
LA
13979:6315
Canada
13979:6321
US
13979:6322
AP
13979:6323
EMEA
13979:6324
LA
13979:6325
Canada
Note that this does not apply to VPNs with IPVCs (e.g. VPNs that were created years ago under IPFR
service; nor does it apply to NNI partner networks which use connections similar to IPVCs.)
As an example, assume that you want sites in AP and EMEA to prefer the default route coming from a
site in AP. See Figure 22, below. The default route site in AP would advertise 0.0.0.0 with CVs
13979:110 and 13979:6313. This causes the AT&T routers to set the BGP local preference for the default
route to 110 in AP and EMEA, respectively. 100 is the default local preference value. The higher the
value, the more preferred the route. Note that setting the local preference to 110 in your local region (CV
13979:110) may not be necessary since the default route coming from anyother region will have a longer
AS path length. Another example might be that the U.S. and Canada are to prefer the US default route.
The CER in the US would advertise 0.0.0.0 with CV=13979:110 and 13979:6315.
ESC Release 2.4

75
Remember that eBGP does not carry the BGP local preference attribute to another AS (thus not to another
region). That is why AT&T configured the route-maps on the borders with other regions to match on the
community values and set the local preference in the local AS (local region).
Figure 22 Inter-Region Default Route Preference
US Default
0.0.0.0/0
CV 13979:110
CV 13979:6315
US
13979
BGP Local Preference
US Originated Default = 110
AP Originated Default = 100
CE
AP Default
0.0.0.0/0
CV 13979:110
CV 13979:6313
CE
Inter-AS RRs
Full-Mesh
Bi-Directional
Route Exchange
Inter-AS RRs
AP
21326
EMEA
21302
Inter-AS RRs
CALA
8035

Canada
8034
Example CER configuration for the AP site that advertises the default route in our example:
router bgp 65000
network 0.0.0.0
! advertise the default route. Note that it must existing in the IP routing
table for BGP to advertise it.
neighbor <PER IP address> send-community

neighbor <PER IP address> route-map SETCV out
route-map SETCV permit 10

match ip address prefix-list DEFAULT
set community 13979:110 13979:6313 additive
route-map SETCV permit 20
ip prefix-list DEFAULT seq 10 permit 0.0.0.0/0
9.4
! adds both these values to any existing values
! matches 0.0.0.0, only

! implicit deny at the end.
Backup Paths
To provide backup paths in a global network, a combination of AS-path prepending and community
values to set local preference in the VPN may be used. Sending a community value to AVPN to set local
preference sets that preference in the AVPN regional autonomous system (not globally in all ASNs). See
ESC Release 2.4

76
Section 14.2.4 for the list of community values supported (note that all regions use the same community
values, starting with 13979 even though the PER ASN may not be 13979).
For example, suppose a customer wants sites in EMEA to use an EMEA data center as their Internet
gateway and Asia Pac to use an Asia Pac data center, but if the regional data center fails, use a data center
in the U.S. for Internet access. See Figure 23 below:
Figure 23 Global Default Routing
If each data center advertised a default route without changing any BGP attributes, sites in a region will
prefer the default route from the data center in that region, because the other default routes will have an
extra AS hop. If the EMEA data centers connection failed, thus no longer advertising the default route,
the PERs in EMEA will see the default from the U.S. and the default from Asia Pac. Both will have the
same AS path length. Each PER in EMEA will choose the default with the lower OSPF metric to its BGP
next hop (which is either the PER serving the data center (DC) in the US or the PE serving the DC in AP,
which ever is approximately the best latency path).
The customer may want to dictate which one is preferred. So, we need to manipulate another BGP
attribute to make this occur. One such attribute is local preference. A customer router when advertising
a route to AVPN can tell the PE to set local preference to a specific value that will then be used within
that AT&T region (AT&T AS). Note that it will not get set in other ASs (regions). Since local preference
does not leave the regional AS but AS path does, one recommended solution is to use local preference to
pick the primary, in-region default route and use AS path length to determine the preferred backup (outof-region) route. This solution would look like the following:
The EMEA and Asia Pac data centers advertise their default routes to AVPN with the BGP community
value 13979:110 that tells AT&T to set the local preference to 110 (the default local preference is 100,
and routers prefer a higher value), and with a longer AS path with the set as-path prepend command. This
works because local preference is only set within the regions AS and a higher local preference beats a
shorter AS path length. PERs in each region will prefer the default route with the highest local preference,
regardless of AS-path length. If the EMEA regional default route fails, all remaining default routes
received in EMEA will have a default local preference of 100. The US route would have a shorter AS
ESC Release 2.4

77
path than Asia Pac, and thus would be preferred (the Asia Pac default route has a longer AS path due to
the prepending). The number of ASNs to prepend depends on the specific topology. An example
configuration is shown below for a site in EMEA:
router bgp 65001
neighbor 192.168.1.2 send-community
neighbor 192.168.1.2 route-map default-route out
..
!
ip bgp-community new-format
!
!
route-map default-route permit 10
match ip address 20
set as-path prepend 65001 65001
set community 13979:110
! note that you use 13979 in all regions
route-map default-route permit 20
! allow advertisement of all other routes but
without altering the AS path and community
Using the above configuration, the default route will show up at the AT&T PER with the following
attributes:
EMEA-PER>sh ip bgp 0.0.0.0
BGP routing table entry for 0.0.0.0/0, version 4
Paths: (1 available, best #1)
65001 65001 65001
192.168.1.1 from 192.168.1.1 (192.168.1.1)
Origin IGP, metric 0, localpref 110, valid, external, best, ref 2
Community: 13979:110
Note: If you are using IPFR service, which uses IP-VCs between regions instead of option C MPeBGP,
refer to the IPFR customer router configuration guide. The configuration for this specific solution may be
the same but other scenarios may require a different solution.
9.5
Load Sharing
The load sharing feature operates differently when the port originating the packets is on a Juniper PER
and that same PER has one or more of the destination load-shared ports. See Figure 24, below (PER1
being a Juniper). All the traffic from remote site CER3 will go to the link on the same PER (to CER1).
This is similar to the static route case discussed in Section 6.2.1 on the AVPN load sharing feature. Note
that PER3 will use automatic load sharing to distribute IP flows from CER4 to CER1 and CER2,
regardless of the port type at CER4.
ESC Release 2.4

78
Figure 24 Load Sharing Inbound, International Sites, NxT1 speeds or Lower
ESC Release 2.4

79
10 AT&T NetBond, IaaS (Infrastructure as a Service)

AT&T NetBond connects your AVPN VPN directly to cloud service providers such as AT&T Computeas-a-Service, Microsoft Azure, IBM CMS and Salesforce.com. There are two different connectivity
models, one for infrastructure-as-a-service providers and one for software-as-a-service providers (such as
Salesforce.com). This section addresses IaaS. The next section addresses SaaS.
A diagram of the IaaS connectivity architecture is shown below in Figure 25. Generally, your AVPN
VPN advertises all routes to your NetBond VPN and vice versa, allowing any-to-any connectivity. There
are some exceptions discussed below.
Figure 25 AT&T NetBond
The key elements in the connection are the AVPN VPN, the cloud VPN and the VLANs to the cloud
service provider (CSP). These make up the customers virtual network connection (VNC).
When the customer creates the VNC on the AT&T web portal, the VLAN page asks the customer to
provide a VLAN IP or Direct Subnet which is the IP subnet AT&T will configure on the VLAN
which connects your VPN to the CSP. The subnet you pick should be out of your private network address
space (no need to be a publicly registered address) and should generally have a /29 prefix length. The
AT&T NetBond orchestrator then splits this subnet in two and configures them on the primary and
backup VLANs to the CSP, as shown in the diagram below.
ESC Release 2.4

80
Figure 26 NetBond VNC Details with VLAN IP Addressing
Note that the exact LAN architecture varies between CSPs (e.g. between AT&T CaaS, IBM CMS,
Microsoft Azure, etc.).
The next three subsections cover:
1. IP Routing
2. Mobility CCS
3. COS
Caveats
Though this guide is not a service description, here are some caveats known when this guide was written:
Multicast traffic will not traverse NetBond.
Route limits with some CSPs. See Section 10.1.2. Note that there may be NetBond options that
will send only the default route to the CSP, to solve CSP route limits.
Hub-Spoke VPN not available with with NetBond.
There are variations to the connections with different CSPs. To confirm details for a specific CSP, please
contact your Technical Sales Consultant.
10.1 IP Routing
In general, all routes in the customers AVPN VPN are passed to the AT&T Cloud Service VPN and
then on to the cloud service providers routers at their data center. And, the cloud service provider
advertises their routes to NetBond which passes them on to the customers AVPN VPN. There are some
exceptions, particularly for CSPs that have low limits on the number of routes they can accept.
The customers AT&T cloud VPN is global for a customer and has the same AS globally (8030). The
cloud VPN serves all CSPs and CSP data centers that the customer subscribes to globally. There is only a
separate cloud VPN if the customer uses a second AVPN VPN.
ESC Release 2.4

81
The customer cloud VPN imports routes from the customer AVPN VPN with an RFC 1998 route-map
that matches on BGP CVs to set BGP local preference within AS 8030. If the customer sets the
following values on routes it advertises from CERs to AVPN, then the cloud VPN (8030) will set the
corresponding local preference value:
BGP CV
Local Preference
set within 8030
13979:80
80
13979:90
90
13979:100
100
13979:110
110
13979:120
120
The following subsections describe load sharing, a route filtering option and considerations for migrating
from an existing internet connection with a CSP to a NetBond connection.
10.1.1 Load Sharing
Traffic going from NetBond to AVPN is not load shared (the IPEs do not have BGP multipath
configured).
Traffic going from AVPN to NetBond is load shared via the normal AVPN behavior (BGP multipath on
the AVPN PERs). This requires the customer to have multiple NetBond VNCs to the same CSP and the
CSP advertising the same routes with the same BGP attributes toward NetBond on both VNCs.
10.1.2 Route Filtering, Route Limits
These two needs are closely related. Route filtering is for a customer to limit subnets advertised to the
CSP, in general, for any reason. The second topic is the specific need to deal with CSPs that cannot accept
the full customer AVPN VPN routing table.
Route Filtering
The customer may want to limit which routes AT&T advertises to the CSP, so there is a route filtering
option using a BGP community value (this is a NetBond option so it works with all CSPs). In the web
portal when creating a VLAN for a VNC the customer can check a box called CV Option or Community
Value Tag to turn on the route filtering. When the option is checked or true, the iPE only imports
customer routes that have the BGP CV 8030:999. The customer must advertise routes with the BGP CV
8030:999 to get the iPE to import and advertise them to the CSP. If the BGP CV 8030:999 is not present
on a route, the iPE will not import the route from your AVPN VPN and the CSP will not receive it. The
portal has this option for each VNC; however, it is applied to the customers cloud VPN, so it applies to
all VNCs in your cloud VPN. If you select the option differently on different VNCs for the same AVPN
VPN, then there may be a conflict. Contact your sales team for technical support.
Route Limits
If you need to meet a CSP route limit on the maximum number of routes they can handle, there are a few
options.
ESC Release 2.4

82
For some CSPs, NetBond is planning to only send the default route or make that an option.
If that is not available or does not fit the customer need, one could limit the number of routes and
maintain full connectivity by using the route filtering option to send only summary routes or a default
route to the CSP. There are a few cautions. First, check with your CSP before relying solely on the
default route to be sure the CSP does not have another default route that might conflict. Second, note that
when the CSP has a packet to send to a remote site on your VPN, the packet will follow the summary
route to your hub site that advertised it. Your hub site router can then send the packet back out to your
AVPN VPN to get to the remote site. (The reason the packet goes back to the hub site, even though the
VPN has the specific route to the remote site is because the iPE has a direct MPLS label switched path to
the next hop of the summary route, which is the customer hub site. MPLS does not do another IP lookup
while traversing the AVPN VPN (to find the more specific route). So packets will tandem route through
the data center, back into the VPN and over to the remote site. See the diagram below.
Figure 27 Sending a Summary Route with Route Filtering
10.1.3 Migrating to NetBond

If the customer has an existing connection with a cloud service provider (non-NetBond, say an IPSec
tunnel over the internet) and migrates to NetBond, the customer will want to control the routing
preferences to migrate traffic smoothly, by having routers choose the right path at the right time.
Assuming the customer wants to use their existing connection until they have tested NetBond, the
customer routers need to continue to prefer the existing connection when NetBond first comes up, since
NetBond will immediately advertise the CSP routes into the AVPN VPN.
ESC Release 2.4

83
Note that some CSPs do not allow both connections up at the same time, such as Microsoft. In this case,
one must just terminate the Internet IPsec tunnel and bring up NetBond.
One way to accomplish a migration when both connections can be up at the same time is to specify local
preference (as in RFC 1998) on your existing CSP route advertised into your AVPN VPN, before turning
up NetBond. You can do this by advertising the CSPs subnet that you receive from your existing internet
connection and adding the BGP community value 13979:110. This tells AVPN to set local preference to
110 (higher than the default of 100). Then, when NetBond comes up, the IP subnet from NetBond will be
less preferred, because it will have the default local preference of 100.
See the network diagram in Figure 28 below.
If your AVPN VPN spans regions, you will also need to add on the BGP CVs for inter-region route
preference, i.e., 13979:6312, 6313, 6314, 6315 for local preference of 110 in each region (see Section
10.3, Inter-Region Route Filtering).
This assumes the subnet from the CSP through NetBond is the same prefix length (mask) as the one
through the internet IPSec tunnel. All routers pick the most specific route first (the smallest IP subnet,
longest prefix).
Routing in the other direction also needs considered. The CSP will receive the customers routes from
both the IPSec tunnel and NetBond. The best solution will depend on how the customer is advertising its
subnets to the CSP and any firewall or NAT functions the customer is doing, compared to NetBonds
advertisements of the AVPN VPN routes to the CSP. NetBond advertises all VPN routes to the CSP as is,
without summarizing or modifying BGP attributes (except adding AS 8030 to the AS path) or filtering
(unless specifically ordered by the customer). Determining the best solution will probably require a
discussion with your CSP.
ESC Release 2.4

84
Figure 28 Migrating from Internet Connection to NetBond
If this is the first NetBond connection enabled on a given AVPN VPN, the AVPN and NetBond route
reflectors first exchange routing tables overnight before routes are received by the CSP. During the time
between enablement and the route reflector update, the customer will see the CSP routes in AVPN but the
CSP will not see the customer routes. This creates a time slot where the above solution will work without
the CSP implementing anything on their end. As soon as the route reflector update occurs in each region
the asymmetric path may occur.
If this is not the first NetBond connection on this AVPN VPN, the routing is updated within minutes of
creating the VNC in the AT&T portal.
When possible migrate during your company maintenance window. This is possible if 24 hours ahead of
time you create your first NetBond connection to a non-production cloud service resource, to force the
route reflector updates from the AVPN VPN. Once that has occurred when you enable the production
NetBond connection, it will complete within minutes, during the same maintenance window that you take
down the IPSEC connection.
10.1.4 Multi-Data Center, Multi-Region Designs
There are more complex network designs that a customer may require, which are not addressed in this
guide. AT&T has documented some specific case studies that your Technical Sales Consultant can assist
customers with. They involve VNCs to multiple CSP data centers and VPNs spanning geographic regions
(U.S., Canada, EMEA, etc.).
ESC Release 2.4

85
10.2 Mobiilty CCS to NetBond

Section 12 describes the capability to connect your AVPN VPN to your AT&T Mobility APN (access
point name) closed user group. If you have this provisioned on your AVPN VPN, the mobility users can
also reach your NetBond CSPs. This works whether you are on the original, pre-LTE CCS or the new
CCS. To make this happen, routes advertised to mobility CCS need a specific BGP community value
attached: 13979:6655. Mobiility CCS only imports routes from your VPN that have that BGP CV.
Original, pre-LTE CCS requires the customer to advertise NetBond routes with the mobility CCS BGP
CVs. That is most likely not possible from the CSP directly so the customer can advertise a summary
route with the mobility BGP CV from a hub site.
For the new CCS, NetBond will automatically add the mobility BGP CV to all CSP routes, so mobility
will then import them. 27
10.3 COS
A customer with NetBond has full use of the NetBond infrastructure resources including bandwidth.
NetBond does not limit bandwidth per customer, so COS is not needed within NetBond. (Note that when
provisioning NetBond the customer does specify a bandwidth commitment but that is just a pricing
commitment, not translated to a technical configuration on routers or switches in the network). Normal
MPLS VPN COS works as usual on the existing AVPN AVPN ports. AT&T is not involved in the cloud
service providers COS, traffic shaping or rate limiting, if any.
To understand COS, one must look end-to-end, from IP host to IP host, in each direction and at each
potential bottleneck. See the diagram below in Figure 29. The most likely bottleneck is at the lowest
speed circuits, which are probably at the remote sites on AVPN. The CER controls COS queuing on that
circuit for packets leaving the CER going to AVPN. For packets going in the other direction, from the
CSP to the AVPN remote site, the PER uses normal AVPN COS, as ordered by the customer. A packet
sourced by the CSP needs marked with an appropriate IP DSCP according to the needs of the customer,
so that the PER can put it in the appropriate COS queue as defined by the customer.
27
NetBond by default adds the CV. The customer does not need to order it.
ESC Release 2.4

86
Figure 29 COS with NetBond
ESC Release 2.4

87
11 AT&T NetBond with SaaS Providers

NetBond with software-as-a-service providers is similar to compute/infrastructure providers, discussed in
the previous section, but with an IP network address translation (NAT) function in NetBond between the
customers VPN and the SaaS providers network. See the diagram below.
Figure 30 NetBond with a SaaS Provider
NetBond advertises the CSPs routes into the customers VPN. All packets going from the customers
VPN to the CSP have their private source IP address translated to an AT&T publicly registered address 28
dedicated to this customers NetBond VNC (virtual network connection; see previous section). The
destination IP address is the CSPs public IP address, usually obtained from a public DNS lookup.
The VNC in this situation is the combination of the customers AVPN VPN, cloud service VPN, and
VLAN to a customer-dedicated vitual NAT router within the NetBond infrastructure.
More information will follow in this documents next release.
28
Though a publicly registered address, it not routable on the internet
ESC Release 2.4

88
12 Mobility CCS (Commercial Connectivity Service)

Mobility CCS connects your mobility APN (access point name) closed user group to your AVPN VPN,
so that mobile users can access your VPN directly, via the AT&T backbone. There are two technical
implementations:
1. Original (pre-LTE) CCS which uses MPLS Option A between mobility and AVPN.
2. New CCS which uses MPLS Option B.
Mobility CCS (both types) imports only routes with the BGP community value 13979:6655 from an
AVPN VPN.
For the original CCS, you only need to add the BGP CV to summary routes (the summary route gets the
packet to an AVPN PER which then does an IP lookup using the customers full VPN routing table).
The rest of this section covers the new CCS.
12.1 New CCS to AVPN Service Architecture Overview
The CCS service binds a mobility APN to a customers AVPN VPN using an RFC 4364 Option B interAS connection. The mobility CCS router is always in BGP AS 20057, which is connected to the AVPN
infrastructure by means of Option B and a series of inter-AS route reflectors. With such an inter-AS path,
traffic between a customer AVPN location and the CCS service will take the best physical path through
the network, regardless of the number of BGP autonomous systems through which the advertisements
pass. The number of BGP hops has no bearing on the actual MPLS label switched path (LSP) that is
created. While routing advertisements will traverse a series of route reflectors in the AT&T MPLS
network, data traffic will be MPLS switched end-to-end over the shortest possible route.
Note that AS 20057 peers with AT&Ts AS 7018, which in turn peers with the regional US AVPN AS
13979, so customers will see these AS numbers in the BGP path of a CCS-connected endpoint. For sites
outside the US on the AVPN network, customers will see the BGP path pass through the US AS of
13979.
For example, say a CCS endpoint in the US was exchanging data with a customer AVPN location in
Germany. The CER at the location Germany would see the prefix that was advertised from CCS as having
the following AS Path: 2005770181397921302. The actual MPLS LSP that carries traffic goes
directly from the CCS router in the US to the AVPN PER in Germany over the best path available end-toend through the network. Figure 30 shows the path of routing advertisements end to end, as well as the
physical path of the data in the LSP.
ESC Release 2.4

89
Figure 31 End to End Routing Path

MPiBGP
MPeBGP
AS 7018
AS 20057
CCS Router
MPLS
Option B
AS 13979
US
AS 21302
EMEA
(or AP, L.Am., Ca)
AT&T PER
MPLS Label Switched Path

PER
Customer CER
12.2 Routing between AVPN and CCS endpoints

Connecting the CCS service to an AVPN VPN allows the CCS endpoints or devices full access to ones
private MPLS-based IP network. There are some routing limitations that one must consider, though.
The CCS service limits the number of individual prefix advertisements it is permitted to receive from
AVPN. Whereas the AVPN network may have hundreds or even thousands of BGP routes, CCS is
limited to receiving only 50 prefixes 29. For this reason, the service allows the customer to control which
specific prefixes are advertised onto the CCS service through the use of BGP Community Value
13979:6655. Only routes which have this community attached will be passed on to the CCS service.
Any routes which do not have CV 13979:6655 will not be advertised to the mobility CCS gateway router.
In the opposite direction, mobility automaticallys advertises the CCS endpoint subnets to the AVPN
service, and all AVPN endpoints will see all CCS advertised routes.
In many cases, CCS endpoints only need to communicate with a data center or a limited number of key
locations. In this case, it is only necessary for the data center to append the CV 13979:6655 onto its route
advertisements, or potentially even just on a summary or default route.
12.3 Use of the Default Route or Summary Route
Customers may consider leveraging a default or summary route to overcome this limit of 50 routes that
can be accepted by the CCS router. This method can work, but there is an important limitation: any
traffic following the default or summary route will be routed to the advertising CER before the packet can
be forwarded onto its final destination.
Consider the example in Figure 31 below. A remote AVPN site advertises subnet 10.20.20.0/24 to the
AVPN service, and CCS advertises subnet 10.10.10.0/24 to AVPN. A third site at a data center advertises
the default route, and also appends BGP community value 13979:6655 to this advertisement. The remote
site now sends a packet to destination 10.10.10.10. The CER forwards the packet to the PER, and the PER
sends the traffic directly to the PER that serves the CCS site.
29
This is a soft limit. AT&T CCS service will consider increasing this number if needed, but cannot increase the
value beyond approximately 300 individual prefixes.
ESC Release 2.4

90
In the return direction, the CCS CER does not have a route back to the 10.20.20.0/24 network, since that
site did not append the CV 13979:6655 to the advertisement. The most specific route it has to this subnet
is the default route advertised from the data center. The MPLS network will forward this packet with a
destination address of 10.20.20.20 to the CER at the data center, which will, in turn, forward the packet
back to its PER and thence to the final destination.
This tandem routing through a data center for traffic from the CCS service to remote endpoints on the
AVPN network can work, but customers should consider the geographic locations of the respective sites
(and hence latency) and the load that traffic in this direction will place on the data center port.
Figure 32 - Tandem routing with a default route
CCS route
advertisement
10.10.10.0/24
10.10.10.0/24
AS 7018
AS 20057
CCS CER
D:10.20.20.20
Data Packet
AS 13979
US PER
US PER
Summary route
advertisement
Data Packet
D:10.10.10.10
Remote Site
CER
10.20.20.0/24
Data Center
CER
0.0.0.0
CV 13979:6655
12.4 CCS with AVPN type-converted VPNs

Some customers initially used AT&T EVPN or IPFR MPLS services, and then later converted these to
AVPN. These converted VPNs can still leverage the full suite of additional services offered by AVPN,
including connections to the CCS service, but there can potentially be routing issues between CCS and
sites on a converted VPN if the CCS service is not specifically configured for this. Customers who
believe they are on VPNs that were converted to AVPN should inquire with their technical sales support
to confirm if this is the case. If so, the AT&T technical support team will ensure the CCS connection is
configured correctly for the type converted VPN.
ESC Release 2.4

91
13 Alternative Access
13.1 ANIRA, Broadband Internet
ANIRA introduces a set of remote office and remote user broadband options for access to AVPN.
Figure 33 AVPN Remote Access, ANIRA
Following is a summary of the ANIRA key features:
Broadband, cellular, analog dial, and ISDN backup, including 3rd party internet service providers.
IP service
Small Office Home Office (SOHO) - Single user (AT&T or third party SW Client) and multi-user
access (select VPN Remote Access Devices, including NetGates & Cisco)
Access via AT&T DSL (aka IP DSL) and AT&T Managed Internet Service
AT&T and customer managed authentication (Secure ID, RADIUS, Safeword) over secure
network paths (Frame Relay Service, ATM Service, IPFR and AVPN)
Dual tunneling (Access to the AVPN and Internet Service)
Support for registered and unregistered IP addresses
Dial IPSEC and L2TP support
No requirement for specialized CPE at the customer network premise
Usage based and fixed rate price plans
Global Roaming
Extended Access
WiFi
Global service
ANIRA introduces some unique routing situations, particularly if it is used together with standard access
to AVPN from the same site. The ANIRA gateway (VIG) between the Internet and AVPN represents a
CER with ASN 65000 for eBGP to a PER. The gateway sets the MED and local preference on routes to
AVPN to set them as less preferred for backup in case the remote site also has primary access via a
private link to AVPN. The customer can request ANIRA to use any AS number for the gateway. They can
ESC Release 2.4

92
also request the gateway to add a BGP community value to all routes that it advertises into the VPN
(same value on all routes). Please contact your AT&T sales team technical support.
13.2 DSL
AT&T offers the ability to access the VPN network with DSL. With a DSL port a customer can order an
AVPN ePVC. While the appeal of this offer is potentially lower priced access to AVPN, there are some
ordering constraints.
First, although you can order COS, AT&T cannot guarantee CoS performance on DSL AVPN
ports because of possible congestion in the DSL providers network.
Second, there is no LMI notification from the DSL network to AT&T on DSL. If a customer
wishes to use dial backup with a DSL ePVC, then BGP should be used.
ESC Release 2.4

93
14 Other Considerations With AVPN Service

The following topics are included in this section:
1. Testing (pinging the PER, large ping packets to PERs, ICMP messages from public addresses and
failover/convergence tests)
2. BGP Attributes (MED, Local Preference and Community Values)
3. BGP Timers
4. VPN Route Limit
5. Route Dampening
6. Broadcast, Directed Broadcast, Bridging, Auto-Install and CDP
7. GRE Tunnels
8. Remote Site Reach-Ability Status
9. Restrictive Route Filtering (Default Route Only)
10. Outbound Route Filtering (ORF)
11. Network Based Firewall
12. MTU
13. Trace-Route
14.1 Testing (Pings, ICMP messages, and Route Convergence)
14.1.1 PERs Rate-Limiting Control Traffic
To keep PERs secure and stable, AT&T configures filters and rate limits on packets destinated for the PE
itself, such as pings, other ICMP packets, BGP, etc. Customers using continuous probes to the PERs
could result in false negativesfailed pings or unusually long delay measurements, because of these
filters, when there is nothing wrong with the data plane (forwarding of user data packets). Thus, AT&T
does not recommend continuous probes or other control protocols such as EIGPR, OSPF, etc.
14.1.2 Large Ping Packets to PERs
During test and turn-up a customer will probably choose to run several different ping tests. Other than a
standard ping test from CE to PE to prove L3 connectivity, AT&T strongly recommends that customers
only ping from CE to CE or from LAN device to LAN device to test L3 connectivity across their VPN.
This is especially true if customers are trying to get a rough idea of latency across their network. In
addition to standard ping tests, some customers may choose to do extended ping tests and vary the size of
the ping packets from small to large. Another possible ping test is to send large packets from the CE to
the PE. These pings will typically fail. The reason for this is either that the PER will either deny any
ICMP fragments or the PER has a higher MTU. Most CE WAN interfaces (e.g., Frame Relay) will have
an MTU (1500 bytes) which is smaller than the MTU of the AT&T PE (4470 bytes). As long as all CEs
have the same MTU, data traffic is unaffected. However, for ICMP echo packets sent directly to and
destined for the PE, the PE generates a reply packet and sends it to the CE. Since the PEs MTU is large
(generally 4470), it does not have to fragment the packet. However, the CE receiving the frame on its
interface configured with a smaller MTU (i.e., 1500 bytes), will see the frame as an error and the ping
will fail. This is why AT&T recommends that customers run all testsother than proving basic layer 3
connectivity to the AT&T PEfrom CE to CE or LAN to LAN.
ESC Release 2.4

94
14.1.3 ICMP Messages from Public Addresses

Under certain conditions the AVPN network may generate an ICMP error message whose source and
destination IP addresses are from AT&T's public registered space. This is not indicative of a
provisioning mistake or an unauthorized site on the customer's VPN, but a customers intrusion detection
systems could falsely report it as such. This issue can occur when a CE-to-PE link is first configured on
the PER with BGP but the CE-PE interface is down on the PE.
The details are as follows: The PER that has the CE-PE interface down is already configured with BGP so
it will attempt to establish the BGP session with the CE by sending a TCP SYN message to the CE's
interface address. Since that directly connected interface (or route) is down on the PER, the PER uses the
next best route to forward the packet. This next best route is usually the default route being advertised by
one of the customers Internet gateway or hub sites. The PE will thus send the TCP SYN packet, with a
time-to-live of one out a trunk towards the MPLS core following the next hop interface for the default
route. The PER originating the TCP SYN packet will use it's internal backbone interface address as the
source address for this packet. The next-hop AT&T router will reduce the TTL of this packet to zero.
Since this router was not the actual destination for the packet, it generates an ICMP error message sourced
with its AT&T assigned node IP address and a destination address of the PER's interface address that
originated the packet. These node IP addresses are from AT&T assigned space. However, since none of
these AT&T node addresses are actually reachable from the customers VPN or from the Internet, the
only route valid for forwarding the ICMP error packet is towards the customers default route. This is
why the ICMP error messages could make it as far the customers Internet gateway.
This is not a mistake, but a customers intrusion detection systems might falsely report it as a possible
security issue. None of the registered AT&T IP addresses used on backbone nodes are reachable from
either the Internet or from within the customer VPN, thus there is no security issue.
14.1.4 Failover, Route Convergence Tests
Routing logic is designed to detect the most common real-world failure scenarios, primarily circuit cuts or
circuit failures. If that occurs, standard long-haul circuits, such as DS1, DS3 and OCx circuits are
designed to detect failures and forward a layer 1 alarm upstream and downstream. Routers, when they see
the layer 1 alarm, declare the interface down (sometimes after a slight delay, called carrier delay or
alarm delay). BGPs default behavior is fast external fallover which means that when the interface
fails it will tear down the BGP session and withdrawal routes, not waiting for the BGP holddown timer to
expire.
When testing this behavior, if you simulate the circuit failure with the shutdown command on the
interface, some routers will not generate the layer 1 alarm. Consequently, the PER will not receive a layer
1 alarm and withdrawal routes immediately, as it would if a circuit actually failed (the PER will not know
of the failure until the layer 2 or layer 3 keepalives fail).
The preferred testing method is to pull the cable on the network side of any CSU/DSU. However, to test
using shutdown, you might try using the loopback local command for T1 and DS3 interfaces and
pos ais-shuton OCx interfaces. This should cause the CER to generate a layer 1 AIS alarm toward the
PER, as a real circuit failure would.
14.2 Primary-Backup via BGP Attributes (MED, Local Preference and Community Values)
In general the best practice for using BGP attributes to set primary and backup links for the same route, is
as follows, for two links at the:
same site, with the same customer ASN: use MED (low on the primary, high on the backup)
ESC Release 2.4

95
same AVPN AS region (both to the same AVPN ASN): set community value so AVPN sets local
preference which stays within that region; recommend different ASNs for the different customer
sites
different AVPN AS regions (e.g. internet gateways in each region of the world): AS path
prepending; recommend different ASNs for the different customer sites.
MEDs: AVPN accepts and uses the multi-exit discriminator on routes received from the customer's CER
in its BGP path selection process. AVPN sets all MEDs to 0 or blank before sending routes to eBGP
neighbors such as customer's CERs and other regions (e.g. Canada, EMEA, Asia Pac). The BGP path
selection process prefers the lowest MED. MED is only compared if the routes come directly from the
same AS. (AVPN does not use always-compare med.) So, AVPN will not compare MEDs between
routes from different regions.
Local Preference: AVPN leaves local preference set to the default (100) unless requested otherwise by
the customer router via a community attributesee next subsection (the highest local preference is
preferred). Local preference is not passed on to other autonomous systesms, such as from the U.S. AS to
the EMEA AS.
Community Strings: AVPN accepts from CERs and acts on the following community values:
No-advertise: this instructs the PER to not advertise this route to any BGP peer, including other
PERs. AT&T does not generally recommend using this attribute since the customer does not
generally know or have control over which PERs their PVCs are homed to.
No-export: this instructs the AT&T routers within the regional AVPN autonomous system (e.g.
13979) to use this route but this AS will not advertise it to another AS, whether a customers AS
or another AVPN AS (in another region). Using no-export on specific routes but not summary
routes or the default route is a way for a customer to reduce the number of routes that AVPN will
advertise to CERs.
13979:80 instructs the PER to set local preference for this route to 80
13979:90 instructs the PER to set local preference for this route of 90
13979:100 instructs the PER to set local preference for this route to 100 (default)
Inter-region community values as described in Section 10.3.
Route group community values. See the route group case study (from your Technical Sales
Consultant) which describes routing using the AVPN route group feature.
Note that 13979 at the beginning of all these values is used in all regions, even though the region is not
using AS 13979.
If AVPN receives a community value from the customer in an eBGP update message, it sets the
corresponding local preference on the route in the regional AVPN autonomous system (not globally in
other AVPN autonomous systems).
AVPN passes community values received from a CER on to all other CERs in the VPN, globally
(except no-export and no-advertise). Thus a CER can pass notes to all other CERs. This is useful to
simplify route filtering or to determine the orgin of a route. Note that this includes the above list of
community values for local preference, however local preference is only set in the local regional AS.
If the customer is going to use community values for their own purposes (for their own filtering, for
example and not a value that AT&T acts on) then do not select one beginning with any AT&T AS
number (e.g. 13979, 8034, 8035, 21302, 21326).
ESC Release 2.4

96
Section 9.3 has an example configuration. Reference IETF RFC 1997, BGP Communities Attribute for
more information on using community attributes.
See APPENDIX A: BGP Path Selection Criteria for a summary of Ciscos use of BGP attributes and
other criteria in picking the best route to use.
14.3 BGP Timers
Initially, when a BGP session is established between peers, all routes are exchanged. After the session has
been established and the initial route exchange has occurred, only incremental updates are sent based on
changes in the network (i.e., networks added or withdrawn.). Otherwise, the only communication ongoing
between BGP peers is a periodic KEEPALIVE message. By default, these messages are sent every 60
seconds (30 seconds on Juniper). BGP also uses a HOLD TIMER, which is used to determine when a
BGP peering relationship is considered dead. By default, the HOLD TIMER is three times the
KEEPALIVE message interval or 180 seconds (90 seconds on Juniper).
For some customers who want fast convergence, short BGP timers are usually not the best answer. If
serial or sonet interfaces, the layer 1 alarming along with the command BGP fast external fallover, will
cause failover without waiting for the BGP keepalives to time out. For ethernet interfaces use
bidirectional forwarding detection (BFD; see the interface section).
router bgp 65000
timers bgp 15 45
bgp fast-external-fallover
! where 15 is the keepalive message interval in seconds and 45

is the hold timer value in seconds
! this is usually enabled by default with no need to configure
Do not use values lower than 15:45 without checking with your AT&T Data Network Consultant or Data
Network Analyst. Setting values lower than 15:45 risks losing keepalives and tearing down BGP during
very short hits or temporary processor overloads. Re-establishing BGP can take 30 to 60 seconds. The
timer values are negotiated and agreed upon between individual BGP peers during the neighbor
establishment process. The PER will accept new values requested by the CER (some PERs will not accept
a holdtime below nine seconds). Different values could be used on different CERs. However, AT&T
strongly recommends that a customer choose a consistent value to use throughout their entire VPN. In
most cases, customers have chosen to stay with the default values - especially customers who are not
using any type of backup path.
14.4 VPN Route Limit
The AVPN service has a limit for the number of routes (prefixes) advertised from a BGP neighbor and
per VPN (VRF table). The purpose of the limit is to protect the VPN from being overwhelmed with an
unusually high number of customer routes. It also ensures that one VPN will not consume so many
resources that it will impact other VPNs. AVPN will discard routes beyond the VRF limit and will close
the BGP session if a neighbor advertises more than the limit (requires the NOC to reset). If you expect to
advertise in excess of 5,000 IPv4 routes in the VPN, contact your AT&T sales team. The limit configured
in the PERs is much higher but we want to know ahead of time for capacity planning purposes if your
VPN will have a large number of routes. We generally expect three to five routes per remote site plus
several hundred from the hub or data center sites.
AT&T strongly encourages customers to summarize routes whenever possible. By doing this, the
customer can limit the number of routes in the VPN. Obvious benefits of lowering the number of routes in
a routing table include less consumption of CPE memory, faster convergence and simpler
troubleshooting.
ESC Release 2.4

97
14.5 Route Dampening

BGP route dampening is normally associated with Internet routing (with 100,000+ routes) where ISPs
need it to prevent churn due to unstable routes. AVPN has not implemented route dampening due to the
relatively small number of routes in individual customer VPNs.
AT&T recommends that CERs not use dampening. Typical default dampening parameters cause the route
to be removed from service for 25 minutes after flapping only 3 times in 15 minutes. BGP route
dampening is outlined in RFC 2439.
14.6 Broadcast, Directed Broadcast, Bridging, Auto-Install and CDP
AVPN does not forward broadcast or bridged traffic (e.g., for NetBEUI, DHCP, bootp, etc.). However,
most customer routers can act as a DHCP relay (Cisco refers to this as helper addressing) which relays
the broadcast packet to a specific IP address (a unicast address or a directed broadcast address - i.e.,
192.168.1.255). Since these packets look like any other IP packet to AVPN, they will be routed
appropriately.
AVPN does not support the Cisco auto-install feature, which is a method to automatically discover the IP
address of a serial interface at startup.
AVPN does not support CDP.
To route non-IP packets (IPX, Appletalk, etc.), a customer can choose to implement GRE tunnels
between CERs.
14.7 GRE Tunnels
GRE (generic routing encapsulation) is a tunneling protocol developed by Cisco that can encapsulate a
wide variety of protocol packet types inside IP tunnels, creating a virtual point-to-point link to Cisco
routers at remote points over an IP network. GRE tunnels are configured between a source router and a
destination router such that packets designated to be forwarded across the tunnel are further encapsulated
with a new header (the GRE header), and placed into the tunnel with a destination address of the tunnel
endpoint (the new next-hop). When the packet reaches the tunnel endpoint, the GRE header is stripped
away, and the packet continues to be forwarded to the destination, as designated in the original IP packet
header.
Customers who need to run non-IP packets across AVPN can choose to configure GRE tunnels to
encapsulate this traffic. Tunnels are configured as point-to-point links between CERs. Tunnel endpoints
can be tied to IP addresses on ethernet interfaces, serial interfaces, or virtual interfaces (loopbacks). All IP
traffic will continue to run normally outside of the tunnel. The example below illustrates a single GRE
tunnel to carry IPX traffic between two sites.
ESC Release 2.4

98
Partial RTA (Hub Site) Configuration

ipx routing 00e0.b05a.5d63
!
interface Tunnel0
description GRE Tunnel (over AVPN) to Remote
no ip address
ipx network AA50401
tunnel source 192.168.1.2
tunnel destination 192.169.1.1
!
interface Ethernet0
ip address 192.168.1.2 255.255.255.0
ipx network 1
!
interface Serial0
description AT&T Frame Relay Port
no ip address
!
interface Serial0.1 point-to-point
description 256K ePVC to AVPN
ip address 192.168.10.1 255.255.255.252
no cdp enable
!
router bgp 65000
network 192.168.1.0 mask 255.255.255.0
no auto-summary
!
ipx router eigrp 1
network AA50401
ESC Release 2.4
! Turn on IPX network AA50401 on the tunnel

interface
! Local Ethernet interface
! RTBs Ethernet interface

99
Partial RTB (Remote Site) Configuration

ipx routing 00e0.b05a.5d61
!
interface Tunnel0
description GRE Tunnel (over AVPN) to Hub
no ip address
ipx network AA50401
tunnel source 192.169.1.1
tunnel destination 192.168.1.2
!
interface Ethernet0
ip address 192.169.1.1 255.255.255.0
ipx network 50401D
!
interface Serial0
description AT&T Frame Relay Port
no ip address
!
interface Serial0.1 point-to-point
description 32K ePVC to AVPN
ip address 192.169.10.1 255.255.255.252
no cdp enable
!
router bgp 65001
network 192.169.1.0 mask 255.255.255.0
no auto-summary
!
ipx router eigrp 1
network AA50401
! Turn on IPX network AA50401 on tunnel

interface
! Local Ethernet interface
! Hub routers Ethernet interface
If tunnels are run between the WAN serial interface addresses, the VPN does not need any routes (other
than the directly connected routes, which the PERs redistribute by default). However, the AVPN ordering
systems require either BGP or at least one static route. To work around this, simply provision one static
route for the CE-PE interface subnet. This is redundant with the redistributed connected on the PER but
it allows the order for the port to proceed.
Note that the CER must advertise any routes through the tunnel, that the far end of the tunnel will need to
know about.
14.8 Remote Site Reach-Ability Status
There are no end-to-end customer VLANs or PVCs in AVPN. All virtual circuits, an OSI layer 2 concept,
from customer sites terminate at the PER. For PVCs, therefore, the A-bit status in LMI will not reflect the
availability of the remote site. This may impact networks that are relying on A-bit status to trigger dial
backup. However, this situation can be remedied by using BGP. BGP routing will recognize end-to-end
path failures by maintaining dynamic routing updates and keep-alive packets. Using BGP, a CER and
ESC Release 2.4

100
PER continually update their routes and send regular keep-alive packets to detect failures. AVPN will
also withdraw the static routes associated with a site if the PER detects a layer 1 or layer 2 failure. Any
sites running BGP will recognize the change in route availability, i.e., will recognize the remote failure.
The primary difference in relying on BGP instead of LMI to indicate that a remote site is unavailable is
that route changes do not generate SNMP traps to management systems. However, LMI may not detect all
failures.
See Section 7, Failure Recovery.
14.9 Restrictive Route Filtering (Default Route Only)
This feature is also called the default route only feature. This capability allows a customer, when
provisioning a link, to request AT&T to only send the default route from the PER to the CER. No other
routes will be advertised over this link. This feature is available on any customer link that is running
BGP. This feature does not require any special commands on the CER. This is a feature that is enabled by
checking a box on the order form.
Important Note: AT&T will not generate (i.e., originate) the default route. The customer originates their
own default route from one (or more) of their sites. Therefore, it is critical that a customer advertise a
reliable default route. If not, the loss of a default route advertisement in the AVPN VPN could cause a site
that is configured with Restrictive Route Filtering to lose all IP connectivity to the network. In cases
where no other connectivity out of a remote site other than the AVPN link exists (i.e., no broadband,
cellular or dial backup), the customer should consider using a floating static route (i.e., ip route 0.0.0.0
0.0.0.0 <PER IP address> 250) to force traffic to the VPN network if the dynamic default is lost for any
reason.
14.10 Outbound Route Filtering (ORF)
ORF is a BGP feature AVPN has implemented that allows a BGP router (e.g., CER) to send a prefix-list
to an eBGP neighbor (e.g., PER). The PER then applies the prefix-list to routes it sends to the CER as a
way to minimize the number of BGP routes sent to the CER. This can help reduce the amount of
resources required for receiving, processing and storing routing updates by filtering out unwanted routing
updates at the source (the PER). It also keeps the CERs routing table simpler and easier to troubleshoot.
Note that ORF is not offered on ports on Juniper PERs.
This feature requires BGP ORF and a prefix-list configured on the CER. (ORF is defined in a draft IETF
contribution.) This feature is enabled on all PERs by default, so it does not require special provisioning.
Also, ORF can apply to any or all routes in the VPN--not just the default route. Cisco requires IOS release
12.2 or later.
Limitation: The PE has a limit of 80 prefixesdo not send more than 80 prefixes (e.g. 10.1.1.0/24 is
counted as one prefix).
With AVPN, the CER advertises the ORF capability in send mode. The PER receives the ORF capability
in receive mode and applies the filter as an outbound (back toward the same CER) policy. The example
below shows a partial CER configuration that would instruct the PER to only send a route for
192.168.1.0/24 network to this CER.
ESC Release 2.4

101
router bgp 65001

neighbor 192.169.10.2 capability orf prefix-list send ! Advertise prefix-based ORF send capabilities
to PER
neighbor 192.169.10.2 prefix-list ORFlist in
! Apply prefix-list ORFlist to ORF process
no auto-summary
!
ip prefix-list ORFlist seq 5 permit 192.168.1.0/24
! Access-list to permit only the network
specified.
14.11 Network Based Firewall (NBFW)
AT&T Managed Firewall ServiceNetwork Based 2.0 (MFS-NB2) provides AT&T's Frame Relay and
ATM customers the advantage of accessing the Internet through secure PVCs that are filtered and
monitored via ADIS's (AT&T Data and Internet Services) Security Data Center. Supplementing the anyto-any connectivity of AVPN, the MFS-NB2 service provides a secure Internet connection across an
AVPN VPN without the double expense of homing remote PVC traffic to a customer-administered
central security location only to be re-routed back out to the Internet.
After a customer signs up for this service, AT&T provisions a logical channel into the customers existing
AVPN VPN. The MFS-NB2 data center will generate a default route into the customer's VPN. Once
Internet-bound traffic reaches the network from a CER, it will be routed to MFS-NB2. Then, this traffic
will be allowed access to the Internet based upon the customer's specific security policy as implemented
by MFS-NB2 into its security data centers. The customer never has to manage the traffic to the Internet,
nor the conglomeration of firewalls, filters, patches, servers, screening applications, virus, and banned site
updates, etc. required to secure Internet access.
The MFS-NB2 will provide customers with choices of fire-walled inbound and outbound access on a
secure platform and network, with additional services allowing them to maintain user authentication,
screen Internet destination, filter traffic from web sites, scan mail for viruses and anti-spamming, and
protect its mail services from misuse.
Important Note: Because the MFS-NB2 service supplies the default route to the customer VPN, the
overall routing design may need to be modified if a customer hub site was using the default route for any
non-Internet traffic.
Dual NBFW Nodes in the Same Region
NBFW has multiple nodes in each region. Each region uses the same ASNs: 65534 for the first node in
the region, 65533 for the second in the region and 65532 for the third, if one. For example, if you have
two nodes in the U.S., the first node would use 65534 while the second one uses 65533. If you also have a
node in EMEA, it would use 65534.
Generally, each node has two connections into your VPN, one primary and the other a backup. All
connections would normally advertise the default route into your VPN but the backup link would
advertise it with a higher cost (e.g. higher BGP MED). All customer sites in a region would naturally
select the default from the in-region NBFW node because defaults injected by a node in another region
would have an additional AS hop.
ESC Release 2.4

102
If dual NBFW nodes are used in the same region, both advertising the default route into the VPN, you
may want to consider how routing will distribute the load among the two nodes. Your sales team can
bring in NBFW routing experts to assist with the design.
14.12 MTU
AT&T recommends setting the maximum transmission unit (MTU) to 1500 bytes on all ports in the
customer VPN, to avoid dropped packets. Here is a list of MTU values on specific PER port types:
Dedicated access IP ports with speeds of DS0, T1 and NxT1 MLPPP, is 1500 bytes.
Frame relay access using LFI via MLPPP is 1500 bytes.

Ethernet MTU on ethernet service provider switches are mostly 1500 bytes. Most PER ethernet
ports are 4470. New GSRs and Juniper PERs are 1500.
All other access types are 4470.
By MTU I mean the maximum size IP datagram (packet) in bytes that the payload of the layer 2 frame
can carry. The layer 2 frame is just the link layer on the circuit between two routers. IP will fragment the
TCP/IP message into multiple packets if necessary, to fit into the layer 2 frames. Note that a given router
may offer the option to adjust the layer 2 frame size, which indirectly sets the maximum frame payload, to
accomadate a larger IP packet, or it might allow setting the maximum IP packet size which you can set,
knowing what the layer 2 frame size and overhead is.
So, in general, leave the MTU on the CER interface at 1500 bytes. While this is the default MTU on
Cisco routers for ethernet and for lower speed serial interface types (e.g., T1s), most DS3 and above
interface typese.g., ATM, HSSI or POS interfaces have a default MTU of 4470. To avoid any
potential issues where a higher speed site could send a packet larger than the maximum MTU for a lower
speed site, configure a 1500 byte MTU for all connection types on your CERs. Assuming that most traffic
carried across the MPLS VPN is initiated from 10/100/1000 Mbps LAN segments where the default MTU
is also 1500, this recommendation should provide the optimum performance for customers.
The more generalized answer is to set the MTU on all CER WAN interfaces to the lowest MTU in the
VPN. For example, if only high speed (DS3 or higher) dedicated ports are used, that MTU could be 4470
(or lower, but the same on all WAN interfaces). Or, if the customer orders a 2000 byte MTU for low
speed ports, then set all MTUs to 2000.
Note 1: the GSR PER running IOS-XR supports a limited packet-per-second rate for packets it needs
to fragment. Packets exceeding the rate are dropped. This applies to unicast and multicast packets. So,
with a VPN using ports on the GSR XR, setting the MTU on all CER WAN interfaces to the lowest MTU
in the VPN is important.
Note 2: Refer to the situation in Figure 33, below. If the CERs are left at the default MTU and CER A
happens to send a packet larger than 1500 bytes (e.g. if originating a GRE or IPSec tunnel) on a WAN
interface with an MTU higher than 1500 bytes (so it does not fragment the packet) and the packets are
going to a PER set to 4470, e.g. PER B, (so it does not fragment the packet) while the CER B is set to
1500 bytes, the receiving CER B may drop frames larger than 1500 bytes.
ESC Release 2.4

103
Figure 34 MTU Causing Dropped Packets
Orderable MTU: AVPN allows the customer to pick a different MTU on dual stack ports, from a set of
predefined values (e.g. 2000, 4470, 9100). For the specific values available for your ports, see your
AT&T sales team. Supporting large MTUs on ethernet is refered to as jumbo frames.
14.13 Trace-Route
In a typical trace-route from a CER at one site to a CER at another site (or device behind this CER), the
first entry will be the IP address of the near-end AT&T PER and the second entry will be the far-end
CER. Any subsequent entries would be any other customer routers in the path to the destination. The farend PER will not show up in a trace-route, nor will any other internal AT&T router or switch hops.
ESC Release 2.4

104
APPENDIX A: BGP Path Selection Criteria

Of the BGP routes (exterior, internal & local) select only one path for a network 30:
1. Next hop must be reachable
2. Largest weight (Cisco proprietary attribute)
3. Largest local preference
4. Shortest AS path length (local route--next hop 0.0.0.0--is shortest)
5. Lowest origin code: i (IGP)< e (EGP) < ? (incomplete)
6. Lowest MED (cost)
7. External path (e-BGP) over an internal path (i-BGP)
8. Shortest internal path to BGP next hop (via IGP)
9. First path received (12.0.11 or later) (or both, if max-paths selected)
10. Lowest BGP router ID (highest loopback or highest interface address on router)
11. Lowest neighbor address
Compare the administrative distance of the resulting single route with administrative distance of
routes from other routing protocols and static and connected networks. Pick the route with the
lowest administrative distance.
When a packet arrives, pick the most specific route (longest prefix length, smallest network).
Note that this process is generalized because the exact process varies depending on the router software
and configuration. With BGP multipath enable (as is true for PERs), the process stops at the line (after
item 6) if the attributes are the same to this point, including the same numbers in the AS Path, then the
router will use load sharing as discussed in Section 6.2.
30
Paraphrased from RFC 1771 and Ciscos documentation. See your router vendor to verify for your router
operating system behavior.
ESC Release 2.4

105
APPENDIX B: ATM IMA Configurations for 26XM, 3660, 37XX Routers

The sample configuration below is based on the use of the AIM-ATM and one or two port VWIC-MFT
card combinations on a 2600XM router platform. The example below uses an AIM-ATM card and one
VWIC-2MFT-T1 card in WIC slot-0 providing a 2xT1 IMA circuit.
network-clock-participate wic 0
! VWIC-2MFT-T1 in WIC slot 0: see note 1
network-clock-select 1 T1 0/0
! One line for each T1 in the IMA group
! using a different priority for each (i.e., 1,2,3)
!
controller T1 0/0
! See note 1 at bottom of page
mode atm aim 0
! On 26xx routers, AIM slot is always 0
framing esf
linecode b8zs
!
add cablelength command under each T1 interface when appropriate
controller T1 0/1
mode atm aim 0
framing esf
linecode b8zs
!
interface ATM0/0
no ip address
ima-group 3
! Creates IMA group 3 and assigns port to group
!
interface ATM0/1
no ip address
ima-group 3
!
interface ATM0/ima3
! Configure IMA interface for group 3
no ip address
!
ip address 192.168.10.1 255.255.255.252
! IP address
pvc att-per 1/155
VPI = 1, VCI = 155 for this sub-interface
oam-pvc manage 0
Note 1: If a second VWIC-2MFT-T1 card is used in WIC sloT1, (e.g., to support a 4xT1 IMA
connection), add network-clock-participate wic 1 and network-clock-select 3 T1 0/2, networkclock-select 4 T1 0/3to the above configuration. The additional controller and ATM commands would
use: controller T1 0/2, controller T1 0/3, ATM 0/2, and ATM 0/3.
ESC Release 2.4

106
This next example assumes the use of an AIM-ATM card and one VWIC-2MFT-T1 card in an NM slot
(e.g., VWIC-MFT seated in a NM-2W card) providing a 2xT1 IMA circuit on a 2600XM platform.
network-clock-participate slot 1
! VWIC seated in a NM card in sloT1

See note 1 at bottom of page
! One line for each T1 in the IMA group
! using a different priority for each (i.e., 1,2,3)
!
controller T1 1/0
mode atm aim 0
framing esf
linecode b8zs
!
add cablelength command under each T1 interface when appropriate
controller T1 1/1
mode atm aim 0
framing esf
linecode b8zs
!
interface ATM0/ima3
! Configure IMA group 3 see note 2
no ip address
!
ip address 192.168.10.1 255.255.255.252
! IP address
pvc att-per 1/155
VPI = 1, VCI = 155 for this sub-interface
oam-pvc manage 0
!
interface ATM1/0
no ip address
ima-group 3
! Creates IMA group 3 and assigns port to group
!
interface ATM1/1
no ip address
ima-group 3
Note 1: When VWICs are seated in an NM (e.g., NM-2W), the network-clock-participate command
refers to the NM slot number, not the WIC (i.e., network-clock-participate slot 1). In addition, for
controller T1 slot#/interface# commands, the NM slot number is the first value (e.g., a VWIC-2MFTT1 in NM slot 1 uses controller T1 1/0 and controller T1 1/1). Adding a second VWIC-2MFT-T1 for
a 4xT1 IMA connection would add network-clock-select 3 T1 1/2, network-clock-select 4 T1 1/3,
controller T1 1/2, controller T1 1/3 and ATM1/2, ATM 1/3 to the above configuration.
Note 2: Regardless of the controller location, the ATM/IMA interface is always ATM0/IMA group#.
ESC Release 2.4

107
Note on 3660/37xx Router Platforms: If AIM-ATM and VWIC-MFT cards are used in the 3660 or
3725/3745 router platforms, the same configurations as shown previously for the 2600XM platforms are
applicable depending on where the VWIC (and AIM-ATM) cards are seated. However, the following
additional command must be added:
network-clock-participate aim 0
ESC Release 2.4

108
APPENDIX C: DS3/E3 Frame Relay Configurations

This appendix includes sample configurations that can be used when DS3 frame relay is used to access to
AVPN.
Main Serial Interface Configurations
The following example configuration is for a Cisco router with a T3/DS3 interface (e.g., PA-T3+) in port
adapter slot 2 on a Cisco 72xx series router.
interface Serial2/0
! DS3 interface with built in CSU
description ** AT&T DS3 FR Port: DNECxxxxxx **
no ip address
encapsulation frame-relay ietf
! Enable Frame-Relay ietf encapsulation for
AVPN connectivity
dsu bandwidth 44210
! Configure internal T3 dsu bandwidth (should
be default value)
framing c-bit
! Configure DS3 Framing type (should be
default value)
cablelength 10
! Specify appropriate cable length in feet
The following example configuration is for a Cisco router with a channelized T3 (Serial) Interface (e.g.,
PA-MC-2T3+) in port adapter slot 2 on a Cisco 72xx series router.
controller T3 2/0
no channelized
! Until this command is entered, the Serial

interface below will not show up.
!
interface Serial2/0
! DS3 interface with built in CSU
description ** AT&T DS3 FR Port: DNECxxxxxx **
no ip address
encapsulation frame-relay ietf
! Enable Frame-Relay ietf encapsulation AVPN
connectivity
dsu bandwidth 44210
! Configure internal T3 dsu bandwidth (should
be default value)
framing c-bit
! Configure DS3 Framing type (should be
default value)
cablelength 10
! Specify appropriate cable length in feet
ESC Release 2.4

109
The following is an example of an E3 configuration on a Cisco NM-1T3/E3:

Card type e3 2
Controller e3 2/0
! Default settings: framing g751, line code HDB3

! The interface configuration is the same as a DS3 with the
following exception:
Interface s2/0
dsu bandwidth 34010
Note in each of the configurations shown above the following commands can be added if desired to
ensure that events are logged in the event of a main interface or sub-interface state change.
logging event status-change
! log sub-interface status changes

! log main serial interface status changes
Additionally, the following command would be used if the default cisco signaling is not used for LMI
signaling on the Frame Relay circuit:
! or ANSI or Q933 if ordered with these options
Sub-Interface Configuration
Configure a sub-interface for each DLCI (configuration is the same regardless of the main interface
configurations shown above).
interface Serial0/0.1 point-to-point
description epvc to AVPN
ip address 192.168.10.1 255.255.255.252
no cdp enable
ESC Release 2.4
! Assign /30 IP address to the interface

! Always Disable CDP on CE-PE links
! Assign appropriate DLCI to the sub-interface

110
APPENDIX D: DIAL BACKUP

Dial backup is aimed at the loss of AVPN connectivity to remote sites where ISDN is sufficient
bandwidth during failure recovery. Because the dial backup bandwidth is limited to ISDN speeds, it is
generally not a viable solution against the loss of connectivity to sites with high-bandwidth access to
AVPN (e.g., sites with DS3 ATM or higher access).
The dial backup solutions described in this document are designed to dial completely around the AVPN
cloud upon failure, not into the cloud. In other words, during dial backup, the AVPN cloud will be
bypassed and customer routers will communicate directly with each other via the dial backup connection.
This approach provides failure recovery for all elements except the CER at the remote location 31.
The following two ways of providing dial backup will serve the needs of most AVPN customers. Please
refer to the Section 0 titled More Complex Dial Backup Environments for comments on other dial
backup scenarios:
Scenario 1: Dial Backup using static routes over the dial backup connection.
Scenario 2: Dial Backup using a dynamic routing protocol, e.g., EIGRP, over the dial backup
connection.
Figure 34 illustrates a typical dial backup network configuration.
Figure 35 Typical Dial Backup Configuration
31
Switched services, in conjunction with a routers dial backup (DBU) function, may be used to provide restoral for
the loss of a sites access into AVPN. Switched services like ISDN and POTS can be used as an effective means of
restoral for primary circuit failures when steps are taken to provide diversity with primary circuits. It should be
noted; however, that if physical diversity does not exist between the primary IFPR access and the switched services
used for dial backup, a singular failure may impact both the primary connection and the ability to establish dial
backup.
ESC Release 2.4

111
Before describing the details of each dial backup scenario, lets discuss attributes common to both.
1. Dial-Around-The-Cloud Solution Both solutions are designed to dial completely around the
AVPN cloud upon failure, not into the cloud. In other words, during dial backup, the AVPN cloud
will be bypassed and customer routers will communicate directly with each other via the dial backup
connection.
2. DBU hub and Spokes One location within the VPN is designated the dial backup (DBU) hub.
The DBU hub is the location that receives dial backup calls from locations on the VPN when failures
occur. Because dial backup creates a circuit-switched connection directly between pairs of customer
routers, the physical topology during dial backup is a hub-and-spoke configuration, where all sites are
spokes, except the DBU hub. However, spokes connected to the DBU hub are able to route through
the DBU hub to reach sites on dial backup and sites on AVPN that have not gone into dial backup.
Therefore while the physical topology is hub-and-spoke, the any-to-any logical connectivity is
maintained. Please note: The DBU hub is typically the location where the Default Route originates
for the network. Typically, this is the location providing Internet access to the network. If default
routing is currently not used and there are no plans to move to default routing to the Internet, the site
chosen as the DBU hub should be a site with one or both of the characteristics below, if possible:
a network data center location with more robust network infrastructure than other locations to
minimize outages and facilitate circuit diversity.
a location where some/most of the network application servers reside, if there is any one such
location in the network. This is recommended to maximize the amount of traffic that terminates in
the DBU hub; and thereby, minimize the amount of traffic needing to route through the site and
out via the AVPN connection to spokes that have not gone into dial backup. It should be noted
that routing through the DBU hub to reach sites that are not on dial backup, while favorable to
preserve the any-to-any connectivity, may require additional bandwidth on the AVPN port at the
DBU hub.
3. Separate DBU Router at the DBU hub - A separate Dial Backup Router is used at the DBU hub to
terminate dial backup calls; instead of combining AVPN and dial backup connectivity on the AVPN
CER. With the separate DBU Router, we avoid the loss of both primary and secondary connectivity
when a single router failure occurs at the DBU hub. With the introduction of a second router for dial
backup, an additional LAN segment and possibly a protocol like HSRP/VRRP may be required to
achieve redundancy under all failure conditions. These topics are beyond the scope of this document.
4. BGP is required on CERs at all sites with dial backup Automatic dial backup initiation relies on
the detection of a loss of primary AVPN connectivity. Static routing with AVPN is not sufficient
when using dial backup because static routes continue to be active after network connectivity is lost
unless the failure is detected by layer 1 or layer 2 of the OSI Reference Model. Because layer 1 and
layer 2 are terminated by various network devices in the local access and AVPN cloud, network
failures not local to the CER or PER may not be detected by layer1 or layer 2 protocols. As such, a
dynamic layer 3 mechanism must be used to detect failures reliably. By using BGP, the CER and the
PER establish a BGP session using TCP as the underlying protocol, and BGP uses keep-alives as a
way to actively check connectivity. Loss of VPN network connectivity can occur in various ways. For
example, a physical failure or a logical configuration issue may bring down the BGP neighbor session
and its associated routes; or, the BGP session may be up but the advertisement of the default route by
the VPN network may stop. Either one of these failure types will cause dynamic routes to drop from
the routing table and activate the floating static routes.
5. Use of Floating Static Routes and Dial-On-Demand Routing (DDR) One of the key factors
looked at by routing processes in deciding which route to select for a given destination is the
Administrative Distance (AD). With other things being equal, a router will select the route with the
ESC Release 2.4

112
lowest Administrative Distance to a destination. Floating static routes are static routes that are
given an Administrative Distance greater than other dynamic routes in the routing table 32. This has
the effect of making the static route less desirable than other dynamic routes; and, it stays inactive, or
floats, until all other routes with lower Administrative Distances are dropped. Once the more
preferred routes are lost, the floating static route is activated, and traffic is sent through path identified
by the static route.
In both dial backup scenarios discussed in the following sections for AVPN, a floating static
default route is configured at all spoke sites with dial backup capabilities. The DBU hub will
either use floating static routes or a dynamic routing protocol depending on the scenario chosen.
AT&T recommends using a dynamic protocol whenever possible. At spokes, the floating default
route points to the IP address of the dial backup interface at the DBU hub as the next hop. As
such, the floating static default route will route traffic to the dial backup link when the eBGPlearned Default Route, which has a lower Administrative Distance, goes away. However, just
because the floating static default route is active does not mean that a call will be placed. With
Dial on Demand Routing (DDR), important traffic is categorized as interesting in the router;
and, it is only interesting traffic that is able to kick off the call. It is also the lack of interesting
traffic that brings down the call 33. As such, the dial backup call will be initiated when there is
traffic classified as interesting to transmit to another site.
6. Use of the Default Route in Conjunction with BGP When using dial backup, it is recommended
that the DBU hub advertise a Default Route into the VPN. As such the DBU hub must be the location
where the Default Route originates for the network. If default routing to the Internet is also a
requirement, the site providing Internet access must be selected to also be the DBU hub.
Note: In environments where a default route (0.0.0.0/0) cannot be advertised from the DBU hub
site to the remotes and/or it is not possible to use a floating static default route (i.e., an all zeros
route) at the remotes, it is usually possible to use some type of supernetted floating static route.
Obviously, the choice of the supernet to use in the floating static route is critical in terms of both
triggering the call at only the appropriate times and also not interfering with any other routing at
the site. In more complex environments such as this, AT&T technical resources should be
engaged by contacting your AT&T sales team.
The most common type of failure protected by dial backup is a failure of the access port at a
remote site. In this case the default route and all other routes learned from AVPN are lost. When
this occurs, the floating static default route becomes active and routes all traffic to the dial backup
interface.
It is also possible that the default route being learned from AVPN via BGP goes away but the
AVPN connection at the remote site stays active continuing to learn other routes on the VPN.
This may occur if access to AVPN is lost at the hub site that advertises the default route into the
VPN. If this scenario occurred, traffic destined for networks still being advertised by other sites
on the VPN would still be routed over the AVPN connection. However, traffic to unknown
networks will be sent to the dial backup interface based on the floating static default route. Then,
if the traffic is classified as interesting in the router it will kick off the call. Otherwise, the
traffic is dropped. This behavior allows sites on the VPN to communicate via dial backup to a hub
location that may have been disconnected from the VPN, while continuing to communicate to
other sites via the VPN.
32
The default AD's for several common dynamic routing protocols (and route types) are: eBGP-20,
EIGRP Internal - 90; EIGRP External-170; OSPF-110; RIP-120; iBGP & eBGP backdoor 200).
33
DDR was implemented after initial dial backup features like Backup Interface to more effectively manage the
amount of time a call is active; and thereby, reduce usage costs.
ESC Release 2.4

113
For the above approach to work, the CER must learn the default route and the other VPN routes
via BGP. If filtering is done on the CER or in the network to only allow the CER to learn a
default route, the CER will dial the Dial Backup Hub when the default route is lost, even if
connectivity to some sites still exists via the VPN. This behavior is generally not desirable
because the bandwidth available on the dial backup connection is typically lower that the primary
connection. It may, however, be acceptable in environments where most traffic is destined to the
Dial Backup Hub anyway.
7. Support of AS Override Both dial backup scenarios support the AVPN AS override feature. As
such, hub and spokes may use any autonomous system number, including the same ASN at all
locations.
Scenario 1: Dial Backup using static routes over the dial backup connection
Scenario 1 is intended for small to medium sized networks where the added administrative overhead of
configuring static routes on the Dial Backup Router is not considered unmanageable; and/or, where there
is a desire to avoid the added complexity of using a new routing protocol over the dial backup connection.
With this scenario, BGP is used to interface with AVPN at the Dial Backup Hub and all spokes requiring
dial backup. For simplicity, dialing can only initiated by the spokes only, and not the hub. The spokes use
a floating default route to send traffic through the dial backup connection upon failure; and, DDR to
initiate and tear down the calls. The DBU Router in the DBU hub is configured with floating static routes
corresponding to all spoke networks.
The following example illustrates the use of dial backup with static routes. The example assumes that
EIGRP is used as the existing IGP at the hub location. Remote locations have a single router, the CE
router, and a single LAN segment. As such, an IGP at the remote sites is not necessary. ISDN BRI
interfaces are used at remote sites for dial backup while an ISDN PRI is assumed at the DBU hub. Below
are the router configuration statements associated with this dial backup example.
ESC Release 2.4

114
Additional RTA Configuration

(ip route 0.0.0.0 0.0.0.0 Null0)
! BGP needs the exact network in the routing

table before it will advertise it. Not required if
default route is already in routing table.
router eigrp 10
redistribute bgp 65000 metric 1536 2000 255 1 1500 route-map bgp-in
! Redistribute routes learned from other sites into
EIGRP
! Make AVPN interface passive for EIGRP
! Log EIGRP neighbor changes
network 192.168.1.0
! Run EIGRP with other routers on this segment
network 192.168.2.0
no auto-summary
router bgp 65000
network 0.0.0.0
network 192.168.1.0 mask 255.255.255.0
network 192.168.2.0 mask 255.255.255.0
timers bgp 15 45

!
match ip address 10
! Private ASN for site

! PER
! Advertise default route to other sites
! Advertise hub site core LAN network(s)
! Advertise hub site core LAN network(s)
! Reduces BGP keep-alive and Hold Time to 15
and 45, from 60 and 180 defaults. This can
improve worst-case failover times.
! Allow only these remote site networks.
! Allow only these remote site networks.
Additional RTE Configuration

username RTB password 7 <encrypted password>
! Username used with CHAP authentication.

One entry per remote site
username RTC password 7 <encrypted password>

!
controller T1 1/0
framing esf
linecode b8zs
pri-group timeslots 1-24
!
isdn switch-type primary dms-100
! Type depends on switch used by ISDN provider
!
interface Serial 1/0:23
! PRI Interface
ip address 192.168.20.1 255.255.255.0
! Default for all interfaces
no ip route-cache
! Turns off fast switching - default on PRI
no ip mroute-cache
! Turns off multicast fast switching - default on PRI
encapsulation ppp
! Defines PPP as layer 2 protocol
dialer in-band
ESC Release 2.4

115
dialer map ip 192.168.20.2 name RTB broadcast

! Optional - not needed in newer IOS
dialer map ip 192.168.20.3 name RTC broadcast
dialer-group 1
! Associates PRI with dialer-list below
dialer idle-timeout 300
! Idle time (in seconds) before the line is disconnected.
Interesting traffic resets timer
no cdp enable
! CDP should typically be disabled on ISDN interfaces
ppp authentication chap
! CHAP authentication enabled for security
ppp multilink
! Optimal setting when remote sites can bring up
multiple B-channels
!
router eigrp 10
redistribute connected
! Advertise dial backup segment to other EIGRP routers
redistribute static metric 64 20000 255 1 1500 ! Redistribute floating statics to other EIGRP routers
with worse metric than redistributed BGP routes on
RTA
passive-interface serial 1/0:23
! Ensures EIGRP won't be run over ISDN - this is true
also since no "network 192.168.20.0" statement is
used.
network 192.168.1.0
network 192.168.2.0
no auto-summary
!
ip classless
! Allows use of supernetting (e.g. default route)
ip route 192.169.0.0 255.255.0.0 192.168.20.2 250
! Floating static to location B
ip route 192.170.0.0 255.255.0.0 192.168.20.3 250
! Floating static to location C
!
access-list 101 deny ip any host 255.255.255.255
! Keeps broadcasts from bringing up call
access-list 101 permit ip any any
! Permit all else(see note below)
dialer-list 1 protocol ip list 101
! Interesting traffic defined by access-list 101
Note: The following ACL defines many of the traffic types which AT&T typically configures on many of
its DBU networks. The appropriate list for different customer environments will depend on
features/protocols running within the network:
access-list 101 deny eigrp any any
access-list 101 deny tcp any any eq tacacs
access-list 101 deny udp any any eq tacacs
access-list 101 deny udp any any eq ntp
access-list 101 deny udp any any eq snmp
access-list 101 deny udp any any eq snmptrap
access-list 101 deny udp any any eq syslog
ESC Release 2.4

116
Additional RTB Configuration

username RTE password 7 <encrypted password>
isdn switch-type basic-ni
!
interface BRI0/0
ip address 192.168.20.2 255.255.255.0
no ip route-cache
no ip mroute-cache
encapsulation ppp
dialer string 93686940
dialer-group 1
dialer wait-for-carrier-time 60
dialer hold-queue 100
dialer load-threshold 200 either
isdn spid1 7323686942 3686942
isdn spid2 7323686943 3686943
ppp multilink
!
router bgp 65010
network 192.169.1.0
timers bgp 30 90
! Used with CHAP authentication. Entry for

DBU hub.
! Depends on switch used by ISDN provider

! Dials DBU hub - could also use dialer map ip
command
! Associates BRI with dialer-list below
! Time (in seconds) the interface waits for a carrier
! Packets to queue during call setup
! Threshold for bringing up additional B-channels ~80%
! From ISDN provisioning information
! Advertise this network

! PER neighbor
! Reduces BGP keep-alive and hold time to 30 and 90,
from 60 and 180. This cuts down time to initiate call.
no auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.20.1 250

! Floating static default route
! Keeps broadcasts from bringing up call
! Permit all else
Scenario 2: Dial Backup Using Dynamic Routing over Dial Backup Connection
Scenario 2 is intended for networks where the added administrative overhead of configuring static routes
on the Dial Backup Router is not desired; or, not practical due to the size of the network. This approach is
more common and typically recommended by AT&T over the static route approach. `With this scenario,
BGP is used to interface with AVPN at the DBU hub and all spokes requiring dial backup. For simplicity,
dialing is initiated by the spokes only, not the hub. The spokes use a floating default route to send traffic
through the dial backup connection upon failure and, DDR to initiate and tear down the calls. An IGP is
used over the dial backup connection to advertise the spoke networks to the hub.
ESC Release 2.4

117
Below are the router configuration statements associated with dial backup Scenario 2. The example below
assumes that EIGRP is used as the existing IGP at hub and spoke locations, ISDN PRI is used at the hub
and ISDN BRI is used at the remotes.
See Figure 34 for a diagram of the network.
Additional RTA Configuration
Same as Scenario 1
Additional RTE Configuration

username RTB password 7 <encrypted password>
username RTC password 7 <encrypted password>
isdn switch-type primary dms-100
! Used with CHAP authentication

interface Serial 1/0:23

! PRI Interface
ip address 192.168.20.1 255.255.255.0
no ip route-cache
no ip mroute-cache
encapsulation ppp
dialer in-band
dialer map ip 192.168.20.2 name RTB broadcast
dialer map ip 192.168.20.3 name RTC broadcast
dialer-group 1
! Associates PRI with dialer-list below
no cdp enable
! CDP should typically be disabled on ISDN interfaces
ppp multilink
! Optimal setting when remote sites can bring up
multiple B-channels
router eigrp 10
network 192.168.1.0
network 192.168.2.0
network 192.168.20.0
distribute-list 20 out serial 1/0:23
no auto-summary
ip classless
access-list 20 deny any
ESC Release 2.4
! To advertise dial backup link

! Run EIGRP on interfaces within this address
! Run EIGRP on dial backup link to learn remote
networks when dial backup is in use
! Controls what EIGRP advertises on dial backup link
! No networks advertised from hub to remote on dial
backup connection

118

! Keeps EIGRP from bringing/keeping up call
access-list 101 deny ip any host 255.255.255.255 ! Keeps broadcasts from bringing up call
! Permit all else
Additional RTB Configuration
username RTE password 7 <encrypted password>
interface BRI0/0
ip address 192.168.20.2 255.255.255.0
no ip route-cache
no ip mroute-cache
encapsulation ppp
dialer string 93686940
dialer-group 1
dialer wait-for-carrier-time 60
dialer load-threshold 200 either
dialer hold-queue 100
isdn spid1 7323686942 3686942
isdn spid2 7323686943 3686943
ppp multilink

! Turns off fast switching

! Turns off multicast fast switching
! Dials dial backup hub
! Associates BRI with dialer-list below
! Idle time (seconds) before the line is
disconnected. interesting traffic resets timer.
! Time (seconds) the interface waits for a carrier
! Threshold for bringing up additional B
channels ~80%
! Number of packets to queue during call setup
! Optimal setting when remotes can bring up
multiple B-channels
router eigrp 10
! To advertise dial backup link
redistribute static
! Redistribute floating default route
redistribute bgp 65010 metric 1536 200 255 1 1500 route-map bgp-in ! Redistribute BGP to learn
dynamic default
network 192.169.1.0
network 192.168.20.0
! Run EIGRP on Dial Backup link to advertise
networks to hub when dial backup is in use
distribute-list 20 out bri0/0
! Controls what EIGRP advertises on dial
backup link
no auto-summary
router bgp 65010
network 192.169.1.0
timers bgp 30 90
ESC Release 2.4
! Advertise this network

! PER neighbor
! Reduces BGP Keepalive and Hold Time to 30
and 90, from 60 and 180. This cuts down
restoral time from approx. 3 minutes to
approx. 1.5 minutes.

119
no auto-summary
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.20.1 250

! Floating default route for dial backup
! Allow only the default to be redistributed
! Only advertise local network to hub site over

dial backup link
! Keeps EIGRP from bringing/keeping up call
! Permit all else
! Route-map for distribution of BGP into EIGRP
! Match access-list 10

match ip address 10
In the example above, the existing IGP at the hub and spoke sites, EIGRP, was used as the routing
protocol on the dial backup connection for simplicity. Youll note, however, that care was taken to ensure
that only the LAN network at the spoke is advertised using EIGRP over the dial backup connection. The
hub does not need to advertise networks over the backup link because of our use of the floating static
default route at the spoke.
When using an existing IGP at the spoke and hub for the dial backup connection, care must be taken to
ensure that running the IGP over the dial backup connection does not introduce instability into the design
(e.g., EIGRP Stuck in Actives, etc). When OSPF is used as the IGP at the hub and spokes, it may be
difficult to maintain the hierarchy required by OSPF if running OSPF over the dial backup connection.
There may also be cases where different IGPs are used at the hub and spokes. For these more complex
cases, contact your AT&T sales team.
Complex Scenarios: Dial Backup Complex Environments
The two scenarios described in the previous sections should cover the majority of VPN networks
requiring basic dial backup. For support of more complex dial backup environments, please engage
AT&T technical resources by working with your local account team. More complex scenarios may
include, but are not limited to, the following requirements:
Multiple DBU hubs
Use of different IGPs over the dial backup link.
Selection of the routing protocol for dial backup when there are different IGPs used at the spoke
and the hub.
Cannot advertise the default route into the VPN.

LAN and Router redundancy at DBU hub
ESC Release 2.4

120
APPENDIX E: Acronyms & Definitions

ACL
Access Control List
AD
Administrative Distance
AS, ASN Autonomous System, Autonomous System Number
AVPN
AT&T VPN Service
BGP-4
Border Gateway Protocol version 4
CDP
Cisco Discovery Protocol
CEF
Cisco Express Forwarding
CER
Customer Edge Router
CO
Central Office
CoS
Class of Service
CPE
Customer Premises Equipment
DBU
Dial Backup
DDR
Dial on Demand Routing
DLCI
Data Link Connection Identifier
DR
Disaster Recovery
DSL
Digital Subscriber Line
eBGP
External Border Gateway Protocol
ePVC
Enterprise PVC, a frame relay or ATM service PVC interfacing with AVPN
EIGRP
Enhanced Interior Gateway Routing Protocol
GRE
Generic Routing Encapsulation
HSRP
Hot Standby Router Protocol
iBGP
Interior Border Gateway Protocol
IETF
Internet Engineering Task Force
IGP
Interior Gateway Protocol (e.g., EIGRP, OSPF, RIP)
IMA
Inverse Multiplexing over ATM
INCS
Integrated Network Connect Service
IOS
Internetworking Operating Software (Cisco router software)
ISDN
Integrated Services Digital Network
ISP
Internet Service Provider
Leased Line A dedicated circuit with only layer 1 services provided between the two points (e.g. no
frame relay or ATM switching provided). One end may terminate on an AT&T router to
provide MPLS or Internet services.
LMI
Local Management Interface (frame relay signaling)
MARO
MIS Access Redundancy Option
MED
Multi-Exit Discriminator
MHSRP Multi-group HSRP
MIS
Managed Internet ServiceAT&Ts brand of Internet service
MPLS
Multi-Protocol Label Switching
MR
Managed Router (at the customer site, managed by AT&T)
ESC Release 2.4

121
Nets
OAM
ORF
OSPF
PBR
PER
PLCP
POP
POTS
PVC
SDO
UBR
VBR
VC
VPN
VRRP
Networks
Operations, Administration & Maintenance (ATM)
Outbound Route Filtering
Open Shortest Path First routing protocol
Policy Based Routing
Provider Edge Router
Physical Layer Convergence Protocol
Point of Presence; sometimes used interchangeably with central office or CO; although, a
POP may be a legal demark not in a central office.
Plain Old Telephone Service
Permanent Virtual Circuit
Switch Diversity Option
Unspecified Bit Rate
Variable Bit Rate
Virtual Circuit
Virtual Private Network
Virtual Router Redundancy Protocol
ESC Release 2.4

122
APPENDIX F: Additional Resources

Other AT&T Customer Guides
AT&T Network-Based Class of Service Features, created to assist AT&T customers with configuring
their routers for class-of-service to match AT&T CoS capabilities.
AT&T PNT AVPN Service Interoperability (SIO) Customer Guide created to assist customers
wanting to migrate from PNT service to AVPN.
AT&T VPN Service Ethernet Access Customer Router Configuration Guide
AT&T VPN Route Groups Customer Guide
AT&T VPN Service Virtual Network Internet Connection Ethernet Access Customer Router
Configuration Guide
AT&T MPLS Service Multicast Customer Router Configuration Guide
AT&T VPN with Wireless Backup Customer Router Configuration Guide
AT&T VPN Dual-Stack (IPv4 / IPv6) Configuration Guide
"AT&T MPLS Services Customer Router Troubleshooting Guide," created to assist MPLS VPN
customers (IPFR, AVPN, PNT) with troubleshooting an IP network.
"AT&T VPN Service Customer Migration Guide, created to assist AT&T customers with migrating
their existing networks to AVPN.
Nortel IPFR/ATM Service Configuration Guide, created to assist IPFR customers with configuring
their Nortel routers for IPFR. Much of this also applies to AT&T VPN Service.
AT&T OPT-E-WAN Customer Configuration Guide (a VPLS service)
AT&T Telepresence Solution Customer Router and Firewall Configuration Guide
AT&T VPN Service, Customer Router Configuration Guide, Addendum for Mobility Commercial
Connectivity Service (CCS) Access
AT&T VPN Sevice, Customer Router Configuration Guide, Addendum for ADSL Access
AT&T MPLS Services Customer VoIP Network Design Guidelines
Case Studies
AT&T also has many customer case study examples that we can quickly modify to your specific
situation.
Reference Books
Halabi, Hassam. Internet Routing Architectures. Indianapolis, IN; Cisco Press, 1997.
A practical guide on IP routing with particular emphasis on BGP and Cisco configurations.
Stewart, John A. BGP4: Inter-Domain Routing in the Internet. Reading, MA; Addison-Wesley, 1999.
A detailed reference book about the BGP4 protocol.
ESC Release 2.4

123
Internet Sites
http://www.cisco.com/warp/public/459/18.html
A BGP Technical Tips page with links to various Cisco tutorials.
http://www.cisco.com/univercd/cc/td/doc/product/software/ios121/121cgcr/ip_c/ipcprt2/1cdbgp.htm
A detailed chapter about configuring Cisco equipment with BGP.
http://www.innosoft.com/rfc/
RFC Index
RFCs of Interest
RFC 1745, BGP4/IDRP for IP---OSPF Interaction
RFC 1771, A Border Gateway Protocol 4 (BGP-4)
RFC 1774, BGP-4 Protocol Analysis
RFC 1930, Guidelines for creation, selection, and registration of an Autonomous System (AS)
RFC 1997, BGP Communities Attribute
RFC 1998, An Application of the BGP Community Attribute in Multi-home Routing
RFC 2547, BGP/MPLS VPNs, describes AVPN implemented using BGP extensions and MPLS
ESC Release 2.4

124

Router Guide

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Router Guide

Încărcat de

Drepturi de autor:

Formate disponibile

AT&T VPN Service

Customer Router Configuration

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

AT&T Proprietary For Use by AT&T Customers

12.2 Routing between AVPN and CCS endpoints ...................................................................... 90

Changes From Previous Releases

ORF limitation update (Section 14.10)

Changes from Release 2.2 to 2.3

Updated the introduction and the interface recommendations

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

ATM Service, including IMA (inverse multiplexing with ATM, multipleT1s)

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

Reference Network and Conventions

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

Figure 1 Reference Network Topology

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

There is no layer 2 keepalives to inform the CER and PER of a failure

AT&T Proprietary For Use by AT&T Customers

! Change timeslot range for fractional T1 access

! note that E3 PPP ports should use no

AT&T Proprietary For Use by AT&T Customers

Here are some example configurations.

SPA-2xT3/E3 in a 7304, 7600, or ASR with frame encapsulation:

Note that scrambling is not enabled (it is disabled by default).

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

OC3, OC12 and OC48 PPP SONET Interfaces

! Default for OC3 VPN ports;

! delays declaring the interface down until after

! shape all traffic on the port

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

BFD, Bidirection Forwarding Detection

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

show bfd neigh details

MLPPP for Multiple T1 or E1s

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

Interface serial x/0 8

! cef distributed for some hardware

! do not use the bandwidth statement 7

! Same group number used on multilink interface 9

! Same group number used on multilink interface

! Same group number used on multilink interface

AT&T Proprietary For Use by AT&T Customers

AT&T Proprietary For Use by AT&T Customers

Frame Relay Encapsulation over Dedicated Access (T1, T3, OC-x)

ESC Release 2.4

AT&T Proprietary For Use by AT&T Customers

Frame Encapsulation Example on DS3 Dedicated Access

! The default is no scrambling, which may not

dsu bandwidth 44210