2013 IEEE Seventh International Symposium on Service-Oriented System Engineering

The Evolution of the Carrier Cloud Networking

Dennis Cai, Sr. Technical Marketing Architect, Cisco Systems
AbstractService providers are looking at cloud solutions to
solve some of their biggest business and technology challenges: reducing costs, creating new levels of efficiency, and facilitating innovative business models that drive revenue growth. This paper begins by outlining the key business drivers and requirements of cloud networking, followed by reviewing existing cloud networking solutions and their limitations. The focus of the paper is on the various feature optimizations and technology solutions to evolve carrier cloud networking. Cloud networking is a very broad area that includes many sub technology areas, such as networking, security, load balancing, storage, VM management, orchestration, etc. The focus of this paper is on the networking part of cloud solutions. Further, the focus is on the carrier cloud providers though the technologies and solutions can also apply to other customer segments like enterprise private cloud. Keywords cloud, virtual overlay, virtual CE, virtual PE, distributed PE, E-VPN, PBB-EVPN, Fabric path, VXLAN, LISP, Data Center

Sai Natarajan, Sr. Product Manager, Cisco Systems

physical location of data centers and classic network design. To efficiently utilize the compute resource pool, different VMs that belong to one tenant can be in different physical locations and VMs must be moved between physical locations on demand. When a VM moves, it needs to retain its IP and MAC address to keep the move transparent to higher layer application. This is a big challenge for the traditional data center networks, where VMs can move within L2 broadcast domain, but cant move across the L3 boundary. To support VM mobility across data center PODs or sites, the network must be designed as a big L2 network, which has potential scale limitations that need to be addressed. C. Seamless integration with existing carrier VPN network Typically, service providers have VPN network footprint to connect their customer branches, headquarters and data centers. Consequently, they can leverage their existing VPN network to provide access to virtual data center as part of the cloud service offering to customers. VPN network access has guaranteed user SLA advantages such as dedicated bandwidth, low latency, security, and so on. Linking the service provider WAN/VPN network with the data center network seamlessly and efficiently is another requirement. D. Heterogeneous application and application transparency Another key requirement of cloud networking is to keep application transparency. From application point of view, it shouldnt matter if it is in a dedicated physical enterprise data center or in a private carrier cloud. Further, service providers need to support multiple enterprise tenants, where different tenants may run different applications. Although most of the applications are IP based, some server clustering applications may require L2 network to support non-IP, non-routable multicast packets as heart beat messages. Typically, L3 technology is more scalable and robust while the L2 technology is simple and provides application transparency. Service providers need to pick one or a mix of L2 and L3 technologies to build carrier cloud network. II. THE FRAMEWORK OF THE CARRIER CLOUD
NETWORKING

I.

BUSINESS DRIVERS AND KEY REQUIREMENTS OF

CARRIER CLOUD NETWORKING

The key business drivers for cloud services include reducing costs, both OPEX and CAPX, creating new levels of efficiency and facilitating innovative business models to drive revenue growth. A. Cloud networking foundation: network virtualization How to reduce CAPX? The answer is quite simple: share the network resource. This is analogous to virtualization of compute resource with virtual machines (VM) that has revolutionized the computing industry. With new networking hardware and technology, network infrastructure can be virtualized and shared among many tenants, without sacrificing user privacy, security, and experience. The fundamental requirement of cloud networking is network virtualization. B. Efficient resource sharing: VM mobility Cloud, or virtual data Center, is made up by many physical data centers. Each physical data center can consist of one or more PODs and can have different resources: compute, storage, network, etc. One of the key requirements of cloud service is elasticity: the ability to provide service on demand for a few hours to a few days and up to years. The physical data center resource must be treated as a virtual resource pool with the ability to efficiently add or remove resources, with minimal operational overhead. In other words, the requirement is to eliminate the data center network boundary due to the
978-0-7695-4944-6/12 $26.00 2012 IEEE DOI 10.1109/SOSE.2013.43 286

A typical carrier cloud network consists of multiple physical data centers: multiple enterprise data centers or private cloud, and multiple SP data centers which offer virtual private cloud or public cloud service. The customer branch offices or client sites can access the private or virtual private cloud through SP L2/L3 VPN network, or even through internet. A typical data center has 3 tiers: Top of rack (ToR) switch, DC aggregation switch and WAN router. Depending on the

physical size of the data center, it may have multiple PODs which are connected by an additional core switch layer.

III.

THE EVOLUTION OF THE CARRIER CLOUD NETWORKING

There are three key areas in the evolution of carrier cloud networking that are described in detail below: x Optimized L2 forwarding x DC fabric evolution: IP virtual overlay x Optimized L3 routing A. Optimized L2 forwarding Most existing cloud networks deploy L2 based carrier cloud solution. L2 solution has the following advantages: x Application transparency, it can work with any application, IP or non-IP x Support VM mobility natively x Plug-n-Play: when a VM moves to a different location, the network can learn the MAC address via data plane learning. It does not require external control plane or network re-provisioning However, as mentioned in the previous section, there are many limitations to the classic L2 solution. One of the evolution paths for carrier cloud networking is to optimize L2 forwarding to address the limitations. This approach is attractive for existing deployments. A few emerging technologies that provide optimized L2 forwarding include IETF E-VPN[1], PBB-EVPN[2], Trill, IEEE 802.1aq, Cisco Fabric path with Segment ID[3]. E-VPN: E-VPN is considered as the technology evolution of VPLS. The evolution in E-VPN is in MAC learning and L2 packet forwarding. From the E-VPN[1] draft: In E-VPN, MAC learning between PEs occurs not in the data plane (as it happens in traditional bridging), but in the control plane. Control plane learning offers greater control over the MAC learning process, such as restricting who learns what, and the ability to apply policies. Furthermore, the control plane chosen to advertise MAC reachability information is multi-protocol (MP) BGP (similar to IP VPNs (RFC 4364)). This provides greater scalability and the ability to preserve "virtualization" or isolation of interacting agent groups (hosts, servers, virtual machines) from each other. In E-VPN, PEs advertise the MAC addresses, learned from the CEs that are connected to them, along with an MPLS label, to other PEs using MP-BGP. Control plane learning enables load balancing of traffic to and from CEs that are multi-homed to multiple PEs. This is in addition to the load balancing across the MPLS core with multiple LSPs between the same pair of PEs. In other words, it allows CEs to connect to multiple active points of attachment. It also improves convergence times in the event of certain network failures In summary, E-VPN provides L2 multi-point bridging service, with control and data plane similar to traditional L3VPN, which has proven to be more scalable and mature. EVPN has the following advantages: x Optimal per-flow packet forwarding, forwarding like equal multi paths with L3

Figure 1: Carrier cloud networking framework Data center traffic typically includes: x Server to server communication: this can be between VMs/servers within one DC or across multiple DCs via DC interconnect network. DC interconnect can be between SP DC sites, or between SP and Enterprise DC sites, or between Enterprise DC sites. x Client to server communication: this can be between customer branch office and the cloud In a traditional data center network design, the L2/L3 boundary is on the aggregation switch. Aggregation switch and WAN router are L3 network for tenant L3VPN. The inter-DC connection may require both L2 extension and L3 VPN connectivity. There are many technologies for the inter-DC L2 extension. The most popular solution is VPLS. The client-DC connection can use existing L3VPN or L2VPN or public internet. The traditional data center design is widely deployed, but has the following limitations: x Not scalable: Classic L2 based DC network has 4K VLAN limit which cannot meet the high tenant scale requirement. Typical DC switches have limited MAC and ARP scale, which cannot meet the high VM scale requirement. Inter-DC L2 extension also has scale limit. x Non-optimized L2 forwarding: Traditional L2 forwarding is per-VLAN. For a large data center with multiple 10G links between switches, per-VLAN load balancing is not efficient compared to the L3 per-flow load balancing. x Other L2 restrictions such as slow network convergence, L2 flooding and broadcast storm, etc. x L2/L3 network boundary limits the VM mobility across PODs

287

x Provide active/active CE to PE multi-homing solution x Support network fast convergence x Support large DC sites scale by avoiding the need of the VPLS pseudowires x Support flexible network topologies: full mesh, partial mesh, tree x Support flexible MAC based policy control Although control plane MAC learning has a lot of advantages, scalability can be a potential limitation. It is not desirable to advertise millions of customer MAC addresses to the BGP routing table. Although the table size is not a big concern for the high end router platforms, dynamic VM movement can cause BGP control plane overhead. One of the potential enhancements to address the scale limitation is PBBEVPN[2]. The details are out of the scope of this paper. Let us review the case study of a tier-1 service providers data center network to demonstrate how E-VPN technology solves the scale limitations of a traditional network. Case study: A tier-1 service provider wants to build a cloud solution with 20 data center sites, each site with two WAN gateway routers, to offer virtual private cloud service. With a classic VPLS solution, each tenant network segment would need 39 VPLS pseudo wires to inter-connect the data center sites. To support 8K tenant segments (a common scale requirement) would require 320,000 VPLS pseudo wires on a single router. Such high control plane scale number is not supported by any of the high end routers in the market today. Next generation HW could potentially support this scale, but it is not a scalable solution. E-VPN technology does not require pseudo wires to inter-connect data center sites and thus, does not have this scale limitation. B. Evolution: End-to-End optimized L2 forwarding Cisco fabric path, Trill and 802.1aq are the technology evolutions for the native L2 forwarding for intra-DC network. The building block of the above technologies is to use ISIS to build the optimal L2 forwarding path, instead of STP or other classic L2 protocols. Although Cisco fabric path (or Trill or 802.1aq) and E-VPN can be deployed both as intra-DC and inter-DC technologies, the most scalable solution is to use EVPN for inter-DC and fabric path (or Trill or 802.1aq) for intra-DC technology. It is very desirable to have a seamless integration between inter-DC technology and inter-DC technology to provide end-to-end optimized L2 forwarding. The example below uses Cisco fabric path seamless integration with E-VPN to demonstrate the solution advantages. The same solution can apply to other DC network technologies integrated with E-VPN, such as IETF Trill over E-VPN, or IETF SPBM over E-VPN. Control plane: For inter-DC, BGP is used to advertise WAN routers switch ID. For intra-DC, ISIS is used to advertise the DC switchs ID. WAN router does the routing redistribution between BGP and ISIS. Another option is to use default route. From control plane perspective, it works like a traditional SP L3 network. With redistribution, all the switches

in the DC (including the WAN router) learn all the other switch IDs. With default route option, DC switches dont need to learn switches/routers ID from other DC sites. Instead, it can use default route pointing to WAN routers. Data plane: Packet received on the fabric path edge port is encapsulated with fabric path header and forwarded to the DC WAN router based on ISIS shortest path. When the packet reaches the WAN router, it retains the data plane encapsulation and adds the MPLS E-VPN encapsulation (MPLS labels). The packet is forwarded to the remote DC based on the BGP table. Advantages: It provides L2 service to the end user application using L3 routing technology between DC switches or routers. It solves the classic L2 forwarding restrictions and has the following benefits: x Supports VM mobility, application transparency server clustering and

x Highly scalable like the classic L3 network x Efficient forwarding, multi-paths and including intra-DC and inter-DC network x Network fast convergence x Flexible policy control with BGP x High MAC scale with conversational MAC learning: customer MAC is kept at the fabric path switch edge port and is learnt only if there is active flow x Segment ID for 16M service instances ECMPs,

Figure 2: Optimized L2 forwarding: Fabric Path seamless integration with E-VPN C. DC fabric evolution: IP virtual overlay Some of the new technologies described in previous section may require HW or SW upgrade on the DC switches, which may not be desirable. In addition, some of the new technologies may result in additional operational complexity compared to the classic L2 and IP routing network. IP virtual overlay solution is to build a virtual network on top of the physical IP only network. The virtual network itself can be L2 or L3 based network.

288

L2 virtual overlay: Customer L2 frame is encapsulated as IP packet over the DC physical IP network. Tunnel end point can be on traditional top of the rack switch (ToR), or on host (Hypervisor). Typically, ToR based virtual overlay provides better performance and host based virtual overlay is easier to design. For L2 virtual overlay, there are a few proposals in the market, such as VXLAN[4], NV-GRE, STT. This paper uses VXLAN as example to explain the L2 virtual overlay technology. The same concept can be extended to other proposals. VXLAN: VXLAN uses L2 over UDP data plane encapsulation, which provides better packet load balancing in the DC physical network compared to IP/GRE encapsulation. VXLAN provides a L2 broadcast domain similar to traditional VLAN. VMs in the same VXLAN can forward packet to each other just like traditional L2 network. The hypervisor or the ToR can work as the VXLAN tunnel point to perform the VXLAN packet encapsulation. VXLAN uses 24 bits for the segment ID to support 16M L2 broadcast domains. The biggest advantage of VXLAN is to create scalable L2 virtual overlay network on top of the IP only physical network. It can work on any existing DC network. Figure 4: VXLAN with control plane MAC learning L3 virtual overlay: as mentioned earlier in the paper, carrier cloud solution needs to support heterogeneous applications and thus cannot eliminate the support for L2 extension in most cases. On the other hand, certain data center such as mobile LTE data center may have IP only applications. In such cases, the solution can be simplified with L3 IP overlay approach. LISP[5] is one of the IETF standard protocol to provide L3 IP overlay solution. The details are out of the scope of this paper. D. Optimized L3 routing Data center traffic can be categorized as follows: x Intra-VLAN L2 traffic within the same DC site x Intra-VLAN L2 traffic across DC sites x Inter-VLAN L3 routing traffic within the same DC site. For example, a single tenant may have multiple L2 segments that require inter-segment routing x L3 WAN routing traffic exiting DC that can be over L3VPN or internet L2 scale requirements such as MAC, VLAN, VPLS VSI and pseudo wires can be addressed by optimized L2 forwarding technologies described in the previous section. However, optimized L3 routing is also required to address high L3 scale requirements for ARP, L3 VRF and FIB. In the classic network design, DC aggregation switch acts as L3 routing gateway for the intra-DC routed traffic. Either DC aggregation switch or WAN router works as L3/L3VPN gateway for WAN routing traffic. In either case, the platform has the burden to support large ARP scale. DC switches are optimized for the performance, cost and power. They typically dont have the CPU processing power and table size to support large scale. To support millions of VMs, optimized L3 routing is required. The principle of the optimized L3 routing is very simple: push L3 gateway deeper into the DC, closer to the host. Instead of using DC aggregation switch as L3 gateway, ToR or host (hypervisor, VM) is used for L3 routing. There are multiple design options to optimize L3 routing. Fundamentally, there are two options: centralized PE routing or distributed PE routing: x Centralized PE Routing: In this design, WAN router provides the L3 VPN PE routing function for northsouth routing traffic. The CE router gateway can have different options. For better ARP scale, CE router gateway can be on the ToR or on the VM. CE routing function on the VM is called virtual CE router. Virtual

Figure 3: L2 virtual overlay using VXLAN The MAC learning in the VXLAN could be the same as classic L2 using data plane learning. In this case, it requires IP multicast in the data center physical network to flood the packet to all VXLAN tunnel end points initially. The other solution is to use control plane for MAC learning. In this case, VXLAN tunnel end point can query the external database for the target VM MAC and VTEP IP gateway address pair instead of flooding for the data plane learning. This provides better scalability. VXLAN can be deployed within one DC, but it can be also deployed among multiple DC sites as DC inter-connect technology.

289

CE routing provides many advantages in the overall cloud network design x Distributed PE Routing: In this design, L3 VPN PE routing is distributed to the ToR or to the host (hypervisor or VM). PE on the host is also called Virtual PE. Here, the WAN router works as ASBR to connect to SP NGN network domain and for some potential interworking function. For the distributed PE routing, there are different design options for the DC fabric. MPLS can be brought to the PE natively or L3 virtual overlay technology such as MPLS over GRE or LISP can be used. In this option, CE router can be the same as PE router or use the virtual CE router options 1) Centralized PE routing, with Virtual CE routing In this design, WAN router is the L3 VPN PE. Virtual router on the VM is the tenant CE router. Virtual CE and PE run common CE-PE routing protocol, which can be BGP, or static/default routing. CE-PE is directly connected through L3 logical link, which could be over native L2 VLAN, L2 Fabric Path or L2 virtual overlay. Virtual CE is one design option. The CE routing could also be done by the physical ToR for higher performance.

x Cross server routing through virtual CE eliminates the ARP scale issue x Cross server bridging through L2 overlay eliminates the VLAN and MAC scale issue x Alleviates the need to extend L3 VPN edge extension into the data center to simplify the network administration x Potential dynamic PE VRF provisioning model to simplify PE/L3VPN provisioning 2) Distributed PE routing

Figure 6: Optimized L3 Routing: distributed PE routing In this solution, SP L3VPN network is extended to the DC network. Traditional L3VPN technology uses BGP as control plane, and MPLS as forwarding plane. However, this is not strictly required for the DC network where MPLS is less popular. The alternative option is MPLS over GRE, or other IP based tunneling encapsulation. For the control plane, BGP is well designed and mature for L3VPN function. LISP could be another option. With LISP pull mode, the PE doesnt need to hold the full tenant routes. It is on-demand basis by pulling the LISP directory server. The PE can be on the ToR switch for better performance or on the hypervisor or VM (virtual PE). The real PE on the ToR can run full BGP or LISP control plane. For the virtual PE on the host, it is desirable not to run full BGP control plane to avoid control plane overhead. Instead, the virtual PE can be managed by an external controller as virtual line card, while the controller works like a virtual routing process. The virtual routing process can run full BGP control plane and communicate with virtual PE via simple protocol such as XMPP, LISP. The biggest advantage of the distributed PE solution is high scalability. The overall tenant scale can be increased by adding more PEs (either virtual or real). On the other hand, the distributed PE model adds complexity with the need to support L3VPN function inside the DC. For the centralized PE model, the overall tenant scale is limited by the WAN router capability. However, high end

Figure 5: Optimized L3 routing: Centralized PE with virtual CE solution This solution offers many advantages: x Flexible MAC based policy control x No cross-tenant dependencies, management and orchestration which simplifies

x Pay as you grow solution Virtual CE per tenant follows same model for tenant routing as other tenant services, e.g. RaaS x Mirrors branch CE model, i.e. can support same features and management models x Allows end-to-end services with enterprise sites (WaaS, LISP, IPSEC, etc)

290

router platform can support tens of thousands of VRFs and millions of VPN routes, which could be sufficient for most of the data center networks. IV. SUMMARY

VXLAN and LISP, and optimized L3 routing with centralized PE/virtual CE solution or distributed PE solution. REFERENCES

Service providers are looking for cloud solutions to solve some of their biggest business and technology challenges: reducing costs, creating new levels of efficiency, and facilitating innovative business models that drive revenue growth. The existing carrier cloud network solutions have several big challenges such as scale limit, non-optimized forwarding and limited VM mobility. This paper described a couple of solutions for the evolution of the carrier cloud networking. It includes end-to-end optimized L2 forwarding using E-VPN and fabric path, IP virtual overlay solution using

[1] [2] [3] [4] [5]

Sajassi, Aggarwal et al. E-VPN: http://datatracker.ietf.org/doc/draft-ietfl2vpn-evpn/ Sajassi et al. PBB-EVPN: http://datatracker.ietf.org/doc/draft-ietf-l2vpnpbb-evpn/ Cisco Fabric Path: http://www.cisco.com/en/US/netsol/ns1151/index.html Mahalingam, Dutt et al. VXLAN: http://datatracker.ietf.org/doc/draftmahalingam-dutt-dcops-vxlan/ LISP:http://datatracker.ietf.org/wg/lisp/

291