Sunteți pe pagina 1din 49
ee
ee

Strategic Technology Infrastructure for Regional Competitiveness in the Network Economy

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

6: Leveraging Advanced Optical and Ethernet Technologies eCorridors Program  2003 Virginia Polytechnic Institute

eCorridors Program

2003 Virginia Polytechnic Institute and State University, Blacksburg, VA. All rights reserved.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Preface

This series of reports, entitled Strategic Technology Infrastructure for Regional Competitiveness in the Network Economy and packaged in eleven Volumes, is the culmination of a dedicated effort of the following individuals and organizations. Each Volume can be viewed as a stand-alone publication; however, it should be noted that each Volume was written in the context of the overall project. The project utilized the Southside and Southwest Virginia regions as a model for a low-cost Geodesic Mesh network design and viable financial model that could be replicated in any region of the U.S.

Volumes

1)

Rationale, Environment, and Strategic Considerations

2)

Connecting the Regional Infrastructure to National and International Networks

3)

A Fiber Optic Infrastructure Design for Southside and Southwest Virginia

4)

Fiber Optic Infrastructure Design Guide

5)

Financial Feasibility and Investment Rationale

6)

Leveraging Advanced Optical and Ethernet Technologies

7)

Speculative and Alternative Technologies

8)

Community, Applications and Services

9)

Demographics for Southside and Southwest Virginia

10)

Health Information Technology and Infrastructure

11)

Education in the 21st Century

i

Volume 1: Rationale, Environment, and Strategic Considerations

Acknowledgements

The following individuals and organizations contributed to the development and preparation of this series of reports.

Allen, Morgan Arellano, Christian Aughenbaugh, John Bevis, Jeff Blythe, Erv Bohland, James Bottom, Beth Bowden, Phillip Brown, Eric Charlton, Garland Chen, Daniel Cohen, Marc Colbert, Joy Croasdale, Hud Crowder, Jeff Dalton, Jody de Vries, Marten Dwyer, Sharon Fisher, Tommy Franklin, Nancy Gaylord, Clark Hach, Richard Hall, Shannon Hares, Glynn Harris, Carl Hey, Bryan Hoover, Maynard Horton, Helen Jones, Brian Jones, Doug Kanter, Theresa

Kidd, Jeff Lee, Steven Lilly, Judy L. Martin, David Martin, Evelyn Mathai, Mathew McCann, Jessica Morrison, Brandon Neidigh, Brenda Nichols, John Pelt, Ranson Perry, Mike Pheley, Al Plymale, V. Jean Plymale, Bill Pollard, John Rodgers, Pat Sanghvi, Harsh Shepherd, Scott Sheppard, Scott Shumaker, Richard Stewart, Jeb Stock, Doris Tyree, Charles Waddell, Bobby Wenrich, John Woods, Cindy Zirkle, Mary

ii

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Organizations and Companies

3com

Adelphia

Advance Fiber Optics

Advanced Network Infrastructure & Services, VA Tech

AEP

AFL

Telecommunications

Alcatel

Anderson &

Associates

Asia Venture Partners

AT&T

Avante

Bristol Virginia Utilities Board

Celion

Center for Wireless Telecommunications, VA Tech

Chamber of Commerce, Richlands

Chilson Enterprises

Cisco Systems

Corning Cable

Systems

Cox Communications

Danvilleonline.com

Dominion Telecom

Economic Development Assistance Center, VA Tech

EngHouse Systems

Enterasys

Extreme Networks

Economic Development Assistance Center, VA Tech

Floyd County High School

Floydva.com

Force 10

Foundry

Future of the Piedmont Foundation

Gamewood, Inc.

GeoTel

Grant County Public Utility District

Hatteras

Hewlett-Packard

IBM

Institute for Advanced Learning and Research, VA Tech

Institute for Connecting Science Research to the Classroom, VA Tech

ION Consulting

KMI Corporation

LENOWISCO

Level 3 Communications, Inc.

MapInfo

Manticom

Marketing Dept., VA Tech

Micrologic, Inc.

Nexans

Nortel

Old Dominion Electric Coop

iii

Pirelli

Prince Edward County Office of Economic Development

Progress Telecom

Qwest

RACO, Inc.

Rinderva.com

Riverstone

Salira

Sprint

Terabeam

Urban Affairs and Planning Dept., VA Tech

Valleynet

Verizon

Wiltel

Worldcom

Worldwide Packets

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Table of Contents

Introduction

Historical Perspective 3 What is Metropolitan/Regional Ethernet?

6

1

Ethernet in the First Mile

7

Backbone Networks for Non-Ethernet Broadband Access

11

Interconnecting Service Provider Networks

14

Cross-connects as Meet Points

14

Packet Switching or Multiplexing as Meet Points

16

Collocation at Meet Points

16

Ethernet-Based Internet Exchange Points

17

Cost and Manageability Benefits of Metro Ethernet

19

Limitations of Current Metro Ethernet Technology

22

Slow Recovery From Link Failures

22

Lack of Remote Fault Isolation

24

Lack of In-Service Performance Monitoring and OAM

25

Limited VLAN Tag Space

25

Spanning Tree Inefficiencies on Highly Meshed Networks

26

Lack of End-to-End Service Guarantees

26

Is Metro Ethernet Ready for Prime Time?

27

Best Practices for Metro Ethernet Networks

29

Architecture: Link Layer or Network Layer Switching

29

Spanning Tree Configuration

30

Forwarding Table Considerations

31

iv

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Protocol Filtering

32

Rate Limits on Broadcast and Multicast Frame Flooding

32

IP Multicast Frame Flooding and Rate-Limiting

33

Quality-of-Service Controls

33

Acronym Glossary 35

38

References 39

Acknowledgements

v

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

List of Figures

Figure 1: MSAP extending access network

8

Figure 2: MSAPs connecting Ethernet rings

9

Figure 3: Schematic of traditional DSL access network

11

Figure 4: MSAP connecting multiple ISPs and access networks

12

Figure 5: Schematic of cross-connect pedestal as meet point

15

Figure 6: Collocation at meet points

17

Figure 7: Link Aggregation between MSAPs

23

vi

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Introduction

Telecommunications services delivery over the last ten years has been dominated by digital circuit-based technologies such as Time Division Multiplexing (TDM) and Synchronous Optical Network (SONET). These technologies are readily adapted to large-scale voice communications, as they effectively multiplex resources with fixed units. For the same reasons, they are less suited for data communications, where the resource demands of the applications are highly variable. Packet switching technologies such as Frame Relay and ATM played an important role in the unprecedented growth in the data telecommunications market, ushered in by the rise of the Internet. With the realization of digital voice and video transmission, the efficiencies of packet switching can also be applied to these applications.

Inarguably, all of these technologies have been extraordinarily effective. Yet, they where designed and developed for a very different telecommunications economy than that which exists today. In today’s telecommunications market, service providers must be agile to seize opportunities. The Internet and all things around it move at a very fast pace. Service providers must be able to scale capacities upward to meet surging demands in order to remain competitive. Today’s service provider cannot assume a decades-long return on capital investments in switching, multiplexing, and line termination equipment.

While SONET, ATM, and Frame Relay will continue to play an important role in telecommunications for many years to come, increasingly, service providers are looking to Ethernet technology as the platform for cost-effective delivery of converged voice, video, and data telecommunications services. Originally developed for use in local area network environments, Ethernet has in recent years evolved such that it is a cost- effective, robust, scalable, manageable platform for metropolitan and regional telecommunications. The ubiquity of Ethernet technology in enterprise networks and its increasing role in metropolitan networks has created an enormous and highly competitive market for Ethernet equipment. This, along with the relative simplicity of the technology, has led to twenty-year history of ten-fold increases in link capacities for approximately three times the cost of the preceding generation of equipment.[4]

1

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Furthermore, the near-total market adoption of each new generation of Ethernet technology has ultimately driven down costs even further.

In this report, we examine the role that Ethernet technology can play delivering telecommunications services on a metropolitan and regional scale. We examine the case in favor of so-called “metro Ethernet” networks, and consider the foremost limitations of the current generation of metro Ethernet equipment.

2

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Historical Perspective

Early wide area data telecommunications networks utilized modems on dial-up or leased telephone lines. The point-to-point nature of such services greatly limited scalability, since a new line was needed for each concurrent data connection to another remote location. True scalability in wide-area data telecommunications networks was ushered in by packet switching. Packet switching protocols such as X.25 became popular because a single dial-up or leased line connection to the network allowed communication with multiple remote locations via the concept of virtual circuits.

The first packet switching protocols were burdened with the need to operate effectively over analog leased lines with very high bit-error rates. The X.25 protocol and its many predecessors and relatives were greatly complicated by error detection and correction mechanisms. With the advent of digital transmission lines, the need for a lightweight packet switching protocol resulted in the development and large scale deployment of Frame Relay. Frame Relay networks retained the fundamental label-swapping techniques of X.25 virtual circuit switching while dispensing with the error correction mechanisms. The simplicity and elegance of Frame Relay allowed it to easily operate at speeds up to 1.5 megabits per second, which was quite impressive at the time.

Motivated by a desire to converge applications such as voice, video, and data communications on the same network, and to meet the future needs for increased bandwidth, in the late 1980s the telecommunications industry developed specifications for Broadband ISDN (B-ISDN). The B-ISDN specification consisted of two significant components: Synchronous Digital Hierarchy (SDH, known in the U.S. and hereafter in this document as SONET) and Asynchronous Transfer Mode (ATM).

SONET provided a robust ring-based architecture for synchronous digital transmission over fiber-optic cable. SONET was critical to meeting the capacity demands imposed by rapid growth and competition in the long distance voice market, as well as the unprecedented demand for bandwidth that followed the privatization and subsequent commercialization of the Internet.

3

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

ATM was intended to address the inefficiencies of time division multiplexing (TDM) that were inherent to SONET and its T–1 roots. Through the use of statistical multiplexing and virtual-circuit-level class-of-service parameters, it allowed delay sensitive traffic classes such as voice and video to be mixed freely with other traffic types on a common digital transmission line. Proponents of ATM believed that it would become the fundamental means of supporting voice, video, and data telecommunications.

Carrier deployments of ATM in the mid-to-late1990s further fueled the rapid growth of the Internet and the use of the Internet Protocol (IP) for intranet, extranet, and other applications, by providing more bandwidth and more flexibility than had previously been available. ATM also contributed to the continued success of Frame Relay, by providing a scalable backbone network that was interoperable with Frame Relay and would allow carriers to meet the growing demands for Frame Relay service that accompanied the rise of the Internet.

While ATM continues to be an important component of carrier networks, it did not achieve the goal of convergence for which it was originally intended. The success of ATM in achieving convergence depended on end-to-end deployment of the technology. In particular, this meant that the enormous installed base of personal computers in enterprise networks would need to be directly connected to local area ATM networks in order to fully leverage the capabilities of ATM. However, in local area networks, there was already a dominant technology that would not be easily displaced: Ethernet.

When ATM emerged as a potential LAN technology, it offered significant advantages over Ethernet. At that time, Ethernet was relatively primitive, relying on broadcast media, and providing only 10 megabits per second of bandwidth shared between all users on the LAN. Furthermore, Ethernet provided no capability for differentiated levels of service that would allow mission-critical or time-sensitive applications to be prioritized above routine traffic on the network. Ethernet networks were interconnected using transparent bridges and multi-protocol routers, which partitioned the Ethernet into smaller shared segments. Partitioning the network in this manner added stability and scalability, at the expense of greater end-to-end packet delay, and increased packet loss. Despite these shortcomings, Ethernet was inexpensive and easy to implement and was adopted by enterprise networks of all sizes.

4

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

The installed base of existing shared Ethernet might not have precluded ATM from taking over the LAN environment. However, the development and rapid deployment of Ethernet switching technology and 100 megabit per second Ethernet delivered a crushing blow to any hope that ATM might one day rule the enterprise network environment. Ethernet switching was a relatively simple variation on the transparent bridge. Advances in semiconductor technology enabled the development of inexpensive high-density Ethernet switches that could be used to replace existing shared hubs. This allowed the existing Ethernet base to migrate to networks providing much greater amounts of bandwidth, while not requiring wholesale replacement of existing components. In particular, existing desktop computer hardware and software could continue to be used on a switched Ethernet network. Fast Ethernet, as the 100 megabit per second variant is known, provided the means to increase the bandwidth on backbone segments by an order of magnitude, in addition to providing very high capacity links for network intensive server applications.

Enterprise network managers found that by simply eliminating congestion on their networks, virtually all applications could be made to work successfully, without the need for ATM’s advanced traffic management capabilities. Any remaining hope for ATM in the enterprise was lost as Ethernet switches evolved to include features such as traffic prioritization, rate limiting, and advanced queuing. These features are critical to supporting real-time applications such as voice and video. The maximum Ethernet link speed also increased by an order of magnitude, to 1000 megabits per second (1 gigabit per second). The extraordinary capacity and advanced traffic management capabilities of Ethernet obviated any need for a technology other than Ethernet at the data link layer of the enterprise.

Indeed, by the turn of the century, rapid advances in Ethernet technology made it feasible for use in metropolitan area networks where SONET, ATM, and Frame Relay technologies ruled. Today’s Ethernet technology supports line speeds from 10 megabits per second to 10,000 megabits per second (10 gigabits per second). Only SONET itself offers line speeds that are comparable to Ethernet, but the inefficiencies of the time division multiplexing inherent to SONET combined with the historically higher capital costs for SONET equipment make Ethernet a far more cost-effective alternative in metropolitan and regional area networks.

5

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

What is Metropolitan/Regional Ethernet?

A metropolitan or regional Ethernet (known herein as a “metro Ethernet”) is a packet

switched network that employs Ethernet technology for wide-area connectivity, especially within a metropolitan area or at a similar regional scale. Metro Ethernet services are typically used by enterprise networks and access service providers for connectivity to the public Internet and to extend the functionality of corporate networks between geographically separate sites.

Nodes in a metro Ethernet are switches operating at either the OSI data-link layer or at the OSI network layer (where the Internet Protocol is the universal network-layer protocol choice). Often, some combination of the services provided by each of these OSI layers is used. Links between nodes in a metro Ethernet are typically a physical point-to-point connection, provisioned over fiber or optical services (such as DWDM), and can operate at any Ethernet line speed (10 megabits per second to 10,000 megabits per second) depending on the service requirements.

Typically, metropolitan and regional fiber networks have ring architectures. While metro Ethernet can successfully operate over ring topologies, today’s standards-based Ethernet is better suited to mesh topologies. Consequently, much of the emphasis in metro Ethernet development is in efficiently supporting ring topologies with a robust, highly-resilient Ethernet technology.

Metro Ethernet network services are provisioned using virtually any combination of logical point-to-point, point-to-multipoint, or multipoint-to-multipoint configurations, over links operating at any Ethernet line speed. Bandwidth can be allocated to such services

in increments as small as 1 kilobit per second. Bandwidth can be dedicated or shared

between multiple service users.

Quality-of-service is implemented in metro Ethernet networks using a combination of many techniques operating at both the data-link layer and the network layer. These techniques include most of the same capabilities found in Frame Relay and ATM

6

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

networks, including packet classification, marking, rate limiting or policing, and transmit queue scheduling with multiple queues.

Some examples of common and proposed uses of metro Ethernet networks are:

Residential and commercial Internet access (so-called Ethernet in the First Mile)

Backbone networks for other broadband access technologies (e.g. DSL, cable modem, wireless broadband)

Corporate LAN extension

Ethernet in the First Mile

Residential and commercial Internet access can be delivered using metro Ethernet networks. In this model, Ethernet connections are extended to the customer premise using a 1000Base-X Ethernet connection over fiber-optic cable. Ethernet switches are located in central-office-like facilities known as Multimedia Service Access Points (MSAP) or in fiber termination pedestals located in utility easements. Figure 1 shows an Ethernet switch located in a community MSAP that is providing 1000Base-X Ethernet connections to homes and businesses in the local community.

7

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

MSAP

Internet Ethernet Ethernet Switch 1000Base-X SONET ADM 1000Base-LX Ethernet ONT Switch 10/100/1000TX Router
Internet
Ethernet
Ethernet
Switch
1000Base-X
SONET ADM
1000Base-LX
Ethernet
ONT
Switch
10/100/1000TX
Router
Wireless
MSAP
IP Phone

Figure 1: MSAP extending access network

Using the regional fiber architecture recommended in this report, Figure 1 shows the metro Ethernet network extending the access network in the community to a distant MSAP (perhaps in a larger city) where an Internet Service Provider can deliver high- capacity access to the Internet using prevailing SONET technologies.

Because of the prevalence of ring topologies in metropolitan and regional fiber networks, Ethernet in the first mile will typically be implemented using a combination of MSAP facilities and smaller distribution switches located in fiber termination pedestals along utility easements. As shown in Figure 2, small Ethernet switches are arranged on fiber rings passing through residential communities and commercial districts. The logical ring topology minimizes the number of fiber pairs consumed on the physical fiber ring. These smaller distribution switches are located close to customers, such that the costs for additional fiber “laterals” to reach each customer premise are substantially lower.

8

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

1000Base-X 1000Base-X 1000Base-X Internet MSAP MSAP 1000Base-X 1000Base-X 1000Base-X
1000Base-X
1000Base-X
1000Base-X
Internet
MSAP
MSAP
1000Base-X
1000Base-X
1000Base-X

Figure 2: MSAPs connecting Ethernet rings

A community may have multiple logical Ethernet rings providing access services in different areas of the community. In this case, the MSAP typically acts as the hub of the community-area distribution networks. Leveraging the regional fiber infrastructure proposed in this report, each MSAP is interconnected to other MSAPs, allowing Internet and application service providers to be located wherever it is most advantageous.

The IEEE Ethernet in the First Mile working group (IEEE 802.3ah) is drafting specifications to make Ethernet-based access networks scalable, manageable, and fault tolerant. Technical proposals for OAM (operations, administration, and management), customer premise network termination, and for both fiber- and copper-based physical layer specifications are being considered by the working group.

9

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Efforts are underway in the IEEE 802.17 Resilient Packet Ring working group to define fault-tolerant ring standards for Ethernet that will ultimately provide a robust architecture that meets or exceeds the resiliency of venerable SONET ring architecture.

The Multimedia Service Access Point is described in greater detail in the section entitled Interconnecting Service Provider Networks.

10

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Backbone Networks for Non-Ethernet Broadband Access

In small communities, broadband access providers employing DSL, cable modem, or even wireless broadband technologies can exploit a regional fiber infrastructure and metro Ethernet to reduce operating costs, and offer revenue-generating broadband access services – even on a small scale.

As shown in Figure 3, traditional approaches to deploying DSL technology have employed SONET-based ATM backbone networks interconnecting telco central offices. The DSL access multiplexer (DSLAM) used to provide connections to residential or commercial subscribers is connected to this ATM backbone network. The ATM backbone provides the means of interconnecting service points to backbone resources such as tier 1 Internet service providers and application service providers. Because of the high infrastructure costs associated with the SONET/ATM architecture, these solutions are not cost effective for smaller communities, where the potential subscriber base is small.

Central Office

Internet DSLAM ATM ATM Switch SONET ADM ATM Switch DSL Modem Router IP Phone Wireless
Internet
DSLAM
ATM
ATM
Switch
SONET ADM
ATM
Switch
DSL Modem
Router
IP Phone
Wireless
POP

Figure 3: Schematic of traditional DSL access network

11

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Using the community MSAP model and regional fiber infrastructure described in this report, it is feasible for even a small rural telephone cooperative to cost-effectively deploy DSL Internet access services. In the lower-left of Figure 4, we see the central office continuing to serve as the termination point for residential and commercial DSL connections. Using 1000Base-X Ethernet, the DSLAM connects to the community MSAP, where access customers are interconnected with an Internet service provider. The MSAP containing Internet service providers (labeled “ISP A” and “ISP B” in Figure 4) could be co-located in the same building facility as the telephone cooperative’s central office, or it could be in some other part of the region, where the costs for traditional high- capacity SONET-based access to the Internet are lower.

1000Base-X 1000Base-X 1000Base-X MSAP Internet Internet ISP A ISP B 1000Base-X 1000Base-X Central Office
1000Base-X
1000Base-X
1000Base-X
MSAP
Internet
Internet
ISP A
ISP B
1000Base-X
1000Base-X
Central Office
Broadband Wireless
Ethernet
DSLAM
Switch

Figure 4: MSAP connecting multiple ISPs and access networks

12

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Figure 4 also illustrates Ethernet-based network access, DSL access, and wireless broadband access all coming together at the community MSAP. This emphasizes a key function performed by the MSAP: interconnection.

13

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Interconnecting Service Provider Networks

Both the Internet and the public switched telephone networks are intricate meshes of connectivity between different service provider networks. Interconnection allows service providers to specialize in network access or higher-layer services, extends the potential market for all service providers, provides better performance for traffic between these networks, and reduces the cost of upstream connectivity to higher-tier providers. There are two basic strategies for interconnecting between such networks: private direct connections and connections at common meet points. When there is more than one other service provider to connect with, the expense of having direct connections to each can be significantly more than connecting to all or most of them at a common meet point.

One of the motivations for direct connections is the desire for service providers to exercise more control over resources and limit the exposure to risk from other entities managing those connections. In order to engender the trust of numerous service providers, meet points are best operated by neutral entities. Such neutrality addresses the concern over potential competitors having control over a service provider’s critical interconnection resources.

The technologies for these meet points may be variable, both in terms of the physical medium and the link-layer technologies used. While the interconnection medium could be copper for some applications, we will focus on fiber as the principal physical interconnection medium. The choice of link-layer is independent from the perspective of interconnection methodology, though the most cost-effective and suitable choices will be important to successful implementation.

Cross-connects as Meet Points

The simplest form of a common public meet point is a cross-connect pedestal in a public right-of-way. In such a scenario, service providers need to make only one fiber build of sufficient capacity to this meet point. Having made this investment once, a service provider can then connect with any others who have likewise constructed facility to this

14

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

point. This may reduce costs significantly over constructing separate facilities to each potential other service provider.

The value of the cross-connect meet point may be increased by having numerous such meet points distributed throughout a geographic region. The entity operating the cross- connect point could connect these distributed meet points via fiber optic cable, and lease access to this dark fiber facility to further reduce the costs to service providers who are closer to some of these. This distributed cross-connect extends the reach of service providers beyond what they may have otherwise been able to cost-effectively construct.

In the model of cross-connect meet points, each connection between service providers is still a dedicated connection and still consumes resources from switching electronics, lasers, etc. For N service providers connecting at such a meet point to each other, this is a total of N(N-1) such connection points; each of the N service providers would have N-1 connections to deploy this “full mesh” approach. The cost of the connecting electronics may still dominate the cost for such interconnections. If the meet point could more efficiently use these connections via shared media, this may further reduce the costs associated with service provider interconnection.

Provider A

Pedestal provides full mesh cross-connect between providers. Provider A Provider B Provider C Provider D
Pedestal provides full mesh cross-connect between providers.
Provider A
Provider B
Provider C Provider D
Patch bay

Provider B

A Provider B Provider C Provider D Patch bay Provider B Provider C Provider D Figure
A Provider B Provider C Provider D Patch bay Provider B Provider C Provider D Figure

Provider C

Provider D

Figure 5: Schematic of cross-connect pedestal as meet point

15

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Packet Switching or Multiplexing as Meet Points

Instead of having dedicated facilities for each possible connection between service providers at meet points, the meet point operator could offer packet-switching or multiplexing services. Using such services, the N service providers may require only one connection each in order to exchange traffic with any of the other service providers. The multiplexing technology could be ATM, Ethernet, WDM, IP, or any of a number of other such technologies. All of these would more efficiently use the physical connection to the meet point, with the trade-off that services would need to be compatible with the chosen multiplexing technology. Most multiplexing technologies, such as SONET or WDM, are based on dedicating virtual resources, for example time slices or frequencies, respectively. To use these dedicated virtual resources, these technologies still may have a significant amount of dedicated resources from the connecting service provider. On the other hand, using a packet-switching technology, such as ATM or Ethernet, allows for more flexible multiplexing of virtual connectivity. Of these, the cost advantages of Ethernet are significant.

The operation of such multiplexing or switching services would require more involvement by the meet point operator. In this scenario, an enclosed space with power would be required, in addition to regular monitoring and management of the switching service provided by the meet point operator. However, the cost efficiencies of such a scenario may be compelling. Offering switching services at the meet point does not preclude the possibility of having physical cross-connects. Those applications or service providers for whom this is more suitable could still use such a meet point strategy and still derive the cost savings relative to independently constructed facilities.

Collocation at Meet Points

So far we have considered the meet point as an isolated point or distributed points to which the service providers would construct fiber facility, keeping all their electronics at their own facilities. Once a meet point operator has made the investment in building, power, HVAC, etc, to operate a switching service, the expansion of such a space to accommodate equipment owned and operated by the constituent service providers may not be significant. This may also provide opportunities for other service providers who

16

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

are better suited by collocation service to connect to the meet points. To more cost-
are better suited by collocation service to connect to the meet points. To more cost-
effectively connect these collocated providers, the meet point operator may provide
copper-based services in addition to the fiber-based services needed to support
connections from outside the facility.
Provider A
Provider B
WWW Server
Mail Server
WWW Server
Mail Server
Media Server
Media Server
Meet Point 1
Provider A
Provider B
Provider C
WWW Server
Mail Server
Meet Point 2
Meet Point 3
Media Server
Provider D
Provider C

Figure 6: Collocation at meet points

Ethernet-Based Internet Exchange Points

As discussed in other sections of this report, the cost benefits of Ethernet as a multiplexing and switching technology make it a compelling choice for meet point services. Using Ethernet as a medium for Internet Protocol connectivity between service providers provides a basis for establishing the meet point as an Internet Exchange Point. In such a scenario, connecting service providers advertise their IP addresses to each other and share a common Ethernet network for exchanging Internet traffic. By connecting to the same Ethernet network, service providers can decide what routing policy they want to have with each other, whether they want to provide transit access to upstream Internet service providers, etc. Just as the cross-connect meet point can be distributed over a geographic region, so too can the Ethernet exchange point. By

17

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

building multiple locations and interconnecting them via Ethernet switches, the meet point operator can distribute the Ethernet exchange point over a region.

Typically, service providers have certain “local” IP addresses they would like all connecting peers to be able to reach via the Ethernet exchange point. To better enable this strategy, the meet point operator can also operate route servers available to all participating service providers for the purpose of exchanging “local” routes.

In addition to having a common Ethernet broadcast domain available for service providers who wish to exchange IP traffic, an Ethernet based meet point can be used to establish virtual private connections, using Ethernet’s Virtual LAN technology, 802.1Q. Using VLANs, connecting service providers can use the same physical Ethernet connection to virtually connect to other providers, in much the same fashion as ATM and Frame Relay provide virtual circuit functionality.

Such an Ethernet meet point, with available route servers and virtual LANs, has been dubbed a “Multimedia Service Access Point” (MSAP). In addition to these stated MSAP services, there may be dark fiber and collocation facilities available, as well as management access to collocated equipment. By expanding from the role of simple cross-connect to an open architecture for Internet based services, the MSAP leverages the cost-effective and ubiquitous nature of Ethernet. This allows for the myriad of Internet applications to be offered by providers at the MSAP: electronic mail, web hosting, streaming media, fiber-based residential and business Internet access … all these become enabled by having an open, provider-neutral infrastructure for service providers.

18

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Cost and Manageability Benefits of Metro Ethernet

Ethernet links operating at 1 gigabit per second can be extended over fiber optic cable at

distances of up to 70 kilometers, without amplifiers or regeneration. For less than

US$40,000 in capital expenditures, it is possible to “light” a fiber span of up to 70

kilometers and immediately provide gigabit Ethernet services between two distant

locations, with all of the physical redundancy and fault resiliency of SONET 1 .

The cost of lighting the same fiber span using OC-12 SONET (which operates at only

622 megabits per second) is almost US$130,000 2 . This difference in capital cost is at

least partly reflective of the different economies of scale for the manufacture of

components needed by each technology. The nearly ubiquitous adoption of Ethernet in

enterprise networks has created a much larger market and far more competitive pricing

for Ethernet products than for SONET products. SONET sales are typically limited to the

service provider market sector.

In addition to the clear cost advantage, Ethernet provides other benefits not available in

the SONET model. Constraints imposed by the SONET architecture preclude using the

entire capacity of the facility (622 megabits per second) for any particular connection.

Typically, only as much as one quarter of the available ring capacity (155 megabits per

second) can be provisioned for any logical circuit on the ring.

Because SONET uses time division multiplexing with very coarse-grained bandwidth

divisions, the capacity of the ring will typically be underutilized even when the ring is fully

provisioned. Service providers must charge customers for more capacity than they are

actually using because the provider has no means with which to recover unused

1 Based on an implementation using Cisco Catalyst 3550 gigabit Ethernet switches and 1000Base-ZX optical transceivers, with two fully redundant physical connections between the switches.

2 Based on an implementation using Cisco 15454 SONET multiplexers with dual rings for redundancy.

19

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

capacity in a SONET-based service delivered to one customer for use in meeting another customers needs. The Ethernet approach uses statistical multiplexing and highly granular bandwidth allocation, just like its cousins, ATM and Frame Relay. These attributes combine to give the service provider an extraordinary degree of flexibility to squeeze as much revenue as possible out the link capacity.

While ATM and Frame Relay can provide similar statistical multiplexing with granular allocation of bandwidth, they cannot compete with Ethernet on the basis of capital cost. The cost for lighting the same fiber span using ATM or Frame Relay (while providing the same level of fault resiliency) includes all of the costs for the SONET solution plus the costs for ATM or Frame Relay switches. Furthermore, neither of these solutions can provide the same bandwidth. As previously noted, the maximum link speed available to ATM or Frame Relay implemented over an OC-12 SONET ring is only 155 megabits per second.

In addition to lower capital costs, the Metro Ethernet Forum cites rapid provisioning as a key benefit of Ethernet in the metropolitan/regional space. “The present lack of customer-centric flexibility, as well as the coarseness of bandwidth granularity for [SONET- and ATM-based] legacy systems are considered major impediments to providing promising, revenue-generating services”[4]. Citing “service velocity” as a “key competitive differentiator”, the Metro Ethernet Forum promotes Ethernet’s ability to offer services with line speeds ranging from 10 megabits per second to 10 gigabits per second, and configurable bandwidth provisioning, provided quickly and on-demand.

Metro Ethernet has other advantages when applied to enterprise LAN extension applications. Because of the cost-effectiveness of the metro Ethernet solution (largely a product of lower equipment costs), it is possible for service providers to offer customers much greater bandwidth for the same money. This allows enterprise networks to be distributed over larger geographic regions without the “WAN penalty” – the traditional difference in bandwidth available between the LAN and the WAN, due largely to the cost of WAN bandwidth. Additionally, because the enterprise network has traditionally been implemented using Ethernet technology, the metro Ethernet solution avoids complex, costly, and difficult-to-manage internetworking solutions to adapt applications to

20

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

prevailing legacy WAN technologies. Ethernet end-to-end results in enterprise network extension that is far more seamless than traditional WAN solutions.

21

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Limitations of Current Metro Ethernet Technology

The Metro Ethernet Forum has identified the following limitations to the use of Ethernet in metropolitan and regional networks:

Slow recovery after link failures

Lack of remote fault management

Lack of in-service performance monitoring and OAM

Limited VLAN tag space

Inefficiencies of spanning tree relative to redundant link utilization in highly meshed topologies

No end-to-end service guarantees.

In the following sections, we will address each of these limitations in some detail, along with a description of some of the protocols and proposals that seek to overcome these limitations.

Slow Recovery From Link Failures

The Spanning Tree protocol (IEEE 802.1D) used in Ethernet networks detects link failures and makes topological adjustments needed to restore network service with a convergence time that is between 50 and 130 seconds. While adequate for some applications, this is a far cry from the 50 millisecond link failure recovery time of SONET’s automatic protection switching (APS). Multiple efforts are underway using different approaches to address this shortcoming. So-called “carrier grade” services are implemented with the fundamental assumption that service restoration occurs in less than 50 milliseconds. Therefore, many in the metro Ethernet camp feel that it is imperative that Ethernet networks be able to match SONET in this regard.

On the near horizon, the Rapid Reconfiguration protocol (IEEE 802.1w) represents an incremental improvement. Using this protocol, Ethernet networks with particular

22

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

topological characteristics can recover from link failures in approximately 1 second. The faster recovery provided by the 802.1w protocol greatly increases the number of applications that can be supported on metro Ethernet. Pre-standard support for the 802.1w protocol is appearing in new Ethernet switches from many vendors, allowing service providers to begin leveraging metro Ethernet to deliver services today. Ratification of the 802.1w specification is expected in the near future.

The Link Aggregation protocol (IEEE 802.3ad) can also be used to vastly improve the resiliency and recovery time of metro Ethernet networks. By employing parallel links between Ethernet switches, and utilizing diverse fiber paths, the IEEE 802.3ad protocol can provide load sharing between the parallel links when both links are operational. When a link is broken, failover to the remaining link occurs with a convergence time on the order of 500 milliseconds. See Figure 7.

MSAP MSAP Ethernet Ethernet 1000Base-X Switch Switch 802.3ad Link Aggregation MSAP Passive Fiber Pass Through
MSAP
MSAP
Ethernet
Ethernet
1000Base-X
Switch
Switch
802.3ad Link
Aggregation
MSAP
Passive Fiber
Pass Through
Figure 7: Link Aggregation between MSAPs

This is obviously an improvement over 802.1w Rapid Reconfiguration (though an order of magnitude worse than SONET), and is very simple to implement. However, many metro fiber networks have ring topologies that do not lend themselves to using Link Aggregation protocol as a means to improve resiliency.

Several vendors (e.g. Extreme, Riverstone) have implemented proprietary approaches to matching or at least approaching the 50 millisecond recovery time in metro Ethernet

23

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

networks. These approaches tend to borrow from the architecture of SONET, creating a dual ring topology that transports Ethernet frames. Extreme claims that their “Ethernet Automatic Protection Switching (EAPS)” dual-ring architecture has a recovery time no worse than 100 milliseconds. The IEEE Resilient Packet Ring working group is presently standardizing such ring-based approaches (as IEEE 802.17), with a goal of matching or even improving upon the SONET benchmark.

Proponents of network layer (IP) switching promote Ethernet-over-MPLS (EoMPLS) as the means by which metro Ethernet networks can achieve the resiliency of SONET protection switching, while at the same time addressing other shortcomings in the area of service-level guarantees. The MPLS functions that provide traffic engineering over an MPLS cloud can be used to provide Ethernet transport with guaranteed bandwidth and 50 millisecond recovery from link failures.

Lack of Remote Fault Isolation

The SONET architecture provides a very effective means of isolating faults to the particular SONET path, line, or section that is experiencing a fault, through the use of overhead bytes in the SONET frame, as well as the Remote Defect Indicator (RDI) and Loss of Signal (LOS) indication at each SONET interface. The 10 gigabit Ethernet standard includes, in the wide area network physical interface specification (the so- called WAN PHY for running over a SONET OC-192c carrier), the ability to map SONET fault isolation into meaningful concepts at the logical interface.

In general, however, Ethernet does not presently possess analogous functionality. In the long-haul applications for which SONET is often employed, remote fault isolation is imperative to network manageability. In metro applications, the degree to which the lack of these capabilities is a detriment to the manageability of Ethernet technology is debatable. Remote fault isolation is less critical in Ethernet in part because its architecture is far less complex. Large enterprise networks, based on Ethernet and spanning very large campuses, have been operated for many years without remote fault isolation. Few enterprise network managers would argue that the lack of remote fault isolation makes their networks more difficult to manage.

24

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Remote fault isolation is an area of active interest and research in the Metro Ethernet Forum and certain IEEE working groups.

Lack of In-Service Performance Monitoring and OAM

Customer services provisioned over SONET-based services such as DS1, DS3, and OC3c are terminated at a demarcation point (typically on the customer premise) using an intelligent network termination device. This device typically provides the means by which overhead bits in frames traveling on the circuit can be used to direct the termination device to loop back the circuit and report the bit error rate (BER). This capability allows the provider to monitor and test the loop extending to the customer premise, prior to dispatching a technician, at great cost savings to the provider.

Two alternative proposals to providing this capability are being considered by the IEEE 802.3ah Ethernet in the First Mile working group. One proposal suggests the use of the Ethernet preamble, and the other offers a frame-based approach.

While there is presently no standards-based approach to providing analogous functionality for metro Ethernet customer access loops, many vendors are developing proprietary approaches to solving this problem. For example, Cisco has an Optical Network Terminator device for use with their Cisco Catalyst 4000 series switches that provides remote OAM functionality for metro Ethernet networks.

Limited VLAN Tag Space

The IEEE 802.1Q standard defines a VLAN tag address space of only 4096 tags. This may be insufficient for a large service provider. Many equipment vendors are implementing so-called “Q-in-Q” approaches to stacking VLAN tags that, along with careful planning and partitioning of the network to allow some tag reuse, should allow networks to grow to reasonably large proportions.

While tag stacking approaches are proprietary, various MPLS techniques being considered by IETF working groups hold the promise of providing standards-based approaches to better scalability of metro Ethernet networks.

25

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Spanning Tree Inefficiencies on Highly Meshed Networks

The advent of standards-based virtual LAN support in the IEEE 802.1Q specification

was not accompanied by a change in the Spanning Tree protocol (IEEE 802.1D).

Standards-based Ethernets with multiple virtual LANs continued to use a single

spanning tree, shared by all virtual LANs in a common broadcast domain. Since there

can be only one loop-free path in a spanning tree, this limitation can result in inefficient

use of the network – redundant paths in a meshed network topology must remain

completely idle by design of the protocol. 3

Per-VLAN Spanning Tree (PVST) is an approach implemented by Cisco and other

vendors. With PVST, each VLAN has a distinct spanning tree. Per-VLAN Spanning Tree

allows for load balancing across VLAN trunks. Each spanning tree instance has its own

configuration messages and other overhead, which can be quite expensive (in terms of

CPU cycles) as the number of VLANs increase. By contrast, the single common

spanning tree of standards-based Ethernet alleviates concerns about protocol overhead

but does not allow VLAN load balancing.

The IEEE 802.1s Multiple Spanning Trees specification will standardize the use of

multiple spanning trees. The draft 802.1s specification combines the best aspects of

per-VLAN spanning tree and the common spanning tree of 802.1Q. In 802.1s, each

spanning tree creates a loop-free logical topology for a configured subset of the VLANs

in the network. This allows VLAN load sharing on highly meshed networks, while limiting

the number of spanning tree instances and associated overhead.

Lack of End-to-End Service Guarantees

Unlike ATM, Ethernet does not have inherent quality-of-service guarantees. This is not

to say that an Ethernet network cannot provide engineered quality assurance to specific

application subsets. Most Ethernet switches designed for metro Ethernet applications

3 An alternative approach that allows parallel redundant links to share the network load is the use of the IEEE 802.3ad Link Aggregation protocol. In this case, the parallel redundant links appear to the Spanning Tree protocol as a single logical link.

26

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

have the ability to classify and mark 802.1Q frames and/or IP packets for elevated priority, to police traffic classes at particular rates, and to provide multi-queue transmit disciplines such as Weighted Round Robin, and Strict Priority. By implementing appropriate trust boundaries, and using these mechanisms, it is possible to provide an appropriate service level for delay-sensitive applications (e.g. voice, video) in the presence of bursty, best-effort data applications.

It is not possible, using current Ethernet technology to, implement quality-of-service guarantees specifying acceptable packet loss, delay, and jitter parameters, with dynamic admission control and optimal path selection. One could argue that while ATM has the capability of signaling QoS requirements in the call setup, it has seen little use in part because of scalability, interoperability, manageability, and policy concerns in the service provider space. Most often, ATM QoS has been applied to permanent virtual circuits, where admission control and optimal path selection are manually determined.

Over the long term, proponents of metro Ethernet point to MPLS and its ability to provide QoS guarantees that are analogous to ATM, with dynamic signaling of QoS parameters and constraint-based routing as the solution to service guarantees for metro Ethernet. Using much of the existing hardware and software used to provide traffic classification and transmit scheduling, and by mapping prioritization between the 802.1Q priority bits and the MPLS EXP bits, metro Ethernet can easily be adapted to provide true end-to- end QoS.

Is Metro Ethernet Ready for Prime Time?

Given these limitations, service providers might be reluctant to deploy metro Ethernet- based services now. While these limitations and their possible solutions are important to understand, it is also important to note that metro Ethernet, even given these limitations, can support the vast majority of today’s applications.

Service providers might well choose to implement a simpler, less robust metro Ethernet network today, and begin generating revenues from the enormous array of applications that can tolerate near-carrier-class service at a substantial price advantage. Indeed, the

27

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

success of providers such as Yipes, Telseon, and Cogent in metro Ethernet services is a testament to the business case for such an approach.

Historically speaking, advances in Ethernet technology have come at a steady pace, and have been quickly adopted by the industry. As the works-in-progress of IEEE working groups and other industry forums come to fruition, the future of Ethernet technology looks very bright, indeed.

28

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Best Practices for Metro Ethernet Networks

Metro Ethernet networks have topological constraints, as well as management and security considerations that are unique to the service provider environment. Enterprise network managers have discovered, through many years of experience with Ethernet technologies, that these networks can provide extraordinary service levels, with very high performance, and excellent resiliency. But, in order to achieve these benefits, careful consideration must be given to the planning and implementation of any Ethernet network.

Given that many incumbent and new service providers have not previously deployed Ethernet-based technologies on any scale, this section focuses on best practices and implementation considerations for metro Ethernet networks. It highlights the pertinent techniques and technology decisions that can benefit from the lessons learned from large-scale Ethernet deployments in both the service provider and enterprise network environments.

Architecture: Link Layer or Network Layer Switching

In planning a metro Ethernet network, one of the foremost considerations is whether the majority of the switching nodes in the network will operate at the OSI data-link layer or at the OSI network layer. Link layer Ethernet switches (also known as Layer 2 or “pure” Ethernet switches) have the functionality needed to perform the role of access network aggregation points in Ethernet-in-the-First Mile deployments, as well as the interconnect capabilities needed in the MSAP. Ethernet switches from a variety of different vendors provide support for QoS mechanisms needed to support the vast majority of applications.

Network layer switches (traditionally known as “routers”) offer much greater functionality, but at a higher cost – typically two to four times the cost of link layer switches. Network layer switches from several vendors can support all of the IP and MPLS functionality needed to provide a robust, high-performance, and cost-effective solution for virtually

29

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

every customer application. Moreover, network layer switches can work over almost any combination of Ethernet and legacy WAN technologies (ATM, PPP/HDLC over SONET), as well as dark-fiber and optical (e.g. DWDM) networks.

Most metro Ethernet service provider networks will employ a combination of switches operating at both the data-link and network layers. In first-mile access and distribution networks, the use of link layer switches interconnected via 1000Base-X Ethernet is a cost-effective and manageable solution. In the core of the network, where access and distribution networks must meet Internet and application service providers, network layer switches provide the greatest flexibility, scalability, and manageability. Additionally, network layer switching solutions support transparent operation over nearly any combination of Ethernet and other link layer environments.

Spanning Tree Configuration

Careful implementation of the Spanning Tree Protocol (IEEE 802.1D) is essential to the success of any metro Ethernet. Even when the nodes of the metro Ethernet are network-layer packet switches (e.g. IP/MPLS switches), there are spanning tree implications whenever Ethernet connections are extended to customer networks. The Spanning Tree protocol is deceptively simple. The implications of the network topology and switch configuration are not always obvious, particularly in larger networks. The perils of giving inadequate consideration to the planning and implementation of Spanning Tree protocol in your network range from inefficient use of valuable (e.g. fiber line) assets, to incredibly disruptive and difficult-to-resolve anomalies known as “forwarding loops”.

The Spanning Tree root bridge election is of critical importance. In many cases, the customer’s local area network will participate, to some degree, in the spanning tree protocol within your domain. To ensure stability of the network service for each customer, it is imperative that the root bridge be completely under the control of the service provider. The root bridge should be at or near the topological center of the bridging domain, and should be a switch with adequate CPU resources to run multiple spanning tree instances.

30

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Service providers are strongly advised to establish internal practices and reviews that ensure proper control over the root bridge election via the following means:

Configuration of root bridge priority parameter values that ensure that the protocol will elect an administratively and topologically appropriate root bridge for each bridging domain.

Configuration of the common so-called “root guard” feature on all customer facing ports. This feature prevents a bridge in the customer’s network from becoming the root bridge in the service provider’s domain.

Many metro Ethernet networks will implement the Spanning Tree protocol using switches interconnected over physical ring topologies. Over this physical topology, Spanning Tree will create a logical hub-and-spoke topology, where one of the switches on the ring is the hub (the root bridge), and one link in the ring (most distant from the root bridge) will be blocked. In this case, best results are achieved when the hub is a bridge that interconnects many such rings, such as the switch shown in the MSAP in Figure 2. Furthermore, such a topology is an ideal candidate for deployment of the IEEE 802.1w Rapid Reconfiguration protocol, which will ensure restoration of the ring in approximately one second, should a fiber cut occur anywhere along the ring.

Forwarding Table Considerations

The forwarding table in a switch contains a list of MAC addresses and corresponding egress ports, typically on a per-VLAN basis. There are three basic considerations regarding the forwarding table:

1)

Overall capacity. Switches used in service provider networks must provide ample capacity to support the maximum number of end systems that might be connected to the provider’s network. Most carrier class switches support on the order of 65,000 or more entries in the forwarding table per VLAN.

2)

Per port capacity. No single port should be allowed to consume all of the available capacity of the forwarding table, since otherwise a denial of service attack is possible by flooding a large number of source addresses into the network from a single port.

31

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

3)

MAC address security. Switches used by service providers should provide the capability to accept and lock-down a limited number of source addresses from customer-facing ports. This can, in some cases, address the denial-of-service vulnerability mentioned in (2), above. This capability can also mitigate the effects of undiscovered topological loops (e.g. resulting from Spanning Tree protocol misconfiguration), by preventing addresses from being learned on any port other than the “correct” port.

Protocol Filtering

While metro Ethernet networks can support any higher-layer protocol that can be encapsulated in an Ethernet frame, service providers may wish to filter unwanted or unsupported protocols received from customer-facing ports. For example, residential access service providers may want to filter all Ethernet frame types other than the conventional encapsulation for IP and ARP. This would ensure that only the protocols needed for supported services are transported on the network and that customers cannot subject the network to protocol families (e.g. AppleTalk) that tend to needlessly consume network resources with overhead traffic, or that otherwise contribute to network instability.

Most Ethernet switches include protocol filtering support. Service providers should implement protocol filtering as appropriate to the services delivered.

Rate Limits on Broadcast and Multicast Frame Flooding

Broadcast and multicast frames are, by default, flooded by switches to all ports on the spanning tree except the port from which the frame was received. Switches deployed by service providers must include the ability to apply rate limits to broadcast and multicast frames. Broadcast and multicast rate controls can mitigate the effects of broadcast storms and undiscovered topological loops and thus allow the network to continue to deliver service even in the face of severe misconfiguration and/or misbehavior on the part of switches in the network.

32

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Service provider Ethernet switches must provide independent configurable rate limits for broadcast and multicast frame flooding. Ideally, the limits should be configurable on a per port basis. True broadcast frame traffic (addressed to the “all ones” destination address) should, under normal conditions, have a very low bit rate per port. Thus, it is desirable to set broadcast rate limits to be a very small portion of the available bandwidth on each port to ensure that broadcast traffic cannot consume valuable resources or destabilize the network. If controls are provided on a per-port basis, it is possible to provide an engineered solution for broadcast frames from edge to core.

IP Multicast Frame Flooding and Rate-Limiting

Since IP multicast (used for many forms of one-to-many multimedia content delivery) uses multicast Ethernet frames, it is not desirable to impose the same highly restrictive rate limiting on IP multicast frames as should be applied to other multicast and broadcast frames. Ideally, switches should provide independent rate-limiting and scope-limiting functionality for IP multicast. Rate-limiting for IP multicast frames could provide an upper bound for all IP multicast frames, or on a per-multicast-group basis, or both. Scope- limiting should provide a means to ensure that most IP multicast frames are flooded only to those ports with downstream IP multicast group receivers, rather than flooding throughout the broadcast domain.

In lieu of fully independent IP multicast rate/scope controls, independent rate controls for multicast frames (of all kinds) on a per port basis can suffice by allowing the multicast frame rate limit to be set to a higher limit than broadcast frames. When combined with protocol filtering (to filter non-IP multicast frames), this can achieve the same effect as an independent rate limit for IP multicast, at the expense of other non-IP protocols (which would in this case be summarily discarded by protocol filters).

Quality-of-Service Controls

In order to provide a reasonable foundation for providing differentiated services, appropriate for a mix of multimedia applications, switches used by service providers should have the following capabilities:

33

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Ability to classify traffic at ingress port based on Ethernet frame fields and network layer attributes. In particular, the classification should allow the classification of traffic by 802.1Q priority, source or destination MAC address, IP precedence and/or DSCP, IP source/destination address, IP protocol field, and transport-layer addresses (i.e. ports) for TCP and UDP.

Ability to set the 802.1Q priority and optionally (strongly recommended) the ability to set the IP DSCP field of outbound frames.

Ability to police at ingress to specified bit rates based on classification as described above. Policing should minimally provide granularity on the order of 100 kilobits per second, and burst sizes ranging from 32 kilobytes up to hundreds of megabytes.

At least two fully independent transmit queues per port. If only two queues are provided, the ability to select either strict priority or weighted round robin disciplines is strongly recommended. Additionally, the ability to direct outbound frames to the appropriate queue of the egress port based on classification as described above.

34

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Acronym Glossary

ARP – Address Resolution Protocol. A link-layer protocol used to discover the associations between Internet Protocol (IP) addresses and Media Access Control (MAC) addresses in an Ethernet network.

ATM – Asynchronous Transfer Mode. A cell-switching paradigm developed as part of the ITU Broadband Integrated Services Digital Network (B-ISDN) specification.

BER – Bit Error Rate

BPDU – Bridge Protocol Data Unit. Refers to an Ethernet frame containing a Spanning Tree protocol message.

DSCP – DiffServ Code Point. A IP packet header field defined to contain a quality-of- service indicator defined by the Differentiated Services (DiffServ) IETF working group.

DSL – Digital Subscriber Line. A technology used to provide digital services on the copper customer loop extending between a customer premise and a telephone company central office.

DWDM – Dense Wave Division Multiplexing.

EoMPLS – Ethernet over MPLS. Refers to the transparent transport of Ethernet frames over an MPLS switching cloud.

IEEE – Institute of Electrical and Electronic Engineers.

IETF – Internet Engineering Task Force.

IP – Internet Protocol. The OSI network layer protocol used on the Internet and in the vast majority of corporate intranets and extranets.

35

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

ISDN – Integrated Services Digital Network.

ITU – International Telecommunications Union, formerly CCITT.

LAN – Local Area Network.

LOS – Loss Of Signal. A SONET fault isolation indicator.

MAC – Media Access Control. Most often used in the context of MAC address, which refers to a link layer frame address (e.g. an Ethernet node’s hardware address).

MST – Multiple Spanning Trees. The approach to multi-VLAN spanning tree specified by the IEEE 802.1s working group, wherein multiple spanning trees are operated, each one providing a loop-free logical topology for a subset of the virtual LANs within the bridged network.

MPLS – Multi-Protocol Label Switching, also cited as Multi-Protocol Label Swapping. Provides label stack switching of IP packets in manner similar to that used in Frame Relay and ATM networks, but with the ability to work over virtually any link layer protocol (PPP/HDLC over SONET, Frame Relay, ATM, Ethernet, etc).

OAM – Operations, Administration, and Management. Sometimes specified OAMP, where the ‘P’ represents Provisioning.

OSI – Open Systems Interconnect. An idealized model for representing the peer and interface interactions between communications protocols, arranged in a stack. Specified by the International Standards Organization (ISO).

PHY – A physical electrical or physical optical network interface component.

PVST – Per-VLAN Spanning Tree. A proprietary approach to implementing the Spanning Tree Protocol (STP) in multi-VLAN networks, wherein each VLAN runs its own instance of STP.

36

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

QoS – Quality of Service. Refers to the mechanisms, specifications, and/or service-level agreements associated with providing end-to-end or node-to-node service guarantees or assurances on the basis of packet loss, delay, and/or jitter.

RDI – Remote Defect Indicator. A SONET fault isolation indicator.

SONET – Synchronous Optical Network. Specified as the Synchronous Digital Hierarchy (SDH) in the ITU Broadband Integrated Services Digital Network (B-ISDN) specification. It provides the basis for synchronous transport services in traditional telco carrier networks and is the underlying transport for both ATM and Frame Relay services, as well as DS1, DS3, OC3c, and higher speed point-to-point services.

STP – Spanning Tree Protocol. A data-link layer protocol for estabilishing a loop-free logical topology over an arbitrary interconnecting of data-link layer bridges.

TDM – Time Division Multiplexing.

TCP – Transport Control Protocol. A transport layer protocol providing reliable bulk data transfer over the Internet Protocol (IP).

UDP – User Datagram Protocol. A transport layer protocol providing message passing (datagram) capability over the Internet Protocol (IP).

VLAN – Virtual Local Area Network. A means of supporting multiple distinct bridging domains on a common Ethernet switching network. While developed for local area networks, the VLAN concept is used in metro Ethernet networks to provision distinct services, providing a virtual private network for each customer.

WAN – Wide Area Network.

37

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Acknowledgements

The authors would like to express their gratitude to Cisco Systems, Inc, and, in particular, Richard Shumaker and Scott Shepard, for their generous contribution of time, effort, and content to this report.

38

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

References

[1]

Black, Ulyess and Waters, Sharleen. Sonet and T1: Architectures for Digital Transport Networks. Prentice Hall, 1997.

[2]

Clark, Kennedy and Hamilton, Kevin. Cisco LAN Switching. Cisco Press, 2001

[3]

Goralski, Walter J. Introduction to ATM Networking. McGraw-Hill, 1995.

[4]

Metro Ethernet Forum. Metro Ethernet Networks: A Technical Overview. 2002

[5]

Norton, William B. Interconnection Strategies for ISPs. Equinex, Inc, 1999.

[6]

Perlman, Radia. Interconnections, 2nd Edition. Addison-Wesley, 2000.

[7]

Spurgeon, Charles. Ethernet: The Definitive Guide. O’Reilly, 2000.

39

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Quick Reference to Frequently Asked Questions

1)

Why is it difficult for an established telecommunications company to make this investment? (Volume 1, Volume 5)

2)

There is already too much fiber in the ground. Why not use what’s there? (Volume 1, Volume 2, Volume 6)

3)

The principal design criterion driving the development of this infrastructure is that every user has the potential to be a “producer” in the network economy. Is this the same as “broadband”, as it is currently hyped in the industry? (Volume 1)

4)

Can we quantify the potential jobs that will be created if a region invests in building advanced telecommunications infrastructure? (Volume 1)

5)

What should be the Tobacco Commission’s role in the deployment of first mile technologies? (Volume 1, Volume 3, Volume 5, Volume 7, Volume 8)

6)

How can localities ensure that they get early access to the network? (Volume 1, Volume 5, Volume 8)

7)

What kind of success have other regions had with the development of network infrastructure for economic development? (Volume 1)

8)

What regulatory factors should be considered when investing in wireless technologies? (Volume 1, Volume 7)

9)

Why do we need to connect to network points outside of the tobacco regions? (Volume 2)

10)

Once the network is in place, what do we do with it? (Volume 2, Volume 8)

11)

Since the business model for inter-regional and inter-county infrastructure did not include the use of conduit facilitating blown fiber strands, what are the circumstances in which this technology is appropriate and financially feasible? (Volume 3, Volume 7)

12)

How do existing community networks fit into the overall design? (Volume 3, Volume 5, Volume 6)

13)

What are some examples for deployment in the first/last mile? (Volume 3, Volume 7)

14)

What type of fiber is recommended? (Volume 3)

15)

What would a network design for my county look like? (Volume 3)

40

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

16)

How much would all this cost? (Volume 3, Volume 5)

17)

What is the appropriate organization model for managing and sustaining the Tobacco Commission’s investment in critical technology infrastructure? (Volume

5)

18)

Tobacco region communities are underserved because the private sector does not see a profitable business case. What makes this feasible from a business perspective? (Volume 5)

19)

If the traditional investment model for developing critical technology infrastructure has failed, what is the alternative? (Volume 5)

20)

How much would it cost for consumers in the region to use the network? (Volume

5)

21)

What technologies enable use of the fiber? (Volume 6)

22)

How does the choice of technology to light the fiber impact the cost? (Volume 6)

23)

How do wireless technologies fit into this framework? (Volume 7)

24)

What is meant by the term “open access”? (Volume 8)

25)

What is the difference between the broadband hype and the “next generation” networks? (Volume 8)

26)

What are some next generation Internet (NGI) applications? (Volume 8)

41