Documente Academic
Documente Profesional
Documente Cultură
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
The Access Layer is becoming more than just a port aggregator Edge of the growing Layer 2 topology
Scaling of STP Edge Ports Virtual embedded switches vPC and loop free designs Layer 2 Multi-Pathing (future)
DCB and Multi-Hop FCoE Support Enhanced Multi-hop FCoE with E-NPV
blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8
blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8
blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8 blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8
blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8 blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8
Core/Aggregation Layer
Be able to design single-hop Unified Networks available today which meet the demands of both SAN and LAN networks
Start the conversation between Network and Storage teams regarding consolidation and FCoE beyond the access layer Understand the Operations and Management aspects of a Unified Network
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
FC Fabric
FC-MAP is the upper 24 bits of the Enodes FCoE MAC FCID is the lower 24 bits of the Enodes MAC
FCoE forwarding decisions still made based on FSPFand the FCID within the Enode MAC Fibre Channel FCID Addressing
FC-MAP (0E-FC-xx)
FC-ID 10.00.01
FC-MAC Address
Data Center Partner Webinar
FC-MAP (0E-FC-xx)
FC-ID 10.00.01
8
Show fcoe
N5K2-60# show fcoe FCF details for interface san-port-channel 200 FCF-MAC is 00:0d:ec:a4:3b:87 FC-MAP is 0e:fc:00 FCF Priority is 128 FKA Advertisement period for FCF is 8 seconds N5K2-60# show fcoe database ------------------------------------------------------------------------------INTERFACE FCID PORT NAME MAC ADDRESS ------------------------------------------------------------------------------vfc1 0x240101 21:00:00:c0:dd:0a:b8:df 00:c0:dd:0a:b8:df vfc201 0x240100 21:00:00:c0:dd:12:04:f2 00:c0:dd:12:04:f2
Operating System
Link
PCIe
10GbE
Ethernet
Ethernet Drivers
10
Eth port
Eth port
Eth port
Eth port
Eth port
Eth port
Eth port
11
VE_Port
VF_Port
VNP_Port
E_NPV Switch
VF_Port
VN_Port
VF_Port
VN_Port Node
**Available NOW
12
Unified Dedicated Wire -- a single DCB Ethernet link capable of carrying all traffic types but actively dedicated to a single traffic type for traffic engineering purposes
Unified Fabric An Ethernet Network made up of Unified Wires everywhere: all protocols network and storage transverse all links simultaneously
13
Fabric A
Fabric B
Core
L3 L2
Aggregation
Shared Access
Unified Wire
14
Core
Aggregation
Access
15
Unified Technology
LAN and SAN networks share the same Unified I/O building blocks: switches and cabling Maintains operations, management and troubleshooting Takes advantage of the Ethernet Roadmap (10G40G100G)
Native Ethernet LAN Fibre Channel over Ethernet SAN
Fabric A Fabric B
Core
L3 L2
Aggregation
Core
Access
Edge
CNA
Multi-pathing
16
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
17
CEE (Converged Enhanced Ethernet) is an informal group of companies that submitted initial inputs to the DCB WGs.
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
18
19
Ethernet Link
Receive Buffers
One Two
PAUSE
B2B Credits
Data Center Partner Webinar
Packet
Eight
2010 Cisco and/or its affiliates. All rights reserved.
Best Practice - use the default COS value of 3 for FCoE/no-drop traffic Can be changed through QOS class-map configuration
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
N5K# show class-map Type qos class-maps =================== class-map type qos match-any class-fcoe match cos 3 class-map type qos match-any class-default match any
21
Create classification rules first by defining and applying policy-map type qos
N5010-2(config)# class-map type qos class-lossless N5010-2(config-cmap-qos)# match cos 4 N5010-2(config-cmap-qos)# policy-map type qos policy-lossless N5010-2(config-pmap-qos)# class type qos class-lossless N5010-2(config-pmap-c-qos)# set qos-group 4 N5010-2(config-pmap-uf)# system qos N5010-2(config-sys-qos)# service-policy type qos input policy-lossless
22
(8)
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
8
16 32 64
00001000
00010000 00100000 01000000
3
4 5 6
Shows ports where PFC is configured, the COS value associated with PFC as well as the PAUSE packets received and sent on that port
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
128
10000000
7
23
Offered Traffic
3G/s
3G/s
2G/s
3G/s
3G/s 3G/s 3G/s
3G/s
3G/s
4G/s
6G/s
3G/s
5G/s
t1
Data Center Partner Webinar
t2
t3
2010 Cisco and/or its affiliates. All rights reserved.
t1
t2
t3 24
Traditional Server
A typical server has equal BW per traffic type Best Practice : FCoE and Ethernet each receive 50% Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs)
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
25
Create classification rules first by defining and applying policy-map type qos
Define and apply policy-map type queuing to configure strict priority and bandwidth sharing
N5010-2(config)# class-map type queuing class-voice N5010-2(config-cmap-que)# match qos-group 2 N5010-2(config-cmap-que)# class-map type queuing class-high N5010-2(config-cmap-que)# match qos-group 3 N5010-2(config-cmap-que)# class-map type queuing class-low N5010-2(config-cmap-que)# match qos-group 4 N5010-2(config-cmap-que)# exit N5010-2(config)# policy-map type queuing policy-BW N5010-2(config-pmap-que)# class type queuing class-voice N5010-2(config-pmap-c-que)# priority N5010-2(config-pmap-c-que)# class type queuing class-high N5010-2(config-pmap-c-que)# bandwidth percent 50 N5010-2(config-pmap-c-que)# class type queuing class-low N5010-2(config-pmap-c-que)# bandwidth percent 20 N5010-2(config-pmap-c-que)# class type queuing class-fcoe N5010-2(config-pmap-c-que)# bandwidth percent 30 N5010-2(config-pmap-c-que)# class type queuing class-default N5010-2(config-pmap-c-que)# bandwidth percent 0 N5010-2(config-pmap-c-que)# system qos N5010-2(config-sys-qos)# service-policy type queuing output policy-BW N5010-2(config-sys-qos)#
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
26
27
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
28
Understanding FCoE
29
FCoE
Mapping of FC Frames over Ethernet
Ethernet Fibre Channel Traffic
Byte 0
Ethernet Header FCoE Header FC Header
Completely based on the FC model Same host-to-switch and switch-to-switch behavior as FC WWNs, FC-IDs, hard/soft zoning, DNS, RSCN
Cisco HP Intel
EMC2
Microsoft
EMULEX NetApp
VMWARE
30
FC Payload
EOF
Redhat
FCoE
Mapping of FC Frames over Ethernet
Ethernet Fibre Channel Traffic
Byte 0
Ethernet Header FCoE Header FC Header
Roadmap of Ethernet
Economy of Scale
FCoE is Ethernet
Byte 2229
CRC FCS
FC Payload
EOF
31
FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FCoE Logical End Point Ethernet Media Access Control Ethernet Physical Layer
FC-1 Encoding
FC-0 Physical Interface
32
FCoE itself
Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic
FIP frames use a different Ethertype from FCoE frames making FIP-Snooping by DCB capable Ethernet bridges
34
VLAN Discovery
FCF Discovery
FCF Discovery
FLOGI/F DISC
FLOGI/FDIS C Accept
FC Command
FC Command Responses
FCoE Protocol
35
The FCF-MAC address is configured on the Nexus 5000 by default once feature fcoe has been configured This is the MAC address returned in step 2 of the FIP exchange
36
37
38
FC-MAC Address
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
FC-MAP (0E-FC-00)
39
FCoE Forwarding
FCF Intermediate switches in the Ethernet cloud
All are Fibre Channel Aware
FC Storage
FCID 7.1.1 FC link FC Fabric VE_port Ethernet Fabric VE_port VF_port Ethernet Fabric VN_port
FC Domain 7
FC Domain 3 MAC A
FC Domain 1 MAC B
FC Frame
FC Frame
Dest. = MAC B Srce. = MAC A D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
Dest. = MAC C Srce. = MAC B D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
FCoE Frame
40
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
41
Fibre-Channel is deterministic.
Flow control is source-based (B2B credits) Services are fabric integrated (no loop concept)
42
? ?
? ? ?
Switch
Switch
Switch
?
? ?
? ? ?
Fabric topology and traffic flows are highly flexible
? ?
?
Client/Server Relationships are not predefined
43
Aggregation
Access
Services deployed in the L2/L3 boundary of the network (loadbalancing, firewall, NAM, etc)
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
STP
44
T1
T2
FSPF
Zone
Switch
Zone
Switch
DNS RSCN
DNS
FSPF Zone
RSCN
Switch RSCN
I5
I4
I2
I3
FC
Core
Core
46
Core
Core
47
?
Switch
?
?
Switch
?
?
? ?
Switch
High Availability redundant network topology (mesh/full mesh) and physically separate redundant fabrics
Bandwidth FC fan-in and oversubscription ratios and Ethernet oversubscription Security FC controls (zoning, port security, ) and IP controls (CISF, ACL, ) Manageability and Visibility Hop by hop visibility for FC and the cloud for Ethernet
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
T0
DNS FSPF
T1
T2
FSPF
Switch
Zone RSCN
Switch
DNS
FSPF Zone
Zone
DNS RSCN
Switch RSCN
I5
I0
I1 I2
I3
I4
48
L3 L2
STP
Aggregation
Core
Virtual PortChannel (VPC)
Core
Access
STP Virtual PortChannel (VPC)
Fold Here
49
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
50
Target
FC
DCBX is used to negotiate the enhanced Ethernet capabilities FIP is use to negotiate the FCoE capabilities as well as the host login process FCoE runs from host to access switch FCF native Ethernet and native FC break off at the access layer
ENode
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
Unified Wire
CNA
51
FC
LAN Fabric
Fabric A
Fabric B
FCoE FC
52
10GbE Ethernet
Link
PCIe
Ethernet Drivers
Operating System
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
53
Utilized Cisco, Intel, Nuova Data Center Bridging Exchange protocol (CIN-DCBX)
Only supports direct attachment of an VN_Port to an VF_Port over the unified wire Generation 2 CNA Utilizes Converged Enhanced Ethernet Data Center Bridging Exchange protocol (CEE-DCBX) Utilizes FCoE Initialization Protocol (FIP) as defined by the T.11 FC-BB-5 specification Supports both direct and multi-hop attachment (through a Nexus 4000 FIP Snooping Bridge)
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
CEE-DCBX
VF VN
CIN-DCBX
Generation 2 CNA
Generation 1 CNA
54
LAN Fabric
Fabric A
Fabric B
FCoE VLANs must not be configured on Ethernet links that are not designate for FCoE
VLAN 10,20
Maintains isolated edge switches for SAN A and B and separate LAN switches for NIC 1 and NIC 2 (standard NIC teaming)
! VLAN 20 is dedicated for VSAN 2 FCoE traffic (config)# vlan 20 (config-vlan)# fcoe vsan 2
55
LAN Fabric
Fabric A
Fabric B
VSAN 2
VSAN 3
VLAN 10,20
! VLAN 20 is dedicated for VSAN 2 FCoE traffic (config)# vlan 20 (config-vlan)# fcoe vsan 2
56
LAN Fabric
Fabric A
Fabric B
spanning-tree mst configuration name FCoE-Fabric revision 5 instance 5 vlan 1-19,40-3967,4048-4093 instance 10 vlan 20-29 instance 15 vlan 30-39
57
Nexus utilizes Virtual Port Channel (vPC) to enable MCEC either between switches or to 802.3ad attached servers
MCEC provides network based load sharing and redundancy without introducing layer 2 loops in the topology MCEC results in diverging LAN and SAN high availability topologies
FC maintains separate SAN A and SAN B topologies
LAN utilizes a single logical topology
Nexus 5000 FCF-A
vPC Peers
MCEC
58
With the NX-OS 4.1(3) releases a vfc interface can only be associated with a vPC which has a single [one (1)] CNA port attached to each edge switch
While the port-channel is the same on N5K-1 and N5K-2, the FCoE VLANs are different vPC configuration works with Gen-2 FIP enabled CNAs ONLY FCoE VLANs are not carried on the vPC peer-link FCoE and FIP ethertypes are not forwarded over the vPC peer link
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
VLAN 10,30
LAN Fabric
Fabric A
Fabric B
VLAN 10,20,30
VLAN 10,20
VLAN 10,30
If SAN design evolves to a shared physical with only VSAN isolation for SAN A and B this could change (currently this appears to be a big if) ISLs between the Nexus 5000 access switches breaks SAN HA requirements
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
Nexus 2232
FEX-2232
Management and configuration handled by the Nexus 5000 Support for Converged Enhanced Ethernet including PFC Part of the Cisco Nexus 2000 Fabric Extender family
61
Nexus 5000 FCF-A Fabric Links Option 1: Single Homed Port Channel Nexus 2232 10GE FEX
SAN A
SAN B
Nexus 7000
Nexus 2232 can only currently be connected to the Nexus 5000 when configured to support FCoE attached servers
Nexus 7000 will support Nexus 2000 in Ethernet only mode in CY2010 (support for FCoE on FEX targeted for CY2011 on next generation N7K line cards)
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
63
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
64
allows multiple applications to share the same Fiber Channel adapter port usage applies to applications such as VMWare, MS Virtual Server and Citrix
Application Server FC NPIV Core Switch
Email I/O N_Port_ID 1 Web I/O N_Port_ID 2 File Services I/O N_Port_ID 3
F_Port
F_Port
Web
File Services
Data Center Partner Webinar
N_Port
65
Nexus 5000, MDS 91xx, MDS blade switches, UCS Fabric Interconnect
F-Port
Eth1/1
NP-Port
F-Port
Eth1/2
F_Port
Eth1/3
N-Port
Data Center Partner Webinar
66
Where does Unified Wire make sense over Unified Dedicated Wire? Unified Wire provides for sharing of a single link for both FC and Ethernet traffic
FCF
67
In Multi-hop FCoE the role of a transit Ethernet bridge needs to be DCB Capable evaluated Ethernet
Avoid Domain ID exhaustion Ease management
Switch
VF
FCF
FCF
VF VN
FIP Snooping is a minimum requirement suggested in FC-BB-5 Fibre Channel over Ethernet NPV (FCoE-NPV) is a new capability intended to solve a number of design and management challenges
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
VN
68
SAN
FCF
FCF MAC 0E.FC.00.DD.EE.FF
Why FIP-Snooping?
Security - Protection from MAC Address spoofing of FCoE end devices (ENode) Fibre Channel links are Point-to-Point
Ethernet bridges can utilize ACLs to provide the equivalent path control (equivalent of point-topoint)
FIP Snooping
ENode
69
69
On the data plane (FCoE ethertype), an FCoE NPV bridge offers more ways to engineer traffic between CNA-facing ports and FCF-facing ports
An FCoE-NPV bridge knows nothing about Fibre Channel, and cant parse packets with FCoE ethertype
70
FCoE-NPV load balance logins from the CNAs evenly across the available FCF uplink ports
FCoE-NPV will take VSAN into account when mapping or pinning logins from a CNA to an FCF uplink
Operations and management process are in line with todays SAN-Admin practices Similar to NPV in a native Fibre Channel network
71
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
71
Target
FABRIC A
FC
FCF
VF
VNP
FCoE_NPV bridge
VF
FCoE_NPV does not consume a domain ID
VN
72
Link Ethernet
PCIe
10GbE
73
VE
74
SAN A
SAN B
FCoE-NPV
FCoE-NPV
75
SAN A
SAN B
FCF
FIP and FcoEframes load shared over MCEC on a per flow basis NO SAN A and SAN B isolation
DCB Enabled
76
Agenda
Why are we here? Background Information FCoE Building Blocks and Terminology DCB Standard FCoE Protocol Information Design Requirements Classical Ethernet + Classical Fibre Channel = ?? Single Hop Designs Multi-Hop Designs FCoE Deployment Considerations Questions
77
LAN and SAN HA models are very different (and not fully compatible) FC and FCoE are prone to HOLB in the network and therefore we are limited in the physical topologies we can build
e.g. 10 x 10G uplinks to LAN aggregation will require 10 x 10G links to a next hop SAN core (with targets attached) No savings, actually spending more to achieve this direct uplinks to SAN core
Targets are attached to the SAN core (the LAN aggregation and SAN core have different topology functions) Where is it more beneficial to deploy two cores SAN and LAN over a unified core topology
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
78
VE Ports
FSPF forwarding for FCoE traffic is end-to-end Hosts will log into the FCF which they are attached to (access FCF) Storage devices will log into the FCF at the core/storage edge Maintains HA requirements from both LAN and SAN perspective
2010 Cisco and/or its affiliates. All rights reserved.
79
FCF
FCoE-NPV
FCF
80
SAN B
81
Larger environments will need dedicated FCoE SAN devices providing target ports
Use connections to a SAN Use a storage edge of other FCoE/DCB capable devices
Data Center Partner Webinar 2010 Cisco and/or its affiliates. All rights reserved.
82
VLAN 10,20
83
VE
VF
VE
VNP
VE
Servers
84
So Remember
All Unified options are important and have different places within the Data Center Network FCoE offers a more flexible and cheaper deployment option over Fibre Channel FCoE IS Fibre Channel Multi-hop FCoE extends the FCoE fabric beyond the access Cisco offers end-to-end FCoE solution with Nexus platform
85
86
87