FCBasics

Fibre Channel Basic
pdobrins@brocade.com +7-985-922-61-33
Fibre Channel Combining SCSI with Networking
19 2008
Brocade
2008 Brocade Communications Systems , Inc. All Rights Reser ved.
Think of Fibre Channel As Super SCSI

Fibre Channel is Fibre Channel is like Super SCSI like Super SCSI
Run faster Better cables Connect more Lower overhead Go longer distance
SCSI
FC FRAME
E P
Fibre Channel Highway

(a new low-level data mover)
Powerful SCSI SCSI FC FRAME FC FRAME management
19 2008
Brocade
Fibre Channel Technical Capabilities

No emitted radio frequency signals (RFI), thus immune to induced electromagnetic signals Serial data transmission at 1.0625, 2.125, and 4.25Gbit with a transfer speed of 0.94 nanoseconds per bit (very low latency) Using 8-bit/10-bit encoding/decoding to translate 8-bit information to 10-bit format for serial transmission (clock signal embedded) Up to 16 million node connectivity (Switched Fabric mode) Variable frame size with a maximum user payload of 2112 bytes. Supports block transfers up to 128MB in size. Superior error correction rate (bit error rate = 1x10-12) 1 bit error each 16 min at 1gbit Supports full duplex transmission(simultaneous send and receive)
19 2008
Brocade
Fibre Channel As an Open Standard

Fibre Channel development began in 1988; the NCITS T11: I/O interface (X3.230-1994) standard was completed in 1994. Reference URL: http://www.t11.org Endorsed by many other standard parties and vendors
FC-SW 1. FC-SB FC-PH (Single Byte Mapping Protocol) 2) 2. FC-SB-2 (Single Byte Protocol Mapping
3. FC-LE (Link Encapsulation) 4. FC-PH (Physical and Signaling) 5. FC-PH (Physical and Signaling) Amendment 1 6. FC-PH (Physical and Signaling) Amendment 2 7. FC-PH-2 (Physical and Signaling 2) 8. FC-PH-3 (Physical and Signaling 3) 9. FC-FG (Fabric Generi c Requirements) 10. FC-GS (Generi c Services) 11. FC-GS-3 (Generic Services 3) 12. FC-SW (Switch Fabric) 13. FC-SW-2 (Switch Fabric 2) 14. FC-AL (Arbitration Loop) 15. FC-AL-2 (Arbitration Loop 2) 16. FC-BB (Backbone) 17. FC-FP (Mapping to HIPPI-FC) 18. HIPPI-FC (FC-PH Encapsulation)
SN IA
Business Solutions
Fi e br
FC-GS
FC-FLA
IA
Open, MultiVendor SANs
li Al an
FC
FC-CT
ce
FC-AL
Product Interoperability
FC-FG
FC-LS
Industry Standards
ANSI
19 2008 Brocade
DMTF IETF
5
Fibre Channel Hardware

Fiber Optic Cables
Glass Core Glass Cladding Coating
Glass Core Glass Cladding Coating
Multimode Fiber
19 2008 Brocade
Single-mode Fiber
Fibre Channel Basics

Wavelengths are expressed in nanometers The speed of light in fibre is 2/3 speed of light in vacuum
Light travels at ~5nsec per meter in glass
Multimode fibre carries numerous modes, or frequencies, and carries short-wave laser light. Single-mode fibre has a smaller core that allows only one mode of light and carries long-wave laser light. Signal loss is caused by dispersion and attenuation. Specific regions in the optical spectrum, windows, have low optical attenuation. 850nm, 1310nm, 1550nm, and 1625nm. Optical power budgets, or link loss budgets, measured in decibels (dBs), are used to manage optical signal loss.
19 2008 Brocade
Fibre Channel Hardware - Connectors
LC Optical Connector
- Standard on 2 Gb/s Switches - Becoming More Popular - Bonded or Unbonded - Usually Purchased as Duplex Cab le Shown as Bonded Duplex
SC Optical Connector
- Standard on 1 Gb/s Switches - Most Widely Used Connector - Bonded or Unbonded - Single or Duplex Cab le Shown as Bonded Duplex
HSSDC Copper Connector

19 2008
Smaller than DB-9 Easier to insert/remove

Brocade
HSSDC2 Copper Connector

- Fits in SFP Media Slots
- Smaller than HSSDC

GBICs, SFPs, and HBAs
Optical LC SFP Optical SC GBIC
HSSDC GBIC
HSSDC2 SFP
Fibre HBA
19 2008
Brocade

Optical Media Comparison Chart
Transceiver Type Form Speed, Factor Gbps Multi-Mode Media Maximum Distance 62.5um/200MHz (OM1) SFP SFP/SFP + SW SFP/SFP + SFP+ XFP SFP LW SFP SFP+ XFP
19 2008
Maximum Distance 9um N/A N/A N/A N/A N/A 30km 50, 80,100km 25km 40km
10
50um/500MHz (OM2) 500m 300m 150m 50m 82m N/A N/A N/A N/A
50um/2000M Hz (OM3) 860m 500m 380m 150m 300m N/A N/A N/A N/A
1 2 4 8 10 2 4 8 10
300m 150m 70m 21m 33m N/A N/A N/A N/A
Brocade
Brocade SilkWorm Hardware

Gigabit Interface Connector (GBIC) Options
SFP LC (2/4/8 Gb/s Products)

Short Wavelength (SWL) Optic (780 nm 850 nm laser, 50/62.5 m Multimode) Long Wavelength (LWL) Optic (1310 nm laser, 9 m Single-mode) Extended Long Wavelength (ELWL) Optic (1550 nm laser, 9 m Single-Mode) Copper (HSSDC2 Female Connector 2Gb/s Only) All Qualified SFPs are Intelligent
SC GBICs (1 Gb/s Products)

19 2008
Short Wavelength (SWL) Optic (780 nm 850 nm laser, 50/62.5 m Multimode) Long Wavelength (LWL) Optic (1310 nm laser, 9 m Single-mode) Extended Long Wavelength (ELWL) Optic (1550 nm laser, 9 m Single-Mode) Passive Copper Active Copper
Brocade
11
Fibre Channel
Fibre Channel Protocol Layers
Audio/Video Common transport for upper-layer protocols FC-4 Channels Networks FC Services
Fast File Transfers
Streams
IPI
SCSI
HIPPI
SBCCS
802.2
IP
BB
FC - CT
FC-3
Common Services
FC-2
Framing Protocol / Flow Control / COS
FC-1
Encode / Decode / PSM & LPSM
FC - PH
FC-0
133 Mbaud
266 Mbaud
531 Mbaud
1,063 Mbaud
2,125 Mbaud
4,250 Mbaud
8,500 Mbaud
10.63 Gbaud
Media:
Optical Laser, LED Copper Coax, Twisted Pair
19 2008
Brocade
12
Fibre Channel Layered Architecture

OSI Reference Model
Application Presentation Session Transport Network Data Link Physical
FC-3: Common Services FC-2: Data Delivery FC-1: Byte Encoding FC-0: Physical Interface
Fibre Channel Data & Applications

FC-4: Upper-Layer Protocols Mapping
With fewer processor interruptions, application can run faster.

19 2008 Brocade
13
Hardware
Flexible Deployment Topologies

Point-to-Point Arbitrated Loop Switched Fabric
Only 2 Devices (Direct Connect)
Up to 126 Devices (Fibre Channel Hubs)
Up to 16 Million Devices (Fibre Channel Switches)

14
19 2008
Brocade
Fibre Channel
Arbitrated Loop
Fibre Channel Arbitrated Loop - the transmit of each port is connected to the receive of the next port Reduced cost path into FC SCSI Replacement May use FC Hub technology Easy for vendors to develop Difficult for customers to deploy Limited possible ports (126) plus the Loop Master (FL_Port) Lower overall throughput 100/200MB maximum bandwidth Limited any to any connectivity - ports on the loop have to arbitrate for control of the loop in order to be able to communicate with a another port on the loop. While this communication is happening all other ports are waiting to get their turn. Primitive flow control causing QOS issues FC-AL was developed to fill the gap between the limited capabilities of point-topoint and the relatively expensive switched fabric. Since only two devices can communicate at a time, the loop is a blocking topology, which means the bandwidth is shared between all devices on the loop.
19 2008 Brocade
15
Fibre Channel
Switched Fabric
Large connectivity on nonshared media, which allows concurrent communicating pairs Highest performance level High scalability Good fault isolation Embedded management and services
19 2008
Brocade
16
FC Protocol Levels
ULP
EXCHA NGE (Transaction) EXCHA NGE (Transaction) EXCHA NGE (Transaction)
1st Information Unit FC-4 1st Sequence
Informat ion Unit
Last Information Unit
Sequence
Last Sequence
FC-3
Frame 1
Frame 2
Frame
Data Data Word Word
Frame n-1
Frame n
FC-2
SO F O rde red Se t
Data Word
Data Word
EO F O rde red Se t
8B/10B
8B/10B Characte r 8B/10B Characte r
8B/10B Characte r
8B/10B Characte r
8B/10B Characte r
FC-1
Characte r
FC-0
bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit bit k l m n o p q r s t u v w x y z a b c d e f g
19 2008
Brocade
17
SCSI-FCP Write Command
Initiator Node
Target Node SI
Frame Frame
IU 1 Command Command Chunk
SR Sequence 1 SI
Request Write to Send Chunk Request
Sequence 2 SR
Frames
Exchange
IU 2 Data Data Chunks
SI SR
Frame
Sequence 3
Sequence 4
SR
SI
IU 3 Status Status Chunk
19 2008
Brocade
18
SCSI-FCP Write Command cont.

Initiator
write req ue st [seq_cn t 0 ] S_ID , D_ID , L UN, LBA, Le ng th
Target Sequence seq_id
nt 0] [seq _c r read y * x _fe th ed Le ng co rre ct , S_ID ,
* De pends
Exchange ox_id
d ata 1 [se q_ cn t 0
on free buffe rs (i.e. 10MB should be writte n but only 2MB free buffe rs avail. 2MB is se nt back and afte r 2MB the nex t x _fe r ready is sent)
d ata n [se q_ cn t n -1
eq good [s s ta tus
_cn t 0]
Have to be last frame!!!
19 2008
Brocade
19
SCSI-FCP Read Command
Initiator Node SI
Frame Frames
Target Node SR Sequence 1
Command IU 1 Command Chunk Exchange

Sequence 2
SR
SI
IU 2 Data Data Chunks Status IU 3 Chunk Status
Sequence 3
SR
Frame
SI
19 2008
Brocade
20
FCP frame control

Data Droop
19 2008
Brocade
21
FCP frame control (i.e. SCSI-FCP Write Command)

Initiator
write req (BB -1 )
r_rd y ) ( BB+1
Switch
write req (BB -1 )
) ( BB+1 r_rd y x _fe r x _fe r BB-1) rd y ( BB-1) rd y (
Target
If
BB=0 (i.e. lost r_rdy frames) the link will be reseted by sending LinkCreditReset (LR) and LinkCreditResetResponse (LRR).
sequence
r_rdy (BB +1)
sequence
r_rdy (BB +1) da ta 1 (BB 1)

r_rd y ) ( BB+1
da ta 1( BB - 1)
) ( BB+1 r_rd y
da ta n (BB 1)
r_rd y ) ( BB+1
sequence
da ta n (BB 1)
) ( BB+1 r_rd y -1) od ( B B tus go s ta
BB- 1) good ( s ta tus
r_rdy (BB +1)
sequence
r_rdy (BB +1)
19 2008
Brocade
22
Fibre Channel
Frame Format
FRAME
C E R O C F 4 4
S O F 4
HEADER
PAYLOAD
24
Up to 2112
2148 Bytes
Word H E A D E R Payload
19 2008
Bits 31-24 R_CTL CS_CTL TYPE SEQ_ID
Bits 23-16
Bits 15-8 D_ID S_ID F_CTL
Bits 7-0
0 1 2 3 4 5 6-n
DF_CTL OX_ID Parameters Payload
SEQ_CNT RX_ID
Brocade
23
Fibre Channel
Classes of Service There are 6 classes of service
Class 1 Dedicated connection port to port with ack. It takes all the bandwidth until one port does a port logoff Class 2 It is is connectionless with ack: one <-> many. No bandwidth is allocated or guaranteed. IP uses this class. > Supported by Brocade Class 3 It is connectionless with no ack: buffer to buffer flow control. Errors are handled at higher level. No bandwidth is allocated or guaranteed. FCP uses this Class. > Supported by Brocade Class 4 It is connection oriented with ack. Ues virtual circuites. Can allocate requested amount of bandwidth (used in video) Class 6 Multicast, also known as uni-directional dedicated connection with ack used sometimes in avionics Class F Switch to switch. It is connectionless with ack. > Supported by Brocade
19 2008 Brocade
24
FC-2 Advanced Fabric Flow Control

How Buffer to Buffer Credits Work
A Fibre channel link is a PAIR of paths A path from "this" transmitter to the "other" receiver and a path from the "other" transmitter to "this receiver The "buffer" resides on each receiver, and that receiver tells the linked transmitter how many BB_Credits are available Sending a frame through the transmitter decrements the B2B Credit Counter Receiving an R-Rdy through the receiver increments the B2B Credit Counter Node N-port
B2B Credit Cnt
Switch F-port Fiber Cable transmit receive
16 Avail.
BC 2 - 60 I have 16 buffer credits Pool 107 Avail.

B2B Credit Cnt
BC I have 107 buffer credits transmit Pool receive
Buffer-toBuffer-to-Buffer Credits are not negotiated! And each receiver on the fiber cable can state a different value!
19 2008
Brocade
25
BB Credit Flow Always Between Units
BB Credits BB Credits
BB Credits
ISL
Server BB Credits Director BB Credits Director BB Credits Storage
19 2008
Brocade
26
Buffers and distance

BB Credit Information Is Exchanged During FLOGI
Typical Data Frame is 2112 Bytes in Size

1 Gbps or 100 MBps 1 Buffer Credit (BC) 1 Buffer Credit (BC)
0.5 BB Credit/Km
2 Gbps or 200 MBps
1 BB Credit/Km
4 Gbps or 400 MBps
2 BB Credit/Km
8 Gbps or 800 MBps 10 Gbps or 1000 MBps
4 BB Credit/Km
6 BB Credit/Km
0 km
19 2008
1km
Brocade
2 km
3 km
4 km
27
FCP frame control
19 2008
Brocade
28
Fibre Channel
Buffer Credits
S O F HEADER PAYLOAD C E R O C F
BUFFER CREDIT
Buffer credits determine the amount of data in transit at any time. A buffer credit allows a device to send a frame before an R_RDY (receiver ready) is received. As the data requirements increase, so should the buffer credits. The physics of light dictate that a fiber has a latency of 5ns/meter, or 5usec/km. The round-trip latency over 10km would be 100usec. The longer the distance, the more credits needed. When talking about speeds of 2Gbps, the credits must be doubled, since the time is halved. HBAs need to be able to support high numbers of credits. Some HBAs store I/O information, others relay it to the server. This can add to the throughput latency. More expensive HBAs will have a context cache to hold such information.
19 2008 Brocade
29
SAN Port Interfaces

FC Node WWN Format
19 2008
Brocade
30
SAN Port Interfaces

FC Port WWN Format
19 2008
Brocade
31
SAN Port Interfaces

Node WWN and Port WWN
19 2008
Brocade
32
Fibre Channel
Port Types
Device Ports
N_Port - Node Port, a Fabric device directly attached NL_Port - Node Loop Port, a device attached to a loop
Switch Ports
U_Port - Universal Port, a port waiting to become some other port type G_Port - Generic Port, a port waiting to be an E or F_Port F_Port - Fabric Port, a port to which an N_Port attaches FL_Port Fabric Loop Port, a port to which a loop attaches E_Port Expansion port used for inter-switch links
19 2008
Brocade
33
Fibre Channel
Expansion Port E_Port
Allows two switches to be connected to create a multi-switch Fabric Connections are called ISLs (Inter Switch Links) Also used to create trunks (eight ISLs combined) Supports 16 virtual channels per ISL
19 2008
Brocade
34
Fibre Channel
Brocade Specialization Virtual Channels
Eight Virtual Channels improve performance

VC 0 - For Link Control Frames VC 1 - For Class 2 ACKs and Link Control Frames VC 2 - For Data VC 3 - For Data VC 4 - For Data VC 5 - For Data VC 6 - For Multicast traffic VC 7 - For Broadcast traffic VC for QoS Switch E
One ISL, 16 Virtual Channels
Condor2 increases Data VCs to 12 (total 32)
Switch
19 2008
Brocade
35
Fibre Channel
Brocade Specialization Virtual Channels (Cont.)
Each physical link is partitioned into eight virtual channels Each VC has independent flow control Bottlenecks on one VC do not impact other VCs An anti-starvation mechanism prevents blocking
Initiators E_Port
1 x ISL 4 x Data VCs
E_Port
Targets
19 2008
Brocade
36
SAN Port Interfaces

FC Addressing
PORT ID: XX YY ZZ where: XX Domain is a value between 0x1 to 0xEF YY ZZ is the port number (0-255) is the AL_PA for a loop device or 00 for a F_Port or any Value for NPIV
19 2008
Brocade
37
Fibre Channel
Address Scheme
Private Loop Address

00 00 PP
Where PP = the local loop address (AL_PA). The private address only uses the last byte (8 bits) of the 24-bit Fabric address.
Fabric Assigned Address
Public Loop Address
NN NN 00
Where NN NN 00 = the address of any Fabric-attached device that has logged into the Fabric. These devices use all 24 bits of the address.
LL LL PP
Where LL LL is assigned by the Fabric at login; and PP = the local loop address (AL_PA). These devices use all 24 bits of the address, and the last 8 bits is the AL_PA.
38
19 2008
Brocade
Fibre Channel
Addressing Examples
Private 00 00 CA NL NL NL Fabric 09 04 00 04 05 06 07 F N Public 09 0E 04 Public 09 01 01
FC-AL
FL NL
FL 00 01 02 03 15 14 13 12
Switch Domain #9
11 10 09 08
Private 00 00 08
F N
Fabric 09 0A 00
39
19 2008
Brocade
Brocade Fabric Services Overview
Key Fabric Services

Distributed Name Server
Standards-based network address assignment Dynamic scalability Auto Discovery Zero Administration
Fabric Controller
SCR registrations, RSCN messages delivery
FSPF Routing
Scalability and Cascading, Any port to any port Redundant, No single points of failure
Zoning
WWN and Port based Software and Hardware Enforcement
19 2008 Brocade
41
Fibre Channel
Recognizing Well Known Fabric Addresses
19 2008
Brocade
42

Distributed Functional Model at FFFFFC
19 2008
Brocade
43

Port and Node Attributes in Name Server database
Port attributes
Port Identifier (Native port address ID) Port Name (World Wide Name) Class of Service (2, 3) FC-4 Types (SCSI, IP) Port Type (N, NL) Symbolic Port Name (free-form information)
Node attributes
Node Name (World Wide Name) Fibre Channel IP Address Initial Process Associator Symbolic Node Name (free-form information)
19 2008 Brocade
44

Device Communication Example
19 2008
Brocade
45

Registered State Change Notifications (RSCN)
HBAs need to be good citizens to work properly with the Name Server. The attributes of a good citizen are:
- It supports RSCNs - It queries the Name Server for available ports - It accesses only ports that are defined by the Name Server
Functional View
Fabric Controller
Well-Known Address FFFFFD
19 2008
Brocade
46

RSCNs
The switch will deliver an RSCN to an Nx_Port only if the following happens: The Nx_Port has registered (using SCR) to receive RSCNs The Nx_Port is still logged into the Fabric The registration function matches the RCSN Type The domain and area fields of the affected Nx_Port are different from the destination Nx_Port
Sent to an end device at most once every 500 ms Three formats of RSCNs: Single PID (same as existing behavior) Multiple PID (multiple PIDs are aggregated into one payload) Fabric RSCN (single fabric format RSCN sent)
19 2008 Brocade
47

State Changes and Notifications
SCN State Change Notification, used for internal state change notifications only. This is the switch logging that the port is online or is an Fx_Port. SCR State Change Registration, issued (requested) by an Nx_Port. RSCN Registered State Change Notification, issued (responded) by the Fabric or an N_Port.
19 2008
Brocade
48
Fibre Channel Routing

Fabric Terminology
Inter-switch Links (ISLs) E_Port-to-E_Port links Communicates using Class F service Will be segmented if link parameters are incompatible Principal Switch Selected when the fabric initializes, before routing is established Manages the assignment of unique domain IDs Provides time synchronization to all other switches in the fabric Principal ISL ISL used to communicate between the Principal Switch and other switches in the fabric
19 2008
Brocade
50

Principal Switch Path
Switch 4 is Principal Switch (fabricprincipal Lowest WWN)
Principal ISL ISL Upstream: none Downstream: 4-3,4-6
Domain 4
Upstream: 3-4 Downstream: 3-2,3-5
Domain 3
Domain 6
Upstream: 6-4 Downstream: none
Domain 2
Domain 5
Upstream: 5-3 Downstream: 5-1
Domain 1
19 2008
Brocade
51

What is a Path?
A Path is a chain of switches from source to destination

Source 011F00 Destination 041600
JBOD
Domain 1 Domain 2 Domain 3

19 2008 Brocade
Domain 4
52

Using a Path
Two valid paths from source host to destination storage
12 3
14 7 13
11
19 2008
Brocade
53

What is a Route?
A route is a map to reach the next hop between an input port and an output E_Port
Domain 1 14 7 Domain 3 13
12 3 Domain 2 11
13 4
19 2008
Brocade
54

FSPF Algorithm
Routing Algorithm Uses cost/weight and bandwidth Can apply static routes if needed
Automatic Failover Fault detection 150ms Self heals in 500ms Alternate route live in 650ms
19 2008
Brocade
55

Path and Route Selection Architecture
Paths: Fully distributed - No single point of failure within the Fabric Fast recovery in case of link or switch failure; identifies switches by domain ID Traffic load sharing over equivalent paths Link State Protocol Fabric Shortest Path First (FSPF) Finds the shortest path to each domain, then programs the hardware routing tables Routes: Dynamically (all SilkWorm switches) Round robin Administrator can configure the route Automatically re-routes upon ISL going away and static routing will again take effect upon ISL return
Brocade
Statically (for the SilkWorm 2000 series and above)
19 2008
56

Routing policy
The routing policy is responsible for selecting a route based on one of two user-selected routing policies

Port-based Routing Exchange-based Routing
2 Gbit/sec ASIC routing is handled by the Fabric Shortest Path First (FSPF) protocol and uses only the port-based routing policy 4 and 8 Gbit/sec ASICs use the FSPF protocol and either portbased routing or exchange-based routing (default) Each switch has its own routing policy Different policies can exist in the same fabric
19 2008 Brocade
57

Port-Based Routing Policy
FSPF calculates the cost of each link and determines the lowest cost path

Within each switch, the input port from the source is assigned to an output port toward the destination (a route) Routes are allocated via round-robin assignment Chosen routes are used until one of the devices in the fabric goes offline or the fabric changes Changes in fabric, when Dynamic Load Sharing is enabled, cause FSPF to recalculate the routes and may reassign the output port to better distribute devices across equal cost routes
Dynamic Load Sharing (DLS) and In-Order Delivery (IOD) options 2 Gbit/sec ASICs support only the port-based routing policy 4 and 8 Gbit/sec ASICs support port-based routing and exchange-based routing
19 2008 Brocade
58

Port Based Routing
Round robin allocates a route from shortest equivalent paths based on link cost FSPF default link costs 1000 at 1 Gbit/sec 500 at 2 / 4 / 8 Gbit/sec
19 2008
Brocade
59

Port Based Routing (cont.)
Source ID and destination domain are used to allocate routes Devices are round-robin allocated to available equal cost routes It is possible to have congestion, if too many high I/O requiring devices are allocated to a single route
19 2008
Brocade
60

Exchange-based Routing
Frames in an exchange are identified by common characteristics called frame parameters Exchange-based routing policy exchange frame parameters include: (DID/SID/OXID)

The default policy on 4 / 8 Gbit/sec switches Available on 4 / 8 Gbit/sec switches only
Cannot create static routes DLS always is enabled and it cannot be switched off
19 2008 Brocade
61

Exchange-based Routing (cont.) The exchange-based routing policy uses an internal hash to allocate a particular SID, DID, OXID exchange to an ISL Exchanges are allocated proportional to the bandwidth available on each of the routes All sequences of frames within a SID, DID, and OXID exchange (the same hash) will traverse an assigned ISL
19 2008
Brocade
62

Display Routing Information Overview
Paths and link connections can be verified using switchshow, fcping, and pathinfo The FSPF protocol determines routing on a local basis Fabric topology information is known at every switch displayed with urouteshow, topologyshow & other routehelp commands No global, edge-to-edge routing table is maintained For a listing of route-related commands use routehelp
19 2008
Brocade
63

Dynamic Load Sharing
dlsSet This command allows load sharing to take place when a fabric change occurs. dlsReset This command prevents load sharing from taking place when a fabric change occurs. dlsShow This command displays the state of the Dynamic Load Sharing option.
19 2008 Brocade
64

Route Selection (dlsSet Example)
Host 3 (On-Line 3rd )
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07 15 14 13 12
00 01 02 03 04 05 06 07 11 10 09 08
Host 2 (On-Line 2nd )
Host 1 (On-Line 1st)

19 2008 Brocade
65

Route Selection (dlsSet Example Cont.)
With DLS not set connections are not re-routed Host 3
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07 15 14 13 12
00 01 02 03 04 05 06 07 11 10 09 08
Host 2 (Off-Line)
Host 1
19 2008 Brocade
66

Route Selection (dlsSet Example Cont.)
With DLS set connections are re-routed Host 3
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07 15 14 13 12
00 01 02 03 04 05 06 07 11 10 09 08
Host 2 (Off-Line)
Host 1
19 2008 Brocade
67

In Order Delivery
iodSet This command enforces in-order delivery of frames during fabric topology changes. iodReset This command allows out-of-order delivery of frames during fabric topology changes. iodShow This command indicates if IOD is turned on or not.
19 2008
Brocade
68

In Order Delivery (iodSet Example)
Host 3
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07 15 14 13 12
00 01 02 03 04 05 06 07 11 10 09 08
Host 2
1
Frames In Queue
Host 1
19 2008 Brocade
69

In Order Delivery (iodSet Example Cont.)
Host 3
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07 15 14 13 12
00 01 02 03 04 05 06 07 11 10 09 08
Host 2 (Off-Line)
1
Frames In Queue
Host 1
19 2008 Brocade
70

In Order Delivery (iodSet Example Cont.)
With iodReset, frame 2 arrives before frame 1
Host 3
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07
00 01 02 03 15 14 13 12 11 10 09 08 04 05 06 07
Host 2 (Off-Line)
1
Frames In Queue
Host 1
19 2008 Brocade
71

Setting a Link Cost Metric
linkCost <port #>,<value> sw2:admin> linkCost 3,250 The default metrics are:
1000 at 1 Gbps 500 at 2/4/8 Gbps
19 2008
Brocade
72

Route Selection Default Behavior
00 01 02 03 15 14 13 12 Domain 1 11 10 09 08
500
linkCost Default = 500

04 05 06 07
0 50
Domain 1 has 3 routes to Domain 3 2 routes have total metric = 500 1 route has total metric = 1000 (never used) Only routes with least total metric will be kept in routing tables
0 50
00 01 02 03 15 14 13 12 Domain 2 11 10 09 08 04 05 06 07
00 01 02 03 15 14 13 12 Domain 3 11 10 09 08 04 05 06 07
500

19 2008 Brocade
73

Route Selection Using Cost
00 01 02 03 15 14 13 12 Domain 1 11 10 09 08
250
04 05 06 07
Command invoked from Domain 1: sw1:admin>linkCost 10,250 Command invoked from Domain 2: sw2:admin>linkCost 7,250
0 50 0 50
00 01 02 03 15 14 13 12 Domain 2 11 10 09 08 04 05 06 07
00 01 02 03 15 14 13 12 Domain 3 11 10 09 08 04 05 06 07
250

19 2008
linkCost is not bi-directional!

Brocade
74

Route Selection Using Cost
00 01 02 03 15 14 13 12 Domain 1 11 10 09 08
500 250
04 05 06 07
No linkCost commands issued from Domain 3 to Domain 2 or Domain 2 to Domain 1
0 50 0 50
00 01 02 03 15 14 13 12 Domain 2 11 10 09 08 04 05 06 07
00 01 02 03 15 14 13 12 Domain 3 11 10 09 08 04 05 06 07
250 500

19 2008 Brocade
75
Switch Latency is NOT an issue

2 microseconds
Disk Subsystem Latency
19 2008
ycnetaL latoT
~4-13,000 Micro-seconds
2 microseconds ~1000
microseconds
Switch Hop Latency
Storage Cache hit

Brocade
Storage Cache miss

76
SAN Design
Key SAN Design Decisions

Fabric size and number of fabrics Selection of switching elements
Number of ports Bandwidth RAS (Reliability/Availability/Serviceability)
Fabric architectural model

Flat Core/Edge Inter-Fabric Routed
Geographical considerations
Storage over MAN/WAN
SAN Management and Diagnostic

19 2008 Brocade
78
Fabric Zoning
The server in the red zone sees one loop of disks The server in the blue zone sees two storage arrays The server in the green zone sees one loop and one array No server sees loop 2
19 2008 Brocade
79
Basic Design Principles

Zoning Enforcement
Session Enforcement
Name Server restricts PLOGIs
Hardware Enforcement
Available through ASIC hardware logic checking Denies illegal access from bad citizens More secure then session
Enforcement based on how members in a zone are defined
19 2008
Brocade
80
Basic Design Principles

Zoning Rules
Devices to Storage: one to many (one HBA port -> to all storage ports) Storage to Storage: one to one Use Aliases for WWNs or Ports Do not mix WWN and Port zoning (if possible)
19 2008
Brocade
81
Basic Design Principles (contd)

ISL Optimization
Condor-ASIC
Brocade (Port Based) Trunking

Provides trunked ISL Bandwith up to 64 Gbit/sec (Condor2)
FOS: Dynamic Path Selection (DPS) EOS: Open Trunking (OT)

Balances loads across trunk groups at the exchange level Preserves in-order delivery within an exchange
19 2008 Brocade
82
Optical Budget
Basic Optical Budget = Output Power Input Sensitivity
This is a parameter used in the design of optical transmission networks. It is the difference between output power level of the source and the receiver sensitivity. It is generally measured in dB (Decibel). Pout = -15 dBm R = -22 dBm
Budget = 7 dB (28km/1550nm)
Optical Budget is affected by:

19 2008
Fiber attenuation (Loss = 0.25db/km for 1550nm) Splices Patch Panels/Connectors (add 0.5db per connector) Optical components (filters, amplifiers, etc) Bends in fiber Contamination (dirt/oil on connectors)
Brocade
83

FCBasics

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

FCBasics

Încărcat de

Drepturi de autor:

Formate disponibile

Fibre Channel Basic

Fibre Channel Combining SCSI with Networking

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Think of Fibre Channel As Super SCSI

Fibre Channel Highway

Powerful SCSI SCSI FC FRAME FC FRAME management

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Fibre Channel Technical Capabilities

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Fibre Channel As an Open Standard

Open, MultiVendor SANs

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Fibre Channel Hardware

Glass Core Glass Cladding Coating

Glass Core Glass Cladding Coating

Fibre Channel Basics

Light travels at ~5nsec per meter in glass

Fibre Channel Hardware - Connectors

HSSDC Copper Connector

Smaller than DB-9 Easier to insert/remove

HSSDC2 Copper Connector

- Smaller than HSSDC

Fibre Channel Hardware

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Fibre Channel Hardware

300m 150m 70m 21m 33m N/A N/A N/A N/A

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Brocade SilkWorm Hardware

SFP LC (2/4/8 Gb/s Products)

SC GBICs (1 Gb/s Products)

Fast File Transfers

Framing Protocol / Flow Control / COS

Encode / Decode / PSM & LPSM

Optical Laser, LED Copper Coax, Twisted Pair

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

Fibre Channel Layered Architecture

Fibre Channel Data & Applications

With fewer processor interruptions, application can run faster.

Flexible Deployment Topologies

Only 2 Devices (Direct Connect)

Up to 126 Devices (Fibre Channel Hubs)

Up to 16 Million Devices (Fibre Channel Switches)

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

1st Information Unit FC-4 1st Sequence

Informat ion Unit

Last Information Unit

8B/10B Characte r 8B/10B Characte r

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

SCSI-FCP Write Command

IU 1 Command Command Chunk

IU 2 Data Data Chunks

IU 3 Status Status Chunk

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

SCSI-FCP Write Command cont.

Target Sequence seq_id

nt 0] [seq _c r read y * x _fe th ed Le ng co rre ct , S_ID ,

Have to be last frame!!!

2008 Brocade Communications Systems , Inc. All Rights Reser ved.

SCSI-FCP Read Command

Target Node SR Sequence 1

Command IU 1 Command Chunk Exchange

IU 2 Data Data Chunks Status IU 3 Chunk Status