Documente Academic
Documente Profesional
Documente Cultură
Configuration requirements for B-Series switches may vary depending on the type of distance
extension device used. The following information provides generic guidelines for determining
how a switch should be configured.It is important to take note of the model and type of line
card connected to the FC switch in order to determine the proper configuration, as incorrect
configuration may result in errors or link failure.
Connecting FC switches directly to dark fiber requires the use of long-wavelength SFP
transceivers. It may be necessary to use the FOS licensed feature Extended Fabric to allocate
sufficient buffers to the long distance E_Ports. In addition, where multiple E_Ports are
required it may be necessary to spread these to different port groups to maximize the number
of buffer credits available.
If appropriate, configure Brocade Extended Fabrics for the appropriate distance.
The following example shows how port 1/0 of a Brocade switch can be configured for LS
Extended Fabrics mode and 100 km distance:
The optics requirements depend on the nature of the WDM device. In addition to concerns
with FC over dark fiber, because CWDM and DWDM products are protocol and bit-rate
transparent, FOS configuration is identical to connecting FC switches directly to dark fiber.
Note that you should “hard” set the FC port speed to the desired rate to ensure that WDM
transponders can lock onto the bit-rate of the ISL.
1.If appropriate, configure Brocade Extended Fabrics for the appropriate distance.
2.Set FC port speed on long distance E_Ports.
The following example shows how an FC port can be configured for a 75 km distance over a
Coarse or Dense WDM device operating at 4 Gbit/sec:
The optics requirements depend on the nature of the WDM device. In addition to concerns
with FC over dark fiber, other factors may apply. TDM and FC-SONET/SDH devices that do
not actively participate in buffer credit management may also require Brocade Extended
Fabrics for optimal configuration (see the next section on “Extended Distance Solutions”). In
addition, it is usually required that you configure the E_Port to operate with Idle rather then
ARB primitives to maintain synchronization. Some devices may also require ports to be
configured to G_Port mode so that loop initialization is not attempted. Note that credit
recovery and quality of Service (QoS) will not be activated on the ISL in these configurations.
1.Configure E_Ports to ensure the use of Idle rather than ARB primitives.
2.Configure Brocade Extended Fabrics for the appropriate distance.
3.Set FC port speed on long-distance E_Ports.
4.Configure E_Ports to G_Port mode.
For FOS version prior to 6.1.1, the port should be configured for R_RDY mode in order to
interoperate with most TDM or SONET devices. The following example shows how a port
can be configured for a 100km link.
FOS versions 6.1.1 and later allow Extended Fabrics to operate in VC_RDY mode using idle
as fill words. This allows Brocade frame-based trunking to be supported over TDM and
SONET/SDH, assuming latency restrictions are met by the distance extension device. Idle fill
words are enabled on the E_Port by setting the “VC Translation Link Init” parameter of the
portCfgLongDistance command to 0 (“zero”) and disabling QoS and credit recovery. The
following example show how a port can be configured:
Summary
The following table describes the recommended guidelines for configuring a B-Series switch
for distance extension devices.
Optical Transport Extended Idle (not ARB) R_RDY Port Speed G_Port
Fabrics Primitive Mode Hard Set Mode
CWDM √ √
DWDM √ √
FC-SONET √ √ √ √
TDM √ √ √ √
Add Extended √ √ √
Distance to above
Key Features
Feature Reason
Virtual Fabrics Cost-effective way to use a single chassis to provide multiple logical
switches with only as many ports dedicated for array replication traffic
as needed.
Large BB_Credit Pool Supports over 1,000 KM distances for ISL connections at full link rate
Allocations to E_Ports providing high bandwidth over extended distances
Extended Distance Enables more BB_Credits for long distances only when required.
License
Partner WDM Optical Ensures product and protocol compatibility between Brocade
Transport and partner products.
Standard E_Ports on Brocade’s FOS-based platforms automatically set both the link
initialization and fill word primitives to ARB regardless of the portcfgfillword setting when
not in R_RDY mode.
In FOS v6.1.2 Brocade introduced support for configuring either ARB or IDLE primitives for
fill words on Extended Fabrics E_ports when in VC_RDY flow control mode. The CLI
portcfglongdistance was enhanced such that if the vc_translation_link_init parameter was
configured as zero, then IDLE primitives were used as fill words. By default ARB primitives
are used as fill words on long distance links.
Mode 0: IDLE/IDLE
Mode 1: ARB/ARB
Mode 2: IDLE/ARB
Mode 3: ARB/ARB then IDLE/ARB
If there is a conflict between the fill word mode and vc_translation_link_init settings the
portcfglongdistance command will warn the user of incompatibility, but it will not prevent the
user from executing the CLI. That is, it will enable vc_translation_link_init as specified by the
user, but it will NOT change the fill word mode.
For example, if the current fill word mode is 0 for the port 2/0, and if a user executes the
command portcfglongdistance to set vc_translation_link_init to 1, then the command reports a
warning message as follows:
Similar warning message is also displayed by the portcfgfillword CLI when it detects
incompatibility with portcfglongdistance command settings. But this warning message from
portcfgfillword is displayed only on FOS v6.2.1 or later releases on the FOS v6.2.x code
stream and FOS v6.3.0 or later releases on the FOS v6.3.x code stream.
To Verify Current Link Init and Fill Word Mode Settings, Users can execute portcfgshow
command to verify the configuration of fill word mode and VC Link Init settings for a given
port.
Prev
Next
Home
SAN/Storage
o Netapp
o EMC VMAX/DMX
o EMC VNX/Clariion
o IBM STORAGE
o Brocade
o Others
MS Product
o MS Server
o MS Visio
o Exchange
Open System
o AIX
o HPUX
o Solaris
The most effective configuration for implementing long-distance SAN fabrics is to deploy
Fibre Channel switches at each location in the SAN. Each switch handles local
interconnectivity and multiplexes traffic across long-distance dark fiber or wave division
multiplexing (WDM) links while the Brocade Extended Fabrics software enables SAN
management over long distances.
For the licensing, L 0(level0) and LE(Level E)do not require a license, use of LD(Dynamic
Long Distance Mode) and LS(Static Long Distance Mode)requires an Extended Fabric
License.
The following items require Extended Fabric License. (Note that the long distance modes
L0.5, L1, and L2 are no longer supported in v5.3.0 and later)
LD calculates BB credits based on the distance measured during port initialization. Brocade
switches use a proprietary algorithm to estimate distance across an ISL. The estimated
distance is used to determine the BB credits required in LD (Dynamic) extended link mode
based on a maximum Fibre Channel payload size of 2,112. An upper limit may be placed on
the calculation by providing a desired_distance value. Fabric OS confines user entries to no
larger than what it has estimated the distance to be. When the measured distance is more than
desired_distance, the desired_distance(the smaller value) is used in the calculation.
Prev
Next
Home
SAN/Storage
o Netapp
o EMC VMAX/DMX
o EMC VNX/Clariion
o IBM STORAGE
o Brocade
o Others
MS Product
o MS Server
o MS Visio
o Exchange
Open System
o AIX
o HPUX
o Solaris
factor. The I/O size, I/O per Second (IOPS) limit, and concurrent or outstanding I/O capability
at the SCSI level of the initiators/targets can be and often are gating factors.
Brocade switches can support two methods of flow control over an ISL or E-port. VC_RDY
and R_RDY flow control are both available options for all Brocade switch types.
VC_RDY
For VC_RDY flow control, Brocade switches require an “Extended Fabric Mode” which will
require to be activated through license code. The supported distances for an E_Port varies
when activating these modes in a Fibre Channel point-to-point switched fabric environment.
Please Keep in mind that each Brocade switch family, ASIC, and mode type (i.e, L1, L2, LD,
etc.) will have unique VC_RDY amounts and characteristics depending on specific fabric
configurations.Please refer to brocade support matrix.
VC_RDY is the default method and uses multiple lanes or channels, each with different buffer
credit allocations, to prioritize traffic types and prevent head-of-line blocking. VC_RDY flow
control differentiates traffic across an ISL. It serves two main purposes:
Fabric internal traffic is generated by switches that communicate with each other to exchange
state information (such as link state information for routing and device information for
Name Service). This type of traffic is given a higher priority so that switches can distribute
the most upto-date information across the fabric even under heavy device traffic.
Additionally, multiple IOs are multiplexed over a single ISL by assigning different VCs to
different IOs and giving them the same priority (unless QoS is enabled). Each IO can have a
fair share of the bandwidth, so that a large-size IO will not consume the whole bandwidth and
starve a small-size IO, thus balancing the performance of different devices communicating
across the ISL.
R_RDY
Receiver Ready (R_RDY) – R_RDY is defined in the ANSI T-11 standards and uses a single
lane or channel for all frame types.
When connecting switches across dark fiber or wave division multiplexing (WDM) optical
links, VC_RDY is the preferred method, but there are some distance extension devices that
require the E_Port use R_RDY. To configure R_RDY flow control on Brocade switches, use
the portCfgISLMode command.
Prev
Next
Home
SAN/Storage
o Netapp
o EMC VMAX/DMX
o EMC VNX/Clariion
o IBM STORAGE
o Brocade
o Others
MS Product
o MS Server
o MS Visio
o Exchange
Open System
o AIX
o HPUX
o Solaris
E_Port Consideration
Standard E_Ports on Brocade’s FOS-based platforms automatically set both the link
initialization and fill word primitives to ARB regardless of the portcfgfillword setting when
not in R_RDY mode.
In FOS v6.1.2 Brocade introduced support for configuring either ARB or IDLE primitives for
fill words on Extended Fabrics E_ports when in VC_RDY flow control mode. The CLI
portcfglongdistance was enhanced such that if the vc_translation_link_init parameter was
configured as zero, then IDLE primitives were used as fill words. By default ARB primitives
are used as fill words on long distance links. So When connecting to extension devices that do
not support ARB primitives, portcfglongdistance vc_translation_link_init parameter should be
set to zero.
It is important to note that some of the combinations of vc_translation_link_init value and fill
word modes are incompatible. The following table provides the supportability matrix between
vc_translation_link_init value and the fill word modes. Users must NOT configure the
unsupported.
0 0 Yes IDLE/IDLE
0 123 No NA
1 0 No NA
1 1, 2, 3 Yes ARB/ARB or
IDLE/ARB
For example, if the current fill word mode is 0 for the port 2/0, and if a user executes the
command portcfglongdistance to set vc_translation_link_init to 1, then the command reports a
warning message as follows:
Note: this warning message from portcfgfillword is displayed only on FOS v6.2.1 or later releases
on the FOS v6.2.x code stream and FOS v6.3.0 or later releases on the FOS v6.3.x code stream.
To Verify Current Link Init and Fill Word Mode Settings, Users can execute portcfgshow
command to verify the configuration of fill word mode and VC Link Init settings
Trunk Port ON
NPIV capability ON
QOS E_Port AE
EX Port OFF
Credit Recovery ON
To Change Long Distance Fill Word Mode Settings, user will need to use portcfglongdistance
CLI after ensure no conflict with the fill word mode setting.
Example:
Assume that a long distance port is currently configured to use IDLE/IDLE primitives. If a
user wants to change the VC Link Init of this port to 1 (to use ARB primitives) then he/she
can execute portcfglongdistance command with vc_translation_link_init parameter set to 1,
and execute portcfgfillword CLI to set the fill word mode to 1, 2 or 3.
Please use portcfgshow CLI to find out the fill word and VC Link Init values configured for a
given port.
Home
SAN/Storage
o Netapp
o EMC VMAX/DMX
o EMC VNX/Clariion
o IBM STORAGE
o Brocade
o Others
MS Product
o MS Server
o MS Visio
o Exchange
Open System
o AIX
o HPUX
o Solaris
Up through the FOS v6.1.1 release, Brocade Fibre Channel switches used IDLE primitives
both during link initialization and for fill words (denoted as “IDLE/IDLE”). This ensured
successful link initialization between Brocade switch ports and end devices operating at
1G/2G/4G speeds.
With the introduction of 8G Fibre Channel, FC-PI-4 and FC-FS-3 standards stated that ARB
should be used as fill words for lowering emission. To comply with the published FC
standards, Brocade introduced options for ARB/ARB and IDLE/ARB link initialization/fill
word support. So What is fillword, what do they do and why are they needed
There has been quit some confusion around the use of fill words with the adoption of the 8G
fibre-channel standard. Some admins have reported that they have problems connecting
devices on this speed as well as numerous headaches in long-distance replication especially
when DWDM/CWDM equipment is involved.
An ordered set is a transmission word used to perform control and signaling functions. There
are 3 types of ordered sets defined:
1. Frame delimiters. These identify the start and end of frames.
2. Primitive signals. These are normally used to indicate events or actions (like IDLE)
3. Primitive Sequences which are used to indicate state or condition changes and are normally
transmitted continuously until something causes the current state to chance. Examples are
NOS,OLS,LR,LRR
So what is a fill-word? A fill-word is a primitive signal which is needed to maintain bit and
word synchronization between two adjacent ports. Is doesn't matter what port type (F-port,E-
port,N-Port etc) it is. They are not data frames in the sense that they transport user-data but
instead they communicate status messages between these two ports. If no user-data is
transmitted the ports will send so called IDLE frames. These are just frames with some bit
pattern where the ports are able to keep there synchronization on a bit-level as well as a word
level. The IDLE frame is a 10-bit frame on the wire, as any ordered set starts with K28.5
which is a fibre-channel notation for 8B10B encoding and three data frames of which the last
20 bits are 1010101010....etc. Depending on the content of these frames it's either a fill-word
or non-fillword.
Examples of fillwords are IDLE, ARB(F0), ARB(FF) and non-fillword are R_RDY,
VC_RDY etc.
So what happened recently with the introduction of the 8G standard.
In the 1,2 and 4G standard the IDLE primitive signal was used to keep bit and word
synchronization. This bitpattern was OK on those speeds however it has been observed that
when increasing the clock speed this pattern caused high emissions which in turn could cause
problems on adjacent ports and links. In order to reduce that the standard now requires links
that are using 8G speed to use the ARB(ff) fill-word. This is a different bit-pattern which
doesn't have this high emission characteristic.
You might wonder what does this have to do with my connection problem? If links negotiate
on 8G speed they both have to use the ARB(FF) fill-word. If that doesn't happen for some
reason then the ports cannot maintain word synchronisation and therefore cannot change the
port into the active state. This causes both ports to be in some sort of deadlock situation and
although you may see that there is a green status light on your HBA and switch port it still is
not able to transfer data.
The standard defines that ports who connect on 8G speed first have to initialize with IDLE
fill-words and as soon as the port changes to the active state it should change the fill-word to
ARB(FF).
It becomes even more complicated with DWDM and CWDM equipment particularly when
multiplexers are used. These TDM devices normally crack open the fibre-channel link on a
frame boundary level and then are able to multiplex this on a higher clock-rate so they are
able to send data from multiple links into one wavelength. If however these TDM devices
cannot open the fibre-channel link because they only look for IDLE fillwords then the end-to-
end link will fail.
Verify with you manufacturer if you use TDM devices and if so do they support ARB(FF)
fillwords. If not than you may have to force the linkspeed to a lower level like 4G.
However, some 8G devices are not capable of properly establishing links with Brocade 8G
Fibre Channel switches when ARB/ARB or IDLE/ARB primitives are used. These 8G
devices require the legacy IDLE/IDLE sequence to achieve successful link initialization. To
address this issue, Brocade has provided the ability to configure any of the three possible
combinations (IDLE/IDLE, ARB/ARB, or IDLE/ARB) for link initialization and fill words.
Any of these modes can be configured on an individual port basis using the CLI
portcfgfillword. This CLI also provides an option to automatically switch from ARB/ARB to
IDLE/ARB if the former is not able to successfully establish a link. This capability is
available in FOS v6.3.1 and later versions of firmware.
So to summarize,
1. For any device connected to switch at levels between 6.2.0c and 6.3.1, the fillword should
remain at default mode which is 0.
2. FOS v6.3.1 and later versions support the following modes for 8G F_Ports as a per port
configuration.
MODE MEANING
It's recommended by brocade to configuring the fill word to mode 3 which typically enables
the port to use the right combination of Fibre Channel primitives to fully interoperate with the
attached end device. Once set to mode 3, user intervention is no longer needed to configure
the right primitives to enable the link between the Brocade 8G switch port and the end device.
Note: If a device is known to require mode 2 (IDLE/ARB) then configuring mode 2 on the 8G
switch port will activate the link sooner. If mode 3 is used in this environment, then the port
will first attempt mode 1 then eventually settle to mode 2 which will cause a delay in
initializing the port. If a device requires IDLE/IDLE combination for link initialization then
one needs to set the needed port to mode 0 using portcfgfillword CLI.
To Verifying Current Fill Word Mode Setting, Users can execute portcfgshow command to
verify the configuration of fill word mode and VC Link Init settings for a given port.
To set Fillword execute portcfgfillword CLI, please note that Changing the “Fill Word” on a
port is disruptive change.
Ports of Slot 0 4 5 6 7
-----------------+---+---+---+-----
AN AN AN AN
Fill Word 0 0 0 0
AL_PA Offset 13 .. .. .. ..
Trunk Port ON ON ON ON
FID128:admin> portcfgfillword 4, 3
FID128:admin> portcfgfillword 5, 3
FID128:admin> portcfgshow -i 4-7
Ports of Slot 0 4 5 6 7
-----------------+---+---+---+-----
AN AN AN AN
Fill Word 3 3 0 0
AL_PA Offset 13 .. .. .. ..
Trunk Port ON ON ON ON
Prev
Next