«DATA CENTER Brocade VDX/VCS Data Center Layer 2 Fabric Design Guide for Brocade Network OS v2.1.1 DATA CENTER DESIGN GUIDE CONTENTS Introduction Building ...»
For a 10 km ISL link, no other ISL links are allowed on the same ASIC. For 5 km and 2 km ISL links, another shortdistance ISL link can be configured. A maximum of three PFCs (per Priority Flow Control) can be supported on a longdistance ISL port.
To configure a long-distance ISL port, use the long-distance-isl command in interface configuration mode.
Please refer to the Network OS Command Reference and Network OS Administrator’s Guide, v2.1.1 for more information on long-distance (LD) ISL.
ECMP Load Balancing Configurable Load Balancing Load balancing allows traffic distribution on static and dynamic LAGs and vLAG. Although it is not common, some traffic patterns fail to distribute, leading to only one ECMP path for the entire traffic. This causes underutilization of ECMP paths, resulting in a loss of data traffic, even though one or more ECMP paths are available to offload the traffic. In Network OS v2.1.0, a new command is introduced to configure ECMP load balancing parameters, in order to offer more flexibility to end users. This command allows users to select various parameters, which can then be used to create a hashing scheme. Please refer to the Network OS Administrator’s Guide, v2.1.1 for more information on hashing scheme and usage guidelines.
• The ECMP hash value is used to add more randomness in selecting the ECMP paths.
The default value of ECMP hash is a random number, set during boot time.
• The default ECMP load balance hashing scheme is based on source and destination IP, MAC address, • VLAN ID (VID), and TCP/UDP port.
In the presence of a large number of traffic streams, load balancing can be achieved without any • additional ECMP-related configuration.
VDX6720_1(config-rbridge-id-2)# fabric ecmpload-balance ?
dst-mac-vid Destination MAC address and VID based load balancing src-dst-ip Source and Destination IP address based load balancing src-dst-ip-mac-vid Source and Destination IP and MAC address and VID based load balancing src-dst-ip-mac-vid-port Source and Destination IP, MAC address, VID and TCP/UDP port based load balancing src-dst-ip-port Source and Destination IP and TCP/UDP port based load balancing src-dst-mac-vid Source and Destination MAC address and VID based load balancing src-mac-vid Source MAC address and VID based load balancing VDX6720_2(config)# rbridge-id 2 VDX6720_2(config-rbridge-id-2)# fabric ecmp load-balance dst-mac-vid VDX6720_2(config-rbridge-id-2)# VDX6720_2(config-rbridge-id-2)# fabric ecmp load-balance-hash-swap 34456 VDX6720_2# show fabric ecmpload-balance Fabric EcmpLoad Balance Information
-----------------------------------Rbridge-Id : 2
BROCADE VDX LAYER 2 FEATURES (EXTERNAL TO FABRIC)Active-Standby Connectivity Active/Standby connectivity is used to provide redundancy at the link layer in a network. The most common protocol used at Layer 2 to provide Active/Standby connectivity is Spanning Tree Protocol (STP). The use of STP was historically acceptable in traditional data center networks, but as the densities of servers in data centers increase, there is a requirement to improve the bandwidth availability of the links by fully utilizing the available link capacity in the networks.
Active-Active Connectivity MCT and vLAG—Multi-Chassis Trunking (MCT) is an industry-accepted solution to eliminate spanning tree in L2 topologies. Link Aggregation Group (LAG)-based MCT is a special case of LAG that is covered in IEEE 802.3ad, in which the LAG ends terminate on two separate chassis that are directly connected to each other. Virtual LAG (vLAG), a Brocade innovation, further extends the concept of LAG by allowing its formation across four physical switches that may not be directly connected to each other (but that participate in the same VCS fabric).
vLAG Enhancements In Brocade Network OS v2.1, the vLAG feature is enhanced to remove several usage restrictions imposed in the
previous Network OS releases. These are highlights of vLAG enhancement in Network OS v2.1:
The ability to specify a minimum number of links that must be active on a vLAG before it can form • aggregation is now supported in VCS mode. This was supported earlier only in standalone mode. The existing “minimum-links” Command Line Interface (CLI) under the port channel is now available in VCS mode.
The ability to validate a remote partner for dynamic vLAG is also added.
• The maximum number of RBridges participating in a vLAG is now increased to 4.
• The maximum number of ports participating in a vLAG is 32, with 16 from a single RBridge.
vLAG Configuration Guidelines with VMware ESX Server and Brocade VDX VMware recommends configuring an EtherChannel when the admin chooses IP-based hashing as Network Interface Card (NIC) teaming. All other options of NIC teaming should use regular switch port configuration.
vLAG Minimum Links The Minimum Links feature allows the user to specify the minimum number of links that must be active on the vLAG before the links can be aggregated. Until the minimum number of links is reached, the port channel shows “protocol down.” The default minimum number of links to form an aggregator is 1. The minimum link configuration allows the user to create vLAGs with a strict lower limit on active participating port members.
As with all other port-channel configuration commands executed on a Brocade VCS operating in Fabric Cluster mode, the user is required to configure the new minimum link value on each RBridge participating in the vLAG. The port channel will be operationally down, if the number of active links in a vLAG falls below the minimum number of links required to keep the port channel logically up. Similarly, the port channel is operationally up when the number of active links in the vLAG becomes equal to the minimum number of links required to bring the port channel logically up. The events that trigger a link active count change are as follows: an active link going up or down, a link added to or deleted from a vLAG, or a participating RBridge coming back up or going down.
Please note that only ports with the same speed are aggregated.
LACP SID and Selection Logic As part of establishing dynamic LACP LAGs (or port channels), LACP PDU frames are exchanged between the switch and the end devices. The exchange includes a unique identifier for both the switch and the device that is used to determine which port channel a link should be associated with. In Network OS v2.1, additional support is added for selecting a unique and consistent Local SID (LACP System ID), which is shared between all RBridges that are connected to the same vLAG. This SID is unique for each switch in the VCS.
During vLAG split and rejoin events, when the member RBridge leaves from and joins into the cluster, SID reselection logic is enhanced to support a knob to control split recovery.
Split Recovery: In Network OS v2.1.0, with no-ignore-split, SID is derived using the actual MAC address of one of the participating RBridges (SID Master). So when a vLAG is formed, the SID from the first RBridge configured into the vLAG is used. If the RBridge that owns the SID of the vLAG leaves the cluster, a new SID is selected from the MAC of one of the other Rbridges, and vLAG converges again.
No Split Recovery: In this mode, a VSID (Virtual SID)—which is a unique identifier derived using the VCS ID, similar to Network OS 2.0—is used as the LACP SID for the vLAG. Upon split, all RBridges continue to advertise the same VSID as their LACP SID. No reconvergence is needed when nodes leave or (re)join the vLAG in the VCS mode.
LACP SID Assignment In Network OS v2.1.0, SID is no longer a virtual MAC address derived using the VCS ID; instead, it is the actual MAC address of the participating RBridges. This allows scaling of the maximum number of RBridges that can participate in a VLAG, from 2 to 16.
LACP Remote Partner Validation A vLAG member’s links in different RBridges sometimes may be connected to other standalone switches. In the previous Network OS releases, there was no validation of the remote partner information for a dynamic vLAG across the RBridges. The user had the responsibility to assure that the links in the dynamic vLAG were all connected to the same remote device and key. In Network OS v2.1.0, remote partner validation support is added, which ensures that the first received partner information is sent out to all member R0Bridges. If the vLAG has a member in another RBridge, the aggregation fails, and vLAG is not formed.
Operational Consideration To ensure that the RBridges that join the fabric (RB3 in Figure 11) pick up the partner state, the remote SID state for vLAGs is included in the local database exchange. The following show command provides information on the SID Master.
show lacp sys-id:
• Port-channel Po 10 - System ID: 0x8000,00-05-33-8d-0e-25 - SID Master: rbridge-id 1 (ignore-split Disabled) Port-channel Po 20 - System ID: 0x8000,01-e0-52-00-00-13 - SID Master: N/A (ignore-split Enabled – • Default) Figure 11: SID Update on LACP Join If both RBridges are connected to the same remote device, the remote SID should match.
• Figure 12: LACP SID Assignment
If the RBridges are configured for the same vLAG but are connected to different remote devices, the remote SID values do not match. Since the local SID state is forced to synchronize between the connecting RBridges, the side whose Local SID is forced to change ends up with disabled links.
Edge Loop Detection (ELD) Edge Loop Detection (ELD) is a Brocade Layer 2 loop detection mechanism. It uses PDUs to detect loops in the network. This protocol is mainly intended for VCS-to-VCS loop prevention operation, but it can also be used in the VCS-to-standalone mode of networks.
The ELD feature is implemented to block redundant links between two VCS fabrics: when a device detects a loop by receiving packets originated from it, it should disable all redundant links in that network. This is to prevent packet storm created due to loops caused by misconfiguration.
The primary purpose of ELD is to block a Layer 2 loop caused by misconfiguration. ELD should be used as a tool to detect any loops in the network, rather than using it to replace Layer 2 protocols such as xSTP, Metro Ring Protocol (MRP), and so on.
The basic ELD functionality is as follows: ELD is enabled on a specific port and VLAN combination. Each ELD-enabled interface on an RBridge sends an ELD PDU. This PDU contains information about the VCS ID and RBridge ID of the node it is sending from, the VLAN associated with this interface on which ELD is enabled, the ELD port priority parameter, and so on. ELD can be configured on Access mode ports and Trunk mode ports, and PDUs follow port configuration for tagging. The port priority parameter decides which port will be shut down in case ELD detects a loop. These PDUs are transmitted from every ELD-enabled interface at the configured ”hello interval” rate. When these PDUs are received back at the originating VCS, ELD detects a loop; this results in shutting down redundant interfaces based on the port parameter of the redundant links. If the port priority value is the same, then the decision is based on the port number.
ELD uses the system MAC address of the primary RBridge of VCS with the multicast (bit 8) and local (bit 7) on. For example, if the base MAC address of the primary RBridge of VCS is 00e0.5200.1800, then the destination MAC address will be 03e0.5200.1800.
Please refer to the Network OS Administrator’s Guide, v2.1.1 for more information on ELD.
Connecting the Fabric to Uplinks Upstream Switches with MCT In order to connect VCS to upstream servers with MCT, set up a normal LACP-based LAG on the MLX side (or equivalent core router), and vLAG will form automatically on the VDX side.
Upstream Switches Without MCT When the upstream devices are not running MCT or cannot support MCT, there are two ways that fabric uplinks can be connected.
Option 1: Form an EtherChannel between the Brocade VDX and upstream devices, as shown in Figure 13. In this case, standard IEEE 802.3ad-based port trunks are formed between the fabric and upstream network devices. This provids link level redundancy, but no node level redundancy. If a core Brocade VDX fails, all of the flows through the MLX/RX (Brocade BigIron® RX Series) that are connected to that device have no path to reach the fabric.
Option 2: Run STP on upstream devices with Active/Standby connections, as shown in Figure 14. Since VCS tunnels all the STP Bridge Protocol Data Units (BPDUs) through, this topology is ideal to provide both node level and link level redundancy. An important caveat to keep in mind here is that if there are more than two upstream switches connected to the fabric, Rapid STP (RSTP) will default to STP, since RSTP requires point-to-point connectivity—which is not provided in the fabric.
Connecting the Servers to Fabric Rack Mount Servers This section describes how rack mount servers, either physical or virtual, can be connected to the Brocade VCS Fabric. When connecting to the fabric in DCB mode, it is important that flow control is enabled on the Converged Network Adapters CNAs.
Blade Servers When connecting blade servers to a VCS Fabric, there are two connectivity options—embedded switches and passthrough modules. Please consult Brocade support to validate the supported solution. Not all qualified solutions have been published as of the release of this document.
Manual Port Profiles Automatic Migration of Port Profiles (AMPP) is a Brocade innovation that allows seamless movement of virtual machines within the Ethernet fabric. In today’s networks, network parameters such as security or VLAN settings need to be physically provisioned before a machine is moved from one physical server to another within the same Layer 2 domain. Using the distributed intelligence feature, a VCS fabric is aware of the port profiles associated with the MAC address of a Virtual Machine (VM) and automatically applies those policies to the new access port that the VM has moved to. Since AMPP is MAC address based, it is hypervisor-agnostic and works with all third-party hypervisors.
In Network OS v2.1.0, AMPP is supported over vLAG.
Dynamic Port Profile with VM Aware Network Automation Server virtualization (VMs, and so forth) is used extensively in current datacenter environments, with VMware being the dominant player in datacenter virtualization. The server hosts (VMware ESXs) are connected directly to the physical switches through switch-ports (or edge-ports in the case of VCS). Many of these server hosts implement an internal switch, called vSwitch, which is created to provide internal connectivity to the VMs and a distributed switch that spans across multiple Server-Hosts. A new layer called the Virtual Access Layer (VAL) virtualizes connectivity between the physical switch and virtual machine via vSwitch or dvSwitch. VAL is not visible to the physical switch;
thus, these VMs and other virtual assets remain hidden to the network administrator. The Brocade VM-aware network automation provides the ability to discover these virtual assets. With VM-aware network automation, a Brocade VDX 67XX switch can dynamically discover virtual assets and offer unprecedented visibility of dynamically created virtual assets.
VM-aware network automation also allows network administrators to view these virtual assets using the Network OS CLI. VM-aware network automation is supported on all the Brocade VDX platforms and can be configured in both VCS and standalone mode. This feature is currently supported on VMware vCenter 4.0 and 4.1.