«DATA CENTER Brocade VDX/VCS Data Center Layer 2 Fabric Design Guide for Brocade Network OS v2.1.1 DATA CENTER DESIGN GUIDE CONTENTS Introduction Building ...»
Brocade VDX/VCS Data Center
Layer 2 Fabric Design Guide for Brocade
Network OS v2.1.1
DATA CENTER DESIGN GUIDE
Building a Multi-Node VCS Fabric
Switch Platform Considerations
VCS Nuts and Bolts (Within the Fabric)
Deciding Which Ports to Use for ISLs
Brocade ISL Trunk
Brocade Long Distance ISL
ECMP Load Balancing
Configurable Load Balancing
Brocade VDX Layer 2 Features (External to Fabric)
vLAG Configuration Guidelines with VMware ESX Server and Brocade VDX
vLAG Minimum Links
LACP SID and Selection Logic
LACP SID Assignment
LACP Remote Partner Validation
show lacp sys-id:
Edge Loop Detection (ELD)
Connecting the Fabric to Uplinks
Upstream Switches with MCT
Upstream Switches Without MCT
Connecting the Servers to Fabric
Rack Mount Servers
Manual Port Profiles
Dynamic Port Profile with VM Aware Network Automation
Data Center Network and vCenter
Network OS Virtual Asset Discovery Process
VM-Aware Network Automation MAC Address Scaling
Port Profile Management
Usage Restriction and Limits
Building a 2-Switch ToR VCS Fabric
Building a 2-switch Aggregation Layer Using VCS
Building the Fabric
24-Switch VCS Reference Architecture
Appendix A: VCS Use Cases
VCS Fabric Technology in the Access Layer
VCS Fabric Technology in the Collapsed Access/Aggregation Layer
VCS Fabric Technology in a Virtualized Environment
VCS Fabric technology in Converged Network Environments
Brocade VDX 6710 Deployment Scenarios
INTRODUCTIONThis document describes and provides high-level design considerations for deploying Brocade VCS® Fabric technology using the Brocade VDX® series switches. It explains the steps and configurations needed to deploy the
A 6-switch VCS Fabric topology providing physical or virtual server connectivity with iSCSI/NAS • A 2-switch Top of Rack (TOR) topology • A 2-switch VCS topology in Aggregation, aggregating 1 GbE switches • A 24-switch VCS topology • The target audience for this document includes sales engineers, field sales, partners, and resellers who want to deploy VCS Fabric technology in a data center. It is assumed the reader of this document is already aware of the VCS Fabric technology, terms, and nomenclatures. Explaining the VCS nomenclature is beyond the scope of this document, and the reader is advised to peruse the publically available documents to become familiar with Brocade VCS Fabric technology.
BUILDING A MULTI-NODE VCS FABRICDesign Considerations
Topology Please note that for all illustrations, the Brocade MLX® is shown as the aggregation switch of choice, which is the current Brocade recommendation. However, Brocade VCS is fully standards-compliant and can be deployed with any standards-compliant third-party switch.
Multi-node fabric design is a function of the number of devices connected to the fabric, type of server/storage connectivity (1 GbE/10 GbE), oversubscription and target latency. There are always tradeoffs that need to be made in order to find the right balance between these four variables. Knowing the application communication pattern will help you tailor the network architecture to maximize network performance.
When designing a fabric with VCS technology, which is topology agnostic, it is important to decide early in the process what the performance goal of the fabric is. If the goal is to provide a low-latency fabric with the most path availability, and scalability is not a priority, then full mesh is the most appropriate topology. However, if a balance between latency, scalability, and availability is the goal, then a Clos topology is more appropriate. There are multiple combinations possible in between these two, including partial mesh, ring, star, or other hybrid networks, which can be designed to handle the tradeoffs between availability and latency. However, Brocade recommends that either a full-mesh or a Clos topology be used to design VCS Fabrics to provide resilient, scalable, and highly available Layer 2 fabrics.
Multi-node fabrics have several use case models. Appendix 1 provides details of various such models for which multi-node fabrics are targeted. Up to and including Network OS v2.0.1, direct connectivity between two separate VCS fabrics is not supported. It is currently required that you place a L2/L3 switch as a hub with multiple VCS fabrics in a spoke, to prevent loops in a network.
Clos Fabrics Figure 1 shows a two-tier Clos Fabric. Generally, the top row of switches acts as core switches and provides connectivity to the edge switches. A Clos Fabric is a scalable architecture with a consistent hop count (3 maximum) for port-to-port connectivity. It is very easy to scale a Clos topology by adding additional switches in either the core or edge. In addition, as a result of the introduction of routing at Layer 2 with VCS technology, traffic load is equally distributed among all equal cost multipaths (ECMP). There are two or more paths between any two edge switches in a resilient core-edge topology.
Figure 1: 2-Tier Clos Fabric Mesh Fabrics Figure 2 shows a full-mesh fabric built with six switches. In a full-mesh topology, every switch is connected directly to every other switch. A full-mesh topology is a resilient architecture with a consistently low number of hop counts (2 hops) between any two ports. A full mesh is the best choice when a minimum number of hops is needed and a future increase in fabric scale is not anticipated, since a change in mesh size can be disruptive to the entire fabric.
For a mesh to be effective, traffic patterns should be evenly distributed with low overall bandwidth consumption.
Figure 2: Full-Mesh Fabric
VCS-to-VCS Connectivity VCS-to-VCS connectivity is supported in Network OS v2.1.0 release. VCS-to-VCS connectivity is supported for only a certain set of topologies, due to the lack of a loop detection mechanism. It is highly recommended that VCS-to-VCS
connectivity be restricted to the following topologies only:
ELD, which is available in Network OS v2.1.1, can be used as a loop detection mechanism between • VCS fabrics. Prior to Network OS v2.1.1 the topology could not have any loops. Also, prior to Network OS v2.1.1, any local loop even within a single cluster caused broadcast storms and brought down the network. A local loop in one cluster impaced the other cluster.
Two VCS clusters can be directly connected to each other—one at the access layer and the other at the • aggregation layer.
Figure 3: VCS-to-VCS Cluster
One VCS cluster at the aggregation layer can be directly connected to up to 16 VCS clusters at the • access layer; however, access clusters must not be connected to each other.
All links connecting to the two clusters must be a part of a single vLAG, and all multicast control traffic • and data traffic should be limited to 10 Gbps within the vLAG. This limitation is due to the fact that there is no distribution of multicast traffic—multicast traffic is always sent out on a primary link.
Figure 4: VCS-to-VCS Cluster Switch Platform Considerations Brocade VCS Fabrics can be designed with the Brocade VDX 6720-24 (16/24 port), VDX 6720-60 (40/50/60 port) switches, VDX 6710 (48×1 GbE Copper + 6×10GbE SFP+), VDX 6730-32 (24×10 GbE SFP+, 8×8 GbE FC ports), and VDX 6730-76 (60×10 GbE SFP+, 16×8 GbE FC ports). The Brocade VDX 6720-24 switch provides a single ASICbased architecture delivering constant latency of ~600ns port-to-port. The Brocade VDX 6720-60 switch is multiASIC based, with latencies ranging from 600 ns to 1.8 us. When designing a Clos topology, using the higher port count switches in the core enables greater scalability. The Brocade VDX 6710 provides cost-effective VCS fabric connectivity to 1 GbE servers, while the Brocade VDX 6730 connects the VCS fabric to the Fibre Channel (FC) Storage Area Network (SAN).
In addition, it is important to consider the airflow direction of the switches. The Brocade VDX is available in both port-side exhaust and port-side intake configurations. Depending upon the hot-aisle, cold-aisle considerations, you can choose the appropriate airflow.
Please note that the Brocade VDX 6710 does not require Port On Demand (POD) or Fibre Channel over Ethernet (FCoE) License, and the VCS fabric must be through 10 GbE ports. However, 10 GbE ports may be used to connect to servers. These server-facing ports will not support Data Center Bridging (DCB). Lastly, VCS fabric is supported only with Brocade VDX 6710.
Oversubscription Ratios Brocade VDX switches do not have dedicated uplinks. Any of the available 10 GbE ports can be used as uplinks to provide desired oversubscription ratios. When designing a mesh network, the oversubscription ratio is directly dependent on the number of uplinks and downlinks. For example, a 120-port, non-blocking mesh can be designed with four 60-port Brocade VDX switches. Each Brocade VDX switch has 30 downstream ports and 30 upstream ports (10 connected to each of three Brocade VDX switches). In this case, the oversubscription ratio is 1:1, as there are 30 upstream ports serving 30 downstream ports. In the case of a two-tier Clos topology, there are two levels of oversubscription, if the fabric uplinks are connected to the core layer. For North-South traffic, oversubscription is the product of the oversubscription of core switches and edge switches. For example, if the core switch with 60 ports
has 20 uplinks and 40 downlinks, then it has an oversubscription ratio of 2:1 (40:20). Furthermore, if the edge switch also has the same oversubscription, then the fabric oversubscription for North-South traffic is 4:1. For EastWest traffic, or if the uplinks are also connected at the edge, the oversubscription is dependent only on the oversubscription of the edge switches, in other words, 2:1 in this example.
Scalability When designing the fabric, it is important to consider the current scalability limits mentioned in the release notes of the software running on the switches. These scalability numbers will be improved in future software releases without requiring any hardware upgrades. This can be referenced in the release notes.
The Brocade VDX series of switches allows for the creation of arbitrary network topologies. Because of this flexibility, it is not possible to cover all architectural scenarios. Therefore, the scope of this document is to provide a baseline of architectural models to give the reader a sense of what can be built.
Implementation Now that the topology, switch, oversubscription, and other variables have been decided, you can decide how to build this network. As mentioned earlier, the Brocade VDX series of switches allows for the creation of arbitrary network topologies. Because of this flexibility, it is not possible to cover all architectural scenarios. Therefore, the scope of this document is to provide a baseline of architectural models, to give the reader a sense of what can be built. This document describes one such model—how to build a core-edge network using 4×24 port edge switches and 2×24 port switches in core with a 4:1 oversubscription ratio. This document discusses how to connect various types of servers and storage to this fabric and how to connect the fabric to upstream devices. Figure 5 shows the basic design of this topology, and Table 1 lists the equipment required for this design.
Table 1: Equipment required for a 6-Switch VCS Fabric Solution The topology used in the above example is a sample topology. Brocade VCS fabrics can be built with a mix and match of any number of switches within the scalability limits of the software version running.
VCS NUTS AND BOLTS (WITHIN THE FABRIC)Deciding Which Ports to Use for ISLs Any port can be used to form an Inter-Switch Link (ISL) on the Brocade VDX series of switches. No special configuration is required to create an ISL. There are two port types supported on the Brocade VDX switches—edge ports and fabric ports. An edge port is used for any non-VDX-to-VDX connectivity—regardless of whether that is a network switch external to the Brocade VDX when running in VCS mode or whether the port is used for end-device connectivity. A fabric port is used to form an ISL to another Brocade VDX switch and can participate within a Brocade VDX ISL Trunk Group.
ISL Trunking When determining which ports to use for ISL Trunking, it is important to understand the concept of port groups on Brocade VDX switches, as shown in Figure 6. ISL Trunks can only be formed between ports of the same port group. In addition, the cable length should be the same to connect the ports forming the ISLs.
Figure 6: Port Groups on the Brocade VDX 6720-60 Figure 7: Port Groups on the Brocade VDX 6720-24
Figure 8: Port Groups on the Brocade VDX 6710-54 Figure 9: Port Groups on the Brocade VDX 6730-32 Figure 10: Port Groups on the Brocade VDX 6730-76 When building a fabric, it is very important to think in terms of ISL bandwidth. Brocade ISL Trunks can be formed using up to 8 links providing up to 80 Gbps of bandwidth. The throughput of ISLs are, however, limited to 80 mpps (million packets/sec), which results in lower bandwidth for small-size packets.
Once it has been decided which ports you will use to form an ISL, VCS needs to be enabled, and the RBridge ID must be defined. The VCS ID needs to be assigned for each switch that will become part of the fabric. The default VCS ID is 1, and the default RBridge ID is 1. Keep in mind that it is disruptive to change these parameters, and a switch reboot will be required. During the reboot process, if there is no predefined fabric configuration, the default fabric configuration will be used upon switch bring-up. Once the switches are in VCS mode, connect the ISLs and the VCS fabric will form. Lastly, please also reference the section on Upgrade/Downgrade Considerations—VCS Fabric Functionality.
Brocade ISL Trunk A Brocade Trunk is a hardware-based Link Aggregation Group (LAG). Brocade Trunks are dynamically formed between two adjacent switches. The trunk formation, which is not driven by Link Aggregation Control Protocol (LACP), is instead controlled by the same FC trunking protocol that controls the trunk formation on FC switches that use the Brocade Fabric OS (FOS). When connecting links between two adjacent Brocade VDX 6720s, Brocade Trunks are enabled automatically, without requiring any additional configuration. Brocade Trunking operates at Layer 2 and is a vastly superior technology compared to the software-based hashing used in standard LAG, which operates at Layer
2. Brocade Trunking evenly distributes traffic across the member links on a frame-by-frame basis, without the need for hashing algorithms, and it can coexist with standard IEEE 802.3ad LAGs.
Brocade Long Distance ISL Normally, an ISL port with Priority Flow Control (PFC) is supported for a 200 m distance on eAnvil-based platforms.
The longdistance-isl command extends that support up to a distance of 10 km, including 2 km and 5 km links.