Virtualization topologies and cabling
Ethernet switch vendors offer a wealth of information on network design for virtualized services. Regardless of the chosen topology, three characteristics related to the connectivity infrastructure will be common:
• The wiring will change.
• The cabling will outlast many computing, storage, and network connection assets.
• Data center network cabling failures can affect even virtualized services.
Since virtualized services require a rethinking of data center connectivity, it is helpful to recognize the cable categories created by End-of-Row and Top-of-Rack topologies.Currently, some device-to-switch links may be sufficient with 1 Gigabit, but the numerous advantages of higher bit rates make future-proofing a wise investment. The good news is that there are many ways to support 10 Gigabit traffic. Table 2 shows the most common 10 Gigabit solutions.
• Twinax: A shielded cable initially used for InfiniBand storage, Twinax supports high bit rates over modest distances. Twinax cables become expensive in longer setups and are less flexible than twisted-pair copper cables. The Twinax SFP+ cable is terminated with compact SFP+ modules instead of large CX4 connectors.
• Unshielded Category 6 Twisted Pair: The IEEE approved the 10GBASE-T standard in 2006 for 10 Gigabits over twisted-pair cable, including shorter Cat 6 UTP links. 10GBASE-T extends the standardization and reduced cost of UTP up to 10 Gigabits. It is backward compatible with Cat 5e/100 Mbps and supports all Ethernet features. Cat 6 patch cords for in-rack applications can be purchased pre-terminated or terminated on-site. Rack-mounted Cat 6 cable is terminated on-site.
• Unshielded Category 6A Twisted Pair: The upgrade to Cat 6 was officially recognized in 2008 in the TIA 568-B.2-10 standard, which specified cable performance up to 500 MHz and defined requirements for alien crosstalk performance. Category 6A cables are thicker and more expensive than Category 6 cables, but they support 10 Gigabit traffic up to 100 meters. Some Cat 6A cables have shielding around each pair of wires for alien crosstalk interference control. Cat 6A patch cords can be purchased pre-terminated for on-site termination. Cat 6A for rack-to-rack or data center applications is terminated on-site.
• Category 7 Shielded Twisted Pair: Although more accurately termed “Class F” as specified in ISO/IEC 11801, Category 7 STP predates Cat 6A and was specifically designed for high-bit-rate applications. Cat 7 mitigates alien crosstalk interference with metallic shielding around each pair of wires and shielding for the entire cable. Cat 7 cable is the most expensive twisted-pair copper solution and requires an additional installation step for shield termination. The inherent noise immunity of the Cat 7 shielding provides better performance than UTP and is theoretically more future-proof. Cat 7 cables do not use standard RJ-45 connectors, so they must be terminated on-site.
• 10GBASE-SR: This is a common fiber option for data centers when coupled with 850 nm multimode transceivers. 10GBASE-SR can be deployed using SFP+ modules to save space and connector power. The maximum distance for 10GBASE-SR is determined by the fiber used. OM1 and OM2 fiber support 10 Gigabits up to 80 meters. The newer OM3 cable supports 10 Gigabits up to 300 meters.
• 10GBASE-LRM: LRM was designed for 10 Gigabit traffic using 1310 nm lasers over multimode fiber for long distances. The IEEE 802.3aq standard was approved in 2006. The cables are duplex FDDI fiber with SC or LC connectors. Patch cords can be purchased with connectors already attached, or connectors can be spliced on-site. Fibers for longer-distance installations are spliced with connectors on-site.
• 10GBASE-LR: LR is the most common option, using single-mode fiber. The cables are duplex fiber with SC or LC connectors. Distance and cost are directly related, making 10GBASE-LR modules very expensive. The numerous newer alternatives make 10GBASE-LR a less desirable choice for virtualized data centers, but it remains a viable option for campus networks.
The Rise of 10GBASE-T:
While the 10GBASE-T standard was ratified in 2006, market adoption was delayed due to density, heat, and power consumption issues that slowed implementation in switches and network interface cards. But recent advances in silicon have resolved these problems, and the rollout of 10GBASE-T is rapidly accelerating.
Broadcom, a market leader in silicon PHY, NIC, and switch networking, began supplying 10GBASE-T solutions in late 2008. Today, virtually every Ethernet vendor offers a 10GBASE-T-capable product, including Cisco® Systems, which announced 10GBASE-T support in January 2010. Server vendors are expected to adopt 10GBASE-T motherboard interfaces soon.
Because 10GBASE-T runs over CAT6/6A/7 copper, it can be used in in-rack, rack-to-rack, and inter-data center applications. 10GBASE-T has the unique characteristic of working well in both Top-of-Rack (ToR) and End-of-Rack (EoR) topologies. It supports automatic data rate negotiation, allowing for gradual migration to 10 Gigabit speeds. While 10GBASE-T has not yet replaced 1GBASE-T as the standard for data center network connectivity, its cost-effectiveness and flexibility make it a likely successor.
Guarantee of success: the cable testing reference model and certification.
Topology and economics are the main determinants of connectivity, but any option requires a reliable cabling infrastructure. Although cable is a passive technology and vendors provide warranties, you should keep in mind that:
• Warranties expire.
• The manufacturer's warranty likely does not cover installation labor.
• Any cable failure could result in service interruptions.
• Repairing a problem is always more expensive than preventing it.
To ensure the proper delivery of virtualized services, the cabling infrastructure must undergo certification testing. Certification is a rigorous evaluation of connectors, installation, and cables performed before the network is put into service. The results of the certification tests are compared to industry standards, resulting in a "Pass" or "Fail" rating for each link. A link that passes certification meets the defined performance specifications. Links that fail certification are repaired, typically at the expense of the network vendor or installer.
The certification traditionally focuses on structured cabling, but if Top-of-Rack topology is used, it can include patch cords used for cables within the rack.
Copper Certification
• Twinax: There is no testing standard for Twinax cables, so they cannot be formally certified.
Network users have no choice but to rely on the manufacturer's warranty or treat Twinax cables as disposable.
• Unshielded twisted pair (UTP): Cat 6/6AUTP certification is performed in two testing phases: channel and alien crosstalk (ATX). All UTP cables, for in-rack, rack-to-rack, and inter-data center applications, must undergo channel testing. This testing certifies thirteen parameters defined by the TIA/EIA-568-B and ISO 11801 standards.
UTP cables used for rack-to-rack and data center links must also undergo alien crosstalk testing. This is a sample test; the ISO/IEC standards suggest 1% or 5% of links, whichever is greater. Alien crosstalk testing ensures that cross-coupling does not affect network performance. Existing Cat 6 cabling up to 50 meters in length can be recertified for 10GBASE-T using the criteria defined in TIA Telecommunications Systems Bulletin (TSB) 155.
• Shielded Twisted Pair: Cat 7 (also called Class F) certification is a channel test.
The ISO 11801 standard specifies the parameter limits and frequency range for this test. Since Cat 7 cable does not use RJ-45 connectors, the cable tester must include an adapter compatible with each vendor's unique connector.
The setup for certifying copper links is shown in Figure 6. A certification tester and its remote unit are connected at both ends of the link to perform all aspects of the certification.
It is important to document the certification test results. A comprehensive, accurate, and easy-to-understand report serves to demonstrate that the infrastructure meets the standards and specifications prescribed by data center management.
A Cat 6A certification test report, showing the link's performance for each test parameter, is shown above.
Fiber Certification.
Basic or Level 1 fiber certification is a loss/length test performed with an Optical Loss Testing System (OLTS). The OLTS measures the loss in the fiber link and compares it to a loss threshold based on the length and bandwidth established by the relevant standard, such as TIA-568-C.0, Generic Telecommunications Cabling for Customer Premises. Based on these measurements, the OLTS displays a Pass or Fail rating for the link. This test is used for both multimode and single-mode fiber.
A detailed results report is essential for fiber certification, just as it is for copper. Figure 9 shows an example.
If a fiber optic link fails the Level 1 test, an optical time-domain reflectometer (OTDR) must be used to identify problem areas. An OTDR is a single-ended troubleshooting and testing instrument that performs a detailed evaluation of each component of a fiber optic link. An OTDR emits pulses of light down the fiber and measures the reflected light. Based on the relative intensity of the return pulses, the loss is plotted as a function of fiber length. This reveals the location of connectors and faults, measures losses, and determines the link length.
An OTDR is a powerful tool for certifying fiber before use. It offers the advantage of measuring losses at each connector and along each cable segment. Figure 10 shows an OTDR trace evaluating the fiber under test.
The "Pass" result indicates that this fiber link meets the specified standards. As previously mentioned, OTDR traces are essential for documenting installation quality and troubleshooting fiber issues in the event of a loss/length certification failure.
Conclusion:
As virtualization advances, it generates fundamental changes in data center networks that are inevitable and, in many cases, desirable. To provide reliable bandwidth for virtualized assets and end users, 10 Gigabit Ethernet will be employed in the virtualized data center. The use of 10 Gigabit Ethernet is important because it is a way to prepare the data center network for the coming years while meeting the immediate need for compatibility with virtualized servers and services.
10 Gigabit Ethernet can be deployed using a variety of copper and fiber options. The 10GBASE-T standard and the newer silicon 10GBASE-T open the door to cost-effective 10 Gigabit Ethernet deployments throughout the virtualized data center. Regardless of the Layer 1 technology chosen, the transition to 10 Gigabit Ethernet requires foresight, detailed planning, and a methodology for testing and troubleshooting.
Fluke Networks certifies 10-GbE copper and fiber cabling installations with Broadcom's range of industry-leading data center networking technologies.
Author Biographies
- David Veneski is the Director of the Data Communications Installation Business Unit at Fluke Networks. His responsibilities include product management and marketing for all copper and fiber certification solutions for enterprise networks. Previously, he was a marketing executive at Cisco Systems, Apple® Computer, and Motorola®. David holds a bachelor's degree from Rensselaer Polytechnic Institute and a master's degree from the University of Connecticut.
- Abhijit Aswath is a Senior Product Manager at Broadcom Corporation. His responsibilities include product management and marketing for all high-speed network controller software. He has extensive experience in the storage, server, and semiconductor industries. Abhijit holds a bachelor's degree in computer networking and a master's degree in business administration from North Carolina State University.
Authors: David Veneski, director of the data communications installation business unit at Fluke Networks;
and Abhijit Aswath, senior product manager at Broadcom Corporation
