The multi-core processors currently used in servers are capable of handling large data flows, and the memory speed of new storage technologies, including solid-state drives (SSDs), can be used to provide high-performance storage in data centers capable of running applications with increasing operational density. Attempting to leverage virtualization could reveal bottlenecks in interconnection or in data input and output. To resolve this, in principle, it would suffice to increase bandwidth by multiplying host bus adapters and adding switches, but this solution would inevitably result in increased network complexity and management costs.
With up to 86% of server workloads projected to be virtualized by 2018[1], major equipment and component manufacturers must keep pace with the demand for inbound and outbound bandwidth in the SAN. Fibre Channel remains the primary protocol used to interconnect virtual servers for storage. An 8G Fibre Channel ramp-up will soon give way to an even faster 16G Fibre Channel in core and top-of-rack storage switches, server host bus adapters (HBAs), interswitch links (ISLs), and Fibre Channel RAID controllers.
16G Fibre Channel, physical interface
The new FC-PI-5 “16G FC” fiber channel standard, finalized in 2009 by the INCITS T11.2 working group, defines the high-speed optical transceivers that will address the input and output bottlenecks that occur with intensive channel usage.
These optical transceivers comply with industry agreements on small pluggable SFP (INF-8074i) and SFP+ (SFF-8431) transceivers regarding low-speed electrical and mechanical specifications, and are backward compatible, providing a simple migration path to improve SAN performance.
By default, 16G Fibre Channel links are defined to operate at a serial line frequency of 14.025 Gb/s. Thanks to a change in data encoding, which increased the 8b/10b used in 8G Fibre Channel to a more efficient 64b/66b, the throughput of a single optical interconnect has been doubled without doubling the serial line frequency. This more efficient encoding scheme enables the link distances required in today's data centers (see Table 1) while simultaneously allowing for relatively inexpensive use of laser technology.
The standard technique defines the use of clock and data recovery (CDR) circuits to ensure good signal integrity and achieve link lengths that meet the physical requirements of growing storage and computing facilities. August 2011 marked the production availability of the first 16G FC SFP transceiver (see Figure 1). The device incorporates onboard CDR functionality in both transmit and receive directions and operates over link distances of up to 125 meters, using OM4 50/125µm multimode optical fiber.
The 16G FC SFP transceivers are capable of operating at 14.025Gb/s, as well as two legacy data rates of 8.5Gb/s (8G FC) and 4.25Gb/s (4G FC), facilitating the adoption of new technology in existing SAN infrastructure. The transmitter and receive path can operate at different data rates, as is typically required during Fibre Channel rate negotiation. Automatic rate negotiation, which requires no user intervention, is key to enabling backward compatibility with low-speed storage devices.
New Features to Facilitate SAN Management:
Digital Diagnostic Monitoring (DMI) information is present in the 16G FC SFP, Small Devices for Industry Agreement, SFF-8472. This provides real-time control information for the laser transceiver, receiver, and environmental conditions via a 2-wire serial interface (TWI).
In addition to these features, the 16G FC SFP transceivers offer several new functionalities (see Figure 2) that ensure host interoperability and improve system fault isolation. Variable electrical input equalization (EQ) and electrical output pre-emphasis (PE) settings, controlled via the TWI interface, allow users additional flexibility in optimizing electrical channels in complex systems. The system manager can select the most suitable SFP configuration for a given interconnect port on-site, optimizing link performance.
To further assist in link optimization and remote troubleshooting, the user can automatically configure the SFP to internally return local traffic to the host ASIC via SFP Electrical Out (EWRAP) or similarly return remotely generated optical data via SFP loopback to the source port via SFP Optical Out (OWRAP).
EWRAP and OWRAP give system architects options for diagnosing storage networks and isolating potential fault locations. Both WRAP functions pass the signal through the internal signal degradation and fluctuation to eliminate CDRs before retransmission. An optional function is also available to simultaneously transmit wrapped information through the module.
Summary
The primary benefit of migrating to 16G FC is increased performance. Higher-speed interconnects create faster data transfers, ensuring that high-demand, resource-intensive applications continue to have a simple migration path across the data center.
Migrating from 8G Fibre Channel to faster, more efficient 16G optical links, along with virtualized servers, will reduce the total number of cables/links managed in the SAN. Link consolidation, combined with the enhanced link diagnostics available in the new SFP, will simplify network maintenance. Reducing the overall number of devices (switches, HBAs, etc.) can further lower IT costs.
For large enterprise data centers or SANs, the improvement in watts per Gb/s should not be overlooked. Migrating from 4G or 8G infrastructure to 16G Fibre Channel can have a significant impact on electricity costs, while also helping to meet the company's environmental goals.
Data center and SAN connectivity is a key area for optical product development initiatives, including high-density products and high-speed serial solutions. The new 16G FC SFP will enable a new generation of high-density switches, ISLs, and host bus adapters. Looking ahead, the T11 working group is addressing the upcoming 32G Fibre Channel FC (FC-PI-6).
New storage-intensive applications, the growth of cloud computing, and server virtualization will continue to drive bandwidth and convergence in the SAN/data center. This evolution opens new avenues for innovation.
Author:
By Robert Hannah, Application Manager, and Randy Clark, Product Manager, Avago Technologies
