Furthermore, increasingly complex scientific, technical, and financial computing applications require ever more powerful supercomputers and ever larger high-performance clusters. The installation of major mass numerical processors, such as Cray's "Jaguar" at Oak Ridge National Labs, IBM's "Roadrunner" at Los Alamos National Labs, and Sun's "Ranger" at the Texas Advanced Computing Center, can utilize over a thousand cables and kilometers of cabling to complete all the connections to servers, switches, and storage devices.
Advances in multi-core processors, virtualization, and consolidation, along with increased host bus speeds, memory, and storage capacity, have led to a greater range of available features that designers can integrate into systems. However, this type of technology places even greater strain on bandwidth capacity, power consumption, and temperature control. When using high-speed I/O ports, space limitations must also be considered. Rack sizes don't appear likely to increase, so the possibilities for increasing the number of I/O ports and bandwidth involve either increasing port density along the switch circuit or using higher signal rates on each line.
High-speed I/O solutions:
The good news is that the I/O systems developed to date already address many of these needs. Leading companies in the sector and some industry organizations are at the forefront of developing specifications that ensure compatibility and network functionality for hardware connections, signaling, and software communication. Organizations such as the Optical Internetworking Forum, the Infiniband Trade Association, the IEEE 802.3 working groups, INCITS T10 and T11, and the SFF committee have made significant contributions.
Currently, SFP+ connections are replacing SFP for Ethernet and Fibre Channel. The SFP+ system occupies the same board space as SFP but offers ten times the bandwidth for Ethernet (10 Gb/s vs. 1 Gb/s) and twice the bandwidth for Fibre Channel (8.5 Gb/s vs. 4.25 Gb/s). The SFP+ system also allows for the free assignment and configuration of any available system port using copper or fiber cabling, depending on the installation requirements.
New and constantly evolving connections are enabling the development of ports with even greater bandwidth. The QSFP+ system uses a 4 x 10 Gb/s configuration for a 40 Gb/s port. The CXP system offers 12 lanes that can support between 100 and 120 Gb/s of bandwidth. QSFP+ and CXP are specified for interconnected InfiniBand Quad Data Rate (QDR) 4x and 12x connections and are expected to be usable for 40 Gb/s and 100 Gb/s connections by the end of the year, when the IEEE 802.3ba specification is released.
Comparing the relative bandwidth density of the three different port types reveals that QSFP+ and CXP can significantly increase the I/O port bandwidth density along the switch circuit. A single SFP+ port operating at 10 Gb/s provides approximately 16 Gb/s of bandwidth per inch, QSFP+ offers a threefold increase (48 Gb/s per inch), and CXP provides 113 Gb/s per inch (a 2.3x increase). Implementing grouped, belly-to-belly, and dual port configurations allows system designers to achieve even higher bandwidth densities with certain port types.
SFP+, QSFP+, or CXP host ports can operate with passive copper-based cable for lengths of 5 to 7 meters (or longer depending on acceptance criteria), active equalized copper-based cable for lengths up to 15 meters (or longer depending on acceptance criteria), or a connectable optical transceiver with an optical connector that allows for passive fiber optic cable connections, further extending cable lengths. This approach allows the installer and user to tailor the cable connection configuration to the specific installation being served.
With the evolution of QSFP+ and CXP, used for Infiniband QDR connections in high-performance computing (HPC) systems, a new cable option known as active optical cable (AOC) has emerged. In an AOC connection, the optical fiber terminates directly into an optical transceiver inside the metal housing located at each end of the cable. The integrated electro-optical connection reduces costs by minimizing the number of components and providing an external electrical interface. These end-to-end solutions are gaining increasing traction in HPC installations because they lower acquisition and operating costs, eliminate interoperability issues, increase eye safety, and improve issues related to optical connector contamination and cleaning; they also enhance reliability. One of the most recent forecasts predicts that the AOC segment will grow by almost 39% annually, reaching $192 million in 2013.
Goals for the future
The debate among various industry organizations has just begun, but it already seems to indicate that the speed of electrical channels will continue to increase. Ethernet, Fibre Channel, and InfiniBand appear to be aiming for 25–28 Gb/s for future high-speed I/O port lines. Considering these speeds, much remains to be done to improve connector and connection capacity before form factor and specification issues are resolved. What we can anticipate, however, is an increase in port bandwidth density. For example, a more compact QSFP form factor with 4 x 25 Gb/s could eventually replace the original 10 x 10 Gb/s configuration for 100 Gb/s connections.
Optical interconnects are expected to continue gaining importance as the technical difficulties and cost of 25–28 Gb/s copper lines begin to limit their use. Some predictions estimate that the maximum length of directly connected passive copper cables will be between 2 and 3 meters. This limitation may well spur the adoption of fiber optics in data centers, whether for integrated AOC connections or as modules for individual transceivers and optical cables.
