Server virtualization counters the trend of using dedicated devices as servers, a trend that was successful because these devices are useful and easy to deploy. But in general, these devices are inefficient. Because they are optimized for a single function, many are future-proof and require constant upgrades. Or perhaps worse, the proliferation of devices has created ever-increasing administrative burdens, sprawling data center networks, and high thermal loads.
The solution to the underutilization problem is virtualization, and the essence of virtualization is a software abstraction layer called a hypervisor. The hypervisor sits between the hardware and the operating system. Virtualization allows multiple operating systems and applications to coexist on a single physical computing platform. The graphic shows an example of virtualization with three logical servers on the same platform. Server virtualization, especially when combined with blade technology, increases computing and storage density while making IT assets more flexible.
The description of a virtualized server in Figure 1 implies a traditional client-server relationship with the user. Virtualization introduces an interesting separation of computing power from the user interface. This article focuses on server virtualization and its effects within the data center.
Virtualized servers will support the full range of enterprise applications, multimedia applications, storage, and management and control services. Virtualization adoption is accelerating. In February 2010, Microsoft® reported that 20% of servers shipped were virtualized. VMware®, a leading hypervisor product provider, reported that customers had virtualized 25% of their servers. No one knows exactly how much virtualization will grow, but vendors and analysts predict a threefold increase over the next five years.
Virtualization isn't limited to large enterprises. New products that unify computing, networking, and storage are designed to maximize the benefits of virtualization for cloud organizations and mid-sized businesses.
Virtualized Server Platform:
The Broadcom NetXtreme® I and NetXtreme II® high-speed controller families offer advanced network virtualization features designed to help customers build and deploy products for a wide range of application profiles. Broadcom's leading market share, with over 70% in the 1G and 10G network controller market segments, is a result of the successful implementation of network virtualization in key growth markets. These markets include high-performance Web 2.0 application servers, low-latency financial trading systems, and high-density cloud computing environments.
Network controller virtualization allows users to consolidate their network hardware resources and run multiple virtual machines simultaneously on consolidated hardware. Virtualization also provides users with a comprehensive set of features such as shared I/O, consolidation, isolation, and migration, as well as simplified management with teaming capabilities and failover.
Broadcom has collaborated with various virtual machine (VM) vendors, including VMware vSphere, Microsoft Hyper-V™, Red Hat® KVM, and Citrix® Xen, to provide a comprehensive set of network virtualization capabilities necessary for the adoption of 10GBASE-T in data center and cloud environments. The functionality and features listed below eliminate virtualization bottlenecks and improve system performance through their additional capabilities:
• Stateless Offloads: Broadcom Ethernet network controllers support stateless offloads such as:
- CP Checksum Offload (CSO), which allows network adapters to compute the TCP checksum on both
transmit and receive.
- Large TCP Send Offload (LSO), which allows the TCP layer to create a TCP message up to 64 KB in length and send it in a stack call across IP and the Ethernet device driver, preventing the host CPU from having to compute the checksum in a virtual environment.
• Jumbo Frame Support: In virtual environments, using jumbo frames saves CPU by reducing interrupts and increases performance by allowing the system to focus on the data within the frames rather than the frames surrounding the data.
• Multiple Queue Support: A Broadcom transport queue manager utilizes on-chip queuing technology with VMware ESX NetQueue. The additional overhead of route finding, data copying, and filtering is offloaded to a network adapter, where the transport queue manager can transmit packets from multiple queues and direct received packets to multiple queues. If a transmit/receive queue pair is dedicated to a virtual machine, the network adapter can provide DMA to and from the virtual machine's memory, and the vSwitch only handles the control plane operation.
• Network offloading: Static VMware, VMDirectPath, or Fixed Pass (FPT) allow Broadcom NetXtreme II network interface cards (C-NICs) to be dedicated entirely to the high-performance virtual machine. FPT uses AMD IOMMU or Intel VT-D to provide DMA data I/O from the physical device to the virtual machine, bypassing the virtualization layer and thus providing full physical access from the Broadcom CNIC to the virtual machine.
• Storage offloading: iSCSI HBA functionality enables on-chip processing of the iSCSI protocol (as well as TCP and IP protocols), freeing up host CPU resources at 10 Gbps line rates over a single Ethernet port.
• iSCSI booting: The server can boot an operating system (OS) on a SAN, completely eliminating the need for local disk storage (which is the primary source of errors in computer systems). In addition to the increased system reliability, the use of diskless servers simplifies the IT administrator's workload by centralizing the creation, distribution, and maintenance of server images, reducing the need for storage capacity with increased disk capacity utilization, and adding data redundancy through the use of replicas and data replication.
Data Center Network Design:
Virtualization affects data center networks in two important ways. The first is bandwidth demand. Consolidated server platforms require higher bitrate connections to support multiple processes. Storage also increases bandwidth demand. Multimedia content and accumulated application data increase storage overhead, and the existence of more data immediately creates the need for high-speed access.
The result is a significant need for a high-bandwidth data center network.
Base Layer: Virtualized Server Platform.
The impact of virtualization stems from the ease with which IT assets can be reused. Virtualization allows hardware to perform multiple functions and enables the transfer of those functions to other hardware platforms. This computing and storage flexibility is coupled with the flexibility of the server's networking capabilities. Figure 2 is an example of a consolidated, virtualized server platform with Broadcom® Corp.'s advanced server networking.
The physical network must adapt to the requirements and benefits that virtualization provides, specifically the increased utilization and bandwidth. To do so, forward-thinking network professionals have implemented End-of-Row (“EoR”) or Top-of-Rack (“ToR”) topologies in their data center networks.
Option 1: The End-of-Row Topology.
As a point of reference, Figure 3 depicts a pre-virtualized data center, where each asset (server, storage device, etc.) is individually connected to an Ethernet switch.
This topology uses structured cabling connections that are difficult to modify. Since virtualization facilitates changes, a network architecture that inhibits them is inherently problematic. The conventional topology is also becoming outdated, as a large number of 1-Gigabit links are incongruous with consolidated servers that require fewer, faster connections. The network must support this shift from many "fine roots" to fewer "thick roots."
The End-of-Row (EoR) network topology, as shown in Figure 4, addresses the limitations of a conventional data center network by dedicating an Ethernet switch to each row of equipment racks. The virtualized assets in each rack, in each row, are connected to an EoR rack switch. This switch also provides a backbone connection to a data center hub.
The EoR topology splits the physical and frame connections of one tier into two, making the network more adaptable. EoR limits cable lengths at the lower tier to the length of a row of racks. Shorter cables are generally easier to install and replace.
The EoR topology confines the impact of asset reconfiguration to a single row of racks, rather than an entire data center. EoR can reuse some elements of the existing physical network, although significant changes and upgrades may be required.
Option 2: The Top-of-Rack Topology.
The Top-of-Rack topology (Figure 5) is very different from the conventional architecture. It dedicates one Ethernet switch to each server rack. The ToR switch interconnects assets in each rack and provides a trunk link connection to an aggregation point in the data center.
Like EoR, the ToR topology divides the switch's physical and frame connections into two tiers. The difference lies in the granularity of the lower tier. While EoR creates modularity in a row of racks, ToR creates modularity in each individual rack.
Note that the ToR design does not limit a server rack to a single switch. The diagram above shows two switches in a rack: one primary and one for redundancy. If the Ethernet switches are deployed as blades, there could be even more switches in a rack.
The Top-of-Rack versus End-of-Row discussion is complex. Many Layer 2 and Layer 3 networking issues must be considered. Some of the relative advantages of each topology are:
Top of Rack
- Less structured cabling.
- Easier to change/expand.
- More modular.
End of Row
- Less disruptive to the infrastructure.
- Fewer switches and trunk connections.
- Easier to manage/more compatible.
Author Biographies
- David Veneski is the Director of the Data Communications Installation Business Unit at Fluke Networks. His responsibilities include product management and marketing for all copper and fiber certification solutions for enterprise networks. Previously, he was a marketing executive at Cisco Systems, Apple® Computer, and Motorola®. David holds a bachelor's degree from Rensselaer Polytechnic Institute and a master's degree from the University of Connecticut.
- Abhijit Aswath is a Senior Product Manager at Broadcom Corporation. His responsibilities include product management and marketing for all high-speed network controller software. He has extensive experience in the storage, server, and semiconductor industries. Abhijit holds a bachelor's degree in computer networking and a master's degree in business administration from North Carolina State University.
