Since its inception, eBay has been building a portfolio of 12 data centers to keep pace with its ever-increasing computing load.
It is not surprising, therefore, that the operating costs of eBay's data processing centers also increase every year as the business continues to grow.
In response, eBay created and implemented a four-year plan for its data centers with an ambitious goal: to cut energy costs in half and double computing performance. At the same time, this new infrastructure design would create greater operational agility and reliability.
To achieve these goals, eBay would consolidate its assets into a smaller number of data centers, each designed from the ground up for maximum energy efficiency without compromising uptime.
“A successful data center strategy assumes that computing demand will grow every year, but allows for that growth without a proportional increase in operating expenses,” explains Dean Nelson, eBay’s head of data center strategy and operations.
“We need to break the linear relationship between computing load and operating costs, which requires regular, disruptive leaps in our energy efficiency, computing capacity, and server utilization profiles.”
In May 2010, eBay opened its flagship facility and the cornerstone of its new data center strategy, Project Topaz.
Located in South Jordan, Utah, Topaz is eBay's largest capital investment ever, and will house more than a third of its global server infrastructure. Despite having all the redundancies required for a Tier IV data center, Topaz boasts an energy efficiency ratio of only 1.4.
Maximum rack efficiency is achieved through best practices.
By employing innovative, energy-efficient mechanisms in its infrastructure, such as improved architectural design, water-saving devices, rainwater harvesting, and dynamic controls in the electrical and cooling subsystems, Topaz's facilities provide energy and cooling to the elevated floor at minimal cost.
In addition to these on-site measures, Topaz has also implemented many best practices in the raised floor space, working closely with the chiller plant to contribute to the LEED Gold design.
There are five features of the Topaz raised floor that are particularly compelling for their value and applicability to all modern data centers:
1. Hot Aisle Containment:
To achieve maximum cooling efficiency, eBay employed a meticulous hot aisle containment design, with a carefully calculated prototype prior to construction.
At its San Jose headquarters, eBay's data center architecture team built full-scale models of various containment designs and experimented with different strategies (hot aisle containment versus cold aisle containment), materials (containment doors versus plastic cladding), and dimensions. This design process determined an optimal balance of cost, flexibility, and reliability before widespread implementation. eBay found that no other hot aisle containment provided a more generous time buffer than cold aisle containment in the event of a CRAC failure. The team deployed containment aisles composed of 45U cabinets and partnered with the distributor to ensure a modular design while minimizing leakage.
2. 400V Distribution for Each Cabinet:
With several blade servers located in certain cabinets, eBay wanted any rack to be able to provide loads of up to 17 kW. While this can be achieved with a traditional 208V distribution system, Topaz significantly reduces costs by extending the high-voltage distribution to all cabinets.
By supplying 400V of power to each cabinet, eBay reduces down-stage transformation and losses in cable transmissions, as well as copper usage.
Specifically, eBay provides 400V three-phase power to two Raritan Smart Rack Power Distribution Units (also known as plug strips or rack PDUs) in each rack. The rack PDUs, with internal line-to-ground (three-phase WYE) wiring, supply 240V single-phase power to each server, within the operating range of virtually any IT equipment.
By eliminating unnecessary voltage transformations, the 400V power supply reduces energy costs by 2-3% compared to 208V distribution and by 4-5% compared to 120V distribution.
As a global distributor of rack power distribution units, Raritan offers a wide range of product configurations that accept a 400V outlet. Although 400V distribution is relatively new in North America, Raritan has deployed 400V systems that are the nominal standard in Australia and many European countries.
3. Detailed Server Power Instrumentation
Arguably, the most important economic metric for the data center operations team is the total cost per search. And because server CPU capacity grows so much with each new generation of products, eBay replaces its servers every two years, as the performance per watt outweighs the monetary costs of the new assets.
With each server acquired, however, eBay can only reduce its cost per search by taking into account all the server's operating costs, not just the capital expenditure. In particular, energy costs are the largest operating expense for any asset.
To maximize savings, eBay installed Raritan rack PDUs that provide accurate energy consumption data for each server's power supply. Compliant with IEC/ANSI billing standards, Raritan's Dominion PX rack PDU family delivers continuous, sub-1% error-rate kilowatt-hour (kWh) data for every asset in the data center. Built with IT and facility standards in mind, Dominion PX can pass this data to building management and real-time asset tracking systems, achieving what Green Grid defines as PUE Category 3 (PUE3) monitoring.
“Dominion PX can give me the precision I need to calculate my true operating costs for each server down to the last penny,” Nelson says. “I can understand the efficiency profiles and power requirements of all the manufacturers that are amortized in the next server RFP.”.
4. Granular Temperature Instrumentation
After incorporating many state-of-the-art mechanisms into its cooling infrastructure, Topaz also improved its cooling efficiency, both in cost savings and reliability, with a real-time understanding of the operating environment of all servers.
Although most data centers monitor setpoint and return temperatures, this data only approximates the actual server environment. And these approximations require operators to create safety margins and cooling gaps, which are, by definition, wasteful.
In contrast, eBay knows the exact inlet and outlet temperature of every rack in the Topaz data center. Specifically, Topaz adheres to ASHRAE recommendations for server environment monitoring, measuring the cold aisle at the top, middle, and bottom, in addition to the hot aisle. Because this information can be constantly monitored, eBay can fine-tune its cooling parameters to optimize efficiency while ensuring instant alerts for any server failures.
Typically, deploying this level of environmental monitoring in every rack would be prohibitively expensive. However, the entire Dominion PX family of smart rack PDUs provides this granular instrumentation without additional infrastructure costs beyond the plug-and-play temperature sensors themselves.
The existing network layer provided by Raritan's rack PDUs is ready to measure and monitor humidity and temperature data
5. Overhead Power Distribution Ductwork:
Most data centers have a maze of cables under the floor, running from the panels to each cabinet. However, this design severely impedes airflow and impairs cooling efficiency. Because this is usually located under the raised floor, the issue is often overlooked and can become material.
Furthermore, a standard design with dedicated underfloor connections for each circuit breaker also consumes a significant amount of copper unnecessarily and underutilized, while simultaneously limiting flexibility when installing new cabinets.
To solve this problem, Topaz uses an overhead power distribution conduit so that any cabinet can be easily connected to a 400 A power source. Two busbars run the length of each row of cabinets, each supplying redundant power and thus completely eliminating the need for underfloor connections. Receptacles can be added anywhere along the row of cabinets in minutes by installing a junction box protected by a circuit breaker, instead of waiting several days for an electrician to install additional outlets.
Summary:
eBay is a company that prioritizes reliability, as downtime costs amount to $2,000/second or $120,000/minute. The sheer scale of eBay's operations means they also need to keep equipment and electricity costs under control. To achieve this, eBay has developed a data center model that makes its computing operations highly reliable and 50% cheaper than the data centers they are decommissioning.
In addition to minimizing failures and reducing operating costs, eBay's facility in South Jordan (Utah, USA) has achieved the gold distinction for Leadership in Energy and Green Design (LEED) from the US Green Building Council.
However, the flexibility designed for eBay's data center is no less important than the LEED efficiency gold badge or Level IV reliability.
This is a data center that can adapt as requirements and equipment change, and, especially for eBay, as demand requires space for future growth.
By implementing the IT industry best practices exemplified by eBay, even smaller data centers can improve uptime, reduce costs, and become more efficient by using existing equipment and simple processes and techniques.
Author:
Article provided by Raritan
