As a result of this increase, data centers are generating more and more heat, and air conditioning (HVAC) systems have to work harder to keep them cool. This, in turn, increases energy costs and is becoming a growing concern for IT departments, as the associated costs could exceed any budget—and it takes advantage of the Green IT Guidelines being launched by prominent organizations in green IT centers.

There is a clear need to monitor energy consumption and temperatures in data centers, as well as to adjust heating, cooling, and ventilation levels to minimize consumption without compromising the uptime of IT systems. But what is the first step forward-thinking corporations should take? What tools are available to obtain the data needed to design a more efficient and environmentally friendly data center?

This white paper, the first in a series, addresses some of the most challenging energy management issues and provides answers. Readers will learn three things:

- Their knowledge gaps regarding what they thought they knew about energy measurement.
- What types of measurement instruments are needed to calculate the energy efficiency of a data center.
- How to establish criteria that bring their data center into compliance with the Green Grid and facilitate their commitment to a more environmentally friendly IT environment.
The Challenge
: When energy management wasn't a top priority, IT managers predicted the total power consumption of their data centers using simple calculations based on equipment specifications.

However, according to HP's "HP Power & Cooling" document, dated August 31, 2006, the average power consumption of a data center increased from 2.1 kilowatts per rack in 1992 to 14 kWh per rack in 2006. Simply multiplying the number of servers by their nominal power consumption does not reflect actual consumption statistics. A survey conducted by DatacenterDynamics at its Spring 2007 conference showed that the average power consumption of all respondents in the US was 5.5 kWh per rack, with an average peak consumption of 11.6 kWh per rack.

On the other hand, many industry analysts, such as Gartner (“25th Annual Data Center Conference,” December 2006), have been predicting significant power supply shortages for over a year. While some companies have the flexibility to relocate their data centers to locations with more reliable and cheaper power, not all can. This means that unresolved discrepancies could force IT and facilities managers to make difficult decisions about maintaining some applications and sacrificing others to keep the business running during peak demand.

With the increasing attention being paid to energy-related issues and the emergence of tools capable of making precise measurements, IT administrators and facility managers no longer have to blindly rely on the nominal consumption figures indicated on their units' nameplates or base their calculations on industry-accepted assumptions. While approximate values ​​were previously acceptable, the looming energy crisis necessitates working with precise figures.

Only by measuring the individual energy consumption of their servers can administrators accurately determine the actual energy use of their equipment and obtain figures that allow them to plan energy-saving measures.
With the data obtained from real-time, individual server monitoring, administrators can manage more effectively and feel more confident when choosing which equipment to shut down, as they can:

- Identify idle processing resources.
- Identify low-performance processing resources (which consume a lot of energy in exchange for reduced computing power).
- Support capacity planning when designing infrastructure for all cooling, protection, and power distribution equipment.

Nominal Capacities and Assumed Data:
Server manufacturers list the power consumption of each server on its nameplate. However, data center managers know that this figure always represents the maximum possible power consumption and that the typical power consumption of a server never reaches the nameplate rating. An easy way to increase server density is to subtract a certain percentage from the nameplate rating, which depends on the workload imposed on the server.
PTS calls this reduced value the equipment's "design power consumption." It is generally 50 to 75 percent of the nameplate rating and is used to size all cooling, protection, and power distribution systems in the data center. Since this percentage is often still higher than the actual consumption, smart PDUs are now used to obtain much more accurate data.

For example, by measuring the electrical consumption of individual servers using smart PDU equipment, PTS's study of Raritan's computer equipment determined that its usual consumption was between 20 and 85 percent of its theoretical (nominal) consumption, with an approximate average of 31 percent, that is, much lower than the nominal consumption of the equipment.

IT Equipment Power Consumption vs. Total Facility Power Consumption
Another important metric is comparing IT equipment power consumption to the total facility power consumption. APC-MGE estimates that 30 percent of a data center's total power consumption is attributed to servers and other IT equipment, while the remaining consumers (cooling, ventilation, power distribution, lighting) account for the other 70 percent. According to EYP Mission Critical Facilities, a division of HP, these figures are split equally, 50/50. However, a preliminary analysis by PTS and Raritan showed that 71 percent of the consumption was attributed to critical computing workloads, but only 29 percent to support systems. The high percentage of energy dedicated to critical computing tasks indicates that PTS and Raritan manage energy in their data centers more efficiently than the industry average. It also suggests an opportunity to improve performance—more computing power with less energy.
This data illustrates the importance of obtaining accurate measurements in data centers, rather than relying on "official industry averages".

Measurement Systems
: How can you perform these measurements efficiently and accurately? Here are some of the pieces of the puzzle to consider:
Branch Circuit Monitors and Individual Device Load Measurement.
Branch circuit monitors are electrical devices that measure the current load on all circuits in an electrical panel and alert operators when the load approaches the circuit breaker's rated limit. This prevention is especially important, for example, in data centers, where additional servers might be plugged into a circuit that is already nearing its maximum capacity. Branch circuit monitors continuously measure the current present on all circuits and ensure that someone is notified before a circuit breaker reaches its rated capacity, or worse, trips.

Environmental Data Collectors in Data Centers:
Environmental data collectors are specifically designed to gather environmental and power supply data from data centers. They can also consolidate and analyze the collected information to help make informed decisions about power consumption for both IT equipment and facilities.
Smart PDUs:
Smart PDUs allow IT staff to monitor the power consumption of any server, storage unit, or other IT device. This is very useful for identifying underutilized equipment and identifying equipment approaching or exceeding its normal maximum utilization levels. They also offer facility-level monitoring and control of data center power consumption.

Smart PDUs can be remotely controlled via a web browser or a command-line interface (CLI). They can measure electrical flow at both the PDU level and at individual outlet levels, allow for user-defined alarm thresholds, provide security through passwords, authentication, authorization, and encryption, and incorporate comprehensive environmental management features. Some models are also highly adaptable, incorporating the latest industry-standard techniques such as SNMP TRAPs/SETs/GETs, IPMI, and SMASH CLP, and can be seamlessly integrated into existing corporate infrastructures such as LDAP servers, Active Directory®, RADIUS, and NFS.

However, these tools cannot reach their full potential if the model is based solely on a static environment. In the real world, data centers are dynamic. Server utilization levels change over time, and with them, power consumption and heat generation and dissipation also vary. This, in turn, requires appropriate cooling for specific racks or computer groups.

Therefore, the next step is to determine the energy efficiency of the data center.
The Green Grid organization: Calculating the PUE Value.
The Green Grid organization defines Power Usage Effectiveness (PUE) with this formula:

PUE = Total Facility Energy Consumption / IT Equipment Energy Consumption.

In this equation, "Total Facility Energy Consumption" is the energy required to keep the entire data center running, including servers, IT equipment, lighting, cooling, ventilation, etc. "IT Equipment Energy Consumption" represents the energy consumed solely by the servers and IT equipment.
The PUE value can range from 1.0 to infinity. A PUE value close to 1.0 would indicate 100 percent efficiency (meaning all the electrical energy would be consumed solely by IT equipment). Currently, there are no comprehensive studies demonstrating the actual distribution of PUE values ​​in data centers. Some preliminary work done by PTS seems to indicate that many data centers may have a PUE of 3.0 or higher, but with proper design a PUE value of 1.6 could be achieved.

Tools for Calculating PUE    
When calculating the PUE value of your data center, it is necessary to accurately measure actual power consumption to establish a baseline for quantifying improvements. To make this possible, Raritan and PTS designed and implemented the following measurement strategy.

A Veris H8820 data capture device was installed to measure the secondary circuit amperage of each load and calculate the power supplied to the air conditioning units (CRAC), lighting, uninterruptible power supplies (UPS), and other non-IT consumers in the data center. This unit was selected because it was easy to install and provided the required data.

Geist Racsense environmental monitors with temperature probes were deployed on the covers and floors of each rack (front and rear), as well as at the power input and output of each CRAC unit.
A humidity probe was also installed to measure the ambient humidity within the room. The Racsense devices enabled the simple and efficient capture of data on the installation and cabling.
Two Dominion® PX intelligent power distribution units were placed in each rack. This allowed for the collection of data on the active power consumption of critical IT workloads, individually for each device. This data on critical IT workloads provided the necessary accuracy to measure PUE and develop improvement proposals.

Results
Nominal vs. Actual Consumption
The histogram in Figure 1 shows the frequency distribution of actual consumption expressed as a percentage of nominal consumption (capacity), based on readings taken at the Raritan data center between February 21 and 26, 2008. The nominal value shown on the nameplate indicates the maximum consumption that a device can reach at any given time under any workload and in any kind of environmental condition within the data center.

Current data center design rules use nominal values ​​for capacity planning. Since the nameplate value represents the maximum achievable value, using this method to size data center power systems is inefficient because it relies on figures higher than actual consumption. Reducing the nominal value by a fixed percentage partially mitigates the problem, but the chart below demonstrates that the range of factors related to actual operational power consumption is so broad that it makes choosing a reduction factor that provides an appropriate balance between energy efficiency and reliable operation extremely difficult. In other words, what we thought we were achieving with a "reduction of values" strategy has turned out to be a myth, as we were sacrificing "real" knowledge for "approximate" knowledge, and thus also sacrificing our ability to become a greener company.

For example, during the measurement of actual average consumption, it was found that 29 devices in Raritan's server room consume between 21% and 40% of this capacity, while 15 consume a higher percentage and 15 consume a lower percentage. The measurement of peak consumption showed that 27 devices in our server room consume between 21% and 40% of this capacity, while 23 consume a higher percentage and 9 consume a lower percentage.

The main conclusion is that actual capacity (actual consumption) has a very wide potential range. Therefore, simply reducing nominal consumption by arithmetic means has two possible consequences: it either compromises reliability or fails to realize the full potential for increased efficiency. A design based on arithmetically reduced consumption might be the first step, but measuring a device's actual consumption during daily operation provides the additional information needed to operate more efficiently and ensure reliable performance.

Calculating Raritan's PUE Value
As shown in Figure 2, approximately 71 percent of Raritan's total average electricity consumption is attributed to critical IT equipment (servers alone account for 49.4 percent), while the
remaining 29 percent is attributed to auxiliary services, such as cooling and lighting.
To calculate Raritan's PUE using the formula above:

Total Consumption = Auxiliary Infrastructure (5.625 kW) + Critical Load (13.68 kW) = 19.3 kW
Total IT Equipment Consumption = Critical Load (13.68 kW)
Raritan's PUE = Total Consumption / Total IT Equipment Consumption = 19.3 kW / 13.68 kW = 1.4

Conclusions:
Rising energy costs and dwindling supply sources were the primary motivations for this analysis. Over a five-month period, Raritan and PTS defined and implemented a plan to equip Raritan's server room and measure power consumption at both the IT equipment level and the secondary circuits for non-IT loads. This study revealed the following:

- Initially assuming its server room would have a low PUE value, Raritan discovered it actually had a respectable PUE of 1.4, with the average consumption of all its IT equipment (servers and combined storage) at 10.5 kW.
- This means that by using real, rather than approximate, data, Raritan can take a smart approach to managing and supplying its data center:
• Raritan does not need to add servers to increase computing power.
• Raritan can become a greener data center.
- 44 devices registered an actual average power consumption equivalent to that of an idle server. This clearly indicated that energy could be saved by consolidating these nearly idle servers. Assuming a typical 10-to-1 consolidation ratio, 40 devices could potentially be taken offline.
• Raritan can prevent the deterioration of its vital services by implementing more efficient workload management.
- Two devices were consuming between 80 and 100 percent of their rated power. If Raritan had applied the industry average and reduced this value by 60 to 80 percent, server performance could have been negatively impacted.
- These same two devices were investigated to ensure they were not creating hot spots and that their power consumption was not near their capacity limits, which would have posed a potential problem.
- Raritan can reduce its operating costs without compromising its vital processing capacity.
The use of smart rack PDUs showed Raritan the actual power consumption at the individual IT device level, providing the data needed to plan the next steps toward reducing energy consumption, increasing efficiency, and consequently, being more
environmentally

. The measurements also yielded a number of unforeseen benefits beyond simply providing the data needed to calculate the PUE value. For example, a sudden drop in power consumption allowed the IT staff to discover that the circuit breaker on one of their CRAC units had tripped. This enabled preventative repairs before the summer heat could cause cooling problems. A temperature sensor located at the top of a rack alerted them that one of the rack fans had failed.

Since industry benchmarks are based on a wide variety of averages, and each data center has its own unique characteristics that differentiate it from those averages, this study by Raritan and PTS reinforced the Green Grid organization's principle:
"To improve the energy performance of data centers, it is first necessary to measure the consumption of the data center as a whole and of each of the subsystems that make it up."

What's the next step?
In the next white paper, we'll examine the use and benefits of Computational Fluid Dynamics (CFD) to help you improve the energy efficiency of your data center.
We'll also compare our results obtained using real-world measurements against CFD-based nominal energy consumption data.

Author:

Greg More, Product Manager at Raritan.

More information or a quote