To fill this gap, a team of specialists from NVIDIA and Vertiv conducted the first major impact analysis of liquid cooling on PUE and power consumption in data centers. The full analysis was published by the American Society of Mechanical Engineers (ASME) in the article "Power Usage Effectiveness Analysis of a High-Density Air-Liquid Hybrid Cooled Data Center." This publication summarizes the methodology, results, and key findings of that analysis.
Methodology for analyzing the power efficiency of liquid cooling in data centers
For our analysis, we selected a Tier 2, medium-sized (1-2 MW) data center in Baltimore, Maryland. The center houses 50 high-density racks arranged in two rows. The benchmark for the analysis was 100% air cooling provided by two chilled water units, with a standard Computer Room Air Handler (CRAH) perimeter air controller and a hot aisle enclosure system. The cooling units are supported by a Vertiv™ Liebert® AFC chiller with free cooling, adiabatic free cooling, hybrid cooling, and adiabatic mechanical cooling capabilities.
Liquid cooling is activated by direct chip cooling via microchannel cold plates mounted on major heat-generating IT components and supported by two Vertiv™ Liebert® XDU Coolant Distribution Units (CDUs) with liquid-to-liquid heat exchangers.
The analysis employed a bottom-up approach by disaggregating the IT load into subsystems, allowing for the precise calculation of the impact of progressively increasing the percentage of liquid-cooled load for each subsystem. Four studies were then conducted, each increasing the percentage of liquid cooling, while simultaneously implementing optimizations to the chilled water temperature, supply air temperature, and secondary inlet temperature through the use of liquid cooling.
Study 1: 100% air-cooled with a chilled water temperature of 7.2°C (45°F), a supply air temperature of 25°C (77°F), and a secondary inlet temperature of 32°C (89.6°F).
Study 2: 61.4% of the load liquid-cooled and 38.6% air-cooled. The chilled water temperature is raised to 18°C (64.4°F), the supply air temperature is maintained at 25°C (77°F), and the secondary inlet temperature is maintained at 32°C (89.6°F).
Study 3: 68.6% of the load liquid-cooled and 31.4% air-cooled. The chilled water temperature rises to 25°C (77°F), the supply air temperature rises to 35°C (95°F), and the secondary inlet temperature is maintained at 32°C (89.6°F).
Study 4: 74.9% of the load is liquid-cooled and 25.1% is air-cooled. The chilled water temperature is maintained at 25°C (77°F), the supply air temperature is maintained at 35°C (95°F), and the secondary inlet temperature rises to 45°C (113°F).
Impact of the introduction of liquid cooling on energy consumption and PUE of data centers
The full implementation of liquid cooling in Study 4 (74.9%) resulted in an 18.1% reduction in core power and a 10.2% reduction in total data center power compared to 100% air cooling. This has the effect of not only reducing energy costs by 10% annually, but also, for data centers using carbon-based energy sources, reducing Scope 2 emissions by the same amount.
The total power consumption of the data center decreased with each increase in the percentage of the load cooled by direct chip cooling. In Studies 1 to 2, power consumption was reduced by 6.4%; an additional 1.8% reduction was achieved between Studies 2 and 3, and a further 2.5% improvement was observed between Studies 3 and 4.
Based on these results, the PUE calculated for the data center in each study may be surprising. The PUE fell by only 3.3%, from 1.38 in Study 1 to 1.34 in Study 4, and actually remained unchanged at 1.35 for Studies 2 and 3.
If you're familiar with how PUE is calculated, you may have already guessed the reason for this discrepancy. PUE is essentially a measure of infrastructure efficiency, calculated by dividing the total power of the data center by the IT power. But liquid cooling not only reduced power consumption within the data center itself, it also reduced IT power consumption (as defined by PUE) by decreasing the demand on server fans.
Server fan power consumption decreased by 41% between Study 1 and Study 2, and by 80% between Study 1 and Study 4. This resulted in a 7% reduction in IT power between Study 1 and Study 4.
Unlike air cooling, liquid cooling affects both the numerator (total data center power) and the denominator (IT equipment power) in the PUE calculation, which doesn't make much sense when comparing the efficiency of liquid and air cooling systems.
In our article, we propose Total Usage Effectiveness (TUE) as a better metric for this purpose, and I will explain why this decision was made in a follow-up publication. The TUE for the data center we analyzed improved by 15.5% between Study 1 and Study 4, leading us to believe that it is an accurate measure of the data center efficiency improvements achieved through the optimized implementation of liquid cooling.
Key findings from the liquid cooling energy efficiency analysis in data centers
The analysis provided several diverse perspectives on the efficiency of liquid cooling in a data center and how it can be optimized. I encourage data center designers to read the full document, which includes the supporting data used to derive the results presented in the previous section. Here are some of the key findings that may be of interest to a wider audience.
In high-density data centers, liquid cooling offers significant improvements in the energy efficiency of IT systems and facilities compared to air cooling. In our fully optimized study, the introduction of liquid cooling resulted in a 10.2% reduction in total data center power consumption and a more than 15% improvement in Total Energy Usage (TUE).
Maximizing the deployment of liquid cooling in data centers—in terms of the percentage of IT load cooled by liquid—offers the greatest efficiency. Direct-to-chip (DTC) cooling does not allow for liquid cooling of the entire load, but approximately 75% of the load can be effectively cooled using DTC liquid cooling.
Liquid cooling can allow for higher temperatures of chilled water, supply air, and secondary inlets, maximizing the efficiency of the facility's infrastructure. In particular, hot water cooling should be considered. Secondary inlet temperatures in our final study reached 45°C (113°F), which contributed to the results achieved and increased opportunities for waste heat reuse.
PUE is not a good measure of liquid cooling efficiency for a data center, and alternative metrics such as TUE will be more useful in guiding design decisions related to the introduction of liquid cooling in an air-cooled data center.
Finally, I want to thank my colleagues at Vertiv and NVIDIA for their work on this groundbreaking analysis. The findings not only quantify the energy savings achievable through liquid cooling, but also provide designers with valuable data that can be used to optimize liquid-cooled data center installations.
For more information on the trends driving the adoption of liquid cooling, see the blog post, Liquid Cooling: Data Center Solutions for High-Density Compute, which summarizes the insights of a panel of liquid cooling experts at the 2022 OCP Global Summit.
Author: Fred Rebarber
As Global Technical Director at Vertiv, Fred serves as the corporate technical interface for large end users and consulting engineers specializing in mission-critical designs. A key function of this role is to provide input on product development based on customer needs and market demands. In his previous role with the OEM group, Fred worked with manufacturers and end users to drive adoption of existing Vertiv™ Liebert® products and create specifications for new products. Prior to his OEM position, Fred was Director of Sales and Marketing at Cooligy, a startup that designs and manufactures chip-level liquid cooling solutions for manufacturers. Fred holds a Bachelor of Science in Mechanical Engineering from the University of California, Berkeley.
