Artificial intelligence (AI) is here to stay. “Every industry will become a technology industry,” according to Jensen Huang, founder and CEO of NVIDIA. AI use cases are virtually limitless, from medical advancements to high-precision fraud prevention. AI is already transforming our lives, just as it is transforming every industry. It is also beginning to fundamentally transform data center infrastructure. AI workloads are driving significant changes in how we power and cool processed data as part of high-performance computing (HPC). A typical IT rack used to run 5-10 kilowatt (kW) workloads and racks running loads above 20 kW were considered high density—a rare sight outside of very specific applications with a narrow scope. IT is being accelerated with GPUs to meet the computing needs of AI models, and these AI chips can require roughly five times more power and five times more cooling capacity in the same space as a traditional server. Mark Zuckerberg announced that by the end of 2024, Meta will spend billions deploying 350,000 NVIDIA H100 GPUs. Rack densities of 40 kW per rack are now at the lower end of what's needed to facilitate AI deployments, with rack densities exceeding 100 kW per rack becoming commonplace and widespread in the near future.
This will require significant capacity increases across the entire powertrain, from the network to the chips in each rack. The introduction of liquid cooling technologies in the data center's white space and, eventually, in enterprise server rooms will be a requirement for most deployments, as traditional cooling methods will be unable to handle the heat generated by GPUs performing AI calculations. Investments to upgrade the infrastructure needed to power and cool AI hardware are substantial, and addressing these new design challenges is critical.
The transition to high density
The transition to accelerated computing won't happen overnight. Data center and server room designers must find ways to future-proof their power and cooling infrastructure, taking into account the future growth of their workloads. Providing sufficient power to each rack requires network-to-rack upgrades. In practice, this likely means high-amperage conduit and high-density rack PDUs. To reject the massive amounts of heat generated by hardware running AI workloads, two liquid cooling technologies are emerging as leading options:
1. Direct-to-chip liquid cooling: Cold plates sit directly on the heat-generating components (typically chips like CPUs and GPUs) to extract heat. Single-phase or two-phase pumped fluid draws heat from the cold plate and sends it away from the data center, exchanging heat but not fluid with the chip. This can remove approximately 70-75% of the heat generated by the equipment in the rack, leaving 25-30% for air-cooling systems to remove.
2. Rear-door heat exchangers: Passive or active heat exchangers replace the rear door of the IT rack with heat exchange coils through which fluid absorbs the heat produced in the rack. These systems are often combined with other cooling systems as a strategy to maintain room neutrality or as a transitional design that begins the journey toward liquid cooling.
While direct-to-chip liquid cooling offers significantly higher density cooling capacity than air, it's important to note that there is still excess heat that the cold plates cannot capture. This heat will be rejected into the data room unless it is contained and removed through other means, such as rear-door heat exchangers or room air cooling. For more information on liquid cooling solutions for data centers, please see our technical report.
AI starter kits for refurbishment and new builds
Power and cooling are becoming integral parts of data room IT solution design, blurring the lines between IT and facility teams. This adds a high degree of complexity to design, implementation, and operation. Partnerships and expertise in comprehensive solutions are ranked as key requirements for smooth transitions to higher densities.
To simplify the move to high density, Vertiv has introduced a range of optimized designs that include power and cooling technology capable of supporting workloads of up to 100 kW per rack in a diverse set of deployment configurations.
These designs offer multiple paths for systems integrators, colocation providers, cloud service providers, or enterprise users to achieve the data center of the future. Each specific installation may have nuances in rack count and density dictated by the IT equipment selection. As such, this collection of designs provides an intuitive way to refine a base design and tailor it precisely to deployment needs.
When retrofitting or repurposing existing environments for AI, our optimized designs help minimize disruption to existing workloads by leveraging existing cooling infrastructure and heat rejection where possible. For example, we can integrate direct-to-chip liquid cooling with a rear-door heat exchanger to maintain a room-neutral cooling solution. In this case, the rear-door heat exchanger prevents excess heat from escaping into the room. For an air-cooled installation looking to add liquid cooling equipment without any on-site modifications, we have liquid-to-air design options available. This same strategy can be implemented in a single rack, in a row, or at scale in a large HPC deployment. For multi-rack designs, we have also included high-density rack PDUs and high-amperage piping to distribute power to each rack.
These options are compatible with a range of different heat rejection solutions that can be combined with liquid cooling. This establishes a clean and cost-effective transition path to high-density liquid cooling without disrupting other workloads in the data room. See our AI data room solutions for more information.
Although many facilities are not designed for high-density systems, Vertiv has extensive experience helping customers develop deployment plans to make a seamless transition to high density for AI and HPC.
About the authors:
Anton Chuchkov is a product manager on Vertiv's IT solutions team, focusing on high-density solutions, including infrastructure to support AI applications. He is responsible for introducing the latest industry technologies to data center vertical markets. He has over ten years of experience in application engineering and product management roles at the chip, board, server, and system levels, enabling time-sensitive AI inference applications at the edge. Anton holds a bachelor's degree in electrical engineering from Stony Brook University.
