roughly Liquid and immersion is the brand new cool at Supercomputing ‘22 • The Register will cowl the newest and most present suggestion on the order of the world. admission slowly because of this you perceive properly and appropriately. will layer your information cleverly and reliably

SC22 It is protected to say that liquid cooling was a scorching subject on the Supercomputing convention in Dallas this week.

So far as the attention may see, the showroom was filled with liquid-cooled servers, oil-filled immersion cooling tanks, and all of the equipment, pumps, and coolant distribution items (CDUs) you might must implement the know-how in a knowledge heart. .

Since this can be a convention on excessive efficiency computing, the emphasis on thermal administration mustn’t come as a shock. However with 400W CPUs and 700W GPUs now out there, it is not simply an HPC or AI drawback. As extra firms look so as to add AI/ML-enabled methods to their knowledge facilities, 3kW, 5kW, and even 10kW methods aren’t so loopy anymore.

So this is a breakdown of the liquid cooling package that caught our eye at this yr’s present.

Direct liquid cooling

The overwhelming majority of liquid cooling methods proven at SC22 are of the direct liquid selection. They commerce copper or aluminum warmth sinks and followers for chilly plates, rubber tubing, and equipment.

If we’re being trustworthy, these chilly dishes look just about the identical. Principally they’re only a hollowed out block of steel with an inlet and outlet for the fluid to undergo. Notice that we’re utilizing the phrase “liquid” right here as a result of liquid-cooled methods can use any variety of refrigerants that aren’t essentially water.

A Supermicro liquid-cooled server equipped with CoolIT cold plates.

A Supermicro liquid-cooled server geared up with CoolIT chilly plates. – Click on to enlarge

In lots of circumstances, OEMs supply their chilly plates from the identical distributors. For instance, CoolIT gives liquid cooling {hardware} for varied OEMs, together with HPE and Supermicro.

Nonetheless, that is to not say there is not a possibility for differentiation. The inside of those chilly plates is crammed with microfin arrays that may be adjusted to optimize fluid movement by them. Relying on the scale or the variety of dies to be cooled, the inside of those chilly plates can fluctuate fairly a bit.

Many of the liquid-cooled methods we noticed on the present flooring used some sort of rubber tubing to attach the chilly plates. Which means the liquid solely cools particular elements just like the CPU and GPU. So whereas the majority of the followers will be eliminated, some airflow remains to be required.

HPE demonstrates its latest liquid-cooled Cray EX blades using AMD's 96-core Epyc 4 CPUs.

HPE demonstrates its newest liquid-cooled Cray EX blades utilizing AMD’s 96-core Epyc 4 CPUs. – Click on to enlarge

The Lenovo Neptune and HPE Cray EX blades had been the exception to this rule. Their methods are designed particularly for liquid cooling and are filled with copper tubing, bus blocks, and chilly plates for the whole lot together with CPUs, GPUs, reminiscence, and NICs.

With this method, HPE has succeeded in packing eight AMD 400W Epyc 4 Genoa CPUs right into a 19-inch chassis.

A liquid-cooled Lenovo Neptune server configured with two AMD Genoa CPUs and four Nvidia H100 GPUs.

A liquid-cooled Lenovo Neptune server configured with two AMD Genoa CPUs and 4 Nvidia H100 GPUs. – Click on to enlarge

Lenovo, in the meantime, confirmed off a 1U Neptune system designed to chill a pair of 96-core Epycs and 4 of Nvidia’s H100 SXM GPUs. Relying on the implementation, producers declare that their direct liquid cooling methods can take away between 80 and 97 % of the warmth generated by the server.

immersion cooling

One of many extra unique liquid cooling applied sciences on show at SC22 was immersion cooling, which has made a comeback lately. These methods can seize one hundred pc of the warmth generated by the system.

Rather than retrofit the server with cold plates, immersion cooling tanks, like this one from Submer, submerge them in a non-conductive liquid.

As a substitute of retrofitting the server with chilly plates, immersion cooling tanks, like this one from Submer, submerge them in a non-conductive liquid. click on to enlarge

As loopy because it sounds, we have been soaking pc elements in non-conductive fluids to maintain them cool for many years. One of the crucial well-known methods for utilizing immersion cooling was the Cray 2 supercomputer.

Whereas the fluids utilized in these methods fluctuate from provider to provider, artificial oils from Exxon or Castrol or specialty coolants from 3M will not be unusual.

Submer was one in all a number of immersion cooling firms to point out off their know-how at SC22 this week. The corporate’s SmartPods look a bit such as you fill a chest freezer filled with oil and begin putting servers vertically from the highest.

Submer presents tanks in varied sizes which might be roughly equal to conventional half and full racks. These tanks are rated for 50-100kW of warmth dissipation, placing them on par with rack-mounted air and liquid cooling infrastructure by way of energy density.

Submer's tank supports OCP OpenRack form factors like these tri-bladed Intel Xeon systems

Submer’s tank helps OCP OpenRack type elements like these tri-bladed Intel Xeon methods – Click on to Develop

The demo tank had three 21-inch servers, every with three dual-socket Intel Sapphire Rapids blades, in addition to a typical 2U AMD system that had been transformed to be used of their tanks.

Nonetheless, we’re advised that the variety of mods required, particularly to the OCP chassis, is fairly negligible, with the one actual modifications being altering the transferring components of issues like energy provides.

Unsurprisingly, immersion cooling complicates upkeep and is a little more concerned than air or direct liquid cooling.

Iceotope Immersion Cooled Server Chassis

Iceotope’s twist on immersion cooling makes use of the server chassis as a reservoir. – Click on to enlarge

Not all present flooring immersion cooling setups require gallons of specialised fluids. Iceotope’s in-chassis immersion cooling system was one instance. The corporate’s sealed server chassis features like a reservoir with the motherboard submerged in a number of millimeters of liquid.

A redundant pump on the rear of the server recirculates oil to hotspots such because the CPU, GPUs, and reminiscence earlier than passing the recent fluids by a warmth exchanger. There, the warmth is transferred to a facility water system or rack-scale Coolant Distribution Models (CDUs).

Assist infrastructure

No matter whether or not you utilize direct-chip or immersion cooling, each methods require further infrastructure to extract and dissipate warmth. For direct liquid-cooled configurations, this will embrace distribution manifolds, rack-level plumbing, and most significantly, a number of CDUs.

Giant rack measurement CDUs can be utilized to chill a whole row of server cupboards. For instance, Cooltera confirmed a number of giant CDUs able to offering as much as 600 kW of cooling to a knowledge heart. Or, for smaller deployments, a rack-mounted CDU can be utilized. We checked out two examples from Supermicro and Cooltera, which supplied between 80 and 100 kW of cooling capability.

A Cooltera rack-mounted refrigerant distribution unit

A Cooltera Rack Mounted Refrigerant Distribution Unit – Click on to Develop

These CDUs are made up of three predominant elements: a warmth exchanger, redundant pumps to flow into coolant by the racks, and a filtration system to forestall particles from clogging crucial elements like chilly plate microfins.

The way in which during which warmth is definitely faraway from the refrigeration system largely is dependent upon the kind of warmth exchanger that’s used. Liquid to air warmth exchangers are among the many easiest as a result of they require the fewest modifications to the set up itself. The Cooltera CDU proven right here makes use of giant radiators to discharge warmth captured by fluid within the knowledge heart’s scorching aisle.

A Cooltera CDU with an integrated liquid-air heat exchanger.

Along with the pumps and filtration, this Cooltera CDU options an built-in liquid to air warmth exchanger. – Click on to enlarge

Nonetheless, a lot of the CDUs we noticed at SC22 used liquid-to-liquid warmth exchangers. The concept right here is to make use of a separate facility-wide water system to maneuver the warmth collected from a number of CDUs to dry coolers on the skin of the constructing the place it’s dissipated to the air. Or, as an alternative of dumping warmth into the ambiance, some knowledge facilities, like Microsoft’s newest facility in Helsinki, have linked their facility’s water methods to district heating methods.

The scenario is far the identical for immersion cooling, though lots of the CDU elements, akin to pumps, liquid-to-liquid warmth exchangers, and filtration methods, are constructed into the tanks. All that’s actually required is that they be linked to the ability’s water system.

Liquid cooling adoption on the rise

Whereas liquid cooling represents solely a fraction of knowledge heart thermal administration spend immediately, hotter elements and better rack energy densities are starting to drive adoption of the know-how.

In accordance with a current Dell’Oro Group report, spending on immersion and liquid cooling tools is anticipated to achieve $1.1 billion or 19% of thermal administration spending by 2026.

In the meantime, rising vitality costs and a better emphasis on sustainability make liquid cooling enticing on different ranges. Placing apart the practicality of cooling a 3kW server with air, 30-40 % of a knowledge heart’s vitality consumption will be attributed to air con and air dealing with tools required to maintain methods at working temperature. .

So whereas server distributors have discovered methods to air-cool servers as much as 10kW, within the case of Nvidia’s DGX H100, at these thermal and energy densities, there are exterior incentives to scale back the facility consumption now getting used for computing. ®

I want the article roughly Liquid and immersion is the brand new cool at Supercomputing ‘22 • The Register provides sharpness to you and is beneficial for additive to your information

Liquid and immersion is the new cool at Supercomputing ‘22 • The Register

By admin