NVIDIA H100 Tensor Core GPU Dominates MLPerf v3.0 Benchmark Outcomes

To know the way a system performs throughout a variety of AI workloads, you have a look at its MLPerf benchmark numbers. AI is quickly evolving, with generative AI workloads turning into more and more outstanding, and MLPerf is evolving with the trade. Its new MLPerf Coaching v3.0 benchmark suite introduces new assessments for advice engines and enormous language mannequin (LLM) coaching.

MLCommons, which oversees MLPerf, launched the newest MLPerf benchmark outcomes at the moment. The NVIDIA H100 dominated practically each class and was the one GPU used within the new LLM benchmarks.

The highest numbers for LLM and BERT pure language processing (NLP) benchmarks belonged to a system collectively developed by NVIDIA and Inflection AI and hosted by CoreWeave, a cloud providers supplier specializing in enterprise-scale GPU-accelerated workloads. To say that the numbers are spectacular is an understatement.

NVIDIA H100 Dominates Each Benchmark

The MLPerf LLM benchmark relies on OpenAI’s GPT-3 LLM skilled with 175 billion parameters (GPT-3 was the newest era GPT accessible when the benchmark was created final yr). Coaching LLMs is a computationally costly process, with Lambda Labs estimating that coaching GPT-3 with 175 billion parameters requires about 3.14E23 FLOPS of computing. That’s quite a lot of costly horsepower.

NVIDIA designed the H100 Tensor Core GPU for precisely these workloads, and it’s shortly turning into some of the well-liked accelerators for coaching giant language fashions. That is for a superb cause. NVIDIA launched a brand new transformer engine within the H100, explicitly designed to speed up transformer mannequin coaching and inference (NVIDIA offers a wonderful description of the machine’s full capabilities in a weblog put up). Transformers are on the coronary heart of generative AI, so it is anticipated that the H100 ought to carry out higher than earlier generations. NVIDIA says all the things is quicker on the H100, with its new transformer engine boosting coaching as much as 6x.

Of the 90 methods included in at the moment’s outcomes, 82 used NVIDIA accelerators (of the 8 non-NVIDIA methods examined, all however one have been submitted by Intel). Just below half of all outcomes have been based mostly on the NVIDIA H100 Tensor Core GPU. The NVIDIA H100 set information on each workload within the MLPerf coaching and inference benchmarks, whereas NVIDIA’s A100 and L4 GPUs offered wholesome inference outcomes.

Wanting deeper into the metrics, the NVIDIA H100 Tensor Core GPU yielded a per-accelerator LLM coaching time of 548 hours (about 23 days). The GPU additionally set per-accelerator information on each benchmark examined.

LLM at Scale: NVIDIA + Inflection AI + CoreWeave

per-accelerator outcomes is attention-grabbing, however real-world manufacturing workloads are not often constructed utilizing single accelerators. There are efficiencies of scale that emerge in a clustered system with a number of GPUs, one thing NVIDIA designed in from the beginning with its ongoing deal with GPU-to-GPU communication utilizing its NVLink expertise. Understanding real-world efficiency requires outcomes on a system degree.

NVIDIA and Inflection AI co-developed a large-scale GPU cluster based mostly on the NVIDIA H100 Tensor Core GPU, hosted and examined by Coreweave. The system combines 3,584 NVIDIA H100 accelerators with 896 4^th era Intel Xeon Platinum 8462Y+ processors. The outcomes are staggering, setting new information on each workload examined.

Delving into the LLM benchmarks reveals off the total capabilities of NVIDIA’s expertise. The three,854 GPU cluster accomplished the huge GPT-3-based coaching benchmark in lower than eleven minutes, whereas a configuration containing half that variety of GPUs accomplished in practically 24 minutes, demonstrating the non-linear scalability potential of the NVIDIA H100 GPU.

Intel was the one different entity to report LLM benchmark outcomes. Intel’s methods mixed 64-96 Intel Xeon Platinum 8380 processors with 256-389 Intel Habana Gaudi2 accelerators. Intel reported LLM coaching instances of 311 minutes for its top-end configuration.

Analyst’s Take

Benchmarks present a point-in-time comparability of methods. That almost each submitted end result was based mostly on an NVIDIA accelerator speaks to the continued dominance of NVIDIA throughout the AI ecosystem. Whereas this dominance is essentially based mostly on its accelerator expertise, the stickiness of NVIDIA throughout the ecosystem continues to be very a lot ruled by the reliance on its software program by the AI neighborhood.

NVIDIA does not simply present the low-level CUDA libraries and instruments upon which practically each AI framework relies, the corporate has moved up the stack to offer full-stack AI instruments and options. Past enabling AI builders, NVIDIA continues to put money into enterprise-level instruments for managing workloads and fashions. NVIDIA’s software program funding is unmatched within the trade and can hold NVIDIA within the driver’s seat for the foreseeable future. There will probably be non-NVIDIA options for coaching, however these will proceed to be the exception.

My largest takeaway from the MLPerf outcomes is not the uncooked efficiency of NVIDIA’s new H100 Tensor Core Accelerators however moderately the ability and effectivity of working AI coaching workloads within the cloud. Constructing a coaching cluster of any measurement is an costly and sophisticated endeavor. NVIDIA does not launch pricing for its H100 accelerator, but it surely’s estimated to be between $30,000-$40,000/every. CoreWeave will lease one to you for $2.23/hour, delivering coaching outcomes in addition to any on-site set up (in an extra plug for CoreWeave, I will level out that it isn’t but attainable to get time on an H100 from any of the highest public cloud suppliers; no CSP has an H100-based occasion usually accessible at the moment).

AI is altering the way in which we interact with expertise. It is altering how companies function and the way we perceive the information surrounding us. NVIDIA sits on the middle of this revolution, quickly increasing its presence into practically each aspect of the information middle. NVIDIA is not the gaming graphics firm we grew up with. It is as an alternative shortly turning into a key enabler for our collective future. Preserve watching, however they’re simply getting began.