NVIDIA H100 vs. AMD MI300X: The Real LLM Benchmark Story

AMD’s benchmark slide from December 2023 claimed the MI300X could hit 1.6x the inference throughput of NVIDIA’s H100 on Llama 2 70B. By the time that number made its way through a few Twitter threads, YouTube explainers, and SEO listicles, it had somehow inflated into “10x faster than H100” — a figure that appears nowhere in AMD’s own published data, NVIDIA’s rebuttal benchmarks, or the MLPerf Inference v4.1 results MLCommons published in August 2024. I went back through the primary sources — AMD’s launch deck, NVIDIA’s January 2024 TensorRT-LLM blog response, and three independent third-party test runs — and the real gap between these two chips is real, measurable, and interesting. It’s just not 10x. It’s closer to 1.2x to 2.1x depending on the workload, the software stack, and whether you’re counting per-GPU throughput or per-dollar throughput. That distinction matters more than the headline number, and it’s where most of the coverage on this comparison falls apart.

6 min read

Where the “10x” Number Actually Comes From

I couldn’t find a single benchmark — AMD’s, NVIDIA’s, or an academic one — that shows a 10x throughput advantage for either chip on a standard LLM inference or training task. What I did find is a plausible mechanism that explains how the number got exaggerated. At extreme context lengths (128K tokens and beyond) with large batch sizes, an 80GB H100 running Llama 3 70B often can’t fit the model weights plus the KV cache on a single GPU, forcing 4-way or 8-way tensor parallelism across NVLink. MI300X’s 192GB of HBM3 can fit the same model and a much larger KV cache on one or two GPUs, avoiding that communication overhead entirely.

In that narrow scenario — long context, single-node, communication-bound rather than compute-bound — latency differences of 5x to 8x have shown up in isolated community tests, mostly because the H100 setup is paying a tensor-parallel tax that has nothing to do with raw FLOPs. That’s a real and useful finding for anyone running long-context inference at scale. It’s also not a general-purpose “MI300X is 10x faster” claim, and treating it as one misleads anyone comparing these chips for a standard fine-tuning or short-context serving job, where the gap shrinks to well under 2x in either direction.

That’s a real and useful finding for anyone running long-context inference at scale.

Two Different Bets on How LLMs Should Run

NVIDIA’s H100, built on the Hopper architecture and fabricated on TSMC’s custom 4N process, ships 80 billion transistors and 80GB of HBM3 with 3.35 TB/s of memory bandwidth on the SXM5 module. Its dense tensor throughput is 989 TFLOPS at FP16 and 1,979 TFLOPS at FP8, doubling to 1,979 and 3,958 TFLOPS respectively with structured sparsity enabled — a feature that requires model-specific sparsification and rarely shows up in production LLM serving. TDP on the SXM5 variant runs up to 700W.

AMD’s MI300X, launched at the “Advancing AI” event in San Jose on December 6, 2023, takes a chiplet approach: eight CDNA3 compute dies (XCDs) stacked with four I/O dies on TSMC’s N5/N6 nodes, totaling roughly 153 billion transistors. It packs 192GB of HBM3 — 2.4x the H100’s capacity — with 5.3 TB/s of bandwidth, and claims 1,307 TFLOPS dense BF16 and 2,615 TFLOPS dense FP8, with no sparsity feature to fall back on. TDP is 750W.

The philosophical split is clear once you line up the specs: NVIDIA bet on raw compute density and a mature software stack (CUDA, cuDNN, TensorRT-LLM), while AMD bet on memory capacity and bandwidth to solve the problem that actually bottlenecks most LLM inference — moving weights and KV cache in and out of memory faster than the compute units can chew through them. Inference for autoregressive transformers is almost always memory-bandwidth-bound during the decode phase, not compute-bound, which is why AMD’s bandwidth advantage matters more in practice than the FLOPS gap suggests.

TDP on the SXM5 variant runs up to 700W.

Memory Bandwidth Is the Real Story, Not FLOPs

Anyone who has profiled an LLM inference server knows the prefill phase (processing the prompt) is compute-heavy, while the decode phase (generating tokens one at a time) is memory-bandwidth-heavy. Since decode dominates wall-clock time in most chat and agent workloads — often 80-90% of total inference time for typical 500-2,000 token generations — memory bandwidth predicts real-world throughput better than FLOPS in almost every serving scenario I’ve tested.

Run the roofline math and MI300X’s 5.3 TB/s versus H100’s 3.35 TB/s is a 58% bandwidth advantage — nowhere near the 10x figure floating around, but a meaningful edge that shows up consistently in decode-heavy benchmarks. Combine that with 192GB of capacity, and MI300X can hold larger batches and longer KV caches without spilling to a second GPU, which is where AMD’s inference throughput claims are strongest and most defensible.

Where H100 claws back ground is in prefill-heavy or training workloads, where NVIDIA’s higher FP8 dense throughput and — critically — the maturity of TensorRT-LLM’s kernel fusion, FlashAttention-2 integration, and paged attention support close much of the gap. NVIDIA’s own January 2024 blog post, “Full-Stack Optimizations Boost H100 Performance,” showed that with TensorRT-LLM 0.7 tuning, H100 matched or beat AMD’s original launch-day MI300X numbers on Llama 2 70B — a rebuttal AMD partially addressed with its own vLLM and ROCm 6.0 tuning updates later in early 2024.

What MLPerf and Independent Benchmarks Actually Show

MLPerf Inference v4.1, published by MLCommons in August 2024, was the first round to include AMD MI300X submissions alongside NVIDIA H100 and H200 results on the Llama 2 70B benchmark. The pattern that emerged was not a blowout in either direction: normalized per-accelerator throughput on the offline scenario put MI300X and H100 within a similar band, with each chip trading leads depending on batch size and the specific software stack version submitted. NVIDIA’s H200 — the memory-upgraded Hopper refresh with 141GB of HBM3e and 4.8 TB/s bandwidth, launched in Q4 2023 and shipped through 2024 — outran both on memory-bound scenarios, which itself is evidence that bandwidth, not raw compute, is the deciding variable in this category.

Databricks’ MosaicML team published its own training-throughput comparisons in early 2024, and the headline finding there was software maturity, not silicon. Training an LLM on ROCm still requires more manual kernel tuning than an equivalent CUDA/TensorRT-LLM pipeline, and Databricks reported needing custom Triton kernels to get MI300X training runs within striking distance of H100 baselines on MPT-class models. That’s an important caveat missing from most comparison articles: raw hardware specs tell you the ceiling, but software stack maturity tells you what teams actually achieve on day one.

• Inference, short context (under 4K tokens): H100 and MI300X trade wins within roughly 20%, with TensorRT-LLM’s kernel optimizations often edging out AMD’s stack.
• Inference, long context (32K+ tokens), large batch: MI300X pulls ahead by 1.5x to 2.1x due to memory capacity avoiding multi-GPU tensor parallelism.
• Training, dense transformer architectures: H100 remains the safer default because of NCCL, Megatron-LM, and DeepSpeed integration maturity; MI300X training pipelines require more ROCm-specific tuning.
• Mixed-precision FP8 workloads: H100’s structured sparsity support gives it a theoretical ceiling MI300X can’t match, though few production models exploit sparsity today.

The Real Cost Math: Cloud Pricing and Cluster Economics

Hardware benchmarks matter less to most teams than what shows up on the cloud bill. As of mid-2024, on-demand H100 SXM instances ran roughly $4.50-$6.98 per GPU-hour on major clouds (AWS p5.48xlarge, Azure ND H100 v5, CoreWeave, Lambda), while Microsoft’s Azure ND MI300X v5 instances launched at roughly $3.40-$4.00 per GPU-hour depending on commitment term and region. Oracle Cloud Infrastructure and select Tier 2 providers followed with MI300X capacity through 2024, generally priced 20-35% below comparable H100 instances.

Run the throughput-per-dollar math using the long-context inference numbers above, and MI300X’s advantage compounds: a 1.8x throughput edge combined with a 25% lower hourly rate produces something closer to a 2.4x effective cost-per-token advantage for that specific workload — a number that’s genuinely impressive without needing to round it up to 10x. For short-context serving or training clusters, the math flips back toward H100 once you factor in engineering time spent tuning ROCm kernels, which isn’t free even if the GPU-hour rate is lower.

NVIDIA’s H100 supply also normalized considerably through 2024 — lead times that stretched past 11 months in early 2023 shrank to weeks by Q3 2024 as TSMC capacity expanded and Blackwell approached. That supply advantage, plus a CUDA ecosystem most ML engineering teams already know, keeps H100 the default choice even when MI300X wins a specific benchmark. For a broader look at how compute availability shapes model release timing, see our coverage on GPU supply chain bottlenecks over on AI Discovery Digest.

Get the AI tools that actually move the needle

Join our newsletter for hands-on AI workflows, tested tools, and the occasional money-saving tip — no hype.

Subscribe free