H200 vs B200 vs H100: The GPU Cloud Cost-Per-Token Reality Check (2026)

The H200 vs B200 vs H100 cost per token comparison reveals an uncomfortable truth: the GPU you rent by the hour is rarely the GPU that minimizes your actual inference bill. A June 2026 paper published on arXiv (arXiv 2606.11690) demonstrated that effective cost-per-million-tokens varies by up to 36.3x on identical H100 hardware, purely as a function of request rate. The GPU market has answered this problem with three distinct memory tiers—H100, H200, and B200—each optimized for a different part of the inference cost curve. Understanding which tier fits your workload is the difference between rational infrastructure planning and paying 3x too much for the same tokens.

Why Your Hourly Rate Is a Lie: The 36x Variance Problem

Most GPU cost calculators assume your silicon runs at peak throughput. They divide the hourly rate by maximum tokens per second and hand you a dollar figure. That figure can be off by a factor of 36, and the arXiv paper proves it.

The research team ran 42 systematic vLLM benchmark configurations across H100 GPUs, varying arrival rate from 1 to 10 requests per second, and measured effective cost-per-million-tokens at each level. At 1 request per second, Mixtral FP16 cost $15.25 per million tokens—more expensive than Claude Sonnet 4.6’s API price at the time. At 5 requests per second, the same setup dropped to $3.16 per million tokens, undercutting every major API tier. The hardware changed nothing. The utilization did everything.

This creates the real framework for comparing H100, H200, and B200: before you ask which GPU is faster or cheaper per hour, you need to ask what your utilization profile looks like—and more critically, which GPU architecture eliminates the constraint that caps your utilization in the first place. For most 70B-and-above inference workloads, that constraint is not compute. It is memory.

The Memory Bottleneck That Splits H100, H200, and B200 Apart

LLM inference is memory-bound, not compute-bound. When a GPU serves a transformer-based model, the bottleneck is almost never tensor core utilization—it is the speed at which weights and KV cache data flow from HBM (High Bandwidth Memory) to the tensor cores. The GPU waits for data. More memory capacity and more bandwidth directly translate to higher concurrent batch sizes, longer context windows, and more tokens per second at equivalent quality.

The engineering intuition behind H200 and B200 starts here. Both chips exist because NVIDIA identified that the H100’s 80 GB HBM3 and 3.35 TB/s bandwidth were becoming the limiting factor for production inference of 70B+ models at scale—not the compute budget. Upgrading the memory subsystem costs less in silicon area and power than upgrading compute, and it directly addresses the bottleneck that matters.

Three different memory architectures define three distinct performance ceilings. What follows is not a review of paper specifications. It is an analysis of which ceiling binds your specific workload—and at what cost.

Takeaway: H200 and H100 share identical compute throughput—the only difference is memory capacity (+76%) and bandwidth (+43%). B200 adds a fundamentally different compute tier (FP4) that has no equivalent on Hopper. Sources: NVIDIA H200 datasheet, FaceOfIT B200 comparison, May 2026.

H100 SXM5: When Ubiquity and Price Beat Everything Else

The H100 SXM5 is the most deployed AI GPU on the market. That ubiquity translates directly into pricing pressure: as of May 2026, neocloud on-demand rates have fallen to $1.49–$3.29 per GPU-hour across more than 36 providers, with spot pricing on Spheron reaching $1.03/hr. Reported by alatirok.com GPU Pricing Index, May 2026: the median on-demand H100 rate is $2.95/hr; RunPod Secure Cloud bills $3.29/hr with per-second granularity, and Lambda Labs starts at $2.49/hr.

For models that fit within 80 GB—Llama 3.1 8B, Mistral 7B, Llama 3.1 70B in INT4 quantization, Qwen 2.5 32B in BF16—the H100 delivers its full performance ceiling. MLPerf Inference benchmarks (MLCommons, 2025) show an 8-GPU H100 SXM node serving approximately 22,290 tokens per second on Llama 3.1 70B FP8 at a node cost of $22.80/hr—translating to roughly $0.28 per million tokens at full utilization.

The H100 also has a practical advantage that specs cannot capture: infrastructure maturity. Every major inference framework (vLLM, TensorRT-LLM, SGLang, llama.cpp) has been profiled and optimized on Hopper. Kernels are tuned, configurations are benchmarked, community support is dense. When you are debugging a production inference bottleneck at 2 AM, that matters. The H100 is the right choice when your model fits in 80 GB, you can maintain high utilization, and the 48% pricing premium of H200 on neoclouds is not justified by your workload.

H200 SXM: The Memory Upgrade That Changes Everything Above 70B

The H200 shares the exact same GH100 die as the H100. It has identical CUDA cores (16,896), identical FP8 throughput (3,958 TFLOPS), identical NVLink bandwidth, and identical power draw (700W). Every performance difference between the H200 and H100 comes from one source: the memory subsystem upgrade to 141 GB HBM3e at 4.8 TB/s—76% more capacity and 43% more bandwidth than the H100.

This precision matters because it tells you exactly when H200 wins and when it does not. If your inference workload fits comfortably in 80 GB and is not memory-bandwidth-constrained, the H200 will perform identically to the H100 at a higher hourly rate. Paying $3.59/hr on RunPod for an H200 instead of $2.69/hr for an H100 Community Cloud pod nets you zero performance improvement for a compute-bound 7B model running at low concurrency.

The inflection point is 70B-class inference at production concurrency. A Llama 3.1 70B model in BF16 requires approximately 140 GB of VRAM for weights alone—which means it does not fit on a single H100. It does fit on a single H200, eliminating one layer of tensor parallelism overhead. For long-context inference (32K–128K context windows), the KV cache grows linearly with sequence length. NVIDIA’s own testing shows the H200 delivers up to 3.4x higher throughput versus H100 for long-context workloads where the H100 runs out of memory (RunPod H200 analysis, 2026). For standard batch inference at 70B, the improvement is more modest: 25–60% higher throughput on memory-bandwidth-bound workloads (emma.ms H200 vs H100, 2026).

Reported by Spheron GPU Cloud Benchmarks, April 2026: an 8-GPU H200 node serving Llama 3.1 70B FP8 reaches approximately 31,700 tokens per second at a node price of roughly $29.76/hr—translating to approximately $0.26 per million tokens. Compared to the H100’s $0.28/MTok, the H200 costs 7% less per token at equivalent utilization, despite a 30% higher hourly sticker. The math inverts entirely when the H100 must split the model across two GPUs: tensor parallelism overhead, doubled inter-GPU communication, and more complex orchestration push H100 cost-per-token above H200 for 70B BF16 serving.

Takeaway: H200 is not universally better. It wins decisively when memory capacity or bandwidth is the bottleneck—which begins reliably at 70B parameters in BF16 at production concurrency. Below that threshold, the H100’s lower price is the winning argument.

B200 SXM6: When FP4 Precision Becomes an Economic Lever

The B200 is not a Hopper upgrade. It is a fundamentally different architecture. NVIDIA’s Blackwell B200 uses a dual-die chiplet design—two GH100-class dies connected via NV-HBI at 10 TB/s—packed into a single module with 192 GB HBM3e and 8.0 TB/s memory bandwidth. That bandwidth figure is 2.4x H100 and 1.67x H200. But the architectural change that most affects cost-per-token economics is FP4 precision support.

FP4 quantization halves the memory footprint of model weights compared to FP8, and the B200’s tensor cores process FP4 at 9,000 TFLOPS—twice the FP8 throughput. For frontier-scale inference (Llama 3.1 405B, Qwen 3 235B-A22B MoE, or similar), FP4 can deliver 2.5–4.5x more tokens per second than an equivalent H100 in FP8 (GPUaaS.com B200 analysis, 2026). At sufficient scale, this throughput advantage offsets the higher hourly rate and produces the lowest cost per token in the market.

The honest caveat is that B200 production supply constraints remain real as of June 2026. NVIDIA’s B200 backlog was reportedly 3.6 million units in mid-2025, and while availability has improved, spot pricing from Spheron ($2.12/hr) reflects that the economics of B200 deployment are still stabilizing. More critically: B200 requires liquid cooling infrastructure. Air-cooled H100/H200 instances can be spun up in seconds on dozens of neoclouds. B200 clusters are substantially rarer, and self-hosting requires engineering investment that most teams cannot justify below 100B-parameter model serving at scale.

Reported by Prompt20 NVIDIA AI GPU Lineup, 2026: The Blackwell jump delivered the largest single-generation leap NVIDIA has shipped in five years. For workloads using FP4, the cost-per-token economics on B200 are roughly half of H100. For workloads that do not use FP4—because the model is too small to benefit, or because the inference framework has not implemented FP4 support—B200’s advantage over H200 narrows to the bandwidth gap (8.0 vs 4.8 TB/s), which still delivers 30–60% higher throughput but at 35–50% higher hourly cost.

H200 vs B200 vs H100 Cost Per Token: Three Models, Three Verdicts

The H200 vs B200 vs H100 cost per token divergence becomes tangible only with concrete numbers. Abstract benchmark figures mean little without a calculation a team can adapt to their workload. The following scenarios use actual benchmark data and published pricing as of June 2026. Each assumes an 8-GPU SXM node at 80% utilization—a reasonable target for a production endpoint with reasonable spare capacity. All “on-demand neocloud” prices use the cheaper end of the neocloud range (RunPod Community/Lambda Labs floor) to represent what a cost-conscious team can actually obtain. Hyperscaler pricing is included to show the premium paid for brand reliability and compliance tiers.

* 405B FP8 requires multi-node inference on H100/H200; single 8-GPU node estimates assume 2-node tensor parallelism overhead included. † Estimated from MLPerf scaling ratios. ‡ B200 FP4 figures from Spheron benchmark data and NVIDIA official projections; FP8 baseline at ~2.5x H100 throughput. Sources: Spheron GPU Cloud Benchmarks 2026, MLCommons Inference Benchmarks, arXiv 2606.11690, June 2026.

Takeaway: For 70B inference, all three GPUs converge to a similar cost-per-token at full utilization on neoclouds. The B200’s advantage emerges at 405B+ scale, where its FP4 support and 2.39x bandwidth over H100 cut 405B inference cost nearly in half compared to H100. The hyperscaler column shows why provider selection matters as much as GPU selection: an H100 on Azure costs more per token than a B200 on RunPod.

Provider Pricing Reality: Neoclouds vs Hyperscalers in June 2026

The GPU you select is only half the cost equation. Where you rent it determines whether the cost-per-token calculation above reflects reality or lives in a spreadsheet. Neocloud pricing (RunPod, Lambda Labs, GMI Cloud, Nebius, Spheron) runs 3–6x below hyperscaler rates for identical silicon, per the alatirok.com GPU Price Index tracking 58 providers as of May 2026. The practical tradeoff is compliance infrastructure and SLAs: AWS, Azure, and GCP offer HIPAA BAA, SOC 2 Type II, GDPR compliance frameworks, and 99.9%+ uptime SLAs. Neoclouds generally do not, or only on specific enterprise tiers.

Sources: alatirok.com GPU Pricing Index, May 2026, RunPod Provider Comparison, 2026, ComputePrices.com, June 2026, GMI Cloud pricing analysis, 2026.

Takeaway: At full utilization, GMI Cloud’s H100 at $2.00/hr is the lowest on-demand rate for teams without compliance requirements. For per-second billing granularity (critical for development workflows with frequent starts/stops), RunPod Secure is the standard reference. For B200 spot access at minimum spend, Spheron’s $2.12/hr spot tier is the market floor—with the caveat that preemptible instances cannot serve production inference endpoints reliably.

Match Your Workload to the Right Silicon: A Decision Framework

Four variables determine your optimal GPU tier. Model size in the precision you will actually deploy (not FP32 paper weights). Context window you serve in production (P95 request length, not maximum possible). Request concurrency (peak simultaneous requests your endpoint must handle). Compliance posture (whether your data requires a specific cloud provider or geographic constraint). The decision tree below addresses each combination in order of most to least common for AI infrastructure teams in 2026.

Takeaway: For the majority of teams running 70B-class inference in 2026, H200 on a neocloud is the rational default—not H100. The pricing premium is real (30–65% higher hourly), but it disappears or inverts at the cost-per-token level for memory-bound workloads above 70B.

The Strongest Objection to Renting H200 Right Now

The objection most worth taking seriously: H200 supply on neoclouds is thinner than H100. As GMI Cloud’s analysis notes, H200 availability is “still thin across all providers” as of June 2026, and listed prices do not help if the instance is not available when your deployment requires it. An H100 you can actually rent beats an H200 you cannot.

This objection is correct in practice—but it applies specifically to on-demand H200 provisioning. For teams that can commit to reserved 1-year or 3-year contracts, H200 reserved pricing has stabilized to $2.00–$2.50/hr across specialist providers (GPUaaS.com, 2026). At that price point—roughly equivalent to H100 spot—the H200’s throughput advantage for 70B+ becomes an unconditional win. The relevant question for infrastructure planning is not “can I get H200 today?” but “what does my utilization forecast look like for the next 12 months, and can I justify a reserved commitment based on that?”

A secondary objection worth addressing: quantized smaller models might substitute for H200’s capacity advantage. Running Llama 3.1 70B in INT4 reduces the VRAM requirement from ~140 GB BF16 to approximately 35 GB—fitting easily on a single H100. If INT4 quality meets your production requirements, the H100 wins on cost. The decision belongs to your evaluation team, not the hardware spec sheet.

Infrastructure Decision Checklist: H100, H200, or B200

The following checklist is ordered by impact on cost-per-token, not chronological setup order. Each step includes the role most responsible for the decision and a measurable success signal.

1. Benchmark your model’s actual VRAM consumption under production batch size (ML Engineer | Before any contract | Signal: measured peak VRAM < 75 GB → H100 candidate; 75–140 GB → H200 candidate; >140 GB or FP4 needed → B200)
2. Measure your P90 request rate over 30 days (Platform Engineer | Before GPU tier selection | Signal: <3 req/s sustained → utilization tax likely; calculate effective $/MTok at actual utilization, not theoretical maximum)
3. Check compliance requirements against provider certifications (Security / Legal | Before provider selection | Signal: HIPAA/SOC2/GDPR required → AWS or Azure; research data only → any neocloud)
4. Get H200 on-demand availability confirmation from your target provider before architecting around it (DevOps | Before architecture commit | Signal: provider confirms <15 min spin-up at your required node count → proceed; longer → design for H100 fallback)
5. Run a 48-hour cost-per-token pilot at target utilization (ML Engineer + Finance | After GPU selection | Signal: effective $/MTok within 20% of benchmark estimate → proceed; larger gap → investigate idle time or cold-start overhead)

What the Next Six Months Will Change

The H200 vs B200 vs H100 cost-per-token equation is not static. Three developments will shift it materially before Q1 2027. First, B200 supply is expanding—NVIDIA is shipping at scale, and reserved pricing will fall as neocloud providers absorb more units. The B200’s $2.25/hr reserved floor is already competitive with H100 on-demand, and that number will decline. Second, FP4 inference framework support is maturing rapidly; TensorRT-LLM and SGLang FP4 kernels are in active development, and production-ready FP4 inference for 70B models on B200 will materially change the comparison table above. Third, AMD MI400—covered in detail in our NVIDIA vs AMD AI Chip War 2026 analysis—will apply pricing pressure on the H100 tier if it ships on schedule, potentially pushing H100 on-demand rates below $2/hr by Q4 2026. Teams signing multi-year H100 reserved contracts today should pressure-test that assumption.

The hardest insight to internalize from the H200 vs B200 vs H100 cost per token data: the GPU tier you choose matters less than the utilization you achieve on it. A well-orchestrated H100 cluster at 85% utilization beats a poorly scheduled B200 cluster at 20% on every cost metric. Before upgrading your silicon, optimize your serving framework—and profile the utilization penalty you are paying right now.

Frequently Asked Questions

Can I run Llama 3.1 405B on a single H200?

No. Llama 3.1 405B in FP8 requires approximately 200 GB of VRAM for weights alone, exceeding the H200’s 141 GB. You need either multiple H200s (typically 2×) or B200 (192 GB per GPU, allowing single-GPU serving with quantization). In BF16 full precision, 405B requires 810 GB—four or more H200s.

Does the H200 support FP4?

No. FP4 tensor core support is exclusive to the Blackwell architecture (B200, B100, B300). Both H100 and H200 share the Hopper architecture’s FP8/INT8 precision ceiling. Workloads that require FP4 for quality or throughput reasons have no option except Blackwell-class hardware.

Is H100 spot pricing reliable enough for production inference?

Spot instances are preemptible by definition—providers can reclaim them with little notice. For production serving, spot is not viable without a fallback on-demand tier to absorb traffic during interruptions. For batch inference jobs, fine-tuning runs, or offline processing where interruption tolerance is built into the workflow, H100 spot at $1.03/hr (Spheron) or GCP A3 spot at $2.25/hr represents genuine cost savings of 50–60% versus on-demand.

At what scale does a B200 reserved contract beat H100 on-demand?

For 405B FP8 inference at sustained throughput, the B200 reserved rate of ~$2.25/hr generates roughly 2.5–3x more tokens per GPU-hour than H100 at the same reservation cost. At that rate, B200 reserved is cheaper per token than H100 on-demand for any team serving more than roughly 3 million tokens per GPU per day at 70%+ utilization. Below that volume, the flexibility of H100 on-demand matters more than the throughput premium.

How does H200 compare to the B200 for training (not inference)?

For training, B200’s FP4 advantage is less decisive—training typically uses BF16 or FP8, where B200 delivers roughly 14% higher TFLOPS than H200 (4,500 vs 3,958 FP8 TFLOPS) but costs 65% more per hour. Memory bandwidth (8.0 vs 4.8 TB/s) and the 2x NVLink 5 interconnect (1.8 TB/s vs 900 GB/s per GPU) benefit large model training at scale. For fine-tuning workloads on single-GPU or 8-GPU nodes, H200 is frequently the better economics unless you are training a 100B+ parameter model.

Related Reading

• NVIDIA vs AMD: The AI Chip Data Center War in 2026 — How the AMD MI400 roadmap and Intel Gaudi 3 are reshaping the competitive landscape that produced the H200 and B200 pricing environment above.
• The Great Unified Memory AI Showdown: DGX Spark vs. Mac Studio M4 Ultra (2026) — When on-device unified memory architectures compete with cloud GPU rental for 70B-class inference, and the cost-per-token calculus changes again.
• The Agent Autonomy Curve: GPU Infrastructure Planning for 2026–2027 — How autonomous AI agents shift the demand curve for the GPU tiers compared in this article.

Sources & Further Reading

• arXiv 2606.11690 — Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation (June 2026)
• NVIDIA H200 Official Datasheet — nvidia.com
• Spheron GPU Cloud Benchmarks 2026 — Updated April 2026
• MLCommons MLPerf Inference Datacenter Benchmarks
• GPU Cloud Pricing 2026: H100, H200, and B200 Costs — alatirok.com
• H100 vs H200 vs B200: Which GPU to Rent in 2026 — GPUaaS.com
• NVIDIA H200 GPU Guide — RunPod, 2026
• NVIDIA H200 vs H100: The 2026 Comparison — emma.ms
• GPU Rental Pricing: RunPod vs Lambda vs CoreWeave vs GMI Cloud — GMI Cloud
• Lambda Labs vs RunPod GPU Pricing — ComputePrices.com, June 2026
• NVIDIA AI GPU Lineup 2026: B200, H100, H200, A100, L40S — Prompt20 Blog
• NVIDIA B200 vs H100 vs A100: GPU Comparison & Benchmarks — FaceOfIT, 2026

Related reading: TCO and power for GPU clusters

Cost-per-token analysis sits inside a broader CapEx decision. Pair this article with GPU cluster TCO 2026 and data center power planning.

Author: Iovanny Olguín Ávila

Computer Systems Engineer with an MSc in Computer Science. I apply quantitative analysis and data-driven methodologies to evaluate financial instruments, investment vehicles, and emerging technologies. My technical background allows me to cut through marketing language and analyze the actual mechanics of financial products — from HELOC structures to Medicare Advantage plan design to business credit card reward algorithms.