In collaboration with NVIDIA, researchers from SGLang have published early benchmarks of the GB200 (Grace Blackwell) NVL72 system, showing up to a 2.7× increase in LLM inference throughput compared to the H100 on the DeepSeek-V2 671B model.
The uplift is attributed to a suite of software optimizations built for the Blackwell architecture, including FP8-optimized matrix multiplication, accelerated attention kernels, and high-speed token routing over NVLink. These enhancements were integrated into the SGLang runtime to fully exploit GB200’s dense multi-GPU fabric and unified memory model.
NVIDIA’s GB200 NVL72 is positioned as a general-purpose platform for large-scale AI, spanning both training and inference. This benchmark, focused solely on inference, offers an early look at how the system performs under realistic load—before broader workloads like multi-trillion parameter training or multimodal serving are publicly tested.
In decoding benchmarks using a 2,000-token prompt, SGLang achieved 7,583 tokens per second per GPU—a 2.7× improvement over H100 HGX systems on the same workload. This level of throughput enables faster responses for large-context inputs and high concurrency, such as technical document summarization, codebase-aware AI assistants, and enterprise-scale retrieval-augmented generation (RAG). It also reduces the number of GPUs required to serve large models interactively, improving both latency and cost efficiency at scale.
The benchmark was conducted using DeepSeek-V2, a 671-billion parameter, decoder-only large language model. Released by DeepSeek, the model follows a Mixture-of-Experts (MoE) design—activating ~21B parameters per token (≈ 9% of the total)—meaning only a fraction of the parameters are used during inference.
This architecture presents a realistic performance challenge: token routing between experts stresses inter-GPU communication, while the large model size and long prompt lengths place sustained pressure on GPU memory.
To achieve the reported speedup, the SGLang team integrated a series of Blackwell-specific optimizations into their runtime. These include DeepGEMM, a high-performance FP8 matrix multiplication library designed to exploit Blackwell’s new UMMA (Unified Matrix Multiplication Accelerator) instructions; FlashInfer FMHA (Fused Multi-Head Attention), a rewritten fused attention kernel optimized for the DeepSeek model’s prefill phase; and DeepEP, a communication library that efficiently shuffles tokens between routed experts using direct NVLink memory mapping.
The team also employed CUTLASS MLA (Multi-Head Latent Attention), a latent-attention kernel tuned for Blackwell’s memory hierarchy, and Mooncake, a custom transfer engine for disaggregated key-value (KV) cache movement.
Collectively, these components formed a software path that minimized compute, memory, and communication overhead during SGLang’s large-scale multi-GPU inference experiments.
The authors note that while the benchmark demonstrates significant gains in decoding throughput, several areas remain under-optimized. In particular, the prefill stage has yet to be fully tuned, and many kernels do not yet saturate the GB200’s memory bandwidth or compute capacity. Communication and compute are also not fully overlapped, leaving further efficiency gains on the table.
Future work will focus on closing these gaps, with plans to optimize prefill latency and improve memory utilization and latency.
