NVIDIA Sets New LLM Speed ​​Record With Llama 4 Maverick | How was chatgpt trained | Large language models tutorial pdf free | Llm python -- tutorial | Turtles AI

NVIDIA Sets New LLM Speed ​​Record With Llama 4 Maverick
Blackwell GPUs and deep software optimizations surpass 1,000 tokens per second per user
Editorial Team24 May 2025

 

NVIDIA has set a new benchmark in large language model (LLM) inference, surpassing the 1,000 tokens per second per user threshold with the 400 billion parameter Llama 4 Maverick model. This performance was achieved using a single DGX B200 node equipped with eight Blackwell GPUs, marking a significant advancement in reducing latency and increasing throughput for generative AI applications.

Key Points:

  • Over 1,000 tokens per second per user on Llama 4 Maverick
  • 72,000 tokens per second per server with DGX B200
  • Software optimizations with TensorRT-LLM and EAGLE-3 speculative decoding
  • Use of FP8 data formats for GEMM, MoE, and Attention operations


NVIDIA’s Blackwell architecture, with its fifth-generation Tensor Cores, enabled this record-breaking performance. Software optimizations, in particular the use of TensorRT-LLM and EAGLE-3 speculative decoding, significantly contributed to the increase in inference speed while maintaining model accuracy. The adoption of the FP8 data format for GEMM, Mixture of Experts (MoE), and Attention operations allowed to reduce the size of the model and take advantage of the high productivity of Tensor Cores.

These innovations demonstrate NVIDIA’s commitment to improving the performance and efficiency of AI models, enabling the deployment of LLM at scale with reduced response times and optimized operating costs.