Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
