Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
The Unsloth documentation provides a guide for running Qwen3.8 models locally, covering the 27B vision-and-reasoning model with 256K context that runs on about 17GB VRAM and the massive 2.4T-A95B model with rival GPT-5 class performance. It details recommended thinking vs instruct sampling settings, the Preserve Thinking feature for continued conversations, and hardware requirements across quantizations from 4-bit to 1-bit. The guide explains running models via Unsloth Desktop with automatic offloading and multi-GPU detection, and via llama.cpp with custom builds for IQ1_XXXS quants, plus NVFP4 options for Blackwell GPUs.
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.
