Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
Amazon outages linked to rapid AI integration were discussed in a recent internal meeting. AI glitches in algorithms managing infrastructure caused disruptions (e.g., issues viewing product details, Freevee streaming). While Amazon is aggressively using AI, sources say the speed is creating instability. The company is focused on reliability amidst growing AI competition. Amazon declined to comment specifically but affirmed commitment to customer experience
Amazon introduces 'Ask this Book,' an AI-powered feature on the Kindle iOS app that allows readers to ask questions about the book they are reading and receive spoiler-free answers. The article also details the 'Recaps' feature for book series, providing a quick refresher on storylines and characters. 'Ask this Book' will expand to Kindle devices and Android OS next year.
Amazon has released Kiro, an AI-powered IDE designed for "vibe coding" that focuses on bringing prototypes into production with features like specs and hooks. It's powered by Claude 4 Sonnet and aims to apply software engineering best practices to the vibe-coding workflow.
AI pioneer Andrej Karpathy proposes an AI application that could enhance the e-book reading experience with interactive features. Amazon is reportedly looking for an applied scientist to improve Kindle's reading and publishing experience using AI.
Amazon's AI shopping assistant Rufus helps customers make informed decisions by answering shopping-related questions using a custom language model and innovative techniques in generative AI.
