Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
Leela Kumili writes about DoorDash's Flux platform, which moved LLM-based coding agent workloads from developer laptops to cloud infrastructure, automating 130,000 engineering tasks in a single month and supporting over 25,000 automated code reviews per week. The platform was built after local execution hit limits on power, uptime, and security, and is organized around four primitives: Firecracker microVM sandboxes, an in-house MCP gateway, YAML-defined playbooks, and multiple invocation surfaces (Slack, GitHub, cron, CLI, conversational interfaces).
- Sandboxes achieve a 95th percentile SLO under 5 seconds for full setup, including microVM start, repo clone, build tool install, and agent harness configuration.
- Playbooks can mix agent-driven steps with deterministic code where predictable execution or validation is required.
- The Agent Gateway enforces scoped permissions and logs all agent activity for auditing and policy enforcement.
- DoorDash switched Slack integration from private channels to public threads so engineers could observe agent executions and see how other teams delegated work.
- Sandboxes achieve a 95th percentile SLO under 5 seconds for full setup, including microVM start, repo clone, build tool install, and agent harness configuration.
- Playbooks can mix agent-driven steps with deterministic code where predictable execution or validation is required.
- The Agent Gateway enforces scoped permissions and logs all agent activity for auditing and policy enforcement.
- DoorDash switched Slack integration from private channels to public threads so engineers could observe agent executions and see how other teams delegated work.
This article explores the emerging category of AI-powered operations agents, comparing AI DevOps engineers and AI SRE agents, how cloud providers are responding, and what engineers should consider when evaluating these tools.
Late last year, startup Platform Engineering Labs made waves in the world of Infrastructure as Code (IaC) by introducing a new IaC platform, called Formae, available initially on Amazon Web Services. This week, Platform Engineering Labs‘ platform gets (beta) support from additional cloud platforms, including Google Cloud Platform, Microsoft Azure, Oracle Cloud Infrastructure, and OVHcloud. The company has also released new AI-enhanced software for managing infrastructure tooling, called the Platform for Infrastructure Builders.
AI is revolutionizing Infrastructure as Code (IaC), enhancing speed, intelligence, and responsiveness. However, human expertise remains crucial for understanding AI-generated outputs and ensuring proper system functionality.
K8S-native cluster-wide deployment for vLLM. Provides a reference implementation for building an inference stack on top of vLLM, enabling scaling, monitoring, request routing, and KV cache offloading with easy cloud deployment.
Microsoft has deployed GPT-4, a large language model, in an isolated, air-gapped Azure Government Top Secret cloud for use by the Department of Defense. Once accredited, Pentagon officials will be able to use the technology in a secure environment. The tool is expected to help DOD officials deal with vast amounts of data and simplify information sorting. Microsoft is a major investor in OpenAI, the maker of GPT-4 and the popular ChatGPT.
