Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
> Lessons from building a fast, reliable scientific agent with local open-weight models, vLLM, and long-context infrastructure
This guide helps engineers build and ship LLM products by covering the full technical stack. It moves from core mechanics (tokenization, embeddings, attention) to training methodologies (pretraining, SFT, RLHF/DPO) and deployment optimizations (LoRA, quantization, vLLM). The focus is on managing critical production tradeoffs between accuracy, latency, memory, and cost
AMD now supports Google’s Gemma 4 models (2B–31B parameters) across its entire hardware lineup, including Instinct GPUs (datacenters), Radeon GPUs (workstations), and Ryzen AI processors (PCs). The integration is compatible with vLLM, SGLang, llama.cpp, Ollama, and Lemonade Server, aiming to optimize AI performance for both cloud and local deployment.
Qwen3-Coder-Next is an 80B MoE model with 256K context designed for fast, agentic coding and local use. It offers performance comparable to models with 10-20x more active parameters and excels in long-horizon reasoning, complex tool use, and recovery from execution failures.
This blog post explains the causes of nondeterminism in LLM inference, arguing that it's not simply due to floating-point non-associativity and concurrency, but rather a lack of batch invariance in kernels. It details how to achieve batch invariance in RMSNorm, matrix multiplication, and attention, and presents experimental results demonstrating deterministic completions and the benefits for on-policy RL.
The article discusses the evolution of model inference techniques from 2017 to a projected 2025, highlighting the progression from simple frameworks like Flask and FastAPI to more advanced solutions like Triton Inference Server and vLLM. It details the increasing demands on inference infrastructure driven by larger and more complex models, and the need for optimization in areas like throughput, latency, and cost.
This article details how to accelerate deep learning and LLM inference using Apache Spark, focusing on distributed inference strategies. It covers basic deployment with `predict_batch_udf`, advanced deployment with inference servers like NVIDIA Triton and vLLM, and deployment on cloud platforms like Databricks and Dataproc. It also provides guidance on resource management and configuration for optimal performance.
