Dmitry Soldatkin, Andrew Smith, and Vinay Arora write about deploying the open-weight Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM to support demanding agentic and reasoning workloads. The article provides a technical walkthrough for hosting this massive 2.4 trillion parameter mixture-of-experts (MoE) model on NVIDIA B300 Blackwell Ultra GPUs, covering infrastructure sizing with NVFP4 quantization, configuration of features like Multi-Token Prediction (MTP), and performance optimization through expert parallelism and prefix caching.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
- The model uses a hybrid architecture combining Gated DeltaNet layers for linear attention and Gated Attention layers for full quadratic attention to manage long context windows up to 1 million tokens.
- NVFP4 quantization reduces the model's weight footprint to ~1.2 TB, allowing it to fit on a single node with 8× NVIDIA B300 GPUs.
- Enabling Multi-Token Prediction (MTP) speculative decoding can reduce Time-To-First-Token (TTFT) by nearly 60%.
- Deployment is managed via the SageMaker HyperPod Inference Operator using Kubernetes (EKS) for automated lifecycle management and resilience.
/u/locbuilds on r/LocalLLM gives advice for an issue where the Qwen 3.8-27b model enters repetitive loops when making tool calls during debugging sessions. Community members suggest that this is often a bug within the agent harness rather than the model itself, recommending several technical mitigations to manage these failures effectively.
- Implement hard loop breakers in the application harness to detect and stop identical consecutive tool calls.
- Provide explicit "error" or "already tried" feedback in tool observations to signal failure back to the model.
- Lower temperature (0.1–0.3) for tool-heavy turns and apply repetition penalties via the sampler.
- Use specialized chat templates, such as Froggeric's Qwen fixed template, which may alleviate looping issues.
- Implement hard loop breakers in the application harness to detect and stop identical consecutive tool calls.
- Provide explicit "error" or "already tried" feedback in tool observations to signal failure back to the model.
- Lower temperature (0.1–0.3) for tool-heavy turns and apply repetition penalties via the sampler.
- Use specialized chat templates, such as Froggeric's Qwen fixed template, which may alleviate looping issues.
noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
This repository provides optimized Jinja chat templates designed to fix critical rendering errors, KV cache invalidation, and agentic stalling issues found in official Qwen 3.5 and 3.6 templates. It is compatible with major inference engines including LM Studio, llama.cpp, vLLM, and MLX.
> Lessons from building a fast, reliable scientific agent with local open-weight models, vLLM, and long-context infrastructure
This guide helps engineers build and ship LLM products by covering the full technical stack. It moves from core mechanics (tokenization, embeddings, attention) to training methodologies (pretraining, SFT, RLHF/DPO) and deployment optimizations (LoRA, quantization, vLLM). The focus is on managing critical production tradeoffs between accuracy, latency, memory, and cost
AMD now supports Google’s Gemma 4 models (2B–31B parameters) across its entire hardware lineup, including Instinct GPUs (datacenters), Radeon GPUs (workstations), and Ryzen AI processors (PCs). The integration is compatible with vLLM, SGLang, llama.cpp, Ollama, and Lemonade Server, aiming to optimize AI performance for both cloud and local deployment.
Qwen3-Coder-Next is an 80B MoE model with 256K context designed for fast, agentic coding and local use. It offers performance comparable to models with 10-20x more active parameters and excels in long-horizon reasoning, complex tool use, and recovery from execution failures.
