This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Brendan McKeag writes about Qwen3.8-27B, a dense 27B-parameter vision-language model from Alibaba that achieves frontier-class agentic coding performance while fitting on a single 24GB GPU. The model's hybrid attention stack (48 linear-attention layers against 16 full attention layers) slashes KV cache memory, and its dense architecture makes it far more practical for fine-tuning than the MoE alternatives dominating the industry. McKeag argues the model serves best as an active tool-calling assistant rather than an encyclopedia, and its small footprint makes it uniquely suited for Runpod Serverless scale-to-zero deployments where cold start is dominated by bytes on disk.
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
simongonzalezdc writes about a serving-side tool-attention failure in Qwen3.8-27B via llama-server (jinja templates) where the position of tools in the rendered payload determines whether the model will invoke them. With an 8-tool list, early-positioned, description-less tools were verbally refused as "not wired up" (0/6 success), while the same tools worked perfectly when moved to the tail (3/3) or given a one-line description (6/6). A follow-up with 13 tools confirmed the pattern: a mid-list pair prescribed by name failed 0/5, but tail placement restored 3/3 clean invocation.
- Hypothesis: later tools in the rendered template output receive more attention; compact single-JSON-blob rendering may amplify the position effect versus per-tool natural-language rendering.
- Controlled matrix (24 runs + 27 curl controls) ruled out tool count, context/style, effort level, and fresh-session as causes.
- Test hardware: AMD Strix Halo (gfx1151/HIP) running Q4_K_XL quantization on current llama-server.
- Full methodology and ledgers published in a companion repo: KyaniteLabs/qwen38-27b-strix-halo.
- Hypothesis: later tools in the rendered template output receive more attention; compact single-JSON-blob rendering may amplify the position effect versus per-tool natural-language rendering.
- Controlled matrix (24 runs + 27 curl controls) ruled out tool count, context/style, effort level, and fresh-session as causes.
- Test hardware: AMD Strix Halo (gfx1151/HIP) running Q4_K_XL quantization on current llama-server.
- Full methodology and ledgers published in a companion repo: KyaniteLabs/qwen38-27b-strix-halo.
Gilbert Pagayon writes that Alibaba'''s Qwen team released Qwen3.8-27B, a 27-billion-parameter open-weight model under Apache 2.0 that can run quantized versions on consumer hardware for coding, research, multimodal tasks and long agent work while keeping data local. The model uses a dense hybrid-attention architecture with flexible reasoning mode and multi-token prediction, and Alibaba'''s benchmarks show it outperforming Claude Opus 4.6 Max on some coding tests but lagging on others. Early hands-on reports praise its capability for its size, though performance varies with quantization, hardware and settings, making the Opus rivalry a signal rather than a verdict.
- Native context is 262,144 tokens, extendable to ~1 million tokens with configuration
- Vision encoder handles images and video natively; scores 91.1 on OmniDocBench 1.5
- Q4_K_M GGUF is ~17.8GB, fitting a 24GB GPU; full BF16 is ~54.7GB
- Multi-token prediction speed gains depend on workload and engine and can be slower in some cases
- Native context is 262,144 tokens, extendable to ~1 million tokens with configuration
- Vision encoder handles images and video natively; scores 91.1 on OmniDocBench 1.5
- Q4_K_M GGUF is ~17.8GB, fitting a 24GB GPU; full BF16 is ~54.7GB
- Multi-token prediction speed gains depend on workload and engine and can be slower in some cases
