This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Brendan McKeag writes about Qwen3.8-27B, a dense 27B-parameter vision-language model from Alibaba that achieves frontier-class agentic coding performance while fitting on a single 24GB GPU. The model's hybrid attention stack (48 linear-attention layers against 16 full attention layers) slashes KV cache memory, and its dense architecture makes it far more practical for fine-tuning than the MoE alternatives dominating the industry. McKeag argues the model serves best as an active tool-calling assistant rather than an encyclopedia, and its small footprint makes it uniquely suited for Runpod Serverless scale-to-zero deployments where cold start is dominated by bytes on disk.
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
