This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Brendan McKeag writes about Qwen3.8-27B, a dense 27B-parameter vision-language model from Alibaba that achieves frontier-class agentic coding performance while fitting on a single 24GB GPU. The model's hybrid attention stack (48 linear-attention layers against 16 full attention layers) slashes KV cache memory, and its dense architecture makes it far more practical for fine-tuning than the MoE alternatives dominating the industry. McKeag argues the model serves best as an active tool-calling assistant rather than an encyclopedia, and its small footprint makes it uniquely suited for Runpod Serverless scale-to-zero deployments where cold start is dominated by bytes on disk.
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
- The model is roughly 3x more token-hungry than Qwen3.6-27B, trading wall-clock speed for better answers
- It crossed 1M Hugging Face downloads within days of release; the community nickname is "local Opus"
- Unsloth shipped QLoRA support the day after weights dropped; 4-bit QLoRA is the well-trodden dense route, while MoE QLoRA at 4-bit is explicitly unsupported
- FlashBoot snapshots require loading the model at worker boot before runpod.serverless.start() to be effective
- Harvey and Engram reported it averaging 67% across 250 legal tasks in a synthetic law firm of ~10,000 documents, ahead of every model in that study
- The built-in multi-token prediction head enables speculative decoding without a separate draft model
Meta Superintelligence Lab writes that Muse Glimmer-30B is a 30-billion-parameter vision-language model optimized for autonomous agentic workflows on consumer-grade hardware. The architecture combines a dense causal transformer with a dedicated ~1.8-billion-parameter vision encoder to process interleaved text and images, enabling multi-step planning, reliable tool invocation, and automatic error recovery. Designed to run locally without cloud dependency, the model employs 4-bit quantization and a novel DFlash speculative decoding drafter to achieve significant speedups on devices with 24 to 32 GB of VRAM. Evaluated against comparable 27 to 31 billion parameter systems, Muse Glimmer demonstrates strong performance across agentic, coding, and multimodal reasoning benchmarks while maintaining strict safety guardrails and supporting over 100 languages.
- Trained on data curated from public sources, third parties, and Meta's internal products, with a knowledge cutoff of January 2026.
- Supports controllable reasoning strength (low, medium, high, xhigh) to balance output quality and inference speed.
- Includes a frozen ViT-G/14 perception encoder and releases both full-precision BF16 weights and two 4-bit quantized variants.
- Recommended inference settings include a temperature of 1.0, top-p of 0.95, and top-k of 64.
- Assessed for moderate or lower risk in cyber, loss-of-control, and chemical/biological domains, though explicit safety guardrails are still recommended for deployment.
- Trained on data curated from public sources, third parties, and Meta's internal products, with a knowledge cutoff of January 2026.
- Supports controllable reasoning strength (low, medium, high, xhigh) to balance output quality and inference speed.
- Includes a frozen ViT-G/14 perception encoder and releases both full-precision BF16 weights and two 4-bit quantized variants.
- Recommended inference settings include a temperature of 1.0, top-p of 0.95, and top-k of 64.
- Assessed for moderate or lower risk in cyber, loss-of-control, and chemical/biological domains, though explicit safety guardrails are still recommended for deployment.
Pedro Cuenca writes Meta released Muse Glimmer-30B, a local, open-source multimodal model distilled from its larger Muse architecture. Designed for agentic workflows, it combines a 28B text decoder with a 2B vision encoder, supporting image, video, and multimodal tool calling out of the box. The release includes immediate compatibility with major inference frameworks like transformers, llama.cpp, and vLLM, alongside built-in speculative decoding for faster generation.
- Features a hybrid attention pattern alternating between three sliding window layers and one full attention layer.
- Incorporates a DFlash block-diffusion drafter to accelerate structured text generation like coding.
- Supports fine-tuning via TRL with practical minimums ranging from one to eight H100 GPUs depending on the method.
- Demonstrates autonomous agent capabilities such as self-quantization, self-deployment, and hardware-specific optimization.
- Features a hybrid attention pattern alternating between three sliding window layers and one full attention layer.
- Incorporates a DFlash block-diffusion drafter to accelerate structured text generation like coding.
- Supports fine-tuning via TRL with practical minimums ranging from one to eight H100 GPUs depending on the method.
- Demonstrates autonomous agent capabilities such as self-quantization, self-deployment, and hardware-specific optimization.
This edition of The Weekly Kaitchup reviews several NVFP4 quantization versions of the Qwen3.6 27B model, comparing NVIDIA's mixed-precision approach with community alternatives like Unsloth and PrismaQuant. It also details DSpark, DeepSeek's new speculative decoding method that uses a parallel draft backbone and a confidence head to significantly accelerate large language model generation speeds.
* Comparison of Qwen3.6 27B NVFP4 quantization variants
* Guidance on selecting models based on accuracy versus memory footprint
* Technical overview of DSpark architecture and suffix decay mitigation
* Performance improvements and vLLM support for DSpark
* Comparison of Qwen3.6 27B NVFP4 quantization variants
* Guidance on selecting models based on accuracy versus memory footprint
* Technical overview of DSpark architecture and suffix decay mitigation
* Performance improvements and vLLM support for DSpark
Google has released Multi-Token Prediction (MTP) drafters for the Gemma 4 model family to significantly accelerate inference speeds. By utilizing a specialized speculative decoding architecture, these drafters can deliver up to a 3x speedup without compromising output quality or reasoning capabilities. This technology addresses memory-bandwidth bottlenecks by allowing a lightweight drafter to predict multiple future tokens that are then verified in parallel by the larger target model.
Key points:
* Improved responsiveness for real-time chat, voice applications, and agentic workflows.
* Faster local development on personal computers and consumer GPUs.
* Enhanced performance and battery efficiency on edge devices.
* Architectural optimizations including KV cache sharing and activation utilization.
* Available now under the Apache 2.0 license via Hugging Face and Kaggle.
Key points:
* Improved responsiveness for real-time chat, voice applications, and agentic workflows.
* Faster local development on personal computers and consumer GPUs.
* Enhanced performance and battery efficiency on edge devices.
* Architectural optimizations including KV cache sharing and activation utilization.
* Available now under the Apache 2.0 license via Hugging Face and Kaggle.
The author explores the common frustration of running local Large Language Models (LLMs), where the gap between potential and usability is often caused by slow inference speeds. Instead of upgrading to larger, more complex models, the author discovered that implementing speculative decoding significantly improved the experience. This technique uses a smaller "draft" model to quickly predict tokens, which a larger "verification" model then checks. This process drastically increases speed and creates a smoother conversational flow without sacrificing the model's intelligence. By focusing on how models are run rather than just which models are used, users can make their self-hosted AI tools much more practical for daily use.
Zed introduces edit prediction powered by Zeta, an open-source model that anticipates developers' next edits, enhancing efficiency. The feature allows users to apply predicted edits with a single keystroke, integrating seamlessly with existing functionalities like language server completions. The article also covers methodologies like supervised fine-tuning, direct preference optimization, and speculative decoding to minimize latency, ensuring a fast editing experience.
