The Unsloth documentation provides a guide for running Qwen3.8 models locally, covering the 27B vision-and-reasoning model with 256K context that runs on about 17GB VRAM and the massive 2.4T-A95B model with rival GPT-5 class performance. It details recommended thinking vs instruct sampling settings, the Preserve Thinking feature for continued conversations, and hardware requirements across quantizations from 4-bit to 1-bit. The guide explains running models via Unsloth Desktop with automatic offloading and multi-GPU detection, and via llama.cpp with custom builds for IQ1_XXXS quants, plus NVFP4 options for Blackwell GPUs.
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.
This guide provides instructions for running Alibaba's Qwen3.6 multimodal hybrid-thinking models locally using Unsloth tools. It covers the 27B and 35B-A3B variants, which support a 256K context window across 201 languages and excel in agentic coding, vision, and chat tasks. The article details hardware requirements for various quantization levels and explains how to leverage Multi Token Prediction (MTP) for significantly faster inference.
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
Unsloth AI presents performance benchmarks for Qwen3.6-35B-A3B GGUF quantizations, claiming state-of-the-art results in mean KL divergence across most model sizes. The discussion includes community analysis regarding SWE-bench Verified performance, where some users noted unexpected discrepancies between Qwen3.5 and Qwen3.6 quantization results during coding tasks.
Key points:
- Unsloth ranks first in 21 of 22 model sizes for mean KL divergence.
- Community debate over SWE-bench testing methodology and sample sizes.
- Reported performance variations between different quantization levels (Q4, Q5, Q6, Q8).
- Discussion on system prompt adherence and error rates in coding benchmarks.
Key points:
- Unsloth ranks first in 21 of 22 model sizes for mean KL divergence.
- Community debate over SWE-bench testing methodology and sample sizes.
- Reported performance variations between different quantization levels (Q4, Q5, Q6, Q8).
- Discussion on system prompt adherence and error rates in coding benchmarks.
This article details benchmarks for Unsloth Dynamic GGUFs of the Qwen3.5 model, including analysis of perplexity, KL divergence, and MXFP4. It covers performance across different bit widths and quant types, highlighting the impact of Imatrix and the limitations of certain quantization approaches. Full benchmark data is also provided.
This article details the performance of Unsloth Dynamic GGUFs on the Aider Polyglot benchmark, showcasing how it can quantize LLMs like DeepSeek-V3.1 to as low as 1-bit while outperforming models like GPT-4.5 and Claude-4-Opus. It also covers benchmark setup, comparisons to other quantization methods, and chat template bug fixes.
