SemanticScuttle - klotz.me

Tags: gguf* + quantization*

0 bookmark(s) - Sort by: Date ↓ / Title /

1. Qwen3.8 27B GGUF Benchmark: Q4 to Q1 Accuracy and Token Efficiency

   Benjamin Marie writes about a comprehensive benchmark of 15 GGUF quantizations of Qwen3.8 27B, ranging from Q4_K_XL down to IQ1_M, evaluated using over 150 million tokens generated across roughly 8 days on an NVIDIA RTX Pro 6000. Using 950 prompts subsampled from MMLU-Pro, LiveCodeBench, and GPQA Diamond, he measures both accuracy and token efficiency to identify the lowest quantization level that retains at least 95% of the original BF16 model's performance.
   
   | Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
   |---|---|---|---|---|---|---|---|
   | 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
   | 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
   | 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
   | 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
   | 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
   | 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
   | 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
   | 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
   | 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
   | 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
   | 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
   | 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
   | 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
   | 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
   | 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
   
   `* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`

   2026-09-02 Tags: qwen3.8 27b, gguf, quantization, llama.cpp, l llm, benchmark, token efficiency, mmlu-pro, livecodebench, gpqa by klotz
2. GLM-5.3-Flash: How to Run Locally

   Unsloth provides documentation and support for running the GLM-5.3-Flash (ox-alpha) multimodal model locally using Unsloth Desktop or llama.cpp. Developed by Z.ai, this 320B parameter model features a hybrid sparse and linear attention architecture designed to improve scaling through Manifold-Constrained Hyper Connections. Users can utilize various quantization levels'' from 1-bit for low RAM requirements (approx. 93GB) up to higher bitrates for improved accuracy'' to run the model on hardware ranging from Mac systems to NVIDIA DGX Spark setups.
   
   - The model features three thinking modes: Low, High, and Max reasoning effort.
   - It is designed to rival Claude Opus 4.8 in coding and agentic benchmarks.
   - Unsloth's dynamic 1-bit quantization retains 71% of top-1% accuracy while being 85% smaller than the BF16 version.
   - The model can be run via a local API using `unsloth run` with llama-server runtime flags.

   2026-08-29 Tags: glm-5.3-flash, ox-alpha, z.ai, unsloth, llm, quantization, local inference, gguf by klotz
3. Qwen3.8 - How to Run Locally

   The Unsloth documentation provides a guide for running Qwen3.8 models locally, covering the 27B vision-and-reasoning model with 256K context that runs on about 17GB VRAM and the massive 2.4T-A95B model with rival GPT-5 class performance. It details recommended thinking vs instruct sampling settings, the Preserve Thinking feature for continued conversations, and hardware requirements across quantizations from 4-bit to 1-bit. The guide explains running models via Unsloth Desktop with automatic offloading and multi-GPU detection, and via llama.cpp with custom builds for IQ1_XXXS quants, plus NVFP4 options for Blackwell GPUs.
   
   - Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
   - Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
   - NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
   - New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD

   2026-08-15 Tags: qwen3.8, unsloth, local inference, gguf, quantization, nvfp4, llm by klotz
4. unsloth/Qwen3.8-27B-GGUF

   Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.

   2026-08-15 Tags: qwen3.8, gguf, unsloth, quantization, vision language model, llm, alibaba, local llm by klotz
5. Qwen3.6 - How to Run Locally

   This guide provides instructions for running Alibaba's Qwen3.6 multimodal hybrid-thinking models locally using Unsloth tools. It covers the 27B and 35B-A3B variants, which support a 256K context window across 201 languages and excel in agentic coding, vision, and chat tasks. The article details hardware requirements for various quantization levels and explains how to leverage Multi Token Prediction (MTP) for significantly faster inference.
   Key topics:
   - Hardware memory requirements for quantized models
   - Faster generation via Multi Token Prediction (MTP)
   - Integration with Unsloth Studio, llama.cpp, and MLX
   - Preserved thinking mode configurations

   2026-08-05 Tags: qwen3.6, alibaba, unsloth, local llm, llm, mtp, gguf, mlx, quantization, multimodal, unaloth by klotz
6. Qwen3.6 GGUF Benchmarks

   Unsloth AI presents performance benchmarks for Qwen3.6-35B-A3B GGUF quantizations, claiming state-of-the-art results in mean KL divergence across most model sizes. The discussion includes community analysis regarding SWE-bench Verified performance, where some users noted unexpected discrepancies between Qwen3.5 and Qwen3.6 quantization results during coding tasks.
   Key points:
   - Unsloth ranks first in 21 of 22 model sizes for mean KL divergence.
   - Community debate over SWE-bench testing methodology and sample sizes.
   - Reported performance variations between different quantization levels (Q4, Q5, Q6, Q8).
   - Discussion on system prompt adherence and error rates in coding benchmarks.

   2026-04-18 Tags: unsloth, qwen3.6, gguf, benchmarks, quantization, swe-bench, llm performance by klotz
7. Bonsai-8B-GGUF-1bit

   Bonsai-8B-GGUF-1bit is an end-to-end 1-bit language model designed for high-efficiency deployment using llama.cpp across CUDA, Metal, and CPU architectures. This model provides a massive 14.1x reduction in memory footprint compared to standard FP16, requiring only 1.15 GB of parameter memory. By leveraging the GGUF Q1_0_g128 format, it achieves significant performance boosts, including 6.2x faster throughput on an RTX 4090 and substantially lower energy consumption per token. It is an ideal solution for on-device assistants, mobile applications, and edge robotics where memory, thermal, and power constraints are paramount.

   2026-04-05 Tags: bonsai-8b, 1-bit language model, gguf, llama.cpp, quantization, prism ml, on-device ai, efficiency by klotz
8. Bonsai 1-bit LLM

   This collection, curated by prism-ml, features a specialized series of 1-bit Bonsai models designed for efficient text generation. The repository includes various model architectures and sizes, such as the 8B, 4B, and 1.7B parameter versions, optimized through extreme quantization. Available in formats like GGUF and MLX-1bit, these models are highly compressed to maximize performance while minimizing the computational footprint. This makes them ideal for running large language model tasks on hardware with limited resources. The collection serves as a hub for exploring the potential of ultra-compact, highly compressed models in the evolving landscape of machine learning and efficient inference.

   2026-04-05 Tags: bonsai, 1-bit models, prism-ml, text generation, gguf, mlx, quantization, llm, machine learning by klotz
9. Qwen3.5 GGUF Benchmarks

   This article details benchmarks for Unsloth Dynamic GGUFs of the Qwen3.5 model, including analysis of perplexity, KL divergence, and MXFP4. It covers performance across different bit widths and quant types, highlighting the impact of Imatrix and the limitations of certain quantization approaches. Full benchmark data is also provided.

   2026-03-01 Tags: qwen3.5, gguf, benchmarks, quantization, perplexity, kl divergence, mxfp4, imatrix, llm, inference, dynamic quantization, unsloth by klotz
10. Unsloth Dynamic GGUFs on Aider Polyglot

    This article details the performance of Unsloth Dynamic GGUFs on the Aider Polyglot benchmark, showcasing how it can quantize LLMs like DeepSeek-V3.1 to as low as 1-bit while outperforming models like GPT-4.5 and Claude-4-Opus. It also covers benchmark setup, comparisons to other quantization methods, and chat template bug fixes.

    2025-10-13 Tags: unsloth, gguf, aider polyglot, llm, quantization, deepseek-v3.1, gpt-4, claude-4, model compression, fine-tuning, inference by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0