Saga Ishtardottir writes that CyberTiel is an abliterated, quantized MoE model designed for high-speed agentic coding and offensive security tasks. Based on the Ornith-1.5-35B-A3B architecture, this version features a grafted multi-token prediction (MTP) head to enable speculative decoding via llama.cpp and uses a custom cyber-weighted importance matrix to maintain performance in low-bit quantizations. The model is optimized for real-world programming problems, showing significant improvements on benchmarks like SWE-bench-Live compared to its censored counterparts.
- Includes an abliterated (uncensored) base that allows the model to perform offensive security work without refusal.
- Features a grafted MTP head at block 40 specifically designed for speculative decoding speedups in llama.cpp.
- Uses Unsloth Dynamic quantization with a custom imatrix calibrated on specialized cybersecurity and code datasets.
- Capable of vision tasks via the mmproj-BF16.gguf projector from Ornith's original model.
- Includes an abliterated (uncensored) base that allows the model to perform offensive security work without refusal.
- Features a grafted MTP head at block 40 specifically designed for speculative decoding speedups in llama.cpp.
- Uses Unsloth Dynamic quantization with a custom imatrix calibrated on specialized cybersecurity and code datasets.
- Capable of vision tasks via the mmproj-BF16.gguf projector from Ornith's original model.
This repository provides quantized GGUF versions of the Tiel coder model, which is a re-quantized and dynamically improved version of Ornith-1.5 35B designed specifically for agentic coding tasks. It excels at multi-turn conversations and solving codebase issues with performance comparable to Opus 4.6 medium in certain benchmarks.
- Optimized via an importance matrix (imatrix) using a calibration corpus heavily weighted toward code.
- Inherits the Ornith-1.5 vision tower, allowing for image-to-text tasks like analyzing screenshots of failing tests.
- Uses the Sharp chat template to produce more concise answers compared to its base model.
- Designed with a Mixture of Experts (MoE) architecture that enables fast inference and small KV cache footprint in RAM.
- Optimized via an importance matrix (imatrix) using a calibration corpus heavily weighted toward code.
- Inherits the Ornith-1.5 vision tower, allowing for image-to-text tasks like analyzing screenshots of failing tests.
- Uses the Sharp chat template to produce more concise answers compared to its base model.
- Designed with a Mixture of Experts (MoE) architecture that enables fast inference and small KV cache footprint in RAM.
Benjamin Marie writes about a comprehensive benchmark of 15 GGUF quantizations of Qwen3.8 27B, ranging from Q4_K_XL down to IQ1_M, evaluated using over 150 million tokens generated across roughly 8 days on an NVIDIA RTX Pro 6000. Using 950 prompts subsampled from MMLU-Pro, LiveCodeBench, and GPQA Diamond, he measures both accuracy and token efficiency to identify the lowest quantization level that retains at least 95% of the original BF16 model's performance.
| Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
|---|---|---|---|---|---|---|---|
| 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
| 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
| 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
| 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
| 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
| 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
| 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
| 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
| 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
| 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
| 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
| 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
| 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
| 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
| 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
`* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`
| Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
|---|---|---|---|---|---|---|---|
| 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
| 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
| 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
| 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
| 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
| 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
| 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
| 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
| 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
| 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
| 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
| 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
| 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
| 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
| 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
`* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`
Unsloth provides documentation and support for running the GLM-5.3-Flash (ox-alpha) multimodal model locally using Unsloth Desktop or llama.cpp. Developed by Z.ai, this 320B parameter model features a hybrid sparse and linear attention architecture designed to improve scaling through Manifold-Constrained Hyper Connections. Users can utilize various quantization levels'' from 1-bit for low RAM requirements (approx. 93GB) up to higher bitrates for improved accuracy'' to run the model on hardware ranging from Mac systems to NVIDIA DGX Spark setups.
- The model features three thinking modes: Low, High, and Max reasoning effort.
- It is designed to rival Claude Opus 4.8 in coding and agentic benchmarks.
- Unsloth's dynamic 1-bit quantization retains 71% of top-1% accuracy while being 85% smaller than the BF16 version.
- The model can be run via a local API using `unsloth run` with llama-server runtime flags.
- The model features three thinking modes: Low, High, and Max reasoning effort.
- It is designed to rival Claude Opus 4.8 in coding and agentic benchmarks.
- Unsloth's dynamic 1-bit quantization retains 71% of top-1% accuracy while being 85% smaller than the BF16 version.
- The model can be run via a local API using `unsloth run` with llama-server runtime flags.
The Unsloth documentation provides a guide for running Qwen3.8 models locally, covering the 27B vision-and-reasoning model with 256K context that runs on about 17GB VRAM and the massive 2.4T-A95B model with rival GPT-5 class performance. It details recommended thinking vs instruct sampling settings, the Preserve Thinking feature for continued conversations, and hardware requirements across quantizations from 4-bit to 1-bit. The guide explains running models via Unsloth Desktop with automatic offloading and multi-GPU detection, and via llama.cpp with custom builds for IQ1_XXXS quants, plus NVFP4 options for Blackwell GPUs.
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
- Qwen3.8-27B GGUFs include Developer Role Support for agentic tools like Codex and MTP enabled for fast inference
- Full-precision Qwen3.8-2.4T-A95B needs 4.9TB storage, reduced to 397GB with Unsloth Dynamic 1-bit GGUF ~91% smaller
- NVFP4 quants run ~1.5x faster than BF16 on Blackwell GPUs with 92-97% top-1 accuracy retention
- New 1-bit types TQ1_0/Q1_0 extend IQ1_S to 1.1875 bpw for large models without QAT/QAD
Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.
Pedro Cuenca writes Meta released Muse Glimmer-30B, a local, open-source multimodal model distilled from its larger Muse architecture. Designed for agentic workflows, it combines a 28B text decoder with a 2B vision encoder, supporting image, video, and multimodal tool calling out of the box. The release includes immediate compatibility with major inference frameworks like transformers, llama.cpp, and vLLM, alongside built-in speculative decoding for faster generation.
- Features a hybrid attention pattern alternating between three sliding window layers and one full attention layer.
- Incorporates a DFlash block-diffusion drafter to accelerate structured text generation like coding.
- Supports fine-tuning via TRL with practical minimums ranging from one to eight H100 GPUs depending on the method.
- Demonstrates autonomous agent capabilities such as self-quantization, self-deployment, and hardware-specific optimization.
- Features a hybrid attention pattern alternating between three sliding window layers and one full attention layer.
- Incorporates a DFlash block-diffusion drafter to accelerate structured text generation like coding.
- Supports fine-tuning via TRL with practical minimums ranging from one to eight H100 GPUs depending on the method.
- Demonstrates autonomous agent capabilities such as self-quantization, self-deployment, and hardware-specific optimization.
This page provides GGUF quantized versions of DiffusionGemma 26B A4B-it, a multimodal model from Google DeepMind based on the Gemma 4 architecture. The model employs discrete text diffusion through block-autoregressive multi-canvas sampling to achieve significantly faster decoding speeds than standard autoregressive models. It is capable of processing interleaved inputs consisting of text, images with variable resolutions and aspect ratios, and video content for generating textual outputs.
Key topics:
- Mixture-of-Experts architecture with 3.8 billion active parameters.
- High-speed generation through parallel denoising of token blocks.
- Multimodal input support including image and video understanding.
- Extensive context window capability up to 256K tokens.
- Integrated reasoning modes for step-by-step thought processes.
Key topics:
- Mixture-of-Experts architecture with 3.8 billion active parameters.
- High-speed generation through parallel denoising of token blocks.
- Multimodal input support including image and video understanding.
- Extensive context window capability up to 256K tokens.
- Integrated reasoning modes for step-by-step thought processes.
This guide provides instructions for running Alibaba's Qwen3.6 multimodal hybrid-thinking models locally using Unsloth tools. It covers the 27B and 35B-A3B variants, which support a 256K context window across 201 languages and excel in agentic coding, vision, and chat tasks. The article details hardware requirements for various quantization levels and explains how to leverage Multi Token Prediction (MTP) for significantly faster inference.
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
This repository provides the GGUF quantized weights for Qwen3.6-27B, a flagship-level coding model designed for stability and real-world utility. The model features significant upgrades in agentic coding capabilities, allowing it to handle frontend workflows and repository-level reasoning with high precision. It also introduces thinking preservation, which enables the model to retain reasoning context from historical messages to improve iterative development.
Key technical highlights:
* Native context length of 262,144 tokens, extensible up to 1,010,000 via RoPE scaling (YaRN).
* Enhanced tool-calling capabilities for complex agentic tasks.
* Support for multimodal inputs including images and video.
* Optimized for various inference frameworks like SGLang, vLLM, and KTransformers.
Key technical highlights:
* Native context length of 262,144 tokens, extensible up to 1,010,000 via RoPE scaling (YaRN).
* Enhanced tool-calling capabilities for complex agentic tasks.
* Support for multimodal inputs including images and video.
* Optimized for various inference frameworks like SGLang, vLLM, and KTransformers.
