Saga Ishtardottir writes that CyberTiel is an abliterated, quantized MoE model designed for high-speed agentic coding and offensive security tasks. Based on the Ornith-1.5-35B-A3B architecture, this version features a grafted multi-token prediction (MTP) head to enable speculative decoding via llama.cpp and uses a custom cyber-weighted importance matrix to maintain performance in low-bit quantizations. The model is optimized for real-world programming problems, showing significant improvements on benchmarks like SWE-bench-Live compared to its censored counterparts.
- Includes an abliterated (uncensored) base that allows the model to perform offensive security work without refusal.
- Features a grafted MTP head at block 40 specifically designed for speculative decoding speedups in llama.cpp.
- Uses Unsloth Dynamic quantization with a custom imatrix calibrated on specialized cybersecurity and code datasets.
- Capable of vision tasks via the mmproj-BF16.gguf projector from Ornith's original model.
- Includes an abliterated (uncensored) base that allows the model to perform offensive security work without refusal.
- Features a grafted MTP head at block 40 specifically designed for speculative decoding speedups in llama.cpp.
- Uses Unsloth Dynamic quantization with a custom imatrix calibrated on specialized cybersecurity and code datasets.
- Capable of vision tasks via the mmproj-BF16.gguf projector from Ornith's original model.
This repository provides optimized serving configurations for the Qwen3.8-27B model running on a single 24 GB consumer GPU (RTX 3090). It achieves high throughput of approximately 1,000 tok/s at 64 concurrent users in batch mode and up to ~133 tok/s for single-user scenarios using speculative decoding techniques like MTP or DFlash2. The project includes custom patches, requantization scripts (such as int8 tensor-core GEMMs), and Docker support to enable extended context windows of 150k to 262k tokens on a single consumer card.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
- Offers two distinct serving profiles: `batch` for high throughput/API backends and `single-user` for low-latency chat experiences.
- Implements advanced speculative decoding modes including MTP (Multi-Token Prediction) and DFlash2 block drafting.
- Supports extreme context lengths up to 262k tokens through KVarN, which utilizes a lossy 4/2-bit KV cache.
- Includes specialized optimizations like int8 activations, quantized lm_head, and split-KV verify attention to maximize VRAM efficiency.
Google has released Multi-Token Prediction (MTP) drafters for the Gemma 4 model family to significantly accelerate inference speeds. By utilizing a specialized speculative decoding architecture, these drafters can deliver up to a 3x speedup without compromising output quality or reasoning capabilities. This technology addresses memory-bandwidth bottlenecks by allowing a lightweight drafter to predict multiple future tokens that are then verified in parallel by the larger target model.
Key points:
* Improved responsiveness for real-time chat, voice applications, and agentic workflows.
* Faster local development on personal computers and consumer GPUs.
* Enhanced performance and battery efficiency on edge devices.
* Architectural optimizations including KV cache sharing and activation utilization.
* Available now under the Apache 2.0 license via Hugging Face and Kaggle.
Key points:
* Improved responsiveness for real-time chat, voice applications, and agentic workflows.
* Faster local development on personal computers and consumer GPUs.
* Enhanced performance and battery efficiency on edge devices.
* Architectural optimizations including KV cache sharing and activation utilization.
* Available now under the Apache 2.0 license via Hugging Face and Kaggle.
