Michal Sutter writes that the vLLM Semantic Router team has released Decision 3.0, a family of five open multimodal decision models ranging from 0.85B to 26.09B parameters. Fine-tuned on Qwen base models, these models accept text, JSON, and images and return calibrated probabilities for typed questions rather than generated text, making them useful for classification and routing tasks.
- All models carry a vision encoder and support multiple image inputs up to 1.6 megapixels each.
- The 9B d3-flash outperforms the previous generation's 27B model, showing significant efficiency gains.
- Benchmarks are self-reported against the Jev Decision Index, with d3 leading at 64.1 on text and 71.6 on vision.
- Median text latency ranges from 21.7 ms for d3-nano to 84 ms for d3 on a single AMD Instinct MI325X GPU.
- All models carry a vision encoder and support multiple image inputs up to 1.6 megapixels each.
- The 9B d3-flash outperforms the previous generation's 27B model, showing significant efficiency gains.
- Benchmarks are self-reported against the Jev Decision Index, with d3 leading at 64.1 on text and 71.6 on vision.
- Median text latency ranges from 21.7 ms for d3-nano to 84 ms for d3 on a single AMD Instinct MI325X GPU.
This documentation introduces how to fine-tune large language models into decision models using Unsloth. These models are designed to make decisions with calibrated probabilities rather than simply generating text. Users can train models like Qwen or Gemma to read input, score provided options, and return a specific choice. The process is supported through the Unsloth Desktop app, free notebooks, or custom Python code, and the resulting model can be integrated directly into the Decision API for practical use.
noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Michal Sutter writes that the Qwen Developer team has released zg (zvec-grep), an open-source local-first search layer designed to streamline how coding agents find information within a workspace. By unifying semantic search, BM25, and ripgrep under a single interface, it reduces tool calls and token usage for LLM agents that would otherwise struggle with manual context assembly or imprecise keyword matching.
- The package is available via npm as `@zvec/zvec-grep` under an Apache 2.0 license.
- It supports four retrieval routes: a hybrid default, BM25 (`--fts`), vector similarity (`--vector`), and literal/regex matching (`--rg`).
- An MCP (Model Context Protocol) integration allows seamless use with tools like Claude Code, Cursor, and Codex.
- Embeddings run locally by default using models such as `potion-code-16m-v2`, though remote Qwen endpoints are also supported via explicit authorization.
- Benchmarks suggest zg can cut tool calls and input tokens for coding agents by approximately 40% to 50%.
- The package is available via npm as `@zvec/zvec-grep` under an Apache 2.0 license.
- It supports four retrieval routes: a hybrid default, BM25 (`--fts`), vector similarity (`--vector`), and literal/regex matching (`--rg`).
- An MCP (Model Context Protocol) integration allows seamless use with tools like Claude Code, Cursor, and Codex.
- Embeddings run locally by default using models such as `potion-code-16m-v2`, though remote Qwen endpoints are also supported via explicit authorization.
- Benchmarks suggest zg can cut tool calls and input tokens for coding agents by approximately 40% to 50%.
This guide outlines the most effective approach for running large language models locally on hardware with 24GB of VRAM. It advises moving away from squeezing extremely large parameter models toward using high-performance 20B to 35B class models that allow room for context and fast processing speeds. The article explains how memory is allocated across model weights, KV cache, and runtime overhead while recommending specific top performers:
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
The article explores how modern local large language models are evolving beyond mere quantization into unique architectures that outperform larger cloud-based counterparts in specific tasks. Rather than being simple smaller versions of existing systems, these new releases employ specialized training and attention mechanisms to handle context management, reasoning, and multimodality on consumer hardware efficiently.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
An experiment exploring whether a local large language model can manage a home server without strict guardrails. By using the Pi agent harness and the Qwen3.6-35B-A3B model, the author successfully enabled an LLM to control a Proxmox VE node. The setup demonstrated impressive capabilities in managing LXC containers, checking system metrics, creating snapshots, and provisioning new virtual machines. Despite these successes, the system encountered difficulties with guest terminal command execution, persistent configuration management, and multi-node scaling.
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
Simon Willison reviews Ornith-1.0, a new series of open-weights models from DeepReinforce designed for agentic coding. Built on Gemma 4 and Qwen 3.5, these models range from 9B to 397B parameters and demonstrate high proficiency in coding benchmarks and tool-use tasks.
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
This article explores how to integrate local Large Language Models (LLMs) with Docker environments using the Model Context Protocol (MCP). By setting up an MCP server, users can enable LLMs to execute container management tasks such as monitoring health, listing volumes, and deploying new services through natural language prompts. The author demonstrates how a high-end MoE model can handle complex instructions, even troubleshooting configuration errors autonomously.
Main points:
- Model Context Protocol (MCP) functions as a bridge between LLMs and external tools.
- Implementation details for the mcp-server-docker package.
- Hardware and model specifications (Qwen3.6-35B-A3B on RTX 3080 Ti).
- Examples of automated deployments for n8n and BentoPDF.
- Security measures for restricting dangerous LLM actions.
Main points:
- Model Context Protocol (MCP) functions as a bridge between LLMs and external tools.
- Implementation details for the mcp-server-docker package.
- Hardware and model specifications (Qwen3.6-35B-A3B on RTX 3080 Ti).
- Examples of automated deployments for n8n and BentoPDF.
- Security measures for restricting dangerous LLM actions.
This repository provides optimized Jinja chat templates designed to fix critical rendering errors, KV cache invalidation, and agentic stalling issues found in official Qwen 3.5 and 3.6 templates. It is compatible with major inference engines including LM Studio, llama.cpp, vLLM, and MLX.
