noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Michal Sutter writes that the Qwen Developer team has released zg (zvec-grep), an open-source local-first search layer designed to streamline how coding agents find information within a workspace. By unifying semantic search, BM25, and ripgrep under a single interface, it reduces tool calls and token usage for LLM agents that would otherwise struggle with manual context assembly or imprecise keyword matching.
- The package is available via npm as `@zvec/zvec-grep` under an Apache 2.0 license.
- It supports four retrieval routes: a hybrid default, BM25 (`--fts`), vector similarity (`--vector`), and literal/regex matching (`--rg`).
- An MCP (Model Context Protocol) integration allows seamless use with tools like Claude Code, Cursor, and Codex.
- Embeddings run locally by default using models such as `potion-code-16m-v2`, though remote Qwen endpoints are also supported via explicit authorization.
- Benchmarks suggest zg can cut tool calls and input tokens for coding agents by approximately 40% to 50%.
- The package is available via npm as `@zvec/zvec-grep` under an Apache 2.0 license.
- It supports four retrieval routes: a hybrid default, BM25 (`--fts`), vector similarity (`--vector`), and literal/regex matching (`--rg`).
- An MCP (Model Context Protocol) integration allows seamless use with tools like Claude Code, Cursor, and Codex.
- Embeddings run locally by default using models such as `potion-code-16m-v2`, though remote Qwen endpoints are also supported via explicit authorization.
- Benchmarks suggest zg can cut tool calls and input tokens for coding agents by approximately 40% to 50%.
This guide outlines the most effective approach for running large language models locally on hardware with 24GB of VRAM. It advises moving away from squeezing extremely large parameter models toward using high-performance 20B to 35B class models that allow room for context and fast processing speeds. The article explains how memory is allocated across model weights, KV cache, and runtime overhead while recommending specific top performers:
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
The article explores how modern local large language models are evolving beyond mere quantization into unique architectures that outperform larger cloud-based counterparts in specific tasks. Rather than being simple smaller versions of existing systems, these new releases employ specialized training and attention mechanisms to handle context management, reasoning, and multimodality on consumer hardware efficiently.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
An experiment exploring whether a local large language model can manage a home server without strict guardrails. By using the Pi agent harness and the Qwen3.6-35B-A3B model, the author successfully enabled an LLM to control a Proxmox VE node. The setup demonstrated impressive capabilities in managing LXC containers, checking system metrics, creating snapshots, and provisioning new virtual machines. Despite these successes, the system encountered difficulties with guest terminal command execution, persistent configuration management, and multi-node scaling.
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
Simon Willison reviews Ornith-1.0, a new series of open-weights models from DeepReinforce designed for agentic coding. Built on Gemma 4 and Qwen 3.5, these models range from 9B to 397B parameters and demonstrate high proficiency in coding benchmarks and tool-use tasks.
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
This article explores how to integrate local Large Language Models (LLMs) with Docker environments using the Model Context Protocol (MCP). By setting up an MCP server, users can enable LLMs to execute container management tasks such as monitoring health, listing volumes, and deploying new services through natural language prompts. The author demonstrates how a high-end MoE model can handle complex instructions, even troubleshooting configuration errors autonomously.
Main points:
- Model Context Protocol (MCP) functions as a bridge between LLMs and external tools.
- Implementation details for the mcp-server-docker package.
- Hardware and model specifications (Qwen3.6-35B-A3B on RTX 3080 Ti).
- Examples of automated deployments for n8n and BentoPDF.
- Security measures for restricting dangerous LLM actions.
Main points:
- Model Context Protocol (MCP) functions as a bridge between LLMs and external tools.
- Implementation details for the mcp-server-docker package.
- Hardware and model specifications (Qwen3.6-35B-A3B on RTX 3080 Ti).
- Examples of automated deployments for n8n and BentoPDF.
- Security measures for restricting dangerous LLM actions.
This repository provides optimized Jinja chat templates designed to fix critical rendering errors, KV cache invalidation, and agentic stalling issues found in official Qwen 3.5 and 3.6 templates. It is compatible with major inference engines including LM Studio, llama.cpp, vLLM, and MLX.
An exploration of high-performing small language models with under 7 billion parameters that can run locally on consumer hardware like laptops and smartphones. The article explains how advancements in training data quality, model distillation from larger frontier models, and architectural improvements like Mixture-of-Experts have enabled these compact models to compete with much larger versions on reasoning benchmarks. It provides a curated guide of top available models on Hugging Face, detailing their specific strengths, benchmark performance, and providing Python code for implementation.
Key models covered:
- Qwen3.5-4B for multilingual tasks and long context windows
- Microsoft Phi-4-mini-instruct for reasoning-heavy English workloads
- Google Gemma 3 4B IT for coding and mathematics
- Google Gemma 3n E4B for efficient mobile and on-device deployment
- Meta Llama 3.2 3B Instruct for tool calling and community support
- SmolLM3-3B for research transparency and open-source projects
- DeepSeek-R1-Distill-Qwen-1.5B for lightweight reasoning on edge devices
- Qwen3-0.6B for ultra-constrained hardware and text classification
Key models covered:
- Qwen3.5-4B for multilingual tasks and long context windows
- Microsoft Phi-4-mini-instruct for reasoning-heavy English workloads
- Google Gemma 3 4B IT for coding and mathematics
- Google Gemma 3n E4B for efficient mobile and on-device deployment
- Meta Llama 3.2 3B Instruct for tool calling and community support
- SmolLM3-3B for research transparency and open-source projects
- DeepSeek-R1-Distill-Qwen-1.5B for lightweight reasoning on edge devices
- Qwen3-0.6B for ultra-constrained hardware and text classification
Small, inexpensive single-board computers like the Raspberry Pi 5 are becoming viable platforms for running local large language models (LLMs). By utilizing quantization techniques to reduce model size and memory requirements, users can run quantized versions of popular models such as Llama 3, Mistral, and Qwen. While processing speeds remain limited compared to high-end GPUs, these devices offer a private and low-cost way to implement AI for specific tasks.
- Quantization allows large models to fit into the Pi's limited RAM by reducing numerical precision.
- Tiny models (1B-3B parameters) run comfortably, while 7B parameter models are usable on 8GB versions with managed expectations.
- Performance is measured in low single-digit tokens per second, making it suitable for non-real-time tasks.
- Hardware upgrades like the Raspberry Pi AI HAT+ or external eGPUs can significantly boost neural processing capabilities.
- Quantization allows large models to fit into the Pi's limited RAM by reducing numerical precision.
- Tiny models (1B-3B parameters) run comfortably, while 7B parameter models are usable on 8GB versions with managed expectations.
- Performance is measured in low single-digit tokens per second, making it suitable for non-real-time tasks.
- Hardware upgrades like the Raspberry Pi AI HAT+ or external eGPUs can significantly boost neural processing capabilities.
