The author examines the trade-offs between running large language models via Ollama versus llama.cpp's WebUI. While Ollama is celebrated for its user-friendly interface and effortless model management, it introduces a small performance overhead by acting as an abstraction layer over the inference engine. In contrast, llama.cpp provides faster generation speeds and highly granular control over hardware offloading and sampling parameters. Although llama.cpp requires more manual effort to launch and switch models, it is preferred for users seeking maximum performance and technical customization.
* Performance comparison of local LLM inference
* Comparison of user experience versus technical control
* Evaluation of abstraction overhead in AI tools
* Performance comparison of local LLM inference
* Comparison of user experience versus technical control
* Evaluation of abstraction overhead in AI tools
An experiment exploring whether a local large language model can manage a home server without strict guardrails. By using the Pi agent harness and the Qwen3.6-35B-A3B model, the author successfully enabled an LLM to control a Proxmox VE node. The setup demonstrated impressive capabilities in managing LXC containers, checking system metrics, creating snapshots, and provisioning new virtual machines. Despite these successes, the system encountered difficulties with guest terminal command execution, persistent configuration management, and multi-node scaling.
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
Simon Willison reviews Ornith-1.0, a new series of open-weights models from DeepReinforce designed for agentic coding. Built on Gemma 4 and Qwen 3.5, these models range from 9B to 397B parameters and demonstrate high proficiency in coding benchmarks and tool-use tasks.
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
From installing Ollama to launching OpenCode with a local model, step by step.
This guide explains how to build a fully local AI coding agent by integrating Ollama, Google's Gemma 4, and OpenCode to ensure data privacy, cost control, and local experimentation. The workflow involves installing Ollama to serve the edge-optimized `gemma4:e4b` model, setting up OpenCode as the agent runtime via npm, and linking them through a local API configuration with a 128K context window. This setup provides a capable, offline development assistant that handles code generation, repository analysis, and file operations without transmitting any data to the cloud.
* **Core Stack:** Uses Ollama for model serving, Gemma 4 (`gemma4:e4b`) as the local LLM, and OpenCode as an open-source agent interface.
* **Setup Steps:** Install Ollama, pull the Gemma 4 model, install OpenCode via `npm install -g opencode-ai`, and connect them by creating a Modelfile for the context window and configuring `opencode.json`.
* **Capabilities:** The local agent supports 128K context, enables coding tasks and workspace automation, and operates entirely offline for maximum security.
This guide explains how to build a fully local AI coding agent by integrating Ollama, Google's Gemma 4, and OpenCode to ensure data privacy, cost control, and local experimentation. The workflow involves installing Ollama to serve the edge-optimized `gemma4:e4b` model, setting up OpenCode as the agent runtime via npm, and linking them through a local API configuration with a 128K context window. This setup provides a capable, offline development assistant that handles code generation, repository analysis, and file operations without transmitting any data to the cloud.
* **Core Stack:** Uses Ollama for model serving, Gemma 4 (`gemma4:e4b`) as the local LLM, and OpenCode as an open-source agent interface.
* **Setup Steps:** Install Ollama, pull the Gemma 4 model, install OpenCode via `npm install -g opencode-ai`, and connect them by creating a Modelfile for the context window and configuring `opencode.json`.
* **Capabilities:** The local agent supports 128K context, enables coding tasks and workspace automation, and operates entirely offline for maximum security.
The author examines the practical benefits of self-hosting large language models locally on powerful hardware like an NVIDIA RTX 5090 rather than subscribing to cloud services such as Claude or Codex. Through testing, it was found that running Qwen 3.6 27B provides a capable coding assistant for tasks like bug detection and data parsing while maintaining complete data privacy and avoiding recurring subscription fees or changing API pricing models.
* Privacy advantages of keeping all processing on-device
* Cost efficiency through one-time hardware investment versus monthly cloud subscriptions
* Performance parity between local Qwen 3.6 and frontier models like Claude Opus for coding tasks
* Privacy advantages of keeping all processing on-device
* Cost efficiency through one-time hardware investment versus monthly cloud subscriptions
* Performance parity between local Qwen 3.6 and frontier models like Claude Opus for coding tasks
This guide provides instructions for running Alibaba's Qwen3.6 multimodal hybrid-thinking models locally using Unsloth tools. It covers the 27B and 35B-A3B variants, which support a 256K context window across 201 languages and excel in agentic coding, vision, and chat tasks. The article details hardware requirements for various quantization levels and explains how to leverage Multi Token Prediction (MTP) for significantly faster inference.
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
> Lessons from building a fast, reliable scientific agent with local open-weight models, vLLM, and long-context infrastructure
Local large language models often struggle with ambiguous prompts because they lack the massive datasets and scale used by cloud-based AI to infer user intent. To improve accuracy, users can implement a custom system prompt that instructs the model to ask up to three targeted clarifying questions before performing complex tasks like coding or writing. This approach reduces errors caused by incorrect assumptions and helps refine user instructions through active dialogue.
>"""When tasked with coding, writing, editing, or summarizing, ask the user up to three targeted clarifying questions. Proceed with the task once you've received answers and understand the prompt fully. If the task is a simple factual question or conversational message, respond directly.
"""
>"""When tasked with coding, writing, editing, or summarizing, ask the user up to three targeted clarifying questions. Proceed with the task once you've received answers and understand the prompt fully. If the task is a simple factual question or conversational message, respond directly.
"""
Running large language models locally often runs into hardware limitations that prevent complex problem-solving. This article explains a hybrid approach where a local model acts as a junior engineer for routine tasks but escalates difficult issues to cloud-based models like Claude when it gets stuck. This orchestration system allows for a privacy-focused, local-first workflow without sacrificing the high-level reasoning power of massive commercial AI.
- Ollama for local inference and model management
- LiteLLM as a routing layer to provide a unified API for both local and cloud models
- OpenRouter or Anthropic's API for flexible cloud escalation
- A simple orchestration system to manage retries and task handovers
- Ollama for local inference and model management
- LiteLLM as a routing layer to provide a unified API for both local and cloud models
- OpenRouter or Anthropic's API for flexible cloud escalation
- A simple orchestration system to manage retries and task handovers
The author explores creating a privacy-focused AI concierge for a Reolink video doorbell using locally hosted tools. By integrating Home Assistant with Piper for text-to-speech, Whisper for speech-to-text, and Ollama to run local large language models, the project aimed to automate interactions with visitors when no one is home. Although real-time two-way conversations were hindered by hardware performance and model latency, a functional system was developed that transcribes visitor messages and sends them as notifications to the owner's phone.
Main points:
Implementing local AI in smart home devices for privacy
Using Home Assistant to orchestrate TTS, STT, and LLM components
Overcoming hardware bottlenecks in real-time speech processing
Automating visitor message transcription and mobile notifications
Main points:
Implementing local AI in smart home devices for privacy
Using Home Assistant to orchestrate TTS, STT, and LLM components
Overcoming hardware bottlenecks in real-time speech processing
Automating visitor message transcription and mobile notifications
