This guide provides instructions for running Alibaba's Qwen3.6 multimodal hybrid-thinking models locally using Unsloth tools. It covers the 27B and 35B-A3B variants, which support a 256K context window across 201 languages and excel in agentic coding, vision, and chat tasks. The article details hardware requirements for various quantization levels and explains how to leverage Multi Token Prediction (MTP) for significantly faster inference.
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
Key topics:
- Hardware memory requirements for quantized models
- Faster generation via Multi Token Prediction (MTP)
- Integration with Unsloth Studio, llama.cpp, and MLX
- Preserved thinking mode configurations
> Lessons from building a fast, reliable scientific agent with local open-weight models, vLLM, and long-context infrastructure
Local large language models often struggle with ambiguous prompts because they lack the massive datasets and scale used by cloud-based AI to infer user intent. To improve accuracy, users can implement a custom system prompt that instructs the model to ask up to three targeted clarifying questions before performing complex tasks like coding or writing. This approach reduces errors caused by incorrect assumptions and helps refine user instructions through active dialogue.
>"""When tasked with coding, writing, editing, or summarizing, ask the user up to three targeted clarifying questions. Proceed with the task once you've received answers and understand the prompt fully. If the task is a simple factual question or conversational message, respond directly.
"""
>"""When tasked with coding, writing, editing, or summarizing, ask the user up to three targeted clarifying questions. Proceed with the task once you've received answers and understand the prompt fully. If the task is a simple factual question or conversational message, respond directly.
"""
Running large language models locally often runs into hardware limitations that prevent complex problem-solving. This article explains a hybrid approach where a local model acts as a junior engineer for routine tasks but escalates difficult issues to cloud-based models like Claude when it gets stuck. This orchestration system allows for a privacy-focused, local-first workflow without sacrificing the high-level reasoning power of massive commercial AI.
- Ollama for local inference and model management
- LiteLLM as a routing layer to provide a unified API for both local and cloud models
- OpenRouter or Anthropic's API for flexible cloud escalation
- A simple orchestration system to manage retries and task handovers
- Ollama for local inference and model management
- LiteLLM as a routing layer to provide a unified API for both local and cloud models
- OpenRouter or Anthropic's API for flexible cloud escalation
- A simple orchestration system to manage retries and task handovers
The author explores creating a privacy-focused AI concierge for a Reolink video doorbell using locally hosted tools. By integrating Home Assistant with Piper for text-to-speech, Whisper for speech-to-text, and Ollama to run local large language models, the project aimed to automate interactions with visitors when no one is home. Although real-time two-way conversations were hindered by hardware performance and model latency, a functional system was developed that transcribes visitor messages and sends them as notifications to the owner's phone.
Main points:
Implementing local AI in smart home devices for privacy
Using Home Assistant to orchestrate TTS, STT, and LLM components
Overcoming hardware bottlenecks in real-time speech processing
Automating visitor message transcription and mobile notifications
Main points:
Implementing local AI in smart home devices for privacy
Using Home Assistant to orchestrate TTS, STT, and LLM components
Overcoming hardware bottlenecks in real-time speech processing
Automating visitor message transcription and mobile notifications
TextGen is an open-source desktop application designed for running large language models locally with complete privacy and zero telemetry. It provides a user interface and API that supports text, vision, tool-calling, and web search functionality. The software allows users to switch between multiple backends such as llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM without restarting the application.
Main topics:
Multimodal support for visual understanding via image attachments
OpenAI/Anthropic compatible API with tool-calling capabilities
Fine-tuning functionality for LoRAs on chat or raw text datasets
Integrated image generation using diffusers models
Support for various installation methods including portable builds and Docker
Main topics:
Multimodal support for visual understanding via image attachments
OpenAI/Anthropic compatible API with tool-calling capabilities
Fine-tuning functionality for LoRAs on chat or raw text datasets
Integrated image generation using diffusers models
Support for various installation methods including portable builds and Docker
Unlike cloud AI services like Claude or Gemini, local LLMs lack built-in workspace features for persistent memory. You can bridge this gap using "context journaling" via system prompts and RAG.
* LM Studio presets for concise system prompts.
* RAG document uploads for background/project history.
* Markdown journal structure (Background, Projects, Corrections).
* “Corrections” section to prevent recurring model errors.
* Session exports for prompt effectiveness records.
* LM Studio presets for concise system prompts.
* RAG document uploads for background/project history.
* Markdown journal structure (Background, Projects, Corrections).
* “Corrections” section to prevent recurring model errors.
* Session exports for prompt effectiveness records.
The author compares the performance of an NVIDIA RTX 5090 against Apple Silicon when running large-scale local Large Language Models. While the 5090 offers superior speed for smaller models that fit within its 32GB VRAM, it struggles with massive models that require significantly more memory. In contrast, Apple's Unified Memory Architecture allows Mac Studio users to access much larger pools of memory, making it a more viable platform for running extremely large LLMs like DeepSeek R1.
This article explores the feasibility of running Large Language Models (LLMs) locally using only a CPU, challenging the assumption that expensive GPUs are strictly necessary. By testing eight different models on an older Intel i5 laptop with 12GB of RAM via Ollama, the author identifies which models offer practical usability for everyday tasks.
Key points include:
- Using tokens per second as a more critical metric for usability than model size or RAM usage alone.
- Why 1B to 2B parameter models provide the best balance of responsiveness and reasoning on low-end hardware.
- The effectiveness of GGUF quantization (specifically Q4_K_M) in reducing resource demands.
- A comparison of various model tiers, from ultra-fast tiny models like Qwen 0.6B to slower, high-capability models like Ministral 3 8B.
Key points include:
- Using tokens per second as a more critical metric for usability than model size or RAM usage alone.
- Why 1B to 2B parameter models provide the best balance of responsiveness and reasoning on low-end hardware.
- The effectiveness of GGUF quantization (specifically Q4_K_M) in reducing resource demands.
- A comparison of various model tiers, from ultra-fast tiny models like Qwen 0.6B to slower, high-capability models like Ministral 3 8B.
The author explores the utility of Google DeepMind's Gemma 4 as a powerful option for running large language models locally on consumer hardware. By testing the E4B variant using tools like LM Studio and llama.cpp, they demonstrate how open-weight models can handle multimodal tasks including text, image analysis, and audio processing with impressive precision and privacy.
