A Reddit thread discussing preferred local Large Language Model (LLM) setups for tasks like summarizing text, coding, and general use. Users share their model choices (Gemma, Qwen, Phi, etc.) and frameworks (llama.cpp, Ollama, EXUI) along with potential issues and configurations.
| Model | Use Cases | Size (Parameters) | Approx. VRAM (Q4 Quantization) | Approx. RAM (Q4) | Notes/Requirements |
|---|---|---|---|---|---|
| Gemma 3 (Meta) | Summarization, conversational tasks, image recognition, translation, simple writing | 3B, 4B, 7B, 8B, 12B, 27B+ | 2-4GB (3B), 4-6GB (7B), 8-12GB (12B) | 4-8GB (3B), 8-12GB (7B), 16-24GB (12B) | Excellent performance for its size. Recent versions have had memory leak issues (see Reddit post – use Ollama 0.6.6 or later, but even that may not be fully fixed). QAT versions are highly recommended. |
| Qwen 2.5 (Alibaba) | Summarization, coding, reasoning, decision-making, technical material processing | 3.5B, 7B, 72B | 2-3GB (3.5B), 4-6GB (7B), 26-30GB (72B) | 4-6GB (3.5B), 8-12GB (7B), 50-60GB (72B) | Qwen models are known for strong performance. Coder versions specifically tuned for code generation. |
| Qwen3 (Alibaba - upcoming) | General purpose, likely similar to Qwen 2.5 with improvements | 70B | Estimated 25-30GB (Q4) | 50-60GB | Expected to be a strong competitor. |
| Llama 3 (Meta) | General purpose, conversation, writing, coding, reasoning | 8B, 13B, 70B+ | 4-6GB (8B), 7-9GB (13B), 25-30GB (70B) | 8-12GB (8B), 14-18GB (13B), 50-60GB (70B) | Current state-of-the-art open-source model. Excellent balance of performance and size. |
| YiXin (01.AI) | Reasoning, brainstorming | 72B | ~26-30GB (Q4) | ~50-60GB | A powerful model focused on reasoning and understanding. Similar VRAM requirements to Qwen 72B. |
| Phi-4 (Microsoft) | General purpose, writing, coding | 14B | ~7-9GB (Q4) | 14-18GB | Smaller model, good for resource-constrained environments, but may not match larger models in complexity. |
| Ling-Lite | RAG (Retrieval-Augmented Generation), fast processing, text extraction | Variable | Varies with size | Varies with size | MoE (Mixture of Experts) model known for speed. Good for RAG applications where quick responses are important. |
Key Considerations:
- Quantization: The VRAM and RAM estimates above are based on 4-bit quantization (Q4). Lower quantization (e.g., Q2) will reduce memory usage further, but may impact quality. Higher quantization (e.g., Q8, FP16) will increase quality but require significantly more memory.
- Frameworks: Popular frameworks for running these models locally include:
- llama.cpp: Highly optimized for CPU and GPU, especially on Apple Silicon.
- Ollama: Simplified setup and management of LLMs. (Be aware of the Gemma 3 memory leak issue!)
- Text Generation WebUI (oobabooga): Web-based interface with many features and customization options.
- Hardware: A dedicated GPU with sufficient VRAM is highly recommended for decent performance. CPU-only inference is possible but can be slow. More RAM is generally better, even if the model fits in VRAM.
- Context Length: The "40k" context mentioned in the Reddit post refers to the maximum number of tokens (words or sub-words) the model can process at once. Longer context lengths require more memory.
This document details how to run Qwen models locally using the Text Generation Web UI (oobabooga), covering installation, setup, and launching the web interface.
"OpenHands LM is built on the foundation of Qwen Coder 2.5 Instruct 32B, leveraging its powerful base capabilities for coding tasks."
A review of the Qwen2.5-VL-32B large language model, noting its performance, capabilities, and how it runs on a 64GB Mac. Includes a demonstration with a map image and performance statistics.
Qwen2.5-VL is a flagship model of the Qwen vision-language series, showcasing advancements in visual recognition, object localization, document parsing, and long-video comprehension. It introduces dynamic resolution processing and absolute time encoding, allowing it to handle complex inputs and maintain native resolution. Available in three sizes, it suits various applications from edge AI to high-performance computing, matching state-of-the-art models in document and diagram understanding while preserving strong linguistic capabilities.
Qwen2.5-VL-3B-Instruct is the latest addition to the Qwen family of vision-language models by Hugging Face, featuring enhanced capabilities in understanding visual content and generating structured outputs. It is designed to directly interact with tools and use computer and phone functions as a visual agent. Qwen2.5-VL can comprehend videos up to an hour long and localize objects within images using bounding boxes or points. It is available in three sizes: 3, 7, and 72 billion parameters.
Qwen2.5-1M models and inference framework support for long-context tasks, with a context length of up to 1M tokens.
A tutorial on using Qwen2.5–7B-Instruct for creating a local, open-source, multi-agentic RAG system.
The implementation described in the article focuses on creating a multi-agentic Retrieval-Augmented Generation (RAG) system using code agents and the Qwen2.5–7B-Instruct model. The system consists of three agents working together in a hierarchical structure:
Manager Agent: This top-level agent breaks down user questions into sub-tasks, utilizes the Wikipedia search agent to find information, and combines the results to provide a final answer. Its system prompt is tailored to guide it through the process of decomposing tasks and coordinating with other agents.
Wikipedia Search Agent: This agent interacts with the Wikipedia search tool to identify relevant pages and their summaries. It further delegates to the page search agent for detailed information retrieval from specific pages if needed. Its prompt is designed to help it navigate Wikipedia effectively and extract necessary information.
Page Search Agent: This agent specializes in extracting precise information from a given Wikipedia page. It uses a semantic search tool to locate specific passages related to the query.
To implement the multi-agent system efficiently, the article mentions several key decisions and modifications to the default Hugging Face implementation:
- Prompting: Customized prompts for each agent, including specific examples that mirror the model’s chat template, to improve task-specific performance.
- History Summarization: Limiting the history passed to each step to avoid excessive context length and improve execution speed.
- Tool Wrapping: Wrapping managed agents as tools to allow better control over the prompts and streamline the architecture.
- Error Handling: Implementing mechanisms to handle tool execution errors effectively.
- Execution Limiting: Setting a maximum number of attempts for the page search agent to prevent infinite loops when searching for information that might not be present on the page.
- Tool Response Modification: Adapting the tool response format to fit the Qwen2.5–7B-Instruct model’s chat template, which supports only system, user, and assistant roles.
By structuring the implementation with these considerations, the system achieves the capability to perform complex, multi-hop question-answering tasks efficiently, despite being powered by a relatively small model running on consumer-grade hardware
This article explores QwQ-32B-Preview, an experimental AI model by Qwen Team, which focuses on advancing AI reasoning capabilities. It discusses the model's performance, limitations, and its deep contemplative abilities on various benchmarks and problems.
Simon Willison reviews the new Qwen2.5-Coder-32B, an open-source LLM by Alibaba, which performs well on various coding benchmarks and can run on personal devices like his MacBook Pro M2.
