- **Inference** – Platforms and engines for running models, plus user interfaces.
- **Models** – LLMs (general, coding, multimodal, image, audio), model providers, and specific model highlights.
- **RAG** – Retrieval-Augmented Generation tools.
- **Safeguards** – Safety and content filtering.
- **Agents & Tools** – Agent frameworks, Model Context Protocol, coding agents, computer/browser automation, memory management, and testing/evaluation.
- **Research, Training & Fine-tuning** – Security, sandboxing, and model development.
- **Hardware** – Local hardware options.
- **Tutorials** – Guides covering models, prompt/context engineering, inference, agents, and RAG.
- **Communities** – Places to connect and share knowledge.
- **Models** – LLMs (general, coding, multimodal, image, audio), model providers, and specific model highlights.
- **RAG** – Retrieval-Augmented Generation tools.
- **Safeguards** – Safety and content filtering.
- **Agents & Tools** – Agent frameworks, Model Context Protocol, coding agents, computer/browser automation, memory management, and testing/evaluation.
- **Research, Training & Fine-tuning** – Security, sandboxing, and model development.
- **Hardware** – Local hardware options.
- **Tutorials** – Guides covering models, prompt/context engineering, inference, agents, and RAG.
- **Communities** – Places to connect and share knowledge.
Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
This guide outlines the most effective approach for running large language models locally on hardware with 24GB of VRAM. It advises moving away from squeezing extremely large parameter models toward using high-performance 20B to 35B class models that allow room for context and fast processing speeds. The article explains how memory is allocated across model weights, KV cache, and runtime overhead while recommending specific top performers:
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
This article explores the feasibility of running Large Language Models (LLMs) locally using only a CPU, challenging the assumption that expensive GPUs are strictly necessary. By testing eight different models on an older Intel i5 laptop with 12GB of RAM via Ollama, the author identifies which models offer practical usability for everyday tasks.
Key points include:
- Using tokens per second as a more critical metric for usability than model size or RAM usage alone.
- Why 1B to 2B parameter models provide the best balance of responsiveness and reasoning on low-end hardware.
- The effectiveness of GGUF quantization (specifically Q4_K_M) in reducing resource demands.
- A comparison of various model tiers, from ultra-fast tiny models like Qwen 0.6B to slower, high-capability models like Ministral 3 8B.
Key points include:
- Using tokens per second as a more critical metric for usability than model size or RAM usage alone.
- Why 1B to 2B parameter models provide the best balance of responsiveness and reasoning on low-end hardware.
- The effectiveness of GGUF quantization (specifically Q4_K_M) in reducing resource demands.
- A comparison of various model tiers, from ultra-fast tiny models like Qwen 0.6B to slower, high-capability models like Ministral 3 8B.
A terminal tool that right-sizes LLM models to your system's RAM, CPU, and GPU. Detects your hardware, scores each model across quality, speed, fit, and context dimensions, and tells you which ones will actually run well on your machine.
