Arsen Apostolov writes about the actual electrical cost of running local Large Language Models on a single NVIDIA RTX 3090 compared to hosted cloud APIs.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
This guide outlines the most effective approach for running large language models locally on hardware with 24GB of VRAM. It advises moving away from squeezing extremely large parameter models toward using high-performance 20B to 35B class models that allow room for context and fast processing speeds. The article explains how memory is allocated across model weights, KV cache, and runtime overhead while recommending specific top performers:
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
An open-source command-line tool designed to identify the optimal local Large Language Model specifically suited for a user's existing or planned hardware. It automatically detects GPU, CPU, and RAM capacity to rank HuggingFace models using real performance benchmarks instead of relying on parameter size alone.
* Hardware auto-detection for NVIDIA, AMD, Apple Silicon, and CPUs
* Intelligent ranking based on benchmark evidence and recency awareness
* Capability to simulate different GPUs for hardware upgrade planning
* Support for GGUF, AWQ, and GPTQ model formats
* Streamlined workflows including one-command chat sessions and Python code snippet generation
* Hardware auto-detection for NVIDIA, AMD, Apple Silicon, and CPUs
* Intelligent ranking based on benchmark evidence and recency awareness
* Capability to simulate different GPUs for hardware upgrade planning
* Support for GGUF, AWQ, and GPTQ model formats
* Streamlined workflows including one-command chat sessions and Python code snippet generation
The author compares the performance of an NVIDIA RTX 5090 against Apple Silicon when running large-scale local Large Language Models. While the 5090 offers superior speed for smaller models that fit within its 32GB VRAM, it struggles with massive models that require significantly more memory. In contrast, Apple's Unified Memory Architecture allows Mac Studio users to access much larger pools of memory, making it a more viable platform for running extremely large LLMs like DeepSeek R1.
AMD CEO Dr. Lisa Su addressed concerns that the rise of agentic AI might cannibalize the GPU market, arguing instead that the demand is largely additive. While GPUs are essential for running foundational models, CPUs play a critical role in orchestration, data movement, and parallel execution required by autonomous agents. This shift could fundamentally change industry-standard CPU-to-GPU ratios, potentially moving from traditional 1:8 configurations toward a more balanced 1:1 ratio as agentic workloads expand.
Personal website of Alex L. Zhang, a PhD student at MIT CSAIL focusing on the efficiency and utilization of language models. His research spans ML systems, language model benchmarks, and specialized model development.
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
AMD now supports Google’s Gemma 4 models (2B–31B parameters) across its entire hardware lineup, including Instinct GPUs (datacenters), Radeon GPUs (workstations), and Ryzen AI processors (PCs). The integration is compatible with vLLM, SGLang, llama.cpp, Ollama, and Lemonade Server, aiming to optimize AI performance for both cloud and local deployment.
This article details the journey of deploying an on-premise Large Language Model (LLM) server, focusing on security considerations. It explores the rationale behind on-premise deployment for privacy and data control, outlining the goals of creating an air-gapped, isolated infrastructure. The authors delve into the hardware selection process, choosing components like an Nvidia RTX Pro 6000 Max-Q for its memory capacity. The deployment process starts with a minimal setup using llama.cpp, then progresses to containerization with Podman and the use of CDI for GPU access. Finally, the article discusses hardening techniques, including kernel module management and file permission restrictions, to minimize the attack surface and enhance security.
>The method, called KV Cache Transform Coding (KVTC), applies ideas from media compression formats like JPEG to shrink the key-value cache behind multi-turn AI systems, lowering GPU memory demands and speeding up time-to-first-token by up to 8x.
Google AI introduces STATIC, a sparse matrix framework that accelerates constrained decoding for LLM-based generative retrieval. It addresses the inefficiency of traditional trie implementations on hardware accelerators by flattening the trie into a static Compressed Sparse Row (CSR) matrix, achieving up to 948x speedup and demonstrating improvements in YouTube video recommendations.
