noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Arsen Apostolov writes about the actual electrical cost of running local Large Language Models on a single NVIDIA RTX 3090 compared to hosted cloud APIs.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
The RTX 3090 offers a compelling combination of performance and 24GB of VRAM, making it a better choice for local LLM and AI workloads than newer Nvidia Blackwell GPUs like the RTX 5070 and even the RTX 5080, due to VRAM limitations and pricing.
A user shares their optimal settings for running the gpt-oss-120b model on a system with dual RTX 3090 GPUs and 128GB of RAM, aiming for a balance between performance and quality.
A user shares their experience running the GPT-OSS 120b model on Ollama with an i7 6700, 64GB DDR4 RAM, RTX 3090, and a 1TB SSD. They note slow initial token generation but acceptable performance overall, highlighting it's possible on a relatively modest setup. The discussion includes comparisons to other hardware configurations, optimization techniques (llama.cpp), and the model's quality.
>I have a 3090 with 64gb ddr4 3200 RAM and am getting around 50 t/s prompt processing speed and 15 t/s generation speed using the following:
>
>`llama-server -m <path to gpt-oss-120b> --ctx-size 32768 --temp 1.0 --top-p 1.0 --jinja -ub 2048 -b 2048 -ngl 99 -fa 'on' --n-cpu-moe 24`
> This about fills up my VRAM and RAM almost entirely. For more wiggle room for other applications use `--n-cpu-moe 26`.
>I have a 3090 with 64gb ddr4 3200 RAM and am getting around 50 t/s prompt processing speed and 15 t/s generation speed using the following:
>
>`llama-server -m <path to gpt-oss-120b> --ctx-size 32768 --temp 1.0 --top-p 1.0 --jinja -ub 2048 -b 2048 -ngl 99 -fa 'on' --n-cpu-moe 24`
> This about fills up my VRAM and RAM almost entirely. For more wiggle room for other applications use `--n-cpu-moe 26`.
A startup called Backprop has demonstrated that a single Nvidia RTX 3090 GPU, released in 2020, can handle serving a modest large language model (LLM) like Llama 3.1 8B to over 100 concurrent users with acceptable throughput. This suggests that expensive enterprise GPUs may not be necessary for scaling LLMs to a few thousand users.
