El Assadi et al. compare ten LLMs (six families) and 26 embedding models (118M - 14B parameters) on 37 tasks, considering cost. In aggregate, the two paradigms are effectively tied (best LLM scores 77.6 versus best embedding model 77.2), yet their strengths diverge by task: LLMs lead on reasoning-heavy retrieval while embedding models lead on classification, and the two match on clustering, STS, and pair classification.
LLMs are significantly more expensive (up to 1,431x) and slower (2.5-736x) than embedding models for certain tasks. The authors suggest using embedding models for similarity, classification, and clustering, and LLMs for reasoning in retrieval.
Reasoning tokens are 28-81% of LLM inference cost; lower budgets maintain or boost retrieval quality for most tested models.
- Only Gemini 3.1 Pro breaks into the Pareto frontier alongside the leading embedding models.
- Accepted to COLM 2026; code, datasets, and results are publicly released on GitHub.
LLMs are significantly more expensive (up to 1,431x) and slower (2.5-736x) than embedding models for certain tasks. The authors suggest using embedding models for similarity, classification, and clustering, and LLMs for reasoning in retrieval.
Reasoning tokens are 28-81% of LLM inference cost; lower budgets maintain or boost retrieval quality for most tested models.
- Only Gemini 3.1 Pro breaks into the Pareto frontier alongside the leading embedding models.
- Accepted to COLM 2026; code, datasets, and results are publicly released on GitHub.
Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
>"I Measured Every Watt on Apple Silicon Five models, sustained generation, real wall-socket energy at $0.31/kWh — and the surprise the RTX-3090 numbers predicted, only bigger."
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
The article discusses the growing trend of running Large Language Models (LLMs) locally on personal machines, exploring the motivations behind this shift – including privacy concerns, cost savings, and a desire for technological sovereignty – as well as the hardware and software advancements making it increasingly feasible.
A technical article explaining how a small change in async Python code—using a semaphore to limit concurrency—reduced LLM request volume and costs by 90% without sacrificing performance.
This article details the billing structure for GitHub Spark, covering costs associated with app creation (based on premium requests) and current limits for deployed apps. It also outlines future billing plans for deployed apps once limits are reached.
Learn how to summarize large documents using LangChain and OpenAI, addressing contextual limits and cost effectively. This tutorial covers text preprocessing, semantic chunking, K-means clustering, and document summarization.
