Yuhao Wu writes about HarnessDev, a benchmark that evaluates LLMs' ability to build and iteratively improve their own agent harness—the model-external execution infrastructure that wraps a model and shapes its task performance. The benchmark has two stages: Creation, where the agent builds a complete execution system from a minimal seed and a few cases, and Evolution, where it revises its own harness using downstream execution feedback. Generated harnesses substantially lag behind mature human-engineered references on code and search/research, while matching or exceeding them on writing and machine-learning experimentation, with large variation in execution cost.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
- Covers six creator LLMs across four domains and five downstream benchmarks (2,207 unique instances).
- Hidden evaluation tasks are withheld from development to prevent overfitting.
- Evolution gains are unstable and transfer only partially to held-out tasks.
- Performance gains depend strongly on which model executes the harness, indicating limited cross-model transfer.
Benjamin Marie writes about a comprehensive benchmark of 15 GGUF quantizations of Qwen3.8 27B, ranging from Q4_K_XL down to IQ1_M, evaluated using over 150 million tokens generated across roughly 8 days on an NVIDIA RTX Pro 6000. Using 950 prompts subsampled from MMLU-Pro, LiveCodeBench, and GPQA Diamond, he measures both accuracy and token efficiency to identify the lowest quantization level that retains at least 95% of the original BF16 model's performance.
| Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
|---|---|---|---|---|---|---|---|
| 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
| 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
| 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
| 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
| 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
| 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
| 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
| 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
| 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
| 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
| 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
| 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
| 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
| 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
| 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
`* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`
| Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
|---|---|---|---|---|---|---|---|
| 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
| 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
| 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
| 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
| 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
| 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
| 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
| 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
| 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
| 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
| 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
| 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
| 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
| 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
| 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
`* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`
El Assadi et al. compare ten LLMs (six families) and 26 embedding models (118M - 14B parameters) on 37 tasks, considering cost. In aggregate, the two paradigms are effectively tied (best LLM scores 77.6 versus best embedding model 77.2), yet their strengths diverge by task: LLMs lead on reasoning-heavy retrieval while embedding models lead on classification, and the two match on clustering, STS, and pair classification.
LLMs are significantly more expensive (up to 1,431x) and slower (2.5-736x) than embedding models for certain tasks. The authors suggest using embedding models for similarity, classification, and clustering, and LLMs for reasoning in retrieval.
Reasoning tokens are 28-81% of LLM inference cost; lower budgets maintain or boost retrieval quality for most tested models.
- Only Gemini 3.1 Pro breaks into the Pareto frontier alongside the leading embedding models.
- Accepted to COLM 2026; code, datasets, and results are publicly released on GitHub.
LLMs are significantly more expensive (up to 1,431x) and slower (2.5-736x) than embedding models for certain tasks. The authors suggest using embedding models for similarity, classification, and clustering, and LLMs for reasoning in retrieval.
Reasoning tokens are 28-81% of LLM inference cost; lower budgets maintain or boost retrieval quality for most tested models.
- Only Gemini 3.1 Pro breaks into the Pareto frontier alongside the leading embedding models.
- Accepted to COLM 2026; code, datasets, and results are publicly released on GitHub.
This article exposes critical flaws in Text-to-SQL benchmarks like BIRD and Spider. An audit of gold queries reveals that several contain incorrect joins, causing mathematically wrong results to be established as ground truth. Since standard execution accuracy measures performance by comparing outputs against these faulty reference answers, models are often penalized for being correct and rewarded for mimicking human errors. To address this, the author proposes a constraint-aware evaluation method that validates SQL logic against declared data semantics rather than relying on potentially incorrect gold results.
- Discrepancies between benchmark gold queries and database schema facts
- The inherent risks of using execution accuracy as the primary metric
- How annotation errors impact model rankings and enterprise deployments
- Introduction of constraint-aware evaluation to ensure semantic validity
- Discrepancies between benchmark gold queries and database schema facts
- The inherent risks of using execution accuracy as the primary metric
- How annotation errors impact model rankings and enterprise deployments
- Introduction of constraint-aware evaluation to ensure semantic validity
MCP-Universe is a comprehensive benchmark designed to evaluate LLMs in realistic tasks through interaction with real-world MCP servers across 6 core domains and 231 tasks. It highlights the challenges of long-context reasoning, unfamiliar tool spaces, and cross-domain variations in LLM performance.
LocalScore is an open benchmark to evaluate local AI task performance across various hardware configurations, measuring Prompt Processing speed, Token Generation speed, Time-to-First-Token (TTFT), and a combined LocalScore.
Hugging Face researchers developed an open-source AI research agent called 'Open Deep Research' in 24 hours, aiming to match OpenAI's Deep Research. The project demonstrates the potential of agent frameworks to enhance AI model capabilities, achieving 55.15% accuracy on the GAIA benchmark. The initiative highlights the rapid development and collaborative nature of open-source AI projects.
A benchmark of large language models, sorted by size (on disk) for each score. Highlighted entries are on the Pareto frontier.
Weaviate introduces StructuredRAG, a benchmark to evaluate LLMs' ability to generate reliable JSON outputs. The study finds that while LLMs perform well on simpler tasks, they struggle with more complex outputs.
This repository contains scripts for benchmarking the performance of large language models (LLMs) served using vLLM.
