SemanticScuttle - klotz.me

klotz: llama.cpp*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

1. New in llama.cpp: Decision Models

   llama.cpp now supports decision models via a `/v1/systemone` endpoint, which accepts a state and typed questions to return probabilities for each provided option in a single forward pass. The API utilizes the System One format introduced with TypeSafe's Jev model, meaning existing clients only need a new base URL. Five initial models are available ranging from 144M to 27B parameters, supporting use cases like request routing, content moderation, and agent action selection with median response times as low as 3 ms.
   - The `state` field accepts text, JSON, screenshots, or a list of chat messages with `image_url` parts.
   - Three question types are supported: `choice` for categorical selection, `score` for a continuous level between 2 and 10 options, and `noul` for yes/no probabilities.
   - Router mode allows loading multiple models on a single server and selecting one per request.
   - Adding descriptions to option labels can significantly improve accuracy, with Julia-1's routing confidence for "charged twice" jumping from a misroute to 0.99 when descriptions were provided.
   - Cloudflare's Clef is the next model planned for integration.

   2026-10-02 Tags: llama.cpp, decision models, systemone, ggml-org, local inference, routing, classification, llm, hallux by klotz
2. Per-tensor layout maps for GGUF quantization

   Bartowski writes about a new method for optimizing GGUF quantizations by using per-tensor layout maps. By analyzing the sensitivity of different tensor types through extensive experimentation, the author developed a solver that allocates more bits to sensitive components (like embeddings and specific attention projections) while "crushing" less critical tensors. This approach aims to maximize performance per bit and restores semantic meaning to quantization naming conventions by ensuring tiered variants like _S, _M, and _L contain consistent proportions of high-bit tensors.
   - Token embeddings are highly sensitive and benefit significantly from higher precision.
   - Sensitivity follows a "U" shape relative to model depth (higher at the beginning and end).
   - Small attention projections such as `attn_v` and `ffn_up` are among the most sensitive tensor types per bit.
   - A "canary" test system is used to verify if new layout maps generalize well across different model architectures before deployment.

   2026-09-13 Tags: gguf quantization, llama.cpp, per-tensor layout mapping, model sensitivity, llm optimization by klotz
3. Tiel-Coder-35B-A3B-GGUF

   This repository provides quantized GGUF versions of the Tiel coder model, which is a re-quantized and dynamically improved version of Ornith-1.5 35B designed specifically for agentic coding tasks. It excels at multi-turn conversations and solving codebase issues with performance comparable to Opus 4.6 medium in certain benchmarks.
   
   - Optimized via an importance matrix (imatrix) using a calibration corpus heavily weighted toward code.
   - Inherits the Ornith-1.5 vision tower, allowing for image-to-text tasks like analyzing screenshots of failing tests.
   - Uses the Sharp chat template to produce more concise answers compared to its base model.
   - Designed with a Mixture of Experts (MoE) architecture that enables fast inference and small KV cache footprint in RAM.

   2026-09-12 Tags: tiel-coder, gguf, moe, agentic coding, quantization, llama.cpp by klotz
4. club-3090/docs/SINGLE_CARD.md at master · noonghunna/club-3090 · GitHub

   noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
   - A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
   - Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
   - The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.

   2026-09-06 Tags: rtx 3090, single gpu, vllm, llama.cpp, kv cache, llm, inference, vram budget, qwen, gemma by klotz
5. Qwen3.8 27B GGUF Benchmark: Q4 to Q1 Accuracy and Token Efficiency

   Benjamin Marie writes about a comprehensive benchmark of 15 GGUF quantizations of Qwen3.8 27B, ranging from Q4_K_XL down to IQ1_M, evaluated using over 150 million tokens generated across roughly 8 days on an NVIDIA RTX Pro 6000. Using 950 prompts subsampled from MMLU-Pro, LiveCodeBench, and GPQA Diamond, he measures both accuracy and token efficiency to identify the lowest quantization level that retains at least 95% of the original BF16 model's performance.
   
   | Chart pt | Quantization | Provider | GGUF file | Size (GB) | Accuracy recovery vs BF16 | Tokens generated | ≥ 95% threshold? |
   |---|---|---|---|---|---|---|---|
   | 1 | IQ3_XXS | bartowski | Qwen3.8-27B-IQ3_XXS.gguf | 12.39 | 97.7% | 11.54M | Yes |
   | 2 | IQ4_XS | bartowski | Qwen3.8-27B-IQ4_XS.gguf | 15.33 | 99.1% | 10.18M | Yes |
   | 3 | IQ2_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ2_S.gguf | 10.85 | 95.9% | 12.24M | Yes |
   | 4 | IQ3_S (AD) | AtomicChat | Qwen3.8-27B-AD-IQ3_S.gguf | 13.60 | 101.1% | 10.48M | Yes* |
   | 5 | Q4_K_M (AD) | AtomicChat | Qwen3.8-27B-AD-Q4_K_M.gguf | 16.84 | 99.9% | 9.99M | Yes |
   | 6 | IQ2_S (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf | 9.26 | 92.2% | 11.81M | **No** |
   | 7 | IQ3_XXS (GSQ-RCO) | ISTA-DASLab | Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf | 10.09 | 96.5% | 10.87M | Yes |
   | 8 | Ridge 3.7 bpw | empero-ai | Qwen3.8-27B-Ridge-3.7bpw.gguf | 12.26 | 97.4% | 10.29M | Yes |
   | 9 | IQ1_M (UD) | unsloth | Qwen3.8-27B-UD-IQ1_M.gguf | 6.73 | 53.4% | 15.41M | **No** |
   | 10 | IQ2_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ2_XXS.gguf | 7.27 | 74.3% | 11.96M | **No** |
   | 11 | IQ3_XXS (UD) | unsloth | Qwen3.8-27B-UD-IQ3_XXS.gguf | 10.58 | 95.5% | 12.27M | Yes |
   | 12 | Q2_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q2_K_XL.gguf | 9.48 | 96.0% | 11.89M | Yes |
   | 13 | Q3_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q3_K_XL.gguf | 12.80 | 100.0% | 9.89M | Yes |
   | 14 | Q4_K_XL (UD) | unsloth | Qwen3.8-27B-UD-Q4_K_XL.gguf | 17.21 | 101.0% | 9.47M | Yes* |
   | 15 | Q4_K_XL abliterated (Huihui UD) | huihui-ai | Huihui-Qwen3.8-27B-abliterated-UD-Q4_K_XL.gguf | 17.03 | 98.8% | 9.93M | Yes |
   
   `* = above 100% BF16 (sampling variance, not a real gain); below-threshold points are 6, 9, and 10.`

   2026-09-02 Tags: qwen3.8 27b, gguf, quantization, llama.cpp, l llm, benchmark, token efficiency, mmlu-pro, livecodebench, gpqa by klotz
6. I turned my old phone into a local LLM server, and it handles productivity tasks better than I expected

   Ayush Pande writes about transforming an outdated Poco M6 Pro smartphone into a functional local LLM server using llama.cpp via Termux. By utilizing lightweight inference engines and specific edge models like Gemma 4 E2B, the author was able to perform productivity tasks such as OCR reports, document summarization, and email proofreading locally on the device with respectable performance levels.
   
   - The setup uses Termux to install dependencies and llama.cpp for ultra-minimalist resource consumption.
   - Gemma 4 E2B is highlighted for its Per-Layer Embeddings architecture, which allows it to maintain high reasoning capabilities despite a small footprint.
   - The phone achieved an average speed of 5-6 tokens per second while running the model and other containerized services.
   - While capable of mobile productivity, the setup is not intended to replace heavy home lab nodes for complex coding or automation tasks.

   2026-09-01 Tags: local llm, llama.cpp, termux, gemma 4 e2b, smartphone server, edge by klotz
7. mozilla-ai/llamafile: Distribute and run LLMs with a single file.

   llamafile is a Mozilla.ai project that packages LLMs into single-file executables by combining llama.cpp with Cosmopolitan Libc, enabling local inference across most operating systems and CPU architectures with no installation required. The project also includes whisperfile (speech-to-text via whisper.cpp) and diffusionfile (image generation via stable-diffusion.cpp). Version 0.10.0 introduced a major build system overhaul to track upstream llama.cpp more closely, adding GPU support (CUDA, ROCm, Metal, Vulkan), an interactive TUI, multimodal input, tool calling, and thinking-mode support.
   
   - The build system uses Cosmopolitan Libc (cosmocc 4.0.2) to produce truly portable binaries in a single file.

   2026-08-25 Tags: llamafile, llama.cpp, single-file executable, local llm inference, cosmopolitan, libc, whisperfile, mozilla, cross-platform, elf, file-formats by klotz
8. Evidence: tool-list POSITION affects tool attention (Qwen3.8-27B) ''� descriptions + tail placement fix "tool not in kit" failures

   simongonzalezdc writes about a serving-side tool-attention failure in Qwen3.8-27B via llama-server (jinja templates) where the position of tools in the rendered payload determines whether the model will invoke them. With an 8-tool list, early-positioned, description-less tools were verbally refused as "not wired up" (0/6 success), while the same tools worked perfectly when moved to the tail (3/3) or given a one-line description (6/6). A follow-up with 13 tools confirmed the pattern: a mid-list pair prescribed by name failed 0/5, but tail placement restored 3/3 clean invocation.
   - Hypothesis: later tools in the rendered template output receive more attention; compact single-JSON-blob rendering may amplify the position effect versus per-tool natural-language rendering.
   - Controlled matrix (24 runs + 27 curl controls) ruled out tool count, context/style, effort level, and fresh-session as causes.
   - Test hardware: AMD Strix Halo (gfx1151/HIP) running Q4_K_XL quantization on current llama-server.
   - Full methodology and ledgers published in a companion repo: KyaniteLabs/qwen38-27b-strix-halo.

   2026-08-22 Tags: llama.cpp, qwen3.8-27b, tool calling, attention position, chat template, jinja, llama-server, llm by klotz
9. The Economics and Engineering of On-Premises LLMs

   Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
   
   - Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
   - Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
   - vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
   - DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
   - Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild

   2026-08-19 Tags: local llm, inference, cost, llama.cpp, vllm, model quantization, llm, acm by klotz
10. Meta is back with Muse Glimmer: local, agentic, multimodal, and open source

    Pedro Cuenca writes Meta released Muse Glimmer-30B, a local, open-source multimodal model distilled from its larger Muse architecture. Designed for agentic workflows, it combines a 28B text decoder with a 2B vision encoder, supporting image, video, and multimodal tool calling out of the box. The release includes immediate compatibility with major inference frameworks like transformers, llama.cpp, and vLLM, alongside built-in speculative decoding for faster generation.
    
    - Features a hybrid attention pattern alternating between three sliding window layers and one full attention layer.
    - Incorporates a DFlash block-diffusion drafter to accelerate structured text generation like coding.
    - Supports fine-tuning via TRL with practical minimums ranging from one to eight H100 GPUs depending on the method.
    - Demonstrates autonomous agent capabilities such as self-quantization, self-deployment, and hardware-specific optimization.

    2026-08-10 Tags: muse glimmer, meta, multimodal, llm, model, agentic workflows, speculative decoding, transformers, llama.cpp, open source, gguf, 24gb by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0