SemanticScuttle - klotz.me

klotz: gguf* + llm*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

1. Unsloth Dynamic GGUFs on Aider Polyglot

   This article details the performance of Unsloth Dynamic GGUFs on the Aider Polyglot benchmark, showcasing how it can quantize LLMs like DeepSeek-V3.1 to as low as 1-bit while outperforming models like GPT-4.5 and Claude-4-Opus. It also covers benchmark setup, comparisons to other quantization methods, and chat template bug fixes.

   2025-10-13 Tags: unsloth, gguf, aider polyglot, llm, quantization, deepseek-v3.1, gpt-4, claude-4, model compression, fine-tuning, inference by klotz
2. guide : running gpt-oss with llama.cpp · Discussion #15396

   A detailed guide for running the new gpt-oss models locally with the best performance using `llama.cpp`. The guide covers a wide range of hardware configurations and provides CLI argument explanations and benchmarks for Apple Silicon devices.

   2025-10-04 Tags: llama.cpp, gpt-oss, large language model, inference, apple silicon, benchmarks, performance, gguf by klotz
3. Gemma 3: How to Run & Fine-tune

   How to run Gemma 3 effectively with our GGUFs on llama.cpp, Ollama, Open WebUI and how to fine-tune with Unsloth! This page details running Gemma 3 on various platforms, including phones, and fine-tuning it using Unsloth, addressing potential issues with float16 precision and providing optimal configuration settings.

   2025-08-16 Tags: gemma 3, llm, fine-tuning, llama.cpp, unsloth, gguf, gpu, colab, vision, audio, oobabooga by klotz
4. DeepSeek-R1-0528-Qwen3-8B-GGUF

   This page details the DeepSeek-R1-0528-Qwen3-8B model, a quantized version of DeepSeek-R1-0528, highlighting its improved reasoning capabilities, evaluation results, usage guidelines, and licensing information. It offers various quantization options (GGUF) for local execution.

   2025-05-30 Tags: deepseek-r1, qwen3, gguf, llm, quantization, reasoning, text generation, transformers, model card, mcp, huggingface by klotz
5. gguf-parser-web

   A web application for parsing GGUF files.

   2025-04-28 Tags: gguf, parser, huggingface, llm, gpu by klotz
6. Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens

   Alibaba's Qwen 2.5 LLM now supports input token limits up to 1 million using Dual Chunk Attention. Two models are released on Hugging Face, requiring significant VRAM for full capacity. Challenges in deployment with quantized GGUF versions and system resource constraints are discussed.

   2025-01-28 Tags: qwen2.5-1m, alibaba, hugging face, gguf, llm, simon willison by klotz
7. Ollama just made it easier to use AI on your laptop — with no internet required

   Ollama now supports HuggingFace GGUF models, making it easier for users to run AI models locally without internet. The GGUF format allows for the use of AI models on modest-sized consumer hardware.

   2024-10-24 Tags: ollama, huggingface, gguf, llm, localllama by klotz
8. TIL: Building llamafiles from Llama 3.2 GGUFs

   A step-by-step guide on building llamafiles from Llama 3.2 GGUFs, including scripting and Dockerization.

   2024-09-28 Tags: llamafile, llama.cpp, llm, llama 3.2, gguf, model quantization, docker, mozilla-ocho by klotz
9. GGUF Quantization with Imatrix and K-Quantization to Run LLMs on Your CPU

   This article explains how to accurately quantize a Large Language Model (LLM) and convert it to the GGUF format for efficient CPU inference. It covers using an importance matrix (imatrix) and K-Quantization method with Gemma 2 Instruct as an example, while highlighting its applicability to other models like Qwen2, Llama 3, and Phi-3.

   2024-09-14 Tags: gguf, quantization, llm, cpu, inference, imatrix by klotz
10. Artifacts Quantized LLM Inference Performance Results on 70b+ Models

    This document contains the quantized LLM inference performance results on 70b+ models.

    2024-06-23 Tags: artifacts, quantization, llm, gguf by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0