SemanticScuttle - klotz.me

Tags: kv cache* + machine learning*

0 bookmark(s) - Sort by: Date ↓ / Title /

1. Google’s TurboQuant Compression May Support Faster Inference, Same Accuracy on Less Capable Hardware

   Google Research has introduced TurboQuant, a new quantization algorithm designed to compress the Key-Value (KV) cache of large language models by up to 6x. By utilizing a two-step process involving randomized Hadamard transforms and Quantized Johnson-Lindenstrauss transforms, the method achieves 3.5-bit compression with near-zero accuracy loss on benchmarks like LongBench. This optimization addresses the massive VRAM requirements of long-context windows, potentially allowing large models to run on significantly less powerful hardware.
   Key points:
   * Compresses KV cache down to 3.5 bits per value.
   * Maintains inference accuracy without requiring model retraining.
   * Uses data vector rotation and QJL transforms to handle outlier distribution skew.
   * Reduces the memory bottleneck for long-context LLM inference.
   * Enables massive context windows on more modest hardware configurations.

   2026-04-17 Tags: machine learning, large language models, kv cache, quantization, turboquant, google research, inference optimization by klotz
2. KV Cache Transform Coding for Compact Storage in LLM Inference

   This paper introduces KVTC, a lightweight transform coder designed to compress key-value (KV) caches, which are crucial for efficient large language model (LLM) serving. KV caches enable reuse across conversation turns, but can consume significant GPU memory. KVTC addresses this by applying techniques from classical media compression – PCA-based decorrelation, adaptive quantization, and entropy coding – to reduce cache size without requiring changes to the underlying model. The authors demonstrate that KVTC achieves up to 20x compression while maintaining reasoning accuracy and long-context performance, and even higher compression for specific applications.

   2026-03-18 Tags: llm, kv cache, kvtc, compression, machine learning, transformers by klotz
3. How LLM Inference Works

   A deep dive into the process of LLM inference, covering tokenization, transformer architecture, KV caching, and optimization techniques for efficient text generation.

   2025-11-26 Tags: llm, inference, transformer, tokenization, kv cache, quantization, deep learning, machine learning, neural networks by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0