Turbovec is an open-source vector index library written in Rust that features Python bindings. It utilizes Google's TurboQuant algorithm to provide highly efficient data quantization without the need for traditional codebook training steps like k-means. The library offers significant memory savings, reducing a 31 GB corpus of 10 million vectors down to just 4 GB, and demonstrates superior search speeds on ARM hardware compared to FAISS.
This article explores TurboQuant, a new vector quantization method introduced by Google researchers to address the massive memory requirements of Large Language Models (LLMs). As LLM parameters and Key-Value (KV) caches grow, memory management becomes a critical bottleneck for performance. TurboQuant utilizes the PolarQuant algorithm and the quantized Johnson-Lindenstrauss (QJL) algorithm to compress the KV cache significantly. Google claims this method can achieve up to 6x compression levels without a noticeable impact on inference times or accuracy. While the article notes that Google's benchmarking data is somewhat vague compared to competitors like NVIDIA's NVFP4, TurboQuant represents a significant development in optimizing AI hardware compatibility and real-time inference performance.
