The article explores the architectural changes that enable DeepSeek's models to perform well with fewer resources, focusing on Multi-Head Latent Attention (MLA). It discusses the evolution of attention mechanisms, from Bahdanau to Transformer's Multi-Head Attention (MHA), and introduces Grouped-Query Attention (GQA) as a solution to MHA's memory inefficiencies. The article highlights DeepSeek's competitive performance despite lower reported training costs.
This post explores optimization techniques for the Key-Value (KV) cache in Large Language Models (LLMs) to enhance scalability and reduce memory footprint, covering methods like Grouped-query Attention, Sliding Window Attention, PagedAttention, and distributed KV cache across multiple GPUs.
