Deploying Large Language Models in streaming applications is limited by growing KV cache memory during decoding and an inability to generalize beyond training sequence length, with naive window attention failing once text exceeds the cache size. The authors observe an attention sink phenomenon where models assign strong attention scores to initial tokens even when semantically irrelevant, and find that retaining the KV of those initial sinks together with a sliding window recovers performance. This motivates StreamingLLM, a zero-shot framework that enables LLMs trained on finite windows to generalize to infinite sequence lengths without fine-tuning, achieving stable language modeling up to 4 million tokens on Llama-2, MPT, Falcon and Pythia.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
