Microsoft researchers introduce LongRoPE2, a method to extend large language model context windows to 128K tokens while maintaining over 97% short-context accuracy, addressing key limitations in positional embeddings.
This article explains the Long RoPE methodology used to expand the context lengths in LLMs without significant performance degradation. It discusses the importance of context length in LLMs and the limitations of previous positional encoding methods. The article then introduces Rotational Positional Encoding (RoPE) and its limitations, and explains how Long RoPE extends RoPE to larger contexts.
