The article proposes a new framework, LongRAG, that aims to improve the performance of Retrieval-Augmented Generation (RAG) by using long retriever and reader components. LongRAG processes Wikipedia into larger 4K-token units, reducing the total units from 22M to 600K, thus decreasing the burden on the retriever. The top-k retrieved units (≈30K tokens) are then fed to a long-context Language Model for zero-shot answer extraction. LongRAG achieves EM of 62.7% on NQ and 64.3% on HotpotQA (full-wiki), which is on par with the state-of-the-art model.
Stay informed about the latest artificial intelligence (AI) terminology with this comprehensive glossary. From algorithm and AI ethics to generative AI and overfitting, learn the essential AI terms that will help you sound smart over drinks or impress in a job interview.
This article explains the Long RoPE methodology used to expand the context lengths in LLMs without significant performance degradation. It discusses the importance of context length in LLMs and the limitations of previous positional encoding methods. The article then introduces Rotational Positional Encoding (RoPE) and its limitations, and explains how Long RoPE extends RoPE to larger contexts.
