This paper addresses the misalignment between traditional IR evaluation metrics and the requirements of modern Retrieval-Augmented Generation (RAG) systems. It proposes a novel annotation schema and the UDCG metric to better evaluate retrieval quality for LLM consumers.
This blog post demonstrates how to create a reusable retrieval evaluation dataset using an LLM to judge query-document pairs. It discusses the process, including building a small labeled dataset, aligning LLM judgments with human preferences, and using the LLM to judge a large set of queries and documents.
