Scaling a simple RAG pipeline from simple notes to full books. This post elaborates on how to utilize larger files with your RAG pipeline by adding an extra step to the process — chunking.
The article explores the concept of writing code that is easy to read by leveraging brain spans: memory span, attention span, and structure span. It suggests guidelines for writing code that flows like a story, such as keeping functions small, using a single level of abstraction, and giving descriptive names to functions. The goal is to make code more readable and understandable, enhancing developer efficiency and collaboration.
The article explains semantic text chunking, a technique for automatically grouping similar pieces of text to be used in pre-processing stages for Retrieval Augmented Generation (RAG) or similar applications. It uses visualizations to understand the chunking process and explores extensions involving clustering and LLM-powered labeling.
This article discusses the importance of chunking, embedding, and indexing in RAGs (Recursive Auto-Segmented Graphs). The author compares recursive character splitting and semantic splitting techniques for text chunking and suggests the use of agentic chunking for superior RAG retrieval.
This article explores the limitations of position-based chunking in Retrieval Augmented Generation (RAG) systems and proposes semantic chunking as a better alternative for improved performance.
