The article explains semantic text chunking, a technique for automatically grouping similar pieces of text to be used in pre-processing stages for Retrieval Augmented Generation (RAG) or similar applications. It uses visualizations to understand the chunking process and explores extensions involving clustering and LLM-powered labeling.
This article discusses the importance of chunking, embedding, and indexing in RAGs (Recursive Auto-Segmented Graphs). The author compares recursive character splitting and semantic splitting techniques for text chunking and suggests the use of agentic chunking for superior RAG retrieval.
This repository showcases various advanced techniques for Retrieval-Augmented Generation (RAG) systems. RAG systems combine information retrieval with generative models to provide accurate and contextually rich responses.
The llmsherpa project provides APIs to accelerate Large Language Model (LLM) projects. It includes features like LayoutPDFReader for PDF text parsing, smart chunking for vector search and Retrieval Augmented Generation, and table analysis. It is open-sourced under Apache 2.0 license.
This article explains Retrieval Augmented Generation (RAG), a method to reduce the risk of hallucinations in Large Language Models (LLMs) by limiting the context in which they generate answers. RAG is demonstrated using txtai, an open-source embeddings database for semantic search, LLM orchestration, and language model workflows.
This article discusses Retrieval-Augmented Generation (RAG) models, a new approach that addresses the limitations of traditional models in knowledge-intensive Natural Language Processing (NLP) tasks. RAG models combine parametric memory from pre-trained seq2seq models with non-parametric memory from a dense vector index of Wikipedia, enabling dynamic knowledge access and integration.
ColBERT is a new way of scoring passage relevance using a BERT language model that substantially solves the problems with dense passage retrieval.
