This article introduces the pyramid search approach using Agentic Knowledge Distillation to address the limitations of traditional RAG strategies in document ingestion.
The pyramid structure allows for multi-level retrieval, including atomic insights, concepts, abstracts, and recollections. This structure mimics a knowledge graph but uses natural language, making it more efficient for LLMs to interact with.
**Knowledge Distillation Process**:
- **Conversion to Markdown**: Documents are converted to Markdown for better token efficiency and processing.
- **Atomic Insights Extraction**: Each page is processed using a two-page sliding window to generate a list of insights in simple sentences.
- **Concept Distillation**: Higher-level concepts are identified from the insights to reduce noise and preserve essential information.
- **Abstract Creation**: An LLM writes a comprehensive abstract for each document, capturing dense information efficiently.
- **Recollections/Memories**: Critical information useful across all tasks is stored at the top of the pyramid.
The pyramid structure allows for multi-level retrieval, including atomic insights, concepts, abstracts, and recollections. This structure mimics a knowledge graph but uses natural language, making it more efficient for LLMs to interact with.
**Knowledge Distillation Process**:
- **Conversion to Markdown**: Documents are converted to Markdown for better token efficiency and processing.
- **Atomic Insights Extraction**: Each page is processed using a two-page sliding window to generate a list of insights in simple sentences.
- **Concept Distillation**: Higher-level concepts are identified from the insights to reduce noise and preserve essential information.
- **Abstract Creation**: An LLM writes a comprehensive abstract for each document, capturing dense information efficiently.
- **Recollections/Memories**: Critical information useful across all tasks is stored at the top of the pyramid.
An open-source project offering a functional RAG UI for document QA, suitable for both end-users and developers. It supports various LLM providers, is customizable, and offers multi-modal QA, citations, and complex reasoning methods.
pip install 'ragna builtin » ' # Install ragna with all extensions
ragna config # Initialize configuration
ragna ui # Launch the web app
ragna config # Initialize configuration
ragna ui # Launch the web app
Image Similarity Search
Reverse Image Search
Object Similarity Search
Robust OCR Document Search
Semantic Search
Cross-modal Retrieval
Probing Perceptual Similarity
Comparing Model Representations
Concept Interpolation
Concept Space Traversal
Image Similarity Search
Reverse Image Search
Object Similarity Search
Robust OCR Document Search
Semantic Search
Cross-modal Retrieval
Probing Perceptual Similarity
Comparing Model Representations
Concept Interpolation
Concept Space Traversal
Image Similarity Search
