β‘ Storage Saving RAG sytem on Consumer Device
Quick Start β’ Features β’ Benchmarks β’ Paper
--- ## π What is Leann? **Leann** revolutionizes Retrieval-Augmented Generation (RAG) by eliminating the storage bottleneck of traditional vector databases. Instead of pre-computing and storing billions of embeddings, Leann dynamically computes embeddings at query time using optimized graph-based search algorithms. ### π― Why Leann? Traditional RAG systems face a fundamental trade-off: - **πΎ Storage**: Storing embeddings for millions of documents requires massive disk space - **π Memory overhead**: The indexes LlamaIndex uses usually face high memory overhead (e.g., in-memory vector databases) - **π° Cost**: Vector databases are expensive to scale **Leann revolutionizes this with Graph-based recomputation and cutting-edge system optimizations:** - β **Zero embedding storage** - Only graph structure is persisted, reducing storage by 94-97% - β **Real-time computation** - Embeddings computed on-demand with low latency - β **Memory efficient** - Runs on consumer hardware with theoretical zero memory overhead - β **Graph-based optimization** - Advanced pruning techniques for efficient search while keeping low storage cost, with batching and overlapping strategies using low-precision search to optimize latency - β **Pluggable backends** - Support for DiskANN, HNSW, and other ANN algorithms (welcome contributions!) ## π Quick Start ### Installation ```bash git clone git@github.com:yichuan520030910320/LEANN-RAG.git leann cd leann git submodule update --init --recursive ``` **macOS:** ```bash brew install llvm libomp boost protobuf export CC=$(brew --prefix llvm)/bin/clang export CXX=$(brew --prefix llvm)/bin/clang++ uv sync ``` **Linux (Ubuntu/Debian):** ```bash sudo apt-get install libomp-dev libboost-all-dev protobuf-compiler libabsl-dev libmkl-full-dev libaio-dev uv sync ``` ### 30-Second Example ```python from leann.api import LeannBuilder, LeannSearcher # 1. Build index (no embeddings stored!) builder = LeannBuilder(backend_name="diskann") builder.add_text("Python is a powerful programming language") builder.add_text("Machine learning transforms industries") builder.add_text("Neural networks process complex data") builder.build_index("knowledge.leann") # 2. Search with real-time embeddings searcher = LeannSearcher("knowledge.leann") results = searcher.search("programming languages", top_k=2) for result in results: print(f"Score: {result['score']:.3f} - {result['text']}") ``` ### Run the Demo ```bash uv run examples/document_search.py ``` or you want to use python ```bash source .venv/bin/activate python ./examples/main_cli_example.py ``` **PDF RAG Demo (using LlamaIndex for document parsing and Leann for indexing/search)** This demo showcases how to build a RAG system for PDF documents using Leann. 1. Place your PDF files (and other supported formats like .docx, .pptx, .xlsx) into the `examples/data/` directory. 2. Ensure you have an `OPENAI_API_KEY` set in your environment variables or in a `.env` file for the LLM to function. ```bash uv run examples/main_cli_example.py ``` ### Regenerating Protobuf Files If you modify any `.proto` files (such as `embedding.proto`), or if you see errors about protobuf version mismatch, **regenerate the C++ protobuf files** to match your installed version: ```bash cd packages/leann-backend-diskann protoc --cpp_out=third_party/DiskANN/include --proto_path=third_party embedding.proto protoc --cpp_out=third_party/DiskANN/src --proto_path=third_party embedding.proto ``` This ensures the generated files are compatible with your system's protobuf library. ## β¨ Features ### π₯ Core Features - **π Multiple Distance Functions**: L2, Cosine, MIPS (Maximum Inner Product Search) - **ποΈ Pluggable Backends**: DiskANN, HNSW/FAISS with unified API - **π Real-time Embeddings**: Dynamic computation using optimized ZMQ servers - **π Scalable Architecture**: Handles millions of documents on consumer hardware - **π― Graph Pruning**: Advanced techniques for memory-efficient search ### π οΈ Technical Highlights - **Zero-copy operations** for maximum performance - **SIMD-optimized** distance computations (AVX2/AVX512) - **Async embedding pipeline** with batched processing - **Memory-mapped indices** for fast startup - **Recompute mode** for highest accuracy scenarios ### π¨ Developer Experience - **Simple Python API** - Get started in minutes - **Extensible backend system** - Easy to add new algorithms - **Comprehensive examples** - From basic usage to production deployment - **Rich debugging tools** - Built-in performance profiling ## π Benchmarks ### Memory Usage Comparison | System | 1M Documents | 10M Documents | 100M Documents | | --------------------- | ---------------- | ---------------- | ---------------- | | Traditional Vector DB | 3.1 GB | 31 GB | 310 GB | | **Leann** | **180 MB** | **1.2 GB** | **8.4 GB** | | **Reduction** | **94.2%** | **96.1%** | **97.3%** | ### Query Performance | Backend | Index Size | Query Time | Recall@10 | | ------------------- | ---------- | ---------- | --------- | | DiskANN | 1M docs | 12ms | 0.95 | | DiskANN + Recompute | 1M docs | 145ms | 0.98 | | HNSW | 1M docs | 8ms | 0.93 | *Benchmarks run on AMD Ryzen 7 with 32GB RAM* ## ποΈ Architecture ``` βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ β Query Text βββββΆβ Embedding βββββΆβ Graph-based β β β β Computation β β Search β βββββββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ β β βΌ βΌ ββββββββββββββββ ββββββββββββββββ β ZMQ Server β β Pruned Graph β β (Cached) β β Index β ββββββββββββββββ ββββββββββββββββ ``` ### Key Components 1. **π§ Embedding Engine**: Real-time transformer inference with caching 2. **π Graph Index**: Memory-efficient navigation structures 3. **π Search Coordinator**: Orchestrates embedding + graph search 4. **β‘ Backend Adapters**: Pluggable algorithm implementations ## π Supported Models & Backends ### π€ Embedding Models - **sentence-transformers/all-mpnet-base-v2** (default) - **sentence-transformers/all-MiniLM-L6-v2** (lightweight) - Any HuggingFace sentence-transformer model - Custom model support via API ### π§ Search Backends - **DiskANN**: Microsoft's billion-scale ANN algorithm - **HNSW**: Hierarchical Navigable Small World graphs - **Coming soon**: ScaNN, Faiss-IVF, NGT ### π Distance Functions - **L2**: Euclidean distance for precise similarity - **Cosine**: Angular similarity for normalized vectors - **MIPS**: Maximum Inner Product Search for recommendation systems ## π¬ Paper If you find Leann useful, please cite: **[LEANN: A Low-Storage Vector Index](https://arxiv.org/abs/2506.08276)** ```bibtex @misc{wang2025leannlowstoragevectorindex, title={LEANN: A Low-Storage Vector Index}, author={Yichuan Wang and Shu Liu and Zhifei Li and Yongji Wu and Ziming Mao and Yilong Zhao and Xiao Yan and Zhiying Xu and Yang Zhou and Ion Stoica and Sewon Min and Matei Zaharia and Joseph E. Gonzalez}, year={2025}, eprint={2506.08276}, archivePrefix={arXiv}, primaryClass={cs.DB}, url={https://arxiv.org/abs/2506.08276}, } ``` ## π Use Cases ### πΌ Enterprise RAG ```python # Handle millions of documents with limited resources builder = LeannBuilder( backend_name="diskann", distance_metric="cosine", graph_degree=64, memory_budget="4GB" ) ``` ### π¬ Research & Experimentation ```python # Quick prototyping with different algorithms for backend in ["diskann", "hnsw"]: searcher = LeannSearcher(index_path, backend=backend) evaluate_recall(searcher, queries, ground_truth) ``` ### π Real-time Applications ```python # Sub-second response times chat = LeannChat("knowledge.leann") response = chat.ask("What is quantum computing?") # Returns in <100ms with recompute mode ``` ## π€ Contributing We welcome contributions! Leann is built by the community, for the community. ### Ways to Contribute - π **Bug Reports**: Found an issue? Let us know! - π‘ **Feature Requests**: Have an idea? We'd love to hear it! - π§ **Code Contributions**: PRs welcome for all skill levels - π **Documentation**: Help make Leann more accessible - π§ͺ **Benchmarks**: Share your performance results ### Development Setup ```bash git clone git@github.com:yichuan520030910320/LEANN-RAG.git leann cd leann git submodule update --init --recursive uv sync --dev uv run pytest tests/ ``` ### Quick Tests ```bash # Sanity check all distance functions uv run python tests/sanity_checks/test_distance_functions.py # Verify L2 implementation uv run python tests/sanity_checks/test_l2_verification.py ``` ## β FAQ ### Common Issues #### NCCL Topology Error **Problem**: You encounter `ncclTopoComputePaths` error during document processing: ``` ncclTopoComputePaths (system=β Star us on GitHub if Leann is useful for your research or applications!
Made with β€οΈ by the Leann team