-> **The numbers speak for themselves:** Index 60 million Wikipedia chunks in just 6GB instead of 201GB. From emails to browser history, everything fits on your laptop. [See detailed benchmarks for different applications below β](#storage-usage-comparison)
+> **The numbers speak for themselves:** Index 60 million Wikipedia chunks in just 6GB instead of 201GB. From emails to browser history, everything fits on your laptop. [See detailed benchmarks for different applications below β](#storage-comparison-on-different-applications)
π **Privacy:** Your data never leaves your laptop. No OpenAI, no cloud, no "terms of service".
@@ -402,7 +402,7 @@ Options:
π **[Simple Example: Compare LEANN vs FAISS β](examples/compare_faiss_vs_leann.py)**
-### Storage Comparison
+### Storage Comparison on Different Applications {#storage-comparison-on-different-applications}
| System | DPR (2.1M) | Wiki (60M) | Chat (400K) | Email (780K) | Browser (38K) |
|--------|-------------|------------|-------------|--------------|---------------|
@@ -439,98 +439,15 @@ If you find Leann useful, please cite:
}
```
-## β¨ Features
+## β¨ [Detailed Features β](docs/features.md)
-### π₯ Core Features
-
-- **π Real-time Embeddings** - Eliminate heavy embedding storage with dynamic computation using optimized ZMQ servers and highly optimized search paradigm (overlapping and batching) with highly optimized embedding engine
-- **π Scalable Architecture** - Handles millions of documents on consumer hardware; the larger your dataset, the more LEANN can save
-- **π― Graph Pruning** - Advanced techniques to minimize the storage overhead of vector search to a limited footprint
-- **ποΈ Pluggable Backends** - DiskANN, HNSW/FAISS with unified API
-
-### π οΈ Technical Highlights
-- **π Recompute Mode** - Highest accuracy scenarios while eliminating vector storage overhead
-- **β‘ Zero-copy Operations** - Minimize IPC overhead by transferring distances instead of embeddings
-- **π High-throughput Embedding Pipeline** - Optimized batched processing for maximum efficiency
-- **π― Two-level Search** - Novel coarse-to-fine search overlap for accelerated query processing (optional)
-- **πΎ Memory-mapped Indices** - Fast startup with raw text mapping to reduce memory overhead
-- **π MLX Support** - Ultra-fast recompute/build with quantized embedding models, accelerating building and search ([minimal example](test/build_mlx_index.py))
-
-### π¨ Developer Experience
-
-- **Simple Python API** - Get started in minutes
-- **Extensible backend system** - Easy to add new algorithms
-- **Comprehensive examples** - From basic usage to production deployment
-
-## π€ Contributing
-
-We welcome contributions! Leann is built by the community, for the community.
-
-### Ways to Contribute
-
-- π **Bug Reports**: Found an issue? Let us know!
-- π‘ **Feature Requests**: Have an idea? We'd love to hear it!
-- π§ **Code Contributions**: PRs welcome for all skill levels
-- π **Documentation**: Help make Leann more accessible
-- π§ͺ **Benchmarks**: Share your performance results
+## π€ [Contributing β](docs/contributing.md)
-
-## FAQ
-
-### 1. My building time seems long
-
-You can speed up the process by using a lightweight embedding model. Add this to your arguments:
-
-```bash
---embedding-model sentence-transformers/all-MiniLM-L6-v2
-```
-**Model sizes:** `all-MiniLM-L6-v2` (30M parameters), `facebook/contriever` (~100M parameters), `Qwen3-0.6B` (600M parameters)
+## [FAQ β](docs/faq.md)
-## π Roadmap
-
-### π― Q2 2025
-
-- [X] DiskANN backend with MIPS/L2/Cosine support
-- [X] HNSW backend integration
-- [X] Real-time embedding pipeline
-- [X] Memory-efficient graph pruning
-
-### π Q3 2025
-
-
-- [ ] Advanced caching strategies
-- [ ] Add contextual-retrieval https://www.anthropic.com/news/contextual-retrieval
-- [ ] Add sleep-time-compute and summarize agent! to summarilze the file on computer!
-- [ ] Add OpenAI recompute API
-
-### π Q4 2025
-
-- [ ] Integration with LangChain/LlamaIndex
-- [ ] Visual similarity search
-- [ ] Query rewrtiting, rerank and expansion
+## π [Roadmap β](docs/roadmap.md)
## π License
@@ -538,11 +455,7 @@ MIT License - see [LICENSE](LICENSE) for details.
## π Acknowledgments
-- **Microsoft Research** for the DiskANN algorithm
-- **Meta AI** for FAISS and optimization insights
-- **HuggingFace** for the transformer ecosystem
-- **Our amazing contributors** who make this possible
-
+This work is done atΒ [**Berkeley Sky Computing Lab**](https://sky.cs.berkeley.edu/)
---
diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..e8d262c --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,11 @@ +# π€ Contributing + +We welcome contributions! Leann is built by the community, for the community. + +## Ways to Contribute + +- π **Bug Reports**: Found an issue? Let us know! +- π‘ **Feature Requests**: Have an idea? We'd love to hear it! +- π§ **Code Contributions**: PRs welcome for all skill levels +- π **Documentation**: Help make Leann more accessible +- π§ͺ **Benchmarks**: Share your performance results \ No newline at end of file diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 0000000..ba06e1a --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,10 @@ +# FAQ + +## 1. My building time seems long + +You can speed up the process by using a lightweight embedding model. Add this to your arguments: + +```bash +--embedding-model sentence-transformers/all-MiniLM-L6-v2 +``` +**Model sizes:** `all-MiniLM-L6-v2` (30M parameters), `facebook/contriever` (~100M parameters), `Qwen3-0.6B` (600M parameters) \ No newline at end of file diff --git a/docs/features.md b/docs/features.md new file mode 100644 index 0000000..a0abf85 --- /dev/null +++ b/docs/features.md @@ -0,0 +1,22 @@ +# β¨ Detailed Features + +## π₯ Core Features + +- **π Real-time Embeddings** - Eliminate heavy embedding storage with dynamic computation using optimized ZMQ servers and highly optimized search paradigm (overlapping and batching) with highly optimized embedding engine +- **π Scalable Architecture** - Handles millions of documents on consumer hardware; the larger your dataset, the more LEANN can save +- **π― Graph Pruning** - Advanced techniques to minimize the storage overhead of vector search to a limited footprint +- **ποΈ Pluggable Backends** - DiskANN, HNSW/FAISS with unified API + +## π οΈ Technical Highlights +- **π Recompute Mode** - Highest accuracy scenarios while eliminating vector storage overhead +- **β‘ Zero-copy Operations** - Minimize IPC overhead by transferring distances instead of embeddings +- **π High-throughput Embedding Pipeline** - Optimized batched processing for maximum efficiency +- **π― Two-level Search** - Novel coarse-to-fine search overlap for accelerated query processing (optional) +- **πΎ Memory-mapped Indices** - Fast startup with raw text mapping to reduce memory overhead +- **π MLX Support** - Ultra-fast recompute/build with quantized embedding models, accelerating building and search ([minimal example](test/build_mlx_index.py)) + +## π¨ Developer Experience + +- **Simple Python API** - Get started in minutes +- **Extensible backend system** - Easy to add new algorithms +- **Comprehensive examples** - From basic usage to production deployment \ No newline at end of file diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..ac6a839 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,21 @@ +# π Roadmap + +## π― Q2 2025 + +- [X] DiskANN backend with MIPS/L2/Cosine support +- [X] HNSW backend integration +- [X] Real-time embedding pipeline +- [X] Memory-efficient graph pruning + +## π Q3 2025 + +- [ ] Advanced caching strategies +- [ ] Add contextual-retrieval https://www.anthropic.com/news/contextual-retrieval +- [ ] Add sleep-time-compute and summarize agent! to summarilze the file on computer! +- [ ] Add OpenAI recompute API + +## π Q4 2025 + +- [ ] Integration with LangChain/LlamaIndex +- [ ] Visual similarity search +- [ ] Query rewrtiting, rerank and expansion \ No newline at end of file diff --git a/examples/compare_faiss_vs_leann.py b/examples/compare_faiss_vs_leann.py index 2a2a55a..ea0ef3e 100644 --- a/examples/compare_faiss_vs_leann.py +++ b/examples/compare_faiss_vs_leann.py @@ -135,6 +135,7 @@ def test_leann_hnsw(): nodes = node_parser.get_nodes_from_documents([doc]) for node in nodes: all_texts.append(node.get_content()) + print(f"Total number of chunks: {len(all_texts)}") tracker.checkpoint("After text chunking")