* refactor: Unify examples interface with BaseRAGExample - Create BaseRAGExample base class for all RAG examples - Refactor 4 examples to use unified interface: - document_rag.py (replaces main_cli_example.py) - email_rag.py (replaces mail_reader_leann.py) - browser_rag.py (replaces google_history_reader_leann.py) - wechat_rag.py (replaces wechat_history_reader_leann.py) - Maintain 100% parameter compatibility with original files - Add interactive mode support for all examples - Unify parameter names (--max-items replaces --max-emails/--max-entries) - Update README.md with new examples usage - Add PARAMETER_CONSISTENCY.md documenting all parameter mappings - Keep main_cli_example.py for backward compatibility with migration notice All default values, LeannBuilder parameters, and chunking settings remain identical to ensure full compatibility with existing indexes. * fix: Update CI tests for new unified examples interface - Rename test_main_cli.py to test_document_rag.py - Update all references from main_cli_example.py to document_rag.py - Update tests/README.md documentation The tests now properly test the new unified interface while maintaining the same test coverage and functionality. * fix: Fix pre-commit issues and update tests - Fix import sorting and unused imports - Update type annotations to use built-in types (list, dict) instead of typing.List/Dict - Fix trailing whitespace and end-of-file issues - Fix Chinese fullwidth comma to regular comma - Update test_main_cli.py to test_document_rag.py - Add backward compatibility test for main_cli_example.py - Pass all pre-commit hooks (ruff, ruff-format, etc.) * refactor: Remove old example scripts and migration references - Delete old example scripts (mail_reader_leann.py, google_history_reader_leann.py, etc.) - Remove migration hints and backward compatibility - Update tests to use new unified examples directly - Clean up all references to old script names - Users now only see the new unified interface * fix: Restore embedding-mode parameter to all examples - All examples now have --embedding-mode parameter (unified interface benefit) - Default is 'sentence-transformers' (consistent with original behavior) - Users can now use OpenAI or MLX embeddings with any data source - Maintains functional equivalence with original scripts * docs: Improve parameter categorization in README - Clearly separate core (shared) vs specific parameters - Move LLM and embedding examples to 'Example Commands' section - Add descriptive comments for all specific parameters - Keep only truly data-source-specific parameters in specific sections * docs: Make example commands more representative - Add default values to parameter descriptions - Replace generic examples with real-world use cases - Focus on data-source-specific features in examples - Remove redundant demonstrations of common parameters * docs: Reorganize parameter documentation structure - Move common parameters to a dedicated section before all examples - Rename sections to 'X-Specific Arguments' for clarity - Remove duplicate common parameters from individual examples - Better information architecture for users * docs: polish applications * docs: Add CLI installation instructions - Add two installation options: venv and global uv tool - Clearly explain when to use each option - Make CLI more accessible for daily use * docs: Clarify CLI global installation process - Explain the transition from venv to global installation - Add upgrade command for global installation - Make it clear that global install allows usage without venv activation * docs: Add collapsible section for CLI installation - Wrap CLI installation instructions in details/summary tags - Keep consistent with other collapsible sections in README - Improve document readability and navigation * style: format * docs: Fix collapsible sections - Make Common Parameters collapsible (as it's lengthy reference material) - Keep CLI Installation visible (important for users to see immediately) - Better information hierarchy * docs: Add introduction for Common Parameters section - Add 'Flexible Configuration' heading with descriptive sentence - Create parallel structure with 'Generation Model Setup' section - Improve document flow and readability * docs: nit * fix: Fix issues in unified examples - Add smart path detection for data directory - Fix add_texts -> add_text method call - Handle both running from project root and examples directory * fix: Fix async/await and add_text issues in unified examples - Remove incorrect await from chat.ask() calls (not async) - Fix add_texts -> add_text method calls - Verify search-complexity correctly maps to efSearch parameter - All examples now run successfully * feat: Address review comments - Add complexity parameter to LeannChat initialization (default: search_complexity) - Fix chunk-size default in README documentation (256, not 2048) - Add more index building parameters as CLI arguments: - --backend-name (hnsw/diskann) - --graph-degree (default: 32) - --build-complexity (default: 64) - --no-compact (disable compact storage) - --no-recompute (disable embedding recomputation) - Update README to document all new parameters * feat: Add chunk-size parameters and improve file type filtering - Add --chunk-size and --chunk-overlap parameters to all RAG examples - Preserve original default values for each data source: - Document: 256/128 (optimized for general documents) - Email: 256/25 (smaller overlap for email threads) - Browser: 256/128 (standard for web content) - WeChat: 192/64 (smaller chunks for chat messages) - Make --file-types optional filter instead of restriction in document_rag - Update README to clarify interactive mode and parameter usage - Fix LLM default model documentation (gpt-4o, not gpt-4o-mini) * feat: Update documentation based on review feedback - Add MLX embedding example to README - Clarify examples/data content description (two papers, Pride and Prejudice, Chinese README) - Move chunk parameters to common parameters section - Remove duplicate chunk parameters from document-specific section * docs: Emphasize diverse data sources in examples/data description * fix: update default embedding models for better performance - Change WeChat, Browser, and Email RAG examples to use all-MiniLM-L6-v2 - Previous Qwen/Qwen3-Embedding-0.6B was too slow for these use cases - all-MiniLM-L6-v2 is a fast 384-dim model, ideal for large-scale personal data * add response highlight * change rebuild logic * fix some example * feat: check if k is larger than #docs * fix: WeChat history reader bugs and refactor wechat_rag to use unified architecture * fix email wrong -1 to process all file * refactor: reorgnize all examples/ and test/ * refactor: reorganize examples and add link checker * fix: add init.py * fix: handle certificate errors in link checker * fix wechat * merge * docs: update README to use proper module imports for apps - Change from 'python apps/xxx.py' to 'python -m apps.xxx' - More professional and pythonic module calling - Ensures proper module resolution and imports - Better separation between apps/ (production tools) and examples/ (demos) --------- Co-authored-by: yichuan520030910320 <yichuan_wang@berkeley.edu>
89 lines
3.0 KiB
Python
89 lines
3.0 KiB
Python
"""
|
|
Simple demo showing basic leann usage
|
|
Run: uv run python examples/basic_demo.py
|
|
"""
|
|
|
|
import argparse
|
|
|
|
from leann import LeannBuilder, LeannChat, LeannSearcher
|
|
|
|
|
|
def main():
|
|
parser = argparse.ArgumentParser(
|
|
description="Simple demo of Leann with selectable embedding models."
|
|
)
|
|
parser.add_argument(
|
|
"--embedding_model",
|
|
type=str,
|
|
default="sentence-transformers/all-mpnet-base-v2",
|
|
help="The embedding model to use, e.g., 'sentence-transformers/all-mpnet-base-v2' or 'text-embedding-ada-002'.",
|
|
)
|
|
args = parser.parse_args()
|
|
|
|
print(f"=== Leann Simple Demo with {args.embedding_model} ===")
|
|
print()
|
|
|
|
# Sample knowledge base
|
|
chunks = [
|
|
"Machine learning is a subset of artificial intelligence that enables computers to learn without being explicitly programmed.",
|
|
"Deep learning uses neural networks with multiple layers to process data and make decisions.",
|
|
"Natural language processing helps computers understand and generate human language.",
|
|
"Computer vision enables machines to interpret and understand visual information from images and videos.",
|
|
"Reinforcement learning teaches agents to make decisions by receiving rewards or penalties for their actions.",
|
|
"Data science combines statistics, programming, and domain expertise to extract insights from data.",
|
|
"Big data refers to extremely large datasets that require special tools and techniques to process.",
|
|
"Cloud computing provides on-demand access to computing resources over the internet.",
|
|
]
|
|
|
|
print("1. Building index (no embeddings stored)...")
|
|
builder = LeannBuilder(
|
|
embedding_model=args.embedding_model,
|
|
backend_name="hnsw",
|
|
)
|
|
for chunk in chunks:
|
|
builder.add_text(chunk)
|
|
builder.build_index("demo_knowledge.leann")
|
|
print()
|
|
|
|
print("2. Searching with real-time embeddings...")
|
|
searcher = LeannSearcher("demo_knowledge.leann")
|
|
|
|
queries = [
|
|
"What is machine learning?",
|
|
"How does neural network work?",
|
|
"Tell me about data processing",
|
|
]
|
|
|
|
for query in queries:
|
|
print(f"Query: {query}")
|
|
results = searcher.search(query, top_k=2)
|
|
|
|
for i, result in enumerate(results, 1):
|
|
print(f" {i}. Score: {result.score:.3f}")
|
|
print(f" Text: {result.text[:100]}...")
|
|
print()
|
|
|
|
print("3. Interactive chat demo:")
|
|
print(" (Note: Requires OpenAI API key for real responses)")
|
|
|
|
chat = LeannChat("demo_knowledge.leann")
|
|
|
|
# Demo questions
|
|
demo_questions: list[str] = [
|
|
"What is the difference between machine learning and deep learning?",
|
|
"How is data science related to big data?",
|
|
]
|
|
|
|
for question in demo_questions:
|
|
print(f" Q: {question}")
|
|
response = chat.ask(question)
|
|
print(f" A: {response}")
|
|
print()
|
|
|
|
print("Demo completed! Try running:")
|
|
print(" uv run python apps/document_rag.py")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|