* Add ty type checker to CI and fix type errors
- Add ty (Astral's fast Python type checker) to GitHub CI workflow
- Fix type annotations across all RAG apps:
- Update load_data return types from list[str] to list[dict[str, Any]]
- Fix base_rag_example.py to properly handle dict format from create_text_chunks
- Fix type errors in leann-core:
- chunking_utils.py: Add explicit type annotations
- cli.py: Fix return type annotations for PDF extraction functions
- interactive_utils.py: Fix readline import type handling
- Fix type errors in apps:
- wechat_history.py: Fix return type annotations
- document_rag.py, code_rag.py: Replace **kwargs with explicit arguments
- Add ty configuration to pyproject.toml
This resolves the bug introduced in PR #157 where create_text_chunks()
changed to return list[dict] but callers were not updated.
* Fix remaining ty type errors
- Fix slack_mcp_reader.py channel parameter can be None
- Fix embedding_compute.py ContextProp type issue
- Fix searcher_base.py method override signatures
- Fix chunking_utils.py chunk_text assignment
- Fix slack_rag.py and twitter_rag.py return types
- Fix email.py and image_rag.py method overrides
* Fix multimodal benchmark scripts type errors
- Fix undefined LeannRetriever -> LeannMultiVector
- Add proper type casts for HuggingFace Dataset iteration
- Cast task config values to correct types
- Add type annotations for dataset row dicts
* Enable ty check for multimodal scripts in CI
All type errors in multimodal scripts have been fixed, so we can now
include them in the CI type checking.
* Fix all test type errors and enable ty check on tests
- Fix test_basic.py: search() takes str not list
- Fix test_cli_prompt_template.py: add type: ignore for Mock assignments
- Fix test_prompt_template_persistence.py: match BaseSearcher.search signature
- Fix test_prompt_template_e2e.py: add type narrowing asserts after skip
- Fix test_readme_examples.py: use explicit kwargs instead of **model_args
- Fix metadata_filter.py: allow Optional[MetadataFilters]
- Update CI to run ty check on tests
* Format code with ruff
* Format searcher_base.py
* Drop Python 3.9 support, require Python 3.10+
Python 3.9 reached end-of-life and the codebase uses PEP 604 union
type syntax (str | None) which requires Python 3.10+.
Changes:
- Remove Python 3.9 from CI build matrix
- Update requires-python to >=3.10 in all pyproject.toml files
- Update classifiers to reflect supported Python versions (3.10-3.13)
* Remove macos-13 from CI build matrix
GitHub Actions deprecated macos-13 runner (brownout started Sept 2025,
fully retired Dec 2025). See: https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/
* Add macos-15-large for Intel Mac builds
Replace deprecated macos-13 with macos-15-large (x86_64 Intel)
to continue supporting Intel Mac users.
* Set MACOSX_DEPLOYMENT_TARGET=13.x for Intel builds
Intel Mac wheels (macos-15-large) now target macOS 13.0/13.3 for
backward compatibility, allowing macOS 13/14/15 Intel users to
install pre-built wheels.
* Remove Intel Mac builds (macos-15-large requires paid plan)
Intel Mac users can build from source. This avoids:
- Paid GitHub Actions runners (macos-15-large)
- Complex cross-compilation setup
* Add macos-15-intel for Intel Mac builds (free runner)
Use macos-15-intel (free standard runner) instead of macos-15-large
(paid). This provides Intel Mac wheel support until Aug 2027.
- MACOSX_DEPLOYMENT_TARGET=13.0 for backward compatibility
- Replaces deprecated macos-13 runner
* Add macOS 26 (beta) to build matrix
Add macos-26 (arm64) runner to the build matrix for testing future
macOS compatibility. This is currently a beta runner that helps ensure
wheels work on upcoming macOS versions.
* Fix macos-15-intel deployment target
The macos-15-intel runner runs macOS 15.7, so Homebrew libraries are
built for macOS 14+. Setting MACOSX_DEPLOYMENT_TARGET=13.0 causes
delocate to fail because system libraries require newer macOS.
Fix by setting deployment target to 15.0 for macos-15-intel, matching
the actual OS version. Intel Mac users will need macOS 15+.
* Exclude macos-15-intel + Python 3.13 (no PyTorch wheels available)
PR #157 changed create_text_chunks() to return list[dict] instead of
list[str] to preserve metadata, but base_rag_example.py was not updated
to handle the new format. This caused all chunks to fail validation
with "All provided chunks are empty or invalid".
* Add custom folder support and improve image loading for multi-vector retrieval
- Enhanced _load_images_from_dir with recursive search support and better error handling
- Added support for WebP format and RGB conversion for all image modes
- Added custom folder CLI arguments (--custom-folder, --recursive, --rebuild-index)
- Improved documentation and removed completed TODO comment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Format code style in leann_multi_vector.py for better readability
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* docs: polish README performance tip section
- Fix typo: 'matrilize' -> 'materialize'
- Improve clarity and formatting of --no-recompute flag explanation
- Add code block for better readability
* format
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
* Add custom folder support and improve image loading for multi-vector retrieval
- Enhanced _load_images_from_dir with recursive search support and better error handling
- Added support for WebP format and RGB conversion for all image modes
- Added custom folder CLI arguments (--custom-folder, --recursive, --rebuild-index)
- Improved documentation and removed completed TODO comment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Format code style in leann_multi_vector.py for better readability
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Add brief introduction and usage guide for ColQwen integration,
similar to other RAG application sections in the README.
- Quick start examples for building, searching, and interactive Q&A
- Setup instructions with prerequisites
- Model options (ColQwen2 vs ColPali)
- Link to detailed ColQwen guide
- Add ColQwen2.5 and ColQwen2_5_Processor imports
- Implement smart model type detection for colqwen2, colqwen2.5, and colpali
- Add task name aliases for easier benchmark invocation
- Add safe model name handling for file paths and index naming
- Support custom model paths including LoRA adapters
- Improve model choice validation and error handling
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* Add timing instrumentation and multi-dataset support for multi-vector retrieval
- Add timing measurements for search operations (load and core time)
- Increase embedding batch size from 1 to 32 for better performance
- Add explicit memory cleanup with del all_embeddings
- Support loading and merging multiple datasets with different splits
- Add CLI arguments for search method selection (ann/exact/exact-all)
- Auto-detect image field names across different dataset structures
- Print candidate doc counts for performance monitoring
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* update vidore
* reproduce docvqa results
* reproduce docvqa results and add debug file
* fix: format colqwen_forward.py to pass pre-commit checks
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add timing instrumentation and multi-dataset support for multi-vector retrieval
- Add timing measurements for search operations (load and core time)
- Increase embedding batch size from 1 to 32 for better performance
- Add explicit memory cleanup with del all_embeddings
- Support loading and merging multiple datasets with different splits
- Add CLI arguments for search method selection (ann/exact/exact-all)
- Auto-detect image field names across different dataset structures
- Print candidate doc counts for performance monitoring
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* update vidore
* reproduce docvqa results
* reproduce docvqa results and add debug file
---------
Co-authored-by: Claude <noreply@anthropic.com>
Fixes#175
Problem:
When --file-types .pdf is specified, PDFs were being processed twice:
1. Separately with PyMuPDF/pdfplumber extractors
2. Again in the 'other file types' section via SimpleDirectoryReader
This caused duplicate processing and potential conflicts.
Solution:
- Exclude .pdf from other_file_extensions when PDFs are already
processed separately
- Only load other file types if there are extensions to process
- Prevents duplicate PDF processing
Changes:
- Added logic to filter out .pdf from code_extensions when loading
other file types if PDFs were processed separately
- Updated SimpleDirectoryReader to use filtered extensions
- Added check to skip loading if no other extensions to process
- Add apps/image_rag.py for indexing and searching images using CLIP embeddings
- Supports text-based image search queries
- Uses CLIP ViT-L/14 model via sentence-transformers
- Follows the same pattern as other RAG apps in the apps directory
- Addresses feature request for CLIP support in apps (issue #94)
* Fix prompt template bugs in build and search
Bug 1: Build template ignored in new format
- Updated compute_embeddings_openai() to read build_prompt_template or prompt_template
- Updated compute_embeddings_ollama() with same fix
- Maintains backward compatibility with old single-template format
Bug 2: Runtime override not wired up
- Wired CLI search to pass provider_options to searcher.search()
- Enables runtime template override during search via --embedding-prompt-template
All 42 prompt template tests passing.
Fixes#155
* Fix: Prevent embedding server from applying templates during search
- Filter out all prompt templates (build_prompt_template, query_prompt_template, prompt_template) from provider_options when launching embedding server during search
- Templates are already applied in compute_query_embedding() before server call
- Prevents double-templating and ensures runtime override works correctly
This fixes the issue where --embedding-prompt-template during search was ignored because the server was applying build_prompt_template instead.
* Format code with ruff
* Add prompt template support and LM Studio SDK integration
Features:
- Prompt template support for embedding models (via --embedding-prompt-template)
- LM Studio SDK integration for automatic context length detection
- Hybrid token limit discovery (Ollama → LM Studio → Registry → Default)
- Client-side token truncation to prevent silent failures
- Automatic persistence of embedding_options to .meta.json
Implementation:
- Added _query_lmstudio_context_limit() with Node.js subprocess bridge
- Modified compute_embeddings_openai() to apply prompt templates before truncation
- Extended CLI with --embedding-prompt-template flag for build and search
- URL detection for LM Studio (port 1234 or lmstudio/lm.studio keywords)
- HTTP→WebSocket URL conversion for SDK compatibility
Tests:
- 60 passing tests across 5 test files
- Comprehensive coverage of prompt templates, LM Studio integration, and token handling
- Parametrized tests for maintainability and clarity
* Add integration tests and fix LM Studio SDK bridge
Features:
- End-to-end integration tests for prompt template with EmbeddingGemma
- Integration tests for hybrid token limit discovery mechanism
- Tests verify real-world functionality with live services (LM Studio, Ollama)
Fixes:
- LM Studio SDK bridge now uses client.embedding.load() for embedding models
- Fixed NODE_PATH resolution to include npm global modules
- Fixed integration test to use WebSocket URL (ws://) for SDK bridge
Tests:
- test_prompt_template_e2e.py: 8 integration tests covering:
- Prompt template prepending with LM Studio (EmbeddingGemma)
- LM Studio SDK bridge for context length detection
- Ollama dynamic token limit detection
- Hybrid discovery fallback mechanism (registry, default)
- All tests marked with @pytest.mark.integration for selective execution
- Tests gracefully skip when services unavailable
Documentation:
- Updated tests/README.md with integration test section
- Added prerequisites and running instructions
- Documented that prompt templates are ONLY for EmbeddingGemma
- Added integration marker to pyproject.toml
Test Results:
- All 8 integration tests passing with live services
- Confirmed prompt templates work correctly with EmbeddingGemma
- Verified LM Studio SDK bridge auto-detects context length (2048)
- Validated hybrid token limit discovery across all backends
* Add prompt template support to Ollama mode
Extends prompt template functionality from OpenAI mode to Ollama for backend consistency.
Changes:
- Add provider_options parameter to compute_embeddings_ollama()
- Apply prompt template before token truncation (lines 1005-1011)
- Pass provider_options through compute_embeddings() call chain
Tests:
- test_ollama_embedding_with_prompt_template: Verifies templates work with Ollama
- test_ollama_prompt_template_affects_embeddings: Confirms embeddings differ with/without template
- Both tests pass with live Ollama service (2/2 passing)
Usage:
leann build --embedding-mode ollama --embedding-prompt-template "query: " ...
* Fix LM Studio SDK bridge to respect JIT auto-evict settings
Problem: SDK bridge called client.embedding.load() which loaded models into
LM Studio memory and bypassed JIT auto-evict settings, causing duplicate
model instances to accumulate.
Root cause analysis (from Perplexity research):
- Explicit SDK load() commands are treated as "pinned" models
- JIT auto-evict only applies to models loaded reactively via API requests
- SDK-loaded models remain in memory until explicitly unloaded
Solutions implemented:
1. Add model.unload() after metadata query (line 243)
- Load model temporarily to get context length
- Unload immediately to hand control back to JIT system
- Subsequent API requests trigger JIT load with auto-evict
2. Add token limit caching to prevent repeated SDK calls
- Cache discovered limits in _token_limit_cache dict (line 48)
- Key: (model_name, base_url), Value: token_limit
- Prevents duplicate load/unload cycles within same process
- Cache shared across all discovery methods (Ollama, SDK, registry)
Tests:
- TestTokenLimitCaching: 5 tests for cache behavior (integrated into test_token_truncation.py)
- Manual testing confirmed no duplicate models in LM Studio after fix
- All existing tests pass
Impact:
- Respects user's LM Studio JIT and auto-evict settings
- Reduces model memory footprint
- Faster subsequent builds (cached limits)
* Document prompt template and LM Studio SDK features
Added comprehensive documentation for new optional embedding features:
Configuration Guide (docs/configuration-guide.md):
- New section: "Optional Embedding Features"
- Task-Specific Prompt Templates subsection:
- Explains EmbeddingGemma use case with document/query prompts
- CLI and Python API examples
- Clear warnings about compatible vs incompatible models
- References to GitHub issue #155 and HuggingFace blog
- LM Studio Auto-Detection subsection:
- Prerequisites (Node.js + @lmstudio/sdk)
- How auto-detection works (4-step process)
- Benefits and optional nature clearly stated
FAQ (docs/faq.md):
- FAQ #2: When should I use prompt templates?
- DO/DON'T guidance with examples
- Links to detailed configuration guide
- FAQ #3: Why is LM Studio loading multiple copies?
- Explains the JIT auto-evict fix
- Troubleshooting steps if still seeing issues
- FAQ #4: Do I need Node.js and @lmstudio/sdk?
- Clarifies it's completely optional
- Lists benefits if installed
- Installation instructions
Cross-references between documents for easy navigation between quick reference and detailed guides.
* Add separate build/query template support for task-specific models
Task-specific models like EmbeddingGemma require different templates for indexing vs searching. Store both templates at build time and auto-apply query template during search with backward compatibility.
* Consolidate prompt template tests from 44 to 37 tests
Merged redundant no-op tests, removed low-value implementation tests, consolidated parameterized CLI tests, and removed hanging over-mocked test. All tests pass with improved focus on behavioral testing.
* Fix query template application in compute_query_embedding
Query templates were only applied in the fallback code path, not when using the embedding server (default path). This meant stored query templates in index metadata were ignored during MCP and CLI searches.
Changes:
- Move template application to before any computation path (searcher_base.py:109-110)
- Add comprehensive tests for both server and fallback paths
- Consolidate tests into test_prompt_template_persistence.py
Tests verify:
- Template applied when using embedding server
- Template applied in fallback path
- Consistent behavior between both paths
* Apply ruff formatting and fix linting issues
- Remove unused imports
- Fix import ordering
- Remove unused variables
- Apply code formatting
* Fix CI test failures: mock OPENAI_API_KEY in tests
Tests were failing in CI because compute_embeddings_openai() checks for OPENAI_API_KEY before using the mocked client. Added monkeypatch to set fake API key in test fixture.
- Fix import order (ast before asyncio)
- Remove NameError from exception handling (ast.literal_eval doesn't raise it)
- Add .lycheeignore to exclude intermittently unavailable star-history API
- Update link-check workflow to exclude star-history API and accept 503 status codes
- Add noqa comments for E402 errors (imports after sys.path modifications)
- Remove unused variable assignment in colqwen_rag.py
- Use importlib.util.find_spec for dependency checks instead of unused imports
- Fix import ordering in test_colqwen_reproduction.py
- Add ColQwenRAG class with easy-to-use CLI for multimodal PDF retrieval
- Support for both ColQwen2 and ColPali models with automatic device selection
- MPS optimization for Apple Silicon with memory-efficient loading
- Complete pipeline: PDF→images→embeddings→HNSW index→search
- Multi-vector indexing for fine-grained document matching
- Comprehensive user guide and reproduction test script
- Resolves#119: ColQwen Doc and Support Management
Features:
- python -m apps.colqwen_rag build --pdfs ./pdfs/ --index my_index
- python -m apps.colqwen_rag search my_index "query text"
- python -m apps.colqwen_rag ask my_index --interactive
- Automatic CPU fallback for memory constraints
- Robust error handling and progress tracking
* feat: enhance token limits with dynamic discovery + AST metadata
Improves upon upstream PR #154 with two major enhancements:
1. **Hybrid Token Limit Discovery**
- Dynamic: Query Ollama /api/show for context limits
- Fallback: Registry for LM Studio/OpenAI
- Zero maintenance for Ollama users
- Respects custom num_ctx settings
2. **AST Metadata Preservation**
- create_ast_chunks() returns dict format with metadata
- Preserves file_path, file_name, timestamps
- Includes astchunk metadata (line numbers, node counts)
- Fixes content extraction bug (checks "content" key)
- Enables --show-metadata flag
3. **Better Token Limits**
- nomic-embed-text: 2048 tokens (vs 512)
- nomic-embed-text-v1.5: 2048 tokens
- Added OpenAI models: 8192 tokens
4. **Comprehensive Tests**
- 11 tests for token truncation
- 545 new lines in test_astchunk_integration.py
- All metadata preservation tests passing
* fix: merge EMBEDDING_MODEL_LIMITS and remove redundant validation
- Merged upstream's model list with our corrected token limits
- Kept our corrected nomic-embed-text: 2048 (not 512)
- Removed post-chunking validation (redundant with embedding-time truncation)
- All tests passing except 2 pre-existing integration test failures
* style: apply ruff formatting and restore PR #154 version handling
- Remove duplicate truncate_to_token_limit and get_model_token_limit functions
- Restore version handling logic (model:latest -> model) from PR #154
- Restore partial matching fallback for model name variations
- Apply ruff formatting to all modified files
- All 11 token truncation tests passing
* style: sort imports alphabetically (pre-commit auto-fix)
* fix: show AST token limit warning only once per session
- Add module-level flag to track if warning shown
- Prevents spam when processing multiple files
- Add clarifying note that auto-truncation happens at embedding time
- Addresses issue where warning appeared for every code file
* enhance: add detailed logging for token truncation
- Track and report truncation statistics (count, tokens removed, max length)
- Show first 3 individual truncations with exact token counts
- Provide comprehensive summary when truncation occurs
- Use WARNING level for data loss visibility
- Silent (DEBUG level only) when no truncation needed
Replaces misleading "truncated where necessary" message that appeared
even when nothing was truncated.
* feat: add metadata output to search results
- Add --show-metadata flag to display file paths in search results
- Preserve document metadata (file_path, file_name, timestamps) during chunking
- Update MCP tool schema to support show_metadata parameter
- Enhance CLI search output to display metadata when requested
- Fix pre-existing bug: args.backend -> args.backend_name
Resolves yichuan-w/LEANN#144
* fix: resolve ZMQ linking issues in Python extension
- Use pkg_check_modules IMPORTED_TARGET to create PkgConfig::ZMQ
- Set PKG_CONFIG_PATH to prioritize ARM64 Homebrew on Apple Silicon
- Override macOS -undefined dynamic_lookup to force proper symbol resolution
- Use PUBLIC linkage for ZMQ in faiss library for transitive linking
- Mark cppzmq includes as SYSTEM to suppress warnings
Fixes editable install ZMQ symbol errors while maintaining compatibility
across Linux, macOS Intel, and macOS ARM64 platforms.
* style: apply ruff formatting
* chore: update faiss submodule to use ww2283 fork
Use ww2283/faiss fork with fix/zmq-linking branch to resolve CI checkout
failures. The ZMQ linking fixes are not yet merged upstream.
* feat: implement true batch processing for Ollama embeddings
Migrate from deprecated /api/embeddings to modern /api/embed endpoint
which supports batch inputs. This reduces HTTP overhead by sending
32 texts per request instead of making individual API calls.
Changes:
- Update endpoint from /api/embeddings to /api/embed
- Change parameter from 'prompt' (single) to 'input' (array)
- Update response parsing for batch embeddings array
- Increase timeout to 60s for batch processing
- Improve error handling for batch requests
Performance:
- Reduces API calls by 32x (batch size)
- Eliminates HTTP connection overhead per text
- Note: Ollama still processes batch items sequentially internally
Related: #151
* fall back to original faiss as i merge the PR
---------
Co-authored-by: yichuan520030910320 <yichuan_wang@berkeley.edu>
Remove paru-bin directory that was incorrectly added as a git submodule.
This directory is an AUR build artifact and should not be tracked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Printing querying time
* Adding source name to chunks
Adding source name as metadata to chunks, then printing the sources when searching
* Printing the context provided to LLM
To check the data transmitted to the LLMs : display the relevance, ID, content, and source of each sent chunk.
* Correcting source as metadata for chunks
* Applying ruff format
* Applying Ruff formatting
* Ruff formatting
* fix: Fix Twitter bookmarks anchor link
- Convert Twitter Bookmarks from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #twitter-bookmarks-your-personal-tweet-library work correctly
Fixes broken link: https://github.com/yichuan-w/LEANN?tab=readme-ov-file#twitter-bookmarks-your-personal-tweet-library
* fix: Fix Slack messages anchor link as well
- Convert Slack Messages from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #slack-messages-search-your-team-conversations work correctly
Both Twitter and Slack MCP sections now have reliable anchor links.
* fix: Point Slack and Twitter links to main MCP section
- Both Slack and Twitter are subsections under MCP Integration
- Links should point to #mcp-integration-rag-on-live-data-from-any-platform
- Users will land on the MCP section and can find both Slack and Twitter subsections there
This matches the actual document structure where Slack and Twitter are under the MCP Integration section.
* Improve Slack MCP integration with retry logic and comprehensive setup guide
- Add retry mechanism with exponential backoff for cache sync issues
- Handle 'users cache is not ready yet' errors gracefully
- Add max-retries and retry-delay CLI arguments for better control
- Create comprehensive Slack setup guide with troubleshooting
- Update README with link to detailed setup guide
- Improve error messages and user experience
* Fix trailing whitespace in slack setup guide
Pre-commit hooks formatting fixes
* Add comprehensive Slack setup guide with success screenshot
- Create detailed setup guide with step-by-step instructions
- Add troubleshooting section for common issues like cache sync errors
- Include real terminal output example from successful integration
- Add screenshot showing VS Code interface with Slack channel data
- Remove excessive emojis for more professional documentation
- Document retry logic improvements and CLI arguments
* Fix formatting issues in Slack setup guide
- Remove trailing whitespace
- Fix end of file formatting
- Pre-commit hooks formatting fixes
* Add real RAG example showing intelligent Slack query functionality
- Add detailed example of asking 'What is LEANN about?'
- Show retrieved messages from Slack channels
- Demonstrate intelligent answer generation based on context
- Add command example for running real RAG queries
- Explain the 4-step process: retrieve, index, generate, cite
* Update Slack setup guide with bot invitation requirements
- Add important section about inviting bot to channels before RAG queries
- Explain the 'not_in_channel' errors and their meaning
- Provide clear steps for bot invitation process
- Document realistic scenario where bot needs explicit channel access
- Update documentation to be more professional and less cursor-style
* Docs: add real RAG example for Sky Lab #random
- Embed screenshot videos/rag-sky-random.png
- Add step-by-step commands and notes
- Include helper test script tests/test_channel_by_id_or_name.py
- Redact example tokens from docs
* Docs/CI: fix broken image paths and ruff lint\n\n- Move screenshot to docs/videos and update references\n- Remove obsolete rag-query-results image\n- Rename variable to satisfy ruff
* Docs: fix image path for lychee (use videos/ relative under docs/)
* Docs: finalize Slack setup guide with Sky random RAG example and image path fixes\n\n- Redact example tokens from docs
* Fix Slack MCP integration and update documentation
- Fix SlackMCPReader to use conversations_history instead of channels_list
- Add fallback imports for leann.interactive_utils and leann.settings
- Update slack-setup-guide.md with real screenshots and improved text
- Remove old screenshot files
* Add Slack integration screenshots to docs/videos
- Add slack_integration.png showing RAG query results
- Add slack_integration_2.png showing additional demo functionality
- Fixes lychee link checker errors for missing image files
* Update Slack integration screenshot with latest changes
* Remove test_channel_by_id_or_name.py
- Clean up temporary test file that was used for debugging
- Keep only the main slack_rag.py application for production use
* Update Slack RAG example to show LEANN announcement retrieval
- Change query from 'PUBPOL 290' to 'What is LEANN about?' for more challenging retrieval
- Update command to use python -m apps.slack_rag instead of test script
- Add expected response showing Yichuan Wang's LEANN announcement message
- Emphasize this demonstrates ability to find specific announcements in conversation history
- Update description to highlight challenging query capabilities
* Update Slack RAG integration with improved CSV parsing and new screenshots
- Fixed CSV message parsing in slack_mcp_reader.py to properly handle individual messages
- Updated slack_rag.py to filter empty channel strings
- Enhanced slack-setup-guide.md with two new query examples:
- Advisor Models query: 'train black-box models to adopt to your personal data'
- Barbarians at the Gate query: 'AI-driven research systems ADRS'
- Replaced old screenshots with four new ones showing both query examples
- Updated documentation to use User OAuth Token (xoxp-) instead of Bot Token (xoxb-)
- Added proper command examples with --no-concatenate-conversations and --force-rebuild flags
* Update Slack RAG documentation with Ollama integration and new screenshots
- Updated slack-setup-guide.md with comprehensive Ollama setup instructions
- Added 6 new screenshots showing complete RAG workflow:
- Command setup, search results, and LLM responses for both queries
- Removed simulated LLM references, now uses real Ollama with llama3.2:1b
- Enhanced documentation with step-by-step Ollama installation
- Updated troubleshooting checklist to include Ollama-specific checks
- Fixed command syntax and added proper Ollama configuration
- Demonstrates working Slack RAG with real AI-generated responses
* Remove Key Features section from Slack RAG examples
- Simplified documentation by removing the bullet point list
- Keeps the focus on the actual examples and screenshots
* fix: Fix Twitter bookmarks anchor link
- Convert Twitter Bookmarks from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #twitter-bookmarks-your-personal-tweet-library work correctly
Fixes broken link: https://github.com/yichuan-w/LEANN?tab=readme-ov-file#twitter-bookmarks-your-personal-tweet-library
* fix: Fix Slack messages anchor link as well
- Convert Slack Messages from collapsible details to proper header
- Update internal link to match new anchor format
- Ensures external links to #slack-messages-search-your-team-conversations work correctly
Both Twitter and Slack MCP sections now have reliable anchor links.
* fix: Point Slack and Twitter links to main MCP section
- Both Slack and Twitter are subsections under MCP Integration
- Links should point to #mcp-integration-rag-on-live-data-from-any-platform
- Users will land on the MCP section and can find both Slack and Twitter subsections there
This matches the actual document structure where Slack and Twitter are under the MCP Integration section.
BREAKING CHANGE: trust_remote_code now defaults to False for security
- Set trust_remote_code=False by default in HFChat class
- Add explicit trust_remote_code parameter to HFChat.__init__()
- Add security warning when trust_remote_code=True is used
- Update get_llm() function to support trust_remote_code parameter
- Update benchmark utilities (load_hf_model, load_vllm_model, load_qwen_vl_model)
- Add comprehensive documentation for security implications
Security Benefits:
- Prevents arbitrary code execution from compromised model repositories
- Requires explicit opt-in for models that need remote code execution
- Shows clear warnings when security is reduced
- Follows security-by-default principle
Migration Guide:
- Most users: No changes needed (more secure by default)
- Users with models requiring remote code: Add trust_remote_code=True explicitly
- Config users: Add 'trust_remote_code': true to LLM config if needed
Fixes#136
