Compare commits
8 Commits
fix/52-inc
...
feature/en
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d2299d2679 | ||
|
|
6bde28584b | ||
|
|
f62632c41f | ||
|
|
27708243ca | ||
|
|
9a1e4652ca | ||
|
|
14e84d9e2d | ||
|
|
2dcfca19ff | ||
|
|
bee2167ee3 |
44
README.md
44
README.md
@@ -5,7 +5,7 @@
|
|||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="https://img.shields.io/badge/Python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue.svg" alt="Python Versions">
|
<img src="https://img.shields.io/badge/Python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue.svg" alt="Python Versions">
|
||||||
<img src="https://github.com/yichuan-w/LEANN/actions/workflows/build-and-publish.yml/badge.svg" alt="CI Status">
|
<img src="https://github.com/yichuan-w/LEANN/actions/workflows/build-and-publish.yml/badge.svg" alt="CI Status">
|
||||||
<img src="https://img.shields.io/badge/Platform-Ubuntu%20%7C%20macOS%20(ARM64%2FIntel)-lightgrey" alt="Platform">
|
<img src="https://img.shields.io/badge/Platform-Ubuntu%20%26%20Arch%20%26%20WSL%20%7C%20macOS%20(ARM64%2FIntel)-lightgrey" alt="Platform">
|
||||||
<img src="https://img.shields.io/badge/License-MIT-green.svg" alt="MIT License">
|
<img src="https://img.shields.io/badge/License-MIT-green.svg" alt="MIT License">
|
||||||
<img src="https://img.shields.io/badge/MCP-Native%20Integration-blue" alt="MCP Integration">
|
<img src="https://img.shields.io/badge/MCP-Native%20Integration-blue" alt="MCP Integration">
|
||||||
</p>
|
</p>
|
||||||
@@ -94,7 +94,9 @@ CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ uv sync
|
|||||||
|
|
||||||
**Linux:**
|
**Linux:**
|
||||||
```bash
|
```bash
|
||||||
sudo apt-get install libomp-dev libboost-all-dev protobuf-compiler libabsl-dev libmkl-full-dev libaio-dev libzmq3-dev
|
# Ubuntu/Debian (For Arch Linux: sudo pacman -S blas lapack openblas libaio boost protobuf abseil-cpp zeromq)
|
||||||
|
sudo apt-get update && sudo apt-get install -y libomp-dev libboost-all-dev protobuf-compiler libabsl-dev libmkl-full-dev libaio-dev libzmq3-dev
|
||||||
|
|
||||||
uv sync
|
uv sync
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -426,21 +428,21 @@ Once the index is built, you can ask questions like:
|
|||||||
**The future of code assistance is here.** Transform your development workflow with LEANN's native MCP integration for Claude Code. Index your entire codebase and get intelligent code assistance directly in your IDE.
|
**The future of code assistance is here.** Transform your development workflow with LEANN's native MCP integration for Claude Code. Index your entire codebase and get intelligent code assistance directly in your IDE.
|
||||||
|
|
||||||
**Key features:**
|
**Key features:**
|
||||||
- 🔍 **Semantic code search** across your entire project
|
- 🔍 **Semantic code search** across your entire project, fully local index and lightweight
|
||||||
- 📚 **Context-aware assistance** for debugging and development
|
- 📚 **Context-aware assistance** for debugging and development
|
||||||
- 🚀 **Zero-config setup** with automatic language detection
|
- 🚀 **Zero-config setup** with automatic language detection
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Install LEANN globally for MCP integration
|
# Install LEANN globally for MCP integration
|
||||||
uv tool install leann-core
|
uv tool install leann-core --with leann
|
||||||
|
claude mcp add --scope user leann-server -- leann_mcp
|
||||||
# Setup is automatic - just start using Claude Code!
|
# Setup is automatic - just start using Claude Code!
|
||||||
```
|
```
|
||||||
Try our fully agentic pipeline with auto query rewriting, semantic search planning, and more:
|
Try our fully agentic pipeline with auto query rewriting, semantic search planning, and more:
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
**Ready to supercharge your coding?** [Complete Setup Guide →](packages/leann-mcp/README.md)
|
**🔥 Ready to supercharge your coding?** [Complete Setup Guide →](packages/leann-mcp/README.md)
|
||||||
|
|
||||||
## 🖥️ Command Line Interface
|
## 🖥️ Command Line Interface
|
||||||
|
|
||||||
@@ -457,7 +459,8 @@ leann --help
|
|||||||
**To make it globally available:**
|
**To make it globally available:**
|
||||||
```bash
|
```bash
|
||||||
# Install the LEANN CLI globally using uv tool
|
# Install the LEANN CLI globally using uv tool
|
||||||
uv tool install leann-core
|
uv tool install leann-core --with leann
|
||||||
|
|
||||||
|
|
||||||
# Now you can use leann from anywhere without activating venv
|
# Now you can use leann from anywhere without activating venv
|
||||||
leann --help
|
leann --help
|
||||||
@@ -481,6 +484,9 @@ leann ask my-docs --interactive
|
|||||||
|
|
||||||
# List all your indexes
|
# List all your indexes
|
||||||
leann list
|
leann list
|
||||||
|
|
||||||
|
# Remove an index
|
||||||
|
leann remove my-docs
|
||||||
```
|
```
|
||||||
|
|
||||||
**Key CLI features:**
|
**Key CLI features:**
|
||||||
@@ -493,7 +499,7 @@ leann list
|
|||||||
<details>
|
<details>
|
||||||
<summary><strong>📋 Click to expand: Complete CLI Reference</strong></summary>
|
<summary><strong>📋 Click to expand: Complete CLI Reference</strong></summary>
|
||||||
|
|
||||||
You can use `leann --help`, or `leann build --help`, `leann search --help`, `leann ask --help` to get the complete CLI reference.
|
You can use `leann --help`, or `leann build --help`, `leann search --help`, `leann ask --help`, `leann list --help`, `leann remove --help` to get the complete CLI reference.
|
||||||
|
|
||||||
**Build Command:**
|
**Build Command:**
|
||||||
```bash
|
```bash
|
||||||
@@ -531,6 +537,28 @@ Options:
|
|||||||
--top-k N Retrieval count (default: 20)
|
--top-k N Retrieval count (default: 20)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**List Command:**
|
||||||
|
```bash
|
||||||
|
leann list
|
||||||
|
|
||||||
|
# Lists all indexes across all projects with status indicators:
|
||||||
|
# ✓ - Index is complete and ready to use
|
||||||
|
# ✗ - Index is incomplete or corrupted
|
||||||
|
```
|
||||||
|
|
||||||
|
**Remove Command:**
|
||||||
|
```bash
|
||||||
|
leann remove INDEX_NAME [OPTIONS]
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--force, -f Force removal without confirmation
|
||||||
|
|
||||||
|
# Smart removal: automatically finds and safely removes indexes
|
||||||
|
# - Shows all matching indexes across projects
|
||||||
|
# - Requires confirmation for cross-project removal
|
||||||
|
# - Interactive selection when multiple matches found
|
||||||
|
```
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
## 🏗️ Architecture & How It Works
|
## 🏗️ Architecture & How It Works
|
||||||
|
|||||||
@@ -4,8 +4,8 @@ build-backend = "scikit_build_core.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-backend-diskann"
|
name = "leann-backend-diskann"
|
||||||
version = "0.2.9"
|
version = "0.3.0"
|
||||||
dependencies = ["leann-core==0.2.9", "numpy", "protobuf>=3.19.0"]
|
dependencies = ["leann-core==0.3.0", "numpy", "protobuf>=3.19.0"]
|
||||||
|
|
||||||
[tool.scikit-build]
|
[tool.scikit-build]
|
||||||
# Key: simplified CMake path
|
# Key: simplified CMake path
|
||||||
|
|||||||
@@ -6,10 +6,10 @@ build-backend = "scikit_build_core.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-backend-hnsw"
|
name = "leann-backend-hnsw"
|
||||||
version = "0.2.9"
|
version = "0.3.0"
|
||||||
description = "Custom-built HNSW (Faiss) backend for the Leann toolkit."
|
description = "Custom-built HNSW (Faiss) backend for the Leann toolkit."
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"leann-core==0.2.9",
|
"leann-core==0.3.0",
|
||||||
"numpy",
|
"numpy",
|
||||||
"pyzmq>=23.0.0",
|
"pyzmq>=23.0.0",
|
||||||
"msgpack>=1.0.0",
|
"msgpack>=1.0.0",
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-core"
|
name = "leann-core"
|
||||||
version = "0.2.9"
|
version = "0.3.0"
|
||||||
description = "Core API and plugin system for LEANN"
|
description = "Core API and plugin system for LEANN"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.9"
|
requires-python = ">=3.9"
|
||||||
|
|||||||
@@ -46,6 +46,7 @@ def compute_embeddings(
|
|||||||
- "sentence-transformers": Use sentence-transformers library (default)
|
- "sentence-transformers": Use sentence-transformers library (default)
|
||||||
- "mlx": Use MLX backend for Apple Silicon
|
- "mlx": Use MLX backend for Apple Silicon
|
||||||
- "openai": Use OpenAI embedding API
|
- "openai": Use OpenAI embedding API
|
||||||
|
- "gemini": Use Google Gemini embedding API
|
||||||
use_server: Whether to use embedding server (True for search, False for build)
|
use_server: Whether to use embedding server (True for search, False for build)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
@@ -306,6 +307,23 @@ class LeannBuilder:
|
|||||||
def build_index(self, index_path: str):
|
def build_index(self, index_path: str):
|
||||||
if not self.chunks:
|
if not self.chunks:
|
||||||
raise ValueError("No chunks added.")
|
raise ValueError("No chunks added.")
|
||||||
|
|
||||||
|
# Filter out invalid/empty text chunks early to keep passage and embedding counts aligned
|
||||||
|
valid_chunks: list[dict[str, Any]] = []
|
||||||
|
skipped = 0
|
||||||
|
for chunk in self.chunks:
|
||||||
|
text = chunk.get("text", "")
|
||||||
|
if isinstance(text, str) and text.strip():
|
||||||
|
valid_chunks.append(chunk)
|
||||||
|
else:
|
||||||
|
skipped += 1
|
||||||
|
if skipped > 0:
|
||||||
|
print(
|
||||||
|
f"Warning: Skipping {skipped} empty/invalid text chunk(s). Processing {len(valid_chunks)} valid chunks"
|
||||||
|
)
|
||||||
|
self.chunks = valid_chunks
|
||||||
|
if not self.chunks:
|
||||||
|
raise ValueError("All provided chunks are empty or invalid. Nothing to index.")
|
||||||
if self.dimensions is None:
|
if self.dimensions is None:
|
||||||
self.dimensions = len(
|
self.dimensions = len(
|
||||||
compute_embeddings(
|
compute_embeddings(
|
||||||
|
|||||||
@@ -680,6 +680,52 @@ class HFChat(LLMInterface):
|
|||||||
return response.strip()
|
return response.strip()
|
||||||
|
|
||||||
|
|
||||||
|
class GeminiChat(LLMInterface):
|
||||||
|
"""LLM interface for Google Gemini models."""
|
||||||
|
|
||||||
|
def __init__(self, model: str = "gemini-2.5-flash", api_key: Optional[str] = None):
|
||||||
|
self.model = model
|
||||||
|
self.api_key = api_key or os.getenv("GEMINI_API_KEY")
|
||||||
|
|
||||||
|
if not self.api_key:
|
||||||
|
raise ValueError(
|
||||||
|
"Gemini API key is required. Set GEMINI_API_KEY environment variable or pass api_key parameter."
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Initializing Gemini Chat with model='{model}'")
|
||||||
|
|
||||||
|
try:
|
||||||
|
import google.genai as genai
|
||||||
|
|
||||||
|
self.client = genai.Client(api_key=self.api_key)
|
||||||
|
except ImportError:
|
||||||
|
raise ImportError(
|
||||||
|
"The 'google-genai' library is required for Gemini models. Please install it with 'uv pip install google-genai'."
|
||||||
|
)
|
||||||
|
|
||||||
|
def ask(self, prompt: str, **kwargs) -> str:
|
||||||
|
logger.info(f"Sending request to Gemini with model {self.model}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Set generation configuration
|
||||||
|
generation_config = {
|
||||||
|
"temperature": kwargs.get("temperature", 0.7),
|
||||||
|
"max_output_tokens": kwargs.get("max_tokens", 1000),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Handle top_p parameter
|
||||||
|
if "top_p" in kwargs:
|
||||||
|
generation_config["top_p"] = kwargs["top_p"]
|
||||||
|
|
||||||
|
response = self.client.models.generate_content(
|
||||||
|
model=self.model, contents=prompt, config=generation_config
|
||||||
|
)
|
||||||
|
return response.text.strip()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error communicating with Gemini: {e}")
|
||||||
|
return f"Error: Could not get a response from Gemini. Details: {e}"
|
||||||
|
|
||||||
|
|
||||||
class OpenAIChat(LLMInterface):
|
class OpenAIChat(LLMInterface):
|
||||||
"""LLM interface for OpenAI models."""
|
"""LLM interface for OpenAI models."""
|
||||||
|
|
||||||
@@ -793,6 +839,8 @@ def get_llm(llm_config: Optional[dict[str, Any]] = None) -> LLMInterface:
|
|||||||
return HFChat(model_name=model or "deepseek-ai/deepseek-llm-7b-chat")
|
return HFChat(model_name=model or "deepseek-ai/deepseek-llm-7b-chat")
|
||||||
elif llm_type == "openai":
|
elif llm_type == "openai":
|
||||||
return OpenAIChat(model=model or "gpt-4o", api_key=llm_config.get("api_key"))
|
return OpenAIChat(model=model or "gpt-4o", api_key=llm_config.get("api_key"))
|
||||||
|
elif llm_type == "gemini":
|
||||||
|
return GeminiChat(model=model or "gemini-2.5-flash", api_key=llm_config.get("api_key"))
|
||||||
elif llm_type == "simulated":
|
elif llm_type == "simulated":
|
||||||
return SimulatedChat()
|
return SimulatedChat()
|
||||||
else:
|
else:
|
||||||
|
|||||||
@@ -84,6 +84,7 @@ Examples:
|
|||||||
leann search my-docs "query" # Search in my-docs index
|
leann search my-docs "query" # Search in my-docs index
|
||||||
leann ask my-docs "question" # Ask my-docs index
|
leann ask my-docs "question" # Ask my-docs index
|
||||||
leann list # List all stored indexes
|
leann list # List all stored indexes
|
||||||
|
leann remove my-docs # Remove an index (local first, then global)
|
||||||
""",
|
""",
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -148,6 +149,12 @@ Examples:
|
|||||||
type=str,
|
type=str,
|
||||||
help="Comma-separated list of file extensions to include (e.g., '.txt,.pdf,.pptx'). If not specified, uses default supported types.",
|
help="Comma-separated list of file extensions to include (e.g., '.txt,.pdf,.pptx'). If not specified, uses default supported types.",
|
||||||
)
|
)
|
||||||
|
build_parser.add_argument(
|
||||||
|
"--include-hidden",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=False,
|
||||||
|
help="Include hidden files and directories (paths starting with '.') during indexing (default: false)",
|
||||||
|
)
|
||||||
build_parser.add_argument(
|
build_parser.add_argument(
|
||||||
"--doc-chunk-size",
|
"--doc-chunk-size",
|
||||||
type=int,
|
type=int,
|
||||||
@@ -245,6 +252,13 @@ Examples:
|
|||||||
# List command
|
# List command
|
||||||
subparsers.add_parser("list", help="List all indexes")
|
subparsers.add_parser("list", help="List all indexes")
|
||||||
|
|
||||||
|
# Remove command
|
||||||
|
remove_parser = subparsers.add_parser("remove", help="Remove an index")
|
||||||
|
remove_parser.add_argument("index_name", help="Index name to remove")
|
||||||
|
remove_parser.add_argument(
|
||||||
|
"--force", "-f", action="store_true", help="Force removal without confirmation"
|
||||||
|
)
|
||||||
|
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
def register_project_dir(self):
|
def register_project_dir(self):
|
||||||
@@ -333,8 +347,6 @@ Examples:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
def list_indexes(self):
|
def list_indexes(self):
|
||||||
print("Stored LEANN indexes:")
|
|
||||||
|
|
||||||
# Get all project directories with .leann
|
# Get all project directories with .leann
|
||||||
global_registry = Path.home() / ".leann" / "projects.json"
|
global_registry = Path.home() / ".leann" / "projects.json"
|
||||||
all_projects = []
|
all_projects = []
|
||||||
@@ -360,58 +372,293 @@ Examples:
|
|||||||
if (current_path / ".leann" / "indexes").exists() and current_path not in valid_projects:
|
if (current_path / ".leann" / "indexes").exists() and current_path not in valid_projects:
|
||||||
valid_projects.append(current_path)
|
valid_projects.append(current_path)
|
||||||
|
|
||||||
if not valid_projects:
|
# Separate current and other projects
|
||||||
print(
|
current_project = None
|
||||||
"No indexes found. Use 'leann build <name> --docs <dir> [<dir2> ...]' to create one."
|
other_projects = []
|
||||||
)
|
|
||||||
return
|
|
||||||
|
|
||||||
total_indexes = 0
|
|
||||||
current_dir = Path.cwd()
|
|
||||||
|
|
||||||
for project_path in valid_projects:
|
for project_path in valid_projects:
|
||||||
indexes_dir = project_path / ".leann" / "indexes"
|
if project_path == current_path:
|
||||||
if not indexes_dir.exists():
|
current_project = project_path
|
||||||
continue
|
|
||||||
|
|
||||||
index_dirs = [d for d in indexes_dir.iterdir() if d.is_dir()]
|
|
||||||
if not index_dirs:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Show project header
|
|
||||||
if project_path == current_dir:
|
|
||||||
print(f"\n📁 Current project ({project_path}):")
|
|
||||||
else:
|
else:
|
||||||
print(f"\n📂 {project_path}:")
|
other_projects.append(project_path)
|
||||||
|
|
||||||
for index_dir in index_dirs:
|
print("📚 LEANN Indexes")
|
||||||
total_indexes += 1
|
print("=" * 50)
|
||||||
index_name = index_dir.name
|
|
||||||
meta_file = index_dir / "documents.leann.meta.json"
|
|
||||||
status = "✓" if meta_file.exists() else "✗"
|
|
||||||
|
|
||||||
print(f" {total_indexes}. {index_name} [{status}]")
|
total_indexes = 0
|
||||||
if status == "✓":
|
current_indexes_count = 0
|
||||||
size_mb = sum(f.stat().st_size for f in index_dir.iterdir() if f.is_file()) / (
|
|
||||||
1024 * 1024
|
|
||||||
)
|
|
||||||
print(f" Size: {size_mb:.1f} MB")
|
|
||||||
|
|
||||||
if total_indexes > 0:
|
# Show current project first (most important)
|
||||||
print(f"\nTotal: {total_indexes} indexes across {len(valid_projects)} projects")
|
if current_project:
|
||||||
print("\nUsage (current project only):")
|
current_indexes_dir = current_project / ".leann" / "indexes"
|
||||||
|
|
||||||
# Show example from current project
|
|
||||||
current_indexes_dir = current_dir / ".leann" / "indexes"
|
|
||||||
if current_indexes_dir.exists():
|
if current_indexes_dir.exists():
|
||||||
current_index_dirs = [d for d in current_indexes_dir.iterdir() if d.is_dir()]
|
current_index_dirs = [d for d in current_indexes_dir.iterdir() if d.is_dir()]
|
||||||
|
|
||||||
|
print("\n🏠 Current Project")
|
||||||
|
print(f" {current_project}")
|
||||||
|
print(" " + "─" * 45)
|
||||||
|
|
||||||
if current_index_dirs:
|
if current_index_dirs:
|
||||||
example_name = current_index_dirs[0].name
|
for index_dir in current_index_dirs:
|
||||||
print(f' leann search {example_name} "your query"')
|
total_indexes += 1
|
||||||
print(f" leann ask {example_name} --interactive")
|
current_indexes_count += 1
|
||||||
|
index_name = index_dir.name
|
||||||
|
meta_file = index_dir / "documents.leann.meta.json"
|
||||||
|
status = "✅" if meta_file.exists() else "❌"
|
||||||
|
|
||||||
|
print(f" {current_indexes_count}. {index_name} {status}")
|
||||||
|
if meta_file.exists():
|
||||||
|
size_mb = sum(
|
||||||
|
f.stat().st_size for f in index_dir.iterdir() if f.is_file()
|
||||||
|
) / (1024 * 1024)
|
||||||
|
print(f" 📦 Size: {size_mb:.1f} MB")
|
||||||
|
else:
|
||||||
|
print(" 📭 No indexes in current project")
|
||||||
|
else:
|
||||||
|
print("\n🏠 Current Project")
|
||||||
|
print(f" {current_path}")
|
||||||
|
print(" " + "─" * 45)
|
||||||
|
print(" 📭 No indexes in current project")
|
||||||
|
|
||||||
|
# Show other projects (reference information)
|
||||||
|
if other_projects:
|
||||||
|
print("\n\n🗂️ Other Projects")
|
||||||
|
print(" " + "─" * 45)
|
||||||
|
|
||||||
|
for project_path in other_projects:
|
||||||
|
indexes_dir = project_path / ".leann" / "indexes"
|
||||||
|
if not indexes_dir.exists():
|
||||||
|
continue
|
||||||
|
|
||||||
|
index_dirs = [d for d in indexes_dir.iterdir() if d.is_dir()]
|
||||||
|
if not index_dirs:
|
||||||
|
continue
|
||||||
|
|
||||||
|
print(f"\n 📂 {project_path.name}")
|
||||||
|
print(f" {project_path}")
|
||||||
|
|
||||||
|
for index_dir in index_dirs:
|
||||||
|
total_indexes += 1
|
||||||
|
index_name = index_dir.name
|
||||||
|
meta_file = index_dir / "documents.leann.meta.json"
|
||||||
|
status = "✅" if meta_file.exists() else "❌"
|
||||||
|
|
||||||
|
print(f" • {index_name} {status}")
|
||||||
|
if meta_file.exists():
|
||||||
|
size_mb = sum(
|
||||||
|
f.stat().st_size for f in index_dir.iterdir() if f.is_file()
|
||||||
|
) / (1024 * 1024)
|
||||||
|
print(f" 📦 {size_mb:.1f} MB")
|
||||||
|
|
||||||
|
# Summary and usage info
|
||||||
|
print("\n" + "=" * 50)
|
||||||
|
if total_indexes == 0:
|
||||||
|
print("💡 Get started:")
|
||||||
|
print(" leann build my-docs --docs ./documents")
|
||||||
|
else:
|
||||||
|
projects_count = len(
|
||||||
|
[
|
||||||
|
p
|
||||||
|
for p in valid_projects
|
||||||
|
if (p / ".leann" / "indexes").exists()
|
||||||
|
and list((p / ".leann" / "indexes").iterdir())
|
||||||
|
]
|
||||||
|
)
|
||||||
|
print(f"📊 Total: {total_indexes} indexes across {projects_count} projects")
|
||||||
|
|
||||||
|
if current_indexes_count > 0:
|
||||||
|
print("\n💫 Quick start (current project):")
|
||||||
|
# Get first index from current project for example
|
||||||
|
current_indexes_dir = current_path / ".leann" / "indexes"
|
||||||
|
if current_indexes_dir.exists():
|
||||||
|
current_index_dirs = [d for d in current_indexes_dir.iterdir() if d.is_dir()]
|
||||||
|
if current_index_dirs:
|
||||||
|
example_name = current_index_dirs[0].name
|
||||||
|
print(f' leann search {example_name} "your query"')
|
||||||
|
print(f" leann ask {example_name} --interactive")
|
||||||
|
else:
|
||||||
|
print("\n💡 Create your first index:")
|
||||||
|
print(" leann build my-docs --docs ./documents")
|
||||||
|
|
||||||
|
def remove_index(self, index_name: str, force: bool = False):
|
||||||
|
"""Safely remove an index - always show all matches for transparency"""
|
||||||
|
|
||||||
|
# Always do a comprehensive search for safety
|
||||||
|
print(f"🔍 Searching for all indexes named '{index_name}'...")
|
||||||
|
all_matches = self._find_all_matching_indexes(index_name)
|
||||||
|
|
||||||
|
if not all_matches:
|
||||||
|
print(f"❌ Index '{index_name}' not found in any project.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
if len(all_matches) == 1:
|
||||||
|
return self._remove_single_match(all_matches[0], index_name, force)
|
||||||
|
else:
|
||||||
|
return self._remove_from_multiple_matches(all_matches, index_name, force)
|
||||||
|
|
||||||
|
def _find_all_matching_indexes(self, index_name: str):
|
||||||
|
"""Find all indexes with the given name across all projects"""
|
||||||
|
matches = []
|
||||||
|
|
||||||
|
# Get all registered projects
|
||||||
|
global_registry = Path.home() / ".leann" / "projects.json"
|
||||||
|
all_projects = []
|
||||||
|
|
||||||
|
if global_registry.exists():
|
||||||
|
try:
|
||||||
|
import json
|
||||||
|
|
||||||
|
with open(global_registry) as f:
|
||||||
|
all_projects = json.load(f)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Always include current project
|
||||||
|
current_path = Path.cwd()
|
||||||
|
if str(current_path) not in all_projects:
|
||||||
|
all_projects.append(str(current_path))
|
||||||
|
|
||||||
|
# Search across all projects
|
||||||
|
for project_dir in all_projects:
|
||||||
|
project_path = Path(project_dir)
|
||||||
|
if not project_path.exists():
|
||||||
|
continue
|
||||||
|
|
||||||
|
index_dir = project_path / ".leann" / "indexes" / index_name
|
||||||
|
if index_dir.exists():
|
||||||
|
is_current = project_path == current_path
|
||||||
|
matches.append(
|
||||||
|
{"project_path": project_path, "index_dir": index_dir, "is_current": is_current}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sort: current project first, then by project name
|
||||||
|
matches.sort(key=lambda x: (not x["is_current"], x["project_path"].name))
|
||||||
|
return matches
|
||||||
|
|
||||||
|
def _remove_single_match(self, match, index_name: str, force: bool):
|
||||||
|
"""Handle removal when only one match is found"""
|
||||||
|
project_path = match["project_path"]
|
||||||
|
index_dir = match["index_dir"]
|
||||||
|
is_current = match["is_current"]
|
||||||
|
|
||||||
|
if is_current:
|
||||||
|
location_info = "current project"
|
||||||
|
emoji = "🏠"
|
||||||
|
else:
|
||||||
|
location_info = f"other project '{project_path.name}'"
|
||||||
|
emoji = "📂"
|
||||||
|
|
||||||
|
print(f"✅ Found 1 index named '{index_name}':")
|
||||||
|
print(f" {emoji} Location: {location_info}")
|
||||||
|
print(f" 📍 Path: {project_path}")
|
||||||
|
|
||||||
|
if not force:
|
||||||
|
if not is_current:
|
||||||
|
print("\n⚠️ CROSS-PROJECT REMOVAL!")
|
||||||
|
print(" This will delete the index from another project.")
|
||||||
|
|
||||||
|
response = input(f" ❓ Confirm removal from {location_info}? (y/N): ").strip().lower()
|
||||||
|
if response not in ["y", "yes"]:
|
||||||
|
print(" ❌ Removal cancelled.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
return self._delete_index_directory(
|
||||||
|
index_dir, index_name, project_path if not is_current else None
|
||||||
|
)
|
||||||
|
|
||||||
|
def _remove_from_multiple_matches(self, matches, index_name: str, force: bool):
|
||||||
|
"""Handle removal when multiple matches are found"""
|
||||||
|
|
||||||
|
print(f"⚠️ Found {len(matches)} indexes named '{index_name}':")
|
||||||
|
print(" " + "─" * 50)
|
||||||
|
|
||||||
|
for i, match in enumerate(matches, 1):
|
||||||
|
project_path = match["project_path"]
|
||||||
|
is_current = match["is_current"]
|
||||||
|
|
||||||
|
if is_current:
|
||||||
|
print(f" {i}. 🏠 Current project")
|
||||||
|
print(f" 📍 {project_path}")
|
||||||
|
else:
|
||||||
|
print(f" {i}. 📂 {project_path.name}")
|
||||||
|
print(f" 📍 {project_path}")
|
||||||
|
|
||||||
|
# Show size info
|
||||||
|
try:
|
||||||
|
size_mb = sum(
|
||||||
|
f.stat().st_size for f in match["index_dir"].iterdir() if f.is_file()
|
||||||
|
) / (1024 * 1024)
|
||||||
|
print(f" 📦 Size: {size_mb:.1f} MB")
|
||||||
|
except (OSError, PermissionError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
print(" " + "─" * 50)
|
||||||
|
|
||||||
|
if force:
|
||||||
|
print(" ❌ Multiple matches found, but --force specified.")
|
||||||
|
print(" Please run without --force to choose which one to remove.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
choice = input(
|
||||||
|
f" ❓ Which one to remove? (1-{len(matches)}, or 'c' to cancel): "
|
||||||
|
).strip()
|
||||||
|
if choice.lower() == "c":
|
||||||
|
print(" ❌ Removal cancelled.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
choice_idx = int(choice) - 1
|
||||||
|
if 0 <= choice_idx < len(matches):
|
||||||
|
selected_match = matches[choice_idx]
|
||||||
|
project_path = selected_match["project_path"]
|
||||||
|
index_dir = selected_match["index_dir"]
|
||||||
|
is_current = selected_match["is_current"]
|
||||||
|
|
||||||
|
location = "current project" if is_current else f"'{project_path.name}' project"
|
||||||
|
print(f" 🎯 Selected: Remove from {location}")
|
||||||
|
|
||||||
|
# Final confirmation for safety
|
||||||
|
confirm = input(
|
||||||
|
f" ❓ FINAL CONFIRMATION - Type '{index_name}' to proceed: "
|
||||||
|
).strip()
|
||||||
|
if confirm != index_name:
|
||||||
|
print(" ❌ Confirmation failed. Removal cancelled.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
return self._delete_index_directory(
|
||||||
|
index_dir, index_name, project_path if not is_current else None
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(" ❌ Invalid choice. Removal cancelled.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except (ValueError, KeyboardInterrupt):
|
||||||
|
print("\n ❌ Invalid input. Removal cancelled.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _delete_index_directory(
|
||||||
|
self, index_dir: Path, index_name: str, project_path: Path | None = None
|
||||||
|
):
|
||||||
|
"""Actually delete the index directory"""
|
||||||
|
try:
|
||||||
|
import shutil
|
||||||
|
|
||||||
|
shutil.rmtree(index_dir)
|
||||||
|
|
||||||
|
if project_path:
|
||||||
|
print(f"✅ Index '{index_name}' removed from {project_path.name}")
|
||||||
|
else:
|
||||||
|
print(f"✅ Index '{index_name}' removed successfully")
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error removing index '{index_name}': {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
def load_documents(
|
def load_documents(
|
||||||
self, docs_paths: Union[str, list], custom_file_types: Union[str, None] = None
|
self,
|
||||||
|
docs_paths: Union[str, list],
|
||||||
|
custom_file_types: Union[str, None] = None,
|
||||||
|
include_hidden: bool = False,
|
||||||
):
|
):
|
||||||
# Handle both single path (string) and multiple paths (list) for backward compatibility
|
# Handle both single path (string) and multiple paths (list) for backward compatibility
|
||||||
if isinstance(docs_paths, str):
|
if isinstance(docs_paths, str):
|
||||||
@@ -455,6 +702,10 @@ Examples:
|
|||||||
|
|
||||||
all_documents = []
|
all_documents = []
|
||||||
|
|
||||||
|
# Helper to detect hidden path components
|
||||||
|
def _path_has_hidden_segment(p: Path) -> bool:
|
||||||
|
return any(part.startswith(".") and part not in [".", ".."] for part in p.parts)
|
||||||
|
|
||||||
# First, process individual files if any
|
# First, process individual files if any
|
||||||
if files:
|
if files:
|
||||||
print(f"\n🔄 Processing {len(files)} individual file{'s' if len(files) > 1 else ''}...")
|
print(f"\n🔄 Processing {len(files)} individual file{'s' if len(files) > 1 else ''}...")
|
||||||
@@ -467,8 +718,12 @@ Examples:
|
|||||||
|
|
||||||
files_by_dir = defaultdict(list)
|
files_by_dir = defaultdict(list)
|
||||||
for file_path in files:
|
for file_path in files:
|
||||||
parent_dir = str(Path(file_path).parent)
|
file_path_obj = Path(file_path)
|
||||||
files_by_dir[parent_dir].append(file_path)
|
if not include_hidden and _path_has_hidden_segment(file_path_obj):
|
||||||
|
print(f" ⚠️ Skipping hidden file: {file_path}")
|
||||||
|
continue
|
||||||
|
parent_dir = str(file_path_obj.parent)
|
||||||
|
files_by_dir[parent_dir].append(str(file_path_obj))
|
||||||
|
|
||||||
# Load files from each parent directory
|
# Load files from each parent directory
|
||||||
for parent_dir, file_list in files_by_dir.items():
|
for parent_dir, file_list in files_by_dir.items():
|
||||||
@@ -479,6 +734,7 @@ Examples:
|
|||||||
file_docs = SimpleDirectoryReader(
|
file_docs = SimpleDirectoryReader(
|
||||||
parent_dir,
|
parent_dir,
|
||||||
input_files=file_list,
|
input_files=file_list,
|
||||||
|
# exclude_hidden only affects directory scans; input_files are explicit
|
||||||
filename_as_id=True,
|
filename_as_id=True,
|
||||||
).load_data()
|
).load_data()
|
||||||
all_documents.extend(file_docs)
|
all_documents.extend(file_docs)
|
||||||
@@ -577,6 +833,8 @@ Examples:
|
|||||||
# Check if file matches any exclude pattern
|
# Check if file matches any exclude pattern
|
||||||
try:
|
try:
|
||||||
relative_path = file_path.relative_to(docs_path)
|
relative_path = file_path.relative_to(docs_path)
|
||||||
|
if not include_hidden and _path_has_hidden_segment(relative_path):
|
||||||
|
continue
|
||||||
if self._should_exclude_file(relative_path, gitignore_matches):
|
if self._should_exclude_file(relative_path, gitignore_matches):
|
||||||
continue
|
continue
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -604,6 +862,7 @@ Examples:
|
|||||||
try:
|
try:
|
||||||
default_docs = SimpleDirectoryReader(
|
default_docs = SimpleDirectoryReader(
|
||||||
str(file_path.parent),
|
str(file_path.parent),
|
||||||
|
exclude_hidden=not include_hidden,
|
||||||
filename_as_id=True,
|
filename_as_id=True,
|
||||||
required_exts=[file_path.suffix],
|
required_exts=[file_path.suffix],
|
||||||
).load_data()
|
).load_data()
|
||||||
@@ -632,6 +891,7 @@ Examples:
|
|||||||
encoding="utf-8",
|
encoding="utf-8",
|
||||||
required_exts=code_extensions,
|
required_exts=code_extensions,
|
||||||
file_extractor={}, # Use default extractors
|
file_extractor={}, # Use default extractors
|
||||||
|
exclude_hidden=not include_hidden,
|
||||||
filename_as_id=True,
|
filename_as_id=True,
|
||||||
).load_data(show_progress=True)
|
).load_data(show_progress=True)
|
||||||
|
|
||||||
@@ -781,7 +1041,9 @@ Examples:
|
|||||||
paragraph_separator="\n\n",
|
paragraph_separator="\n\n",
|
||||||
)
|
)
|
||||||
|
|
||||||
all_texts = self.load_documents(docs_paths, args.file_types)
|
all_texts = self.load_documents(
|
||||||
|
docs_paths, args.file_types, include_hidden=args.include_hidden
|
||||||
|
)
|
||||||
if not all_texts:
|
if not all_texts:
|
||||||
print("No documents found")
|
print("No documents found")
|
||||||
return
|
return
|
||||||
@@ -918,6 +1180,8 @@ Examples:
|
|||||||
|
|
||||||
if args.command == "list":
|
if args.command == "list":
|
||||||
self.list_indexes()
|
self.list_indexes()
|
||||||
|
elif args.command == "remove":
|
||||||
|
self.remove_index(args.index_name, args.force)
|
||||||
elif args.command == "build":
|
elif args.command == "build":
|
||||||
await self.build_index(args)
|
await self.build_index(args)
|
||||||
elif args.command == "search":
|
elif args.command == "search":
|
||||||
@@ -929,10 +1193,15 @@ Examples:
|
|||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
|
import logging
|
||||||
|
|
||||||
import dotenv
|
import dotenv
|
||||||
|
|
||||||
dotenv.load_dotenv()
|
dotenv.load_dotenv()
|
||||||
|
|
||||||
|
# Set clean logging for CLI usage
|
||||||
|
logging.getLogger().setLevel(logging.WARNING) # Only show warnings and errors
|
||||||
|
|
||||||
cli = LeannCLI()
|
cli = LeannCLI()
|
||||||
asyncio.run(cli.run())
|
asyncio.run(cli.run())
|
||||||
|
|
||||||
|
|||||||
@@ -57,6 +57,8 @@ def compute_embeddings(
|
|||||||
return compute_embeddings_mlx(texts, model_name)
|
return compute_embeddings_mlx(texts, model_name)
|
||||||
elif mode == "ollama":
|
elif mode == "ollama":
|
||||||
return compute_embeddings_ollama(texts, model_name, is_build=is_build)
|
return compute_embeddings_ollama(texts, model_name, is_build=is_build)
|
||||||
|
elif mode == "gemini":
|
||||||
|
return compute_embeddings_gemini(texts, model_name, is_build=is_build)
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"Unsupported embedding mode: {mode}")
|
raise ValueError(f"Unsupported embedding mode: {mode}")
|
||||||
|
|
||||||
@@ -244,6 +246,16 @@ def compute_embeddings_openai(texts: list[str], model_name: str) -> np.ndarray:
|
|||||||
except ImportError as e:
|
except ImportError as e:
|
||||||
raise ImportError(f"OpenAI package not installed: {e}")
|
raise ImportError(f"OpenAI package not installed: {e}")
|
||||||
|
|
||||||
|
# Validate input list
|
||||||
|
if not texts:
|
||||||
|
raise ValueError("Cannot compute embeddings for empty text list")
|
||||||
|
# Extra validation: abort early if any item is empty/whitespace
|
||||||
|
invalid_count = sum(1 for t in texts if not isinstance(t, str) or not t.strip())
|
||||||
|
if invalid_count > 0:
|
||||||
|
raise ValueError(
|
||||||
|
f"Found {invalid_count} empty/invalid text(s) in input. Upstream should filter before calling OpenAI."
|
||||||
|
)
|
||||||
|
|
||||||
api_key = os.getenv("OPENAI_API_KEY")
|
api_key = os.getenv("OPENAI_API_KEY")
|
||||||
if not api_key:
|
if not api_key:
|
||||||
raise RuntimeError("OPENAI_API_KEY environment variable not set")
|
raise RuntimeError("OPENAI_API_KEY environment variable not set")
|
||||||
@@ -658,3 +670,83 @@ def compute_embeddings_ollama(
|
|||||||
logger.info(f"Generated {len(embeddings)} embeddings, dimension: {embeddings.shape[1]}")
|
logger.info(f"Generated {len(embeddings)} embeddings, dimension: {embeddings.shape[1]}")
|
||||||
|
|
||||||
return embeddings
|
return embeddings
|
||||||
|
|
||||||
|
|
||||||
|
def compute_embeddings_gemini(
|
||||||
|
texts: list[str], model_name: str = "text-embedding-004", is_build: bool = False
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
Compute embeddings using Google Gemini API.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
texts: List of texts to compute embeddings for
|
||||||
|
model_name: Gemini model name (default: "text-embedding-004")
|
||||||
|
is_build: Whether this is a build operation (shows progress bar)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Embeddings array, shape: (len(texts), embedding_dim)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
import os
|
||||||
|
|
||||||
|
import google.genai as genai
|
||||||
|
except ImportError as e:
|
||||||
|
raise ImportError(f"Google GenAI package not installed: {e}")
|
||||||
|
|
||||||
|
api_key = os.getenv("GEMINI_API_KEY")
|
||||||
|
if not api_key:
|
||||||
|
raise RuntimeError("GEMINI_API_KEY environment variable not set")
|
||||||
|
|
||||||
|
# Cache Gemini client
|
||||||
|
cache_key = "gemini_client"
|
||||||
|
if cache_key in _model_cache:
|
||||||
|
client = _model_cache[cache_key]
|
||||||
|
else:
|
||||||
|
client = genai.Client(api_key=api_key)
|
||||||
|
_model_cache[cache_key] = client
|
||||||
|
logger.info("Gemini client cached")
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Computing embeddings for {len(texts)} texts using Gemini API, model: '{model_name}'"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Gemini supports batch embedding
|
||||||
|
max_batch_size = 100 # Conservative batch size for Gemini
|
||||||
|
all_embeddings = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
total_batches = (len(texts) + max_batch_size - 1) // max_batch_size
|
||||||
|
batch_range = range(0, len(texts), max_batch_size)
|
||||||
|
batch_iterator = tqdm(
|
||||||
|
batch_range, desc="Computing embeddings", unit="batch", total=total_batches
|
||||||
|
)
|
||||||
|
except ImportError:
|
||||||
|
# Fallback when tqdm is not available
|
||||||
|
batch_iterator = range(0, len(texts), max_batch_size)
|
||||||
|
|
||||||
|
for i in batch_iterator:
|
||||||
|
batch_texts = texts[i : i + max_batch_size]
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Use the embed_content method from the new Google GenAI SDK
|
||||||
|
response = client.models.embed_content(
|
||||||
|
model=model_name,
|
||||||
|
contents=batch_texts,
|
||||||
|
config=genai.types.EmbedContentConfig(
|
||||||
|
task_type="RETRIEVAL_DOCUMENT" # For document embedding
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Extract embeddings from response
|
||||||
|
for embedding_data in response.embeddings:
|
||||||
|
all_embeddings.append(embedding_data.values)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Batch {i} failed: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
embeddings = np.array(all_embeddings, dtype=np.float32)
|
||||||
|
logger.info(f"Generated {len(embeddings)} embeddings, dimension: {embeddings.shape[1]}")
|
||||||
|
|
||||||
|
return embeddings
|
||||||
|
|||||||
@@ -2,11 +2,15 @@
|
|||||||
|
|
||||||
import importlib
|
import importlib
|
||||||
import importlib.metadata
|
import importlib.metadata
|
||||||
|
import logging
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from leann.interface import LeannBackendFactoryInterface
|
from leann.interface import LeannBackendFactoryInterface
|
||||||
|
|
||||||
|
# Set up logger for this module
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
BACKEND_REGISTRY: dict[str, "LeannBackendFactoryInterface"] = {}
|
BACKEND_REGISTRY: dict[str, "LeannBackendFactoryInterface"] = {}
|
||||||
|
|
||||||
|
|
||||||
@@ -14,7 +18,7 @@ def register_backend(name: str):
|
|||||||
"""A decorator to register a new backend class."""
|
"""A decorator to register a new backend class."""
|
||||||
|
|
||||||
def decorator(cls):
|
def decorator(cls):
|
||||||
print(f"INFO: Registering backend '{name}'")
|
logger.debug(f"Registering backend '{name}'")
|
||||||
BACKEND_REGISTRY[name] = cls
|
BACKEND_REGISTRY[name] = cls
|
||||||
return cls
|
return cls
|
||||||
|
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ leann build docs-and-configs --docs $(git ls-files "*.md" "*.yml" "*.yaml" "*.js
|
|||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
**Try this in Claude Code:**
|
## **Try this in Claude Code:**
|
||||||
```
|
```
|
||||||
Help me understand this codebase. List available indexes and search for authentication patterns.
|
Help me understand this codebase. List available indexes and search for authentication patterns.
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann"
|
name = "leann"
|
||||||
version = "0.2.9"
|
version = "0.3.0"
|
||||||
description = "LEANN - The smallest vector index in the world. RAG Everything with LEANN!"
|
description = "LEANN - The smallest vector index in the world. RAG Everything with LEANN!"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.9"
|
requires-python = ">=3.9"
|
||||||
|
|||||||
Reference in New Issue
Block a user