Compare commits
22 Commits
ast-fork
...
fix-arm64-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2159a29073 | ||
|
|
185bd38112 | ||
|
|
936fa525de | ||
|
|
f6c83898b8 | ||
|
|
40cb39ed8a | ||
|
|
b74718332e | ||
|
|
2c0e4ec58d | ||
|
|
9836ce049d | ||
|
|
478b10c7c1 | ||
|
|
1fce9ad445 | ||
|
|
d452b1ffa3 | ||
|
|
e0085da8ba | ||
|
|
377c952134 | ||
|
|
0ff18a7d79 | ||
|
|
08f9757c45 | ||
|
|
c5c8a57441 | ||
|
|
a5ef3e66d0 | ||
|
|
5079a8b799 | ||
|
|
07f8129f65 | ||
|
|
45ef563bda | ||
|
|
e9d447ac2a | ||
|
|
141e498329 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -22,7 +22,6 @@ demo/experiment_results/**/*.json
|
|||||||
*.sh
|
*.sh
|
||||||
*.txt
|
*.txt
|
||||||
!CMakeLists.txt
|
!CMakeLists.txt
|
||||||
!llms.txt
|
|
||||||
latency_breakdown*.json
|
latency_breakdown*.json
|
||||||
experiment_results/eval_results/diskann/*.json
|
experiment_results/eval_results/diskann/*.json
|
||||||
aws/
|
aws/
|
||||||
|
|||||||
4
.gitmodules
vendored
4
.gitmodules
vendored
@@ -14,7 +14,3 @@
|
|||||||
[submodule "packages/leann-backend-hnsw/third_party/libzmq"]
|
[submodule "packages/leann-backend-hnsw/third_party/libzmq"]
|
||||||
path = packages/leann-backend-hnsw/third_party/libzmq
|
path = packages/leann-backend-hnsw/third_party/libzmq
|
||||||
url = https://github.com/zeromq/libzmq.git
|
url = https://github.com/zeromq/libzmq.git
|
||||||
[submodule "packages/astchunk-leann"]
|
|
||||||
path = packages/astchunk-leann
|
|
||||||
url = git@github.com:yichuan-w/astchunk-leann.git
|
|
||||||
branch = main
|
|
||||||
|
|||||||
13
README.md
13
README.md
@@ -656,19 +656,6 @@ results = searcher.search(
|
|||||||
|
|
||||||
📖 **[Complete Metadata filtering guide →](docs/metadata_filtering.md)**
|
📖 **[Complete Metadata filtering guide →](docs/metadata_filtering.md)**
|
||||||
|
|
||||||
### 🔍 Grep Search
|
|
||||||
|
|
||||||
For exact text matching instead of semantic search, use the `use_grep` parameter:
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Exact text search
|
|
||||||
results = searcher.search("banana‑crocodile", use_grep=True, top_k=1)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Use cases**: Finding specific code patterns, error messages, function names, or exact phrases where semantic similarity isn't needed.
|
|
||||||
|
|
||||||
📖 **[Complete grep search guide →](docs/grep_search.md)**
|
|
||||||
|
|
||||||
## 🏗️ Architecture & How It Works
|
## 🏗️ Architecture & How It Works
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
|
|||||||
@@ -26,21 +26,6 @@ leann build my-code-index --docs ./src --use-ast-chunking
|
|||||||
uv pip install -e "."
|
uv pip install -e "."
|
||||||
```
|
```
|
||||||
|
|
||||||
#### For normal users (PyPI install)
|
|
||||||
- Use `pip install leann` or `uv pip install leann`.
|
|
||||||
- `astchunk` is pulled automatically from PyPI as a dependency; no extra steps.
|
|
||||||
|
|
||||||
#### For developers (from source, editable)
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/yichuan-w/LEANN.git leann
|
|
||||||
cd leann
|
|
||||||
git submodule update --init --recursive
|
|
||||||
uv sync
|
|
||||||
```
|
|
||||||
- This repo vendors `astchunk` as a git submodule at `packages/astchunk-leann` (our fork).
|
|
||||||
- `[tool.uv.sources]` maps the `astchunk` package to that path in editable mode.
|
|
||||||
- You can edit code under `packages/astchunk-leann` and Python will use your changes immediately (no separate `pip install astchunk` needed).
|
|
||||||
|
|
||||||
## Best Practices
|
## Best Practices
|
||||||
|
|
||||||
### When to Use AST Chunking
|
### When to Use AST Chunking
|
||||||
|
|||||||
@@ -1,149 +0,0 @@
|
|||||||
# LEANN Grep Search Usage Guide
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
LEANN's grep search functionality provides exact text matching for finding specific code patterns, error messages, function names, or exact phrases in your indexed documents.
|
|
||||||
|
|
||||||
## Basic Usage
|
|
||||||
|
|
||||||
### Simple Grep Search
|
|
||||||
|
|
||||||
```python
|
|
||||||
from leann.api import LeannSearcher
|
|
||||||
|
|
||||||
searcher = LeannSearcher("your_index_path")
|
|
||||||
|
|
||||||
# Exact text search
|
|
||||||
results = searcher.search("def authenticate_user", use_grep=True, top_k=5)
|
|
||||||
|
|
||||||
for result in results:
|
|
||||||
print(f"Score: {result.score}")
|
|
||||||
print(f"Text: {result.text[:100]}...")
|
|
||||||
print("-" * 40)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Comparison: Semantic vs Grep Search
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Semantic search - finds conceptually similar content
|
|
||||||
semantic_results = searcher.search("machine learning algorithms", top_k=3)
|
|
||||||
|
|
||||||
# Grep search - finds exact text matches
|
|
||||||
grep_results = searcher.search("def train_model", use_grep=True, top_k=3)
|
|
||||||
```
|
|
||||||
|
|
||||||
## When to Use Grep Search
|
|
||||||
|
|
||||||
### Use Cases
|
|
||||||
|
|
||||||
- **Code Search**: Finding specific function definitions, class names, or variable references
|
|
||||||
- **Error Debugging**: Locating exact error messages or stack traces
|
|
||||||
- **Documentation**: Finding specific API endpoints or exact terminology
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Find function definitions
|
|
||||||
functions = searcher.search("def __init__", use_grep=True)
|
|
||||||
|
|
||||||
# Find import statements
|
|
||||||
imports = searcher.search("from sklearn import", use_grep=True)
|
|
||||||
|
|
||||||
# Find specific error types
|
|
||||||
errors = searcher.search("FileNotFoundError", use_grep=True)
|
|
||||||
|
|
||||||
# Find TODO comments
|
|
||||||
todos = searcher.search("TODO:", use_grep=True)
|
|
||||||
|
|
||||||
# Find configuration entries
|
|
||||||
configs = searcher.search("server_port=", use_grep=True)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Technical Details
|
|
||||||
|
|
||||||
### How It Works
|
|
||||||
|
|
||||||
1. **File Location**: Grep search operates on the raw text stored in `.jsonl` files
|
|
||||||
2. **Command Execution**: Uses the system `grep` command with case-insensitive search
|
|
||||||
3. **Result Processing**: Parses JSON lines and extracts text and metadata
|
|
||||||
4. **Scoring**: Simple frequency-based scoring based on query term occurrences
|
|
||||||
|
|
||||||
### Search Process
|
|
||||||
|
|
||||||
```
|
|
||||||
Query: "def train_model"
|
|
||||||
↓
|
|
||||||
grep -i -n "def train_model" documents.leann.passages.jsonl
|
|
||||||
↓
|
|
||||||
Parse matching JSON lines
|
|
||||||
↓
|
|
||||||
Calculate scores based on term frequency
|
|
||||||
↓
|
|
||||||
Return top_k results
|
|
||||||
```
|
|
||||||
|
|
||||||
### Scoring Algorithm
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Term frequency in document
|
|
||||||
score = text.lower().count(query.lower())
|
|
||||||
```
|
|
||||||
|
|
||||||
Results are ranked by score (highest first), with higher scores indicating more occurrences of the search term.
|
|
||||||
|
|
||||||
## Error Handling
|
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
#### Grep Command Not Found
|
|
||||||
```
|
|
||||||
RuntimeError: grep command not found. Please install grep or use semantic search.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution**: Install grep on your system:
|
|
||||||
- **Ubuntu/Debian**: `sudo apt-get install grep`
|
|
||||||
- **macOS**: grep is pre-installed
|
|
||||||
- **Windows**: Use WSL or install grep via Git Bash/MSYS2
|
|
||||||
|
|
||||||
#### No Results Found
|
|
||||||
```python
|
|
||||||
# Check if your query exists in the raw data
|
|
||||||
results = searcher.search("your_query", use_grep=True)
|
|
||||||
if not results:
|
|
||||||
print("No exact matches found. Try:")
|
|
||||||
print("1. Check spelling and case")
|
|
||||||
print("2. Use partial terms")
|
|
||||||
print("3. Switch to semantic search")
|
|
||||||
```
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
```python
|
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
Grep Search Example
|
|
||||||
Demonstrates grep search for exact text matching.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from leann.api import LeannSearcher
|
|
||||||
|
|
||||||
def demonstrate_grep_search():
|
|
||||||
# Initialize searcher
|
|
||||||
searcher = LeannSearcher("my_index")
|
|
||||||
|
|
||||||
print("=== Function Search ===")
|
|
||||||
functions = searcher.search("def __init__", use_grep=True, top_k=5)
|
|
||||||
for i, result in enumerate(functions, 1):
|
|
||||||
print(f"{i}. Score: {result.score}")
|
|
||||||
print(f" Preview: {result.text[:60]}...")
|
|
||||||
print()
|
|
||||||
|
|
||||||
print("=== Error Search ===")
|
|
||||||
errors = searcher.search("FileNotFoundError", use_grep=True, top_k=3)
|
|
||||||
for result in errors:
|
|
||||||
print(f"Content: {result.text.strip()}")
|
|
||||||
print("-" * 40)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
demonstrate_grep_search()
|
|
||||||
```
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
"""
|
|
||||||
Grep Search Example
|
|
||||||
|
|
||||||
Shows how to use grep-based text search instead of semantic search.
|
|
||||||
Useful when you need exact text matches rather than meaning-based results.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from leann import LeannSearcher
|
|
||||||
|
|
||||||
# Load your index
|
|
||||||
searcher = LeannSearcher("my-documents.leann")
|
|
||||||
|
|
||||||
# Regular semantic search
|
|
||||||
print("=== Semantic Search ===")
|
|
||||||
results = searcher.search("machine learning algorithms", top_k=3)
|
|
||||||
for result in results:
|
|
||||||
print(f"Score: {result.score:.3f}")
|
|
||||||
print(f"Text: {result.text[:80]}...")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Grep-based search for exact text matches
|
|
||||||
print("=== Grep Search ===")
|
|
||||||
results = searcher.search("def train_model", top_k=3, use_grep=True)
|
|
||||||
for result in results:
|
|
||||||
print(f"Score: {result.score}")
|
|
||||||
print(f"Text: {result.text[:80]}...")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Find specific error messages
|
|
||||||
error_results = searcher.search("FileNotFoundError", use_grep=True)
|
|
||||||
print(f"Found {len(error_results)} files mentioning FileNotFoundError")
|
|
||||||
|
|
||||||
# Search for function definitions
|
|
||||||
func_results = searcher.search("class SearchResult", use_grep=True, top_k=5)
|
|
||||||
print(f"Found {len(func_results)} class definitions")
|
|
||||||
28
llms.txt
28
llms.txt
@@ -1,28 +0,0 @@
|
|||||||
# llms.txt — LEANN MCP and Agent Integration
|
|
||||||
product: LEANN
|
|
||||||
homepage: https://github.com/yichuan-w/LEANN
|
|
||||||
contact: https://github.com/yichuan-w/LEANN/issues
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
install: uv tool install leann-core --with leann
|
|
||||||
|
|
||||||
# MCP Server Entry Point
|
|
||||||
mcp.server: leann_mcp
|
|
||||||
mcp.protocol_version: 2024-11-05
|
|
||||||
|
|
||||||
# Tools
|
|
||||||
mcp.tools: leann_list, leann_search
|
|
||||||
|
|
||||||
mcp.tool.leann_list.description: List available LEANN indexes
|
|
||||||
mcp.tool.leann_list.input: {}
|
|
||||||
|
|
||||||
mcp.tool.leann_search.description: Semantic search across a named LEANN index
|
|
||||||
mcp.tool.leann_search.input.index_name: string, required
|
|
||||||
mcp.tool.leann_search.input.query: string, required
|
|
||||||
mcp.tool.leann_search.input.top_k: integer, optional, default=5, min=1, max=20
|
|
||||||
mcp.tool.leann_search.input.complexity: integer, optional, default=32, min=16, max=128
|
|
||||||
|
|
||||||
# Notes
|
|
||||||
note: Build indexes with `leann build <name> --docs <files...>` before searching.
|
|
||||||
example.add: claude mcp add --scope user leann-server -- leann_mcp
|
|
||||||
example.verify: claude mcp list | cat
|
|
||||||
Submodule packages/astchunk-leann deleted from a4537018a3
@@ -4,8 +4,8 @@ build-backend = "scikit_build_core.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-backend-diskann"
|
name = "leann-backend-diskann"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
dependencies = ["leann-core==0.3.3", "numpy", "protobuf>=3.19.0"]
|
dependencies = ["leann-core==0.3.2", "numpy", "protobuf>=3.19.0"]
|
||||||
|
|
||||||
[tool.scikit-build]
|
[tool.scikit-build]
|
||||||
# Key: simplified CMake path
|
# Key: simplified CMake path
|
||||||
|
|||||||
@@ -6,10 +6,10 @@ build-backend = "scikit_build_core.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-backend-hnsw"
|
name = "leann-backend-hnsw"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
description = "Custom-built HNSW (Faiss) backend for the Leann toolkit."
|
description = "Custom-built HNSW (Faiss) backend for the Leann toolkit."
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"leann-core==0.3.3",
|
"leann-core==0.3.2",
|
||||||
"numpy",
|
"numpy",
|
||||||
"pyzmq>=23.0.0",
|
"pyzmq>=23.0.0",
|
||||||
"msgpack>=1.0.0",
|
"msgpack>=1.0.0",
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann-core"
|
name = "leann-core"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
description = "Core API and plugin system for LEANN"
|
description = "Core API and plugin system for LEANN"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.9"
|
requires-python = ">=3.9"
|
||||||
|
|||||||
@@ -6,8 +6,6 @@ with the correct, original embedding logic from the user's reference code.
|
|||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import pickle
|
import pickle
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import time
|
import time
|
||||||
import warnings
|
import warnings
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
@@ -655,7 +653,6 @@ class LeannSearcher:
|
|||||||
expected_zmq_port: int = 5557,
|
expected_zmq_port: int = 5557,
|
||||||
metadata_filters: Optional[dict[str, dict[str, Union[str, int, float, bool, list]]]] = None,
|
metadata_filters: Optional[dict[str, dict[str, Union[str, int, float, bool, list]]]] = None,
|
||||||
batch_size: int = 0,
|
batch_size: int = 0,
|
||||||
use_grep: bool = False,
|
|
||||||
**kwargs,
|
**kwargs,
|
||||||
) -> list[SearchResult]:
|
) -> list[SearchResult]:
|
||||||
"""
|
"""
|
||||||
@@ -682,10 +679,6 @@ class LeannSearcher:
|
|||||||
Returns:
|
Returns:
|
||||||
List of SearchResult objects with text, metadata, and similarity scores
|
List of SearchResult objects with text, metadata, and similarity scores
|
||||||
"""
|
"""
|
||||||
# Handle grep search
|
|
||||||
if use_grep:
|
|
||||||
return self._grep_search(query, top_k)
|
|
||||||
|
|
||||||
logger.info("🔍 LeannSearcher.search() called:")
|
logger.info("🔍 LeannSearcher.search() called:")
|
||||||
logger.info(f" Query: '{query}'")
|
logger.info(f" Query: '{query}'")
|
||||||
logger.info(f" Top_k: {top_k}")
|
logger.info(f" Top_k: {top_k}")
|
||||||
@@ -802,96 +795,9 @@ class LeannSearcher:
|
|||||||
logger.info(f" {GREEN}✓ Final enriched results: {len(enriched_results)} passages{RESET}")
|
logger.info(f" {GREEN}✓ Final enriched results: {len(enriched_results)} passages{RESET}")
|
||||||
return enriched_results
|
return enriched_results
|
||||||
|
|
||||||
def _find_jsonl_file(self) -> Optional[str]:
|
|
||||||
"""Find the .jsonl file containing raw passages for grep search"""
|
|
||||||
index_path = Path(self.meta_path_str).parent
|
|
||||||
potential_files = [
|
|
||||||
index_path / "documents.leann.passages.jsonl",
|
|
||||||
index_path.parent / "documents.leann.passages.jsonl",
|
|
||||||
]
|
|
||||||
|
|
||||||
for file_path in potential_files:
|
|
||||||
if file_path.exists():
|
|
||||||
return str(file_path)
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _grep_search(self, query: str, top_k: int = 5) -> list[SearchResult]:
|
|
||||||
"""Perform grep-based search on raw passages"""
|
|
||||||
jsonl_file = self._find_jsonl_file()
|
|
||||||
if not jsonl_file:
|
|
||||||
raise FileNotFoundError("No .jsonl passages file found for grep search")
|
|
||||||
|
|
||||||
try:
|
|
||||||
cmd = ["grep", "-i", "-n", query, jsonl_file]
|
|
||||||
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
|
||||||
|
|
||||||
if result.returncode == 1:
|
|
||||||
return []
|
|
||||||
elif result.returncode != 0:
|
|
||||||
raise RuntimeError(f"Grep failed: {result.stderr}")
|
|
||||||
|
|
||||||
matches = []
|
|
||||||
for line in result.stdout.strip().split("\n"):
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
parts = line.split(":", 1)
|
|
||||||
if len(parts) != 2:
|
|
||||||
continue
|
|
||||||
|
|
||||||
try:
|
|
||||||
data = json.loads(parts[1])
|
|
||||||
text = data.get("text", "")
|
|
||||||
score = text.lower().count(query.lower())
|
|
||||||
|
|
||||||
matches.append(
|
|
||||||
SearchResult(
|
|
||||||
id=data.get("id", parts[0]),
|
|
||||||
text=text,
|
|
||||||
metadata=data.get("metadata", {}),
|
|
||||||
score=float(score),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
|
|
||||||
matches.sort(key=lambda x: x.score, reverse=True)
|
|
||||||
return matches[:top_k]
|
|
||||||
|
|
||||||
except FileNotFoundError:
|
|
||||||
raise RuntimeError(
|
|
||||||
"grep command not found. Please install grep or use semantic search."
|
|
||||||
)
|
|
||||||
|
|
||||||
def _python_regex_search(self, query: str, top_k: int = 5) -> list[SearchResult]:
|
|
||||||
"""Fallback regex search"""
|
|
||||||
jsonl_file = self._find_jsonl_file()
|
|
||||||
if not jsonl_file:
|
|
||||||
raise FileNotFoundError("No .jsonl file found")
|
|
||||||
|
|
||||||
pattern = re.compile(re.escape(query), re.IGNORECASE)
|
|
||||||
matches = []
|
|
||||||
|
|
||||||
with open(jsonl_file, encoding="utf-8") as f:
|
|
||||||
for line_num, line in enumerate(f, 1):
|
|
||||||
if pattern.search(line):
|
|
||||||
try:
|
|
||||||
data = json.loads(line.strip())
|
|
||||||
matches.append(
|
|
||||||
SearchResult(
|
|
||||||
id=data.get("id", str(line_num)),
|
|
||||||
text=data.get("text", ""),
|
|
||||||
metadata=data.get("metadata", {}),
|
|
||||||
score=float(len(pattern.findall(data.get("text", "")))),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
|
|
||||||
matches.sort(key=lambda x: x.score, reverse=True)
|
|
||||||
return matches[:top_k]
|
|
||||||
|
|
||||||
def cleanup(self):
|
def cleanup(self):
|
||||||
"""Explicitly cleanup embedding server resources.
|
"""Explicitly cleanup embedding server resources.
|
||||||
|
|
||||||
This method should be called after you're done using the searcher,
|
This method should be called after you're done using the searcher,
|
||||||
especially in test environments or batch processing scenarios.
|
especially in test environments or batch processing scenarios.
|
||||||
"""
|
"""
|
||||||
@@ -947,7 +853,6 @@ class LeannChat:
|
|||||||
expected_zmq_port: int = 5557,
|
expected_zmq_port: int = 5557,
|
||||||
metadata_filters: Optional[dict[str, dict[str, Union[str, int, float, bool, list]]]] = None,
|
metadata_filters: Optional[dict[str, dict[str, Union[str, int, float, bool, list]]]] = None,
|
||||||
batch_size: int = 0,
|
batch_size: int = 0,
|
||||||
use_grep: bool = False,
|
|
||||||
**search_kwargs,
|
**search_kwargs,
|
||||||
):
|
):
|
||||||
if llm_kwargs is None:
|
if llm_kwargs is None:
|
||||||
|
|||||||
@@ -322,17 +322,9 @@ Examples:
|
|||||||
|
|
||||||
return basic_matches
|
return basic_matches
|
||||||
|
|
||||||
def _should_exclude_file(self, file_path: Path, gitignore_matches) -> bool:
|
def _should_exclude_file(self, relative_path: Path, gitignore_matches) -> bool:
|
||||||
"""Check if a file should be excluded using gitignore parser.
|
"""Check if a file should be excluded using gitignore parser."""
|
||||||
|
return gitignore_matches(str(relative_path))
|
||||||
Always match against absolute, posix-style paths for consistency with
|
|
||||||
gitignore_parser expectations.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
absolute_path = file_path.resolve()
|
|
||||||
except Exception:
|
|
||||||
absolute_path = Path(str(file_path))
|
|
||||||
return gitignore_matches(absolute_path.as_posix())
|
|
||||||
|
|
||||||
def _is_git_submodule(self, path: Path) -> bool:
|
def _is_git_submodule(self, path: Path) -> bool:
|
||||||
"""Check if a path is a git submodule."""
|
"""Check if a path is a git submodule."""
|
||||||
@@ -404,9 +396,7 @@ Examples:
|
|||||||
print(f" {current_path}")
|
print(f" {current_path}")
|
||||||
print(" " + "─" * 45)
|
print(" " + "─" * 45)
|
||||||
|
|
||||||
current_indexes = self._discover_indexes_in_project(
|
current_indexes = self._discover_indexes_in_project(current_path)
|
||||||
current_path, exclude_dirs=other_projects
|
|
||||||
)
|
|
||||||
if current_indexes:
|
if current_indexes:
|
||||||
for idx in current_indexes:
|
for idx in current_indexes:
|
||||||
total_indexes += 1
|
total_indexes += 1
|
||||||
@@ -445,14 +435,9 @@ Examples:
|
|||||||
print(" leann build my-docs --docs ./documents")
|
print(" leann build my-docs --docs ./documents")
|
||||||
else:
|
else:
|
||||||
# Count only projects that have at least one discoverable index
|
# Count only projects that have at least one discoverable index
|
||||||
projects_count = 0
|
projects_count = sum(
|
||||||
for p in valid_projects:
|
1 for p in valid_projects if len(self._discover_indexes_in_project(p)) > 0
|
||||||
if p == current_path:
|
)
|
||||||
discovered = self._discover_indexes_in_project(p, exclude_dirs=other_projects)
|
|
||||||
else:
|
|
||||||
discovered = self._discover_indexes_in_project(p)
|
|
||||||
if len(discovered) > 0:
|
|
||||||
projects_count += 1
|
|
||||||
print(f"📊 Total: {total_indexes} indexes across {projects_count} projects")
|
print(f"📊 Total: {total_indexes} indexes across {projects_count} projects")
|
||||||
|
|
||||||
if current_indexes_count > 0:
|
if current_indexes_count > 0:
|
||||||
@@ -469,22 +454,9 @@ Examples:
|
|||||||
print("\n💡 Create your first index:")
|
print("\n💡 Create your first index:")
|
||||||
print(" leann build my-docs --docs ./documents")
|
print(" leann build my-docs --docs ./documents")
|
||||||
|
|
||||||
def _discover_indexes_in_project(
|
def _discover_indexes_in_project(self, project_path: Path):
|
||||||
self, project_path: Path, exclude_dirs: Optional[list[Path]] = None
|
"""Discover all indexes in a project directory (both CLI and apps formats)"""
|
||||||
):
|
|
||||||
"""Discover all indexes in a project directory (both CLI and apps formats)
|
|
||||||
|
|
||||||
exclude_dirs: when provided, skip any APP-format index files that are
|
|
||||||
located under these directories. This prevents duplicates when the
|
|
||||||
current project is a parent directory of other registered projects.
|
|
||||||
"""
|
|
||||||
indexes = []
|
indexes = []
|
||||||
exclude_dirs = exclude_dirs or []
|
|
||||||
# normalize to resolved paths once for comparison
|
|
||||||
try:
|
|
||||||
exclude_dirs_resolved = [p.resolve() for p in exclude_dirs]
|
|
||||||
except Exception:
|
|
||||||
exclude_dirs_resolved = exclude_dirs
|
|
||||||
|
|
||||||
# 1. CLI format: .leann/indexes/index_name/
|
# 1. CLI format: .leann/indexes/index_name/
|
||||||
cli_indexes_dir = project_path / ".leann" / "indexes"
|
cli_indexes_dir = project_path / ".leann" / "indexes"
|
||||||
@@ -523,17 +495,6 @@ Examples:
|
|||||||
continue
|
continue
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
# Skip meta files that live under excluded directories
|
|
||||||
try:
|
|
||||||
meta_parent_resolved = meta_file.parent.resolve()
|
|
||||||
if any(
|
|
||||||
meta_parent_resolved.is_relative_to(ex_dir)
|
|
||||||
for ex_dir in exclude_dirs_resolved
|
|
||||||
):
|
|
||||||
continue
|
|
||||||
except Exception:
|
|
||||||
# best effort; if resolve or comparison fails, do not exclude
|
|
||||||
pass
|
|
||||||
# Use the parent directory name as the app index display name
|
# Use the parent directory name as the app index display name
|
||||||
display_name = meta_file.parent.name
|
display_name = meta_file.parent.name
|
||||||
# Extract file base used to store files
|
# Extract file base used to store files
|
||||||
@@ -1061,8 +1022,7 @@ Examples:
|
|||||||
|
|
||||||
# Try to use better PDF parsers first, but only if PDFs are requested
|
# Try to use better PDF parsers first, but only if PDFs are requested
|
||||||
documents = []
|
documents = []
|
||||||
# Use resolved absolute paths to avoid mismatches (symlinks, relative vs absolute)
|
docs_path = Path(docs_dir)
|
||||||
docs_path = Path(docs_dir).resolve()
|
|
||||||
|
|
||||||
# Check if we should process PDFs
|
# Check if we should process PDFs
|
||||||
should_process_pdfs = custom_file_types is None or ".pdf" in custom_file_types
|
should_process_pdfs = custom_file_types is None or ".pdf" in custom_file_types
|
||||||
@@ -1071,15 +1031,10 @@ Examples:
|
|||||||
for file_path in docs_path.rglob("*.pdf"):
|
for file_path in docs_path.rglob("*.pdf"):
|
||||||
# Check if file matches any exclude pattern
|
# Check if file matches any exclude pattern
|
||||||
try:
|
try:
|
||||||
# Ensure both paths are resolved before computing relativity
|
|
||||||
file_path_resolved = file_path.resolve()
|
|
||||||
# Determine directory scope using the non-resolved path to avoid
|
|
||||||
# misclassifying symlinked entries as outside the docs directory
|
|
||||||
relative_path = file_path.relative_to(docs_path)
|
relative_path = file_path.relative_to(docs_path)
|
||||||
if not include_hidden and _path_has_hidden_segment(relative_path):
|
if not include_hidden and _path_has_hidden_segment(relative_path):
|
||||||
continue
|
continue
|
||||||
# Use absolute path for gitignore matching
|
if self._should_exclude_file(relative_path, gitignore_matches):
|
||||||
if self._should_exclude_file(file_path_resolved, gitignore_matches):
|
|
||||||
continue
|
continue
|
||||||
except ValueError:
|
except ValueError:
|
||||||
# Skip files that can't be made relative to docs_path
|
# Skip files that can't be made relative to docs_path
|
||||||
@@ -1122,11 +1077,10 @@ Examples:
|
|||||||
) -> bool:
|
) -> bool:
|
||||||
"""Return True if file should be included (not excluded)"""
|
"""Return True if file should be included (not excluded)"""
|
||||||
try:
|
try:
|
||||||
docs_path_obj = Path(docs_dir).resolve()
|
docs_path_obj = Path(docs_dir)
|
||||||
file_path_obj = Path(file_path).resolve()
|
file_path_obj = Path(file_path)
|
||||||
# Use absolute path for gitignore matching
|
relative_path = file_path_obj.relative_to(docs_path_obj)
|
||||||
_ = file_path_obj.relative_to(docs_path_obj) # validate scope
|
return not self._should_exclude_file(relative_path, gitignore_matches)
|
||||||
return not self._should_exclude_file(file_path_obj, gitignore_matches)
|
|
||||||
except (ValueError, OSError):
|
except (ValueError, OSError):
|
||||||
return True # Include files that can't be processed
|
return True # Include files that can't be processed
|
||||||
|
|
||||||
|
|||||||
@@ -2,8 +2,6 @@
|
|||||||
|
|
||||||
Transform your development workflow with intelligent code assistance using LEANN's semantic search directly in Claude Code.
|
Transform your development workflow with intelligent code assistance using LEANN's semantic search directly in Claude Code.
|
||||||
|
|
||||||
For agent-facing discovery details, see `llms.txt` in the repository root.
|
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
Install LEANN globally for MCP integration (with default backend):
|
Install LEANN globally for MCP integration (with default backend):
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "leann"
|
name = "leann"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
description = "LEANN - The smallest vector index in the world. RAG Everything with LEANN!"
|
description = "LEANN - The smallest vector index in the world. RAG Everything with LEANN!"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.9"
|
requires-python = ">=3.9"
|
||||||
|
|||||||
@@ -99,7 +99,6 @@ wechat-exporter = "wechat_exporter.main:main"
|
|||||||
leann-core = { path = "packages/leann-core", editable = true }
|
leann-core = { path = "packages/leann-core", editable = true }
|
||||||
leann-backend-diskann = { path = "packages/leann-backend-diskann", editable = true }
|
leann-backend-diskann = { path = "packages/leann-backend-diskann", editable = true }
|
||||||
leann-backend-hnsw = { path = "packages/leann-backend-hnsw", editable = true }
|
leann-backend-hnsw = { path = "packages/leann-backend-hnsw", editable = true }
|
||||||
astchunk = { path = "packages/astchunk-leann", editable = true }
|
|
||||||
|
|
||||||
[tool.ruff]
|
[tool.ruff]
|
||||||
target-version = "py39"
|
target-version = "py39"
|
||||||
|
|||||||
45
uv.lock
generated
45
uv.lock
generated
@@ -1,5 +1,5 @@
|
|||||||
version = 1
|
version = 1
|
||||||
revision = 2
|
revision = 3
|
||||||
requires-python = ">=3.9"
|
requires-python = ">=3.9"
|
||||||
resolution-markers = [
|
resolution-markers = [
|
||||||
"python_full_version >= '3.12'",
|
"python_full_version >= '3.12'",
|
||||||
@@ -201,7 +201,7 @@ wheels = [
|
|||||||
[[package]]
|
[[package]]
|
||||||
name = "astchunk"
|
name = "astchunk"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
source = { editable = "packages/astchunk-leann" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" },
|
{ name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" },
|
||||||
{ name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.10.*'" },
|
{ name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.10.*'" },
|
||||||
@@ -214,31 +214,10 @@ dependencies = [
|
|||||||
{ name = "tree-sitter-python" },
|
{ name = "tree-sitter-python" },
|
||||||
{ name = "tree-sitter-typescript" },
|
{ name = "tree-sitter-typescript" },
|
||||||
]
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/db/2a/7a35e2fac7d550265ae2ee40651425083b37555f921d1a1b77c3f525e0df/astchunk-0.1.0.tar.gz", hash = "sha256:f4dff0ef8b3b3bcfeac363384db1e153f74d4c825dc2e35864abfab027713be4", size = 18093, upload-time = "2025-06-19T04:37:25.34Z" }
|
||||||
[package.metadata]
|
wheels = [
|
||||||
requires-dist = [
|
{ url = "https://files.pythonhosted.org/packages/be/84/5433ab0e933b572750cb16fd7edf3d6c7902b069461a22ec670042752a4d/astchunk-0.1.0-py3-none-any.whl", hash = "sha256:33ada9fc3620807fdda5846fa1948af463f281a60e0d43d4f3782b6dbb416d24", size = 15396, upload-time = "2025-06-19T04:37:23.87Z" },
|
||||||
{ name = "black", marker = "extra == 'dev'", specifier = ">=22.0.0" },
|
|
||||||
{ name = "flake8", marker = "extra == 'dev'", specifier = ">=5.0.0" },
|
|
||||||
{ name = "isort", marker = "extra == 'dev'", specifier = ">=5.10.0" },
|
|
||||||
{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.0.0" },
|
|
||||||
{ name = "myst-parser", marker = "extra == 'docs'", specifier = ">=0.18.0" },
|
|
||||||
{ name = "numpy", specifier = ">=1.20.0" },
|
|
||||||
{ name = "pre-commit", marker = "extra == 'dev'", specifier = ">=2.20.0" },
|
|
||||||
{ name = "pyrsistent", specifier = ">=0.18.0" },
|
|
||||||
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=7.0.0" },
|
|
||||||
{ name = "pytest", marker = "extra == 'test'", specifier = ">=7.0.0" },
|
|
||||||
{ name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=4.0.0" },
|
|
||||||
{ name = "pytest-cov", marker = "extra == 'test'", specifier = ">=4.0.0" },
|
|
||||||
{ name = "pytest-xdist", marker = "extra == 'test'", specifier = ">=2.5.0" },
|
|
||||||
{ name = "sphinx", marker = "extra == 'docs'", specifier = ">=5.0.0" },
|
|
||||||
{ name = "sphinx-rtd-theme", marker = "extra == 'docs'", specifier = ">=1.0.0" },
|
|
||||||
{ name = "tree-sitter", specifier = ">=0.20.0" },
|
|
||||||
{ name = "tree-sitter-c-sharp", specifier = ">=0.20.0" },
|
|
||||||
{ name = "tree-sitter-java", specifier = ">=0.20.0" },
|
|
||||||
{ name = "tree-sitter-python", specifier = ">=0.20.0" },
|
|
||||||
{ name = "tree-sitter-typescript", specifier = ">=0.20.0" },
|
|
||||||
]
|
]
|
||||||
provides-extras = ["dev", "docs", "test"]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "asttokens"
|
name = "asttokens"
|
||||||
@@ -1585,7 +1564,7 @@ name = "importlib-metadata"
|
|||||||
version = "8.7.0"
|
version = "8.7.0"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "zipp", marker = "python_full_version < '3.10'" },
|
{ name = "zipp" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/76/66/650a33bd90f786193e4de4b3ad86ea60b53c89b669a5c7be931fac31cdb0/importlib_metadata-8.7.0.tar.gz", hash = "sha256:d13b81ad223b890aa16c5471f2ac3056cf76c5f10f82d6f9292f0b415f389000", size = 56641, upload-time = "2025-04-27T15:29:01.736Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/76/66/650a33bd90f786193e4de4b3ad86ea60b53c89b669a5c7be931fac31cdb0/importlib_metadata-8.7.0.tar.gz", hash = "sha256:d13b81ad223b890aa16c5471f2ac3056cf76c5f10f82d6f9292f0b415f389000", size = 56641, upload-time = "2025-04-27T15:29:01.736Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
@@ -2138,7 +2117,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "leann-backend-diskann"
|
name = "leann-backend-diskann"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
source = { editable = "packages/leann-backend-diskann" }
|
source = { editable = "packages/leann-backend-diskann" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "leann-core" },
|
{ name = "leann-core" },
|
||||||
@@ -2150,14 +2129,14 @@ dependencies = [
|
|||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
{ name = "leann-core", specifier = "==0.3.3" },
|
{ name = "leann-core", specifier = "==0.3.2" },
|
||||||
{ name = "numpy" },
|
{ name = "numpy" },
|
||||||
{ name = "protobuf", specifier = ">=3.19.0" },
|
{ name = "protobuf", specifier = ">=3.19.0" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "leann-backend-hnsw"
|
name = "leann-backend-hnsw"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
source = { editable = "packages/leann-backend-hnsw" }
|
source = { editable = "packages/leann-backend-hnsw" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "leann-core" },
|
{ name = "leann-core" },
|
||||||
@@ -2170,7 +2149,7 @@ dependencies = [
|
|||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
{ name = "leann-core", specifier = "==0.3.3" },
|
{ name = "leann-core", specifier = "==0.3.2" },
|
||||||
{ name = "msgpack", specifier = ">=1.0.0" },
|
{ name = "msgpack", specifier = ">=1.0.0" },
|
||||||
{ name = "numpy" },
|
{ name = "numpy" },
|
||||||
{ name = "pyzmq", specifier = ">=23.0.0" },
|
{ name = "pyzmq", specifier = ">=23.0.0" },
|
||||||
@@ -2178,7 +2157,7 @@ requires-dist = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "leann-core"
|
name = "leann-core"
|
||||||
version = "0.3.3"
|
version = "0.3.2"
|
||||||
source = { editable = "packages/leann-core" }
|
source = { editable = "packages/leann-core" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "accelerate" },
|
{ name = "accelerate" },
|
||||||
@@ -2318,7 +2297,7 @@ test = [
|
|||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
{ name = "astchunk", editable = "packages/astchunk-leann" },
|
{ name = "astchunk", specifier = ">=0.1.0" },
|
||||||
{ name = "beautifulsoup4", marker = "extra == 'documents'", specifier = ">=4.13.0" },
|
{ name = "beautifulsoup4", marker = "extra == 'documents'", specifier = ">=4.13.0" },
|
||||||
{ name = "black", marker = "extra == 'dev'", specifier = ">=23.0" },
|
{ name = "black", marker = "extra == 'dev'", specifier = ">=23.0" },
|
||||||
{ name = "boto3" },
|
{ name = "boto3" },
|
||||||
|
|||||||
Reference in New Issue
Block a user