- Comment out llama-index-readers-docling and llama-index-node-parser-docling
- These packages require Python >= 3.10 and were causing CI failures on Python 3.9
- Regenerate uv.lock file to resolve dependency conflicts
* fix: auto-detect normalized embeddings and use cosine distance
- Add automatic detection for normalized embedding models (OpenAI, Voyage AI, Cohere)
- Automatically set distance_metric='cosine' for normalized embeddings
- Add warnings when using non-optimal distance metrics
- Implement manual L2 normalization in HNSW backend (custom Faiss build lacks normalize_L2)
- Fix DiskANN zmq_port compatibility with lazy loading strategy
- Add documentation for normalized embeddings feature
This fixes the low accuracy issue when using OpenAI text-embedding-3-small model with default MIPS metric.
* style: format
- Add pre-commit configuration with ruff and black
- Fix lint CI job to use uv tool install instead of sync
- Add essential LlamaIndex dependencies to leann-core
Co-Authored-By: Yichuan Wang <73766326+yichuan-w@users.noreply.github.com>
- Fix ambiguous fullwidth characters (commas, parentheses) in strings and comments
- Replace Chinese comments with English equivalents
- Fix unused imports with proper noqa annotations for intentional imports
- Fix bare except clauses with specific exception types
- Fix redefined variables and undefined names
- Add ruff noqa annotations for generated protobuf files
- Add lint and format check to GitHub Actions CI pipeline
* chore: shorter build time
* chore: update faiss
* fix: no longger do embedding server reuse
* fix: do not reuse emb_server and close it properly
* feat: cli tool
* feat: cli more args
* fix: same embedding logic
* fix: diskann zmq port and passages
* feat: auto discovery of packages and fix passage gen for diskann
* docs: embedding pruning
* refactor: passage structure
* feat: reproducible research datas, rpj_wiki & dpr
* refactor: chat and base searcher
* feat: chat on mps
