- Convert List[str] to list[str], Dict[str, Any] to dict[str, Any], etc.
- Use ruff --unsafe-fixes to automatically apply all type annotation updates
- Remove deprecated typing imports (List, Dict, Tuple) where no longer needed
- Keep Optional[str] syntax (union operator | not supported in Python 3.9)
Now all type annotations are Python 3.9 compatible with modern builtin generics.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Format diskann_backend.py and conftest.py according to ruff standards
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit provides a minimal, focused fix for CI hanging issues by addressing the root causes:
**Key Changes:**
1. **ZMQ Resource Management:**
- Remove `context.term()` calls that were causing hangs
- Add `socket.setsockopt(zmq.LINGER, 0)` to prevent blocking on close
- Keep socket operations simple with default timeouts (no artificial limits)
2. **Process Cleanup:**
- Add timeout (1s) to final `process.wait()` in embedding server manager
- Prevent infinite waiting that was causing CI hangs
3. **Resource Cleanup Methods:**
- Add simple `cleanup()` methods to searchers and API classes
- Focus on C++ object destruction for DiskANN backend
- Avoid complex cleanup logic that could introduce new issues
4. **Basic Test Safety:**
- Simple pytest-timeout configuration (300s)
- Basic test session cleanup using psutil
- Minimal conftest.py without complex logic
**Philosophy:**
This solution avoids the complex multi-layered fixes from the previous PR chain.
Instead, it targets the specific root causes:
- ZMQ context termination blocking
- Process wait() without timeout
- C++ resource leaks in backends
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace 'int | None' with 'Optional[int]' everywhere
- Replace 'subprocess.Popen | None' with 'Optional[subprocess.Popen]'
- Add Optional import to all affected files
- Update ruff target-version from py310 to py39
- The '|' syntax for Union types was introduced in Python 3.10 (PEP 604)
Fixes TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
- Add intelligent memory calculation based on data size and system specs
- search_memory_maximum: 1/10 of embedding size (controls PQ compression)
- build_memory_maximum: 50% of available RAM (controls sharding)
- Provides optimal balance between performance and memory usage
- Automatic fallback to default values if parameters are explicitly provided
* fix: auto-detect normalized embeddings and use cosine distance
- Add automatic detection for normalized embedding models (OpenAI, Voyage AI, Cohere)
- Automatically set distance_metric='cosine' for normalized embeddings
- Add warnings when using non-optimal distance metrics
- Implement manual L2 normalization in HNSW backend (custom Faiss build lacks normalize_L2)
- Fix DiskANN zmq_port compatibility with lazy loading strategy
- Add documentation for normalized embeddings feature
This fixes the low accuracy issue when using OpenAI text-embedding-3-small model with default MIPS metric.
* style: format
- Fix ambiguous fullwidth characters (commas, parentheses) in strings and comments
- Replace Chinese comments with English equivalents
- Fix unused imports with proper noqa annotations for intentional imports
- Fix bare except clauses with specific exception types
- Fix redefined variables and undefined names
- Add ruff noqa annotations for generated protobuf files
- Add lint and format check to GitHub Actions CI pipeline
* chore: shorter build time
* chore: update faiss
* fix: no longger do embedding server reuse
* fix: do not reuse emb_server and close it properly
* feat: cli tool
* feat: cli more args
* fix: same embedding logic
* fix: diskann zmq port and passages
* feat: auto discovery of packages and fix passage gen for diskann
* docs: embedding pruning
* refactor: passage structure
* feat: reproducible research datas, rpj_wiki & dpr
* refactor: chat and base searcher
* feat: chat on mps
