4671ed9b36
* fix: auto-detect normalized embeddings and use cosine distance - Add automatic detection for normalized embedding models (OpenAI, Voyage AI, Cohere) - Automatically set distance_metric='cosine' for normalized embeddings - Add warnings when using non-optimal distance metrics - Implement manual L2 normalization in HNSW backend (custom Faiss build lacks normalize_L2) - Fix DiskANN zmq_port compatibility with lazy loading strategy - Add documentation for normalized embeddings feature This fixes the low accuracy issue when using OpenAI text-embedding-3-small model with default MIPS metric. * style: format * feat: add OpenAI embeddings support to google_history_reader_leann.py - Add --embedding-model and --embedding-mode arguments - Support automatic detection of normalized embeddings - Works correctly with cosine distance for OpenAI embeddings * feat: add --use-existing-index option to google_history_reader_leann.py - Allow using existing index without rebuilding - Useful for testing pre-built indices * fix: Improve OpenAI embeddings handling in HNSW backend * fix: improve macOS C++ compatibility and add CI tests * refactor: improve test structure and fix main_cli example - Move pytest configuration from pytest.ini to pyproject.toml - Remove unnecessary run_tests.py script (use test extras instead) - Fix main_cli_example.py to properly use command line arguments for LLM config - Add test_readme_examples.py to test code examples from README - Refactor tests to use pytest fixtures and parametrization - Update test documentation to reflect new structure - Set proper environment variables in CI for test execution * fix: add --distance-metric support to DiskANN embedding server and remove obsolete macOS ABI test markers - Add --distance-metric parameter to diskann_embedding_server.py for consistency with other backends - Remove pytest.skip and pytest.xfail markers for macOS C++ ABI issues as they have been fixed - Fix test assertions to handle SearchResult objects correctly - All tests now pass on macOS with the C++ ABI compatibility fixes * chore: update lock file with test dependencies * docs: remove obsolete C++ ABI compatibility warnings - Remove outdated macOS C++ compatibility warnings from README - Simplify CI workflow by removing macOS-specific failure handling - All tests now pass consistently on macOS after ABI fixes * fix: update macOS deployment target for DiskANN to 13.3 - DiskANN uses sgesdd_ LAPACK function which is only available on macOS 13.3+ - Update MACOSX_DEPLOYMENT_TARGET from 11.0 to 13.3 for DiskANN builds - This fixes the compilation error on GitHub Actions macOS runners * fix: align Python version requirements to 3.9 - Update root project to support Python 3.9, matching subpackages - Restore macOS Python 3.9 support in CI - This fixes the CI failure for Python 3.9 environments * fix: handle MPS memory issues in CI tests - Use smaller MiniLM-L6-v2 model (384 dimensions) for README tests in CI - Skip other memory-intensive tests in CI environment - Add minimal CI tests that don't require model loading - Set CI environment variable and disable MPS fallback - Ensure README examples always run correctly in CI * fix: remove Python 3.10+ dependencies for compatibility - Comment out llama-index-readers-docling and llama-index-node-parser-docling - These packages require Python >= 3.10 and were causing CI failures on Python 3.9 - Regenerate uv.lock file to resolve dependency conflicts * fix: use virtual environment in CI instead of system packages - uv-managed Python environments don't allow --system installs - Create and activate virtual environment before installing packages - Update all CI steps to use the virtual environment * add some env in ci * fix: use --find-links to install platform-specific wheels - Let uv automatically select the correct wheel for the current platform - Fixes error when trying to install macOS wheels on Linux - Simplifies the installation logic * fix: disable OpenMP parallelism in CI to avoid libomp crashes - Set OMP_NUM_THREADS=1 to avoid OpenMP thread synchronization issues - Set MKL_NUM_THREADS=1 for single-threaded MKL operations - This prevents segfaults in LayerNorm on macOS CI runners - Addresses the libomp compatibility issues with PyTorch on Apple Silicon * skip several macos test because strange issue on ci --------- Co-authored-by: yichuan520030910320 <yichuan_wang@berkeley.edu>
64 lines
2.4 KiB
CMake
64 lines
2.4 KiB
CMake
cmake_minimum_required(VERSION 3.24)
|
|
project(leann_backend_hnsw_wrapper)
|
|
set(CMAKE_C_COMPILER_WORKS 1)
|
|
set(CMAKE_CXX_COMPILER_WORKS 1)
|
|
|
|
# Set OpenMP path for macOS
|
|
if(APPLE)
|
|
set(OpenMP_C_FLAGS "-Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include")
|
|
set(OpenMP_CXX_FLAGS "-Xpreprocessor -fopenmp -I/opt/homebrew/opt/libomp/include")
|
|
set(OpenMP_C_LIB_NAMES "omp")
|
|
set(OpenMP_CXX_LIB_NAMES "omp")
|
|
set(OpenMP_omp_LIBRARY "/opt/homebrew/opt/libomp/lib/libomp.dylib")
|
|
|
|
# Force use of system libc++ to avoid version mismatch
|
|
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++")
|
|
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -stdlib=libc++")
|
|
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -stdlib=libc++")
|
|
|
|
# Set minimum macOS version for better compatibility
|
|
set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum macOS version")
|
|
endif()
|
|
|
|
# Use system ZeroMQ instead of building from source
|
|
find_package(PkgConfig REQUIRED)
|
|
pkg_check_modules(ZMQ REQUIRED libzmq)
|
|
|
|
# Add cppzmq headers
|
|
include_directories(third_party/cppzmq)
|
|
|
|
# Configure msgpack-c - disable boost dependency
|
|
set(MSGPACK_USE_BOOST OFF CACHE BOOL "" FORCE)
|
|
add_compile_definitions(MSGPACK_NO_BOOST)
|
|
include_directories(third_party/msgpack-c/include)
|
|
|
|
# Faiss configuration - streamlined build
|
|
set(FAISS_ENABLE_PYTHON ON CACHE BOOL "" FORCE)
|
|
set(FAISS_ENABLE_GPU OFF CACHE BOOL "" FORCE)
|
|
set(FAISS_ENABLE_EXTRAS OFF CACHE BOOL "" FORCE)
|
|
set(BUILD_TESTING OFF CACHE BOOL "" FORCE)
|
|
set(FAISS_ENABLE_C_API OFF CACHE BOOL "" FORCE)
|
|
set(FAISS_OPT_LEVEL "generic" CACHE STRING "" FORCE)
|
|
|
|
# Disable additional SIMD versions to speed up compilation
|
|
set(FAISS_ENABLE_AVX2 OFF CACHE BOOL "" FORCE)
|
|
set(FAISS_ENABLE_AVX512 OFF CACHE BOOL "" FORCE)
|
|
|
|
# Additional optimization options from INSTALL.md
|
|
set(CMAKE_BUILD_TYPE "Release" CACHE STRING "" FORCE)
|
|
set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) # Static library is faster to build
|
|
|
|
# Avoid building demos and benchmarks
|
|
set(BUILD_DEMOS OFF CACHE BOOL "" FORCE)
|
|
set(BUILD_BENCHS OFF CACHE BOOL "" FORCE)
|
|
|
|
# NEW: Tell Faiss to only build the generic version
|
|
set(FAISS_BUILD_GENERIC ON CACHE BOOL "" FORCE)
|
|
set(FAISS_BUILD_AVX2 OFF CACHE BOOL "" FORCE)
|
|
set(FAISS_BUILD_AVX512 OFF CACHE BOOL "" FORCE)
|
|
|
|
# IMPORTANT: Disable building AVX versions to speed up compilation
|
|
set(FAISS_BUILD_AVX_VERSIONS OFF CACHE BOOL "" FORCE)
|
|
|
|
add_subdirectory(third_party/faiss)
|