Commit Graph

278 Commits

Author SHA1 Message Date
Andy Lee
971653fa1a upgrade: switch from manylinux2014 to manylinux_2_35
- Use manylinux_2_35 (GLIBC 2.35) instead of manylinux2014 (GLIBC 2.17)
- Still compatible with Google Colab (requires ≤2.35)
- Benefits: newer toolchain, better performance, modern C++ features
- Switch from yum to dnf package manager
- Remove pyzmq version cap as manylinux_2_35 has newer ZeroMQ
- Update documentation to reflect the change
2025-07-25 13:40:07 -07:00
Andy Lee
02672c040d update: DiskANN submodule to support both build environments
- Point to fix/python-finding-compatibility branch
- Support both Development and Development.Module for Python finding
- Fixes compatibility with both standard and manylinux builds
2025-07-25 13:33:36 -07:00
Andy Lee
f55108feda fix: resolve CI conflicts and Python finding issues
- Remove duplicate ci-cibuildwheel.yml workflow to avoid confusion
- Fix DiskANN CMakeLists.txt to support both standard and manylinux builds
- Try Development.Module first (manylinux), fallback to Development (standard)
- Keep build-reusable.yml and build-cibuildwheel.yml as separate workflows
- They build different wheel types and are selected at release time
2025-07-25 13:21:22 -07:00
Andy Lee
74d485c908 fix: remove Chinese comments from code
- Replace all Chinese comments with English
- Ensure code is internationalization-friendly
2025-07-25 12:55:28 -07:00
Andy Lee
d1fefb6378 fix: correct DiskANN subdirectory path
- Change from src/third_party/DiskANN to third_party/DiskANN
- This was causing CMake to fail finding the subdirectory
2025-07-25 12:51:47 -07:00
Andy Lee
732384f4f8 fix: improve Python finding for DiskANN in manylinux environment
- Add explicit Python finding logic to DiskANN CMakeLists.txt
- Change Development to Development.Module to avoid Embed requirement
- Pass Python variables through CMake cache to submodules
2025-07-25 12:22:27 -07:00
Andy Lee
ae38e10d1b chore: focus on Linux builds for Colab compatibility
- Remove macOS from build matrix to simplify CI
- Keep macOS configurations for future reference but they won't be used
- This speeds up CI and focuses on the primary target (Colab/Linux)
2025-07-25 12:20:32 -07:00
Andy Lee
ca0fd88934 chore: clean up temporary test scripts 2025-07-25 11:53:23 -07:00
Andy Lee
3c8d32f156 fix: prevent pyzmq compilation during tests
- Cap pyzmq version to <27 for manylinux2014 compatibility
- Pre-install pyzmq binary wheel before tests using CIBW_BEFORE_TEST
- Force pip to use only binary wheels with --only-binary :all:
2025-07-25 11:35:31 -07:00
Andy Lee
b8ff00fc6a fix: address macOS deployment target and pyzmq compilation issues
- Set MACOSX_DEPLOYMENT_TARGET=11.0 for macOS builds
- Add pyzmq to test requirements to use pre-built wheels
- Configure deployment target in both workflow and pyproject.toml
- Skip ARM64 tests on GitHub Actions to avoid cross-compilation issues
2025-07-25 11:13:51 -07:00
Andy Lee
3c836766f8 fix: update faiss submodule to fix-python-finding-manylinux branch 2025-07-25 10:55:28 -07:00
Andy Lee
b4a1dfb9c7 fix: update faiss submodule pointer with Python finding fixes 2025-07-25 10:53:05 -07:00
Andy Lee
a4d66e95d8 fix: improve Python finding for Faiss in manylinux environment
- Use Development.Module instead of Development component
- Find Python in main Faiss CMakeLists.txt before python subdirectory
- Add debug output to trace Python variable passing
- Set Python_FIND_VIRTUALENV=ONLY for Faiss
2025-07-25 10:52:34 -07:00
Andy Lee
cf58b3e31b fix: re-enable Faiss Python bindings and improve Python finding
- Re-enable FAISS_ENABLE_PYTHON since we need the Python bindings
- Use Development.Module component for better compatibility
- Pass Python information to Faiss through CMake cache variables
- Add CMAKE_PREFIX_PATH= to help CMake find Python in manylinux
2025-07-25 10:45:07 -07:00
Andy Lee
e9c2ca7936 fix: remove cmake.verbose from scikit-build config
- cmake.verbose is not allowed when minimum-version is set to 0.10 or higher
- This was causing build failures in cibuildwheel
2025-07-25 10:37:06 -07:00
Andy Lee
dab154a77b fix: disable Faiss Python bindings to avoid CMake Python finding issues
- Set FAISS_ENABLE_PYTHON to OFF since we use our own Cython bindings
- This avoids the CMake Python finding issues in manylinux environments
- Simplify CMakeLists.txt by removing unnecessary Python finding logic
- Keep swig installation for other potential uses
2025-07-25 10:33:31 -07:00
Andy Lee
13413dfae5 fix: improve Python detection in manylinux environment
- Modify faiss CMakeLists.txt to try both FindPython and FindPython3
- Add scikit-build configuration to help with Python detection
- Simplify Linux environment variables in cibuildwheel
- Add CMake Python detection before faiss configuration
2025-07-25 10:27:04 -07:00
Andy Lee
0543cc9816 fix: add missing dependencies for CI builds
- Add libomp for macOS to fix OpenMP linking error
- Add python3-devel for Linux (though may not be needed in manylinux)
- Install numpy before building to satisfy faiss CMake requirements
- Add before-build configuration to cibuildwheel
2025-07-25 10:20:02 -07:00
Andy Lee
fb53ed9a0e fix: use manylinux2014 for Colab compatibility
- Switch from manylinux_2_28 to manylinux2014 (provides manylinux_2_17)
- This should produce wheels compatible with manylinux_2_35_x86_64 requirement
- Update package manager from dnf to yum for CentOS 7
- Use cmake3 with symlink for compatibility
2025-07-25 10:15:05 -07:00
Andy Lee
015f43733a fix: adjust configuration for Colab compatibility
- Remove Windows support as not needed
- Move Python finding hints to cibuildwheel environment variables
- Keep pyproject.toml clean to avoid breaking normal builds
- Target manylinux_2_28 for better Colab compatibility
2025-07-25 10:09:21 -07:00
Andy Lee
2957c8bf5a docs: add manylinux build strategy documentation 2025-07-25 10:03:57 -07:00
Andy Lee
a73194c3f6 feat: simplify build using cibuildwheel with standard configuration
- Add Python_FIND_VIRTUALENV hints to pyproject.toml for CMake
- Create standardized cibuildwheel workflow using manylinux_2_28
- Simplify system dependency installation
- Add global cibuildwheel configuration in root pyproject.toml
- Create streamlined test workflow for manylinux compatibility
2025-07-25 10:03:23 -07:00
Andy Lee
5eb893c62b ci: add Python 3.13 support to build matrix 2025-07-25 09:53:36 -07:00
yichuan520030910320
d91ce2e94d readme 2025-07-25 02:19:54 -07:00
yichuan520030910320
5c2ff8a641 clean research stuff 2025-07-25 02:14:15 -07:00
yichuan520030910320
d4f474c9b7 update broken link 2025-07-25 02:13:22 -07:00
yichuan520030910320
170f7644e9 simplify readme 2025-07-25 02:11:02 -07:00
yichuan520030910320
cd8b970eff Merge branch 'main' of https://github.com/yichuan-w/LEANN 2025-07-25 01:45:57 -07:00
yichuan520030910320
52153bbb69 update faiss compare 2025-07-25 01:45:50 -07:00
GitHub Actions
e1ae087207 chore: release v0.1.8 2025-07-25 08:24:40 +00:00
Andy Lee
48c5e12ac1 fix: use absolute path for passages_file to prevent FileNotFoundError
When embedding server is launched as a subprocess, it may run in a different
working directory. Using absolute paths ensures the server can always find
the metadata file regardless of where it's launched from.
2025-07-25 01:23:47 -07:00
yichuan520030910320
f8b5c97190 Merge branch 'main' of https://github.com/yichuan-w/LEANN 2025-07-25 00:37:33 -07:00
yichuan520030910320
d038c81b8b update benchmard section 2025-07-25 00:37:27 -07:00
Andy Lee
29cbbbd0d6 fix: resolve libzmq pkg-config issues in manylinux containers
- Add gcc-c++ and cmake to dependencies
- Create libzmq.pc file if missing (CentOS 7 issue)
- Set PKG_CONFIG_PATH through CIBW_ENVIRONMENT_LINUX
- Add protobuf-devel to ensure all headers are available
- Fix shell variable escaping in heredoc
2025-07-25 00:35:52 -07:00
Andy Lee
179f30bc36 fix: improve system dependency installation in manylinux containers
- Add yum cache cleaning and updating
- Make package installations more resilient with fallbacks
- Use pkgconfig instead of pkg-config (CentOS 7 naming)
- Handle optional packages that might not be available
- Add error handling for package installation failures
2025-07-25 00:30:29 -07:00
Andy Lee
c4a0a68581 fix: handle pure Python packages in cibuildwheel workflow
- Build pure Python packages (leann-core, leann) with standard build tool
- Only use cibuildwheel for C extension packages (leann-backend-hnsw, leann-backend-diskann)
- Build pure Python packages only once on ubuntu-latest
- Add Python setup for building pure packages
- Add package listing step for debugging
2025-07-25 00:26:15 -07:00
Andy Lee
5c836ad08e fix: handle git dubious ownership error in manylinux containers
- Add multiple safe.directory configurations to cover different possible paths
- This fixes 'detected dubious ownership in repository' error
- Ensures git works properly in manylinux2014 containers
2025-07-25 00:22:01 -07:00
Andy Lee
673fd9b7cd fix: upgrade to actions v4 and handle manylinux2014 compatibility
- Upgrade all GitHub Actions to v4 (v3 is deprecated)
- Use manual git checkout in manylinux2014 containers to avoid Node.js issues
- Update artifact naming to ensure uniqueness (required by v4)
- Add fail-fast: false to build strategies
- This maintains manylinux2014 compatibility while using latest actions
2025-07-25 00:20:21 -07:00
Andy Lee
84b24b233d feat: add cibuildwheel option to release workflow
- Add optional use_cibuildwheel parameter to release workflow
- Create separate CI workflow for testing cibuildwheel
- Support conditional build workflow selection in release process
- This allows building wheels compatible with Google Colab and older systems
- Maintains backward compatibility with existing build process
2025-07-25 00:16:08 -07:00
Andy Lee
499cdd7822 feat: add cibuildwheel workflow for better platform compatibility
- Use cibuildwheel for professional wheel building
- Specifically use manylinux2014 for Google Colab compatibility
- Supports Python 3.9-3.12 on Linux and macOS
- Handles monorepo structure with separate builds per package
- Includes basic import tests for each package
- This should resolve compatibility issues with older systems like Google Colab
2025-07-25 00:16:08 -07:00
yichuan520030910320
800d4cf111 Merge branch 'main' of https://github.com/yichuan-w/LEANN 2025-07-25 00:12:47 -07:00
yichuan520030910320
b6d43f5fd9 add gif 2025-07-25 00:12:35 -07:00
Andy Lee
3603cd5034 fix: downgrade GitHub Actions versions for manylinux2014 compatibility
- Use actions/checkout@v3 instead of v4 (Node.js 16 vs 20)
- Use actions/setup-python@v4 instead of v5
- Use actions/upload-artifact@v3 and download-artifact@v3
- This fixes GLIBC version errors in manylinux2014 containers
- manylinux2014 (CentOS 7) has glibc 2.17 but Node.js 20 needs 2.25+
2025-07-25 00:12:05 -07:00
Andy Lee
6df7893173 feat: use manylinux2014 containers for better Linux compatibility
- Add manylinux2014 Docker containers for Linux builds
- This will generate wheels compatible with older Linux systems (CentOS 7+, Ubuntu 16.04+)
- Separate build logic for container vs regular environments
- Install appropriate system dependencies for yum-based manylinux environment
- Use pip instead of uv in containers for better compatibility
- Fix Python version format for manylinux container paths
2025-07-25 00:08:42 -07:00
GitHub Actions
e64b599276 chore: release v0.1.7 2025-07-25 04:47:57 +00:00
Andy Lee
2dd59c4ba1 fix: let auditwheel auto-detect manylinux platform tag
- Remove --plat manylinux2014_x86_64 flag that was causing build failures
- Let auditwheel automatically determine the appropriate manylinux tag
- Add auditwheel show command to display compatibility info
- This fixes the 'too-recent versioned symbols' error
v0.1.7
2025-07-24 21:44:15 -07:00
GitHub Actions
166986d5e6 chore: release v0.1.6 2025-07-25 04:30:07 +00:00
Andy Lee
a6aec68f32 fix: use manylinux2014 for better Linux compatibility
- Change auditwheel --plat to manylinux2014_x86_64
- This ensures wheels work on Ubuntu 16.04+ instead of requiring 24.04+
- Fixes compatibility issues for users on Ubuntu 22.04 and similar systems
2025-07-24 21:26:28 -07:00
GitHub Actions
ed27a127d5 chore: release v0.1.5 2025-07-25 04:00:54 +00:00
Andy Lee
d8b4ea7564 fix: add write permissions for GitHub Actions to push commits v0.1.5 2025-07-24 20:55:24 -07:00