diff --git a/.github/workflows/build-cibuildwheel.yml b/.github/workflows/build-cibuildwheel.yml index e129ebd..7e219c5 100644 --- a/.github/workflows/build-cibuildwheel.yml +++ b/.github/workflows/build-cibuildwheel.yml @@ -46,14 +46,15 @@ jobs: CIBW_BUILD: cp39-* cp310-* cp311-* cp312-* cp313-* CIBW_SKIP: "*-win32 *-manylinux_i686 pp* *musllinux*" - # Use manylinux_2_28 for better compatibility - CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28 - CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28 + # 使用manylinux2014生成兼容性更好的wheels + CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014 + CIBW_MANYLINUX_AARCH64_IMAGE: manylinux2014 - # Install dependencies before building + # Linux dependencies - 使用yum因为manylinux2014基于CentOS 7 CIBW_BEFORE_ALL_LINUX: | - dnf install -y epel-release - dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake + yum install -y epel-release + yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3 + ln -sf /usr/bin/cmake3 /usr/bin/cmake CIBW_BEFORE_ALL_MACOS: | brew install boost zeromq openblas cmake @@ -77,12 +78,13 @@ jobs: CIBW_BUILD: cp39-* cp310-* cp311-* cp312-* cp313-* CIBW_SKIP: "*-win32 *-manylinux_i686 pp* *musllinux*" - CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28 - CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28 + CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014 + CIBW_MANYLINUX_AARCH64_IMAGE: manylinux2014 CIBW_BEFORE_ALL_LINUX: | - dnf install -y epel-release - dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake + yum install -y epel-release + yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3 + ln -sf /usr/bin/cmake3 /usr/bin/cmake CIBW_BEFORE_ALL_MACOS: | brew install boost zeromq openblas cmake diff --git a/MANYLINUX_BUILD_STRATEGY.md b/MANYLINUX_BUILD_STRATEGY.md index 11d39c7..58684e2 100644 --- a/MANYLINUX_BUILD_STRATEGY.md +++ b/MANYLINUX_BUILD_STRATEGY.md @@ -1,77 +1,50 @@ # Manylinux Build Strategy -## 问题概述 +## Problem +Google Colab requires wheels compatible with `manylinux_2_35_x86_64` or earlier. Our previous builds were producing `manylinux_2_39_x86_64` wheels, which are incompatible. -Colab和许多Linux环境需要`manylinux`兼容的wheels，而不是特定于某个Linux发行版的wheels。我们之前的尝试过于复杂，试图手动处理所有的依赖关系。 +## Solution +We're using `cibuildwheel` with `manylinux2014` images to build wheels that are compatible with a wide range of Linux distributions, including Google Colab. -## 新策略 +### Key Changes -基于社区最佳实践和其他成功项目的经验，我们采用了以下简化方案： +1. **cibuildwheel Configuration** + - Using `manylinux2014` images (provides `manylinux_2_17` compatibility) + - Using `yum` package manager (CentOS 7 based) + - Installing `cmake3` and creating symlink for compatibility -### 1. 使用 cibuildwheel +2. **Build Matrix** + - Python versions: 3.9, 3.10, 3.11, 3.12, 3.13 + - Platforms: Linux (x86_64), macOS + - No Windows support (not required) -`cibuildwheel`是PyPA推荐的构建多平台wheels的工具，它： -- 自动处理manylinux环境设置 -- 管理不同Python版本的构建 -- 提供标准化的测试框架 +3. **Dependencies** + - Linux: gcc-c++, boost-devel, zeromq-devel, openblas-devel, cmake3 + - macOS: boost, zeromq, openblas, cmake (via Homebrew) -### 2. 使用 manylinux_2_28 +4. **Environment Variables** + - `CMAKE_BUILD_PARALLEL_LEVEL=8`: Speed up builds + - `Python_FIND_VIRTUALENV=ONLY`: Help CMake find Python in cibuildwheel env + - `Python3_FIND_VIRTUALENV=ONLY`: Alternative variable for compatibility -- 从`manylinux2014`升级到`manylinux_2_28` -- 提供更现代的基础库 -- 使用`dnf`而不是`yum` -- 更好地支持现代Python版本 +## Testing Strategy -### 3. CMake配置 +1. **CI Pipeline**: `test-manylinux.yml` + - Triggers on PR to main, manual dispatch, or push to `fix/manylinux-*` branches + - Builds wheels using cibuildwheel + - Tests installation on Ubuntu 22.04 (simulating Colab) -在`pyproject.toml`中添加： -```toml -[tool.scikit-build.cmake.define] -Python_FIND_VIRTUALENV = "ONLY" -Python3_FIND_VIRTUALENV = "ONLY" -``` +2. **Local Testing** + ```bash + # Download built wheels + # Test in fresh environment + python -m venv test_env + source test_env/bin/activate + pip install leann_core-*.whl leann_backend_hnsw-*manylinux*.whl leann-*.whl + python -c "from leann import LeannBuilder; print('Success!')" + ``` -这帮助CMake在cibuildwheel的虚拟环境中正确找到Python。 - -### 4. 简化的依赖安装 - -只安装必要的系统依赖： -- `gcc-c++` -- `boost-devel` -- `zeromq-devel` -- `openblas-devel` -- `cmake` - -## 测试策略 - -1. **手动触发**: 使用`workflow_dispatch`在GitHub Actions上测试 -2. **自动测试**: 在`fix/manylinux-*`分支上推送时自动运行 -3. **PR测试**: 当PR修改相关文件时自动测试 - -## 如何使用 - -### 在本地测试 -```bash -# 安装cibuildwheel -pip install cibuildwheel - -# 构建wheels -cibuildwheel --platform linux packages/leann-backend-hnsw -``` - -### 在GitHub Actions测试 -1. 推送到`fix/manylinux-compatibility`分支 -2. 或手动触发"Test Manylinux Build"工作流 - -## 下一步 - -1. **监控CI运行结果** -2. **根据错误调整依赖** -3. **测试生成的wheels在Colab的兼容性** -4. **如果成功，将更改合并到main** - -## 参考 - -- [cibuildwheel文档](https://cibuildwheel.readthedocs.io/) -- [manylinux规范](https://github.com/pypa/manylinux) -- [scikit-build-core文档](https://scikit-build-core.readthedocs.io/) \ No newline at end of file +## References +- [cibuildwheel documentation](https://cibuildwheel.readthedocs.io/) +- [manylinux standards](https://github.com/pypa/manylinux) +- [PEP 599 - manylinux2014](https://peps.python.org/pep-0599/) \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index ac58cf1..b241169 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -65,15 +65,16 @@ leann-backend-hnsw = { path = "packages/leann-backend-hnsw", editable = true } # 跳过32位和PyPy构建 skip = "*-win32 *-manylinux_i686 pp* *musllinux*" -# 使用更新的manylinux镜像以获得更好的兼容性 -manylinux-x86_64-image = "manylinux_2_28" -manylinux-aarch64-image = "manylinux_2_28" +# 使用manylinux2014以获得最大兼容性（支持GLIBC 2.17） +manylinux-x86_64-image = "manylinux2014" +manylinux-aarch64-image = "manylinux2014" # Linux系统依赖 [tool.cibuildwheel.linux] before-all = """ - dnf install -y epel-release - dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake + yum install -y epel-release + yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3 + ln -sf /usr/bin/cmake3 /usr/bin/cmake """ # macOS系统依赖