TBNilles/LEANN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: use manylinux2014 for Colab compatibility

Browse Source 
- Switch from manylinux_2_28 to manylinux2014 (provides manylinux_2_17)
- This should produce wheels compatible with manylinux_2_35_x86_64 requirement
- Update package manager from dnf to yum for CentOS 7
- Use cmake3 with symlink for compatibility
This commit is contained in:
Andy Lee
2025-07-25 10:15:05 -07:00
parent 015f43733a
commit fb53ed9a0e
3 changed files with 56 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
.github/workflows/build-cibuildwheel.yml vendored
View File

@@ -46,14 +46,15 @@ jobs:
CIBW_BUILD: cp39-* cp310-* cp311-* cp312-* cp313-*
CIBW_SKIP: "*-win32 *-manylinux_i686 pp* *musllinux*"
# Use manylinux_2_28 for better compatibility
CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
# 使用manylinux2014生成兼容性更好的wheels
CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux2014
# Install dependencies before building
# Linux dependencies - 使用yum因为manylinux2014基于CentOS 7
CIBW_BEFORE_ALL_LINUX: |
dnf install -y epel-release
dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake
yum install -y epel-release
yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3
ln -sf /usr/bin/cmake3 /usr/bin/cmake
CIBW_BEFORE_ALL_MACOS: |
brew install boost zeromq openblas cmake
@@ -77,12 +78,13 @@ jobs:
CIBW_BUILD: cp39-* cp310-* cp311-* cp312-* cp313-*
CIBW_SKIP: "*-win32 *-manylinux_i686 pp* *musllinux*"
CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux2014
CIBW_BEFORE_ALL_LINUX: |
dnf install -y epel-release
dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake
yum install -y epel-release
yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3
ln -sf /usr/bin/cmake3 /usr/bin/cmake
CIBW_BEFORE_ALL_MACOS: |
brew install boost zeromq openblas cmake

103
MANYLINUX_BUILD_STRATEGY.md
View File

@@ -1,77 +1,50 @@
# Manylinux Build Strategy
## 问题概述
## Problem
Google Colab requires wheels compatible with `manylinux_2_35_x86_64` or earlier. Our previous builds were producing `manylinux_2_39_x86_64` wheels, which are incompatible.
Colab和许多Linux环境需要`manylinux`兼容的wheels，而不是特定于某个Linux发行版的wheels。我们之前的尝试过于复杂，试图手动处理所有的依赖关系。
## Solution
We're using `cibuildwheel` with `manylinux2014` images to build wheels that are compatible with a wide range of Linux distributions, including Google Colab.
## 新策略
### Key Changes
基于社区最佳实践和其他成功项目的经验，我们采用了以下简化方案：
1. **cibuildwheel Configuration**
- Using `manylinux2014` images (provides `manylinux_2_17` compatibility)
- Using `yum` package manager (CentOS 7 based)
- Installing `cmake3` and creating symlink for compatibility
### 1. 使用 cibuildwheel
2. **Build Matrix**
- Python versions: 3.9, 3.10, 3.11, 3.12, 3.13
- Platforms: Linux (x86_64), macOS
- No Windows support (not required)
`cibuildwheel`是PyPA推荐的构建多平台wheels的工具，它：
- 自动处理manylinux环境设置
- 管理不同Python版本的构建
- 提供标准化的测试框架
3. **Dependencies**
- Linux: gcc-c++, boost-devel, zeromq-devel, openblas-devel, cmake3
- macOS: boost, zeromq, openblas, cmake (via Homebrew)
### 2. 使用 manylinux_2_28
4. **Environment Variables**
- `CMAKE_BUILD_PARALLEL_LEVEL=8`: Speed up builds
- `Python_FIND_VIRTUALENV=ONLY`: Help CMake find Python in cibuildwheel env
- `Python3_FIND_VIRTUALENV=ONLY`: Alternative variable for compatibility
- 从`manylinux2014`升级到`manylinux_2_28`
- 提供更现代的基础库
- 使用`dnf`而不是`yum`
- 更好地支持现代Python版本
## Testing Strategy
### 3. CMake配置
1. **CI Pipeline**: `test-manylinux.yml`
- Triggers on PR to main, manual dispatch, or push to `fix/manylinux-*` branches
- Builds wheels using cibuildwheel
- Tests installation on Ubuntu 22.04 (simulating Colab)
在`pyproject.toml`中添加：
```toml
[tool.scikit-build.cmake.define]
Python_FIND_VIRTUALENV = "ONLY"
Python3_FIND_VIRTUALENV = "ONLY"
```
2. **Local Testing**
```bash
# Download built wheels
# Test in fresh environment
python -m venv test_env
source test_env/bin/activate
pip install leann_core-*.whl leann_backend_hnsw-*manylinux*.whl leann-*.whl
python -c "from leann import LeannBuilder; print('Success!')"
```
这帮助CMake在cibuildwheel的虚拟环境中正确找到Python。
### 4. 简化的依赖安装
只安装必要的系统依赖：
- `gcc-c++`
- `boost-devel`
- `zeromq-devel`
- `openblas-devel`
- `cmake`
## 测试策略
1. **手动触发**: 使用`workflow_dispatch`在GitHub Actions上测试
2. **自动测试**: 在`fix/manylinux-*`分支上推送时自动运行
3. **PR测试**: 当PR修改相关文件时自动测试
## 如何使用
### 在本地测试
```bash
# 安装cibuildwheel
pip install cibuildwheel
# 构建wheels
cibuildwheel --platform linux packages/leann-backend-hnsw
```
### 在GitHub Actions测试
1. 推送到`fix/manylinux-compatibility`分支
2. 或手动触发"Test Manylinux Build"工作流
## 下一步
1. **监控CI运行结果**
2. **根据错误调整依赖**
3. **测试生成的wheels在Colab的兼容性**
4. **如果成功，将更改合并到main**
## 参考
- [cibuildwheel文档](https://cibuildwheel.readthedocs.io/)
- [manylinux规范](https://github.com/pypa/manylinux)
- [scikit-build-core文档](https://scikit-build-core.readthedocs.io/)
## References
- [cibuildwheel documentation](https://cibuildwheel.readthedocs.io/)
- [manylinux standards](https://github.com/pypa/manylinux)
- [PEP 599 - manylinux2014](https://peps.python.org/pep-0599/)

11
pyproject.toml
View File

@@ -65,15 +65,16 @@ leann-backend-hnsw = { path = "packages/leann-backend-hnsw", editable = true }
# 跳过32位和PyPy构建
skip = "*-win32 *-manylinux_i686 pp* *musllinux*"
# 使用更新的manylinux镜像以获得更好的兼容性
manylinux-x86_64-image = "manylinux_2_28"
manylinux-aarch64-image = "manylinux_2_28"
# 使用manylinux2014以获得最大兼容性（支持GLIBC 2.17）
manylinux-x86_64-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"
# Linux系统依赖
[tool.cibuildwheel.linux]
before-all = """
dnf install -y epel-release
dnf install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake
yum install -y epel-release
yum install -y gcc-c++ boost-devel zeromq-devel openblas-devel cmake3
ln -sf /usr/bin/cmake3 /usr/bin/cmake
"""
# macOS系统依赖