Infra: add bindings/python into root workspace#2506
Conversation
kevinjqliu
left a comment
There was a problem hiding this comment.
i also made some refactor to .github/actions/overwrite-package-version/action.yml, .github/workflows/release_python.yml, and .github/workflows/release_python_nightly.yml to align Cargo workspace version with pyproject toml version
blackmwk
left a comment
There was a problem hiding this comment.
Thanks @kevinjqliu for this pr!
| anyhow = "1.0.72" | ||
| apache-avro = { version = "0.21", features = ["zstandard"] } | ||
| array-init = "2" | ||
| arrow = { version = "58", features = ["pyarrow", "chrono-tz"] } |
There was a problem hiding this comment.
This adds all arrow crates, we should follow conventions for other crates, e.g only add necessary sub arrow crates.
There was a problem hiding this comment.
done, moved all features to crate (bindings/python/Cargo.toml)
| zeroize = "1.7" | ||
| zstd = "0.13.3" | ||
|
|
||
| [profile.py-release] |
There was a problem hiding this comment.
Does this have to be in root workspace? If not, I prefer to keep it in pyiceberg_core crate.
There was a problem hiding this comment.
done, i dont like this here either. moved to pyproject.toml, verified locally that the configs are applied when building wheels
Which issue does this PR close?
What changes are included in this PR?
Migrates
bindings/pythonfrom a standalone, excluded crate into the root Cargo workspace.This lets
pyiceberg_core_rustinherit workspace versions, dependency declarations, package metadata, and the root lockfile instead of maintaining a parallel Rust dependency setup underbindings/python.Changes
bindings/pythonas a workspace member.bindings/python/Cargo.tomlto inherit shared package metadata and dependencies with{ workspace = true }.bindings/python/Cargo.lock; Python bindings now use the rootCargo.lock.pyo3/extension-modulefrom workspace Cargo features and requirematurin >= 1.9.4, so maturin sets extension-module build mode only for Python extension builds.py-releaseCargo profile for Python wheel builds and configure maturin/release workflows to use it.check-rustCI job; Rust validation is covered by the main workspace CI.cargo metadata.overwrite-package-versionto accept a PEP 440versioninput and write it topyproject.toml.pyproject.tomlinstead of the workspaceCargo.toml, since.devand-rc.Nversions are not valid Cargo semver and can break workspace dependency resolution.Notes
Plain Cargo workspace builds now include
pyiceberg_core_rust, while actual Python wheels continue to be built through maturin. The Python-specific release optimization settings are isolated in thepy-releaseprofile so normal Rust release builds are not affected.make buildnow buildsbindings/pythonas part of the workspace because it runscargo build --all-targets --all-features --workspace. This is intentional for now: after removingpyo3/extension-module, the Python Rust crate can be built by plain Cargo, so the main workspace build can validate it.A possible follow-up improvement is to exclude
bindings/pythonfrom the default root build/test path withworkspace.default-members, while keeping full validation incargo check --workspace, clippy, and the maturin-based Python CI/release workflows. That would avoid building the Python cdylib in normal Rust builds while still keeping workspace dependency inheritance and CI coverage.Are these changes tested?
Yes
Tested affected workflows on my fork:
.github/workflows/release_python.yml, https://github.com/kevinjqliu/iceberg-rust/actions/runs/26416510024.github/workflows/release_python_nightly.yml, https://github.com/kevinjqliu/iceberg-rust/actions/runs/26416510760