Skip to content

Fix/python cargo lock drift - #1899

Merged
coderfender merged 3 commits into
apache:mainfrom
djanand:fix/python-cargo-lock-drift
Jul 3, 2026
Merged

Fix/python cargo lock drift#1899
coderfender merged 3 commits into
apache:mainfrom
djanand:fix/python-cargo-lock-drift

Conversation

@djanand

@djanand djanand commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #1887.

Rationale for this change

python/ is a separate Cargo workspace (excluded from the root workspace) with its own python/Cargo.lock, and it path-depends on the main ballista crates. Two gaps make that lock drift out of sync:

  1. Dependabot only covers the root workspace (directory: "/")- there is no /python entry, so dep bumps update /Cargo.lock but never python/Cargo.lock.
  2. No CI catches the drift early. rust.yml runs --locked only against the main workspace (python is excluded). A stale python lock only surfaces later and cryptically, in build.yml's maturin steps (which run with locked = true from python/pyproject.toml): error: cannot update the lock file python/Cargo.lock because --locked was passed
  3. Added an explicit least-privilege permissions: contents: read block to dependencies.yml (resolves a CodeQL finding; matches the other workflows).

Net effect: when a dep moves in a ballista crate manifest (e.g. itertools, tower-http were bumped to 0.15/0.7), python/Cargo.lock is left behind on main, and the next contributor inherits a red maturin build with an unhelpful message. This PR fixes the current drift and adds a guard so future drift is caught on the PR that introduces it.

What changes are included in this PR?

  • python/Cargo.lock - minimally reconciled with the manifests (picks up the already-published itertools 0.15.0 / tower-http 0.7.0 that flow in via the ballista path-deps; lockfile format v4 preserved). This fixes the current drift so the new guard passes from day one.
  • .github/workflows/dependencies.yml - new python-lock job that runs cargo metadata --locked in python/ and fails with an actionable message (run 'cd python && cargo update' and commit) if the lock is stale. The workflow already triggers on **/Cargo.toml / **/Cargo.lock, so both main-workspace and python changes exercise it.
  • .github/dependabot.yml - new cargo entry for directory: "/python" (mirroring the root block's manual-bump ignore list for arrow/datafusion/sqlparser), so python's direct deps stay fresh.

Are there any user-facing changes?

No runtime/API changes - this is CI/tooling only.

One contributor-facing note: dependabot bumps a transitive dep into python via the ballista path-deps (which python doesn't declare directly, so dependabot can't sync python's lock for it). Such a PR will now trip the python-lock guard and needs a one-line cd python && cargo update + commit on that branch. This isn't new breakage - that drift already broke the next python/** PR cryptically; the guard just relocates it to the PR that caused it, with a clear fix message.

No breaking changes to public APIs.

Comment thread .github/workflows/dependencies.yml Fixed
@djanand
djanand marked this pull request as draft June 26, 2026 05:53
@djanand
djanand marked this pull request as ready for review June 27, 2026 01:11
@djanand

djanand commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

This requires #1919 to be reviewed & merged sincemain has diverged

@coderfender

Copy link
Copy Markdown
Contributor

@djanand seems like #1919 is merged now ?

@djanand
djanand force-pushed the fix/python-cargo-lock-drift branch from 62cba08 to 4d07e10 Compare July 3, 2026 03:48
@github-actions github-actions Bot removed the python label Jul 3, 2026
@djanand

djanand commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@djanand seems like #1919 is merged now ?

thanks @coderfender. Can you please review/approve/merge this PR ?

@coderfender
coderfender merged commit 84512cb into apache:main Jul 3, 2026
5 checks passed
coderfender pushed a commit to coderfender/datafusion-ballista that referenced this pull request Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve python/Cargo.lock drifts - CI deps

4 participants