feat(CI): Keep local and Github CI test in sync with `cargo xtask` by 2010YOUY01 · Pull Request #23414 · apache/datafusion

2010YOUY01 · 2026-07-09T08:41:29Z

Summary

For easier review, start with this PR writeup, then read the top comments in .github/workflows/rust.yml, and then follow along with the code.

This PR introduces an easy way to reproduce CI checks locally.

To reproduce .github/workflows/rust.yml:

cargo xtask ci workflow rust

Example output:

yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci workflow rust

ci workflow `rust` total execution time: 565.131385s
    ci job `linux-rustdoc`: 210.992471s
    ci job `linux-test-doc`: 117.395768s
    ci job `linux-test`: 85.890313s
    ci job `linux-test-example`: 39.367084s
    ci job `linux-test-datafusion-cli`: 37.235765s
    ci job `clippy`: 24.750692s
    ci job `msrv`: 16.421963s
    ci job `macos-aarch64`: 12.399443s
    ci job `config-docs-check`: 6.801263s
    ci job `check-fmt`: 3.139787s
    ci job `linux-cargo-check-datafusion`: 2.715153s
    ci job `cargo-toml-formatting-checks`: 1.94445s
    ci job `examples-docs-check`: 1.518475s
    ci job `linux-cargo-check-datafusion-functions`: 1.121708s
    ci job `sqllogictest-substrait`: 991.08ms
    ci job `linux-build-lib`: 600.417ms
    ci job `linux-datafusion-proto-features`: 596.001ms
    ci job `linux-datafusion-substrait-features`: 528.707ms
    ci job `vendor`: 346.94ms
    ci job `linux-datafusion-common-features`: 208.197ms
    ci job `verify-clean`: 83.283ms
    ci job `verify-clean`: 82.425ms

To reproduce single job or step similarly,

yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci job check-fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`

ci job `check-fmt` total execution time: 2.817591s
    ci step `fmt`: 2.817591s
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci step fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`

ci step `fmt` completed in 2.793302s

Background

The existing GitHub CI follows a 3-layer structure:

step: an atomic check, for example cargo fmt
job: one or more steps, for example check-fmt
workflow: a list of jobs, for example .github/workflows/rust.yml

For example:

workflow: rust
  job: check-fmt
    step: fmt
      command: cargo fmt --all -- --check

  job: clippy
    step: clippy
      command: cargo clippy ...

  job: linux-test
    step: rust-test
      command: cargo test ...
    step: verify-clean
      command: git diff --exit-code

This PR mirrors that structure in xtask, so the same CI units can be used by both GitHub Actions and local runs.

Motivation

Making CI easy to reproduce locally improves developer experience.

If a specific job fails on GitHub, we can reproduce it locally with cargo xtask ci job <job>.
After a small change, we can run a subset of the full CI locally. This shortens the development cycle while still giving good confidence that the full CI will pass.

Possible future extensions:

cargo xtask ci workflow changed

Detect changed Rust files, then only run unit tests and clippy on crates affected by the change, plus relevant integration tests.

cargo xtask ci workflow doc

For documentation-only changes, only run formatting, typo checks, docs checks, and other lightweight checks.

Implementation

The implementation adds reusable units for atomic checks, such as cargo fmt. In GitHub Actions terminology, these are "steps".

The key idea is:

GitHub CI orchestrates these atomic checks remotely.
cargo xtask ci ... orchestrates the same checks locally.

There are two viable ways to organize these atomic checks:

cargo xtask, for example cargo xtask ci step fmt
shell scripts, for example ./check_fmt.sh

I previously tried the shell script approach for dev.yml:

https://github.com/apache/datafusion/blob/main/dev/rust_lint.sh

For the current CI complexity, I found cargo xtask easier to keep organized. I suggest we continue with this approach and later migrate the existing scripts to use xtask as well.

Scope of this PR

This PR:

Introduces the xtask framework.
Migrates the steps in rust.yml to xtask runners. The commands are unchanged, except for the small difference noted below.
Supports local orchestration with cargo xtask ci workflow rust.

Notes:

This PR only achieves CI check command equivalence. Setup steps and tool installation are not fully reproduced yet, and are expected to be done manually for now.
Three jobs are skipped in the local run because they need slightly different handling on macOS. I would like to add them in follow-up PRs to keep this PR easier to review.

Testing

Individual 'step's will be tested by CI.

For workflow/jobs, I have tested locally on MacOS. I think we can let CI to test some fast workflows in the future.

2010YOUY01 · 2026-07-09T08:44:04Z

+                "--release-source",
+                "rust-dist",


These two lines added should be the only changed command (I found the default option fails on my local setting)

2010YOUY01 · 2026-07-09T09:02:27Z

cc who might be interested, @alamb @comphead @kumarUjjawal @blaginin

comphead

Thanks @2010YOUY01 does that mean CI changes should be reflected in xtask and vice versa?

WDYT about https://github.com/nektos/act to try run DataFusion CI locally, although I heard not all things possible with this tool

kumarUjjawal · 2026-07-09T15:04:39Z

+        .iter()
+        .find(|job| job.name == VERIFY_CLEAN_JOB_NAME)
+        .unwrap_or_else(|| panic!("missing `{VERIFY_CLEAN_JOB_NAME}` job"));
+    let mut jobs = vec![verify_clean];


fails immediately if we have there are any uncommitted changes before it runs any CI check. Was this intentional?

Yes, this is intentional, now the whole workflow requires a clean local git repo to run (no uncommitted changes), though I think there is room to improve it later.

To better simulate GitHub CI, we should try to reproduce the actual commands it runs as closely as possible.

Currently, GitHub CI runs several verify-clean checks between tests to ensure that no generated or modified files are left behind after each test, using git diff.

So for the local reproducer to behave the same way, we need to ensure there are no uncommitted changes before running the full workflow.

kumarUjjawal · 2026-07-09T15:07:59Z

+            format!("failed to create {}: {error}", tpch_data.display())
+        })?;
+
+        self.command("git")


if the step fails partway through and I re-run, git clone immediately fails with "destination path already exists". I'd have to manually rm -rf tpch-dbgen before retrying.

It keeps the existing command, we could fix it later. See #23414 (comment)

kumarUjjawal · 2026-07-09T15:09:48Z

+            .env("RUSTFLAGS", rustflags)
+            .run()?;
+
+        let chrome_driver = format!(


This results in wasm-pack being called with --chromedriver /chromedriver , which fails with a confusing "not found" error. The run_rust_sqllogictest function handles the missing-env-var case much better by returning an explicit error, we can use the same here

Similarly, #23414 (comment)

I found this job also a bit tricky to setup locally on my Mac

kumarUjjawal · 2026-07-09T15:15:14Z

+            let result = runner(self, &args);
+            job_timing.add_step_timing(step, start.elapsed().unwrap_or_default());
+
+            if let Err(error) = result


we should consider building the full message in a single map_err rather than chaining, there are several instances

2010YOUY01 · 2026-07-10T01:14:04Z

Thanks @2010YOUY01 does that mean CI changes should be reflected in xtask and vice versa?

Yes, exactly. That's the end goal, and this PR is the initial step with a limited scope.

WDYT about https://github.com/nektos/act to try run DataFusion CI locally, although I heard not all things possible with this tool

Good to know about act! I went to do some investigation, its limitations are

Not mature enough (lots of unsupported features for github .yml, bugs)
Expensive to run locally (Docker based, pulls large images)

Overall I think xtask is a better choice.

2010YOUY01 · 2026-07-10T01:20:59Z

+use crate::ci_workflows::WorkflowInfo;
+
+const VERIFY_CLEAN_JOB_NAME: &str = "verify-clean";
+const SKIPPED_JOBS: &[&str] = &[


These jobs are not supported by the local runner yet. I found that they need some modifications to run on macOS.

This PR does not change any commands, to make the review easier.

alamb

Thank you @2010YOUY01 for this PR -- the idea of making the deveopment experience better is good, but I am worried about adding another tool requirement

Making CI easy to reproduce locally improves developer experience.

I do agree with this, but I also think making the development experience more bespoke (requiring yet another tool -- we already have taplo, prettier, etc) works against this purpose

For example InfluxData uses just for this purpose, and I have also used Make for this purpose. This has resulted in several times it not being possible to use standard cargo commands to run the tests (b/c the indirection layers add some magic flag)

In my opinion these tools all tend to make understanding how to do things in the repo less standard and thus put up a larger barrier to entry.

On the other hand AI tools lower this barrier, so maybe it is not as much of a problem anymore 🤔

I previously tried the shell script approach for dev.yml:
https://github.com/apache/datafusion/blob/main/dev/rust_lint.sh

For the current CI complexity, I found cargo xtask easier to keep organized. I suggest we continue with this approach and later migrate the existing scripts to use xtask as well.

I agree we can't mirror CI 1:1 in shell scripts without unwarranted complexity

However I think we can get like 95% of the common failures from CI via shell scripts and that would be good enough for the repo

alamb · 2026-07-10T17:53:46Z

+# Architecture
+# -------------
+#
+# Each atomic CI check lives in an individual `xtask` step. The GitHub workflow


I have never heard of xtask -- I am sure it is awesome, but it does now add one more development dependency for developers and makes it that much harder (maybe just a little) to develop in DataFuson. (as now you need to have yet another tool installed)

alamb · 2026-07-10T17:54:02Z

+#
+# (@ means no `needs` dependency.)
+#
+# @ -> linux-build-lib


Is the idea that xtask can model this type of dependency tree?

alamb · 2026-07-10T18:00:16Z

So in other words maybe we don't have to have 100% parity with what CI does -- instead we can focus on getting the most of the coverage via standard commands.

And maybe if there is something that CI is doing that is non standard, we can try to reduce the complexity there

2010YOUY01 · 2026-07-11T04:31:12Z

Thank you for the detailed feedback, @alamb!

I agree that the current solution adds friction for new developers. I think this approach could eventually become frictionless, but it needs more work. Let me run some experiments to see whether I can make even the initial version easier to use. Here are a few ideas:

Provide a one-click setup/bootstrap script that installs the required tools.
Skip checks whose tools are unavailable. With only the Rust toolchain installed, most critical checks should already be runnable.
Provide help messages showing the exact underlying commands, for example: cargo xtask ci step fmt help → command: cargo fmt.

One minor clarification: xtask itself does not require installing any additional tools. It is more like a Cargo builtin feature (or convention): https://github.com/matklad/cargo-xtask

Shell scripts vs. xtask commands

To reach our end goal, the leaf nodes (atomic checks), should be wrapped as reusable units. The main design question is whether those units should be shell scripts or xtask steps.

I still think xtask will be easier to use and maintain in the long term. The existing leaf scripts already contain additional logic rather than just invoking plain commands. Personally, I find Rust/xtask easier to work with: Rust is DataFusion's native language, and it provides more structure than shell scripts.

Let me explore ways to reduce the setup friction and see how the approach looks afterward.

Additional references

Here are a few projects I would like to examine more closely. My impression is that most projects do not provide a great developer experience for running tests locally or reproducing CI. ChatGPT suggested these three as particularly strong examples, and I would like to study their approaches and consider how we could implement something similar:

ClickHouse: https://clickhouse.com/docs/development/continuous-integration
Materialize: https://github.com/MaterializeInc/materialize/blob/main/doc/developer/mzcompose.md
RisingWave: https://risingwave.com/blog/developer-experience-counts-behind-the-scenes-of-risedev/

alamb · 2026-07-12T10:30:45Z

One minor clarification: xtask itself does not require installing any additional tools. It is more like a Cargo builtin feature (or convention): https://github.com/matklad/cargo-xtask

I think you need to do

cargo install cargo-xtask

I am also somewhat worried about taking on a new (admittedly dev only) dependency. For example, I took a look at https://github.com/matklad/cargo-xtask and it was last updated 9 months ago and it doesn't have an obvious long term maintenance model (e.g. bugs,. etc) beyond someone's personal project

I still think xtask will be easier to use and maintain in the long term. The existing leaf scripts already contain additional logic rather than just invoking plain commands. Personally, I find Rust/xtask easier to work with: Rust is DataFusion's native language, and it provides more structure than shell scripts.

I agree that shell scripts are not great for complex logic. What I was wondering is can we avoid complicated shell scripts by simplifying that logic (like does it need to follow all the dependency checks that the current CI does?) rather than finding a some system (like xtask) to help manage that complexity

2010YOUY01 · 2026-07-12T12:49:16Z

One minor clarification: xtask itself does not require installing any additional tools. It is more like a Cargo builtin feature (or convention): https://github.com/matklad/cargo-xtask

I think you need to do
cargo install cargo-xtask

That cargo install is not needed, it's a binary crate within DataFusion

cargo run --bin datafusion-xtask -- (extra args)

# xtask is an alias for the above command
cargo xtask (extra args)

It's more like a convention that is used in Rust ecosystem (e.g. cargo itself https://github.com/rust-lang/cargo/blob/master/.cargo/config.toml) to implement project-specific development commands and automation in Rust.

The linked cargo-xtask repository only documents this convention, below is a quote from https://github.com/matklad/cargo-xtask

cargo-xtask is way to add free-form automation to a Rust project, a-la make, npm run or bespoke bash scripts.

The two distinguishing features of xtask are:

* It doesn't require any other binaries besides `cargo` and `rustc`, it fully bootstraps from them
* Unlike bash, it can more easily be cross platform, as it doesn't use the shell.

I still think xtask will be easier to use and maintain in the long term. The existing leaf scripts already contain additional logic rather than just invoking plain commands. Personally, I find Rust/xtask easier to work with: Rust is DataFusion's native language, and it provides more structure than shell scripts.

I agree that shell scripts are not great for complex logic. What I was wondering is can we avoid complicated shell scripts by simplifying that logic (like does it need to follow all the dependency checks that the current CI does?) rather than finding a some system (like xtask) to help manage that complexity

I agree we could pul such logic outside existing scripts to keep things easier to maintain.

comphead · 2026-07-12T19:24:55Z

I'm trying to get a bigger picture. When we say rust CI does it mean to run locally the flow .github/workflows/rust.yml? if it is I would recommend again to look into apache/datafusion-comet#4570, so the contributor would run the flow in their own repo without having a local clone.

Do we anticipate supporting 2 flows might be hard to maintain and keep them in sync and eventually the local part might be untrusted and not used as the result? 🤔

If the end goal to catch issues as early as possible or save ASF resources then contributor forks delegation would help, otherwise we may want to try to break down rust.yml into more manageable parts and have critical part for testing in separate, locally testable yml

2010YOUY01 · 2026-07-13T01:15:17Z

I'm trying to get a bigger picture. When we say rust CI does it mean to run locally the flow .github/workflows/rust.yml? if it is I would recommend again to look into apache/datafusion-comet#4570, so the contributor would run the flow in their own repo without having a local clone.

Do we anticipate supporting 2 flows might be hard to maintain and keep them in sync and eventually the local part might be untrusted and not used as the result? 🤔

If the end goal to catch issues as early as possible or save ASF resources then contributor forks delegation would help, otherwise we may want to try to break down rust.yml into more manageable parts and have critical part for testing in separate, locally testable yml

Regarding the end goal: Another important goal is faster local checks. GitHub-hosted runners can be slow, especially for forks, while local runs can be much faster. This also saves time otherwise spent fixing CI failures: we can simply ask a coding agent to run the local CI suite on the final commit before opening a PR, I found checking Github CI status after sending PR, and potentially fix it later is particularly annoying, and this can be avoided. I think this is actually the more important motivation.
Regarding maintenance: After experimenting with this approach, I have two observations:
1. Moving the shell scripts to a Rust-based cargo xtask solution has already reduced the maintenance burden. In general, I find it more ergonomic to maintain.
2. Personally, I think keeping the local CI runner and GitHub Actions in sync will be actually simple. Most critical tests require only the Rust toolchain.

feat(CI): Keep local and Github CI test in sync with cargo xtask

17d62e7

github-actions Bot added the development-process Related to development process of DataFusion label Jul 9, 2026

2010YOUY01 commented Jul 9, 2026

View reviewed changes

2010YOUY01 mentioned this pull request Jul 9, 2026

EPIC: Add Local Scripts to Reproduce Full CI #21048

Open

12 tasks

comphead reviewed Jul 9, 2026

View reviewed changes

kumarUjjawal reviewed Jul 9, 2026

View reviewed changes

2010YOUY01 commented Jul 10, 2026

View reviewed changes

alamb reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

2010YOUY01 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Motivation

Implementation

Scope of this PR

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Jul 9, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 10, 2026

Uh oh!

2010YOUY01 commented Jul 11, 2026

Shell scripts vs. xtask commands

Additional references

Uh oh!

alamb commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2010YOUY01 commented Jul 12, 2026

Uh oh!

comphead commented Jul 12, 2026

Uh oh!

2010YOUY01 commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

2010YOUY01 commented Jul 9, 2026 •

edited

Loading

2010YOUY01 commented Jul 10, 2026 •

edited

Loading

alamb commented Jul 12, 2026 •

edited

Loading