Skip to content

Use R2 bucket for duckdb libraries - #8486

Merged
myrrc merged 1 commit into
developfrom
myrrc/duckdb-r2-infra
Jun 23, 2026
Merged

Use R2 bucket for duckdb libraries#8486
myrrc merged 1 commit into
developfrom
myrrc/duckdb-r2-infra

Conversation

@myrrc

@myrrc myrrc commented Jun 18, 2026

Copy link
Copy Markdown
Contributor
  • Use ci-builds.vortex.dev R2 bucket as source for duckdb release and commit builds.
  • Mirror release builds from duckdb github releases page. Build commits from source.
  • For commit builds also try to download from R2 (useful for testing pre-release in CI)
  • Gate test_geometry for duckdb under release builds only. Running it for commits means we need to bundle "spatial" extension which is hard on macos due to openssl-dev symbols

@myrrc
myrrc requested a review from a team June 18, 2026 09:30
@myrrc
myrrc force-pushed the myrrc/duckdb-r2-infra branch from 8129660 to 8af34b4 Compare June 18, 2026 09:31
@myrrc
myrrc marked this pull request as draft June 18, 2026 09:35
@codspeed-hq

codspeed-hq Bot commented Jun 18, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 19.21%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 6 improved benchmarks
❌ 1 regressed benchmark
✅ 1576 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 168.9 µs 205.6 µs -17.87%
Simulation chunked_bool_canonical_into[(1000, 10)] 27.1 µs 16.2 µs +67.38%
Simulation bitwise_not_vortex_buffer_mut[128] 244.4 ns 186.1 ns +31.34%
Simulation bitwise_not_vortex_buffer_mut[1024] 304.7 ns 246.4 ns +23.68%
Simulation bitwise_not_vortex_buffer_mut[2048] 398.6 ns 340.3 ns +17.14%
Simulation chunked_varbinview_canonical_into[(100, 100)] 258.9 µs 223.8 µs +15.7%
Simulation chunked_varbinview_into_canonical[(100, 100)] 306 µs 270.8 µs +13.01%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing myrrc/duckdb-r2-infra (451a71e) with develop (5a764e6)

Open in CodSpeed

@myrrc
myrrc force-pushed the myrrc/duckdb-r2-infra branch from c3d1f15 to d3fc1da Compare June 18, 2026 10:44
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 10:45 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 10:45 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 10:45 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 11:17 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 11:17 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 11:17 — with GitHub Actions Inactive
@myrrc
myrrc requested a review from joseph-isaacs June 18, 2026 11:19
@myrrc
myrrc marked this pull request as ready for review June 18, 2026 11:19
@myrrc
myrrc marked this pull request as draft June 18, 2026 11:19
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 13:39 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 18, 2026 13:39 — with GitHub Actions Inactive
@myrrc

myrrc commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Macos deployment error:

error:  cannot create /Users/runner/work/vortex/vortex/duckdb-08e34c447bae34eaee3723cac61f2878b6bdf787/data/csv/issue2628_������.csv
        Illegal byte sequence

@myrrc
myrrc temporarily deployed to duckdb-build June 19, 2026 09:53 — with GitHub Actions Inactive
@myrrc myrrc mentioned this pull request Jun 19, 2026
@myrrc
myrrc requested a review from AdamGS June 19, 2026 10:16
myrrc added a commit that referenced this pull request Jun 19, 2026
DuckdbFS implementation for Vortex was introduced in
#6198 as opt-out, but changed
to opt-in in #6564 due to
performance regressions.
There were multiple issues
(#6709,
#6565
#6685) associated with it
which differ from vortex's file system behaviour.

It also requires additional dependencies CI which are a blocker for 
#8486 since MacOS runner
doesn't bundle openssl for x86_64 on arm, and builds fail.

As a long term goal, calling duckdb's blocking IO inside our event loop
isn't the right abstraction. We want to allow duckdb to use its own IO
outside vortex.

Duckdb fs is  also not maintaned actively so we're removing it

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc
myrrc temporarily deployed to duckdb-build June 19, 2026 13:06 — with GitHub Actions Inactive
@myrrc
myrrc force-pushed the myrrc/duckdb-r2-infra branch from b32cb14 to 3c6ac96 Compare June 19, 2026 13:44
@myrrc
myrrc temporarily deployed to duckdb-build June 19, 2026 13:44 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 19, 2026 13:44 — with GitHub Actions Inactive
@myrrc
myrrc temporarily deployed to duckdb-build June 19, 2026 13:44 — with GitHub Actions Inactive
@myrrc
myrrc marked this pull request as ready for review June 19, 2026 13:46
@myrrc
myrrc requested review from robert3005 and removed request for joseph-isaacs June 19, 2026 13:47
@myrrc

myrrc commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

I've verified manually this works for commits, they are uploaded and used from R2 as well.

@myrrc myrrc added the ext/duckdb Relates to the DuckDB integration label Jun 22, 2026
@myrrc
myrrc enabled auto-merge (squash) June 22, 2026 13:22
@robert3005
robert3005 requested a review from 0ax1 June 22, 2026 13:25

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to see this coming into shape! Couple of questions inline. In general, will this be able to handle diff build configs: debug, release, asan etc.? The build config should prob be part of the key with which we store builds.

Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/duckdb-r2.yml Outdated
Comment thread .github/workflows/duckdb-r2.yml
Comment thread .github/workflows/duckdb-r2.yml Outdated
Comment thread .github/workflows/duckdb-r2.yml Outdated
Comment thread .github/workflows/duckdb-r2.yml Outdated
@myrrc

myrrc commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

In general, will this be able to handle diff build configs: debug, release, asan etc.?

Yes, but for the first version it won't to keep the diff small

@vortex-data vortex-data deleted a comment from github-actions Bot Jun 22, 2026
@vortex-data vortex-data deleted a comment from 0ax1 Jun 22, 2026
Comment thread vortex-duckdb/build.rs
Comment thread .github/workflows/ci.yml
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc
myrrc force-pushed the myrrc/duckdb-r2-infra branch from 3c6ac96 to 451a71e Compare June 23, 2026 09:42
@myrrc
myrrc requested a review from 0ax1 June 23, 2026 09:43
Comment thread scripts/duckdb-r2-prepare.sh
Comment thread scripts/duckdb-r2-prepare.sh
Comment thread scripts/duckdb-r2-resolve.sh
Comment thread scripts/duckdb-r2-resolve.sh

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love the long shell scripts and we should prob consider switching to python here eventually but let's give this a spin in prod.

@myrrc
myrrc merged commit e6d0e08 into develop Jun 23, 2026
135 of 136 checks passed
@myrrc
myrrc deleted the myrrc/duckdb-r2-infra branch June 23, 2026 12:59
@myrrc myrrc linked an issue Jun 23, 2026 that may be closed by this pull request
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/ci ext/duckdb Relates to the DuckDB integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking Issue: own infrastructure for duckdb builds

2 participants