Use R2 bucket for duckdb libraries - #8486
Conversation
8129660 to
8af34b4
Compare
Merging this PR will improve performance by 19.21%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
168.9 µs | 205.6 µs | -17.87% |
| ⚡ | Simulation | chunked_bool_canonical_into[(1000, 10)] |
27.1 µs | 16.2 µs | +67.38% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
244.4 ns | 186.1 ns | +31.34% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[1024] |
304.7 ns | 246.4 ns | +23.68% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[2048] |
398.6 ns | 340.3 ns | +17.14% |
| ⚡ | Simulation | chunked_varbinview_canonical_into[(100, 100)] |
258.9 µs | 223.8 µs | +15.7% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(100, 100)] |
306 µs | 270.8 µs | +13.01% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-r2-infra (451a71e) with develop (5a764e6)
c3d1f15 to
d3fc1da
Compare
|
Macos deployment error: |
DuckdbFS implementation for Vortex was introduced in #6198 as opt-out, but changed to opt-in in #6564 due to performance regressions. There were multiple issues (#6709, #6565 #6685) associated with it which differ from vortex's file system behaviour. It also requires additional dependencies CI which are a blocker for #8486 since MacOS runner doesn't bundle openssl for x86_64 on arm, and builds fail. As a long term goal, calling duckdb's blocking IO inside our event loop isn't the right abstraction. We want to allow duckdb to use its own IO outside vortex. Duckdb fs is also not maintaned actively so we're removing it Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
b32cb14 to
3c6ac96
Compare
|
I've verified manually this works for commits, they are uploaded and used from R2 as well. |
0ax1
left a comment
There was a problem hiding this comment.
Great to see this coming into shape! Couple of questions inline. In general, will this be able to handle diff build configs: debug, release, asan etc.? The build config should prob be part of the key with which we store builds.
Yes, but for the first version it won't to keep the diff small |
3c6ac96 to
451a71e
Compare
Uh oh!
There was an error while loading. Please reload this page.