Skip to content

Increse duckdb retries when operating on parquet s3 #8903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robert3005 merged 1 commit into from
Jul 23, 2026
Merged

Increse duckdb retries when operating on parquet s3 #8903

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion benchmarks/duckdb-bench/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ use vortex_bench::runner::filter_queries;
use vortex_bench::setup_logging_and_tracing;
use vortex_bench::v3;

const S3_HTTP_INIT_SQL: [&str; 3] = [
"SET http_retries = 8",
"SET http_retry_wait_ms = 250",
"SET http_retry_backoff = 2",
];

/// Common arguments shared across benchmarks
#[derive(Parser)]
struct Args {
Expand Down Expand Up @@ -159,6 +165,11 @@ fn main() -> anyhow::Result<()> {
)?;

let benchmark_name = benchmark.dataset().to_string();
let mut duckdb_init_sql = Vec::new();
if benchmark.data_url().scheme() == "s3" {
duckdb_init_sql.extend(S3_HTTP_INIT_SQL.map(String::from));
}
duckdb_init_sql.extend(benchmark.engine_init_sql(Engine::DuckDB));

let mode = if args.explain {
BenchmarkMode::Explain
Expand All @@ -178,7 +189,7 @@ fn main() -> anyhow::Result<()> {
args.delete_duckdb_database,
args.threads,
)?;
ctx.set_init_sql(benchmark.engine_init_sql(Engine::DuckDB))?;
ctx.set_init_sql(duckdb_init_sql.clone())?;
ctx.register_tables(&*benchmark, format)?;

// Duckdb doesn't support octet_length for strings but we need this
Expand Down
Loading