Bump duckdb to v1.5.3

[sfc-gh-abozkurt]

Bump DuckDB to v1.5.3

Version Updates

• DuckDB: 1.4.4 → 1.5.3
• Python DuckDB package: 1.4.3 → 1.5.3
• DuckDB extension tags updated: httpfs, aws, azure pinned to new commits compatible with 1.5.3

Native GeoParquet / GEOMETRY Support

DuckDB 1.5 promotes GEOMETRY to a first-class LogicalTypeId (previously it was a BLOB with a type alias). This required changes across the stack:

pgduck_server — Wire protocol adaptation

• duckdb_column_type() now returns DUCKDB_TYPE_INVALID for GEOMETRY columns. Two new helpers added via duckdb_pglake:
  • duckdb_pglake_is_geometry_type() — checks whether a logical type is GEOMETRY
  • duckdb_pglake_geometry_get_srid() — extracts SRID from the CRS attached to the type (e.g. EPSG:4326 → 4326, OGC:CRS84 → 4326)
• When a GEOMETRY column is detected, it is mapped to BLOB internally so the existing hex-WKB serialization path handles it.
• SRID injection: A new inject_srid_into_hex_wkb() function transforms ISO WKB hex into EWKB hex by setting the SRID flag and inserting the SRID bytes, so PostGIS can recover the SRID from the wire format.

pg_lake_engine — Read path (read_data.c)

• Parquet/Iceberg reads now keep GEOMETRY native instead of downgrading to BLOB. DuckDB reads GeoParquet columns natively; for old files without GeoParquet metadata, DuckDB casts BLOB → GEOMETRY automatically.
• CRS stripping via ST_SetCRS(col, ''): Replaces the old ST_GeomFromWKB(col::blob) projection. Stripping CRS is needed because DuckDB 1.5+ requires plain GEOMETRY (without CRS) for implicit POINT_2D casts used by spheroid functions like ST_Distance_Spheroid. SRID preservation relies on PostGIS applying the column typmod.
• Iceberg schema override: When Iceberg metadata maps a geometry column to binary (per Iceberg spec), but the PostgreSQL column type is geometry, the schema entry is overridden to GEOMETRY so read_parquet() reads it natively.
• Iceberg fixed[N] type handling: GetSchemaType() now maps fixed[N] → binary to avoid DuckDB parse errors.
• GDAL format: New projection using COALESCE(hex(TRY_CAST(col AS BLOB)), ST_AsHexWKB(TRY_CAST(col AS GEOMETRY))) to handle both BLOB and GEOMETRY sources.

pg_lake_engine — Write path (write_data.c)

• GeoParquet write with CRS preservation: Instead of ST_AsWKB(col) (which wrote a raw blob), the column is now kept as native GEOMETRY. When the PostgreSQL column carries an SRID (via typmod), ST_SetCRS(col, 'EPSG:<srid>') is applied so DuckDB's GeoParquet writer emits CRS metadata.

DuckDB HTTP Filesystem API Changes

DuckDB 1.5.1 refactored the HTTP filesystem API — PostRequest and PutRequest now take HTTPInput & instead of FileHandle &, and HTTPFSUtil::GetHTTPUtil() returns a reference instead of shared_ptr.

PgLakeS3FileSystem

• PostRequest / PutRequest signatures updated to accept HTTPInput &.
• New context registry (RegisterContext / LookupContext) with a mutex-protected map that associates HTTPInput* → ClientContext*. This replaces the previous approach of casting FileHandle to access the ClientContext for encryption settings.
• SetEncryptionFields now takes optional_ptr<ClientContext> directly instead of a handle reference.

RegionAwareS3FileSystem / CachingFileSystem

• Updated HTTPFSUtil::GetHTTPUtil() usage from -> to . (reference instead of pointer).
• Added required GetEstimatedCacheMemory() override to cache-related classes.

Glob Handling for Special Characters in Partition Paths

Iceberg partition paths can contain glob characters (e.g., * in directory names like specialChars!@#$%^&*()_+). DuckDB's glob machinery interprets these as wildcards. The caching filesystem now checks whether a path with glob characters refers to an actual existing file first, returning it directly without globbing.

S3 URL Decoding Fix

Common prefixes returned by S3 ListObjectsV2 are now URL-decoded via S3FileSystem::UrlDecode() before being used as paths, fixing issues with encoded characters in S3 prefixes.

Removed enable_geoparquet_conversion Setting

The SET GLOBAL enable_geoparquet_conversion TO 'false' initialization command was removed from pgduck_server, as native GeoParquet support is now the intended behavior.

DuckDB Patches

Patch	Status	Description
return_stats.patch	Removed	Parquet stats fixes for decimals and booleans — merged upstream in DuckDB 1.5.1
parquet-virtual-column-stats.patch	Added	Prevents out-of-bounds crash when getting column statistics for virtual/added columns not present in Parquet row groups
composite-type-resolution.patch	Updated	Rebased for new duckdb-postgres API (e.g., connection->Query(nullptr, ...))
numeric-nan.patch	Updated	Rebased for line offsets
snapshot.patch	Updated	Rebased for new duckdb-postgres API changes

pg_lake Bugs

• Fixed wrong file_sequence_number in manifest entry
• Gdal supports multicurve.

Test Updates

• Error message changes: DuckDB 1.5.1 changed several error messages:
  • "HTTP Error: Unable to connect to URL" → "HTTP Error:"
  • "NOT FOUND" assertions now use case-insensitive matching (.upper())
  • "Could not establish connection error" → "Could not connect to server error" / "Could not resolve hostname error"
  • "READ_PARQUET " (trailing space) → "READ_PARQUET" (trimmed)
• GDAL tests: Added "Unsupported geometry type in WKB" and "Could not find layer" as alternative accepted error messages.
• pgbench test: Replaced pgbench -i -I t initialization (which relied on DuckDB supporting pgbench's DDL through the t init step) with explicit CREATE TABLE statements (pgbench runs then by adding WITH options that are not supported by duckdb).
• Caching tests: Double-remove of a cached file now expects a potential 404 error instead of silent success.
• Test fixture: iceberg_catalog fixture now depends on extension to ensure extensions are created before catalog setup.

Fixes #357

[sfc-gh-mslot]

we need to unflip argument order when pushing down functions like st_area when we switch to DuckDB 1.5 https://duckdb.org/2026/03/09/announcing-duckdb-150#breaking-change-flipping-of-axis-order

[sfc-gh-mslot]

Rebased onto current main. Conflicts and notable resolutions:

• pg_lake_engine/src/pgduck/{read,write}_data.c — kept this PR's ST_SetCRS-based geometry handling (intentional for DuckDB 1.5+ spheroid functions), but updated all quote_identifier calls to duckdb_quote_identifier to align with the rename in Fix DuckDB identifier and struct value quoting for special characters #297.
• numeric-nan.patch — kept main's version with +/-Inf handling from postgres_scanner fix +-inf #308 (this PR was based on the older NaN-only version). Updated context lines so it applies cleanly against the new duckdb-postgres SHA.
• postgis-geometry-wkb.patch (new on main from postgres_scanner: convert PostGIS geometry to canonical WKB #377) — updated context lines for the new submodule's LogicalType::GEOMETRY() return.
• Pipfile.lock — took main's lock (which now includes `pytest-rerunfailures`) and swapped in this PR's duckdb 1.5.1 entry. CI uses `pipenv install --dev` so the lockfile hash mismatch will resolve on its own.

Patch fixups landed as a separate commit so the original commits stay yours. Local `make install` clean. CI rerunning on the rebased branch.

[sfc-gh-mslot]

CI on the rebased branch: lint, all 6 build-and-install platforms, and most test groups pass. Remaining failures are isolated to spatial / geometry handling:

• `test-check (17, check-pgduck_server)` and `test-check (18, check-pgduck_server)` — `test_postgres_scanner_geometry.py` hits `Binder Error: No function matches the given name and argument types 'st_geomfromwkb(GEOMETRY)'`. The `postgis-geometry-wkb` patch (from postgres_scanner: convert PostGIS geometry to canonical WKB #377) tags geometry columns and rewrites the projection with `ST_AsBinary(...)`, which returns bytea — but the bind type still resolves to `LogicalType::GEOMETRY()` because that's what `pgtypename == "geometry"` returns under DuckDB 1.5.1. So the column ends up typed as GEOMETRY but carrying BLOB data, and `ST_GeomFromWKB` (BLOB → GEOMETRY) doesn't match. The patch likely needs to return `LogicalType::BLOB` when it sets `POSTGIS_GEOMETRY`, or the tests need to drop the explicit `ST_GeomFromWKB` wrapping.

• `test-check (17|18, check-pg_lake_spatial)` and `test-installcheck (*, installcheck-pg_lake_spatial)` — first failure is in `test_gdal_zip_gml`: pgduck_server crashes on `SELECT shape FROM test_gdal.fdw`, after which every subsequent test sees "connection to server on socket ... refused". The test was updated in this PR to expect 5 rows back (DuckDB 1.5.1 supports MULTICURVE), but pgduck_server is dying before it can return. Likely a real bug in the GDAL / native-GEOMETRY-from-Parquet path.

These look like genuine PR-level issues with the DuckDB 1.5.1 geometry semantics rather than rebase artifacts, so leaving them for you to investigate.

[sfc-gh-mslot]

did we unflip the arguments per #357?

[sfc-gh-mslot]

may have some performance implications

[sfc-gh-mslot]

why is this needed? could use a short comment

[sfc-gh-mslot]

could use a comment.

Why do we need to track multiple contexts now?

[sfc-gh-mslot]

kind of wondering whether we should still map to blob to support reading when the SRID is not 4326.

[sfc-gh-mslot]

should we add it to the typemap instead?

[sfc-gh-mslot]

Safe to assume we're running latest DuckDB here, it's whatever we build with, but probably there should just be a separate geometry_to_text for DUCKDB_TYPE_GEOMETRY