Skip to content

fix: add DuckDB dialect UDFs so flush partition filters run over SPI - #228

Merged
qsliu2017 merged 1 commit into
mainfrom
fix/223-flush-partition-filter-dialect
Jul 16, 2026
Merged

fix: add DuckDB dialect UDFs so flush partition filters run over SPI#228
qsliu2017 merged 1 commit into
mainfrom
fix/223-flush-partition-filter-dialect

Conversation

@qsliu2017

@qsliu2017 qsliu2017 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

CALL ducklake.set_partition(tbl, 'month(col)') (or any non-identity transform: year/month/day/hour/bucket) is accepted, but every subsequent ducklake.flush_inlined_data() fails with IO Error: SPI execution failed: function month(timestamp with time zone) does not exist (or murmur3_32(...) for bucket partitions). The maintenance worker keeps retrying the flush and leaves orphaned parquet files behind on each attempt. Root cause: DuckLake embeds the partition expression in DuckDB dialect in the flush's per-file deleted-rows bookkeeping query, and pg_ducklake ships that query to PostgreSQL over SPI, whose parser has no such functions.

Per maintainer decision, the bookkeeping query stays on the PG/SPI path and the dialect gap is filled instead: the new extension objects are ducklake.year/month/day/hour (date, timestamp, timestamptz, plus hour(time) -- the only transform DuckDB accepts on TIME) and ducklake.murmur3_32 overloads for boolean, all integer widths, real, double precision, text, bytea, date, time, timestamp, timestamptz, numeric, and uuid. The murmur3 implementations reuse the vendored DuckLake murmur3 header, so they are bit-exact with the writer by construction; the SPI executor forces search_path=ducklake for metadata queries (via the existing GUC nest-level pattern) so the unqualified names resolve deterministically and cannot collide with e.g. orafce's month() in public.

Verifying against writer-side ground truth surfaced two additional silent-corruption bugs, both fixed here. First, the inlined table stores text columns as BYTEA, so the upstream filter's CAST(col AS VARCHAR) compared PG's hex form ('\x6170706c65') instead of the string -- every text bucket mis-hashed and every text identity comparison missed, writing delete tombstones onto the wrong rows. Second, without an exact numeric overload PostgreSQL resolves murmur3_32(numeric) to the double precision one (the preferred numeric type) and silently mis-buckets. Fixing the first required a small new vendored seam (ducklake-010 patch): a metadata-manager virtual through which the backend renders each partition column per its inlined storage type (convert_from() for text-as-BYTEA, raw column for bytea, ::text for decimals wider than 18 digits, PG-parseable type names otherwise).

One documented limitation: timestamptz calendar transforms depend on the session TimeZone in both engines, and DuckDB syncs its TimeZone from PostgreSQL only once, at instance creation -- a mid-session SET timezone can make writer and bookkeeping disagree near month/year boundaries. Accepted as inherent to evaluating in two engines.

Tests: nine flush-with-delete scenarios (month on timestamptz incl. NULL partitions, bucket on int/text/numeric(18,3)/numeric(30,10)/uuid/bytea, identity on text, hour on time) plus a PG-vs-DuckDB equivalence matrix pinned to the Iceberg spec reference vectors that fails loudly on any dialect drift. Full regression suite: 52/52 on main.

This PR now targets main because it adds SQL interface (new extension objects, shipped in the in-development 1.1.0 update script). Orphaned parquet files left by past failed flushes are recoverable with ducklake.cleanup_orphaned_files().

Fixes #223

🤖 Generated with Claude Code

flush_inlined_data() on a table with a non-identity partition transform
(year/month/day/hour/bucket) failed with "function month(timestamp with
time zone) does not exist": DuckLake embeds the partition expression in
DuckDB dialect (month(col), (murmur3_32(col) & 2147483647) % N) in the
per-file deleted-rows bookkeeping query, and pg_ducklake ships that query
to PostgreSQL over SPI, whose parser has no such functions. Beyond the
missing functions, the filter also mis-rendered columns: the inlined table
stores VARCHAR as BYTEA, so CAST(col AS VARCHAR) produced PG's hex form
('\x6170706c65') and silently mis-hashed every text bucket / mismatched
every text identity comparison, mislaying delete tombstones (silent data
corruption, not an error).

Fix, keeping the query on the PG/SPI path: provide the DuckDB functions
as UDFs in the ducklake schema with bit-exact semantics -
year/month/day/hour over EXTRACT (date/timestamp/timestamptz, hour(time)),
and murmur3_32 overloads reusing the vendored DuckLakeMurmur3 header
(bool/int2/int4/int8/float4/float8/text/bytea/date/time/timestamp/
timestamptz/uuid/numeric; numeric hashes the unscaled value as an 8-byte
long like DuckDB decimals of width <= 18, and an exact numeric overload is
required or PG resolves to the double one and silently mis-buckets).
SPIExecuteInSubtransaction forces search_path=ducklake for metadata SPI
queries via the existing GUC nest-level pattern, so the unqualified names
resolve deterministically. A new ducklake-010 patch adds a metadata-manager
virtual (GetFlushPartitionSQLExpressions) so the backend renders each
partition column per its inlined storage type: convert_from() for
VARCHAR-as-BYTEA, raw column for BLOB, ::text for decimals wider than 18
digits (DuckDB hashes their string form), and PG-parseable type names
otherwise (e.g. DOUBLE PRECISION, avoiding unparseable CAST(col AS DOUBLE)).

The new SQL objects ship in the in-development 1.1.0 update script
(pg_ducklake--1.0.0--1.1.0.sql). The regression test covers
flush-with-deletes for month(timestamptz) incl. NULL partitions, bucket on
int/text/numeric(18,3)/numeric(30,10)/uuid/bytea, identity on text,
hour(time), plus a PG-vs-DuckDB equivalence matrix (Iceberg reference
vectors included) that fails loudly on any dialect drift. Known limits:
identity partitions on bytea fail upstream while building partition values
from non-UTF8 bytes; DuckDB accepts only hour() on TIME; timestamptz
transforms depend on the session TimeZone, which DuckDB syncs from PG only
at instance creation (a later SET timezone can make writer and bookkeeping
disagree near boundaries - accepted limitation of evaluating in two
engines).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qsliu2017
qsliu2017 force-pushed the fix/223-flush-partition-filter-dialect branch from 2b9f879 to 0ee8039 Compare July 15, 2026 09:55
@qsliu2017
qsliu2017 changed the base branch from v1.0 to main July 15, 2026 09:55
@qsliu2017 qsliu2017 closed this Jul 15, 2026
@qsliu2017 qsliu2017 reopened this Jul 15, 2026
@qsliu2017 qsliu2017 changed the title fix: run flush partition filter query in DuckDB dialect fix: add DuckDB dialect UDFs so flush partition filters run over SPI Jul 15, 2026
@qsliu2017
qsliu2017 merged commit e43c6b8 into main Jul 16, 2026
9 checks passed
@qsliu2017
qsliu2017 deleted the fix/223-flush-partition-filter-dialect branch July 16, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: time partition transforms (month/year/day/hour) silently accepted on TIMESTAMPTZ columns but fail at flush time

1 participant