fix: add DuckDB dialect UDFs so flush partition filters run over SPI - #228
Merged
Conversation
flush_inlined_data() on a table with a non-identity partition transform
(year/month/day/hour/bucket) failed with "function month(timestamp with
time zone) does not exist": DuckLake embeds the partition expression in
DuckDB dialect (month(col), (murmur3_32(col) & 2147483647) % N) in the
per-file deleted-rows bookkeeping query, and pg_ducklake ships that query
to PostgreSQL over SPI, whose parser has no such functions. Beyond the
missing functions, the filter also mis-rendered columns: the inlined table
stores VARCHAR as BYTEA, so CAST(col AS VARCHAR) produced PG's hex form
('\x6170706c65') and silently mis-hashed every text bucket / mismatched
every text identity comparison, mislaying delete tombstones (silent data
corruption, not an error).
Fix, keeping the query on the PG/SPI path: provide the DuckDB functions
as UDFs in the ducklake schema with bit-exact semantics -
year/month/day/hour over EXTRACT (date/timestamp/timestamptz, hour(time)),
and murmur3_32 overloads reusing the vendored DuckLakeMurmur3 header
(bool/int2/int4/int8/float4/float8/text/bytea/date/time/timestamp/
timestamptz/uuid/numeric; numeric hashes the unscaled value as an 8-byte
long like DuckDB decimals of width <= 18, and an exact numeric overload is
required or PG resolves to the double one and silently mis-buckets).
SPIExecuteInSubtransaction forces search_path=ducklake for metadata SPI
queries via the existing GUC nest-level pattern, so the unqualified names
resolve deterministically. A new ducklake-010 patch adds a metadata-manager
virtual (GetFlushPartitionSQLExpressions) so the backend renders each
partition column per its inlined storage type: convert_from() for
VARCHAR-as-BYTEA, raw column for BLOB, ::text for decimals wider than 18
digits (DuckDB hashes their string form), and PG-parseable type names
otherwise (e.g. DOUBLE PRECISION, avoiding unparseable CAST(col AS DOUBLE)).
The new SQL objects ship in the in-development 1.1.0 update script
(pg_ducklake--1.0.0--1.1.0.sql). The regression test covers
flush-with-deletes for month(timestamptz) incl. NULL partitions, bucket on
int/text/numeric(18,3)/numeric(30,10)/uuid/bytea, identity on text,
hour(time), plus a PG-vs-DuckDB equivalence matrix (Iceberg reference
vectors included) that fails loudly on any dialect drift. Known limits:
identity partitions on bytea fail upstream while building partition values
from non-UTF8 bytes; DuckDB accepts only hour() on TIME; timestamptz
transforms depend on the session TimeZone, which DuckDB syncs from PG only
at instance creation (a later SET timezone can make writer and bookkeeping
disagree near boundaries - accepted limitation of evaluating in two
engines).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qsliu2017
force-pushed
the
fix/223-flush-partition-filter-dialect
branch
from
July 15, 2026 09:55
2b9f879 to
0ee8039
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CALL ducklake.set_partition(tbl, 'month(col)')(or any non-identity transform: year/month/day/hour/bucket) is accepted, but every subsequentducklake.flush_inlined_data()fails withIO Error: SPI execution failed: function month(timestamp with time zone) does not exist(ormurmur3_32(...)for bucket partitions). The maintenance worker keeps retrying the flush and leaves orphaned parquet files behind on each attempt. Root cause: DuckLake embeds the partition expression in DuckDB dialect in the flush's per-file deleted-rows bookkeeping query, and pg_ducklake ships that query to PostgreSQL over SPI, whose parser has no such functions.Per maintainer decision, the bookkeeping query stays on the PG/SPI path and the dialect gap is filled instead: the new extension objects are
ducklake.year/month/day/hour(date, timestamp, timestamptz, plushour(time)-- the only transform DuckDB accepts on TIME) andducklake.murmur3_32overloads for boolean, all integer widths, real, double precision, text, bytea, date, time, timestamp, timestamptz, numeric, and uuid. The murmur3 implementations reuse the vendored DuckLake murmur3 header, so they are bit-exact with the writer by construction; the SPI executor forcessearch_path=ducklakefor metadata queries (via the existing GUC nest-level pattern) so the unqualified names resolve deterministically and cannot collide with e.g. orafce'smonth()in public.Verifying against writer-side ground truth surfaced two additional silent-corruption bugs, both fixed here. First, the inlined table stores text columns as BYTEA, so the upstream filter's
CAST(col AS VARCHAR)compared PG's hex form ('\x6170706c65') instead of the string -- every text bucket mis-hashed and every text identity comparison missed, writing delete tombstones onto the wrong rows. Second, without an exactnumericoverload PostgreSQL resolvesmurmur3_32(numeric)to thedouble precisionone (the preferred numeric type) and silently mis-buckets. Fixing the first required a small new vendored seam (ducklake-010patch): a metadata-manager virtual through which the backend renders each partition column per its inlined storage type (convert_from()for text-as-BYTEA, raw column for bytea,::textfor decimals wider than 18 digits, PG-parseable type names otherwise).One documented limitation: timestamptz calendar transforms depend on the session TimeZone in both engines, and DuckDB syncs its TimeZone from PostgreSQL only once, at instance creation -- a mid-session
SET timezonecan make writer and bookkeeping disagree near month/year boundaries. Accepted as inherent to evaluating in two engines.Tests: nine flush-with-delete scenarios (month on timestamptz incl. NULL partitions, bucket on int/text/numeric(18,3)/numeric(30,10)/uuid/bytea, identity on text, hour on time) plus a PG-vs-DuckDB equivalence matrix pinned to the Iceberg spec reference vectors that fails loudly on any dialect drift. Full regression suite: 52/52 on main.
This PR now targets
mainbecause it adds SQL interface (new extension objects, shipped in the in-development 1.1.0 update script). Orphaned parquet files left by past failed flushes are recoverable withducklake.cleanup_orphaned_files().Fixes #223
🤖 Generated with Claude Code