Add DuckLake table format support (pg_lake_ducklake)#414
Draft
sfc-gh-mslot wants to merge 1 commit into
Draft
Conversation
sfc-gh-mslot
force-pushed
the
marcoslot/ducklake-rebased
branch
4 times, most recently
from
June 22, 2026 12:31
851ca4b to
3850a5f
Compare
Introduces a new pg_lake_ducklake extension that hosts a DuckLake catalog inside PostgreSQL while continuing to share pg_lake_table's foreign-data-wrapper machinery. PostgreSQL stores the catalog in lake_ducklake.* and exposes the spec-defined ducklake_* views in public; DuckDB ATTACHes the same database over the postgres bridge and reads/writes through those views. Highlights: - DuckLake v1 metadata catalog (snapshots, schemas, tables, columns, data_file/delete_file, partition_info, file_column_stats, ...) with INSTEAD-OF triggers on the public ducklake_* views so DuckDB writes go through validated paths. - pg_lake_ducklake.so (separate from pg_lake_table.so) owns DuckLake- specific catalog functions, replay handlers, drop hooks, and the privileged-SPI helpers; pg_lake_table calls in via PGDLLEXPORT externs. - USING ducklake / USING pg_lake_ducklake access methods rewrite CREATE TABLE into a foreign table on the dedicated pg_lake_ducklake server. - Bidirectional DDL replay: CREATE/DROP/ALTER TABLE, ADD/DROP/RENAME COLUMN, ALTER SCHEMA RENAME, CREATE SCHEMA all flow PG <-> DuckDB with a DucklakeInDDLReplay guard preventing duplicate version rows. - PG-side INSERT/UPDATE/DELETE participate in the DuckLake snapshot loop with row-id allocation, position-delete files, and per-table partition routing. - Object-access hook end-snapshots lake_ducklake.schema rows on PG DROP SCHEMA and revives them on re-CREATE so grants survive. - Concurrency: PG-side snapshot allocation serialises behind a transaction-scoped advisory lock (class 102) so concurrent PG writers don't trip DuckLake's PK-based OCC; external DuckDB writers still rely on the spec's retry loop. End-of-statement data_file / delete_file lookups in DucklakeRemoveDataFileByPath / DucklakeUpdateDeleteFileDataFileId run with read_only=false so SPI takes a fresh snapshot after we acquire the class-101 update lock -- read_only=true reused the statement's pre-lock snapshot and missed rows committed by another transaction that already released its lock, leaving stale parquet files end-snapshotted incorrectly. PG-side UPDATE/DELETE on ducklake tables always uses merge-on-read (position-delete file + new data file), never copy-on-write -- COW end-snapshots the source data_file, but DuckDB's extension always writes merge-on-read deletes and does not take the class-101 lock, so a concurrent DuckDB UPDATE landing while PG COW'd would produce a delete_file pointing at a now-dead source plus PG's COW survivors file holding rows DuckDB had updated separately. Forcing merge-on-read on the PG side makes both engines' delete_files compose additively at read time. - DucklakeRegisterTable persists data_path / schema-relative paths so catalogs are portable across object-store prefixes. - DuckLake roles (ducklake_superuser, ducklake_writer, ducklake_reader) follow the upstream taxonomy (ducklake.select/docs/stable/duckdb/guides/access_control), granted on the public.ducklake_* views rather than the underlying lake_ducklake catalog tables. - ~100 pytest cases covering catalog spec, lifecycle, partitioning, schema rename/revive, DuckDB compaction, three-leg PG <-> DuckDB round-trip, and 8 exotic-type / weird-name / edge-value interop tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Slot <marco.slot@snowflake.com>
sfc-gh-mslot
force-pushed
the
marcoslot/ducklake-rebased
branch
from
June 22, 2026 12:59
3850a5f to
7a44b3a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new
pg_lake_ducklakeextension implementing the DuckLake v1 spec with the catalog stored entirely in PostgreSQL. DuckLake tables are foreign tables on a dedicated server backed by the same FDW machinery aspg_lake_table: PostgreSQL owns transaction boundaries and DDL, DuckDB (inside pgduck_server) does the columnar execution, and the catalog itself is plain PostgreSQL tables. The catalog is exposed in two layers,lake_ducklake.*(internal metadata tables) andpublic.ducklake_*(the spec views DuckDB expects, with INSTEAD-OF triggers routing writes back). Writes participate in the surrounding PG transaction, snapshots are allocated lazily and committed atomically, and DDL flows both ways: PG-sideCREATE/ALTER/DROPupdate the catalog, and DuckDB-side DDL replays back into PostgreSQL via the view triggers.Concurrent PG-side writers serialise behind a transaction-scoped advisory lock around snapshot allocation. That lock doesn't cover external DuckDB writers, but conflicts between PG and DuckDB writes are still resolved by DuckLake's primary-key-based optimistic concurrency control on the snapshot id, so a losing writer retries.
From PostgreSQL, a DuckLake table is just
USING ducklake:From DuckDB, attach the same PG-hosted catalog over the postgres bridge and read/write the same tables and data files:
Full usage, options, partitioning, role model, and limitations are in
docs/ducklake.md. Supporting changes inpg_lake_engineandpg_lake_table(data-file stats, batch catalog inserts, field-id mapping, partition-spec catalog) are shared with the existing Iceberg path.Limitations
CREATE TABLE. The spec is fixed at create time.lake_ducklake.snapshotis fully versioned, but nothing plumbs a historicalsnapshot_idinto scans.data_inlining_row_limit = 0so DuckDB always writes a Parquet file rather than materialising small INSERTs into a catalog table, because the FDW can't read inlined rows yet.CREATE VIEWfrom the DuckDB side persistslake_ducklake.viewrows but isn't materialised as a PostgreSQL view, so those views aren't visible from PostgreSQL.DROP SCHEMAon the DuckDB side isn't propagated back to drop the PostgreSQL schema (intentionally, to preserve grants).column_tag/tag/sort_info/column_mapping/name_mappingcolumns are stored but not yet surfaced or exercised by reads.Test plan
pg_lake_ducklake(basic, writes, CTAS, alter table, partitioning, snapshots, role grants, metadata spec, injection/crash)test_e2e_interop.py,test_duckdb_round_trip.py,test_concurrency_pg_vs_duckdb.py,test_interop_types_names_values.py) covering bidirectional read/write and DuckDB-driven compactionpg_lake_tableDuckLake DDL/DML tests + isolation specs for concurrent DuckLake DDL and DML