Skip to content

Add DuckLake table format support (pg_lake_ducklake)#414

Draft
sfc-gh-mslot wants to merge 1 commit into
mainfrom
marcoslot/ducklake-rebased
Draft

Add DuckLake table format support (pg_lake_ducklake)#414
sfc-gh-mslot wants to merge 1 commit into
mainfrom
marcoslot/ducklake-rebased

Conversation

@sfc-gh-mslot

@sfc-gh-mslot sfc-gh-mslot commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

This PR adds a new pg_lake_ducklake extension implementing the DuckLake v1 spec with the catalog stored entirely in PostgreSQL. DuckLake tables are foreign tables on a dedicated server backed by the same FDW machinery as pg_lake_table: PostgreSQL owns transaction boundaries and DDL, DuckDB (inside pgduck_server) does the columnar execution, and the catalog itself is plain PostgreSQL tables. The catalog is exposed in two layers, lake_ducklake.* (internal metadata tables) and public.ducklake_* (the spec views DuckDB expects, with INSTEAD-OF triggers routing writes back). Writes participate in the surrounding PG transaction, snapshots are allocated lazily and committed atomically, and DDL flows both ways: PG-side CREATE/ALTER/DROP update the catalog, and DuckDB-side DDL replays back into PostgreSQL via the view triggers.

Concurrent PG-side writers serialise behind a transaction-scoped advisory lock around snapshot allocation. That lock doesn't cover external DuckDB writers, but conflicts between PG and DuckDB writes are still resolved by DuckLake's primary-key-based optimistic concurrency control on the snapshot id, so a losing writer retries.

From PostgreSQL, a DuckLake table is just USING ducklake:

-- set a default storage prefix once; tables land at <prefix>/<schema>/<table>/
set pg_lake_ducklake.default_location_prefix to 's3://mybucket/lake';

-- create a DuckLake table and write to it like any other table
create table measurements (
  station_name text not null,
  measurement  double precision not null
) using ducklake;

insert into measurements values ('Istanbul', 18.5), ('Athens', 21.7);

-- reads, UPDATE/DELETE (position deletes), partitioning, CTAS all work
select station_name, avg(measurement) from measurements group by 1;

From DuckDB, attach the same PG-hosted catalog over the postgres bridge and read/write the same tables and data files:

-- in DuckDB
INSTALL ducklake; INSTALL postgres; LOAD ducklake;

-- point ducklake at the PG-hosted catalog (no DATA_PATH needed if the
-- catalog's data_path is set on the PG side)
ATTACH 'postgres:host=pg.example dbname=lakehouse user=alice' AS dl (TYPE DUCKLAKE);

-- tables created from PostgreSQL show up here; writes land in the same catalog
SELECT * FROM dl.public.measurements;
INSERT INTO dl.public.measurements VALUES ('Berlin', 12.0);

Full usage, options, partitioning, role model, and limitations are in docs/ducklake.md. Supporting changes in pg_lake_engine and pg_lake_table (data-file stats, batch catalog inserts, field-id mapping, partition-spec catalog) are shared with the existing Iceberg path.

Limitations

  • No partition spec evolution after CREATE TABLE. The spec is fixed at create time.
  • No time-travel reads yet. lake_ducklake.snapshot is fully versioned, but nothing plumbs a historical snapshot_id into scans.
  • No incremental (change-feed) reads. Scans always read the latest snapshot; there's no way to read just the rows changed between two snapshots.
  • Inlined data is disabled. The extension forces data_inlining_row_limit = 0 so DuckDB always writes a Parquet file rather than materialising small INSERTs into a catalog table, because the FDW can't read inlined rows yet.
  • CREATE VIEW from the DuckDB side persists lake_ducklake.view rows but isn't materialised as a PostgreSQL view, so those views aren't visible from PostgreSQL.
  • DROP SCHEMA on the DuckDB side isn't propagated back to drop the PostgreSQL schema (intentionally, to preserve grants).
  • DuckDB macros and the column_tag / tag / sort_info / column_mapping / name_mapping columns are stored but not yet surfaced or exercised by reads.

Test plan

  • pytest suite for pg_lake_ducklake (basic, writes, CTAS, alter table, partitioning, snapshots, role grants, metadata spec, injection/crash)
  • DuckDB interop suite (test_e2e_interop.py, test_duckdb_round_trip.py, test_concurrency_pg_vs_duckdb.py, test_interop_types_names_values.py) covering bidirectional read/write and DuckDB-driven compaction
  • pg_lake_table DuckLake DDL/DML tests + isolation specs for concurrent DuckLake DDL and DML
  • CI (build + full check)

@sfc-gh-mslot
sfc-gh-mslot force-pushed the marcoslot/ducklake-rebased branch 4 times, most recently from 851ca4b to 3850a5f Compare June 22, 2026 12:31
Introduces a new pg_lake_ducklake extension that hosts a DuckLake
catalog inside PostgreSQL while continuing to share pg_lake_table's
foreign-data-wrapper machinery. PostgreSQL stores the catalog in
lake_ducklake.* and exposes the spec-defined ducklake_* views in
public; DuckDB ATTACHes the same database over the postgres bridge
and reads/writes through those views.

Highlights:

- DuckLake v1 metadata catalog (snapshots, schemas, tables, columns,
  data_file/delete_file, partition_info, file_column_stats, ...) with
  INSTEAD-OF triggers on the public ducklake_* views so DuckDB writes
  go through validated paths.
- pg_lake_ducklake.so (separate from pg_lake_table.so) owns DuckLake-
  specific catalog functions, replay handlers, drop hooks, and the
  privileged-SPI helpers; pg_lake_table calls in via PGDLLEXPORT
  externs.
- USING ducklake / USING pg_lake_ducklake access methods rewrite
  CREATE TABLE into a foreign table on the dedicated pg_lake_ducklake
  server.
- Bidirectional DDL replay: CREATE/DROP/ALTER TABLE, ADD/DROP/RENAME
  COLUMN, ALTER SCHEMA RENAME, CREATE SCHEMA all flow PG <-> DuckDB
  with a DucklakeInDDLReplay guard preventing duplicate version rows.
- PG-side INSERT/UPDATE/DELETE participate in the DuckLake snapshot
  loop with row-id allocation, position-delete files, and per-table
  partition routing.
- Object-access hook end-snapshots lake_ducklake.schema rows on PG
  DROP SCHEMA and revives them on re-CREATE so grants survive.
- Concurrency: PG-side snapshot allocation serialises behind a
  transaction-scoped advisory lock (class 102) so concurrent PG
  writers don't trip DuckLake's PK-based OCC; external DuckDB writers
  still rely on the spec's retry loop. End-of-statement data_file /
  delete_file lookups in DucklakeRemoveDataFileByPath /
  DucklakeUpdateDeleteFileDataFileId run with read_only=false so SPI
  takes a fresh snapshot after we acquire the class-101 update lock --
  read_only=true reused the statement's pre-lock snapshot and missed
  rows committed by another transaction that already released its
  lock, leaving stale parquet files end-snapshotted incorrectly.
  PG-side UPDATE/DELETE on ducklake tables always uses merge-on-read
  (position-delete file + new data file), never copy-on-write -- COW
  end-snapshots the source data_file, but DuckDB's extension always
  writes merge-on-read deletes and does not take the class-101 lock,
  so a concurrent DuckDB UPDATE landing while PG COW'd would produce
  a delete_file pointing at a now-dead source plus PG's COW survivors
  file holding rows DuckDB had updated separately. Forcing
  merge-on-read on the PG side makes both engines' delete_files
  compose additively at read time.
- DucklakeRegisterTable persists data_path / schema-relative paths so
  catalogs are portable across object-store prefixes.
- DuckLake roles (ducklake_superuser, ducklake_writer, ducklake_reader)
  follow the upstream taxonomy
  (ducklake.select/docs/stable/duckdb/guides/access_control), granted
  on the public.ducklake_* views rather than the underlying
  lake_ducklake catalog tables.
- ~100 pytest cases covering catalog spec, lifecycle, partitioning,
  schema rename/revive, DuckDB compaction, three-leg PG <-> DuckDB
  round-trip, and 8 exotic-type / weird-name / edge-value interop
  tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Slot <marco.slot@snowflake.com>
@sfc-gh-mslot
sfc-gh-mslot force-pushed the marcoslot/ducklake-rebased branch from 3850a5f to 7a44b3a Compare June 22, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant