feat: maintain inlined-data column stats in the direct-insert writer

[GavinPizza]

The direct-insert writer (CreateSnapshotForDirectInsert) does not maintain ducklake_table_column_stats min/max or contains_null for the rows it inlines. Reads stay correct only because the vendored guard ducklake-006-inlined-scan-stats-guard.patch suppresses statistics for the whole table, permanently, for anything that has ever inlined. Details in the issue.

#217 fixed the same function on v1.0 by advancing next_file_id and degrading the column stats to NULL, and named the precise-stats version as a refactor targeting main. This is that refactor. main has neither half today, so in this function it is behind the release branch.

Widening instead of degrading

Each inserted batch widens the persisted bounds: never shrink, never blank, never seed. "Never seed" is per bound rather than per row, because a bound derived from one batch is not a bound over the table. So the writer never creates a stats row for a column that has none, and never fills a missing min or max on a row that already carries the other one. A NULL bound only costs pruning; a too-narrow one is silently wrong, and not only for pruning, since DuckDB's compressed materialization uses a table-level min as an exact re-base constant.

Decoding goes through FromGlobalStats and merging through MergeStats, so the comparison semantics cannot drift from the engine's. Per snapshot that is one SELECT plus one UPDATE, skipped when no bound moved.

Write-path cost, measured on this branch: +7% at batch=1000 and +20% at batch=100. The residual is flat per-statement overhead rather than a width effect (it shows identically at one column, 143 vs 114 ms), so cost no longer scales with column count: width sensitivity is 5.7 ms/col against a 5.4 baseline, where a naive per-column implementation cost 26.7. The accumulator folds PG Datums instead of building a duckdb::Value per cell, which is a deliberate departure from reusing the engine's types and worth about 6x on this path for byte-identical bounds.

This supersedes v1.0's degrade-to-NULL rather than porting it: blanking wipes bounds for already-flushed parquet data too, which bakes in wrong results after a later flush.

contains_null

Retiring 006 means the reader trusts the whole stats row, not just the bounds, and contains_null is part of it. Left stale at false while an inlined NULL exists, DuckDB folds IS NULL to false and the row disappears from the result, while IS NOT NULL reports it as non-NULL. That is a wrong answer rather than lost pruning, so retiring the guard has to cover it for the same reason it has to cover the bounds.

It rides the same MergeStats call, as null_count, which the catalog persists as the boolean. The OR, the monotone false->true and the sticky unknown-degrade are all the engine's rather than a second implementation here. MergeStats folds null_count before its !AnyValid() early return, so an all-NULL batch updates null-ness without touching bounds.

Counting is deliberately not gated on the min/max eligibility test. That test gates comparability, which has no bearing on null-ness, and native's inline writer (TemplatedUpdateStats) counts NULLs for every column whatever its type. Gating it would make the fast path disagree with the normal path on the same INSERT. All three inline writers do it -- VALUES, UNNEST and COPY FROM STDIN -- each having its own null branch.

On a table that has only ever held inlined rows, contains_null stays SQL NULL rather than becoming true: MergeStats' has_null_count degrade is sticky and no widen can recover it. That is native behavior and it is safe, since unknown makes no HasNoNull claim and nothing folds.

next_file_id

Forward-ports #217's first fix, load-bearing here for a second reason: DuckLake keys its table-stats cache on next_file_id, which an inline insert does not advance, so a scan taken between two inline inserts in one session serves the pre-widening cached zone map however truthful the catalog is. With both halves in place, 006 retires with no replacement vendored patch: net -1.

It also fixes #217's Drift 1 corruption on main, which is data loss rather than lost pruning. The stale cached stats carry next_row_id, so a normal-path insert after direct inserts in the same backend reuses row ids: on main today such a table holds 4 rows with 3 distinct row_ids, and deleting one row destroys an unrelated one. Verified by A/B on both binaries.

Disclosures

Retiring 006 changes behavior on tables whose persisted bound can no longer be cast to the column's current type: four query shapes that work today start throwing at plan time. Those tables are already broken at the data level (SELECT * throws either way), and the throw is upstream's, confirmed on stock DuckDB and stock ducklake with no vendored patches. 006 only moved it from planning to execution. The 11-shape matrix is in the issue. If you would rather land the upstream read-path fix first, re-adding the guard is a one-hunk revert and I can split it out.

The accumulator forces ISO DateStyle inside Finalize and restores it from a destructor, which a PG elog(ERROR) longjmps past, so an error there can leave the session pinned to ISO/YMD; neither ROLLBACK nor RESET ALL undoes it. direct_insert.cpp already does the same thing twice unguarded across a much wider window (:1526-1531 restored at :1604-1607), spanning the conversion loop where errors actually happen, so fixing only the new and narrower instance would leave the one users hit. I would rather fix the pattern in a follow-up, either with the NewGUCNestLevel() idiom or by encoding temporal bounds without touching the global, but can fold it in here if you would prefer not to merge another instance.

Tests

inlined_scan_stats_guard reworked to assert maintained bounds: fully-inlined widening, multi-file seed/flush/insert/flush, a half-populated stats row reached through supported SQL, DateStyle independence, and the between-inserts stale-cache case. Both properties have negative controls.

Also forward-ports #217's direct_insert_metadata, which main never received. Its next_file_id and row-id-uniqueness sections pass byte-for-byte against v1.0's expected output. Its third section asserted that the fast path degrades stats to NULL because it cannot maintain them, which this change makes false, so it now asserts the widened bounds and contains_null instead.

Three further sections cover contains_null: one column per inline writer, so a fix reaching VALUES and COPY but missing UNNEST still fails; the min/max-ineligible types (DOUBLE, BLOB), which only this change makes reach the widen at all; and the inline-only table, asserting the value stays unknown and that IS NULL still returns the row across a flush.

Regression 54/54 and isolation 3/3 on PG 18.4.

If this is backported to v1.0: that branch degrades the stats to NULL in the same function (pgducklake_metadata_manager.cpp:833), and the backport must delete that statement. Left in, it runs after the widen and blanks the bounds this change just computed, silently and without failing any test.

On the follow-up refactor #217 also promised, replacing this hand-written SQL with UpdateGlobalTableStatsSql: deliberately not attempted here. That builder also writes contains_nan, extra_stats and ducklake_table_stats, and its INSERT branch would create the stats rows this change is careful never to create. It is worth doing, but as its own PR rather than inside the one making these bounds correct.

Fixes #237