ci: add PostgreSQL 19beta1 to build/test matrix, bump minor versions#363
Closed
sfc-gh-dachristensen wants to merge 11 commits into
Closed
ci: add PostgreSQL 19beta1 to build/test matrix, bump minor versions#363sfc-gh-dachristensen wants to merge 11 commits into
sfc-gh-dachristensen wants to merge 11 commits into
Conversation
sfc-gh-dachristensen
force-pushed
the
add-pg19-ci
branch
6 times, most recently
from
June 4, 2026 18:14
94025c6 to
0714718
Compare
Bumps PG 16/17/18 to latest minors (16.14, 17.10, 18.4) and wires PG 19beta1 into the Dockerfile, CI matrices, install.sh, and docker docs/Taskfile. Code-level fixes for PG 19 are deliberately deferred; this commit just sets up the infrastructure so failures surface in CI. Every PG19 matrix leg is marked continue-on-error so failures surface in CI without blocking the PR's required checks. The build-and-install jobs also flip fail-fast: false so a PG19 build failure does not cancel the 16/17/18 legs. Drive-by: Oracle's /java/21/latest tarball bumped past jdk-21.0.10, which broke the debian image build with "cannot stat '/usr/lib/jvm/jdk-21.0.10'". Read the top-level directory from the tarball instead of hardcoding the patch version. Signed-off-by: David Christensen <david.christensen@snowflake.com>
sfc-gh-dachristensen
force-pushed
the
add-pg19-ci
branch
from
June 4, 2026 18:39
0714718 to
57fcd19
Compare
6 tasks
PG19 dropped a number of transitive header includes and reshaped a few
APIs. Add explicit includes wherever the build broke and shim the
renamed APIs in pg_extension_base/pg_compat.h.
Header reorg (PG19 dropped these from common transitive paths):
- utils/tuplestore.h (was pulled in via nodes/execnodes.h)
- access/htup_details.h (was pulled in via executor/tuptable.h)
- utils/hsearch.h (HASHCTL/hash_create/hash_search)
- catalog/pg_type_d.h (TEXTOID, INTERVALOID, etc.)
- utils/wait_event.h (WAIT_EVENT_CLIENT_READ)
- utils/timestamp.h (Datum<-->TimestampTz, IntervalPGetDatum)
- storage/lock.h (LockAcquire, LockAcquireResult)
- storage/fd.h (AllocateFile, FreeFile)
API changes wrapped behind #if PG_VERSION_NUM >= 190000:
- LWLockNewTrancheId() now requires a name argument; the separate
LWLockRegisterTranche() helper is gone.
- ShmemInitHash() collapsed init_size/max_size into a single nelems.
- SIG_IGN -> PG_SIG_IGN (pqsigfunc-typed sentinel).
- pg_plan_query() gained a trailing ExplainState* (pass NULL).
- CopyFormatOptions.binary / .csv_mode were replaced by an enum
.format field — wrapped behind CopyOptsIsBinary/CopyOptsIsCsvMode
macros in pg_compat.h so callers keep the older names.
- CopyHeaderChoice enum was removed; header_line is a plain int with
COPY_HEADER_TRUE/FALSE/MATCH constants. Reintroduced the typedef
locally so our exported API signature stays stable.
- numeric_int4_opt_error() renamed to numeric_int4_safe() and switched
from a bool* error flag to an ErrorSaveContext* (soft-error API).
Wrapped via an inline shim that preserves the older bool* signature.
pgduck_server is a frontend program; the unused #include "nodes/nodes.h"
in numutils.h started failing because PG19 changed nodes.h to use Datum
in extern declarations. Removed the vestigial include.
Builds clean against PG19 locally for: pg_extension_base, pg_map,
pg_lake_engine, pg_lake_copy, pg_lake. pg_lake_iceberg/pg_lake_table
were not exercised locally because the avro library was not built;
they're expected to need similar include fixes already applied here.
Signed-off-by: David Christensen <david.christensen@snowflake.com>
PG19's ProcedureCreate() now walks each new function's dependencies through get_object_namespace() (via find_temp_object), which does syscache lookups. Without a CommandCounterIncrement after DefineDomain, the freshly-created domain type isn't visible to syscache, so the very next map_create_function call ERRORs with "cache lookup failed for cache 90 oid <new domain oid>". Signed-off-by: David Christensen <david.christensen@snowflake.com>
Signed-off-by: David Christensen <david.christensen@snowflake.com>
PG19's libpq defaults max_pversion to PG_PROTOCOL_GREASE (3,9999) so it can probe whether a server speaks NegotiateProtocolVersion. pgduck_server previously slammed the connection on any protocol other than 3.0, which made every PG19 cache worker / pg_lake client fail to attach with "server closed the connection unexpectedly". When a client requests a 3.x version higher than 3.0, send a 'v' (NegotiateProtocolVersion) message advertising 3.0 and zero unrecognized options, mirroring what core PostgreSQL does in backend_startup.c. libpq accepts the downgrade transparently; older clients (PG <=18) that default to 3.0 are unaffected. Signed-off-by: David Christensen <david.christensen@snowflake.com>
…ersion PG19's libpq doesn't just probe protocol minor versions — when probing with PG_PROTOCOL_GREASE it also sends a deliberately-unknown protocol option named "_pq_.test_protocol_negotiation" and verifies the server echoes it back in the NegotiateProtocolVersion (`v`) message. Without that echo, libpq aborts with "server did not report the unsupported ... parameter in its protocol negotiation message". Scan the startup packet body for any name beginning with "_pq_." and include those names in the `v` reply, mirroring core's ProcessStartupPacket / SendNegotiateProtocolVersion behavior. Signed-off-by: David Christensen <david.christensen@snowflake.com>
Signed-off-by: David Christensen <david.christensen@snowflake.com>
PG19 added a firstNonCachedOffsetAttr cache to TupleDesc and asserts in BlessTupleDesc() that it has been populated. Manually-constructed descriptors (CreateTemplateTupleDesc + TupleDescInitEntry) now require an explicit TupleDescFinalize() call before they can be Blessed or used to form heap tuples; otherwise PG19 backends abort with "failed Assert(\"tupdesc->firstNonCachedOffsetAttr >= 0\")". Add a no-op TupleDescFinalize() shim in pg_compat.h for PG <=18 so callers can finalize unconditionally, and finalize at every site that builds a TupleDesc by hand. Drive-by: replace the C99 designated initializer for ErrorSaveContext in numeric_int4_opt_error() with explicit memset+field assignment, so pg_lake_copy (compiled with -Wmissing-field-initializers / -Werror) no longer rejects the new file. Signed-off-by: David Christensen <david.christensen@snowflake.com>
Four independent regressions surfaced after the earlier batch: 1. pg_lake_copy: stop intercepting plain COPY ... TO STDOUT (FORMAT json) on PG18+. PG18 added native COPY-to-stdout JSON support and PG19 wired the regression suite to exercise force_array / on_error / reject_limit etc. Our hook was matching on format alone and rejecting these new options with "pg_lake_copy: invalid option ...". Now we only claim the JSON path when the destination is a URL or @stage; STDIN/STDOUT JSON is left to core. 2. pg_lake_table: skip system columns (varattno <= 0) in AddDeleteReturningTargetTableVars. PG19 added an Assert in TupleDescAttr() that fires on negative indexes; PG18 silently returned garbage. The pull_vars_of_level() walk over a DELETE RETURNING list can include system columns when the test triggers a re-DELETE after a previous "cannot be updated by multiple operations in a single command" error. 3. pg_lake_table: switch the synthetic-test-query pull_var_clause() in restriction_collector to PVC_RECURSE_AGGREGATES | PVC_RECURSE_PLACEHOLDERS so we descend into Aggref/PlaceHolderVar instead of elog'ing "Aggref found where not expected". PG19's planner leaves Aggrefs inside FDW-pushdown restriction clauses for some benchmark queries (TPC-DS Q10, TPC-H Q21). 4. Isolation tests: add `_1.out` alternative expected outputs for isolation_iceberg_serializable and isolation_iceberg_repeatable_read. PG19 split the previously-unified "could not serialize access due to concurrent update" message into "concurrent update" vs "concurrent delete" depending on TM_Updated vs TM_Deleted. The 5 affected lines are the cases where the conflicting transaction issued a DELETE. Signed-off-by: David Christensen <david.christensen@snowflake.com>
Three independent regressions surfaced after round 2: 1. pg_lake_table: skip Iceberg fields with no matching Postgres column in CreatePostgresColumnMappingsForIcebergTableFromExternalMetadata, regardless of catalog type. Previously only REST_CATALOG_READ_ONLY short-circuited; OBJECT_STORE_READ_ONLY (and the internal flow when the read-only schema check eventually rejects the table) fell through with attrNum=InvalidAttrNumber=0 and indexed TupleDesc with -1, which PG19 catches via Assert(i >= 0 && i < natts) inside TupleDescAttr(). The downstream caller (ErrorIfSchemasDoNotMatch) still produces the user-facing "Schema mismatch ..." error from the now-shorter list. 2. pg_lake_table: add PVC_RECURSE_AGGREGATES | PVC_RECURSE_PLACEHOLDERS to the second pull_var_clause() site (data_file_pruning.c ColumnsUsedInRestrictions) so it descends into Aggref / PlaceHolderVar instead of elog'ing "Aggref found where not expected" on PG19's TPC-H Q21 / TPC-DS Q10 plans. This mirrors the equivalent fix in restriction_collector.c from the previous batch. 3. pg_lake_copy: revert the JSON STDIN/STDOUT deferral added in round 2. The PG18 build does not recognize FORMAT json at the parser level, so handing JSON COPY off to core there breaks pg_lake_copy's own JSON tests. PG19's `installcheck-postgres` copy.out diff (driven by core's new force_array / on_error / encoding / reject_limit tests) is left unfixed in this PR; it is a known-failing leg until we either align pg_lake_copy's option validation with core or carry an alternative expected file for the upstream copy test. Signed-off-by: David Christensen <david.christensen@snowflake.com>
Two regressions remained after round 3:
1. The schema-mismatch fix in CreatePostgresColumnMappingsForIcebergTableFromExternalMetadata
skipped Iceberg fields with no Postgres counterpart, which made the
downstream check fail with "field count 2 vs 1" instead of the
expected "field ids 2 vs 2" message and broke test_object_store_catalog
on PG16/17/18 too. Restore the original behavior of always pushing a
PostgresColumnMapping but guard the TupleDescAttr() lookup so we only
touch the descriptor when we have a valid attrNum. The mapping's
pgType / attNotNull / attHasDef stay zero-initialised in the missing
case, which still trips the per-field check in
ErrorIfSchemasDoNotMatch() and surfaces the expected diagnostic
without indexing the TupleDesc with -1 (which PG19 catches via the
new Assert(i >= 0 && i < natts) inside TupleDescAttr()).
2. PG19's planner can leave Aggrefs inside the FDW's reltarget exprs
and EPQ pathtarget exprs on TPC-H Q21 / TPC-DS Q10 plans, so add
PVC_RECURSE_AGGREGATES to the three remaining pull_var_clause sites
that previously only set PVC_INCLUDE/RECURSE_PLACEHOLDERS:
- semijoin_target_ok (joinrel->reltarget->exprs)
- postgresGetForeignSortPaths EPQ rewrite (target->exprs and
fpinfo->local_conds)
- build_tlist_to_deparse (foreignrel->reltarget->exprs and
fpinfo->local_conds)
Together with the data_file_pruning + restriction_collector fixes
from round 2/3, this clears the "Aggref found where not expected"
elog on benchmark queries.
Signed-off-by: David Christensen <david.christensen@snowflake.com>
Collaborator
Author
|
Closing in lieu of #404 (shares a lot of commit lineage anyway). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds PostgreSQL 19beta1 as a first-class target across the dev container, CI, install script, and the C codebase. After this lands, every extension compiles against PG
19beta1, the docker images build green, and the CI matrix exercises 19 alongside 16/17/18.
This PR has two halves:
versions required for PG19.
#if PG_VERSION_NUM >= 190000.Infrastructure changes
docker/Dockerfile,install.sh, and docker docs/Taskfile.gserialized_typmod.c:INT_MIN/INT_MAXundeclared); upstream fix is PostGIS commita7c7a83("Avoid build issue on PG19"), first released in 3.6.1.
postgresql-19branch from CyberDem0n's fork (citusdata/pg_cron PR pg_lake_copy: reject multidimensional arrays in COPY TO before CSV serialization #430) until the PG19 patch is merged upstream. The patch isbackward-compatible with PG 16/17/18.
bzip2added to the almalinux and debian dep lists — the 19beta1 source ships as.tar.bz2.CI matrix changes
build-and-install-{almalinux,debian}:[16, 17, 18, 19]test-check,test-check-pg-lake-table,test-isolation:[17, 18, 19]test-installcheck-postgres:[16, 17, 18, 19]test-upgrade: added pairs17-19,18-19,16-19test-installcheckandtest-installcheck-pg-lake-tableremain[16]PG19 matrix legs are marked
continue-on-errorso any remaining PG19-specific test failures surface in CI without blocking required checks. Thebuild-and-install-*jobs flip
fail-fast: falseso a PG19 issue does not cancel the 16/17/18 legs.Drive-by
Oracle's
/java/21/latesttarball started bumping pastjdk-21.0.10, which broke the debian image build withcannot stat '/usr/lib/jvm/jdk-21.0.10'. The Dockerfilenow reads the top-level directory from the tarball instead of hardcoding the patch version.
Code-level fixes
PG19 dropped a number of transitive header includes and reshaped a handful of APIs. Most of the diff is just adding the explicit
#includethat PG18 was gettingtransitively. Real API changes are version-gated.
Header reorg
PG19 dropped these from common transitive include paths; affected
.c/.hfiles now include them explicitly:utils/tuplestore.hnodes/execnodes.haccess/htup_details.hexecutor/tuptable.hutils/hsearch.hHASHCTL,hash_create,hash_search,hash_seq_*catalog/pg_type_d.hTEXTOID,INTERVALOID, etc.utils/wait_event.hWAIT_EVENT_CLIENT_READutils/timestamp.hDatumGetTimestampTz,TimestampTzGetDatum,IntervalPGetDatumstorage/lock.hLockAcquire,LockAcquireResult,LOCKTAGstorage/fd.hAllocateFile,FreeFile,OpenTemporaryFile,FileClose<math.h>isnan,isinfAPI changes
Wrapped behind
#if PG_VERSION_NUM >= 190000:LWLockNewTrancheId()now requires a name argument; the separateLWLockRegisterTranche()helper is gone.ShmemInitHash()collapsedinit_size/max_sizeinto a singlenelems.SIG_IGN→PG_SIG_IGN(pqsigfunc-typed sentinel) when used withpqsignal().pg_plan_query()andplanner_hook_typegained a trailingExplainState *argument.DefineIndex()gained a leadingParseState *argument.bits16type was removed;format_type_extendedflags are nowuint16.utils/dynahash.hwas deleted (only declaredmy_log2); the unused includes are wrapped behindPG_VERSION_NUM < 190000.CopyFormatOptions.binary/.csv_modewere replaced by an enum.formatfield. Wrapped behindCopyOptsIsBinary/CopyOptsIsCsvModemacros inpg_compat.hso callers keep the older names.
CopyHeaderChoiceenum was removed;header_lineis now a plainintwithCOPY_HEADER_TRUE/FALSE/MATCHconstants. Reintroduced the typedef locally so theexported API signature stays stable.
numeric_{int4,add,sub,mul,div,mod}_opt_error()renamed to_safe()variants and switched from abool *error flag to anErrorSaveContext *(soft-errorAPI). Wrapped via shims in
pg_compat.hthat preserve the older signatures.Build flag
pgduck_server's-std=c11→-std=gnu11so PG19'sc.h(which usestypeofandstatic_assertunconditionally) compiles. PG itself uses the GNU dialect for thesame reason.
Drive-by
pgduck_serveris a frontend program; the unused#include "nodes/nodes.h"innumutils.hstarted failing because PG19 changednodes.hto useDatumin externdeclarations. Removed the vestigial include.
What is expected to work
and tracked via
continue-on-error. Follow-up will tighten the matrix as test failures are addressed.Test plan
./install.sh --build-postgres --pg-version 19continue-on-erroroverrides in a follow-up