[Bug]: ERROR: pgclone: could not import snapshot

[herwigg]

pgclone version

4.3.0

PostgreSQL version

18

Operating system

RHEL 9

Installation method

Built from source (make install)

Feature area

Table clone (pgclone_clone_table)

Sync or async?

Synchronous (foreground call)

What happened?

when cloning a schema I got the error: ERROR: pgclone: could not import snapshot 000000A6-000C4299-1 on source: ERROR: invalid snapshot identifier: "000000A6-000C4299-1"
and effectively, not all tables were copied.

Reproduction steps

itiniris_cl=> \timing
Timing is on.
itiniris_cl=> SELECT pgclone.schema(
itiniris_cl(>     'postgresql://00104529@<host alias>:5432/itiniris'
itiniris_cl(>     ,'itiniris'            -- schema to clone
itiniris_cl(>     ,true                  -- include table data);
itiniris_cl(> );
ERROR:  pgclone: could not import snapshot 000000A6-000C4299-1 on source: ERROR:  invalid snapshot identifier: "000000A6-000C4299-1"

Time: 8153325.487 ms (02:15:53.325)

PostgreSQL logs

from the postgres log on the master or target isntance:
2026-05-11 18:54:35.992 CEST [3943869] [local] itinirisuser@itiniris_cl/[unknown] STATEMENT:  CREATE INDEX IF NOT EXISTS perchildamount_payment_fk_i ON itiniris.per_child_amounts USING btree (payment_id)
2026-05-11 18:54:36.020 CEST [3943869] [local] itinirisuser@itiniris_cl/[unknown] LOG:  temporary file: path "base/pgsql_tmp/pgsql_tmp3943869.2.fileset/1.0", size 280887296
2026-05-11 18:54:36.020 CEST [3943869] [local] itinirisuser@itiniris_cl/[unknown] STATEMENT:  CREATE INDEX IF NOT EXISTS perchildamount_payment_fk_i ON itiniris.per_child_amounts USING btree (payment_id)
2026-05-11 18:54:36.029 CEST [3943869] [local] itinirisuser@itiniris_cl/[unknown] LOG:  duration: 26734.475 ms
2026-05-11 18:54:36.142 CEST [3908164] [local] itinirisuser@itiniris_cl/psql ERROR:  pgclone: could not import snapshot 000000A6-000C4299-1 on source: ERROR:  invalid snapshot identifier: "000000A6-000C4299-1"

pgclone_jobs_view (async only)

Additional context

No response

Pre-flight checks

• I've redacted any passwords from connection strings and logs
• I've checked existing issues for duplicates
• This is not a security vulnerability (those go through SECURITY.md)

[valehdba]

Hi @herwigg — thanks for the detailed report, especially the timing and the log excerpt showing CREATE INDEX completing 113 ms before the error. That timeline pointed directly at the root cause.

Diagnosis

pgclone.schema() exports a snapshot on a "keeper" connection at the start, then opens one fresh source connection per table (plus extra ones for the FK / view / function / trigger sub-phases) and binds each to the keeper's snapshot via SET TRANSACTION SNAPSHOT. The keeper itself sits idle in transaction for the rest of the run — in your case ~2h, which is well past the idle-drop threshold of most perimeter firewalls (commonly 30–60 min) and past any non-zero idle_in_transaction_session_timeout on the source.

When the keeper transaction dies for any reason, PostgreSQL deletes the exported-snapshot file. The next SET TRANSACTION SNAPSHOT then fails with the misleading message ERROR: invalid snapshot identifier — the same string PG emits for a malformed ID, even though in this case the ID was perfectly well-formed (000000A6-000C4299-1 is the canonical PG 17/18 procNumber-lxid-counter layout) and the actual cause was that the keeper's file had been reaped.

Your <host alias>:5432 setup makes path 1 (firewall idle drop) the most likely culprit, but the fix covers all three known paths.

Fix in PR #10 (v4.3.1)

Four layers of protection, all PG 14–18 compatible:

1. TCP keepalives auto-injected into every source conninfo (keepalives=1 keepalives_idle=30 keepalives_interval=10 keepalives_count=6) so the OS keeps the firewall NAT entry warm and detects silent drops in ~90 s. Works for both URI (postgresql://...) and keyword-form conninfo strings.
2. SET LOCAL idle_in_transaction_session_timeout = 0 and SET LOCAL statement_timeout = 0 inside the keeper transaction, so a non-zero source-side setting can't kill it. SET LOCAL reverts at COMMIT — nothing leaks into pooled connections.
3. Clearer error when SET TRANSACTION SNAPSHOT does fail, with an errhint naming the most likely cause and pointing at the {"consistent": false} emergency opt-out.
4. Keeper liveness ping between sub-phases so the failure surfaces before a per-table COPY wastes time, with an error message that names the keeper instead of the misleading "invalid snapshot identifier".

A regression test (test/test_snapshot_keeper.sh) deliberately sets idle_in_transaction_session_timeout = 1s on the source role and clones five tables; without the fix the keeper dies mid-loop. It runs on all five PG versions in CI.

After the PR merges and v4.3.1 is tagged

cd /path/to/pgclone-source
git pull
make clean && make && sudo make install
psql -d itiniris_cl -c "ALTER EXTENSION pgclone UPDATE;"

Then re-run the same SELECT pgclone.schema(...) call.

If for any reason the clone still fails with the same error after upgrading, please pass '{"consistent": false}' as a fourth argument to bypass cross-table snapshot sharing entirely — every per-table copy will still be internally consistent, but cross-table referential consistency will not be guaranteed:

SELECT pgclone.schema(
    'postgresql://00104529@<host alias>:5432/itiniris',
    'itiniris',
    true,
    '{"consistent": false}'
);

Will close this issue automatically when #10 merges.

[herwigg]

Hi, Well thanks for building this. Unfortunately when I try to make install it as root user I get a cd '/usr/pgsql-18/lib/bitcode' && /usr/lib64/llvm20/bin/llvm-lto -thinlto -thinlto-action=thinlink -o pgclone.index.bc pgclone/src/pgclone.bc pgclone/src/pgclone_bgw.bc pgclone/src/pgclone_diff.bc pgclone/src/pgclone_preflight.bc /bin/sh: /usr/lib64/llvm20/bin/llvm-lto: No such file or directory make: *** [/usr/pgsql-18/lib/pgxs/src/makefiles/pgxs.mk:241: install] Error 127 on NAME="Red Hat Enterprise Linux" VERSION="8.10 (Ootpa)" I make it as user postgres with pg_config in /usr/pgsql-18/bin/ did not have such issues with previous builds. And idea or should I file this as a bug/problem or did I not read the isntallation manual? Herwig Op wo 13 mei 2026 om 22:24 schreef Valeh Agayev ***@***.***>:

…

*valehdba* left a comment (valehdba/pgclone#9) <#9 (comment)> Hi @herwigg <https://github.com/herwigg> — thanks for the detailed report, especially the timing and the log excerpt showing CREATE INDEX completing 113 ms before the error. That timeline pointed directly at the root cause. Diagnosis pgclone.schema() exports a snapshot on a "keeper" connection at the start, then opens one fresh source connection per table (plus extra ones for the FK / view / function / trigger sub-phases) and binds each to the keeper's snapshot via SET TRANSACTION SNAPSHOT. The keeper itself sits *idle in transaction* for the rest of the run — in your case ~2h, which is well past the idle-drop threshold of most perimeter firewalls (commonly 30–60 min) and past any non-zero idle_in_transaction_session_timeout on the source. When the keeper transaction dies for any reason, PostgreSQL deletes the exported-snapshot file. The next SET TRANSACTION SNAPSHOT then fails with the misleading message ERROR: invalid snapshot identifier — the same string PG emits for a *malformed* ID, even though in this case the ID was perfectly well-formed (000000A6-000C4299-1 is the canonical PG 17/18 procNumber-lxid-counter layout) and the actual cause was that the keeper's file had been reaped. Your <host alias>:5432 setup makes path 1 (firewall idle drop) the most likely culprit, but the fix covers all three known paths. Fix in PR #10 <#10> (v4.3.1) Four layers of protection, all PG 14–18 compatible: 1. *TCP keepalives* auto-injected into every source conninfo (keepalives=1 keepalives_idle=30 keepalives_interval=10 keepalives_count=6) so the OS keeps the firewall NAT entry warm and detects silent drops in ~90 s. Works for both URI (postgresql://...) and keyword-form conninfo strings. 2. *SET LOCAL idle_in_transaction_session_timeout = 0* and *SET LOCAL statement_timeout = 0* inside the keeper transaction, so a non-zero source-side setting can't kill it. SET LOCAL reverts at COMMIT — nothing leaks into pooled connections. 3. *Clearer error* when SET TRANSACTION SNAPSHOT does fail, with an errhint naming the most likely cause and pointing at the {"consistent": false} emergency opt-out. 4. *Keeper liveness ping* between sub-phases so the failure surfaces *before* a per-table COPY wastes time, with an error message that names the keeper instead of the misleading "invalid snapshot identifier". A regression test (test/test_snapshot_keeper.sh) deliberately sets idle_in_transaction_session_timeout = 1s on the source role and clones five tables; without the fix the keeper dies mid-loop. It runs on all five PG versions in CI. After the PR merges and v4.3.1 is tagged cd /path/to/pgclone-source git pull make clean && make && sudo make install psql -d itiniris_cl -c "ALTER EXTENSION pgclone UPDATE;" Then re-run the same SELECT pgclone.schema(...) call. If for any reason the clone still fails with the same error after upgrading, please pass '{"consistent": false}' as a fourth argument to bypass cross-table snapshot sharing entirely — every per-table copy will still be internally consistent, but cross-table referential consistency will not be guaranteed: SELECT pgclone.schema( 'postgresql://00104529@<host alias>:5432/itiniris', 'itiniris', true, '{"consistent": false}' ); Will close this issue automatically when #10 <#10> merges. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACR3SXX6ZXQELUZNGJE7WT342TKW3AVCNFSM6AAAAACY4KDILSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DINBUHEZDIOBSHE> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[valehdba]

Hi @herwigg, thanks for the really thorough report. The timing output plus those Postgres log lines actually make this one pretty clear.

First thing worth flagging: you're on 4.3.0, and this exact bug got fixed in 4.3.1 and 4.3.2. So the short version is please upgrade, but let me explain what's going on so it makes sense.

In consistent mode (which became the default in 4.3.0), pgclone opens one "keeper" connection on the source. That connection runs pg_export_snapshot() inside a REPEATABLE READ transaction, and then every table it clones does SET TRANSACTION SNAPSHOT against that same snapshot so the whole clone sees one consistent point in time. The catch is that the keeper transaction has to stay alive, sitting idle in transaction, for the entire run. The instant it dies, Postgres throws away the snapshot file, and every table after that point fails with the error you hit:

invalid snapshot identifier: "000000A6-000C4299-1"

That message is honestly a bit of a red herring. The ID isn't malformed at all. The snapshot it pointed to just got cleaned up because the keeper transaction was killed. Your clone ran for 2h15m, and I can see in the log that a single index build (perchildamount_payment_fk_i) took almost 27 seconds on its own with a 280MB temp file, so the keeper was idle for a very long stretch while the rest of the work happened. Something on the source side terminated that idle session before the clone finished, which is exactly why only some of your tables came across.

On 4.3.0 the keeper had basically no protection against this. The 4.3.1 and 4.3.2 releases fixed it three ways: they set idle_in_transaction_session_timeout, transaction_timeout (on PG17+), and statement_timeout all to 0 on the keeper transaction so server side timeouts can't reap it; they turn on TCP keepalives so an idle connection drop from a firewall gets caught; and they ping the keeper before each table import so if it does die you get a clear error instead of that cryptic snapshot message.

So, to upgrade to the latest (4.4.0):

git pull
make clean && make && sudo make install

Then reload the library. If pgclone is in shared_preload_libraries you'll need a full restart, not just a reconnect, otherwise the old .so stays loaded and none of the timeout fixes actually run:

sudo pg_ctlcluster 18 main restart -m fast    # or systemctl restart postgresql-18

And then:

ALTER EXTENSION pgclone UPDATE TO '4.4.0';

Now, if it still fails after all that, the cause is probably something between pgclone and the source that the extension can't control from inside its own session. A couple of things worth checking:

Is there a firewall, NAT gateway, or load balancer on the path to :5432? The keepalives help a lot, but a really aggressive idle timeout on a middlebox can still cut a long-idle connection. On a 2+ hour clone that's a real possibility.

Is there a connection pooler in the way? This one's important. PgBouncer in transaction mode in particular will break SET TRANSACTION SNAPSHOT, because the keeper and the importing connection won't end up on the same backend. If there's a pooler between you and the source, connect pgclone straight to the source instead.

And if you just need it working today regardless of the root cause, you can turn off cross table snapshot sharing:

SELECT pgclone.schema(
    'postgresql://00104529@<host alias>:5432/itiniris',
    'itiniris', true,
    '{"consistent": false}'
);

That drops the keeper entirely and clones each table in its own transaction. You lose cross table consistency (each table is internally consistent, but they're not guaranteed consistent as a set), which is totally fine if nothing is writing to the source during the clone, like a maintenance window or a replica.

Could you let me know the version once you've upgraded, and whether there's a pooler or firewall on the way to the source? If 4.4.0 with a direct connection still reproduces it, I'd love to see the new error message, since the keeper ping should now tell us exactly where it's dying.