feat: Adding pg_legacy_replication verified source using decoderbufs#589

Open

neuromantik33 wants to merge 88 commits into

dlt-hub:masterfrom

neuromantik33:pg-legacy-replcation

Commits on Dec 16, 2024

fix: finally got pg_replication tests working as is
Nicolas ESTRADA
committed
feat: got decoderbufs to run and compile in docker
Nicolas ESTRADA
committed
chore: updated protobuf to latest compatible version
Nicolas ESTRADA
committed
chore: copying all files from pg_replication; format-lint is reformatting other sources -_-
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: removed all references to publications
Nicolas ESTRADA
committed
fix: applied suggested changes mentioned here https://github.com/dlt-hub/dlt/issues/1920
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: finally got snapshot to work
Nicolas ESTRADA
committed
chore: simply cleaning up
Nicolas ESTRADA
committed
chore: need to find a better way to clean up the underlying engine
Nicolas ESTRADA
committed
wip: handling begin/commit
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: making progress
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
refactor: some test refactoring
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: saving work
Nicolas ESTRADA
committed
wip: cleaning up + refactor
Nicolas ESTRADA
committed
wip: cleaning up + refactor
Nicolas ESTRADA
committed
wip: cleaning up + refactor
Nicolas ESTRADA
committed
wip: slowly progressing
Nicolas ESTRADA
committed
wip: all tests pass now to update docs and cleanup
Nicolas ESTRADA
committed
wip: still trying to get it work with all versions of dlt
Nicolas ESTRADA
committed
wip
Nicolas ESTRADA
committed
wip: changing signature
Nicolas ESTRADA
committed
wip: finally got rid of those errors
Nicolas ESTRADA
committed
wip: correcting failing tests
Nicolas ESTRADA
committed
wip: fixed working examples
Nicolas ESTRADA
committed
wip: more refactoring now docs... -_-
Nicolas ESTRADA
committed
wip: cleaning up
Nicolas ESTRADA
committed
wip: cleaning up
Nicolas ESTRADA
committed
wip: attempting to refactor to use dlt resources
Nicolas ESTRADA
committed
wip: second test passing
Nicolas ESTRADA
committed
wip: all tests pass again now for refactoring
Nicolas ESTRADA
committed
wip: init_replication is now a dlt source
Nicolas ESTRADA
committed
wip: more refactoring
Nicolas ESTRADA
committed
wip: saving work until I can get hinting to work
Nicolas ESTRADA
committed
wip: finally got something somewhat working
Nicolas ESTRADA
committed
wip: done with coding now docs
Nicolas ESTRADA
committed
fix: various performance improvements
Nicolas ESTRADA
committed
fix: minor corrections to handle old versions of postgres
Nicolas ESTRADA
committed
fix: small type corrections for pg9.6
Nicolas ESTRADA
committed
fix: exposing table options for later arrow support
Nicolas ESTRADA
committed
wip: saving work for arrow
Nicolas ESTRADA
committed
wip: first test with arrow passing
Nicolas ESTRADA
committed
wip: almost done passing all tests
Nicolas ESTRADA
committed
wip: some arrow tests are still not passing
Nicolas ESTRADA
committed
fix: done with pyarrow; too many issues with duckdb atm
Nicolas ESTRADA
committed
wip: some bug fixes
Nicolas ESTRADA
committed
wip: small refactoring
Nicolas ESTRADA
committed
wip: duckdb needs patching, trying out new max_lsn
Nicolas ESTRADA
committed
wip: some refactoring of options to make certain features togglable
Nicolas ESTRADA
committed
wip: lsn and deleted ts are optional
Nicolas ESTRADA
committed
feat: added optional transaction id
Nicolas ESTRADA
committed
feat: added optional commit timestamp
Nicolas ESTRADA
committed
fix: never handled missing type and added text oid mapping
Nicolas ESTRADA
committed
fix: added some logging and bug fixes
Nicolas ESTRADA
committed
chore: basic refactoring
Nicolas ESTRADA
committed
fix: minor corrections
Nicolas ESTRADA
committed
chore: reverting back to prev state
Nicolas ESTRADA
committed
chore: rebasing 1.x branch onto my own
Nicolas ESTRADA
committed

Commits on Dec 19, 2024

fix: corrected bug regarding column names
Nicolas ESTRADA
committed
chore: minor fixes
Nicolas ESTRADA
committed

Commits on Dec 20, 2024

chore: small perf fixes and aligning with more adt
Nicolas ESTRADA
committed
chore: refactoring and cleaning
Nicolas ESTRADA
committed

Commits on Dec 22, 2024

chore: finished docstrings
Nicolas ESTRADA
committed
bugfix: misuse of defaultdict
Nicolas ESTRADA
committed

Commits on Dec 23, 2024

Finally done with docs
Nicolas ESTRADA
committed

Commits on Dec 30, 2024

fix: wasn't able to execute local tests without these settings
Nicolas ESTRADA
committed

Commits on Jan 14, 2025

feat: added basic support for scalar array types
Nicolas ESTRADA
committed

Commits on Jan 15, 2025

chore: slight perf improvments for pg_arrays
Nicolas ESTRADA
committed

Commits on Jan 16, 2025

fix: it turns out pg_arrays are annoying found temp workaround
Nicolas ESTRADA
committed

Commits on Jan 22, 2025

refactor: all sqlalchemy event code is done at engine configuration
Nicolas ESTRADA
committed
chore: bumped python to 3.9; small refactorings
Nicolas ESTRADA
committed
refactor: init_replication is now in pkg ns
Nicolas ESTRADA
committed

Commits on Jan 28, 2025

fix: corrected bugs regarding inferring nullability wrong; refactored tests used to choose best schema
Nicolas ESTRADA
committed

Commits on Jan 29, 2025

fix: rolling back on managing conn lifecycle using context mgrs: it doesn't work as per https://www.psycopg.org/docs/usage.html#with-statement
Nicolas ESTRADA
committed

Commits on Jan 30, 2025

fix: corrected regression with occasional datum_missinng values
Nicolas ESTRADA
committed

Commits on Feb 7, 2025

fix: add support for ordinary json pg_type
Nicolas ESTRADA
committed

Commits on Feb 19, 2025

fix: various fixes of bugs encountered during production
Nicolas ESTRADA
committed

Commits on Mar 3, 2025

fix: various fixes related to pyarrow backends
Nicolas ESTRADA
committed

Commits on Jul 1, 2025

chore: updating poetry + dlt
Nicolas ESTRADA
committed