Merge to upstream#20
Open
perangel wants to merge 82 commits into
Open
Conversation
This change is necessary to avoid potential data corruption. Unless overridden, the changes schema should always be used when writing to the target. Since data integrity checks only check to see if the two changesets tables on both the source and target are the same, it will not detect if data is written to the wrong schema.
…a-is-target-schema-by-default refactor: Change schema is target schema by default
…urns-errors refactor: Axon returns errors rather than logging
fix: Correct empty array in prepareQueryArgs
Allows the ability to set the starting changeset id in the Axon configuration, which will be used during the run.
feat: Axon start from ID
improve: Align row insert/update/delete log msgs
* Bring up source and target database containers * Setup a Test Schema on the source containers * Write rows asynchronously on source * Sync with Axon * Wait for all changes to source to complete * Resync Axon * Run axon.Verify to check if sync completed successfully.
Duplicate INSERTS in 1:1 DB migrations potentially indicate a migration error, especially when changeset record verification will be used as the non-duplicate errors will show as extra change records.
When we start processing "live" changeset from the NOTIFY channel, we should make sure to _skip_ any events with a lower ID than the changeset we last processed. Otherwise we end up "jumping back in time" by replaying buffered notifications.
…ed-changesets bugfix: skip notifications before last processed
Needs more work to map column types a la loadPrimaryKeys.
We need to use the col, colArgs when constructing the setClauses for `DO UPDATE SET` in the upsert query.
This allows us to do data type specific processing like bytea.
Reducing workers and data changes. Currently adds test time, but does not increase reliability.
A quick implementation which needs more clean up, but works
Changesets are not provided by the notification system in strict ID order. That may be a bug, but this queue enforces the sequential record ordering. TODO: Is this a bug workaround? If so solve the core bug.
feat: Verbose record-by-record VerifyChangesets on diff, add a strictly ordered buffer queue to NotifyListener
this is not a content repo
Testing for LogicalReplication listener mode revealed that newer PSQL images include additional build information in the 'SHOW server_version' call. This is a slightly more robust version of the initial implementation (with added tests).
fix: More durable parsing of semver from db
The previous implementation of only calling conn.Close() explicitly when the shutdown signal is received means that connections are not closed cleanly when axon closes due to an error.
fix: Defer Close()s after Open()s, not in shutdown()
feat: Add replication mode config/skeleton to axon.Run
chore: go1.15 --> go1.17
Add linter, address errors
fix: Actually vendor the linter deps
This test mostly demonstrates that out-of-the-box replication connection is only supported on postgres images > 9. As we are currently running 12.6 in production, this shouldn't be an issue for actual use.
test: Ensure replication connection can be formed
Use PSQL/CRDB compatible query to fetch table information
This stored info is not used yet, but we will be using it to confirm if the target DB is in-sync with the source.
Clean up LogicalReplicationListener
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.