[APIPUB-113] - Add option --processDeletesAndKeyChangesOnFullPublish#137
Open
ea-mtenhoor wants to merge 1 commit into
Open
Conversation
stevenarnold-ed-fi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new option to the API Publisher allowing for deletes and key changes to be processed even when publishing from change version 1. The original code attempts to optimize the first publish by skipping these steps. This works under the assumption that the target ODS is empty when publishing from change version 1, and thus there would be no keys to change and no records to delete.
However, this assumption is wrong quite often in the field. For example, if the first full publish fails for any reason, the data in the target ODS remains and the API Publisher does not store a value for lastChangeVersionsProcessed. Then, as source systems like a SIS continue to make updates to the source ODS, it's possible (and increasingly likely over time) that data will be deleted or have its keys changed. When the second (or any subsequent) full publish happens there are now deletes and/or keyChanges that need to be processed, but the API Publisher continues to skip these steps under the assumption that the target ODS is empty, because it is starting from change version 1.
The code in this PR defaults to the original behavior to ensure that this is a non-breaking change. However, it is our intention to enable this option everywhere we use API Publisher. It might be worth considering changing the default.
This code has been tested and is in use in Education Analytics production systems.