feat(backend): keep only latest preprocessing pipeline version#6813
Draft
corneliusroemer wants to merge 1 commit into
Draft
feat(backend): keep only latest preprocessing pipeline version#6813corneliusroemer wants to merge 1 commit into
corneliusroemer wants to merge 1 commit into
Conversation
Previously the pipeline-version garbage collector kept the two most recent preprocessing pipeline versions (passing `latestVersion - 1` as the earliest version to keep). Change it to keep only the latest version (`latestVersion`), so outdated preprocessed data is no longer retained unnecessarily. Also updates the GC test assertions to expect only the latest version surviving, and clarifies the backend AGENTS.md test instructions to prefer the Docker route when Docker is present. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
|
At first glance I'm not sure about this. I can imagine various race conditions that might result (not necessarily with current code, unsure, but with possible code) and I'm not sure DB size is such a big issue |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The pipeline-version garbage collector (
UseNewerProcessingPipelineVersionTask) runs after the backend bumps an organism to a newer preprocessing pipeline version, deleting outdated preprocessed data. Previously it retained the two most recent pipeline versions by passinglatestVersion - 1as the earliest version to keep.This PR changes it to keep only the latest version (passing
latestVersion), so older preprocessed data is no longer retained unnecessarily — keeping only what's needed.Changes
UseNewerProcessingPipelineVersionTask.kt: passlatestVersioninstead oflatestVersion - 1tocleanUpOutdatedPreprocessingData. This is the only behavioral change; the deletion is still scoped per-organism and only triggers when a version upgrade actually happens.UseNewerProcessingPipelineVersionTaskTest.kt: updated the existing GC test assertions to expect only the latest version surviving ([2L]after the v2 bump,[3L]after v3), with clarified comments. The per-organism scoping assertions (OTHER_ORGANISMuntouched) are unchanged.backend/AGENTS.md: clarified test instructions to check for Docker first and prefer the Docker route, falling back toUSE_NONDOCKER_INFRA=trueonly when Docker is absent or its run fails.Testing
./gradlew test --tests "...UseNewerProcessingPipelineVersionTaskTest"passes.🤖 Generated with Claude Code
🚀 Preview: Add
previewlabel to enable