-
Notifications
You must be signed in to change notification settings - Fork 2
feat: add Flyway migrations and S3 orphaned object cleanup #269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
johanbriger
merged 12 commits into
main
from
114-feat-implement-durable-retry-logic-for-failed-s3minio-deletions
Apr 27, 2026
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
5e44b5a
feat: add Flyway migrations and S3 orphaned object cleanup
johanbriger 67e9f4b
test: disable Flyway in integration tests
johanbriger 77a0c64
fix: prioritize new orphans in cleanup queue
johanbriger 2844da8
- Added OrphanedS3Enqueuer with REQUIRES_NEW to ensure cleanup records
johanbriger 86f24b1
refactor: simplify delete logic by removing unreachable else branch
johanbriger b690d2b
refactor(attachment): enforce transaction-aware S3 deletion
johanbriger d07b230
fix(cleanup): improve error logging by preserving stack traces
johanbriger 72c334c
refactor(db): migrate demo data to Flyway and fix schema race condition
johanbriger a448cc5
security(db): isolate demo data to dev-only flyway location
johanbriger 438b754
fix(service): improve error handling and traceability in AttachmentSe…
johanbriger e16586c
fix(cleanup): add alerting for orphaned S3 objects exceeding max retries
johanbriger 56102f9
fix(cleanup): make OrphanedS3Enqueuer idempotent using find-or-create
johanbriger File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
61 changes: 61 additions & 0 deletions
61
src/main/java/org/example/vet1177/entities/OrphanedS3Object.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| package org.example.vet1177.entities; | ||
|
|
||
| import jakarta.persistence.*; | ||
| import java.time.Instant; | ||
| import java.util.UUID; | ||
|
|
||
| @Entity | ||
| @Table(name = "orphaned_s3_objects") | ||
| public class OrphanedS3Object { | ||
|
|
||
| @Id | ||
| @GeneratedValue(strategy = GenerationType.UUID) | ||
| private UUID id; | ||
|
|
||
| @Column(name = "s3_key", unique = true, nullable = false) | ||
| private String s3Key; | ||
|
|
||
| @Column(name = "s3_bucket", nullable = false) | ||
| private String s3Bucket; | ||
|
|
||
| @Column(name = "created_at", nullable = false, updatable = false) | ||
| private Instant createdAt = Instant.now(); | ||
|
|
||
| @Column(name = "retry_count", nullable = false) | ||
| private int retryCount = 0; | ||
|
|
||
| @Column(name = "last_attempt_at") | ||
| private Instant lastAttemptAt; | ||
|
|
||
| @Column(name = "last_error", columnDefinition = "TEXT") | ||
| private String lastError; | ||
|
|
||
| public OrphanedS3Object() {} | ||
|
|
||
| public OrphanedS3Object(String s3Key, String s3Bucket) { | ||
| this.s3Key = s3Key; | ||
| this.s3Bucket = s3Bucket; | ||
| this.createdAt = Instant.now(); | ||
| this.retryCount = 0; | ||
| } | ||
|
|
||
| public UUID getId() { return id; } | ||
| public void setId(UUID id) { this.id = id; } | ||
|
|
||
| public String getS3Key() { return s3Key; } | ||
| public void setS3Key(String s3Key) { this.s3Key = s3Key; } | ||
|
|
||
| public String getS3Bucket() { return s3Bucket; } | ||
| public void setS3Bucket(String s3Bucket) { this.s3Bucket = s3Bucket; } | ||
|
|
||
| public Instant getCreatedAt() { return createdAt; } | ||
|
|
||
| public int getRetryCount() { return retryCount; } | ||
| public void setRetryCount(int retryCount) { this.retryCount = retryCount; } | ||
|
|
||
| public Instant getLastAttemptAt() { return lastAttemptAt; } | ||
| public void setLastAttemptAt(Instant lastAttemptAt) { this.lastAttemptAt = lastAttemptAt; } | ||
|
|
||
| public String getLastError() { return lastError; } | ||
| public void setLastError(String lastError) { this.lastError = lastError; } | ||
| } |
27 changes: 27 additions & 0 deletions
27
src/main/java/org/example/vet1177/repository/OrphanedS3ObjectRepository.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| package org.example.vet1177.repository; | ||
|
|
||
| import org.example.vet1177.entities.OrphanedS3Object; | ||
| import org.springframework.data.domain.Pageable; | ||
| import org.springframework.data.jpa.repository.JpaRepository; | ||
| import org.springframework.data.jpa.repository.Query; | ||
| import org.springframework.data.repository.query.Param; | ||
| import org.springframework.stereotype.Repository; | ||
| import java.util.List; | ||
| import java.util.Optional; | ||
| import java.util.UUID; | ||
|
|
||
| @Repository | ||
| public interface OrphanedS3ObjectRepository extends JpaRepository<OrphanedS3Object, UUID> { | ||
|
|
||
| @Query(""" | ||
| SELECT o FROM OrphanedS3Object o | ||
| WHERE o.retryCount < :maxRetries | ||
| ORDER BY o.lastAttemptAt ASC NULLS FIRST, o.createdAt ASC | ||
| """) | ||
|
|
||
| List<OrphanedS3Object> findNextBatchToProcess(@Param("maxRetries") int maxRetries, Pageable pageable); | ||
|
|
||
| long countByRetryCountGreaterThanEqual(int maxRetries); | ||
|
|
||
| Optional<OrphanedS3Object> findByS3Key(String s3Key); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.