[FEAT] Redis 장애 복구 배치 프레임워크 구현 및 복구 정합성 결함 개선#6
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughRedis 장애 복구 배치 전 과정(타깃 스키마/매퍼/서비스, Lua, 정책 플래그 게이트, 검증·보정, 관리자 API), 설정/키팩토리 추가, 런타임 차단 적용, 마이그레이션과 대규모 테스트·문서가 일괄 도입되었습니다. ChangesRedis 트래픽 복구 파이프라인
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant Gate as TrafficRestoreTrafficGateService
participant Stream as TrafficStreamConsumer/Enqueue/Reclaim
participant Admin as AdminTrafficRestoreController
participant Orchestrator as TrafficRestoreOrchestratorService
participant Exec as TrafficRestoreExecutionService
participant Redis as Redis/Lua
participant DB as DB Targets/Logs/Policy
Client->>Gate: shouldBlockTraffic()
Gate-->>Client: true or false
alt Gate blocks
Client-->>Stream: block enqueue/consume/reclaim
else Proceed
Client->>Stream: enqueue/consume/reclaim
end
Admin->>Orchestrator: start(failureDate) or resume(anchorDate)
Orchestrator->>Redis: activate restore flag (policy_id=8)
Orchestrator->>Exec: execute P0->P1->P2
Exec->>DB: insert/select hydrate & daily targets
Exec->>Redis: replay usage via restore_usage_replay.lua
Exec->>DB: mark DONE / FAILED
Orchestrator->>Redis: verify and correct mismatches
Orchestrator->>Redis: cleanup restore idempotency keys
Orchestrator->>Redis: deactivate restore flag
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Suggested labels
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
|
Owner
Author
|
@CodeRabbit review |
Owner
Author
|
@CodeRabbit review |
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayService.java (1)
29-60: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win
replay트랜잭션 흐름도 단계 주석으로 고정해 두는 편이 좋겠습니다.
replay → terminal 상태 전환 → afterCommit 정리는 분기와 상태 전이가 모두 걸린 핵심 경로입니다. 성공/실패 분기와 commit 이후 cleanup 의도를 짧은 한국어 단계 주석으로 남겨 두면 이후 정합성 회귀를 줄이기 쉽습니다. As per coding guidelines:**/*.{java,kt}: For method bodies, add concise step-by-step comments that describe the logic flow when the method contains meaningful branching, sequencing, transactions, external calls, or state transitions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayService.java` around lines 29 - 60, 메서드 replay의 핵심 트랜잭션 흐름을 짧은 한국어 단계 주석으로 고정해 주세요: 메서드 시작부에 "1) replay 명령 조회 및 실행", 실행 결과 분기부 위에 "2) 실패 시 terminal=FAILED 및 즉시 반환", 성공 경로의 상태 전환 직전에 "3) terminal=DONE으로 업데이트(동시성 체크)", 그리고 registerIdempotencyCleanupAfterCommit 내부에 "4) DB 커밋 확정 후 Redis idempotencyKey 삭제"와 같이 replay(), dailyAppTargetMapper.markTargetTerminalIfProcessing(...), TrafficRestoreReplayResult.status(), registerIdempotencyCleanupAfterCommit(String) 및 replayLuaExecutor.deleteIdempotencyKey(idempotencyKey) 호출들을 참조하는 간결한 한두 문장씩 넣어 주십시오.src/main/java/com/pooli/traffic/service/restore/TrafficRestoreOrchestratorService.java (1)
42-57:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift고정 TTL만으로는 장시간 복구의 단일 실행 보장이 깨집니다.
manager lock이 6시간 고정인데 worker 실행 중 lease 연장이 전혀 없습니다. 복구가 TTL을 넘기면 첫 번째 worker가 아직 phase를 돌리는 중에도 두 번째
start()/resume()가 같은 lock을 다시 획득해 hydrate, replay, flag lifecycle이 겹칠 수 있습니다. 이 경로는 lock heartbeat/lease 연장이나 batch metadata와 결합된 영속 세션 검증으로 보강해야 합니다.Also applies to: 198-218
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/pooli/traffic/service/restore/TrafficRestoreOrchestratorService.java` around lines 42 - 57, The manager lock currently uses a fixed TTL (RESTORE_MANAGER_LOCK_TTL_MS) without any lease/heartbeat, so long-running restores can exceed the TTL and allow a second start()/resume() to acquire the same lock; modify TrafficRestoreOrchestratorService to implement a lock lease extension or heartbeat while a worker is active (e.g., periodically refresh the Redis lock via cacheStringRedisTemplate/trafficLuaScriptInfraService) and/or add persistent session validation tied to batch metadata (using LineDailyBatchJobMapper/batchJobMapper or related hydrate/replay lifecycle checks) to refuse a second run; ensure the taskExecutor-driven worker loop extends the lock until completion and that start()/resume() verify the active lease/session before proceeding.
♻️ Duplicate comments (2)
src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase0HydrateService.java (1)
28-30:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
targetMonthStart를 변환 전에 명시적으로 검증해 주세요.지금은
YearMonth.from(target.getTargetMonthStart())가 일반적인NullPointerException을 던져 새 테스트가 기대하는 계약도 맞지 않고, 복구 실패 원인도 흐려집니다.targetMonthStart를 먼저 검증해서 owner 정보를 포함한 메시지로 빠르게 실패시키는 편이 안전합니다.수정 예시
public void hydrate(`@NonNull` TrafficRestoreHydrateTarget target) { + Objects.requireNonNull( + target.getTargetMonthStart(), + "targetMonthStart must not be null for owner " + target.getTargetOwnerId() + ); YearMonth targetMonth = YearMonth.from(target.getTargetMonthStart()); switch (target.getTargetType()) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase0HydrateService.java` around lines 28 - 30, Before calling YearMonth.from in TrafficRestorePhase0HydrateService.hydrate, explicitly validate target.getTargetMonthStart() (from TrafficRestoreHydrateTarget) and fail fast with a clear exception that includes owner/context info from the target (e.g., target.getOwner() or target identifier) so the error is actionable; update the hydrate method to check for null/invalid targetMonthStart and throw an IllegalArgumentException (or similar) with a message containing the owner/target identifier instead of letting YearMonth.from throw a raw NullPointerException.src/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayServiceTest.java (1)
58-146: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win
FAILEDreplay 경로도 테스트로 고정해 주세요.현재는
APPLIED/SKIPPED와 ownership 상실만 검증하고 있습니다. 서비스가 비정상 status에서FAILED로 닫히고idempotency key를 삭제하지 않는 계약도 같이 고정해야 phase 1 복구 회귀를 막을 수 있습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayServiceTest.java` around lines 58 - 146, Add a unit test that covers the FAILED replay path: create a TrafficRestoreDailyAppTarget and TrafficRestoreReplayCommand, mock replayLuaExecutor.replay(command) to return a TrafficRestoreReplayResult with status "FAILED", mock dailyAppTargetMapper.markTargetTerminalIfProcessing(target.getId(), TrafficRestoreTargetStatus.FAILED, WORKER_ID) to return 1 (or the intended ownership result), call service.replay(target, WORKER_ID) inside TransactionSynchronizationManager.initSynchronization()/clearSynchronization(), verify that markTargetTerminalIfProcessing was called with TrafficRestoreTargetStatus.FAILED, and verify that replayLuaExecutor.deleteIdempotencyKey(command.getIdempotencyKey()) is never called both before and after invoking TransactionSynchronization::afterCommit (i.e., ensure no deletion is scheduled for FAILED).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/plans/2026-05-31-global-policy-preflight-hydrate.md`:
- Around line 1-10: 문서의 언어와 런타임 표기를 규정과 맞게 수정하세요: 문서 상단의 제목/섹션 라벨("Goal",
"Architecture", "Tech Stack" 등)을 모두 한국어로 바꾸고 본문 전체를 한국어로 통일하며, Tech Stack 블록에서
"Java 17"을 "Java 21"로 수정하고 Spring Boot 표기("Spring Boot 3.5.10")와 기타 항목은 그대로
유지하도록 업데이트하여 docs/superpowers 규칙(docs/superpowers/**/*.md: 모든 사양 및 계획 문서는 한국어로
작성)과 src/main/java 기준(Java 21 기반)을 반영하세요.
In `@docs/superpowers/plans/2026-05-31-traffic-restore-async-flag-lifecycle.md`:
- Around line 1-10: 문서 본문 전체를 영어가 섞이지 않도록 모두 한국어로 번역 및 통일하세요—문서 제목, 섹션 헤더(예:
"Goal", "Architecture", "Tech Stack"), 체크박스 작업 지시(- [ ])와 실행/기대 결과 설명을 포함해 파일
전체를 한국어로 교체하고 표현을 자연스럽게 다듬으세요; 문서 내에 언급된 식별자
TrafficRestoreOrchestratorService.start()와 resume() 등은 그대로 남겨두고 해당 메서드
설명(아키텍처/동작 흐름)은 한국어로 상세히 설명하도록 수정하며, 문서 규칙(docs/superpowers/**/*.md)에 따라 변경사항을
반영해 47-599 관련 참조도 한국어로 업데이트하세요.
In
`@src/main/java/com/pooli/traffic/service/decision/TrafficDeductOrchestratorService.java`:
- Line 45: The GLOBAL_POLICY_IDS constant currently omits the fail-closed policy
(policy_id 8), so add 8L to the list in the declaration of GLOBAL_POLICY_IDS and
ensure any code paths that use globalPolicyKeys(),
executePreflightKeyExistence(...), or hydrateOnDemandIfAnyPolicyKeyMissing(...)
will see it; better yet, replace the hard-coded List.of(...) with a shared
source — re-use the bootstrap service's policy list (or call the single method
that provides the canonical policy IDs) so this constant no longer diverges from
the bootstrap policy set.
In
`@src/main/java/com/pooli/traffic/service/policy/TrafficPolicyBootstrapService.java`:
- Around line 113-134: In hydrateOnDemandIfAnyPolicyKeyMissing add concise
Korean step comments inside the method body that outline: (1) hot-path quick
check using allPolicyKeysExist(policyKeys), (2) build lockKey/lockOwner and
attempt distributed lock via tryAcquireLock(lockKey, lockOwner), (3) on lock
success re-check allPolicyKeysExist(policyKeys) before proceeding, (4) call
synchronizePolicyActivationSnapshotWithoutLock("on_demand_preflight", false) to
hydrate snapshot, and (5) always release lock via releaseLock(lockKey,
lockOwner); place one-line comments near the corresponding statements
(referencing allPolicyKeysExist, lockKey/lockOwner, tryAcquireLock,
synchronizePolicyActivationSnapshotWithoutLock, releaseLock) to document the
sequencing and intent.
In
`@src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase2ReplayService.java`:
- Around line 87-93: registerIdempotencyCleanupAfterCommit currently registers a
TransactionSynchronization whose afterCommit calls
replayLuaExecutor.deleteIdempotencyKey(idempotencyKey) directly; wrap that call
in a try-catch that catches Throwable (or at least Exception/RuntimeException)
so any exception is swallowed and does not propagate to the transaction caller,
log the failure with context (include idempotencyKey) via your logger and
increment/emit a metric for cleanup failures, and keep the registration via
TransactionSynchronizationManager.registerSynchronization unchanged so cleanup
remains best-effort and non-fatal to the phase2 worker.
In
`@src/main/java/com/pooli/traffic/service/runtime/TrafficBalanceSnapshotHydrateService.java`:
- Around line 47-55: The current short-circuit uses
trafficRemainingBalanceCacheService.hasKey(balanceKey) and returns
TrafficBalanceSnapshotHydrateResult.hydrated(), which falsely treats partially
corrupted hashes as valid; change the pre-check and the post-lock re-check (the
checks around remainingIndivAmountKey and the similar checks at the other
locations noted) to verify that the required hash fields are present and
non-empty instead of just key existence: for individual snapshots ensure the
hash contains non-empty "amount" and "qos" fields (use the cache service's
hget/hmget-style API or add an isIndividualReady(balanceKey) helper), for shared
snapshots ensure the "amount" field is present (implement isSharedReady), and
only return hydrated() when those validations succeed; otherwise proceed to
hydrateWithLock (same lock key use: indivHydrateLockKey and
hydrateIndividualSnapshotUnlocked) so the lock/recheck logic uses the same
field-readiness criteria.
In
`@src/main/java/com/pooli/traffic/service/runtime/TrafficLuaScriptInfraService.java`:
- Around line 114-127: executePreflightKeyExistence calls executeListSingle with
TrafficLuaScriptType.PREFLIGHT_KEY_EXISTENCE but registerScript only registers
the restore list script, so requireListScript throws "Lua list script is not
registered" on first preflight; update registerScript to also register the
PREFLIGHT_KEY_EXISTENCE entry in the list-script registry (the same place that
registers restore scripts) so that requireListScript finds a script for
TrafficLuaScriptType.PREFLIGHT_KEY_EXISTENCE; reference the methods
registerScript, requireListScript, executeListSingle and the enum/value
PREFLIGHT_KEY_EXISTENCE when making the change.
---
Outside diff comments:
In
`@src/main/java/com/pooli/traffic/service/restore/TrafficRestoreOrchestratorService.java`:
- Around line 42-57: The manager lock currently uses a fixed TTL
(RESTORE_MANAGER_LOCK_TTL_MS) without any lease/heartbeat, so long-running
restores can exceed the TTL and allow a second start()/resume() to acquire the
same lock; modify TrafficRestoreOrchestratorService to implement a lock lease
extension or heartbeat while a worker is active (e.g., periodically refresh the
Redis lock via cacheStringRedisTemplate/trafficLuaScriptInfraService) and/or add
persistent session validation tied to batch metadata (using
LineDailyBatchJobMapper/batchJobMapper or related hydrate/replay lifecycle
checks) to refuse a second run; ensure the taskExecutor-driven worker loop
extends the lock until completion and that start()/resume() verify the active
lease/session before proceeding.
In
`@src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayService.java`:
- Around line 29-60: 메서드 replay의 핵심 트랜잭션 흐름을 짧은 한국어 단계 주석으로 고정해 주세요: 메서드 시작부에
"1) replay 명령 조회 및 실행", 실행 결과 분기부 위에 "2) 실패 시 terminal=FAILED 및 즉시 반환", 성공 경로의
상태 전환 직전에 "3) terminal=DONE으로 업데이트(동시성 체크)", 그리고
registerIdempotencyCleanupAfterCommit 내부에 "4) DB 커밋 확정 후 Redis idempotencyKey
삭제"와 같이 replay(), dailyAppTargetMapper.markTargetTerminalIfProcessing(...),
TrafficRestoreReplayResult.status(),
registerIdempotencyCleanupAfterCommit(String) 및
replayLuaExecutor.deleteIdempotencyKey(idempotencyKey) 호출들을 참조하는 간결한 한두 문장씩 넣어
주십시오.
---
Duplicate comments:
In
`@src/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase0HydrateService.java`:
- Around line 28-30: Before calling YearMonth.from in
TrafficRestorePhase0HydrateService.hydrate, explicitly validate
target.getTargetMonthStart() (from TrafficRestoreHydrateTarget) and fail fast
with a clear exception that includes owner/context info from the target (e.g.,
target.getOwner() or target identifier) so the error is actionable; update the
hydrate method to check for null/invalid targetMonthStart and throw an
IllegalArgumentException (or similar) with a message containing the owner/target
identifier instead of letting YearMonth.from throw a raw NullPointerException.
In
`@src/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayServiceTest.java`:
- Around line 58-146: Add a unit test that covers the FAILED replay path: create
a TrafficRestoreDailyAppTarget and TrafficRestoreReplayCommand, mock
replayLuaExecutor.replay(command) to return a TrafficRestoreReplayResult with
status "FAILED", mock
dailyAppTargetMapper.markTargetTerminalIfProcessing(target.getId(),
TrafficRestoreTargetStatus.FAILED, WORKER_ID) to return 1 (or the intended
ownership result), call service.replay(target, WORKER_ID) inside
TransactionSynchronizationManager.initSynchronization()/clearSynchronization(),
verify that markTargetTerminalIfProcessing was called with
TrafficRestoreTargetStatus.FAILED, and verify that
replayLuaExecutor.deleteIdempotencyKey(command.getIdempotencyKey()) is never
called both before and after invoking TransactionSynchronization::afterCommit
(i.e., ensure no deletion is scheduled for FAILED).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: f953385c-dc16-4e7e-84b8-8b81111b9760
📒 Files selected for processing (30)
.gitignoredocs/superpowers/plans/2026-05-31-global-policy-preflight-hydrate.mddocs/superpowers/plans/2026-05-31-restore-replay-remaining-atomicity.mddocs/superpowers/plans/2026-05-31-traffic-restore-async-flag-lifecycle.mddocs/superpowers/plans/2026-05-31-traffic-restore-idempotency-after-commit.mdsrc/main/java/com/pooli/traffic/controller/AdminTrafficRestoreController.javasrc/main/java/com/pooli/traffic/domain/dto/request/TrafficRestoreStartReqDto.javasrc/main/java/com/pooli/traffic/domain/dto/response/TrafficRestoreStartResDto.javasrc/main/java/com/pooli/traffic/domain/enums/TrafficLuaScriptType.javasrc/main/java/com/pooli/traffic/service/decision/TrafficDeductOrchestratorService.javasrc/main/java/com/pooli/traffic/service/policy/TrafficPolicyBootstrapService.javasrc/main/java/com/pooli/traffic/service/restore/TrafficRestoreExecutionService.javasrc/main/java/com/pooli/traffic/service/restore/TrafficRestoreOrchestratorService.javasrc/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase0HydrateService.javasrc/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayService.javasrc/main/java/com/pooli/traffic/service/restore/TrafficRestorePhase2ReplayService.javasrc/main/java/com/pooli/traffic/service/runtime/TrafficBalanceSnapshotHydrateService.javasrc/main/java/com/pooli/traffic/service/runtime/TrafficLuaScriptInfraService.javasrc/main/resources/lua/traffic/preflight_key_existence.luasrc/main/resources/lua/traffic/restore_usage_replay.luasrc/test/java/com/pooli/traffic/controller/AdminTrafficRestoreControllerTest.javasrc/test/java/com/pooli/traffic/service/decision/TrafficDeductOrchestratorServiceTest.javasrc/test/java/com/pooli/traffic/service/policy/TrafficPolicyBootstrapServiceTest.javasrc/test/java/com/pooli/traffic/service/restore/TrafficRestoreOrchestratorServiceTest.javasrc/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase0HydrateServiceTest.javasrc/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase1ReplayServiceTest.javasrc/test/java/com/pooli/traffic/service/restore/TrafficRestorePhase2ReplayServiceTest.javasrc/test/java/com/pooli/traffic/service/runtime/TrafficBalanceSnapshotHydrateServiceTest.javasrc/test/java/com/pooli/traffic/service/runtime/TrafficLuaPolicyContractTest.javasrc/test/java/com/pooli/traffic/service/runtime/TrafficRestoreLuaContractTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PULL REQUEST DESCRIPTION
개요
deduct_unified.lua)이 상정하는 Redis 데이터 구조(자료형 및 키 생성 계약)와 복구 배치 데이터 모델 간의 불일치를 해소하는 정합성 정렬 작업 수행.상세 아키텍처 및 복구 흐름
%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart TD A["Admin API: 복구 시작 요청"] --> B["장애 복구 Flag DB 활성화<br/>(policy_id = 8)"] B --> C["전역 정책 Redis Hydrate 실행"] C --> D["최악 처리 시간 + 1초 대기<br/>(In-flight 트래픽 유입 완전 차단)"] D --> E["Phase 0: Target 생성 및 Hydrate<br/>(월별 잔량 snapshot 및 전역 정책 복구)"] E --> F["Phase 1: Target 생성 및 Replay<br/>(DAILY_APP_TOTAL_DATA 기반 복구)"] F --> G["Phase 2: Done Log Replay<br/>(TRAFFIC_DEDUCT_DONE 기반 복구)"] G --> H["전체 Redis 데이터 검증 및 자동 보정<br/>(100% 전수 검사 & Correction Lua)"] H --> I["Idempotency Key 일괄 제거<br/>(restore:idempotency:* Cleanup)"] I --> J["장애 복구 Flag DB 비활성화"] J --> K["전역 정책 Redis Hydrate 재실행"] K --> L["복구 정상 완료 및 트래픽 게이트 개방"]Phase 1/2 Redis Replay 및 멱등성(Idempotency) 흐름
%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% sequenceDiagram autonumber actor Worker as 복구 Batch Worker participant DB as MySQL DB participant Redis as Redis Cache Worker->>DB: 복구 대상 레코드 선점 (FOR UPDATE SKIP LOCKED) DB-->>Worker: 대상 레코드 반환 및 status = PROCESSING 변경 Worker->>Redis: Idempotency Key 존재 확인 & Replay Lua 실행 alt Idempotency Key 미존재 시 (최초 Replay) Redis-->>Redis: 사용량 증가/잔량 차감 반영 및 Idempotency Key 생성 (TTL 무제한) Redis-->>Worker: APPLIED 반환 else Idempotency Key 존재 시 (중복 Replay 방지) Redis-->>Worker: SKIPPED 반환 end Worker->>DB: MySQL 대상 레코드 status = DONE 변경 및 Commit Worker->>Redis: Idempotency Key 제거 (Clean)주요 변경 사항 및 기술 의사 결정
1. 트래픽 진입 통제 및 안전 대기 체계
policy_id = 8에 대응하는 장애 복구 전역 정책 도입 및 해당 정책 활성화 시 트래픽 유입(Enqueue, Poll, Reclaim, Worker 차감 전 단계)을 완전 차단.worst processing time + 1000ms지연 대기를 적용하여 처리 중인 메시지의 안전한 잔존 처리를 보장하고 데이터 상태 변경 경합 차단.2. 다단계(Phase) 기반 점진적 복구 파이프라인 구축
remaining_indiv_amount), 공유 잔량 snapshot(remaining_shared_amount) 및 전역 정책 복구.DAILY_APP_TOTAL_DATA기준으로 Redis 일별/앱별 사용량 복구 및 잔량 차감.TRAFFIC_DEDUCT_DONE로그 기반 정밀 Replay 수행.각 복구 배치별 단계적 절차 시퀀스 다이어그램
[Phase 0] Target Insert & Redis Hydrate 절차
%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% sequenceDiagram autonumber actor Admin as 관리자 API / Orchestrator participant M_DB as MySQL DB (RESTORE_HYDRATE_TARGET) participant Worker as Hydrate Worker participant Redis as Redis Cache Admin->>M_DB: 1. 복구 대상 회선 식별 및 Hydrate Target 적재 (PENDING) Worker->>M_DB: 2. PENDING 대상 선점 (FOR UPDATE SKIP LOCKED) M_DB-->>Worker: 3. LINE/FAMILY/POLICY 대상 정보 반환 및 PROCESSING 전환 opt LINE 복구 Worker->>M_DB: 4-1. MySQL 최종 월별 잔량 snapshot 조회 Worker->>Redis: 4-2. snapshot 기반 개인 잔량(remaining_indiv_amount) HSET 복구 end opt FAMILY 복구 Worker->>M_DB: 5-1. MySQL 최종 월별 잔량 snapshot 조회 Worker->>Redis: 5-2. snapshot 기반 공유 잔량(remaining_shared_amount) HSET 복구 end opt GLOBAL_POLICY 복구 Worker->>M_DB: 6-1. MySQL POLICY 테이블 전역 정책 정보 조회 Worker->>Redis: 6-2. 전역 정책 정보(policy:*) HSET 반영 end Worker->>M_DB: 7. 선점 대상 DONE 상태 갱신[Phase 1] Target Insert & Daily App Replay 절차
%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% sequenceDiagram autonumber actor Admin as 관리자 API / Orchestrator participant M_DB as MySQL DB (RESTORE_DAILY_APP_TARGET) participant Worker as Replay Worker participant Redis as Redis Cache Admin->>M_DB: 1. DAILY_APP_TOTAL_DATA 기반 Replay Target 적재 (PENDING) Worker->>M_DB: 2. PENDING 대상 선점 (FOR UPDATE SKIP LOCKED) M_DB-->>Worker: 3. Replay 대상 정보 반환 및 PROCESSING 전환 Worker->>Redis: 4. Replay Lua 실행 (멱등키 검증 + 사용량/잔량 누적 차감) alt 멱등키 존재 시 Redis-->>Worker: 5-1. SKIPPED 반환 else 멱등키 미존재 시 Redis-->>Redis: 5-2. daily_total_usage(String INCRBY), daily_app_usage(Hash HINCRBY), daily/monthly_shared_usage 누적 업데이트 및 멱등키 생성 Redis-->>Worker: 5-3. APPLIED 반환 end Worker->>M_DB: 6. 선점 대상 DONE 상태 갱신 및 Commit Worker->>Redis: 7. 복구 멱등키 제거 (Clean)[Phase 2] Done Log Replay & Verification/Correction 절차
%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% sequenceDiagram autonumber actor Admin as 관리자 API / Orchestrator participant M_DB as MySQL DB (TRAFFIC_DEDUCT_DONE) participant Worker as Done Log Worker participant Redis as Redis Cache Admin->>M_DB: 1. 복구 범위 내 Done Log 중 restore_status가 NONE/RETRYABLE인 건 직접 조회 M_DB-->>Admin: 2. Done Log 레코드 반환 및 PROCESSING 전환 Admin->>Redis: 3. Replay Lua 실행 (멱등성 보장 하에 실시간 사용량/잔량 Replay 반영) Admin->>M_DB: 4. Done Log restore_status = DONE 커밋 Admin->>Redis: 5. 멱등성 임시 키 즉시 제거 Note over Admin, Redis: 모든 Replay 완료 후 전수 검증 및 자동 보정 개시 Admin->>M_DB: 6. 복구 기점 원천 기준 복구 기대치 데이터셋 일괄 조회 Admin->>Redis: 7. Redis 실측값 병렬 조회 및 비교 대조 alt 불일치 데이터 감지 시 Admin->>Redis: 8. Correction Lua 실행 (DEL 후 재생성 및 기대값으로 원자 보정) end Admin->>Redis: 9. 복구용 멱등키 프리픽스(restore:idempotency:*) 일괄 삭제각 복구 배치별 세부 단계적 진행 절차
Step 1: RESTORE_P0_TARGET_INSERT (Phase 0 복구 대상 적재)
LineMonthUsage및 Done Log 이력 기반 대상 회선 일괄 추출.RESTORE_HYDRATE_TARGET테이블에 LINE, FAMILY, GLOBAL_POLICY 유형별 복구 대상 레코드 삽입.PENDING상태로 초기화 기입하며, Unique 제약 조건을 통해 동일 식별자 중복 유입 예방.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart LR A["LineMonthUsage / Done Log 스캔"] --> B["중복 식별자 제거<br/>(Unique Key 필터)"] B --> C["RESTORE_HYDRATE_TARGET 테이블에<br/>PENDING 레코드 삽입"]Step 2: RESTORE_P0_REDIS_HYDRATE (Phase 0 Redis Hydrate)
RESTORE_HYDRATE_TARGET에서 청크 단위(FOR UPDATE SKIP LOCKED)로PENDING및RETRYABLE상태 대상을 선점하여PROCESSING상태로 전환.remaining_indiv_amount) 복원.remaining_shared_amount) 복원. (결합 회선 한정)POLICY테이블 기준으로 전역 정책 정보 및 장애 복구 플래그를 Redis에 반영.DONE갱신. 실패 시 에러 사유 마킹 및RETRYABLE/FAILED상태 분기 적용.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart TD A["PENDING/RETRYABLE 선점<br/>(SKIP LOCKED)"] --> B{"대상 타입 판별"} B -- "LINE" --> C["MySQL 잔량 snapshot 조회"] --> F["Redis remaining_indiv_amount 복구"] B -- "FAMILY" --> D["MySQL 잔량 snapshot 조회"] --> G["Redis remaining_shared_amount 복구"] B -- "GLOBAL_POLICY" --> E["MySQL POLICY 데이터 조회"] --> H["Redis policy:* 전역정책 반영"] F & G & H --> I["MySQL DONE 마킹"]Step 3: RESTORE_P1_TARGET_INSERT (Phase 1 복구 대상 적재)
DAILY_APP_TOTAL_DATA전체 정보 추출.RESTORE_DAILY_APP_TARGET테이블에 개별 앱 사용량 단위 복구 데이터 삽입 및PENDING상태 초기화.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart LR A["복구 대상 날짜 범위 확정"] --> B["DAILY_APP_TOTAL_DATA 조회"] B --> C["RESTORE_DAILY_APP_TARGET 테이블에<br/>PENDING 적재"]Step 4: RESTORE_P1_DAILY_APP_REPLAY (Phase 1 Redis Replay)
RESTORE_DAILY_APP_TARGET에서 청크 단위로 미완료 상태 레코드 선점 및PROCESSING마킹.restore_usage_replay.lua)를 호출하여 일별/앱별 누적 사용량 증가 및 잔량 차감 수행.restore:idempotency:p1:daily_app:...) 유무 판단을 수행하여 중복 반영 차단 및 처리 완료 후 멱등 키 생성.DONE상태 Commit 종결 후 Redis 멱등 키 즉각 제거 처리.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart TD A["DAILY_APP_TARGET 선점"] --> B["Replay Lua 스크립트 실행"] B --> C{"Idempotency Key 검증"} C -- "존재 (중복)" --> D["Replay 스킵 (SKIPPED)"] C -- "미존재 (최초)" --> E["Redis 사용량/잔량 누적 갱신<br/>& 멱등키 생성"] D & E --> F["MySQL DONE 상태 커밋"] F --> G["Redis 멱등성 키 제거"]Step 5: RESTORE_P2_DONE_LOG_REPLAY (Phase 2 Done Log Replay)
TRAFFIC_DEDUCT_DONE테이블을 복구 날짜 범위 기반으로 직접 쿼리하여restore_status가NONE/RETRYABLE인 건 선점.restore:idempotency:p2:done_log:...) 검증 하에 누적 사용량 및 잔량 Replay 반영.restore_status = DONE) 커밋 직후 멱등 키 즉시 제거.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart TD A["TRAFFIC_DEDUCT_DONE 스캔<br/>(NONE/RETRYABLE)"] --> B["Replay Lua 실행 (멱등키 검증)"] B --> C{"멱등키 존재 여부"} C -- "미존재" --> D["Redis 차감 및 누적 실행<br/>& 멱등키 생성"] C -- "존재" --> E["Replay 생략"] D & E --> F["MySQL restore_status = DONE 변경"] F --> G["Redis 멱등 키 제거"]Step 6: 전체 검증 및 자동 보정 단계 (Verification & Correction)
TrafficRestoreVerificationMapper를 활용하여 복구 원천 데이터 기준의 전수 기대치(개인/공유 잔량 및 사용량) 데이터셋 일괄 산출.restore_usage_correction.lua)를 호출해 강제 보정(자료형 불일치 시DEL후 재기입) 적용 및 오류 상태 추적.restore:idempotency:*계열 멱등성 임시 키 전체 제거 및 장애 복구 플래그 해제.%%{init: {"theme": "default", "themeVariables": {"background": "#ffffff"}}}%% flowchart TD A["원천 데이터 기반 기대치 계산<br/>(VerificationMapper)"] --> B["Redis 실측값 조회 및 비교"] B --> C{"정합성 일치 여부"} C -- "일치" --> F["통과"] C -- "불일치" --> D["Correction Lua 호출"] D --> E["오류 자료형 DEL 후<br/>목표 구조 재작성 & 보정"] E --> F F --> G["임시 멱등 키 일괄 삭제<br/>(restore:idempotency:*)"] G --> H["장애 복구 Flag 해제 및 게이트 오픈"]3. 실시간 트래픽 차감 계약 기반 Redis 정합성 정렬
deduct_unified.lua) 및 조회기(LineDailyUsageRedisReader) 계약에 맞춰 StringINCRBY카운터 형식으로 복구하도록 수정.daily_shared_usage및monthly_shared_usage키 자체를 생성하지 않도록 설계 정비 (불필요한 Hash 키 생성 방지).amount단독 검증에서 탈피하여 개인 잔량 Hash의amount와qos필드를 일괄 검증 및 동시 보정하도록 구현체 수정.DEL처리 후 정상 자료형으로 강제 재생성하도록restore_usage_correction.lua보완.관련 BackLog
Resolves: N/A (장애 복구 전용 배치 태스크 전용 브랜치 작업 건)
PR 유형
PR Checklist
상세 변경 이력 요약
1. Redis Key별 데이터 명세 및 복구 계약 요약
remaining_indiv_amount:{lineId}:{yyyyMM}amount(개인 잔량),qos(QoS 한도)remaining_shared_amount:{familyId}:{yyyyMM}amount(공유 잔량)daily_total_usage:{lineId}:{yyyyMMdd}INCRBY계약 준수 (Hash 오복구 교정 완료)daily_app_usage:{lineId}:{yyyyMMdd}app:{appId}:individualapp:{appId}:sharedapp:{appId}:qosdaily_shared_usage:{lineId}:{yyyyMMdd}usage_amount,family_idmonthly_shared_usage:{lineId}:{yyyyMM}usage_amount,family_id2. 정합성 결함 수정 전/후 동작 비교
INCRBY구조로 정형화 복구amount단독 검증으로qos유실 및 오염 가능성 존재amount와qos를 묶어 1회 트랜잭션으로 원자적 검증 및 보정DEL처리 선행 후 정상 자료형 구조로 리빌딩MISSING_FAMILY_ID에러 반환 및 실패 기록3. 변경 파일별 책임 구조도