SentinelPay: Exactly-Once Payment Execution Service

👥 Owners

• Aman Kumar — @001ak
• Prerna Kumari — @prerna456

A fault-tolerant payment execution engine guaranteeing exactly-once processing across client retries, JVM crashes, and external gateway failures.

📖 Project Overview

SentinelPay is a heavily resilient payment execution service designed to handle financial transactions where duplicate processing is unacceptable. It serves as a hardened intermediary between client applications and external payment gateways.

The core objective of this system is to enforce Idempotency and Data Consistency under chaos (network drops, database timeouts, and external provider failures). It intentionally avoids caching layers to keep the transactional boundary strictly within the SQL database, preventing dual-write inconsistencies.

🏗️ Technical Architecture & Stack

• Language/Framework: Java 17+, Spring Boot 3.x
• Database: PostgreSQL (Single source of truth enforcing ACID, unique constraints, and row-level locks)
• Message Broker: RabbitMQ
• Data Access: Spring Data JPA / Hibernate
• Testing & Chaos: JUnit 5, Testcontainers (Real DB/Broker instances), WireMock (External API failure simulation)

🛡️ Core Engineering Patterns & Guarantees

1. The Idempotency Engine

The Challenge: Clients often retry requests (e.g., clicking "Pay" twice or network drops after sending). The system must never charge the user twice. The Implementation:

• Every API request requires an Idempotency-Key header.
• The request payload is hashed (SHA-256). If a request arrives with an existing key but a mismatched hash, it is rejected with a 409 Conflict.
• The Guarantee: Handled entirely via a database UNIQUE constraint, removing race conditions that occur in application-level checks.

2. Strict State Machine & Optimistic Locking

The Challenge: Concurrent processes might attempt to modify the same payment simultaneously (e.g., a webhook arriving while the synchronous API is still processing). The Implementation:

• Payments follow a strict FSM: CREATED -> PROCESSING -> SUCCESS | FAILED. Invalid transitions throw an IllegalStateException.
• Utilizes JPA @Version for Optimistic Locking. If a concurrent modification occurs, the transaction rolls back, preventing dirty writes.

3. The Transactional Outbox Pattern

The Challenge: After a successful payment, downstream systems (like notification or billing services) need to be alerted. If the application updates the database but crashes before sending the Kafka/RabbitMQ message, the event is lost forever (the dual-write problem). The Implementation:

• Payment state mutation and the insertion of an OutboxEvent (delivery_status = PENDING) occur within a single @Transactional PostgreSQL commit.
• An asynchronous Outbox Poller guarantees at-least-once delivery to RabbitMQ, decoupling the business logic from broker reliability.

4. Concurrency Control: The Poller (Skip Locked)

The Challenge: If multiple instances of the Outbox Poller run simultaneously (horizontal scaling), they will pull the same events, causing duplicate downstream messages (the thundering herd problem). The Implementation:

• The background worker fetches pending events using a native SELECT ... FOR UPDATE SKIP LOCKED query.
• This safely allows multiple application instances to poll the same database table concurrently without row contention or duplicate processing.

5. The Reconciliation Worker

The Challenge: When the external payment provider times out (e.g., returns a 504), the internal state is stuck in PROCESSING. The application doesn't know if the money moved or not. The Implementation:

• A scheduled job periodically sweeps for payments stuck in the PROCESSING state past a certain threshold.
• It queries the external provider for the true source-of-truth status and safely transitions the internal payment to SUCCESS or FAILED, ensuring eventual consistency.

🗄️ Database Schema Details

The schema relies heavily on PostgreSQL constraints and JSONB flexibility.

• payments: The core transactional table.
  • Enforces uniqueness on idempotency_key.
  • Contains a partial index: CREATE INDEX idx_payments_processing_time ON payments (status, created_at) WHERE status = 'PROCESSING'; (Crucial for the Reconciliation Worker).
• outbox_events: The event staging table.
  • Contains a partial index: CREATE INDEX idx_outbox_pending_retry ON outbox_events (delivery_status, next_retry_at) WHERE delivery_status = 'PENDING'; (Crucial for the fast polling).