Loyalty Points Program - Temporal Workflow Demo

Demonstration of Temporal patterns through a customer loyalty program

This project illustrates the implementation of core Temporal patterns (Entity Workflow, Update Methods, Queries, Saga) in a realistic business context: a loyalty points system with tier management (Basic → Gold → Platinum), point accumulation/redemption, and grace period upon program departure.

🎯 Why This Project?

Temporal excels at managing long-running business workflows, and this project demonstrates:

Entity Workflow Pattern: One workflow per customer, living for years
Update Methods: State modification with transactional guarantees
Queries: Synchronous workflow state reads
Saga Pattern: Automatic compensation on failure (point spending)
Resilience: Chaos injection to test robustness
Multi-language: Go, Java, Python, TypeScript in a single system

🏗️ Architecture

graph TD
    Gateway["Gateway<br/>(Caddy :8080)"]

    Dashboard["Dashboard<br/>(Vue/Nuxt)"]
    ChaosService["Chaos Service<br/>(Java/Spring)"]
    ChaosEngine["Chaos Engine<br/>(Toxiproxy)"]

    CustomerService["Customer Service<br/>(Go)"]
    LoyaltyService["Loyalty Service<br/>(Java/Temporal)"]
    EmailService["Email Service<br/>(Python/FastAPI)"]
    RedemptionService["Redemption Service<br/>(TypeScript/Node.js)"]

    Redis[("Redis")]
    Temporal[("Temporal Server")]

    Gateway -->|"/*"| Dashboard
    Gateway -->|"/api/chaos/*"| ChaosService
    Gateway -->|"/api/*"| ChaosEngine

    ChaosService --> ChaosEngine

    ChaosEngine -->|"proxies"| CustomerService
    ChaosEngine -->|"proxies"| LoyaltyService
    ChaosEngine -->|"proxies"| EmailService
    ChaosEngine -->|"proxies"| RedemptionService

    CustomerService --> Redis

    LoyaltyService -->|"orchestrates"| Temporal
    LoyaltyService -.->|"activity: sendEmail"| EmailService
    LoyaltyService -.->|"activity: processRedemption"| RedemptionService

Components

Service	Technology	Role
Dashboard	Vue.js + Nuxt (static)	User interface
Gateway	Caddy	Unified reverse proxy
Loyalty Service	Java + Spring Boot + Temporal SDK	Loyalty workflows (core logic)
Customer Service	Go + Redis	Customer CRUD
Email Service	Python + FastAPI	Notifications (mock)
Redemption Service	TypeScript + Node.js	Redemption processing (mock)
Chaos Service	Java + Spring Boot	Chaos control API
Chaos Helper	Go	HTTP error generator
Chaos Engine	Toxiproxy	Failure injection
Temporal Server	Temporal	Workflow orchestration

🚀 Launch the Demo (Production)

# Start all containerized services
docker compose up -d

# Check status
docker compose ps

# Access the application
open http://localhost:8080

# Access Temporal UI
open http://localhost:8233

Available features:

Create customers and enroll them in the program
Earn points (automatic tier upgrade at 500 and 1000 points)
Spend points (with Saga compensation on failure)
Leave and rejoin the program (30-day grace period)
Inject chaos (ON/OFF/SLOW/ERROR) to test resilience

🛠️ Development Workflow

Development Mode with Hot Reload

# 1. Start infrastructure (in Docker)
docker compose -f compose-dev.yaml up -d

# 2. Separate terminal: Dashboard with hot reload
cd dashboard
npm install
npm run dev

# 3. Separate terminal: Loyalty Service with hot reload
cd loyalty
mvn spring-boot:run

Unified access: Always via http://localhost:8080 (gateway proxies to host)

Useful Commands

# Rebuild a specific image
docker compose build loyalty-service

# View service logs
docker compose logs -f loyalty-service

# Restart a service
docker compose restart customer-service

# Stop everything
docker compose down

📐 Temporal Patterns Implemented

1. Entity Workflow Pattern

Each customer has their own long-running workflow:

Workflow ID: loyalty-customer-{id}
Lifetime: Potentially years
Persistent state: Points, tier, transactions, grace period

2. Update Methods (Signals)

Synchronous workflow state modification:

earnPoints(purchaseId, amount) - Earn points (idempotent)
spendPoints(redemptionId, points, reason) - Spend points (saga)
leaveProgram(reason) - Leave program (starts grace period)
rejoinProgram() - Rejoin program (within 30 days)

3. Queries

Synchronous reads without state modification:

getBalance() - Current points and tier
getCustomerStatus() - Full status (Active/Left/Expired)
getTransactions() - Transaction history

4. Saga Pattern

Automatic compensation on failure during point spending:

Deduct points from workflow
Call redemption service
If failure → Automatically credit points back (rollback)
Send email notification of result

5. Chaos Engineering

Controlled failure injection via Toxiproxy:

ON: Service operational
OFF: Service unreachable (timeout)
SLOW: 30-second latency injection
ERROR: HTTP 400 responses
KILL: Abrupt process termination (auto-restart in 10-20s)

🔄 Saga Pattern in Detail

The Saga pattern ensures data consistency when orchestrating distributed transactions across multiple services. This project implements the Saga pattern for the spend points operation, which involves external service calls that may fail.

Compensation Mechanism

When a customer spends points, the system must:

Deduct points from the workflow state
Process the redemption via an external service
Handle failures gracefully by compensating (rolling back) the deduction

This prevents inconsistent states where points are deducted but the redemption fails.

Sequence Diagram

sequenceDiagram
    participant User as Dashboard
    participant API as Loyalty API
    participant WF as Loyalty Workflow
    participant TS as Temporal Server
    participant RS as Redemption Service
    participant ES as Email Service

    Note over User,ES: Success Scenario
    User->>API: POST /spend (points: 100)
    API->>TS: Send Update Method
    TS->>WF: spendPoints(redemptionId, 100)
    WF->>WF: Reserve points (-100)
    WF->>RS: Activity: processRedemption(100)
    RS-->>WF: Success (value: $100)
    WF->>WF: Confirm spend (update state)
    WF->>ES: Activity: sendEmail(SUCCESS)
    ES-->>WF: Email sent
    WF-->>API: Update complete
    API-->>User: 200 OK

    Note over User,ES: Failure Scenario with Compensation
    User->>API: POST /spend (points: 200)
    API->>TS: Send Update Method
    TS->>WF: spendPoints(redemptionId, 200)
    WF->>WF: Reserve points (-200)
    WF->>RS: Activity: processRedemption(200)
    RS-->>WF: Error (insufficient funds)
    WF->>WF: Compensate: Restore points (+200)
    WF->>ES: Activity: sendEmail(FAILED)
    ES-->>WF: Email sent
    WF-->>API: Update complete (compensated)
    API-->>User: 200 OK (with compensation message)

Implementation Details

The implementation uses Temporal's native Saga class to manage compensation automatically:

Step 1: Initialize Saga

final var sagaOptions = new Saga.Options.Builder()
    .setParallelCompensation(false)  // Sequential compensation
    .setContinueWithError(false)     // Stop on first error
    .build();
final var saga = new Saga(sagaOptions);

Step 2: Reserve Points & Register Compensation

// Deduct points optimistically (before external call)
final var transaction = state.reservePoints(redemptionId, points, reason);

// Register compensation BEFORE calling external service
// This ensures we have a rollback plan ready
saga.addCompensation(state::compensateSpend, points, transaction);

Step 3: Call External Service

try {
    // Attempt to process redemption via Temporal Activity
    activities.processRedemption(redemptionId, customerId, points, reason);

    // Success: Confirm the transaction
    state.confirmSpend(redemptionId, transaction);

} catch (ActivityFailure e) {
    // Failure: Execute ALL registered compensations in LIFO order
    saga.compensate();

    // Extract error code for reporting
    var errorCode = LoyaltyErrorCode.UNKNOWN_ERROR;
    if (e.getCause() instanceof ApplicationFailure appFailure) {
        errorCode = LoyaltyErrorCode.valueOf(appFailure.getType());
    }
    return SpendResult.failure(state.getCurrentPoints(), currentLevel, errorCode);
}

Key Benefits

Automatic Rollback: Points are restored if redemption fails
Guaranteed Execution: Temporal ensures compensation runs even after process crashes
Audit Trail: All compensation events are recorded in transaction history
Idempotency: Using redemptionId prevents duplicate processing
User Notification: Email activities inform users of both success and failure

Error Handling

The saga handles multiple failure scenarios:

Service Unavailable: Temporal retries the activity automatically
Business Rejection: Immediate compensation without retry
Timeout: Retry with backoff, then compensate if exhausted
Network Error: Retry with exponential backoff

All failures trigger the compensation logic, ensuring the workflow state remains consistent.

🔧 Chaos Engineering System

This project includes a comprehensive chaos engineering system to demonstrate Temporal's resilience patterns under various failure conditions. The system allows controlled injection of failures to test how the workflow handles real-world distributed system challenges.

Architecture

The chaos system consists of three components working together:

graph LR
    Gateway["Gateway"] -->|"/api/*"| Toxiproxy
    Dashboard -->|"Control API"| ChaosService["Chaos Service<br/>(Spring Boot)"]
    ChaosService -->|"Configure"| Toxiproxy["Toxiproxy<br/>(Shopify)"]

    Toxiproxy -->|"proxy"| Customer["Customer Service"]
    Toxiproxy -->|"proxy"| Email["Email Service"]
    Toxiproxy -->|"proxy"| Redemption["Redemption Service"]
    Toxiproxy -.->|"redirect (ERROR)"| Helper["Chaos Helper<br/>(Go)"]

Component	Role	Technology
Toxiproxy	Network proxy that injects failures	Shopify Toxiproxy
Chaos Service	REST API to control Toxiproxy	Java/Spring Boot
Chaos Helper	Mock service returning HTTP 400 errors	Go

Failure Modes

Each service can be set to one of four states to simulate different failure scenarios:

State	Implementation	HTTP Response	Temporal Behavior
ON	No failures, normal operation	200 OK	Activities succeed
OFF	Connection reset (TCP reset)	Connection failed	Activities retry with exponential backoff
SLOW	30-second latency injection	Timeout	Activities timeout, retry until exhausted
ERROR	Redirect to mock service	400 Bad Request	Activities fail immediately (non-retryable)

Implementation Details:

ON: No Toxiproxy toxics applied, traffic flows normally
OFF: reset_peer toxic immediately closes TCP connections
SLOW: latency toxic adds 30,000ms delay to all requests
ERROR: Upstream redirected to Chaos Helper returning HTTP 400 + 2s delay

Testing Temporal Resilience

The chaos system demonstrates key Temporal capabilities:

1. Automatic Activity Retries (OFF State)

When a service is OFF, Temporal automatically retries failed activities:

T+0s  : Activity starts → Connection refused
T+1s  : Retry #1 → Connection refused
T+3s  : Retry #2 → Connection refused
T+7s  : Retry #3 → Connection refused
T+15s : Retry #4 → Connection refused
...   : (Turn service ON)
T+31s : Retry #5 → Success!

Retry Policy:

RetryOptions.newBuilder()
    .setInitialInterval(Duration.ofSeconds(1))
    .setBackoffCoefficient(2.0)
    .setMaximumAttempts(5)
    .setMaximumInterval(Duration.ofSeconds(30))
    .build();

ActivityOptions.newBuilder()
    .setStartToCloseTimeout(Duration.ofSeconds(10))
    .setRetryOptions(retryOptions)
    .build()

2. Saga Compensation (OFF/ERROR States)

When the Redemption Service fails during spend points:

// Service OFF → Connection failure → Retries exhausted → Saga compensates
// Service ERROR → HTTP 400 → Immediate failure → Saga compensates

Expected behavior:

Points deducted optimistically
Redemption activity fails
Compensation automatically restores points
Transaction marked as COMPENSATED in audit trail

3. Timeout Handling (SLOW State)

When latency exceeds activity timeout, Temporal retries until exhausted:

T+0s  : spendPoints() called
T+0s  : Reserve 100 points (workflow state)
T+0s  : processRedemption activity starts
T+10s : Activity timeout (30s latency > 10s timeout)
T+11s : Retry #1 starts
T+21s : Retry #1 timeout
T+23s : Retry #2 starts
...   : Retries continue with exponential backoff
T+63s : All retries exhausted (5 attempts)
T+63s : Compensation triggered → Points restored

Activity timeout (10 seconds) is less than 30s latency injection, causing retries to exhaust and saga compensation to trigger.

4. Non-Retryable Errors (ERROR State)

HTTP 400 errors trigger immediate failure without retries:

catch (ActivityFailure e) {
    if (e.getCause() instanceof ApplicationFailure appFailure) {
        String errorCode = appFailure.getType();
        // errorCode = "REDEMPTION_REJECTED" or similar
    }
    saga.compensate(); // Immediate rollback
}

Demo Scenarios

Scenario 1: Email Service Resilience

Setup: Set Email Service to OFF

Test:

Enroll customer (triggers welcome email)
Email activity fails immediately
Temporal retries automatically (5 attempts with exponential backoff)
Set Email Service to ON
Next retry succeeds

Result: Demonstrates Temporal's automatic retry mechanism

Scenario 2: Saga Compensation with Connection Failure

Setup: Set Redemption Service to OFF

Test:

Customer has 500 points
Spend 100 points
Points deducted → 400 points
Redemption activity fails (retries exhausted)
Compensation triggers → 500 points restored

Result: Demonstrates saga compensation on network failure

Scenario 3: Saga Compensation with Business Error

Setup: Set Redemption Service to ERROR

Test:

Customer has 500 points
Spend 100 points
Points deducted → 400 points
Redemption returns HTTP 400 (immediate failure, no retries)
Compensation triggers immediately → 500 points restored

Result: Demonstrates fast compensation on non-retryable errors

Scenario 4: Timeout and Retry Exhaustion

Setup: Set Redemption Service to SLOW

Test:

Spend 100 points
Activity starts processing
Activity times out after 10 seconds (latency is 30s)
Temporal retries automatically (5 attempts)
All retries exhaust (each times out after 10s)
Compensation triggers → Points restored

Result: Demonstrates timeout handling and compensation when latency exceeds activity timeout

Chaos Control API

Control chaos states via REST API or Dashboard UI:

# Get all service states
curl http://localhost:8080/api/chaos/services

# Set service to OFF
curl -X POST http://localhost:8080/api/chaos/services/email/state \
  -H "Content-Type: application/json" \
  -d '{"state": "off"}'

# Reset all services to ON
curl -X POST http://localhost:8080/api/chaos/reset

# Emergency: Kill a service process
curl http://localhost:8080/api/chaos/services/email/kill

Key Insights

The chaos system validates that:

Workflows are durable: State persists through service failures
Activities retry automatically: Network failures don't require manual intervention
Compensation executes reliably: Saga pattern maintains consistency even after retry exhaustion
Timeouts protect workflows: Activities exceeding timeout trigger retries and eventual compensation
Error codes matter: Retryable vs non-retryable errors trigger different behavior

This demonstrates Temporal's core value proposition: reliable execution in unreliable environments.

📊 Loyalty Tiers

Tier	Points Required	Description
Basic	0	Default tier upon enrollment
Gold	500	Mid-tier
Platinum	1000	Premium tier

Tier upgrades are automatic when accumulating points.

📁 Project Structure

.
├── dashboard/          # Vue.js/Nuxt frontend (static)
├── loyalty/            # Java/Spring Boot + Temporal SDK service
├── customer/           # Go + Redis service
├── email/              # Python/FastAPI service (mock)
├── redemption/         # TypeScript/Node.js service (mock)
├── chaos/              # Chaos Service (Java) + Chaos Helper (Go)
├── gateway/            # Caddy configuration
├── specs/              # Complete documentation
│   ├── index.md        # Overview
│   ├── domain.md       # Domain model
│   ├── workflow.md     # Workflow specification
│   ├── architecture.md # Detailed architecture
│   ├── api.md          # REST endpoints
│   ├── chaos.md        # Chaos strategies
│   └── testing.md      # Manual testing guide
├── compose.yaml        # Production (all services containerized)
└── compose-dev.yaml    # Development (Dashboard + Loyalty on host)

📚 Complete Documentation

Detailed documentation is located in the specs/ folder:

specs/index.md - Entry point, overview
specs/workflow.md - Temporal workflow specification
specs/architecture.md - Architecture and communication
specs/domain.md - Business domain model
specs/api.md - REST API documentation
specs/chaos.md - Chaos engineering strategies
specs/testing.md - Complete manual testing guide

🎓 Temporal Concepts Demonstrated

This project is designed as a learning reference for:

Long-running workflows: Complete lifecycle of a business entity
Determinism: Guaranteed workflow replay
Activities: Non-deterministic external calls (Redis, HTTP)
Persistent state: State management without external database
Idempotency: Deduplication via processedTransactionIds
Timers: 30-day grace period with Workflow.await()
Compensation: Automatic rollback with Saga pattern
Resilience: Automatic recovery after injected failures

⚙️ Prerequisites

Docker + Docker Compose (for services)
Java 25+ + Maven (for Loyalty Service in dev)
Node.js 24+ + npm (for Dashboard in dev)

🤖 Built with Claude Code

This project was designed and implemented with assistance from Claude Code, Anthropic's CLI tool for software development. Claude Code contributed to:

Specification Design: Detailed technical specifications in the specs/ folder
Architecture Planning: Multi-service architecture with chaos engineering components
Implementation: Code generation across multiple languages (Java, Go, Python, TypeScript)
Documentation: Comprehensive README and inline code documentation
Testing Scenarios: Manual test scenarios and chaos engineering strategies

📄 License

This project is distributed under the Apache 2.0 license. See LICENSE for details.

🤝 Contributing

This project is an educational demonstration. Contributions are welcome for:

Adding automated tests
Improving documentation
Proposing new Temporal patterns
Fixing bugs

→ Quick start: docker compose up -d then open http://localhost:8080

Name		Name	Last commit message	Last commit date
Latest commit History 79 Commits
.claude		.claude
.github/workflows		.github/workflows
chaos		chaos
customer		customer
dashboard		dashboard
email		email
gateway		gateway
it		it
loyalty		loyalty
redemption		redemption
specs		specs
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
app.png		app.png
chaos.png		chaos.png
compose-dev.yaml		compose-dev.yaml
compose.yaml		compose.yaml
temporal.png		temporal.png

Folders and files

Latest commit

History

Repository files navigation

Loyalty Points Program - Temporal Workflow Demo

🎯 Why This Project?

🏗️ Architecture

Components

🚀 Launch the Demo (Production)

🛠️ Development Workflow

Development Mode with Hot Reload

Useful Commands

📐 Temporal Patterns Implemented

1. Entity Workflow Pattern

2. Update Methods (Signals)

3. Queries

4. Saga Pattern

5. Chaos Engineering

🔄 Saga Pattern in Detail

Compensation Mechanism

Sequence Diagram

Implementation Details

Key Benefits

Error Handling

🔧 Chaos Engineering System

Architecture

Failure Modes

Testing Temporal Resilience

1. Automatic Activity Retries (OFF State)

2. Saga Compensation (OFF/ERROR States)

3. Timeout Handling (SLOW State)

4. Non-Retryable Errors (ERROR State)

Demo Scenarios

Scenario 1: Email Service Resilience

Scenario 2: Saga Compensation with Connection Failure

Scenario 3: Saga Compensation with Business Error

Scenario 4: Timeout and Retry Exhaustion

Chaos Control API

Key Insights

📊 Loyalty Tiers

📁 Project Structure

📚 Complete Documentation

🎓 Temporal Concepts Demonstrated

⚙️ Prerequisites

🤖 Built with Claude Code

📄 License

🤝 Contributing

About

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages