Skip to content
BranchesTags

Repository files navigation

EventFlow

EventFlow

Production-grade distributed event processing — Kafka ingestion, saga workflows, retries, DLQ, replay, and cloud-native observability.

CI Go Kafka PostgreSQL Redis Kubernetes Terraform MIT

Quick Start · Live Demo · Architecture · Case Study · Recruiter Guide

Live Demo

EventFlow live demo — publish, saga, retry, DLQ, replay

ShipPurchased → GalacticCommerce saga → retry → DLQ → replay → Grafana metrics — ~80 seconds, real APIs. 

# Full stack + 9-act demo
.\scripts\demo.ps1

# Re-run when stack is already up
.\scripts\demo.ps1 -SkipStackStart
Resource Link
Demo script docs/demo/demo-script.md
Video guide docs/demo/video.md
Record a GIF docs/assets/generate-gifs.md

Why EventFlow?

Modern services emit millions of events per day. A single purchase fans out to payment, inventory, and notifications — and failures are inevitable.

EventFlow is the infrastructure layer that makes async processing reliable:

Capability What operators get
Durable ingestion Kafka topics with idempotent publish APIs
Saga orchestration Multi-step workflows with LIFO compensation
Transient failure handling Exponential backoff retry engine
Poison message quarantine {topic}-dlq topics + inspection APIs
Incident recovery DLQ and time-range replay without redeploys
Production visibility Prometheus metrics + Grafana dashboards

Built in Go with REST and gRPC, deployable via Docker Compose, Helm, and Terraform (AWS).

Architecture

EventFlow architecture 

Clients ──► API Gateway (REST :8080 / gRPC :9090)
                │
                ├──► Kafka (ship-orders, orders, payments, …)
                │         └──► Consumer Workers ──► Retry Engine ──► DLQ
                ├──► Workflow Engine (sagas + compensation)
                ├──► Replay Service (DLQ / time-range)
                ├──► PostgreSQL (durable state) + Redis (locks, idempotency)
                └──► Prometheus :9091 ──► Grafana :3000
Layer Components
Ingress API Gateway — REST :8080, gRPC :9090
Messaging Apache Kafka — partitioned topics + paired DLQ topics
Processing Consumer Workers — consumer groups, manual offset commits
Orchestration Workflow Engine — saga steps + LIFO compensation
Reliability Retry Engine — exponential backoff → DLQ routing
Recovery Replay Service — DLQ-only, time-range, partition replay
State PostgreSQL (durable) + Redis (locks, idempotency)
Observability Prometheus + Grafana dashboards
Additional diagrams
Diagram Description
Workflow Engine Saga state machine
Retry & DLQ Failure handling flow
Event Replay Replay sequence
Observability Metrics → dashboards pipeline

Re-render: .\scripts\render-diagrams.ps1

Core Features

Feature Description
Topic Administration Create, list, delete Kafka topics via REST/gRPC
Event Publishing Single + batch publish, idempotency keys, Snappy compression
Consumer Groups Partition assignment, offset commits, at-least-once delivery
Workflow Sagas Multi-step processes with compensating transactions
Retry Engine Exponential backoff, configurable max attempts
Dead Letter Queue {topic}-dlq with stats and inspection APIs
Event Replay DLQ-only, time-range, and partition replay
Observability 15+ Prometheus metrics, 2 Grafana dashboards
Cloud Deploy Helm chart + Terraform (EKS, MSK, RDS, ElastiCache)

Quick Start

Prerequisites

  • Docker Desktop (or Docker Engine + Compose v2)
  • Go 1.22+ (for local builds)
  • Make (optional)

Start the platform

# Start Kafka, PostgreSQL, Redis, all services, Prometheus, Grafana
make docker-up

# Or with Galactic Commerce demo overlay
docker compose -f docker/docker-compose.yml -f docker/docker-compose.demo.yml up -d --wait

# Verify
curl http://localhost:8080/healthz

Service endpoints

Service URL Purpose
REST API http://localhost:8080 Topics, events, workflows, replay
gRPC localhost:9090 Same operations via gRPC
Workflow Engine http://localhost:8081/metrics Saga metrics
Consumer Worker http://localhost:8082/metrics Consumer metrics
Prometheus http://localhost:9091 Metrics scrape UI
Grafana http://localhost:3000 Dashboards (admin / admin)

Example APIs

Publish an event

curl -s -X POST http://localhost:8080/api/v1/events \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "ship-orders",
    "eventType": "ShipPurchased",
    "idempotencyKey": "ship-001",
    "payload": {"pilotId": 42, "shipId": "falcon-x", "credits": 45000}
  }' | jq

Response:

{
  "id": "6c26f76b-dd6d-4b1b-9c2a-50a7af5a050f",
  "topic": "ship-orders",
  "partition": 3,
  "offset": 1,
  "eventType": "ShipPurchased",
  "publishedAt": "2026-06-09T23:44:02Z"
}

Create a topic

curl -s -X POST http://localhost:8080/api/v1/topics \
  -H "Content-Type: application/json" \
  -d '{"name":"ship-orders","partitions":6,"replicationFactor":1,"retentionHours":168}' | jq

Inspect consumer offsets

curl -s http://localhost:8080/api/v1/consumer-groups/galactic-commerce-workers/offsets | jq

Workflow Examples

GalacticCommerce saga

# Create workflow
WF=$(curl -s -X POST http://localhost:8080/api/v1/workflows \
  -H "Content-Type: application/json" \
  -d '{"name":"GalacticCommerce","input":{"pilotId":42,"shipId":"falcon-x","credits":45000}}' \
  | jq -r '.id')

# Run async
curl -s -X POST "http://localhost:8080/api/v1/workflows/$WF/run" | jq

# Poll status
curl -s "http://localhost:8080/api/v1/workflows/$WF" | jq '.workflow.status, .steps[].name, .steps[].status'

Steps: ProcessPayment → ReserveInventory → SendConfirmation

Inject saga failure (demo)

curl -s -X POST http://localhost:8080/api/v1/workflows \
  -H "Content-Type: application/json" \
  -d '{"name":"GalacticCommerce","input":{"pilotId":99,"demoFailStep":"ReserveInventory"}}' | jq

On failure, RefundPayment compensation runs (LIFO rollback).

Workflow state machine

Retry and DLQ Examples

# Publish a failing event
curl -s -X POST http://localhost:8080/api/v1/events \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "ship-orders",
    "eventType": "ShipPurchased",
    "idempotencyKey": "toxic-001",
    "payload": {"pilotId": 1, "simulateFailure": true}
  }' | jq -r '.id'

# Inspect retries
curl -s "http://localhost:8080/api/v1/retries?topic=ship-orders&eventId=<EVENT_ID>" | jq

# DLQ stats and messages
curl -s http://localhost:8080/api/v1/dlq/ship-orders/stats | jq
curl -s "http://localhost:8080/api/v1/dlq/ship-orders?limit=5" | jq

Retry and DLQ flow

Replay Examples

# Replay all unreplayed DLQ messages back to source topic
curl -s -X POST http://localhost:8080/api/v1/replay \
  -H "Content-Type: application/json" \
  -d '{"topic":"ship-orders","dlqOnly":true,"targetTopic":"ship-orders"}' | jq

# Replay by time range
curl -s -X POST http://localhost:8080/api/v1/replay \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "orders",
    "startTime": "2026-01-01T00:00:00Z",
    "endTime": "2026-06-01T00:00:00Z"
  }' | jq

Replay sequence

Observability

# Prometheus targets
open http://localhost:9091

# Grafana demo dashboard (after demo.ps1)
open http://localhost:3000/d/eventflow-demo/eventflow-demo-dashboard

Observability pipeline

Metric Description
eventflow_events_published_total Events published per topic
eventflow_events_processed_total Consumer throughput
eventflow_consumer_lag Partition lag
eventflow_dlq_messages_total DLQ insertions
eventflow_retry_attempts_total Retry scheduled / failed / dlq
eventflow_workflow_duration_seconds Saga step latency

Scaling and Reliability

Horizontal scale

Dimension Pattern
Publish throughput Increase topic partitions; batch publish API
Consume throughput Add consumer-worker replicas (≤ partition count)
API ingress Stateless api-gateway behind load balancer
Workflow execution Redis distributed locks prevent duplicate saga runs

Reliability guarantees

  • At-least-once delivery — producer acks + manual offset commits
  • Idempotency — Redis deduplication on idempotency keys (24h TTL)
  • Poison messages — max-retry → DLQ topic + PostgreSQL audit row
  • Partial failure — saga compensation (LIFO) instead of distributed 2PC
  • Operator recovery — replay API without code changes

Production hardening checklist

For real deployments beyond this reference architecture: mTLS/OAuth2, schema registry, rate limiting, SLO-based alerting, multi-AZ Kafka, and chaos testing. See docs/case-study.md for trade-off analysis.

Demo Screenshots


Event Publishing
Saga Failure
Retry Engine

DLQ Insertion
DLQ Replay
Grafana Dashboard

Regenerate assets: .\scripts\generate-screenshots.ps1

Demo Video

Record and publish a portfolio walkthrough:

  1. Run .\scripts\demo.ps1 (~80s)
  2. Follow docs/demo/video.md for narrative and OBS settings
  3. Upload to YouTube/Loom and embed:
[![EventFlow demo](docs/assets/hero-demo-preview.png)](https://www.youtube.com/watch?v=YOUR_VIDEO_ID)

Repository Structure

EventFlow/
├── cmd/                        # api-gateway, consumer-worker, workflow-engine, demo-generator
├── internal/                   # api, grpc, workflow, retry, replay, storage, topic
├── pkg/                        # config, kafka, metrics, models
├── api/                        # proto, gen/go, openapi
├── migrations/                 # PostgreSQL schema
├── docker/                     # Compose + Dockerfiles
├── deployments/                # k8s, monitoring/grafana
├── helm/eventflow/             # Helm chart
├── terraform/                  # AWS modules (EKS, MSK, RDS, ElastiCache)
├── tests/integration/          # Testcontainers suite
├── docs/
│   ├── diagrams/               # Architecture PNGs (Mermaid source)
│   ├── demo/screenshots/       # README demo captures
│   ├── case-study.md           # System design depth
│   ├── recruiter-guide.md      # Hiring manager overview
│   ├── interview-guide.md      # Interview prep
│   └── resume-snippets.md      # ATS bullet points
└── scripts/                    # demo.ps1, render-diagrams.ps1, generate-screenshots.ps1

Design Decisions

Decision Rationale
Kafka as event log Durable, ordered-per-partition, replayable
PostgreSQL for state ACID workflow/retry/DLQ records, queryable by operators
Redis for locks Prevent duplicate workflow execution across replicas
At-least-once delivery Simpler than exactly-once; idempotency keys compensate
Embedded retry/replay Fewer moving parts; invoked from gateway and consumer
Saga over 2PC Availability and partition tolerance in distributed deploys
Convention-based DLQ {topic}-dlq mirrors industry patterns (SQS, Service Bus)

Full analysis: docs/case-study.md

Resume Bullet Points

  • Built EventFlow, a distributed event platform in Go with Kafka, saga workflows, exponential retry/DLQ, and replay across 16 REST endpoints and gRPC
  • Engineered at-least-once delivery with consumer offset tracking, idempotency keys, and Redis deduplication
  • Deployed via Docker Compose, Kubernetes/Helm, and Terraform (AWS EKS, MSK, RDS, ElastiCache)

More versions: docs/resume-snippets.md (Backend · Platform · Infrastructure)

Build and Test

make build              # Build all services
make test               # Unit tests
make test-integration   # Testcontainers (Kafka, Postgres, Redis)
make proto              # Generate gRPC stubs
make helm-install       # Deploy Helm chart

Documentation

Document Audience
Case Study Engineers — system design depth
Recruiter Guide Hiring managers — plain language
Interview Guide Candidates — technical interview prep
Resume Snippets Portfolio — ATS bullet points
Project Metrics Portfolio stats
Demo Script Live presentation
Architecture Technical reference
Deployment Ops runbook
CHANGELOG Release notes

Release Notes

See CHANGELOG.md for the full history. Latest stable: v1.0.0 — portfolio release with README, diagrams, recruiter materials, live demo, and contribution guide.

License

MIT — see LICENSE.

EventFlow v1.0.0 — If this looks like a real distributed systems platform, star the repo.

About

Distributed event processing platform built with Go, Kafka, PostgreSQL, and Redis featuring workflow orchestration, retries, DLQs, event replay, and cloud-native deployment.

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages