Kafkaesque

Repository for learning Kafka, event streaming and event-driven architectural patterns.

Architecture

graph LR
    subgraph InfraTools["Infrastructure Tools"]
        TopicConfig[kafka-topics.yml]
        TopicRegTool[Topic Register Tool]
        SchemaFiles[Avro Schemas]
        SchemaGen[Schema Generator]
        SchemaRegTool[Schema Register Tool]
        
        TopicConfig --> TopicRegTool
        SchemaFiles --> SchemaGen
        SchemaGen --> |Generates C# Classes| SchemaFiles
    end

    subgraph Client
        HTTP[HTTP Client]
    end

    subgraph OrderService["Order Service (Producer)"]
        API[REST API]
        OrderSvc[Order Service]
        OrderRepo[Order Repository]
        OutboxRepo[Outbox Repository]
        DB[(PostgreSQL<br/>orders DB)]
        OutboxWorker[Outbox Worker<br/>Background Service]
        
        API --> OrderSvc
        OrderSvc --> OrderRepo
        OrderSvc --> OutboxRepo
        OrderRepo --> DB
        OutboxRepo --> DB
        OutboxWorker --> OutboxRepo
    end

    subgraph KafkaInfra["Kafka Infrastructure"]
        SchemaReg[Schema Registry]
        Broker[Kafka Broker]
        Topic[Topic: order.placed]
        
        Broker --> Topic
    end

    subgraph FulfillmentService["Fulfillment Service (Consumer)"]
        Consumer[Kafka Consumer]
        FulfillSvc[Fulfillment Service<br/>Background Service]
        
        Consumer --> FulfillSvc
    end

    TopicRegTool --> |Creates Topics| Broker
    SchemaRegTool --> |Registers Schemas| SchemaReg
    HTTP -->|POST /orders| API
    OutboxWorker -->|Publishes Events| Broker
    OutboxWorker -.->|Validates Schema| SchemaReg
    Topic -->|Consumes Events| Consumer
    Consumer -.->|Validates Schema| SchemaReg

  

Loading

Key Components:

• Order Service: REST API that creates orders and stores them in PostgreSQL with an outbox pattern for reliable event publishing
• Outbox Worker: Background service that polls the outbox table and publishes events to Kafka
• Kafka Infrastructure: Message broker with Schema Registry for Avro schema validation
• Fulfillment Service: Background consumer that processes order events from Kafka
• Infrastructure Tools:
  • Topic Register: Automatically registers Kafka topics from configuration on startup
  • Schema Register: Registers Avro schemas with the Schema Registry
  • Schema Generator: Generates C# classes from Avro schema definitions

Flink Stream Analytics

Real-time event processing using Apache Flink. See flink-job-submitter/README.md for full documentation.

Components

• flink-job-manager: Coordinates job execution, checkpointing, provides Web UI
• flink-task-manager: Executes stream processing operators
• flink-job-submitter: Generic job submitter (short-lived container that exits after submission)

Building

cd flink-job-submitter
mvn clean package

Running

docker-compose up -d flink-job-submitter

Getting Started

Prerequisites

• Docker and Docker Compose
• .NET 9 (with tools installed via dotnet tool restore)
• Java 17
• Maven 3.9+

Running services

• Run the following bash script to start up the infrastructure:

  ./run.sh

• When making changes, we can rebuild and run a specific container by supplying it as an argument to the run script

  ./run.sh order-service # fulfillment-service, schema-register, topic-register etc

Managing topics

Topics are automatically created on startup via the topic-register service, which reads from kafka-topics.yml.

Adding new topics:

1. Edit kafka-topics.yml to add your topic definition
2. Restart the infrastructure: ./run.sh

Topics can also be managed manually through the Kafka Control Center at http://localhost:9021.

Updating schemas

See the following guidance for creating or updating schemas