catalog-storage (GenericCatalog)

A high-throughput, schema-flexible resource catalog service in Go — designed to store and serve arbitrary business entities (products, policies, vehicles, contracts, …) behind one gRPC API, without one-table-per-type schema sprawl.

                    ┌──────────────┐
   gRPC writes ───▶ │   catalog    │──▶ sharded write-behind buffer
                    └──────┬───────┘            │ batched
                           │ reads              ▼
                    ┌──────▼───────┐    ┌──────────────┐     ┌────────────────┐
                    │ Redis (state │    │ Redpanda /   │ ──▶ │ ingest-worker  │
                    │ + tiered     │    │ Kafka        │     │ (1/partition)  │
                    │ storage)     │    └──────────────┘     └───────┬────────┘
                    └──────────────┘                                 │ COPY
                                                              ┌──────▼────────┐
                                                              │ PostgreSQL    │
                                                              │ (partitioned) │
                                                              └───────────────┘

Core ideas

• Generic resources + extension tables — a stable base schema (resources, resource_links) with per-domain extension tables (pricing, vehicle, insurance, …) instead of EAV or JSON-only blobs.
• Field scoping — queries declare which field groups they need; the service only joins/hydrates those extensions.
• Hierarchical bundles — composable resource bundles with slug-based routing (112-policy), bundle components, and progressive field filtering.
• Write-behind ingestion — writes are acknowledged from an in-memory sharded buffer, drained to Redpanda/Kafka, and bulk-COPYed into partition-aware PostgreSQL by per-partition ingest workers.
• Closure tables — materialized hierarchy paths maintained by a dedicated closure-worker.

Components

Path	What it is
code/cmd/catalog	gRPC API server (write-behind buffer, field scoping)
code/cmd/ingest-worker	Kafka consumer; partition-aware bulk COPY into PostgreSQL
code/cmd/closure-worker	Maintains closure tables for hierarchies
code/cmd/load-producer, grpc-loadtest, …	Load-generation and benchmark tools
proto/	Protobuf/gRPC API definitions (catalog, bundles, resources, events, queue)
migrations/	PostgreSQL schema (partitioning, event sourcing, monitoring views)
deploy/	docker-compose stack, PgBouncer, Grafana/OTel observability, k6 suites
k8s/	Kubernetes manifests

Quick start

# Infrastructure (PostgreSQL, PgBouncer, Redis, Redpanda)
cd deploy && docker compose up -d

# Configuration
cd ../code && cp .env.example .env

# Build & run
go build ./...
go run ./cmd/catalog          # gRPC :9090, HTTP :8080
go run ./cmd/ingest-worker    # in a second shell

Protobuf regeneration: make proto-gen (see make help).

Performance work

Ingestion throughput was tuned iteratively — buffer sharding vs. lock-free ring buffers, batch sizing, per-partition COPY strategies, PgBouncer pooling, consumer batching. k6 load-test suites live in deploy/k6/.

License

MIT — see LICENSE.