feat: Infrastructure optimization for millions of TPS (all 14 services)

[devin-ai-integration]

Summary

Production-grade high-throughput tuning for all 14 infrastructure services, plus polyglot transaction processing engines in Go, Rust, and Python. Target: millions of transactions per second.

Supersedes #34 (closed due to merge conflict).

Mojaloop — MySQL, NOT PostgreSQL

Mojaloop's Knex.js migrations use MySQL-specific syntax (AUTO_INCREMENT, ENUM, ON DUPLICATE KEY UPDATE). Postgres is not supported without forking. Strategy: aggressive MySQL tuning + ProxySQL pooling.

# mysql-production.cnf (key settings)
innodb_buffer_pool_size = 96G          # 75% of 128GB
innodb_flush_log_at_trx_commit = 2     # 10x throughput (async commit)
innodb_io_capacity = 20000             # NVMe baseline

# proxysql.cnf
max_connections = 20000                # Pool down to 500 backend
query rules: SELECT → replica, writes → primary

Infrastructure configs (35 files, +4,359 lines)

Service	Key Optimization	Target
PostgreSQL	64GB shared_buffers, 32 parallel workers, zstd WAL, PgBouncer 20K→500	1M+ queries/s
Kafka	5-broker KRaft, zstd, 24 partitions, 10K batch	2M+ msg/s
Redis	8 I/O threads, 48GB, LFU eviction, pipeline	5M+ ops/s
TigerBeetle	6-node cluster, 8GB grid cache, 8190 batch	10M+ transfers/s
APISIX	auto workers, 65K connections, upstream keepalive 320	1M+ req/s
OpenSearch	30s refresh, async translog, hot-warm-cold ISM	500K+ docs/s
Temporal	4096 history shards, 16 task queue partitions, 30K persistence QPS	10K+ workflows/s
Mojaloop	ProxySQL R/W split, 4 central-ledger replicas, batch settlement	100K+ transfers/s
Fluvio	16MB batch, zstd, 12 partitions	1M+ events/s
Dapr	gRPC, bulk pub/sub, 1% sampling, 10K batch producer	N/A (middleware)
Permify	200 DB connections, 4 replicas, HPA→20, circuit breaker	100K+ checks/s
Lakehouse	Bronze/Silver/Gold medallion, BRIN indexes, COPY ingestion	100K+ rows/s

Polyglot high-performance engines

Go (services/go/high-perf-tx-engine/): Goroutine pool with sync.Pool zero-allocation, BatchAccumulator (8190/batch, 10ms flush), circuit breakers per downstream.

Rust (services/rust/high-perf-tx-engine/): Tokio + crossbeam lock-free channels, DashMap idempotency cache, atomic metrics. services/rust/tigerbeetle-batch-client/: DoubleEntryBuilder for GL transfers with fee/commission legs.

Python (services/python/high-perf-analytics/): uvloop + httptools, async BatchProcessor with configurable flush, AggregationEngine with percentile computation, LakehouseOptimizer with Bronze→Silver ETL and Gold materialized views.

Bug fix

• Fixed pre-existing missing github.com/lib/pq dependency in services/go/mojaloop-connector-pos/go.mod

Kernel tuning

infra/kernel-tuning.sh: TCP BBR, 65K backlog, NVMe scheduler, huge pages for PostgreSQL, THP disabled, io_uring AIO limit.

Test Results (12/12 passed)

All YAML/TOML/JSON configs parse, Go/Rust/Python code compiles, 8 key tuning values verified, TypeScript integrity confirmed (8 pre-existing errors only), test suite 4,286 passed with 6 pre-existing failures.

Link to Devin session: https://app.devin.ai/sessions/3ebd42bf0430422a9a2bd85ed9f9cd4c
Requested by: @munisp

[devin-ai-integration]

Original prompt from Patrick

https://drive.google.com/file/d/1ko3y7OBp1tJIXGTbe2QGFRHMQfxMTWHX/view?usp=sharing

1. Extract ALL everything in the archive
2. how do ensure and assess that features for example domain and business logic/rules/requirements are fully impemented and production ready and complete - can you thoroughly assess each files and features to determine there are ready for production

1. Database integration (replace in-memory with real Postgres)
2. Inter-service HTTP wiring with retries/circuit breakers
3. Security hardening (JWT everywhere, remove hardcoded creds, mTLS)
4. Integration tests for critical flows
5. Graceful shutdown, observability, alerting
   3)search for orphan, partially and generic scaffolded features across the platform - fully implement them end to end -generic CRUD-only patterns , modules with no domain logic, disconnected features, and incomplete implementations.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring