Deliver Data API billing events to S3 for downstream billing

[Hazel-Datastax]

The Data API emits per-call billing events (BillingEvent JSON) on the billing.events logger via DefaultBilling.emitEvent(). We need to publish these events to the downstream's S3 buckets.

This is a parent tracking ticket, add sub tickets for the different work needed.

Approach

Deliver via a custom log handler attached to the billing.events logger: it batches the logged JSON lines and writes .jsonl objects to S3 (AWS SDK v2). The handler is attached at startup only when stargate.jsonapi.billing.s3.enabled=true (off by default). Because the destination is a config choice, on-prem deployments route billing.events to their own stack via log config with no code change.

Object contract

• Buckets: serverless-usage-{dev,test,prod}
• Path: s3://<bucket>/<event_type>/<YYYY>/<MM>/<DD>/<HH>/<MM>/<uuid>.jsonl
• Format: NDJSON (one event per line), uncompressed .jsonl, ≤ ~2MB / ~50 events per file
• Dedup: downstream dedups on id; reuse the object key on retry
• Auth: IRSA (cross-account) on AWS; GCP/Azure via OIDC federation

Constraints

• On/off is the config billing.s3.enabled + a heartbeat metric/alert — not the log level.
• The handler uses its own bounded buffer; no silent drop (drop → metric).
• Idempotent: one object key per sealed batch, reused on retry; downstream dedups on id.
• Delivery requires billing-events-logging enabled (events reach the logger) AND billing.s3.enabled=true.

Reliability

• Async (never block the request path) + retry/backoff + never silently drop (Micrometer metric) + graceful-shutdown flush + dual-write reconciliation.
• Disk spool / PVC are deferred — to be sized against an explicit acceptable-loss budget.