From 38b14bb989e4abd4d21216361929004a8b361047 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 09:18:03 -0600 Subject: [PATCH 1/2] docs(analytics): document transaction-commit-time metric Add a Storage Metrics section documenting transaction-commit-time (write commit submit->durable duration distribution) and how its upper percentiles serve as the leading indicator for the storage overload (503) rejection. Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/analytics/overview.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6..d9549cc4 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -169,6 +169,25 @@ Harper automatically tracks the following metrics for all services. Applications | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | | `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | +### Storage Metrics + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | + +`transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, +`p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard +uses: when the oldest outstanding commit exceeds `storage.maxTransactionQueueTime` (default 45s), +Harper rejects new write transactions with `Outstanding write transactions have too long of queue, +please try again later` (HTTP 503). + +Because it is measured on that same clock, it is the leading indicator for that rejection: a rising +`p99`/`p999` means commits are taking longer to drain — from write volume, large transactions, or a +saturated storage volume — and is the signal to shed or throttle write load before commits start +timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared +write path rather than per-thread submission concurrency, so alert on the upper percentiles trending +toward `maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. + ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | From 340817646b104543dda30f97f25a428e65f93eb0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 10:29:14 -0600 Subject: [PATCH 2/2] fix(analytics): satisfy prettier table formatting --- reference/analytics/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index d9549cc4..69e2bb29 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,8 +171,8 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------- | | `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`,