Grafana dashboard for prerender_html jobs, cross-linked with Indexing#5438
Merged
habdelra merged 4 commits intoJul 9, 2026
Merged
Conversation
…810) Mirror the Indexing dashboard for prerender_html jobs: pending/in-flight/ oldest-pending stats, an outcome row (completed / failed / retried / publishes-coalesced), throughput and stock timeseries, active jobs with job_progress gauges, per-realm status, queued/completed tables, and a longest-jobs pair split by priority tier. Job tables carry generation and spawned_by columns; spawned_by pivots to the Indexing dashboard's new job-id filter, and the Indexing dashboard's Active/Completed job ids pivot back through the prerender dashboard's spawning-job filter. The prerender coalesce merge now counts folded-in publishes (coalescedPublishes) so the dashboard can report coalescing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Observability diff (vs staging)Diff truncated (77229 bytes; limit 60000). Full diff: https://github.com/cardstack/boxel/actions/runs/28982400792 Show diffdiff --git a/tmp/remote-canon.bHtid9/dashboards/boxel-status/indexing.json b/tmp/committed-canon.SqeM7r/dashboards/boxel-status/indexing.json
index a39cf75..fb727fb 100644
--- a/tmp/remote-canon.bHtid9/dashboards/boxel-status/indexing.json
+++ b/tmp/committed-canon.SqeM7r/dashboards/boxel-status/indexing.json
@@ -39,6 +39,17 @@
"title": "Realms",
"type": "dashboards"
},
+ {
+ "asDropdown": false,
+ "icon": "external link",
+ "includeVars": false,
+ "keepTime": true,
+ "tags": [
+ "workflow:prerender-html"
+ ],
+ "title": "Prerender HTML",
+ "type": "dashboards"
+ },
{
"asDropdown": false,
"icon": "external link",
@@ -69,6 +80,10 @@
"uid": "cef5v5sl9k7i8f"
},
"description": "System-wide operator action: queue a full reindex across every realm. The button disables itself while a `full-reindex` orchestration job is already pending or running. Per-realm reindex moved to the Realms dashboard. Click POSTs with `Authorization: Bearer ${grafana_secret}` (substituted from SSM at apply time, CS-10929).",
+ "fieldConfig": {
+ "defaults": {},
+ "overrides": []
+ },
"gridPos": {
"h": 8,
"w": 24,
@@ -589,7 +604,7 @@
"type": "grafana-postgresql-datasource",
"uid": "cef5v5sl9k7i8f"
},
- "description": "One row per indexing job currently held by a worker (across all realm-server / worker tasks). `progress` and `current files` come from `job_progress` (CS-10930), populated by each realm-server's IndexingEventSink write-through. Click the realm cell to drill into the live activity feed.",
+ "description": "One row per indexing job currently held by a worker (across all realm-server / worker tasks). `progress` and `current files` come from `job_progress` (CS-10930), populated by each realm-server's IndexingEventSink write-through. Click the realm cell to drill into the live activity feed; the job_id cell pivots to the Prerender HTML dashboard filtered to the prerender_html job this pass spawned.",
"fieldConfig": {
"defaults": {
"color": {
@@ -786,6 +801,24 @@
}
]
},
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "job_id"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View spawned prerender job",
+ "url": "/d/b5eou6wleg7i6o?${__url_time_range}&var-spawning_job_id=${__data.fields.job_id}&orgId=1"
+ }
+ ]
+ }
+ ]
+ },
{
"matcher": {
"id": "byName",
@@ -852,7 +885,7 @@
"editorMode": "code",
"format": "table",
"rawQuery": true,
- "rawSql": "SELECT\n job_id,\n realm,\n realm_url,\n job_type,\n files_completed,\n total_files,\n percent,\n percent AS pct,\n elapsed_seconds,\n started_at,\n worker_id,\n reservation_id\nFROM (\n SELECT\n j.id AS job_id,\n COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n ) AS realm,\n COALESCE(j.args->>'realmURL','') AS realm_url,\n j.job_type,\n COALESCE(jp.files_completed, 0) AS files_completed,\n COALESCE(jp.total_files, 0) AS total_files,\n CASE WHEN COALESCE(jp.total_files, 0) > 0\n THEN (jp.files_completed::float / jp.total_files) * 100\n ELSE 0\n END AS percent,\n EXTRACT(EPOCH FROM (NOW() - jr.created_at)) AS elapsed_seconds,\n jr.created_at AS started_at,\n jr.worker_id,\n jr.id AS reservation_id\n FROM jobs j\n JOIN job_reservations jr ON jr.job_id = j.id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n LEFT JOIN job_progress jp ON jp.job_id = j.id\n WHERE j.job_type IN ('from-scratch-index','incremental-index')\n AND j.finished_at IS NULL\n) active\nORDER BY started_at DESC;",
+ "rawSql": "SELECT\n job_id,\n realm,\n realm_url,\n job_type,\n files_completed,\n total_files,\n percent,\n percent AS pct,\n elapsed_seconds,\n started_at,\n worker_id,\n reservation_id\nFROM (\n SELECT\n j.id AS job_id,\n COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n ) AS realm,\n COALESCE(j.args->>'realmURL','') AS realm_url,\n j.job_type,\n COALESCE(jp.files_completed, 0) AS files_completed,\n COALESCE(jp.total_files, 0) AS total_files,\n CASE WHEN COALESCE(jp.total_files, 0) > 0\n THEN (jp.files_completed::float / jp.total_files) * 100\n ELSE 0\n END AS percent,\n EXTRACT(EPOCH FROM (NOW() - jr.created_at)) AS elapsed_seconds,\n jr.created_at AS started_at,\n jr.worker_id,\n jr.id AS reservation_id\n FROM jobs j\n JOIN job_reservations jr ON jr.job_id = j.id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n LEFT JOIN job_progress jp ON jp.job_id = j.id\n WHERE j.job_type IN ('from-scratch-index','incremental-index')\n AND j.finished_at IS NULL\n AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\n) active\nORDER BY started_at DESC;",
"refId": "A"
}
],
@@ -996,7 +1029,7 @@
"type": "grafana-postgresql-datasource",
"uid": "cef5v5sl9k7i8f"
},
- "description": "Indexing jobs (from-scratch-index, incremental-index) waiting on a worker. Same query as Job Queue → Waiting Jobs, narrowed to indexing job types.",
+ "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) waiting on a worker — no live reservation holds them, so jobs whose lease expired and are claimable again are included.",
"fieldConfig": {
"defaults": {
"color": {
@@ -1120,7 +1153,7 @@
"editorMode": "code",
"format": "table",
"rawQuery": true,
- "rawSql": "SELECT \n j.id, \n j.priority, \n j.job_type, \n CASE\n WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n ELSE COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n )\n END AS realm,\n CASE \n WHEN j.concurrency_group ~ '^indexing:https://[^/]+/.+' THEN REGEXP_REPLACE(j.concurrency_group, '^indexing:https://[^/]+/', '') \n WHEN j.concurrency_group ~ '^indexing:https://[^/]+/?$' THEN REGEXP_REPLACE(j.concurrency_group, '^indexing:https://', '') \n ELSE j.concurrency_group \n END AS concurrency_group, \n j.status AS status, \n j.created_at AS created_at, \n\n\n -- Wait time in seconds\n CASE \n WHEN jr.created_at IS NOT NULL \n THEN EXTRACT(EPOCH FROM (jr.created_at - j.created_at))\n ELSE \n EXTRACT(EPOCH FROM (NOW() - j.created_at))\n END\n AS wait_seconds,\n j.id as job_id\n\nFROM \n jobs j\n \nLEFT JOIN \n job_reservations jr ON j.id = jr.job_id\n\nWHERE\njr.job_id IS NULL AND j.status = 'unfulfilled' AND j.job_type IN ('from-scratch-index','incremental-index','full-reindex') \n \nORDER BY \n j.created_at ASC\nLIMIT 500;",
+ "rawSql": "SELECT \n j.id, \n j.priority, \n j.job_type, \n CASE\n WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n ELSE COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n )\n END AS realm,\n CASE \n WHEN j.concurrency_group ~ '^indexing:https://[^/]+/.+' THEN REGEXP_REPLACE(j.concurrency_group, '^indexing:https://[^/]+/', '') \n WHEN j.concurrency_group ~ '^indexing:https://[^/]+/?$' THEN REGEXP_REPLACE(j.concurrency_group, '^indexing:https://', '') \n ELSE j.concurrency_group \n END AS concurrency_group, \n j.status AS status, \n j.created_at AS created_at, \n\n\n -- Wait time in seconds\n CASE \n WHEN jr.created_at IS NOT NULL \n THEN EXTRACT(EPOCH FROM (jr.created_at - j.created_at))\n ELSE \n EXTRACT(EPOCH FROM (NOW() - j.created_at))\n END\n AS wait_seconds,\n j.id as job_id\n\nFROM \n jobs j\n \nLEFT JOIN \n job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n\nWHERE\njr.id IS NULL AND j.status = 'unfulfilled' AND j.job_type IN ('from-scratch-index','incremental-index','full-reindex') AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\n \nORDER BY\n j.created_at ASC\nLIMIT 500;",
"refId": "A",
"sql": {
"columns": [
@@ -1149,7 +1182,7 @@
"type": "grafana-postgresql-datasource",
"uid": "cef5v5sl9k7i8f"
},
- "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
+ "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). `generation` is the realm index generation the pass committed (NULL for full-reindex orchestration and results from builds that predate generation stamping). The id cell pivots to the Prerender HTML dashboard filtered to the prerender_html job this pass spawned. Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
"fieldConfig": {
"defaults": {
"color": {
@@ -1179,6 +1212,24 @@
}
},
"overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "id"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View spawned prerender job",
+ "url": "/d/b5eou6wleg7i6o?${__url_time_range}&var-spawning_job_id=${__data.fields.id}&orgId=1"
+ }
+ ]
+ }
+ ]
+ },
{
"matcher": {
"id": "byName",
@@ -1286,7 +1337,7 @@
"editorMode": "code",
"format": "table",
"rawQuery": true,
- "rawSql": "SELECT\n j.id,\n lr.reservation_id,\n j.job_type,\n CASE\n WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n ELSE COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n )\n END AS realm,\n j.status,\n j.created_at,\n lr.started_at,\n j.finished_at,\n EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n ORDER BY jr.created_at DESC\n LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n SELECT MIN(jr.created_at) AS first_started_at\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n) fr ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n AND j.finished_at IS NOT NULL\n AND j.finished_at > NOW() - INTERVAL '24 hours'\nORDER BY j.finished_at DESC\nLIMIT 500;",
+ "rawSql": "SELECT\n j.id,\n lr.reservation_id,\n j.job_type,\n CASE\n WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n ELSE COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n )\n END AS realm,\n j.status,\n j.created_at,\n lr.started_at,\n j.finished_at,\n EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n (j.result->>'generation')::bigint AS generation,\n lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n ORDER BY jr.created_at DESC\n LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n SELECT MIN(jr.created_at) AS first_started_at\n FROM job_reservations jr\n WHERE jr.job_id = j.id\n) fr ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n AND j.finished_at IS NOT NULL\n AND j.finished_at > NOW() - INTERVAL '24 hours'\n AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\nORDER BY j.finished_at DESC\nLIMIT 500;",
"refId": "A"
}
],
@@ -1607,6 +1658,21 @@
"query": "REDACTED",
"skipUrlSync": true,
"type": "constant"
+ },
+ {
+ "current": {
+ "selected": false,
+ "text": "",
+ "value": ""
+ },
+ "description": "Filter the job tables to a single indexing job id. Prefilled by the 'View originating index job' drill-through on the Prerender HTML dashboard. Empty = no filter.",
+ "hide": 0,
+ "label": "Job id filter",
+ "name": "job_id",
+ "options": [],
+ "query": "",
+ "skipUrlSync": false,
+ "type": "textbox"
}
]
},
diff --git a/tmp/committed-canon.SqeM7r/dashboards/boxel-status/prerender-html.json b/tmp/committed-canon.SqeM7r/dashboards/boxel-status/prerender-html.json
new file mode 100644
index 0000000..213deeb
--- /dev/null
+++ b/tmp/committed-canon.SqeM7r/dashboards/boxel-status/prerender-html.json
@@ -0,0 +1,1789 @@
+{
+ "apiVersion": "dashboard.grafana.app/v1beta1",
+ "kind": "Dashboard",
+ "metadata": {
+ "annotations": {
+ "grafana.app/folder": "defd2d156sav4d"
+ },
+ "name": "b5eou6wleg7i6o"
+ },
+ "spec": {
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": {
+ "type": "grafana",
+ "uid": "-- Grafana --"
+ },
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "fiscalYearStartMonth": 0,
+ "graphTooltip": 0,
+ "links": [
+ {
+ "asDropdown": false,
+ "icon": "external link",
+ "includeVars": false,
+ "keepTime": true,
+ "tags": [
+ "workflow:indexing"
+ ],
+ "title": "Indexing",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": false,
+ "icon": "external link",
+ "includeVars": false,
+ "keepTime": true,
+ "tags": [
+ "entity:realm"
+ ],
+ "title": "Realms",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": false,
+ "icon": "external link",
+ "includeVars": false,
+ "keepTime": true,
+ "tags": [
+ "workflow:queue"
+ ],
+ "title": "Job Queue",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": false,
+ "icon": "external link",
+ "includeVars": false,
+ "keepTime": true,
+ "tags": [
+ "forensics"
+ ],
+ "title": "Logs",
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs waiting for a worker. Reconciles with `SELECT count(*) FROM jobs WHERE status='unfulfilled' AND job_type = 'prerender_html'` minus those with an active reservation.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 50
+ },
+ {
+ "color": "red",
+ "value": 200
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 8,
+ "x": 0,
+ "y": 0
+ },
+ "id": 1,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COUNT(*) AS pending\n FROM jobs j\n LEFT JOIN job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n WHERE j.status = 'unfulfilled'\n AND j.job_type = 'prerender_html'\n AND jr.id IS NULL;",
+ "refId": "A"
+ }
+ ],
+ "title": "Pending Prerender Jobs",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs currently held by a worker (live reservation, finished_at NULL).",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "blue"
+ },
+ {
+ "color": "green",
+ "value": 1
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 8,
+ "x": 8,
+ "y": 0
+ },
+ "id": 2,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COUNT(*) AS in_flight\n FROM jobs j\n JOIN job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n WHERE j.finished_at IS NULL\n AND j.job_type = 'prerender_html';",
+ "refId": "A"
+ }
+ ],
+ "title": "In-flight Prerender Jobs",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "Age of the oldest pending prerender_html job. Red after 5 minutes. System-tier HTML jobs run at priority 0 and are only claimed by workers whose priority floor is 0, so sustained values here usually mean that pool is saturated or stuck.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 60
+ },
+ {
+ "color": "red",
+ "value": 300
+ }
+ ]
+ },
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 8,
+ "x": 16,
+ "y": 0
+ },
+ "id": 3,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT EXTRACT(EPOCH FROM (NOW() - MIN(j.created_at))) AS oldest_pending_seconds\n FROM jobs j\n LEFT JOIN job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n WHERE j.status = 'unfulfilled'\n AND j.job_type = 'prerender_html'\n AND jr.id IS NULL;",
+ "refId": "A"
+ }
+ ],
+ "title": "Oldest Pending",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs that resolved in the dashboard's time range.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 0,
+ "y": 4
+ },
+ "id": 4,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COUNT(*) AS completed\n FROM jobs\n WHERE job_type = 'prerender_html'\n AND status = 'resolved'\n AND $__timeFilter(finished_at);",
+ "refId": "A"
+ }
+ ],
+ "title": "Completed (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs rejected (failed all retries) in the dashboard's time range. A rejected job leaves the affected cards serving their previous HTML; a later index pass for the realm re-publishes the work.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 1
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 6,
+ "y": 4
+ },
+ "id": 5,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COUNT(*) AS failed\n FROM jobs\n WHERE job_type = 'prerender_html'\n AND status = 'rejected'\n AND $__timeFilter(finished_at);",
+ "refId": "A"
+ }
+ ],
+ "title": "Failed (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs claimed more than once in the dashboard's time range — a lease expired or a worker died mid-run, and the queue re-issued the job.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 1
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 12,
+ "y": 4
+ },
+ "id": 6,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COUNT(*) AS retried\n FROM (\n SELECT j.id\n FROM jobs j\n JOIN job_reservations jr ON jr.job_id = j.id\n WHERE j.job_type = 'prerender_html'\n AND $__timeFilter(jr.created_at)\n GROUP BY j.id\n HAVING COUNT(*) > 1\n ) multi;",
+ "refId": "A"
+ }
+ ],
+ "title": "Retried Jobs (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "Publishes merged into an already-pending prerender_html job for the same realm (per-URL update-wins merge), summed from the surviving jobs' `coalescedPublishes` arg over jobs created in the dashboard's time range. Piggybacks on an in-flight job aren't counted — a running worker holds its args in memory, so the queue records nothing for them.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 18,
+ "y": 4
+ },
+ "id": 7,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT COALESCE(SUM((j.args->>'coalescedPublishes')::int), 0) AS coalesced\n FROM jobs j\n WHERE j.job_type = 'prerender_html'\n AND $__timeFilter(j.created_at);",
+ "refId": "A"
+ }
+ ],
+ "title": "Publishes Coalesced (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html job flow rates. Three series — `arrived` (jobs queued), `started` (a worker reserved them), `completed` (jobs.finished_at set). Bucketed by 30s. A publish that coalesces into a pending job updates it in place rather than creating a new row, so merged publishes don't show as arrivals — see 'Publishes Coalesced'. A persistent gap between arrived and completed indicates worker saturation.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisGridShow": true,
+ "axisLabel": "jobs / 30s",
+ "axisPlacement": "auto",
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "lineInterpolation": "smooth",
+ "lineWidth": 1,
+ "pointSize": 4,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": true,
+ "stacking": {
+ "mode": "none"
+ }
+ },
+ "mappings": [],
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 8
+ },
+ "id": 8,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "lastNotNull"
+ ],
+ "displayMode": "list",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "none"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "time_series",
+ "rawQuery": true,
+ "rawSql": "SELECT $__timeGroupAlias(j.created_at, '30s'),\n COUNT(*) AS arrived\n FROM jobs j\n WHERE j.job_type = 'prerender_html'\n AND $__timeFilter(j.created_at)\n GROUP BY 1\n ORDER BY 1;",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "time_series",
+ "rawQuery": true,
+ "rawSql": "SELECT $__timeGroupAlias(jr.created_at, '30s'),\n COUNT(*) AS started\n FROM job_reservations jr\n JOIN jobs j ON j.id = jr.job_id\n WHERE j.job_type = 'prerender_html'\n AND $__timeFilter(jr.created_at)\n GROUP BY 1\n ORDER BY 1;",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "time_series",
+ "rawQuery": true,
+ "rawSql": "SELECT $__timeGroupAlias(j.finished_at, '30s'),\n COUNT(*) AS completed\n FROM jobs j\n WHERE j.job_type = 'prerender_html'\n AND j.finished_at IS NOT NULL\n AND $__timeFilter(j.finished_at)\n GROUP BY 1\n ORDER BY 1;",
+ "refId": "C"
+ }
+ ],
+ "title": "Prerender throughput (arrived / started / completed)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "Stocks of prerender_html jobs over time. `pending` = queued, no live reservation; `in_flight` = a worker has an open reservation; `completed` = cumulative finished. Bucketed at 1 minute over the panel's time range. Cost: O(buckets × prerender-jobs in window) per refresh — if this gets slow, switch to a snapshot/materialized table.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisGridShow": true,
+ "axisLabel": "jobs",
+ "axisPlacement": "auto",
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "lineInterpolation": "smooth",
+ "lineWidth": 1,
+ "pointSize": 4,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": true,
+ "stacking": {
+ "mode": "none"
+ }
+ },
+ "mappings": [],
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 16
+ },
+ "id": 9,
+ "options": {
+ "legend": {
+ "calcs": [
+ "max",
+ "lastNotNull"
+ ],
+ "displayMode": "list",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "none"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "time_series",
+ "rawQuery": true,
+ "rawSql": "WITH buckets AS (\n SELECT generate_series($__timeFrom()::timestamptz, $__timeTo()::timestamptz, '1 minute') AS bucket\n),\nprerender_jobs AS (\n SELECT j.id, j.created_at, j.finished_at,\n (SELECT MIN(jr.created_at) FROM job_reservations jr WHERE jr.job_id = j.id) AS first_started_at\n FROM jobs j\n WHERE j.job_type = 'prerender_html'\n AND j.created_at <= $__timeTo()::timestamptz\n)\nSELECT b.bucket AS time,\n COUNT(*) FILTER (WHERE pj.created_at <= b.bucket\n AND (pj.first_started_at IS NULL OR pj.first_started_at > b.bucket)\n AND (pj.finished_at IS NULL OR pj.finished_at > b.bucket)) AS pending,\n COUNT(*) FILTER (WHERE pj.first_started_at IS NOT NULL\n AND pj.first_started_at <= b.bucket\n AND (pj.finished_at IS NULL OR pj.finished_at > b.bucket)) AS in_flight,\n COUNT(*) FILTER (WHERE pj.finished_at IS NOT NULL AND pj.finished_at <= b.bucket) AS completed\n FROM buckets b LEFT JOIN prerender_jobs pj ON TRUE\n GROUP BY b.bucket\n ORDER BY b.bucket;",
+ "refId": "A"
+ }
+ ],
+ "title": "Pending vs in-flight vs completed (over time)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "One row per prerender_html job currently held by a worker. `progress` and file counts come from `job_progress`, populated by the same event-sink write-through the indexing jobs use. Click the realm cell for the live activity feed; `spawned_by` pivots to the Indexing dashboard focused on the index job whose invalidation set this job renders.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "left",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false,
+ "minWidth": 100
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "percent"
+ },
+ "properties": [
+ {
+ "id": "custom.cellOptions",
+ "value": {
+ "mode": "gradient",
+ "type": "gauge",
+ "valueDisplayMode": "hidden"
+ }
+ },
+ {
+ "id": "unit",
+ "value": "percent"
+ },
+ {
+ "id": "min",
+ "value": 0
+ },
+ {
+ "id": "max",
+ "value": 100
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "pct"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "percent"
+ },
+ {
+ "id": "decimals",
+ "value": 1
+ },
+ {
+ "id": "displayName",
+ "value": " "
+ },
+ {
+ "id": "custom.align",
+ "value": "left"
+ },
+ {
+ "id": "custom.minWidth",
+ "value": 70
+ },
+ {
+ "id": "custom.width",
+ "value": 70
+ },
+ {
+ "id": "custom.filterable",
+ "value": false
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "elapsed_seconds"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "s"
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "realm_url"
+ },
+ "properties": [
+ {
+ "id": "custom.hidden",
+ "value": true
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "reservation_id"
+ },
+ "properties": [
+ {
+ "id": "actions",
+ "value": [
+ {
+ "confirmation": "Cancel running reservation ${__value.raw}? The worker will stop processing it.",
+ "fetch": {
+ "body": "",
+ "headers": [
+ [
+ "Authorization",
+ "Bearer ${grafana_secret}"
+ ]
+ ],
+ "method": "POST",
+ "queryParams": [
+ [
+ "reservation_id",
+ "${__value.raw}"
+ ]
+ ],
+ "url": "${realm_server}_grafana-complete-job"
+ },
+ "oneClick": false,
+ "title": "Delete reservation ${__value.raw}",
+ "type": "fetch"
+ }
+ ]
+ },
+ {
+ "id": "mappings",
+ "value": [
+ {
+ "options": {
+ "from": 0,
+ "result": {
+ "color": "red",
+ "index": 0,
+ "text": "Delete"
+ },
+ "to": 9999999999999
+ },
+ "type": "range"
+ }
+ ]
+ },
+ {
+ "id": "displayName",
+ "value": "Action"
+ },
+ {
+ "id": "custom.filterable",
+ "value": false
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "realm"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View activity feed",
+ "url": "/d/fetquzizsej28b?${__url_time_range}&var-realm_url=${__data.fields.realm_url:queryparam}&var-job_id=${__data.fields.job_id}.${__data.fields.reservation_id}&orgId=1&viewPanel=11"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "spawned_by"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View originating index job",
+ "url": "/d/aetquzjcf2znke?${__url_time_range}&var-job_id=${__data.fields.spawned_by}&orgId=1"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "worker_id"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View job logs",
+ "url": "/d/fetquzizsej28b?${__url_time_range}&var-indexing_job_full=${__data.fields.job_id}.${__data.fields.reservation_id}&var-indexing_job_short=${__data.fields.job_id}&orgId=1&viewPanel=20"
+ }
+ ]
+ },
+ {
+ "id": "mappings",
+ "value": [
+ {
+ "options": {
+ "pattern": "^(.{6}).*$",
+ "result": {
+ "index": 0,
+ "text": "View logs ($1)"
+ }
+ },
+ "type": "regex"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 24
+ },
+ "id": 10,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "enablePagination": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true,
+ "sortBy": []
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT\n job_id,\n realm,\n realm_url,\n generation,\n spawned_by,\n files_completed,\n total_files,\n percent,\n percent AS pct,\n elapsed_seconds,\n started_at,\n worker_id,\n reservation_id\nFROM (\n SELECT\n j.id AS job_id,\n COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n ) AS realm,\n COALESCE(j.args->>'realmURL','') AS realm_url,\n (j.args->>'generation')::bigint AS generation,\n (j.args->>'spawningJobId')::bigint AS spawned_by,\n COALESCE(jp.files_completed, 0) AS files_completed,\n COALESCE(jp.total_files, 0) AS total_files,\n CASE WHEN COALESCE(jp.total_files, 0) > 0\n THEN (jp.files_completed::float / jp.total_files) * 100\n ELSE 0\n END AS percent,\n EXTRACT(EPOCH FROM (NOW() - jr.created_at)) AS elapsed_seconds,\n jr.created_at AS started_at,\n jr.worker_id,\n jr.id AS reservation_id\n FROM jobs j\n JOIN job_reservations jr ON jr.job_id = j.id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n LEFT JOIN job_progress jp ON jp.job_id = j.id\n WHERE j.job_type = 'prerender_html'\n AND j.finished_at IS NULL\n AND (${spawning_job_id:sqlstring} = '' OR j.args->>'spawningJobId' = ${spawning_job_id:sqlstring})\n) active\nORDER BY started_at DESC;",
+ "refId": "A"
+ }
+ ],
+ "title": "Active Prerendering",
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "Per-realm aggregate of prerender_html job state. `oldest_pending_seconds` red after 5 min flags realms whose HTML backlog isn't draining.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "left",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false,
+ "minWidth": 100
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ }
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "realm_url"
+ },
+ "properties": [
+ {
+ "id": "custom.hidden",
+ "value": true
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "oldest_pending_seconds"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "s"
+ },
+ {
+ "id": "thresholds",
+ "value": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 60
+ },
+ {
+ "color": "red",
+ "value": 300
+ }
+ ]
+ }
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "realm"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View activity feed",
+ "url": "/d/fetquzizsej28b?${__url_time_range}&var-realm_url=${__data.fields.realm_url:queryparam}&orgId=1&viewPanel=11"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 6,
+ "w": 24,
+ "x": 0,
+ "y": 34
+ },
+ "id": 11,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "enablePagination": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true,
+ "sortBy": []
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT\n COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n ) AS realm,\n COALESCE(j.args->>'realmURL','') AS realm_url,\n COUNT(*) FILTER (WHERE j.status = 'unfulfilled' AND jr.id IS NULL) AS pending,\n COUNT(*) FILTER (WHERE j.status = 'unfulfilled' AND jr.id IS NOT NULL) AS in_flight,\n MAX(j.finished_at) AS last_completed_at,\n EXTRACT(EPOCH FROM (NOW() - MIN(j.created_at)\n FILTER (WHERE j.status = 'unfulfilled' AND jr.id IS NULL))) AS oldest_pending_seconds\n FROM jobs j\n LEFT JOIN job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n WHERE j.job_type = 'prerender_html'\n GROUP BY j.args->>'realmURL'\n ORDER BY pending DESC, in_flight DESC, last_completed_at DESC NULLS LAST\n LIMIT 200;",
+ "refId": "A"
+ }
+ ],
+ "title": "Per-realm prerender status",
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs waiting on a worker. `urls` is the size of the job's merged change set; `coalesced` counts publishes folded into the job while it waited; `spawned_by` is the index job whose publish created it (the newest publish wins when jobs merge).",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "left",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false,
+ "minWidth": 150
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "wait_seconds"
+ },
+ "properties": [
+ {
+ "id": "unit",
+ "value": "s"
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "spawned_by"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "View originating index job",
+ "url": "/d/aetquzjcf2znke?${__url_time_range}&var-job_id=${__data.fields.spawned_by}&orgId=1"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "job_id"
+ },
+ "properties": [
+ {
+ "id": "actions",
+ "value": [
+ {
+ "confirmation": "Delete waiting job ${__value.raw}? This marks it as completed without running it.",
+ "fetch": {
+ "body": "",
+ "headers": [
+ [
+ "Authorization",
+ "Bearer ${grafana_secret}"
+ ]
+ ],
+ "method": "POST",
+ "queryParams": [
+ [
+ "job_id",
+ "${__value.raw}"
+ ]
+ ],
+ "url": "${realm_server}_grafana-complete-job"
+ },
+ "oneClick": false,
+ "title": "Delete job ${__value.raw}",
+ "type": "fetch"
+ }
+ ]
+ },
+ {
+ "id": "mappings",
+ "value": [
+ {
+ "options": {
+ "from": 0,
+ "result": {
+ "color": "red",
+ "index": 0,
+ "text": "Delete"
+ },
+ "to": 9999999999999
+ },
+ "type": "range"
+ }
+ ]
+ },
+ {
+ "id": "displayName",
+ "value": "Action"
+ },
+ {
+ "id": "custom.filterable",
+ "value": false
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 40
+ },
+ "id": 12,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "enablePagination": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true,
+ "sortBy": []
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "editorMode": "code",
+ "format": "table",
+ "rawQuery": true,
+ "rawSql": "SELECT\n j.id,\n j.priority,\n COALESCE(\n NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n ) AS realm,\n j.status AS status,\n j.created_at AS created_at,\n EXTRACT(EPOCH FROM (NOW() - j.created_at)) AS wait_seconds,\n (j.args->>'generation')::bigint AS generation,\n (j.args->>'spawningJobId')::bigint AS spawned_by,\n jsonb_array_length(COALESCE(j.args->'changes', '[]'::jsonb)) AS urls,\n COALESCE((j.args->>'coalescedPublishes')::int, 0) AS coalesced,\n j.id AS job_id\nFROM jobs j\nLEFT JOIN job_reservations jr ON j.id = jr.job_id\n AND jr.completed_at IS NULL AND jr.locked_until > NOW()\nWHERE jr.id IS NULL\n AND j.status = 'unfulfilled'\n AND j.job_type = 'prerender_html'\n AND (${spawning_job_id:sqlstring} = '' OR j.args->>'spawningJobId' = ${spawning_job_id:sqlstring})\nORDER BY j.created_at ASC\nLIMIT 500;",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued Prerender Jobs",
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "grafana-postgresql-datasource",
+ "uid": "cef5v5sl9k7i8f"
+ },
+ "description": "prerender_html jobs that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). `generation` is the realm index generation stamped on the HTML rows the job wrote; `spawned_by` pivots to the originating index job. Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
+ "fieldConfig": { |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fd02d0235
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review feedback on the dashboards: interpolate the job-id textbox vars through Grafana's sqlstring formatter so a quote in the URL can't break or alter the panel SQL, and make the Queued tables join only live reservations (matching the Pending stat) so a job whose lease expired and is claimable again stays visible. Also type coalescedPublishes as number|null — an optional number violates WorkerArgs' JSON index signature, which is what broke CI lint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot stopped reviewing on behalf of
habdelra due to an error
July 8, 2026 23:28
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds a coalescedPublishes counter to prerender_html jobs so the queue can track how many publishes were merged into a pending job, and introduces observability dashboards/links to surface these metrics.
Changes:
- Add
coalescedPublishesto the prerender job args, parse it safely, and accumulate it during coalesce joins - Initialize the new arg at enqueue time and add tests covering count accumulation across merges
- Add a new Grafana “Prerender HTML” dashboard and cross-links from the Indexing dashboard; exclude generated Grafana resources from prettier
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/runtime-common/tasks/prerender-html.ts | Adds coalescedPublishes to args, parses it, and increments it during job coalescing. |
| packages/runtime-common/jobs/prerender-html.ts | Seeds coalescedPublishes as null when enqueueing new jobs. |
| packages/realm-server/tests/prerender-html-split-test.ts | Adds assertions to verify coalescedPublishes increments and accumulates across merges. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/prerender-html.json | Adds a new Grafana dashboard for prerender job health and coalescing metrics. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/indexing.json | Adds navigation to the new dashboard and job-id filtering/drill-through links. |
| .prettierignore | Prevents prettier from reformatting generated Grafana resource JSON. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
lukemelia
approved these changes
Jul 8, 2026
habdelra
changed the base branch from
cs-11763-prerender-split-prerender_html-job-enqueue-coalesce-swap
to
main
July 9, 2026 13:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Grafana dashboard for
prerender_htmljobs, cross-linked with the Indexing dashboardNow that HTML prerendering runs as its own
prerender_htmlbackground job, operators need the same visibility into that job type that the Indexing dashboard gives them for indexing jobs — and a way to hop between the two halves of what used to be a single job.The new Prerender HTML dashboard
packages/observability/grafanactl/resources/dashboards/boxel-status/prerender-html.jsonmirrors the Indexing dashboard panel-for-panel, keyed tojob_type = 'prerender_html', so the two read identically: pending / in-flight / oldest-pending stats, throughput (arrived / started / completed), pending-vs-in-flight-vs-completed stocks over time, an Active table with live per-file progress bars, a per-realm rollup, and queued / completed job tables with the same delete actions and log drill-throughs. The per-file progress bars work because the prerender pass reports through the same event-sink write-through tojob_progressthat indexing uses.Two deliberate departures from a literal mirror:
It also adds an outcome-stats row the indexing dashboard doesn't have, because these are the questions specific to a deferred channel: completed / failed / retried counts over the dashboard's time range, plus publishes coalesced — how many enqueues were folded into an already-pending job for the same realm. Coalescing is invisible in the
jobstable (a merged publish updates the surviving row in place rather than inserting), so the coalesce handler now stamps acoalescedPublishescount into the merged job's args, the same way it already stampsspawningJobId. That's the only product-code change here, it's additive, and it's covered by new coalesce-decision tests.Cross-links in both directions
Every job table on the new dashboard carries
generationandspawned_bycolumns. Clickingspawned_byopens the Indexing dashboard with its new Job id filter prefilled, landing you on exactly the indexing job whose invalidation set the prerender job renders. In the other direction, the job-id cells in the Indexing dashboard's Active and Completed tables link to the Prerender HTML dashboard filtered by Spawning index job, showing the HTML job that indexing pass published. The Completed Indexing Jobs table also gains agenerationcolumn (from the job result), so the(realm, generation)pair is visible on both sides.One asymmetry to be aware of: when several publishes coalesce, the surviving job carries the newest publish's
spawningJobId. A drill-through from an older, merged-away indexing pass can come up empty — the filter's description says so, and clearing it shows the realm's jobs.Non-regression
The Indexing dashboard changes: a dashboards-nav link, the Job id filter (defaults to empty = no filter), the generation column, and the two pivot links. Filter textbox values on both dashboards interpolate through Grafana's
sqlstringformatter, so a quote in the URL or textbox can't break or alter the panel SQL. Both dashboards' Queued tables count a job as queued when no live reservation holds it — the same definition the Pending stat uses — so a job whose lease expired and is claimable again stays visible and deletable. All panels were exercised against a local Grafana + Postgres with seeded jobs..prettierignorenow excludesgrafanactl/resources/so commit-time formatting can't fight thegrafanactl resources pullround-trip format (the observability package's lint already excluded dashboard JSON from prettier for exactly this reason).Tests
prerender-html-split-test— the pending-join merge stampscoalescedPublishes: 1on the first fold and accumulates across merges.prerender_htmljobs in every state (pending, in-flight with progress, resolved, rejected, retried, coalesced) verifying each panel's SQL and both drill-through directions.🤖 Generated with Claude Code
https://claude.ai/code/session_011CV4mDDVQaBS16qmqCVXqE