Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -411,9 +411,12 @@ jobs:
exit 0
fi
subject="$(printf '%s' "${COMMIT_MSG}" | head -n 1)"
# Canonical alarms embed: [source] severity — what fired, one link to
# the run, and the critical hue shared with the Axiom notifiers.
payload="$(jq -n \
--arg title "main pipeline failed: ${subject}" \
--arg title "[CI] critical — main pipeline failed" \
--arg url "${RUN_URL}" \
--arg desc "${FAILED}" \
'{embeds: [{title: $title, url: $url, description: $desc, color: 15548997}]}')"
--arg desc "${subject}" \
--arg failed "${FAILED}" \
'{embeds: [{title: $title, url: $url, description: $desc, color: 15026253, fields: [{name: "Failed jobs", value: $failed}]}]}')"
curl -fsS -H 'content-type: application/json' -d "${payload}" "${WEBHOOK_URL}"
4 changes: 4 additions & 0 deletions apps/bugsink/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ Discord messaging-service config pointing at the channel's incoming webhook (the
webhook added in its settings. No outgoing email is configured — alerting is webhook-only, and
per-project alert toggles live in each Bugsink project's settings.

Bugsink's Discord messaging service renders a fixed body the app doesn't template, so these alerts
keep their own shape rather than the structured embed the CI path posts to the same channel
(`docs/architecture/platform/observability.md` § Alarms channel).

## Housekeeping

The `.github/workflows/bugsink-vacuum.yml` workflow runs `bugsink-manage vacuum_files` monthly (and
Expand Down
8 changes: 8 additions & 0 deletions docs/architecture/platform/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,11 @@ The `vers verification lag` threshold monitor watches `vers.verification.lag` an
`vers alarms`. It alerts on no data as well as on the threshold. The gauge exports from
`service-replay`'s always-warm machine ([deployment](./deployment.md)), so a silent dataset means
the exporter or the process around it is down, never a healthy quiet system.

## Alarms channel

Axiom monitors, the CI pipeline, and Bugsink post to one Discord channel. The CI `alert` job posts a
structured embed — a `[CI] critical — …` title, the failing run's link, and a red severity colour
(`#e5484d`, decimal `15026253`) — in `.github/workflows/main.yml`. Axiom and Bugsink post their
tools' stock formats: Axiom's custom-webhook notifier, the one templated body it offers, is not
enabled on the plan, and Bugsink's Discord messaging service exposes no templating.
Loading