feat: post the canonical alarms embed from axiom via custom-webhook

[zgeoff]

Axiom's custom-webhook notifier can post the canonical alarms embed natively — a [Axiom] severity — what fired title, a severity colour, and a recovery colour when a monitor clears — matching the CI embed shipped in #695. The API refuses it today (400 feature not enabled: notifiersCustomWebhook), so this is blocked on enabling that feature on the Axiom plan.

Scope

• Replace the stock discordWebhook notifier in infra/axiom.ts with two custom-webhook notifiers (critical, warning) posting the canonical embed, once notifiersCustomWebhook is enabled on the Axiom account.
• Wire the 5xx monitor to critical and the verification-lag monitor to warning; bun run up.
• Update docs/architecture/platform/observability.md § Alarms channel and the Bugsink README note to reflect Axiom posting the canonical embed.

Notes

Blocked on a plan/paid decision (enabling notifiersCustomWebhook), not on another issue — no blockedBy edge to record. Verified before the block: the rendered embed parses as valid JSON on both the firing (.Action == "Open") and recovery ("Closed") branches, and the colour placeholder renders as the integer Discord requires.

Palette

Severity	Meaning	Hex	Discord decimal
critical	a failure needing action now	#e5484d	15026253
warning	a degradation worth attention	#ffb224	16757284
recovery	a cleared alert returning to green	#30a46c	3187820

Implementation

Replace the alarmsNotifier block:

const alarmsWebhookURL = pulumi.secret(requireEnv('DISCORD_ALARMS_WEBHOOK'));
const alarmsWebhookHeaders = { 'Content-Type': 'application/json' } as const;

const criticalAlarmsNotifier = new axiom.Notifier(
  'vers-alarms-critical',
  {
    name: 'vers alarms — critical (Discord)',
    properties: {
      customWebhook: {
        url: alarmsWebhookURL,
        headers: alarmsWebhookHeaders,
        body: buildAlarmsWebhookBody({ label: 'critical', firingColor: 15_026_253 }),
      },
    },
  },
  { provider: axiomProvider },
);

const warningAlarmsNotifier = new axiom.Notifier(
  'vers-alarms-warning',
  {
    name: 'vers alarms — warning (Discord)',
    properties: {
      customWebhook: {
        url: alarmsWebhookURL,
        headers: alarmsWebhookHeaders,
        body: buildAlarmsWebhookBody({ label: 'warning', firingColor: 16_757_284 }),
      },
    },
  },
  { provider: axiomProvider },
);

Point serverErrorsMonitor at [criticalAlarmsNotifier.id], verificationLagMonitor at [warningAlarmsNotifier.id]. Swap the export alarmsNotifierName for criticalAlarmsNotifierName / warningAlarmsNotifierName (and in infra/index.ts).

The body builder (private helper, alongside requireEnv):

interface AlarmsSeverity {
  readonly label: string;
  readonly firingColor: number;
}

/**
 * Renders the canonical alarms embed as a Discord webhook payload. Axiom fills
 * the Go-template placeholders per alert: .Action is "Open" when the monitor
 * trips and "Closed" when it recovers, selecting the firing or recovery colour
 * and title word. The colour placeholder sits unquoted so it renders as the
 * integer Discord requires. Monitor descriptions interpolate straight into this
 * JSON string, so they must stay free of double quotes and newlines.
 */
function buildAlarmsWebhookBody(severity: AlarmsSeverity): string {
  const recoveryColor = 3_187_820;

  return [
    '{',
    '  "embeds": [',
    '    {',
    `      "title": "[Axiom] {{ if eq .Action "Open" }}${severity.label}{{ else }}recovery{{ end }} — {{ .Title }}",`,
    '      "description": "{{ .Description }}",',
    '      "url": "https://app.axiom.co",',
    `      "color": {{ if eq .Action "Open" }}${severity.firingColor}{{ else }}${recoveryColor}{{ end }},`,
    '      "fields": [',
    '        { "name": "State", "value": "{{ .Action }}", "inline": true },',
    '        { "name": "Value", "value": "{{ .Value }}", "inline": true }',
    '      ]',
    '    }',
    '  ]',
    '}',
  ].join('\n');
}

Deep-link caveat

The embed links to https://app.axiom.co (console home), not the specific monitor. A per-monitor deep link is https://app.axiom.co/<org>/monitors/{{.MonitorID}}, which needs the org slug — Axiom offers no template variable for it, and the management token 500s on the org endpoints. Parameterise the org slug (a non-secret env var) to build the deep link when adopting this.