diff --git a/CLAUDE.md b/CLAUDE.md index cebfa1f..5e0f929 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,11 +120,12 @@ The token must be an **API Token** (not Global API Key), created at: https://dash.cloudflare.com/profile/api-tokens Required permissions: -- Account > Account Analytics > Read +- Account > Account Analytics > Read (without this the CF half reads empty — the checker now surfaces this as an auth failure instead of a false all-clear) - Account > Workers Scripts > Edit - Account > Workers R2 Storage > Read - Account > D1 > Read -- Zone > Zone > Read +- Zone > Zone > Read (also required to enumerate zones for Workers Routes removal on auto-disconnect) +- Zone > Workers Routes > Edit (auto-disconnect removes zone [[routes]], not just workers.dev + custom domains) Or use the "Edit Cloudflare Workers" template. @@ -136,7 +137,7 @@ Required permissions: - Write access for auto-kill actions (stop/terminate pods, scale endpoints) ## Supported Cloud Providers -- **Cloudflare** — Workers, Durable Objects, R2, D1, Queues, Stream, Zones +- **Cloudflare** — Workers, Durable Objects, R2, D1, Queues, Stream, Zones, Workers AI (Neurons), AI Gateway (upstream LLM $), Vectorize - **GCP** — Cloud Run, Compute Engine, GKE, BigQuery, Cloud Functions, Cloud Storage - **AWS** — EC2, Lambda, RDS, ECS, EKS, S3, SageMaker, Cost Explorer - **RunPod** — GPU Pods (on-demand & spot), Serverless Endpoints, Network Volumes diff --git a/FEEDBACK-from-divinci-deployment.md b/FEEDBACK-from-divinci-deployment.md new file mode 100644 index 0000000..e73b0db --- /dev/null +++ b/FEEDBACK-from-divinci-deployment.md @@ -0,0 +1,146 @@ +# Feedback from the Divinci self-hosted deployment (2026-06-16/17) + +> **Status (2026-06-17):** Addressed on branch `feat/cf-ai-coverage-and-kill-safety`. +> - **`packages/kill-switch-cf`** (self-hosted worker): #1, #2, #3, #4, #5, #8, #9, #10, #11, #12 implemented. +> - **`packages/api`** (managed engine): #1, #2, #9 mirrored. #4 (6h cooldown) and #5 +> (metric-category gating) already existed here. +> - Deferred: #6/#7 (uniform $/day thresholds + cross-cloud catch-all clarity) — mostly +> docs/UI; the managed engine already computes `estimatedDailyCostUSD` per service. + + +Real-world findings from running a fork of `kill-switch-cf` in production against +a busy Cloudflare + GCP account (Workers, DO, R2, D1, Queues, **Workers AI**, AI +Gateway, Vectorize + Cloud Run). Ordered roughly by impact. Each is a candidate +issue/PR for the upstream repo. + +--- + +## 1. No Workers AI (Neurons) monitoring — biggest coverage gap (HIGH) + +The CF coverage (Workers/DO/R2/D1/Queues/Stream/Pages) omits **Workers AI +inference**, which is today's scariest cost runaway. Our single worst spike was +Workers AI: one model did **3.67M Neurons (~$40) in a day** vs a <$0.35/day +baseline — and nothing watched it. + +Add the `aiInferenceAdaptiveGroups` dataset: +- dimensions: `modelId`, `neurons`, `errorCode`, `requestSource`, `tag` +- sum: `totalNeurons`, `totalInputTokens`, `totalOutputTokens`, `totalInferenceTimeMs` +- top-level: `count` (request count) +- pricing: **$0.011 per 1,000 Neurons** + +Threshold on daily total Neurons (we used 500k ≈ $5.50/day). Detect-and-alert +only — there's no CF API to throttle Workers AI per-model, so document that the +mitigation is in-app (drop the model from the caller's fallback chain). + +## 2. AI Gateway cost monitoring (HIGH for anyone using it as an LLM proxy) + +`aiGatewayRequestsAdaptiveGroups` has a `sum.cost` that attributes **upstream +provider** cost (OpenAI/Anthropic/Gemini/Vertex) for traffic routed through AI +Gateway. This catches external LLM spend that neither Workers AI Neurons nor the +CF bill can see. +- dimensions: `gateway`, `provider`, `model`, `cached`, `error`, `rateLimited`, `statusCode` +- sum: `cost`, `erroredRequests`, `cachedRequests`, `uncachedTokensIn/Out`, `cachedTokensIn/Out` +- Gotcha: `cost` is **$0 for the `workers-ai` provider** (Neurons bills that + separately) — combine #1 and #2 to avoid double-counting and to avoid blind spots. + +## 3. Vectorize monitoring (LOW, but easy) + +`vectorizeQueriesAdaptiveGroups`: +- dimensions: `vectorizeIndexId` only +- sum: `queriedVectorDimensions` +- **Gotcha: this dataset has NO top-level `count` field** (querying it errors). +- pricing: $0.01 per 1M queried dimensions. + +## 4. Alert de-dup / edge-triggering should be built into the self-hosted worker (HIGH) + +With a 5-min cron and **daily-cumulative** thresholds (requests/neurons/rows that +accumulate over the UTC day), once a threshold is crossed it STAYS crossed for the +rest of the day → the same alert re-fires every 5 min (~288×/day). PagerDuty +self-dedups via `dedup_key`, but **Discord/Slack/custom webhooks do not** — we +got an all-day alert stream on Slack. The managed `monitoring-engine` has a 6h +cooldown; the self-hosted `kill-switch-cf` should ship the same. We implemented it +with a KV namespace keyed by a **digit-stripped signature** of the violation set +(so the changing numbers in messages don't defeat the key), TTL = cooldown, +PagerDuty exempt. + +## 5. Separate the ALERT threshold from the KILL threshold (HIGH — caused an outage) + +Gating auto-disconnect on the SAME threshold as alerting means either (a) any +over-threshold service is auto-killed, or (b) you raise the threshold and lose +early alerting. **This caused a production incident for us:** a legitimate +high-traffic Durable Object (a voice agent, ~1.5M reqs/day = **$0.23/day**) was +**auto-disconnected** because the default DO threshold (1M reqs) doubled as the +kill threshold. A $0.23 cost signal took down a live service at 2am. + +Fix we adopted: a `DISCONNECT_THRESHOLD_MULTIPLIER` (default 10) — alerts fire at +1× the threshold, auto-disconnect/delete only above N×. Strongly recommend the +self-hosted package **default to alert-only**, or ship this multiplier defaulting +high. (This is a simpler cousin of the managed product's metric-category gating.) + +## 6. Default thresholds should be dollar-calibrated, not raw counts (MED) + +`DO_REQUEST_THRESHOLD` default of 1M reqs = **$0.15/day** — not a meaningful cost +signal, yet it triggered a kill. The product already computes +`estimatedDailyCostUSD` per service in `UsageResult`; consider letting operators +threshold on **$/day uniformly** instead of per-metric raw counts, or at least +pick raw-count defaults that map to a sane dollar floor. + +## 7. Catch-all "total spend" threshold semantics (MED) + +A cross-cloud "total daily spend" catch-all that includes a dominant provider +(GCP Cloud Run, ~$130/day for us) alongside CF (~$12/day) fires constantly if you +size it against CF intuition (we mis-set $50 vs a $140 baseline). Either scope +catch-alls per-provider, or make the docs/UI loudly state the total is cross-cloud +and show the current baseline when setting it. + +## 8. Batch GraphQL datasets into one query — rate limits (MED) + +Firing many sequential GraphQL queries per check (we now have ~9) hits CF GraphQL +Analytics rate limits (**10429**), which `cfGraphQLSafe` swallows → silent +monitoring gaps. The GraphQL API supports multiple top-level dataset fields under +one `account {}` selection in a single request — batching cuts request count ~8× +and dodges the limit. + +## 9. Silent auth-error blindness (HIGH — security/reliability) + +A CF token missing **`Account Analytics:Read`** returns `10000 Authentication +error`, which `cfGraphQLSafe` swallows → the **entire CF half reads empty** and +the operator believes they're protected. We were blind for an unknown period +without noticing (only GCP was working). Recommendations: +- Surface a prominent `analyticsAuth: "FAILING"` status on the health/`/spend` + endpoint instead of silently returning zeros. +- Document the EXACT token scopes: **Account Analytics:Read** (monitoring) + + **Workers Scripts:Edit** + **Workers Routes:Edit** (auto-disconnect). + +## 10. `disconnectWorker` doesn't remove zone Workers Routes (MED — correctness) + +The disconnect path disables the workers.dev subdomain and deletes custom +domains, but does **not** remove zone Workers **routes** (`[[routes]]` patterns). +A worker served via a route (not workers.dev) would NOT actually be disconnected, +despite the docs claiming "Workers Routes: Edit" is required. Either implement +route removal or fix the docs so operators don't over-trust the kill. + +## 11. No restore/undo for false positives (MED — operability) + +When auto-disconnect fires on a false positive there's no built-in recovery — we +had to manually `POST /workers/scripts//subdomain {enabled:true}`. A +`kill-switch restore ` CLI/endpoint (re-enable subdomain, re-add removed +domains/routes from the forensic snapshot) would make false positives recoverable +in seconds. + +## 12. Endpoints fail-open without ADMIN_SECRET (MED — security) + +The self-hosted worker's HTTP endpoints (`/spend`, `/usage`, `/check`, +`/test-alert`) are **unauthenticated unless `ADMIN_SECRET` is set**, and `/check` +can trigger destructive actions while `/test-alert` can be spammed. Default to +**fail-closed** (refuse if no `ADMIN_SECRET` configured) rather than fail-open. + +--- + +### Net +The product's biggest near-term value-add is **LLM cost coverage** (#1, #2) — +that's the runaway everyone fears in 2026 and it's currently unmonitored. The +biggest safety bug is **alert-threshold == kill-threshold** (#5), which turned a +$0.23 signal into an outage. Everything else is polish. + +_— Compiled from the Divinci billing-monitor deployment, 2026-06-17._ diff --git a/packages/api/src/providers/cloudflare/checker.ts b/packages/api/src/providers/cloudflare/checker.ts index de8878e..8d6829f 100644 --- a/packages/api/src/providers/cloudflare/checker.ts +++ b/packages/api/src/providers/cloudflare/checker.ts @@ -31,6 +31,12 @@ const CLOUDFLARE_METRIC_CATEGORIES: Record = { streamMinutesPerDay: "load", argoGBPerDay: "load", r2StorageGB: "storage", + // LLM/AI spend. Categorized "load" (not "cost") on purpose: it's genuinely a + // cost runaway, but there's NO Cloudflare API to throttle Workers AI per-model + // or to cap AI Gateway's upstream provider spend — so these are alert-only and + // must not be in the default auto-kill set. Mitigation is in-app. + aiNeuronsPerDay: "load", + aiGatewayCostUSD: "load", }; const CF_GRAPHQL = `${CF_API}/graphql`; @@ -45,6 +51,20 @@ async function cfFetch(path: string, token: string, options: RequestInit = {}): }); } +/** + * Classified Cloudflare GraphQL error. `kind === "auth"` is the dangerous one: + * a token missing `Account Analytics: Read` returns a 10000 "Authentication + * error", and if that were swallowed the whole CF half would read EMPTY while + * the operator believes they're protected (the silent-blindness bug, #9). Auth + * errors must propagate so the account is flagged, never masked as "all clear". + */ +export class CfGraphQLError extends Error { + constructor(message: string, public kind: "auth" | "rate-limit" | "graphql" | "parse" | "http") { + super(message); + this.name = "CfGraphQLError"; + } +} + async function cfGraphQL(token: string, accountId: string, query: string): Promise { const res = await fetch(CF_GRAPHQL, { method: "POST", @@ -60,11 +80,25 @@ async function cfGraphQL(token: string, accountId: string, query: string): Promi try { data = JSON.parse(text); } catch { - throw new Error(`CF GraphQL parse error: ${text.substring(0, 200)}`); + throw new CfGraphQLError(`CF GraphQL parse error: ${text.substring(0, 200)}`, "parse"); } - if (data.errors) { - throw new Error(`CF GraphQL error: ${JSON.stringify(data.errors)}`); + if (data.errors && data.errors.length) { + const blob = JSON.stringify(data.errors); + if (/\b10000\b|9106|9109|authentication error|not authorized|account analytics/i.test(blob)) { + throw new CfGraphQLError( + `Cloudflare analytics auth FAILING — the API token is missing 'Account Analytics: Read'. Monitoring is BLIND until fixed. ${blob.substring(0, 200)}`, + "auth", + ); + } + if (/\b10429\b|rate.?limit|too many request/i.test(blob)) { + throw new CfGraphQLError(`Cloudflare GraphQL rate-limited (10429): ${blob.substring(0, 200)}`, "rate-limit"); + } + throw new CfGraphQLError(`CF GraphQL error: ${blob.substring(0, 300)}`, "graphql"); + } + + if (!res.ok) { + throw new CfGraphQLError(`CF GraphQL HTTP ${res.status}: ${text.substring(0, 150)}`, "http"); } return data; } @@ -140,6 +174,89 @@ async function queryWorkerUsage(token: string, accountId: string): Promise { + const today = new Date().toISOString().split("T")[0]; + const query = `{ + viewer { + accounts(filter: {accountTag: "${accountId.replace(/[^a-zA-Z0-9-]/g, "")}"}) { + aiInferenceAdaptiveGroups( + limit: 100, + filter: {date_geq: "${today}"}, + orderBy: [sum_totalNeurons_DESC] + ) { + count + dimensions { modelId } + sum { totalNeurons totalInputTokens totalOutputTokens } + } + } + } + }`; + + try { + const data = await cfGraphQL(token, accountId, query); + const groups = data?.data?.viewer?.accounts?.[0]?.aiInferenceAdaptiveGroups ?? []; + return groups.map((g: any) => { + const neurons = g.sum.totalNeurons || 0; + // Workers AI pricing: $0.011 per 1,000 Neurons + const cost = (neurons / 1000) * 0.011; + return { + serviceName: `ai:${g.dimensions.modelId}`, + metrics: [ + { name: "Workers AI Neurons", value: neurons, unit: "neurons", thresholdKey: "aiNeuronsPerDay" }, + ], + estimatedDailyCostUSD: cost, + }; + }); + } catch (e) { + if (e instanceof CfGraphQLError && e.kind === "auth") throw e; + return []; + } +} + +async function queryAIGatewayUsage(token: string, accountId: string): Promise { + const today = new Date().toISOString().split("T")[0]; + const query = `{ + viewer { + accounts(filter: {accountTag: "${accountId.replace(/[^a-zA-Z0-9-]/g, "")}"}) { + aiGatewayRequestsAdaptiveGroups( + limit: 100, + filter: {date_geq: "${today}"}, + orderBy: [sum_cost_DESC] + ) { + dimensions { gateway provider model } + sum { cost erroredRequests cachedRequests } + } + } + } + }`; + + try { + const data = await cfGraphQL(token, accountId, query); + const groups = data?.data?.viewer?.accounts?.[0]?.aiGatewayRequestsAdaptiveGroups ?? []; + return groups + // sum.cost is $0 for the workers-ai provider (Neurons bill that separately, + // counted in queryAIUsage) — skip to avoid double-counting / false zeros. + .filter((g: any) => g.dimensions.provider !== "workers-ai") + .map((g: any) => { + const cost = g.sum.cost || 0; + return { + serviceName: `aigw:${g.dimensions.gateway}/${g.dimensions.provider}/${g.dimensions.model}`, + metrics: [ + { name: "AI Gateway Upstream Cost", value: cost, unit: "USD", thresholdKey: "aiGatewayCostUSD" }, + ], + estimatedDailyCostUSD: cost, + }; + }); + } catch (e) { + if (e instanceof CfGraphQLError && e.kind === "auth") throw e; + return []; + } +} + // ─── R2, D1, Queues, Stream Usage Queries ─────────────────────────────────── async function queryR2Usage(token: string, accountId: string): Promise { @@ -188,7 +305,9 @@ async function queryR2Usage(token: string, accountId: string): Promise ({ serviceName: `r2:${b.name}`, metrics: [], @@ -243,7 +362,8 @@ async function queryD1Usage(token: string, accountId: string): Promise(); - for (const s of [...doServices, ...workerServices, ...r2Services, ...d1Services, ...queueServices, ...streamServices]) { + for (const s of [...doServices, ...workerServices, ...r2Services, ...d1Services, ...queueServices, ...streamServices, ...aiServices, ...aiGatewayServices]) { const existing = serviceMap.get(s.serviceName); if (existing) { existing.metrics.push(...s.metrics); @@ -515,6 +637,18 @@ export const cloudflareProvider: CloudProvider = { return pauseZone(apiToken, serviceId || serviceName); } + // Workers AI / AI Gateway are alert-only: there's no Cloudflare API to + // throttle per-model inference or cap upstream provider spend. Return a + // non-success result rather than attempting a bogus worker disconnect. + if (serviceType === "ai" || serviceType === "aigw") { + return { + success: false, + action, + serviceName, + details: "Workers AI / AI Gateway spend cannot be throttled via the Cloudflare API — alert-only. Mitigate in-app (drop the model from the fallback chain) or set an account spend limit.", + }; + } + // Service-specific kill actions switch (serviceType) { case "r2": @@ -591,6 +725,8 @@ export const cloudflareProvider: CloudProvider = { queueOpsPerDay: 1_000_000, streamMinutesPerDay: 10_000, argoGBPerDay: 100, + aiNeuronsPerDay: 500_000, // ~$5.50/day @ $0.011/1k Neurons + aiGatewayCostUSD: 10, // upstream LLM $/day routed through AI Gateway }; }, }; diff --git a/packages/api/src/providers/types.ts b/packages/api/src/providers/types.ts index a808ce2..7b9d9cb 100644 --- a/packages/api/src/providers/types.ts +++ b/packages/api/src/providers/types.ts @@ -168,6 +168,8 @@ export interface ThresholdConfig { streamMinutesPerDay?: number; // Stream minutes stored/delivered argoGBPerDay?: number; // Argo Smart Routing GB routed pagesRequestsPerDay?: number; // Pages Functions requests + aiNeuronsPerDay?: number; // Workers AI Neurons/day (alert-only; no CF throttle API) + aiGatewayCostUSD?: number; // AI Gateway upstream provider $/day (alert-only) // ─── GCP Cost Thresholds ──────────────────────────────────────────────────── gcpBudgetPercent?: number; diff --git a/packages/api/tests/providers/cloudflare.test.ts b/packages/api/tests/providers/cloudflare.test.ts index 4f74afe..cbb5462 100644 --- a/packages/api/tests/providers/cloudflare.test.ts +++ b/packages/api/tests/providers/cloudflare.test.ts @@ -788,5 +788,93 @@ describe("Cloudflare Provider", () => { expect.objectContaining({ method: "DELETE" }) ); }); + + it("treats Workers AI / AI Gateway as alert-only (no CF kill API)", async () => { + const ai = await cloudflareProvider.executeKillSwitch(credential, "ai:@cf/meta/llama-3", "disconnect"); + expect(ai.success).toBe(false); + expect(ai.details).toMatch(/alert-only/i); + + const gw = await cloudflareProvider.executeKillSwitch(credential, "aigw:my-gw/openai/gpt-4o", "disconnect"); + expect(gw.success).toBe(false); + expect(gw.details).toMatch(/alert-only/i); + // No destructive call should have been attempted + expect(mockFetch).not.toHaveBeenCalled(); + }); + }); + + // ─── LLM / AI coverage (#1 Workers AI, #2 AI Gateway) ────────────────────── + + function mockAllDatasets(overrides: Record = {}) { + const ds = (key: string) => overrides[key] ?? []; + mockFetch.mockImplementation(async (url: string, options: any) => { + const query = JSON.parse(options?.body || "{}").query || ""; + const urlStr = typeof url === "string" ? url : ""; + const wrap = (field: string, rows: any[]) => ({ + ok: true, + text: async () => JSON.stringify({ data: { viewer: { accounts: [{ [field]: rows }] } } }), + }); + if (query.includes("durableObjects")) return wrap("durableObjectsInvocationsAdaptiveGroups", ds("do")); + if (query.includes("workersInvocations")) return wrap("workersInvocationsAdaptive", ds("worker")); + if (query.includes("r2Storage")) return wrap("r2StorageAdaptiveGroups", ds("r2")); + if (query.includes("d1Analytics")) return wrap("d1AnalyticsAdaptiveGroups", ds("d1")); + if (query.includes("aiInference")) return wrap("aiInferenceAdaptiveGroups", ds("ai")); + if (query.includes("aiGatewayRequests")) return wrap("aiGatewayRequestsAdaptiveGroups", ds("aigw")); + if (urlStr.includes("/r2/buckets") || urlStr.includes("/d1/database") || + urlStr.includes("/queues") || urlStr.includes("/stream")) { + return { ok: true, text: async () => JSON.stringify({ result: [] }) }; + } + return { ok: true, text: async () => "{}" }; + }); + } + + describe("Workers AI (#1)", () => { + it("flags a Neuron spike as an alert-only (load) violation with $ estimate", async () => { + mockAllDatasets({ + ai: [{ count: 1200, dimensions: { modelId: "@cf/meta/llama-3" }, sum: { totalNeurons: 3_670_000, totalInputTokens: 1, totalOutputTokens: 2 } }], + }); + const result = await cloudflareProvider.checkUsage(credential, { aiNeuronsPerDay: 500_000 }); + const svc = result.services.find(s => s.serviceName === "ai:@cf/meta/llama-3"); + expect(svc).toBeDefined(); + // $0.011 / 1k Neurons → 3.67M Neurons ≈ $40.37 + expect(svc!.estimatedDailyCostUSD).toBeCloseTo(40.37, 1); + const v = result.violations.find(v => v.serviceName === "ai:@cf/meta/llama-3"); + expect(v).toBeDefined(); + expect(v!.category).toBe("load"); // alert-only — not in default auto-kill set + }); + }); + + describe("AI Gateway (#2)", () => { + it("flags upstream provider cost but skips the workers-ai provider (no double-count)", async () => { + mockAllDatasets({ + aigw: [ + { dimensions: { gateway: "g", provider: "openai", model: "gpt-4o" }, sum: { cost: 25, erroredRequests: 0, cachedRequests: 0 } }, + { dimensions: { gateway: "g", provider: "workers-ai", model: "@cf/x" }, sum: { cost: 0, erroredRequests: 0, cachedRequests: 0 } }, + ], + }); + const result = await cloudflareProvider.checkUsage(credential, { aiGatewayCostUSD: 10 }); + expect(result.services.some(s => s.serviceName.includes("workers-ai"))).toBe(false); + const v = result.violations.find(v => v.serviceName === "aigw:g/openai/gpt-4o"); + expect(v).toBeDefined(); + expect(v!.currentValue).toBe(25); + expect(v!.category).toBe("load"); + }); + }); + + describe("analytics auth blindness (#9)", () => { + it("throws (does NOT silently report all-clear) when the token lacks Account Analytics:Read", async () => { + mockFetch.mockImplementation(async (url: string, options: any) => { + const query = JSON.parse(options?.body || "{}").query || ""; + const urlStr = typeof url === "string" ? url : ""; + if (query) { + return { ok: true, text: async () => JSON.stringify({ errors: [{ message: "Authentication error", code: 10000 }] }) }; + } + if (urlStr.includes("/r2/buckets") || urlStr.includes("/d1/database") || + urlStr.includes("/queues") || urlStr.includes("/stream")) { + return { ok: true, text: async () => JSON.stringify({ result: [] }) }; + } + return { ok: true, text: async () => "{}" }; + }); + await expect(cloudflareProvider.checkUsage(credential, defaultThresholds)).rejects.toThrow(/auth|analytics|blind/i); + }); }); }); diff --git a/packages/kill-switch-cf/README.md b/packages/kill-switch-cf/README.md index c86a548..9aa54b3 100644 --- a/packages/kill-switch-cf/README.md +++ b/packages/kill-switch-cf/README.md @@ -8,13 +8,31 @@ Born from an **$80,000 Durable Objects bill**. Cloudflare has no native spending Every 6 hours (configurable), this Worker: -1. Queries Cloudflare's GraphQL Analytics API for per-worker usage -2. Checks Durable Object requests, wall-time, and Worker request volume against your thresholds -3. If any worker exceeds limits: +1. Queries Cloudflare's GraphQL Analytics API for per-worker usage (in two batched + requests — see [Coverage](#coverage) — to stay under the GraphQL rate limit) +2. Checks Durable Objects, Workers, R2, D1, **Workers AI (Neurons)**, **AI Gateway + (upstream LLM $)**, and **Vectorize** against your thresholds +3. If any resource exceeds its **alert** threshold: - **Alerts** you via PagerDuty (phone call), Discord, Slack, or custom webhook - - **Auto-disconnects** the offending worker by removing its routes and custom domains + (webhook alerts are de-duped within a cooldown window so a daily-cumulative + breach doesn't re-fire every cron tick) + - **Auto-disconnects** the offending worker — only if it also crosses the + **kill** threshold (`DISCONNECT_THRESHOLD_MULTIPLIER`× the alert threshold) — + by removing its zone routes, custom domains, and workers.dev subdomain - Worker code stays intact — just stops receiving traffic (reversible) +### Coverage + +| Service | Dataset | Action | +|---------|---------|--------| +| Durable Objects | `durableObjectsInvocationsAdaptiveGroups` | alert + disconnect/delete | +| Workers | `workersInvocationsAdaptive` | alert + disconnect | +| R2 | `r2StorageAdaptiveGroups` | alert + delete (opt-in) | +| D1 | `d1AnalyticsAdaptiveGroups` | alert + delete (opt-in) | +| **Workers AI** | `aiInferenceAdaptiveGroups` (Neurons) | **alert only** — no CF throttle API; mitigate in-app | +| **AI Gateway** | `aiGatewayRequestsAdaptiveGroups` (`sum.cost`) | **alert only** — catches upstream OpenAI/Anthropic/Gemini $ | +| **Vectorize** | `vectorizeQueriesAdaptiveGroups` | **alert only** | + ``` Normal: Worker ← Traffic ← Routes/Domains Kill switch: Worker Traffic ✗ Routes removed @@ -39,12 +57,29 @@ wrangler deploy wrangler secret put CLOUDFLARE_ACCOUNT_ID # API token with these permissions: -# - Account Analytics: Read -# - Account Workers Scripts: Edit -# - Account Workers Routes: Edit +# - Account > Account Analytics: Read (monitoring — without it, the CF half reads EMPTY and you're silently blind) +# - Account > Workers Scripts: Edit (auto-disconnect: subdomain + custom domains) +# - Zone > Workers Routes: Edit (auto-disconnect: zone [[routes]] removal) +# - Zone > Zone: Read (enumerate zones to find routes to remove) wrangler secret put CLOUDFLARE_API_TOKEN + +# Bearer token gating /check, /usage, /test-alert. REQUIRED — these endpoints +# FAIL CLOSED (refuse) if ADMIN_SECRET is unset, since /check can take destructive +# actions and /test-alert can be spammed. +wrangler secret put ADMIN_SECRET +``` + +### (Recommended) Enable alert de-dup + kill snapshots + +```bash +# One-time: create the KV namespace, then paste the id into wrangler.toml +# under the commented-out [[kv_namespaces]] block (binding = "ALERT_STATE"). +wrangler kv namespace create ALERT_STATE ``` +Without this binding, webhook alerts cannot be de-duped and will re-fire on every +cron tick once a daily-cumulative threshold is crossed. + ### 3. Set up alerting (at least one) ```bash @@ -88,10 +123,24 @@ All thresholds are set in `wrangler.toml` under `[vars]`: | `DO_REQUEST_THRESHOLD` | `1000000` | Max Durable Object requests per day before alerting | | `DO_WALLTIME_HOURS_THRESHOLD` | `100` | Max DO wall-time hours per day | | `WORKER_REQUEST_THRESHOLD` | `10000000` | Max Worker requests per day (catches feedback loops) | -| `AUTO_DISCONNECT` | `true` | Auto-remove routes when threshold exceeded (reversible) | +| `AI_NEURONS_THRESHOLD` | `500000` | Max Workers AI Neurons/day (~$5.50) — alert only | +| `AI_GATEWAY_COST_THRESHOLD` | `10` | Max AI Gateway upstream $/day — alert only | +| `VECTORIZE_DIMENSIONS_THRESHOLD` | `100000000` | Max Vectorize queried dimensions/day (~$1) — alert only | +| `DISCONNECT_THRESHOLD_MULTIPLIER` | `10` | Kill only above N× the alert threshold (set `1` for alert==kill) | +| `ALERT_COOLDOWN_SECONDS` | `21600` | Webhook alert de-dup window (needs `ALERT_STATE` KV) | +| `AUTO_DISCONNECT` | `true` | Auto-remove routes/domains when kill threshold exceeded (reversible) | | `AUTO_DELETE` | `false` | Auto-delete worker script (nuclear, irreversible) | | `PROTECTED_WORKERS` | `cloudflare-billing-kill-switch` | Comma-separated workers to never kill | +### Alert threshold vs. kill threshold + +Alerts fire at **1×** the thresholds above. Auto-disconnect/delete only fire above +**`DISCONNECT_THRESHOLD_MULTIPLIER`×** (default 10×). This prevents a cheap-but-busy +service — e.g. a 1.5M-req/day voice-agent Durable Object that costs ~$0.23/day — from +being auto-killed off a low alert floor. A breach between 1× and the kill threshold is +reported as `[alert-only]`; above it as `[KILL-ELIGIBLE]`. Set the multiplier to `1` +to restore the old behavior where alerting and killing share one threshold. + ### Cron Schedule Default: every 6 hours. Change in `wrangler.toml`: @@ -116,24 +165,43 @@ PROTECTED_WORKERS = "cloudflare-billing-kill-switch,my-critical-api,my-website" ## How Auto-Disconnect Works -When a worker exceeds thresholds, the kill switch: +When a worker crosses the **kill** threshold, the kill switch: 1. **Disables the workers.dev subdomain** — stops traffic via `*.workers.dev` URLs 2. **Removes custom domains** — detaches any custom domains bound to the worker +3. **Removes zone Workers Routes** — deletes `[[routes]]` patterns across the + account's zones (requires `Zone: Read` + `Workers Routes: Edit`). A worker + served only via a zone route would otherwise keep receiving traffic. -The worker script, Durable Objects, and KV data are **not** deleted. To restore service: +The worker script, Durable Objects, and KV data are **not** deleted. If `ALERT_STATE` +is bound, a forensic snapshot of what was removed (domains + routes) is written to KV +under `kill: