Skip to content

Harden production deploy pipeline (GitHub Actions deploy + pre-merge guard) - #1

Merged
borland667 merged 4 commits into
mainfrom
ops/harden-netlify-deploy-pipeline
Jun 14, 2026
Merged

Harden production deploy pipeline (GitHub Actions deploy + pre-merge guard)#1
borland667 merged 4 commits into
mainfrom
ops/harden-netlify-deploy-pipeline

Conversation

@borland667

Copy link
Copy Markdown
Owner

Summary

Move production deploys to GitHub Actions so the Netlify noble build-image rollout (June 2026) and similar future regressions stop being a single-point-of-failure that takes the site offline. Netlify's git auto-deploy had been silently failing for ~7 weeks before we noticed.

What this PR adds

  1. netlify.toml[build.environment] NODE_VERSION = "24.1.0" (commit 85a5091). Defence-in-depth so even the now-secondary Netlify-side build path boots the right Node.
  2. CI pre-merge guard netlify-build-check (commit 049e006). Runs netlify build --offline on every PR. Future Netlify-Build regressions show up as a red PR check, not as a silent 7-week outage.
  3. GitHub Actions deploy-production job (commit 0223efe). New source of truth for prod deploys:
    • waits for verify + netlify-build-check + migrate-production
    • pinned Node 24.1.0 via .nvmrc + actions/setup-node@v4
    • npm cidb:generatebuildnetlify deploy --prod --skip-functions-cache
    • 5-attempt live-API smoke check against /api/tournaments?status=active,upcoming; fails the workflow if prod doesn't 200
    • gated on NETLIFY_AUTH_TOKEN + NETLIFY_SITE_ID (already set as repo secrets)
  4. Documentation (commit 7a9469a). docs/ARCHITECTURE.md §15.4 describes the full pipeline, secrets contract, and the --skip-functions-cache requirement. AGENTS.md §3 adds deploy ops rules and rollback guidance.

Why each piece

Failure mode we've seen Mitigation in this PR
Netlify auto-build silently failed for 7 weeks netlify-build-check on every PR → red CI
Function cache reused a stale Prisma client --skip-functions-cache mandatory
closingDate race (DB dropped, Lambda still selecting it) deploy-production needs: migrate-production, ordered explicitly
Old auth/env (April 28 build) live for weeks live-API smoke check fails the workflow
Future Netlify image rollouts Node pinned in netlify.toml AND in GH Actions; deploy is independent of Netlify Build

Secrets configured (out of band)

  • NETLIFY_AUTH_TOKEN — set 2026-06-14
  • NETLIFY_SITE_ID — set 2026-06-14
  • PRODUCTION_DATABASE_URL — pre-existing
  • RESULTS_IMPORT_API_KEY — added to Netlify project env (not a GH secret; the Lambda needs it)
  • NODE_VERSION — added to Netlify project env (24.1.0)

Open follow-up (intentionally not in this PR)

Once we've seen the first GH Actions deploy-production run go green on main, we can disable Netlify's git auto-deploy (skip_automatic_builds = true via Netlify API) to remove the duplicated/failing parallel build. Leaving it on for one merge cycle as a safety net.

Test plan

  • npm run verify passes locally (38 tests, build clean)
  • YAML parses cleanly
  • CI runs both verify and netlify-build-check green on this PR
  • Merge → deploy-production job runs on main
  • Live smoke check (curl /api/tournaments?status=active,upcoming) returns 200
  • Confirm new commit SHA shows up in Netlify deploy title (ci: <sha7> (main))
  • After confirming the above, disable Netlify git auto-deploy in a follow-up

Made with Cursor

borland667 and others added 4 commits June 14, 2026 16:02
Adds a `[build.environment]` block setting `NODE_VERSION = "24.1.0"` so
the Netlify-side build (now secondary to GitHub Actions deploy) uses the
same Node we declare in .nvmrc and package.json#engines. This is
defence-in-depth: even if the primary GH Actions deploy path is bypassed
and a stray Netlify auto-build runs, it boots the right Node.

Context: the June 2026 `noble` build-image rollout silently broke
Netlify auto-deploys for ~7 weeks. Subsequent work in this branch moves
production deploys to GitHub Actions so the noble incompatibility is no
longer in the critical path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a `netlify-build-check` job that runs `netlify build --offline` on
every PR after `verify` passes. This simulates Netlify Build's pipeline
(executing netlify.toml's build command and bundling functions) so we
catch Netlify-Build-specific regressions before they merge to main.

Why we need this: the June 2026 `noble` image rollout silently broke
auto-deploys for ~7 weeks because nothing failed in our PR pipeline.
Running `netlify build` in CI surfaces that class of failure on the PR.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a `deploy-production` job that becomes the source of truth for
production deploys. The job:

  1. Waits for `verify` and `netlify-build-check` to pass
  2. Waits for `migrate-production` (when configured) so the schema is
     ready before the new Lambda boots
  3. Builds with Node pinned via `.nvmrc` (`actions/setup-node@v4`)
  4. Generates the Prisma client against the current schema
  5. Runs `netlify deploy --prod --skip-functions-cache` so the function
     bundle is always fresh (no stale cached Prisma client)
  6. Smoke-checks the live API and fails the job if it does not 200

Gated by NETLIFY_AUTH_TOKEN + NETLIFY_SITE_ID via a
`check-production-deploy` job (secrets cannot be referenced in
job-level `if:`).

Background: Netlify's own git auto-deploy has been failing silently on
the `noble` build image since the June 2026 rollout, leaving production
on the April 28 Lambda for ~7 weeks. Moving the deploy into GitHub
Actions gives us full logs, pinned Node, and a built-in smoke check, so
a future Netlify image regression no longer takes the site down.

Requires GitHub secrets: NETLIFY_AUTH_TOKEN, NETLIFY_SITE_ID (both set
out-of-band when this commit lands).

Co-authored-by: Cursor <cursoragent@cursor.com>
- ARCHITECTURE.md §15.4: new section describing the
  verify → netlify-build-check → migrate → deploy pipeline, the
  required GitHub secrets, the --skip-functions-cache requirement,
  and the live-API smoke check
- ARCHITECTURE.md "Destructive Migrations And The Deploy Race":
  updated to reflect that migrate and deploy are now sequenced in
  the same workflow, noting the residual race only exists while
  Netlify's git auto-deploy stays on as a fallback
- AGENTS.md §3 "Production Deploy Pipeline": ops rules for deploys,
  rollback guidance (prefer restoreSiteDeploy), and the required
  GitHub secrets contract

Co-authored-by: Cursor <cursoragent@cursor.com>
@borland667
borland667 merged commit 471a052 into main Jun 14, 2026
6 checks passed
@borland667
borland667 deleted the ops/harden-netlify-deploy-pipeline branch June 14, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant