Template Sync fails portfolio-wide: automation App lacks the Workflows permission

[devantler]

🤖 Generated by the Daily AI Assistant

Problem

The weekly 🔄 Template Sync workflow is failing on every tenant whose pending template diff touches .github/workflows/**. Confirmed on two consumers (both private), latest scheduled run (Mon 2026-06-22):

• devantler-tech/wedding-app → run 27938670743
• devantler-tech/ascoachingogvaner → run 27938158294

Both fail at the push step with the identical error:

! [remote rejected] chore/template-sync_749b2e9 -> chore/template-sync_749b2e9
  (refusing to allow a GitHub App to create or update workflow `.github/workflows/cd.yaml` without `workflows` permission)
error: failed to push some refs

The sync diff in that run modifies cd.yaml/ci.yaml/release.yaml and adds sync-labels.yaml/validate-scaffold.yaml — any change under .github/workflows/** triggers the rejection. It recurs every Monday 06:00 UTC and will keep failing until the App permission below is granted.

Root cause — the automation App lacks the Workflows repository permission

This is not a workflow bug. The reusable workflow reusable-workflows/.github/workflows/template-sync.yaml (pinned @v5.6.4 by the consumers) already requests permission-workflows: write when it mints the scoped App token via actions/create-github-app-token@v3 (from vars.APP_CLIENT_ID + secrets.APP_PRIVATE_KEY — the same automation App tracked for the Members grant under #53).

The failing run proves the token is minted but without the workflows scope:

Run actions/create-github-app-token@…
  permission-contents: write
  permission-pull-requests: write
  permission-workflows: write        # ← requested
…
- Token: ghs_…                       # ← minted successfully (no 422)
…
! [remote rejected] … without `workflows` permission   # ← but the token can't push workflow files

actions/create-github-app-token@v3 does not 422 when you request a permission the installation lacks — it silently grants the intersection of the requested scopes and the App's installed permissions. The App installation has contents + pull_requests but not workflows, so the minted ghs_ token carries only the former, and GitHub rejects any push that creates/updates a workflow file. (The GITHUB_TOKEN fallback can never push workflow files either — a GitHub safety restriction — so an App token with the Workflows scope is the only path that can sync workflow changes.)

Fix (maintainer-only — no safe gh/declarative path)

Grant the automation GitHub App (the APP_CLIENT_ID app) the Workflows: Read and write repository permission:

1. App settings → Permissions → Repository permissions → Workflows → Read and write.
2. Save, then re-approve the installation on the org (new App permissions require installation re-approval).

GitHub App permission-set edits aren't manageable via the Crossplane deploy/ source of truth or any gh write, so this is a genuine maintainer carve-out — same class as the Members grant tracked under #53. No agent-actionable PR exists for the grant itself.

Verify after granting

Re-run Template Sync on one consumer (gh workflow run "🔄 Template Sync" --repo devantler-tech/wedding-app) and confirm the sync PR opens and the push succeeds.

Minor follow-up (deferred)

The reusable workflow's inline NOTE claims create-github-app-token 422s when the App lacks Workflows — empirically it does not (it silently drops the scope, which is exactly why this failed at push instead of at token-mint). Worth correcting the comment / failing loudly when the minted token lacks a requested scope. Deferred because reusable-workflows is mid-merge into actions (actions#314).

Relates to #56 (declarative GitHub-org-as-code roadmap) and #53 (same automation App).

[devantler]

Fix (maintainer-only — no safe gh/declarative path) - Grant the automation GitHub App (the APP_CLIENT_ID app) the Workflows: Read and write repository permission.

This has already been done. So you should have the go ahead signal already!

[devantler]

🤖 Generated by the Daily AI Assistant

Verification after the grant — still failing, and the simple explanation no longer fits

Thanks for granting it. I re-ran Template Sync to verify (workflow_dispatch on wedding-app, run 28313704719, 06:23Z today) and it still fails at the same push:

 ! [remote rejected] chore/template-sync_5faf3ed -> chore/template-sync_5faf3ed
   (refusing to allow a GitHub App to create or update workflow `.github/workflows/cd.yaml` without `workflows` permission)
error: failed to push some refs to '.../wedding-app'

But digging in, the original "App lacks the Workflows permission" diagnosis looks incomplete — the permission appears to already be present, yet the minted token still drops it:

• APP_CLIENT_ID resolves to botantler-1 (client-id Iv23liNRTPaU9Vf1lXJr).
• botantler-1's org installation already has "workflows": "write" — and has since updated_at: 2026-04-02, repository_selection: all (so wedding-app is covered):

  GET /orgs/devantler-tech/installations → botantler-1:
    permissions.workflows = "write"  (contents=write, pull_requests=write, members=read, …)
  

• The reusable workflow is fine: v5.6.4 (wedding-app's pin) does set use-app-token: true and request permission-contents/pull-requests/workflows: write, and uses the app token (not GITHUB_TOKEN) for the push. The run log confirms use-app-token: true and all three permission-*: write inputs were passed to create-github-app-token.

So: the app has workflows:write at the installation, the RW requests it, the token step didn't 422 — yet the scoped installation token used for the push lacked workflows.

What this points at

create-github-app-token is invoked with repositories: <caller>,<template> (scoped token). The most likely remaining cause is that the repositories-scoped installation token isn't being minted with the workflows scope even though the install grants it — i.e. the gap is in the scoped-token mint, not a missing org-level grant. Worth checking:

1. In the App's settings UI, confirm Workflows: Read and write is actually saved (not just selected) and that there's no pending installation permission-update awaiting acceptance on the org install (the API can list a permission while a pending acceptance still gates token-minting).
2. As a quick isolation test, try a token without the repositories: scoping (org-wide token) to see whether the workflows scope then survives — if it does, the repositories-scoped mint is the culprit and the RW should drop/adjust that input.

I can't take this further from my side (verifying the actual minted-token scope needs the App private key / App-settings access). Leaving #68 open; happy to re-run the verification dispatch once you've checked the App UI.