feat(github): manage GitHub org declaratively via Crossplane provider-upjet-github

AGENTS.md

@@ -198,7 +198,7 @@ You **cannot** decrypt existing secrets without the proper Age keys. For local d
 - **Small, focused changes** — one concern per PR.
 - **Never commit plaintext secrets** — all secrets must be SOPS-encrypted with the `.enc.yaml` suffix.
 - **Base files are immutable** — use Kustomize `patches:` in overlays; never edit `k8s/bases/` directly from a provider or cluster overlay.
-- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`.
+- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. One prod-only side layer hangs off `infrastructure` without gating `apps`: `infrastructure-overprovisioning` (apply-only autoscaler buffer). Declarative GitHub org management runs as a normal **app** (`github-config`) consuming the `devantler-tech/.github` artifact, with its Crossplane provider in the `infrastructure` layer — see [`docs/github-management.md`](docs/github-management.md).
 
 ### Infrastructure File Structure Convention
 

docs/github-management.md

@@ -0,0 +1,127 @@
+# Declarative GitHub management
+
+The devantler-tech GitHub org is managed declaratively as GitOps: GitHub
+repositories (and, over time, rulesets, labels, branch protection, …) are
+**Crossplane managed resources** reconciled by
+[provider-upjet-github](https://github.com/crossplane-contrib/provider-upjet-github)
+against the GitHub API. Change a manifest, merge, and Flux + Crossplane converge
+GitHub to it — including reverting out-of-band UI edits.
+
+The desired state does **not** live in this repo. It lives in
+[`devantler-tech/.github`](https://github.com/devantler-tech/.github) under
+`deploy/`, onboarded here as a **normal app** (`github-config`) using the same
+pattern as every other tenant (cosign-verified OCI artifact → namespace-scoped
+`Kustomization`) — only the artifact carries managed resources instead of a
+workload. There is **no** bespoke Flux Kustomization for GitHub; it rides the
+existing `apps` layer.
+
+## Why an app, and why namespaced
+
+GitHub's managed resources ship in two flavours: cluster-scoped
+(`repo.github.upbound.io`) and **namespaced** (`repo.github.m.upbound.io`,
+Crossplane v2). We use the **namespaced** variants. That choice is what lets
+`github-config` be a *genuinely isolated* app: a namespaced managed resource can
+be applied by a namespace-scoped `ServiceAccount` with a plain `Role` — no
+`ClusterRoleBinding`, no cluster RBAC. The app's authority to touch GitHub is
+confined to its own namespace and is **not** aggregated into the built-in `edit`
+ClusterRole, so no other tenant inherits it.
+
+## Architecture
+
+Three tiers, each in its conventional place — controller in `controllers/`, its
+CRs one tier later in `infrastructure/`, the workload in `apps/`:
+
+| Piece | Where | Notes |
+|---|---|---|
+| Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only. `provider.defaultActivations: []` keeps unused provider CRDs inactive. Installs the pkg.crossplane.io CRDs. |
+| Provider + activation policy | `k8s/providers/hetzner/infrastructure/crossplane/` | The `Provider` package + `DeploymentRuntimeConfig` + `ManagedResourceActivationPolicy` (namespaced MRDs only). One tier after the controller (needs its CRDs), like Coroot/Flagger CRs. Establishes the namespaced github CRDs. |
+| The `github-config` app (platform scaffolding) | `k8s/bases/apps/github-config/` | namespace, SA, least-privilege `Role`, the namespaced `SecretStore`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). The package is **public**, so the OCIRepository pulls anonymously — no ghcr pull credential. |
+| Desired state (tenant-owned) | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The `Repository`/ruleset/label managed resources, **plus the tenant-specific `github-app-credentials` `ExternalSecret` and the namespaced `ProviderConfig`** (applied by the app's own SA). Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. |
+| Credentials | OpenBao `secret/infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`). **Placeholders are seeded declaratively (only-if-absent) by the vault-config bootstrap Job** — no SOPS; the maintainer overwrites them in place via the OpenBao UI/CLI (see below). The tenant's `ExternalSecret` reads them via the **namespaced `SecretStore`** (auth role `github-config`, scoped to `infrastructure/github/*` only — never the broad cluster store) and renders the provider's JSON credential format. |
+
+Ordering is the normal chain: `apps` `dependsOn` `infrastructure`, so by the
+time the app reconciles, the provider (infrastructure tier) has installed the
+namespaced github CRDs the `ProviderConfig` and managed resources need. On a
+fresh install the app's `Kustomization` retries benignly until those CRDs exist.
+
+The GitHub App credentials are **not** SOPS-managed. The vault-config bootstrap
+Job seeds **placeholder** values into OpenBao (`secret/infrastructure/github/app`)
+**only when the secret is absent**, and the maintainer overwrites them in place
+(below). A later Job re-run never clobbers the real values (the seed is guarded
+by an existence check), and OpenBao is backed up (Raft snapshots → R2 via
+Velero), so the manually-set values are durable without a GitOps source of truth.
+
+## Credential setup (one-time, maintainer)
+
+1. Create a **GitHub App** on the devantler-tech org (Settings → Developer
+   settings → GitHub Apps): permissions *Repository: Administration
+   (read/write), Contents (read)* and *Organization: Administration
+   (read/write)* to start — widen as more resource kinds come under
+   management. Install it on **all repositories** of the org. No webhook.
+2. **Overwrite the placeholders in OpenBao** with the App's real values — the
+   keys already exist (seeded by the vault-config Job), so just set them, e.g.:
+   ```sh
+   bao kv put -mount=secret infrastructure/github/app \
+     app_id="<app id>" installation_id="<installation id>" pem=@app.private-key.pem
+   ```
+   (or via the OpenBao UI). No SOPS, no re-encryption.
+3. The `ExternalSecret` refreshes (≤1h) and renders the provider credential into
+   `github-config`; the app reconciles once `.github` has published its first
+   `v*` artifact. Until the real values are set, the placeholders materialize a
+   (non-working) credential and the provider's auth simply fails — isolated to
+   the leaf `apps` layer, blocking nothing else.
+
+## Adopting an existing repository (Observe-first)
+
+Desired-state manifests live in `devantler-tech/.github` under `deploy/`. Never
+let Crossplane *create* what already exists — **bind** to it:
+
+1. In `.github`, add `deploy/repositories/<name>.yaml`: a namespaced
+   `Repository` (`apiVersion: repo.github.m.upbound.io/v1alpha1`), external-name
+   annotation = the repo name, `managementPolicies: ["Observe"]`,
+   `providerConfigRef: {kind: ProviderConfig, name: default}`. (Namespaced MRs
+   have no `deletionPolicy`; "never delete" is expressed by omitting `Delete`
+   from `managementPolicies`.)
+2. Tag a `v*` release of `.github`; wait for the managed resource to become
+   Ready and inspect what Crossplane observed:
+   `kubectl -n github-config get repository <name> -o yaml` →
+   `status.atProvider` is the live GitHub state.
+3. Backfill `spec.forProvider` from `status.atProvider` (the fields you want to
+   pin: `visibility`, `hasIssues`, `deleteBranchOnMerge`, …).
+4. Promote `managementPolicies` to the full set **except `Delete`**
+   (`["Observe", "Create", "Update", "LateInitialize"]`). From now on GitHub
+   follows the manifest, drift included, but deleting the CR never deletes the
+   real repository.
+
+Roll the org over incrementally: one or two repos per release, watch them
+reconcile, then batch the rest.
+
+## Adding more GitHub resource kinds
+
+1. Activate the namespaced MRD in
+   `k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml`
+   (plural.group form, e.g. `repositoryrulesets.repo.github.m.upbound.io`); add
+   its API group to the app `Role` in `k8s/bases/apps/github-config/rbac.yaml`
+   if it is a new group.
+2. Add the managed resources in `.github`'s `deploy/` — same Observe-first flow
+   where an external object already exists. Each active MRD costs the apiserver
+   ~3 MiB; activate only what is used.
+
+## Safety rails
+
+- **Never `Delete`** — managed resources omit `Delete` from
+  `managementPolicies`, so neither CR deletion nor a Flux prune can ever delete
+  a real GitHub repository.
+- **Observe-first adoption** — a new managed resource for an existing object
+  always starts `managementPolicies: ["Observe"]` (read-only).
+- **Least privilege (RBAC)** — the app SA's authority is a namespaced `Role`
+  over the GitHub MR groups + its own `ProviderConfig`/`ExternalSecret` only,
+  never aggregated into `edit`.
+- **Least privilege (secrets)** — the tenant reads OpenBao through a
+  **namespaced `SecretStore`** backed by a dedicated `github-config` Vault role
+  scoped to `infrastructure/github/*` only; it never touches the shared
+  cluster-scoped `openbao` ClusterSecretStore (the `restrict-tenant-secret-stores`
+  Kyverno policy enforces this for tenant-applied ExternalSecrets).
+- The GitHub App credential is org-scoped and short-lived per request; the
+  provider pod's egress is FQDN-pinned to `api.github.com` by the
+  `crossplane-system` CiliumNetworkPolicy.

k8s/bases/apps/github-config/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - serviceaccount.yaml
+  - rbac.yaml
+  - secretstore.yaml
+  - sync.yaml

k8s/bases/apps/github-config/namespace.yaml

@@ -0,0 +1,15 @@
+# Namespace for the github-config app: where the devantler-tech org's GitHub
+# state is reconciled. No pods run here — it holds the namespaced Crossplane
+# managed resources (repo.github.m.upbound.io/…), the namespaced ProviderConfig,
+# the GitHub App credential, and the Flux source/sync that pull the org's
+# desired state from the devantler-tech/.github repo's published artifact.
+# (A default-deny CiliumNetworkPolicy is auto-generated by the add-default-deny
+# Kyverno policy; no per-app NetworkPolicy is needed since there are no pods.)
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+    pod-security.kubernetes.io/enforce: restricted
+    pod-security.kubernetes.io/enforce-version: latest

k8s/bases/apps/github-config/rbac.yaml

@@ -0,0 +1,69 @@
+# Least-privilege RBAC for the github-config app's ServiceAccount.
+#
+# Deliberately NOT the stock app pattern (a RoleBinding to the built-in `edit`
+# ClusterRole) and deliberately NOT aggregated into `edit`: GitHub managed
+# resources let their holder manipulate the org's real repositories, so that
+# authority must be confined to THIS app — never inherited by every tenant that
+# has an `edit` binding. Instead, a dedicated namespaced Role grants the SA full
+# verbs over the namespaced provider-upjet-github managed-resource API groups,
+# and only within the github-config namespace. Because the managed resources are
+# namespaced (Crossplane v2 `*.github.m.upbound.io`), a plain Role + RoleBinding
+# suffices — no ClusterRole/ClusterRoleBinding, no cluster RBAC. Widen apiGroups
+# here only as new GitHub MR groups are activated in the
+# ManagedResourceActivationPolicy.
+#
+# The .github artifact the SA applies also contains the namespaced ProviderConfig
+# (github.m.upbound.io) and the GitHub App credential ExternalSecret
+# (external-secrets.io), so the SA needs verbs on those two groups too — still
+# confined to this namespace.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: github-config-managed-resources
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+rules:
+  - apiGroups:
+      - repo.github.m.upbound.io
+      - actions.github.m.upbound.io
+      - enterprise.github.m.upbound.io
+      - github.m.upbound.io
+    resources:
+      - "*"
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - external-secrets.io
+    resources:
+      - externalsecrets
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: github-config-managed-resources
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: github-config-managed-resources
+subjects:
+  - kind: ServiceAccount
+    name: github-config
+    namespace: github-config

k8s/bases/apps/github-config/secretstore.yaml

@@ -0,0 +1,34 @@
+# Tenant-scoped, namespaced SecretStore for github-config — provisioned by the
+# platform so the github-config app only has to ship an ExternalSecret in its
+# own deploy/ artifact to read the GitHub App credential. The tenant must NOT
+# use the shared cluster-scoped `openbao` ClusterSecretStore (whose role can read
+# every infra + app path); the `restrict-tenant-secret-stores` Kyverno policy
+# blocks tenant ExternalSecrets from referencing it.
+#
+# Authenticates as the app's own `github-config` ServiceAccount via the dedicated
+# `github-config` OpenBao Kubernetes auth role
+# (k8s/bases/infrastructure/vault-config/job.yaml), which carries only the
+# `infra-github-readonly` policy — read on secret/data/infrastructure/github/*
+# and nothing else. The tenant can therefore reach the GitHub App credential and
+# no other secret.
+apiVersion: external-secrets.io/v1
+kind: SecretStore
+metadata:
+  name: openbao
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+spec:
+  provider:
+    vault:
+      # openbao-active = unsealed Raft leader only; the plain `openbao` Service
+      # also round-robins onto sealed-but-Ready standbys (503 "Vault is sealed").
+      server: "http://openbao-active.openbao.svc.cluster.local:8200"
+      path: "secret"
+      version: "v2"
+      auth:
+        kubernetes:
+          mountPath: "kubernetes"
+          role: "github-config"
+          serviceAccountRef:
+            name: "github-config"

k8s/bases/apps/github-config/serviceaccount.yaml

@@ -0,0 +1,14 @@
+# The github-config app's ServiceAccount. The app's Flux Kustomization
+# (sync.yaml) runs AS this SA, so the manifests in the .github artifact are
+# applied with exactly the namespace-scoped authority granted by rbac.yaml —
+# nothing more. No imagePullSecrets: the github-config/manifests package is
+# public (it carries only non-secret org config), so the OCIRepository pulls it
+# anonymously and this SA runs no pods.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: github-config
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+automountServiceAccountToken: false

k8s/bases/apps/github-config/sync.yaml

@@ -0,0 +1,61 @@
+# The github-config app's source + sync — the same delivery pattern as every
+# other app/tenant (cosign-verified OCI artifact → namespace-scoped
+# Kustomization), but the artifact carries Crossplane managed resources (the
+# org's desired GitHub state) instead of a workload. The desired state lives in
+# devantler-tech/.github under deploy/ and is published as a signed artifact by
+# that repo's cd.yaml on v* tags.
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: OCIRepository
+metadata:
+  name: github-config
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+spec:
+  interval: 1m
+  ref:
+    semver: ">=1.0.0"
+  # .github is an invalid OCI path component (leading dot), so the artifact is
+  # published under the `github-config` name — see devantler-tech/.github cd.yaml.
+  # No secretRef: the package is public (non-secret org config), so it is pulled
+  # anonymously — the tenant needs no GHCR pull credential.
+  url: oci://ghcr.io/devantler-tech/github-config/manifests
+  verify:
+    provider: cosign
+    matchOIDCIdentity:
+      # Keyless-signed by the shared publish-manifests reusable workflow that
+      # .github's cd.yaml calls. Signing runs INSIDE that reusable workflow, so
+      # the cosign OIDC subject is the reusable workflow's path (devantler-tech/
+      # reusable-workflows), NOT .github's own cd.yaml — same convention as the
+      # publish-app.yaml-signed app artifacts.
+      - issuer: '^https://token\.actions\.githubusercontent\.com$'
+        subject: '^https://github\.com/devantler-tech/reusable-workflows/\.github/workflows/publish-manifests\.yaml@.+$'
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: github-config
+  namespace: github-config
+  labels:
+    app.kubernetes.io/managed-by: ksail
+  annotations:
+    ksail.devantler.tech/reconcile-exclude: "true"
+spec:
+  interval: 1m
+  timeout: 5m
+  retryInterval: 2m
+  path: .
+  prune: true
+  wait: true
+  force: true
+  # Runs as the app SA: the GitHub managed resources are applied with only the
+  # namespace-scoped authority in rbac.yaml. This Kustomization is itself applied
+  # by the `apps` Flux Kustomization; it must NOT dependsOn it (deadlock) — it
+  # retries benignly until the provider (infrastructure layer) has established
+  # the namespaced CRDs.
+  serviceAccountName: github-config
+  sourceRef:
+    kind: OCIRepository
+    name: github-config
+  targetNamespace: github-config

k8s/bases/apps/kustomization.yaml

@@ -11,6 +11,11 @@ resources:
   # infra-layer plumbing (OpenBao policies/seeds, NetworkPolicies, Kubescape
   # security exceptions, prod config-map vars) is intentionally left idle.
   # - fleetdm/
+  # Declarative GitHub org management: the devantler-tech/.github repo's deploy/
+  # artifact (Crossplane managed resources) onboarded as a normal app/tenant.
+  # Prod-only in practice (its Crossplane provider is Hetzner-only); the docker
+  # provider deploys no apps, so it is naturally excluded from local clusters.
+  - github-config/
   - headlamp/
   - homepage/
   - umami/