From a6801388179e31788eea771bcce5d7ce3bdcf0ad Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 12 Jun 2026 18:33:48 +0200 Subject: [PATCH 1/6] feat(github): manage GitHub org declaratively via Crossplane provider-upjet-github MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub repos/rulesets/labels become Kubernetes custom resources reconciled by provider-upjet-github, replacing manual GitHub UI administration. - crossplane core HelmRelease (prod-only controllers overlay, chart 2.3.2) with provider.defaultActivations=[] so only explicitly activated MRDs become CRDs (~3MiB apiserver each; the provider ships 77) - dedicated `github` Flux Kustomization (dependsOn: infrastructure) so a GitHub-side stall never holds the infrastructure→apps deploy chain red - GitHub App auth: PushSecret seeds OpenBao from variables-cluster (SOPS), ExternalSecret renders the provider's app_auth JSON credential - Observe-first adoption: first two Repository MRs (platform, ksail) bound by external-name with managementPolicies=[Observe], deletionPolicy=Orphan - docs/github-management.md: architecture, credential setup, adoption flow Co-Authored-By: Claude Fable 5 --- AGENTS.md | 2 +- docs/github-management.md | 75 +++++++++++++++++++ .../infrastructure/vault-config/job.yaml | 20 ++++- .../prod/github-flux-kustomization.yaml | 39 ++++++++++ k8s/clusters/prod/kustomization.yaml | 4 + .../hetzner/github/external-secret.yaml | 37 +++++++++ .../hetzner/github/kustomization.yaml | 28 +++++++ .../managed-resource-activation-policy.yaml | 16 ++++ .../hetzner/github/provider-config.yaml | 15 ++++ k8s/providers/hetzner/github/provider.yaml | 34 +++++++++ .../hetzner/github/repositories/ksail.yaml | 15 ++++ .../github/repositories/kustomization.yaml | 12 +++ .../hetzner/github/repositories/platform.yaml | 17 +++++ .../hetzner/github/seed-github-app.yaml | 37 +++++++++ .../controllers/crossplane/helm-release.yaml | 49 ++++++++++++ .../crossplane/helm-repository.yaml | 7 ++ .../controllers/crossplane/kustomization.yaml | 8 ++ .../controllers/crossplane/namespace.yaml | 4 + .../controllers/crossplane/networkpolicy.yaml | 50 +++++++++++++ .../controllers/kustomization.yaml | 5 ++ 20 files changed, 472 insertions(+), 2 deletions(-) create mode 100644 docs/github-management.md create mode 100644 k8s/clusters/prod/github-flux-kustomization.yaml create mode 100644 k8s/providers/hetzner/github/external-secret.yaml create mode 100644 k8s/providers/hetzner/github/kustomization.yaml create mode 100644 k8s/providers/hetzner/github/managed-resource-activation-policy.yaml create mode 100644 k8s/providers/hetzner/github/provider-config.yaml create mode 100644 k8s/providers/hetzner/github/provider.yaml create mode 100644 k8s/providers/hetzner/github/repositories/ksail.yaml create mode 100644 k8s/providers/hetzner/github/repositories/kustomization.yaml create mode 100644 k8s/providers/hetzner/github/repositories/platform.yaml create mode 100644 k8s/providers/hetzner/github/seed-github-app.yaml create mode 100644 k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml create mode 100644 k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml create mode 100644 k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml create mode 100644 k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml create mode 100644 k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml diff --git a/AGENTS.md b/AGENTS.md index 0c08a4941..859b6b9cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -198,7 +198,7 @@ You **cannot** decrypt existing secrets without the proper Age keys. For local d - **Small, focused changes** — one concern per PR. - **Never commit plaintext secrets** — all secrets must be SOPS-encrypted with the `.enc.yaml` suffix. - **Base files are immutable** — use Kustomize `patches:` in overlays; never edit `k8s/bases/` directly from a provider or cluster overlay. -- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. +- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. Two prod-only side layers hang off `infrastructure` without gating `apps`: `infrastructure-overprovisioning` (apply-only autoscaler buffer) and `github` (declarative GitHub org management via Crossplane — see [`docs/github-management.md`](docs/github-management.md)). ### Infrastructure File Structure Convention diff --git a/docs/github-management.md b/docs/github-management.md new file mode 100644 index 000000000..f1e4af151 --- /dev/null +++ b/docs/github-management.md @@ -0,0 +1,75 @@ +# Declarative GitHub management + +The devantler-tech GitHub org is managed declaratively from this repo: +GitHub repositories (and, over time, rulesets, labels, branch protection, …) +are **Crossplane managed resources** reconciled by +[provider-upjet-github](https://github.com/crossplane-contrib/provider-upjet-github) +against the GitHub API. Change a manifest, merge, and Flux + Crossplane +converge GitHub to it — including reverting out-of-band UI edits. + +## Architecture + +| Piece | Where | Notes | +|---|---|---| +| Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only (external reconcilers must run on exactly one always-on cluster). `provider.defaultActivations: []` keeps unused provider CRDs inactive. | +| Provider + org state | `k8s/providers/hetzner/github/` | Provider package, `ManagedResourceActivationPolicy`, `ProviderConfig`, credentials wiring, and `repositories/` (one file per repo). | +| Flux wiring | `k8s/clusters/prod/github-flux-kustomization.yaml` | Dedicated `github` Flux Kustomization, `dependsOn: infrastructure`. Isolated so a GitHub-side stall never holds `infrastructure`/`apps` red. | +| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by a PushSecret, materialized into `crossplane-system/github-app-credentials` by an ExternalSecret that renders the provider's JSON credential format. | + +The layering follows the repo's Flagger/Coroot convention: the controller +(HelmRelease) ships one layer earlier than the CRs that need its CRDs. + +## Credential setup (one-time, maintainer) + +1. Create a **GitHub App** on the devantler-tech org (Settings → Developer + settings → GitHub Apps): permissions *Repository: Administration + (read/write), Contents (read)* and *Organization: Administration + (read/write)* to start — widen as more resource kinds come under + management. Install it on **all repositories** of the org. No webhook. +2. Add three keys to the **prod** `variables-cluster` SOPS secret + (`k8s/clusters/prod/bootstrap/variables-cluster-secret.enc.yaml`): + `github_app_id`, `github_app_installation_id`, and `github_app_pem` + (the private key, as a normal multiline PEM block). +3. Merge. The PushSecret seeds OpenBao, the ExternalSecret renders the + provider credential, and the `github` Flux Kustomization turns Ready. + Until step 2 lands, that Kustomization sits red — expected, and isolated. + +## Adopting an existing repository (Observe-first) + +Never let Crossplane *create* what already exists — **bind** to it: + +1. Add `repositories/.yaml` copying an existing file: external-name + annotation = the repo name, `managementPolicies: ["Observe"]`, + `deletionPolicy: Orphan`, empty `forProvider`. +2. Merge; wait for the MR to become Ready and inspect what Crossplane + observed: `kubectl get repository.repo.github.upbound.io -o yaml` + → `status.atProvider` is the live GitHub state. +3. Backfill `spec.forProvider` from `status.atProvider` (only the fields you + want to pin — unset fields are left alone by the provider's late + initialization, but be explicit about the ones that matter: + `visibility`, `hasIssues`, `deleteBranchOnMerge`, …). +4. Flip `managementPolicies` to `["*"]`. From now on GitHub follows the + manifest, drift included. Keep `deletionPolicy: Orphan` — deleting the CR + (or a Flux prune) must orphan, never delete, the real repository. + +Roll the org over incrementally: one or two repos per PR, watch them +reconcile, then batch the rest. + +## Adding more GitHub resource kinds + +1. Activate the MRD in + `k8s/providers/hetzner/github/managed-resource-activation-policy.yaml` + (plural.group form, e.g. `repositoryrulesets.repo.github.upbound.io`). +2. Add the CRs — same Observe-first flow where an external object already + exists. Each active MRD costs the apiserver ~3 MiB; activate only what is + used. + +## Safety rails + +- **`deletionPolicy: Orphan` on every repository MR, always** — even fully + managed ones. CR deletion must never cascade to GitHub. +- **Observe-first adoption** — a new MR for an existing object never starts + with write policies. +- The GitHub App credential is org-scoped and short-lived per request; the + provider pod's egress is FQDN-pinned to `api.github.com` by the + `crossplane-system` CiliumNetworkPolicy. diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 6e4f4e079..b5eb57908 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -402,6 +402,18 @@ spec: } POLICY + # Read the GitHub App credential for provider-upjet-github + # (declarative GitHub management). Consumed by the + # github-app-credentials ExternalSecret in crossplane-system via + # the `external-secrets` ClusterSecretStore role; seeded from + # variables-cluster by the seed-github-app PushSecret (see + # k8s/providers/hetzner/github/). + bao policy write infra-github-readonly - <<'POLICY' + path "secret/data/infrastructure/github/*" { + capabilities = ["read"] + } + POLICY + bao policy write app-fleetdm-readonly - <<'POLICY' path "secret/data/apps/fleetdm/*" { capabilities = ["read"] @@ -506,6 +518,9 @@ spec: path "secret/data/infrastructure/ghcr/*" { capabilities = ["create", "update", "read"] } + path "secret/data/infrastructure/github/*" { + capabilities = ["create", "update", "read"] + } path "secret/data/apps/fleetdm/*" { capabilities = ["create", "update", "read"] } @@ -542,6 +557,9 @@ spec: path "secret/metadata/infrastructure/ghcr/*" { capabilities = ["create", "update", "read"] } + path "secret/metadata/infrastructure/github/*" { + capabilities = ["create", "update", "read"] + } path "secret/metadata/apps/fleetdm/*" { capabilities = ["create", "update", "read"] } @@ -568,7 +586,7 @@ spec: bao write auth/kubernetes/role/external-secrets \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets \ - policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,infra-oidc-readonly,vault-seed-write \ + policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,infra-github-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,infra-oidc-readonly,vault-seed-write \ ttl=1h # Dedicated role for the fleetdm VaultDynamicSecret generator. diff --git a/k8s/clusters/prod/github-flux-kustomization.yaml b/k8s/clusters/prod/github-flux-kustomization.yaml new file mode 100644 index 000000000..08b438a53 --- /dev/null +++ b/k8s/clusters/prod/github-flux-kustomization.yaml @@ -0,0 +1,39 @@ +# Prod-only: reconciles the devantler-tech GitHub org's declarative state +# (providers/hetzner/github/ — Crossplane provider-upjet-github package, +# credentials wiring, and the Repository/… managed resources) via its OWN +# Flux Kustomization instead of the `infrastructure` layer. +# +# Why it cannot ride along in `infrastructure`: GitHub org state is orthogonal +# to cluster workloads, but `infrastructure` reconciles with `wait: true` and +# gates `apps` (and with it every merge-queue / tag deploy). A GitHub-side +# stall — credentials not yet seeded in variables-cluster, a GitHub API +# outage, a misconfigured managed resource — must only ever hold THIS +# Kustomization red (same isolation reasoning as +# overprovisioning-flux-kustomization.yaml, though this layer keeps +# `wait: true`: steady-state, everything in it is expected to be Ready). +# +# Expected on a fresh install: the first reconcile fails with unknown-kind +# errors until the Provider package installs and the activation policy +# creates the github.upbound.io CRDs (~1-2 min), then converges on retry. +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: github + namespace: flux-system +spec: + interval: 60m + timeout: 10m + retryInterval: 2m + path: providers/hetzner/github + sourceRef: + kind: OCIRepository + name: flux-system + # After `infrastructure` so the Crossplane core HelmRelease (controllers + # layer), the openbao ClusterSecretStore and the vault-config policies it + # needs are all in place first. + dependsOn: + - name: infrastructure + wait: true + prune: true + force: true diff --git a/k8s/clusters/prod/kustomization.yaml b/k8s/clusters/prod/kustomization.yaml index bdc176135..2d77d90c0 100644 --- a/k8s/clusters/prod/kustomization.yaml +++ b/k8s/clusters/prod/kustomization.yaml @@ -7,6 +7,10 @@ resources: # `infrastructure` layer (its pause pods are expected-Pending at the # autoscaler ceiling and must not hold deploys red) — see the file header. - overprovisioning-flux-kustomization.yaml + # Prod-only: declarative GitHub org management (Crossplane managed + # resources), isolated in its own Flux Kustomization so a GitHub-side stall + # never holds the infrastructure→apps deploy chain red — see the file header. + - github-flux-kustomization.yaml # Template inputs for this cluster. cluster-meta drives the replacements # below; it is annotated local-config and never applied to the cluster. diff --git a/k8s/providers/hetzner/github/external-secret.yaml b/k8s/providers/hetzner/github/external-secret.yaml new file mode 100644 index 000000000..31b8452c2 --- /dev/null +++ b/k8s/providers/hetzner/github/external-secret.yaml @@ -0,0 +1,37 @@ +# Materializes the provider-upjet-github credentials Secret from OpenBao. +# The provider expects ONE JSON document ({"app_auth":[…],"owner":…}) whose +# pem_file value carries \n-escaped newlines — the template builds it from the +# three plain keys seeded by seed-github-app.yaml (toJson does the escaping), +# so the PEM can live in OpenBao/SOPS as a normal multiline string. +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: github-app-credentials + namespace: crossplane-system +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: github-app-credentials + creationPolicy: Owner + template: + engineVersion: v2 + type: Opaque + data: + credentials: '{"app_auth":[{"id":"{{ .app_id }}","installation_id":"{{ .installation_id }}","pem_file":{{ .pem | toJson }}}],"owner":"devantler-tech"}' + data: + - secretKey: app_id + remoteRef: + key: infrastructure/github/app + property: app_id + - secretKey: installation_id + remoteRef: + key: infrastructure/github/app + property: installation_id + - secretKey: pem + remoteRef: + key: infrastructure/github/app + property: pem diff --git a/k8s/providers/hetzner/github/kustomization.yaml b/k8s/providers/hetzner/github/kustomization.yaml new file mode 100644 index 000000000..93d854113 --- /dev/null +++ b/k8s/providers/hetzner/github/kustomization.yaml @@ -0,0 +1,28 @@ +# Declarative GitHub management — the devantler-tech org's GitHub state +# (repositories, rulesets, labels, …) as Crossplane managed resources, +# reconciled by provider-upjet-github. See docs/github-management.md. +# +# This layer is applied by its OWN Flux Kustomization (`github`, see +# k8s/clusters/prod/github-flux-kustomization.yaml), NOT by the health-gated +# `infrastructure` layer: GitHub org state is orthogonal to cluster workloads, +# and a stalled GitHub reconcile (e.g. credentials not yet seeded, GitHub API +# outage) must never hold `infrastructure`/`apps` — and with them every +# merge-queue / tag deploy — red. +# +# Layering follows the Flagger/Coroot convention (CRDs ship one layer earlier +# than the CRs that use them): the Crossplane core HelmRelease lives in +# ../infrastructure/controllers/crossplane/, this layer installs the provider +# package and the managed resources. On a FRESH install the first reconcile of +# this layer fails benignly (the ProviderConfig/Repository CRDs only exist +# once the Provider package and the activation policy below have done their +# work) and converges on the next retries. +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - provider.yaml + - managed-resource-activation-policy.yaml + - provider-config.yaml + - seed-github-app.yaml + - external-secret.yaml + - repositories/ diff --git a/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml b/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml new file mode 100644 index 000000000..263e2d864 --- /dev/null +++ b/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml @@ -0,0 +1,16 @@ +# provider-upjet-github declares SafeStart, so its 77 ManagedResourceDefinitions +# install Inactive; only the ones named here become CRDs (each active MRD costs +# the apiserver ~3MiB). The chart's activate-everything default MRAP is disabled +# via `provider.defaultActivations: []` in the crossplane HelmRelease. Extend +# this list as more of the org's GitHub surface comes under management. +apiVersion: apiextensions.crossplane.io/v1alpha1 +kind: ManagedResourceActivationPolicy +metadata: + name: github +spec: + activate: + - repositories.repo.github.upbound.io + - defaultbranches.repo.github.upbound.io + - branchprotections.repo.github.upbound.io + - repositoryrulesets.repo.github.upbound.io + - issuelabels.repo.github.upbound.io diff --git a/k8s/providers/hetzner/github/provider-config.yaml b/k8s/providers/hetzner/github/provider-config.yaml new file mode 100644 index 000000000..1c24e6c3b --- /dev/null +++ b/k8s/providers/hetzner/github/provider-config.yaml @@ -0,0 +1,15 @@ +# Authenticates provider-upjet-github against the devantler-tech org as a +# GitHub App (not a PAT: short-lived installation tokens, org-scoped, survives +# personal-account changes). The referenced Secret is materialized by +# external-secret.yaml from OpenBao. +apiVersion: github.upbound.io/v1beta1 +kind: ProviderConfig +metadata: + name: default +spec: + credentials: + source: Secret + secretRef: + namespace: crossplane-system + name: github-app-credentials + key: credentials diff --git a/k8s/providers/hetzner/github/provider.yaml b/k8s/providers/hetzner/github/provider.yaml new file mode 100644 index 000000000..0f9b61cdb --- /dev/null +++ b/k8s/providers/hetzner/github/provider.yaml @@ -0,0 +1,34 @@ +# provider-upjet-github — Upjet-generated Crossplane provider wrapping the +# integrations/github Terraform provider. ghcr.io is the project's publish +# registry (xpkg.crossplane.io merely proxies it) and is already the trust +# root for every first-party image in this cluster. +apiVersion: pkg.crossplane.io/v1 +kind: Provider +metadata: + name: provider-upjet-github +spec: + package: ghcr.io/crossplane-contrib/provider-upjet-github:v0.19.1 + runtimeConfigRef: + name: provider-upjet-github +--- +# Bounded resources for the provider runtime pod (created by Crossplane, so +# the HelmRelease values can't reach it). The empty selector is the documented +# placeholder — Crossplane fills in the real match labels. +apiVersion: pkg.crossplane.io/v1beta1 +kind: DeploymentRuntimeConfig +metadata: + name: provider-upjet-github +spec: + deploymentTemplate: + spec: + selector: {} + template: + spec: + containers: + - name: package-runtime + resources: + requests: + cpu: 10m + memory: 128Mi + limits: + memory: 512Mi diff --git a/k8s/providers/hetzner/github/repositories/ksail.yaml b/k8s/providers/hetzner/github/repositories/ksail.yaml new file mode 100644 index 000000000..ee6772fe4 --- /dev/null +++ b/k8s/providers/hetzner/github/repositories/ksail.yaml @@ -0,0 +1,15 @@ +apiVersion: repo.github.upbound.io/v1alpha1 +kind: Repository +metadata: + name: ksail + annotations: + # Binds this CR to the EXISTING devantler-tech/ksail repository + # (owner comes from the ProviderConfig credentials). + crossplane.io/external-name: ksail +spec: + # Observe-only while adopting — see repositories/kustomization.yaml. + managementPolicies: ["Observe"] + deletionPolicy: Orphan + forProvider: {} + providerConfigRef: + name: default diff --git a/k8s/providers/hetzner/github/repositories/kustomization.yaml b/k8s/providers/hetzner/github/repositories/kustomization.yaml new file mode 100644 index 000000000..4b597104c --- /dev/null +++ b/k8s/providers/hetzner/github/repositories/kustomization.yaml @@ -0,0 +1,12 @@ +# One file per managed repository. Adoption is Observe-first (the Crossplane +# import flow): every repo starts with managementPolicies: ["Observe"] — +# read-only, zero risk to the live repo — and is flipped to ["*"] only after +# its spec.forProvider has been backfilled from status.atProvider. Keep +# deletionPolicy: Orphan on every repo even when fully managed, so no Flux +# prune or CR mishap can ever delete a real GitHub repository. +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - platform.yaml + - ksail.yaml diff --git a/k8s/providers/hetzner/github/repositories/platform.yaml b/k8s/providers/hetzner/github/repositories/platform.yaml new file mode 100644 index 000000000..90c8c09a4 --- /dev/null +++ b/k8s/providers/hetzner/github/repositories/platform.yaml @@ -0,0 +1,17 @@ +apiVersion: repo.github.upbound.io/v1alpha1 +kind: Repository +metadata: + name: platform + annotations: + # Binds this CR to the EXISTING devantler-tech/platform repository + # (owner comes from the ProviderConfig credentials). + crossplane.io/external-name: platform +spec: + # Observe-only while adopting: Crossplane fills status.atProvider and never + # writes to GitHub. Flip to ["*"] after backfilling forProvider — see + # repositories/kustomization.yaml and docs/github-management.md. + managementPolicies: ["Observe"] + deletionPolicy: Orphan + forProvider: {} + providerConfigRef: + name: default diff --git a/k8s/providers/hetzner/github/seed-github-app.yaml b/k8s/providers/hetzner/github/seed-github-app.yaml new file mode 100644 index 000000000..239bd768d --- /dev/null +++ b/k8s/providers/hetzner/github/seed-github-app.yaml @@ -0,0 +1,37 @@ +# PushSecret for the GitHub App credentials used by provider-upjet-github. +# Same durable-seed pattern as seed-hcloud: re-pushing hourly from the SOPS +# source (variables-cluster) is what re-seeds the keys automatically after +# OpenBao data loss. Lives in THIS layer (not infrastructure/vault-seed/) so +# that, until the maintainer adds the github_app_* keys to the prod +# variables-cluster secret, the resulting InProgress stall holds only the +# isolated `github` Flux Kustomization red — never `infrastructure`. +--- +apiVersion: external-secrets.io/v1alpha1 +kind: PushSecret +metadata: + name: seed-github-app + namespace: flux-system +spec: + refreshInterval: 1h + secretStoreRefs: + - name: openbao + kind: ClusterSecretStore + selector: + secret: + name: variables-cluster + data: + - match: + secretKey: github_app_id + remoteRef: + remoteKey: infrastructure/github/app + property: app_id + - match: + secretKey: github_app_installation_id + remoteRef: + remoteKey: infrastructure/github/app + property: installation_id + - match: + secretKey: github_app_pem + remoteRef: + remoteKey: infrastructure/github/app + property: pem diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml new file mode 100644 index 000000000..7ba73ddbe --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml @@ -0,0 +1,49 @@ +# Crossplane core — the control plane for managing EXTERNAL resources (GitHub +# org/repo state, see ../../../github/) as Kubernetes custom resources. Prod-only: +# external-state reconcilers must run on exactly one always-on cluster, or two +# clusters would race to converge the same GitHub org (same reasoning as +# external-dns living in this overlay). +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: crossplane + namespace: crossplane-system + labels: + helm.toolkit.fluxcd.io/crds: enabled + helm.toolkit.fluxcd.io/remediation: enabled +spec: + interval: 10m + timeout: 10m + install: + crds: CreateReplace + upgrade: + crds: CreateReplace + chart: + spec: + chart: crossplane + version: 2.3.2 + sourceRef: + kind: HelmRepository + name: crossplane + # https://github.com/crossplane/crossplane/blob/main/cluster/charts/crossplane/values.yaml + values: + provider: + # Providers in this cluster declare SafeStart, so their ManagedResource- + # Definitions install Inactive. An empty list here stops the chart from + # creating the "default" activate-everything MRAP — each MRD costs the + # apiserver ~3MiB once active, and provider-upjet-github alone ships 77. + # The MRDs we actually use are activated by an explicit + # ManagedResourceActivationPolicy in providers/hetzner/github/. + defaultActivations: [] + resourcesCrossplane: + requests: + cpu: 25m + memory: 256Mi + limits: + memory: 512Mi + resourcesRBACManager: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml new file mode 100644 index 000000000..961f20061 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml @@ -0,0 +1,7 @@ +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: crossplane + namespace: crossplane-system +spec: + url: https://charts.crossplane.io/stable diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml new file mode 100644 index 000000000..e6c1f2415 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml @@ -0,0 +1,8 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - helm-repository.yaml + - helm-release.yaml + - networkpolicy.yaml diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml new file mode 100644 index 000000000..8e6246cea --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: crossplane-system diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml new file mode 100644 index 000000000..3453a63b0 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml @@ -0,0 +1,50 @@ +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: allow-crossplane + namespace: crossplane-system +spec: + # Namespace-wide: covers the Crossplane core pod (package manager pulls + # provider OCI packages itself, in-pod — unlike runtime images, which the + # kubelet pulls at node level) AND the provider pods Crossplane creates here + # (provider-upjet-github talks to the GitHub API). + endpointSelector: {} + egress: + # Kube API for watching/updating managed resources and leader election. + - toEntities: + - kube-apiserver + # Pinned by FQDN rather than world:443 so a compromised pod cannot reach + # arbitrary external services: + # ghcr.io + pkg-containers.githubusercontent.com — provider package + # pulls by the core package manager (ghcr blob fetches redirect to + # pkg-containers.githubusercontent.com). + # api.github.com — the GitHub REST/GraphQL API used by + # provider-upjet-github to reconcile org/repo state. + - toFQDNs: + - matchName: "ghcr.io" + - matchName: "pkg-containers.githubusercontent.com" + - matchName: "api.github.com" + toPorts: + - ports: + - port: "443" + protocol: TCP + # DNS resolution. rules.dns activates Cilium's L7 DNS proxy so the + # toFQDNs rule above can be enforced; the visibility list is restricted + # to the names these pods actually need to resolve, closing off DNS + # tunneling exfil (same pattern as external-dns). + - toEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: kube-system + k8s-app: kube-dns + toPorts: + - ports: + - port: "53" + protocol: UDP + - port: "53" + protocol: TCP + rules: + dns: + - matchName: "ghcr.io" + - matchName: "pkg-containers.githubusercontent.com" + - matchName: "api.github.com" + - matchPattern: "*.cluster.local" diff --git a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml index e67b35204..35ddd4637 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml @@ -13,6 +13,11 @@ resources: # so the policy lives here, not in the base — see cilium/ for the rationale. - cilium/ - coredns/ + # Prod-only: Crossplane core for declarative GitHub management — external + # reconcilers must run on exactly one always-on cluster. The provider + # package + managed resources live in the dedicated `github` layer + # (../../github/), applied by its own Flux Kustomization. + - crossplane/ - external-dns/ - flux-instance/ - hcloud-ccm/ From dfd246e12c9839e2c22fd2d619f070f24e01bea9 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sat, 13 Jun 2026 11:05:26 +0200 Subject: [PATCH 2/6] refactor(github): deliver GitHub state as the .github tenant via namespaced MRs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rework: instead of holding cluster-scoped Repository CRs in the platform tree, the org's desired GitHub state lives in devantler-tech/.github under deploy/ and is onboarded as an isolated tenant. - switch the activation policy + ProviderConfig + managed resources to the Crossplane v2 NAMESPACED variants (repo.github.m.upbound.io), so a namespace-scoped tenant SA with a plain Role can apply them — no cluster RBAC, no ClusterRoleBinding - add the github-config tenant: namespace, least-privilege SA + Role (NOT aggregated into edit, so other tenants don't inherit GitHub powers), ghcr + GitHub App credential ExternalSecrets, namespaced ProviderConfig, and a cosign-verified OCIRepository + Kustomization that pull .github's artifact and apply it as the tenant SA - engine (provider package + activation policy + credential seed) stays platform-side; desired-state managed resources move to .github/deploy/ - remove the cluster-scoped repositories/ CRs (now in .github) - docs rewritten for the tenant + namespaced architecture Co-Authored-By: Claude Fable 5 --- docs/github-management.md | 103 ++++++++++++------ .../hetzner/github/external-secret.yaml | 2 +- .../github/ghcr-auth-externalsecret.yaml | 35 ++++++ .../hetzner/github/kustomization.yaml | 48 +++++--- .../managed-resource-activation-policy.yaml | 25 +++-- k8s/providers/hetzner/github/namespace.yaml | 13 +++ .../hetzner/github/provider-config.yaml | 13 ++- k8s/providers/hetzner/github/rbac.yaml | 51 +++++++++ .../hetzner/github/repositories/ksail.yaml | 15 --- .../github/repositories/kustomization.yaml | 12 -- .../hetzner/github/repositories/platform.yaml | 17 --- .../hetzner/github/serviceaccount.yaml | 14 +++ k8s/providers/hetzner/github/sync.yaml | 56 ++++++++++ 13 files changed, 293 insertions(+), 111 deletions(-) create mode 100644 k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml create mode 100644 k8s/providers/hetzner/github/namespace.yaml create mode 100644 k8s/providers/hetzner/github/rbac.yaml delete mode 100644 k8s/providers/hetzner/github/repositories/ksail.yaml delete mode 100644 k8s/providers/hetzner/github/repositories/kustomization.yaml delete mode 100644 k8s/providers/hetzner/github/repositories/platform.yaml create mode 100644 k8s/providers/hetzner/github/serviceaccount.yaml create mode 100644 k8s/providers/hetzner/github/sync.yaml diff --git a/docs/github-management.md b/docs/github-management.md index f1e4af151..348ca6e6d 100644 --- a/docs/github-management.md +++ b/docs/github-management.md @@ -1,23 +1,45 @@ # Declarative GitHub management -The devantler-tech GitHub org is managed declaratively from this repo: -GitHub repositories (and, over time, rulesets, labels, branch protection, …) -are **Crossplane managed resources** reconciled by +The devantler-tech GitHub org is managed declaratively as GitOps: GitHub +repositories (and, over time, rulesets, labels, branch protection, …) are +**Crossplane managed resources** reconciled by [provider-upjet-github](https://github.com/crossplane-contrib/provider-upjet-github) -against the GitHub API. Change a manifest, merge, and Flux + Crossplane -converge GitHub to it — including reverting out-of-band UI edits. +against the GitHub API. Change a manifest, merge, and Flux + Crossplane converge +GitHub to it — including reverting out-of-band UI edits. + +The desired state does **not** live in this repo. It lives in +[`devantler-tech/.github`](https://github.com/devantler-tech/.github) under +`deploy/`, which is onboarded here as a **tenant**: the platform pulls its +cosign-signed OCI artifact and applies it, exactly like an application tenant — +only the artifact carries managed resources instead of a workload. + +## Why a tenant, and why namespaced + +GitHub's managed resources ship in two flavours: cluster-scoped +(`repo.github.upbound.io`) and **namespaced** (`repo.github.m.upbound.io`, +Crossplane v2). We use the **namespaced** variants. That choice is what lets +`.github` be a *genuinely isolated* tenant: a namespaced managed resource can be +applied by a namespace-scoped `ServiceAccount` with a plain `Role` — no +`ClusterRoleBinding`, no cluster RBAC. The tenant's authority to touch GitHub is +confined to its own namespace and is **not** aggregated into the built-in `edit` +ClusterRole, so no other tenant inherits it. ## Architecture | Piece | Where | Notes | |---|---|---| | Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only (external reconcilers must run on exactly one always-on cluster). `provider.defaultActivations: []` keeps unused provider CRDs inactive. | -| Provider + org state | `k8s/providers/hetzner/github/` | Provider package, `ManagedResourceActivationPolicy`, `ProviderConfig`, credentials wiring, and `repositories/` (one file per repo). | +| Engine | `k8s/providers/hetzner/github/` (`provider.yaml`, `managed-resource-activation-policy.yaml`, `seed-github-app.yaml`) | The provider package + runtime; activates only the **namespaced** MRDs we use; seeds the GitHub App credential into OpenBao. | +| Tenant wiring | `k8s/providers/hetzner/github/` (`namespace.yaml`, `serviceaccount.yaml`, `rbac.yaml`, `ghcr-auth-externalsecret.yaml`, `external-secret.yaml`, `provider-config.yaml`, `sync.yaml`) | The `github-config` namespace, its least-privilege SA + Role, the GitHub App + ghcr credentials, the namespaced `ProviderConfig`, and the OCIRepository + Kustomization that pull and apply `.github`'s artifact as the tenant SA. | +| Desired state | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The actual `Repository`/ruleset/label managed resources. Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags. | | Flux wiring | `k8s/clusters/prod/github-flux-kustomization.yaml` | Dedicated `github` Flux Kustomization, `dependsOn: infrastructure`. Isolated so a GitHub-side stall never holds `infrastructure`/`apps` red. | -| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by a PushSecret, materialized into `crossplane-system/github-app-credentials` by an ExternalSecret that renders the provider's JSON credential format. | +| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by a PushSecret, materialized into `github-config/github-app-credentials` by an ExternalSecret that renders the provider's JSON credential format. | -The layering follows the repo's Flagger/Coroot convention: the controller -(HelmRelease) ships one layer earlier than the CRs that need its CRDs. +> Why the tenant wiring lives in the `github` layer (not `k8s/bases/apps/`): +> it is prod-only and coupled to the Crossplane provider, and we want it +> isolated from the `apps` deploy chain. It is still a tenant *in substance* — +> dedicated namespace, isolated least-privilege SA, cosign-verified OCI delivery +> from its own repo, own pruning Kustomization. ## Credential setup (one-time, maintainer) @@ -30,46 +52,55 @@ The layering follows the repo's Flagger/Coroot convention: the controller (`k8s/clusters/prod/bootstrap/variables-cluster-secret.enc.yaml`): `github_app_id`, `github_app_installation_id`, and `github_app_pem` (the private key, as a normal multiline PEM block). -3. Merge. The PushSecret seeds OpenBao, the ExternalSecret renders the - provider credential, and the `github` Flux Kustomization turns Ready. - Until step 2 lands, that Kustomization sits red — expected, and isolated. +3. Merge. The PushSecret seeds OpenBao, the ExternalSecret renders the provider + credential into `github-config`, and the `github` Flux Kustomization turns + Ready once `.github` has published its first `v*` artifact. Until then that + Kustomization sits red — expected, and isolated. ## Adopting an existing repository (Observe-first) -Never let Crossplane *create* what already exists — **bind** to it: +Desired-state manifests live in `devantler-tech/.github` under `deploy/`. Never +let Crossplane *create* what already exists — **bind** to it: -1. Add `repositories/.yaml` copying an existing file: external-name +1. In `.github`, add `deploy/repositories/.yaml`: a namespaced + `Repository` (`apiVersion: repo.github.m.upbound.io/v1alpha1`), external-name annotation = the repo name, `managementPolicies: ["Observe"]`, - `deletionPolicy: Orphan`, empty `forProvider`. -2. Merge; wait for the MR to become Ready and inspect what Crossplane - observed: `kubectl get repository.repo.github.upbound.io -o yaml` - → `status.atProvider` is the live GitHub state. -3. Backfill `spec.forProvider` from `status.atProvider` (only the fields you - want to pin — unset fields are left alone by the provider's late - initialization, but be explicit about the ones that matter: - `visibility`, `hasIssues`, `deleteBranchOnMerge`, …). -4. Flip `managementPolicies` to `["*"]`. From now on GitHub follows the - manifest, drift included. Keep `deletionPolicy: Orphan` — deleting the CR - (or a Flux prune) must orphan, never delete, the real repository. - -Roll the org over incrementally: one or two repos per PR, watch them + `providerConfigRef: {kind: ProviderConfig, name: default}`. (Namespaced MRs + have no `deletionPolicy`; "never delete" is expressed by omitting `Delete` + from `managementPolicies`.) +2. Tag a `v*` release of `.github`; wait for the managed resource to become + Ready and inspect what Crossplane observed: + `kubectl -n github-config get repository -o yaml` → + `status.atProvider` is the live GitHub state. +3. Backfill `spec.forProvider` from `status.atProvider` (the fields you want to + pin: `visibility`, `hasIssues`, `deleteBranchOnMerge`, …). +4. Promote `managementPolicies` to the full set **except `Delete`** + (`["Observe", "Create", "Update", "LateInitialize"]`). From now on GitHub + follows the manifest, drift included, but deleting the CR never deletes the + real repository. + +Roll the org over incrementally: one or two repos per release, watch them reconcile, then batch the rest. ## Adding more GitHub resource kinds -1. Activate the MRD in +1. Activate the namespaced MRD in `k8s/providers/hetzner/github/managed-resource-activation-policy.yaml` - (plural.group form, e.g. `repositoryrulesets.repo.github.upbound.io`). -2. Add the CRs — same Observe-first flow where an external object already - exists. Each active MRD costs the apiserver ~3 MiB; activate only what is - used. + (plural.group form, e.g. `repositoryrulesets.repo.github.m.upbound.io`); add + its API group to the tenant `Role` in `rbac.yaml` if it is a new group. +2. Add the managed resources in `.github`'s `deploy/` — same Observe-first flow + where an external object already exists. Each active MRD costs the apiserver + ~3 MiB; activate only what is used. ## Safety rails -- **`deletionPolicy: Orphan` on every repository MR, always** — even fully - managed ones. CR deletion must never cascade to GitHub. -- **Observe-first adoption** — a new MR for an existing object never starts - with write policies. +- **Never `Delete`** — managed resources omit `Delete` from + `managementPolicies`, so neither CR deletion nor a Flux prune can ever delete + a real GitHub repository. +- **Observe-first adoption** — a new managed resource for an existing object + always starts `managementPolicies: ["Observe"]` (read-only). +- **Least privilege** — the tenant SA's authority is a namespaced `Role` over + the GitHub MR API groups only, never aggregated into `edit`. - The GitHub App credential is org-scoped and short-lived per request; the provider pod's egress is FQDN-pinned to `api.github.com` by the `crossplane-system` CiliumNetworkPolicy. diff --git a/k8s/providers/hetzner/github/external-secret.yaml b/k8s/providers/hetzner/github/external-secret.yaml index 31b8452c2..938e20969 100644 --- a/k8s/providers/hetzner/github/external-secret.yaml +++ b/k8s/providers/hetzner/github/external-secret.yaml @@ -8,7 +8,7 @@ apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: github-app-credentials - namespace: crossplane-system + namespace: github-config spec: refreshInterval: 1h secretStoreRef: diff --git a/k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml b/k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml new file mode 100644 index 000000000..c5e1d3231 --- /dev/null +++ b/k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml @@ -0,0 +1,35 @@ +# GHCR image-pull secret for the .github tenant, sourced from OpenBao (same +# shape and path as every other tenant's ghcr-auth). Consumed by the tenant's +# OCIRepository (secretRef) and ServiceAccount (imagePullSecrets) to pull the +# cosign-signed manifests artifact published by devantler-tech/.github. +# +# PLATFORM-side resource: reconciled by flux-system (this layer), never by the +# tenant ServiceAccount, so it uses the shared cluster-scoped `openbao` +# ClusterSecretStore reading the platform path infrastructure/ghcr/auth — the +# `restrict-tenant-secret-stores` Kyverno policy carves out flux-system-applied +# ExternalSecrets. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: ghcr-auth + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: ghcr-auth + creationPolicy: Owner + template: + engineVersion: v2 + type: kubernetes.io/dockerconfigjson + data: + .dockerconfigjson: "{{ .dockerconfigjson }}" + data: + - secretKey: dockerconfigjson + remoteRef: + key: infrastructure/ghcr/auth + property: dockerconfigjson diff --git a/k8s/providers/hetzner/github/kustomization.yaml b/k8s/providers/hetzner/github/kustomization.yaml index 93d854113..45879791c 100644 --- a/k8s/providers/hetzner/github/kustomization.yaml +++ b/k8s/providers/hetzner/github/kustomization.yaml @@ -1,28 +1,44 @@ # Declarative GitHub management — the devantler-tech org's GitHub state -# (repositories, rulesets, labels, …) as Crossplane managed resources, -# reconciled by provider-upjet-github. See docs/github-management.md. +# (repositories, rulesets, labels, …) as Crossplane managed resources. This +# layer holds the ENGINE + the .github TENANT wiring; the desired-state managed +# resources themselves live in the devantler-tech/.github repo under deploy/ and +# are pulled in as a cosign-signed OCI artifact (sync.yaml). See +# docs/github-management.md. # -# This layer is applied by its OWN Flux Kustomization (`github`, see -# k8s/clusters/prod/github-flux-kustomization.yaml), NOT by the health-gated +# Applied by its OWN Flux Kustomization (`github`, see +# k8s/clusters/prod/github-flux-kustomization.yaml), NOT the health-gated # `infrastructure` layer: GitHub org state is orthogonal to cluster workloads, -# and a stalled GitHub reconcile (e.g. credentials not yet seeded, GitHub API -# outage) must never hold `infrastructure`/`apps` — and with them every -# merge-queue / tag deploy — red. +# and a stalled GitHub reconcile (credentials not seeded, GitHub API outage, +# .github not yet published) must never hold `infrastructure`/`apps` — and with +# them every merge-queue / tag deploy — red. # -# Layering follows the Flagger/Coroot convention (CRDs ship one layer earlier -# than the CRs that use them): the Crossplane core HelmRelease lives in -# ../infrastructure/controllers/crossplane/, this layer installs the provider -# package and the managed resources. On a FRESH install the first reconcile of -# this layer fails benignly (the ProviderConfig/Repository CRDs only exist -# once the Provider package and the activation policy below have done their -# work) and converges on the next retries. +# Two parts: +# ENGINE (cluster infra) — provider.yaml (the provider package + runtime), +# managed-resource-activation-policy.yaml (activate only the namespaced MRDs +# we use), seed-github-app.yaml (seed the GitHub App credential into OpenBao). +# TENANT (.github) — namespace, ServiceAccount + least-privilege Role/Binding, +# ghcr pull + GitHub App credential ExternalSecrets, the namespaced +# ProviderConfig, and the OCIRepository + Kustomization that pull and apply +# the org's managed resources as the tenant SA. +# +# Crossplane core (the controller) lives one layer earlier in +# ../infrastructure/controllers/crossplane/. On a FRESH install the first +# reconcile here fails benignly (the namespaced CRDs only exist once the Provider +# package + activation policy have run; the OCI artifact only exists once .github +# has published a v* release) and converges on the next retries. --- apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: + # Engine (cluster infra) - provider.yaml - managed-resource-activation-policy.yaml - - provider-config.yaml - seed-github-app.yaml + # Tenant (.github) + - namespace.yaml + - serviceaccount.yaml + - rbac.yaml + - ghcr-auth-externalsecret.yaml - external-secret.yaml - - repositories/ + - provider-config.yaml + - sync.yaml diff --git a/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml b/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml index 263e2d864..1035774b4 100644 --- a/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml +++ b/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml @@ -1,16 +1,23 @@ -# provider-upjet-github declares SafeStart, so its 77 ManagedResourceDefinitions +# provider-upjet-github declares SafeStart, so its ManagedResourceDefinitions # install Inactive; only the ones named here become CRDs (each active MRD costs -# the apiserver ~3MiB). The chart's activate-everything default MRAP is disabled -# via `provider.defaultActivations: []` in the crossplane HelmRelease. Extend -# this list as more of the org's GitHub surface comes under management. +# the apiserver ~3MiB, and the provider ships ~78). The chart's +# activate-everything default MRAP is disabled via +# `provider.defaultActivations: []` in the crossplane HelmRelease. +# +# We activate the NAMESPACED (Crossplane v2, `*.github.m.upbound.io`) variants +# only — namespaced managed resources can be applied by a namespace-scoped +# tenant ServiceAccount with a plain Role (no cluster RBAC), which is what makes +# the .github tenant a genuinely isolated tenant. Extend this list as more of +# the org's GitHub surface comes under management (org rulesets, Actions +# secrets, …) — see docs/github-management.md. apiVersion: apiextensions.crossplane.io/v1alpha1 kind: ManagedResourceActivationPolicy metadata: name: github spec: activate: - - repositories.repo.github.upbound.io - - defaultbranches.repo.github.upbound.io - - branchprotections.repo.github.upbound.io - - repositoryrulesets.repo.github.upbound.io - - issuelabels.repo.github.upbound.io + - repositories.repo.github.m.upbound.io + - defaultbranches.repo.github.m.upbound.io + - branchprotections.repo.github.m.upbound.io + - repositoryrulesets.repo.github.m.upbound.io + - issuelabels.repo.github.m.upbound.io diff --git a/k8s/providers/hetzner/github/namespace.yaml b/k8s/providers/hetzner/github/namespace.yaml new file mode 100644 index 000000000..4b662be8d --- /dev/null +++ b/k8s/providers/hetzner/github/namespace.yaml @@ -0,0 +1,13 @@ +# Namespace for the .github "tenant": where the devantler-tech org's GitHub +# state is reconciled. No pods run here — it holds the namespaced Crossplane +# managed resources (repo.github.m.upbound.io/…), the namespaced ProviderConfig, +# the GitHub App credential, and the Flux source/sync that pull the org's +# desired state from the devantler-tech/.github repo's published artifact. +apiVersion: v1 +kind: Namespace +metadata: + name: github-config + labels: + app.kubernetes.io/managed-by: ksail + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: latest diff --git a/k8s/providers/hetzner/github/provider-config.yaml b/k8s/providers/hetzner/github/provider-config.yaml index 1c24e6c3b..0b9b86e21 100644 --- a/k8s/providers/hetzner/github/provider-config.yaml +++ b/k8s/providers/hetzner/github/provider-config.yaml @@ -1,15 +1,18 @@ -# Authenticates provider-upjet-github against the devantler-tech org as a -# GitHub App (not a PAT: short-lived installation tokens, org-scoped, survives -# personal-account changes). The referenced Secret is materialized by +# Namespaced ProviderConfig (Crossplane v2): authenticates provider-upjet-github +# against the devantler-tech org as a GitHub App (not a PAT — short-lived +# installation tokens, org-scoped, survives personal-account changes). Living in +# the github-config namespace, it is referenced by the namespaced Repository/… +# managed resources in their own namespace, so no cluster-scoped config is +# needed. The credentials Secret is local (same namespace), materialized by # external-secret.yaml from OpenBao. -apiVersion: github.upbound.io/v1beta1 +apiVersion: github.m.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default + namespace: github-config spec: credentials: source: Secret secretRef: - namespace: crossplane-system name: github-app-credentials key: credentials diff --git a/k8s/providers/hetzner/github/rbac.yaml b/k8s/providers/hetzner/github/rbac.yaml new file mode 100644 index 000000000..7f3c8e81a --- /dev/null +++ b/k8s/providers/hetzner/github/rbac.yaml @@ -0,0 +1,51 @@ +# Least-privilege RBAC for the .github tenant's ServiceAccount. +# +# Deliberately NOT the stock tenant pattern (a RoleBinding to the built-in +# `edit` ClusterRole) and deliberately NOT aggregated into `edit`: GitHub +# managed resources let their holder manipulate the org's real repositories, so +# that authority must be confined to THIS tenant — never inherited by every +# tenant that has an `edit` binding. Instead, a dedicated namespaced Role grants +# the SA full verbs over the namespaced provider-upjet-github managed-resource +# API groups, and only within the github-config namespace. Because the managed +# resources are namespaced (Crossplane v2 `*.github.m.upbound.io`), a plain +# Role + RoleBinding suffices — no ClusterRole/ClusterRoleBinding, no cluster +# RBAC. Widen apiGroups here only as new GitHub MR groups are activated in the +# ManagedResourceActivationPolicy. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: github-config-managed-resources + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +rules: + - apiGroups: + - repo.github.m.upbound.io + - actions.github.m.upbound.io + - enterprise.github.m.upbound.io + resources: + - "*" + verbs: + - get + - list + - watch + - create + - update + - patch + - delete +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: github-config-managed-resources + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: github-config-managed-resources +subjects: + - kind: ServiceAccount + name: github-config + namespace: github-config diff --git a/k8s/providers/hetzner/github/repositories/ksail.yaml b/k8s/providers/hetzner/github/repositories/ksail.yaml deleted file mode 100644 index ee6772fe4..000000000 --- a/k8s/providers/hetzner/github/repositories/ksail.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: repo.github.upbound.io/v1alpha1 -kind: Repository -metadata: - name: ksail - annotations: - # Binds this CR to the EXISTING devantler-tech/ksail repository - # (owner comes from the ProviderConfig credentials). - crossplane.io/external-name: ksail -spec: - # Observe-only while adopting — see repositories/kustomization.yaml. - managementPolicies: ["Observe"] - deletionPolicy: Orphan - forProvider: {} - providerConfigRef: - name: default diff --git a/k8s/providers/hetzner/github/repositories/kustomization.yaml b/k8s/providers/hetzner/github/repositories/kustomization.yaml deleted file mode 100644 index 4b597104c..000000000 --- a/k8s/providers/hetzner/github/repositories/kustomization.yaml +++ /dev/null @@ -1,12 +0,0 @@ -# One file per managed repository. Adoption is Observe-first (the Crossplane -# import flow): every repo starts with managementPolicies: ["Observe"] — -# read-only, zero risk to the live repo — and is flipped to ["*"] only after -# its spec.forProvider has been backfilled from status.atProvider. Keep -# deletionPolicy: Orphan on every repo even when fully managed, so no Flux -# prune or CR mishap can ever delete a real GitHub repository. ---- -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: - - platform.yaml - - ksail.yaml diff --git a/k8s/providers/hetzner/github/repositories/platform.yaml b/k8s/providers/hetzner/github/repositories/platform.yaml deleted file mode 100644 index 90c8c09a4..000000000 --- a/k8s/providers/hetzner/github/repositories/platform.yaml +++ /dev/null @@ -1,17 +0,0 @@ -apiVersion: repo.github.upbound.io/v1alpha1 -kind: Repository -metadata: - name: platform - annotations: - # Binds this CR to the EXISTING devantler-tech/platform repository - # (owner comes from the ProviderConfig credentials). - crossplane.io/external-name: platform -spec: - # Observe-only while adopting: Crossplane fills status.atProvider and never - # writes to GitHub. Flip to ["*"] after backfilling forProvider — see - # repositories/kustomization.yaml and docs/github-management.md. - managementPolicies: ["Observe"] - deletionPolicy: Orphan - forProvider: {} - providerConfigRef: - name: default diff --git a/k8s/providers/hetzner/github/serviceaccount.yaml b/k8s/providers/hetzner/github/serviceaccount.yaml new file mode 100644 index 000000000..67f988509 --- /dev/null +++ b/k8s/providers/hetzner/github/serviceaccount.yaml @@ -0,0 +1,14 @@ +# The .github tenant's ServiceAccount. The tenant's Flux Kustomization (sync.yaml) +# runs AS this SA, so the GitHub managed resources are applied with exactly the +# namespace-scoped authority granted by rbac.yaml — nothing more. imagePullSecrets +# lets it pull the cosign-signed manifests artifact from GHCR. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +automountServiceAccountToken: false +imagePullSecrets: + - name: ghcr-auth diff --git a/k8s/providers/hetzner/github/sync.yaml b/k8s/providers/hetzner/github/sync.yaml new file mode 100644 index 000000000..b69c8526d --- /dev/null +++ b/k8s/providers/hetzner/github/sync.yaml @@ -0,0 +1,56 @@ +# The .github tenant's source + sync — the same delivery pattern as every other +# tenant (cosign-verified OCI artifact → namespace-scoped Kustomization), but the +# artifact carries Crossplane managed resources (the org's desired GitHub state) +# instead of a workload. The desired state lives in devantler-tech/.github under +# deploy/ and is published as a signed artifact by that repo's cd.yaml on v* tags. +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: OCIRepository +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +spec: + interval: 1m + ref: + semver: ">=1.0.0" + # .github is an invalid OCI path component (leading dot), so the artifact is + # published under the `github-config` name — see devantler-tech/.github cd.yaml. + url: oci://ghcr.io/devantler-tech/github-config/manifests + secretRef: + name: ghcr-auth + verify: + provider: cosign + matchOIDCIdentity: + # Keyless-signed by the .github repo's own cd.yaml workflow (the repo is + # literally named `.github`, hence the doubled `\.github` in the path). + - issuer: '^https://token\.actions\.githubusercontent\.com$' + subject: '^https://github\.com/devantler-tech/\.github/\.github/workflows/cd\.yaml@.+$' +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail + annotations: + ksail.devantler.tech/reconcile-exclude: "true" +spec: + interval: 1m + timeout: 5m + retryInterval: 2m + path: . + prune: true + wait: true + force: true + # Runs as the tenant SA: the GitHub managed resources are applied with only + # the namespace-scoped authority in rbac.yaml. No dependsOn the `github` Flux + # Kustomization (which applies THIS object) — that would deadlock; instead it + # retries benignly until the Provider has established the namespaced CRDs. + serviceAccountName: github-config + sourceRef: + kind: OCIRepository + name: github-config + targetNamespace: github-config From 6a0a2087a1edd5899fc578573cd489426042726b Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sat, 13 Jun 2026 11:58:41 +0200 Subject: [PATCH 3/6] fix(github): verify github-config artifact against publish-manifests signer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The .github repo's cd.yaml now publishes the github-config OCI artifact by calling the shared publish-manifests reusable workflow (reusable-workflows#319) instead of signing inline. Cosign keyless signing runs INSIDE that reusable workflow, so the certificate's OIDC subject is the reusable workflow's path — not .github's own cd.yaml. Update the github-config OCIRepository's verify.matchOIDCIdentity.subject to match the new signer: ^…/devantler-tech/reusable-workflows/.github/workflows/publish-manifests.yaml@.+$ (same convention as the publish-app.yaml-signed app artifacts). This must land together with .github#39 (the caller cutover) so the verified identity always matches the signer. The github-config artifact is not yet in production (both PRs are drafts), so there is no live verification to break. Validated: `kubectl kustomize k8s/providers/hetzner/github/` builds clean with the updated subject. Co-Authored-By: Claude Opus 4.8 (1M context) --- k8s/providers/hetzner/github/sync.yaml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/k8s/providers/hetzner/github/sync.yaml b/k8s/providers/hetzner/github/sync.yaml index b69c8526d..c5293a762 100644 --- a/k8s/providers/hetzner/github/sync.yaml +++ b/k8s/providers/hetzner/github/sync.yaml @@ -23,10 +23,13 @@ spec: verify: provider: cosign matchOIDCIdentity: - # Keyless-signed by the .github repo's own cd.yaml workflow (the repo is - # literally named `.github`, hence the doubled `\.github` in the path). + # Keyless-signed by the shared publish-manifests reusable workflow that + # .github's cd.yaml calls. Signing runs INSIDE that reusable workflow, so + # the cosign OIDC subject is the reusable workflow's path (devantler-tech/ + # reusable-workflows), NOT .github's own cd.yaml — same convention as the + # publish-app.yaml-signed app artifacts. - issuer: '^https://token\.actions\.githubusercontent\.com$' - subject: '^https://github\.com/devantler-tech/\.github/\.github/workflows/cd\.yaml@.+$' + subject: '^https://github\.com/devantler-tech/reusable-workflows/\.github/workflows/publish-manifests\.yaml@.+$' --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization From 7154978a2adc987994923aa8bd918affc0c06d97 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Tue, 16 Jun 2026 09:46:41 +0200 Subject: [PATCH 4/6] refactor(github): onboard GitHub management as a standard app, not a bespoke Flux Kustomization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dedicated `github` Flux Kustomization (clusters/prod/github-flux-kustomization.yaml + providers/hetzner/github/) diverged from the proven pattern. Rework to the standard tiers: - Crossplane core HelmRelease stays in infrastructure/controllers/crossplane/ - Provider package + ManagedResourceActivationPolicy move to the infrastructure layer (providers/hetzner/infrastructure/crossplane/), one tier after the controller — same controller-then-CR ordering as Coroot/Flagger - the GitHub state is onboarded as a normal app, bases/apps/github-config/ (namespace, SA, least-priv Role, ESO creds, namespaced ProviderConfig, seed PushSecret, cosign-verified OCIRepository+Kustomization), consumed by the existing `apps` Flux Kustomization — no bespoke kustomization - the credential seed PushSecret lives with the app (leaf apps layer) so a missing-key stall can never block the infrastructure→apps chain - delete the bespoke github layer + github-flux-kustomization.yaml; update docs/github-management.md + AGENTS.md Co-Authored-By: Claude Fable 5 --- AGENTS.md | 2 +- docs/github-management.md | 63 +++++++++++-------- .../apps/github-config}/external-secret.yaml | 0 .../ghcr-auth-externalsecret.yaml | 14 ++--- .../apps/github-config/kustomization.yaml | 12 ++++ .../apps/github-config}/namespace.yaml | 4 +- .../apps/github-config}/provider-config.yaml | 4 +- .../apps/github-config}/rbac.yaml | 22 +++---- .../apps/github-config}/seed-github-app.yaml | 15 +++-- .../apps/github-config/serviceaccount.yaml | 14 +++++ .../apps/github-config}/sync.yaml | 20 +++--- k8s/bases/apps/kustomization.yaml | 5 ++ .../infrastructure/vault-config/job.yaml | 8 +-- .../prod/github-flux-kustomization.yaml | 39 ------------ k8s/clusters/prod/kustomization.yaml | 4 -- .../hetzner/github/kustomization.yaml | 44 ------------- .../hetzner/github/serviceaccount.yaml | 14 ----- .../controllers/crossplane/helm-release.yaml | 3 +- .../crossplane/kustomization.yaml | 6 ++ .../managed-resource-activation-policy.yaml | 6 +- .../crossplane}/provider.yaml | 11 +++- .../hetzner/infrastructure/kustomization.yaml | 7 +++ 22 files changed, 143 insertions(+), 174 deletions(-) rename k8s/{providers/hetzner/github => bases/apps/github-config}/external-secret.yaml (100%) rename k8s/{providers/hetzner/github => bases/apps/github-config}/ghcr-auth-externalsecret.yaml (64%) create mode 100644 k8s/bases/apps/github-config/kustomization.yaml rename k8s/{providers/hetzner/github => bases/apps/github-config}/namespace.yaml (69%) rename k8s/{providers/hetzner/github => bases/apps/github-config}/provider-config.yaml (78%) rename k8s/{providers/hetzner/github => bases/apps/github-config}/rbac.yaml (50%) rename k8s/{providers/hetzner/github => bases/apps/github-config}/seed-github-app.yaml (60%) create mode 100644 k8s/bases/apps/github-config/serviceaccount.yaml rename k8s/{providers/hetzner/github => bases/apps/github-config}/sync.yaml (67%) delete mode 100644 k8s/clusters/prod/github-flux-kustomization.yaml delete mode 100644 k8s/providers/hetzner/github/kustomization.yaml delete mode 100644 k8s/providers/hetzner/github/serviceaccount.yaml create mode 100644 k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml rename k8s/providers/hetzner/{github => infrastructure/crossplane}/managed-resource-activation-policy.yaml (82%) rename k8s/providers/hetzner/{github => infrastructure/crossplane}/provider.yaml (59%) diff --git a/AGENTS.md b/AGENTS.md index 859b6b9cf..4774bcdf1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -198,7 +198,7 @@ You **cannot** decrypt existing secrets without the proper Age keys. For local d - **Small, focused changes** — one concern per PR. - **Never commit plaintext secrets** — all secrets must be SOPS-encrypted with the `.enc.yaml` suffix. - **Base files are immutable** — use Kustomize `patches:` in overlays; never edit `k8s/bases/` directly from a provider or cluster overlay. -- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. Two prod-only side layers hang off `infrastructure` without gating `apps`: `infrastructure-overprovisioning` (apply-only autoscaler buffer) and `github` (declarative GitHub org management via Crossplane — see [`docs/github-management.md`](docs/github-management.md)). +- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. One prod-only side layer hangs off `infrastructure` without gating `apps`: `infrastructure-overprovisioning` (apply-only autoscaler buffer). Declarative GitHub org management runs as a normal **app** (`github-config`) consuming the `devantler-tech/.github` artifact, with its Crossplane provider in the `infrastructure` layer — see [`docs/github-management.md`](docs/github-management.md). ### Infrastructure File Structure Convention diff --git a/docs/github-management.md b/docs/github-management.md index 348ca6e6d..ead5730c7 100644 --- a/docs/github-management.md +++ b/docs/github-management.md @@ -9,37 +9,45 @@ GitHub to it — including reverting out-of-band UI edits. The desired state does **not** live in this repo. It lives in [`devantler-tech/.github`](https://github.com/devantler-tech/.github) under -`deploy/`, which is onboarded here as a **tenant**: the platform pulls its -cosign-signed OCI artifact and applies it, exactly like an application tenant — -only the artifact carries managed resources instead of a workload. +`deploy/`, onboarded here as a **normal app** (`github-config`) using the same +pattern as every other tenant (cosign-verified OCI artifact → namespace-scoped +`Kustomization`) — only the artifact carries managed resources instead of a +workload. There is **no** bespoke Flux Kustomization for GitHub; it rides the +existing `apps` layer. -## Why a tenant, and why namespaced +## Why an app, and why namespaced GitHub's managed resources ship in two flavours: cluster-scoped (`repo.github.upbound.io`) and **namespaced** (`repo.github.m.upbound.io`, Crossplane v2). We use the **namespaced** variants. That choice is what lets -`.github` be a *genuinely isolated* tenant: a namespaced managed resource can be -applied by a namespace-scoped `ServiceAccount` with a plain `Role` — no -`ClusterRoleBinding`, no cluster RBAC. The tenant's authority to touch GitHub is +`github-config` be a *genuinely isolated* app: a namespaced managed resource can +be applied by a namespace-scoped `ServiceAccount` with a plain `Role` — no +`ClusterRoleBinding`, no cluster RBAC. The app's authority to touch GitHub is confined to its own namespace and is **not** aggregated into the built-in `edit` ClusterRole, so no other tenant inherits it. ## Architecture +Three tiers, each in its conventional place — controller in `controllers/`, its +CRs one tier later in `infrastructure/`, the workload in `apps/`: + | Piece | Where | Notes | |---|---|---| -| Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only (external reconcilers must run on exactly one always-on cluster). `provider.defaultActivations: []` keeps unused provider CRDs inactive. | -| Engine | `k8s/providers/hetzner/github/` (`provider.yaml`, `managed-resource-activation-policy.yaml`, `seed-github-app.yaml`) | The provider package + runtime; activates only the **namespaced** MRDs we use; seeds the GitHub App credential into OpenBao. | -| Tenant wiring | `k8s/providers/hetzner/github/` (`namespace.yaml`, `serviceaccount.yaml`, `rbac.yaml`, `ghcr-auth-externalsecret.yaml`, `external-secret.yaml`, `provider-config.yaml`, `sync.yaml`) | The `github-config` namespace, its least-privilege SA + Role, the GitHub App + ghcr credentials, the namespaced `ProviderConfig`, and the OCIRepository + Kustomization that pull and apply `.github`'s artifact as the tenant SA. | -| Desired state | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The actual `Repository`/ruleset/label managed resources. Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags. | -| Flux wiring | `k8s/clusters/prod/github-flux-kustomization.yaml` | Dedicated `github` Flux Kustomization, `dependsOn: infrastructure`. Isolated so a GitHub-side stall never holds `infrastructure`/`apps` red. | -| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by a PushSecret, materialized into `github-config/github-app-credentials` by an ExternalSecret that renders the provider's JSON credential format. | - -> Why the tenant wiring lives in the `github` layer (not `k8s/bases/apps/`): -> it is prod-only and coupled to the Crossplane provider, and we want it -> isolated from the `apps` deploy chain. It is still a tenant *in substance* — -> dedicated namespace, isolated least-privilege SA, cosign-verified OCI delivery -> from its own repo, own pruning Kustomization. +| Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only. `provider.defaultActivations: []` keeps unused provider CRDs inactive. Installs the pkg.crossplane.io CRDs. | +| Provider + activation policy | `k8s/providers/hetzner/infrastructure/crossplane/` | The `Provider` package + `DeploymentRuntimeConfig` + `ManagedResourceActivationPolicy` (namespaced MRDs only). One tier after the controller (needs its CRDs), like Coroot/Flagger CRs. Establishes the namespaced github CRDs. | +| The `github-config` app | `k8s/bases/apps/github-config/` | Standard app onboarding: namespace, SA, least-privilege `Role`, GitHub App + ghcr credential `ExternalSecret`s, the credential seed `PushSecret`, the namespaced `ProviderConfig`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). | +| Desired state | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The actual `Repository`/ruleset/label managed resources. Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. | +| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by the app's `PushSecret`, materialized into `github-config/github-app-credentials` by an `ExternalSecret` that renders the provider's JSON credential format. | + +Ordering is the normal chain: `apps` `dependsOn` `infrastructure`, so by the +time the app reconciles, the provider (infrastructure tier) has installed the +namespaced github CRDs the `ProviderConfig` and managed resources need. On a +fresh install the app's `Kustomization` retries benignly until those CRDs exist. + +The credential seed `PushSecret` lives **with the app** (apps layer), not in +`infrastructure/vault-seed/`: until the GitHub App keys are in `variables-cluster` +it stalls, and the apps layer is the leaf of the dependency chain — so a stall +can never hold `infrastructure` (and through it every deploy) red. ## Credential setup (one-time, maintainer) @@ -51,11 +59,11 @@ ClusterRole, so no other tenant inherits it. 2. Add three keys to the **prod** `variables-cluster` SOPS secret (`k8s/clusters/prod/bootstrap/variables-cluster-secret.enc.yaml`): `github_app_id`, `github_app_installation_id`, and `github_app_pem` - (the private key, as a normal multiline PEM block). + (the private key, as a normal multiline PEM block). Do this *before* the + first rollout so the seed `PushSecret` doesn't sit red. 3. Merge. The PushSecret seeds OpenBao, the ExternalSecret renders the provider - credential into `github-config`, and the `github` Flux Kustomization turns - Ready once `.github` has published its first `v*` artifact. Until then that - Kustomization sits red — expected, and isolated. + credential into `github-config`, and the app reconciles once `.github` has + published its first `v*` artifact. ## Adopting an existing repository (Observe-first) @@ -85,9 +93,10 @@ reconcile, then batch the rest. ## Adding more GitHub resource kinds 1. Activate the namespaced MRD in - `k8s/providers/hetzner/github/managed-resource-activation-policy.yaml` + `k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml` (plural.group form, e.g. `repositoryrulesets.repo.github.m.upbound.io`); add - its API group to the tenant `Role` in `rbac.yaml` if it is a new group. + its API group to the app `Role` in `k8s/bases/apps/github-config/rbac.yaml` + if it is a new group. 2. Add the managed resources in `.github`'s `deploy/` — same Observe-first flow where an external object already exists. Each active MRD costs the apiserver ~3 MiB; activate only what is used. @@ -99,8 +108,8 @@ reconcile, then batch the rest. a real GitHub repository. - **Observe-first adoption** — a new managed resource for an existing object always starts `managementPolicies: ["Observe"]` (read-only). -- **Least privilege** — the tenant SA's authority is a namespaced `Role` over - the GitHub MR API groups only, never aggregated into `edit`. +- **Least privilege** — the app SA's authority is a namespaced `Role` over the + GitHub MR API groups only, never aggregated into `edit`. - The GitHub App credential is org-scoped and short-lived per request; the provider pod's egress is FQDN-pinned to `api.github.com` by the `crossplane-system` CiliumNetworkPolicy. diff --git a/k8s/providers/hetzner/github/external-secret.yaml b/k8s/bases/apps/github-config/external-secret.yaml similarity index 100% rename from k8s/providers/hetzner/github/external-secret.yaml rename to k8s/bases/apps/github-config/external-secret.yaml diff --git a/k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml b/k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml similarity index 64% rename from k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml rename to k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml index c5e1d3231..0ec541513 100644 --- a/k8s/providers/hetzner/github/ghcr-auth-externalsecret.yaml +++ b/k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml @@ -1,13 +1,13 @@ -# GHCR image-pull secret for the .github tenant, sourced from OpenBao (same -# shape and path as every other tenant's ghcr-auth). Consumed by the tenant's +# GHCR image-pull secret for the github-config app, sourced from OpenBao (same +# shape and path as every other tenant's ghcr-auth). Consumed by the app's # OCIRepository (secretRef) and ServiceAccount (imagePullSecrets) to pull the # cosign-signed manifests artifact published by devantler-tech/.github. # -# PLATFORM-side resource: reconciled by flux-system (this layer), never by the -# tenant ServiceAccount, so it uses the shared cluster-scoped `openbao` -# ClusterSecretStore reading the platform path infrastructure/ghcr/auth — the -# `restrict-tenant-secret-stores` Kyverno policy carves out flux-system-applied -# ExternalSecrets. +# PLATFORM-side resource: reconciled by flux-system (the apps layer applies the +# app scaffolding), never by the app ServiceAccount, so it uses the shared +# cluster-scoped `openbao` ClusterSecretStore reading the platform path +# infrastructure/ghcr/auth — the `restrict-tenant-secret-stores` Kyverno policy +# carves out flux-system-applied ExternalSecrets. apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: diff --git a/k8s/bases/apps/github-config/kustomization.yaml b/k8s/bases/apps/github-config/kustomization.yaml new file mode 100644 index 000000000..17a3267db --- /dev/null +++ b/k8s/bases/apps/github-config/kustomization.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - serviceaccount.yaml + - rbac.yaml + - ghcr-auth-externalsecret.yaml + - external-secret.yaml + - provider-config.yaml + - seed-github-app.yaml + - sync.yaml diff --git a/k8s/providers/hetzner/github/namespace.yaml b/k8s/bases/apps/github-config/namespace.yaml similarity index 69% rename from k8s/providers/hetzner/github/namespace.yaml rename to k8s/bases/apps/github-config/namespace.yaml index 4b662be8d..d582a6b11 100644 --- a/k8s/providers/hetzner/github/namespace.yaml +++ b/k8s/bases/apps/github-config/namespace.yaml @@ -1,8 +1,10 @@ -# Namespace for the .github "tenant": where the devantler-tech org's GitHub +# Namespace for the github-config app: where the devantler-tech org's GitHub # state is reconciled. No pods run here — it holds the namespaced Crossplane # managed resources (repo.github.m.upbound.io/…), the namespaced ProviderConfig, # the GitHub App credential, and the Flux source/sync that pull the org's # desired state from the devantler-tech/.github repo's published artifact. +# (A default-deny CiliumNetworkPolicy is auto-generated by the add-default-deny +# Kyverno policy; no per-app NetworkPolicy is needed since there are no pods.) apiVersion: v1 kind: Namespace metadata: diff --git a/k8s/providers/hetzner/github/provider-config.yaml b/k8s/bases/apps/github-config/provider-config.yaml similarity index 78% rename from k8s/providers/hetzner/github/provider-config.yaml rename to k8s/bases/apps/github-config/provider-config.yaml index 0b9b86e21..afb4cf120 100644 --- a/k8s/providers/hetzner/github/provider-config.yaml +++ b/k8s/bases/apps/github-config/provider-config.yaml @@ -4,7 +4,9 @@ # the github-config namespace, it is referenced by the namespaced Repository/… # managed resources in their own namespace, so no cluster-scoped config is # needed. The credentials Secret is local (same namespace), materialized by -# external-secret.yaml from OpenBao. +# external-secret.yaml from OpenBao. The github.m.upbound.io CRD is installed by +# the provider package in the infrastructure layer (one tier earlier), so this +# applies once that has reconciled. apiVersion: github.m.upbound.io/v1beta1 kind: ProviderConfig metadata: diff --git a/k8s/providers/hetzner/github/rbac.yaml b/k8s/bases/apps/github-config/rbac.yaml similarity index 50% rename from k8s/providers/hetzner/github/rbac.yaml rename to k8s/bases/apps/github-config/rbac.yaml index 7f3c8e81a..77eed2b87 100644 --- a/k8s/providers/hetzner/github/rbac.yaml +++ b/k8s/bases/apps/github-config/rbac.yaml @@ -1,15 +1,15 @@ -# Least-privilege RBAC for the .github tenant's ServiceAccount. +# Least-privilege RBAC for the github-config app's ServiceAccount. # -# Deliberately NOT the stock tenant pattern (a RoleBinding to the built-in -# `edit` ClusterRole) and deliberately NOT aggregated into `edit`: GitHub -# managed resources let their holder manipulate the org's real repositories, so -# that authority must be confined to THIS tenant — never inherited by every -# tenant that has an `edit` binding. Instead, a dedicated namespaced Role grants -# the SA full verbs over the namespaced provider-upjet-github managed-resource -# API groups, and only within the github-config namespace. Because the managed -# resources are namespaced (Crossplane v2 `*.github.m.upbound.io`), a plain -# Role + RoleBinding suffices — no ClusterRole/ClusterRoleBinding, no cluster -# RBAC. Widen apiGroups here only as new GitHub MR groups are activated in the +# Deliberately NOT the stock app pattern (a RoleBinding to the built-in `edit` +# ClusterRole) and deliberately NOT aggregated into `edit`: GitHub managed +# resources let their holder manipulate the org's real repositories, so that +# authority must be confined to THIS app — never inherited by every tenant that +# has an `edit` binding. Instead, a dedicated namespaced Role grants the SA full +# verbs over the namespaced provider-upjet-github managed-resource API groups, +# and only within the github-config namespace. Because the managed resources are +# namespaced (Crossplane v2 `*.github.m.upbound.io`), a plain Role + RoleBinding +# suffices — no ClusterRole/ClusterRoleBinding, no cluster RBAC. Widen apiGroups +# here only as new GitHub MR groups are activated in the # ManagedResourceActivationPolicy. apiVersion: rbac.authorization.k8s.io/v1 kind: Role diff --git a/k8s/providers/hetzner/github/seed-github-app.yaml b/k8s/bases/apps/github-config/seed-github-app.yaml similarity index 60% rename from k8s/providers/hetzner/github/seed-github-app.yaml rename to k8s/bases/apps/github-config/seed-github-app.yaml index 239bd768d..d348bfd09 100644 --- a/k8s/providers/hetzner/github/seed-github-app.yaml +++ b/k8s/bases/apps/github-config/seed-github-app.yaml @@ -1,10 +1,15 @@ # PushSecret for the GitHub App credentials used by provider-upjet-github. # Same durable-seed pattern as seed-hcloud: re-pushing hourly from the SOPS -# source (variables-cluster) is what re-seeds the keys automatically after -# OpenBao data loss. Lives in THIS layer (not infrastructure/vault-seed/) so -# that, until the maintainer adds the github_app_* keys to the prod -# variables-cluster secret, the resulting InProgress stall holds only the -# isolated `github` Flux Kustomization red — never `infrastructure`. +# source (variables-cluster) re-seeds the keys automatically after OpenBao data +# loss. +# +# Lives with the github-config app (the apps layer), NOT in +# infrastructure/vault-seed/, on purpose: until the maintainer adds the +# github_app_* keys to the prod variables-cluster secret this PushSecret stalls, +# and the apps layer is the leaf of the Flux dependency chain — so the stall can +# never hold `infrastructure` (and through it every deploy) red. A vault-seed +# placement would block the chain. Seed the keys before/with rollout to avoid +# even a transient red apps layer. --- apiVersion: external-secrets.io/v1alpha1 kind: PushSecret diff --git a/k8s/bases/apps/github-config/serviceaccount.yaml b/k8s/bases/apps/github-config/serviceaccount.yaml new file mode 100644 index 000000000..60d94dd4b --- /dev/null +++ b/k8s/bases/apps/github-config/serviceaccount.yaml @@ -0,0 +1,14 @@ +# The github-config app's ServiceAccount. The app's Flux Kustomization +# (sync.yaml) runs AS this SA, so the GitHub managed resources are applied with +# exactly the namespace-scoped authority granted by rbac.yaml — nothing more. +# imagePullSecrets lets it pull the cosign-signed manifests artifact from GHCR. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +automountServiceAccountToken: false +imagePullSecrets: + - name: ghcr-auth diff --git a/k8s/providers/hetzner/github/sync.yaml b/k8s/bases/apps/github-config/sync.yaml similarity index 67% rename from k8s/providers/hetzner/github/sync.yaml rename to k8s/bases/apps/github-config/sync.yaml index c5293a762..f6ccfa074 100644 --- a/k8s/providers/hetzner/github/sync.yaml +++ b/k8s/bases/apps/github-config/sync.yaml @@ -1,8 +1,9 @@ -# The .github tenant's source + sync — the same delivery pattern as every other -# tenant (cosign-verified OCI artifact → namespace-scoped Kustomization), but the -# artifact carries Crossplane managed resources (the org's desired GitHub state) -# instead of a workload. The desired state lives in devantler-tech/.github under -# deploy/ and is published as a signed artifact by that repo's cd.yaml on v* tags. +# The github-config app's source + sync — the same delivery pattern as every +# other app/tenant (cosign-verified OCI artifact → namespace-scoped +# Kustomization), but the artifact carries Crossplane managed resources (the +# org's desired GitHub state) instead of a workload. The desired state lives in +# devantler-tech/.github under deploy/ and is published as a signed artifact by +# that repo's cd.yaml on v* tags. --- apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository @@ -48,10 +49,11 @@ spec: prune: true wait: true force: true - # Runs as the tenant SA: the GitHub managed resources are applied with only - # the namespace-scoped authority in rbac.yaml. No dependsOn the `github` Flux - # Kustomization (which applies THIS object) — that would deadlock; instead it - # retries benignly until the Provider has established the namespaced CRDs. + # Runs as the app SA: the GitHub managed resources are applied with only the + # namespace-scoped authority in rbac.yaml. This Kustomization is itself applied + # by the `apps` Flux Kustomization; it must NOT dependsOn it (deadlock) — it + # retries benignly until the provider (infrastructure layer) has established + # the namespaced CRDs. serviceAccountName: github-config sourceRef: kind: OCIRepository diff --git a/k8s/bases/apps/kustomization.yaml b/k8s/bases/apps/kustomization.yaml index f01718fe4..275a82879 100644 --- a/k8s/bases/apps/kustomization.yaml +++ b/k8s/bases/apps/kustomization.yaml @@ -11,6 +11,11 @@ resources: # infra-layer plumbing (OpenBao policies/seeds, NetworkPolicies, Kubescape # security exceptions, prod config-map vars) is intentionally left idle. # - fleetdm/ + # Declarative GitHub org management: the devantler-tech/.github repo's deploy/ + # artifact (Crossplane managed resources) onboarded as a normal app/tenant. + # Prod-only in practice (its Crossplane provider is Hetzner-only); the docker + # provider deploys no apps, so it is naturally excluded from local clusters. + - github-config/ - headlamp/ - homepage/ - umami/ diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 3b3962bc6..bbb5e5126 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -481,10 +481,10 @@ spec: # Read the GitHub App credential for provider-upjet-github # (declarative GitHub management). Consumed by the - # github-app-credentials ExternalSecret in crossplane-system via - # the `external-secrets` ClusterSecretStore role; seeded from - # variables-cluster by the seed-github-app PushSecret (see - # k8s/providers/hetzner/github/). + # github-app-credentials ExternalSecret in the github-config + # namespace via the `external-secrets` ClusterSecretStore role; + # seeded from variables-cluster by the seed-github-app PushSecret + # (see k8s/bases/apps/github-config/). bao policy write infra-github-readonly - <<'POLICY' path "secret/data/infrastructure/github/*" { capabilities = ["read"] diff --git a/k8s/clusters/prod/github-flux-kustomization.yaml b/k8s/clusters/prod/github-flux-kustomization.yaml deleted file mode 100644 index 08b438a53..000000000 --- a/k8s/clusters/prod/github-flux-kustomization.yaml +++ /dev/null @@ -1,39 +0,0 @@ -# Prod-only: reconciles the devantler-tech GitHub org's declarative state -# (providers/hetzner/github/ — Crossplane provider-upjet-github package, -# credentials wiring, and the Repository/… managed resources) via its OWN -# Flux Kustomization instead of the `infrastructure` layer. -# -# Why it cannot ride along in `infrastructure`: GitHub org state is orthogonal -# to cluster workloads, but `infrastructure` reconciles with `wait: true` and -# gates `apps` (and with it every merge-queue / tag deploy). A GitHub-side -# stall — credentials not yet seeded in variables-cluster, a GitHub API -# outage, a misconfigured managed resource — must only ever hold THIS -# Kustomization red (same isolation reasoning as -# overprovisioning-flux-kustomization.yaml, though this layer keeps -# `wait: true`: steady-state, everything in it is expected to be Ready). -# -# Expected on a fresh install: the first reconcile fails with unknown-kind -# errors until the Provider package installs and the activation policy -# creates the github.upbound.io CRDs (~1-2 min), then converges on retry. ---- -apiVersion: kustomize.toolkit.fluxcd.io/v1 -kind: Kustomization -metadata: - name: github - namespace: flux-system -spec: - interval: 60m - timeout: 10m - retryInterval: 2m - path: providers/hetzner/github - sourceRef: - kind: OCIRepository - name: flux-system - # After `infrastructure` so the Crossplane core HelmRelease (controllers - # layer), the openbao ClusterSecretStore and the vault-config policies it - # needs are all in place first. - dependsOn: - - name: infrastructure - wait: true - prune: true - force: true diff --git a/k8s/clusters/prod/kustomization.yaml b/k8s/clusters/prod/kustomization.yaml index 2d77d90c0..bdc176135 100644 --- a/k8s/clusters/prod/kustomization.yaml +++ b/k8s/clusters/prod/kustomization.yaml @@ -7,10 +7,6 @@ resources: # `infrastructure` layer (its pause pods are expected-Pending at the # autoscaler ceiling and must not hold deploys red) — see the file header. - overprovisioning-flux-kustomization.yaml - # Prod-only: declarative GitHub org management (Crossplane managed - # resources), isolated in its own Flux Kustomization so a GitHub-side stall - # never holds the infrastructure→apps deploy chain red — see the file header. - - github-flux-kustomization.yaml # Template inputs for this cluster. cluster-meta drives the replacements # below; it is annotated local-config and never applied to the cluster. diff --git a/k8s/providers/hetzner/github/kustomization.yaml b/k8s/providers/hetzner/github/kustomization.yaml deleted file mode 100644 index 45879791c..000000000 --- a/k8s/providers/hetzner/github/kustomization.yaml +++ /dev/null @@ -1,44 +0,0 @@ -# Declarative GitHub management — the devantler-tech org's GitHub state -# (repositories, rulesets, labels, …) as Crossplane managed resources. This -# layer holds the ENGINE + the .github TENANT wiring; the desired-state managed -# resources themselves live in the devantler-tech/.github repo under deploy/ and -# are pulled in as a cosign-signed OCI artifact (sync.yaml). See -# docs/github-management.md. -# -# Applied by its OWN Flux Kustomization (`github`, see -# k8s/clusters/prod/github-flux-kustomization.yaml), NOT the health-gated -# `infrastructure` layer: GitHub org state is orthogonal to cluster workloads, -# and a stalled GitHub reconcile (credentials not seeded, GitHub API outage, -# .github not yet published) must never hold `infrastructure`/`apps` — and with -# them every merge-queue / tag deploy — red. -# -# Two parts: -# ENGINE (cluster infra) — provider.yaml (the provider package + runtime), -# managed-resource-activation-policy.yaml (activate only the namespaced MRDs -# we use), seed-github-app.yaml (seed the GitHub App credential into OpenBao). -# TENANT (.github) — namespace, ServiceAccount + least-privilege Role/Binding, -# ghcr pull + GitHub App credential ExternalSecrets, the namespaced -# ProviderConfig, and the OCIRepository + Kustomization that pull and apply -# the org's managed resources as the tenant SA. -# -# Crossplane core (the controller) lives one layer earlier in -# ../infrastructure/controllers/crossplane/. On a FRESH install the first -# reconcile here fails benignly (the namespaced CRDs only exist once the Provider -# package + activation policy have run; the OCI artifact only exists once .github -# has published a v* release) and converges on the next retries. ---- -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: - # Engine (cluster infra) - - provider.yaml - - managed-resource-activation-policy.yaml - - seed-github-app.yaml - # Tenant (.github) - - namespace.yaml - - serviceaccount.yaml - - rbac.yaml - - ghcr-auth-externalsecret.yaml - - external-secret.yaml - - provider-config.yaml - - sync.yaml diff --git a/k8s/providers/hetzner/github/serviceaccount.yaml b/k8s/providers/hetzner/github/serviceaccount.yaml deleted file mode 100644 index 67f988509..000000000 --- a/k8s/providers/hetzner/github/serviceaccount.yaml +++ /dev/null @@ -1,14 +0,0 @@ -# The .github tenant's ServiceAccount. The tenant's Flux Kustomization (sync.yaml) -# runs AS this SA, so the GitHub managed resources are applied with exactly the -# namespace-scoped authority granted by rbac.yaml — nothing more. imagePullSecrets -# lets it pull the cosign-signed manifests artifact from GHCR. -apiVersion: v1 -kind: ServiceAccount -metadata: - name: github-config - namespace: github-config - labels: - app.kubernetes.io/managed-by: ksail -automountServiceAccountToken: false -imagePullSecrets: - - name: ghcr-auth diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml index 7ba73ddbe..4df1112b1 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml @@ -33,7 +33,8 @@ spec: # creating the "default" activate-everything MRAP — each MRD costs the # apiserver ~3MiB once active, and provider-upjet-github alone ships 77. # The MRDs we actually use are activated by an explicit - # ManagedResourceActivationPolicy in providers/hetzner/github/. + # ManagedResourceActivationPolicy in + # providers/hetzner/infrastructure/crossplane/. defaultActivations: [] resourcesCrossplane: requests: diff --git a/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml b/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml new file mode 100644 index 000000000..dbc6e5ec0 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml @@ -0,0 +1,6 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - provider.yaml + - managed-resource-activation-policy.yaml diff --git a/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml b/k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml similarity index 82% rename from k8s/providers/hetzner/github/managed-resource-activation-policy.yaml rename to k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml index 1035774b4..222e5ae3e 100644 --- a/k8s/providers/hetzner/github/managed-resource-activation-policy.yaml +++ b/k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml @@ -7,9 +7,9 @@ # We activate the NAMESPACED (Crossplane v2, `*.github.m.upbound.io`) variants # only — namespaced managed resources can be applied by a namespace-scoped # tenant ServiceAccount with a plain Role (no cluster RBAC), which is what makes -# the .github tenant a genuinely isolated tenant. Extend this list as more of -# the org's GitHub surface comes under management (org rulesets, Actions -# secrets, …) — see docs/github-management.md. +# github-config a genuinely isolated app/tenant. Extend this list as more of the +# org's GitHub surface comes under management (org rulesets, Actions secrets, …) +# — see docs/github-management.md. apiVersion: apiextensions.crossplane.io/v1alpha1 kind: ManagedResourceActivationPolicy metadata: diff --git a/k8s/providers/hetzner/github/provider.yaml b/k8s/providers/hetzner/infrastructure/crossplane/provider.yaml similarity index 59% rename from k8s/providers/hetzner/github/provider.yaml rename to k8s/providers/hetzner/infrastructure/crossplane/provider.yaml index 0f9b61cdb..62a9bbd9b 100644 --- a/k8s/providers/hetzner/github/provider.yaml +++ b/k8s/providers/hetzner/infrastructure/crossplane/provider.yaml @@ -1,7 +1,12 @@ # provider-upjet-github — Upjet-generated Crossplane provider wrapping the -# integrations/github Terraform provider. ghcr.io is the project's publish -# registry (xpkg.crossplane.io merely proxies it) and is already the trust -# root for every first-party image in this cluster. +# integrations/github Terraform provider. Lives in the `infrastructure` layer +# (not `infrastructure/controllers/`) because the Provider/DeploymentRuntimeConfig +# are pkg.crossplane.io CRs that need the Crossplane CRDs the core HelmRelease in +# controllers/crossplane/ installs first — the same controller-then-CR ordering +# as Coroot and Flagger. Installing this provider also establishes the namespaced +# github CRDs the github-config app (k8s/bases/apps/github-config/) consumes. +# ghcr.io is the project's publish registry (xpkg.crossplane.io merely proxies +# it) and is already the trust root for every first-party image in this cluster. apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: diff --git a/k8s/providers/hetzner/infrastructure/kustomization.yaml b/k8s/providers/hetzner/infrastructure/kustomization.yaml index 6f3d73f6b..959c11d14 100644 --- a/k8s/providers/hetzner/infrastructure/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/kustomization.yaml @@ -5,6 +5,13 @@ resources: - ../../../bases/infrastructure/ - certificates/ - cluster-issuers/ + # Crossplane provider-upjet-github package + activation policy. Lives in this + # `infrastructure` layer (not `infrastructure/controllers/`) because these are + # pkg.crossplane.io CRs that need the Crossplane CRDs the core HelmRelease in + # controllers/crossplane/ installs first — same controller-then-CR ordering as + # Coroot/Flagger. Establishes the namespaced github CRDs the github-config app + # (bases/apps/github-config/) consumes. Prod-only (Crossplane is Hetzner-only). + - crossplane/ # external-dns lives in this `infrastructure` layer (not # `infrastructure/controllers/`) because its Cloudflare token is a user-fed # OpenBao secret materialised by an ExternalSecret, and the openbao From 3f9fc2bfbd2a2aaa53574cebbd490042adea7e7b Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Tue, 16 Jun 2026 23:13:21 +0200 Subject: [PATCH 5/6] refactor(github): seed GitHub App creds as OpenBao placeholders, drop SOPS PushSecret MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the variables-cluster (SOPS) PushSecret seed with a declarative, idempotent placeholder seed in the vault-config bootstrap Job: it writes REPLACE_WITH_* values to secret/infrastructure/github/app only when the secret is absent, so the maintainer overwrites them in place via the OpenBao UI/CLI and a later Job re-run never clobbers the real values. OpenBao is backed up (Raft snapshots → R2), so the manually-set values are durable without a GitOps source. - add the only-if-absent kv seed to vault-config/job.yaml (section 1b) - remove seed-github-app.yaml PushSecret + its kustomization entry - drop the now-unused vault-seed-write github paths (keep infra-github-readonly, which the github-app-credentials ExternalSecret still needs to read) - docs: credential setup is now 'overwrite the OpenBao placeholders', no SOPS Co-Authored-By: Claude Fable 5 --- docs/github-management.md | 34 ++++++++------- .../apps/github-config/kustomization.yaml | 1 - .../apps/github-config/seed-github-app.yaml | 42 ------------------- .../infrastructure/vault-config/job.yaml | 25 ++++++++--- 4 files changed, 39 insertions(+), 63 deletions(-) delete mode 100644 k8s/bases/apps/github-config/seed-github-app.yaml diff --git a/docs/github-management.md b/docs/github-management.md index ead5730c7..dff8aa548 100644 --- a/docs/github-management.md +++ b/docs/github-management.md @@ -35,19 +35,21 @@ CRs one tier later in `infrastructure/`, the workload in `apps/`: |---|---|---| | Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only. `provider.defaultActivations: []` keeps unused provider CRDs inactive. Installs the pkg.crossplane.io CRDs. | | Provider + activation policy | `k8s/providers/hetzner/infrastructure/crossplane/` | The `Provider` package + `DeploymentRuntimeConfig` + `ManagedResourceActivationPolicy` (namespaced MRDs only). One tier after the controller (needs its CRDs), like Coroot/Flagger CRs. Establishes the namespaced github CRDs. | -| The `github-config` app | `k8s/bases/apps/github-config/` | Standard app onboarding: namespace, SA, least-privilege `Role`, GitHub App + ghcr credential `ExternalSecret`s, the credential seed `PushSecret`, the namespaced `ProviderConfig`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). | +| The `github-config` app | `k8s/bases/apps/github-config/` | Standard app onboarding: namespace, SA, least-privilege `Role`, GitHub App + ghcr credential `ExternalSecret`s, the namespaced `ProviderConfig`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). | | Desired state | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The actual `Repository`/ruleset/label managed resources. Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. | -| Credentials | OpenBao `infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`), seeded from the SOPS `variables-cluster` secret by the app's `PushSecret`, materialized into `github-config/github-app-credentials` by an `ExternalSecret` that renders the provider's JSON credential format. | +| Credentials | OpenBao `secret/infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`). **Placeholders are seeded declaratively (only-if-absent) by the vault-config bootstrap Job** — no SOPS; the maintainer overwrites them in place via the OpenBao UI/CLI (see below). The `github-app-credentials` `ExternalSecret` reads them and renders the provider's JSON credential format. | Ordering is the normal chain: `apps` `dependsOn` `infrastructure`, so by the time the app reconciles, the provider (infrastructure tier) has installed the namespaced github CRDs the `ProviderConfig` and managed resources need. On a fresh install the app's `Kustomization` retries benignly until those CRDs exist. -The credential seed `PushSecret` lives **with the app** (apps layer), not in -`infrastructure/vault-seed/`: until the GitHub App keys are in `variables-cluster` -it stalls, and the apps layer is the leaf of the dependency chain — so a stall -can never hold `infrastructure` (and through it every deploy) red. +The GitHub App credentials are **not** SOPS-managed. The vault-config bootstrap +Job seeds **placeholder** values into OpenBao (`secret/infrastructure/github/app`) +**only when the secret is absent**, and the maintainer overwrites them in place +(below). A later Job re-run never clobbers the real values (the seed is guarded +by an existence check), and OpenBao is backed up (Raft snapshots → R2 via +Velero), so the manually-set values are durable without a GitOps source of truth. ## Credential setup (one-time, maintainer) @@ -56,14 +58,18 @@ can never hold `infrastructure` (and through it every deploy) red. (read/write), Contents (read)* and *Organization: Administration (read/write)* to start — widen as more resource kinds come under management. Install it on **all repositories** of the org. No webhook. -2. Add three keys to the **prod** `variables-cluster` SOPS secret - (`k8s/clusters/prod/bootstrap/variables-cluster-secret.enc.yaml`): - `github_app_id`, `github_app_installation_id`, and `github_app_pem` - (the private key, as a normal multiline PEM block). Do this *before* the - first rollout so the seed `PushSecret` doesn't sit red. -3. Merge. The PushSecret seeds OpenBao, the ExternalSecret renders the provider - credential into `github-config`, and the app reconciles once `.github` has - published its first `v*` artifact. +2. **Overwrite the placeholders in OpenBao** with the App's real values — the + keys already exist (seeded by the vault-config Job), so just set them, e.g.: + ```sh + bao kv put -mount=secret infrastructure/github/app \ + app_id="" installation_id="" pem=@app.private-key.pem + ``` + (or via the OpenBao UI). No SOPS, no re-encryption. +3. The `ExternalSecret` refreshes (≤1h) and renders the provider credential into + `github-config`; the app reconciles once `.github` has published its first + `v*` artifact. Until the real values are set, the placeholders materialize a + (non-working) credential and the provider's auth simply fails — isolated to + the leaf `apps` layer, blocking nothing else. ## Adopting an existing repository (Observe-first) diff --git a/k8s/bases/apps/github-config/kustomization.yaml b/k8s/bases/apps/github-config/kustomization.yaml index 17a3267db..e89dddb0e 100644 --- a/k8s/bases/apps/github-config/kustomization.yaml +++ b/k8s/bases/apps/github-config/kustomization.yaml @@ -8,5 +8,4 @@ resources: - ghcr-auth-externalsecret.yaml - external-secret.yaml - provider-config.yaml - - seed-github-app.yaml - sync.yaml diff --git a/k8s/bases/apps/github-config/seed-github-app.yaml b/k8s/bases/apps/github-config/seed-github-app.yaml deleted file mode 100644 index d348bfd09..000000000 --- a/k8s/bases/apps/github-config/seed-github-app.yaml +++ /dev/null @@ -1,42 +0,0 @@ -# PushSecret for the GitHub App credentials used by provider-upjet-github. -# Same durable-seed pattern as seed-hcloud: re-pushing hourly from the SOPS -# source (variables-cluster) re-seeds the keys automatically after OpenBao data -# loss. -# -# Lives with the github-config app (the apps layer), NOT in -# infrastructure/vault-seed/, on purpose: until the maintainer adds the -# github_app_* keys to the prod variables-cluster secret this PushSecret stalls, -# and the apps layer is the leaf of the Flux dependency chain — so the stall can -# never hold `infrastructure` (and through it every deploy) red. A vault-seed -# placement would block the chain. Seed the keys before/with rollout to avoid -# even a transient red apps layer. ---- -apiVersion: external-secrets.io/v1alpha1 -kind: PushSecret -metadata: - name: seed-github-app - namespace: flux-system -spec: - refreshInterval: 1h - secretStoreRefs: - - name: openbao - kind: ClusterSecretStore - selector: - secret: - name: variables-cluster - data: - - match: - secretKey: github_app_id - remoteRef: - remoteKey: infrastructure/github/app - property: app_id - - match: - secretKey: github_app_installation_id - remoteRef: - remoteKey: infrastructure/github/app - property: installation_id - - match: - secretKey: github_app_pem - remoteRef: - remoteKey: infrastructure/github/app - property: pem diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index bbb5e5126..2a3b2650c 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -419,6 +419,25 @@ spec: echo "KV v2 already enabled at secret/." fi + # --- 1b. Seed placeholder GitHub App credentials (only if absent) --- + # provider-upjet-github (the github-config app) authenticates as a + # GitHub App; its credentials live at secret/infrastructure/github/app + # and are read by the github-app-credentials ExternalSecret. We seed + # PLACEHOLDERS here declaratively (no SOPS) ONLY when the secret is + # absent, so the maintainer can overwrite them in place via the + # OpenBao UI/CLI and a later Job re-run won't clobber the real values. + # OpenBao is backed up (Raft snapshots → R2 via Velero), so these + # manually-set values are durable — no GitOps re-seed is needed. + if ! bao kv get -mount=secret infrastructure/github/app >/dev/null 2>&1; then + echo "Seeding placeholder GitHub App credentials at secret/infrastructure/github/app..." + bao kv put -mount=secret infrastructure/github/app \ + app_id="REPLACE_WITH_GITHUB_APP_ID" \ + installation_id="REPLACE_WITH_GITHUB_APP_INSTALLATION_ID" \ + pem="REPLACE_WITH_GITHUB_APP_PRIVATE_KEY_PEM" + else + echo "GitHub App credentials already present at secret/infrastructure/github/app (leaving as-is)." + fi + # --- 2. Enable Kubernetes auth --- if ! bao auth list -format=json | grep -q '"kubernetes/"'; then echo "Enabling Kubernetes auth method..." @@ -623,9 +642,6 @@ spec: path "secret/data/infrastructure/ghcr/*" { capabilities = ["create", "update", "read"] } - path "secret/data/infrastructure/github/*" { - capabilities = ["create", "update", "read"] - } path "secret/data/apps/fleetdm/*" { capabilities = ["create", "update", "read"] } @@ -662,9 +678,6 @@ spec: path "secret/metadata/infrastructure/ghcr/*" { capabilities = ["create", "update", "read"] } - path "secret/metadata/infrastructure/github/*" { - capabilities = ["create", "update", "read"] - } path "secret/metadata/apps/fleetdm/*" { capabilities = ["create", "update", "read"] } From f36f9299896ed1cc3bc20c4f25ce2c9124ed018e Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Wed, 17 Jun 2026 22:01:54 +0200 Subject: [PATCH 6/6] refactor(github): namespaced SecretStore + tenant-owned creds, drop ghcr-auth Tighten the github-config tenant's privileges and move tenant-specific manifests out of the platform: - add a namespaced SecretStore (k8s/bases/apps/github-config/secretstore.yaml) backed by a dedicated 'github-config' OpenBao role scoped to infrastructure/github/* only (mirrors the wedding-app pattern); the tenant no longer uses the broad cluster-scoped openbao ClusterSecretStore - remove infra-github-readonly from the shared external-secrets role bundle (now carried solely by the dedicated github-config role) - move the github-app-credentials ExternalSecret + the namespaced ProviderConfig to devantler-tech/.github deploy/ (tenant-applied); the app SA Role gains verbs on external-secrets.io + github.m.upbound.io accordingly - drop ghcr-auth entirely: the github-config/manifests package is public, so the OCIRepository pulls anonymously (no secretRef, no ExternalSecret, no SA imagePullSecrets) - docs updated Co-Authored-By: Claude Fable 5 --- docs/github-management.md | 16 +++++--- .../apps/github-config/external-secret.yaml | 37 ------------------- .../ghcr-auth-externalsecret.yaml | 35 ------------------ .../apps/github-config/kustomization.yaml | 4 +- .../apps/github-config/provider-config.yaml | 20 ---------- k8s/bases/apps/github-config/rbac.yaml | 18 +++++++++ k8s/bases/apps/github-config/secretstore.yaml | 34 +++++++++++++++++ .../apps/github-config/serviceaccount.yaml | 10 ++--- k8s/bases/apps/github-config/sync.yaml | 4 +- .../infrastructure/vault-config/job.yaml | 15 +++++++- 10 files changed, 85 insertions(+), 108 deletions(-) delete mode 100644 k8s/bases/apps/github-config/external-secret.yaml delete mode 100644 k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml delete mode 100644 k8s/bases/apps/github-config/provider-config.yaml create mode 100644 k8s/bases/apps/github-config/secretstore.yaml diff --git a/docs/github-management.md b/docs/github-management.md index dff8aa548..1c01de8c8 100644 --- a/docs/github-management.md +++ b/docs/github-management.md @@ -35,9 +35,9 @@ CRs one tier later in `infrastructure/`, the workload in `apps/`: |---|---|---| | Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only. `provider.defaultActivations: []` keeps unused provider CRDs inactive. Installs the pkg.crossplane.io CRDs. | | Provider + activation policy | `k8s/providers/hetzner/infrastructure/crossplane/` | The `Provider` package + `DeploymentRuntimeConfig` + `ManagedResourceActivationPolicy` (namespaced MRDs only). One tier after the controller (needs its CRDs), like Coroot/Flagger CRs. Establishes the namespaced github CRDs. | -| The `github-config` app | `k8s/bases/apps/github-config/` | Standard app onboarding: namespace, SA, least-privilege `Role`, GitHub App + ghcr credential `ExternalSecret`s, the namespaced `ProviderConfig`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). | -| Desired state | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The actual `Repository`/ruleset/label managed resources. Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. | -| Credentials | OpenBao `secret/infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`). **Placeholders are seeded declaratively (only-if-absent) by the vault-config bootstrap Job** — no SOPS; the maintainer overwrites them in place via the OpenBao UI/CLI (see below). The `github-app-credentials` `ExternalSecret` reads them and renders the provider's JSON credential format. | +| The `github-config` app (platform scaffolding) | `k8s/bases/apps/github-config/` | namespace, SA, least-privilege `Role`, the namespaced `SecretStore`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). The package is **public**, so the OCIRepository pulls anonymously — no ghcr pull credential. | +| Desired state (tenant-owned) | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The `Repository`/ruleset/label managed resources, **plus the tenant-specific `github-app-credentials` `ExternalSecret` and the namespaced `ProviderConfig`** (applied by the app's own SA). Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. | +| Credentials | OpenBao `secret/infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`). **Placeholders are seeded declaratively (only-if-absent) by the vault-config bootstrap Job** — no SOPS; the maintainer overwrites them in place via the OpenBao UI/CLI (see below). The tenant's `ExternalSecret` reads them via the **namespaced `SecretStore`** (auth role `github-config`, scoped to `infrastructure/github/*` only — never the broad cluster store) and renders the provider's JSON credential format. | Ordering is the normal chain: `apps` `dependsOn` `infrastructure`, so by the time the app reconciles, the provider (infrastructure tier) has installed the @@ -114,8 +114,14 @@ reconcile, then batch the rest. a real GitHub repository. - **Observe-first adoption** — a new managed resource for an existing object always starts `managementPolicies: ["Observe"]` (read-only). -- **Least privilege** — the app SA's authority is a namespaced `Role` over the - GitHub MR API groups only, never aggregated into `edit`. +- **Least privilege (RBAC)** — the app SA's authority is a namespaced `Role` + over the GitHub MR groups + its own `ProviderConfig`/`ExternalSecret` only, + never aggregated into `edit`. +- **Least privilege (secrets)** — the tenant reads OpenBao through a + **namespaced `SecretStore`** backed by a dedicated `github-config` Vault role + scoped to `infrastructure/github/*` only; it never touches the shared + cluster-scoped `openbao` ClusterSecretStore (the `restrict-tenant-secret-stores` + Kyverno policy enforces this for tenant-applied ExternalSecrets). - The GitHub App credential is org-scoped and short-lived per request; the provider pod's egress is FQDN-pinned to `api.github.com` by the `crossplane-system` CiliumNetworkPolicy. diff --git a/k8s/bases/apps/github-config/external-secret.yaml b/k8s/bases/apps/github-config/external-secret.yaml deleted file mode 100644 index 938e20969..000000000 --- a/k8s/bases/apps/github-config/external-secret.yaml +++ /dev/null @@ -1,37 +0,0 @@ -# Materializes the provider-upjet-github credentials Secret from OpenBao. -# The provider expects ONE JSON document ({"app_auth":[…],"owner":…}) whose -# pem_file value carries \n-escaped newlines — the template builds it from the -# three plain keys seeded by seed-github-app.yaml (toJson does the escaping), -# so the PEM can live in OpenBao/SOPS as a normal multiline string. ---- -apiVersion: external-secrets.io/v1 -kind: ExternalSecret -metadata: - name: github-app-credentials - namespace: github-config -spec: - refreshInterval: 1h - secretStoreRef: - name: openbao - kind: ClusterSecretStore - target: - name: github-app-credentials - creationPolicy: Owner - template: - engineVersion: v2 - type: Opaque - data: - credentials: '{"app_auth":[{"id":"{{ .app_id }}","installation_id":"{{ .installation_id }}","pem_file":{{ .pem | toJson }}}],"owner":"devantler-tech"}' - data: - - secretKey: app_id - remoteRef: - key: infrastructure/github/app - property: app_id - - secretKey: installation_id - remoteRef: - key: infrastructure/github/app - property: installation_id - - secretKey: pem - remoteRef: - key: infrastructure/github/app - property: pem diff --git a/k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml b/k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml deleted file mode 100644 index 0ec541513..000000000 --- a/k8s/bases/apps/github-config/ghcr-auth-externalsecret.yaml +++ /dev/null @@ -1,35 +0,0 @@ -# GHCR image-pull secret for the github-config app, sourced from OpenBao (same -# shape and path as every other tenant's ghcr-auth). Consumed by the app's -# OCIRepository (secretRef) and ServiceAccount (imagePullSecrets) to pull the -# cosign-signed manifests artifact published by devantler-tech/.github. -# -# PLATFORM-side resource: reconciled by flux-system (the apps layer applies the -# app scaffolding), never by the app ServiceAccount, so it uses the shared -# cluster-scoped `openbao` ClusterSecretStore reading the platform path -# infrastructure/ghcr/auth — the `restrict-tenant-secret-stores` Kyverno policy -# carves out flux-system-applied ExternalSecrets. -apiVersion: external-secrets.io/v1 -kind: ExternalSecret -metadata: - name: ghcr-auth - namespace: github-config - labels: - app.kubernetes.io/managed-by: ksail -spec: - refreshInterval: 1h - secretStoreRef: - name: openbao - kind: ClusterSecretStore - target: - name: ghcr-auth - creationPolicy: Owner - template: - engineVersion: v2 - type: kubernetes.io/dockerconfigjson - data: - .dockerconfigjson: "{{ .dockerconfigjson }}" - data: - - secretKey: dockerconfigjson - remoteRef: - key: infrastructure/ghcr/auth - property: dockerconfigjson diff --git a/k8s/bases/apps/github-config/kustomization.yaml b/k8s/bases/apps/github-config/kustomization.yaml index e89dddb0e..81340772c 100644 --- a/k8s/bases/apps/github-config/kustomization.yaml +++ b/k8s/bases/apps/github-config/kustomization.yaml @@ -5,7 +5,5 @@ resources: - namespace.yaml - serviceaccount.yaml - rbac.yaml - - ghcr-auth-externalsecret.yaml - - external-secret.yaml - - provider-config.yaml + - secretstore.yaml - sync.yaml diff --git a/k8s/bases/apps/github-config/provider-config.yaml b/k8s/bases/apps/github-config/provider-config.yaml deleted file mode 100644 index afb4cf120..000000000 --- a/k8s/bases/apps/github-config/provider-config.yaml +++ /dev/null @@ -1,20 +0,0 @@ -# Namespaced ProviderConfig (Crossplane v2): authenticates provider-upjet-github -# against the devantler-tech org as a GitHub App (not a PAT — short-lived -# installation tokens, org-scoped, survives personal-account changes). Living in -# the github-config namespace, it is referenced by the namespaced Repository/… -# managed resources in their own namespace, so no cluster-scoped config is -# needed. The credentials Secret is local (same namespace), materialized by -# external-secret.yaml from OpenBao. The github.m.upbound.io CRD is installed by -# the provider package in the infrastructure layer (one tier earlier), so this -# applies once that has reconciled. -apiVersion: github.m.upbound.io/v1beta1 -kind: ProviderConfig -metadata: - name: default - namespace: github-config -spec: - credentials: - source: Secret - secretRef: - name: github-app-credentials - key: credentials diff --git a/k8s/bases/apps/github-config/rbac.yaml b/k8s/bases/apps/github-config/rbac.yaml index 77eed2b87..25a4e194d 100644 --- a/k8s/bases/apps/github-config/rbac.yaml +++ b/k8s/bases/apps/github-config/rbac.yaml @@ -11,6 +11,11 @@ # suffices — no ClusterRole/ClusterRoleBinding, no cluster RBAC. Widen apiGroups # here only as new GitHub MR groups are activated in the # ManagedResourceActivationPolicy. +# +# The .github artifact the SA applies also contains the namespaced ProviderConfig +# (github.m.upbound.io) and the GitHub App credential ExternalSecret +# (external-secrets.io), so the SA needs verbs on those two groups too — still +# confined to this namespace. apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: @@ -23,6 +28,7 @@ rules: - repo.github.m.upbound.io - actions.github.m.upbound.io - enterprise.github.m.upbound.io + - github.m.upbound.io resources: - "*" verbs: @@ -33,6 +39,18 @@ rules: - update - patch - delete + - apiGroups: + - external-secrets.io + resources: + - externalsecrets + verbs: + - get + - list + - watch + - create + - update + - patch + - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding diff --git a/k8s/bases/apps/github-config/secretstore.yaml b/k8s/bases/apps/github-config/secretstore.yaml new file mode 100644 index 000000000..b7ca098ba --- /dev/null +++ b/k8s/bases/apps/github-config/secretstore.yaml @@ -0,0 +1,34 @@ +# Tenant-scoped, namespaced SecretStore for github-config — provisioned by the +# platform so the github-config app only has to ship an ExternalSecret in its +# own deploy/ artifact to read the GitHub App credential. The tenant must NOT +# use the shared cluster-scoped `openbao` ClusterSecretStore (whose role can read +# every infra + app path); the `restrict-tenant-secret-stores` Kyverno policy +# blocks tenant ExternalSecrets from referencing it. +# +# Authenticates as the app's own `github-config` ServiceAccount via the dedicated +# `github-config` OpenBao Kubernetes auth role +# (k8s/bases/infrastructure/vault-config/job.yaml), which carries only the +# `infra-github-readonly` policy — read on secret/data/infrastructure/github/* +# and nothing else. The tenant can therefore reach the GitHub App credential and +# no other secret. +apiVersion: external-secrets.io/v1 +kind: SecretStore +metadata: + name: openbao + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +spec: + provider: + vault: + # openbao-active = unsealed Raft leader only; the plain `openbao` Service + # also round-robins onto sealed-but-Ready standbys (503 "Vault is sealed"). + server: "http://openbao-active.openbao.svc.cluster.local:8200" + path: "secret" + version: "v2" + auth: + kubernetes: + mountPath: "kubernetes" + role: "github-config" + serviceAccountRef: + name: "github-config" diff --git a/k8s/bases/apps/github-config/serviceaccount.yaml b/k8s/bases/apps/github-config/serviceaccount.yaml index 60d94dd4b..1cd813d81 100644 --- a/k8s/bases/apps/github-config/serviceaccount.yaml +++ b/k8s/bases/apps/github-config/serviceaccount.yaml @@ -1,7 +1,9 @@ # The github-config app's ServiceAccount. The app's Flux Kustomization -# (sync.yaml) runs AS this SA, so the GitHub managed resources are applied with -# exactly the namespace-scoped authority granted by rbac.yaml — nothing more. -# imagePullSecrets lets it pull the cosign-signed manifests artifact from GHCR. +# (sync.yaml) runs AS this SA, so the manifests in the .github artifact are +# applied with exactly the namespace-scoped authority granted by rbac.yaml — +# nothing more. No imagePullSecrets: the github-config/manifests package is +# public (it carries only non-secret org config), so the OCIRepository pulls it +# anonymously and this SA runs no pods. apiVersion: v1 kind: ServiceAccount metadata: @@ -10,5 +12,3 @@ metadata: labels: app.kubernetes.io/managed-by: ksail automountServiceAccountToken: false -imagePullSecrets: - - name: ghcr-auth diff --git a/k8s/bases/apps/github-config/sync.yaml b/k8s/bases/apps/github-config/sync.yaml index f6ccfa074..dc9e86726 100644 --- a/k8s/bases/apps/github-config/sync.yaml +++ b/k8s/bases/apps/github-config/sync.yaml @@ -18,9 +18,9 @@ spec: semver: ">=1.0.0" # .github is an invalid OCI path component (leading dot), so the artifact is # published under the `github-config` name — see devantler-tech/.github cd.yaml. + # No secretRef: the package is public (non-secret org config), so it is pulled + # anonymously — the tenant needs no GHCR pull credential. url: oci://ghcr.io/devantler-tech/github-config/manifests - secretRef: - name: ghcr-auth verify: provider: cosign matchOIDCIdentity: diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 2a3b2650c..5c0ce711c 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -704,7 +704,7 @@ spec: bao write auth/kubernetes/role/external-secrets \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets \ - policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,infra-github-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \ + policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \ ttl=1h # Dedicated role for the fleetdm VaultDynamicSecret generator. @@ -727,6 +727,19 @@ spec: policies=app-umami-db-readonly \ ttl=1h + # Dedicated tenant role for github-config. Authenticates the app's + # own `github-config` ServiceAccount (used by its namespaced + # SecretStore, k8s/bases/apps/github-config/secretstore.yaml) and + # carries ONLY `infra-github-readonly` (read on + # secret/data/infrastructure/github/*) — never the broad + # `external-secrets` bundle — so the tenant can read the GitHub App + # credential and nothing else. + bao write auth/kubernetes/role/github-config \ + bound_service_account_names=github-config \ + bound_service_account_namespaces=github-config \ + policies=infra-github-readonly \ + ttl=1h + # Dedicated tenant role for wedding-app. Authenticates the tenant's # own `wedding-app` ServiceAccount (used by its namespaced # SecretStore + deploy/ ExternalSecret) and carries ONLY the