diff --git a/AGENTS.md b/AGENTS.md index 0c08a4941..4774bcdf1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -198,7 +198,7 @@ You **cannot** decrypt existing secrets without the proper Age keys. For local d - **Small, focused changes** — one concern per PR. - **Never commit plaintext secrets** — all secrets must be SOPS-encrypted with the `.enc.yaml` suffix. - **Base files are immutable** — use Kustomize `patches:` in overlays; never edit `k8s/bases/` directly from a provider or cluster overlay. -- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. +- **Flux dependency order** — `bootstrap` → `infrastructure-controllers` → `infrastructure` → `apps`. One prod-only side layer hangs off `infrastructure` without gating `apps`: `infrastructure-overprovisioning` (apply-only autoscaler buffer). Declarative GitHub org management runs as a normal **app** (`github-config`) consuming the `devantler-tech/.github` artifact, with its Crossplane provider in the `infrastructure` layer — see [`docs/github-management.md`](docs/github-management.md). ### Infrastructure File Structure Convention diff --git a/docs/github-management.md b/docs/github-management.md new file mode 100644 index 000000000..1c01de8c8 --- /dev/null +++ b/docs/github-management.md @@ -0,0 +1,127 @@ +# Declarative GitHub management + +The devantler-tech GitHub org is managed declaratively as GitOps: GitHub +repositories (and, over time, rulesets, labels, branch protection, …) are +**Crossplane managed resources** reconciled by +[provider-upjet-github](https://github.com/crossplane-contrib/provider-upjet-github) +against the GitHub API. Change a manifest, merge, and Flux + Crossplane converge +GitHub to it — including reverting out-of-band UI edits. + +The desired state does **not** live in this repo. It lives in +[`devantler-tech/.github`](https://github.com/devantler-tech/.github) under +`deploy/`, onboarded here as a **normal app** (`github-config`) using the same +pattern as every other tenant (cosign-verified OCI artifact → namespace-scoped +`Kustomization`) — only the artifact carries managed resources instead of a +workload. There is **no** bespoke Flux Kustomization for GitHub; it rides the +existing `apps` layer. + +## Why an app, and why namespaced + +GitHub's managed resources ship in two flavours: cluster-scoped +(`repo.github.upbound.io`) and **namespaced** (`repo.github.m.upbound.io`, +Crossplane v2). We use the **namespaced** variants. That choice is what lets +`github-config` be a *genuinely isolated* app: a namespaced managed resource can +be applied by a namespace-scoped `ServiceAccount` with a plain `Role` — no +`ClusterRoleBinding`, no cluster RBAC. The app's authority to touch GitHub is +confined to its own namespace and is **not** aggregated into the built-in `edit` +ClusterRole, so no other tenant inherits it. + +## Architecture + +Three tiers, each in its conventional place — controller in `controllers/`, its +CRs one tier later in `infrastructure/`, the workload in `apps/`: + +| Piece | Where | Notes | +|---|---|---| +| Crossplane core | `k8s/providers/hetzner/infrastructure/controllers/crossplane/` | HelmRelease; prod-only. `provider.defaultActivations: []` keeps unused provider CRDs inactive. Installs the pkg.crossplane.io CRDs. | +| Provider + activation policy | `k8s/providers/hetzner/infrastructure/crossplane/` | The `Provider` package + `DeploymentRuntimeConfig` + `ManagedResourceActivationPolicy` (namespaced MRDs only). One tier after the controller (needs its CRDs), like Coroot/Flagger CRs. Establishes the namespaced github CRDs. | +| The `github-config` app (platform scaffolding) | `k8s/bases/apps/github-config/` | namespace, SA, least-privilege `Role`, the namespaced `SecretStore`, and the cosign-verified `OCIRepository` + `Kustomization`. Applied by the existing `apps` Flux Kustomization. Prod-only in practice (the docker provider deploys no apps). The package is **public**, so the OCIRepository pulls anonymously — no ghcr pull credential. | +| Desired state (tenant-owned) | [`devantler-tech/.github`](https://github.com/devantler-tech/.github) `deploy/` | The `Repository`/ruleset/label managed resources, **plus the tenant-specific `github-app-credentials` `ExternalSecret` and the namespaced `ProviderConfig`** (applied by the app's own SA). Published as a cosign-signed OCI artifact to `ghcr.io/devantler-tech/github-config/manifests` on `v*` tags by the shared `publish-manifests` reusable workflow. | +| Credentials | OpenBao `secret/infrastructure/github/app` | GitHub App (`app_id`, `installation_id`, `pem`). **Placeholders are seeded declaratively (only-if-absent) by the vault-config bootstrap Job** — no SOPS; the maintainer overwrites them in place via the OpenBao UI/CLI (see below). The tenant's `ExternalSecret` reads them via the **namespaced `SecretStore`** (auth role `github-config`, scoped to `infrastructure/github/*` only — never the broad cluster store) and renders the provider's JSON credential format. | + +Ordering is the normal chain: `apps` `dependsOn` `infrastructure`, so by the +time the app reconciles, the provider (infrastructure tier) has installed the +namespaced github CRDs the `ProviderConfig` and managed resources need. On a +fresh install the app's `Kustomization` retries benignly until those CRDs exist. + +The GitHub App credentials are **not** SOPS-managed. The vault-config bootstrap +Job seeds **placeholder** values into OpenBao (`secret/infrastructure/github/app`) +**only when the secret is absent**, and the maintainer overwrites them in place +(below). A later Job re-run never clobbers the real values (the seed is guarded +by an existence check), and OpenBao is backed up (Raft snapshots → R2 via +Velero), so the manually-set values are durable without a GitOps source of truth. + +## Credential setup (one-time, maintainer) + +1. Create a **GitHub App** on the devantler-tech org (Settings → Developer + settings → GitHub Apps): permissions *Repository: Administration + (read/write), Contents (read)* and *Organization: Administration + (read/write)* to start — widen as more resource kinds come under + management. Install it on **all repositories** of the org. No webhook. +2. **Overwrite the placeholders in OpenBao** with the App's real values — the + keys already exist (seeded by the vault-config Job), so just set them, e.g.: + ```sh + bao kv put -mount=secret infrastructure/github/app \ + app_id="" installation_id="" pem=@app.private-key.pem + ``` + (or via the OpenBao UI). No SOPS, no re-encryption. +3. The `ExternalSecret` refreshes (≤1h) and renders the provider credential into + `github-config`; the app reconciles once `.github` has published its first + `v*` artifact. Until the real values are set, the placeholders materialize a + (non-working) credential and the provider's auth simply fails — isolated to + the leaf `apps` layer, blocking nothing else. + +## Adopting an existing repository (Observe-first) + +Desired-state manifests live in `devantler-tech/.github` under `deploy/`. Never +let Crossplane *create* what already exists — **bind** to it: + +1. In `.github`, add `deploy/repositories/.yaml`: a namespaced + `Repository` (`apiVersion: repo.github.m.upbound.io/v1alpha1`), external-name + annotation = the repo name, `managementPolicies: ["Observe"]`, + `providerConfigRef: {kind: ProviderConfig, name: default}`. (Namespaced MRs + have no `deletionPolicy`; "never delete" is expressed by omitting `Delete` + from `managementPolicies`.) +2. Tag a `v*` release of `.github`; wait for the managed resource to become + Ready and inspect what Crossplane observed: + `kubectl -n github-config get repository -o yaml` → + `status.atProvider` is the live GitHub state. +3. Backfill `spec.forProvider` from `status.atProvider` (the fields you want to + pin: `visibility`, `hasIssues`, `deleteBranchOnMerge`, …). +4. Promote `managementPolicies` to the full set **except `Delete`** + (`["Observe", "Create", "Update", "LateInitialize"]`). From now on GitHub + follows the manifest, drift included, but deleting the CR never deletes the + real repository. + +Roll the org over incrementally: one or two repos per release, watch them +reconcile, then batch the rest. + +## Adding more GitHub resource kinds + +1. Activate the namespaced MRD in + `k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml` + (plural.group form, e.g. `repositoryrulesets.repo.github.m.upbound.io`); add + its API group to the app `Role` in `k8s/bases/apps/github-config/rbac.yaml` + if it is a new group. +2. Add the managed resources in `.github`'s `deploy/` — same Observe-first flow + where an external object already exists. Each active MRD costs the apiserver + ~3 MiB; activate only what is used. + +## Safety rails + +- **Never `Delete`** — managed resources omit `Delete` from + `managementPolicies`, so neither CR deletion nor a Flux prune can ever delete + a real GitHub repository. +- **Observe-first adoption** — a new managed resource for an existing object + always starts `managementPolicies: ["Observe"]` (read-only). +- **Least privilege (RBAC)** — the app SA's authority is a namespaced `Role` + over the GitHub MR groups + its own `ProviderConfig`/`ExternalSecret` only, + never aggregated into `edit`. +- **Least privilege (secrets)** — the tenant reads OpenBao through a + **namespaced `SecretStore`** backed by a dedicated `github-config` Vault role + scoped to `infrastructure/github/*` only; it never touches the shared + cluster-scoped `openbao` ClusterSecretStore (the `restrict-tenant-secret-stores` + Kyverno policy enforces this for tenant-applied ExternalSecrets). +- The GitHub App credential is org-scoped and short-lived per request; the + provider pod's egress is FQDN-pinned to `api.github.com` by the + `crossplane-system` CiliumNetworkPolicy. diff --git a/k8s/bases/apps/github-config/kustomization.yaml b/k8s/bases/apps/github-config/kustomization.yaml new file mode 100644 index 000000000..81340772c --- /dev/null +++ b/k8s/bases/apps/github-config/kustomization.yaml @@ -0,0 +1,9 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - serviceaccount.yaml + - rbac.yaml + - secretstore.yaml + - sync.yaml diff --git a/k8s/bases/apps/github-config/namespace.yaml b/k8s/bases/apps/github-config/namespace.yaml new file mode 100644 index 000000000..d582a6b11 --- /dev/null +++ b/k8s/bases/apps/github-config/namespace.yaml @@ -0,0 +1,15 @@ +# Namespace for the github-config app: where the devantler-tech org's GitHub +# state is reconciled. No pods run here — it holds the namespaced Crossplane +# managed resources (repo.github.m.upbound.io/…), the namespaced ProviderConfig, +# the GitHub App credential, and the Flux source/sync that pull the org's +# desired state from the devantler-tech/.github repo's published artifact. +# (A default-deny CiliumNetworkPolicy is auto-generated by the add-default-deny +# Kyverno policy; no per-app NetworkPolicy is needed since there are no pods.) +apiVersion: v1 +kind: Namespace +metadata: + name: github-config + labels: + app.kubernetes.io/managed-by: ksail + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: latest diff --git a/k8s/bases/apps/github-config/rbac.yaml b/k8s/bases/apps/github-config/rbac.yaml new file mode 100644 index 000000000..25a4e194d --- /dev/null +++ b/k8s/bases/apps/github-config/rbac.yaml @@ -0,0 +1,69 @@ +# Least-privilege RBAC for the github-config app's ServiceAccount. +# +# Deliberately NOT the stock app pattern (a RoleBinding to the built-in `edit` +# ClusterRole) and deliberately NOT aggregated into `edit`: GitHub managed +# resources let their holder manipulate the org's real repositories, so that +# authority must be confined to THIS app — never inherited by every tenant that +# has an `edit` binding. Instead, a dedicated namespaced Role grants the SA full +# verbs over the namespaced provider-upjet-github managed-resource API groups, +# and only within the github-config namespace. Because the managed resources are +# namespaced (Crossplane v2 `*.github.m.upbound.io`), a plain Role + RoleBinding +# suffices — no ClusterRole/ClusterRoleBinding, no cluster RBAC. Widen apiGroups +# here only as new GitHub MR groups are activated in the +# ManagedResourceActivationPolicy. +# +# The .github artifact the SA applies also contains the namespaced ProviderConfig +# (github.m.upbound.io) and the GitHub App credential ExternalSecret +# (external-secrets.io), so the SA needs verbs on those two groups too — still +# confined to this namespace. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: github-config-managed-resources + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +rules: + - apiGroups: + - repo.github.m.upbound.io + - actions.github.m.upbound.io + - enterprise.github.m.upbound.io + - github.m.upbound.io + resources: + - "*" + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - external-secrets.io + resources: + - externalsecrets + verbs: + - get + - list + - watch + - create + - update + - patch + - delete +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: github-config-managed-resources + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: github-config-managed-resources +subjects: + - kind: ServiceAccount + name: github-config + namespace: github-config diff --git a/k8s/bases/apps/github-config/secretstore.yaml b/k8s/bases/apps/github-config/secretstore.yaml new file mode 100644 index 000000000..b7ca098ba --- /dev/null +++ b/k8s/bases/apps/github-config/secretstore.yaml @@ -0,0 +1,34 @@ +# Tenant-scoped, namespaced SecretStore for github-config — provisioned by the +# platform so the github-config app only has to ship an ExternalSecret in its +# own deploy/ artifact to read the GitHub App credential. The tenant must NOT +# use the shared cluster-scoped `openbao` ClusterSecretStore (whose role can read +# every infra + app path); the `restrict-tenant-secret-stores` Kyverno policy +# blocks tenant ExternalSecrets from referencing it. +# +# Authenticates as the app's own `github-config` ServiceAccount via the dedicated +# `github-config` OpenBao Kubernetes auth role +# (k8s/bases/infrastructure/vault-config/job.yaml), which carries only the +# `infra-github-readonly` policy — read on secret/data/infrastructure/github/* +# and nothing else. The tenant can therefore reach the GitHub App credential and +# no other secret. +apiVersion: external-secrets.io/v1 +kind: SecretStore +metadata: + name: openbao + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +spec: + provider: + vault: + # openbao-active = unsealed Raft leader only; the plain `openbao` Service + # also round-robins onto sealed-but-Ready standbys (503 "Vault is sealed"). + server: "http://openbao-active.openbao.svc.cluster.local:8200" + path: "secret" + version: "v2" + auth: + kubernetes: + mountPath: "kubernetes" + role: "github-config" + serviceAccountRef: + name: "github-config" diff --git a/k8s/bases/apps/github-config/serviceaccount.yaml b/k8s/bases/apps/github-config/serviceaccount.yaml new file mode 100644 index 000000000..1cd813d81 --- /dev/null +++ b/k8s/bases/apps/github-config/serviceaccount.yaml @@ -0,0 +1,14 @@ +# The github-config app's ServiceAccount. The app's Flux Kustomization +# (sync.yaml) runs AS this SA, so the manifests in the .github artifact are +# applied with exactly the namespace-scoped authority granted by rbac.yaml — +# nothing more. No imagePullSecrets: the github-config/manifests package is +# public (it carries only non-secret org config), so the OCIRepository pulls it +# anonymously and this SA runs no pods. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +automountServiceAccountToken: false diff --git a/k8s/bases/apps/github-config/sync.yaml b/k8s/bases/apps/github-config/sync.yaml new file mode 100644 index 000000000..dc9e86726 --- /dev/null +++ b/k8s/bases/apps/github-config/sync.yaml @@ -0,0 +1,61 @@ +# The github-config app's source + sync — the same delivery pattern as every +# other app/tenant (cosign-verified OCI artifact → namespace-scoped +# Kustomization), but the artifact carries Crossplane managed resources (the +# org's desired GitHub state) instead of a workload. The desired state lives in +# devantler-tech/.github under deploy/ and is published as a signed artifact by +# that repo's cd.yaml on v* tags. +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: OCIRepository +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail +spec: + interval: 1m + ref: + semver: ">=1.0.0" + # .github is an invalid OCI path component (leading dot), so the artifact is + # published under the `github-config` name — see devantler-tech/.github cd.yaml. + # No secretRef: the package is public (non-secret org config), so it is pulled + # anonymously — the tenant needs no GHCR pull credential. + url: oci://ghcr.io/devantler-tech/github-config/manifests + verify: + provider: cosign + matchOIDCIdentity: + # Keyless-signed by the shared publish-manifests reusable workflow that + # .github's cd.yaml calls. Signing runs INSIDE that reusable workflow, so + # the cosign OIDC subject is the reusable workflow's path (devantler-tech/ + # reusable-workflows), NOT .github's own cd.yaml — same convention as the + # publish-app.yaml-signed app artifacts. + - issuer: '^https://token\.actions\.githubusercontent\.com$' + subject: '^https://github\.com/devantler-tech/reusable-workflows/\.github/workflows/publish-manifests\.yaml@.+$' +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: github-config + namespace: github-config + labels: + app.kubernetes.io/managed-by: ksail + annotations: + ksail.devantler.tech/reconcile-exclude: "true" +spec: + interval: 1m + timeout: 5m + retryInterval: 2m + path: . + prune: true + wait: true + force: true + # Runs as the app SA: the GitHub managed resources are applied with only the + # namespace-scoped authority in rbac.yaml. This Kustomization is itself applied + # by the `apps` Flux Kustomization; it must NOT dependsOn it (deadlock) — it + # retries benignly until the provider (infrastructure layer) has established + # the namespaced CRDs. + serviceAccountName: github-config + sourceRef: + kind: OCIRepository + name: github-config + targetNamespace: github-config diff --git a/k8s/bases/apps/kustomization.yaml b/k8s/bases/apps/kustomization.yaml index f01718fe4..275a82879 100644 --- a/k8s/bases/apps/kustomization.yaml +++ b/k8s/bases/apps/kustomization.yaml @@ -11,6 +11,11 @@ resources: # infra-layer plumbing (OpenBao policies/seeds, NetworkPolicies, Kubescape # security exceptions, prod config-map vars) is intentionally left idle. # - fleetdm/ + # Declarative GitHub org management: the devantler-tech/.github repo's deploy/ + # artifact (Crossplane managed resources) onboarded as a normal app/tenant. + # Prod-only in practice (its Crossplane provider is Hetzner-only); the docker + # provider deploys no apps, so it is naturally excluded from local clusters. + - github-config/ - headlamp/ - homepage/ - umami/ diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 86d81a29e..5c0ce711c 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -419,6 +419,25 @@ spec: echo "KV v2 already enabled at secret/." fi + # --- 1b. Seed placeholder GitHub App credentials (only if absent) --- + # provider-upjet-github (the github-config app) authenticates as a + # GitHub App; its credentials live at secret/infrastructure/github/app + # and are read by the github-app-credentials ExternalSecret. We seed + # PLACEHOLDERS here declaratively (no SOPS) ONLY when the secret is + # absent, so the maintainer can overwrite them in place via the + # OpenBao UI/CLI and a later Job re-run won't clobber the real values. + # OpenBao is backed up (Raft snapshots → R2 via Velero), so these + # manually-set values are durable — no GitOps re-seed is needed. + if ! bao kv get -mount=secret infrastructure/github/app >/dev/null 2>&1; then + echo "Seeding placeholder GitHub App credentials at secret/infrastructure/github/app..." + bao kv put -mount=secret infrastructure/github/app \ + app_id="REPLACE_WITH_GITHUB_APP_ID" \ + installation_id="REPLACE_WITH_GITHUB_APP_INSTALLATION_ID" \ + pem="REPLACE_WITH_GITHUB_APP_PRIVATE_KEY_PEM" + else + echo "GitHub App credentials already present at secret/infrastructure/github/app (leaving as-is)." + fi + # --- 2. Enable Kubernetes auth --- if ! bao auth list -format=json | grep -q '"kubernetes/"'; then echo "Enabling Kubernetes auth method..." @@ -479,6 +498,18 @@ spec: } POLICY + # Read the GitHub App credential for provider-upjet-github + # (declarative GitHub management). Consumed by the + # github-app-credentials ExternalSecret in the github-config + # namespace via the `external-secrets` ClusterSecretStore role; + # seeded from variables-cluster by the seed-github-app PushSecret + # (see k8s/bases/apps/github-config/). + bao policy write infra-github-readonly - <<'POLICY' + path "secret/data/infrastructure/github/*" { + capabilities = ["read"] + } + POLICY + bao policy write app-fleetdm-readonly - <<'POLICY' path "secret/data/apps/fleetdm/*" { capabilities = ["read"] @@ -696,6 +727,19 @@ spec: policies=app-umami-db-readonly \ ttl=1h + # Dedicated tenant role for github-config. Authenticates the app's + # own `github-config` ServiceAccount (used by its namespaced + # SecretStore, k8s/bases/apps/github-config/secretstore.yaml) and + # carries ONLY `infra-github-readonly` (read on + # secret/data/infrastructure/github/*) — never the broad + # `external-secrets` bundle — so the tenant can read the GitHub App + # credential and nothing else. + bao write auth/kubernetes/role/github-config \ + bound_service_account_names=github-config \ + bound_service_account_namespaces=github-config \ + policies=infra-github-readonly \ + ttl=1h + # Dedicated tenant role for wedding-app. Authenticates the tenant's # own `wedding-app` ServiceAccount (used by its namespaced # SecretStore + deploy/ ExternalSecret) and carries ONLY the diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml new file mode 100644 index 000000000..4df1112b1 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-release.yaml @@ -0,0 +1,50 @@ +# Crossplane core — the control plane for managing EXTERNAL resources (GitHub +# org/repo state, see ../../../github/) as Kubernetes custom resources. Prod-only: +# external-state reconcilers must run on exactly one always-on cluster, or two +# clusters would race to converge the same GitHub org (same reasoning as +# external-dns living in this overlay). +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: crossplane + namespace: crossplane-system + labels: + helm.toolkit.fluxcd.io/crds: enabled + helm.toolkit.fluxcd.io/remediation: enabled +spec: + interval: 10m + timeout: 10m + install: + crds: CreateReplace + upgrade: + crds: CreateReplace + chart: + spec: + chart: crossplane + version: 2.3.2 + sourceRef: + kind: HelmRepository + name: crossplane + # https://github.com/crossplane/crossplane/blob/main/cluster/charts/crossplane/values.yaml + values: + provider: + # Providers in this cluster declare SafeStart, so their ManagedResource- + # Definitions install Inactive. An empty list here stops the chart from + # creating the "default" activate-everything MRAP — each MRD costs the + # apiserver ~3MiB once active, and provider-upjet-github alone ships 77. + # The MRDs we actually use are activated by an explicit + # ManagedResourceActivationPolicy in + # providers/hetzner/infrastructure/crossplane/. + defaultActivations: [] + resourcesCrossplane: + requests: + cpu: 25m + memory: 256Mi + limits: + memory: 512Mi + resourcesRBACManager: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml new file mode 100644 index 000000000..961f20061 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/helm-repository.yaml @@ -0,0 +1,7 @@ +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: crossplane + namespace: crossplane-system +spec: + url: https://charts.crossplane.io/stable diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml new file mode 100644 index 000000000..e6c1f2415 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/kustomization.yaml @@ -0,0 +1,8 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - helm-repository.yaml + - helm-release.yaml + - networkpolicy.yaml diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml new file mode 100644 index 000000000..8e6246cea --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: crossplane-system diff --git a/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml b/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml new file mode 100644 index 000000000..3453a63b0 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/crossplane/networkpolicy.yaml @@ -0,0 +1,50 @@ +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: allow-crossplane + namespace: crossplane-system +spec: + # Namespace-wide: covers the Crossplane core pod (package manager pulls + # provider OCI packages itself, in-pod — unlike runtime images, which the + # kubelet pulls at node level) AND the provider pods Crossplane creates here + # (provider-upjet-github talks to the GitHub API). + endpointSelector: {} + egress: + # Kube API for watching/updating managed resources and leader election. + - toEntities: + - kube-apiserver + # Pinned by FQDN rather than world:443 so a compromised pod cannot reach + # arbitrary external services: + # ghcr.io + pkg-containers.githubusercontent.com — provider package + # pulls by the core package manager (ghcr blob fetches redirect to + # pkg-containers.githubusercontent.com). + # api.github.com — the GitHub REST/GraphQL API used by + # provider-upjet-github to reconcile org/repo state. + - toFQDNs: + - matchName: "ghcr.io" + - matchName: "pkg-containers.githubusercontent.com" + - matchName: "api.github.com" + toPorts: + - ports: + - port: "443" + protocol: TCP + # DNS resolution. rules.dns activates Cilium's L7 DNS proxy so the + # toFQDNs rule above can be enforced; the visibility list is restricted + # to the names these pods actually need to resolve, closing off DNS + # tunneling exfil (same pattern as external-dns). + - toEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: kube-system + k8s-app: kube-dns + toPorts: + - ports: + - port: "53" + protocol: UDP + - port: "53" + protocol: TCP + rules: + dns: + - matchName: "ghcr.io" + - matchName: "pkg-containers.githubusercontent.com" + - matchName: "api.github.com" + - matchPattern: "*.cluster.local" diff --git a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml index 7ef4d98ce..4dbd5a030 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml @@ -13,6 +13,13 @@ resources: # so the policy lives here, not in the base — see cilium/ for the rationale. - cilium/ - coredns/ + # Prod-only: Crossplane core for declarative GitHub management — external + # reconcilers must run on exactly one always-on cluster. The provider + # package + managed resources live in the dedicated `github` layer + # (../../github/), applied by its own Flux Kustomization. (Crossplane core + # has no ExternalSecret dependency, so unlike external-dns it stays in this + # wait-gated controllers layer.) + - crossplane/ # NB: external-dns moved to ../external-dns/ (the `infrastructure` layer): # its Cloudflare token now arrives via an OpenBao ExternalSecret, and the # ClusterSecretStore only exists once `infrastructure` reconciles — in this diff --git a/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml b/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml new file mode 100644 index 000000000..dbc6e5ec0 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/crossplane/kustomization.yaml @@ -0,0 +1,6 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - provider.yaml + - managed-resource-activation-policy.yaml diff --git a/k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml b/k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml new file mode 100644 index 000000000..222e5ae3e --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/crossplane/managed-resource-activation-policy.yaml @@ -0,0 +1,23 @@ +# provider-upjet-github declares SafeStart, so its ManagedResourceDefinitions +# install Inactive; only the ones named here become CRDs (each active MRD costs +# the apiserver ~3MiB, and the provider ships ~78). The chart's +# activate-everything default MRAP is disabled via +# `provider.defaultActivations: []` in the crossplane HelmRelease. +# +# We activate the NAMESPACED (Crossplane v2, `*.github.m.upbound.io`) variants +# only — namespaced managed resources can be applied by a namespace-scoped +# tenant ServiceAccount with a plain Role (no cluster RBAC), which is what makes +# github-config a genuinely isolated app/tenant. Extend this list as more of the +# org's GitHub surface comes under management (org rulesets, Actions secrets, …) +# — see docs/github-management.md. +apiVersion: apiextensions.crossplane.io/v1alpha1 +kind: ManagedResourceActivationPolicy +metadata: + name: github +spec: + activate: + - repositories.repo.github.m.upbound.io + - defaultbranches.repo.github.m.upbound.io + - branchprotections.repo.github.m.upbound.io + - repositoryrulesets.repo.github.m.upbound.io + - issuelabels.repo.github.m.upbound.io diff --git a/k8s/providers/hetzner/infrastructure/crossplane/provider.yaml b/k8s/providers/hetzner/infrastructure/crossplane/provider.yaml new file mode 100644 index 000000000..62a9bbd9b --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/crossplane/provider.yaml @@ -0,0 +1,39 @@ +# provider-upjet-github — Upjet-generated Crossplane provider wrapping the +# integrations/github Terraform provider. Lives in the `infrastructure` layer +# (not `infrastructure/controllers/`) because the Provider/DeploymentRuntimeConfig +# are pkg.crossplane.io CRs that need the Crossplane CRDs the core HelmRelease in +# controllers/crossplane/ installs first — the same controller-then-CR ordering +# as Coroot and Flagger. Installing this provider also establishes the namespaced +# github CRDs the github-config app (k8s/bases/apps/github-config/) consumes. +# ghcr.io is the project's publish registry (xpkg.crossplane.io merely proxies +# it) and is already the trust root for every first-party image in this cluster. +apiVersion: pkg.crossplane.io/v1 +kind: Provider +metadata: + name: provider-upjet-github +spec: + package: ghcr.io/crossplane-contrib/provider-upjet-github:v0.19.1 + runtimeConfigRef: + name: provider-upjet-github +--- +# Bounded resources for the provider runtime pod (created by Crossplane, so +# the HelmRelease values can't reach it). The empty selector is the documented +# placeholder — Crossplane fills in the real match labels. +apiVersion: pkg.crossplane.io/v1beta1 +kind: DeploymentRuntimeConfig +metadata: + name: provider-upjet-github +spec: + deploymentTemplate: + spec: + selector: {} + template: + spec: + containers: + - name: package-runtime + resources: + requests: + cpu: 10m + memory: 128Mi + limits: + memory: 512Mi diff --git a/k8s/providers/hetzner/infrastructure/kustomization.yaml b/k8s/providers/hetzner/infrastructure/kustomization.yaml index fcb3fda92..1f7c8b91f 100644 --- a/k8s/providers/hetzner/infrastructure/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/kustomization.yaml @@ -4,6 +4,13 @@ kind: Kustomization resources: - ../../../bases/infrastructure/ - cluster-issuers/ + # Crossplane provider-upjet-github package + activation policy. Lives in this + # `infrastructure` layer (not `infrastructure/controllers/`) because these are + # pkg.crossplane.io CRs that need the Crossplane CRDs the core HelmRelease in + # controllers/crossplane/ installs first — same controller-then-CR ordering as + # Coroot/Flagger. Establishes the namespaced github CRDs the github-config app + # (bases/apps/github-config/) consumes. Prod-only (Crossplane is Hetzner-only). + - crossplane/ # external-dns lives in this `infrastructure` layer (not # `infrastructure/controllers/`) because its Cloudflare token is a user-fed # OpenBao secret materialised by an ExternalSecret, and the openbao