feat: add EtcdBackup CR for on-demand snapshots to object storage (S3/GCS, pluggable)

[xrl]

Refs #385

What

Adds a new namespaced EtcdBackup CRD that takes a point-in-time snapshot of a referenced EtcdCluster and uploads it to an object store. This implements the roadmap item "Create on-demand backup of a cluster" (docs/roadmap.md).

apiVersion: operator.etcd.io/v1alpha1
kind: EtcdBackup
metadata:
  name: nightly
spec:
  clusterRef: my-cluster
  destination:
    provider: s3                 # s3 | gcs
    prefix: etcd/backups
    secretRef: { name: s3-creds } # optional; omit for IRSA / Workload Identity
    s3: { bucket: my-backups, region: us-east-1 }
  retention: { retainCount: 7 }
status:
  phase: Completed               # Pending|Snapshotting|Uploading|Completed|Failed
  snapshotLocation: s3://my-backups/etcd/backups/my-cluster/nightly-20260617T010203Z.db
  snapshotSizeBytes: 20480
  completionTime: "2026-06-17T01:02:05Z"

Architecture decision (delegated) — and rationale

The task was to choose between a standalone backup operator and growing the core operator, with a recommendation to isolate the heavy cloud SDKs. I chose a dedicated controller + an isolated provider package, wired into the existing operator binary — not a second binary, for now.

Two seams keep the cloud SDKs and exec machinery out of the core reconcile path and make everything unit-testable without cloud creds or a live cluster:

1. Snapshotter (internal/controller/snapshotter.go) produces the snapshot byte stream; the default execs etcdctl snapshot save in a member pod via the exec subresource and streams stdout.
2. pkg/objectstore.Store (interface.go + s3.go + gcs.go) is the minimal Upload/List/Delete surface. A Factory registry dispatches on the provider name, so a new backend is a new file calling objectstore.Register. The AWS and GCP SDKs are imported only from this package — the controller and API types pull in zero cloud SDK.

The snapshot is streamed snapshotter→uploader through an io.Pipe for backpressure/bounded memory.

Why in-binary rather than cmd/backup-manager: because the SDKs are already quarantined behind the interface and the controller is a self-contained file set, the code is trivially liftable into a separate binary later if the cloud-SDK CVE surface becomes a problem for the core operator. Doing it in-binary now keeps this one independently-mergeable PR with one Deployment / one RBAC set / one image (the standard kubebuilder multi-controller layout), and avoids paying the operational cost of a second binary before there's a reason to. The trade-off (core image gains the AWS/GCP SDK dependency tree) is documented and reversible. See docs/snapshot-backups.md.

Credentials

secretRef is optional. Absent it, providers use their ambient chain (IRSA on AWS, Workload Identity on GCP) — the recommended production posture. Present, the keys are accessKeyID/secretAccessKey(/sessionToken) for s3 and serviceAccountJSON for gcs.

Testing

• Unit (pkg/objectstore): JoinKey/destination validation, factory dispatch + duplicate-register panic, and the S3 and GCS stores exercised against in-memory fakes of the narrowed SDK interfaces — no cloud credentials required.
• Unit/envtest (internal/controller): full reconcile happy-path, missing-cluster / no-ready-members / snapshot-error / upload-error → Failed, terminal no-op, snapshot timeout, credential resolution (s3/gcs/ambient/missing-secret), retention pruning, and CRD→destination mapping — all via a fake Snapshotter + fake Store.
• e2e (test/e2e/backup_test.go): a MinIO-backed (S3-compatible, no real cloud) end-to-end test, compile-verified and gated behind ETCD_E2E_BACKUP=true so it doesn't run in the default kind matrix. All its helpers are local to the file to avoid colliding with other in-flight e2e PRs.
• Sample CRs in config/samples/operator_v1alpha1_etcdbackup.yaml.

go build ./..., go vet ./..., golangci-lint run, the objectstore + controller (envtest) suites, and go test -c ./test/e2e/ all pass locally; go mod tidy is a no-op.

Notes for reviewers

• CRDs/deepcopy/RBAC were regenerated with the repo-pinned controller-gen. That pinned version (v0.21.0) is newer than the one that last generated operator.etcd.io_etcdclusters.yaml (v0.20.1), so that file shows a one-line version-annotation bump — a side effect of using the pinned tool, not a hand edit.
• Restore (new cluster from a backup) and scheduled backups (EtcdBackupSchedule) are intentionally out of scope; this CRD + provider interface are the foundation for both.

🤖 Generated with Claude Code

---

PR series — operability fixes & TLS

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

	PR	Lands	After
	Reviewable now — small, any order
🟢	#374	Requeue instead of swallowing the client-cert provisioning error	—
🟢	#391	Grant events.k8s.io RBAC so operator Events are actually recorded	—
🟢	#392	Correlate members[]/leaderID from one health snapshot (consistent leader)	—
🟢	#393	Propagate user-supplied altNames.ipAddresses into certificates	—
🟢	#394	Accept day-suffix validityDuration (365d, 100d12h) as documented	—
🟢	#395	Surface early reconcile errors as a Degraded condition (was empty status)	—
🟢	#379	Configurable reconcile worker pool (--max-concurrent-reconciles)	—
🟢	#369	kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)	—
	TLS stack — in order
🟢	#376	Independent spec.tls.{peer,client} surfaces (breaking alpha API)	—
⚪	#377	TLSReady condition + TLS lifecycle Events	#376
⚪	#378	Multi-member TLS quorum e2e + PeerCANotShared	#377
	Parked as drafts pending #363
⚪	#382	Per-cluster domain metrics on the operator /metrics endpoint	—
⚪	#384	EtcdCluster admission webhooks (consolidating with #328)	#363
⚪→	#386	EtcdBackup CR → object storage (S3/GCS)	#363
⚪	#387	Automatic quorum-loss disaster recovery (bootstrap-latch guarded)	#363

🟢 ready · ⚪ draft

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign jberkus for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign jberkus for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[xrl]

Rebased onto current main; converting to draft while #363 lands.