Skip to content

feat(controller): reconcile dependents when their cluster becomes ready#138

Open
bmorton wants to merge 9 commits into
mainfrom
feat/reconcile-cluster-readiness-watch
Open

feat(controller): reconcile dependents when their cluster becomes ready#138
bmorton wants to merge 9 commits into
mainfrom
feat/reconcile-cluster-readiness-watch

Conversation

@bmorton

@bmorton bmorton commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Problem

When a TemporalCluster and a dependent object (e.g. TemporalNamespace) are created at the same time, the dependent registers much later than the cluster becomes ready. Two mechanics cause this:

  1. No event-driven trigger. The cluster-dependent controllers (TemporalNamespace, TemporalSchedule, TemporalSearchAttribute, TemporalWorkflowRun, TemporalClusterConnection) resolve their target via resolveTarget and, when the cluster isn't Ready, just requeue on a timer. None of them watch the TemporalCluster/TemporalDevServer, so nothing re-triggers a reconcile the moment the cluster flips to Ready.
  2. Exponential backoff on transient not-ready. Once the cluster reports Ready, the controllers immediately call the frontend. If it's up but not yet serving RPCs, the call returns a raw error and the controllers return ctrl.Result{}, err — so controller-runtime applies exponential backoff (default max ~16.7 min). The object is "caught in a backoff" and registers long after the cluster is usable.

This also slowed the e2e suites, which sequenced cluster-first-then-dependent to sidestep the delay.

Changes

  • Shared helpers (internal/controller/cluster_watch.go): a clusterReadinessChanged predicate (fires on create, generation change, or a Ready-condition transition — ignores routine status writes), a refTargets matcher (same-namespace, empty Kind defaults to TemporalCluster), an isTransientClusterErr classifier (gRPC Unavailable/DeadlineExceeded/Canceled), and clusterUnavailableRequeue = 5s. New reason const ReasonFrontendUnavailable.
  • Event-driven readiness: each dependent controller now .Watches TemporalCluster and TemporalDevServer, mapping a changed target to same-namespace CRs referencing it. Dependents reconcile the instant their cluster becomes Ready.
  • No backoff on transient not-ready: the four RPC-registering controllers convert transient frontend errors into a short fixed requeue (Ready=False / FrontendUnavailable) instead of erroring. Terminal errors still surface. TemporalClusterConnection already swallowed transient per-peer errors, so it gets the watch only.
  • e2e: test/e2e/namespace now applies the cluster and namespace concurrently, proving the fix and trimming wall-clock time.

Testing

  • make lint — 0 issues
  • make test (unit + envtest) — all pass, no generated drift
  • New unit tests for the predicate, matcher, transient classifier, and map function; new envtest spec asserting a transient frontend error yields a short requeue (not a returned error).

No API/CRD schema changes.

Design & plan: docs/superpowers/specs/2026-07-24-cluster-readiness-watch-design.md, docs/superpowers/plans/2026-07-24-cluster-readiness-watch.md.

bmorton and others added 9 commits July 24, 2026 05:21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Brian Morton <brian@mmmhm.com>
Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 136e3309-64e9-4f7a-8a1d-4bab174e96e1
Signed-off-by: Brian Morton <brian@mmmhm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant