[Feature/Bug] Managed reconciler needs a mechanism to express "async operation in progress" without triggering `ReconcileSuccess` or `Update()`

[jacob-hudson]

This issue is filed against crossplane/crossplane-runtime because the root constraint exists here, not in upjet or any provider.

The managed reconciler in pkg/reconciler/managed unconditionally calls status.MarkConditions(xpv1.ReconcileSuccess()) whenever Observe() returns ResourceUpToDate: true with no error. There is currently no way for an ExternalClient to signal "I am deliberately returning ResourceUpToDate: true to suppress Update(), but reconciliation has not actually succeeded — an async operation is in progress." The Synced condition is therefore always overwritten to True in this path, regardless of what the provider set before returning.

Upjet exploits ResourceUpToDate: true as the mechanism to suppress Update() during async operations, but pays the cost of a permanently false Synced=True signal. This is not a bug in upjet's logic — it is an absence of a contract in crossplane-runtime that would allow the two concerns (suppress Update, signal pending) to be expressed independently.

A corresponding issue has been filed at crossplane/upjet: [link — fill in once filed].

---

What happened?

When an upjet-based provider (e.g. upbound/provider-aws-kafka) is performing a long-running async operation — such as an MSK cluster broker instance type change taking 60+ minutes — the managed resource's Synced condition remains True for the entire duration.

The AsyncOperation=False (Reason: Ongoing) condition is correctly set by upjet throughout. But Synced continues to read True with Reason: ReconcileSuccess, actively misleading operators, alerting systems, and automation that polls Synced to determine whether a resource has converged.

Expected: Synced=False while an async operation is in progress, transitioning back to True when it completes.

Actual: Synced=True/ReconcileSuccess is actively written on every reconcile pass for the full duration of the operation.

Root cause

In pkg/reconciler/managed/reconciler.go, the reconcile loop unconditionally sets ReconcileSuccess when Observe() returns {ResourceExists: true, ResourceUpToDate: true} with no error:

// Pseudocode — see pkg/reconciler/managed/reconciler.go for the full implementation
observation, err := external.Observe(ctx, managed)
if err != nil {
    // ...error path sets ReconcileError...
}
if observation.ResourceUpToDate {
    status.MarkConditions(xpv1.ReconcileSuccess())
    return reconcile.Result{RequeueAfter: r.pollInterval}, nil
}
// ... else calls Update()

There is no branch for "ResourceUpToDate=true but not actually reconciled successfully." The signal is binary: either the resource is up to date (success) or it needs updating (call Update()).

Upjet's ASyncInProgress branch in pkg/controller/external.go is forced to return ResourceUpToDate: true to prevent the reconciler from calling Update() — which could trigger a redundant or harmful second async operation. The consequence is that crossplane-runtime then overwrites any pending/in-progress condition upjet attempted to set, with ReconcileSuccess.

Why this cannot be fixed in upjet alone

The two naive fixes are both broken:

Returning ResourceUpToDate: false: Causes the reconciler to call Update(), potentially queuing a second async operation against an already in-flight AWS change.

Setting ReconcileError before returning: Fails for two independent reasons. First, crossplane-runtime overwrites it with ReconcileSuccess unconditionally on the same pass (since ResourceUpToDate: true is returned). Second, ReconcileError triggers exponential backoff, which is the opposite of what is needed — frequent requeues are necessary so that async completion is detected promptly.

There is also no ReconcilePending constructor in crossplane-runtime v1.x. The only Synced-related constructors available are ReconcileSuccess, ReconcileError, and ReconcilePaused — none model "deferred pending an in-flight async operation."

Proposed solutions

Option 1: Add ReconcilePending condition constructor

Add a new constructor to crossplane-runtime:

// ReconcilePending returns a condition indicating reconciliation is deferred
// pending an in-flight async operation. Unlike ReconcileError, this does not
// trigger exponential backoff.
func ReconcilePending(msg string) xpv1.Condition {
    return xpv1.Condition{
        Type:    xpv1.TypeSynced,
        Status:  corev1.ConditionFalse,
        Reason:  "ReconcilePending",
        Message: msg,
    }
}

The managed reconciler would need a corresponding hook — for example, checking whether the provider set a ReconcilePending condition after Observe() returns but before the ResourceUpToDate branch overwrites it, and if so, honouring it without calling status.MarkConditions(ReconcileSuccess()).

This correctly models the semantic distinction: "reconciliation failed" vs "reconciliation is deferred pending an in-flight operation." However, it requires condition inspection inside the reconciler, which is slightly awkward — see Option 2 for a cleaner alternative.

Option 2 (preferred): Extend ExternalObservation with AsyncOperationInProgress bool

ExternalObservation is defined in crossplane-runtime. A new field would allow providers to express both signals simultaneously — suppress Update() and suppress ReconcileSuccess — without the reconciler needing to inspect conditions written as side effects during Observe():

type ExternalObservation struct {
    ResourceExists           bool
    ResourceUpToDate         bool
    AsyncOperationInProgress bool // suppress Update() AND suppress ReconcileSuccess
    // ...
}

The managed reconciler would handle this in the ResourceUpToDate branch:

if observation.ResourceUpToDate {
    if observation.AsyncOperationInProgress {
        status.MarkConditions(xpv1.ReconcilePending("Async operation in progress"))
        // r.pollInterval is illustrative; a shorter interval would be preferable
        // to detect async completion promptly
        return reconcile.Result{RequeueAfter: r.pollInterval}, nil
    }
    status.MarkConditions(xpv1.ReconcileSuccess())
    return reconcile.Result{RequeueAfter: r.pollInterval}, nil
}

Upjet would then set AsyncOperationInProgress: true in its ASyncInProgress branch instead of relying on ResourceUpToDate: true alone. This keeps semantics within the observation contract and avoids condition inspection logic in the reconciler.

Option 3 (upjet-only, partial): Dedicated async condition type

Accept that Synced semantics are constrained by crossplane-runtime and document the AsyncOperation=False (Reason: Ongoing) condition as the canonical signal for async-in-progress state. This is the least invasive option but leaves Synced incorrect and does not address the root issue. Not recommended as a resolution.

Operational impact

For operators running alerting on Synced=False persisting beyond a threshold, the current behaviour silently swallows the alert: Synced=True is written on every reconcile pass during a genuine in-progress mutation, so the alert condition never triggers. Any tooling or automation that polls Synced to determine whether a resource has converged will receive a false positive for the full duration of the async update.

Related

• Companion upjet issue (direct fix site once crossplane-runtime unblocks it): [Bug] Synced condition remains True during ongoing async operations upjet#609
• Original async failure fix in upjet v1.3.0 (async failures correctly set Synced=False, in-progress case was not addressed): crossplane-contrib/provider-upjet-aws#1164, crossplane/upjet v1.3.0
• Prior discussion on deprecating Synced entirely: crossplane/crossplane-runtime#198

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[bobh66]

/fresh