Skip to content

[BUG] CycleNodeStatus with nil timeoutTimestamp panics in timedOut() and wedges DrainingPods #146

Description

@aamir296

Describe the bug
CycleNodeStatusTransitioner.timedOut() dereferences status.timeoutTimestamp without checking for nil. If a CycleNodeStatus reaches DrainingPods with status.timeoutTimestamp unset, the controller panics, controller-runtime recovers/requeues, and the node cycle never progresses.

Affected code:

  • pkg/controller/cyclenodestatus/transitioner/util.go
  • CycleNodeStatusTransitioner.timedOut()

Current behavior:

return time.Now().After(t.cycleNodeStatus.Status.TimeoutTimestamp.Time)

TimeoutTimestamp is defined as *metav1.Time and is optional, so this can be nil.

To Reproduce

  1. Have a CycleNodeStatus in phase DrainingPods.
  2. Ensure .status.timeoutTimestamp is absent/null.
  3. Let the cyclenodestatus.controller reconcile the object.
  4. Observe a nil pointer panic from timedOut().

Expected behavior
The controller should never panic on a persisted CR state. If timeoutTimestamp is nil, Cyclops should either initialize it, treat the CNS as not timed out yet, or fail the CNS with a clear error.

Screenshots or Logs
Redacted production stack trace:

ERROR  Observed a panic  {"controller": "cyclenodestatus.controller",
  "object": {"name":"<cycle-node-status-name>","namespace":"<operator-namespace>"},
  "panic": "runtime error: invalid memory address or nil pointer dereference",
  "stacktrace": "... cyclenodestatus/transitioner.(*CycleNodeStatusTransitioner).timedOut()
      .../pkg/controller/cyclenodestatus/transitioner/util.go:42
      .../transitioner.(*CycleNodeStatusTransitioner).transitionDraining()
      .../pkg/controller/cyclenodestatus/transitioner/transitions.go:164 ..."}

Kubernetes Cluster Version
AWS EKS 1.33 (v1.33.11-eks-40737a8).

Cyclops Version
v1.10.5

Additional context
This can leave node cycles stuck in DrainingPods / WaitingTermination. In AWS environments, that can strand detached EC2 worker instances that are no longer ASG-managed but continue billing.

Suggested fix:

func (t *CycleNodeStatusTransitioner) timedOut() bool {
    if t.cycleNodeStatus.Status.TimeoutTimestamp == nil {
        return false
    }
    return time.Now().After(t.cycleNodeStatus.Status.TimeoutTimestamp.Time)
}

It would be helpful to add a regression test covering a CNS in DrainingPods with nil status.timeoutTimestamp.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions