Determine final list of fields for 0.2 Status

[jberkus]

Per #136 we need to decide the final list of fields for cluster status. Here's the current proposal from @ballista01 :

status:
  observedGeneration: 5                    # NEW: Top-level field indicating the latest .metadata.generation processed by the controller
  currentReplicas: 3                       # Unchanged: Matches the StatefulSet replica count
  currentVersion: 3.6.0-rc.3               # Unchanged: Represents the running etcd version of the cluster
  leaderId: 975eab669fe7b560               # Unchanged: Leader member ID for status observability
  memberCount: 3                           # Unchanged: Total number of etcd members
  readyReplicas: 3

  # lastDefragTime could be optionally added if controller manages defragmentation
  # lastDefragTime: "2025-07-20T12:00:00Z"   # OPTIONAL: Cluster-level defragmentation completion time; not included in this PR

  members:
    - id: 22f4b6fa7639917a
      name: test-cluster-2
      version: 3.6.0-rc.3
      isHealthy: true
      isLearner: false                     # NEW: Added to support learner promotion logic and troubleshooting
      # REMOVED: dbSize/dbSizeInUse → considered too dynamic; suggested to be exposed via /metrics
      # clientURL and peerURL omitted here for minimal diagnostic view, can be re-added under a feature gate

    - id: 52117ab448fed65d
      name: test-cluster-1
      version: 3.6.0-rc.3
      isHealthy: true
      isLearner: false

    - id: 4f8c9e671cbd2233
      name: test-cluster-0
      version: 3.6.0-rc.3
      isHealthy: true
      isLearner: false

  conditions:
    - type: Available
      status: "True"
      reason: ClusterReady
      message: Cluster is fully available and healthy
      observedGeneration: 5
      lastTransitionTime: "2025-07-02T18:52:06Z"
      # Unchanged: Matches K8s convention for status.conditions

    - type: Progressing
      status: "False"
      reason: ReconcileSuccess
      message: Cluster reconciled to desired state
      observedGeneration: 5
      lastTransitionTime: "2025-07-02T18:52:06Z"
      # Unchanged: Indicates that spec and observed state are aligned

    - type: Degraded
      status: "False"
      reason: ClusterReady
      message: Cluster is healthy
      observedGeneration: 5
      lastTransitionTime: "2025-07-02T18:50:57Z"
      # Unchanged: Used to signal partial failures if any member is unhealthy

  # Note: dbSize and dbSizeInUse fields are REMOVED in this revision.
  # Rationale: These are volatile metrics not suitable for status fields;
  # better exposed via Prometheus-compatible /metrics endpoint for observability.

Please discuss below!

[jberkus]

Question 1:

These three fields seem confusingly similar for the user:

currentReplicas: 3
readyReplicas: 3
memberCount: 3

Do we actually need both ReadyReplicas and MemberCount? Under what circumstances would these two numbers be different?

[jberkus]

Aside from that one question, I'm pretty happy with the list of fields above; it seems like a good starting point.

[hakman]

Maybe also add isLeader?

[ivanvc]

I agree with the points above, @jberkus (consolidating readyReplicas and memberCount SGTM) and @hakman. This is looking good for a first version.

[ahrtr]

Overall looks good for now, with a couple of minor comments:

• leaderId or IsLeader can be dynamic, but the controller may not get notified. It is unlikely in a stable environment. So I'm to add leaderId for now. (Note: we don't have a isLeader field/attribute by default in etcd; it's just a derived attribute in etcdctl by comparing leaderId with each member ID)
• suggest to keep both currentReplicas and memberCount for now. Usually they should match, if not then means the cluster isn't probably healthy or in progress of reconciliation/converge.

[ivanvc]

We discussed this issue at the community meeting. We agreed that it’s better to have more status fields, as the user may not be able to use the etcd API.

[ballista01]

Hey everyone, I’ve got the new status & condition reporting wired through the reconcile loop, and the e2e recorder is streaming those transitions into the CI artifacts. If you want to audit a run:

• open the failing/passing CI job -> Details link
• in Prow, switch to the Artifacts tab
• the etcdcluster-status/ directory now contains a JSONL per test (e.g. TestPodRecovery-…/etcd-recovery-test-status.jsonl), each line a raw watch event

For convenience, here’s a prettified snapshot from TestPodRecovery-multi-node-pod-recovery:

Formatted status timeline

[
  {
    "timestamp": "2025-10-22T01:39:54.844207705Z",
    "event": "ADDED",
    "generation": 1,
    "observedGeneration": 0,
    "resourceVersion": "1513",
    "status": {}
  },
  {
    "timestamp": "2025-10-22T01:39:54.855472522Z",
    "event": "MODIFIED",
    "generation": 1,
    "observedGeneration": 1,
    "resourceVersion": "1520",
    "status": {
      "observedGeneration": 1,
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "CreatingResources",
          "message": "StatefulSet does not exist yet"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "InitializingCluster",
          "message": "StatefulSet created with 0 replicas, requires scaling to 1"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "Reconciling",
          "message": "Reconciliation started"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:03.865251448Z",
    "event": "MODIFIED",
    "generation": 1,
    "observedGeneration": 1,
    "resourceVersion": "1581",
    "status": {
      "observedGeneration": 1,
      "currentReplicas": 1,
      "readyReplicas": 1,
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "CreatingResources",
          "message": "StatefulSet does not exist yet"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "InitializingCluster",
          "message": "Scaling StatefulSet from 0 to 1 replica"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "Reconciling",
          "message": "Reconciliation started"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:03.870760011Z",
    "event": "MODIFIED",
    "generation": 1,
    "observedGeneration": 1,
    "resourceVersion": "1582",
    "status": {
      "observedGeneration": 1,
      "currentReplicas": 1,
      "readyReplicas": 1,
      "memberCount": 1,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "True",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:03Z",
          "reason": "ClusterReady",
          "message": "Cluster is fully available and healthy"
        },
        {
          "type": "Progressing",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:03Z",
          "reason": "ReconcileSuccess",
          "message": "All reconciliation operations complete"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:04.862887354Z",
    "event": "MODIFIED",
    "generation": 1,
    "observedGeneration": 1,
    "resourceVersion": "1586",
    "status": {
      "observedGeneration": 1,
      "currentReplicas": 1,
      "readyReplicas": 1,
      "memberCount": 1,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "True",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:03Z",
          "reason": "ClusterReady",
          "message": "Cluster is fully available and healthy"
        },
        {
          "type": "Progressing",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:04Z",
          "reason": "ReconcileSuccess",
          "message": "All reconciliation operations complete"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:09.864513273Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 1,
    "resourceVersion": "1596",
    "status": {
      "observedGeneration": 1,
      "currentReplicas": 1,
      "readyReplicas": 1,
      "memberCount": 1,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "True",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:03Z",
          "reason": "ClusterReady",
          "message": "Cluster is fully available and healthy"
        },
        {
          "type": "Progressing",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:40:04Z",
          "reason": "ReconcileSuccess",
          "message": "All reconciliation operations complete"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:18.878687972Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1654",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 2,
      "readyReplicas": 2,
      "memberCount": 1,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "ScalingUp",
          "message": "Cluster scaling operation in progress"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "ScalingUp",
          "message": "Adding learner member to scale to 3"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:18.887329945Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1655",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 2,
      "readyReplicas": 2,
      "memberCount": 2,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-1",
          "id": "2231b243372c40fe",
          "version": "3.6.5",
          "isHealthy": true,
          "isLearner": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "PromotingLearner",
          "message": "Learner promotion in progress"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "PromotingLearner",
          "message": "Promoting learner 2463946472906572030 to voting member"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:27.901594809Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1710",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 3,
      "readyReplicas": 3,
      "memberCount": 2,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-1",
          "id": "2231b243372c40fe",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "ScalingUp",
          "message": "Cluster scaling operation in progress"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "ScalingUp",
          "message": "Adding learner member to scale to 3"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:27.910238125Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1711",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 3,
      "readyReplicas": 3,
      "memberCount": 3,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-1",
          "id": "2231b243372c40fe",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-2",
          "id": "3071b84edade751c",
          "version": "3.6.5",
          "isHealthy": true,
          "isLearner": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "PromotingLearner",
          "message": "Learner promotion in progress"
        },
        {
          "type": "Progressing",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:09Z",
          "reason": "PromotingLearner",
          "message": "Promoting learner 3490773835007816988 to voting member"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 1,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:27.9180419Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1712",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 3,
      "readyReplicas": 3,
      "memberCount": 3,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-1",
          "id": "2231b243372c40fe",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-2",
          "id": "3071b84edade751c",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:27Z",
          "reason": "ClusterReady",
          "message": "Cluster is fully available and healthy"
        },
        {
          "type": "Progressing",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:27Z",
          "reason": "ReconcileSuccess",
          "message": "All reconciliation operations complete"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  },
  {
    "timestamp": "2025-10-22T01:40:28.886726729Z",
    "event": "MODIFIED",
    "generation": 2,
    "observedGeneration": 2,
    "resourceVersion": "1716",
    "status": {
      "observedGeneration": 2,
      "currentReplicas": 3,
      "readyReplicas": 3,
      "memberCount": 3,
      "currentVersion": "3.6.5",
      "leaderId": "3ba519e4d7c4460e",
      "members": [
        {
          "name": "etcd-scaleoutfrom1to3-1",
          "id": "2231b243372c40fe",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-2",
          "id": "3071b84edade751c",
          "version": "3.6.5",
          "isHealthy": true
        },
        {
          "name": "etcd-scaleoutfrom1to3-0",
          "id": "3ba519e4d7c4460e",
          "version": "3.6.5",
          "isHealthy": true,
          "isLeader": true
        }
      ],
      "conditions": [
        {
          "type": "Available",
          "status": "True",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:27Z",
          "reason": "ClusterReady",
          "message": "Cluster is fully available and healthy"
        },
        {
          "type": "Progressing",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:40:28Z",
          "reason": "ReconcileSuccess",
          "message": "All reconciliation operations complete"
        },
        {
          "type": "Degraded",
          "status": "False",
          "observedGeneration": 2,
          "lastTransitionTime": "2025-10-22T01:39:54Z",
          "reason": "ClusterHealthy",
          "message": "All members are healthy"
        }
      ]
    }
  }
]

Looking forward to your comments!

[ivanvc]

Closing, as we already merged the PR that implements the status fields.