Dipowell/node readiness timing

[diamondpowell]

Summary

Adds node_readiness_time as a separate metric in the open-source CRUD module to match internal repo behavior. The internal repo captures how long K8s nodes take to become Ready independently from when the ARM API completes. The open-source repo was missing this - it only had combined duration.

Azure API says "done" when the control plane finishes, but nodes might not be schedulable yet. Capturing both timestamps separately enables regression analysis:

• command_execution_time > node_readiness_time → ARM layer is the bottleneck
• node_readiness_time > command_execution_time → K8s layer is the bottleneck

Changes

kubernetes_client.py

• Added return_timestamp=False parameter to wait_for_nodes_ready()
• When True, returns (ready_nodes, timestamp) tuple instead of just ready_nodes

aks_client.py

• Added _run_concurrent_arm_and_readiness() helper using asyncio.gather()
• Updated create_node_pool(), scale_node_pool(), and _progressive_scale() to capture concurrent timing

Timing metadata stored via op.add_metadata():

• node_readiness_time: seconds from start until K8s nodes were Ready
• command_execution_time: seconds from start until ARM operation completed

Implementation notes

• Uses asyncio.to_thread() to run ARM polling and K8s readiness checks concurrently
• return_exceptions=True ensures both tasks complete even if one fails, enabling partial diagnostics

[Copilot]

Pull request overview

This PR adds independent node-readiness timing to the Python AKS CRUD flow by extending the Kubernetes wait helper to optionally return a readiness timestamp, then running the ARM poller and the K8s readiness wait concurrently so both timings can be recorded for regression analysis.

Changes:

• Extend KubernetesClient.wait_for_nodes_ready() with return_timestamp to optionally return (ready_nodes, ready_timestamp).
• Add concurrent ARM + K8s readiness execution in AKSClient and record node_readiness_time / command_execution_time metadata.
• Update AKS and Kubernetes client unit tests to cover the new return shape and timing metadata recording.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
modules/python/clients/kubernetes_client.py	Adds return_timestamp option so callers can capture when nodes became Ready.
modules/python/clients/aks_client.py	Runs ARM and readiness concurrently and stores separate timing metadata.
modules/python/tests/clients/test_kubernetes_client.py	Adds a unit test validating the timestamp-returning behavior.
modules/python/tests/clients/test_aks_client.py	Updates tests to expect timestamp return and to assert timing metadata is recorded.

[liyu-ma]

asyncio requires event loop. With this implementation it embed an implicit requirement that caller must not be in an event loop already. For example if clients want to invoke create_node_pool in an async fashion at higher level, the asyncio.run here will fail as an it is called from within an event loop already. Ideally if we want to use async it would be good to propagate the async chain all the way up but this is much bigger refactoring.

How about just spinning up a small ThreadPoolExecutor similar like this:

def _run_concurrent_arm_and_readiness(self, ...):
    with ThreadPoolExecutor(max_workers=2) as ex:
        arm_future = ex.submit(...)
        k8s_future = ex.submit(...)
        wait([arm_future, k8s_future])

[liyu-ma]

I think _instrument_nodepool_provisioning is a better name for the method

[liyu-ma]

I see only begin_create_or_update is passed in so no need to have a separate poller param

[liyu-ma]

I think start_time can be captured within this function, no need to be passed from outside