machinelearning_run() 400 INVALID_ARGUMENT: GKE createTime passed verbatim to proto Timestamp (not RFC 3339)

[lokic233]

Summary

machinelearning_run() fails with HTTP 400 INVALID_ARGUMENT at the control-plane create POST when the GKE workload createTime is not strict RFC 3339. The SDK passes workload_details["creation-timestamp"] verbatim into the google.protobuf.Timestamp field workload_details.gke.create_time.

Environment

• google-cloud-mldiagnostics 1.0.2 / 1.0.3
• GKE TPU (tpu7x, GKE 1.35.3-gke.1522000), single-controller Pathways (JAX 0.10.0)
• --enable-managed-mldiagnostics on; injection-webhook injecting GKE_DIAGON_*

Error (verbatim)

POST .../machineLearningRuns?machine_learning_run_id=<id>
400 INVALID_ARGUMENT
fieldViolations[0]:
  field: machine_learning_run.workload_details.gke.create_time
  description: "Invalid value at '...gke.create_time' (google.protobuf.Timestamp),
    Field 'createTime', Illegal timestamp format; timestamps must end with 'Z'
    or have a valid timezone offset."

Root cause

clients/control_plane_client.py (around L259-261):

creation_timestamp = workload_details.get("creation-timestamp")
if creation_timestamp:
  gke_workload_details["createTime"] = creation_timestamp   # verbatim → proto Timestamp

No RFC-3339 normalization. The source creation-timestamp (from GKE_DIAGON_METADATA) isn't guaranteed to be RFC 3339. Also host_utils.get_identifier itself formats timestamps as "%Y%m%d-%H%M%S" (no Z), so this is latent even for intended inputs.

Fix

PR #1 adds a _normalize_rfc3339() helper applied at the createTime assignment (best-effort; unparseable values pass through). Verified: the 400 cleared and the MLRun was created.

— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).

[lokic233]

Proposed fix: #1 (adds _normalize_rfc3339(), verified clears this 400). — via Navi on behalf of @lokic233