Model-metrics dashboard charts require Log Analytics on _Default bucket (undocumented; misleading "0 time series" error)

[lokic233]

Summary

The Model-metrics (and performance-metrics) time series charts in the ML Diagnostics dashboard (Cluster Director → Diagnostics, and GKE → AI/ML → Diagnostics) render their data by running a Log Analytics SQL query against the project's _Default log bucket. If Log Analytics is not enabled on that bucket, every model/performance-metric widget fails — even though the metric data is present and correct in Cloud Logging, the MLRun is healthy, and Profiles + System metrics render fine.

This prerequisite is not documented, and the failure mode is a generic, misleading error.

Impact

• Symptom: Model-metrics widgets show "Something went wrong in the backend" / "0 time series", while Profiles and System metrics render normally.
• The metric log entries (log_name=".../ml_diagnostics_metric") are present and correctly shaped in Cloud Logging the whole time — the data is fine; only the chart's query path fails.
• Directly querying reveals the real cause: the widget runs Log Analytics SQL over `PROJECT.global._Default._Default` and the bucket returns "Bucket .../buckets/_Default ... does not have Log Analytics enabled."

Root cause

_Default had analyticsEnabled: false. The dashboard's model-metric charts depend on Log Analytics being enabled on that bucket, but nothing in the managed onboarding (gcloud container clusters update --enable-managed-mldiagnostics) enables it, and the docs don't mention it.

Fix (for users)

Enable Log Analytics on the _Default bucket (non-destructive upgrade — existing logs stay queryable, no added cost, cannot be disabled):

gcloud logging buckets update _Default --location=global --enable-analytics --project=PROJECT

Requires logging.buckets.update (e.g. roles/logging.configWriter).

⚠️ Important gotcha: Log Analytics does not backfill

Enabling Log Analytics only indexes logs ingested after the flag is turned on. Runs that completed before enablement will still show empty charts — their rows aren't in the analytics view. After enabling, launch a new run to see charts populate. This is easy to misdiagnose as "the fix didn't work."

Requests

1. Auto-enable (or gate on) Log Analytics as part of managed ML Diagnostics onboarding, since the model-metric dashboard hard-depends on it.
2. Improve the failure UX — a Log-Analytics-disabled bucket should surface an actionable error (e.g. "Log Analytics is not enabled on the _Default bucket; enable it to view model metrics") instead of the generic "Something went wrong in the backend / 0 time series."
3. Document the prerequisite in the README (Viewing Logs, Metrics and Profiles) and setup guide — the exact gcloud logging buckets update ... --enable-analytics command, the least-privilege role, and the no-backfill behavior.

I'll attach a small README docs PR covering (3).

Environment

google-cloud-mldiagnostics 1.0.3, JAX 0.10.0, GKE workload on TPU (plain-SPMD), managed-mldiagnostics enabled (connection-operator + injection-webhook present). Confirmed: enabling Log Analytics on _Default, then running a fresh MLRun, makes the model-metric charts render natively (loss / learning rate / throughput / tflops, over Time and over Steps).