[FEATURE] Expose Foreman CRD status (Workload/AgenticTask/FleetNode) as Prometheus metrics via kube-state-metrics Custom Resource State

[Defilan]

Feature Description

Ship a kube-state-metrics Custom Resource State (CRS) config that maps the Foreman CRD status fields (Workload, AgenticTask, FleetNode) to Prometheus gauges, so Foreman job and fleet state is scrapable and alertable in the Prometheus + Grafana stack operators already run. This is pure configuration against the kube-state-metrics already deployed in most clusters: no new operator code.

Problem Statement

As an operator, I want Foreman job and fleet state in Prometheus so I can build Grafana panels and alerts without polling kubectl.

Today the only ways to see Foreman state are kubectl get and the CRDs' additionalPrinterColumns. None of it reaches the metrics stack, so you cannot, for example, alert on a Workload stuck in Pending, a FleetNode whose heartbeat has gone stale, or a spike in AgenticTask Verdict=NO-GO/INCOMPLETE. The operator already emits some inference metrics (InferenceService phase, model download duration), but the Foreman CRDs' status is not surfaced at all.

The two axes that matter most and are unique to Foreman: AgenticTask decouples status.phase (Pending/Scheduled/Running/Verifying/Succeeded/Failed) from status.verdict (GO/NO-GO/INCOMPLETE/GATE-PASS/GATE-FAIL). A task can be Phase=Succeeded yet Verdict=INCOMPLETE: finished without actually working. That combination is exactly what an SRE wants to alert on and cannot see today.

Proposed Solution

Add a CustomResourceStateMetrics config, gated behind a chart values toggle, that maps at least:

• foreman_agentictask_info{namespace,name,workload,agent,phase,verdict} (gauge = 1)
• foreman_agentictask_phase / foreman_agentictask_verdict as enum-style gauges (one series per state, value 1 on the active state) for easy alerting
• foreman_workload_tasks_total and foreman_workload_tasks_failed
• foreman_fleetnode_info{node,accelerator,ram} (gauge = 1)
• foreman_fleetnode_heartbeat_timestamp_seconds (gauge) and foreman_fleetnode_phase

Example CRS mapping (illustrative):

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: foreman.llmkube.dev
        version: v1alpha1
        kind: AgenticTask
      metricNamePrefix: foreman_agentictask
      metrics:
        - name: info
          help: "AgenticTask phase and verdict"
          each:
            type: Info
            info:
              labelsFromPath:
                phase: [status, phase]
                verdict: [status, verdict]
                workload: [metadata, labels, "foreman.llmkube.dev/workload"]

Ship it as a chart template (a standalone CRS ConfigMap wired into the kube-state-metrics --custom-resource-state-config-file, or the kube-prometheus-stack kube-state-metrics.customResourceState values path), plus a starter Grafana dashboard JSON and a couple of example PrometheusRule alerts (stuck-Pending Workload, stale FleetNode heartbeat).

Alternatives Considered

• Bespoke exporter in the operator: more code to maintain, duplicates what kube-state-metrics already does well.
• Grafana Infinity/JSON datasource hitting the K8s API: read-only, brittle, no alerting, scrape-interval mismatch.
• The forthcoming LLMKube command-center dashboard: a complementary, separate concern (a resource-aware UI). CRS is the lowest-effort path that lands Foreman state in the metrics stack users already run, and it is useful regardless of the dashboard.

Additional Context

• Related issues: [BUG] foreman audit ConfigMaps accumulate forever — no ownerReferences, no retention #990 (audit ConfigMap retention, adjacent Foreman-observability gap)
• Similar features in other projects: kube-state-metrics Custom Resource State Metrics, OpenCost's KSM-based approach
• Workarounds currently used: kubectl get workload/agentictask/fleetnode and the printcolumns

Priority

• Medium - Nice to have (a same-day observability win on existing infra)

Willingness to Contribute

• Yes, I can submit a PR