Cadence Cockpit • Closed-Loop Fleet Observability for Field Robots

Forenly AI Platform's Entry for the Splunk Agentic Ops Hackathon (2026)

A Physical AI observability platform that maintains a real-time digital twin of a deployed field-robot fleet — closing the loop between on-site telemetry and autonomous corrective action through Splunk Cloud HEC, the Splunk MCP Server, and GCP Gemini multi-agent orchestration.

---

Community

Building this in the open — join the team chat on Discord: https://discord.gg/7JXRwTy2EE

---

Where this fits in the Forenly lifecycle

Forenly builds one continuous life-cycle for autonomous field robots (robotic mowing / field service), split across three hackathon entries on the same customer and the same fleet:

Stage	Project	What it does
1. Acquire	Lawn Advisor (Forenly/gcp)	Generates leads, recommends the right robot for a site, builds a deployment plan, sends the proposal → customer onboards and robots get installed on site.
2. Operate	FleetMind / Gemini XPRIZE (Forenly/gemini-xprize)	Runs the deployed fleet day-to-day: intake → schedule → dispatch → verify → invoice.
3. Observe & Sustain	Cadence Cockpit (this repo)	Watches the running fleet's telemetry in Splunk, detects faults and wear before they cause downtime, and closes the loop with autonomous corrective action — feeding repairs and schedule changes back to the Operate layer.

This repo is stage 3: once robots are mowing real sites, how do you keep the whole fleet healthy, on-SLA, and running with as little human intervention as possible?

---

The Core Idea: from alerts to a closed loop

Most observability stacks stop at the alert — a human still has to read the dashboard, diagnose, and act. Cadence Cockpit treats the deployed fleet as a live digital twin and closes the loop: every robot's telemetry flows into Splunk, a Gemini multi-agent orchestrator triages anomalies through the Splunk MCP Server (a guided query chain to root cause), and instead of just paging someone it executes the corrective action and writes the outcome back to the twin.

                  ┌─────────────────────────────────────────┐
                  │     Splunk Cloud HEC — Fleet Telemetry   │
                  │  (battery · motors · GPS/boundary ·      │
                  │   coverage · dock cycles · faults)       │
                  └────────────────────┬────────────────────┘
                                       │
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │   Gemini Orchestrator  +  Splunk MCP     │
                  │   (root-cause query chain → triage)      │
                  └────────────────────┬────────────────────┘
                                       │
                       [ Multi-Dimensional Fleet Triage ]
                                       │
         ┌─────────────────────────────┼─────────────────────────────┐
         ▼                             ▼                             ▼
┌──────────────────┐          ┌──────────────────┐          ┌──────────────────┐
│   Operational    │          │   Predictive     │          │   SLA & Coverage │
│   Health         │          │   Maintenance    │          │                  │
├──────────────────┤          ├──────────────────┤          ├──────────────────┤
│ • Stuck / tipped │          │ • Battery decay  │          │ • Coverage % vs  │
│ • Slope torque   │          │ • Blade wear     │          │   contracted SLA │
│   loss           │          │ • Motor temp     │          │ • Missed windows │
│ • Boundary/GPS   │          │   trend          │          │ • Fleet balance  │
│   drift          │          │                  │          │   across sites   │
└──────────────────┘          └──────────────────┘          └──────────────────┘

---

The 3 Simulation Channels

Cadence Cockpit programmatically demonstrates autonomous triage and resolution of fleet anomalies across three channels. Each one follows the same shape: Anomaly → Triage → Closed-loop Execution.

1. 🛠️ Operational Health Loop (real-time faults)

• The Anomaly: Robot MOW-07 reports sustained torque loss on a slope plus a boundary-GPS drift — telemetry into Splunk shows it has stopped covering Zone B.
• The Triage: The Gemini agent runs a Splunk MCP query chain (recent faults, last-good run, neighboring robots) and classifies it as a localized, recoverable fault — not a hardware failure.
• The Execution: The system re-issues a safe return-to-dock + re-route command, reassigns Zone B to an idle robot on the same site, and posts a status update — no human paged unless the retry fails.

2. 🔋 Predictive Maintenance Loop (wear before failure)

• The Anomaly: Battery charge-cycle data and blade-runtime counters trend toward end-of-life across several robots; one motor's temperature curve is climbing run-over-run.
• The Triage: The Gemini agent projects a failure window from the Splunk time-series and flags which robots will breach threshold first.
• The Execution: Instead of waiting for a breakdown, the agent schedules a preventive part swap during a non-mow window, raises the maintenance ticket, and notes the part needed — turning unplanned downtime into planned service.

3. 📊 SLA & Coverage Loop (service-level assurance)

• The Anomaly: A site's measured coverage drops below its contracted SLA after two missed mow windows, while another site's fleet sits underutilized.
• The Triage: The Gemini agent flags an SLA-breach risk and identifies the imbalance across sites.
• The Execution: It rebalances scheduling/dispatch, feeds the change back to the Operate layer (FleetMind), and surfaces the SLA status so billing/credits stay accurate — keeping the customer commitment intact.

---

Technology Stack

The platform demonstrates a cohesive cyber-physical loop built from:

1. Splunk Cloud (HEC): Ingests big-data telemetry from the deployed fleet — robot logs, battery/motor metrics, GPS/boundary events, coverage, and dock cycles.
2. Splunk MCP Server: The guided query layer the agent uses to walk from symptom to root cause (the hackathon's MCP bonus track).
3. GCP Gemini Orchestrator: The "core mind" that triages fleet anomalies and decides the corrective action.
4. Real-time Digital Twin: A live model of every robot and site, updated from telemetry and used to validate actions before they execute.
5. Slack (Socket Mode): Interactive escalation/approval interface for field-ops when an action needs a human in the loop.
6. FleetMind / Gemini XPRIZE node: The Operate layer this stage feeds corrective actions and schedule changes back into.
7. Lawn Advisor node: The Acquire layer that originates each deployment in the first place.

---

🚀 Setup & Run Instructions (Local Sandbox)

The sandbox serves the live dashboard on port 8050 and acts as the bridge to Splunk Cloud.

1. Prerequisites

Python 3.8+ on your host. No external pip libraries are strictly required for the lightweight server — it relies on core modules (http.server, urllib, json) for execution reliability.

2. Configure Environment Variables

Create a .env file in the project root:

PORT=8050
SPLUNK_HEC_URL=https://<your-stack>.splunkcloud.com:8088/services/collector
SPLUNK_HEC_TOKEN=<your-splunk-hec-token>

The HEC URL and token come from your Splunk Cloud stack (Settings → Data Inputs → HTTP Event Collector). No self-hosted Splunk instance is required — the dashboard talks to Splunk Cloud directly over HTTPS.

3. Start the Server

python3 server.py

The server will initialize on port 8050 and print: Forenly Splunk-Agentic-Ops Dashboard serving on port 8050

4. Open the Cockpit

Open http://localhost:8050 in your browser to interact with the Cockpit. The dashboard connects to your Splunk Cloud stack over HTTPS using the HEC credentials configured above — no SSH tunnel or self-hosted instance involved.

---

⚖️ License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

---

👥 Team

Forenly AI Platform · github.com/Forenly