Data Farm — Centralized Entity Data Layer & Realtime Sync

[n-lark]

Data Farm — Centralized Entity Data Layer & Realtime Sync

Parent: Epic #6519 — Scalable Frontend Architecture & Data Flow

Problem

Entity data (teams, applications, instances, devices) is fetched ad-hoc inside Vue components — created()/mounted() hooks, route watchers, mixins — and held as local component state. No single source of truth. Consequences:

• After an MCP tool succeeds (e.g. "create an instance" from expert chat), the UI doesn't update — the entity exists server-side but the list has no way to learn about it without a reload, which loses the chat.
• The same entity is fetched repeatedly at different levels (list + detail + status) and copies diverge (team.deviceCount vs deviceCountDeltaSincePageLoad; context-store team vs account-store team).
• Loading is a scatter of independent spinners with no coherent skeleton state.

Solution

Centralized Pinia stores, one per entity (data-farm-<entity>), that own that entity's data. The flow becomes unidirectional — the epic's "Golden Path" (concept #5):

Component  →  data-farm store action  →  API client / service  →  store state
   ▲                                                                    │
   └──────────────────── reactive read (getters) ──────────────────────┘
                                   ▲
        subscriber (MQTT/WS)  ─────┘   pushes realtime CRUD + status into the same state

• Stores own data. Components never call API clients directly; they call store actions and read getters.
• Subscribers update stores. The existing subscribers pattern (team-channel.subscriber.ts → live-status.ts) is generalized from "status only" to "full entity CRUD", so an MCP-triggered create/update/delete flows into the store and every mounted component reflects it.
• Backend publishes lifecycle events. app.comms.team in packages/flowfuse/forge/comms/index.js already emits t/updated, u/<uid>/membership, p/<id>/state, d/<id>/state. The BE half adds lifecycle publishers (created/updated/deleted) and calls them from mutation routes.
• Components go dumb + get skeletons. Components render store state and show skeletons while a store reports loading.

Relationship to Epic #6519

Epic sub-issue	Relationship
#6522 Modular State with Pinia (✅ closed)	Prerequisite, done. Data-farm builds on Pinia.
#6520 Hydrate-Before-Route (open)	Complementary. Data-farm stores are what hydration populates. Skeleton loaders overlap directly with #6520's "Tiered Hydration" — keep skeleton components shared, not duplicated.
#6521 GBAC / router guards (open)	Complementary. Guards "validate permissions against pre-loaded store data" — that pre-loaded data is data-farm.
Epic concepts #3 / #5 / #7	This sub-issue is the implementation of these for real entities.

Net-new here (not in the epic): wiring the subscribers pattern to entity CRUD (not just status), enabled by infra that already exists (subscribers/, services/mqtt.service.ts, transport/mqtt.transport.ts, comms/).

PR shape per entity

Tasks map 1:1 to PRs, drawn so each is independently mergeable and demonstrable — no PR ships a store nothing reads.

PR	Contains	Depends on
Backend	lifecycle publisher call sites + ACL	— (independent)
Store + first consumer	store + unit tests, primary list view migrated with skeleton, create/update/delete upsert/remove — fixes the same-session bug	Teams pilot (conventions)
Subscriber	route CRUD topics → applyRealtime — adds cross-session live sync	Store PR (+ Backend PR to fire)
Remaining vue	migrate rest off local fetching, skeletons, delete dead code	Store PR

Per entity: store first (backend in parallel) → subscriber + remaining-vue.

Entities (child sub-issues)

First wave, each broken into the PRs above:

#	Entity	Notes
1	Teams	Pilot; realtime topics already exist → no backend PR (3 PRs).
2	Applications	Highest value for the originating bug.
3	Hosted Instances	Status already live; entity data not centralized.
4	Remote Instances (Devices)	Polymorphic ownership; worst scattering.

Future waves (same shape): Snapshots · Pipelines · Device Groups · Members & Invitations · Team Library · Blueprints · Broker topics · Tables.