Collaboration: fleet-llm-d as operations layer (tenant governance, cost, compliance)

[jkershawrh]

Summary

I've built fleet-llm-d, an open-source (Apache 2.0) operations layer for fleet-level inference orchestration. After studying ModelPlane's architecture, I believe our projects are complementary — ModelPlane handles infrastructure (provisioning, scheduling, caching, routing), while fleet-llm-d handles operations (tenant governance, cost/tokenomics, compliance, SLO-aware autoscaling).

We've already built the integration layer and proven it works.

What fleet-llm-d adds to ModelPlane

Capability	What It Does
Tenant Governance	Per-tenant quotas, rate limits, GPU budgets, cost caps via TenantProfile CRD
Cost Model	GPU pricing (6 types × 3 tiers), cost-per-token, chargeback reports, budget alerts
Compliance/Audit	ARE Immutable Ledger — hash-chained, tamper-evident decision records (EU AI Act, SOC 2, NIST AI RMF)
SLO-Aware Autoscaling	Scales based on TTFT/throughput targets, not just replica count
Lifecycle Management	SLO-gated canary rollouts with automatic rollback

Three-layer architecture

fleet-llm-d (operations)     ← tenant governance, cost, compliance, SLO scaling
     ↕
ModelPlane (infrastructure)   ← provisioning, scheduling, caching, routing
     ↕
llm-d (within-cluster)       ← EPP, KV cache, P/D disagg, flow control

Integration already built

1,942 lines of Go integration code, 50 architecture proof tests passing:

• pkg/modelplane/watcher.go — polls ModelPlane API for InferenceCluster/ModelDeployment/ModelEndpoint changes
• pkg/modelplane/adapter.go — converts ModelPlane types to fleet-llm-d internal types
• pkg/modelplane/policy.go — injects placement annotations, replica counts, canary weights into ModelDeployments
• pkg/modelplane/compliance.go — records all ModelPlane state changes to ARE Immutable Ledger
• pkg/cost/modelplane.go — computes deployment cost from InferenceClass GPU pricing

Tested against a mock ModelPlane API on OpenShift — fleet-controller successfully consumed 3 InferenceClusters, 2 ModelDeployments, and computed cost ($20.60/hr for a 4-replica granite deployment).

Ask

1. Package access — xpkg.crossplane.io/modelplaneai/modelplane:v0.1.0 is private on GHCR. We'd love access for live integration testing.
2. Joint testing — deploy ModelPlane + fleet-llm-d together to validate the full stack.
3. CRD alignment — ensure our policy injection annotations don't conflict with ModelPlane's scheduling.

Project

• GitHub: https://github.com/jkershawrh/fleet-llm-d
• Stats: 50 architecture proofs, 500+ tests, 22 Go packages, deployed on OpenShift with real Granite inference
• License: Apache 2.0

Happy to jump on a call to discuss. This could be a strong joint story for the CNCF ecosystem.