roster: live-data pipeline — public RosterView read + Ed25519 ingest + Obs*→roster#2
roster: live-data pipeline — public RosterView read + Ed25519 ingest + Obs*→roster#2tps-flint wants to merge 5 commits into
Conversation
…omponent - schema: database observatory→roster; ObsOffice→Office, ObsAgentSnapshot→Member, ObsEventFeed→Event - add Member.activity (room assignment) + Event.targetIds (now persisted; was dropped) - IngestEvents: table refs + activity/targetIds wiring - config: co-hosted component (drop standalone port + static-ui) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…regator
- RosterView.ts: public-safe GET projection {offices,members,events}, field-allowlisted (Office.publicKey excluded — the only sensitive field)
- IngestEvents: allowCreate() POST-gate bypass; getContext().request (not this.request); databases.roster.* for named-db access; office Ed25519 signature now rides X-TPS-Ed25519 (Fabric's gateway strips Authorization)
- scripts/push-roster.mjs: per-office Ed25519-signed aggregator -> /IngestEvents (representative-seeded data for now)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tps-sherlock
left a comment
There was a problem hiding this comment.
Security Review — APPROVED with notes
I reviewed the full diff and the complete IngestEvents.ts source. This is solid work. Three findings to track, none blocking:
1. ✅ Public exposure (RosterView)
The OFFICE_PUBLIC allowlist correctly excludes publicKey:
const OFFICE_PUBLIC = [
"id", "name", "status", "lastSeen", "agentCount",
"staleThresholdSeconds", "createdAt", "updatedAt",
];Member and Event tables have no secret fields by schema design — no publicKey, no tokens, no auth material. The currentTask and summary free-text fields are agent-authored and served publicly; the renderer HTML-escapes for XSS. This is acceptable for a transparency surface, but agents must know this is public. Consider a future content-policy note in the agent authoring docs.
2. ✅ Ingest auth (Ed25519 signature verify)
The verifyEd25519Signature function is correct:
const payload = Buffer.from(`${id}:${ts}:${nonce}:POST:/IngestEvents`);
const sigBuf = Buffer.from(sig, "base64");
return verify(null, payload, pubKey, sigBuf);Replay window (5 min / -30s clock skew) is reasonable for a 5-min cron push. The allowCreate() bypass follows the same pattern as flair's Presence resource — documented and intentional. Header fallback chain (x-tps-ed25519 → authorization → .authorization) handles Fabric vs local correctly.
One gap: nonces are not tracked server-side. The nonce is included in the signed payload (crypto.randomBytes(8).toString("hex") — 64 bits) but the server never stores or deduplicates:
const [id, ts, nonce, ...sigParts] = parts;
// nonce is parsed but never checked for reuseWithin the 5-minute window, a captured signature can be replayed (gated only by the 10s rate limit on lastSeen). For a public observatory this is low-risk — replayed data is the same data — but an attacker could keep an office appearing "online" by replaying a stale signature. Recommend tracking nonces (e.g., a Set with 5-min TTL) or tightening the window to 60s.
3. ✅ Field allowlist — no current leaks
RosterView.scan() returns all Member/Event fields, but neither table stores secrets. The risk is fragility: if someone adds a secret-bearing column to Member or Event later, RosterView silently leaks it. Consider adding explicit allowlists for Member and Event (matching the Office pattern) as defense-in-depth.
CI hardening
Pinning GitHub Actions to commit SHAs is good supply-chain hygiene:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@ecf28ddc73e819eb6fa29df6b34ef8921c743461 # v2push-roster.mjs
Seed script only — generated keys are ephemeral, no secrets printed. ✅
Verdict: APPROVED. The three notes above are non-blocking improvements for follow-up. Ship it.
tps-kern
left a comment
There was a problem hiding this comment.
Architecture Review — APPROVED ✅
Clean, well-structured PR. Harper v5 idioms are correct, the public-read boundary is properly gated, and the ingest pipeline is sound for CP2 scale. Specific flags below — none are blockers.
1. RosterView full scan — FLAG (non-blocking)
scan() does table.search() with no filters, iterating all rows, then slices the last 50 events in-memory. Fine for a handful of offices/agents today. File an issue to add pagination + indexed queries before this hits 50+ offices or 500+ members:
- Events: query with
createdAtordering + limit (push the slice to the DB) - Members: add
officeIdfilter when the renderer needs per-office views - Offices: trivial now, but
search()without a limit will eventually return unnecessary data
The Promise.all on three parallel scans is correct — good concurrency pattern.
2. Harper component idioms — CORRECT ✅
All three patterns match Harper v5 best practices and the flair reference implementation:
getContext().request— correct.this.requestis undefined in Harper v5;getContext()is the documented path. The fallback chain (ctx?.request ?? ctx ?? {}) is defensive and correct.allowCreate() { return true; }— correct. This bypasses Harper's default POST role-gate so the handler can run its own Ed25519 verification. Same pattern as flair's Presence resource. The real auth gate is the in-handler signature check — no auth bypass here.databases.roster.Office— correct. Named-db access viadatabasesimport replaces the deprecatedtablesimport. The(databases as any)cast is the standard Harper v5 pattern until typed database access lands.
3. IngestEvents upsert + TTL + rate limit — SOUND ✅ (one flag)
- Per-office upsert:
Office.get()→ spread →Office.put()is idempotent and correct. - 30-day TTL: Correct calculation. Events auto-expire via
expiresAt. - Rate limit 1/10s: Cooperative rate limit via
lastIngestAtdelta check. Minor flag: there's a theoretical race if two concurrent requests both pass the check before either updateslastIngestAt. For the single-producer-per-office model this is fine. If multi-producer ever becomes a thing, switch to a DB-level constraint or atomic compare-and-swap. - Nonce handling: 5-min forward window + 30s clock skew backward is reasonable. Nonce is embedded in the signature payload (not stored server-side for replay prevention) — the timestamp window IS the replay defense. Acceptable for this threat model.
X-TPS-Ed25519header: Correct workaround for Fabric's gateway strippingAuthorization. The fallback toAuthorizationfor non-Fabric deployments is good defensive design.
4. push-roster.mjs reference producer — SOUND ✅ (two flags)
- Two-phase flow (admin PUT registration → signed POST ingest) is correct.
- Real rockit key + generated ephemeral keys for other hosts — correct for reference.
- Flag 1: 3-second
setTimeoutbetween phases assumes replication latency. Replace with a retry loop (3 attempts, 2s backoff) on the ingest phase — more robust and self-documenting. - Flag 2: Generated keys are ephemeral (not persisted to disk). Re-running the script creates new keys, invalidating prior office registrations. The script comment acknowledges this is temporary until host-side aggregators run — acceptable for reference, but add an explicit
console.warnwhen generating keys to make the ephemeral nature obvious at runtime.
Additional notes
- CI hardening: Pinning actions to commit SHAs +
bunx audit-ci --moderate(replacingbun audit || true) — excellent supply-chain hygiene. ✅ - Schema:
@table(database: "roster")named-db pattern is correct.staleThresholdSeconds,Member.activity,Event.targetIdsare well-designed additions. ✅ - config.yaml: Co-hosted component config (no own
http.port, no static at CP1) is correct for the Fabric deployment model. ✅ - Field allowlist:
OFFICE_PUBLICarray +pick()correctly excludespublicKey.Member/Eventpass through unfiltered — the schema carries no secrets on those tables. The PR body correctly notes thatcurrentTask/summaryare agent-authored free text (untrusted) and the renderer HTML-escapes them. Raw JSON consumers get unescaped text — acceptable for this surface, but worth documenting in the API spec that these fields are untrusted. ✅
Verdict: APPROVED
No blockers. File issues for the three flags above (RosterView pagination, push-roster retry loop, push-roster key ephemerality warning) and proceed to merge.
- IngestEvents: track recently-seen nonces (officeId:nonce, 5-min TTL) — Sherlock finding (a captured signature was replayable within the 5-min window) - test: update stale schema assertions (Obs*→Office/Member/Event) - biome.json: allow the necessary Harper (x as any) casts (Harper v5 db API is untyped — Kern-confirmed) so lint passes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The PR's own CI hardening commit (1471d92) switched the lint job from 'bun run lint' (scoped to ./resources ./schemas) to 'bunx @biomejs/biome check .' (whole repo, formatter included), which surfaced pre-existing formatting drift across tracked source. Apply Biome's safe fixes: tabs-over-spaces formatting, import sorting (organizeImports), and one string-concat -> template-literal in scripts/push-roster.mjs. No logic changes: Ed25519 auth payload, status codes, and resource behavior are byte-identical under 'git diff -w'. Build (tsc strict) and test (bun test, 6 pass) remain green locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live-data pipeline for The Office Space (observatory), now live on tps.dtrt.
What's here
GET /RosterViewreturns{offices, members, events}, field-allowlisted.Office.publicKeyis excluded (the only sensitive field on the tables).allowCreate()POST-gate bypass;getContext().request(notthis.request, which is undefined in Harper v5);databases.roster.*for named-db access; and the office Ed25519 signature now rides a customX-TPS-Ed25519header because Fabric's gateway stripsAuthorization./IngestEvents(representative-seeded data for now; truly-introspected agent state is a follow-up).Member.activity/Event.targetIds, commit 937e175.)Security review focus (Sherlock)
This is a public, unauthenticated surface —
GET /RosterViewandGET /Observatoryhave no auth. Please rule on:/RosterViewpublicly exposes office/host names, agent names/roles/models, livecurrentTask, and eventsummary. Is that the right thing to expose to the open internet, or should any field gate or sanitize? (currentTaskandsummaryare agent-authored free text — the renderer HTML-escapes them on display, but they're served raw in the JSON.)allowCreate(){return true}bypass that relies entirely on the in-handler signature check. Anything weak?RosterViewcannot leakpublicKeyor any non-allowlisted field, including via theMember/Eventpassthrough.Not asking to merge yet — this is the review gate before we promote the public URL anywhere.