Bug: Unbounded growth of adhoc.json session metadata — no pruning of stale entries

[edspencer]

Summary

The adhoc.json session metadata file (stored at .herdctl/session-metadata/adhoc.json) grows unboundedly because there is no pruning, TTL, or cleanup mechanism for stale metadata entries. On production (devbox.valfenda.net), there are 110 adhoc session metadata entries that are never cleaned up.

This is separate from #164 (attribution race condition for new sessions) and distinct from #167 (cross-agent metadata contamination). Issue #167 addresses metadata written to the wrong agent's file; this issue addresses the fact that the adhoc.json file itself grows indefinitely with entries for sessions that may no longer exist on disk.

Root Cause

How adhoc metadata entries are created

When SessionDiscoveryService.getAllSessions() scans ~/.claude/projects/, directories that don't match any known fleet agent use metadataKey = "adhoc" (line 552 of session-discovery.ts). Every session in these unattributed directories gets an autoName and preview extracted from its JSONL file and cached into .herdctl/session-metadata/adhoc.json via batchSetAutoNames("adhoc", ...) and batchSetPreviews("adhoc", ...).

The critical amplification path: getAllSessionGroups() in web-chat-manager.ts calls getAllSessions() without a limit parameter, so Phase 2's top-N filtering is skipped and every single non-sidechain session in every non-temp directory is enriched and has metadata written.

Why entries are never removed

The SessionMetadataStore class has no cleanup, pruning, TTL, or expiry logic whatsoever. The only "delete" operation is removeCustomName(), which removes a user-set custom name but doesn't address the autoName/preview cache entries. There is:

• No TTL or expiry check on metadata entries
• No periodic pruning job
• No max-entries limit
• No reconciliation against existing JSONL files
• No staleness detection

Growth pattern

Every unique Claude Code CLI session file discovered in any unattributed directory gets an entry added to adhoc.json. These entries persist even after:

• The underlying JSONL session file is deleted by Claude Code
• The working directory no longer exists
• The session hasn't been accessed in months

On a development machine with active Claude Code usage, new sessions are created frequently, and old ones are never cleaned up, leading to monotonic growth.

Key files

File	Line	Role
packages/core/src/state/session-metadata.ts	Entire file	SessionMetadataStore — no cleanup logic exists
packages/core/src/state/session-discovery.ts	552	metadataKey = agentName ?? "adhoc" — adhoc key assignment
packages/core/src/state/session-discovery.ts	660-666	Batch writes metadata for all enriched sessions
packages/web/src/server/chat/web-chat-manager.ts	211	getAllSessions(agentList) called without limit — enriches ALL sessions

Impact

Severity: Medium — Not a crash or data loss bug, but causes degradation over time:

1. Growing file size: adhoc.json grows with every newly discovered unattributed session. At 110 entries with autoName + autoNameMtime + preview + previewMtime per entry, the file is still manageable (~50-100KB), but on machines with heavy CLI usage it will grow to thousands of entries over months.

2. Increasing I/O per request: Every call to getAllSessions() (triggered by /api/chat/all and /api/chat/recent) reads the entire adhoc.json into memory, parses it, and potentially writes it back with new entries. As the file grows, this read/parse/write cycle gets slower.

3. Memory overhead: The in-memory cache in SessionMetadataStore holds the full parsed adhoc.json object, which grows proportionally.

4. Wasted JSONL parsing: The autoName/preview cache entries reference sessions whose JSONL files may no longer exist. When resolveAutoName() checks the cache and finds a valid entry, it skips the JSONL parse — but the entry itself is dead weight that was created by parsing a file that no longer matters.

Proposed Fix

Add a prune(agentName, validSessionIds: Set<string>) method to SessionMetadataStore that removes entries whose sessionId is not in the provided set. Call it at the end of getAllSessions() for each directory group, passing the set of sessionIds that were actually found during the scan.

// In SessionMetadataStore
async prune(agentName: string, validSessionIds: Set<string>): Promise<number> {
  const metadata = await this.loadMetadata(agentName);
  if (!metadata) return 0;

  let pruned = 0;
  for (const sessionId of Object.keys(metadata.sessions)) {
    if (!validSessionIds.has(sessionId)) {
      delete metadata.sessions[sessionId];
      pruned++;
    }
  }

  if (pruned > 0) {
    await this.saveMetadata(agentName, metadata);
  }
  return pruned;
}

Additionally, consider:

• A configurable max-entries limit (e.g., keep most recent 500 entries by mtime)
• Only running the prune on every Nth call or at startup to avoid prune-on-every-request overhead

Relationship to other issues

• Bug: Attribution race condition causes new chat sessions to be invisible in sidebar #164 (attribution race condition): Separate issue. Bug: Attribution race condition causes new chat sessions to be invisible in sidebar #164 is about new sessions not appearing; this is about old sessions never being purged.
• Bug: getAllSessions() writes cross-agent metadata into agent-specific metadata files #167 (cross-agent metadata contamination): Related but distinct. Bug: getAllSessions() writes cross-agent metadata into agent-specific metadata files #167 addresses metadata being written to the wrong agent's file. Even after Bug: getAllSessions() writes cross-agent metadata into agent-specific metadata files #167 is fixed, adhoc.json would still grow unboundedly for genuinely unattributed sessions.