Refactor snapshot-mismatch cache invalidation out of FileTreeService.ValidateAsync

[woutervanranst]

Summary

FileTreeService.ValidateAsync() currently owns both filetree cache validation and chunk-index cache invalidation on snapshot mismatch. The invalidation appears necessary for correctness, but the ownership looks like a domain-boundary leak: FileTreeService is coordinating another service's cache lifecycle.

Current behavior

Today FileTreeService.ValidateAsync():

• compares latest local vs remote snapshot state
• materializes remote filetree marker files for ExistsInRemote(...)
• deletes chunk-index L2 cache files on snapshot mismatch
• calls ChunkIndexService.InvalidateL1() on snapshot mismatch

That means ArchiveCommandHandler currently must call ValidateAsync() before _chunkIndex.FlushAsync() to avoid merging against stale shard cache state.

Why this is a problem

The current placement makes the dependency surprising and obscures ownership:

• FileTreeService nominally owns filetree traversal, persistence, and filetree cache state
• ChunkIndexService nominally owns chunk-index cache state
• snapshot mismatch is really a repository-wide cache-coordination concern, not purely a filetree concern

This also makes archive tail ordering harder to understand. The need to validate filetrees before flushing the chunk index is driven by hidden cross-cache invalidation behavior rather than an explicit orchestration step.

Desired direction

Refactor the snapshot-mismatch coordination so chunk-index invalidation is no longer owned implicitly by FileTreeService.ValidateAsync().

Likely options:

1. Introduce a repository-level cache validation/coordinator service that compares snapshot state and coordinates both filetree and chunk-index cache refresh behavior.
2. Split the current behavior into narrower responsibilities, for example:
   • filetree-only remote knowledge/materialization in FileTreeService
   • chunk-index invalidation owned by ChunkIndexService
   • feature handler or shared orchestrator coordinates both after snapshot comparison.

Constraints

• Do not lose the current correctness guarantee: on snapshot mismatch, chunk-index caches may be stale and must still be invalidated before shard reads/merges that rely on trusted cache state.
• Keep filetree remote-existence optimization intact.
• Preserve crash safety and cross-machine correctness semantics.

Related context

• ArchiveCommandHandler tail ordering currently depends on this behavior.
• AGENTS.md already documents that chunk-index cache ownership belongs to ChunkIndexService and that snapshot comparisons determine whether local tree/chunk cache state can be trusted.
• docs/decisions/adr-0007-separate-phase-and-detail-logging-in-pipeline-handlers.md was prompted in part by this hidden dependency becoming visible during archive logging cleanup.