Skip to content

Enhancement: Query result pagination with cached result sets #29

Description

@Ian-stetsenko

Idea

When a user runs a query like lore context src/ --limit 10, the CLI scans git history, parses atoms, filters superseded entries, and returns 10 results. If the user then wants the next 10, they have to re-run the entire pipeline from scratch.

Proposed Behavior

lore context src/ --limit 10            # page 1 — full pipeline runs, result cached
lore context src/ --limit 10 --page 2   # page 2 — served from cache, no git scan

How It Could Work

  1. After the full pipeline (git scan → parse → supersession filter), cache the complete result set to disk (e.g., .lore/cache/<query-hash>.json)
  2. Include a cache key based on: command + target + options + HEAD commit hash
  3. Subsequent calls with --page N check for a valid cache entry and serve results.slice((N-1)*limit, N*limit)
  4. Cache is invalidated when HEAD moves (new commits)

Why

  • Large repos with thousands of Lore atoms — full scans are expensive
  • Common workflow: browse through results page by page
  • Enables future interactive/TUI modes where pagination is expected

Open Questions

  • Cache storage format (JSON? MessagePack?)
  • TTL vs HEAD-based invalidation
  • Should --offset be supported as an alternative to --page?
  • Memory-mapped vs file-based for very large result sets

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions