Skip to content

Persistent indexer storage (PostgreSQL backend) #31

@LiranCohen

Description

@LiranCohen

Summary

Replace the in-memory `IndexerStore` with a persistent storage backend (PostgreSQL) for production deployments.

Context

The current `IndexerStore` (`src/indexer/store.ts`) uses in-memory `Map`/`Set` collections with FIFO eviction (added in the hardening PR #22). This is fine for development but loses all indexed data on restart. A production indexer needs persistent storage.

Scope

  • Create `IndexerStorePostgres` implementing the same public API as `IndexerStore`
  • Use the existing PostgreSQL infrastructure from `docker-compose.test.yaml`
  • Add database migrations for the indexer schema (repos, stars, follows, cursors)
  • Make the store backend configurable via environment variable (`DWN_GIT_INDEXER_STORE=postgres`)
  • Keep in-memory store as default for development

Design considerations

  • The store interface is already well-defined — `putRepo`, `putStar`, `putFollow`, `getRepo`, `search`, etc.
  • Full-text search (`search()` method) maps naturally to PostgreSQL `tsvector`/`tsquery`
  • Trending calculation can use SQL window functions
  • Connection pooling for the crawl loop + API server

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions