Skip to content

feat: add replication plugin for external-to-internal data sync#239

Open
kartikganesh wants to merge 2 commits into
outerbase:mainfrom
kartikganesh:feat/replication-plugin
Open

feat: add replication plugin for external-to-internal data sync#239
kartikganesh wants to merge 2 commits into
outerbase:mainfrom
kartikganesh:feat/replication-plugin

Conversation

@kartikganesh
Copy link
Copy Markdown

Summary

Fixes #72. New ReplicationPlugin pulls data from configured external sources (Postgres, etc.) into internal DO SQLite using cursor-based incremental sync.

/claim #72

Changes

  • plugins/replication/index.ts: Full plugin implementation with REST API, cursor-based sync, batch processing, DO alarm scheduling
  • plugins/replication/meta.json: Plugin metadata
  • plugins/replication/README.md: Usage documentation with examples

How it works

  1. Admin configures tables via POST /replication/tables with table name, cursor column, and interval
  2. Plugin stores config in tmp_replication_config and schedules DO alarm
  3. On each sync cycle, queries external source: SELECT * FROM table WHERE cursorCol > lastCursor ORDER BY cursorCol LIMIT 1000
  4. Upserts rows into internal SQLite with INSERT OR REPLACE
  5. Updates cursor position in tmp_replication_state
  6. Reschedules alarm for next interval

API

  • POST /replication/tables — configure tables to replicate
  • GET /replication/status — show sync status per table
  • POST /replication/sync — trigger manual sync
  • DELETE /replication/tables/:name — remove table from replication

Test plan

  • Configure replication for a table, verify data appears in internal SQLite
  • Verify incremental sync only fetches new rows
  • Test manual sync via POST /replication/sync
  • Verify status endpoint shows correct cursor and row count
  • Test removal of table from replication
  • Verify alarm-based scheduling fires at configured intervals

Fixes outerbase#72. New ReplicationPlugin pulls data from external sources
(Postgres, etc.) into internal DO SQLite using cursor-based
incremental sync. Configurable per-table intervals, batch processing
(1000 rows/pull), and DO alarm scheduling. REST API under /replication.

/claim outerbase#72

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13 tests covering syncTable, syncAllTables, onAlarm, and
scheduleNextAlarm with mocked external queries and DO storage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kartikganesh
Copy link
Copy Markdown
Author

/claim

@kartikganesh
Copy link
Copy Markdown
Author

Hey team — this PR implements the replication feature as a proper StarbasePlugin, following the existing plugin architecture (register, beforeQuery, afterQuery hooks + REST routes).

Key design choices aligned with the issue requirements:

Issue requirement Implementation
Pull mechanism (not push) Cursor-based polling: SELECT * FROM table WHERE cursorCol > ? ORDER BY cursorCol LIMIT 1000
User-defined intervals Per-table intervalMs config, alarm scheduled based on shortest interval
User-defined tables Admin selects which tables to replicate via REST API
Column-based cursor tracking User specifies cursor column (e.g. id, created_at) per table
Append-only polling Only fetches rows newer than last cursor position

Architecture notes:

  • Plugin pattern — extends StarbasePlugin, registers routes under /replication/*, uses pathPrefix for auth scoping
  • Incremental sync — never re-fetches already-synced rows; cursor persisted in tmp_replication_state
  • DO alarm integrationonAlarm() method for the DO to delegate, schedules next sync automatically
  • Error isolation — one table failing doesn't block others from syncing
  • Tested — 13 unit tests covering cursor queries, first sync, empty results, upsert logic, state updates, alarm scheduling, error handling

Other open PRs either don't use the plugin system (making them incompatible with StarbaseDB's architecture), skip incremental cursors, or don't handle DO alarm scheduling for continuous sync.

Happy to iterate on any feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant