sync engine specs

[priyanshujain]

Goal

• Store history of changes
• Transfer minimal data to client to get most recent data
• Can rollback to previous state

This is how the data flow look like

[priyanshujain]

Actions

• Add Transaction
• Edit Transaction
• Create Collection
• Edit Collection
• Record Payment
• Edit Payment
• Create Invoice
• Edit Invoice

[priyanshujain]

Rough notes

When a user logs in, the client starts by downloading a copy of all events. From that point on, updates to that business in both directions are synced over the WebSocket connection. It allows user to go offline for an arbitrary amount of time and continue performing actions. When you come back online, the client downloads a new list of events from starting from last commit hash/reference, reapplies any offline events on top of this latest state, and then continues syncing updates over a new WebSocket connection. This means that connecting and reconnecting are very simple and all of the complexity with sync engine is in dealing with updates to already connected clients.

It will use protobuf for transport and use single structure for events.

interface Event {
    path: string;
    ...
}

We should also maintain a value for every entity sync state to make sure every client has a copy of it. If any point in time I want to know that how many of clients don't have Invoice#34 then I should be able to answer it on server side.

One interesting insight is that every client does not need all the data. It might depend of the role of the user. For example only managers or admin need to have approval requests data for all users.

[priyanshujain]

Git as event sourcing

Commits represent code changes that are recorded sequentially and become the main source of truth. This means that you can view a history of all changes, time travel by checking to a previous commit, rollback changes, troubleshoot issues by using a binary search etc.

[priyanshujain]

Object Pool Layer

We can use dexie/indexeddb for storing object pool with outbox pattern.

The Outbox pattern allows to atomically update data in a database and record messages that need to be sent in order to guarantee data consistency.

When performing regular database record changes, we can also insert event records in an “ephemeral” outbox table within the same transaction.

BEGIN;
  UPDATE shopping_cart_items SET quantity = 2 WHERE id = 1;
  UPDATE products SET in_stock_count = in_stock_count - 1 WHERE id = 123;
  INSERT INTO outbox_events (event_type, entity_type, entity_id, payload) VALUES (...);
COMMIT;

After the transaction completes, the domain-specific events can be reliably captured by CDC and permanently stored in an event store similarly to a traditional Event Sourcing approach.

We can also use tauri with rust backend on client that uses sqlite as main DB. Either way we should try to support both via an interface so if the user uses the app on pwa or web it will use indexeddb otherwise sqlite on desktop.