Design proposal: Persistent Replica Runtime

design/2025-10-persistent-replica-runtime.md

@@ -0,0 +1,265 @@
+# Deterministic Projection System
+
+TODO: Safing policy. Intent revision issuers should always run against the
+entire input revisions, not deltas. The safing policy determines when the
+reducers arm. This stops intent issuers running during catch up, and only after
+catch up. Hm i'm not sure this is correct. The members patching policies style
+revision issuer should be what an intent policy looks like, and the output
+replica should really be the replica that is safed. It would seem strange to
+convert an intent replica into soemthing like this thoug. No it is fine. Intent
+replicas always produce recallable effects and model a desired state. Which
+outcome replicas map to the real world. Intent replicas do not produce commands
+for outcome replicas to implement, but instead an overall desired state for them
+to implement. In this sense, the safing policy is not for the intent replica,
+it's for the outcome replica. So we really have an Obligation replica as well as
+an intent replica. Where Obligations are one shot and Intents produce a steady
+state.
+
+TODO: When policy. Intent revision issuers have an evaluation policy
+
+TODO: Intent replica throttle indicator to help the scheduler choose which
+replicas to sacrifice in a CPU exhaustion event.
+
+TODO: Explicitly mention that revisions and deltas can safely be held in memory
+on different workers. since they are immutable.
+
+TODO: Focus on explaining the Replica as the core concept, and revise
+terminology to say snapshot instead of revision and just say "delta stream"
+instead of whatever we use for that. Just call it a state snapshot.
+
+TODO: How to handle replica version changes. Any change to the schema is going
+to cause problems for dependencies. Code changes matter less but obviously
+migration does need to happen. Code changes can effect the contract version
+though if e.g. scope changes massively so it can't just be a schema version.
+
+TODO: Rename replica -> projection + projection node
+
+The persistent replica runtime is a deterministic incremental computation
+environment.
+
+Background: https://github.com/the-draupnir-project/Draupnir/issues/979
+
+The matrix-protection-suite has evolved to around a "revision issuers" concept
+to provide deterministic, incremental computation and data-flow in order to
+manage complex interactions and resource intensive operations.
+
+![](./images/2025-10-draupnir-data-flow.png)
+
+This abstraction has not been generalized and is ad-hoc. And the
+matrix-protection-suite had specific and constrained design goals in its early
+development in order to build a new core for the draupnir project:
+
+- Revision issuers are entirely in-memory structures. There are no asynchronous
+  methods to provide data for persistent storage.
+- Attempts to persist specific revision issuers has required for backing stores
+  to be written that are in reality a downstream consumer of the deltas produced
+  by revisions outside of startup time.
+- The backing stores are not generalized either and have virtually no support
+  for rollbacks in failed updates etc.
+- Transactions between revision issuers are not formalized and it is possible
+  for revision issuers to become unsynchronised, if a delta fails to process or
+  is dropped, that effect is permanent until the system is restarted.
+- Further the lack of "transaction" complicates persistence of revisions issuers
+  that are not sources of non-deterministic input e.g. the PolicyRoomRevisions
+  were persisted, we'd want to be confident that all deltas from the source
+  revision (room state) were applied in order.
+
+Most importantly, the matrix-protection-suite was designed to be used entirely
+within one JS process. Using workers was never a consideration. This holds the
+Draupnir project back from being able to use more centralized service
+architectures. Which would be required to provide a popular service.
+
+## Proposal
+
+We propose that all computation in Draupnir to be expressed through _replica
+actors_ using a new framework called the persistent replica runtime. This
+transition would happen incrementally, rather than as a focussed effort like the
+transition to the matrix-protection-suite and Draupnir 2.0.
+
+This would give us the following advantages:
+
+- Restart can be immediate and instantaneous.
+- Workload can be balanced across worker threads using actor migration.
+- All core computation is deterministic and reproducible, easing development and
+  testing.
+- All data-flow interactions are explicit and auditable.
+- Effects and effect handlers are first-class, providing not only pre-viewable
+  and auditable consequences for protections, but for all side-effects.
+- The system is fudamentally fault tolerant: all computation is incremental,
+  transactional, and deterministic. And can be restarted at the last delta.
+
+_Revision issuers_ become _persistent replicas_. The live in-memory instance of
+a _revision issuer_ becomes a _replica actor_. A meta-object is introduced to
+describe the schema of the revisions and the logic of the actor called a
+_replica description_.
+
+Persistence is implemented by a storage backend that takes the _replica
+description_ and creates tables or documents from them.
+
+A scheduler is introduced that schedules _replica actors_ across worker threads
+in Node.JS.
+
+## Concepts
+
+### Projection node
+
+Projection nodes are immutable snapshots that accurately reflect a system state
+at the point the node was created.
+
+Projection nodes can reduce input _deltas_ into an _output_ delta, which can
+then be used to produce another projection node that reflects a new system
+state.
+
+### Projection delta
+
+Deltas represent the outcome of a computation related to a projection node that
+provide a reproducible transformation of the projection state.
+
+Delta's have their own identity, a delta ID and backlink to a previous delta.
+
+#### Source delta
+
+Source deltas are special because they encapsulate the source of non-determinism
+into the deterministic system. All computation in the system is caused by source
+deltas.
+
+### Provenance data
+
+Each delta and projection node needs to include information about upstream
+inputs it depends on.
+
+Each delta refers to the single source projection node that is the direct cause
+for the current data flow.
+
+Each projection node refers to the current transient source projection nodes
+that have been used to derive the current state.
+
+### Projection node reducer
+
+A projection node reducer is a deterministic function that takes a projection
+node and an input delta and produces a new output delta specific to the
+projection node.
+
+### Projection definition
+
+A projection definition defines how to build projections from partition data,
+projection node reducers, and what the inputs to the projection are.
+
+### Rebuild delta
+
+A rebuild delta is a specific kind of delta that is the direct result of a
+change in implementation of a projection, ie a change to the projection
+definition. It may be important to issue a rebuild delta when bugs that caused
+projections to produce faulty data are fixed.
+
+The change in projection definition is used as the source delta within a rebuild
+delta.
+
+### Partitioning
+
+When projection definitions are combined with partition data when instantiated.
+
+replicas are partitioned by a key. This key is usually something like a Matrix
+room ID, or the user identifier of a Draupnir instance.
+
+### Intent projection (effect)
+
+A intent projection is a special kind of projection that produces an intention
+to see some outcome enacted. These intents are then carried out by outcome
+projections (effect handlers).
+
+Enabled protections should be intent projections.
+
+### Outcome projection (effect handler)
+
+An outcome replica is a special kind of projection that records whether outcomes
+have been issued for intents. The semantic meaning of the outcome is not
+relevant, e.g. failure to ban a user is still an outcome that needs to be
+recorded and is the outcome of the intent.
+
+## Semantics
+
+### Actor migration
+
+Actors need to be transportable across node.js worker threads in order to
+balance CPU.
+
+### Preemption
+
+replica actor client code will be interruptive in order to gracefully handle
+shutdown and migration of actors.
+
+This will be achieved by abusing I/O operations to signal interrupts. CPU bound
+actor code will still have to cooperatively check for an interrupt signal.
+
+### Garbage collection
+
+Replica dependents will formally be tracked. Transitive dependencies will also
+need to be considered for rollbacks to be successful. replicas will only be able
+to discard an old revision when all transitive dependents have successfully
+migrated to a superseding revision.
+
+### Scheduling
+
+The scheduler is responsible for starting replica actors and migrating them when
+CPU becomes exhausted in worker threads. The scheduler gives each actor a
+controller that signals interrupts. The store is expected to also receive the
+controller so that it can preempt on I/O.
+
+The scheduler will have to be communicated `os.loadavg()` from each worker
+thread and use this to trigger actor migration. Realistically we can probably
+only move actors from high utilisation to low utilisation and have some strategy
+that tries to identify and isolate the cpu intensive actors for reporting.
+
+### Crashing
+
+What happens when an actor crashes?
+
+#### Poisoned delta
+
+If the revision reducer crashes when processing a delta, and continues to crash
+when re-attempted, the delta will be skipped with an explicit acknowledgement.
+This signifies a bug in the implementation of the revision reducer and is really
+a critical issue. It's important not to bring down the whole system, but quite
+honestly the existence of a reducer bug in critical areas can be as bad as that.
+Protections obviously can have faulty reducers and don't effect other
+functionality.
+
+#### Other issues
+
+The actor should be restarted with exponential backoff.
+
+### Messaging
+
+How do deltas get communicated to dependencies? especially if the in-memory
+actors can crash and be unreliable for message delivery.
+
+The deltas get written to the persistent store and a message is sent to the
+other threads about the write so that the delta can be read back from the store
+and propagated to dependencies.
+
+## Non-deterministic inputs
+
+The PRR needs non-deterministic inputs in order to do anything useful, such as
+matrix events, commands, and time scheduling.
+
+These are provided by _source replicas_.
+
+### Source replica
+
+A source replica is a system or client provided replica with no dependencies
+that produces deltas. Source replicas provide no data in their records, only the
+deltas are significant.
+
+### Time source replica
+
+Some protections in Draupnir have time-scheduled background tasks, such as
+DraupnirNews.
+
+These should be implemented by having a revision reducer that accepts a tick
+from a time service.
+
+The time source replica is a system provided source replica that persists
+records about when to send ticks to other replicas. Each replica that requires
+time scheduling is given a adaptor replica to bridge them to the time source. So
+that they are not informed about every single tick.