diff --git a/design/2025-10-persistent-replica-runtime.md b/design/2025-10-persistent-replica-runtime.md new file mode 100644 index 0000000..cd3ce7b --- /dev/null +++ b/design/2025-10-persistent-replica-runtime.md @@ -0,0 +1,265 @@ +# Deterministic Projection System + +TODO: Safing policy. Intent revision issuers should always run against the +entire input revisions, not deltas. The safing policy determines when the +reducers arm. This stops intent issuers running during catch up, and only after +catch up. Hm i'm not sure this is correct. The members patching policies style +revision issuer should be what an intent policy looks like, and the output +replica should really be the replica that is safed. It would seem strange to +convert an intent replica into soemthing like this thoug. No it is fine. Intent +replicas always produce recallable effects and model a desired state. Which +outcome replicas map to the real world. Intent replicas do not produce commands +for outcome replicas to implement, but instead an overall desired state for them +to implement. In this sense, the safing policy is not for the intent replica, +it's for the outcome replica. So we really have an Obligation replica as well as +an intent replica. Where Obligations are one shot and Intents produce a steady +state. + +TODO: When policy. Intent revision issuers have an evaluation policy + +TODO: Intent replica throttle indicator to help the scheduler choose which +replicas to sacrifice in a CPU exhaustion event. + +TODO: Explicitly mention that revisions and deltas can safely be held in memory +on different workers. since they are immutable. + +TODO: Focus on explaining the Replica as the core concept, and revise +terminology to say snapshot instead of revision and just say "delta stream" +instead of whatever we use for that. Just call it a state snapshot. + +TODO: How to handle replica version changes. Any change to the schema is going +to cause problems for dependencies. Code changes matter less but obviously +migration does need to happen. Code changes can effect the contract version +though if e.g. scope changes massively so it can't just be a schema version. + +TODO: Rename replica -> projection + projection node + +The persistent replica runtime is a deterministic incremental computation +environment. + +Background: https://github.com/the-draupnir-project/Draupnir/issues/979 + +The matrix-protection-suite has evolved to around a "revision issuers" concept +to provide deterministic, incremental computation and data-flow in order to +manage complex interactions and resource intensive operations. + +![](./images/2025-10-draupnir-data-flow.png) + +This abstraction has not been generalized and is ad-hoc. And the +matrix-protection-suite had specific and constrained design goals in its early +development in order to build a new core for the draupnir project: + +- Revision issuers are entirely in-memory structures. There are no asynchronous + methods to provide data for persistent storage. +- Attempts to persist specific revision issuers has required for backing stores + to be written that are in reality a downstream consumer of the deltas produced + by revisions outside of startup time. +- The backing stores are not generalized either and have virtually no support + for rollbacks in failed updates etc. +- Transactions between revision issuers are not formalized and it is possible + for revision issuers to become unsynchronised, if a delta fails to process or + is dropped, that effect is permanent until the system is restarted. +- Further the lack of "transaction" complicates persistence of revisions issuers + that are not sources of non-deterministic input e.g. the PolicyRoomRevisions + were persisted, we'd want to be confident that all deltas from the source + revision (room state) were applied in order. + +Most importantly, the matrix-protection-suite was designed to be used entirely +within one JS process. Using workers was never a consideration. This holds the +Draupnir project back from being able to use more centralized service +architectures. Which would be required to provide a popular service. + +## Proposal + +We propose that all computation in Draupnir to be expressed through _replica +actors_ using a new framework called the persistent replica runtime. This +transition would happen incrementally, rather than as a focussed effort like the +transition to the matrix-protection-suite and Draupnir 2.0. + +This would give us the following advantages: + +- Restart can be immediate and instantaneous. +- Workload can be balanced across worker threads using actor migration. +- All core computation is deterministic and reproducible, easing development and + testing. +- All data-flow interactions are explicit and auditable. +- Effects and effect handlers are first-class, providing not only pre-viewable + and auditable consequences for protections, but for all side-effects. +- The system is fudamentally fault tolerant: all computation is incremental, + transactional, and deterministic. And can be restarted at the last delta. + +_Revision issuers_ become _persistent replicas_. The live in-memory instance of +a _revision issuer_ becomes a _replica actor_. A meta-object is introduced to +describe the schema of the revisions and the logic of the actor called a +_replica description_. + +Persistence is implemented by a storage backend that takes the _replica +description_ and creates tables or documents from them. + +A scheduler is introduced that schedules _replica actors_ across worker threads +in Node.JS. + +## Concepts + +### Projection node + +Projection nodes are immutable snapshots that accurately reflect a system state +at the point the node was created. + +Projection nodes can reduce input _deltas_ into an _output_ delta, which can +then be used to produce another projection node that reflects a new system +state. + +### Projection delta + +Deltas represent the outcome of a computation related to a projection node that +provide a reproducible transformation of the projection state. + +Delta's have their own identity, a delta ID and backlink to a previous delta. + +#### Source delta + +Source deltas are special because they encapsulate the source of non-determinism +into the deterministic system. All computation in the system is caused by source +deltas. + +### Provenance data + +Each delta and projection node needs to include information about upstream +inputs it depends on. + +Each delta refers to the single source projection node that is the direct cause +for the current data flow. + +Each projection node refers to the current transient source projection nodes +that have been used to derive the current state. + +### Projection node reducer + +A projection node reducer is a deterministic function that takes a projection +node and an input delta and produces a new output delta specific to the +projection node. + +### Projection definition + +A projection definition defines how to build projections from partition data, +projection node reducers, and what the inputs to the projection are. + +### Rebuild delta + +A rebuild delta is a specific kind of delta that is the direct result of a +change in implementation of a projection, ie a change to the projection +definition. It may be important to issue a rebuild delta when bugs that caused +projections to produce faulty data are fixed. + +The change in projection definition is used as the source delta within a rebuild +delta. + +### Partitioning + +When projection definitions are combined with partition data when instantiated. + +replicas are partitioned by a key. This key is usually something like a Matrix +room ID, or the user identifier of a Draupnir instance. + +### Intent projection (effect) + +A intent projection is a special kind of projection that produces an intention +to see some outcome enacted. These intents are then carried out by outcome +projections (effect handlers). + +Enabled protections should be intent projections. + +### Outcome projection (effect handler) + +An outcome replica is a special kind of projection that records whether outcomes +have been issued for intents. The semantic meaning of the outcome is not +relevant, e.g. failure to ban a user is still an outcome that needs to be +recorded and is the outcome of the intent. + +## Semantics + +### Actor migration + +Actors need to be transportable across node.js worker threads in order to +balance CPU. + +### Preemption + +replica actor client code will be interruptive in order to gracefully handle +shutdown and migration of actors. + +This will be achieved by abusing I/O operations to signal interrupts. CPU bound +actor code will still have to cooperatively check for an interrupt signal. + +### Garbage collection + +Replica dependents will formally be tracked. Transitive dependencies will also +need to be considered for rollbacks to be successful. replicas will only be able +to discard an old revision when all transitive dependents have successfully +migrated to a superseding revision. + +### Scheduling + +The scheduler is responsible for starting replica actors and migrating them when +CPU becomes exhausted in worker threads. The scheduler gives each actor a +controller that signals interrupts. The store is expected to also receive the +controller so that it can preempt on I/O. + +The scheduler will have to be communicated `os.loadavg()` from each worker +thread and use this to trigger actor migration. Realistically we can probably +only move actors from high utilisation to low utilisation and have some strategy +that tries to identify and isolate the cpu intensive actors for reporting. + +### Crashing + +What happens when an actor crashes? + +#### Poisoned delta + +If the revision reducer crashes when processing a delta, and continues to crash +when re-attempted, the delta will be skipped with an explicit acknowledgement. +This signifies a bug in the implementation of the revision reducer and is really +a critical issue. It's important not to bring down the whole system, but quite +honestly the existence of a reducer bug in critical areas can be as bad as that. +Protections obviously can have faulty reducers and don't effect other +functionality. + +#### Other issues + +The actor should be restarted with exponential backoff. + +### Messaging + +How do deltas get communicated to dependencies? especially if the in-memory +actors can crash and be unreliable for message delivery. + +The deltas get written to the persistent store and a message is sent to the +other threads about the write so that the delta can be read back from the store +and propagated to dependencies. + +## Non-deterministic inputs + +The PRR needs non-deterministic inputs in order to do anything useful, such as +matrix events, commands, and time scheduling. + +These are provided by _source replicas_. + +### Source replica + +A source replica is a system or client provided replica with no dependencies +that produces deltas. Source replicas provide no data in their records, only the +deltas are significant. + +### Time source replica + +Some protections in Draupnir have time-scheduled background tasks, such as +DraupnirNews. + +These should be implemented by having a revision reducer that accepts a tick +from a time service. + +The time source replica is a system provided source replica that persists +records about when to send ticks to other replicas. Each replica that requires +time scheduling is given a adaptor replica to bridge them to the time source. So +that they are not informed about every single tick. diff --git a/design/images/2025-10-draupnir-data-flow.png b/design/images/2025-10-draupnir-data-flow.png new file mode 100644 index 0000000..e92bf5d Binary files /dev/null and b/design/images/2025-10-draupnir-data-flow.png differ