From 71d2859b86a9aa9325972eb6effee766be6fd626 Mon Sep 17 00:00:00 2001 From: gnuxie Date: Mon, 24 Nov 2025 10:29:08 +0000 Subject: [PATCH 1/3] Projections exploration report. https://github.com/the-draupnir-project/planning/issues/88. --- exploration/25/projections.md | 79 +++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 exploration/25/projections.md diff --git a/exploration/25/projections.md b/exploration/25/projections.md new file mode 100644 index 0000000..26975ff --- /dev/null +++ b/exploration/25/projections.md @@ -0,0 +1,79 @@ +# Projections Exploration Report + +## Context + +In order to implement previews for +[policy list subscription](https://github.com/the-draupnir-project/planning/issues/2), +we would need a generic _revision_ interface that is accessible on the +_protection_ interface. Having a generic _revision_ type would also make it +easier to derive new sets of revisions for the purpose of observing the effects. +For example, viewing the effects of subscribing to a _policy room_ on the +_member ban synchronisation protection_ would require _revising_ the _policy +list revision_ for Draupnir's _watched policy rooms_, the _membership policies +revision_, and a new _intent_ _revision_ that could explain what the protection +was going to do. + +## Unknowns + +## Experiment + +We generalised the _revision_ and _revision issuer_ abstraction into a generic +_projection node_ and _projection_ type. This work was inspired by the _replica_ +concept from the Persistent Replica Runtime proposal. We then implemented a +_projection_ for the _member ban synchronisation protection_ and another for the +_server ban synchronisation protection_. + +Policy previews were then derived by destructuring the active protections from +the protections manager. Then hand coding a function to create deltas from each +input starting from the _policy list revision_ for Draupnir's _watched policy +rooms_. + +### Exploration points + +- Determine whether if it is reasonable to rewrite all the revisions in the data + flow from the _policy list revision issuer_ all the way to the _protection + projection intents_. +- Determine if cloning a snapshot of the pipeline and reducing with arbitrary + data in isolation of the original is possible. +- Determine how two inputs to a _projection_ interact when the data flow of two + inputs are in a fork because they share a transient dependency. And thus the + _projection_ is convergence of the fork. +- Determine how to handle hooks for plugins without making the plugin handle the + allocation of the listener. And while keeping the "hook registry" generic. + +## Outcome + +- A generic `Projection` and `ProjectionNode` type were successfully created and + used. + +- We were unable to arbitrarily clone a pipeline of _projections_ because only + the projection intents of protections were moved to the new abstraction. + However, we were able to reduce the deltas from each source revision to + provide a preview. And this was surprisingly simple. So it should be straight + forward to make this process generic later. Remember, the trick is to focus on + the deltas, not the projection nodes. + +- We were unable to experiment with how two inputs to a _projection_ interact + around a fork and merge. The suspicion I have is that if there needs to be + synchronisation or progress updates then the revision is badly designed in + some way. But it does need investigating. + +- When experimenting with _projection nodes_ it became clear that in some + situations the reducer for producing the delta had similar code as the reducer + for producing the next _projection node_ when reducing that delta. We thought + about what it would look like to merge the two steps into one join. But it + became apparent that doing so would mean maintaining an independent code path + that is rarely used for reducing the delta into a projection node. Ie if you + wanted to restore a projection node from the deltas it produces then you would + have to use a rarely used code path. Which is very unideal. The only way you + could do this is by only reproducing projection nodes from input deltas, not + the output deltas. It needs thought about whether this is something we want to + do for PRR. + +- Additionally we spared some thought for how to persist projection nodes and + run reducers without loading the entire projection. One idea was to shard the + projection node data and input delta and only run part of the reducer at a + time. But it's not clear to me how feasible it is to find common keys in input + to partition on especially when there are multiple input projections. It would + be highly desirable to do this over adding IO into reducer code which is meant + to be deterministic. From a03adecf6c12c283322f610c2f13595b0aa602b9 Mon Sep 17 00:00:00 2001 From: gnuxie Date: Tue, 2 Dec 2025 22:58:58 +0000 Subject: [PATCH 2/3] idk --- exploration/25/projections.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/exploration/25/projections.md b/exploration/25/projections.md index 26975ff..ba4d4a6 100644 --- a/exploration/25/projections.md +++ b/exploration/25/projections.md @@ -77,3 +77,10 @@ rooms_. to partition on especially when there are multiple input projections. It would be highly desirable to do this over adding IO into reducer code which is meant to be deterministic. + +- While we didn't have time to explore implementation of a protection hook + registry. We think that it is possible to create a protection hook registry + where each handle on a protection after construction is connected up with the + associated input. And disconnected when the protection is disposed. We think + this can be done by maintaining a metaobject with a description of each hook + that takes the protected rooms set, context, and protection itself as input. From 50fe9285d85d91667e2fcef74d69bc812acbe364 Mon Sep 17 00:00:00 2001 From: gnuxie Date: Fri, 9 Jan 2026 15:19:14 +0000 Subject: [PATCH 3/3] I think this is ready to commit. --- exploration/25/projections.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/exploration/25/projections.md b/exploration/25/projections.md index ba4d4a6..02beb30 100644 --- a/exploration/25/projections.md +++ b/exploration/25/projections.md @@ -84,3 +84,7 @@ rooms_. associated input. And disconnected when the protection is disposed. We think this can be done by maintaining a metaobject with a description of each hook that takes the protected rooms set, context, and protection itself as input. + +- We discovered that forks in dataflow have to be handled by keeping either side + of the fork's source stream dependencies in sync and reducing deltas from both + sides of the fork into a single delta.