Exploration of projections#94
Conversation
| - We were unable to experiment with how two inputs to a _projection_ interact | ||
| around a fork and merge. The suspicion I have is that if there needs to be | ||
| synchronisation or progress updates then the revision is badly designed in | ||
| some way. But it does need investigating. |
There was a problem hiding this comment.
The way to fix this is to reduce and merge deltas before reducing a new projection node and update track via source progress ID's. You only need update tracking relative to source streams.
| - When experimenting with _projection nodes_ it became clear that in some | ||
| situations the reducer for producing the delta had similar code as the reducer | ||
| for producing the next _projection node_ when reducing that delta. We thought | ||
| about what it would look like to merge the two steps into one join. But it | ||
| became apparent that doing so would mean maintaining an independent code path | ||
| that is rarely used for reducing the delta into a projection node. Ie if you | ||
| wanted to restore a projection node from the deltas it produces then you would | ||
| have to use a rarely used code path. Which is very unideal. The only way you | ||
| could do this is by only reproducing projection nodes from input deltas, not | ||
| the output deltas. It needs thought about whether this is something we want to | ||
| do for PRR. |
There was a problem hiding this comment.
It's probably an essential trade-off of the system. And honestly seems like a small price to pay for the benefits
| - Additionally we spared some thought for how to persist projection nodes and | ||
| run reducers without loading the entire projection. One idea was to shard the | ||
| projection node data and input delta and only run part of the reducer at a | ||
| time. But it's not clear to me how feasible it is to find common keys in input | ||
| to partition on especially when there are multiple input projections. It would | ||
| be highly desirable to do this over adding IO into reducer code which is meant | ||
| to be deterministic. |
There was a problem hiding this comment.
Deltas can be merged deterministically so the number of inputs shouldn't be relevant. I think the problem here though is that the projection node itself has to be partitioned (so the whole thing can be loaded into memory and run reducers). It's just a theory as to whether we can do that. And we also need to consider that if this is what we depend on, then we need to make sure that there aren't fundamental limitations to partitioning projection nodes or we can do that arbitrarily and infinity. So that's what a future experiment should test.
1 participant
No description provided.