Decoupling the codebase from transportation details will unlock two things:
- People who need a transportation other then gRPC can implement it while keeping the core logic of the project intact
- We can have an in-memory implementation that is always chosen for intra-worker comms
While this was helpful for keeping a decoupled codebase, it also unlocks further optimizations:
- If a worker needs to pull data from another worker, and both happen to be identified with the same URL, it's safe to assume that the worker is trying to talk to itself, so it can fallback to a full in-memory connection.
- If the coordinator needs to communicate with a worker for feeding it work, and it happens to be co-located with the worker to which it should feed work, it can fallback to in-memory comms without paying network + serialization.
- If the coordinator decides that one task is more than enough for certain stages, it can colocate them in itself, and essentially execute the plan fully in-memory as if it was single-node.
- The AQE machinery can be used in a single-node context, relying fully on in-memory comms.
This is a collection of PRs an issues that will need to happen to consider this task done:
Decoupling the codebase from transportation details will unlock two things:
While this was helpful for keeping a decoupled codebase, it also unlocks further optimizations:
This is a collection of PRs an issues that will need to happen to consider this task done:
grpcon and off #579