Skip to content

[Epic] Abstract transportation protocol details #580

Description

@gabotechs

Decoupling the codebase from transportation details will unlock two things:

  1. People who need a transportation other then gRPC can implement it while keeping the core logic of the project intact
  2. We can have an in-memory implementation that is always chosen for intra-worker comms

While this was helpful for keeping a decoupled codebase, it also unlocks further optimizations:

  • If a worker needs to pull data from another worker, and both happen to be identified with the same URL, it's safe to assume that the worker is trying to talk to itself, so it can fallback to a full in-memory connection.
  • If the coordinator needs to communicate with a worker for feeding it work, and it happens to be co-located with the worker to which it should feed work, it can fallback to in-memory comms without paying network + serialization.
  • If the coordinator decides that one task is more than enough for certain stages, it can colocate them in itself, and essentially execute the plan fully in-memory as if it was single-node.
  • The AQE machinery can be used in a single-node context, relying fully on in-memory comms.

This is a collection of PRs an issues that will need to happen to consider this task done:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions