With the WorkerChannel abstraction shipped in #512, we can have a fully in-memory implementation that communicates any WorkerChannel client call with a local Worker.
The implementation will likely be small and pretty much a proxy to the Worker methods.
About wiring this up with the codebase, the in-memory comms should likely interact well with the present LocalWorkerContext in the TaskContext at the moment of calling ChannelResolver::get_worker_client_for_url:
- If
LocalWorkerContext.self_url == url: fallback to the in-memory implementation of the WorkerChannel
- Otherwise, use whatever other
WorkerChannel implementation is registered
This probably means that the ChannelResolver::get_worker_client_for_url should be extended with another task_ctx: &Arc<TaskContext> argument so that we can retrieve the WorkerChannel implementation from there.
One idea that comes to mind, is to just have a helper function that is used in the places where WorkerChannels are retrieved for specific URLs, and apply this logic of returning the in-memory implementation instead of the real one if LocalWorkerContext.self_url == url:
|
let mut client = ch_resolver.get_worker_client_for_url(&target_url).await?; |
|
let mut client = channel_resolver.get_worker_client_for_url(&url).await?; |
With the
WorkerChannelabstraction shipped in #512, we can have a fully in-memory implementation that communicates anyWorkerChannelclient call with a localWorker.The implementation will likely be small and pretty much a proxy to the
Workermethods.About wiring this up with the codebase, the in-memory comms should likely interact well with the present LocalWorkerContext in the
TaskContextat the moment of callingChannelResolver::get_worker_client_for_url:LocalWorkerContext.self_url == url: fallback to the in-memory implementation of theWorkerChannelWorkerChannelimplementation is registeredThis probably means that the
ChannelResolver::get_worker_client_for_urlshould be extended with anothertask_ctx: &Arc<TaskContext>argument so that we can retrieve theWorkerChannelimplementation from there.One idea that comes to mind, is to just have a helper function that is used in the places where
WorkerChannels are retrieved for specific URLs, and apply this logic of returning the in-memory implementation instead of the real one ifLocalWorkerContext.self_url == url:datafusion-distributed/src/worker/worker_connection_pool.rs
Line 130 in e3ac0fb
datafusion-distributed/src/coordinator/query_coordinator.rs
Line 199 in e3ac0fb