Register the connector-service (webapi) plugin in the Flyte 2 executor#7565
Register the connector-service (webapi) plugin in the Flyte 2 executor#7565dejanzele wants to merge 3 commits into
Conversation
The OSS Flyte 2 executor never calls RegisterConnectorPlugin, so task types backed by external connectors (agents) have no plugin and fall through to the pod plugin. Register it during executor setup, and clear ResourceQuotas since the v2 executor wires no ResourceRegistrar/ResourceManager. Enables routing tasks (e.g. armada) to a connector service over gRPC. Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
There was a problem hiding this comment.
Pull request overview
Registers the Flyte webapi connector-service plugin in the v2 executor and restores required pluginmachinery contracts by providing no-op quota registration/allocation implementations, so connector-backed task types can be routed and executed without crashing.
Changes:
- Register the connector-service (webapi) plugin during executor startup so
connector-servicetasks resolve to a handler. - Provide a no-op
ResourceRegistrar(setup-time) andResourceManager(execution-time) to satisfy webapi allocation-token/quota paths. - Add unit tests validating the no-op registrar/manager behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
executor/setup.go |
Registers connector plugin before registry init; wires in no-op resource registrar and resource manager. |
executor/pkg/plugin/resource_manager.go |
Introduces no-op implementations of ResourceRegistrar and ResourceManager for executor usage. |
executor/pkg/plugin/resource_manager_test.go |
Adds unit tests for the no-op registrar/manager. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
|
Thanks, good catches. Added a no-op |
|
@pingsutw there was some work already on this right |
Registers the connector-service (webapi) plugin in the Flyte 2 executor so it can run connector tasks. Today a task whose type resolves to
connector-servicehas no plugin to handle it, so it never runs.The webapi machinery also expects a ResourceManager and a SecretManager from the plugin setup context, which the executor doesn't supply. This adds a no-op ResourceManager and registrar so the allocation-token path doesn't nil-dereference.
Closes #7564.
Tested with unit tests for the no-op ResourceManager and registrar, and manually end to end: a real connector (armada-flyte) runs
@env.taskfunctions as Armada jobs through this change.