Context
When no ImageJ plugin is connected to the CopilotJ server, today every plugin call fails and the user gets an opaque error. The short-term fix (branch fix-plugin-not-connected) makes this safe: a dedicated PluginNotConnectedError is raised at the source, the agent short-circuits at the tool boundary instead of burning retries, and the user sees a single curated, actionable message via the existing new:error channel — e.g.:
ImageJ is not connected. Please start Fiji with the CopilotJ plugin installed and wait for it to connect to the server, then resend your request.
Unknown crashes are also surfaced (sanitized) so the stream is never silently empty. That solves the immediate "user is left in the dark" problem.
This issue captures the long-term goal: the agent itself should understand plugin-connection state and adapt, rather than emitting a canned static string the moment a tool call fails.
Long-term vision
1. Plugin health awareness
Surface plugin connection state (connected / disconnected, last-seen timestamp, plugin version) to both the agent and the UI. The agent should be able to reason about connection state up front instead of discovering the failure mid-tool-call. Concretely: a plugin_status tool or system-prompt context the leader can query before acting, plus a connection indicator in the frontend.
2. Graceful degradation
Not every request needs ImageJ. Today the leader typically opens every dialog with an ImageJ perception/snapshot call, so a missing plugin blocks even plugin-independent questions. Long-term, plugin-independent requests (knowledge-bank lookups, research/web search, general Q&A about workflows or macros) should succeed without a connected plugin; only image/macro/perception actions should require it. The agent should recognize when a request doesn't need the plugin and answer directly.
3. Coaching
When the plugin is needed but absent, the agent should walk the user through resolving it — how to start Fiji, how to confirm the CopilotJ plugin is installed and connected, and how to troubleshoot common failures — rather than a one-line banner. This can reuse the existing version-skew / install troubleshooting FAQ already in web/src/assets/manual.md. The static PluginNotConnectedError message becomes a fallback, not the only response.
4. Recovery
Detect mid-conversation (re)connection of the plugin and let the user resume the interrupted task seamlessly — ideally without re-sending the whole message. Today a disconnect mid-task ends the dialog; the user must reconnect and retype.
Why now / what's missing
- No concept of "plugin health" in the agent's context — failures are only discoverable by calling a tool and catching the error.
- No behavioral guidance in the leader/executor prompts for the disconnected state, so the model can't coach or degrade gracefully on its own.
- The connection lifecycle (connect / disconnect / reconnect) isn't surfaced as events the agent or UI can react to.
Foundation
Builds on the short-term fix (fix-plugin-not-connected), which introduced the PluginRequestError / PluginNotConnectedError hierarchy and sanitized crash surfacing — the primitives a richer agent behavior can be built on.
Open questions
- Should the agent get a dedicated
check_plugin_status tool, or should connection state be injected into the system prompt each turn?
- How much of the coaching content lives in the prompt vs. the knowledge bank (
macro/, research/)?
- Should plugin-independent routing be explicit (the model decides) or enforced (the framework gates tools by required-capability)?
/cc any discussion on scope welcome.
Context
When no ImageJ plugin is connected to the CopilotJ server, today every plugin call fails and the user gets an opaque error. The short-term fix (branch
fix-plugin-not-connected) makes this safe: a dedicatedPluginNotConnectedErroris raised at the source, the agent short-circuits at the tool boundary instead of burning retries, and the user sees a single curated, actionable message via the existingnew:errorchannel — e.g.:Unknown crashes are also surfaced (sanitized) so the stream is never silently empty. That solves the immediate "user is left in the dark" problem.
This issue captures the long-term goal: the agent itself should understand plugin-connection state and adapt, rather than emitting a canned static string the moment a tool call fails.
Long-term vision
1. Plugin health awareness
Surface plugin connection state (connected / disconnected, last-seen timestamp, plugin version) to both the agent and the UI. The agent should be able to reason about connection state up front instead of discovering the failure mid-tool-call. Concretely: a
plugin_statustool or system-prompt context the leader can query before acting, plus a connection indicator in the frontend.2. Graceful degradation
Not every request needs ImageJ. Today the leader typically opens every dialog with an ImageJ perception/snapshot call, so a missing plugin blocks even plugin-independent questions. Long-term, plugin-independent requests (knowledge-bank lookups, research/web search, general Q&A about workflows or macros) should succeed without a connected plugin; only image/macro/perception actions should require it. The agent should recognize when a request doesn't need the plugin and answer directly.
3. Coaching
When the plugin is needed but absent, the agent should walk the user through resolving it — how to start Fiji, how to confirm the CopilotJ plugin is installed and connected, and how to troubleshoot common failures — rather than a one-line banner. This can reuse the existing version-skew / install troubleshooting FAQ already in
web/src/assets/manual.md. The staticPluginNotConnectedErrormessage becomes a fallback, not the only response.4. Recovery
Detect mid-conversation (re)connection of the plugin and let the user resume the interrupted task seamlessly — ideally without re-sending the whole message. Today a disconnect mid-task ends the dialog; the user must reconnect and retype.
Why now / what's missing
Foundation
Builds on the short-term fix (
fix-plugin-not-connected), which introduced thePluginRequestError/PluginNotConnectedErrorhierarchy and sanitized crash surfacing — the primitives a richer agent behavior can be built on.Open questions
check_plugin_statustool, or should connection state be injected into the system prompt each turn?macro/,research/)?/cc any discussion on scope welcome.