CopilotJ as an MCP server?

[ctrueden]

In troubleshooting issues relating to #11, I wondered whether CopilotJ would be better as an MCP server that can be plugged into existing agentic AI clients, rather than using its own bespoke client. It would almost certainly simplify the codebase. The question is whether the tradeoffs would be worth it. Below is an exchange I had with Claude about the issue, in case it's interesting to you. To be clear: I'm not necessarily advocating for any particular architecture. But my impression when studying the codebase is that CopilotJ reimplements many features that already exist in other agentic AI tools, so I think it's worth asking the question seriously: would a different architecture work better for CopilotJ's goals?

What about implementing Fiji-specific tools, skills, even an MCP server, whatever the right layers of abstraction are to allow existing agentic AI assistants to develop image analysis workflows in Fiji -- i.e. meeting the same requirements CopilotJ seems designed to meet -- but without so many of the pitfalls that happen from reimplementing so much of the user interaction stack?

The right architecture: an MCP server

An MCP server wrapping the Fiji bridge is the correct abstraction. Here's what the stack looks like:

Claude Desktop / Claude Code / any MCP-capable client
            ↕  MCP protocol (tools, resources)
       Fiji MCP Server  (Python, ~300–500 lines)
            ↕  WebSocket (same protocol as current CopilotJ bridge)
   CopilotJBridge Plugin  (already written, already installed)
            ↕
         Fiji / ImageJ

You get for free — from the AI client — everything CopilotJ reinvents less well: conversation history, multi-turn context management, prompt caching, streaming, token efficiency, and the UI. You only implement the Fiji-specific layer.

What the MCP tools would actually be

Looking at CopilotJ's tool list, the core set is small:

@mcp.tool()
def run_macro(script: str, timeout: int = 30) -> str:
    """Execute an ImageJ macro script in the running Fiji instance."""

@mcp.tool()
def capture_fiji() -> ImageContent:
    """Capture the current state of Fiji as an image."""
    # Return image bytes — Claude sees it natively, no separate VLM needed

@mcp.tool()
def execute_python(script: str) -> str:
    """Execute a Python script via PyImageJ."""

@mcp.tool()
def list_windows() -> list[str]:
    """List all currently open image windows in Fiji."""

@mcp.tool()
def folder_summary(path: str) -> str:
    """List files in a directory."""

That's essentially it. The perception tool becomes capture_fiji() returning an image directly — Claude handles the vision natively, no COPILOTJ_VLM_MODEL needed. The elaborate imagej_perception orchestration in CopilotJ exists only because it had to glue together a separate VLM call; native tool-use with vision collapses that to one tool.

What you'd need to build

The WebSocket bridge protocol is already implemented on both sides — the Java plugin is installed in your Fiji, and the Python bridge code is in copilotj/server/bridge.py and copilotj/plugin/. The MCP server is a thin wrapper: reuse the bridge connection code, expose the tools, done. The hard Fiji integration work is already written.

The perception / VLM handling is actually simpler with MCP than in CopilotJ — instead of orchestrating a separate model call, you return an image and the client model sees it directly.

What you'd give up

• Knowledge bank / RAG: CopilotJ's kb_retrieve for past sessions. This could be implemented as an MCP resource or tool if needed, but honestly for a basic "segment blobs" workflow it's just overhead.
• Workflow save/replay: Useful, but can be a separate MCP tool, or just left to the AI to write a script.
• The multi-agent routing: Tool Agent, Research Agent, etc. With a capable model this is largely unnecessary — the model decides which tools to call.

The honest assessment

CopilotJ's genuine contribution is the bridge plugin and the tool implementations. Everything else — the web UI, the ReAct parsing, the multi-agent orchestration, the prompt engineering, the conversation management — is reinvention of things the AI providers do better. An MCP server distills it to exactly the valuable part.

[yzx9]

Hi @ctrueden, this is a great point. We share similar concerns about these architectural tradeoffs.

One way we’ve been thinking about this is along two architectures: 1. building a more domain-specific agent system. 2. adapting a general-purpose agent with domain-specific extensions (e.g., via MCP)

Each comes with its own tradeoffs, especially around flexibility vs. control. We chose 1 because bioimage analysis is a complex, multi-step, and stateful task. It requires tight control over planning, tool use, validation, and prompt engineering. For this reason, we prefer to build a custom system, as it allows us to better explore and identify best practices for smart bioimage analysis.

We are also actively exploring how to integrate with existing agent and have already developed an MCP prototype. However, making MCP + general agents work well still requires additional domain-specific support (e.g., state handling, workflow control, and prompt engineering).

[yzx9]

Hello. We have reached a final decision in #26: we have exposed the CopilotJ bridge as an MCP server.

This is a pure Java-side implementation that runs in Fiji Latest. Its primary functions include retrieving Fiji’s status and running macro scripts. However, based on our testing, it still requires substantial prompt engineering for the large language model to use it correctly.

Moving forward, we would like to continue exploring additional solutions. As described in #30, many Fiji plugins are still designed primarily around GUI interaction, which can be especially valuable for human-in-the-loop workflows in bioimage analysis. To make these workflows more LLM-friendly, I am currently working on an MCP-based visual window perception and interaction scheme for Fiji (#152). Similar to playwright-mcp, this approach aims to turn Fiji windows into structured text representations of available views and actions. I hope this can make GUI interactions more recordable, replayable, and reproducible.

We will close this issue for now. Thank you for your patience.