LLM standardization in Fiji

[hinerm]

Hi @yzx9!

I just wanted to let you know that I've been working on the Fiji-llm project with the aims of creating a central core for LLM integration in ImageJ/Fiji.

Right now, the key elements are:

• A built-in MCP server, connecting tool definition to the SciJava plugin framework so that the community can add LLM-facing API without reinventing the underlying utilities, and to define core useful functionality (e.g. querying application context)
• A similarly extensible framework for model providers
• A simple chat interface (my personal goal is to focus on local models assisting with scripting and macros, and not on agentic image analysis)

Everything is still in active development and regularly being refactored 😓 but I'd love to know if there are areas that would be useful to developers like yourself.

[yzx9]

Hi @hiner!

Great to hear that you’re also interested in using LLMs to interact with Fiji. I’m very interested in this direction as well. I especially like the idea of treating Fiji-LLM as a shared layer for LLM integration, instead of having each project build its own separate bridge. Hooking that into the SciJava plugin framework also seems like the right place to start, since that is where a lot of Fiji’s extensibility already lives.

I’ve also been working recently on adding an MCP implementation to our CopilotJ Bridge, based on the MCP Java SDK (#26). Our approach is a little different from Fiji-llm: we’re leaning more toward providing a smaller set of more opinionated APIs, while giving the model a bit more autonomy in how it uses them.

One concern I have with letting developers define tools completely freely is that the number of tools could grow very quickly. From our experiments, simply having more tools doesn’t necessarily make tool use more effective for the model. A good example is how Claude Code deals with too much context: when MCP-related content takes up too much space, it gets folded out of the active context and exposed through search tools instead. Low-quality tools, or an already-saturated tool set, can make that search less effective as well.

That said, I still think giving plugin developers more control is a worth trying. It lets them manage their own complexity and expose the parts of their plugins that they understand best. At the same time, I think we’ll still need better ways to shape the experience around this. For example, helping developers provide better scripting support, documentation, and searchable examples could make a big difference. In many cases, even three to five good examples, or examples showing alternative ways to call the same functionality, might be enough.

The current shape of your run_macro API also seems like it could support more advanced behavior, such as parameter validation. LLMs sometimes call parameters that don’t exist, or parameters that are incompatible with the installed version of a plugin. It would be really useful if those issues could be caught before the script actually runs. This is especially important because Fiji doesn’t currently have very strong rollback support. I noticed that Ctrl-Z supports one level of undo, but for an LLM-driven workflow, being able to roll back an entire script would be much more meaningful.

Another related issue is that many Fiji plugins are still designed primarily around GUI interaction. In our code, we’ve also experimented with a Playwright-like mode, where we turn windows into a text representation of views and available actions and send that to the agent. Given the success of Playwright and agent-browser, I think this might be another promising direction for handling GUI-based interaction in Fiji.

Apologies if this is a bit of a brain dump. I’ll try to test your code soon and organize my thoughts more clearly

[hinerm]

Apologies if this is a bit of a brain dump.

@yzx9 No apology needed! I appreciate the thoughtful comments 😄

I’ve also been working recently on adding an MCP implementation to our CopilotJ Bridge, based on the MCP Java SDK (#26).

Very cool - I want the MCP Java SDK route as well.

One concern I have with letting developers define tools completely freely is that the number of tools could grow very quickly. From our experiments, simply having more tools doesn’t necessarily make tool use more effective for the model.

I completely agree with this and have similar fears. But ultimately this is very similar as plugins in general. I think it is important for community developers to have the ability to add their own extensions, and I believe our role is to provide architecture to encourage best practices and positive outcomes, but the quality of the final product is up to the individual developers, driven by community need.

In Fiji we have a couple mechanisms that help here: first, devs can ship tools on update sites, allowing users to select what pools of tools are actually installed locally.

Second, I created a ToolContext which is poorly named and implemented at the moment. It is actually "tool scope" and the intent is to allow users to select the actual tools that are provided to the model (much like in VS code or other harnesses you can see the full list of tools, with checkboxes to turn them on/off). This creation was premature and I'm not sure what this should look like in the end, or if we actually need it (it may be sufficient to turn update sites on/off).

Long term I am curious if there are general frameworks for tool searching/folding. I built the LLM support on langchain4j and it's pretty easy to filter the tool set, so we do have the ability to control this, I'm just not 100% sure what the best way to do so is (or if we should use some existing, established practice/library)

LLMs sometimes call parameters that don’t exist, or parameters that are incompatible with the installed version of a plugin. It would be really useful if those issues could be caught before the script actually runs

Yes this is a very annoying issue that we run into quickly when writing tools. I am hoping that the framework I'm building helps standardize tool definition in Fiji, with good descriptions and named parameters, in such a way to give agents the best chance at successful use. Also, langchain4j does expose tool exception handlers. My goal is to provide extensible hooks in case tool developers need to handle particular errors in a particular way.

You may also be interested in this issue as it sounds like the "best practices" may be moving towards wrapping MCP surfaces in something more familiar to LLMs.

Another related issue is that many Fiji plugins are still designed primarily around GUI interaction.

Yes this is a good point. In some ways I see this as a benefit because of my own desire for the agents to guide a human user.. so that agents run interactive commands to pop dialogs and advise the user how to parameterize them.

Another reason I am trying to focus on scripts is that they are inspectable, shareable, and reproducible. My understanding of CopilotJ right now is that you have an output format for your workflows that the agent can read in? I am hoping that by shipping core tools for script writing to encourage developers to use scripts as their "workflow" data structure. But I still need to add some support for LLMs to understand the unique syntax and limitations of script writing in Fiji.

In any case, I am interested in your Playwright direction, as that might be functionality that makes sense to maintain in core Fiji!

Thanks again for your thoughts. No rush, I'm just glad to connect with community devs 😄

[yzx9]

@hinerm Yes, MCP is not the only possible approach. Whether it is a CLI, a script, or MCP, the most important thing is to provide agents with an intuitive way to access the resources users need

From this perspective, I think Fiji’s existing scripting capabilities are already sufficient to provide many of these abilities. What seems more important to me is making the scripts more LLM-friendly, for example through more intuitive and robust design, better documentation, programmatic and semantic validation, and rollback support

At the same time, this approach helps avoid some of the issues that come with exposing too many tools. Rather than providing a dedicated interface for every capability, agents can access functionality through scripts and look up well-documented scripting APIs and examples to discover and use the resources they need

I also agree that the GUI-oriented nature of many Fiji plugins can be especially valuable for human-in-the-loop workflows in bioimage analysis. To make these GUI-based workflows more LLM-friendly, it would be helpful if GUI interactions had some structured representation, so they can be recordable, replayable, and reproducible