Imaging Data Commons integration relies on incorrect, outdated and incomplete information

[fedorov]

I am a technical lead of the Imaging Data Commons team. I did a quick experiment with your platform, and here are some quick suggestions.

1. Based on the responses of the agent, you rely on a notebook that is deprecated. The below is incorrect.

2. Your integration is unaware of the latest (v23) version of IDC, and is using an old version of idc-index. I could not get the answer why!

3. Is the document/prompt describing integration with IDC available publicly? I am happy to review and offer suggestions for improvement.

4. You may want to look at the officially supported IDC Claude skill https://github.com/ImagingDataCommons/idc-claude-skill, and consider integrating it with your system.

The capabilities of your system in terms of supporting IDC are critically diminished due to the way integration is implemented at the moment. As an example, it is unaware of how to access various metadata attributes that are readily available via idc-index, and comes up with heuristics to determine pathology slide stain, leading to incorrect answers and the false claim below.

Here's the response from Claude equipped with the aforementioned imaging-data-commons skill: https://claude.ai/share/4088ce4c-17fc-4359-8d70-6e4ee0d68af6. I confirm this response is accurate (and falling back to parsing SeriesDescription is exactly what your system did!).

I am sorry if this is not the right place to ask questions, but the notebook interface does not appear to have any place to contact maintainers, and its interface includes pointer to this repo. I hope this is the right place!

[brandomr]

@fedorov thanks for flagging this! I'm not at all surprised you're finding that Beaker is using an outdated version of idc-index--we added that integration quite a while ago and didn't revisit it. We can definitely update this and I'd be curious your feedback once it's fixed. One question for you: can we get the same functionality now through cdapython as we could through idc-index?

[fedorov]

we added that integration quite a while ago and didn't revisit it

Is that "integration" - as it is right now - part of some repository? What is the process for refining it? Is that part of some proprietary component? I hope not!

can we get the same functionality now through cdapython as we could through idc-index?

I am not an expert in cdapython, but from the little I know, I can't imaging how you possibly could.

[brandomr]

The "integration" is in our biomedical repo which supports the system you've tested: https://github.com/jataware/biome/tree/main/adhoc_data/specifications/idc

It provides a specialist agent this README https://github.com/jataware/biome/blob/main/adhoc_data/specifications/idc/attachments/idc.md which is probably outdated

The version of idc-index isn't pinned https://github.com/jataware/biome/blob/main/pyproject.toml#L30 but we haven't rebuilt that system in some time since our work on BDF ended. But we'll be restarting work on this soon and either way we can pretty easily update the "integration".

In Beaker an "integration" is very similar to a Claude code skill--though captured a bit differently (via a YAML file). We can easily update this using the official skill and the updated doc notebook.

[fedorov]

Thanks for the explanation!

I would strongly encourage you to move away from your internal Yaml convention, and instead adopt skills. Among other advantages, you might be able to leverage numerous skills already available, such as those in https://github.com/K-Dense-AI/claude-scientific-skills/tree/main/scientific-skills. Of course, you can use Claude to rewrite IDC skill and fit it into your Yaml format, but your format is unlikely to be used by anyone else, and you won't be able to leverage the continuous refinements and benefit from broad adoption of skills format.

It is also important to build your framework to accommodate continuous refinement of both the resources referenced, and the capabilities enabling access to those resources. Ideally, you would point to the IDC skill that we are maintaining, and pull in new versions as those are released. I will be updating our skill regularly, but I can't update various platform-specific interfaces to IDC...

[brandomr]

That's a totally fair point--we have a bespoke format we developed that pre-dates Claude skills. Our system handles these a bit differently; in order to avoid the main agent pulling the integration / (skill) into its context we rely on a subagent topology which handles queries against an integration to more efficiently manage context. We also have custom logic that tries to match a user query against a known example; if an exact match is found we short circuit and rely on that example code to provide a less stochastic experience. I don't think it would be a big lift for us to natively support skills though and I love the idea of dynamically fetching the official skill you're maintaining. I'll keep you posted on progress to that end.

[fedorov]

in order to avoid the main agent pulling the integration / (skill) into its context we rely on a subagent topology which handles queries against an integration to more efficiently manage context.

This sounds like something similar to the concept of "references" in skills: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#progressive-disclosure-patterns.

Looking forward to seeing how things develop in your system. And in either case, I am happy to answer any questions related to IDC and to help you make your resource more capable in in this regard! Please reach out any time!

[brandomr]

Hi @fedorov we've implemented the official skill via a couple PR's jataware/biome#79 and jataware/biome#80. This has actually given us some ideas on how we can improve our system to dynamically support fetching skills like this at runtime and a more dynamic approach to managing them in context.

For your demo query I am now seeing the correct results, though our system gets there in a slightly less elegant way. That's what I think we can really improve upon. If you run other A/B tests in Beaker vs Claude (+skill) please let me know, this was quite useful

[fedorov]

@brandomr great, happy to hear this was not a huge lift!

What about references, which are part of the skill and are referenced from SKILL.md? See https://github.com/ImagingDataCommons/idc-claude-skill/tree/main/references.

You might want to consider setting up a cron GitHub Actions workflow that would check for skill updates (there are some new features that I am planning to add, and we have release v24 coming up in the next 2-3 weeks, which will also result in a new version of the skill).

[brandomr]

Oh actually we included the references in that second PR since I missed them in the first! We now dynamically fetch the files directly from your Github at runtime; if that's unavailable it falls back on what we store in the repo. See https://github.com/jataware/biome/blob/main/entrypoint.sh

[fedorov]

Nice! Sorry I missed it! Very cool!

I will let you know after I manage to try this out. I will definitely make time to do that!