add claude and local llm functionality

[ginberg]

see #46

[ginberg]

did you have the possibility to try this out @Faptimus420?

[Faptimus420]

did you have the possibility to try this out @Faptimus420?

Unfortunately, not yet - we are missing some system-level dependencies required by one of the R packages required by the tool on our system at the moment. Will test once that's been resolved

[Faptimus420]

Hi @ginberg. Apologies this took so long, getting all the dependencies required by the lock file on our system took a few days, as we're working on an offline system.

Nonetheless, I got the package running. I can point it to our local ollama models - when I create a new chat, I get "✔ ollama chat created with llama3.1:8b". However, when I try to actually send a prompt, I get
Error in req_perform() at ellmer/R/httr2.R:23:3: ! HTTP 500 Internal Server Error. ℹ Failed to create new sequence: unable to create sampling context
This is only happening from within PatientGenerator - if I prompt the same model directly using the ellmer library, I get a response.

I only set the model, provider, and echo arguments, I did not touch jsonSchemaPath or system_prompt. Let me know if you have any ideas - I'll ask the sysadmin to see if I can get any more detailed logs.

[ginberg]

hi @Faptimus420, thanks for trying this out!
When searching for this error, I found a few github issues this one for example. But I am not totally sure, how we need to adjust this package. We will look into it more. But please let us know if you have any ideas.

[Faptimus420]

I think I have figured out the cause of the HTTP 500 issue - it's not a bug per-se, but rather the default CDM schema (cdm54schema-complete.json) is far too long and complex - producing far too many rules - than our local Ollama is able to handle. I am able to run using the smaller cdm54schema-short_deprecated.json (even if it takes quite some time).

In the issue (#46), @cebarboza mentions that "Gemma 4 takes forever" - this is most likely why. Getting structured responses is quite a complex ask, and it might actually just take forever for a local model to generate. Having looked into Ollama's documentation, they actually explicitly say structured responses cannot use GPU or NPU acceleration.

I've asked the sysadmin to see if we can get the phi4 model added to our Ollama instance - I've read around that it's good for both tasks needing structured responses, and for running in resource-constrained environments.

[Faptimus420]

I have now tried with phi4 - I can confirm that using the smaller schema works - it generated a response in about 15 minutes on our hardware. I have not inspected it thoroughly yet, but on a first glance it seems fine.

However, getting the same error when using the bigger, default schema. From what I've been reading around, this might be a more fundamental issue with Ollama itself. When a structured response is requested using a JSON schema, Ollama parses the schema into a GBNF grammar, and uses it to make sure generated responses fit the grammar/schema. From what I've read, Ollama has a hardcoded limit on how complex a grammar can be, and I think the schema just hits the limit, producing too many rules. I'll see again about getting access to Ollama logs on the system as the error is coming from Ollama, not R, and maybe it could me in a more concrete direction.

[Faptimus420]

I managed to get access to Ollama logs, and found the problem - luckily, it has nothing to do with schema complexity. In cdm54schema-complete.json, on line 949, the RegEx pattern used is supported by the OpenAI API, but not by Ollama's parser. The fix (that makes this work on Ollama, without breaking OpenAI API), is to use this more explicit pattern instead: ^([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]$. With that, on our hardware using phi4, I managed to get a correct-looking response in about 30 minutes (I halso had to extend ellmer's timeout of 15 minutes with options(ellmer_timeout_s=<seconds>).

I am unable to commit this change myself as I am running on a system that doesn't have Internet access, so I cannot access GitHub from there. I am leaving for holiday as of tomorrow, so I will be unavailable. I will loop in one of my colleagues, so that he can assist in making this available to my colleagues who actually need to use it - I have informed them on how to modify the code examples in ReadMe.md to actually use the local model.

[ginberg]

@Faptimus420 thanks a lot for diving into this, much appreciated! Have a nice holiday. Great if you can pass this knowledge on to your coworkers and if they can make a PR to make this work with Ollama