How to implement phi-3-vision-128k-instruct (onnx_runtime_genai) with simpleAI ?

[Youho99]

Bravo for your work, there is potential in this project!

I would like to use this project to create a compatible openai api for the model: phi-3-vision-128k-instruct (onnx version in the onnx_runtime_genai project).

First link is the tutorial for the onnx model, second one is the github project.

Only, I don't really understand how to go about it. Could you help me ?

[Youho99]

I think i need the image support modification

[lhenault]

Hi @Youho99, thanks for the kind words! I've been busy elsewhere these last months and haven't worked on this for a while, but happy to answer your question.

We would indeed have to implement the /images/ endpoints to make it work with vision models. That shouldn't be too hard to do (I might even find a bit of time for this) and would be a nice addition to the project.

[Youho99]

Hi @lhenault !

I can help implement this feature.

This will allow me to understand in more depth how the whole system evolves.

[Youho99]

We would indeed have to implement the /images/ endpoints to make it work with vision models. That shouldn't be too hard to do (I might even find a bit of time for this) and would be a nice addition to the project.

In my use, I do not want to "generate" an image however!

I want to use a VLM, so call the chat/completions endpoint, giving the image and the prompt as input

[lhenault]

Oh ok, thanks for clarifying. I've done a bit of research about that, and I've found this thread explaining how to structure a message for this purpose.

Looks like it's still in beta, and expects the user to first upload the image to provide its URL.

If you want to implement this, I think you might want to skip the uploading part in a first time, and use an existing image online. On the model side of things, you can do something like this and then simply pass the input to the VLM.

[Youho99]

In my case, I will use base64 images
I already have my model which works correctly in an API that I created (which is not Openai compatible)

So the backend of the model, I know it works, my question marks are not there.

Rather, it's the whole system. I started to look in depth, and I already saw that the code could not handle "content" correctly for an image for example.

I'm currently working on changing this, I'm going from bugs to crashes 😆