multimodal function response

[falmanna]

Seems like it is not possible to return multimodal responses to tool calls even if the model support that like gemini.

[dosco]

maybe possible for all models function returns certain type of return value we set a function result thats just a placeholder and then add another user role entry with the multi-modal response. might work

[falmanna]

maybe possible for all models function returns certain type of return value we set a function result thats just a placeholder and then add another user role entry with the multi-modal response. might work

Yes that's how I worked around it but it needed a lot of funky code and debugging to figure out what is happening. I also had to figure out that I need to add processor tag to keep the loop working running.

Supporting this natively similar to ai-sdk would be a nice feature.

[morgan-coded]

@dosco — your earlier suggestion here (set a placeholder function result, then add a following user-role turn carrying the multimodal response) looks like the right provider-neutral shape to me: it works for any model instead of leaning on one provider's tool-result format.

Before I implement it — would you want that wrapped into AxChatRequest, so a tool can return a typed multimodal result and Ax expands it into the placeholder + follow-up turn, or handled at the memory/adapter layer instead? Once you confirm the shape, I'll build it — scoped to the function-result, memory, and Gemini mapping path first.