feat: add openai api support and chunking#8
Conversation
| def _is_model_loading(exception: BaseException) -> bool: | ||
| return ( | ||
| isinstance(exception, openai.InternalServerError) | ||
| and exception.status_code == 503 |
There was a problem hiding this comment.
Is there any risk that a 503/not responding might mean that something is long-term offline and not just still loading? Do we need a timeout for example?
There was a problem hiding this comment.
I'm wondering whether it's worth getting rid of this behaviour. It was all so it worked on cloud run - because the model would wake up on your first request. Now we're just hosting models it's probably not necessary anymore.
There was a problem hiding this comment.
Although saying that I don't think it does any harm. It does give up on retrying - and I suppose 503 errors are something where it's always worth trying a retry.
| """ | ||
| We frequently end up chunking input sentences when we do things with Gen AI. | ||
| This has become replicated across multiple projects, | ||
| so we made some laughably simple helper functions here. |
There was a problem hiding this comment.
All true, but remember this is an open-source library :-)
There was a problem hiding this comment.
I just thought it was worth the context for why chunking functionality was in there
2 participants
Summary
As I've been adding stuff recently which all runs the same code to run requests on OpenAI API Compatible LLMs, I have added the functionality to genai utils.
This is mostly so we can keep things consistent like
Specifically for this PR I have added:
Pull request checklist
/reviewmy PR, and reviewed any LLM generated code myself