Limit total token output

[eliott07]

In order to prevent exploits that try to overload servers by generating endless responses, we should establish a limit for the number of output tokens.

If the users requests, we should maybe add a "keep generating" button, depending on complexity and @robin-lecomte 's opinion on this.

[maxenceh]

@robin-lecomte you can see current behaviour here -> an additional text explaining the response cut off

[robin-lecomte]

Thank you @maxenceh,
A couple of points on this:

Minor, but I'd move the info text above the buttons and set it in bold. That keeps it visually tied to the response instead of leaving a stray line floating underneath.
The sentence "Try rephrasing your question to get a shorter answer" feels a bit blaming, since it implies the user misused the assistant. I'd go with something like: "This response reached its maximum length. Ask the assistant to continue, or rephrase your question to be more specific."

One question on this: can the user actually ask the assistant to continue the response? Or do we only invite them to start over with better instructions? If it's the second case, drop the "Ask the assistant to continue" part.

[maxenceh]

One question on this: can the user actually ask the assistant to continue the response? Or do we only invite them to start over with better instructions? If it's the second case, drop the "Ask the assistant to continue" part.

Thanks @robin-lecomte 🙏
I'd say we don't allow the user to continue, because we consider it's a misusage. @eliott07 what's your take on this?