Create strategies for continuing amidst token limit issues

[rgbkrk]

Sometimes ChatLab is off to the races cranking on analysis and then it runs out of tokens.

We need to do two things:

• Count the number of tokens with tiktoken
• Remove earlier messages with a strategy. Allow users to pick from various strategies while we figure out a good default

The simplest strategy could be "remove the first message until under the token limit". Advanced strategies could include taking out the system message or even parts of it.

[nicosuave]

Thoughts on summarizing prior messages?

[DLFinney]

While i agree its the obvious default, removing starting with the first message doesnt seem like an assumptiom that would work well for most of my chats. Whilst i sometimes go off on tangents i try to keep to a contained task in each chat most of the time.

As a result the first message usually includes the setup of the whole task. i think if its removed then the chat may lose focus.

i would like chatGPT to include the option of deleting parts of the conversation. It sometimes isnt apparent for a few prompts that an answer wasnt helpful, and then i dont think one can get rid of it. anyway, i dont think you can control that.

being able to define important prompts and responses might help? then the default could say delete the oldest prompt and response (but leave the first one and any others that the user protects). and then user could have a way or protecting prompts and responses. im not sure if thats feasible?

[DLFinney]

maybe when the point is reached that old messages need removing, chstgpt asks you to go bacn through and thumbs up and thumbs down all the answers that have been helpful so far? and use that to prioritise what to remove.

[rgbkrk]

Thoughts on summarizing prior messages?

@nicosuave Now that's a cool idea. We could send (a portion of) the prior messages over to a model for summarization. Hopefully, it still has a context for any recent debugging and identifiers it needs. What's a good prompt we can get started with?

being able to define important prompts and responses might help? then the default could say delete the oldest prompt and response (but leave the first one and any others that the user protects). and then user could have a way or protecting prompts and responses. im not sure if thats feasible?

We could store that as metadata alongside the actual messages. We could have an interactive widget in the notebook to clean up the prior messages or tag them.

[DLFinney]

@nicosuave Now that's a cool idea. We could send (a portion of) the prior messages over to a model for summarization. Hopefully, it still has a context for any recent debugging and identifiers it needs. What's a good prompt we can get started with?

I just tried this two prompts on one of my longest and most successful coding chats with gpt.
"Please create a summary of this chat so far. The summary should include all the information you would need at the start of a new chat in order to continue assisting me."
and
"can you produce a highly concise psudocode that we can start another chat with. it should include any corrections or modification weve discussed, and you should be able to use it to broadly reproduce the python code here."
i havent checked or tested them but both outputs looked fairly good. i put them seperately into new chats, and then together (summary as one prompt, with a followup prompt of the pseudo code). the response to the combination looked, at a glance, to give a reasonable fresh starting point. the responses to the individual prompts felt like they each lacked some details and that more work would be needed to get chatgpt back up to speed.