local llm support?

[itsPreto]

any plans to support local backends like llama.cpp, text gen. ui, etc?

[arjunbansal]

👋 @itsPreto

There are a couple of ways to use local backends.

1. If the local LLM is wrapped in an openAI compatible endpoint then you can use the OpenAI Python lib and point the openai.api_base variable to localhost:port. This approach is described in the context of a self-hosted Llama model here
2. Alternatively, you could implement something more native following a similar pattern as our recipes & implementation for Langchain, MosaicML or Together

We'd welcome your contribution and happy to help if you decide on a native implementation.