idiot question

[tommasofavaron1]

Hi, is it possible to use the library having only the response, the query and the context without pass throght llm model? I need to use it with vLLM engine.
Thanks a lot and very nice work.

[bencw99]

Hi! The method requires additional inference passes with the model used to generate the response, so it isn't possible to use with just the query, context and response. While it's possible to apply our method using vLLM, we currently only support huggingface.

If the model you're using to generate cannot be run with huggingface, another option would be to provide a smaller model to our huggingface implementation that is used only to "cite" and to provide the original response generated using vLLM. To provide the original response, you'd write the following after constructing your ContextCiter:

response = ### Your response from vLLM
_, prompt = citer._get_prompt_ids(return_prompt=True)
citer._cache["output"] = prompt + response

Here, we're effectively pretending that the small model generated this response, and then asking it to cite its sources. We've found this to be a reasonably effective way to cite sources for large model for which we don't want to perform additional inference calls.