Describe the bug
We should remove dependence on Hugot. The project is awesome, but it provides us more than what we need for DynaRAG. The tokenizer bindings by Daulet are still needed, and do a lot of the heavy lifting. Instead, write some simple C bindings to the onnxruntime.so object.
Describe the bug
We should remove dependence on Hugot. The project is awesome, but it provides us more than what we need for DynaRAG. The tokenizer bindings by Daulet are still needed, and do a lot of the heavy lifting. Instead, write some simple C bindings to the onnxruntime.so object.