1) I think Talos should add
Support Maximal Update Parametrization as it could help get optimal HPs from small proxy models and apply them on large LLMs
More details, please refer to : https://github.com/microsoft/mup
2) Once implemented, I can see how this feature will
It will acclerate the LLMs development as optimal HPs could be found by small proxy models and apply them on large LLMS. And this will save a lot of training cost. So there are quite a lot benefits for the LLM community.
3) I believe this feature is
- [YES] critically important
- [YES] must have
- [YES] nice to have
Is there any plan for this?
1) I think Talos should add
Support Maximal Update Parametrization as it could help get optimal HPs from small proxy models and apply them on large LLMs
More details, please refer to : https://github.com/microsoft/mup
2) Once implemented, I can see how this feature will
It will acclerate the LLMs development as optimal HPs could be found by small proxy models and apply them on large LLMS. And this will save a lot of training cost. So there are quite a lot benefits for the LLM community.
3) I believe this feature is
Is there any plan for this?