Adapting ALMA source code to LLaMa 3.1

[aiyubx]

Hey there!

I am considering adapting ALMA to train LLaMa 3.1.

I was wondering if you have tried it and if the results were similar or better than with LLaMa 2.

Regarding the source code changes, on the first glance, only the eos and padding related parts seem to need some fixes. Any tips would be appreciated!

Thank you.

[zhl606]

Hey there!

I am considering adapting ALMA to train LLaMa 3.1.

I was wondering if you have tried it and if the results were similar or better than with LLaMa 2.

Regarding the source code changes, on the first glance, only the eos and padding related parts seem to need some fixes. Any tips would be appreciated!

Thank you.

Hello, I would like to use this training approach to fine-tune LLaMa 3 as well. I look forward to more discussions.

[pratik443]

Hello, were you guys able to try it?
@zhl606 @aiyubx