Advice to reproduce for other languages / newer version of UD

[OrianeN]

Hello, I'm working on Occitan and it would be interesting to explore the evaluation of LLMs on an Occitan version of MultiBLIMP.

Unfortunately the Occitan treebank was only added officially to UD in v2.16, while your dataset was built from v2.15 😢.

Unless you are willing to build it in the next few days/weeks, I can try to use your scripts and the pipeline description from your paper. Any advice (small or large) you might have is welcome (order and usage of scripts, format of files, where to do manual spot checks...) !

[jumelet]

Hi Oriane, thanks for reaching out!

Let me discuss this asap with my collaborators, we are actually planning on releasing an updated version of v1 sometime soon and it is good to hear people would benefit from this. Otherwise I'd be glad to help you out running it yourself.

[jshrdt]

Hi Oriane,

Great to hear that you are working on Occitan!
While we are working on the update, I can share my recent notes from setting up to work on MultiBLiMP with you (see txt file).

multiblimp-setup.txt

[OrianeN]

Thank you very much !!