-
Notifications
You must be signed in to change notification settings - Fork 0
HowDoesThisWork
Importantly, we're doing WSD, such that the sense inventory "is built up on the basis of the Europarl parallel corpus".
We have some English polysemous nouns, and we have their translations in five European languages: Dutch, French, German, Spanish, Italian.
So the source language here is English, and we want to predict translations in the five target languages. Right?
Specification for the shared task is here: Trial_Data_Task13.pdf
"Predict translation in one language" or alternatively do the multilingual evaluation task "in which the answer consists of translations in all five supported languages". (TODO(alexr): explain what this means? ...)
Also there are two types of scoring: "best" where you have to pick the 1-best, and "oof" ("out of five") where you pick the five best. In the latter case, the
See task description, section 2.2. But:
- there are multiword translations: one Dutch/German word may correspond with several English words.
- Also one English word might correspond with several Spanish words.
The data for 2013 task that has come out so far comes with two trials, Trial 1 and Trial 2. We have a bunch of files called WORD.data (where WORD is the word that we're trying to learn to disambiguate), each of which contains a bunch of instances, which are just sentences that provide context that we're supposed to use to disambiguate.
For example, the data might look like:
<corpus lang="english">
<lexelt item="test.n">
<instance id="1">
<context>A worthy father should pass the <head>tests</head> of strength and endurance, and so she rewards the efforts of those who are still fit enough to subdue and defend her at the end of the arduous rut.</context>
</instance>
</lexelt>
</corpus>
Note that the word in context here is actually tests, not test. So maybe morphological features will be helpful...
Disambiguation for just five nouns: bank, movement, occupation, passage, plant.
Disambiguation for: coach, education, execution, figure, job, letter, match, mission, mood, paper, post, pot, range rest, ring, scene, side, soil, strain, test.
For the one-best, it looks like this:
bank.n.fr 10 :: bord;
bank.n.fr 11 :: cisjordanie;rive;
(you can have a list of translation suggestions, separated by semicolons)
Note that this is different from (though similar to) the gold standard data format, where the words have "the number of times a translation has been chosen by the human annotators" with it.
Then for the out-of-five, you want to pick up to five choices: the target-language words have three colons in front of them, like so.
movement.n.it 3 ::: circolare;fluttuazione;mosso;movimento;movimiento;muoversi;
movement.n.it 4 ::: circolazione;libera circolazione;traffico;trasferimento;
ScorerTask3.pl is perhaps not the greatest piece of software ever constructed.
It works better (at all) if all of the files are in the same directory; alexr
has made an eval directory where all of the gold standard files can sit.
This trial run is what happens when you always guess "banco" for "bank", and evaluate on Spanish. Run it as follows:
alex@dan:~/semeval2013/src$ python3 run_experiment.py --sourceword=bank --targetlang=es
alex@dan:~/semeval2013/src$ cd ../eval/
alex@dan:~/semeval2013/eval$ perl ScorerTask3.pl bank.output bank.gold.es -v
alex@dan:~/semeval2013/eval$ cat bank.output.results
Scoring 'best' goldfile = bank.gold.es sysfile = bank.output
bank.n.es Item 1 mode 'banco' : system 'banco' correct
bank.n.es Item 1 credit 0.714285714285714 guesses 1 human responses 7: score is 0.714285714285714
bank.n.es Item 2 mode 'banco' : system 'banco' correct
bank.n.es Item 2 credit 0.571428571428571 guesses 1 human responses 7: score is 0.571428571428571
bank.n.es Item 3 mode 'cisjordania' : system 'banco' wrong
bank.n.es Item 3 credit 0 guesses 1 human responses 10: score is 0
bank.n.es Item 4 mode 'cisjordania' : system 'banco' wrong
bank.n.es Item 4 credit 0 guesses 1 human responses 10: score is 0
bank.n.es Item 5 mode 'borde' : system 'banco' wrong
bank.n.es Item 5 credit 0 guesses 1 human responses 11: score is 0
bank.n.es Item 6 credit 0 guesses 1 human responses 11: score is 0
bank.n.es Item 7 credit 0.5 guesses 1 human responses 8: score is 0.5
bank.n.es Item 8 credit 0.333333333333333 guesses 1 human responses 12: score is 0.333333333333333
bank.n.es Item 9 credit 0.5 guesses 1 human responses 8: score is 0.5
bank.n.es Item 10 mode 'borde' : system 'banco' wrong
bank.n.es Item 10 credit 0 guesses 1 human responses 11: score is 0
bank.n.es Item 11 mode 'cisjordania' : system 'banco' wrong
bank.n.es Item 11 credit 0 guesses 1 human responses 10: score is 0
bank.n.es Item 12 mode 'banco' : system 'banco' correct
bank.n.es Item 12 credit 0.714285714285714 guesses 1 human responses 7: score is 0.714285714285714
bank.n.es Item 13 credit 0.4 guesses 1 human responses 10: score is 0.4
bank.n.es Item 14 mode 'banco' : system 'banco' correct
bank.n.es Item 14 credit 1 guesses 1 human responses 4: score is 1
bank.n.es Item 15 mode 'banco' : system 'banco' correct
bank.n.es Item 15 credit 0.2 guesses 1 human responses 10: score is 0.2
bank.n.es Item 16 credit 0.111111111111111 guesses 1 human responses 9: score is 0.111111111111111
bank.n.es Item 17 credit 0.363636363636364 guesses 1 human responses 11: score is 0.363636363636364
bank.n.es Item 18 credit 0.5 guesses 1 human responses 8: score is 0.5
bank.n.es Item 19 mode 'borde' : system 'banco' wrong
bank.n.es Item 19 credit 0 guesses 1 human responses 11: score is 0
bank.n.es Item 20 credit 0.444444444444444 guesses 1 human responses 9: score is 0.444444444444444
Total = 20, attempted = 20
precision = 31.76, recall = 31.76
Total with mode 11 attempted 11
Mode precision = 45.45, Mode recall = 45.45
Precisions and recalls are out of 100.