Feat/open cultural

[gunicsroland]

Summary

This PR introduces a new open-ended cultural evaluation task to HugMe.

Key Features

Open-Ended Question Format

The questions use 3 answer types:

• entity - short factual answers
• short_answer - brief sentence-length responses
• explanation - longer, more detailed answers

Evaluation Methodology

The evaluation uses a hybrid scoring approach depending on the answer type.

Entity answers

For entity questions, the dataset provides a list of accepted aliases. Evaluation is performed using:

Exact string matching
Alias matching
Substring checks against accepted answers

This enables robust evaluation of short factual responses while remaining deterministic.

Short Answer & Explanation Answers

For short_answers and explanation type answers it uses an LLM-as-judge to evaluate the responses.
The judge assigns one of four verdicts according to a task-specific rubric:

• Correct: if everything is good
• Partially correct: If there are some good parts
• Incorrect: If the answer is wrong
• Unceratin: If the judge can't decide or we output is malformed

Scoring and Reporting

It saves the score categorically and a commulative score like in other tasks, but it also saves a stat of the verdict cases:

{
   "cultural-open" {
         "category_scores": {
                       ....
                },
        "stat_summary": {
            "total": 51,
            "correct": 8,
            "partially_correct": 2,
            "incorrect": 17,
            "uncertain": 24
            }
        }

Handling Uncertain Cases

Responses that receive an Uncertain verdict are automatically exported to a separate JSON file. This enables:

• Manual review of ambiguous cases
• Quality control and validity of the LLM judge
• Re-scoring after rubric improvements

Result file modification

Furthermore the eval-result files in case of the cultural task now contain the question_id so later the multiple choice and open ended results can be cross evaluated.

[matyasosvath]

this file is not needed in the PR

[matyasosvath]

this fn can be replaced with: entry.get("gold_answer")

>>> d = {"a": 0}
>>> d.get("b")
>>> d.get("b") is None
True

[matyasosvath]

you can check these in one if condition