UAlberta at SemEval-2026 Task 5: Disambiguating Stories via Task Decomposition

This repository contains our system for SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding. Given a short narrative containing an ambiguous word (a homonym) with two candidate senses, the task is to predict, on a 1–5 scale, how plausible each sense is in context. Our approach centers on task decomposition (TD): rather than predicting a score directly, we break the problem into simpler subtasks and combine their outputs, then ensemble complementary signals from word sense disambiguation, fine-tuned embeddings, and large language models.

🏆 Best Paper Award at SemEval-2026
🥈 2nd Place on the Official Leaderboard

---

Overview

The input story is processed in parallel by several components — task decomposition, direct LLM prompting, word sense disambiguation, a fine-tuned story-ending model, and a translation-based method (one homonym per translation). Their outputs are combined with a ridge regression ensemble to produce the final plausibility score. Task decomposition is the strongest individual component, accounting for more than half of the ensemble's contribution.

---

Repository Structure

.
├── TaskDecomposition/   # TD: binary-decision prompts + regressor
├── DirectPrompting/     # Direct LLM prompting variants (e.g., Qwen)
├── WSD/                 # ConSec continuous WSD interface
├── StoryEnding/         # Fine-tuned DeBERTa story-ending model
├── OHPT/                # One Homonym Per Translation
├── Ensemble/            # Ridge regression ensemble + scaling
├── data/                # Train/dev/test data and dev-set gold labels
├── predictions/         # Per-system predictions on dev/ and test/, consumed by Ensemble/
├── eval/                # Shared scoring and format-check utilities
└── assets/              # Paper, poster, slides, and award certificate

Each component directory (TaskDecomposition/, DirectPrompting/, WSD/, StoryEnding/, OHPT/, Ensemble/) has its own README and requirements.txt, since components depend on different, sometimes incompatible, package versions (e.g. WSD/ requires an older, ConSec-specific environment). Install a component's dependencies from within its own directory rather than at the repository root.

🚀 Quick Start & Replication

To replicate our system end to end:

1. Generate Component Outputs: Follow the instructions in TaskDecomposition/, DirectPrompting/, WSD/, StoryEnding/, and OHPT/ to generate each system's predictions and drop them into predictions/dev/ and predictions/test/.
2. Run the Ensemble: Once all component predictions are in place, follow Ensemble/README.md to run the ridge regression ensemble and produce the final plausibility scores, verifiable against data/dev_solution.jsonl via eval/scoring.py.

---

Citation

@inproceedings{basil-etal-2026-ualberta,
    title = "{UA}lberta at {S}em{E}val-2026 Task 5: Disambiguating Stories via Task Decomposition",
    author = "Basil, David  and
      Cho, Junhyeon  and
      Girigowda, Chirooth  and
      Luo, Guoqing  and
      Momin, Sahir  and
      Robinson, Sevryn  and
      Shi, Ning  and
      Kondrak, Grzegorz",
    booktitle = "Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026)",
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
}