BioCLEAR Benchmark
(1) Create a new environment
conda create --name my_env
conda activate my_env
(2) Get the evaluation package
README.md
evaluate_sentence_level.py
evaluate.py
sari.py
sources/abstracts.json
sources/source_sentences.json
references/abstracts.json
references/reference_sentences.json
runs/Test_task11_gpt4o.json
runs/Test_task12_gpt4o.json
(3) You may have to install the SARI implementation of EASSE
git clone https://github.com/feralvam/easse.git
cd easse
pip install -e .
(4) How to evaluate?
(a) Any run can be evaluated against doc-level references
python3 evaluate.py runs/Test_task11_gpt4o.json Cochrane
python3 evaluate.py runs/Test_task12_gpt4o.json Cochrane
(b) Sentence-level runs can be evaluated against sentence-level references.
python3 evaluate_sentence_level.py runs/Test_task11_gpt4o.json Cochrane
(c) The output is (only) corpus_sari as implemented in EASSE.
For Cochrane data, we evaluate against Cochrane auto-aligned sentences (a smaller set that includes only reference sentences that align well with the source).
For document-level evaluation, we report results against the original PLMs (first SARI score) and the Cochrane-auto-aligned references (second SARI score).
(d) We included various sources, used as the final option of the script.
- Cochrane: this is the new Cochrane-auto data of the CLEF 2025 SimpleText Track (test data).
- "Cochrane-auto val": this is the validation split of EMNLP/TSAR 2024 Cochrane-auto (train data).
- "Cochrane-auto test": this is the test split of EMNLP/TSAR 2024 Cochrane-auto (train data).
- Medline: These are Medline abstracts with three human sentence-level simplifications (TREC PLABA Track data). SARI scores against all three adaptations.
- SimpleText2024: These are the manually simplified Arminer abstracts from earlier years of CLEF SimpleText.