How to map human_annotated_data to raw subsets?

[yx-lu]

Thanks a lot for releasing this dataset.

I’m trying to use human_annotated_data as supervision labels and map it to the raw review text from the other subsets (ICLR2024, NIPS2023, F1000Journal, SemanticWebJournal), but I’m stuck on the join.

What I observed:

human_annotated_data rows contain only:
paper_id
submitted_at
metrics (keys like review__Overall_Quality, etc.)
The four raw subsets use different ID formats:
ICLR2024 / NIPS2023: OpenReview-style IDs (e.g., zzv4Bf50RW)
SemanticWebJournal: IDs like 3654-4868
F1000Journal: no obvious paper_id field aligned with human_annotated_data

Human_annotated_data.paper_id (e.g., small numeric strings like 42, 120) does not match IDs in those raw subsets.
Could you clarify the intended mapping?

Is there an official mapping file from human_annotated_data.paper_id to raw paper/review records?

[sadjadeb]

Hi @yx-lu ,

Thanks for your question and for your interest in the dataset.

The base file corresponding to the human annotations is data/processed/merged_200_papers.jsonl. We do not have human annotations for the full dataset; instead, annotations were collected for 200 papers randomly sampled across the different data sources. At the moment, this file is only available via our Google Drive. Sorry about the inconvenience. We will add it to Hugging Face as well for consistency.

A few clarifications on the structure:

• merged_200_papers.jsonl contains multiple rows per paper, where each row corresponds to a single review.
• data/human_annotation_data.jsonl contains one row per paper, with the aggregated human evaluation results.
• The paper_id fields in these two files are aligned with each other, but not with the IDs used in the raw subsets (ICLR2024, NIPS2023, F1000Journal, SemanticWebJournal).

To map reviews to annotations:

• Each review in merged_200_papers.jsonl has a reviewer identifier (for example, "reviewer": "Reviewer_EGJf" for "paper_id": 123).
• In human_annotation_data.jsonl, the corresponding evaluation can be found under the same paper_id (for example, "123") within the metrics field, using keys of the form review_Reviewer-EGJf_*.

In short, merged_200_papers.jsonl serves as the bridge between raw review text and the human annotations. There is currently no separate official mapping file beyond this.

Hope this helps. Let us know if anything is still unclear.