尊敬的作者/审稿人：
您好！我目前正在学习贵团队发表在 IJOC 上的论文 “Mitigating Age-Related Bias in Large Language Models”，十分钦佩您们在语言模型公平性方向的系统研究及对 BMIL 框架的创新设计。
在阅读过程中，我注意到文中提到为了验证专家标注的一致性，您们报告了 Cohen’s Kappa = 0.816，并表示这是“几乎完全一致”（almost perfect agreement）。由于经典的 Cohen’s Kappa 通常适用于两位评分者的一致性评估，我想请教以下几个问题以进一步理解您们的实现方式：
1.您们是否是对五位专家之间的所有两两 Cohen’s Kappa 进行计算并取平均？
2.还是实际使用的是适用于多评分者场景的 Fleiss’s Kappa，只是文中沿用了 “Cohen’s Kappa” 的表述？
3.如果可以的话，是否方便简要说明或分享一下一致性矩阵（Consistency Matrix）和实现中所用的 Python/R 计算方法？
由于我们团队也在开展类似方向的研究，理解这一指标的计算过程将对我们准确复现您们的评估体系大有帮助。非常感谢您们的宝贵工作，也期待您的指导与回复！
此致
敬礼！
刘洋
大连理工大学
Dear [Authors / Reviewers],
I’m currently studying your excellent paper “Mitigating Age-Related Bias in Large Language Models” published in IJOC, and I greatly appreciate your rigorous work on bias mitigation and the introduction of the BMIL framework.
While reviewing your methodology, I noticed that the paper reports a Cohen’s Kappa value of 0.816 to indicate near-perfect agreement among five human experts. However, as Cohen’s Kappa is traditionally designed for two raters, I am curious about the exact calculation method you employed in your study.
Specifically, I’d like to clarify:
-
Was this Kappa value computed by averaging all pairwise Cohen’s Kappa values across the five annotators?
-
Or was it based on a multi-rater agreement method such as Fleiss' Kappa, despite being labeled as “Cohen’s Kappa”?
-
If possible, could you briefly describe or share the consistency matrix or the corresponding Python/R function used in your implementation?
Understanding this point will greatly help us when referencing your evaluation protocol in our own work. Thank you again for your pioneering contribution to responsible AI research!
Sincerely,
Liu Yang
Dalian University of Technology
尊敬的作者/审稿人：
您好！我目前正在学习贵团队发表在 IJOC 上的论文 “Mitigating Age-Related Bias in Large Language Models”，十分钦佩您们在语言模型公平性方向的系统研究及对 BMIL 框架的创新设计。
在阅读过程中，我注意到文中提到为了验证专家标注的一致性，您们报告了 Cohen’s Kappa = 0.816，并表示这是“几乎完全一致”（almost perfect agreement）。由于经典的 Cohen’s Kappa 通常适用于两位评分者的一致性评估，我想请教以下几个问题以进一步理解您们的实现方式：
1.您们是否是对五位专家之间的所有两两 Cohen’s Kappa 进行计算并取平均？
2.还是实际使用的是适用于多评分者场景的 Fleiss’s Kappa，只是文中沿用了 “Cohen’s Kappa” 的表述？
3.如果可以的话，是否方便简要说明或分享一下一致性矩阵（Consistency Matrix）和实现中所用的 Python/R 计算方法？
由于我们团队也在开展类似方向的研究，理解这一指标的计算过程将对我们准确复现您们的评估体系大有帮助。非常感谢您们的宝贵工作，也期待您的指导与回复！
此致
敬礼！
刘洋
大连理工大学
Dear [Authors / Reviewers],
I’m currently studying your excellent paper “Mitigating Age-Related Bias in Large Language Models” published in IJOC, and I greatly appreciate your rigorous work on bias mitigation and the introduction of the BMIL framework.
While reviewing your methodology, I noticed that the paper reports a Cohen’s Kappa value of 0.816 to indicate near-perfect agreement among five human experts. However, as Cohen’s Kappa is traditionally designed for two raters, I am curious about the exact calculation method you employed in your study.
Specifically, I’d like to clarify:
Was this Kappa value computed by averaging all pairwise Cohen’s Kappa values across the five annotators?
Or was it based on a multi-rater agreement method such as Fleiss' Kappa, despite being labeled as “Cohen’s Kappa”?
If possible, could you briefly describe or share the consistency matrix or the corresponding Python/R function used in your implementation?
Understanding this point will greatly help us when referencing your evaluation protocol in our own work. Thank you again for your pioneering contribution to responsible AI research!
Sincerely,
Liu Yang
Dalian University of Technology