SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

要約

タイトル：SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

要約：
– 自然言語処理のデータセットは、人間の判断による注釈の不一致が多い。
– 主観的な判断に依存するタスク、例えばセンチメント分析や攻撃的な言語の識別などでは、不一致が特に多い。
– 多くの自然言語処理の研究者は、不一致を排除するのではなく、保存することが重要だと結論づけている。
– LeWiDiというシリーズの共有タスクは、このアプローチを促進することを目的としており、統一されたフレームワークを提供する。
– 第2回LeWiDi共有タスクは、第1回目と異なり、NLPに特化しており、主観的なタスクに焦点を当てている。
– 評価においては、ソフトアプローチが集中的に行われる。
– 13件の共有タスク提出論文があった。

要約(オリジナル)

NLP datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the NLP community has come to realize that the approach of ‘reconciling’ these different subjective interpretations is inappropriate. Many NLP researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them-indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus creation for NLP has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing NLP models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements-as training with aggregated labels for subjective NLP tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.

arxiv情報

提供元, 利用サービス

arxiv.jp, OpenAI