Multi-label classification of open-ended questions with BERT

要約

タイトル：BERTを用いたオープンエンドの複数ラベル分類

要約：
– サーベイでのオープンエンドの質問に答えることは、回答者の答えを制限せず、偏見を回避する点で価値がある。
– しかし、オープンエンドの回答は難解で解析が困難。
– この論文は、社会科学サーベイでのオープンエンドのテキスト回答の複数ラベル分類に焦点を当てている。
– 研究は、 transformer-based architectureであるBERTのドイツ語におけるパフォーマンスを、伝統的な複数ラベルアルゴリズムと比較し、最小の0/1損失率（13.1％）を報告。
– BERTを使用した分類には、少なくとも1つのラベルを強制することが推奨されるが、BERTは回答のわずか1.5％でラベルを予測しないため、0/1損失率を大幅に低下させることはない。
– この研究の重要な意義は、 1）BERTを使用したオープンエンドの複数ラベル分類がドイツ語でも機能することが示されたこと。 2）軽度の複数ラベル分類タスクでは、損失率が十分小さいため、完全自動分類が可能になったこと。 3）BERTを使用した複数ラベル分類には単一のモデルのみが必要で、競合アルゴリズム（ECC）は個々の単一ラベル予測を反復処理する必要がある。

要約(オリジナル)

Open-ended questions in surveys are valuable because they do not constrain the respondent’s answer, thereby avoiding biases. However, answers to open-ended questions are text data which are harder to analyze. Traditionally, answers were manually classified as specified in the coding manual. Most of the effort to automate coding has gone into the easier problem of single label prediction, where answers are classified into a single code. However, open-ends that require multi-label classification, i.e., that are assigned multiple codes, occur frequently. This paper focuses on multi-label classification of text answers to open-ended survey questions in social science surveys. We evaluate the performance of the transformer-based architecture BERT for the German language in comparison to traditional multi-label algorithms (Binary Relevance, Label Powerset, ECC) in a German social science survey, the GLES Panel (N=17,584, 55 labels). We find that classification with BERT (forcing at least one label) has the smallest 0/1 loss (13.1%) among methods considered (18.9%-21.6%). As expected, it is much easier to correctly predict answer texts that correspond to a single label (7.1% loss) than those that correspond to multiple labels ($\sim$50% loss). Because BERT predicts zero labels for only 1.5% of the answers, forcing at least one label, while recommended, ultimately does not lower the 0/1 loss by much. Our work has important implications for social scientists: 1) We have shown multi-label classification with BERT works in the German language for open-ends. 2) For mildly multi-label classification tasks, the loss now appears small enough to allow for fully automatic classification (as compared to semi-automatic approaches). 3) Multi-label classification with BERT requires only a single model. The leading competitor, ECC, iterates through individual single label predictions.

arxiv情報

提供元, 利用サービス

arxiv.jp, OpenAI