Self-discipline on multiple channels

要約

タイトル： Self-discipline on multiple channels
要約：
– 自己蒸留は自身の情報を利用してモデルの汎化能力を向上するため、将来性がある。
– 現存する自己蒸留の方法は、追加のモデル、モデルの変更、またはトレーニングのためのバッチサイズ拡張が必要で、使用の難しさ、メモリ消費量、および計算コストが増大する。
– この論文では、複数のチャンネルの概念を用いたSelf-discipline on multiple channels（SMC）を開発した。SMCは、一貫性正則化と自己蒸留を組み合わせてモデルの汎化能力およびノイズのあるラベルへの堅牢性を向上させる。
– SMC-2はSMCのうち2つのチャンネルを含むものであり、SMC-2は他の方法よりも優れた性能を示した。
– SMC-2とデータ拡張を使用すると、モデルの汎化能力が0.28％から1.80％向上することが実験結果から示された。
– ラベルノイズによる干渉の実験結果から、SMC-2がラベルノイズの干渉によりモデルの汎化能力が遅れる傾向を抑制することが示された。　
– ソースコードは以下のリンクから入手可能：https://github.com/JiuTiannn/SMC-Self-discipline-on-multiple-channels。

要約(オリジナル)

Self-distillation relies on its own information to improve the generalization ability of the model and has a bright future. Existing self-distillation methods either require additional models, model modification, or batch size expansion for training, which increases the difficulty of use, memory consumption, and computational cost. This paper developed Self-discipline on multiple channels(SMC), which combines consistency regularization with self-distillation using the concept of multiple channels. Conceptually, SMC consists of two steps: 1) each channel data is simultaneously passed through the model to obtain its corresponding soft label, and 2) the soft label saved in the previous step is read together with the soft label obtained from the current channel data through the model to calculate the loss function. SMC uses consistent regularization and self-distillation to improve the generalization ability of the model and the robustness of the model to noisy labels. We named the SMC containing only two channels as SMC-2. Comparative experimental results on both datasets show that SMC-2 outperforms Label Smoothing Regularizaion and Self-distillation From The Last Mini-batch on all models, and outperforms the state-of-the-art Sharpness-Aware Minimization method on 83% of the models.Compatibility of SMC-2 and data augmentation experimental results show that using both SMC-2 and data augmentation improves the generalization ability of the model between 0.28% and 1.80% compared to using only data augmentation. Ultimately, the results of the label noise interference experiments show that SMC-2 curbs the tendency that the model’s generalization ability decreases in the late training period due to the interference of label noise. The code is available at https://github.com/JiuTiannn/SMC-Self-discipline-on-multiple-channels.

arxiv情報

提供元, 利用サービス

arxiv.jp, OpenAI