Learning Compositional Functions with Transformers from Easy-to-Hard Data

要約

トランスベースの言語モデルは、さまざまな複雑な推論タスクにわたって印象的な機能を実証しています。
トランスの表現力を調査する以前の理論的研究により、並列化可能な計算を含むマルチステップ推論タスクを効率的に実行できることが示されています。
ただし、そのような構造の学習性、特に勾配ベースの最適化を介して効率的な学習を可能にするデータ分布の条件は、未解決の問題のままです。
この質問への回答に向けて、この作業では、$ k $ fold構成タスクの学習性を研究します。これは、$ k $入力順列と$ k $の隠された順列のインターリーブ組成を計算する必要があり、$ o（\ log k）$レイヤーの変圧器で表現できます。
負の面では、$ k $ -foldコンポジションタスク分布のためにSQ Oracleに多項式的に多くのクエリのみを作成するSQ学習者は、$ K $でサンプルサイズを指数関数的に行う必要があることを示す統計クエリ（SQ）の下限を証明します。
On the other hand, we show that this function class can be efficiently learned, with runtime and sample complexity polynomial in $k$, by gradient descent on an $O(\log k)$-depth transformer via two different curriculum learning strategies: one in which data consists of $k’$-fold composition functions with $k’ \le k$ presented in increasing difficulty, and another in which all such data is presented simultaneously.
私たちの作品は、複雑な組成タスクを学習するためのトランスのデータ分布に簡単な例と困難な例の両方を持っていることの必要性と十分性に光を当てています。

要約(オリジナル)

Transformer-based language models have demonstrated impressive capabilities across a range of complex reasoning tasks. Prior theoretical work exploring the expressive power of transformers has shown that they can efficiently perform multi-step reasoning tasks involving parallelizable computations. However, the learnability of such constructions, particularly the conditions on the data distribution that enable efficient learning via gradient-based optimization, remains an open question. Towards answering this question, in this work we study the learnability of the $k$-fold composition task, which requires computing an interleaved composition of $k$ input permutations and $k$ hidden permutations, and can be expressed by a transformer with $O(\log k)$ layers. On the negative front, we prove a Statistical Query (SQ) lower bound showing that any SQ learner that makes only polynomially-many queries to an SQ oracle for the $k$-fold composition task distribution must have sample size exponential in $k$, thus establishing a statistical-computational gap. On the other hand, we show that this function class can be efficiently learned, with runtime and sample complexity polynomial in $k$, by gradient descent on an $O(\log k)$-depth transformer via two different curriculum learning strategies: one in which data consists of $k’$-fold composition functions with $k’ \le k$ presented in increasing difficulty, and another in which all such data is presented simultaneously. Our work sheds light on the necessity and sufficiency of having both easy and hard examples in the data distribution for transformers to learn complex compositional tasks.

arxiv情報

提供元, 利用サービス

arxiv.jp, Google