Judgment of Learning: A Human Ability Beyond Generative Artificial Intelligence

要約

大規模な言語モデル（LLM）は、さまざまな言語ベースのタスクで人間の認知をますます模倣しています。
ただし、特にメモリパフォーマンスの予測におけるメタ認知の能力は未調査のままです。
ここでは、CHATGPTベースのLLMSが、個人が自分の将来のメモリパフォーマンスを予測するメタ認知尺度であるChatGPTベースのLLMが、人間の学習の判断（JOL）と一致するかどうかを評価するために、クロスエージェント予測モデルを紹介します。
人間とLLMを文章のペアでテストしましたが、その1つは庭のパス文です。これは、再分析を必要とする前に、最初は読者を誤った解釈に誤解させる文です。
コンテキストの適合を操作することで（フィッティングと不満な文章）、本質的なキュー（つまり、関連性）がLLMとヒトJOLの両方にどのように影響するかを調べました。
Our results revealed that while human JOL reliably predicted actual memory performance, none of the tested LLMs (GPT-3.5-turbo, GPT-4-turbo, and GPT-4o) demonstrated comparable predictive accuracy.
この矛盾は、文章がフィッティングに現れたのか、それとも不適切な文脈に現れたのかに関係なく出現しました。
これらの発見は、LLMSがオブジェクトレベルで人間の認知をモデル化する能力を実証したにもかかわらず、メタレベルで苦労し、個々のメモリ予測の変動性をキャプチャできないことを示しています。
By identifying this shortcoming, our study underscores the need for further refinements in LLMs’ self-monitoring abilities, which could enhance their utility in educational settings, personalized learning, and human-AI interactions.
Strengthening LLMs’ metacognitive performance may reduce the reliance on human oversight, paving the way for more autonomous and seamless integration of AI into tasks requiring deeper cognitive awareness.

要約(オリジナル)

Large language models (LLMs) increasingly mimic human cognition in various language-based tasks. However, their capacity for metacognition – particularly in predicting memory performance – remains unexplored. Here, we introduce a cross-agent prediction model to assess whether ChatGPT-based LLMs align with human judgments of learning (JOL), a metacognitive measure where individuals predict their own future memory performance. We tested humans and LLMs on pairs of sentences, one of which was a garden-path sentence – a sentence that initially misleads the reader toward an incorrect interpretation before requiring reanalysis. By manipulating contextual fit (fitting vs. unfitting sentences), we probed how intrinsic cues (i.e., relatedness) affect both LLM and human JOL. Our results revealed that while human JOL reliably predicted actual memory performance, none of the tested LLMs (GPT-3.5-turbo, GPT-4-turbo, and GPT-4o) demonstrated comparable predictive accuracy. This discrepancy emerged regardless of whether sentences appeared in fitting or unfitting contexts. These findings indicate that, despite LLMs’ demonstrated capacity to model human cognition at the object-level, they struggle at the meta-level, failing to capture the variability in individual memory predictions. By identifying this shortcoming, our study underscores the need for further refinements in LLMs’ self-monitoring abilities, which could enhance their utility in educational settings, personalized learning, and human-AI interactions. Strengthening LLMs’ metacognitive performance may reduce the reliance on human oversight, paving the way for more autonomous and seamless integration of AI into tasks requiring deeper cognitive awareness.

arxiv情報

提供元, 利用サービス

arxiv.jp, Google